René Pickhardt

Create a Screencast in Ubuntu with recordmydesktop and do Soundengineering and post production

René Pickhardt — Fri, 17 May 2013 18:36:55 +0000

I promised to create some screen casts for the autocomplete service with GWT based on neo4j. After I have created all the screencasts I had to go through quite some hassle in order to do so.

So let me share my experience and toolchain in order to produce a somewhat accaptable screen cast. I am still not happy with the quality of the results and if you have any suggestions feel free to tell me.

First of all you download and install recordmydesktop. Once this is done you can just do a screencast by calling the following line.

recordmydesktop --quick-subsampling --full-shots --no-shared --v_bitrate 2000000 --on-the-fly-encoding

bear in mind that you should have at least a dual core processor if you really want to do video encoding on the fly.

After creating this screencast I realized that my audio and video tracks where not in sync. Even worse. I realized that the video track was always a few seconds shorter than the audio track. At the end of the video when the audio wasn’t finnished yet the last video frame would just be displayed. This might be alright on a 20 second screencast but some of my screencasts where up to 14 minutes and that really sucked! because speaking and video would just not be synchrone anymore.
In another screen cast that I did today I had the problems with the sound not being loud enough so Here is the stack of software that works fine with corrupt .ogv files produced by recordmydesktop.

After searching quite a while and trying several programms (pitivi, Avidemux,oggtools(!),…) that all crashed on the .ogv file that came out of recordmydesktop I reallized that with ffmpeg2theora I could actually extract the audio and video files seperately. to do so just enter:

ffmpeg2theora video.ogv --noaudio
ffmpeg2theora video.ogv --novideo

The problem was the Audiofile. I was able to play it with my Audioplayer but again not able to open it with any editing software but oggconvert!
With this little tool I was able to open the corrupt .oga audio file and save it to an .ogg file (with vorbis audio codec!).

Now after I had a non corrupted audiofile I was able to adjusting the speed of the audiofile adjusting it to the length of the recoreded video with audacity. Easy instructions how to do this are available at http://wiki.audacityteam.org/wiki/Change_Speed Audacity also provided the options for me to amplify the volume of the sound.

Finally I had two files of the same length just need to merge them again. The oggTools again failed to help me on this but there is pitivi which is able to do the job.

So you see if one keeps patient one will reach his goal. Some of the screen casts are already published and more will come. So far I hope you enjoyed reading my experiences and solutions

by the way on wikipedia there are quite some tutorials on how to edit ogg videos.
http://commons.wikimedia.org/wiki/Help:Converting_video

Building an Autocompletion on GWT screencast Part 1: Getting Warm – Reviewing remote procedure calls Quite a while ago I promised to create some screencasts...
How to create animations between two vector graphics? I am just about to prepare a talk about my...
Building an Autocompletion on GWT screencast Part 2: Invoking The Remote Procedure Call Hey everyone after posting my first screencast in this series...

The best way to create an autocomplete service: And the winner is…. Giuseppe Ottaviano

René Pickhardt — Wed, 15 May 2013 13:06:42 +0000

Over one year ago I was starting to think about indexing scored stings for auto completion queries. I stumbled upon this problem after seeing the strength of the predictions of the typology approach for next word prediction on smartphones. The typology approach had one major drawback: Though its suggestions had a high precision the speed with 50 milliseconds per suggestion was rather slow especially when working on a server side application.

On August 16 th 2012 I found a first solution building on Nicolai Diethelms Suggest Tree. Though the speedup was great the suggest tree at this time had several major drawbacks (1. the amount of suggestions had to be known before building the tree 2.) large memory overhead and high redundancy 3.) no possibility of updating weights or even inserting new strings after building the tree) (the last 2 issues have been fixed just last month)
So I tried to find a solution which required less redundancy. But still for indexing Gigabytes of 5-grams we needed a persistent method. We tried Lucene and MySql in December and January. After seeing that MySQL does not provide any indices for this kind of query I decided to misuse multidimensional trees of MySQL in a highly redundant way to somehow be able to evaluate the strength of typology on large data sets with gigabytes of n-grams. Creating one of the dirtiest hacks in my life I could at least handle the data but the solution was rather engineered and consisted of throwing hardware at the problem.
After Christoph tried to solve this using bitmap indices which was quite fast but had issues with scaling and index maintainability we had a discussion and finally the solution popped in my mind in the beginning of march this year.

Even though I was thinking of scored tries before they always lacked the problem that they could only find the top-1 element efficiently. Then I realized that one had to sort the children of a node by score and use a priority queue during retrieval. In this way one would get the maximum possible runtime I was doing this in a rather redundant way because I was aiming for fast prefix retrieval of the trie node and then fast retrieval of the top children.

After I came up with my solution and after talking to Lucene contributers from IBM in Haifa I realized that Lucene had a pretty similar solution as a less popular “hidden feature” which I tested. Anyway in my experiment I also experienced a large memory overhead with the Lucene solution so my friend Heinrich and me started to develop my trie based solution and benchmark it with various baselines in order to produce a good solid output.

The developement started last month and we had quite some progress. Our goal was always to be about as fast as Nicolai Diethelms suggest tree but not running into all the drawbacks of his solution. In our coding session yesterday we realized that Nicolai improved his data structure a lot by getting rid of his memory overhead and also being able to update, insert and delete new items to his index (still the amount of suggestions has to be known before the tree was build)

Yet while learning more about the ternary tree data structure he used to build up his solution I found a paper that will be presented TODAY at WWW conference. Guess what: Independently of us Giuseppe Ottaviano explains in Chapter 4 the exact solution and algorithm that I came up with this march. Combined with an efficient implementation of the tries and many compression techniques (even respecting cache locality of the processor) he even beats Nicolai Diethelms suggest tree.

I looked up Giuseppe Ottaviano and the only thing two things I have to say are:

Congratulations Giuseppe. You really worked on that kind of problems for a long time an created an amazing paper. This is also reflected by the related work section and all the small details that are in your paper which we were still in the process of figuring out.
If anyone needs an auto completion service this is the way to go. Being able to provide suggestions from a dictionary with 10 Mio. entries in a few micro seconds (yes micro not milli!) means that a single computer can handle about 100’000 requests per second which is certainly web scale. Even the updated suggest tree by Nicolai is now the way to go and maybe much easier to use since it is java based and not C++ and the full code is open sourced.

Ok so much for the history of events and the congratulations to Giuseppe. I am happy to see that the algorithm really performs that well but there is one little thing that really bothers me a lot:

How come our community of researchers hasn’t come up with a good way of sharing credits to a person like me who came up independently with the solution? As for me I feel that the strongest chapter of my dissertation just collapsed and one year of research just burnt away. I mean personally I gained and learnt a lot from it but from a carrier point of view this seems rather like a huge drawback.

Anyway life goes on and by thinking about the trie based solution we already came up with a decent list of future work which we can most certainly use for follow up work and I will certainly contact the authors maybe a collaboration in future will be possible.

No related posts.

Please help me to realize my Web science massive open online course

René Pickhardt — Wed, 01 May 2013 09:59:57 +0000

I am asking you for a big favor in this blog post! You can help me to achieve one of my childhood dreams:

I am an enthusiastic teacher and love to share information (as you might have seen by reading my blog) Over the last month I have designed a structure for an online course on Web Science together with a short video. In this blog post I will introduce the course to you but I am also asking you to vote for the course since only 10 of the 250 courses that applied for the fellowship will be sponsored and thus be realized.

So please go to https://moocfellowship.org/submissions/web-science an learn more about the course and vote for it. You can find almost all details of the course in this blog post.

Why creating such a cours?

The web has become important to its 2.3 billion users. Yet only a small group of people understand the processes that take place on it and quickly steer its development into new directions.

Novelty of the subject

Web Science is an upcoming academic field. Much information about the web already exists online, but no course that comprises all of it.

High value for every web user

The MOOC would be of high value and of relevance for anybody using the web e.g:

A programmer who is building the next web application
A company deciding their web strategy
A judge who has to decide a case regarding net neutrality or copy right infringements
The Government as well as public authorities which have to make decisions on how to regulate the web
…

The web is the right place to learn about the web

The web itself is the best platform to educate people about the web since you can always point directly to the object of study. By creating a MOOC we will be able to aggregate, organize and filter much of the available information.

Integration within our institution

The MOOC will be a core element for the web science lecture of our web science master program. The goal is that students will work with the material provided by the MOOC and the instructors will replace classical lectures with public Q&A sessions. Additionally the Web Science lecture of 2013/2014 will serve as an internal testing of the MOOC such that the improved MOOC can launch on iversity in 2014.

Course content

This MOOC consists of ten lessons divided into three parts.

Lesson 1 – 3: Foundations of the web
Lesson 4 – 7: Theoretical results of web user behavior
Lesson 8 – 10: Web & society

Lesson 1 & 2: History of the Web & Web Architecture

You will understand the historical development of the web and see how the cold war in combination with advances in technical developments led to the Internet Protocol suite.

On each Layer you will know one protocol and understand how these protocols build an open, inter operable and decentralized system. Furthermore you will learn about the domain name system and find out why the concepts of URI and Hypertext were crucial for the success of the web.

Lesson 3: Structure of the Web

You will learn about the six degrees of seperation and understand concepts like small world networks by studying ‘the other’ Milgram experiment. You will be able to use power law distributions to describe the structure of the web, its content and its users.

Lesson 4 & 5: Micro and Macro behavior of web users & Social Network (Analysis)

You will be introduced to theories from Microsociology and see how applying them to the behavior of people on the web leads to macro structures such as:

Analyzing social network data from the Koblenz Network Collection using Octave you will gain a deeper understanding of social theories and social networks.

Lesson 6 & 7: Information Retrieval & Recommender systems

Completing this section you will understand the basic architecture of a (web) search engine. You can name the fundamental (non technical) difficulties one has in order to create a good information retrieval system. You will learn about the connection to recommender systems that are (not only!) used by large web shops to increase cross selling.
You will be able to discuss the danger of such algorithms like the relevance paradox and the filter bubble.

Lesson 8: Trust and Security

You will learn how third parties act as trust providers on the web and how this issue is related to markets with asymmetric information. You will see that trust issues in the online word differ from the offline problems. You will know of ways like cryptography, secure communication and certificates to resolve trust issues and how those techniques can even lead to a new currency.

Lesson 9: Web Economics

You will know of e-commerce models like online shopping & auctions as well as online advertising and marketing. You will be able to interpret and apply metrics for web analytics such as

Lesson 10: Web Governance and Web Ethics

Finally you will understand the important role of institutions like W3C, IETF and ICANN . You will use your understanding of the web architecture to discuss and explain the connections between

Net neutrality
Piracy and copy right infringement
Internet censorship and the freedom of speach

So please go to https://moocfellowship.org/submissions/web-science an learn more about the course and vote for it.

Web science education and web science mooc The slides of my talk in the web science education...
Teaching Web Science (web architecture and Web ethics) to students In July 2012 we taught a course for the German...
Open access and data from my research. Old resources for various topics finally online. Being strong pro on the topic of open access I...

Web science education and web science mooc

René Pickhardt — Wed, 01 May 2013 08:38:14 +0000

The slides of my talk in the web science education workshop can be found here. The talk was about two things:

Our application for the web science mooc (<-- vote now!) which I already blogged about this morning
Teaching web architecture and web ethics to high school students

Please help me to realize my Web science massive open online course I am asking you for a big favor in this...
Internet educates africa! Open education – a success story! Obviously education is one of the topics that is important...
Teaching Web Science (web architecture and Web ethics) to students In July 2012 we taught a course for the German...

Teaching Web Science (web architecture and Web ethics) to students

René Pickhardt — Tue, 26 Mar 2013 13:52:51 +0000

In July 2012 we taught a course for the German National Summer School for high school students. The course consisted of 50 hours over 14 days. Due to some specific settings of the Summer School we had to make a few adjustments to the format of our curriculum and lectures. Still we gathered some good experiences for future teaching. The main lesson learnt was that knowledge of the Internet protocol suite contributes to a better understanding of the decentralized and open aspects of the web. This leads to a better comprehension of the ethical aspects of the web like net neutrality, copyright, relevance paradox, censorship and others. We propose that any curriculum about Web Science should include a fair part of lectures on Web Architecture and the Internet protocol Stack.

Course context (level, students, discipline, etc.)

The course was designed for 16 highly gifted high school students (11th and 12th grade). The level was supposed to be manageable for a second year undergraduate student. Since our students came from different grades and schools we were forced to sacrifice some course time to teach some basic programming skills. Thus we could not cover all the aspects of Web Science. Instead we focused on three main course objectives:

Course objectives and targeted competencies

By the end of the course our students should…

understand the current web architecture in particular the decentralized and open aspects.
gain the ability to form and defend a solid opinion on currently ongoing ethical discussions related to the Web.
realize that the study of the Web needs much broader skill set than knowledge about Computer Science.

Course content (Structure, sections, topics, references)

All students were asked to prepare a talk and read the book ”Weaving the Web” by Sir Tim Berners-Lee before the summer school started. Ten of the talks included the technical foundations starting with binary numbers going all the way to the application layer and all the necessary protocols. This included the theoretical study of IP, TCP and HTTP as well as routing algorithms (BGP ) and DNS. To ensure a better understanding the students had to form groups and implement a simple Web Server and a Web Client that were able to process HTTP1.0 GET requests during course time. This was done using the Java Programming Language and the socket classes from the Java API. These topics have been covered in the first week of the course. In the second half we focused on the ethics of the web. After each talk on an ethical topic which was supposed to give an overview for about 20 minutes we entered a 2 hour group discussion. For example for the discussion on net neutrality we knew the following groups of interests from the overview talk: Large internet providers, big web companies, small web companies, politicians, consumers. Students were randomly assigned to one of these groups. Within 10 minutes they had to prepare a list of arguments that would reflect the interests of their particular group as well as arguments they would expect from other groups. While discussing the issue on a round table they had to find a good solution respecting the technical nature of the web and the interests of their group.

Evaluation methods (Tests, projects, papers,etc.)

Even though the Summer School is very competitive participation is voluntary so there can’t be an exam or something similar in the end. Also all work had to be completed during the 50 hours course time without any home work assignments. We had three evaluation methods to ensure the comprehension of the course content.

1. Hacking Project: As already mentioned students implemented a Web Server and Web Client during the first half of the course. Being in groups of 2 or 3 students and being new to programming we teachers helped students out which gave us a nice feedback whether or not students understood the content.

2. Oral presentation: After the middle of the course students had to prepare and give a presentation to be consumed by an interdisciplinary audience i.e the students from other courses of the summer school, which are all not covering any IT topics. We asked the students to create a theatre role-play of what happens if someone types www.wikipedia.org into a web browser and hits the enter key. All students placed routing tables on the seats for the audience, created TCP / IP packets (filled with candy that represented the time to live) and routed DNS requests as well as HTTP requests together with the TCP handshake around the audience in the class room demonstrating that the basic decentralized web architecture was understood by everyone in the course.

3. Paper Writing: During the last days of the course the students were expected to collectively prepare a 25 pages documentation with scientific standards of what they have learned during the summer school. The process of creating this documentation is not only guided by us teachers but gives also a nice feedback loop to see if the goals of the course have been achieved.

Overall we can say that the concept of the course worked really well. Especially putting such a high focus on the Web Architecture and actually letting students implement protocols helped to gain a deeper understanding.

Web science education and web science mooc The slides of my talk in the web science education...
Please help me to realize my Web science massive open online course I am asking you for a big favor in this...
My Students Bachlor thesis I advised Martin Körner – Thesis on Generalized...

Organization of the Open Access event 2013 in Oxford.

René Pickhardt — Tue, 19 Mar 2013 19:08:07 +0000

During the past two months I invested quite some of my spare free time to contribute to the organization of the open access event Rigor and Openness in 21st century science which will take place in the University of Oxford on April 11th and 12th.

The Idea of the conference came up during Heinrich’s time in Oxford where he and me have been working a lot on related work and he was working closely together with people from the Akorn project (c.f. Heinrichs blog article why openess benefits research)

Thanks to a great effort of the organizing team (mostly students from Oxford + Heinrich and me) we can finally publicly announce the conference.

As you can see from the web page

http://www.rigourandopenness.org/

we have been able to attract quite some famous speakers to Oxford for the various sessions, keynotes and the debate. I am particularly proud that we could even get hold of Amelia Andersdotter MEP and member of the Swedish pirate party for the public debate together with people from publishers…

Please help us to spread the word of the conference. Even if you cannot come to Oxford or live abroad some of your friends might be close and might want to attend.

The topic of open access is important since we have to preserve or knowledge (c.f. http://en.wikipedia.org/wiki/The_Cost_of_Knowledge)

Open Access and the Boycott of Elsevier! Let uns not stop here and take the digital revolution one step further! Believing in open models and supporting ideas of copy left...
Big step towards open access by Great Britain and a comment from Neelie Kroes During my vaccation a lot of stuff has been happened...
Open access and data from my research. Old resources for various topics finally online. Being strong pro on the topic of open access I...

Building an Autocompletion on GWT screencast Part 2: Invoking The Remote Procedure Call

René Pickhardt — Tue, 12 Mar 2013 07:25:00 +0000

Hey everyone after posting my first screencast in this series reviewing the basic process for creating remote procedure calls in GWT we are now finally starting with the real tutorial for building an autocomplete service.

This tutorial (again hosted on wikipedia) covers the basic user interface meaning

how to integreate a SuggestBox instead of a textfield into the GWT Starter project
how to set up the neccessary stuff (extending a SuggestOracle) to fire a remote procedure call that requests suggestions if the user has typed something.
how to override the necessary methods from the SuggestOracle Interface

So here we go with the second part of the screencast which you can of course directly download from wikipedia:

Feel free to ask questions, give comments and improve the screencast!

Building an Autocompletion on GWT screencast Part 1: Getting Warm – Reviewing remote procedure calls Quite a while ago I promised to create some screencasts...
Building an Autocompletion on GWT with RPC, ContextListener and a Suggest Tree: Part 0 Over the last weeks there was quite some quality programming...
Create a Screencast in Ubuntu with recordmydesktop and do Soundengineering and post production I promised to create some screen casts for the autocomplete...

Building an Autocompletion on GWT screencast Part 1: Getting Warm – Reviewing remote procedure calls

René Pickhardt — Tue, 19 Feb 2013 09:11:29 +0000

Quite a while ago I promised to create some screencasts on how to build a (personalized) Autocompletion in GWT. Even though the screencasts have been created for quite some time now I had to wait publishing them for various reasons.

Finally it is now the time to go public with the first video. I do really start from scratch. So the first video might be a little bit boaring since I am only reviewing the Remote Procedure calls of GWT.

A litte Note: The video is hosted on Wikipedia! I think it is important to spread knowledge under a creative commons licence and the youtubes, vimeos,… of this world are rather trying to do a vendor lock in. So If the embedded player is not so well you can go directly to wikipedia for a fullscreen version or direct download of the video.

Another note: I did not publish the source code! This has a pretty simple reason (and yes you can call me crazy): If you really want to learn something, copying and pasting code doesn’t help you to get the full understanding. Doing it step by step e.g. watching the screencasts and reproducing the steps is the way to go.

As always I am open to suggestions and feedback but please have in mind that the entire course of videos is already recorded.

Building an Autocompletion on GWT screencast Part 2: Invoking The Remote Procedure Call Hey everyone after posting my first screencast in this series...
Building an Autocompletion on GWT with RPC, ContextListener and a Suggest Tree: Part 0 Over the last weeks there was quite some quality programming...
Create a Screencast in Ubuntu with recordmydesktop and do Soundengineering and post production I promised to create some screen casts for the autocomplete...

Slides of Related work application presented in the Graphdevroom at FOSDEM

René Pickhardt — Sat, 02 Feb 2013 15:13:02 +0000

Download the slidedeck of our talk at fosdem 2013 including all the resources that we pointed to.

Most important other links are:

was great talking here and again we are open source, open data and so on. So if you have suggestions or want to contribute feel free. Just do it or contact us. We are really looking forward to meet some other hackers that just want to go geek and change the world

Related-work.net – Product Requirement Document released! Recently I visited my friend Heinrich Hartmann in Oxford. We...
Related work of the Reading club on distributed graph data bases (Beehive, Scalable SPARQL Querying of Large RDF Graphs, memcached) Today we finally had our reading club and discussed several...
Michael Hunger talks about High Availability of Neo4j built on Paxos in the GraphDevroom @ FOSDEM As we know neo4j has a master slave replication with...