René Pickhardt

Birds of a feather: Graph processing future trends in Graph Devroom

Rene — Sun, 05 Feb 2012 11:00:56 +0000

Since one of the talks got canceled the organisers of the Graph Devroom at Fosdem used the opportunity to make a public discussions with all the developers to talk about some future trends in graph processing. I really liked the idea but unfortunately the discussion wasn’t really kicking off well. I guess for a discussion like this people have to prepared in a better way.

Topics were Blueprints (a common graph accass api) created by Marko Rodriguez as
Problem of real time graph processing
Benchmarking issues (we need standards for benchmarking).
A guy from OrientDB raised the question weather Graph databases should really have ACID transaction?
Max De Marzi raised the question about are graphs changing while processing or are they rather static?
Achim Pointed out that Relational databases are actually a special case of Graph databases. He demands vendors to generalize more and consolidate the technologies…

The room was not as full as the talks before but sill half of the seats have been filled as you can see on this short video:

My thoughts on acid discussion

I think that the ACID question was interesting. Alistair from neo4j gave a fine response to this saying that it clearly depends on the usecase and the kind of transactions that you really need. He compared to the relational data base world where you might have the option of switching ACID off.

He says that “in neo4j you cannot shut off ACID as neo4j believes that for most of their customers this is the best choice. But he admits that there are use cases where you migh want to shutoff ACID.

Changing graphs vs static graphs

I think that this is also a very important question. On one hand we have static models like Giraph that are able to find answers on huge static graphs on the other side you have situations like graphity where you have fast fluctuation in data. Unfortunately for the later there is no technology I know of (besides hidden Facebook and Twitter and Googleplus) .

Nils Grunwald from Linkfluence talks at FOSDEM about Cascalog for graph processing

Rene — Sun, 05 Feb 2012 10:10:03 +0000

Nils Grunwald works at the french startup Linkefluence. Their product is more or less social network analysis and graph processing. They crawl the web and blogs or get other social network data and provide solutions with statistics and insights for their customers.

In this scenario obviously big data is envolved and the data carries a natural structure of a graph. He sais a system to process the data has the following constrains:

The processing should not compromise the rest of the system
Low maintenance costs
Used for queries and rapid prototyping (so they want a “general” graph processing solution as customer needs changes)
Flexible, hard to tell which field or metadata will be used beforehand.

He afterwards introduces their solution Cascalog based on Hadoop and is also inspired by cascading a workflow managment system and datalog a subset of prolog which as a declarative, expressive language is very concise way of writing queries and enable quick prototyping

For me personally it is not a very interesting solution since it is not able to answer queries in realtime which of course is obvious if you consider the technologies it is based on. But I quess for people that have time and just do analysis this solution will properly work pretty well!

What I really liked about his the solution is that after processing the graph you can export the data to Gephi or to Neo4j to have fast query processing.

Hey then explained alot specific details about the syntax of cascalog:

nils grundwald from linkfluence talks about cascalog at fosdem

Claudio Martella talks @ FOSDEM about Apache Giraph: Distributed Graph Processing in the Cloud

Rene — Sun, 05 Feb 2012 09:01:45 +0000

Claudio Martella introduces Apache Giraph which according to him is a loose implementation of Google Pregel which was introduced on SIGMOD in 2010. He points out that Map Reduce cannot be used to do graph processing.

He then gave an example on how MapReduce can be used to to do page rank calculation. He points out that Pagerank can be calculated as a local property of a graph in a distributed way by calculating local pagerank from the knowledge of the neighbours. He did this to show what the Drawbacks of this method are in his oppinion:

job boostrap take some time
disk is hit about 6 times
Data is sorted
Graph is passed through

Like in the Pregel Paper he says that other Graphalgorithms like singlesource shortest paths have the same problems.

: Claudio Martella from Apache explains how giraph works at in the graph dev room @ Fosdem 2012

After introducing more about implementing Pregle ontop of the existing MapReduce structure for distributing he says that this system has some advantages over MapReduce

it’s a stateful computation
Disk is hit if/only for checkpoints
No sorting is necessary
Only messages hit the network

He points out that the advantages of Giraph over other methods (Hama, GoldenOrb, Signal/Collect) are especially an active community (Facebook, Yahoo, Linkedin, Twitter) behind this project. I personally think another advantage is that it is run by Apache who already run MapReduce (Hadoop) with great success. So it is something that people trust…

Claudio points out explicitly that they are searching for more contributors and I think this is really an interesting topic to work on! So thank Claudio for your inspiring work!

Attending Mozilla Devroom at FOSDEM: Hacking Gecko by Bobby Holley

Rene — Sat, 04 Feb 2012 14:04:24 +0000

Since I went to Brussesl to attend the Free Open Source Developer European Meeting and give a talk about graphity I had also some time to attend some interesting talks. Since I will teach a class in summer for students about creating a basic webserver and webbrowser I decided to listen to some talks of the mozilla devrooom.

I first attended a very refreshing and fun talk given by Bobby Holley a platform engineer at mozilla about hacking the Rendering Engine Gecko. I think I realized some things that are not only of interest for hacking gecko but also for hacking metalcon or some other software!

you can find his slides on http://people.mozilla.com/~bholley/hacking-gecko-fosdem2012/hacking-gecko.html (when you go there just use the arrow keys to navigate!)

Some messages I found particular interesting together with my thoughts and comments:

Making Mozilla is great!

So this is actually why Mozilla thinks making Mozilla is great and what their motivation is. They think firefox enables them to change the way the web is developed. If they don’t like something they won’t implement it. So you clearly see one asset of Firefox is being in the gate keeper position. And being there is valuable in its own. That is probably one of the reasons why Google put so much effort on chrome and android. Both products go very deep and make google a gatekeeper giving them tons of power to steer developments in the “right” direction.

Look at the feature!

Bobby pointed out that if you want to contribute to Mozilla are Gecko you should not even consider to follow a top down approach. He said that it is impossible to understand 7 mio. lines of (fast changing) code. He rather suggests to look at the specific feature you want to develope and look at the documentation. You should play around and get it running. In my own experience that is on of the things that makes the difference betweed some coder and an amazing hacker. If you are really able to hack your things in an entire system without overview of the entire thing then your “in business”

Tests are awesome!

Ok that was the most interesting take away. Bobby said that tests are important not only so you can see that your code works properly and that you can check that your code doesn’t interfere with others. But tests are also great so if people change their parts they can check your feature will still work with their. He said your tools won’t continue working for longer than a couple weeks unless you provide tests so others see their plugins don’t destroy your work… Metalcon was only 40’000 lines of code and pretty manageable but we already saw weired things getting messed up once we checked in some code at some other point. To quote Bobby:

If you check in some new lines you can smoke test them but if you check them in and 200 test lamps go green it feels good. More importantly. A huge community will make new stuff so if you didn’t write a test within a month your new feature might not work…

Open Access and the Boycott of Elsevier! Let uns not stop here and take the digital revolution one step further!

Rene — Sat, 28 Jan 2012 13:48:22 +0000

Believing in open models and supporting ideas of copy left I am more than happy to see the current developement of scientists worldwide publicly making statements of not supporting Elsevier in the future which I will obviously join!

You can find the page where scientists make thos statements under: http://thecostofknowledge.com/

And you find much more resources on the whole discussion under http://michaelnielsen.org/polymath1/index.php?title=Journal_publishing_reform

Is this enough or do we need to go even further?

in my oppinion we need to take much more steps. Journals and publications where good institutions in times 50 years ago. Where it took mankind much efford to spread valuable information. But heaving the web and network technologies a scientific journal seems to be pointless in a certain way. The entire reviewing process of course gives some trend of what papers have good quality and what papers don’t. Afterall science is envolving at a faster and faster speed after all.

But as we can see from open projects like wikipedia, linux, wordpress and many more those procedures also yield amazingly good results. They take less effort and are much faster in their deciession process. I have observed that open repositories like http://arxiv.org/ helped mathematicians a lot. But as I know from my good friend Dr. Heinrich Hartmann who is a postdoc at Oxford is that many advanced scientific research discussions among the younger generation already proceed on http://mathoverflow.net/ an “unscientific”, not citeable, not driving your citation count webservice. There people can ask questions and post answers. The best thing is they can rate the answers and receive “carma points” for providing good answers and receiving much feedback.

We see this outside of academia envolving even faster. When I have a question about anything I type it into google search. Chances are high that someone asked the question on yahoo answers or some similar service where best answers are voted and crowed sourced! If the entire society has figured out the strength of this system, why not using it in academia as well?

Obviously these kind of communications and collective intelligence efforts are possible in todays world. So why do we still stick to our old fashoined “good” working methods despite the fact that everyone I talk too is complaining about the reviewing process? Never heard someone complaining about mathoverflows feedback mechanisms!

It is also much cheaper!

It is unbelievable that Elsevier is making a revenue of $3 bn / year. This money comes from our education system! This money could be invested to researches. Maintaining a reliabale citeable academic website compared to mathoverflow would cost us a fraction of this money. Making it possible to afford more researchers actually working on problems!

But actually the best part is, we don’t even need to maintain such a site. Servieces exist. Maybe not designed specifically for academic research but we could still use them. Why do we need a conference or e journal. Every idea every solution can be made publicly and discussed with a broad audience. My first research paper about graphity is still in the reviewing process where some comitee decides weather the idea and results are good enough to be published. This is happening while the corresponding blogpost already received 1465 views on my blog and about twice as much on dzone and made quite some buzz already setting me in the discussion with some co’workers from linked in, yahoo and microsoft… proving that my research results are actually of interest to people. All this happens with me being a new fish in science having no precompiled trust or authority on any topic at all.

So everyone it is your descission how you act. Of course it is easy and probably efficient as well as time saving to rely on some authority to select high quality information sources for you. And I am not saying that the quality in top conferences and journals is not high. But – besides the money – you might pay a very high price in the sense that these authorities filter a lot of also good information for you. Why not using todays modern technologies and have the crowd decide which resources and ideas are worthwile spreading among people and which ones are not?

Google 2011 Q4 Earnings

Rene — Wed, 25 Jan 2012 19:04:30 +0000

Ok no secret here that I am a Google Fan. But listening to the Google Report of 2011 I am just amazed and speechless.

Everything is growing:

$10 bn revenue / quartal ==> more than $100 mio. / day!!!
90 Mio Google+ users
over 60% of plus users engage daily with it and over 80% weekly!
350 Mio active Gmail users
Youtube makes $5 bn revenue
700’000 android devices installed daily
250 Mio. Android devices in total!
11 bn downloads from the android market
chrome is growing (sadly no numbers) But in an interesting (on its own) blog post of reddit you can see 42% of reddit users use chrome (which might not be representative)
Google apps has 5000 new businesses signing up per day (among them: harvard, berkley, states (like wyoming), and a major bank bbva >100’000 employees) ,…)
1 mio. Google+ pages have been created by brands (it is mentioned that there exists a sales team (I knew it all the time (-: )

Larry points out again:

Like I always said: “Emerging highest quality products can generate huge new businesses for Google on the long term. Just like search. And we have a ton of experience monetizing those products over time!”

But have a look for yourself and listen to the annual report!

Question by Filip Stilin (House on Mars): What do you think of Bandcamp?

Rene — Sat, 21 Jan 2012 00:36:10 +0000

Filip Stilin is the frontman of House on Mars a promesing young croation band (check out their music on bandcamp). He loves music and online marketing so he read my blog and sent me and email with a couple of interesting observations and questions. I got his permission to publish parts of his mail and answer the questions to a wider audiance in my blog.

Filip: Even though I wasn’t agreeing with your Facebook skepticism in the beginning, I realized that I was overestimating Facebook in its promotion role. I’ve been creating extremely successful, targeted (extremely low budget though – just 5 or 10 euros at a time) campaigns for my band on Facebook.

Results?

Even though I managed to inflate the number of fans (with 12 fans/1 euro average), the interaction stayed the same. These campaigns aren’t entirely useless, though – having this number of fans or more looks nice in a smaller Croatian market and can help in booking bigger shows (thus getting to more fans)..but still dissappointing. I won’t campaign until we release a website/album/have something to sell.

Rene: I like your observation. The first was that gaining these paid likes does not really increase interactions and increase your reach. As I am saying to the In Legend guys all the time: “Money invested in facebook or even effort in facebook reach is not the best way to increase one’s reach” I am very glad that you came to the same conclusion and shared your insights!

Secondly I partially agree with the effect of large fan numbers while booking gigs. It certainly looks good to business partners like bookers, labels, distributers,… if your social media numbers burst. But again I would say the price is too high. With 1 Euro / 12 fans you would need to invest 1000 Euro for 12’000 fans an 12’000 isn’t even skyhigh (well I don’t know about croation standards). But as we know only a very small fraction of these 12’000 fans would actually become real fans and start interacting with you. All this for getting a gig! I guess this money could much better be invested in a high quality video which especially for a young band is a very good investment. With the video in combination with smart music downloads you will be able to increase your reach. Maybe not to 12’000 fans but still to a solid number of real fans that actually come to your concert because they really care! In this way your social media fancount (especially facebook) will also grow.

Filip: Question1 – sharing music!
About the thesis of providing music only on the band site – I think it’s hard for someone to become our fan if there is no music on Facebook. Choosing one song for preview and directing a fan to .com might work, but they are attracting entirely different audiences – think poppy indie rock vs. oriental, modern metal ballad. Is it okay to let this promo run free and spread like wildfire until the album release? Or should I provide 2/3 of songs for free, and ask for a mail adress for the 3rd one? This might be a good model.

Rene: I agree with what you say. At the time of writing the blog post you are referring too I wasn’t aware of the existing facebook music apps. The important thing is getting a sustainable contact to the person interested in your music. This is achieved ultimately by his email adress. But your question is very important. Of course you have to give people a bait. This could be

snippets
entire song(s) on streeming
a music video (In legend offers downloads under every music video)
a free download (without registration)

and I really don’t know where to set the boarder.

In the beginning times of In Legend we had 3 songs for streaming on myspace and 4 songs on the ep for download in exchange of an email adress. That turned out to be a good solution. People who liked the first songs where curious to download them together with one additional song. So I guess a 2/3 split would work as well. I will just warn you. Asking people for their mail address scares 4 of 5 people away. But hey at least you get the adresses of your fans that are really willing to give something for the music!

Now about the place where to make the connection. If you achive getting the fans mail adress via a smart Facebook music player or via download on your homepage I don’t care. Once people like your music (and chances are higher once you can talk to them frequently) they will also turn into facebook fans. So I recommend switching from rootmusic which you are using right now on your facebook profile to bandRX or Songpier since both services allow you to give access to your music in return of mail adresses. Songpier is a very new service but they also offer a cool mobile app (right now also without collecting mail adresses)
Here is a video about bandRX

Filip: Question2 – What do you think of Bandcamp?
I personally think it’s a great platform for selling music and there is an option of collecting mail adresses. The downside is that there is no valuable content that can be published, like blogs. It would be perfect if it was just a music-streaming, checkout widget on my site.

Rene: One of my Favourite (but retired) bands Jester’s Funeral have just published all their songs to bandcamp and have linked to their homepage and from the homepage to bandcamp. It is not quite the widged you are asking for but I guess this stays an option especially if you don’t bring the technical know how of programming a homepage that enables you to offer your music as a download in exchange of mail adresses.
There is only one thing that bothers me about bandcamp. They only let 200 fans per month download your music for free (email exchange) and offer a pay as much as you want option. From the money raised they keep 15% as a service charge. If you want more free downloads you have to buy them or have people pay for your music.
To some extend they offer a fair deal. It is a good service for a reasonable price. I as a programmer would just do it on my own have the full controll of my data and keep the 15% but I am pretty convinced that bandcamp should be a pretty good option for many musicians

I hope I could answer your questions to your satisfaction! Sorry that there isn’t always the clear black our white, right or wrong. Things are complex on the web but by reading your mail I am very convinced that you are asking the right questions which means that at least in online marketing you are far ahead of 95% of all musicians!

Wikipedia to Blackout for 24 hours to fight SOPA and PIPA – Copy of the user discussion and poll on my blog

Rene — Tue, 17 Jan 2012 11:31:49 +0000

I am one of the web pioneers but this is about the most amazing thing that I will be witnessing on the web as long as I can remember. Tomorrow on January 18th the english version of Wikipedia will shut down for 24 hours to protest two upcoming (?) american laws (SOPA and PIPA) that set the legal foundations to censor the web. This is happening in the country that is so proud of it’s freedom of speech.

Press release: http://wikimediafoundation.org/wiki/Press_releases/English_Wikipedia_to_go_dark
Letter to Users: http://wikimediafoundation.org/wiki/English_Wikipedia_anti-SOPA_blackout
User discussion: http://en.wikipedia.org/wiki/Wikipedia:SOPA_initiative/Action see below for a saved copy!

This is such an important move of democracy that I was standing still for a couple of minutes after I heard of this! 1’800 active wikipedia authors moderators and administrators collectively agreed to make this move in order to show a protest! I am very excited to see where this will be going and what impact this has. Freedom of the internet is what makes this such a beautiful space. Everyone spread this word! discuss this! Don’t let anyone take the freedom of speech and information sharing from you!

Since the user discussion and poll won’t be available tomorrow I attached them to my blogpost.

http://www.rene-pickhardt.de/wp-content/uploads/2012/01/Wikipedia-SOPA-initiative-Action-Wikipedia-the-free-encyclopedia.html

I will not comment on this any further. Please everyone Have your own oppinion and act with responsability.

Ebay auction by Van canto, Ordan Ogen and In Legend for charity.

Rene — Fri, 13 Jan 2012 15:20:09 +0000

Doing charity is of course always a good way of doing marketing. But I think the important rule here is that it really has to be honest and come from the hart and not just be ment to be a marketing campaign. I just realized this after three bands I am friend with did something great. I put this under online music marketing, but again the marketing of something like this should just be the side effect (at most):

After their third tour together Van Canto, Ordan Ogan and In Legend announced that they will post a piece of Bastian Emigs drumset on ebay (feel free to join the auction). The piece is signed from all band members. They said they would donate the money to the charity organization (Médecins Sans Frontières) Bastian Emig the front man and composer of in legend and drummer of van canto is working for. After less than 6 hours the auction is at already 905 Euro. Today the 1000 Euro barrier was taken!

See everyone that is the amazing thing about the web! If something is really great the information about it will spread very fast. This movement costs nothing for Van canto! Social meadia will really help to push this (but it would have been pushed without social media) Here nothing is sold. Just good things happen. Noone seeks for attention. Interesting how on the other side people wonder what weired things they can do in order to get recognition and how simple it is to start a real viral effect by just doing something really good which does not even cost you money or effort!

I am sure many bands could do this or similar things. Even though I have worked together with in legend quite a lot and also with van canto quite some time I like those guys much better now (not that I didn’t like them before. but using your popularity to do good things is very charismatic) Things like this are good for your brand value and brand reputation (besides the fact that they probably also feel very good to do!). Especially this auction really helps charity organizations which is much better than DJ Sammy who stole the idea from a charity organization and didn’t donate any money yet!

Hope the move of Van Canto, In Legend and Ordan Ogan inspires others to do similar things.

And since I really like their Idea here is the current video of van canto:

Algorithms exercise: Find mistakes in Wikipedia articles

Rene — Wed, 11 Jan 2012 14:47:20 +0000

Today I started an experiment I created an excercise for coursework in algorithms and data structures that is very unusuale and many people have been criticle if this was a good idea. The idea behind the exercise is that studens should read wikipedia articles to topics related to lectures and find mistakes or suggest things that could be improoved. Thereby I hope that people will do something that many people in science don’t do often enough: Read something critically and carefully and question the things that you have learnt. (more discussions after the exercise)

Read the following wikipedia articles:

Find at least 5 mistakes or passages that could be improved. Write down what is wrong or what could be improved. Give a justification for your statements and write down your suggested new version of this very passage.

To get inspired you can find mistakes by looking at the discussion page of the articles or at the version history. You might also be able to look the same article in versions of other languages!

Here are some example types / things that could possibly be improoved:

pure mistakes
semantics of links
semantics of pictures
articles could explain easy concepts in difficult words
missing cites
missing links to original scientific work
…

Further discussion

I am really excited how many students will try to do this exercise and how well it is accapted and what the quality of the answers will be…

I would also love to receive your feedback, thoughts and comments about this kind of exercise! Mabe you have some ideas that could be extended or you asked students to do similar coursework?