Over the last weeks there was quite some quality programming time for me. First of all I built some indices on the typology data base in which way I was able to increase the retrieval speed of typology by a factor of over 1000 which is something that rarely happens in computer science. I will blog about this soon. But heaving those techniques at hand I also used them to built a better auto completion for the search function of my online social network metalcon.de.

The search functionality is not deployed to the real site yet. But on the demo page you can find a demo showing how the completion is helping you typing. Right now the network requests are faster than google search (which I admit it is quite easy if you only have to handle a request a second and also have a much smaller concept space).  Still I was amazed by the ease and beauty of the program and the fact that the suggestions for autocompletion are actually more accurate than our current data base search. So feel free to have a look at the demo:

http://gwt.metalcon.de/GWT-Modelling/#AutoCompletionTest

Right now it consists of about 150 thousand concepts which come from 4 different data sources (Metal Bands, Metal records, Tracks and Germen venues for Heavy metal) I am pretty sure that increasing the size of the concept space by 2 orders of magnitude should not be a problem. And if everything works out fine I will be able to test this hypothesis on my joint project related work which will have a data base with at least 1 mio. concepts that need to be autocompleted.

Even though everyting I used but the ContextListener and my small but effective caching strategy can be found at http://developer-resource.blogspot.de/2008/07/google-web-toolkit-suggest-box-rpc.html and the data structure (suggest tree) is open source and can be found at http://sourceforge.net/projects/suggesttree/ I am planning to produce a series of screencasts and release the source code of my implementation together with some test data over the next weeks in order to spread the knowledge of how to built strong auto completion engines. The planned structure of these articles will be:

part 1: introduction of which parts exist and where to find them

  • Set up a gwt project
  • Erease all files that are not required
  • Create a basic Design

part 2: AutoComplete via RPC

  • Neccesary client side Stuff
  • Integration of SuggestBox and Suggest Oracle
  • Setting up the Remote procedure call

part 3: A basic AutoComplete Server

  • show how to fill data with it and where to include it in the autocomplete
  • disclaimer! of not a good solution yet
  • Always the same suggestions

part 4: AutoComplete Pulling suggestions from a data base

  • inlcuding a data base
  • locking the data base for every auto complete http request
  • show how this is a poor design
  • demonstrate low response times speed

part 5: Introducing the context Listener

  • introducing a context listener.
  • demonstrate lacks in speed with every network request

part 6: Introducing a fast Index (Suggest Tree)

  • inlcude the suggest tree
  • demonstrate increased speed

part 7: Introducing client side caching and formatting

  • introducing caching
  • demonstrate no network traffic for cached completions

not covered topics: (but for some points happy for hints)

  • on user login: create personalized suggest tree save in some context data structure
  • merging from personalized AND gobal index (google will only display 2 or 3 personalized results)
  • index compression
  • schedualing / caching / precalculation of index
  • not prefix retrieval (merging?)
  • css of retrieval box
  • parallel architectures for searching

If you like this post, you might like these related posts:

  1. Building an Autocompletion on GWT screencast Part 1: Getting Warm – Reviewing remote procedure calls Quite a while ago I promised to create some screencasts...
  2. Building an Autocomplete Service in GWT screencast Part 4: Integrating the neo4j Data base In this screencast of my series I explain at a...
  3. Building an Autocompletion on GWT screencast Part 2: Invoking The Remote Procedure Call Hey everyone after posting my first screencast in this series...
  4. Building an Autocomplete Service in GWT screencast Part 3: Getting the Server code to send a basic response In this screencast of my series on building an autocomplete...
  5. GWT + database connection in Servlet ContextListener – Auto Complete Video Tutorial Part 5 Finally we have all the basics that are needed for...

Sharing:

Tags: , , , , ,

6 Comments on Building an Autocompletion on GWT with RPC, ContextListener and a Suggest Tree: Part 0

  1. [...] Building an Autocompletion on GWT with RPC, ContextListener and a Suggest Tree: Part 0 [...]

  2. wil.pannell says:

    looking forward to your series…can you use some help with the gwt?

  3. [...] this screencast of my series on building an autocomplete service you will learn how to implement a Server servlet in GWT such that autocomplete queries receive a [...]

  4. [...] to users. Since our development was not proceeding we never deployed Graphity within metalcon. Also building an autocomplete service for the site should not be a problem [...]

  5. [...] this screencast of my series I explain at a very basic level how to integrate a data base to pull data for autocomplete queries. [...]

  6. [...] we have all the basics that are needed for building an Autocomplete service and now comes the juicy part. From now on we are looking at how to make it fast and robust. In the [...]

Leave a Reply

*

Close

Subscribe to my newsletter

You don't like mail?