Springy Web

Tip: Content assist for static imports in Eclipse

2012-02-01T10:30:00.001Z

You can add a list of "Favourites" in your Eclipse settings to help with content assist for standard static methods. For example, assertNotNull() and friends when writing unit tests.

To enable this, in your Eclipse Preferences window, navigate to Java / Editor / Content Assist / Favorites. I've got the following types in mine:

org.hamcrest.Matchers
org.junit.Assert
org.springframework.test.web.server.request.MockMvcRequestBuilders
org.springframework.test.web.server.result.MockMvcResultHandlers
org.springframework.test.web.server.result.MockMvcResultMatchers
org.springframework.test.web.server.setup.MockMvcBuilders

Add them one at a time with the New Type… button.

Hierarchical Faceting With Elastic Search

2012-01-24T09:52:00.001Z

Our current project is using ElasticSearch for, er search.Hierarchical categorization of content found itself pinned to our sprint planning board this week and so I decided to take a look into what options were available.Our classification is a pretty basic hierarchy dividing and redividing things into groups, where each new group is a sub-species of its parent group.
Essentially, at this stage we have two main requirements:

1. Hierarchical Search

By classifying X as:

/a/b/c/d

We must be able to search for and find X for the following paths:

/a
/a/b
/a/b/c
/a/b/c/d

(I'm using /a/b/c/d for brevity, but imagine /Computer Peripherals/Hard Drives/External/USB for example).

2. Drill Down with facets

To allow for searching from the more general to the more specific we want to able to allow children of a given facet to be returned.e.g when we ask for the direct children of /a as facets, the system should return /a/b/.

A useful starting point came not from the ElasticSearch documentation but from the Solr wiki, which is a worthwhile read in my opinion. Much of what follows is explained in fuller detail there.

A nice solution to requirement 1 is provided by Lucene's PathHierarchyTokenizer, which is exposed by ElasticSearch.
As described in the docs this tokenizer allows us to take input like:

/a/b/c/d

and produce the following tokens:

/a
/a/b
/a/b/c
/a/b/c/d

This works pretty well, however by using this tokenizer alone it isn’t particularly easy to constrain the depth of the classification (Requirement 2). Therefore we decided to follow the seemingly hacky but certainly workable idea (See facet.prefix in the Solr article) of storing depth information along with the path.

So our indexed category terms will now be stored as:

0/a
1/a/b
2/a/b/c
3/a/b/c/d

i.e. 1/a/b represents the fact that a/b is 1 level below /a.With this information we can now use a Regex Pattern to get facets for a particular depth - More on this later in the article.

Encoding the depth

We gave some thought to how we might go about encoding depth along with our categories and opted for writing a custom TokenFilter with input provided from the token stream of the PathHierarchyTokenizer.

The remainder of this article is intended to illustrate how to write and test a TokenFilter as well as how to deploy and reference it for use with ElasticSearch. The full source code is available on GitHub.

We are using Maven 3 for managing this customisation so having created a new Maven (jar) project in my IDE, I setup the required dependencies (Full POM is here).

   
    
        
            org.elasticsearch
            elasticsearch
            ${elasticsearch.version}
            compile
        
        
        
            org.apache.lucene
            lucene-test-framework
            ${lucene.version}
            test
        
        
        
            junit
            junit
            ${junit.version}
            test

Note: the lucene-test-framework allows us to easily test our TokenFilter without having to keep deploying to ElasticSearch.

Step 1 Write a failing test

public class TokenCountFilterTest {

    private class TestAnalyzer extends Analyzer {
      @Override
      public TokenStream tokenStream(final String fieldName, final Reader reader) {
        return new TokenCountFilter(new PathHierarchyTokenizer(reader));
      }   
    }
    
    /**
     * Create a string of ${pathElementCount} path elements
     * and assert that each path in the hierarchy is prepended with its depth.
     * 
     * e.g given the input /a/a/a/a
     * 
     * We would expect the output
     * 
     * 0/a
     * 1/a/a
     * 2/a/a/a
     * 3/a/a/a/a
     * @throws IOException 
     * 
     */
    @Test
    public void testTokens() throws IOException {
      int pathElementCount = 10;
      StringBuffer input = new StringBuffer();
      String[] output = new String[pathElementCount];
      String pathElement = "/a";
      for(int i = 0; i< pathElementCount; i++) {
       input.append(pathElement);
       output[i] = i + input.toString(); 
      }
      
      Analyzer testAnalyzer = new TestAnalyzer();
      BaseTokenStreamTestCase.assertAnalyzesTo(testAnalyzer, input.toString(),output);
    }
}

On lines 3-8 we create an Analyzer for use with the test.TokenCountFilter is the filter that we are going to write to amend the tokens produced by the instance of PathHierarchyTokenizer which is passed as an argument to the constructor on line 6.

The javadoc for the testTokens method describes how the test will work. Note that most of the work is managed by BaseTokenStreamTestCase on line 38. This Lucene test class provides the static method assertAnalyzesTo, which accepts an Analyzer to test, an input string to analyze e.g "/a/b/c" and an array of expected output tokens e.g ["0/a", "1/a/b", "2a/b/c"] which should be emitted after tokenization. Internally BaseTokenStreamTestCase utilizes the JUnit framework to make test assertions based on the input and expected output.

Step 2 Write the TokenFilter

Note: This may not be the most efficient way of achieving our goal but performs well enough in our tests. It is important to note that token filters operate on every token produced for a stream. With this in mind one must consider their use carefully. Our category paths will seldom contain more that 5 elements and consequently this filter will not greatly affect performance when used with the PathHierarchyTokenizer. Any future enhancements by us will be posted in a future article and of course we welcome any advice from your own experience.

public class TokenCountFilter extends TokenFilter {

private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
private int count = 0;

public TokenCountFilter(final TokenStream input) {
  super(input);
}

/**
 * Prepend the value of ${count} to each token and increment ${count}
 */
@Override
public final boolean incrementToken() throws IOException {
  final boolean increment = input.incrementToken();
  if (increment) {
    final char[] count = String.valueOf(this.count++).toCharArray();
    final int newLength = termAtt.length() + count.length;
    final char[] resizedBuffer = termAtt.resizeBuffer(newLength);
    termAtt.setLength(newLength); 
    System.arraycopy(resizedBuffer, 0, resizedBuffer, count.length, termAtt.length());
    System.arraycopy(count, 0, resizedBuffer, 0, count.length);
  }
  return increment;
}

@Override 
public void end() throws IOException {
  super.end();
  count = 0;
}

@Override
public void reset() throws IOException {
  super.reset();
  count = 0;
}

For each call to input.incrementToken (line 15) that returns true we end up with a new token in the termAtt buffer (declared on line 3). The remainder of the method manages resizing the termAtt buffer before refilling it with the value of count followed by the original chars from the buffer hence providing us with the required depth encoded before the term in the index.

Running our test class will now give us a green bar.

Step 3 Create a factory to provide the TokenFilter to ElasticSearch

@AnalysisSettingsRequired
public class TokenCountFilterFactory extends AbstractTokenFilterFactory {
  
  @Inject
  public TokenCountFilterFactory(Index index, @IndexSettings Settings indexSettings,
      @Assisted String name, @Assisted Settings settings) { 
      super(index, indexSettings, name, settings);
  }
  
  @Override
  public TokenStream create(TokenStream tokenStream) {
      return new TokenCountFilter(tokenStream);
  }
}

This code is fairly self-explanatory and is basically cut and paste code which follows the same pattern as the other factories from ElasticSearch i.e. with a factory method creating a new instance of our TokenCountFilter around the TokenStream.

Step 4 Make the new code available to ElasticSearch

Running:

mvn clean package

Will compile and the code and generate a jar file. This jar should then be added to $ES_HOME/lib. A restart of ElasticSearch will ensure that our classes get loaded.

Step 5 Try it out

First we can setup a new 'test' index which will use our TokenFilter as part of its analysis:

curl -XPUT 'http://localhost:9200/test/' -d '
{
  "index": {},
  "settings": {
    "analysis": {
      "analyzer": {
        "depth_path_hierarchy": { "type": "custom", "tokenizer": "path_hierarchy", "filter": ["token_count"] }
      },
      "filter" : {
  "token_count": {"type": "com.springyweb.elasticsearch.index.analysis.TokenCountFilterFactory"}
      }
    }
  }
}'

Lines 9-11 define our new "token_count" filter. Note that we actually define the type as TokenCountFilterFactory which is the class responsible for creating our TokenFilter.Lines 6-8 define a ""depth_path_hierarchy" analyzer. Note that we specify the "path_hierarchy" tokenizer which will provide the tokens for our "token_count" filter.

Next we setup a mapping for a type called 'content' which we shall be storing in our 'test' index.

curl -XPUT 'http://localhost:9200/test/content/_mapping' -d '
{
     "test_mapping" : {
         "properties" : {
       "categories" : {"type" : "string", "index_analyzer" : "depth_path_hierarchy"}
         }
     }
}'

A Mapping defines how a document should be mapped to the search engine. In this case we have specified that when indexing our "categories" property that the "depth_path_hierarchy" analyzer should be used. Note: For this particular analyzer to be used solely for indexing (i.e. not for search) we used the key "index_analyzer" in our request. When creating a mapping for a property it is also possible to specify which analyzer is to be used when searching by adding the key/value "search_analyzer" : "<analyzer name>", or one can simply use the single key/value "analyzer" : "<analyzer name>" which will use the same analyzer for indexing and searching the property. Further details regarding mapping can be found here.

We are now in a position to add some test content to our 'test' index. (Take note of the categories)

curl -XPUT http://localhost:9200/test/content/1 -d '{
    "name": "test 1",
  "categories": ["/foo/bar/baz"]
}'

curl -XPUT http://localhost:9200/test/content/2 -d '{
    "name": "test 2",
  "categories": ["/foo/baz/bar"]
}'

The search shown below verifies that we can find both pieces of content beneath the root "0/foo" category which means that Requirement 1 is satisfied.
Note that we use a term query which is not analyzed, the reason for this is that we want search for an entire term e.g. /a/b/c/d without that term being tokenized in any way. This makes sense as otherwise a search for /a/b/c/d would be tokenized into path elements (if we use the "depth_path_hierarchy" analyzer for example) and therefore could find something categorized more generally, /a/b/c for example, which we don't want.

curl -XGET http://localhost:9200/test/content/_search -d '{
    "query" : {
        "term" : { "categories": "0/foo" }
    }
}'

returns:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 0.5945348,
    "hits" : [ {
      "_index" : "test",
      "_type" : "content",
      "_id" : "1",
      "_score" : 0.5945348, "_source" : {
    "name": "test 1",
"categories": ["/foo/bar/baz"]
}
    }, {
      "_index" : "test",
      "_type" : "content",
      "_id" : "2",
      "_score" : 0.5945348, "_source" : {
    "name": "test 2",
"categories": ["/foo/baz/bar"]
}
    } ]
  }
}

Our final test is to try to fulfill Requirement 2, namely to return direct children as facets based on a given path.

curl -XGET http://localhost:9200/test/content/_search?pretty=true -d '{
    "query" : {
            "match_all" : {  }
        },
    "facets" : {
        "category" : { 
            "terms" : {
                "field" : "categories",
                "regex" : "^1/foo/.*"
            } 
       }
    }
}'

Note the regex on line 9. Here we are limiting the facets returned to those that start with 1/foo i.e a depth of 1 beneath /foo.The response shows that we get the required facet terms, '1/foo/baz' and '1/foo/bar'

"facets" : {
   "category" : {
     "_type" : "terms",
     "missing" : 0,
     "total" : 6,
     "other" : 4,
     "terms" : [ {
       "term" : "1/foo/baz",
       "count" : 1
     }, {
       "term" : "1/foo/bar",
       "count" : 1
     } ]
   }

So that wraps this rather long post up. No doubt there are numerous approaches to solving this kind of problem but this article has been intended partially as a solution but more generally as an attemptto illustrate some of the features of ElasticSearch that you may not be familiar with.

You can download the source code for this article: here.

Happy searching!

Error building Erlang on Lion using Macports

2011-12-13T10:31:00.001Z

Macports was hanging during my install of the rabbitmq-server port.Running port -v showed me that an error was occurring during Erlang compilation (RabbitMQ is writtien in Erlang).The issue, it seems to due to the optimisation settings used by llvm-gcc-4.2.The solution is to patch the Erlang Portfile.Edit:


/opt/local/var/macports/sources/rsync.macports.org/release/ports/lang/erlang/Portfile

Search for:


if {${configure.compiler} == "clang"} { 
    configure.compiler llvm-gcc-4.2 
}

And insert the following directly below it:


configure.cflags-delete "-std=c99"
configure.cflags-delete -O2
configure.cxxflags-delete -O2
configure.cflags-append -O0
configure.cxxflags-append -O0

Save the file and then re-run the port install.

Useful MongoDB book

2011-12-08T19:58:00.001Z

Kristina Chodorow's '50 Tips and Tricks for MongoDB Developers' is a concise and really useful resource. It backed up some of my assumptions and taught me a great deal that I had not considered.

Java Virtual Machine Process Status Tool

2011-11-30T13:03:00.001Z

I'm sure I'm just way behind the times but I never knew about the Java Virtual Machine Process Status Tool even though it's been around since Java 1.5.Running jps on the command line with no options, shows me that my Hadoop processes are running.

18513 DataNode
18582 SecondaryNameNode
18761 Jps
18699 TaskTracker
18631 JobTracker

With the the -l option I can see the package names too

18513 org.apache.hadoop.hdfs.server.datanode.DataNode
18763 sun.tools.jps.Jps
18582 org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
18699 org.apache.hadoop.mapred.TaskTracker
18631 org.apache.hadoop.mapred.JobTracker

With -v I can see the the arguments passed to the JVM.

18513 DataNode -Dfile.encoding=UTF-8 -Xserver -Dproc_datano...

18582 SecondaryNameNode -Dfile.encoding=UTF-8 -Dproc_seconda...

18766 Jps -Dfile.encoding=UTF-8 -Dapplication.home=/System/Li...

18699 TaskTracker -Dfile.encoding=UTF-8 -Dproc_tasktracker -Xm ...

18631 JobTracker -Dfile.encoding=UTF-8 -Dproc_jobtracker -Xmx ...

A useful tool that is certainly easier than

 ps aux | grep java

Although do note that what looks like the OS process id is actually the local VM identifier which according to jps docs: "is typically, but not necessarily, the operating system's process identifier for the JVM proces"

Hadoop on OSX 10.7

2011-11-30T12:35:00.000Z

When starting Hadoop using start-all.sh I kept seeing the following in my log file


"Unable to load realm info from SCDynamicStore"

A workaround is suggested on the Hadoop Jira under HADOOP-7489which is to specify kerberos config on the command line:


 -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk

This works perfectly but is easier if you add it into the HADOOP_OPTS environment variable in $HADOOP_HOME/conf/hadoop-env.shMine looks like this:


export HADOOP_OPTS="-server -Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

ElasticSearch is full of Springy goodness

2011-11-29T17:46:00.000Z

Firstly, I am a huge fan of Solr. It is easy to test, blisteringly fast, and incredibly stable. I would recommend it as a solution to many search problems in a heartbeat. However, despite the recent efforts of SolrCloud I have simply found that as a scalable cloud solution, Solr is still in its infancy.

Thanks to a useful blog article I was able to test some multi-core examples out (note: If you do follow the blog example then beware of capitalisation issues with attribute names. - e.g "instancedir" should be camel case - "instanceDir"). My major issue is that the whole approach feels too clunky, particularly for a solution such as ours which requires multi-tennancy and therefore a large number of cores. At the time of writing SolrCloud doesn't seem to have a nice way of discovering other nodes in a cluster. It all works but it isn't slick.

ElasticSearch on the other hand has been designed for the cloud. James Cook has written a brilliant tutorial on running ElasticSearch on EC2. In just a few hours I had created some test EC2 instances running ElasticSearch.
Discovery of other ElasticSearch instances was trivial. The cloud-aws plugin allows for several options, e.g security groups or tags. When I stopped the "master" instance, the "slave" instance noticed and took over as master. Asynchronous backup to EC3 just worked.
Multi-tennancy has been made really easy too - If you post to an index that doesn't exist, it gets created. Trivial to get going and so far very impressive.

Developing an API service with Ruby

2011-11-29T13:32:00.001Z

Our current project involves creating a json-based API service, which can be called either from a web browser, or from other code. We started developing it directly in Rails, but then found the grape project on github.
Grape seems to be a much better fit for the project than Rails was. We will still use Rails for the front end website, but using grape for the API service means that the code is a lot simpler.