Extending Neo4j

One of the great things about Neo4j is how easy it is to extend it. You can extend Neo4j with Plugins and Unmanaged Extensions. Two great examples of plugins are the Gremlin Plugin (which lets you use the Gremlin library with Neo4j) and the Spatial Plugin (which lets you perform spatial operations like searching for data within specified regions or within a specified distance of a point of interest).

Plugins are meant to extend the capabilities of the database, nodes, or relationships. Unmanaged extensions are meant to let you do anything you want. This great power comes with great responsibility, so be careful what you do here. David Montag cooked up an unmanaged extension template for us to use on github so lets give it a whirl. We are going to clone the project, compile it, download Neo4j, configure Neo4j to use the extension, test the extension and tweak it a bit.

Start by cloning the project:

git clone git://github.com/dmontag/neo4j-unmanaged-extension-template.git
cd neo4j-unmanaged-extension-template

Let’s add Neography to our gemfile and install it:

echo "gem 'neography' " > Gemfile
bundle install 

Then let’s go ahead and install Neo4j. For this project I am going to use Neo4j 1.8 enterprise edition:

echo "require 'neography/tasks'" > Rakefile
rake neo4j:install[enterprise,1.8]

Make sure you are not running Neo4j already as the next step will download and build the unmanaged extension template which includes some tests that will fail if you already have Neo4j running somewhere.

mvn clean package

This will download and do its thing unless you don’t have Maven installed (type mvn -version to see if you do). Finally you should see:

...
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ unmanaged-extension-template ---
[INFO] Building jar: /Users/maxdemarzi/Projects/neo4j-unmanaged-extension-template/target/unmanaged-extension-template-1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.153s
[INFO] Finished at: Tue Nov 13 22:16:40 CST 2012
[INFO] Final Memory: 27M/218M
[INFO] ------------------------------------------------------------------------

Now we can copy the jar file it created to our Neo4j plugins directory:

cp target/unmanaged-extension-template-1.0.jar neo4j/plugins/

Before this will work, we need to tell Neo4j about our new unmanaged extension by adding a line to the neo4j-server.properties file in the neo4j/conf directory:

echo "org.neo4j.server.thirdparty_jaxrs_classes=org.neo4j.example.unmanagedextension=/example" >> neo4j/conf/neo4j-server.properties

We are telling Neo4j to load this unmanaged extension and make “/example” the rest api end-point. Now we can start the Neo4j server…

rake neo4j:start

… and test this out using curl:

curl http://localhost:7474/example/service/helloworld
Hello World!

or via the http console in the web admin:

The reason this works is because the jar file contains the contents of /src/main/java/org/neo4j/example/unmanagedextension/MyService.java in which you will find:

@Path("/service")
public class MyService {

    @GET
    @Path("/helloworld")
    public String helloWorld() {
        return "Hello World!";
    }

This very simple method just returns a string from the server whenever we call it. Let’s try something a little more useful, like warming up the Node and Relationship Caches. We need to first create some nodes and relationships. Edit your Rakefile to look like this:

require 'neography'
require 'neography/tasks'

namespace :neo4j do
  task :create do
    neo = Neography::Rest.new
    nodes = {}
    5.times do |x|
      5.times do |y|
       node = neo.create_node({:x => x.to_f, :y=> y.to_f}) 
       nodes["#{x}-#{y}"] = node["self"].split('/').last
      end
    end
    
    4.times do |x|
      4.times do |y|
       neo.create_relationship("next_to", 
                               nodes["#{x}-#{y}"], 
                               nodes["#{x+1}-#{y}"], 
                               {:time => (1 + rand(4)).to_f }) 
       neo.create_relationship("next_to", 
                               nodes["#{x}-#{y}"], 
                               nodes["#{x}-#{y+1}"], 
                               {:time => (1 + rand(4)).to_f }) 
      end
      neo.create_relationship("next_to", 
                              nodes["#{x}-4"], 
                              nodes["#{x+1}-4"], 
                              {:time => (1 + rand(4)).to_f }) 
      neo.create_relationship("next_to", 
                              nodes["4-#{x}"], 
                              nodes["4-#{x+1}"], 
                              {:time => (1 + rand(4)).to_f }) 
    end
        
  end
end

Then run:

rake neo4j:create

… to create your graph. I’ll explain why we created this particular graph in a follow up blog post, but for now back to our MyService.java file. We will be using the GlobalGraphOperations tool and adding the following:

import org.neo4j.tooling.GlobalGraphOperations;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Relationship;

    @GET
    @Path("/warmup")
    public String warmUp(@Context GraphDatabaseService db) {
		Node start;
		for ( Node n : GlobalGraphOperations.at( db ).getAllNodes() ) {
		   n.getPropertyKeys();
		   for ( Relationship relationship : n.getRelationships() ) {
              start = relationship.getStartNode();
           }
        }
        for ( Relationship r : GlobalGraphOperations.at( db ).getAllRelationships() ) {
          r.getPropertyKeys();
          start = r.getStartNode();
        }
    return "Warmed up and ready to go!";
    }

The code above first iterates through every node, brings up its properties, then calls up the relationships that node has. The code then iterates through every relationship, brings up its properties, then calls up the nodes it connects. The sequence above effectively warms up your caches. If your graph fits in memory, this will prime the database. If your graph doesn’t fit in memory maybe you want to specify a certain set of nodes or relationships to warm up. Either way you probably don’t want to run this over and over again or expose it to your end users. Remember what I said about being responsible with what you do here.

You’ll have to stop Neo4j, recompile it, copy the new jar file over the existing one, restart Neo4j, and run the command.

rake neo4j:stop
mvn clean package
cp target/unmanaged-extension-template-1.0.jar neo4j/plugins/
rake neo4j:start
curl http://localhost:7474/example/service/warmup

If everything went well you should see:

Warmed up and ready to go!

…and looking at your node and relationship caches in the Neo4j web admin server info page:

If you don’t see these you are probably running the community edition instead of the enterprise edition. If the numbers are all zero then something went wrong. Retrace your steps and let me know if any part of this is confusing so I can do a better job explaining it. You can see my copy of MyService.java on github.

By the way, if you just want a warm cache you can do the same with Cypher by using:

START n=node(*) 
MATCH n -[r?]-> () 
RETURN count(n.property_i_do_not_have?)+count(r.property_i_do_not_have?)

Did you know that “Cypher” is a term used to describe freestyle rap in a group setting?

Well, now you know. Stay tuned for part two of this series.

Tagged , , , , , , , , , ,

18 thoughts on “Extending Neo4j

  1. […] Extending Neo4j I showed you how to create an unmanaged extension to warm up the node and relationship caches. […]

  2. […] Extending Neo4j by Max De Marzi. […]

  3. […] Extending Neo4j I showed you how to create an unmanaged extension to warm up the node and relationship caches. […]

  4. Mak says:

    Hi, and thanks for the useful page, however, what I am trying to stick to is pure neo4J for the measurement of centrality degrees, clustering coeff. on my twitter analysis,
    from reports, I am have understood that performance is seriously affected by additional plugins to neo4J.
    I truely wonder if this can be supprted using neo4j traversals, without getting another layer such as cypher or Gremlin involved, and if there is some clue about this to speed up my work. I have been through neo4J documentation.

    Cheers,

  5. Peter Boling says:

    Trying to figure out how to install the spatial plugin into Neo4j (which I have successfully installed). I am coming up empty on all my searches. Is spatial an unmanaged extension, and therefore should be similar to this procedure?

  6. […] Extending Neo4j « Max De Marzi […]

  7. […] start with I needed to add spatial as an unmanaged extension to my neo4j plugins folder which involved doing the […]

  8. […] going to walk-through the code using the Neo4j Core API inside an unmanaged extension so it can be accessed via REST and I’ll show you how I wrote a couple of performance tests […]

  9. kamran says:

    I’ve noticed that the Cypher endpoint for neo4j outputs incredibly heavy JSON as a response, a lot of it is not useful, and much of it is structured in a way that is not desirable (relationships are returned at the same level as nodes). Would it make sense in this situation to write a server plugin that somehow extends the cypher endpoint to return “sane” JSON output instead of the endless arrays it currently returns?

  10. […] we are using an Unmanaged Extension that takes the name of a relationship type. For every node in the graph, we check to see if it does […]

  11. […] you are looking at is an unmanaged extension that takes a node id parameter and uses that node as the starting point of a traversal. The […]

  12. […] blog post we saw how we could get about 1,250 requests per second (with a 10ms latency) using an Unmanaged Extension running inside the Neo4j server… but what if we wanted to go […]

  13. I found that the code at https://github.com/dmontag/neo4j-unmanaged-extension-template does not work for Neo4j 2.0. So I forked it at https://github.com/craigtaverner/neo4j-unmanaged-extension-template and fixed it for 2.0.1. I’ve sent a pull request, but until that goes through I guess people could just use my fork. No work appears to be happening on the template, so hopefully it is stable (aside from the minor changes required to follow releases).

    For following releases, I suspect that most of the time it is sufficient to merely change the version number in the pom. For example, this blog talks about 1.8, but dmontag master is at 1.9, and my master is now at 2.0.1. Without having to do a real upgrade, it could in many cases be sufficient to just say that the command:

    rake neo4j:install[community,2.0.1]

    must match the pom.xml line:

    2.0.1

    (except for the 1.9->2.0 change which seemed to require a little more, as seen in the pull request)

  14. […] I am going to show you how to run Neo4j Embedded and Neo4j Server at the same time…and an Unmanaged Extension inside that Neo4j Server. There aren’t any real good reasons why you’d want to do this, […]

  15. Gagan says:

    Why do you say unmanged extensions are dangerous . What should we keep in mind while coding them ?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,682 other followers

%d bloggers like this: