Max De Marzi

Relationships

maxdemarzi — Thu, 10 May 2012 03:18:57 +0000

When working with relational databases, the join tables are sometimes treated as second class citizens. If they are lucky, they’ll get some additional fields, but are often just placeholder tables connecting your main object tables together. It leads you to think about objects first, and relationships second. With graphs, you will want to switch up your thinking. You want to start thinking about how things are connected. Think about the different ways things are connected. Two people can be friends, co-workers, and neighbors all at the same time.

If you’ve ever been crazy enough to go back to school after your bachelor’s degree, you know all about research papers, journals, and citations. So what kind of relationships can we find here? Papers are rarely authored alone. Having multiple authors means you can create a graph of co-authors. Which students and scientists work together on papers, and which should work together. Scientific research builds upon the work of others, and previous work can be used to build a graph of citations. Keywords in the abstracts or extracted from the papers can be used to build a graph of similar content.

What about a social network like Twitter? How about a Follower graph, a Reply graph, a Mention graph, a Co-Listed graph, a HashTag graph, a Co-Location graph, a Shared-URL graph? What about blogs? How about a Link graph, a Blogroll graph, a Tag graph?

You will be surprised by how connected your data is once you start exploring it. Don’t be afraid of your model. Just start with whatever you have and start adding nodes and relationships as they become apparent to you. Your model will probably evolve, so don’t waste your time second guessing yourself.

When folks ask me about modeling, I tell them to start with the questions they want to answer. It doesn’t end there however. Once you’ve answered those questions, you’ll discover more. You’ll say, if I had this extra piece of data, I could ask and answer a ton new questions I didn’t think about before.

I’ll spend some time in the next few posts digging in a little deeper.

Filed under: Random Tagged: graph database, relationship graph

Slides from Chicago Graph Database April Meet-up

maxdemarzi — Mon, 07 May 2012 00:51:55 +0000

Introduction to Graph Databases

View more PowerPoint from Max De Marzi

Introduction to Gremlin

View more PowerPoint from Max De Marzi

Thank you again to Groupon Engineering! They hosted our Graph Database Meet-up at their Headquarters.

Join us May 31st, 2012 for our next meet-up Add data from your existing Application into Neo4j.

Filed under: Gremlin Tagged: graph database, gremlin, neo4j

Using Three.js with Neo4j

maxdemarzi — Mon, 16 Apr 2012 13:18:16 +0000

Last week we saw Sigma.js, and as promised here is a graph visualization with Three.js and Neo4j. Three.js is a lightweight 3D library, written by Mr. Doob and a small army of contributors.

The things you can do with Three.js are amazing, and my little demo here doesn’t give it justice, but nonetheless I’ll show you how to build it.

We need to pass the nodes and relationships to Three.js, one of the ways we can do that easily is with the Gon gem, since we’re using sinatra, we’ll use the special gon-sinatra gem.

class App < Sinatra::Base
  register Gon::Sinatra
  
  def nodes
    neo = Neography::Rest.new
    cypher_query =  " START node = node:nodes_index(type='User')"
    cypher_query << " RETURN ID(node), node"
    neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0]}.merge(n[1]["data"])}
  end  
  
  def edges
    neo = Neography::Rest.new
    cypher_query =  " START source = node:nodes_index(type='User')"
    cypher_query << " MATCH source -[rel]-> target"
    cypher_query << " RETURN ID(rel), ID(source), ID(target)"
    neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0], "source" => n[1], "target" => n[2]} }
  end
  
  get '/' do
    neo = Neography::Rest.new
    gon.nodes = nodes  
    gon.edges = edges
    erb :index
  end
end

In our view we will add”include_gon” which ties our nodes and edges to our html page.



	
		Three.js and Neo4j
		<%= include_gon %>

If you view the source of index.html, you’ll see our nodes and edges.

window.gon = {};
gon.nodes=[{"id":1,
            "rotation_x":5.061454830783556,
            "name":"zfbushqe",
            "position_y":256,
            "position_x":658,
            "position_z":577,
            "rotation_y":3.543018381548489},              
           {"id":2,
            "rotation_x":4.572762640225143,
            "name":"afntayhh",
            "position_y":-22,
            "position_x":510,
            "position_z":404,
            "rotation_y":2.2689280275926285}
            ...
gon.edges=[{"id":3,"source":1,"target":198},
           {"id":2,"source":1,"target":39},
           {"id":1,"source":1,"target":21}
           ...
           ]

To use Three.js, we’ll need to create a camera, a scene and pick a renderer to use.

  camera = new THREE.PerspectiveCamera( 75, window.innerWidth / window.innerHeight, 1, 10000 );
  camera.position.z = 100;

  scene = new THREE.Scene();

  scene.add( camera );

  renderer = new THREE.CanvasRenderer();
  renderer.setSize( window.innerWidth, window.innerHeight );
  container.appendChild( renderer.domElement );

To create our nodes, we’ll use Spheres and use a normal material. We’ll grab the nodes from gon and use their properties to position and set the orientation of our spheres.

  var geometry = new THREE.SphereGeometry( 50, 8, 7, false );
  var material = new THREE.MeshNormalMaterial();
    
  group = new THREE.Object3D();
		
  for (n in gon.nodes) {
	
    var mesh = new THREE.Mesh( geometry, material );
    mesh.position.x = gon.nodes[n].position_x;
    mesh.position.y = gon.nodes[n].position_y;
    mesh.position.z = gon.nodes[n].position_z;
    mesh.rotation.x = gon.nodes[n].rotation_x;
    mesh.rotation.y = gon.nodes[n].rotation_y;
    mesh.matrixAutoUpdate = false;
    mesh.updateMatrix();
    group.add( mesh );

  }

  scene.add( group );

For our edges, we’ll create simple lines of random colors between the source and target nodes.

  for (n in gon.edges) {
    var line_segment = new THREE.Geometry();
    line_segment.vertices.push( new THREE.Vertex( group.children[gon.edges[n].source - 1].position ) );
    line_segment.vertices.push( new THREE.Vertex( group.children[gon.edges[n].target - 1].position ) );
    var line = new THREE.Line( line_segment, 
                              new THREE.LineBasicMaterial( { color: Math.random() * 0xffffff, 
                                                             opacity: 0.5 } ) );

    scene.add(line)
  }

We’ll animate our visualization by rendering the scene and controlling the camera with the mouse.

document.addEventListener( 'mousemove',  onDocumentMouseMove,  false );

function onDocumentMouseMove(event) {
  mouseX = event.clientX - windowHalfX;
  mouseY = event.clientY - windowHalfY;
}

function animate() {
  requestAnimationFrame( animate );
  render();
}

function render() {
  camera.position.x += ( mouseX - camera.position.x ) * .05;
  camera.position.y += ( - mouseY + 200 - camera.position.y ) * .05;
  camera.lookAt( scene.position );
  renderer.render( scene, camera );
}

I’m going to skip the graph creation but all the code is available on github as usual, and you’ll definitively want to see this live on Heroku.

If you want to learn more about Three.js, check out these tutorial and these videos.

Filed under: Cypher, Neography, Visualization Tagged: cypher, graph database, heroku, javascript, neo4j, ruby, visualization

Using Sigma.js with Neo4j

maxdemarzi — Thu, 12 Apr 2012 16:23:59 +0000

I’ve done a few posts recently using D3.js and now I want to show you how to use two other great Javascript libraries to visualize your graphs. We’ll start with Sigma.js and soon I’ll do another post with Three.js.

We’re going to create our graph and group our nodes into five clusters. You’ll notice later on that we’re going to give our clustered nodes colors using rgb values so we’ll be able to see them move around until they find their right place in our layout. We’ll be using two Sigma.js plugins, the GEFX (Graph Exchange XML Format) parser and the ForceAtlas2 layout.

You can see what a GEFX file looks like below. Notice it comes from Gephi which is an interactive visualization and exploration platform, which runs on all major operating systems, is open source, and is free.

In order to build this file, we will need to get the nodes and edges from the graph and create an XML file.

get '/graph.xml' do
  @nodes = nodes  
  @edges = edges
  builder :graph
end

We’ll use Cypher to get our nodes and edges:

def nodes
  neo = Neography::Rest.new
  cypher_query =  " START node = node:nodes_index(type='User')"
  cypher_query << " RETURN ID(node), node"
  neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0]}.merge(n[1]["data"])}
end

We need the node and relationship ids, so notice I’m using the ID() function in both cases.

def edges
  neo = Neography::Rest.new
  cypher_query =  " START source = node:nodes_index(type='User')"
  cypher_query << " MATCH source -[rel]-> target"
  cypher_query << " RETURN ID(rel), ID(source), ID(target)"
  neo.execute_query(cypher_query)["data"].collect{|n| {"id" => n[0], 
                                                       "source" => n[1], 
                                                       "target" => n[2]} 
                                                      }
end

So far we have seen graphs represented as JSON, and we’ve built these manually. Today we’ll take advantage of the Builder Ruby Gem to build our graph in XML.

xml.instruct! :xml
xml.gexf 'xmlns' => "http://www.gephi.org/gexf", 'xmlns:viz' => "http://www.gephi.org/gexf/viz"  do
  xml.graph 'defaultedgetype' => "directed", 'idtype' => "string", 'type' => "static" do
    xml.nodes :count => @nodes.size do
      @nodes.each do |n|
        xml.node :id => n["id"],    :label => n["name"] do
	   xml.tag!("viz:size",     :value => n["size"])
	   xml.tag!("viz:color",    :b => n["b"], :g => n["g"], :r => n["r"])
	   xml.tag!("viz:position", :x => n["x"], :y => n["y"]) 
	 end
      end
    end
    xml.edges :count => @edges.size do
      @edges.each do |e|
        xml.edge:id => e["id"], :source => e["source"], :target => e["target"] 
      end
    end
  end
end

You can get the code on github as usual and see it running live on Heroku at neosigma.herokuapp.com. You will want to see it live on Heroku so you can see the nodes in random positions and then move to form clusters. Use your mouse wheel to zoom in, and click and drag to move around.

Credit goes out to Alexis Jacomy and Mathieu Jacomy.

You’ve seen me create numerous random graphs, but for completeness here is the code for this graph. Notice how I create 5 clusters and for each node I assign half its relationships to other nodes in their cluster and half to random nodes? This is so the ForceAtlas2 layout plugin clusters our nodes neatly.

def create_graph
  neo = Neography::Rest.new
  graph_exists = neo.get_node_properties(1)
  return if graph_exists && graph_exists['name']

  names = 500.times.collect{|x| generate_text}
  clusters = 5.times.collect{|x| {:r => rand(256),
                                  :g => rand(256),
                                  :b => rand(256)} }
  commands = []
  names.each_index do |n|
    cluster = clusters[n % clusters.size]
    commands << [:create_node, {:name => names[n], 
                                :size => 5.0 + rand(20.0), 
                                :r => cluster[:r],
                                :g => cluster[:g],
                                :b => cluster[:b],
                                :x => rand(600) - 300,
                                :y => rand(150) - 150
                                 }]
  end
 
  names.each_index do |from| 
    commands << [:add_node_to_index, "nodes_index", "type", "User", "{#{from}}"]
    connected = []

    # create clustered relationships
    members = 20.times.collect{|x| x * 10 + (from % clusters.size)}
    members.delete(from)
    rels = 3
    rels.times do |x|
      to = members[x]
      connected << to
      commands << [:create_relationship, "follows", "{#{from}}", "{#{to}}"]  unless to == from
    end    

    # create random relationships
    rels = 3
    rels.times do |x|
      to = rand(names.size)
      commands << [:create_relationship, "follows", "{#{from}}", "{#{to}}"] unless (to == from) || connected.include?(to)
    end

  end

  batch_result = neo.batch *commands
end

Filed under: Cypher, Heroku, Visualization Tagged: cypher, graph database, heroku, neo4j, nosql, ruby, visualization

Slides from Chicago Graph Database March Meet-up

maxdemarzi — Wed, 28 Mar 2012 16:41:18 +0000

Cypher

View more PowerPoint from Max De Marzi

Thank you very much to Groupon Engineering! They hosted our Graph Database Meet-up at their Headquarters.

Join us April 30th, 2012 for Neo4j Basics and an introduction to Gremlin.

Filed under: Cypher Tagged: cypher, graph database, neo4j, nosql

Visualizing a set of Hiveplots with Neo4j

maxdemarzi — Mon, 26 Mar 2012 16:47:15 +0000

What should a graph look like and how can I tell two graphs apart?

These are questions Martin Krzywinski (Genome Sciences Center, Vancouver, BC) has been asking. Take a look at the picture below:

It’s the same graph, the same data, visualized 8 different ways. Which is the right way? What advantage does one layout give over the other? Can you tell it’s the same network? I can’t.

Eight layouts might be too much, so let’s just look at one on the next picture:

Martin took the spring embedded visualization and tweaked it around. Can you tell it’s the same graph, the same data underneath? I can’t.

To tackle this problem, Martin invented the Hive Plot, a perceptually uniform and scalable layout visualization for network visual analytics.

If you want to learn more about Hive Plots, take a look at his website and this presentation (it is quite large at 20 MB). I cannot do it justice in this short blog post, and in all honestly haven’t had the time to study it properly.

Today I just want to give you a little taste of Hiveplots. I am going to visualize the github graphs of nine languages you might not have heard of: Boo, Dylan, Factor, Gosu, Mirah, Nemerle, Nu, Parrot, Self. I’m not going to show you how to create the graph this time, because this is real data we are using. You can take a look at it on the data folder in github.

The graph is basically: (Language)–(Repository)–(User). There are two relationships between Repository and User, wrote and forked.

I’ll show you how to get the data out and into our visualization.

def wroterepos(language)
  neo = Neography::Rest.new
  neo.execute_script("m = [:]
                      g.V.filter{it.type == 'language' && it.name == '#{language}'}
                       .in.transform{m[it.name] = it.in('wrote').gather{it.name}.next()}
                       .iterate()
                      m")
end

We do the same thing but for forked. This may seem a bit strange to you, but what I am doing is kind of like the SQL equivalent of a LEFT OUTER JOIN with Gremlin.

def forkedrepos(language)
  neo = Neography::Rest.new
  neo.execute_script("m = [:]
                      g.V.filter{it.type == 'language' && it.name == '#{language}'}
                       .in.transform{m[it.name] = it.in('forked').gather{it.name}.next()}
                       .iterate()
                      m")
end

Now we do some ruby magic to put our data into the JSON format the visualization wants.

get '/hive/:name' do
  repos        = []
  writers      = [] 
  forkers      = []
  temp_forkers = []
  temp_writers = []

  wroterepos(params[:name]).each_pair do |key, value|
    repos << {"name" => key, "imports" => value, "node_type" => "repo"}
    temp_writers << { "name" => value[0] }
  end

  i = 0
  forkedrepos(params[:name]).each_pair do |key, value|
    repos[i]["imports"] =  repos[i]["imports"] + value
    temp_writers[i]["imports"] = value
    temp_forkers << value
    i += 1
  end

  temp_writers.group_by {|i| i["name"]}.each do |w, f|
    writers << {"name" => w, 
                "imports" => f.collect{|y| y["imports"]}.flatten.uniq, 
                "node_type" => "writer"}
  end

  temp_forkers.flatten.uniq.delete_if{|x| writers.collect{|y| y["name"]}.include?(x)}.each do |f|
    forkers << {"name" => f, 
                "imports" => [], 
                "node_type" => "forker"}
  end

  (repos + writers + forkers).to_json
end

The blue color nodes are our repositories, the yellow nodes are our writers, and the green nodes are our forkers. The 12 o’clock axis (the top) shows nodes with only outgoing relationships. The bottom-left axis shows nodes with only incoming relationships. These are the writers without any forks, and the forkers who never started their own public projects. The remaining nodes in the bottom-right have both incoming and outgoing relationships. These are the repository writers who created projects other people found worth forking.

The graphs are ordered across for each row in the following manner:

Boo, Dylan, Factor
Gosu, Mirah, Nemerle
Nu, Parrot, Self

Can you see the similarities between Boo, Factor and Numerle? See how different they are from Gosu and Self? What does the hive plot tell you about these Language github repositories?

You can try a live version at hiveplot.herokuapp.com/index.html and as always the code is available on github.

Our visualization was done by Rich Morin and Mike Bostock with D3.js. Is is a hot off the press work in progress. You can follow the action on this D3.js google group thread.

Filed under: Gremlin, Visualization Tagged: d3.js, graph database, gremlin, javascript, neo4j, network, visualization

MadCoderTV is live on Roku

maxdemarzi — Tue, 20 Mar 2012 11:00:09 +0000

I’ve had a Roku streaming player for my TV for a few years now and a few months ago I got interested in how it actually worked. I started seeing more channels pop-up and I thought how hard could it be to put one up? So I found their SDK, grabbed their sample application and after a few tweaks, some nice artwork and finding content, it was approved.

Why go through the trouble of doing this to watch videos on my TV when I can just watch them on the laptop? One word… Distraction.

If a video is more than 3 to 5 minutes long, there is a good chance I won’t make it all the way through. It’s hard to sit still when you have the full power of your laptop and the internet at your fingertips. On the TV, as long as there are no commercials, I’m pretty much going to just sit there and watch. Maybe it is some mild form of ADD, maybe it’s normal.

I am using Github Pages to host the xml for the feeds, and the videos are hosted on Amazon cloudfront by their owners. You can find some RailsCasts and Confreaks videos at the moment, but the feeds like the channel are open source, so if you have any content you’d like to add, please send me a pull request to the gh-pages branch.

So what does this have to do with Ruby + Graphs?

Well, I asked the Neo4j guys and they’re letting me have their videos on there as well. So if you want to see Andreas Kollegger tell you how to get started with Neo4j on your 60″ big screen at home, now you can.

The code is available at https://github.com/maxdemarzi/madcodertv be sure to check out the gh-pages branch for the video content and send me a pull request if you have any videos you want to add!

Checkout a screenshot:

Update: So far 6500 developers have installed the channel. If you have no idea what I’m talking about, a Roku is a little box you attach to your TV and the internet and let’s you stream videos from over 400 channels.

Filed under: Random Tagged: channel, github, neo4j, roku, television, video

JUNG in Neo4j – Part 2

maxdemarzi — Fri, 16 Mar 2012 06:35:22 +0000

A few weeks ago I showed you how to visualize a graph using the chord flare visualization and how to visualize a network using a force directed graph visualization from D3.js.

On Twitter Claire Willett from Riparian Data asked:

Visualizing a 200-node network w/ #Cypher + #D3js –cool method, but maybe a node quilt would work better? ow.ly/92wDE @maxdemarzi—
(@RiparianData) February 13, 2012

This post on Graphs Beyond the Hairball by Robert Kosara explains why some non-traditional graph visualizations may work better and links us to an article explaining what a Node Quilt is and how it’s useful. We’re going to just take the first step and build a Matrix representation of a graph. We will use one of the JUNG clustering algorithms to help us understand it.

The study of social networks goes back to at least the ancient Greeks, but we won’t go back that far in time today… just to 1977. A man named Wayne Zachary recorded the interactions of a Karate Club at a University for 2 years. During this time a conflict developed between the administrator and the instructor and the club broke up into two. Turns out you could predict which club each member would belong to by building a graph of their weighted relationships and spliting it along the minimum-cut.

We’re working with Neo4j, and you all know Neo knows Kung Fu, so we’ll do something a little different. Plus looking at a two node cluster is kinda lame… let’s go bigger. We’re going to be using Neo4j with the JUNG jars already in the lib directory. Refer to JUNG in Neo4j – Part 1 if you need help with that. Let’s create our graph:

def create_graph
  neo = Neography::Rest.new
  graph_exists = neo.get_node_properties(1)
  return if graph_exists && graph_exists['name']

  names = %w[Aaron Achyuta Adam Adel Agam Alex Allison Amit Andreas Andrey 
             Andy Anne Barry Ben Bill Bob Brian Bruce Chris Corey 
             Dan Dave Dean Denis Eli Eric Esteban Ezl Fawad Gabriel 
             James Jason Jeff Jennifer Jim Jon Joe John Jonathan Justin 
             Kim Kiril LeRoy Lester Mark Max Maykel Michael Musannif Neil]

  commands = names.map{ |n| [:create_node, {"name" => n}]}

  names.each_index do |from| 
    commands << [:add_node_to_index, "nodes_index", "type", "User", "{#{from}}"]
    connected = []

    # create clustered relationships
    members = 5.times.collect{|x| x * 10 + (from % 10)}
    members.delete(from)
    rels = 1 + rand(4)
    rels.times do |x|
      to = members[x]
      connected << to
      commands << create_rel(from, to) unless to == from
    end    

    # create random relationships
    rels = 1 + rand(4)
    rels.times do |x|
      to = rand(names.size)
      commands << create_rel(from, to) unless (to == from) || connected.include?(to)
    end    
   end
   batch_result = neo.batch *commands
end

I am once again using the first 50 names from the Graph DataBase- Chicago Meet-Up group. You’ve seen me do this once or twice already, so I won’t go over it in detail, but as you can see above, I am forcing small clusters of relationships to exist along with random relationships. Our relationships have a weight property, so the create_rel method is just this:

def create_rel(x,y,z= 1 + rand(10))
  [:create_relationship, "knows", "{#{x}}", "{#{y}}", {:weight => z}]
end

Our visualization is expecting a list of nodes already in groups, and a list of relationships that includes it’s strength. The JSON object looks like this:

{"nodes":[{"name":"Myriel","group":1},
          {"name":"Napoleon","group":1},
          {"name":"Mlle.Baptistine","group":2}],
 "links":[{"source":1,"target":0,"value":1},
          {"source":2,"target":0,"value":8},
          {"source":3,"target":0,"value":10}]
}

Getting our nodes is pretty easy, we’ll use Gremlin to retrieve all but the root node.

def get_nodes
  neo = Neography::Rest.new
  neo.execute_script("g.V.filter{it.id != 0}.transform{[it.id,it.name]}")
end

Getting the relationships is also pretty ease, to switch it up, we’ll use Cypher.

def get_relationships
  neo = Neography::Rest.new
  cypher_query =  " START a = node:nodes_index(type='User')"
  cypher_query << " MATCH a-[r:knows]-b"
  cypher_query << " RETURN ID(a), ID(b), r.weight"
  neo.execute_query(cypher_query)["data"]
end

Now comes the fun part. To get our clusters, we’ll be using the Voltage Clusterer Class. We don’t want the root node getting in the way, so we’ll create a sub-graph using TinkerGraph that excludes it (you could do something similar to cluster just a small part of a larger graph). We then pass this graph on to GraphJung and set our VoltageClusterer to try to get 10 clusters for us.

def get_voltage_clusters
  neo = Neography::Rest.new
  lg = neo.execute_script("import edu.uci.ics.jung.algorithms.cluster.VoltageClusterer;
                            to = new TinkerGraph();
                            g.V.filter{it.id != 0}.sideEffect{toVertex = to.addVertex(it.getId()); 
                                           ElementHelper.copyProperties(it, toVertex);
                            }.iterate();   
                            g.E.sideEffect{toEdge = to.addEdge(it.getId(), 
                                                    to.v(it.getOutVertex().getId()), 
                                                    to.v(it.getInVertex().getId()),
                                                    it.getLabel());
                                            ElementHelper.copyProperties(it, toEdge);
                            }.iterate();
                            vc = new VoltageClusterer(new GraphJung(to), 10);
                            vc.cluster(10).id;
                            ")
end

We can then put it all together to build that JSON object.

get '/cluster' do
  clusters = Hash.new
  get_voltage_clusters.each_with_index {|item, index| item.each{|i| clusters[i.to_i] = index + 1} }
  nodes = get_nodes.map{|n| {"name" => n[1], "group" => clusters[n[0]]}}
  relationships = get_relationships.map{|r| {"source" => r[0] - 1, "target" => r[1] - 1, "value" => r[2]} }
  {:nodes => nodes, :links => relationships}.to_json
end

Our visualization was done by Mike Bostock with D3.js. As always, the code is on Github. Click on the image below to see the live example on Heroku.

Note: The version on Heroku is using a pre-generated JSON file.

Filed under: Cypher, Gremlin, Heroku, Visualization Tagged: cluster, cypher, d3.js, graph database, neo4j, visualization

JUNG in Neo4j – Part 1

maxdemarzi — Tue, 13 Mar 2012 16:34:09 +0000

It’s nice to have an arsenal. In the world of graph databases, one such stock room is the Java Universal Network/Graph Framework(JUNG) which contains a cache of algorithms from graph theory, data mining, and social network analysis, such as routines for clustering, decomposition, optimization, random graph generation, statistical analysis, and calculation of network distances, flows, and importance measures (centrality, PageRank, HITS, etc.).

We can use JUNG via the Blueprints ouplementation and access it via Gremlin. It doesn’t come pre-packaged with Neo4j, but Michael Hunger playing the role of “Tank” and was able to load up our stock room with a few key strokes.

We’ll create a directory for our project.

mkdir neojung
cd neojung

Let’s install Neography

echo "source 'http://rubygems.org'
gem 'neography' " > Gemfile
bundle install

… add the tasks, and install noe4j.

echo "require 'neography/tasks'" >> Rakefile
rake neo4j:install

At this point you have a barebones Neo4j installation in the neojung/neo4j directory. To add JUNG we need to build a few jar files and add them to the neo4j lib directory.

We are going to be using Maven, so install it if you don’t have it.

sudo apt-get install maven2

First thing we need to do is grab blueprints and build it.

git clone git://github.com/tinkerpop/blueprints.git
cd blueprints/blueprints-graph-jung/
mvn clean -DskipTests install

We also need to get a copy of the dependencies.

mvn dependency:copy-dependencies

Now we can copy all these jar files into Neo4j.

cp target/*.jar ../../neo4j/lib
cp target/dependency/*.jar ../../neo4j/lib

Finally we need to add executable permissions to these jar files and start Neo4j.

cd ../..
chmod +x neo4j/lib/*.jar
rake neo4j:start

If you bring up the Neo4j Power Tool Console on localhost:7474 and go into the Gremlin web shell you are now able to use JUNG algorithms. Let’s import the pagerank algorithm:

import edu.uci.ics.jung.algorithms.scoring.PageRank  
==>import com.tinkerpop.gremlin.*
==>import com.tinkerpop.gremlin.java.*
==>import com.tinkerpop.gremlin.pipes.*
...
==>import edu.uci.ics.jung.algorithms.scoring.PageRank

We’ll now create a small toy graph and run pagerank on it.

j = new GraphJung(TinkerGraphFactory.createTinkerGraph());
==> graphjung[tinkergraph[vertices:6 edges:6]]
pr = new PageRank(j, 0.15d)
==> edu.uci.ics.jung.algorithms.scoring.PageRank@8f2588
pr.evaluate()
==>
j.getVertices().collect{ [it, pr.getVertexScore(it)] }
==> [v[3], 0.30472082661863664]
==> [v[2], 0.14598540145985392]
==> [v[1], 0.11375485828040566]
==> [v[6], 0.11375485828040566]
==> [v[5], 0.1757986539008436]
==> [v[4], 0.14598540145985392]

If you have large graphs or low memory servers, you’ll want to heed this warning by Marko Rodriguez.

NOTE: JUNG is a library that was developed for in-memory graph structures. As such, many of the aspects of its various classes are memory based. For instance, given the above PageRank example, the method pageRank.getVertexScore() is pulling from an in-memory Map that contains the score for each vertex. If the number of vertices in the graph is large, then such in-memory structures will ultimately throw an OutOfMemoryError. As such, be wary of such situations when using GraphJung.

I’ll be writing more about JUNG and how to use some of its other algorithms. Be sure to
Follow @maxdemarzi to know as soon as they are posted.

Filed under: Gremlin, Java Tagged: graph database, gremlin, java, jung, neo4j, pagerank

Connections in Time

maxdemarzi — Fri, 09 Mar 2012 04:47:43 +0000

Some relationships change over time. Think about your friends from high school, college, work, the city you used to live in, the ones that liked you ex- better, etc. When exploring a social network it is important that we understand not only the strength of the relationship now, but over time. We can use communication between people as a measure.

I ran into a visualization that explored how multiple parties where connected by communications in multiple projects. We’re going to reuse it to explore how multiple people interact with each other. So let’s make a network of 50 friends and connect them to each other multiple times. Think of it as people writing on your facebook wall.

We will use the names of the first 50 members of the Graph DataBase- Chicago Meet-Up group for our names, but we’ll need a way to generate random times and offset them to simulate when people joined the social network.

def generate_time(from = Time.local(2004, 1, 1), to = Time.now)
  Time.at(from + rand * (to.to_f - from.to_f)).strftime('%Y-%m-%d')
end

def time_offset(n)
    Time.local(2004, 1, 1) + ((60*60*24*58) * n)
end

Let’s give our network a little something special. In your group of friends, most are quiet, some mingle, and some are social butterflies. We’ll create a random function with a power law distribution to model our connections and add a few outliers.

def powerlaw(min=1,max=500,n=20,o=0.05)
    max += 1
    pl = ((max**(n+1) - min**(n+1))*rand() + min**(n+1))**(1.0/(n+1))
    rand > o ? (max-1-pl.to_i)+min : rand(max).to_i
end

The code to create a relationship is pretty simple, we’ll use the Batch commands again and reference the nodes we create.

def create_rel(from,to,start_date,end_date)
  [:create_relationship, "wrote", "{#{from}}", "{#{to}}", {:date => generate_time(start_date,end_date)}]
end

Let’s put it together to create our graph. In order for our data to make sense, we are limiting the messaging between people to when they were both using the social network. You’ll see this in the maximum of the two nodes being passed into our time_offset method.

def create_graph
  neo = Neography::Rest.new
  graph_exists = neo.get_node_properties(1)
  return if graph_exists && graph_exists['name']
  commands = []
  names = %w[Aaron Achyuta Adam Adel Agam Alex Allison Amit Andreas Andrey 
             Andy Anne Barry Ben Bill Bob Brian Bruce Chris Corey 
             Dan Dave Dean Denis Eli Eric Esteban Ezl Fawad Gabriel 
             James Jason Jeff Jennifer Jim Jon Joe John Jonathan Justin 
             Kim Kiril LeRoy Lester Mark Max Maykel Michael Musannif Neil]

  commands = names.map{ |n| [:create_node, {"name" => n}]}

  names.each_index do |from|
    commands << [:add_node_to_index, "nodes_index", "type", "user", "{#{from}}"]  
    powerlaw.times do
      to = rand(50)
      commands << create_rel(from,to,time_offset([from,to].max),Time.now) 
    end
  end
  batch_result = neo.batch *commands
end

Our visualization was built using D3.js and it makes a web request expecting to see a JSON object that looks like:

{"cases":[{"title":"Aaron",
           "initiated_at":"2005-01-14",
           "last_correspondance_at":"2012-02-14",
           "exchanges":[{"incoming":true,
                         "sender_or_recipent":"2",
                         "journal_date":"2007-04-09"},
                        {"incoming":true,
                         "sender_or_recipent":"2",
                         "journal_date":"2008-10-02"}]}],
"parties":[{"id":1,"name":"Aaron","value":60},
          {"id":2,"name":"Achyuta","value":144}]}

We spent some time getting our data into our graph, now let’s get it all back out. Instead of getting everything in one shot, we’ll split the work into 3 queries. The first one called get_parties will get the users of the social network using Cypher. Notice that to get the ID of a node we use the ID() function and not me.id. We are also using the count, min and max cypher aggregate functions to get the data we need.

def get_parties
  neo = Neography::Rest.new
  cypher_query =  " START me = node:nodes_index(type = 'user')"
  cypher_query << " MATCH (me)-[r?:wrote]-()"
  cypher_query << " RETURN ID(me), me.name, count(r), min(r.date), max(r.date)"
  cypher_query << " ORDER BY ID(me)"
  neo.execute_query(cypher_query)["data"]
end

We’ll write another query to get the incoming relationships for each node. We are using the collect function to get two arrays back, one for the ids of friends and one for the date of the relationships.

def get_incoming_matrix
  neo = Neography::Rest.new
  cypher_query =  " START me = node:nodes_index(type = 'user')"
  cypher_query << " MATCH (me)<-[r?:wrote]-(friends)"
  cypher_query << " RETURN ID(me), me.name, collect(ID(friends)), collect(r.date)"
  cypher_query << " ORDER BY ID(me)"
  neo.execute_query(cypher_query)["data"]
end

A second query gives us the outgoing relationships for each node. Notice we are ordering both queries by the ID function.

def get_outgoing_matrix
  neo = Neography::Rest.new
  cypher_query =  " START me = node:nodes_index(type = 'user')"
  cypher_query << " MATCH (me)-[r?:wrote]->(friends)"
  cypher_query << " RETURN ID(me), me.name, collect(ID(friends)), collect(r.date)"
  cypher_query << " ORDER BY ID(me)"
  neo.execute_query(cypher_query)["data"]
end

Now we put it all together by combining the parties and their exchanges into a JSON object.

get '/communication' do
  p = get_parties
  parties = p.map{|p| {"id" => p[0], "name" => p[1], "value" =>p[2]} }
  cases = p.map{|p| {"title" => p[1], "initiated_at" => p[3], "last_correspondance_at" =>p[4], "exchanges" => []} }
  
  gim = get_incoming_matrix
  gim.each_index do |im|
    sors = gim[im][2][1..(gim[im][2].size - 2)].split(", ")
    jds  = gim[im][3][1..(gim[im][3].size - 2)].split(", ")
    sors.size.times do |t|
      cases[im]["exchanges"] <<  {"incoming" => true, "sender_or_recipent" => sors[t], "journal_date" => jds[t]}  
    end
  end

  gom = get_outgoing_matrix
  gom.each_index do |om|
    sors = gom[om][2][1..(gom[om][2].size - 2)].split(", ")
    jds  = gom[om][3][1..(gom[om][3].size - 2)].split(", ")
    sors.size.times do |t|
      cases[om]["exchanges"] <<  {"incoming" => false, "sender_or_recipent" => sors[t], "journal_date" => jds[t]}  
    end
  end
  {:cases => cases, :parties => parties}.to_json
end

The string wrangling you see to populate sors and jds is necessary because Cypher returns an Array wrapped inside a string instead of a proper array. One day we’ll get proper JSON objects back and this ugly little hack won’t be necessary. Update: We won’t have to wait long, the fix has been committed. All of the code is available on github as usual and you can play with it live on heroku.

You can use this visualization in many situations. Use it to visualize github commits to multiple projects by your development team. Conversations on Twitter about a range of Topics. Wikipedia page edits, patient records, resource utilization, etc.

Credits:
This post is based on the work of Even Westvang and his SeePlan series. He will be open sourcing his work soon. Watch this space for a link.

Filed under: Cypher, Heroku, Visualization Tagged: cypher, graph database, neo4j, network, ruby, visualization