Visualizing the news with Vivagraph.js

neo_news

Today I want to introduce you to VivaGraphJS – a JavaScript Graph Drawing Library made by Andrei Kashcha of Yasiv. It supports rendering graphs using WebGL, SVG or CSS formats and currently supports a force directed layout. The Library provides an API which tracks graph changes and reflect changes on the rendering surface which makes it fantastic for graph exploration.

Today we will be integrating it with Neo4j and the Alchemy API.

alchemyapi

The Alchemy API provides a set of natural language processing tools, and we’ll make use of their Entity Extraction capabilities.

We are going to take a look at the news of the world, extract the entities mentioned in them, connect them all together in Neo4j and visualize them. To get the news, we’ll use Feedzilla.

feedzilla

A sidekiq job will run every six hours to collect the latest 100 news articles:

module Job
  class GetNews
    include Sidekiq::Worker
    sidekiq_options :retry => false

    def perform
      Job::GetNews.perform_in(6.hours)

      feed = HTTPClient.get("http://api.feedzilla.com/v1/categories/19/articles.json?order=date&count=100&title_only=1")
      parsed_feed = Oj.load(feed.body)
      parsed_feed["articles"].each do |article|
        Job::GetArticle.perform_async(article["url"])
      end
    end

  end
end

From each article, we’ll get the story and pass it on to Alchemy API and ask it to find the entities in the article.

@entities = Oj.load(HTTPClient.post_content("http://access.alchemyapi.com/calls/url/URLGetRankedNamedEntities",
      {:url => article_url, 
       :apikey => ENV['ALCHEMY_API'],
       : outputMode => "json"}),
       {:'Accept-encoding' => "gzip"})["entities"]

We’ll create a node for each article and connect each entity found by a “MENTIONED” relationship:

commands = []
@batch_result.each do |b|
  commands << [:create_relationship, "MENTIONED", @article_node, b["body"]["self"].split("/").last]
end

Some of these entities have interesting descriptions, so we’ll query dbpedia for more information.

dbpedia_logo

Moving on to our front-end application, we’ll provide an auto-complete search box which will query the Entities index of our graph:

  get '/search' do 
    content_type :json
    neo = Neography::Rest.new    

    cypher = "START me=node:entities({query}) 
              RETURN ID(me), me.text
              ORDER BY me.text
              LIMIT 15"

    neo.execute_query(cypher, {:query => "text:*#{params[:term]}* OR uri:*#{params[:term]}*" })["data"].map{|x| { label: x[1], value: x[0]}}.to_json   
  end

We’ll also provide an end-point to pull a node and all the nodes connected to it and return a JSON object with this data:

get '/edges/:id' do
    content_type :json
    neo = Neography::Rest.new    

    cypher = "START me=node(#{params[:id]}) 
              MATCH me -- related
              RETURN ID(me), me.text, me.description, me.type, ID(related), related.text, related.description, related.type"

    connections = neo.execute_query(cypher)["data"]   
    connections.collect{|n| {"source" => n[0], "source_data" => {:label => n[1], 
                                                                 :description => n[2],
                                                                 :type => n[3] },
                             "target" => n[4], "target_data" => {:label => n[5], 
                                                                 :description => n[6],
                                                                 :type => n[7]}} }.to_json
  end

Our javascript will contain the calls to Vivagraph:

var graph = Viva.Graph.graph();	

  var layout = Viva.Graph.Layout.forceDirected(graph, {
      springLength:100,
      springCoeff:0.0001,
      dragCoeff:0.02,
      gravity:-1
  });	

Our nodes will display both the Text found in the entity as well as an image to represent the entity type.

var ui = Viva.Graph.svg('g'),
        svgText = Viva.Graph.svg('text').attr('y', '-4px').text(node.data.label),
        img = Viva.Graph.svg('image')
           .attr('width', 32)
           .attr('height', 32)
           .link('/img/' + node.data.type + '.png');

    ui.append(svgText);
    ui.append(img);

We’ll add a mouseover event to update the side-panel with the description of our node:

$(ui).hover(function() { // mouse over
	                    highlightRelatedNodes(node.id, true);
	                    $('#explanation').html(node.data.description);
	                }, function() { // mouse out
	                    highlightRelatedNodes(node.id, false);
	                });

…and to load the nodes related to this entity on click:

$(ui).click(function() { 
				        console.log("click", node);
			        	if (!node || !node.position) return;
			        	renderer.rerender();
			        	loadData(graph,node.id);
			}
	);

As usual, the code is available on github, and you can see it running live on Heroku.

Check out some more of the Vivagraph.js demos:

Tagged , , , , , , ,

2 thoughts on “Visualizing the news with Vivagraph.js

  1. I really love it :-)
    I am very impressed. Wish you great success with it…

  2. […] Visualizing the news with Vivagraph.js (maxdemarzi.com) […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 1,579 other followers

%d bloggers like this: