In the last blog post we managed to run Neo4j at Ludicrous Speed over http using Undertow and get to about 8000 requests per second. If we needed more speed we can scale up the server or we can scale out to multiple servers by switching out the GraphDatabaseFactory and using the HighlyAvailableGraphDatabaseFactory class instead in Neo4j Enterprise Edition.
But can we go faster on a single server without new hardware? Well… yes, if we’re willing to drop http and switch to Web Sockets.
In the last blog post we saw how we could get about 1,250 requests per second (with a 10ms latency) using an Unmanaged Extension running inside the Neo4j server… but what if we wanted to go faster?
The easy answer is to Scale Up. However, trying to add more cores to my Apple laptop doesn’t sound like a good time. Another answer is running a Neo4j Cluster and (almost) linearly scaling our read requests as we add more servers. So a 3 server cluster would give us between 3,500 and 3,750 requests per second.
But can we go faster on a single server without new hardware? Well… yes.
Continue reading
I really like this saying by Corey Lanum:
Finding the relationships that should not be there is a great use case for Neo4j, and today I want to highlight an example of why. When you purchase something online, the merchant hands off your information to the payment gateway which processes your actual payment. Before they accept the transaction, they run it via series of risk management tests to validate that it is a real transaction and protect themselves from fraud. One of the hardest things for SQL based systems to do is cross check the incoming payment information against existing data looking for relationships that shouldn’t be there.
Continue reading
One of the benefits of Open Source Software is that if you want to change how something is done, you can. At Neo Technology, we have a small team of “Field Engineers” who don’t really work ON the product but rather WITH the product. We help our customers with issues of all kinds, answer questions, give suggestions and whatever we need to do to make people’s project successful. A little while back I had a support ticket for a traversal that was taking longer than they hoped it would.
Think about a social network, one of the things you may want to do is tell the user how big their friends network is. But why stop there? How about their friends of friends or even friends of friends of friends network? These are the kind of questions graph databases excel at compared to relational databases. Let’s take a look at what they were doing:
Continue reading
The expressive power of Cypher is already awesome and getting better with the Neo4j 2.0 release. Let’s take a step back from the bleeding edge and see Cypher in 1.9.4 and how it can be translated into Java. First a simple example where we look up a User node by an index and return a list of usernames belonging to the people who are that user’s friends:
START me = node:Users(username='maxdemarzi') MATCH me -[:FRIENDS]-> people RETURN people.username
The Cypher statement expresses what I want even better than my botched explanation in English. So how would we do this in the Neo4j Java API?
Continue reading
Connected: The Surprising Power of Our Social Networks and How They Shape Our Lives
Continue reading
Let’s add a couple of performance tests to the mix. We learned about Gatling in a previous blog post, we’re going to use it here again. The first test will randomly choose users and documents (from the graph we created in part 2) and write the results to a file, the second test will re-use the results of the first one and run consistently so we can change hardware, change Neo4j parameters, tune the JVM, etc. and see how they affect our performance.
The full code for the Random Permissions test is here, I’ll just highlight the main parts:
Continue reading
People produce a lot of content. Messages, text files, spreadsheets, presentations, reports, financials, etc, the list goes on. Usually organizations want to have a repository of all this content centralized somewhere (just in case a laptop breaks, gets lost or stolen for example). This leads to some kind of grouping and permission structure. You don’t want employees seeing each other’s HR records, unless they work for HR, same for Payroll, or unreleased quarterly numbers, etc. As this data grows it no longer becomes easy to simply navigate and a search engine is required to make sense of it all.
But what if your search engine returns 1000 results for a query and the user doing the search is supposed to only have access to see 4 things? How do you handle this? Check the user permissions on each file realtime? Slow. Pre-calculate all document permissions for a user on login? Slow and what if new documents are created or permissions change between logins? Does the system scale at 1M documents, 10M documents, 100M documents?
Continue reading
Did you know you can write Javascript in the Neo4j console to access the Neo4j API?
Try it. Open up your Neo4j Web Admin Console and type:
neo4j-sh (0)$ eval db
EmbeddedGraphDatabase [data/graph.db]
OMG! I know, Neo4j is crazy. So much to play with, I’ve been at it for a few years and I haven’t even dug into this area. What else can we do here?
Continue reading
In Extending Neo4j I showed you how to create an unmanaged extension to warm up the node and relationship caches. Let’s try doing something more interesting like exposing the A* (A Star) search algorithm through the REST API. The graph we created earlier looks like this:
Continue reading
Max De Marzi 