Problems | Max De Marzi

Jan 18 2017

Our own Multi-Model Database – Part 5

shitty5

In part 4 I promised metrics and a shell, so that’s what we’ll tackle today. We are lucky that the Metrics library can be plugged into Jooby without much effort… and double lucky that the Crash library can also be plugged into Jooby without much effort. This is what we are all about here because we’re a bunch of lazy, impatient developers who are ignorant of the limits of our capabilities and who would rather reuse open source code instead of falling victim to the “Not Invented Here” syndrome and do everything from scratch.
Continue reading →

Tagged graph, graph database, graph databases, java, network, performance, software, technology, visualization

Jan 10 2017

1 Comment

Java, Problems

Our own Multi-Model Database – Part 4

shitty4

Please read parts 1, 2 and 3 before continuing or you’ll be lost.

We started adding an HTTP server to our database last time and created just a couple of end points. Today we’ll finish out the rest of the end points. We’ll also be good open source developers by hooking in Continuous Integration , Test Coverage and Continuous Deployment.

Continue reading →

Tagged graph database, graph databases, java, network, performance, relationship graph, software, technology, testing

Jan 06 2017

2 Comments

Java, Problems

Multi-Threading a Traversal

multi-threads

What would you think if I ran out of time,
Would you stand up and walk out on me?
Lend me your eyes and I’ll write you a post
And I’ll try not to run out of memory.

Oh, I get by with a little help from my threads
Mm, I get high with a little help from my threads
Mm, gonna try with a little help from my threads

Today we are going to take a look at how to take a Neo4j traversal and split it up into lots of smaller traversals. I promise it will be electrifying.
Continue reading →

Tagged graph, graph database, graph databases, java, neo4j, network, performance, technology

Jan 02 2017

3 Comments

Java, Problems, Random, Testing

Our own Multi-Model Database – Part 3

shitty3

If you haven’t read part 1 and part 2 then do that first or you’ll have no clue what I’m doing, and I’d like to be the only one not knowing what I’m doing.

We’ve built the beginnings of this database but so far it’s just a library and for it to be a proper database we need to be able to talk to it. Following the Neo4j footsteps, we will wrap a web server around our database and see how it performs.

There are a ton of Java based frameworks and micro-frameworks out there. Not as bad as the Javascript folks, but that still leaves us with a lot of choices. So as any developer would do I turn to benchmarks done by other people of stuff that doesn’t apply to me, and you won’t believe what I found –scratch that, yes you will, I got benchmarks.
Continue reading →

Tagged github, graph, graph database, java, relationship graph, software, technology, testing

Dec 31 2016

5 Comments

Java, Problems, Random

OUR OWN MULTI-MODEL DATABASE – PART 2

shitty2

If you haven’t read part 1 then do that first or this won’t make sense, well nothing makes sense but this specially won’t.

So before going much further I decided to benchmark our new database and found that our addNode speed is phenomenal, but it was taking forever to create relationships. See some JMH benchmarks below:

Benchmark                                                           Mode  Cnt     Score     Error  Units
ChronicleGraphBenchmark.measureCreateEmptyNodes                    thrpt   10  1548.235 ± 556.615  ops/s
ChronicleGraphBenchmark.measureCreateEmptyNodesAndRelationships    thrpt   10     0.165 ±   0.007  ops/s

Each time I was creating 1000 users, so this test shows us we can create over a million empty nodes in one second. Yeah ChronicleMap is damn fast. But then when I tried to create 100 relationships for each user (100,000 total) it was taking forever (about 6 seconds). So I opened up YourKit and you won’t believe what I found out next (come on that’s some good clickbait).
Continue reading →

Tagged github, graph, graph database, performance, relationship graph, software, technology, testing

Dec 27 2016

Connected

The Stereo MC’s song “Connected” could be about some recently gained insight and the realization that maybe some of the people you held dear are phonies and while the reality of the situation is scary, you cannot allow yourself to turn a blind eye anymore or allow yourself to backslide by disconnecting from the real world.

Or it could be a warning about how we’ve all been blinded by SQL databases for too long and we must instead look to connect our data with Graph Databases. About how those new connections may be scary (like because of fraud detection) but they are necessary to better understand reality.

Either way, we may want to see if two nodes in Neo4j are connected and I’m going to show you how to do that faster.
Continue reading →

Tagged big data, graph, graph database, graph databases, java, neo4j, performance, relationship graph, software, technology

Oct 28 2016

2 Comments

Cypher, Problems

News Feeds

Ron Burgundy (in Anchorman) gets Hungry

The “News Feed” is a core feature of social networks like Twitter, Facebook, or Vine (RIP). Let’s take a look at how we could model and implement this in Neo4j. Our social network needs Users (otherwise it would be kinda empty) that FOLLOW each other (otherwise it would not be very social). Those users need to POST some Messages (otherwise it would be boring). Here is our first attempt at a model (using Arrows):
Continue reading →

Tagged graph, graph database, neo4j, network, news feed, nosql, relationship graph, social network

Oct 27 2016

Catalogs and Hierarchies

When I was younger, friends and I would play a role playing game called “Shadowrun“. The game draws elements from science fiction, crime dramas, and magic and blends them all together to make a fun mess. You could be a Dwarf Shaman, an Elf Decker, a Human Rigger, an Orc Adept, a Troll Street Samurai or whatever combination your heart desired. Choosing a gender, race and archetype was just the beginning a more important question: “What is your character going to wear and take on missions?”
Continue reading →

Tagged catalog, graph database, graph databases, hierarchy, neo4j, nosql, performance, relationship graph, technology

Sep 28 2016

2 Comments

Java, Problems, Random

Delivering a Graph Based Search solution to slightly wrong data

When it comes to databases, having good clean data is always important. More so with Graphs which deal with concepts as nodes and their relationships between them. Inevitably, you will run into messy data and have to deal with it. In a lot of the projects our customers work on they are dealing with connecting multiple data sources to get to a “golden record” or single source of truth. A lofty goal, sometimes impossible to achieve, but we can use the relationships of the data to help us come close.

One option is to extract the features (or tags) of a composite object and see if any other object shares most of these features. If that is the case then they are possibly the same object and should be merged instead of creating a new record. A partial subgraph match is something akin to a recommendation engine in Neo4j and pretty trivial to write. Take a look back at a few old blog posts for ideas.
Continue reading →

Tagged big data, graph database, graph search, java, neo4j, network, performance, relationship graph, search, software, technology

Sep 26 2016

Custom Importers

imports

When it comes to getting data into Neo4j, you have a ton of options. You can use LOAD CSV from Cypher, you can use the Import Tool, you can use the JDBC connector in APOC, and possibly a few more options I’m forgetting. Some of these require the data to be in a specific format, others that you write a little custom cypher. These work very well most of the time, but sometimes you run into data in weird shapes and coming in from vendors who aren’t willing to change just for you. What do you do in that case? Well, you write a custom importer. I’m going to show you how by importing the Cities database from MaxMind.
Continue reading →

Tagged graph database, import, maxmind, neo4j, nosql, performance

Max De Marzi

Graphs with Neo4j

Category Archives: Problems

Our own Multi-Model Database – Part 5

Our own Multi-Model Database – Part 4

Multi-Threading a Traversal

Our own Multi-Model Database – Part 3

OUR OWN MULTI-MODEL DATABASE – PART 2

Connected

Catalogs and Hierarchies

Delivering a Graph Based Search solution to slightly wrong data

Custom Importers