Our own Multi-Model Database – Part 1

shittydb

I may be remembering this wrong, but I think it was Henry Rollins who once asked, “What came first, the shitty Multi-Model Databases or the Drugs?” His confusion was over whether:

A) there were a bunch of developers dicking around with their Mac laptops and they wrote a shitty database, put it on github, posted on hacker news, and then other developers who were on drugs started using it or…

B) there were a bunch of developers on ketamine and ecstasy and somebody said lets write a shitty database

I think “A” is what probably happens and how we end up with over 300 databases on DB Engines. But what about “B” ? Well I don’t have any good stuff lying around, but I did hurt my foot the other day and the doctors gave me some Tramadol, so lets down some of that and see what happens.
Continue reading

Tagged , , , , , , , ,

Connected

connected

The Stereo MC’s song “Connected” could be about some recently gained insight and the realization that maybe some of the people you held dear are phonies and while the reality of the situation is scary, you cannot allow yourself to turn a blind eye anymore or allow yourself to backslide by disconnecting from the real world.

Or it could be a warning about how we’ve all been blinded by SQL databases for too long and we must instead look to connect our data with Graph Databases. About how those new connections may be scary (like because of fraud detection) but they are necessary to better understand reality.

Either way, we may want to see if two nodes in Neo4j are connected and I’m going to show you how to do that faster.
Continue reading

Tagged , , , , , , , , ,

Enabling Legacy Automatic Full Text Search on Neo4j 3.x

solar-elastic-lucene

Neo4j 3.x has made inroads toward Full Text Search capabilities using Cypher keywords “STARTS WITH”, “ENDS WITH” and “CONTAINS”. However this search capability is limited to a single Schema Index and can be a problem when you need a very flexible search interface. To search across multiple models you can do this trick:
Continue reading

Tagged , , , , ,

News Feeds

Ron Burgundy Gets Hungry

Ron Burgundy (in Anchorman) gets Hungry

The “News Feed” is a core feature of social networks like Twitter, Facebook, or Vine (RIP). Let’s take a look at how we could model and implement this in Neo4j. Our social network needs Users (otherwise it would be kinda empty) that FOLLOW each other (otherwise it would not be very social). Those users need to POST some Messages (otherwise it would be boring). Here is our first attempt at a model (using Arrows):
Continue reading

Tagged , , , , , , ,

Catalogs and Hierarchies

street-samurai-catalog-large

When I was younger, friends and I would play a role playing game called “Shadowrun“. The game draws elements from science fiction, crime dramas, and magic and blends them all together to make a fun mess. You could be a Dwarf Shaman, an Elf Decker, a Human Rigger, an Orc Adept, a Troll Street Samurai or whatever combination your heart desired. Choosing a gender, race and archetype was just the beginning a more important question: “What is your character going to wear and take on missions?”
Continue reading

Tagged , , , , , , , ,

Delivering a Graph Based Search solution to slightly wrong data

oops

When it comes to databases, having good clean data is always important. More so with Graphs which deal with concepts as nodes and their relationships between them. Inevitably, you will run into messy data and have to deal with it. In a lot of the projects our customers work on they are dealing with connecting multiple data sources to get to a “golden record” or single source of truth. A lofty goal, sometimes impossible to achieve, but we can use the relationships of the data to help us come close.

One option is to extract the features (or tags) of a composite object and see if any other object shares most of these features. If that is the case then they are possibly the same object and should be merged instead of creating a new record. A partial subgraph match is something akin to a recommendation engine in Neo4j and pretty trivial to write. Take a look back at a few old blog posts for ideas.
Continue reading

Tagged , , , , , , , , , ,

Custom Importers

imports

When it comes to getting data into Neo4j, you have a ton of options. You can use LOAD CSV from Cypher, you can use the Import Tool, you can use the JDBC connector in APOC, and possibly a few more options I’m forgetting. Some of these require the data to be in a specific format, others that you write a little custom cypher. These work very well most of the time, but sometimes you run into data in weird shapes and coming in from vendors who aren’t willing to change just for you. What do you do in that case? Well, you write a custom importer. I’m going to show you how by importing the Cities database from MaxMind.
Continue reading

Tagged , , , , ,

Speeding up Traversals

roots

A few folks have come to us recently with the need to trace lineages of nodes of variable depth many hops away. You can run into this need if you are looking at the ancestries of living things, tracing data as it flows through an ETL, large network connectivity maps, etc. These types of queries tend to be murder on relational databases because of the massive recursive joins they have to deal with. Let’s give them a try in Neo4j.
Continue reading

Tagged , , , , ,

Scaling Cypher Writes

salt-pepa-writes

Let’s talk about writes, baby. Let’s talk about you and me. Let’s talk about all the good things. And the bad things that may be. Let’s talk about writes, and indexing and batching, and transactions in Neo4j. Let’s start with my environment. A 3 year old MacBook Pro (dying to get the new ones… once they finally come out) running a 4 core 2.3 GHz Intel Core i7 that is hyper-threading and pretending to have 8. An Apple SM256E SSD that is about average as far as SSDs go. So definitely not a production grade server, so bear that in mind.
Continue reading

Tagged , , , , , ,

Bidirectional Traversals in Space

firefly

If you have never watched Firefly, then stop whatever you are doing and get to it, you can come back and read this post later. Ok good, now where were we. Firefly. The series is set a few hundred years from now, after people begin to terraform a new star system and it follows the adventures of the renegade crew of Serenity, a “Firefly-class” spaceship whose work consists of cargo runs or smuggling while failing to stay out of trouble. There is no faster than light travel in this series, so ships can’t just “warp” where ever they want. Instead they travel about from planets and moons, exchanging cargo, refueling and trying to make a living. We are going to model “The Verse” of Firefly in Neo4j, and see how we can find routes to move our illicit cargo from one place to another.
Continue reading

Tagged , , , , , , , , ,