Max De Marzi

I may be remembering this wrong, but I think it was Henry Rollins who once asked, “What came first, the shitty Multi-Model Databases or the Drugs?” His confusion was over whether:

A) there were a bunch of developers dicking around with their Mac laptops and they wrote a shitty database, put it on github, posted on hacker news, and then other developers who were on drugs started using it or…

B) there were a bunch of developers on ketamine and ecstasy and somebody said lets write a shitty database

I think “A” is what probably happens and how we end up with over 300 databases on DB Engines. But what about “B” ? Well I don’t have any good stuff lying around, but I did hurt my foot the other day and the doctors gave me some Tramadol, so lets down some of that and see what happens.

Alright most multi-model databases start out from some key-value backing. There are plenty of choices to choose from, but I’m gonna go embedded first (an homage to Neo4j which started life as embedded only) so I’ll use ChronicleMap.

Chronicle Map is built by the Open HFT folks (Roman Leventov, Rob Austin, Peter Lawrey, and others) and is known to be pretty damn fast, so we’ll use that.

Now, I’ll need to handle Keys and their values, but those values can be Objects and those Objects can be connected together. Yes our multi-model database will be a KV Store, Object Database and Graph Database rolled into one shitty mess. Let’s call this thing ChronicleGraph and setup our storage.

public class ChronicleGraph {

    private static ChronicleMap<String, Object> nodes;

So we have this “nodes” Map that accepts any String as a Key and stores an Object. Now on to our addNode method, we need to accept both empty nodes and nodes with properties, but we’ll force the users to give us a primary key of some sort. This will make getting the node back easy.

    public boolean addNode (String key) {
        return addNode(key,"");
    }

    public boolean addNode (String key, Object properties) {
        nodes.put(key, properties);
        return true;
    }

Let’s test this out… does this pass?

    @Test
    public void shouldAddNode() {
        boolean created = cg.addNode("key");
        Assert.assertTrue(created);
        Assert.assertEquals("", cg.getNode("key"));
    }

Yes… what about with complex properties?

    @Test
    public void shouldAddNodeWithObjectProperties() {
        HashMap<String, Object> address = new HashMap<>();
        address.put("Country", "USA");
        address.put("Zip", "60601");
        address.put("State", "TX");
        address.put("City", "Chicago");
        address.put("Line1 ", "175 N. Harbor Dr.");
        HashMap<String, Object> properties = new HashMap<>();
        properties.put("name", "max");
        properties.put("email", "maxdemarzi@hotmail.com");
        properties.put("address", address);
        boolean created = cg.addNode("complex", properties);
        Assert.assertTrue(created);
        Assert.assertEquals(properties, cg.getNode("complex"));
    }

Yes, no problem. Ok so that was the easy part. How do we join these guys together in relationships though? First, let’s add another map for “relationships” which will store the relationship properties if any. We want to be able to store relationship properties, without those we’d be reduced to the most useless of all NoSQL databases the dreaded “RDFs”. Ewwwww.

    private static ChronicleMap<String, Object> relationships;
    private static HashMap<String, ChronicleMap<String, Set<String>>> related = new HashMap<>();

We are also adding a map of maps called “related” which will store our relationships by type and direction. To add a new relationship, we ask the user for the Type and the keys of the two nodes we’re connecting. If the related map doesn’t know about this relationship type, then we add the type before trying to create the relationship. This addRelationshipType will create two ChronicleMaps one for the Incoming and one for the Outgoing direction and our addEdge method is called twice to match each direction.

    public boolean addRelationship (String type, String from, String to) {
        if(!related.containsKey(type+"-out")) {
            addRelationshipType(type, DEFAULT_MAXIMUM_RELATIONSHIPS, 
                                DEFAULT_OUTGOING, DEFAULT_INCOMING);
        }
        addEdge(related.get(type + "-out"), from, to);
        addEdge(related.get(type + "-in"), to, from);

        return true;
    }

This means of course, that when we delete a node, we have to delete it from the nodes map as well as from both its incoming and outgoing relationships, and delete any relationship properties it may have had from both directions. That method was painful, so I’ll spare you the details but take a look at it here if you want.

Now, what good would this database be if it could not traverse. Let’s add a new method to get the nodes linked by an outgoing relationship (and we’ll do the same for incoming) by first getting the node ids in the type+”out” ChronicleMap of “related” and then using “nodes” to get their properties:

    public Set<Object> getOutgoingRelationshipNodes(String type, String from) {
        Set<Object> results = new HashSet<>();
        for (String key : related.get(type+"-out").get(from) ) {
            HashMap<String, Object> properties = new HashMap<>();
            properties.put("_id", key);
            properties.put("properties", nodes.get(key));
            results.add(properties);
        }
        return results;
    }

… and now for a crazy test:

 @Test
    public void shouldGetNodeOutgoingRelationshipNodes() {
        cg.addNode("one", 1);
        cg.addNode("two", "node two");

        HashMap<String, Object> node3props = new HashMap<> ();
        node3props.put("property1", 3);
        cg.addNode("three", node3props);

        cg.addRelationshipType("FRIENDS", 10000, 100, 100);
        cg.addRelationship("FRIENDS", "one", "two");
        cg.addRelationship("FRIENDS", "one", "three");
        Set<Object> actual = cg.getOutgoingRelationshipNodes("FRIENDS", "one");

        Set<Object> expected = new HashSet<Object>() {{
            add( new HashMap<String, Object>() {{
                put("_id", "two");
                put("properties", "node two");
            }});
            add( new HashMap<String, Object>() {{
                put("_id", "three");
                put("properties", node3props);
            }});
        }};
        Assert.assertEquals(expected, actual);
    }

Beautiful isn’t it. The Tramadol is wearing off, so we’ll stop here. Take a look at the source code on github as always. I’ll try to continue this later maybe by adding a web server and an API to go with it. In the meantime enjoy the shitty music and the drugs.

On to Part 2