Max De Marzi

In the movie “Field of Dreams“, a voice can be heard saying “If you build it, he will come“. For some reason people got that mixed up with “if you build it, they will come” and it became a bit of a trap that many engineers fall for. The myth being that if you build a better contraption everyone will want to use it. But that is not how the world works. You have to win the hearts and minds of the people who may want to use your product… just ask Apple which posted record results yet again.

Jamie Brandon wrote a monster of a blog post the other day against SQL. This is my favorite part:

To take an example close to my heart: Differential dataflow is a dataflow engine that includes support for automatic parallel execution, horizontal scaling and incrementally maintained views. It totals ~16kloc and was mostly written by a single person. Materialize adds support for SQL and various data sources. To date, that has taken ~128kloc (not including dependencies) and I estimate ~15-20 engineer-years. Just converting SQL to the logical plan takes ~27kloc, more than than the entirety of differential dataflow.

Similarly, sqlite looks to have ~212kloc and duckdb ~141kloc. The count for duckdb doesn’t even include the parser that they (sensibly) borrowed from postgres, which at ~47kloc is much larger than the entire ~30kloc codebase for lua.

Jamie Brandon – Against SQL

I don’t know why, but I need things to be fast. Nothing drives me up the wall more than waiting a long time for some database query to finish. It’s some kind of disease I tell you. So today we’re doing to do a little performance checking to see where RageDB is at. So far all we can do is create nodes and relationships, but that’s enough to get us started.

Good relationships are hard. They don’t just happen. They take time, patience and about two thousand lines of code to work together. I’m not convinced I have it right, maybe one of you out there has a better design we can implement. In the original design I was storing the full relationships complete with starting/ending node ids, properties and type information. This time Relationships are only temporarily created when requested, and we’re just going to store their pieces in different vectors. Let’s dive in to the code:

Whatever your views on the game Monopoly, you play by getting properties, changing them and making sure anybody that uses them pays a tidy sum. That’s also true of graph databases. Finding a property, changing a property, filtering on a property and sometimes even retrieving properties can be really expensive. Part of the issue is that it was decided at some point that it would be great if any nodes of the same label could have different properties and if they had the same property keys, they didn’t necessarily have to have the same property types. This mean one Person node could have a height property and the other not. One node’s height property could be an integer representing inches or centimeters while another could use a float and another just write out “170 cm” or ” 5 foot 6″ as strings. Dealing with properties was completely left to the developer. Many times, there isn’t even an option to enforce property types.

When I was first introduced to graph databases I had a hard time trusting them. When a node gets created, where does it go? There are no tables in graph databases, so I was missing that loving embrace, I mean arrangement of rows and columns. It made me a little paranoid, like what if I lost them? I built some projects storing all the data in Postgres first, so if anything happened to the graph I could rebuild it. That warm protective security blanket is something we’re bringing back. We’re taking another walk through a door and arranging all nodes and properties of each type in a set of Vectors (well, one per Shard of course).