Max De Marzi

A few days ago I taught the Neo4j Modeling class in Dallas… well my own version of the class since I teach some of the older material plus some of my blog posts. If you ever get the chance, take the class, it will open your eyes at what is possible when “third normal form” gets thrown out the window and a whole new world of possibilities is before you. The next modeling class I’m teaching will be at Graph Connect in NYC on September 20-21, 2018. I recommend you join me, unless of course you are looking at this after it occurred, then catch the next one.

Regardless, I am to the point where I want to model messaging. There are a couple of ways of doing it. The first one is the simplest:

A user node has a MESSAGED relationship to another user node, the message and the time are stored as properties on the relationship and that’s it. It’s really easy to understand, but there is a problem with this model. As time grows and our user starts to have more conversations with various people, their node will be full of these MESSAGED relationships. How do we known which ones are new? We would have to traverse them all, get their “when” property, sort all the messages by time and then show the user the most recent ones… this will make our query slower and slower as we add more data, and we want to avoid that. So what do we do? We could try “dated” relationship types:

By “promoting” the date property up to the relationship type, we can traverse just to the most recent messages and since every node in Neo4j knows which relationship types are connected to it in BOTH directions (and their counts) it will make a “get recent messages” query fast. But we don’t want a sorted list of most recent messages. We want a sorted list of the most recent messages grouped by the users involved. So in this case switching to “dated” relationship types doesn’t help us. We need to try a different strategy. What if we promoted what is really happening into a node. In this case I’m talking about “conversations”:

By connecting both users to a Conversation node and then adding the messages to it, we can get the grouping of our messages by user that we want. But what about getting the most recent messages from this conversation? Should we change the ADDED_TO relationship to a “dated” relationship type? Well… how many messages would two people on a dating site exchange before they switch to another medium like a phone call, text messages or a date? According to some sources, about a dozen messages should be exchanged before asking for the digits. So in a “worst case” scenario of very chatty users, we’re probably looking at less than a thousand messages for sure, probably less than 50 in the great majority of cases, so I do not think it is worth it to bother with that.

This model choice opens up some interesting possibilities. Can more than 2 users be involved in a conversation? Maybe not for this feature, but what if we wanted to add Groups to our dating site. Then we could use the same ideas and connect multiple people to a conversation. This reminded me that we have a Stream… so we could get fancy and add things like Event nodes, where someone proposes a get together (like going to a zombie pub crawl, going to see a show, or for the movie lovers… Netflix and Chill). These events could be public and have “IM_GOING” relationships or they could require approval, so whoever created the event would moderate who is coming and private information about the event could be revealed once approved. We have lots of possibilities with a Dating Stream rather than just profiles. But let’s not get ahead of ourselves, we have to get messaging in first.

Let’s start with creating the messages and conversations and then we’ll figure out how to display them.

 
@Path("/users/{username}/conversations")
public class Conversations {

    @POST
    @Path("/{username2}")
    public Response addToConversation(String body,
                                      @PathParam("username") final String username,
                                      @PathParam("username2") final String username2,
                                      @Context GraphDatabaseService db) throws IOException {

We will use the PostValidator to clean the input like before and then find our two users in the graph:

 
        Map<String, Object> results;
        HashMap<String, Object> input = PostValidator.validate(body);
        ZonedDateTime dateTime = ZonedDateTime.now(utc);

        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            Node user2 = Users.findUser(username2, db);

Once we have our users, we need to make sure the user receiving the message has not blocked the user sending the message. We should probably double check all avenues of communication on both the creation and retrieval against blocked users.

 
            HashSet<Node> blocked = new HashSet<>();
            for (Relationship r1 : user2.getRelationships(Direction.OUTGOING, RelationshipTypes.BLOCKS)) {
                blocked.add(r1.getEndNode());
            }
            if (blocked.contains(user)) {
                throw ConversationExceptions.conversationNotAllowed;
            }

Next we need to find their conversation. We could have used a composite index or NODE KEY as Neo4j calls it on Conversation, but the likeliness of anyone talking to millions of people is zero. I would expect the average to be under 100 with a thousand or so at the top. So what we can do instead is simply traverse the PART_OF relationship twice, looking for the second user.

 
            Node conversation = null;
            outerloop:
            for (Relationship r1 : user.getRelationships(Direction.OUTGOING, RelationshipTypes.PART_OF)) {
                conversation = r1.getEndNode();
                for (Relationship r2 : conversation.getRelationships(Direction.INCOMING, RelationshipTypes.PART_OF)) {
                    if (user2.equals(r2.getStartNode())) {
                        break outerloop;
                    }
                }
            }

We can optimize this part by finding the user with the least amount of PART_OF relationships and traversing those looking for the other user. We can do this using the getDegree method:

 
if (user.getDegree(RelationshipTypes.PART_OF, Direction.OUTGOING) 
    < user2.getDegree(RelationshipTypes.PART_OF, Direction.OUTGOING)) {

What if we can’t find a conversation? Well then we have to create one, but we need to check we are allowed to message this user. According to our rules, they need to have given a High Five to one of our posts in the last 5 days. So let’s get the user’s posts:

 
            if (conversation == null) {
                ZoneId zoneId = ZoneId.of((String) user.getProperty(TIMEZONE));
                ZonedDateTime startOfFiveDays = ZonedDateTime.now(zoneId).with(LocalTime.MIN).minusDays(5);

                // Get their posts
                ArrayList<RelationshipType> types = new ArrayList<>();
                for (RelationshipType t : user.getRelationshipTypes()) {
                    if (t.name().startsWith("POSTED_ON")) {
                        types.add(t);
                    }
                }

                types.sort(relTypeComparator);

…and check for a high five from them. We only need to find one, so we break out of the loop as soon as we do. We are going to assume it is more likely that user2 high fived a recent post, so we will order our POSTED_ON relationship types before traversing them.

 
                boolean allowed = false;
                outerloop:
                for (Relationship r1 : user.getRelationships(types.toArray(new RelationshipType[0]))) {
                    Node post = r1.getEndNode();
                    for (Relationship r : post.getRelationships(RelationshipTypes.HIGH_FIVED, Direction.INCOMING)) {
                        // Check the user first, then get the time
                        if (user2.equals(r.getStartNode())) {
                            ZonedDateTime when = (ZonedDateTime)r.getProperty(TIME);
                            if (when.isAfter(startOfFiveDays)) {
                                allowed = true;
                                break outerloop;
                            }
                        }
                    }
                }

If we find a recent high five, we go ahead and create the conversation and make both users part of it, otherwise we deny the request.

 
                if (allowed) {
                    conversation = db.createNode(Labels.Conversation);
                    user.createRelationshipTo(conversation, RelationshipTypes.PART_OF);
                    user2.createRelationshipTo(conversation, RelationshipTypes.PART_OF);
                } else {
                    throw ConversationExceptions.conversationNotAllowed;
                }

Now you may be thinking, couldn’t we have gone from user2 and look for the high five relationships, filter them by date and then see if any of them belong to the first user? Yes we could have. If we decide not to keep the “expired” high fives and delete them instead, it would make more sense go this route. We may end up changing our query to do that… or we could check the sum of the number of posted relationships from the first user against the number of high fives from the second user and choose a different path on a case by case basis. As a monetization feature, we could have high fives that last longer than 5 days, maybe 10 days, or they could last forever, but that takes away the urgency to respond, so maybe doubling their time is better than infinite time. We also probably want to keep expired high fives to help us build recommendations.

We are getting ahead of ourselves again, for now we need to just create the actual Message, and connect it to our Conversation:

 
            Node message = db.createNode(Labels.Message);
            message.setProperty(STATUS, input.get(STATUS));
            message.setProperty(TIME, dateTime);
            message.setProperty(AUTHOR, username);
            message.createRelationshipTo(conversation, RelationshipTypes.ADDED_TO);

            results = message.getAllProperties();
            tx.success();

Ok, so we can create messages and conversations, but we also need to be able to see them. For this we need a getConversation method:

 
    @GET
    @Path("/{username2}")
    public Response getConversation(@PathParam("username") final String username,
                                    @PathParam("username2") final String username2,
                                     @QueryParam("limit") @DefaultValue("25") final Integer limit,
                                     @QueryParam("since") final String since,
                                     @Context GraphDatabaseService db) throws IOException {

It turns out the methods are pretty similar so I won’t go over it in as much detail. The real difference is that once we have our conversation, we need to get the messages, sort them and return them.

 
            for (Relationship r1 :  conversation.getRelationships(Direction.INCOMING, RelationshipTypes.ADDED_TO)) {
                Node message = r1.getStartNode();
                if (latest.isAfter((ZonedDateTime) message.getProperty(TIME))) {
                    results.add(message.getAllProperties());
                }
            }
            tx.success();
        }

        results.sort(timedComparator);

Turning to our front end, we can add them both to our API:

 
    @GET("users/{username}/conversations/{username2}")
    Call<List<Message>> getConversation(@Path("username") String username,
                                       @Path("username2") String username2);

    @POST("users/{username}/conversations/{username2}")
    Call<Message> createMessage(@Path("username") String username,
                                @Path("username2") String username2,
                                @Body Message message);

…and then add them to our application. I’ll skip that for now since it’s pretty straight forward, but take a look at the source code if you are interested. So lets see what our work yielded:

Nice! Our users can now send messages to each other… there is only one problem, we haven’t written a way to show the person who high fived a post that their crush has started a conversation with them. We will add that next.