Max De Marzi

In part four, we continued cloning Twitter by adding hashtag and mentions functionality. Then we went beyond it by adding the ability to edit a post. So we have a social network where people can follow each other and post stuff. Today we’re adding the ability to say a user likes a post, reposts a post and the most important query of all, being finally able to see our feed or timeline.

To create a like, we find the user liking and the author of the post, then we use get getPost method we created in the last blog post to find the post we want to like. Before we can add a LIKES relationship, we have to first check if one already exists. That’s what he userLikesPost method does below. I’m not going to show you that since we see the same code in removeLike a little bit later. From there it is pretty straight forward. We create the LIKES relationship, add timestamps and return the result.

 
    @POST
    @Path("/{username2}/{time}")
    public Response createLike(@PathParam("username") final String username,
                               @PathParam("username2") final String username2,
                               @PathParam("time") final Long time,
                               @Context GraphDatabaseService db) throws IOException {
        Map<String, Object> results;

        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            Node user2 = Users.findUser(username2, db);
            Node post = getPost(user2, time);

            if (userLikesPost(user, post)) {
                throw LikeExceptions.alreadyLikesPost;
            }

            Relationship like = user.createRelationshipTo(post, RelationshipTypes.LIKES);
            LocalDateTime dateTime = LocalDateTime.now(utc);
            like.setProperty(TIME, dateTime.toEpochSecond(ZoneOffset.UTC));
            results = post.getAllProperties();
            results.put(USERNAME, user2.getProperty(USERNAME));
            results.put(NAME, user2.getProperty(NAME));
            results.put(LIKES, post.getDegree(RelationshipTypes.LIKES));
            results.put(REPOSTS, post.getDegree(Direction.INCOMING)
                    - 1 // for the Posted Relationship Type
                    - post.getDegree(RelationshipTypes.LIKES)
                    - post.getDegree(RelationshipTypes.REPLIED_TO));
            results.put(LIKED, true);
            results.put(REPOSTED, userRepostedPost(user, post));
            tx.success();
        }
        return Response.ok().entity(objectMapper.writeValueAsString(results)).build();
    }

Now our users are armed with the LIKES functionality. On Twitter most posts don’t get any likes, on Facebook however, it seems most post have some likes. The alternative way to model likes is to skip the relationship altogether and use a set of bitmaps. One bitmap per user where we store the post node ids they liked, and one bitmap per post where we store the user node ids of the users who liked the post. We have seen an example of half of this before on a previous post creating one way relationships. If we used a bitmap we’d lose the order of the likes. So another alternative is to use a fast integer array compression library which lets us keep the order, but not the LIKES times… We could use the Post times as a proxy. For now we will continue with proper relationships.

We could also build the functionality to unlike a post. The missing “thumbs down” button in Facebook and “broken heart” in Twitter. It may be a useful feature when building a bulletin board system, but we’ll skip it for now. Instead we need to add the ability to remove a like when somebody accidentally clicked on a like or later decided they didn’t actually like what the author had to say or what they linked to. We start off the same was as creating a like, but then must find the relationship between the user and the post. We are going to be smart about this by looking at all the likes of the user or all the likes of the post based on which one has the least amount of relationships and try to find the other node. Once we do, we can delete that relationship and we are good to go. The userLikesPost method mentioned earlier just returns true or false instead of deleting the relationship.

 
    @DELETE
    @Path("/{username2}/{time}")
    public Response removeLike(@PathParam("username") final String username,
                               @PathParam("username2") final String username2,
                               @PathParam("time") final Long time,
                               @Context GraphDatabaseService db) throws IOException {
        boolean liked = false;
        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            Node user2 = Users.findUser(username2, db);
            Node post = getPost(user2, time);

            if (user.getDegree(RelationshipTypes.LIKES, Direction.OUTGOING)
                    < post.getDegree(RelationshipTypes.LIKES, Direction.INCOMING) ) {
                for (Relationship r1 : user.getRelationships(Direction.OUTGOING, RelationshipTypes.LIKES)) {
                    if (r1.getEndNode().equals(post)) {
                        r1.delete();
                        liked = true;
                        break;
                    }
                }
            } else {
                for (Relationship r1 : post.getRelationships(Direction.INCOMING, RelationshipTypes.LIKES)) {
                    if (r1.getStartNode().equals(user)) {
                        r1.delete();
                        liked = true;
                        break;
                    }
                }
            }
            tx.success();
        }

        if(!liked) {
            throw LikeExceptions.notLikingPost;
        }

        return Response.noContent().build();
    }

We are almost there with our LIKES functionality. We also need to get all the likes of our user. This is going to flow just like our getFollowers method seen in part 3. From our user, we will traverse out the LIKES relationships. The “limit” and “since” will trim and order our output. I realized when I wrote this that I would need to account for seeing the LIKES of a user when logged on, and when browsing anonymously. To accomplish this we will add a “username2” parameter and with it find out if our browsing user has also liked or reposted the posts on our list.

 
@Path("/users/{username}/likes")
public class Likes {

    private static final ObjectMapper objectMapper = new ObjectMapper();

    @GET
    public Response getLikes(@PathParam("username") final String username,
                             @QueryParam("limit") @DefaultValue("25") final Integer limit,
                             @QueryParam("since") final Long since,
                             @QueryParam("username2") final String username2,
                             @Context GraphDatabaseService db) throws IOException {
        ArrayList<Map<String, Object>> results = new ArrayList<>();
        LocalDateTime dateTime;
        if (since == null) {
            dateTime = LocalDateTime.now(utc);
        } else {
            dateTime = LocalDateTime.ofEpochSecond(since, 0, ZoneOffset.UTC);
        }
        Long latest = dateTime.toEpochSecond(ZoneOffset.UTC);

        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            Node user2 = null;
            if (username2 != null) {
                user2 = Users.findUser(username2, db);
            }
            for (Relationship r1 : user.getRelationships(Direction.OUTGOING, RelationshipTypes.LIKES)) {
                Node post = r1.getEndNode();
                Map<String, Object> properties = post.getAllProperties();
                Long time = (Long)r1.getProperty("time");
                if(time < latest) {
                    Node author = getAuthor(post, (Long)properties.get(TIME));
                    properties.put(LIKED_TIME, time);
                    properties.put(USERNAME, author.getProperty(USERNAME));
                    properties.put(NAME, author.getProperty(NAME));
                    properties.put(HASH, author.getProperty(HASH));
                    properties.put(LIKES, post.getDegree(RelationshipTypes.LIKES));
                    properties.put(REPOSTS, post.getDegree() - 1 - post.getDegree(RelationshipTypes.LIKES));
                    if (user2 != null) {
                        properties.put(LIKED, userLikesPost(user2, post));
                        properties.put(REPOSTED, userRepostedPost(user2, post));
                    }
                    results.add(properties);
                }
            }
            tx.success();
        }

        results.sort(Comparator.comparing(m -> (Long) m.get(LIKED_TIME), reverseOrder()));

        return Response.ok().entity(objectMapper.writeValueAsString(
                results.subList(0, Math.min(results.size(), limit))))
                .build();
    }

We’re going to have to add this LIKED and REPOSTED functionality to our getPosts and getMentions methods as well. Really wherever we return a Post.

Speaking of posts, lets add the createRepost method. It’s pretty similar to our createLikes method, but instead of a straight forward LIKES relationships, we are going to be adding a “dated REPOSTED_ON” relationship.

 
    @POST
    @Path("/{username2}/{time}")
    public Response createRepost(@PathParam("username") final String username,
                               @PathParam("username2") final String username2,
                               @PathParam("time") final Long time,
                               @Context GraphDatabaseService db) throws IOException {
        Map<String, Object> results;

        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            Node user2 = Users.findUser(username2, db);
            Node post = getPost(user2, time);

            LocalDateTime dateTime = LocalDateTime.now(utc);
            if (userRepostedPost(user, post)) {
                throw PostExceptions.postAlreadyReposted;
            } else {
                Relationship r1 = user.createRelationshipTo(post, RelationshipType.withName("REPOSTED_ON_" +
                        dateTime.format(dateFormatter)));
                r1.setProperty(TIME, dateTime.toEpochSecond(ZoneOffset.UTC));
                results = post.getAllProperties();
                results.put(REPOSTED_TIME, dateTime.toEpochSecond(ZoneOffset.UTC));
                results.put(TIME, time);
                results.put(USERNAME, user2.getProperty(USERNAME));
                results.put(NAME, user2.getProperty(NAME));
                results.put(LIKES, post.getDegree(RelationshipTypes.LIKES));
                results.put(REPOSTS, post.getDegree(Direction.INCOMING)
                        - 1 // for the Posted Relationship Type
                        - post.getDegree(RelationshipTypes.LIKES)
                        - post.getDegree(RelationshipTypes.REPLIED_TO));
                results.put(LIKED, userLikesPost(user, post));
                results.put(REPOSTED, true);

            }
            tx.success();
        }
        return Response.ok().entity(objectMapper.writeValueAsString(results)).build();
    }

Since we are creating a repost, we know the REPOSTED value is going to be true. However I’ve been calling “userRepostedPost” to get REPOSTED, without showing you what that looks like. It’s a bit ugly I admit. Most posts will have few if any reposts, so if this is the case, we can traverse all its REPOSTED_ON_{date} relationships looking for the user. If however this is a very popular post, we will follow a different strategy. We will find the date of the post and check if the user reposted it on the same day it was posted. If not we will check the REPOSTED_ON_{date} relationship types from creation day forward until today and check the user or the post for a connection. We could optimize away this method by caching the results of the REPOSTED_ON_{date} relationships for either the user or the post. If the user only has a few reposts, we can even cache them all. Caching is an optimization strategy we can employ if needed, but don’t forget Neo4j can traverse millions of relationships per second, so it may not be necessary at all.

 
    public static boolean userRepostedPost(Node user, Node post) {
        boolean alreadyReposted = false;
        LocalDateTime now = LocalDateTime.now(utc);
        LocalDateTime dateTime = LocalDateTime.ofEpochSecond((Long)post.getProperty(TIME), 0, ZoneOffset.UTC).truncatedTo(ChronoUnit.DAYS);

        if (post.getDegree(Direction.INCOMING) < 1000) {
            for (Relationship r1 : post.getRelationships(Direction.INCOMING)) {
                if (r1.getStartNode().equals(user) && r1.getType().name().startsWith("REPOSTED_ON_")) {
                    alreadyReposted = true;
                    break;
                }
            }
        }

        while (dateTime.isBefore(now) && !alreadyReposted) {
            RelationshipType repostedOn = RelationshipType.withName("REPOSTED_ON_" +
                    dateTime.format(dateFormatter));

            if (user.getDegree(repostedOn, Direction.OUTGOING)
                    < post.getDegree(repostedOn, Direction.INCOMING)) {
                for (Relationship r1 : user.getRelationships(Direction.OUTGOING, repostedOn)) {
                    if (r1.getEndNode().equals(post)) {
                        alreadyReposted = true;
                        break;
                    }
                }
            } else {
                for (Relationship r1 : post.getRelationships(Direction.INCOMING, repostedOn)) {
                    if (r1.getStartNode().equals(user)) {
                        alreadyReposted = true;
                        break;
                    }
                }
            }
            dateTime = dateTime.plusDays(1);
        }
        return alreadyReposted;
    }

…and after all that I think we are ready for the timeline query.

About 100 lines of Java goodness taking us from the user to all of the people the user follows, to all of their posts and reposts on a set of dates. Ordered by date, and limited to a specified value for pagination. Here it is in all its splendor:

 
    @GET
    public Response getTimeline(@PathParam("username") final String username,
                             @QueryParam("limit") @DefaultValue("100") final Integer limit,
                             @QueryParam("since") final Long since,
                             @Context GraphDatabaseService db) throws IOException {
        ArrayList<Map<String, Object>> results = new ArrayList<>();
        LocalDateTime dateTime;
        if (since == null) {
            dateTime = LocalDateTime.now(utc);
        } else {
            dateTime = LocalDateTime.ofEpochSecond(since, 0, ZoneOffset.UTC);
        }
        Long latest = dateTime.toEpochSecond(ZoneOffset.UTC);

        try (Transaction tx = db.beginTx()) {
            Node user = Users.findUser(username, db);
            HashSet<Long> seen = new HashSet<>();
            ArrayList<Node> follows = new ArrayList<>();
            follows.add(user); // Adding user to see their posts on timeline as well
            for (Relationship r : user.getRelationships(Direction.OUTGOING, RelationshipTypes.FOLLOWS)) {
                follows.add(r.getEndNode());
            }

            LocalDateTime earliest = LocalDateTime.ofEpochSecond((Long)user.getProperty(TIME), 0, ZoneOffset.UTC);

            while (seen.size() < limit && (dateTime.isAfter(earliest))) {
                RelationshipType posted = RelationshipType.withName("POSTED_ON_" +
                        dateTime.format(dateFormatter));
                RelationshipType reposted = RelationshipType.withName("REPOSTED_ON_" +
                        dateTime.format(dateFormatter));

                for (Node follow : follows) {
                    Map followProperties = follow.getAllProperties();

                    for (Relationship r1 : follow.getRelationships(Direction.OUTGOING, posted)) {
                        Node post = r1.getEndNode();
                        if(seen.add(post.getId())) {
                            Long time = (Long)r1.getProperty("time");
                            Map<String, Object> properties = r1.getEndNode().getAllProperties();
                            if (time < latest) {
                                properties.put(TIME, time);
                                properties.put(USERNAME, followProperties.get(USERNAME));
                                properties.put(NAME, followProperties.get(NAME));
                                properties.put(HASH, followProperties.get(HASH));
                                properties.put(LIKES, post.getDegree(RelationshipTypes.LIKES));
                                properties.put(REPOSTS, post.getDegree(Direction.INCOMING)
                                        - 1 // for the Posted Relationship Type
                                        - post.getDegree(RelationshipTypes.LIKES)
                                        - post.getDegree(RelationshipTypes.REPLIED_TO));
                                properties.put(LIKED, userLikesPost(user, post));
                                properties.put(REPOSTED, userRepostedPost(user, post));
                                results.add(properties);
                            }
                        }
                    }

                    for (Relationship r1 : follow.getRelationships(Direction.OUTGOING, reposted)) {
                        Node post = r1.getEndNode();
                        if(seen.add(post.getId())) {
                            Map<String, Object> properties = r1.getEndNode().getAllProperties();
                            Long reposted_time = (Long)r1.getProperty(TIME);
                            if (reposted_time < latest) {
                                properties.put(REPOSTED_TIME, reposted_time);
                                properties.put(REPOSTER_USERNAME, followProperties.get(USERNAME));
                                properties.put(REPOSTER_NAME, followProperties.get(NAME));
                                properties.put(HASH, followProperties.get(HASH));
                                properties.put(LIKES, post.getDegree(RelationshipTypes.LIKES));
                                properties.put(REPOSTS, post.getDegree(Direction.INCOMING)
                                        - 1 // for the Posted Relationship Type
                                        - post.getDegree(RelationshipTypes.LIKES)
                                        - post.getDegree(RelationshipTypes.REPLIED_TO));
                                properties.put(LIKED, userLikesPost(user, post));
                                properties.put(REPOSTED, userRepostedPost(user, post));

                                Node author = getAuthor(post, (Long)properties.get(TIME));
                                properties.put(USERNAME, author.getProperty(USERNAME));
                                properties.put(NAME, author.getProperty(NAME));
                                results.add(properties);
                            }
                        }
                    }
                }
                dateTime = dateTime.minusDays(1);
            }
            tx.success();
        }

        results.sort(Comparator.comparing(m -> (Long) m.get("time"), reverseOrder()));

        return Response.ok().entity(objectMapper.writeValueAsString(
                results.subList(0, Math.min(results.size(), limit))))
                .build();
    }

…and that is how you build the back-end data service of a Twitter clone with Neo4j using Extensions to the existing Neo4j REST API. We’ll look at building our front end in an upcoming post. On to part 6.