Remote Profiling Neo4j with YourKit on AWS

remote_profile_sideways

A few months ago, Mark Needham blogged about how to setup remote monitoring of Neo4j using YourKit. I was asked the other day about getting a few more details on how to do this on Amazon, so here is my attempt at that. The first thing we’ll do is setup Neo4j on a Virtual Private Cloud. It’s good practice to not put your databases directly on the public internet.

vpc_wizard

We’ll use the VPC Wizard and create a VPC with public and private subnets. Our application servers will go on the public subnet, our database servers will go on the private subnet.

vpc_public_private

Follow along with the defaults and we’ll have a public subnet at 10.0.0.0/24 with a private subnet at 10.0.1.0/24.

vpc_public_private_step2

The wizard will create a NAT instance for us. Leave that alone for now, we’ll get back to it later. Next create a new EC2 instance for our Neo4j to run inside our private subnet.

create-inside-vpc

We’ll configure it’s security group (sg-912300f4) to allow ssh, port 7474 (Neo4j’s port) and port 8888 where we’ll setup monitoring later.

neo_ports

Once the instance is running, make a note of it’s private ip.

private_ip

Now before we can connect to it, we need to connect to our NAT instance… and before we can SSH into our NAT instance, we need to change the Security Group settings to allow inbound traffic over port 22. But don’t stop there. We are also going to accept all incoming traffic from the instances in the private subnet. We can point to them by the security group we just created (sg-912300f4).

letmeout2

Now we should be able to connect to our NAT instance. Let’s setup an ssh tunnel to our private instance:

ssh ec2-user@52.11.16.76 -i ec2-us-west.pem -t -A -L 2222:10.0.1.185:22 'watch -n60 echo "jump"'

With that in place we can ssh to our private instance using port 2222 on localhost:

ssh -p 2222 -i ec2-us-west.pem ubuntu@localhost

If you get a prompt, then congratulations on making it this far. Now let’s get Java installed:

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer

Download and extract Neo4j:

curl -O http://dist.neo4j.org/neo4j-enterprise-2.2.0-M04-unix.tar.gz
tar -xvzf neo4j-enterprise-2.2.0-M04-unix.tar.gz

…and let’s download yourkit as well.

curl -O https://www.yourkit.com/download/yjp-2014-build-14120-linux.tar.bz2
tar -xvjf yjp-2014-build-14120-linux.tar.bz2

From here, we’ll just need to update the neo4j-wrapper.conf file:

vi neo4j-enterprise-2.2.0-M04/conf/neo4j-wrapper.conf

to use the yourkit agent:

wrapper.java.additional=-agentpath:/home/ubuntu/yjp-2014-build-14120/bin/linux-x86-64/libyjpagent.so=port=8888

Edit the neo4j-server.properties file:

vi neo4j-enterprise-2.2.0-M04/conf/neo4j-wrapper.conf

…to let Neo4j listen on all IPs:

org.neo4j.server.webserver.address=0.0.0.0

and start neo4j.

neo4j-enterprise-2.2.0-M04/bin/neo4j start

Now finally we can connect Yourkit to our remote instance:

Screen Shot 2015-02-24 at 9.51.24 AM

Screen Shot 2015-02-24 at 9.51.36 AM

and start profiling Neo4j:

Screen Shot 2015-02-24 at 1.25.18 AM

You can use the ssh tunnel you made earlier to make requests to Neo4j. Don’t forget to lock down your NAT instance, so only you have access to it.

Tagged , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: