Thursday, June 19, 2014

Moving to Silvrback

I will no longer be using Blogger as my platform.  There are some things to like about the platform, but more and more I find it to be suboptimal, particularly for developers who want to post code.  I won't be closing the site down, but I do plan to not write any more content within Blogger.

You can view my new blog at http://rclayton.silvrback.com.

Friday, June 13, 2014

Enable JMX on Amazon EC2

I ran into an issue this last week enabling JMX for a Java process on an Amazon EC2 instance.  After a couple of hours researching (and finding a couple of answers on StackOverflow), I wanted to share a complete answer to how this is done.

1.  Unblock RMI and JMX Ports.

First, you have to ensure both the JMX and RMI ports are available to the process that's going to make the JMX connection.  One issue many devs/admins are having within EC2 is that their JMX clients are unable to establish a connection to RMI.  Therefore, you need to make sure your instance's security policy, as well as your firewall (if using one), allow both ports to be accessed.  In this example, I'm specifying 9011 and 9010 for JMX and RMI respectively.

2.  Resolve the Public DNS Name of your EC2 Instance.

Another problem is that when your JMX client makes a connection, the JMX server returns the wrong connection information for the server's RMI port.  If you don't override the java.rmi.server.hostname, the server will return the default hostname for the EC2 instance, which is not publicly resolvable.  To fix this, you can request the public DNS name of an instance from the "metadata server" within EC2.  A call made by an EC2 instance to the URL http://169.254.169.254/latest/meta-data/public-hostname will return the public DNS name of that instance.  In the example below, you will see the call made using wget for this information.

3.  Prefer the IPv4 Stack.

Finally, the last issue we ran into was the JMX server binding to IPv6 addresses instead of IPv4.  You can instruct Java to prefer the IPv4 stack with the following flag: java.net.preferIPv4Stack=true.

The rest of the configuration is pretty standard.  This is a fragment of a BASH script showing the composition of JAVA_OPTS you may use when starting your Java process:

#!/bin/bash

JMX_PORT=9011

RMI_PORT=9010

RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`

JAVA_OPTS="-Dcom.sun.management.jmxremote "
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.local.only=false "
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.port=${JMX_PORT} "
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.authenticate=false"
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.ssl=false"
JAVA_OPTS="${JAVA_OPTS} -Djava.rmi.server.hostname=${RMI_HOST}"
JAVA_OPTS="${JAVA_OPTS} -Djava.net.preferIPv4Stack=true "
JAVA_OPTS="${JAVA_OPTS} -Dcom.sun.management.jmxremote.rmi.port=${RMI_PORT}"

#

I would like to thank the StackOverflow commentors for help with this issue.  You can see the original posts on these threads:

http://stackoverflow.com/questions/13734646/how-to-connect-to-java-instances-running-on-ec2-using-jmx

http://stackoverflow.com/questions/19130877/jmx-connection-to-amazon-ec2-fails