The Docker community is full of wide-eyed idealists who see the world as a place of default configurations and simple use cases using only the latest things. The Docker examples include Node.js, CouchDB, and Riak, among others. No mention of, say, Rails, despite it being one of the most popular frameworks over the past 5 years. I’ll try to fill that void.

What this tutorial covers

This blog post is an introduction to Docker for folks who use Rails. You don’t have to know anything about Docker, but you do need to know how to get a Rails app running. I’ll walk you through setting up Docker for development on a Mac, creating a Dockerfile, and running that Dockerfile on your development Docker host.

If you don’t have a Rails application, go make one. There are numerous tutorials on creating a simple Rails app. This tutorial assumes you have an existing Rails app that you want to deploy with Docker.

Setting up Docker

Setting up a development environment for Docker is pretty easy using their provided instructions—the Provisioning instructions are what you want. Some things I mention below are using those instructions on a Mac; namely, I’m using boot2docker, which automates the management of a virtual machine that acts as your Docker server.

One note: boot2docker creates a VirtualBox VM that uses NAT, but it does not map any ports to be accessible from the host machine. After issuing boot2docker init, you should run:

VBoxManage controlvm boot2docker-vm natpf1 "rails,tcp,127.0.0.1,3000,,3000"

Which will map port 3000 on your local machine to port 3000 on the Docker VM. Later, we will have our Docker container listen on port 3000 and without this mapping it will be inaccessible.

Creating a Docker file

The only thing you need to Dockerize your app is a Dockerfile. This defines how docker build will assemble the container for your app and how docker run will execute it. The beginning of the Docker file defines what base image to use:

FROM ubuntu

Like a lot of Rails developers, we use a Ruby runtime manager to ensure consistency across different developer systems. In our case, we use RVM, but a similar approach could be used for rbenv. The first thing we need to do is install RVM in our container and install the Ruby version we use:

RUN curl -sSL https://get.rvm.io | bash -s stable
RUN /bin/bash -c -l 'rvm requirements'
RUN /bin/bash -c -l 'rvm install 1.9.3-p448'
RUN /bin/bash -c -l 'rvm use 1.9.3-p448'
RUN /bin/bash -c -l 'gem install bundler --no-ri --no-rdoc'
RUN /bin/bash -c -l 'bundle config path "$HOME/bundler"'

The first command is the familiar one-line install for RVM. With it installed, we run the subsequent commands using the rvm-shell so that they have the rvm environment. After installing the Ruby that our app prefers, we install Bundler.

The last bit of setup is to pull the Rails application code into the container and install the gems that it needs:

ADD . pl-site
WORKDIR pl-site
RUN /bin/bash -c -l 'bundle install'

The final bit of our Dockerfile tells Docker what port to expose and how to run the Rails server:

EXPOSE 80
CMD /bin/bash -c -l 'ruby script/rails server -p 80'

The entirety of the Dockerfile looks like this:

FROM ubuntu

RUN apt-get update -q
RUN apt-get install -qy curl

RUN curl -sSL https://get.rvm.io | bash -s stable
RUN /bin/bash -c -l 'rvm requirements'
RUN /bin/bash -c -l 'rvm install 1.9.3-p448'
RUN /bin/bash -c -l 'rvm use 1.9.3-p448'
RUN /bin/bash -c -l 'gem install bundler --no-ri --no-rdoc'

RUN /bin/bash -c -l 'bundle config path "$HOME/bundler"'

ADD . pl-site
WORKDIR pl-site
RUN /bin/bash -c -l 'bundle install'

EXPOSE 80
CMD /bin/bash -c -l 'ruby script/rails server -p 80'

Building the container

With the Dockerfile written, the container can be built with docker build:

docker build -t container_name .

This command will take a while to run and produce a lot of output as Docker assembles your container: downloading the base image, installing RVM & gems, and putting your code in place.

Running the container

With the container built, you can run it on your development docker instance:

docker run -p 3000:80 container_name

This runs the container, mapping port 3000 on your local machine to 80 on the container, where the Rails server is listening.

Stopping the container

The docker run command executes in the foreground but ignores SIGINT (i.e. ctrl+c) by default. You can stop & remove the container with this one liner:

docker rm -f `docker ps -lq`

Resetting the Docker host

Docker aggressively caches throughout the build process—each step of your Dockerfile is saved so that subsequent runs of docker build can skip parts that have completed successfully. Unfortunately the caching, like that in so many projects, can cause problems when you’re developing. Sometimes it can be fixed by clearing the cache of Docker steps with docker rm `docker ps --no-trunc -a -q`. More often than not, though, I’ve had to go a step further and blow away the Docker host altogether. Since it is a VM controlled by boot2docker, removing & recreating it is easy. I’ve settled on this one liner that I run whenever I want to really rebuild the container:

boot2docker stop && boot2docker delete && boot2docker init && \
VBoxManage controlvm boot2docker-vm natpf1 "rails,tcp,127.0.0.1,3000,,3000" \
&& boot2docker start

Code

The example code used here is on GitHub. Note that there is an ultra-simple branch, echo, that makes an echo server in Docker.

The post Introduction to Docker with Rails appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

The Proximo add-on solves this problem by providing you with a proxy to send traffic through, with an IP address that won’t change. Their integration instructions suggest running your application in a wrapper that they provide in their stacklet.  Unfortunately, running our application within the wrapper mucks up the creation of the listening socket for the webserver:

Exception in thread "main" java.net.BindException: Cannot assign requested address

As an alternative to running your application within their wrapper, Proximo offers some examples of using it as an HTTP proxy, though they don’t provide an example for doing this in Java and for all my trying, I couldn’t get this to work globally through my Java application.

We looked through the aforementioned wrapper that Proximo distributes and found that it connects to the Proximo server as a SOCKS proxy.  That’s easy to do in Java, so I added this to the startup of my app:

URL proximo = new URL(proximoUrl)
String userInfo = proximo.getUserInfo()
String user = userInfo.substring(0, userInfo.indexOf(':'))
String password = userInfo.substring(userInfo.indexOf(':') + 1)

System.setProperty('socksProxyHost', proximo.getHost())
System.setProperty('socksProxyPort', PROXIMO_PORT)
Authenticator.setDefault(new ProxyAuthenticator(user, password))

That ProxyAuth is defined as an inner class elsewhere:

private class ProxyAuthenticator extends Authenticator {
    private final PasswordAuthentication passwordAuthentication;

    private ProxyAuth(String user, String password) {
        passwordAuthentication = new PasswordAuthentication(user, password.toCharArray())
    }

    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        return passwordAuthentication;
    }
}

With this configured, all requests initiated by the Java application are sent via SOCKS through the Proximo proxy.  We just white list the proxy’s IP address in our Salesforce settings and everything works.

We have encapsulated this Proximo SOCKS setup code into a module, proximo-socks, that you can use in your application.  Simply add it as a dependency in your pom.xml or build.gradle and then initialize it in your app’s startup:

import com.palominolabs.heroku.Proximo
…
Proximo.setup()

See also

• The summary version on Stack Overflow
• Setting up a SOCKS proxy in Java

1. Note that this only applies to username/password based authentication. Using the preferred Oauth method requires no whitelisting. ↩

The post Using Proximo as a SOCKS proxy in Java appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

Some simple benchmarking illustrates the performance difference between the two—for the following benchmarks we used an m3.2xlarge EC2 instance, which provides access to all 8 cores of an Intel Xeon E5-2670.

With a single thread, the new LongAdder is one third slower, but when threads are in contention to increment the field, LongAdder shows its value.  Note that the only thing each thread is doing is attempting to increment the counter—this is a synthetic benchmark of the most extreme kind. The contention here is higher than you’re likely to see in most real-world apps, but sometimes you do need this sort of shared counter, and LongAdder will be a big help.

You can find the code for these benchmarks in our java-8-benchmarks repository.  It uses JMH to do all of the real work and Marshall‘s gradle-jmh-demo for plumbing.  JMH makes benchmarking easy by doing all of the fiddly little bits for you, ensuring the resulting numbers represent the state of the art in JVM-based benchmarking accuracy.  JMH isn’t amenable to running under perf, though, so we also have some simple standalone benchmarks for that.

More details with perf-stat

We wrote standalone benchmarks so that we could have more control and run them under perf-stat to get some more details about what is going on.  The most basic thing is the wall clock time that each benchmark run took.  These benchmarks are all run on an Intel Core  i7-2600K (real hardware, not virtualized).

While AtomicLong is a bit quicker in the single-threaded case, it quickly loses ground to LongAdder, being nearly 4 times slower with two threads, and nearly 5x with threads matching the machine’s cores.  More impressive is that LongAdder’s performance is constant until the number of threads exceeds the CPU’s number of physical cores (in this case 4).

Instructions per cycle

Instructions per cycle measures how much of the time the CPU has work to do vs. when it’s waiting for memory to load or cache coherency protocols to settle. In this case, we see that AtomicLong has disastrously bad IPC with many threads, while LongAdder maintains a much healther IPC. The falloff from 4 to 8 is likely because this CPU has 4 cores with 2 hardware threads each, and the hardware threads don’t actually help in this case.

Idle time

The  execution pipeline on the processor is divided into two major groups: the front end, responsible for fetching & decoding operations, and the back end, which executes the instructions. There isn’t much interesting happening with operation fetching, so let’s skip the front end.

Activity on the back end gives more insight as to what is going on, showing the AtomicLong implementation leaving more than twice as many cycles idle.  AtomicLong’s high idle time is analogous to its poor instructions per cycle: the CPU’s cores are spending a lot of time deciding which of them gets to control the cache line containing the AtomicLong.

See also

• A more detailed look at LongAdder
• Using JMH to Benchmark Multi-Threaded Code

The post Java 8 Performance Improvements: LongAdder vs AtomicLong appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

Last year we created a tool to make comparing the price of EC2 instances easier.  With the advent of these new instance types, we’ve updated WhichInstance.com to reflect the new options, improving the interface a bit in the process.  Give WhicInstance a look to more easily understand the costs of running an instance over time, and remember that Amazon reserved instances can save you a lot of money!

The post WhichInstance.com updated for new M3 instances appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

At Palomino Labs, we recently completed a project that involved two main pieces: a webapp written in Rails and a data storage application in Java. In our scenario, a portion of the webapp’s functionality depended upon being able to communicate with the running Java app. In the development environment, this is a minor nuisance, requiring the developer to have the Java application running. Running the test automatically using Jenkins was a bigger problem, and since we know that automated testing is integral to catching regressions quickly, the problem needed to be solved. One option would be to have a long-running version of the Java service available to the Jenkins box, but that would far from ideal because we want to make sure that the versions of the Java and Rails projects that are used for testing stay in sync with each other. Instead, we have Jenkins start and stop a fresh, up-to-date instance of the Java service for every test run.

Adding Java Convenience

While we have scripts to run our application in the production and staging environments, we usually depend on our IDE for running the application on our development machines. This is a minor issue for developers (“Why are the tests failing? Whoops, I forgot to start that other thing…”) but a show stopper for Jenkins (née Hudson) who doesn’t want to run an IDE. The first task was to create a way to run the Java application from the command line.

Using The Exec Maven Plugin

We use Maven for building our Java projects. The Exec Maven Plugin provides an easy way to run you classes from the command line with the necessary classpath. Usage begins with including it in your pom.xml

<project>

    ...

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

Actually running a main() method is easy—you just invoke mvn exec:java and pass it the class you want to run:

mvn exec:java -Dexec.mainClass="com.palominolabs.widgets.Widgeter"

Start & Stop Scripts

We start by creating a set of simple process management scripts to run the Java application in the background and stop it when we’re done testing. Our project uses modules, so this script lives within the widgeter-service module, which has the main method we want to run, at widgets/widgets-service/bin/.

Note the bashism "$(cd "$(dirname "$(dirname "$0")" )"; pwd)" which finds the directory the script lives in. This script also employs lock directories, the use of which removes a race condition when using lock files.

#!/bin/bash
WIDGETER_PATH="$(cd "$(dirname "$(dirname "$0")" )"; pwd)"
RUNDIR="$WIDGETER_PATH/run"
PIDFILE="$RUNDIR/widgeter-service.pid"
LOGFILE="$RUNDIR/widgeter-service.log"

cd $WIDGETER_PATH
if mkdir "$RUNDIR"; then
    mvn exec:java -Dexec.mainClass="com.palominolabs.widgets.Widgeter" 2>&1 > $LOGFILE &
    PID=$!
    echo -n "$PID" >> $PIDFILE
    echo "Started $PID"
else
    PID=$(cat $PIDFILE)
    echo >&2 "Lockdir $RUNDIR exists; is pid $PID still running?"
    exit 1
fi

The stop script lives in the same directory and looks similar:

#!/bin/bash
WIDGETER_PATH="$(cd "$(dirname "$(dirname "$0")" )"; pwd)"
RUNDIR="$WIDGETER_PATH/run"
PIDFILE="$RUNDIR/widgeter-service.pid"

cd $WIDGETER_PATH
if [[ -f "$PIDFILE" ]]; then
    PID=$(cat $PIDFILE)
    echo "Killing $PID"
    kill $PID

    rm -r $RUNDIR
else
    echo "Widgeter isn't running ($PIDFILE doesn't exist)"
fi

With the Java now easy to run from the command line, we just need to wire up the Rails side of things.

Cucumber Integration

We want the Widgeter to be automatically started and whenever Selenium tests (written in Cucumber & Capybara) are run—either by a developer in their IDE or by Jenkins. To do so, we edit the cucumber script provided by the cucumber gem. The one assumption made here is that the widgets project exists at the same level as the Rails project and that this script is run from the root of the rails project.

#!/usr/bin/env ruby
WIDGETS_DIR="../widgets"

system("cd $WIDGETS_DIR && ./widgeter-service/bin/widgeter-service-start.sh")
Signal.trap("EXIT") do
    system("cd $WIDGETS_DIR && ./widgeter-service/bin/widgeter-service-stop.sh")
end

cucumber_bin = Gem.bin_path('cucumber', 'cucumber')
if cucumber_bin
  load cucumber_bin
else
  require 'rubygems' unless ENV['NO_RUBYGEMS']
  require 'cucumber'
  load Cucumber::BINARY
end

Jenkins

Jenkins runs the tests using a script, jenkins-cucumber.sh. Originially, this simply called jenkins-build.sh which sources the project’s .rvmrc, executes bundle install, and performs migrations, before running cucumber under xvfb. Adding lines similar to the cucumber script above to start Widgeter before Cucumber, and stop it when the tests completed or are killed. This script is only for use by Jenkins, so we simply hardcode the location of the Widgets project.

#!/bin/bash -e
source script/jenkins-build.sh

RAILS_DIR=`pwd`
WIDGETS_DIR="/var/lib/jenkins/jobs/widgets/workspace"

cd $WIDGETS_DIR && ./widgeter-service/bin/widgeter-service-start.sh
cleanup() {
    if [[ $WIDGETER_CLEANED -eq 1 ]]; then
        exit 0
    fi
    WIDGETER_CLEANED=1
    cd $WIDGETS_DIR && ./widgeter-service/bin/widgeter-service-stop.sh
}
trap cleanup EXIT

xvfb-run bundle exec cucumber -p ci || true

The post Ruby Testing and External Dependencies appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

Security Context

First, let’s take a look at what we’re going to do here. This post covers symmetric encryption, wherein a single key is used for encryption and decryption. In such a system there is a single thing, the encryption key, that must be kept secret and shared between anyone wanting to encrypt or decrypt data. In our example, we are also using a message authentication code (MAC), which requires its own key that must be secret & shared. Pieces of information like these are appropriately called shared secrets.

The MAC ensures that the outputs of our encryption (namely the ciphertext and initialization vector) have not been altered. It is theoretically possible to alter the initialization vector (IV) and ciphertext in concert to change the message without knowing the secret key. Additionally, either of those pieces could be manipulated so as to cause a fault when they are fed into the decryption system, causing an unexpected exception in your decryption system.

Prerequisites

Generally, you should employ the highest strength encryption readily available. Java ships by default with a security policy that complies with United States cryptography export control regulations that limits the encryption algorithms & key lengths that can be used. To get all of the available cryptographic tools you must install the Java Cryptography Extension Unlimited Strength Jurisdiction Policy Files. Without these, you will get InvalidAlgorithmParameterException because Java can’t find the requested algorithm or InvalidKeyException if the given key length exceeds that which is permitted under the jurisdiction policy files. Look out for these misleading errors when deploying your code to new machines.

Check out our basic-java-encryption repository to follow along at home.

Getting Started

The javax.crypto package deals exclusively in bytes. If you have strings you want to encrypt, you’ll need to represent them as bytes. For arbitrary things like your encryption keys, the easiest way is to store them as Base64 encoded strings and read or write them to disk using something like Commons Codec’s Base64.

If you’re dealing with strings of unknown stripe, be sure to use explicit encodings—given how opaque encryption programming is, the last thing you want is to get hung up because what you thought was a UTF-8 string was actually interpreted as US-ASCII.

Encrypting

You need a few different objects to perform encryption: the actual Cipher, the initialization vector, and the key. It’s also a good idea to generate a message authentication code (MAC) of the ciphertext, to protect against tampering with the ciphertext or IV that could be used to manipulate the Cipher object upon decryption.

The encryption flow looks like this:

SecureRandom secureRandom = new SecureRandom();
Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
Mac mac = Mac.getInstance("HmacSHA512");

// Sizes appropriate for AES: 128 bit IV, 256 bit key
private static final int IV_SIZE = 16;
private static final int KEY_SIZE = 32;

byte[] iv = new byte[IV_SIZE];
byte[] aesKey = new byte[KEY_SIZE];
byte[] macKey = new byte[KEY_SIZE];
byte[] ciphertext;
byte[] macBytes;

// Generate the keys & iv
secureRandom.nextBytes(iv);
secureRandom.nextBytes(aesKey);
secureRandom.nextBytes(macKey);

try {
    SecretKeySpec secretKeySpec = new SecretKeySpec(aesKey, "AES");
    IvParameterSpec ivParameterSpec = new IvParameterSpec(iv);
    cipher.init(Cipher.ENCRYPT_MODE, secretKeySpec, ivParameterSpec);
    ciphertext = cipher.doFinal(byteArrayToEncrypt);

    mac.init(macKey);
    mac.update(iv);
    macBytes = mac.doFinal(ciphertext);
} catch (IllegalBlockSizeException | BadPaddingException |
        InvalidAlgorithmParameterException | InvalidKeyException e) {
    logger.warn("Problem encrypting", e);
    // throw exception or other error handling
}

The ciphertext, IV, and MAC can be stored wherever you keep data—they can be serialized into a single object, for example—without any special considerations. The keys, however, are sensitive and should be stored securely.

Decrypting

Decryption is a similar flow, but in reverse. You start by checking that the stored MAC matches the ciphertext & IV that you want to decrypt, and if they do, then you decrypt.

Assuming the objects from above still exist:

byte[] plaintext = new byte[0];

try {
    mac.init(hmacKey);
    mac.update(iv);
    // macBytes from above
    if (MessageDigest.isEqual(macBytes, mac.doFinal(ciphertext))) {
        cipher.init(Cipher.DECRYPT_MODE, secretKeySpec, ivParameterSpec);
        plaintext = cipher.doFinal(ciphertext);
    } else {
        logger.warn("MAC mismatch");
        // do something
    }
} catch (IllegalBlockSizeException | BadPaddingException |
        InvalidAlgorithmParameterException | InvalidKeyException e) {
    logger.warn("Problem encrypting", e);
    // throw exception or other error handling
}

Demo Code

Play around with a working example by cloning our basic-java-encryption repository.

Bits & Baubles

If encrypting multiple things, you must call init() and doFinal() for each plaintext with a new initialization vector each time.

Having explained this whole thing, the real answer to encryption is to abstract away as much as possible. If possible, use libraries like keyczar or Jasypt that will aid you in performing high-level cryptographic functions, while taking care of many of the minutiae for you.

The post Symmetric Encryption in Java appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

This is not a cut & paste tutorial for using Curator, but rather a summariztion of how we implemented Curator in BenchPress. While the class names are unchanged, their content is simplified. Pull up the BenchPress code for further detail on employing Curator’s Service Discovery.

To begin with, put some junk in your POM:

<dependency>
    <groupId>com.netflix.curator</groupId>
    <artifactId>curator-framework</artifactId>
    <version>1.1.9</version>
    <exclusions>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.netflix.curator</groupId>
    <artifactId>curator-x-discovery</artifactId>
    <version>1.1.9</version>
</dependency>

Curator is the base Netflix framework and they have broken out service discovery into a separate artifact. Unfortunately, Curator follows the lead of many Java projects that don’t properly declare their dependencies, and the curator-framework artifact will pull in a (likely unwanted) version of Log4j, so I suggest using the exclusions.

You need some configgy things in various places; for the purposes of this tutorial I’ll put them in a Config object:

public class Config {
    public static final String zkConnectionString = "127.0.0.1:2181";
    public static final String basePath = "/myApp/serviceDiscovery";
}

Talking to ZooKeeper

Easy parts done, let’s create a CuratorFramework, which is the conduit through which ZooKeeper communication happens:

try {
    curatorFramework = CuratorFrameworkFactory.builder()
        .connectionTimeoutMs(1000)
        .retryPolicy(new RetryNTimes(10, 500))
        .connectString(Config.zkConnectionString)
        .build();
} catch (IOException e) {
    throw Throwables.propagate(e);
}
curatorFramework.start();

As mentioned above, ZooKeeper is incredibly flexible. This flexibility comes from its inherent simplicity—ZooKeeper is arranged like a filesystem as a series of nodes in a hierarchy, with each of the nodes having an optional data attachment. For this application, we don’t particularly care how Curator Service Discovery uses ZooKeeper to list services as available, but we do need to give Curator a place where this information can live. ZooKeeper paths are not autovivified and must be exlicitly created. Like other frameworks (see the delightfully undocumented zkClient), Curator provides a method to create a path if it doesn’t exist. This is a good thing to call whenever you create a CuratorFramework.

try {
    new EnsurePath(Config.basePath).ensure(curatorFramework.getZookeeperClient());
} catch (Exception e) {
    throw Throwables.propagate(e);
}

Handling Metadata

With Curator configured and ZooKeeper ready to receive, we need to prepare our wares. In this example, our workers are services that listen on the netowrk for work to do and the farmer has work that it wants to distribut to workers. In order to communicate with workers, the farmer needs to know the hostname of a worker and what port it is listening on. This information is what we want to insert into ZooKeeper via Curator. To make things tidy, we will encapsulate this worker metadata in an object:

public final class WorkerMetadata {
    @JsonProperty("workerId")
    private final UUID workerId;

    @JsonProperty("listenAddress")
    private final String listenAddress;

    @JsonProperty("listenPort")
    private final int listenPort;

    @JsonCreator
    public WorkerMetadata(@JsonProperty("workerId") UUID workerId,
            @JsonProperty("listenAddress") String listenAddress,
            @JsonProperty("listenPort") int listenPort) {
        this.workerId = workerId;
        this.listenAddress = listenAddress;
        this.listenPort = listenPort;
    }

    public UUID getWorkerId() {
        return workerId;
    }

    public String getListenAddress() {
        return listenAddress;
    }

    public int getListenPort() {
        return listenPort;
    }
}

A simple object with some annotations to let Jackson know how to interact with it. Curator doesn’t natively understand how to apply Jackson to custom objects, so the last thing we need is an InstanceSerializer to pass to the ServiceDiscovery:

public class InstanceSerializerFactory {
    private final ObjectReader objectReader;
    private final ObjectWriter objectWriter;

    InstanceSerializerFactory(ObjectReader objectReader, ObjectWriter objectWriter) {
        this.objectReader = objectReader;
        this.objectWriter = objectWriter;
    }

    public <T> InstanceSerializer<T> getInstanceSerializer(
            TypeReference<ServiceInstance<T>> typeReference) {
        return new JacksonInstanceSerializer<T>(objectReader, objectWriter, typeReference);
    }

}

Advertising Workers

There are two sides to service discovery: those with something to offer (services) and those who want to utilize those services. In our example, workers provide services and the farmer is looking to utilize those services. Let’s start with how workers register themselves as providing a service:

public final class WorkerAdvertiser {
    private final CuratorFramework curatorFramework;
    private final InstanceSerializer<WorkerMetadata> jacksonInstanceSerializer;
    private final UUID workerId = UUID.randomUUID();

    private final String serviceName;
    private final String listenAddress;
    private final int listenPort;

    WorkerAdvertiser(CuratorFramework curatorFramework, InstanceSerializerFactory instanceSerializerFactory
            String serviceName, String listenAddress, int listenPort) {
        this.curatorFramework = curatorFramework;
        this.jacksonInstanceSerializer = instanceSerializerFactory.getInstanceSerializer(
            new TypeReference<ServiceInstance<WorkerMetadata>>() {}
        );
        this.listenAddress = listenAddress;
        this.listenPort = listenPort;
    }

    public void advertiseAvailability() {
        try {
            ServiceDiscovery<WorkerMetadata> discovery = getDiscovery();
            discovery.start();
            discovery.registerService(getInstance());
            discovery.close();
        } catch (Exception e) {
            throw Throwables.propagate(e);
        }
    }

    public void deAdvertiseAvailability() {
        try {
            ServiceDiscovery<WorkerMetadata> discovery = getDiscovery();
            discovery.start();
            discovery.unregisterService(getInstance());
            discovery.close();
        } catch (Exception e) {
            throw Throwables.propagate(e);
        }
    }

    private ServiceInstance<WorkerMetadata> getInstance() throws Exception {
        WorkerMetadata workerMetadata = new WorkerMetadata(workerId, listenAddress, listenPort);
        return ServiceInstance.<WorkerMetadata>builder()
            .name(serviceName)
            .address(listenAddress)
            .port(listenPort)
            .id(workerId.toString())
            .payload(workerMetadata)
            .build();
    }

    private ServiceDiscovery<WorkerMetadata> getDiscovery() {
        return ServiceDiscoveryBuilder.builder(WorkerMetadata.class)
            .basePath(Config.basePath)
            .client(curatorFramework)
            .serializer(jacksonInstanceSerializer)
            .build();
    }
}

The core things here are the two public methods for advertising availability—call them does exactly what it says on the tin. From the worker’s side, this is it: create a WorkerAdvertiser, initializing it with the metadata about this worker (in the example case, how to contact the worker), and advertise as appropriate. Curator will create entries in ZooKeeper that can be found by a farmer looking to pass out work.

Finding Workers

Using Curator from a farmer’s perspective is a similarly simple endeavor:

public final class WorkerFinder {
    private final ServiceDiscovery<WorkerMetadata> discovery;

    WorkerFinder(CuratorFramework curatorFramework,
                 InstanceSerializerFactory instanceSerializerFactory) {
        discovery = ServiceDiscoveryBuilder.builder(WorkerMetadata.class)
            .basePath(Config.basePath())
            .client(curatorFramework)
            .serializer(instanceSerializerFactory
                .getInstanceSerializer(new TypeReference<ServiceInstance<WorkerMetadata>>() {}))
            .build();

        try {
            discovery.start();
        } catch (Exception e) {
            throw Throwables.propagate(e);
        }
    }

    public Collection<ServiceInstance<WorkerMetadata>> getWorkers(String serviceName) {
        Collection<ServiceInstance<WorkerMetadata>> instances;

        try {
            instances = discovery.queryForInstances(serviceName));
        } catch (Exception e) {
            throw Throwables.propagate(e);
        }

        return instances;
    }
}

The WorkerFinder cretaes a Curator ServiceDiscovery object with which we can find workers matching the service we desire. The returned ServiceInstances contain a WorkerMetadata object in their payload:

for (ServiceInstance<WorkerMetadata> instance : workerFinder.getWorkers()) {
    WorkerMetadata workerMetadata = instance.getPayload();
    // Do something useful here
}

Netflix Curator does a lot more than service discovery— check out their documentation for further details. Also peruse the BenchPress source for further details on how we used Curator in that project.

The post Using Netflix Curator for Service Discovery appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.

We recently ran across the no known serializer problem while trying to use Jackson (by way of Jersey) with an object that contained a Joda Time Duration. For future reference, the exception produced by Jersey looks like this:

com.sun.jersey.spi.container.ContainerResponse - The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
com.fasterxml.jackson.databind.JsonMappingException: No suitable constructor found for type [simple type, class org.joda.time.Duration]: can not instantiate from JSON object (need to add/enable type information?)
 at [Source: org.eclipse.jetty.server.HttpInput@1c7b0f4d; line: 1, column: 47]
    at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:163) ~[jackson-databind-2.0.1.jar:na]

Since our application employed the Joda data type extensions module for Jackson, this error seemed curious, but looking into that project it became clear that translation is not provided for all of the Joda data types. Duration is an exceedingly simple object (it can be represented as a number of milliseconds), so writing our own serializer & deserializer is an easy endeavor.

public final class DurationSerializer extends StdScalarSerializer<Duration> {
    public DurationSerializer() { super(Duration.class); }

    @Override
    public void serialize(Duration value, JsonGenerator jgen, SerializerProvider provider) throws IOException,
        JsonProcessingException {
        jgen.writeNumber(value.getMillis());
    }
}

Exceedingly simple, no? As mentioned before, a Duration is nothing more than an elapsed amount of time which we represent as milliseconds for ease of JSONing. Writing the deserializer to pull a Duration out of JSON is a similarly trivial bit of code:

public final class DurationDeserializer extends StdScalarDeserializer<Duration> {
    public DurationDeserializer() { super(Duration.class); }

    @Override
    public Duration deserialize(JsonParser jsonParser, DeserializationContext deserializationContext)
            throws IOException, JsonProcessingException {
        if (jsonParser.getCurrentToken() == VALUE_NUMBER_INT) {
            return new Duration(jsonParser.getLongValue());
        }

        throw deserializationContext.mappingException("Expected JSON Number");
    }
}

Easy, we’re just verifying that the JSON token is an integer and then passing it to the Duration constructor. These two classes are the actual functionality of JSON reading and writing, but we need to tell Jackson about them. To do so, we’ll create a Jackson data-binding module:

public final class JodaDurationModule extends SimpleModule {
    public JodaDurationModule() {
        addDeserializer(Duration.class, new DurationDeserializer());
        addSerializer(Duration.class, new DurationSerializer());
    }
}

Now, tell Jackson about the module:

ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(new JodaDurationModule());

With that, Jackson now knows how to translate Joda Durations into JSON and back:

System.out.println(objectMapper.writer().writeValueAsString(new Duration(8)));

For completeness, here are the Maven dependency clauses for Jackson. The above code only requires jackson-core, but I mention the data-binding annotations and the Joda Time extensions:

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>2.0.1</version>
</dependency>
<!-- For the aforementioned data-binding annotations -->
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.0.1</version>
</dependency>
<!-- Extensions for Joda Time, which don't include Duration -->
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-datatype-joda</artifactId>
    <version>2.0.1</version>
</dependency>

The post Writing a Custom Jackson Serializer & Deserializer appeared first on Palomino Labs Blog.



Palomino Labs unlocks the potential of software to change people and industries. Our team of experienced software developers, designers, and product strategists can help turn any idea into reality. Check out our website for more information, or send us an email and let's start talking about how we can work together.