Getting Started Developing with ZooKeeper using Node.js

Getting Started W/Zookeeper

A guide to learning and using Zookeeper.

  1. Install OS level dependencies
  2. Configure server and play with CLI facilities
  3. Starting development using nodejs and node-zookeeper-client

Install

yum install zookeeper zookeeper-server

At the time of this writing, version available to Fedora 20, zookeeper-3.4.5-12.fc20, is a little different than the source install. One of the main differences is that this installation doesn't come with the zkCli script to launch the command line.

rpm -ql zookeeper zookeeper-server

/usr/bin/cli_mt
/usr/bin/cli_st
/usr/bin/load_gen
/usr/bin/zktreeutil
/usr/share/doc/zookeeper
/usr/share/doc/zookeeper/ChangeLog
/usr/share/doc/zookeeper/LICENSE
/usr/share/doc/zookeeper/NOTICE.txt
/usr/share/doc/zookeeper/README
/usr/share/doc/zookeeper/README.txt
/etc/zookeeper
/etc/zookeeper/log4j.properties
/etc/zookeeper/zoo.cfg
/etc/zookeeper/zoo_sample.cfg
/usr/lib/systemd/system/zookeeper.service
/var/lib/zookeeper
/var/lib/zookeeper/data
/var/lib/zookeeper/data/myid
/var/lib/zookeeper/log
/var/log/zookeeper

Instead there are a couple of binaries, cli_mt and cli_st, for multi-threaded and single-threaded respectively.

Configure

The installation provides only a sample configuration which is suitable for development. Copy that into place.

cp /etc/zookeeper/zoo_sample.cfg /etc/zookeeper/zoo.cfg

Launch the server manually. First check the systemd service file for exact syntax.

cat /usr/lib/systemd/system/zookeeper.service
[Unit]
Description=Apache ZooKeeper
After=network.target
ConditionPathExists=/etc/zookeeper/zoo.cfg
ConditionPathExists=/etc/zookeeper/log4j.properties
ConditionPathExists=/var/lib/zookeeper/data/myid

[Service]
Type=simple
User=zookeeper
SyslogIdentifier=zookeeper
WorkingDirectory=/var/lib/zookeeper
UMask=0027
Environment="CP=/etc/zookeeper:/usr/share/java/slf4j/slf4j-log4j12.jar:/usr/share/java/slf4j/slf4j-api.jar:/usr/share/java/netty.jar:/usr/share/java/log4j.jar:/usr/share/java/jline.jar:/usr/share/java/zookeeper/zookeeper.jar"
Environment="IPv6=-Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true"
#Environment="JMX=-Dcom.sun.management.jmxremote"
ExecStart=/usr/lib/jvm/jre-1.7.0/bin/java -cp $CP $JMX $IPv6 org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/zoo.cfg

[Install]
WantedBy=multi-user.target

The exact command:

sudo /usr/lib/jvm/jre-1.7.0/bin/java -cp /etc/zookeeper:/usr/share/java/slf4j/slf4j-log4j12.jar:/usr/share/java/slf4j/slf4j-api.jar:/usr/share/java/netty.jar:/usr/share/java/log4j.jar:/usr/share/java/jline.jar:/usr/share/java/zookeeper/zookeeper.jar -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=true org.apache.zookeeper.server.quorum.QuorumPeerMain /etc/zookeeper/zoo.cfg

And the output:

[myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /etc/zookeeper/zoo.cfg
[myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
[myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
[myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
[myid:] - WARN  [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running  in standalone mode
[myid:] - INFO  [main:QuorumPeerConfig@101] - Reading configuration from: /etc/zookeeper/zoo.cfg
[myid:] - INFO  [main:ZooKeeperServerMain@95] - Starting server
[myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.5--1, built on 07/30/2013 13:20 GMT
[myid:] - INFO  [main:Environment@100] - Server environment:host.name=localhost.localdomain
[myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.7.0_71
[myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
[myid:] - INFO  [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.0.fc20.x86_64/jre
[myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=/etc/zookeeper:/usr/share/java/slf4j/slf4j-log4j12.jar:/usr/share/java/slf4j/slf4j-api.jar:/usr/share/java/netty.jar:/usr/share/java/log4j.jar:/usr/share/java/jline.jar:/usr/share/java/zookeeper/zookeeper.jar
[myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
[myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=/tmp
[myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
[myid:] - INFO  [main:Environment@100] - Server environment:os.name=Linux
[myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
[myid:] - INFO  [main:Environment@100] - Server environment:os.version=3.16.6-203.fc20.x86_64
[myid:] - INFO  [main:Environment@100] - Server environment:user.name=root
[myid:] - INFO  [main:Environment@100] - Server environment:user.home=/root
[myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/home/adarias
[myid:] - INFO  [main:ZooKeeperServer@726] - tickTime set to 2000
[myid:] - INFO  [main:ZooKeeperServer@735] - minSessionTimeout set to -1
[myid:] - INFO  [main:ZooKeeperServer@744] - maxSessionTimeout set to -1
[myid:] - INFO  [main:NIOServerCnxnFactory@94] - binding to port ::/0:0:0:0:0:0:0:0:2181
[myid:] - INFO  [main:FileSnap@83] - Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.0
[myid:] - INFO  [main:FileTxnSnapLog@240] - Snapshotting: 0x75 to /var/lib/zookeeper/data/version-2/snapshot.75

At this point you can connect to the server and interact via the command line. For this we'll use the single-threaded cli binary.

cli_st localhost:2181
Watcher SESSION_EVENT state = CONNECTED_STATE
Got a new session id: 0x149639d3a060001

Play around:

ls /
time = 6 msec
/: rc = 0
        httpservers
        zookeeper
time = 6 msec
help
    create [+[e|s]] <path>
    delete <path>
    set <path> <data>
    get <path>
    ls <path>
    ls2 <path>
    sync <path>
    exists <path>
    wexists <path>
    myid
    verbose
    addauth <id> <scheme>
    quit

    prefix the command with the character 'a' to run the command asynchronously.
    run the 'verbose' command to toggle verbose logging.
    i.e. 'aget /foo' to get /foo asynchronously
get /
time = 7011 msec
/: rc = 0
 value_len = 0

Stat:
        ctime = Wed Dec 31 16:00:00 1969
        czxid=0
        mtime=Wed Dec 31 16:00:00 1969
        mzxid=0
        version=0       aversion=0
        ephemeralOwner = 0

Disconnect the client, kill the server, enable the systemd service and start it that way.

systemctl status zookeeper
sudo systemctl enable zookeeper.service
sudo systemctl start zookeeper.service

Starting Development

At this point we are ready to start developing against the ZK server. For this exercise, we'll use nodejs and the node-zookeeper-client module.

Start the project any way you want. I do it like this:

  1. Create project directory.
  2. Create the package.json file using npm.
  3. Install the node-zookeeper-client module and --save with npm to add it to your package.json.

At this point your project tree should look something like this:

ls -1
index.js
node_modules
package.json

Add the following to your index.js:

var zk = require('node-zookeeper-client')
        , events = require('events')
        , util = require('util')
        , os = require('os')
        , http = require('http')
        ;

zkaddr = 'localhost:2181';
httpport = process.env.HTTPPORT || 3000;

function Server() {
        this.zkclient = zk.createClient(zkaddr);

        this.nodeid = os.hostname() + '-' + process.pid;
        this.nodepath = '/httpservers/' + this.nodeid;

        var t = this;

        process.on('SIGINT', function() {
                console.log('[server] Received SIGINT signal. Shutting down.');
                t.zkclient.close();
                t.httpserver.close();
                process.exit(130);
        });

        // register the http request handler
        this.httpserver = http.createServer(function(req, res) {
                console.log('[httpserver] Incoming request.');
                res.send('OK');
        });

        // Node tree preperation includes checking for the httpservers parent node
        // and creating it if it does already exists. Finally, the server object
        // is notified when zk is ready.
        this.prepZK = function(t) {
                // check the /httpservers node exists...
                t.zkclient.exists('/httpservers', function(err, stat) {
                        if(err) {
                                console.log(err.stack);
                        } else {
                                if(!stat) {
                                        // create it if not.
                                        t.zkclient.mkdirp('/httpservers', function(err, pth) {
                                                if(err) {
                                                        console.log('[zkclient] Unable to create /httpservers node.');
                                                        console.log('[zkclient] %s', err.stack);
                                                        // log exception and exit because that node is absolutely necessary.
                                                        exit(1);
                                                }
                                        });
                                }
                                // signal that zk is ready and ephemeral nodes can now be created.
                                t.emit('zkclient:ready');
                        }
                });
        };

        this.on('zkclient:ready', function() {

                // Register this http server with the zk server
                this.zkclient.create(this.nodepath, zk.CreateMode.EPHEMERAL, function(err, pth) {
                        if(err) {
                                console.log('[zkclient] Failed to create node %s due to %s.', t.nodepath, err);
                        } else {
                                console.log('[zkclient] Node: %s is successfully created.', t.nodepath);
                        }
                });

        });
}

util.inherits(Server, events.EventEmitter);

Server.prototype.start = function() {

        var t = this;

        // when httpserver is ready...
        this.httpserver.on('listening', function() {
                console.log('[httpserver] listening on port %d.', httpport);

                // wait for the zkclient to be connected
                t.zkclient.once('connected', function() {
                        console.log('[zkclient] Connected to the zookeeper server.');

                        // and then prep the node tree
                        t.prepZK(t);
                });

                // connect to zookeeper server
                t.zkclient.connect()
        });

        this.httpserver.listen(httpport);
};

Server.prototype.stop = function() {
        this.zkclient.close();
        this.httpserver.close();
        exit(0);
};

var server = new Server();
server.start();

This code implements a very basic http server which when started will register with the zookeeper server. This is only one part of a solution which will be able to dynamically reconfigure ipvsadm based on ephemeral http server nodes.