Using dynamically generated configs with puppet

After using Puppet with an external node classifier for a while one starts questioning what other information could be generated by this instead of just YAML to feed the puppetmaster. When supervisor was being rolled out there was a need to a large number of near identical config files to be generated, however any special information about the configs really had no place in Puppet. So the solution to this was to have the Django app generate the config files and then have puppet pull them down with a custom parser.

In /var/lib/puppet/lib/puppet/parser/functions lives the file webcontent.rb which has the following contents:

require 'open-uri'
module Puppet::Parser::Functions
    newfunction(:webcontent, :type => :rvalue) do |args|
        server = args[0]
        configpath = args[1]
        config = ""
        beginopen( "http://#{server}/#{configpath}/" ) do |f|
             f.each_line do |line|
                 config = "#{config}#{line}"
             end
            endrescue OpenURI::HTTPError => e
            raise Puppet::ParseError, "404 for http://#{server}/#{configpath}/"
 
        rescue Exception => e
            raise Puppet::ParseError, "content string is http://#{server}/#{configpath}/ #{e}"
        end
        return config
    end
end

Using the Ruby module open-uri content is grabbed by the puppetmaster and placed into the catalog. Using the following Django model, view and template a config file is easily generated and passed along to Puppet

class SupervisorProgram(models.Model):
    name = models.CharField(max_length=128)
    command = models.CharField(max_length=512)
    autostart = models.BooleanField(default=True)
    autorestart = models.CharField(max_length=32,choices=(('false','false'),('true','true'),('unexpected','unexpected')))
    startsecs = models.IntegerField(default=10)
    startretries = models.IntegerField(default=3)
    exitcodes = models.CharField(max_length=64,default="0,2")
    stopsignal = models.CharField(max_length=5,choices=(('TERM','TERM'),('HUP','HUP'),('INT','INT'),('QUIT','QUIT'),('KILL','KILL'),('USR1','USR1'),('USR2','USR2')),default="TERM")
    stopwaitsecs = models.IntegerField(default=10)
    user = models.CharField(max_length=16 ,default="nagios")
    redirect_stderr = models.BooleanField(default=False)
    stdout_logfile = models.CharField(max_length=256,default="AUTO")
    stdout_logfile_maxbytes = models.CharField( max_length = 8,default="50MB")
    stdout_logfile_backups = models.IntegerField(default=10)
    stderr_logfile = models.CharField(max_length=256,default="AUTO")
    stderr_logfile_maxbytes = models.CharField( max_length = 8,default="50MB")
    stderr_logfile_backups = models.IntegerField(default=10)
    environment = models.CharField( max_length=512,blank=True,null=True)
    directory = models.CharField(max_length=128,default="/")
    umask = models.IntegerField(blank=True,null=True)
    priority = models.IntegerField(default=999)
 
    def __unicode__(self):
        return self.name
    class Meta:
        ordering = ('name',)
 
class SupervisorProgramAdmin(admin.ModelAdmin):
    list_display = ('name','command','autorestart','stopsignal','exitcodes','user','stdout_logfile','stderr_logfile')

The following is the view used:

def getSupervisorConfig(request,service):
    print "getSupervisorConfig has been called for %s" % service
    service = get_object_or_404(SupervisorProgram,name=service)
    directives = {}
    directives["command"] = str(service.command)
    directives["process_name"] = str(service.name)
    directives["priority"] = int(service.priority)
    directives["autostart" ] = service.autostart
    directives["autorestart"] = service.autorestart
    directives["startsecs"] = int(service.startsecs)
    directives["startretries"] = int(service.startretries)
    directives["exitcodes"] = str(service.exitcodes)
    directives["stopsignal"] = str(service.stopsignal)
    directives["stopwaitsecs"] = int(service.stopwaitsecs)
    directives["user"] = str(service.user)
    directives["redirect_stderr"] = service.redirect_stderr
    directives["stdout_logfile"] = str(service.stdout_logfile)
    directives["stdout_logfile_maxbytes"] = str(service.stdout_logfile_maxbytes)
    directives["stdout_logfile_backups"] = int(service.stdout_logfile_backups)
    directives["stderr_logfile"] = str(service.stderr_logfile)
    directives["stderr_logfile_maxbytes"] = str(service.stderr_logfile_maxbytes)
    directives["stderr_logfile_backups"] = int(service.stderr_logfile_backups)
    directives["directory"] = str(service.directory)
    if service.environment:
        directives["environment"] = str(service.environment)
 
    return render_to_response("sock/supervisor.conf",directives)

With the 20 configuration options per supervisord controlled process there are far too many options that should be sanely passed to puppetmaster from the external node classifier.

Here is the Django template:

#generated config
[program:{{ process_name }}]
command={{ command }}
process_name=%(program_name)s
priority={{ priority }}
autostart={{ autostart }}
autorestart={{ autorestart }}
startsecs={{ startsecs }}
startretries={{ startretries }}
exitcodes={{ exitcodes }}
stopsignal={{ stopsignal }}
stopwaitsecs={{ stopwaitsecs }}
user={{ user }}
redirect_stderr={{ redirect_stderr }}
stdout_logfile={{ stdout_logfile }}
stdout_logfile_maxbytes={{ stdout_logfile_maxbytes }}
stdout_logfile_backups={{ stdout_logfile_backups }}
stderr_logfile={{ stderr_logfile }}
stderr_logfile_maxbytes={{ stderr_logfile_maxbytes }}
stderr_logfile_backups={{ stderr_logfile_backups }}
{% if environment %}
environment={{ environment }}
{% endif %}

Finally all of this can be referenced with a custom define as follows:

define supervisorconfig(
        $program,
        $server = "${rserver}"
) {
    file {
        "${name}.conf":
            owner => root,
            group => root,
            mode => 0644,
            path => "/etc/supervisord.d/${name}.conf",
            content => webcontent( $server, "dpuppet/sock3/getsupervisorconfig/$program")
    }
}

Database Fixtures for Isolated Testing in PHP

Long ago, Ryan wrote the history of our fixture frameworks. Now, you too can have the awesome Team Lazer Beez database fixture for your own project. With the release of Team Lazer Beez Open Source (formerly Genius Open Source) version 1.2, our YAML-backed, easy-to-setup fixture framework has been integrated into the gosTest framework.

Here is a simple example from our Fixture package:

<?php
// Include the Genius config file
require_once dirname(dirname(__FILE__)) . '/Core/gosConfig.inc.php';
 
/**
 * A function to get single values from a database table
 */
function getThingFromDB($id) {
    $db = gosDB_Helper::getDBByName('main');
 
    return $db->getOne("SELECT s1 FROM fixture_test WHERE i1 = " . $id);
}

In a nearby test file:

<?php
// Include the Genius fixture configuration. This geneartes a database
// and applies the schema to it.  See fixtureTestConfig.inc.php
// for more details.
require_once(GOS_ROOT . 'Fixture/fixtureTestConfig.inc.php');
 
function testGetThingFromDB() {
    // Create a fixture
    $fixture = gosTest_Fixture_Controller::getByDBName('main');
 
    // Load the fixture into the database
    $fixture->parseFixtureFile(GOS_ROOT . 'Fixture/example_fixture.yaml');
 
    // Directly access the fixture, which is identical to what the database contains
    $idToGet = $fixture->get('fixtureName.fixture_test.i1');
 
    // Pull the value we want directly from the fixture
    $fixtureThing = $fixture->get('fixtureName.fixture_test.s1');
    // Pull the value we want from the DB via the function we're testing
    $thing = getThingFromDB($idToGet);
 
    echo "The fixture put $fixtureThing into the DB.\n";
    echo "Our function selected $thing from the DB.\n";
}
 
// Run the test
testGetThingFromDB();

The first magical line here is that require_once—fixtureTestConfig.inc.php uses the database connection information defined in Fixture/fixtureConfig.yaml to create a database is used for the duration of this PHP process only. Subsequent test runs (or invocations of this example script) will generate an entirely new database. See gosDB_AutoDBGenerator for details on these ephemeral databases. The takeaway is that you must have a user listed in that fixture config YAML file that can create databases.

In the test function itself, we create a fixture on the appropriate database, in this case our main database, and load the data from our fixture file, a simple YAML file defining what data we want in the database:

# The name of this fixture
fixtureName:
    # A table we will insert data into
    fixture_test:
        i1: 11
        s1: str1

At this point, we’re ready to start testing. First we get the ID that we need to pass to getThingFromDB, which we simply pull from the fixture. Next, we get the value that we are expecting from the same table in the fixture. Now that we have the ID we want to give to the function we’re testing, and the value we expect to get back, we can execute that function and compare the results. Go ahead and try this all for yourself; the above example works all on its own. The easiest way is to download the example.php and the necessary YAML file.

The gosFixture class automatically cleans up the database tables that it touched (see Core/lib/gosTest/Framework/TestCase.acls.php), so you need to re-apply the fixture for every test. The easiest way to do this is to put the fixture initialization in the setUpExtension() of a test class, giving you completely fresh fixture data for each and every test.

So go forth and test your code with better isolation and known clean data.

Puppet External Node Classifier

In Puppet the initial method of holding information about your machines is through the site.pp config file, this rapidly becomes tiresome when you have more than 5 servers. This is where an external node classier comes in as a handy tool.

The first step in setting up an external node classifier is developing what will be generating the YAML for the puppetmaster to read. In this situation Django chosen as a web framework as the built in admin interface saved a bit of development time for management. Additional thought behind using an ORM framework was cutting down on development time on other projects that might need access to the truth database and could simply pull YAML from it. Within the Django application there are classes describing hosts, environments, server class, puppet classes, hard drive and filesystem layout. Here is the class describing a host:

class Host(models.Model):
    hostname = models.CharField(null=True,blank=False,max_length=128)
    basename = models.CharField(null=True,blank=True,max_length=128)
    maintenance = models.BooleanField(default=False)
    datacenter = models.CharField(max_length=5,choices=(('sjc','san jose'),
        ('smc','server room')),blank=True,null=True)
    environment = models.CharField(max_length=5,choices=(('app','production'),
        ('stg1','staging 1'),('stg2','staging 2'),('dev','development')),blank=True,null=True)
    env = models.ForeignKey(Environment)
    ostype = models.CharField(max_length=32,choices=(('linux','linux'),
        ('windows','windows'),('mac','mac')),blank=True,null=True)
    info = models.TextField(blank=True,null=True)
    bit = models.CharField(blank=True,null=True,max_length=5,choices=(('32','32 bit'),('64','64 bit')))
    puppetrun = models.IntegerField(default=1800)
    serverclass = models.ForeignKey(SvrClass)
 
    classes = models.ManyToManyField(PuppetClass,blank=True,null=True)
    drives = models.ManyToManyField(Drive,blank=True,null=True)
    interfaces = models.ManyToManyField(NetInterface,blank=True,null=True)
    script = models.ManyToManyField(Script,blank=True,null=True)
    osver = models.ManyToManyField(Osversion,blank=True,null=True)
    services = models.ManyToManyField(Service,blank=True,null=True)
    supervisors = models.ManyToManyField(SupervisorProgram,blank=True,null=True)
 
    def getBranch(self):
        return self.env.branch.name
 
    def getEnvironment(self):
        return self.env.name
 
    def __str__(self):
        return self.__repr__()
 
    def __repr__(self):
        return "hostname: %s environment: %s" % (self.hostname,self.env.name)
 
class HostAdmin(admin.ModelAdmin):
    list_display    =   ('hostname','environment','bit','ostype','datacenter','maintenance')
    list_filter     =   ('environment','datacenter')
    ordering        =   ('hostname','datacenter','environment')
    search_fields   =   ('hostname',)

and now the code for the Puppet class in Python, the Puppet class class:

class PuppetClass(models.Model):
    name = models.CharField(max_length=128)
    def __unicode__(self):
        return self.name
    class Meta:
        ordering = ('name',)
 
class PuppetClassAdmin(admin.ModelAdmin):
    list_display = ('name',)

With these two classes a host and what Puppet classes it has assigned to it can readily be described and the following code will output YAML describing the host as the puppetmaster understands:

import yaml
 
def hostinfo(request,hostname):
    host = get_object_or_404(Host,hostname=hostname)
    classlist = []
    [classlist.append(str(x.name)) for x in host.classes.all()]
    #if a host's classlist is empty at this point pull from the default list
    if not classlist:
        #default list is simply a list of puppet class objects
        defaultlist = DefaultList.objects.get(name="standard")
        [classlist.append(str(x.name)) for x in defaultlist.classes.all()]
    #add in some extra parameters
    params = {'datacenter':str(host.datacenter),'machine':str(host.basename),'serverclass':str(host.serverclass.name),'env':str(host.env.name),'branch':'%s' % str(host.env.branch.name),'envmaint':str(host.env.maintenance),'hostmaint':str(host.maintenance)}
    params['rserver'] = str(host.env.rserver)
    yamlsrc = yaml.dump({'classes':classlist,'parameters':params})
    return  render_to_response("sock/hostinfo.yaml",{'hostname':hostname,'yaml':yamlsrc})

One issue that was encountered with exporting data from django to yaml was the usage of utf for strings, hence the continual usage of str(). The result of the function is dumped out through the following incredibly complex template

---
{{ yaml }}

At the end of all of this we finally get data that comes out as something similar to the following:

---
classes: [bots, puppet-classes, puppet-master, repos, yamlsvn]
parameters: {branch: '33', datacenter: sjc, env: app, envmaint: '0', hostmaint: '0',
  machine: repo, rserver: sjc-repo.genops.net, serverclass: none}

Next we need a command that the puppetmaster can run to get the YAML, here is the current script:

#!/bin/sh
curl http://example.genius.com/dpuppet/sock3/hostinfo/$1/ 2>/dev/null | \
sed "s/&#39;/'/g"
exit 0;

Finally the puppetmaster needs to be configured to pull from the external node classifier:

[main]
    # Where Puppet stores dynamic and growing data.
    # The default value is '/var/puppet'.
    vardir = /var/lib/puppet
 
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet
 
    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet
 
    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl
    # use external nodes ftw
    external_nodes = /usr/local/bin/puppetinfo.sh
    node_terminus = exec
 
[puppetd]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt
 
    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig
    server =  puppetmaster.genius.com
    runinterval = 300

Puppet at Genius

Puppet is a configuration management system  that was created with the goal of making more portions of system administration tasks reuseable. At Genius puppet is used in all of our environments: production, staging, and development. Within this setup each environment has its own puppetmaster, with each master pulling against its own version of the puppet classes. All of the masters in turn pull all of their information nodes from a central external node classifier.

The masters act as their own clients, syncing against themselves with a puppet class that controls checkouts of an svn repository holding all puppet classes.  All of the servers have been configured with custom defines to handle subervision checkouts and configs for supervisor, additionally a custom parser has been written to allow pulling down webpages as part a content description. All servers run on an interval of 5 minutes with clients running on a 15 minute interval, with some special exceptions for dns servers.  Additionally the web app that powers the external node classier also handles generating config files and maintaining a truth database for other tasks, like imaging.

This is the first in a series of post regarding our setup, following posts will cover our external node classifier, truth database, customer parser, custom defines.

Using HornetQ without a separate JNDI server

HornetQ is JBoss’s latest messaging product. It provides a JMS implementation as well as its own alternate API. JMS is the Java Message Service: a set of APIs built around asynchronous messaging through topics (think bulletin boards) and queues (self-explanatory). For more, see the standard JMS tutorial.

This tutorial shows how to access HornetQ JMS resources via JNDI without needing a separate JNDI server. It also provides a test utility to run an embedded HornetQ server (perfect for unit tests that need a JMS broker running!).

Why JNDI?

Since JMS is part of Java EE, you’ll often see tutorials that have JMS resources (e.g. connection factories or queues) being provided to your code is via JNDI. This isn’t the only choice, though: you could instead manually instantiate your preferred provider’s implementations of the resources you need.

If you choose the latter approach, you might use the following to get a Queue if you were using ActiveMQ (another JMS implementation):

Queue queue = new ActiveMQQueue("queueName");

This works fine until you switch to another provider, at which point you need to change all your queues (and connection factories and topics and…) to look like this:

Queue queue = new HornetQQueue("queueName");

This is why it’s recommended to instead pull those objects out of JNDI. If you’re getting the queue out of JNDI with the following code, you wouldn’t need to change anything if you switch providers other than changing how the JNDI context gets populated.

Context context = new InitialContext();
Queue queue;
try {
    queue = (Queue) context.lookup("jndiNameForQueue");
} finally {
    // release your context's resources when you're done with it
    context.close();
}

This isn’t meant to be a JNDI tutorial, but it should be clear why getting JMS resources (as well as things like JDBC DataSource objects and other “managed” objects) out of JNDI is a good thing: it makes it easier to program to interfaces, not implementations, and maintain vendor neutrality.

HornetQ’s JNDI support

HornetQ comes with all you need to be able to connect to a JNDI server and pull JMS resources out of that server. You can set up the HornetQ server to serve JNDI in addition to JMS by enabling the appropriate bean in hornetq-beans.xml.

This approach is straightforward, but it becomes problematic if you want your HornetQ servers to be in a HA (high availability) pair. It’s hard enough to configure and test HA without also needing JNDI service to fail over as well. Though there is some documentation on how to set up HA JNDI, it would be simpler (and surely faster as well) if there was no need to talk to a server at all to get JMS resources. This approach makes sense if your deployment setup is simple enough that you know which queues and JMS servers you will be using and can put that information in a few config files and bundle it with your code during deployment. This is the case for many uses of JMS. If, on the other hand, you’re already tied to using a Java EE server that provides JNDI, then you might as well use that.

ActiveMQ provides a simple way to configure JMS resources in local JNDI, but HornetQ doesn’t come with anything similar. This tutorial provides a way to do local JNDI with HornetQ.

Local JNDI Configuration

The starting point to accessing JNDI objects is creating a new InitialContext. The specific way that the resulting Context gets populated with data is controlled by the implementation of InitialContextFactory that you’re using. You can change the implementation by specifying the java.naming.factory.initial property in jndi.properties to be the fully qualified class name of an impementation of InitialContextFactory.

We can use this mechanism to specify an implementation of InitialContextFactory that reads HornetQ’s config files. More sophisticated implementations might produce a Context that accesses data on some remote server, but our goal here is something that reads data out of config files and populates a memory-only Context. We’ll need to read the core HornetQ config file to get connection factory information and the JMS HornetQ config file to get the JNDI names to assign to queues, topics, etc. We’ll also need something to actually create a Context implementation since we don’t want to have to write that ourselves. Implementing Context requires a non-trivial amount of work to do well. simple-jndi provides a basic in-memory InitialContextFactory, so we’ll use that.

Your jndi.properties would look something like this:

java.naming.factory.initial = com.genius.hornetq.XmlHornetQInitialContextFactory
hornetq.jndi.wrapped.initialcontextfactory.impl = org.osjava.sj.memory.MemoryContextFactory
hornetq.xml.jms.path = /path-to-hornetq-jms.xml
hornetq.xml.config.path = /path-to-hornetq-config.xml

The java.naming.factory.initial tells InitialContext which class to instantiate to return the underlying Context (look at the source that comes with the JDK if you’re curious how). In this case, we want InitialContext to use our custom implementation that reads HornetQ XML files.

The other properties are only relevant to our custom InitialContextFactory. See the HornetQ documentation for more about what to put in the XML config files.
hornetq.jndi.wrapped.initialcontextfactory.impl defines what InitialContextFactory we’ll use to actually create a Context object. In this case, it’s the simple-jndi memory-only InitialContextFactory.
hornetq.xml.jms.path is the path in the classpath to the HornetQ JMS config file. This is typically “hornetq-jms.xml”.
hornetq.xml.config.path is the path in the classpath to the HornetQ core config file. This is typically “hornetq-configuration.xml”.

Now that we’ve got jndi.properties ready, I’ve added the custom InitialContextFactory source below. The only tricky part is handling HornetQ’s LogDelegateFactory selection. The LogDelegateFactory system lets you change the logging system used by HornetQ and is configurable via the core config file. (I’d link to the documentation, but this option is undocumented. See FileConfigurationParser#parseMainConfig() in the HornetQ source.) Unfortunately, a bug in the way the LogDelegateFactory instance is set causes some classes to not properly pick up your custom implementation, so if you wish to use a custom LogDelegateFactory, note the commented out call to Logger.setDelegateFactory. Other than that, it’s pretty simple. We use HornetQ’s configuration parsing code to get the information we need out of the config files, then populate the Context we get from simple-jndi with the resulting JMS objects.

The only dependency outside of HornetQ is SLF4J for logging.

package com.genius.hornetq;
 
import org.hornetq.api.core.Pair;
import org.hornetq.api.core.TransportConfiguration;
import org.hornetq.api.jms.HornetQJMSClient;
import org.hornetq.core.config.Configuration;
import org.hornetq.core.deployers.impl.FileConfigurationParser;
import org.hornetq.jms.client.HornetQConnectionFactory;
import org.hornetq.jms.client.HornetQDestination;
import org.hornetq.jms.server.JMSServerConfigParser;
import org.hornetq.jms.server.config.ConnectionFactoryConfiguration;
import org.hornetq.jms.server.config.JMSConfiguration;
import org.hornetq.jms.server.config.JMSQueueConfiguration;
import org.hornetq.jms.server.config.TopicConfiguration;
import org.hornetq.jms.server.impl.JMSServerConfigParserImpl;
import org.hornetq.spi.core.logging.LogDelegateFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import javax.jms.Queue;
import javax.jms.Topic;
import javax.naming.Context;
import javax.naming.NamingException;
import javax.naming.spi.InitialContextFactory;
import java.io.InputStream;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Hashtable;
import java.util.List;
import java.util.Map;
 
/**
 * Parses hornetq jms and core configuration files and populates the resulting Context appropriately.
 *
 * The following properties are needed:
 *
 * hornetq.xml.jms.path = path in the classpath to hornetq jms xml file
 *
 * hornetq.xml.config.path = path in classpath to hornetq core config file
 *
 * hornetq.jndi.wrapped.initialcontextfactory.impl = fully qualified classname of InitialContextFactory to use to
 * provide the Context instance that will be populated with configured JMS objects. An instance of this class will be
 * created and passed an empty environment.
 */
public class XmlHornetQInitialContextFactory implements InitialContextFactory {
 
    private static final String JMS_XML_PATH_KEY = "hornetq.xml.jms.path";
    private static final String CONFIG_XML_PATH_KEY = "hornetq.xml.config.path";
    private static final String INTERNAL_ICF_IMPL_KEY = "hornetq.jndi.wrapped.initialcontextfactory.impl";
    private static final Logger logger = LoggerFactory.getLogger(XmlHornetQInitialContextFactory.class);
 
    public XmlHornetQInitialContextFactory() {
        logger.trace("Instantiated");
    }
 
    @Override
    public Context getInitialContext(Hashtable<?, ?> environmentHashtable) throws NamingException {
        logger.trace("Creating InitialContext");
        Map<string, String> env = new HashMap<string, String>();
        for (Map.Entry<?, ?> entry : environmentHashtable.entrySet()) {
            if (entry.getKey() instanceof String && entry.getValue() instanceof String) {
                env.put((String) entry.getKey(), (String) entry.getValue());
            }
        }
 
        this.checkKeys(env, JMS_XML_PATH_KEY, INTERNAL_ICF_IMPL_KEY, CONFIG_XML_PATH_KEY);
 
        Context ctx = this.getContext(env);
 
        populateContext(env, ctx);
 
        return ctx;
    }
 
    private void populateContext(Map<string, String> env, Context ctx) throws NamingException {
        final InputStream configXmlStream = getStream(env.get(CONFIG_XML_PATH_KEY));
 
        /*
        * A bug in HQ 2.1.2 means that the log delegate factory will not be applied to any classes that use static
        * loggers and were loaded before the config-specified delegate factory is applied. A fix is scheduled for 2.2.0.
         * Until then, we'll hardcode the delegate factory to slf4j for the first few loggers.
        */
        //org.hornetq.core.logging.Logger.setDelegateFactory(new YourCustomLogDelegateFactory());
 
        final Configuration config;
        try {
            config = new FileConfigurationParser().parseMainConfig(configXmlStream);
        } catch (Exception e) {
            throw getNamingException("Couldn't get configuration from xml", e);
        }
 
        final LogDelegateFactory logDelegateFactory =
                (LogDelegateFactory) instantiate(config.getLogDelegateFactoryClassName());
        org.hornetq.core.logging.Logger.setDelegateFactory(logDelegateFactory);
 
 
        final InputStream jmsXmlStream = getStream(env.get(JMS_XML_PATH_KEY));
        final JMSServerConfigParser jmsServerConfigParser = new JMSServerConfigParserImpl();
        JMSConfiguration jmsConfig;
        try {
            jmsConfig = jmsServerConfigParser.parseConfiguration(jmsXmlStream);
        } catch (Exception e) {
            throw getNamingException("Couldn't get jms info from xml", e);
        }
 
        for (ConnectionFactoryConfiguration cfConfig : jmsConfig.getConnectionFactoryConfigurations()) {
            final HornetQConnectionFactory hqcf =
                    getHornetQConnectionFactory(cfConfig, config.getConnectorConfigurations());
            for (String jndiName : cfConfig.getBindings()) {
                ctx.bind(jndiName, hqcf);
            }
        }
 
        for (JMSQueueConfiguration queueConfiguration : jmsConfig.getQueueConfigurations()) {
            Queue queue = HornetQDestination.createQueue(queueConfiguration.getName());
            for (String jndiName : queueConfiguration.getBindings()) {
                ctx.bind(jndiName, queue);
            }
        }
 
        for (TopicConfiguration topicConfiguration : jmsConfig.getTopicConfigurations()) {
            Topic topic = HornetQDestination.createTopic(topicConfiguration.getName());
            for (String jndiName : topicConfiguration.getBindings()) {
                ctx.bind(jndiName, topic);
            }
        }
    }
 
    private InputStream getStream(String xmlPath) throws NamingException {
        final InputStream xmlStream = this.getClass().getResourceAsStream(xmlPath);
        if (xmlStream == null) {
            throw new NamingException("Cannot find resource at <" + xmlPath + ">");
        }
        return xmlStream;
    }
 
    private Context getContext(Map<string, String> env) throws NamingException {
        final String icfClassName = env.get(INTERNAL_ICF_IMPL_KEY);
 
        InitialContextFactory icf = (InitialContextFactory) instantiate(icfClassName);
 
        //noinspection UseOfObsoleteCollectionType
        return icf.getInitialContext(new Hashtable<object, Object>());
    }
 
    /**
     * Instantiate the supplied class name via its 0-arg ctor.
     *
     * @param className class name to instantiate
     *
     * @return instance of className
     *
     * @throws NamingException if the class cannot be instantiated
     */
    private Object instantiate(String className) throws NamingException {
        Object obj;
        try {
            final Class<?> klass = Class.forName(className);
            final Constructor<?> ctor = klass.getConstructor();
            obj = ctor.newInstance();
        } catch (ClassNotFoundException e) {
            throw getNamingException("Couldn't find class " + className, e);
        } catch (NoSuchMethodException e) {
            throw getNamingException("Couldn't find 0-arg constructor for " + className, e);
        } catch (InvocationTargetException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        } catch (InstantiationException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        } catch (IllegalAccessException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        }
        return obj;
    }
 
    private static NamingException getNamingException(String message, Exception e) {
        final NamingException ne = new NamingException(message);
        ne.initCause(e);
        return ne;
    }
 
    /**
     * @param env  the env to look in
     * @param keys the keys to check for
     *
     * @throws NamingException if any key is not found in the env or if the value is null
     */
    private void checkKeys(Map<string, String> env, String... keys) throws NamingException {
        for (String key : keys) {
            if (env.get(key) == null) {
                throw new NamingException("Couldn't find " + key + " property in " + env);
            }
        }
    }
 
    static HornetQConnectionFactory getHornetQConnectionFactory(ConnectionFactoryConfiguration cfConfig,
            Map<string, TransportConfiguration> connectorConfigurations) throws NamingException {
 
        /*
         * Implementation largely lifted from JMSServerManagerImpl
         */
 
        List<pair<transportConfiguration, TransportConfiguration>> connectorConfigs =
                new ArrayList<pair<transportConfiguration, TransportConfiguration>>();
 
        for (Pair<string, String> configConnector : cfConfig.getConnectorNames()) {
            String connectorName = configConnector.a;
            String backupConnectorName = configConnector.b;
 
            TransportConfiguration connector = connectorConfigurations.get(connectorName);
 
            if (connector == null) {
                throw new NamingException("No configured connector with name <" + connectorName + ">");
            }
 
            logger.debug("Found connector config for <" + connectorName + ">");
            TransportConfiguration backupConnector = null;
 
            if (backupConnectorName != null) {
                backupConnector = connectorConfigurations.get(backupConnectorName);
 
                if (backupConnector == null) {
                    throw new NamingException("No configured backup connector with name <" + connectorName + ">");
                }
 
                logger.debug("Found backup connector config for <" + backupConnectorName + ">");
            } else {
                logger.debug("Not using a backup connector for main connector <" + connectorName + ">");
            }
 
            connectorConfigs.add(new Pair<transportConfiguration, TransportConfiguration>(connector, backupConnector));
        }
 
 
        HornetQConnectionFactory cf = HornetQJMSClient.createConnectionFactory(connectorConfigs);
        cf.setClientID(cfConfig.getClientID());
        cf.setClientFailureCheckPeriod(cfConfig.getClientFailureCheckPeriod());
        cf.setConnectionTTL(cfConfig.getConnectionTTL());
        cf.setCallTimeout(cfConfig.getCallTimeout());
        cf.setCacheLargeMessagesClient(cfConfig.isCacheLargeMessagesClient());
        cf.setMinLargeMessageSize(cfConfig.getMinLargeMessageSize());
        cf.setConsumerWindowSize(cfConfig.getConsumerWindowSize());
        cf.setConsumerMaxRate(cfConfig.getConsumerMaxRate());
        cf.setConfirmationWindowSize(cfConfig.getConfirmationWindowSize());
        cf.setProducerWindowSize(cfConfig.getProducerWindowSize());
        cf.setProducerMaxRate(cfConfig.getProducerMaxRate());
        cf.setBlockOnAcknowledge(cfConfig.isBlockOnAcknowledge());
        cf.setBlockOnDurableSend(cfConfig.isBlockOnDurableSend());
        cf.setBlockOnNonDurableSend(cfConfig.isBlockOnNonDurableSend());
        cf.setAutoGroup(cfConfig.isAutoGroup());
        cf.setPreAcknowledge(cfConfig.isPreAcknowledge());
        cf.setConnectionLoadBalancingPolicyClassName(cfConfig.getLoadBalancingPolicyClassName());
        cf.setTransactionBatchSize(cfConfig.getTransactionBatchSize());
        cf.setDupsOKBatchSize(cfConfig.getDupsOKBatchSize());
        cf.setUseGlobalPools(cfConfig.isUseGlobalPools());
        cf.setScheduledThreadPoolMaxSize(cfConfig.getScheduledThreadPoolMaxSize());
        cf.setThreadPoolMaxSize(cfConfig.getThreadPoolMaxSize());
        cf.setRetryInterval(cfConfig.getRetryInterval());
        cf.setRetryIntervalMultiplier(cfConfig.getRetryIntervalMultiplier());
        cf.setMaxRetryInterval(cfConfig.getMaxRetryInterval());
        cf.setReconnectAttempts(cfConfig.getReconnectAttempts());
        cf.setFailoverOnInitialConnection(cfConfig.isFailoverOnInitialConnection());
        cf.setFailoverOnServerShutdown(cfConfig.isFailoverOnServerShutdown());
        cf.setGroupID(cfConfig.getGroupID());
 
        return cf;
    }
}

Testing JMS code

As a bonus for reading this far, here’s JmsTestServerManager. This class makes it easy to start and stop an embedded HornetQ server. This is ideal for testing code that reads from or writes to JMS queues or topics, for instance. You should create one instance per test class (via @BeforeClass if you’re using JUnit 4) and call start() in setup (@Before) and stop() in teardown (@After). This means you’ll have a fresh server instance for each test. No more needing to clean up leftover messages in queues between each test! Fortunately, HornetQ stops and starts quite quickly so this is unlikely to be a performance problem for your tests.

 
package com.genius.testutil.jms;
 
import net.jcip.annotations.NotThreadSafe;
import org.hornetq.api.core.TransportConfiguration;
import org.hornetq.core.config.Configuration;
import org.hornetq.core.config.impl.ConfigurationImpl;
import org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory;
import org.hornetq.core.server.HornetQServer;
import org.hornetq.core.server.HornetQServers;
import org.hornetq.jms.server.JMSServerConfigParser;
import org.hornetq.jms.server.JMSServerManager;
import org.hornetq.jms.server.config.JMSConfiguration;
import org.hornetq.jms.server.config.JMSQueueConfiguration;
import org.hornetq.jms.server.config.TopicConfiguration;
import org.hornetq.jms.server.impl.JMSServerConfigParserImpl;
import org.hornetq.jms.server.impl.JMSServerManagerImpl;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.io.InputStream;
import java.util.HashSet;
import java.util.Set;
 
/*
 * See org.hornetq.tests.util.JMSTestBase
 */
 
/**
 * Runs an embedded HornetQ server suitable for use in unit tests that need JMS. It should be re-used across test
 * methods; just call start() in @Before and stop() in @After.
 */
@NotThreadSafe
public class JmsTestServerManager {
    private static final Logger logger = LoggerFactory.getLogger(JmsTestServerManager.class);
 
    private static final String INVM_ACCEPTOR_FACTORY = InVMAcceptorFactory.class.getCanonicalName();
 
    private static final String DEFAULT_HORNETQ_TEST_JMS_XML = "/hornetq-test-jms.xml";
 
    /**
     * Queues to add when the server is started
     */
    @NotNull
    private final Set<string> queueNames;
 
    /**
     * Topics to add when the server is started
     */
    @NotNull
    private final Set<string> topicNames;
 
    /**
     * The context passed to the jms server (proxied to prevent close()). No objects are registered in it; it is simply
     * provided to the server so that the server doesn't try to create its own.
     */
    @NotNull
    private final Context context;
 
    /**
     * null if the server is stopped
     */
    @Nullable
    private JMSServerManager jmsServer;
 
    /**
     * null if the server is stopped
     */
    @Nullable
    private HornetQServer server;
 
    /**
     * @param context    the context to give to the server.
     * @param queueNames set of queue names to create in the server
     * @param topicNames set of topic names to create in the server
     */
    private JmsTestServerManager(@NotNull Context context, @NotNull Set<string> queueNames,
            @NotNull Set<string> topicNames) {
        this.queueNames = queueNames;
 
        // avoid having the server initialize its own InitialContext, and don't let it close the InitialContext
        // since we want to keep it open across invocations
 
        this.context = new ReadOnlyContextProxy(context);
        this.topicNames = topicNames;
    }
 
    /**
     * @param context      The JNDI context to pass to the server
     * @param pathToJmsXml The classpath path to the hornetq jms file to read topics and queues from
     *
     * @return a configured JmsTestServerManager in the stopped state
     */
    static JmsTestServerManager getNew(@NotNull Context context, @NotNull String pathToJmsXml) {
        final JMSConfiguration configuration = getJmsConfiguration(pathToJmsXml);
        return new JmsTestServerManager(context, getQueueNames(configuration), getTopicNames(configuration));
    }
 
    /**
     * Create a new InitialContext and read queues & topics from the default xml path.
     *
     * @return a configured JmsTestServerManager in the stopped state
     *
     * @throws NamingException if an InitialContext can't be created
     * @see JmsTestServerManager#getNew(Context, String)
     */
    static JmsTestServerManager getNew() throws NamingException {
        // It's ok to not close this context since we want it to be left open
        //noinspection JNDIResourceOpenedButNotSafelyClosed
        return getNew(new InitialContext(), DEFAULT_HORNETQ_TEST_JMS_XML);
    }
 
    /**
     * Provides access to the context used by this object to avoid needing to re-create it.
     *
     * @return the context used by this reader
     */
    @NotNull
    public Context getContext() {
        return this.context;
    }
 
    /**
     * Start the server and add all queues to it.
     */
    public void start() {
        if (this.server != null) {
            throw new IllegalStateException("Server already started");
        }
 
        this.server = HornetQServers.newHornetQServer(createDefaultConfig());
        try {
            this.jmsServer = new JMSServerManagerImpl(this.server);
            this.jmsServer.setContext(this.context);
            this.jmsServer.start();
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't start JMS server", e);
        }
 
        try {
            for (String queue : this.queueNames) {
                logger.debug("Adding queue " + queue);
                this.jmsServer.createQueue(false, queue, null, false);
            }
            for (String topicName : this.topicNames) {
                logger.debug("Adding topic " + topicName);
                this.jmsServer.createTopic(false, topicName);
            }
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't create destination", e);
        }
    }
 
    /**
     * Shut down the server.
     */
    public void stop() {
        if (this.server == null) {
            throw new IllegalStateException("Server not started");
        }
 
        try {
            this.jmsServer.stop();
            this.server.stop();
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't stop JMS server", e);
        }
 
        this.jmsServer = null;
        this.server = null;
    }
 
    private static JMSConfiguration getJmsConfiguration(String pathToJmsXml) {
        final InputStream xmlStream = JmsTestServerManager.class.getResourceAsStream(pathToJmsXml);
        if (xmlStream == null) {
            throw new YourCustomTestException("Couldn't find hornetq jms xml");
        }
 
        final JMSServerConfigParser jmsServerConfigParser = new JMSServerConfigParserImpl();
        JMSConfiguration jmsConfig;
        try {
            jmsConfig = jmsServerConfigParser.parseConfiguration(xmlStream);
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't get jms info from xml", e);
        }
 
        return jmsConfig;
    }
 
    private static Set<string> getTopicNames(JMSConfiguration jmsConfig) {
        Set<string> topicNames = new HashSet<string>();
        for (TopicConfiguration topicConfiguration : jmsConfig.getTopicConfigurations()) {
            topicNames.add(topicConfiguration.getName());
        }
 
        return topicNames;
    }
 
    private static Set<string> getQueueNames(JMSConfiguration jmsConfig) {
        Set<string> queueNames = new HashSet<string>();
        for (JMSQueueConfiguration jmsQueueConfiguration : jmsConfig.getQueueConfigurations()) {
            queueNames.add(jmsQueueConfiguration.getName());
        }
 
        return queueNames;
    }
 
    private Configuration createDefaultConfig() {
        Configuration configuration = new ConfigurationImpl();
        configuration.setSecurityEnabled(false);
        configuration.setJMXManagementEnabled(true);
        configuration.setPersistenceEnabled(false);
 
        configuration.setFileDeploymentEnabled(false);
 
        configuration.getAcceptorConfigurations().add(new TransportConfiguration(INVM_ACCEPTOR_FACTORY));
 
        // avoid the log about using the default cluster password
        configuration.setClusterPassword("asdf");
        configuration.setLogDelegateFactoryClassName(SLF4JLogDelegateFactory.class.getCanonicalName());
 
        return configuration;
    }
}

You may have noticed ReadOnlyContextProxy being used in JmsTestServerManager. That’s a rather boring Context implementation that proxies all read-only method calls to an internal Context instance and has a no-op close() implementation. This is to prevent HornetQ from closing the Context since we want to keep the Context open through all test method invocations to prevent pointlessly re-creating an InitialContext every time.

The rather uninteresting source of ReadOnlyContextProxy follows to save you from writing your own.

package com.genius.testutil.jms;
 
import net.jcip.annotations.NotThreadSafe;
 
import javax.naming.Binding;
import javax.naming.Context;
import javax.naming.Name;
import javax.naming.NameClassPair;
import javax.naming.NameParser;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import java.util.Hashtable;
 
/**
 * Proxies Context methods to let us control what happens to a Context that we pass in to the embedded JMS server.
 */
@NotThreadSafe
class ReadOnlyContextProxy implements Context {
 
    private final Context context;
 
    ReadOnlyContextProxy(Context context) {
        this.context = context;
    }
 
    @Override
    public void close() throws NamingException {
        // no op
    }
 
    /*
     * State-change methods not allowed. Could also no-op them if the exception becomes problematic, but
     * org.hornetq.tests.unit.util.InVMContext UOEs on many operations.
     */
 
    @Override
    public void bind(Name name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void bind(String name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rebind(Name name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rebind(String name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void unbind(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void unbind(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rename(Name oldName, Name newName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rename(String oldName, String newName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void destroySubcontext(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void destroySubcontext(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Object addToEnvironment(String propName, Object propVal) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Object removeFromEnvironment(String propName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Context createSubcontext(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Context createSubcontext(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    /*
     * Read-only methods left alone.
     */
 
    @Override
    public NamingEnumeration<nameClassPair> list(Name name) throws NamingException {
        return context.list(name);
    }
 
    @Override
    public NamingEnumeration<nameClassPair> list(String name) throws NamingException {
        return context.list(name);
    }
 
    @Override
    public NamingEnumeration<binding> listBindings(Name name) throws NamingException {
        return context.listBindings(name);
    }
 
    @Override
    public NamingEnumeration<binding> listBindings(String name) throws NamingException {
        return context.listBindings(name);
    }
 
    @Override
    public Object lookupLink(Name name) throws NamingException {
        return context.lookupLink(name);
    }
 
    @Override
    public Object lookupLink(String name) throws NamingException {
        return context.lookupLink(name);
    }
 
    @Override
    public NameParser getNameParser(Name name) throws NamingException {
        return context.getNameParser(name);
    }
 
    @Override
    public NameParser getNameParser(String name) throws NamingException {
        return context.getNameParser(name);
    }
 
    @Override
    public Name composeName(Name name, Name prefix) throws NamingException {
        return context.composeName(name, prefix);
    }
 
    @Override
    public String composeName(String name, String prefix) throws NamingException {
        return context.composeName(name, prefix);
    }
 
    @Override
    public Hashtable<?, ?> getEnvironment() throws NamingException {
        return context.getEnvironment();
    }
 
    @Override
    public Object lookup(Name name) throws NamingException {
        return context.lookup(name);
    }
 
    @Override
    public Object lookup(String name) throws NamingException {
        return context.lookup(name);
    }
 
    @Override
    public String getNameInNamespace() throws NamingException {
        return context.getNameInNamespace();
    }
}

With these classes, you’ve got all you need to get started writing JMS code (and testing it, too).

Making PHP Regex Errors Real

PHP employs Perl Compatible Regular Expressions (PCRE) in the built-in collection of preg_* functions, such as preg_match(). While PCRE is certainly the preferred regular expression library, PHP’s implementation allows the functions to fail without any explicit warning—the user must check preg_last_error() to know that an error occurred. Often, the return of a regular expression match is checked, and different operations are performed if the regex matched or not.

/**
 * Find primes with a regex.
 * http://montreal.pm.org/tech/neil_kandalgaonkar.shtml
 */
function isPrime($num) {
    $num = str_repeat('1', $num);
    $ret = preg_match('/^1?$|^(11+?)\1+$/', $num);
 
    echo 'Return value is ';
    var_dump($ret);
 
    if ($ret === 0) {
        echo "Prime\n";
    } else {
        echo "Not prime\n";
    }
}

Looks perfectly sensible. Through some mathematical regex trickery, we determine whether or not a number is prime. For reasons beyond the scope of this article, this regex fails under default PHP configurations beginning at the number 22201 because PHP’s regular expression backtracking limit is exceeded. While the documentation for preg_match() claims it will return boolean false if a PREG_BACKTRACK_LIMIT_ERROR occurs, the function actually returns integer 0. In the case of the above function, PHP will start calling everything above 22200 a prime number. Even if the documentation were correct we wouldn’t be much better off—every number would be classified as composite number.

How do we deal with this? You must check preg_last_error() every time a PCRE function is used. That warning is bold for a reason: the results of failing to check preg_last_error() can be even more destructive than improperly classifying integers. The function preg_replace() returns null when an error occurs, which PHP will happily coerce to 0 or the empty string depending on context. It is very easy to assume that your regular expression replacement went through successfully and keep trucking along, but your users will not be happy with that null value when it’s used in a string context.

The solution to these ails is the newly released gosRegex module of the Genius Open Source library. This new module provides simple wrappers for all of the PCRE functions in PHP, checking preg_last_error() for you and turning any errors into exception.

// Use the gosRegex functions exactly like their preg_* counterparts
gosRegex::match('/foo (bar)/', 'foo foo bar foo baz foo', $matches);
print_r($matches);
 
// If you do something that causes an error, the gosRegex functions let you know
try {
    // Example from http://us.php.net/preg_last_error
    gosRegex::match('/(?:\D+|<\d+>)*[!?]/', 'foobar foobar foobar');
} catch (gosException_RegularExpression $e) {
    print "Got a regex error: " . $e->getMessage() . "\n";
}

So grab the Genius Open Source library and start being safe with your regular expressions in PHP.