Team Lazer Beez Blog

Using dynamically generated configs with puppet

Jon Heise — Thu, 04 Nov 2010 18:41:44 +0000

After using Puppet with an external node classifier for a while one starts questioning what other information could be generated by this instead of just YAML to feed the puppetmaster. When supervisor was being rolled out there was a need to a large number of near identical config files to be generated, however any special information about the configs really had no place in Puppet. So the solution to this was to have the Django app generate the config files and then have puppet pull them down with a custom parser.

In /var/lib/puppet/lib/puppet/parser/functions lives the file webcontent.rb which has the following contents:

require 'open-uri'
module Puppet::Parser::Functions
    newfunction(:webcontent, :type =&gt; :rvalue) do |args|
        server = args[0]
        configpath = args[1]
        config = ""
        begin⋅
            open( "http://#{server}/#{configpath}/" ) do |f|
             f.each_line do |line|
                 config = "#{config}#{line}"
             end
            end⋅
        rescue OpenURI::HTTPError =&gt; e
            raise Puppet::ParseError, "404 for http://#{server}/#{configpath}/"
 
        rescue Exception =&gt; e
            raise Puppet::ParseError, "content string is http://#{server}/#{configpath}/ #{e}"
        end
        return config
    end
end

Using the Ruby module open-uri content is grabbed by the puppetmaster and placed into the catalog. Using the following Django model, view and template a config file is easily generated and passed along to Puppet

class SupervisorProgram(models.Model):
    name = models.CharField(max_length=128)
    command = models.CharField(max_length=512)
    autostart = models.BooleanField(default=True)
    autorestart = models.CharField(max_length=32,choices=(('false','false'),('true','true'),('unexpected','unexpected')))
    startsecs = models.IntegerField(default=10)
    startretries = models.IntegerField(default=3)
    exitcodes = models.CharField(max_length=64,default="0,2")
    stopsignal = models.CharField(max_length=5,choices=(('TERM','TERM'),('HUP','HUP'),('INT','INT'),('QUIT','QUIT'),('KILL','KILL'),('USR1','USR1'),('USR2','USR2')),default="TERM")
    stopwaitsecs = models.IntegerField(default=10)
    user = models.CharField(max_length=16 ,default="nagios")
    redirect_stderr = models.BooleanField(default=False)
    stdout_logfile = models.CharField(max_length=256,default="AUTO")
    stdout_logfile_maxbytes = models.CharField( max_length = 8,default="50MB")
    stdout_logfile_backups = models.IntegerField(default=10)
    stderr_logfile = models.CharField(max_length=256,default="AUTO")
    stderr_logfile_maxbytes = models.CharField( max_length = 8,default="50MB")
    stderr_logfile_backups = models.IntegerField(default=10)
    environment = models.CharField( max_length=512,blank=True,null=True)
    directory = models.CharField(max_length=128,default="/")
    umask = models.IntegerField(blank=True,null=True)
    priority = models.IntegerField(default=999)
 
    def __unicode__(self):
        return self.name
    class Meta:
        ordering = ('name',)
 
class SupervisorProgramAdmin(admin.ModelAdmin):
    list_display = ('name','command','autorestart','stopsignal','exitcodes','user','stdout_logfile','stderr_logfile')

The following is the view used:

def getSupervisorConfig(request,service):
    print "getSupervisorConfig has been called for %s" % service
    service = get_object_or_404(SupervisorProgram,name=service)
    directives = {}
    directives["command"] = str(service.command)
    directives["process_name"] = str(service.name)
    directives["priority"] = int(service.priority)
    directives["autostart" ] = service.autostart
    directives["autorestart"] = service.autorestart
    directives["startsecs"] = int(service.startsecs)
    directives["startretries"] = int(service.startretries)
    directives["exitcodes"] = str(service.exitcodes)
    directives["stopsignal"] = str(service.stopsignal)
    directives["stopwaitsecs"] = int(service.stopwaitsecs)
    directives["user"] = str(service.user)
    directives["redirect_stderr"] = service.redirect_stderr
    directives["stdout_logfile"] = str(service.stdout_logfile)
    directives["stdout_logfile_maxbytes"] = str(service.stdout_logfile_maxbytes)
    directives["stdout_logfile_backups"] = int(service.stdout_logfile_backups)
    directives["stderr_logfile"] = str(service.stderr_logfile)
    directives["stderr_logfile_maxbytes"] = str(service.stderr_logfile_maxbytes)
    directives["stderr_logfile_backups"] = int(service.stderr_logfile_backups)
    directives["directory"] = str(service.directory)
    if service.environment:
        directives["environment"] = str(service.environment)
 
    return render_to_response("sock/supervisor.conf",directives)

With the 20 configuration options per supervisord controlled process there are far too many options that should be sanely passed to puppetmaster from the external node classifier.

Here is the Django template:

#generated config
[program:{{ process_name }}]
command={{ command }}
process_name=%(program_name)s
priority={{ priority }}
autostart={{ autostart }}
autorestart={{ autorestart }}
startsecs={{ startsecs }}
startretries={{ startretries }}
exitcodes={{ exitcodes }}
stopsignal={{ stopsignal }}
stopwaitsecs={{ stopwaitsecs }}
user={{ user }}
redirect_stderr={{ redirect_stderr }}
stdout_logfile={{ stdout_logfile }}
stdout_logfile_maxbytes={{ stdout_logfile_maxbytes }}
stdout_logfile_backups={{ stdout_logfile_backups }}
stderr_logfile={{ stderr_logfile }}
stderr_logfile_maxbytes={{ stderr_logfile_maxbytes }}
stderr_logfile_backups={{ stderr_logfile_backups }}
{% if environment %}
environment={{ environment }}
{% endif %}

Finally all of this can be referenced with a custom define as follows:

define supervisorconfig(
        $program,
        $server = "${rserver}"
) {
    file {
        "${name}.conf":
            owner => root,
            group => root,
            mode => 0644,
            path => "/etc/supervisord.d/${name}.conf",
            content => webcontent( $server, "dpuppet/sock3/getsupervisorconfig/$program")
    }
}

Database Fixtures for Isolated Testing in PHP

Drew Stephens — Wed, 03 Nov 2010 18:31:40 +0000

Long ago, Ryan wrote the history of our fixture frameworks. Now, you too can have the awesome Team Lazer Beez database fixture for your own project. With the release of Team Lazer Beez Open Source (formerly Genius Open Source) version 1.2, our YAML-backed, easy-to-setup fixture framework has been integrated into the gosTest framework.

Here is a simple example from our Fixture package:


// Include the Genius config file
require_once dirname(dirname(__FILE__)) . '/Core/gosConfig.inc.php';
 
/**
 * A function to get single values from a database table
 */
function getThingFromDB($id) {
    $db = gosDB_Helper::getDBByName('main');
 
    return $db->getOne("SELECT s1 FROM fixture_test WHERE i1 = " . $id);
}

In a nearby test file:


// Include the Genius fixture configuration. This geneartes a database
// and applies the schema to it.  See fixtureTestConfig.inc.php
// for more details.
require_once(GOS_ROOT . 'Fixture/fixtureTestConfig.inc.php');
 
function testGetThingFromDB() {
    // Create a fixture
    $fixture = gosTest_Fixture_Controller::getByDBName('main');
 
    // Load the fixture into the database
    $fixture->parseFixtureFile(GOS_ROOT . 'Fixture/example_fixture.yaml');
 
    // Directly access the fixture, which is identical to what the database contains
    $idToGet = $fixture->get('fixtureName.fixture_test.i1');
 
    // Pull the value we want directly from the fixture
    $fixtureThing = $fixture->get('fixtureName.fixture_test.s1');
    // Pull the value we want from the DB via the function we're testing
    $thing = getThingFromDB($idToGet);
 
    echo "The fixture put $fixtureThing into the DB.\n";
    echo "Our function selected $thing from the DB.\n";
}
 
// Run the test
testGetThingFromDB();

The first magical line here is that require_once—fixtureTestConfig.inc.php uses the database connection information defined in Fixture/fixtureConfig.yaml to create a database is used for the duration of this PHP process only. Subsequent test runs (or invocations of this example script) will generate an entirely new database. See gosDB_AutoDBGenerator for details on these ephemeral databases. The takeaway is that you must have a user listed in that fixture config YAML file that can create databases.

In the test function itself, we create a fixture on the appropriate database, in this case our main database, and load the data from our fixture file, a simple YAML file defining what data we want in the database:

# The name of this fixture
fixtureName:
    # A table we will insert data into
    fixture_test:
        i1: 11
        s1: str1

At this point, we’re ready to start testing. First we get the ID that we need to pass to getThingFromDB, which we simply pull from the fixture. Next, we get the value that we are expecting from the same table in the fixture. Now that we have the ID we want to give to the function we’re testing, and the value we expect to get back, we can execute that function and compare the results. Go ahead and try this all for yourself; the above example works all on its own. The easiest way is to download the example.php and the necessary YAML file.

The gosFixture class automatically cleans up the database tables that it touched (see Core/lib/gosTest/Framework/TestCase.acls.php), so you need to re-apply the fixture for every test. The easiest way to do this is to put the fixture initialization in the setUpExtension() of a test class, giving you completely fresh fixture data for each and every test.

So go forth and test your code with better isolation and known clean data.

Puppet External Node Classifier

Jon Heise — Wed, 13 Oct 2010 18:06:59 +0000

In Puppet the initial method of holding information about your machines is through the site.pp config file, this rapidly becomes tiresome when you have more than 5 servers. This is where an external node classier comes in as a handy tool.

The first step in setting up an external node classifier is developing what will be generating the YAML for the puppetmaster to read. In this situation Django chosen as a web framework as the built in admin interface saved a bit of development time for management. Additional thought behind using an ORM framework was cutting down on development time on other projects that might need access to the truth database and could simply pull YAML from it. Within the Django application there are classes describing hosts, environments, server class, puppet classes, hard drive and filesystem layout. Here is the class describing a host:

class Host(models.Model):
    hostname = models.CharField(null=True,blank=False,max_length=128)
    basename = models.CharField(null=True,blank=True,max_length=128)
    maintenance = models.BooleanField(default=False)
    datacenter = models.CharField(max_length=5,choices=(('sjc','san jose'),
        ('smc','server room')),blank=True,null=True)
    environment = models.CharField(max_length=5,choices=(('app','production'),
        ('stg1','staging 1'),('stg2','staging 2'),('dev','development')),blank=True,null=True)
    env = models.ForeignKey(Environment)
    ostype = models.CharField(max_length=32,choices=(('linux','linux'),
        ('windows','windows'),('mac','mac')),blank=True,null=True)
    info = models.TextField(blank=True,null=True)
    bit = models.CharField(blank=True,null=True,max_length=5,choices=(('32','32 bit'),('64','64 bit')))
    puppetrun = models.IntegerField(default=1800)
    serverclass = models.ForeignKey(SvrClass)
 
    classes = models.ManyToManyField(PuppetClass,blank=True,null=True)
    drives = models.ManyToManyField(Drive,blank=True,null=True)
    interfaces = models.ManyToManyField(NetInterface,blank=True,null=True)
    script = models.ManyToManyField(Script,blank=True,null=True)
    osver = models.ManyToManyField(Osversion,blank=True,null=True)
    services = models.ManyToManyField(Service,blank=True,null=True)
    supervisors = models.ManyToManyField(SupervisorProgram,blank=True,null=True)
 
    def getBranch(self):
        return self.env.branch.name
 
    def getEnvironment(self):
        return self.env.name
 
    def __str__(self):
        return self.__repr__()
 
    def __repr__(self):
        return "hostname: %s environment: %s" % (self.hostname,self.env.name)
 
class HostAdmin(admin.ModelAdmin):
    list_display    =   ('hostname','environment','bit','ostype','datacenter','maintenance')
    list_filter     =   ('environment','datacenter')
    ordering        =   ('hostname','datacenter','environment')
    search_fields   =   ('hostname',)

and now the code for the Puppet class in Python, the Puppet class class:

class PuppetClass(models.Model):
    name = models.CharField(max_length=128)
    def __unicode__(self):
        return self.name
    class Meta:
        ordering = ('name',)
 
class PuppetClassAdmin(admin.ModelAdmin):
    list_display = ('name',)

With these two classes a host and what Puppet classes it has assigned to it can readily be described and the following code will output YAML describing the host as the puppetmaster understands:

import yaml
 
def hostinfo(request,hostname):
    host = get_object_or_404(Host,hostname=hostname)
    classlist = []
    [classlist.append(str(x.name)) for x in host.classes.all()]
    #if a host's classlist is empty at this point pull from the default list
    if not classlist:
        #default list is simply a list of puppet class objects
        defaultlist = DefaultList.objects.get(name="standard")
        [classlist.append(str(x.name)) for x in defaultlist.classes.all()]
    #add in some extra parameters
    params = {'datacenter':str(host.datacenter),'machine':str(host.basename),'serverclass':str(host.serverclass.name),'env':str(host.env.name),'branch':'%s' % str(host.env.branch.name),'envmaint':str(host.env.maintenance),'hostmaint':str(host.maintenance)}
    params['rserver'] = str(host.env.rserver)
    yamlsrc = yaml.dump({'classes':classlist,'parameters':params})
    return  render_to_response("sock/hostinfo.yaml",{'hostname':hostname,'yaml':yamlsrc})

One issue that was encountered with exporting data from django to yaml was the usage of utf for strings, hence the continual usage of str(). The result of the function is dumped out through the following incredibly complex template

---
{{ yaml }}

At the end of all of this we finally get data that comes out as something similar to the following:

---
classes: [bots, puppet-classes, puppet-master, repos, yamlsvn]
parameters: {branch: '33', datacenter: sjc, env: app, envmaint: '0', hostmaint: '0',
  machine: repo, rserver: sjc-repo.genops.net, serverclass: none}

Next we need a command that the puppetmaster can run to get the YAML, here is the current script:

#!/bin/sh
curl http://example.genius.com/dpuppet/sock3/hostinfo/$1/ 2>/dev/null | \
sed "s/'/'/g"
exit 0;

Finally the puppetmaster needs to be configured to pull from the external node classifier:

[main]
    # Where Puppet stores dynamic and growing data.
    # The default value is '/var/puppet'.
    vardir = /var/lib/puppet
 
    # The Puppet log directory.
    # The default value is '$vardir/log'.
    logdir = /var/log/puppet
 
    # Where Puppet PID files are kept.
    # The default value is '$vardir/run'.
    rundir = /var/run/puppet
 
    # Where SSL certificates are kept.
    # The default value is '$confdir/ssl'.
    ssldir = $vardir/ssl
    # use external nodes ftw
    external_nodes = /usr/local/bin/puppetinfo.sh
    node_terminus = exec
 
[puppetd]
    # The file in which puppetd stores a list of the classes
    # associated with the retrieved configuratiion.  Can be loaded in
    # the separate ``puppet`` executable using the ``--loadclasses``
    # option.
    # The default value is '$confdir/classes.txt'.
    classfile = $vardir/classes.txt
 
    # Where puppetd caches the local configuration.  An
    # extension indicating the cache format is added automatically.
    # The default value is '$confdir/localconfig'.
    localconfig = $vardir/localconfig
    server =  puppetmaster.genius.com
    runinterval = 300

Puppet at Genius

Jon Heise — Tue, 12 Oct 2010 18:00:18 +0000

Puppet is a configuration management system that was created with the goal of making more portions of system administration tasks reuseable. At Genius puppet is used in all of our environments: production, staging, and development. Within this setup each environment has its own puppetmaster, with each master pulling against its own version of the puppet classes. All of the masters in turn pull all of their information nodes from a central external node classifier.

The masters act as their own clients, syncing against themselves with a puppet class that controls checkouts of an svn repository holding all puppet classes. All of the servers have been configured with custom defines to handle subervision checkouts and configs for supervisor, additionally a custom parser has been written to allow pulling down webpages as part a content description. All servers run on an interval of 5 minutes with clients running on a 15 minute interval, with some special exceptions for dns servers. Additionally the web app that powers the external node classier also handles generating config files and maintaining a truth database for other tasks, like imaging.

This is the first in a series of post regarding our setup, following posts will cover our external node classifier, truth database, customer parser, custom defines.

Using HornetQ without a separate JNDI server

Marshall Pierce — Fri, 08 Oct 2010 22:10:19 +0000

HornetQ is JBoss’s latest messaging product. It provides a JMS implementation as well as its own alternate API. JMS is the Java Message Service: a set of APIs built around asynchronous messaging through topics (think bulletin boards) and queues (self-explanatory). For more, see the standard JMS tutorial.

This tutorial shows how to access HornetQ JMS resources via JNDI without needing a separate JNDI server. It also provides a test utility to run an embedded HornetQ server (perfect for unit tests that need a JMS broker running!).

Why JNDI?

Since JMS is part of Java EE, you’ll often see tutorials that have JMS resources (e.g. connection factories or queues) being provided to your code is via JNDI. This isn’t the only choice, though: you could instead manually instantiate your preferred provider’s implementations of the resources you need.

If you choose the latter approach, you might use the following to get a Queue if you were using ActiveMQ (another JMS implementation):

Queue queue = new ActiveMQQueue("queueName");

This works fine until you switch to another provider, at which point you need to change all your queues (and connection factories and topics and…) to look like this:

Queue queue = new HornetQQueue("queueName");

This is why it’s recommended to instead pull those objects out of JNDI. If you’re getting the queue out of JNDI with the following code, you wouldn’t need to change anything if you switch providers other than changing how the JNDI context gets populated.

Context context = new InitialContext();
Queue queue;
try {
    queue = (Queue) context.lookup("jndiNameForQueue");
} finally {
    // release your context's resources when you're done with it
    context.close();
}

This isn’t meant to be a JNDI tutorial, but it should be clear why getting JMS resources (as well as things like JDBC DataSource objects and other “managed” objects) out of JNDI is a good thing: it makes it easier to program to interfaces, not implementations, and maintain vendor neutrality.

HornetQ’s JNDI support

HornetQ comes with all you need to be able to connect to a JNDI server and pull JMS resources out of that server. You can set up the HornetQ server to serve JNDI in addition to JMS by enabling the appropriate bean in hornetq-beans.xml.

This approach is straightforward, but it becomes problematic if you want your HornetQ servers to be in a HA (high availability) pair. It’s hard enough to configure and test HA without also needing JNDI service to fail over as well. Though there is some documentation on how to set up HA JNDI, it would be simpler (and surely faster as well) if there was no need to talk to a server at all to get JMS resources. This approach makes sense if your deployment setup is simple enough that you know which queues and JMS servers you will be using and can put that information in a few config files and bundle it with your code during deployment. This is the case for many uses of JMS. If, on the other hand, you’re already tied to using a Java EE server that provides JNDI, then you might as well use that.

ActiveMQ provides a simple way to configure JMS resources in local JNDI, but HornetQ doesn’t come with anything similar. This tutorial provides a way to do local JNDI with HornetQ.

Local JNDI Configuration

The starting point to accessing JNDI objects is creating a new InitialContext. The specific way that the resulting Context gets populated with data is controlled by the implementation of InitialContextFactory that you’re using. You can change the implementation by specifying the java.naming.factory.initial property in jndi.properties to be the fully qualified class name of an impementation of InitialContextFactory.

We can use this mechanism to specify an implementation of InitialContextFactory that reads HornetQ’s config files. More sophisticated implementations might produce a Context that accesses data on some remote server, but our goal here is something that reads data out of config files and populates a memory-only Context. We’ll need to read the core HornetQ config file to get connection factory information and the JMS HornetQ config file to get the JNDI names to assign to queues, topics, etc. We’ll also need something to actually create a Context implementation since we don’t want to have to write that ourselves. Implementing Context requires a non-trivial amount of work to do well. simple-jndi provides a basic in-memory InitialContextFactory, so we’ll use that.

Your jndi.properties would look something like this:

java.naming.factory.initial = com.genius.hornetq.XmlHornetQInitialContextFactory
hornetq.jndi.wrapped.initialcontextfactory.impl = org.osjava.sj.memory.MemoryContextFactory
hornetq.xml.jms.path = /path-to-hornetq-jms.xml
hornetq.xml.config.path = /path-to-hornetq-config.xml

The java.naming.factory.initial tells InitialContext which class to instantiate to return the underlying Context (look at the source that comes with the JDK if you’re curious how). In this case, we want InitialContext to use our custom implementation that reads HornetQ XML files.

The other properties are only relevant to our custom InitialContextFactory. See the HornetQ documentation for more about what to put in the XML config files.
hornetq.jndi.wrapped.initialcontextfactory.impl defines what InitialContextFactory we’ll use to actually create a Context object. In this case, it’s the simple-jndi memory-only InitialContextFactory.
hornetq.xml.jms.path is the path in the classpath to the HornetQ JMS config file. This is typically “hornetq-jms.xml”.
hornetq.xml.config.path is the path in the classpath to the HornetQ core config file. This is typically “hornetq-configuration.xml”.

Now that we’ve got jndi.properties ready, I’ve added the custom InitialContextFactory source below. The only tricky part is handling HornetQ’s LogDelegateFactory selection. The LogDelegateFactory system lets you change the logging system used by HornetQ and is configurable via the core config file. (I’d link to the documentation, but this option is undocumented. See FileConfigurationParser#parseMainConfig() in the HornetQ source.) Unfortunately, a bug in the way the LogDelegateFactory instance is set causes some classes to not properly pick up your custom implementation, so if you wish to use a custom LogDelegateFactory, note the commented out call to Logger.setDelegateFactory. Other than that, it’s pretty simple. We use HornetQ’s configuration parsing code to get the information we need out of the config files, then populate the Context we get from simple-jndi with the resulting JMS objects.

The only dependency outside of HornetQ is SLF4J for logging.

package com.genius.hornetq;
 
import org.hornetq.api.core.Pair;
import org.hornetq.api.core.TransportConfiguration;
import org.hornetq.api.jms.HornetQJMSClient;
import org.hornetq.core.config.Configuration;
import org.hornetq.core.deployers.impl.FileConfigurationParser;
import org.hornetq.jms.client.HornetQConnectionFactory;
import org.hornetq.jms.client.HornetQDestination;
import org.hornetq.jms.server.JMSServerConfigParser;
import org.hornetq.jms.server.config.ConnectionFactoryConfiguration;
import org.hornetq.jms.server.config.JMSConfiguration;
import org.hornetq.jms.server.config.JMSQueueConfiguration;
import org.hornetq.jms.server.config.TopicConfiguration;
import org.hornetq.jms.server.impl.JMSServerConfigParserImpl;
import org.hornetq.spi.core.logging.LogDelegateFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import javax.jms.Queue;
import javax.jms.Topic;
import javax.naming.Context;
import javax.naming.NamingException;
import javax.naming.spi.InitialContextFactory;
import java.io.InputStream;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Hashtable;
import java.util.List;
import java.util.Map;
 
/**
 * Parses hornetq jms and core configuration files and populates the resulting Context appropriately.
 *
 * The following properties are needed:
 *
 * hornetq.xml.jms.path = path in the classpath to hornetq jms xml file
 *
 * hornetq.xml.config.path = path in classpath to hornetq core config file
 *
 * hornetq.jndi.wrapped.initialcontextfactory.impl = fully qualified classname of InitialContextFactory to use to
 * provide the Context instance that will be populated with configured JMS objects. An instance of this class will be
 * created and passed an empty environment.
 */
public class XmlHornetQInitialContextFactory implements InitialContextFactory {
 
    private static final String JMS_XML_PATH_KEY = "hornetq.xml.jms.path";
    private static final String CONFIG_XML_PATH_KEY = "hornetq.xml.config.path";
    private static final String INTERNAL_ICF_IMPL_KEY = "hornetq.jndi.wrapped.initialcontextfactory.impl";
    private static final Logger logger = LoggerFactory.getLogger(XmlHornetQInitialContextFactory.class);
 
    public XmlHornetQInitialContextFactory() {
        logger.trace("Instantiated");
    }
 
    @Override
    public Context getInitialContext(Hashtable, ?> environmentHashtable) throws NamingException {
        logger.trace("Creating InitialContext");
        Map<string, String> env = new HashMap<string, String>();
        for (Map.Entry, ?> entry : environmentHashtable.entrySet()) {
            if (entry.getKey() instanceof String && entry.getValue() instanceof String) {
                env.put((String) entry.getKey(), (String) entry.getValue());
            }
        }
 
        this.checkKeys(env, JMS_XML_PATH_KEY, INTERNAL_ICF_IMPL_KEY, CONFIG_XML_PATH_KEY);
 
        Context ctx = this.getContext(env);
 
        populateContext(env, ctx);
 
        return ctx;
    }
 
    private void populateContext(Map<string, String> env, Context ctx) throws NamingException {
        final InputStream configXmlStream = getStream(env.get(CONFIG_XML_PATH_KEY));
 
        /*
        * A bug in HQ 2.1.2 means that the log delegate factory will not be applied to any classes that use static
        * loggers and were loaded before the config-specified delegate factory is applied. A fix is scheduled for 2.2.0.
         * Until then, we'll hardcode the delegate factory to slf4j for the first few loggers.
        */
        //org.hornetq.core.logging.Logger.setDelegateFactory(new YourCustomLogDelegateFactory());
 
        final Configuration config;
        try {
            config = new FileConfigurationParser().parseMainConfig(configXmlStream);
        } catch (Exception e) {
            throw getNamingException("Couldn't get configuration from xml", e);
        }
 
        final LogDelegateFactory logDelegateFactory =
                (LogDelegateFactory) instantiate(config.getLogDelegateFactoryClassName());
        org.hornetq.core.logging.Logger.setDelegateFactory(logDelegateFactory);
 
 
        final InputStream jmsXmlStream = getStream(env.get(JMS_XML_PATH_KEY));
        final JMSServerConfigParser jmsServerConfigParser = new JMSServerConfigParserImpl();
        JMSConfiguration jmsConfig;
        try {
            jmsConfig = jmsServerConfigParser.parseConfiguration(jmsXmlStream);
        } catch (Exception e) {
            throw getNamingException("Couldn't get jms info from xml", e);
        }
 
        for (ConnectionFactoryConfiguration cfConfig : jmsConfig.getConnectionFactoryConfigurations()) {
            final HornetQConnectionFactory hqcf =
                    getHornetQConnectionFactory(cfConfig, config.getConnectorConfigurations());
            for (String jndiName : cfConfig.getBindings()) {
                ctx.bind(jndiName, hqcf);
            }
        }
 
        for (JMSQueueConfiguration queueConfiguration : jmsConfig.getQueueConfigurations()) {
            Queue queue = HornetQDestination.createQueue(queueConfiguration.getName());
            for (String jndiName : queueConfiguration.getBindings()) {
                ctx.bind(jndiName, queue);
            }
        }
 
        for (TopicConfiguration topicConfiguration : jmsConfig.getTopicConfigurations()) {
            Topic topic = HornetQDestination.createTopic(topicConfiguration.getName());
            for (String jndiName : topicConfiguration.getBindings()) {
                ctx.bind(jndiName, topic);
            }
        }
    }
 
    private InputStream getStream(String xmlPath) throws NamingException {
        final InputStream xmlStream = this.getClass().getResourceAsStream(xmlPath);
        if (xmlStream == null) {
            throw new NamingException("Cannot find resource at <" + xmlPath + ">");
        }
        return xmlStream;
    }
 
    private Context getContext(Map<string, String> env) throws NamingException {
        final String icfClassName = env.get(INTERNAL_ICF_IMPL_KEY);
 
        InitialContextFactory icf = (InitialContextFactory) instantiate(icfClassName);
 
        //noinspection UseOfObsoleteCollectionType
        return icf.getInitialContext(new Hashtable<object, Object>());
    }
 
    /**
     * Instantiate the supplied class name via its 0-arg ctor.
     *
     * @param className class name to instantiate
     *
     * @return instance of className
     *
     * @throws NamingException if the class cannot be instantiated
     */
    private Object instantiate(String className) throws NamingException {
        Object obj;
        try {
            final Class klass = Class.forName(className);
            final Constructor ctor = klass.getConstructor();
            obj = ctor.newInstance();
        } catch (ClassNotFoundException e) {
            throw getNamingException("Couldn't find class " + className, e);
        } catch (NoSuchMethodException e) {
            throw getNamingException("Couldn't find 0-arg constructor for " + className, e);
        } catch (InvocationTargetException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        } catch (InstantiationException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        } catch (IllegalAccessException e) {
            throw getNamingException("Couldn't instantiate " + className, e);
        }
        return obj;
    }
 
    private static NamingException getNamingException(String message, Exception e) {
        final NamingException ne = new NamingException(message);
        ne.initCause(e);
        return ne;
    }
 
    /**
     * @param env  the env to look in
     * @param keys the keys to check for
     *
     * @throws NamingException if any key is not found in the env or if the value is null
     */
    private void checkKeys(Map<string, String> env, String... keys) throws NamingException {
        for (String key : keys) {
            if (env.get(key) == null) {
                throw new NamingException("Couldn't find " + key + " property in " + env);
            }
        }
    }
 
    static HornetQConnectionFactory getHornetQConnectionFactory(ConnectionFactoryConfiguration cfConfig,
            Map<string, TransportConfiguration> connectorConfigurations) throws NamingException {
 
        /*
         * Implementation largely lifted from JMSServerManagerImpl
         */
 
        List<pair<transportConfiguration, TransportConfiguration>> connectorConfigs =
                new ArrayList<pair<transportConfiguration, TransportConfiguration>>();
 
        for (Pair<string, String> configConnector : cfConfig.getConnectorNames()) {
            String connectorName = configConnector.a;
            String backupConnectorName = configConnector.b;
 
            TransportConfiguration connector = connectorConfigurations.get(connectorName);
 
            if (connector == null) {
                throw new NamingException("No configured connector with name <" + connectorName + ">");
            }
 
            logger.debug("Found connector config for <" + connectorName + ">");
            TransportConfiguration backupConnector = null;
 
            if (backupConnectorName != null) {
                backupConnector = connectorConfigurations.get(backupConnectorName);
 
                if (backupConnector == null) {
                    throw new NamingException("No configured backup connector with name <" + connectorName + ">");
                }
 
                logger.debug("Found backup connector config for <" + backupConnectorName + ">");
            } else {
                logger.debug("Not using a backup connector for main connector <" + connectorName + ">");
            }
 
            connectorConfigs.add(new Pair<transportConfiguration, TransportConfiguration>(connector, backupConnector));
        }
 
 
        HornetQConnectionFactory cf = HornetQJMSClient.createConnectionFactory(connectorConfigs);
        cf.setClientID(cfConfig.getClientID());
        cf.setClientFailureCheckPeriod(cfConfig.getClientFailureCheckPeriod());
        cf.setConnectionTTL(cfConfig.getConnectionTTL());
        cf.setCallTimeout(cfConfig.getCallTimeout());
        cf.setCacheLargeMessagesClient(cfConfig.isCacheLargeMessagesClient());
        cf.setMinLargeMessageSize(cfConfig.getMinLargeMessageSize());
        cf.setConsumerWindowSize(cfConfig.getConsumerWindowSize());
        cf.setConsumerMaxRate(cfConfig.getConsumerMaxRate());
        cf.setConfirmationWindowSize(cfConfig.getConfirmationWindowSize());
        cf.setProducerWindowSize(cfConfig.getProducerWindowSize());
        cf.setProducerMaxRate(cfConfig.getProducerMaxRate());
        cf.setBlockOnAcknowledge(cfConfig.isBlockOnAcknowledge());
        cf.setBlockOnDurableSend(cfConfig.isBlockOnDurableSend());
        cf.setBlockOnNonDurableSend(cfConfig.isBlockOnNonDurableSend());
        cf.setAutoGroup(cfConfig.isAutoGroup());
        cf.setPreAcknowledge(cfConfig.isPreAcknowledge());
        cf.setConnectionLoadBalancingPolicyClassName(cfConfig.getLoadBalancingPolicyClassName());
        cf.setTransactionBatchSize(cfConfig.getTransactionBatchSize());
        cf.setDupsOKBatchSize(cfConfig.getDupsOKBatchSize());
        cf.setUseGlobalPools(cfConfig.isUseGlobalPools());
        cf.setScheduledThreadPoolMaxSize(cfConfig.getScheduledThreadPoolMaxSize());
        cf.setThreadPoolMaxSize(cfConfig.getThreadPoolMaxSize());
        cf.setRetryInterval(cfConfig.getRetryInterval());
        cf.setRetryIntervalMultiplier(cfConfig.getRetryIntervalMultiplier());
        cf.setMaxRetryInterval(cfConfig.getMaxRetryInterval());
        cf.setReconnectAttempts(cfConfig.getReconnectAttempts());
        cf.setFailoverOnInitialConnection(cfConfig.isFailoverOnInitialConnection());
        cf.setFailoverOnServerShutdown(cfConfig.isFailoverOnServerShutdown());
        cf.setGroupID(cfConfig.getGroupID());
 
        return cf;
    }
}

Testing JMS code

As a bonus for reading this far, here’s JmsTestServerManager. This class makes it easy to start and stop an embedded HornetQ server. This is ideal for testing code that reads from or writes to JMS queues or topics, for instance. You should create one instance per test class (via @BeforeClass if you’re using JUnit 4) and call start() in setup (@Before) and stop() in teardown (@After). This means you’ll have a fresh server instance for each test. No more needing to clean up leftover messages in queues between each test! Fortunately, HornetQ stops and starts quite quickly so this is unlikely to be a performance problem for your tests.

 
package com.genius.testutil.jms;
 
import net.jcip.annotations.NotThreadSafe;
import org.hornetq.api.core.TransportConfiguration;
import org.hornetq.core.config.Configuration;
import org.hornetq.core.config.impl.ConfigurationImpl;
import org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory;
import org.hornetq.core.server.HornetQServer;
import org.hornetq.core.server.HornetQServers;
import org.hornetq.jms.server.JMSServerConfigParser;
import org.hornetq.jms.server.JMSServerManager;
import org.hornetq.jms.server.config.JMSConfiguration;
import org.hornetq.jms.server.config.JMSQueueConfiguration;
import org.hornetq.jms.server.config.TopicConfiguration;
import org.hornetq.jms.server.impl.JMSServerConfigParserImpl;
import org.hornetq.jms.server.impl.JMSServerManagerImpl;
import org.jetbrains.annotations.NotNull;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
 
import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import java.io.InputStream;
import java.util.HashSet;
import java.util.Set;
 
/*
 * See org.hornetq.tests.util.JMSTestBase
 */
 
/**
 * Runs an embedded HornetQ server suitable for use in unit tests that need JMS. It should be re-used across test
 * methods; just call start() in @Before and stop() in @After.
 */
@NotThreadSafe
public class JmsTestServerManager {
    private static final Logger logger = LoggerFactory.getLogger(JmsTestServerManager.class);
 
    private static final String INVM_ACCEPTOR_FACTORY = InVMAcceptorFactory.class.getCanonicalName();
 
    private static final String DEFAULT_HORNETQ_TEST_JMS_XML = "/hornetq-test-jms.xml";
 
    /**
     * Queues to add when the server is started
     */
    @NotNull
    private final Set<string> queueNames;
 
    /**
     * Topics to add when the server is started
     */
    @NotNull
    private final Set<string> topicNames;
 
    /**
     * The context passed to the jms server (proxied to prevent close()). No objects are registered in it; it is simply
     * provided to the server so that the server doesn't try to create its own.
     */
    @NotNull
    private final Context context;
 
    /**
     * null if the server is stopped
     */
    @Nullable
    private JMSServerManager jmsServer;
 
    /**
     * null if the server is stopped
     */
    @Nullable
    private HornetQServer server;
 
    /**
     * @param context    the context to give to the server.
     * @param queueNames set of queue names to create in the server
     * @param topicNames set of topic names to create in the server
     */
    private JmsTestServerManager(@NotNull Context context, @NotNull Set<string> queueNames,
            @NotNull Set<string> topicNames) {
        this.queueNames = queueNames;
 
        // avoid having the server initialize its own InitialContext, and don't let it close the InitialContext
        // since we want to keep it open across invocations
 
        this.context = new ReadOnlyContextProxy(context);
        this.topicNames = topicNames;
    }
 
    /**
     * @param context      The JNDI context to pass to the server
     * @param pathToJmsXml The classpath path to the hornetq jms file to read topics and queues from
     *
     * @return a configured JmsTestServerManager in the stopped state
     */
    static JmsTestServerManager getNew(@NotNull Context context, @NotNull String pathToJmsXml) {
        final JMSConfiguration configuration = getJmsConfiguration(pathToJmsXml);
        return new JmsTestServerManager(context, getQueueNames(configuration), getTopicNames(configuration));
    }
 
    /**
     * Create a new InitialContext and read queues & topics from the default xml path.
     *
     * @return a configured JmsTestServerManager in the stopped state
     *
     * @throws NamingException if an InitialContext can't be created
     * @see JmsTestServerManager#getNew(Context, String)
     */
    static JmsTestServerManager getNew() throws NamingException {
        // It's ok to not close this context since we want it to be left open
        //noinspection JNDIResourceOpenedButNotSafelyClosed
        return getNew(new InitialContext(), DEFAULT_HORNETQ_TEST_JMS_XML);
    }
 
    /**
     * Provides access to the context used by this object to avoid needing to re-create it.
     *
     * @return the context used by this reader
     */
    @NotNull
    public Context getContext() {
        return this.context;
    }
 
    /**
     * Start the server and add all queues to it.
     */
    public void start() {
        if (this.server != null) {
            throw new IllegalStateException("Server already started");
        }
 
        this.server = HornetQServers.newHornetQServer(createDefaultConfig());
        try {
            this.jmsServer = new JMSServerManagerImpl(this.server);
            this.jmsServer.setContext(this.context);
            this.jmsServer.start();
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't start JMS server", e);
        }
 
        try {
            for (String queue : this.queueNames) {
                logger.debug("Adding queue " + queue);
                this.jmsServer.createQueue(false, queue, null, false);
            }
            for (String topicName : this.topicNames) {
                logger.debug("Adding topic " + topicName);
                this.jmsServer.createTopic(false, topicName);
            }
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't create destination", e);
        }
    }
 
    /**
     * Shut down the server.
     */
    public void stop() {
        if (this.server == null) {
            throw new IllegalStateException("Server not started");
        }
 
        try {
            this.jmsServer.stop();
            this.server.stop();
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't stop JMS server", e);
        }
 
        this.jmsServer = null;
        this.server = null;
    }
 
    private static JMSConfiguration getJmsConfiguration(String pathToJmsXml) {
        final InputStream xmlStream = JmsTestServerManager.class.getResourceAsStream(pathToJmsXml);
        if (xmlStream == null) {
            throw new YourCustomTestException("Couldn't find hornetq jms xml");
        }
 
        final JMSServerConfigParser jmsServerConfigParser = new JMSServerConfigParserImpl();
        JMSConfiguration jmsConfig;
        try {
            jmsConfig = jmsServerConfigParser.parseConfiguration(xmlStream);
        } catch (Exception e) {
            throw new YourCustomTestException("Couldn't get jms info from xml", e);
        }
 
        return jmsConfig;
    }
 
    private static Set<string> getTopicNames(JMSConfiguration jmsConfig) {
        Set<string> topicNames = new HashSet<string>();
        for (TopicConfiguration topicConfiguration : jmsConfig.getTopicConfigurations()) {
            topicNames.add(topicConfiguration.getName());
        }
 
        return topicNames;
    }
 
    private static Set<string> getQueueNames(JMSConfiguration jmsConfig) {
        Set<string> queueNames = new HashSet<string>();
        for (JMSQueueConfiguration jmsQueueConfiguration : jmsConfig.getQueueConfigurations()) {
            queueNames.add(jmsQueueConfiguration.getName());
        }
 
        return queueNames;
    }
 
    private Configuration createDefaultConfig() {
        Configuration configuration = new ConfigurationImpl();
        configuration.setSecurityEnabled(false);
        configuration.setJMXManagementEnabled(true);
        configuration.setPersistenceEnabled(false);
 
        configuration.setFileDeploymentEnabled(false);
 
        configuration.getAcceptorConfigurations().add(new TransportConfiguration(INVM_ACCEPTOR_FACTORY));
 
        // avoid the log about using the default cluster password
        configuration.setClusterPassword("asdf");
        configuration.setLogDelegateFactoryClassName(SLF4JLogDelegateFactory.class.getCanonicalName());
 
        return configuration;
    }
}

You may have noticed ReadOnlyContextProxy being used in JmsTestServerManager. That’s a rather boring Context implementation that proxies all read-only method calls to an internal Context instance and has a no-op close() implementation. This is to prevent HornetQ from closing the Context since we want to keep the Context open through all test method invocations to prevent pointlessly re-creating an InitialContext every time.

The rather uninteresting source of ReadOnlyContextProxy follows to save you from writing your own.

package com.genius.testutil.jms;
 
import net.jcip.annotations.NotThreadSafe;
 
import javax.naming.Binding;
import javax.naming.Context;
import javax.naming.Name;
import javax.naming.NameClassPair;
import javax.naming.NameParser;
import javax.naming.NamingEnumeration;
import javax.naming.NamingException;
import java.util.Hashtable;
 
/**
 * Proxies Context methods to let us control what happens to a Context that we pass in to the embedded JMS server.
 */
@NotThreadSafe
class ReadOnlyContextProxy implements Context {
 
    private final Context context;
 
    ReadOnlyContextProxy(Context context) {
        this.context = context;
    }
 
    @Override
    public void close() throws NamingException {
        // no op
    }
 
    /*
     * State-change methods not allowed. Could also no-op them if the exception becomes problematic, but
     * org.hornetq.tests.unit.util.InVMContext UOEs on many operations.
     */
 
    @Override
    public void bind(Name name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void bind(String name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rebind(Name name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rebind(String name, Object obj) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void unbind(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void unbind(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rename(Name oldName, Name newName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void rename(String oldName, String newName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void destroySubcontext(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public void destroySubcontext(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Object addToEnvironment(String propName, Object propVal) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Object removeFromEnvironment(String propName) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Context createSubcontext(Name name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    @Override
    public Context createSubcontext(String name) throws NamingException {
        throw new UnsupportedOperationException();
    }
 
    /*
     * Read-only methods left alone.
     */
 
    @Override
    public NamingEnumeration<nameClassPair> list(Name name) throws NamingException {
        return context.list(name);
    }
 
    @Override
    public NamingEnumeration<nameClassPair> list(String name) throws NamingException {
        return context.list(name);
    }
 
    @Override
    public NamingEnumeration<binding> listBindings(Name name) throws NamingException {
        return context.listBindings(name);
    }
 
    @Override
    public NamingEnumeration<binding> listBindings(String name) throws NamingException {
        return context.listBindings(name);
    }
 
    @Override
    public Object lookupLink(Name name) throws NamingException {
        return context.lookupLink(name);
    }
 
    @Override
    public Object lookupLink(String name) throws NamingException {
        return context.lookupLink(name);
    }
 
    @Override
    public NameParser getNameParser(Name name) throws NamingException {
        return context.getNameParser(name);
    }
 
    @Override
    public NameParser getNameParser(String name) throws NamingException {
        return context.getNameParser(name);
    }
 
    @Override
    public Name composeName(Name name, Name prefix) throws NamingException {
        return context.composeName(name, prefix);
    }
 
    @Override
    public String composeName(String name, String prefix) throws NamingException {
        return context.composeName(name, prefix);
    }
 
    @Override
    public Hashtable, ?> getEnvironment() throws NamingException {
        return context.getEnvironment();
    }
 
    @Override
    public Object lookup(Name name) throws NamingException {
        return context.lookup(name);
    }
 
    @Override
    public Object lookup(String name) throws NamingException {
        return context.lookup(name);
    }
 
    @Override
    public String getNameInNamespace() throws NamingException {
        return context.getNameInNamespace();
    }
}

With these classes, you’ve got all you need to get started writing JMS code (and testing it, too).

Making PHP Regex Errors Real

Drew Stephens — Mon, 30 Aug 2010 18:16:24 +0000

PHP employs Perl Compatible Regular Expressions (PCRE) in the built-in collection of preg_* functions, such as preg_match(). While PCRE is certainly the preferred regular expression library, PHP’s implementation allows the functions to fail without any explicit warning—the user must check preg_last_error() to know that an error occurred. Often, the return of a regular expression match is checked, and different operations are performed if the regex matched or not.

/**
 * Find primes with a regex.
 * http://montreal.pm.org/tech/neil_kandalgaonkar.shtml
 */
function isPrime($num) {
    $num = str_repeat('1', $num);
    $ret = preg_match('/^1?$|^(11+?)\1+$/', $num);
 
    echo 'Return value is ';
    var_dump($ret);
 
    if ($ret === 0) {
        echo "Prime\n";
    } else {
        echo "Not prime\n";
    }
}

Looks perfectly sensible. Through some mathematical regex trickery, we determine whether or not a number is prime. For reasons beyond the scope of this article, this regex fails under default PHP configurations beginning at the number 22201 because PHP’s regular expression backtracking limit is exceeded. While the documentation for preg_match() claims it will return boolean false if a PREG_BACKTRACK_LIMIT_ERROR occurs, the function actually returns integer 0. In the case of the above function, PHP will start calling everything above 22200 a prime number. Even if the documentation were correct we wouldn’t be much better off—every number would be classified as composite number.

How do we deal with this? You must check preg_last_error() every time a PCRE function is used. That warning is bold for a reason: the results of failing to check preg_last_error() can be even more destructive than improperly classifying integers. The function preg_replace() returns null when an error occurs, which PHP will happily coerce to 0 or the empty string depending on context. It is very easy to assume that your regular expression replacement went through successfully and keep trucking along, but your users will not be happy with that null value when it’s used in a string context.

The solution to these ails is the newly released gosRegex module of the Genius Open Source library. This new module provides simple wrappers for all of the PCRE functions in PHP, checking preg_last_error() for you and turning any errors into exception.

// Use the gosRegex functions exactly like their preg_* counterparts
gosRegex::match('/foo (bar)/', 'foo foo bar foo baz foo', $matches);
print_r($matches);
 
// If you do something that causes an error, the gosRegex functions let you know
try {
    // Example from http://us.php.net/preg_last_error
    gosRegex::match('/(?:\D+|<\d+>)*[!?]/', 'foobar foobar foobar');
} catch (gosException_RegularExpression $e) {
    print "Got a regex error: " . $e->getMessage() . "\n";
}

So grab the Genius Open Source library and start being safe with your regular expressions in PHP.

Building High Performance Teams

Shirley Foster — Tue, 03 Aug 2010 19:44:52 +0000

As an Engineering and Operations leader in a high-growth privately held company, it’s my responsibility to deliver the most features in the shortest amount of time with the highest quality at the lowest risk. In my experience, this is not achieved by browbeating teams into working longer hours, or writing scads of detailed requirements, but to establish a process that is iterative and transparent, and a culture founded on collaboration and communication.

High performance teams are typically flat (not hierarchical), self-organizing, and cross-functional. Adopting the following guidelines will help you build a team that maximizes their combined strengths to outperform expectations while having a great time!

Keys to building high performance teams

Alignment of Tactical Deliverables with Strategic Objectives
It’s important that your team understands how their day-to-day contributions ‘fit’ the strategic goals of the company. This is especially true in a high-growth start-up since the team are also (typically) shareholders and have a vested interest in ensuring the work they do is positively contributing to the company’s success. Share the strategic goals and how specific features and release deliverables will help the company achieve them. Also, highlight sales and customer successes where particular features or capabilities played a key role.

Team-based Goals
Measuring the team on their collective accomplishments increases teamwork by creating a sense of joint ownership and accountability. By focusing on team goals, rather than individual goals, team members will raise issues earlier (so as not to impede the team’s progress), and seek ways to help one another to complete tasks. Mentoring and knowledge transfer will also happen more naturally as teams seek ways to become more productive.

Transparency
Adopting agile practices will help increase transparency to your team’s progress, provide earlier visibility to issues and potential delays and establish a regular time for the team to reflect on what they’ve accomplished. The best way to increase transparency is to adopt Agile software development practices. At Genius, we follow the tenets of both Scrum and Lean software development. Our daily stand-up meetings keep everyone in the team on the same page and a product backlog that is visible to the whole company ensures that there is clarity to what’s on tap for future releases.

Self-organizing Teams
While on the surface this may seem counter-intuitive to those who are more familiar with hierarchical organizations; self-organizing teams are highly effective. Providing teams with clearly understood expectations and allowing them to self-organize to meet those goals – in combination with a transparent process – results in highly motivated and effective teams. Technical teams understand how each of their skills can be best leveraged to achieve the desired result and take greater ownership in the outcome rather than simply following management direction.

Communication and Collaboration
The most effective teams communicate and collaborate regularly. Creating a culture where open communication is valued and encouraged will ensure that your teams seek clarity of understanding, raise issues or concerns openly, that they are willing to challenge the status quo and that they validate ideas before implementing them. At Genius, everyone (whether junior or senior) is capable of having the best idea in the room and is encouraged to share their thinking. This results in a better product and creates an environment where everyone on the team is contributing fully.

Communication must happen at every level – from technical discussions to updates on the business. As a leader it’s important to set the example by being a great communicator and by demonstrating that you value everyone’s contributions and that there are no negative implications to speaking up.

Establish Accountability
Establishing ownership for deliverables, in addition to setting clear goals and measuring the results, is key to improving the performance of your team. It’s best to establish a relatively short timeframe to meet a set of goals. The list of goals should be determined with the team so that they are clear on the expectations and their ability to meet them. At Genius, we have adopted 2 weeks sprints and release new features to production every 2 weeks.

While it’s desirable for the team to deliver on all their commitments, absolute perfection in estimation is impossible, It is important to be flexible to some degree – when issues that may affect the team’s ability to deliver on their commitments come up early they can be dealt with effectively. This is especially important at the outset as your team is transitioning to a results-based model. Remember, the goal is to build a framework for success so that the team will be motivated to do more.

Measure, Review and Adapt
Performance improvement can only be made if you are willing to openly discuss things that have not gone well and to identify improvements that will increase the team’s rate of success. Examples can range from how the team is organizing itself to how well they (really) understood the requirements. It’s also important to reflect on what went well and to discuss how the team can adopt this into their process. Keep a record of what you’ve discussed so you can periodically look back with the team and see how far you’ve come.

Prioritize Career Development
High performance teams need (and expect) to be recognized for their achievements. Not only on an ongoing basis (related to deliverables), but also in career advancement. Provide clear definitions of what’s required to achieve greater responsibility and establish a regular dialog where you articulate areas of strength and opportunities to improve. This will inspire high performers to step up to meet the next level.

Implement Source Control and Build Management
One of the keys to success in an agile environment is continuous integration (check-ins to a single codebase) and continuous builds. Having effective and automatic builds ensures that the team is alerted at the earliest time about build problems. This is best accomplished with a Build Engineer whose responsibility it is to ensure that code is branched (and merged) appropriately, that builds are correctly executed and that problems are investigated.

Celebrate
Having led several Engineering and Operations teams over the past 20 years, I have seen the transformation and the results of adopting these practices multiple times. As you begin to adopt these with your team, don’t forget the most important thing of all…CELEBRATE! At Genius, we celebrate (as a company) the teams’ accomplishments every two weeks at the Sprint Review. We also celebrate our most recent release at the weekly Engineering and TechOps meeting and recognize what’s been completed daily at the stand-up meetings.

Not only does this give the team the recognition they deserve, but it also provides regular feedback for how they are doing. At Genius, the team has had a lot to celebrate with over 70 consecutive successful sprints and 15 on-time releases since adopting agile; providing new capabilities and features that meet our customer’s ongoing business goals.

Good luck! I look forward to hearing your thoughts and experiences.

Genius Open Source Libraries

Drew Stephens — Thu, 29 Jul 2010 19:12:48 +0000

Some time ago, Genius Engineering decided to unify the manner in which we encode values that contain user input. We previously depended upon the PHP built-in htmlentities() and some simple wrappers around it for our encoding needs, but this function alone can’t safely sanitize tainted data in all contexts. Furthermore, we didn’t have a unified vision of whether encoding should happen immediately upon receipt of data from the user or when we display that data to the user. The ambiguity of our security arrangement, and the lack of encoding functions appropriate for all contexts led the engineering team to look for better options in PHP security for the prevention of cross-site scripting (XSS) and SQL injection vulnerabilities. While there is plenty of information about these issues and what must be done to fix them, there is a distinct dearth of libraries in PHP to properly encode strings for all of the situations.

When the right tool for the job doesn’t exist, you build it. We came up with a set of functions to sanitize tainted data in any of the places that it is output to the user. The functions are very straightforward: give them a string and you get back one that is fully escaped. Output from the gosSanitizer functions can be safely used as a double-quoted string in an HTML attribute or JavaScript context, or as a single-quoted string in an SQL context.

// Output an unsafe string, presumably user input
$xss = '';
echo 'If your entered your name as ' . $xss . ', we\'d be in trouble.
' . "\n";
 
// Sanitize that string, and output it safely
$htmlContentContext = gosSanitizer::sanitizeForHTMLContent($xss);
echo "But if we sanitize your name, " . $htmlContentContext . ", then all is well.
\n";
 
echo 'HTML Attribute
';
// We can also safely sanitize it for an HTML attribute context
$htmlAttributeContext = gosSanitizer::sanitizeForHTMLAttribute($xss);
echo 'Tainted strings can also be used in an
    . $htmlAttributeContext . '">HTML attribute
    context.
' . "\n";
 
echo 'JavaScript string
';
// And we can even make strings used in JavaScript safe
$jsString = '\';alert(1);var b =\'';
echo '';

We have created a project on Launchpad to host the Genius text sanitizing libraries. The project consists of three modules: Core and Utility which provide general purpose support functions, and Sanitizer, which holds the functions used above. In the case of Sanitizer, all of the functions are static, and can be accessed through the gosSanitizer class. To use the Genius Sanitizer, you’ll need all three modules: Core, Utility, and Sanitizer itself. All of the Genius modules are loaded using the autoloader defined in Core/gosConfig.inc.php, so including this file is all that is needed to use any of the Genius Open Source libraries.

// Include the Genius config file
require_once 'Core/gosConfig.inc.php';
// Use gos* classess & functions here

We plan to continue adding modules to the Genius Open Source libraries collection in the future. Keep an eye on this blog for announcements!

Edited 2010-08-30 to reflect prefix change from “sg” to “gos”

Releasing Every Fortnight

Shirley Foster — Mon, 26 Jul 2010 21:08:21 +0000

Genius.com’s successful adoption of agile practices has been covered at some length in earlier postings, including Presenting on Going Agile with Scrum and An Agile Fortnight. Building on this success, we have most recently reached the point where the completed user stories for any given sprint at not only ‘potentially shippable’ but are actually deployed to production. So, how did we get here and how long did it take?

Testing as the foundation

One of the key elements of our success in bi-weekly product releases is the commitment to increasing automated test coverage – both unit tests and functional automation tests.

With rapid rate of change – and new features in every release – it is imperative that developers know immediately if their check-ins have caused a build to break. This is only possible with a concerted investment in unit testing and QA automation. In our cases, we proceeded in phases, each taking approximately 4 months to implement:

All check-ins must have associated unit tests. While we did not take the time to retrofit existing code, all new or modified code was required to have associated unit tests
All product builds must run the complete unit test suite. We use Hudson, integrated with JUnit, mbUnit, Test::Unit, jsUnity, and PHPUnit to execute all the unit tests with every build and to report on failures at any stage
Run builds on every checkin.
All regression tests in TestRun (our test plan management tool) must be automated using Selenium and added to the nightly build. This took some time and had to be done incrementally. With an end-to-end test that required 3 days of manual testing by the entire QA team when we started, the impact of incremental investments in test automation began to pay off quickly. Automation of existing regression tests became a background task for the QA Engineers for each sprint. Developers also pitched in, writing helper functions to ease automation and writing automated tests themselves.
All stories must have associated Selenium RC automated functional tests checked in and added to the nightly build test. In addition to the manual functional testing, every new story must have associated automated tests checked in and executing (via Hudson) nightly so that we were not adding to the regression debt.
Run an acceptance test of functional tests on every checkin.

When is a story done?

We established a very rigorous definition of ‘done’ for stories to ensure a consistent quality level. We also adopted ‘story swarming’ (applying as many developers/QA/DB to the story) to shorten times on individual stories and to avoid having many stories open at once.

For a story to be done:

All phases completed (in our case ‘To Do’, ‘In Progress’, ‘Security Review’, ‘Ready for QA’, ‘In QA’, ‘Validated’)
Unit testing complete
Security reviewed (code reviewed for web application security vulnerabilities)
Validated by QA
Test cases documented in TestRun
Automated QA testing complete
Validated by Product Owner
All Operational considerations have been addressed

Providing all these conditions have been met, the story will be demonstrated to the company at the Sprint Review on the second Friday of the two-week Sprint and released to customers the following Tuesday.

What else needs to be considered?

One of the things I often get asked about when moving so quickly is the coherency of the architecture and the user experience. At Genius, we employ several methods to ensure the architecture is appropriately scalable and maintainable and that the product is easy to use:

NMI (needs more information) stories. For user stories that have a significant impact on user experience or the underlying architecture, the team will first complete an NMI. NMI stories are focused on a subset of the team determining user flow (with leadership from the Product Designer) and/or underlying architecture (with leadership from the Technical Leads and the Development Director). The input to an NMI story is a list of questions that need answering (such as “how will the Marketing user…?” or “How can we ensure continuous availability of this feature during system maintenance?” The output of NMIs is a user flow or technical design, and a documented list of tasks for an upcoming sprint.
Development framework. Ease of use is a key differentiator at Genius, as is performance. We evaluated several frameworks and determined that to achieve the level of user interactivity required (Ajax) we would need to build our own lightweight PHP framework. This framework is now the basis for all new functionality added to the product – not only speeding development, but further ensuring consistency in coding and usability.
Designated ‘leads’ in each of the major technical components or code bases of the product, Technical Operations and User Experience with primary responsibly to making the team productive – and secondary responsibility to completing story tasks for the sprint.

Another concern with bi-weekly deployments is releasing partially complete features. As a SaaS provider, all the software we release to our production servers is immediately available to customers, so our goal is to complete at least a minimal feature set within each release. That said, we do make use of a beta flag (set by the provisioning team) to preview new features with customers or internally. This, combined with feature-based provisioning, can provide a lot of control over what an individual customer user can see or access. Of course, in the case that work on an existing feature is partially complete, we will typically rollback the code to the prior version (excluding it from the current sprint) to prevent user inconsistencies.

What’s up next?

The next step in our process evolution is to parallelize the nightly functional build tests (which currently contains over 600 Selenium scripts and runs for over 3 hours) so they can be run with every build. We are taking a two-pronged approach to this:

Virtualized Selenium servers in-house. These will be used to run functional tests against every build for a single browser.
Sauce Labs Sauce On Demand for cross-browser Selenium testing of all the automated functional tests on a daily basis.

In the future we will provide updates on our experiences with Sauce Labs and any other process developments.

Serving RESTful URLs with mod_rewrite

Ben VanEvery — Wed, 26 May 2010 00:10:13 +0000

We’ve been experimenting with an internal API to our app to facilitate development of UI tests by our QA team. After much discussion (likely to described in a blog post some time down the line), the decision was made to provide the API in a RESTful style over HTTPS. Rather than make a separate PHP script at each location an API request could possibly land, it made more sense to have all requests routed to a single handler that delegated the requests to the appropriate class. Doing this in Apache required a little configuration and research.

Apache provides a module for rewriting incoming URLs, thus giving us the ability to route all requests for the API to the PHP handler. Configuring this required a simple regular expression.

RewriteRule ^/api.*$ /api/handler.php [L]

That was simple enough. No more information is required in the URL by the handler, it has access to the originally requested resource through $_SERVER['REQUEST_URI']. This now means that a request to /api/team/1, would be actually be sent to /api/handler.php, which would then determine that a request for the details of the team with id 1 should be processed.

You may have noticed something odd about the rewrite above: there is no query string. Indeed, the query string is not specified and that is wrong. Two things are at play here, (1) RewriteRule does not match against any text in the query string, and (2) the substitution URL completely rewrites the URL, so the original query string is not magically carried over. This is simultaneously useful and troublesome. The mod_rewrite module provides several directives for working with the URL, one of which is named RewriteCond, which, in addition to the URL itself, lets you match against several of the other HTTP things going on (aka Server Variables). One of these is the query string. Matching against the query string (or anything for that matter) in a RewriteCond gives you access to back reference those matches in an ensuing RewriteRule directive using the %n syntax in addition the canonical $n syntax of the regular expression used for RewriteRule.

As useful as this is, in our situation we want to maintain the entire query string, which did not require use of the RewriteCond directive at all. We accomplish this by,

RewriteRule ^/api.*$ /api/handler.php?%{QUERY_STRING} [L]

For full details of the mod_rewrite module, check out the doc, http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html. The list of Server Variables available for substitution can be found in the discussion of the RewriteCond directive. For some examples of the RewriteCond directive, check out: http://fantomaster.com/faarticles/rewritingurls.txt.