Configuration management with Archaius (from Netflix)

IMPORTANT: This blog has been moved to jlordiales.me. The content here might be outdated. To see this post on the new blog go to http://jlordiales.me/2014/10/07/configuration-management-with-archaius-from-netflix
So you usually need to read some configuration variables from your app. Maybe you read them from the system properties like:

String prop = System.getProperty("myProperty");
int x = DEFAULT_VALUE;
try {
    x = Integer.parseInt(prop);
} catch (NumberFormatException e) {
    // handle format issues
}
myMethod(x);

Or maybe you have a properties file that you read using Spring. Or maybe you have a simple key/value table in your DB with some properties that you read from there? Or you get them from an external REST endpoint? Or from other type of key/value store like Redis or Memcached?

Whatever the case might be your configuration variables might be coming from a lot of different sources and, specially if your app uses more than one, this can become difficult to maintain. Additionally, you don’t want to do a re-deploy every time you need to change the value of one of your properties, particularly for feature toogles.

Luckily, the Netflix folks already had these issues and came up with a solution that they were kind enough to open source. If you haven’t seen Netflix Github repository I strongly recommend that you take a look. They have some serious cool projects that could be just the thing your application needs. One of those projects is the one that concerns us today: Archaius.

Their wiki and examples should give you a very good idea of what Archaius is and what it is useful for. For now I’ll just say that Archaius is an extension of Apache’s Common Configuration library that allows you to retrieve properties from several dynamic sources and that it solves all the issues mentioned previously (heterogeneous sources of properties, run-time changes, etc.).

Rather than going into much details about what the tool is and does I’ll go with some examples instead.
What is the simplest possible example to use Archaius to read a property file? The well known “Hello World”:

public class ApplicationConfig {

    public String getStringProperty(String key, String defaultValue) {
        final DynamicStringProperty property = DynamicPropertyFactory.getInstance().getStringProperty(key,
            defaultValue);
        return property.get();
    }
}

public class ApplicationConfigTest {
    private ApplicationConfig appConfig = new ApplicationConfig();

    @Test
    public void shouldRetrieveThePropertyByKey() {
        String property = appConfig.getStringProperty("hello.world.message", "default message");

        assertThat(property, is("Hello Archaius World!"));
    }

    @Test
    public void shouldRetrieveDefaultValueWhenKeyIsNotPresent() {
        String property = appConfig.getStringProperty("some.key", "default message");

        assertThat(property, is("default message"));
    }
}

That code, together with a “config.properties” file somewhere in your classpath (src/main/resources by convention in Maven and Gradle for example).

Notice that you don’t need to tell Archaius where to find your properties file because the default name that he is going to be looking for is “config.properties”. This example doesn’t seem like a big improvement over what you would usually do with Spring or any other tool but remember that this is just a “Hello World” and bear with me to see more interesting use cases.

What if you don’t want or can’t name your property file “config.property”? In that case you need to tell Archaius where to look for this file. You can easily do this changing the system property “archaius.configurationSource.defaultFileName”, either passing it as a parameter to the vm when you start your application (java … -Darchaius.configurationSource.defaultFileName=customName.properties) or in the code itself:

public class ApplicationConfig {
    static {
        System.setProperty("archaius.configurationSource.defaultFileName", "customConfig.properties");
    }

    public String getStringProperty(String key, String defaultValue) {
        final DynamicStringProperty property = DynamicPropertyFactory.getInstance().getStringProperty(key,
            defaultValue);
        return property.get();
    }
}

Simple enough, right?

Now, what if you want to read several properties files? You can easily define a chain of property files and the order in which they should be loaded starting from the default file which is loaded first. From there, you can specify a special property with key “@next=nextFile.properties” to tell Archaius which is the next file that should be loaded.
In our example so far, we could add the following line to our “customConfig.properties” file:
“@next=secondConfig.properties” and add the corresponding “secondConfig.properties” to our resources folder with the following content:
“cascade.property=cascade value”.

We can see this working by adding the following test to our ApplicationConfigTest class:

@Test
public void shouldReadCascadeConfigurationFiles() {
   String property = appConfig.getStringProperty("cascade.property", "not found");

   assertThat(property, is("cascade value"));
}

Note that we are getting the property from the new file without any additional change to our ApplicationConfig class. This is completely transparent from the point of view of our client.

Until now, we have been reading properties just from different property files but what if you want to read them from a different source?
In the most general case, you can code your own logic by implementing “com.netflix.config.PolledConfigurationSource”.
If that new source is a database that can be accessed through JDBC then Archaius already provides a “JDBCConfigurationSource” that you can use. You only need to tell him what query he should use to get the properties and which columns represent the property key and property value.

So our example would look like:

@Component
public class ApplicationConfig {
    static {
        System.setProperty("archaius.configurationSource.defaultFileName", "customConfig.properties");
    }

    private final DataSource dataSource;

    @Autowired
    public ApplicationConfig(DataSource dataSource) {
        this.dataSource = dataSource;
        installJdbcSource();
    }

    public String getStringProperty(String key, String defaultValue) {
        final DynamicStringProperty property = DynamicPropertyFactory.getInstance().getStringProperty(key,
            defaultValue);
        return property.get();
    }

    private void installJdbcSource() {
        if (!isConfigurationInstalled()) {
            PolledConfigurationSource source = new JDBCConfigurationSource(dataSource,
                "select distinct property_key, property_value from properties", "property_key", "property_value");
            DynamicConfiguration configuration = new DynamicConfiguration(source,
                new FixedDelayPollingScheduler(100, 1000, true));

            ConfigurationManager.install(configuration);
        }
    }
}

We are using Spring to autowire a data source that will use an in-memory H2 database with a simple key/value table. Note how we create a new PolledConfigurationSource using the JDBCConfigurationSource already provided by Archaius and then we register the new configuration using the ConfigurationManager. After doing this we can get any property from the DB exactly the same way we do it for properties files (i.e., using the DynamicPropertyFactory).

We can now add a couple of tests to make sure that we are actually reading properties from the DB and that we can update their values and see the changes reflected in our dynamic configuration.

    @Test
    public void shouldRetrievePropertyFromDB() {
        String property = appConfig.getStringProperty("db.property", "default message");

        assertThat(property, is("this is a db property"));
    }

    @Test
    public void shouldReadTheNewValueAfterTheSpecifiedDelay() throws InterruptedException {
        template.update("update properties set property_value = 'changed value' where property_key = 'db.property'");
        String property = appConfig.getStringProperty("db.property", "default message");

        //We updated the value on the DB but Archaius polls for changes every 1000 milliseconds so it still sees the old value
        assertThat(property, is("this is a db property"));

        Thread.sleep(1500);

        property = appConfig.getStringProperty("db.property", "default message");
        assertThat(property, is("changed value"));
    }

To conclude this post, another really cool feature offered by Archaius is the possibility to register our configurations as MBeans via JMX. We can do this by default setting the system property archaius.dynamicPropertyFactory.registerConfigWithJMX=true or programmatically with ConfigJMXManager.registerConfigMbean(config);.

After doing this we can connect via JConsole and not only get the value of all properties but also update them and see their new value reflected in Archaius. This would allow us, for instance, to change the values of properties defined statically in property files during runtime without the need for a server push.
We can modify our ApplicationConfig class a little bit to add a main method that will keep running printing the values of different properties, allowing us to play around in JConsole.

public class ApplicationConfig extends Thread {

    private final DataSource dataSource;

    @Autowired
    public ApplicationConfig(DataSource dataSource) {
        this.dataSource = dataSource;
        cascadeDefaultConfiguration();
        DynamicConfiguration jdbcSource = installJdbcSource();
        registerMBean(jdbcSource);
    }

    public String getStringProperty(String key, String defaultValue) {
        final DynamicStringProperty property = DynamicPropertyFactory.getInstance().getStringProperty(key,
            defaultValue);
        return property.get();
    }

    @Override
    public void run() {
        while (true) {
            try {
                sleep(100);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

        }
    }

    private void registerMBean(DynamicConfiguration jdbcSource) {
        setDaemon(false);
        ConfigJMXManager.registerConfigMbean(jdbcSource);
    }

    public static void main(String[] args) {
        ApplicationContext applicationContext = new ClassPathXmlApplicationContext("archaiusContext.xml");
        ApplicationConfig applicationConfig = (ApplicationConfig) applicationContext.getBean("applicationConfig");

        applicationConfig.start();

        while (true) {
            try {
                System.out.println(applicationConfig.getStringProperty("hello.world.message", "default message"));
                System.out.println(applicationConfig.getStringProperty("cascade.property", "default message"));
                System.out.println(applicationConfig.getStringProperty("db.property", "default message"));
                sleep(3000);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }

        }
    }
}

And that’s it for now. You can find all the code showed in this post at https://github.com/jlordiales/archaius-example.

There’s a lot more you can do with Archaius, like Callbacks everytime a property changes, integration with Zookeeper and other services. Check out the Archaius project and examples at their Github repo.

Advertisements

When git ignores your… .gitignore?

IMPORTANT: This blog has been moved to jlordiales.me. The content here might be outdated. To see this post on the new blog go to http://jlordiales.me/2012/12/28/when-git-ignores-your-gitignore

I feel like I should start this post saying that I absolutely love git. If you’ve never heard of it, is a source control system like CVS or Subversion but, unlike those two, is a distributed version control system. I’m not going to get into much details about the history and capabilities of git but if you’re curious about it you can go to http://git-scm.com/book which is an amazing resource with everything from intro to advanced concepts.

I imagine that by now most professional software developers use some form of version control at their daily jobs but you shouldn’t stop there. I use git for personal, one-man projects as well. While some people might think that this could be an overkill, I completely disagree. There is nothing like the comfort in knowing that all the history of all your files is safe and ready to be brought back if you ever need them. We all make mistakes sometimes after all. With git this is as easy as writing 3 simple commands:

mkdir myNewProject
cd myNewProject
git init

That’s it! Every file you create or modify on “myNewProject” will be now tracked by git.

A pretty useful feature that you get with every source control tool is the possibility to ignore certain files from the tool’s tracking. Generally speaking, you don’t want to get into your code repository any file that can be computed as a result of another file. In a typical Java Maven project this would be for example the “target” directory, or if you are using Eclipse the “.metadata”, “.project” or “.settings” files. In git the easiest way to do this is to have a special file named “.gitignore” at the root of your project with all the exclusion rules you want to set. The syntax of this file is fairly straightforward. You can also have a “.gitignore” file for each subdirectory in your project, but this is less common.

A tricky thing with git and ignore rules is that if the file you want to ignore is already being tracked by git then adding it to “.gitignore” won’t make git to automatically forget about the file. To illustrate this point, consider the following example. First we create a repository with two initial files and commit them:

mkdir gitExample
cd gitExample
touch file1 file2
git init
git add .
git commit -m "Initial commit"

Let’s create now the .gitignore file to try to ignore “file2” and commit that:

echo file2 > .gitignore
git add .
git commit -m "Added gitignore for file2"

Now, let’s modify “file2” and see what happens:

echo "Hello World" >> file2
git status

We get:

# On branch master
# Changes not staged for commit:
# (use "git add ..." to update what will be committed)
# (use "git checkout -- ..." to discard changes in working directory)
#
# modified: file2
#
no changes added to commit (use "git add" and/or "git commit -a")

Git is effectively still tracking file2 even though is already on our .gitignore. Like I said before, this happens because git was already tracking the file when we added it to our ignores. So let’s see what happens when we add the “.gitignore” before adding the file to git:

mkdir gitExample
cd gitExample
touch file1 file2
git init
echo "file2" > .gitignore
git status

And now we get:

# On branch master
#
# Initial commit
#
# Untracked files:
# (use "git add ..." to include in what will be committed)
#
# .gitignore
# file1
nothing added to commit but untracked files present (use "git add" to track)

Cool! No mention of file2 anywhere! But if, as our first example, we forgot to add the files to our .gitignore initially? How do we stop git from tracking them? A nice command we can use for this cases is git rm --cached file. In our first example:

git rm --cached file2
git commit -m "Removed file2"

If we now modify the file again and do a git status we get:

echo "Hello World Again" >> file2
git status


# On branch master
nothing to commit (working directory clean)

Exactly what we wanted!

Note that this little command will remove the file from git index but it won’t do anything with your working copy. That means that you will still have the file on your directory but the file won’t be a part of the git repository anymore. This also implies that the next time you do a push to a remote repository the file won’t be pushed and, if it already existed on the remote repository, it will be deleted.

This is fine for a typical use case when you added a file that you never intended to have on the repository. But consider this other use case. In a lot of projects the developers upload into the remote central repository a set of config files for their IDE with default values for things like code formatting and style checking. But at the same time, when developers clone the repository they customize those config files to suit their personal preferences. However, those changes that apply to each particular developer should not be commited and pushed back to the repository. The problem with using git rm --cached here is that, while each developer will still have their own copy, the next time they push to the server they’ll remove the default config from there. In cases like this, there is another pretty useful command that will do the trick: git update-index --assume-unchanged <file>. Let’s see that with an example:

mkdir gitExample
cd gitExample
touch file1 file2
git init
git add .
git commit -m "Initial commit"

There we have our default “file2”. Now let’s use the git update-index command and make some changes to the file:

git update-index --assume-unchanged file2
echo "Hello World" >> file2
git status

The result:

# On branch master
nothing to commit (working directory clean)

Magic! Changes to our file are no longer seen by git and the original file is still on the repository to be cloned by other users with its default value.

Hope this helps someone! Cheers!