Ansible projects usually start small and “under control” but as time passes by and your codebase grows (along with the product) things can start getting a little messy.
You all know this mesh of “we’re going microservices but not really,” which require more roles and more platforms to support.

We see more and more companies embracing the Microservices architecture, and each team decides on the platform and language for the individual application. While Docker made it easier to run, the system as a whole started to get a little more complicated to run, maintain, and test from an ops point of view.

In recent years, I’ve been putting much emphasis on unit and integration test for my clients, as a prerequisite for a successful CI / CD project. Without it, the confidence level in the application drops, and the whole process breaks. It usually goes like this:

Me: So does your team has a good Unit Test coverage?

Team lead: Yeah, it’s a project that we just started, we have *token fresh junior developer name* working on them.

Me: And how many tests are currently running?

Team lead: 0, at the moment we skip them, but it’s a real priority.

I’m a little harsh, but eventually, unit tests are something that developers do embrace, and it does increase the quality of the application.

Back to us DevOps people. I have to admit that when it comes for the deployment code, I rarely practice what I preach, and mostly rely on the CI process to validate my deployment code. Unfortunately, that cycle tends to take a lot of build log staring time, as I make changes, add features, and then have to test.

Because the deployments can take quite a bit of time, getting feedback while developing it is not immediate or requires some hacks.

When I need to add or modify a role, I will usually do one of two things:

1. Work directly on the project, add or change a few things, and then cross my fingers that everything will run smoothly. Then, I push and test, only to discover a few issues, bang my head, and repeat. Needless to say, it is not very effective.

2. I will start with a clean ansible-playbook on the side, prepare an environment that will accommodate my tests, which takes time, and then once everything is ready, and the code is working, I will move it to the real repo. Only to discover I broke a few other things.

Maybe that’s just me, but this whole workflow is not very efficient and prone to errors.

Enter Molecule

There are a few things we want to test during and after writing ansible roles: 1. YAML and Ansible syntax. We’ve all been there, trying to figure exactly what is wrong with line 9 position 4 “- name: give me a break ansible.” 2. That the role is indeed running. 3. That the role did what we expected it to do. 4. The role runs well on all of our supported platforms.

“Molecule is designed to aid in the development and testing of Ansible roles.

Molecule provides support for testing with multiple instances, operating systems and distributions, virtualization providers, test frameworks, and testing scenarios.

Molecule encourages an approach that results in consistently developed roles that are well-written, easily understood, and maintained.”

I have to say that after spending some time with Molecule, it did live up to some of its promises, and I was able to shave off some time working on a few roles.

The rough idea is that Molecule runs these tests * Linting - Syntax and YAML style. * Test the role on a locally provisioned Docker container or one of the other providers (EC2, Azure, GCE, Vagrant, etc.). * “Verify” the environment by running the unit test using TestInfra.

The great thing about it that just like with unit tests it helps you during the development phase and not just something you use on your build pipeline. Heck, you can even start working in TDD.

Molecule does have its drawbacks, and I will cover them later, but I do think that it is well worth it, especially as it looks like Ansible is still here with growing projects and DevOps engineers come can go.

Installing Molecule

For the official install guide - https://molecule.readthedocs.io/en/stable/installation.html Before you can install Molecule make sure that you have: 1. Ansible 2. Python 3. pip 4. Docker

Once you have all the prerequisites, you can go ahead and install molecule. I would recommend doing so in a virtual environment, but for simplicity, we’ll go straight ahead and install it.

    pip install molecule docker

Depending on your setup, you may have to install other modules as well. I had to do so after getting a few errors at runtime.

Using Molecule to boilerplate a role

This is where Molecule starts getting a little opinionated. You use it to create a role with all of its files and the molecule folder.

  molecule init role -r copy-files

Molecule will create a folder with the name of the new role with the following structure:

    .
    ├── README.md
    ├── defaults
    │   └── main.yml
    ├── handlers
    │   └── main.yml
    ├── meta
    │   └── main.yml
    ├── molecule
    │   └── default
    │       ├── Dockerfile.j2
    │       ├── INSTALL.rst
    │       ├── molecule.yml
    │       ├── playbook.yml
    │       └── tests
    │           ├── test_default.py
    │           └── test_default.pyc
    ├── tasks
    │   └── main.yml
    └── vars
        └── main.yml
    
    8 directories, 12 files

You can see Molecule created the role files, and also a molecule folder. Under that, the “default” folder is the initial “scenario”, and it includes:

• Dockerfile that represents your target inventory

• the INSTALL.rst includes additional requirements for testing

• molecule.yml includes the scenario configuration, such as driver, platforms, test library, etc.

• playbook.yml will run your role when testing this scenario.

The scenario concept allows you to run a variety of tests to cover your supported platforms and different architecture. For example, when running in a cluster or a standalone mode, or maybe supporting multiple target operating systems.

Molecule is based on Ansible Galaxy, which you may or may not use, and for that, it adds strict checks for things like the metadata, which may not be relevant for your project. For most of my use cases, I found it a little “bloaty” and added a little overhead, especially when migrating existing projects.

Testing with Molecule

Before we can run our little test, let’s add a simple copy task for our role. I created a small text file, placed it under the “files” folder, and added the copy task:

    mkdir files  
    echo "testing molecule" > files/test.txt

My tasks/main.yml file looks like this:

    ---
    # tasks file for copy-files
    - name: Copy the test file
      copy:
        src: test.txt
        dest: /test.txt

I added a tiny test:

    def test_test_txt_file(host):
        f = host.file('/test.txt')
    
        assert f.exists

Running the test

    molecule test

That didn’t go very well:

    [701] Role info should contain 
    [703] Should change default metadata: author
    [703] Should change default metadata: description
    [703] Should change default metadata: company
    [703] Should change default metadata: license

To save you some screen real-estate:

        =========================== 2 passed in 2.45 seconds ===========================
    Verifier completed successfully.

So what happens if we make a little change in the task, and change the destination file name?

The output is much more verbose, and there are many other things to show, but it does give the basic idea of what this tool provides.

The good

1. Can save you time by failing sooner than later.
2. Makes sure that your project is uniform documented and follow guidelines.
3. Helps you test on multiple platforms.
4. Unit tests are always a good idea in our world.
5. Isolate your work on just the building blocks.

What didn’t work for me

1. In more complicated projects, where sometimes the environment is more dynamic, it becomes a little harder to test the roles. You have to do more work on the Molecule playbooks to simulate the real environment and deployment process, which leads to duplicate code and may result in nulling the point of these tests.
2. In deployments that are docker heavy, using swarm or other orchestrators, the local tests will not be enough.
3. Added overhead for simple roles.

Final thoughts

Will I start using Molecule on my projects? I believe that I will. From my limited experience, it has already started to prove itself. Even though it may not cover all of my use cases, at least while developing the roles, there is value in this tool.

At the end of the day, DevOps code is not exempt from testing, and we have to push through the initial resistance of writing unit tests just like when promoting the habit in development teams.