Selectively running Android modularized unit tests on your CI server

Header Photo by Icons8 Team on Unsplash

Modularizing your Android projects can bring a number of different advantages to your team. Some of these include reduced build times, a greater separation of concerns and the ability to reuse components throughout our applications. As we started to get more and more modules in our projects, I started to think more about how these were being run on our CI server. For example, we open a pull request and for that changed code all of our tests and checks are run for the entire project. When you have a couple of modules you probably won’t see a concern here. But what if we have 30 modules, each with plenty of code / tests, and we open a pull request that only makes changes to one of those modules? In this article I want to share how we’ve made some additions to our CI to help here!

We’ve been trying to reduce our CI times recently, so with our modularisation this seemed like a good place to start looking. We have unit tests in every feature module in our application and we have around 20 modules currently. Whilst unit tests don’t take too long to run, being able to shave some time off of each build that occurs will add up over the days, weeks and months that our CI is building our tasks. With restricted concurrent builds on our current CI plan, that saved time helps to free up our CI server quicker, keeping us more productive in our work.

Unfortunately, there’s no magic way to detect what modules have changes to them and only run the tests for those modules. In Android we can either run a gradle test task from the root of our project, or individually for each of the modules in our project. Even on our CI server (bitrise) the test step takes a single test command, which by default uses the test task from the root of the project. When it comes to running unit tests via gradle, we can however provide a list of test commands to run during our test task, for example:

./gradlew :moduleA:testDebugUnitTest :moduleB:testDebugUnitTest

That would solve all of our problems when it comes to running our unit tests, but how do we get there? There are a couple of things that we need to do in order to build our test commands dynamically.

We need to begin by detecting the modules in our code that have changed files in them. Again, there’s no straightforward way to detect this from within the CI server – so we’re going to need to perform some git diffing and calculate the changed modules using those diffs. This is going to look something like so:

dest=origin/_branch_merging_into_
branch=origin/_branch_for_pull_request_

changed_modules=""

git diff --name-only $dest..$branch | { while read line
    do
      module_name=${line%%/*}

      if [[ ${module_name} != "buildSrc" && 
            ${changed_modules} != *"$module_name"* ]]; then 
              changed_modules="${test_modules} ${module_name}"
      fi
    done
}

We need to start by retrieving the destination that our branch is being merged into, along with the actual branch for the pull request that has been opened. You can’t hardcode these as everytime you open a pull request this code is going to be run. On bitrise you can access environment variables to get these values:

dest=origin/$BITRISEIO_GIT_BRANCH_DEST
branch=origin/$BITRISE_GIT_BRANCH

Next we’re going to perform the git diff operation against these two branches. From the code above, the section below takes our two branches and loops through each line that is presented in the diff. However, we don’t care for the actual diff content, we only want the names of the files that have changes. Using the –name-only command when performing the diff means that we be presented only with the file names, instead of the file diff content.

git diff --name-only $dest..$branch | { while read line
    do

    done
}

Now that we have our changed files, we need to pull out the module name from each of them. In the line below, we pull out the string content up until the first forward slash character.

module_name=${line%%/*}

To note, this isn’t a sure fire way of getting the module name as it can yield unexpected values. For example, if we change a gradle or text file in the root of our project which isn’t within a module, then this module_name variable could be assigned with something that doesn’t represent a module. The same goes for modules that we have deleted – whilst these would appear in the diff, we wouldn’t want to run the tests for them as the module no longer exists. The next script that we write will handle this, for now we just want to get a list of everything that has changed. This way, we can also re-use this script if we decide to selectively run other things on our CI server.

The last piece of code in our script will be used to build our list of changed module names. So for each module_name variable we’re going to add it to our changed_modules variable, in the end this will result in a single string representing separated module names.

You may notice that this is all wrapped in an if statement – this checks as to whether  the changed_modules already contains the current module_name and if so, we don’t want to re-add it to our changed_modules variable (otherwise we will end up with duplicates!).

if [[ ${changed_modules} != *"$module_name"* ]]; then 
      
      changed_modules="${changed_modules} ${module_name}"

fi

At this point we have a list of module names in a single string. Depending on your CI service, you may need to pass this to another script step. In the case of bitrise, you can write this value to an environment variable to re-use in other script steps for build module specific commands:

envman add --key CHANGED_MODULES --value "${changed_modules}"

From the above operations we’re going to now have a string that represents all of the different module names in our application. This might look something like:

moduleA moduleB moduleC

Now that we have a collection of these module names, we need to go ahead and build the test task using those names so that we can run the unit tests for that module. For this we’re going to need to take each one of the module names from our string and create the test command for each one. For running our unit tests we’re going to want to end up with a string that looks like:

:moduleA:testDebugUnitTest :moduleB:testDebugUnitTest ...

However, as previously mentioned, it might be the case that some module names that we’ve acquired aren’t actually modules within our application. For example, if you’ve edited the build.gradle / gradle.properties files, your buildSrc module or even deleted moduleA from your application, then these will all be contained within your module name string. For this reason, before we build our test command string we need to filter out anything that doesn’t support us running unit tests for the module within our application. The code to achieve this looks like so:

AVAILABLE_TASKS=$(./gradlew tasks --all)
modules=$CHANGED_MODULES

test_commands=""

for module in $commands
do 
    if [[ $AVAILABLE_TASKS =~ $module":" ]]; then 
        test_commands=
            ${test_commands}" :"${module}":testDebugUnitTest"
    fi
done

if [[ $test_commands == "" ]]; then
    test_commands="test"
fi

envman add --key UNIT_TEST_COMMANDS --value "${test_commands}"

We begin by retrieving all of the available gradle tasks within our application:

AVAILABLE_TASKS=$(./gradlew tasks --all)

Whilst this doesn’t provide us with the actual commands for running tests, it does tell us the modules that can have commands run against them. For example:

• If we make changes to the buildSrc module, this has no gradle tasks to run against it that will come back from our tasks command

• A deleted module would not show any gradle commands that can be run for it

• If root files have been edited (root build.gradle, gradle.properties) that are not in a module, these names will not match any modules and their commands

With the tasks –all command we can retrieve a collection of modules and check our module names against them. With this collection of commands we can now take our generated module names from the last script and check these agains the available commands. We’ll begin by retrieving this module names from the environment variable that we saved them to:

modules=$CHANGED_MODULES

Next we need to check whether our available tasks contains a reference to the modules within our module names string. For this we loop through each of the module names in our string and verify that the module name is supported for our needs:

for module in $modules
do 
    if [[ $AVAILABLE_TASKS =~ $module":" ]]; then 
        test_commands=
            ${test_commands}" :"${module}":testDebugUnitTest"
    fi
done

If you run the tasks –all command then you’ll see something like the following:

moduleA:someTask moduleA: anotherTask moduleB:someTask ...

And in our script above we have the following line:

if [[ $AVAILABLE_TASKS =~ $module":" ]];

Here we are taking our module name, appending it with a colon and asserting whether our variable containing the tasks has a reference to this string value. If so, we can presume that the module supports the unit test command that we want to run. If so, then we append our test_commands variable with the command to run the unit tests for our module.

test_commands=
            ${test_commands}" :"${module}":testDebugUnitTest"

If for some reason out test_commands variable is empty, either something has gone wrong or no modules have been changed – maybe only the build.gradle file has only been changed in the current PR. Here you can either not run any tests, or have a safeguard in place that will just run all of the tests for the project.  This can be done by assigning the “test” string value to our test_commands variable.

if [[ $test_commands == "" ]]; then
    test_commands="test"
fi

With the above done we should now have a string variable that looks something like so:

:moduleA:testDebugUnitTest :moduleB:testDebugUnitTest

This is great! Now we have  a collection of the commands that need to be run for our unit test task. The only thing left to do is to save this to an environment variable so that our CI can use it within the unit test task.

envman add --key UNIT_TEST_COMMANDS --value "${test_commands}"

This next part will really depend on the service you are using for your CI. For us we are using Bitrise, Bitrise provides a Gradle Unit Test step which is used to run the unit tests in a project. This step tasks a Test task input variable – this is where we are now gong to pass a reference to our UNIT_TEST_COMMANDS variable

Now when our unit tests are run, only the unit tests for the changed modules are run. This will help us to shave some times off our builds, allowing us to be more productive and efficient when building our products!

With all of the above you are able to put something in place that allows you to selectively run unit tests for your modularised android project. Even if the above isn’t exactly what you’re looking to put in place, you may be able to use some of the module specific scripting for something else within your CI server.

Are you already using scripting for these kind of things, or looking to put something in place? Feel free to reach out and I’ll be happy to chat over any of these things!