Shadow Jobs

We take developer productivity pretty seriously at Slack. It’s multi-faceted, spreading across build speed, reliability, modernity, and more. One thing we really pride ourselves in is using the latest build tools when possible. Not necessarily bleeding edge (alphas, betas, RC, etc.), but the latest stable versions. Aside from the new features and improvements they bring, we found developers are generally happier feeling like they are using the latest and greatest. Life’s too short to not use modern tools.

Updates come with inherent risk though, especially when some build tools have long release cycles like the Android Gradle Plugin (AGP) or Kotlin. Updating on day one can impose a risk you’re not willing to take, as you’re trusting these tools’ tests to ensure they have no regressions. There’s no test as thorough as running it in your own codebase, but that’s not something library and tool maintainers can do.

Or can they?

Consider this: most major build tools have some degree of public alpha, beta, RC, or snapshot channel to allow developers to try out new versions and features before they’re stabilized. You wouldn’t check this in to your code base immediately, but you could test this! Many shops do but often wait until late in the development cycle to test something out, if ever.

At Slack, we test a number of core build tools continuously to catch regressions early and give feedback to maintainers as soon as possible and to give ourselves peace of mind updating to stable when it’s released. These are run as custom Continuous Integration (CI) jobs on GitHub Actions that build our repo against new, pre-release versions of the tools we use. These are called Shadow Jobs, because they shadow the main branch and just apply a set of changes on top of it to build it against these newer tools. In this post, we’ll cover how to design a simple shadow job and how to integrate it in your build. It’s oriented toward an Android/Kotlin/Gradle codebase, but the concepts should be pretty transferable!

Composing a Build

In a GitHub Actions workflow, this is easy to set up! You don’t have to use Actions, but we’ve found it very amenable to this case. Most modern CI systems should be able to facilitate this pattern.

Our workflows are pretty complex, but let’s look at a simple example workflow for testing newer Java versions, AGP versions, and Kotlin IR.

name: (🔮) Build Tools

on:
  schedule:
    - cron: '0 8 * * *'

jobs:
  build:
    # Names are shortened for readability in CI
    name: 'J=${{ matrix.java }} | AGP=${{ matrix.agp }} | IR=${{ matrix.kotlinIR }}'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        java: [1.8, 11, 15]
        agp: [4.1.2, 4.2.0-beta04, 7.0.0-alpha05]
        kotlinIR: [true, false]
        exclude:
          - agp: 7.0.0-alpha05
            java: 1.8
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Gradle Wrapper Validation
        uses: gradle/wrapper-validation-action@v1

      - name: Install JDK ${{ matrix.java }}
        uses: actions/setup-java@v1
        with:
          java-version: ${{ matrix.java }}

      - name: Build
        id: gradle
        uses: eskatos/gradle-command-action@v1
        with:
          arguments: :app:assembleExternalRelease :app:lintExternalRelease test --stacktrace -PagpVersion=${{ matrix.agp }} -PkotlinIREnabled=${{ matrix.kotlinIR }}

Let’s break this down section by section.

First, we schedule this to run on a nightly basis via the cron scheduling event. We could run it more often if we wanted, but found this cadence to be enough for us to check in on tools. In this case, it runs once a day at 8am UTC (giving us a “nightly” run for our largely-North-American-based team).

on:
  schedule:
    - cron: '0 8 * * *'

We set fail-fast: false because we want all the runs of our matrix to complete since each tests a different combination. By default, Actions will cancel all other pending jobs upon the first failure.

fail-fast: false

We define a matrix via the matrix strategy section with all the versions we want to test.

matrix:
  java: [1.8, 11, 15]
  agp: [4.1.2, 4.2.0-beta04, 7.0.0-alpha05]
  kotlinIR: [true, false]
  exclude:
    - agp: 7.0.0-alpha05
      java: 1.8

It’s fairly straightforward, but a couple things warrant added context:

• Even though we can only target JDK 8 for Android binaries, we want developers to be able to use modern tools and plan to move up to a higher version soon. AGP 7.0 will also require a minimum of JDK 11. We’re not sure which version we’ll move up to formally, so we test both!
• Since AGP 7 requires JDK 11, we don’t want to run a build with AGP 7 and Java 8. We use exclude: here to exclude all combinations that include AGP 7 and Java 8. This is a powerful way to cut down on redundant or otherwise not-usable combinations.

The first place we use a matrix input is in the setup-java action

- name: Install JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}

This is pretty simple and handled right in the workflow. The setup-java action will use the version we define here.

The next section is a bit more complex. This is where we actually run our build (unit tests, lint, and assembling the production app).

In order to pass our matrix values into the build, we expose them via Gradle Properties and pass them via -Pproperty=value syntax in the command.

- name: Build
  id: gradle
  uses: eskatos/gradle-command-action@v1
  with:
  arguments: :app:assembleExternalRelease :app:lintExternalRelease test --stacktrace -PagpVersion=${{ matrix.agp }} -PkotlinIREnabled=${{ matrix.kotlinIR }}

To pick these up in our build, we simply look for these properties. Here is an example build file for a simple project that reads the above properties and uses them for setting the AGP version and Kotlin IR.

buildscript {
  dependencies {
    // AGP dependency. Must go before Kotlin's
    val agpVersion = findProperty("agpVersion")?.toString() ?: "4.1.2"
    classpath("com.android.tools.build:gradle:$agpVersion}")
    classpath(kotlin("gradle-plugin", version = "1.4.30"))
  }
}

apply(plugin = "com.android.application")
plugins {
  kotlin("android")
}

val irEnabled = properties.findProperty("kotlinIREnabled")?.toBoolean() == true

tasks.withType<KotlinCompile>().configureEach {
  kotlinOptions {
    useIR = irEnabled
  }
}

// Rest of the build file...

Now we’ll have a matrix of builds that cover all possible combinations. Running these nightly allow us to get early insight into how our build reacts to these types. Running them as a matrix allows us to also test their interaction with newer versions of other tools, too!

You can reuse the above patterns to configure a number of other things as well. Here are some examples of other things we currently test, or have tested in the past:

• Kotlin milestone/RC versions via Gradle property
• New Android SDK versions via Gradle property
• Upcoming Gradle RCs and nightlies via the eskatos/gradle-command-action action, which supports different configurations as arguments. See its README.
• Snapshots/RCs of regular app library dependencies and building against them early too. This includes libraries in Sonatype’s snapshots repo, Kotlin EAP snapshots, and AndroidX’s snapshots.
• Internal changes even! You don’t have to just limit this pattern to external dependencies, another great use case is for incubating an invasive internal change that you want to give extra time to bake.
  • One example of this for us was an RxJava 3 migration script that we wanted to test for a couple of weeks before committing to applying it in the codebase. This allowed us to migrate to RxJava 3 with virtually no issues.

Closing Notes

We’ve been using shadow jobs for about a year now and in that time we’ve caught dozens of issues early. I highly recommend doing the same in your own project! Especially if you use Kotlin, Jetbrains is actively asking developers to try out the new IR backend and this is an excellent way to do that.

Not only do you give your team confidence in updating to new dependencies, but you also give valuable early feedback to developers of these tools. On top of all that, these tools often introduce improvements that find existing issues in your codebase (particularly Kotlin updates), and you can usually backport fixes for these issues into your codebase today.

Lastly – by staying on top of these we also give team members the ability to try new tools that depend on these, like Android Studio Arctic Fox or Jetpack Compose. Even if it’s only local usage, we better empower them to forge ahead.

If you find this interesting and like using modern tools, come join us! https://slack.com/careers

Appendix

To really drive home the value we’ve gotten out of this, here is a comprehensive list of every issue we’ve filed as a result of these jobs over the past year:

• Kotlin: KT-44420, KT-43538, KT-44045, KT-43242, KT-43241, KT-42915, KT-41493, KT-42257, KT-42752, KT-41484, KT-41494, KT-41702, KT-41732, KT-41486, KT-40351, KT-40485, KT-40091, KT-37944, KT-36478, KT-33052
• Android: b/177562904, b/170026127, b/162155191, b/161920328, b/161727305, b/159516509, b/157723604, b/157681341, b/156657359, b/148284064, b/148284065, b/162528480, b/162446295, b/162259266, b/161381697, b/160815393, b/160714171, b/158597249, b/154315491, b/148929520
• Gradle: #15674, #13333, #13302, #13811, #12997, #12452, #12198