Migration Automation: Easing the Jenkins → GHA shift with help from AI

Overview

The past few months have been exciting times for Slack’s CI infrastructure. After years of developer frustration with Jenkins (everything from security issues to downtime to generally poor UX) internal pressure led us to move a majority of Slack’s CI jobs from Jenkins to GitHub Actions. 

My intern project at Slack this summer involved creating a conversion tool that could automatically migrate Jenkins pipelines to GitHub Actions, saving developers time and accelerating the migration. The project was successful, and is expected to cut the migration time by half and save over 1,300 hours. This blog is focused on that conversion tool, and the 7 week journey to design, implement, and improve it. 

Before we start though, you should know some terminology:

• Pipeline: A set of Jenkins plugins used for organizing Jenkins jobs. “Jenkins pipeline” refers to a set of one or more ordered Jenkins jobs. At the beginning of this project we had242 Jenkins pipelines to migrate.
• Instance: A Jenkins instance is where Jenkins pipelines are hosted. We were migrating off of 3 Jenkins instances.
• Workflow: a GitHub Actions CI job.
• GHA: GitHub Actions.

Planning for the Project

Pondering the Problem

The success criteria for the project are as follows:

1. Create tooling that automates the difficulties of manually migrating a Jenkins pipeline to GHA
2. Run this tooling on a large subset of existing Jenkins jobs
3. Create documentation around the conversion process 

Tools of the Trade

After doing some research, we identified a few useful tools to use to convert from Jenkins to GHA:

• The GitHub Actions Importer. GitHub publishes a tool that can audit a Jenkins instance and either fully convert, partially convert, or fail to convert all pipelines on it to GitHub Actions. Although generally most pipelines fully convert, not all do. So a few more tools will be needed after using the Importer.

1. Python scripting: It seemed likely that some amount of the Importer’s errors would be easy to fix with just regex scripts. We expected to use Large Language Models (LLMs) for any errors that were too intricate to fix with regular expressions.

The Solution

Part 1 – The Importer

Over the span of a week, we ran the Importer on several of our Jenkins instances. The tool was simple. You install it from the GitHub CLI, configure it, and run a command like this:

gh actions-importer audit jenkins --output-dir <output-dir>

The Importer would then output GitHub Actions workflow files for each Jenkins pipeline on the Jenkins instance being audited. Upon running the Importer, we were curious as to how well it performed. Fortunately, the Importer creates an audit summary upon completing its audit of a Jenkins instance. The audit file contains information about how many Jenkins jobs were fully converted, partially converted, and failed to convert from Jenkins to GHA.

About 50% of the time the Importer moved the pipeline from Jenkins to GHA without error. 5% of the time the importer failed completely, and the rest of the jobs were partially-imported — it created a workflow YAML file, but wasn’t able to populate it fully.

In addition to the success and failure rates, we were curious as to exactly what Jenkins build steps and environments were unsupported by the Importer. Fortunately the audit summaries contain information about this as well. Glancing at the audit summary, it was initially concerning to see that there were hundreds of Jenkins pipelines that weren’t able to fully convert to GitHub Actions. A deeper inspection revealed a pleasant surprise – only eight unsupported Jenkins build steps and environments were responsible for over 90% of those failures.

This was good news. Instead of needing to spread ourselves thin looking at dozens of unsupported Jenkins steps, we instead only had to look at eight. For those key few unsupported items, we’d need to look at the following:

1. Is this actually unsupported by GHA, or do equivalents exist?
2. If equivalents do exist, is it easy to write scripts to edit the Importer’s outputs accordingly?

Part 2 – Research

At this point in the project, it was known that the Importer’s workflows had a lot of room for improvement. Around half of the pipelines didn’t fully convert, so a big objective was to look into the unsupported Jenkins steps that caused them to not fully convert. Another task though was to investigate the pipelines that fully converted. Can we really trust the Importer when it claims that it could fully convert something? So really every pipeline that got (or didn’t get) converted to a workflow had to be looked at. 

After looking through several dozen of the YAML files the Importer made, it was clear that there were 4 main types of corrections we’d need to make to the workflow YAML files:

1. Replacing rate-limited actions with internal mirrors 
2. Replacing actions with other actions
3. Adding useful comments for end users 
4. Removing unnecessary comments

Over the course of 2 weeks, we studied each of these:

1 – Replacing rate limited actions with internal mirrors

RoI: Very High

Overview

By default, if you use an arbitrary action from the GitHub Actions marketplace, like say actions/github-script, any requests made by the action will hit the github.com API. There are rate limits for this. Actions that hit our self-hosted GitHub Enterprise instance have far higher rate limits however. So a task we had to do was replace rate-limited actions with internal mirrors of those actions that we have on our GitHub Enterprise instance.

End-User Value

Correcting rate-limited actions is simple to do on our end, and prevents end users from dealing with the headache that is having jobs fail from rate limits. So high RoI here.

2 – Replacing actions with other actions

RoI: High

Overview

There were a few instances where, to perform a certain task, the Importer would use one action whereas we’d prefer if it had used another. For instance, for sending Slack messages the Importer often elected to use the rtCamp/action-slack-notify action. Naturally, Slack has internal actions for sending Slack messages, and we’d rather use internal libraries for this purpose. So something to look into was replacing rtCamp/action-slack-notify with Slack’s internal messaging action.

This type of task seemed suitable for an LLM to do, since two actions often have disparate syntax which is hard to convert between by just using Python’s string methods. Preliminary testing also showed perfect LLM performance for this task which was reassuring.

End-User Value

Replacing one action with a different action would require end users to know both actions well. So automating this saves quite a bit of time and effort on their end, and it’s not that hard to write LLM prompts for these tasks. The RoI is thus high.

3 – Adding Useful Comments for End Users

RoI: Medium

Overview

Some action items in the workflow files could only be done manually. For example, the syntax used in the workflow files to access secrets assumed that the secrets would be stored on GitHub. However, we don’t store our secrets on GitHub, and it would be time prohibitive and against our security policies to move them to GitHub. It would instead be better for end users to edit the workflow files such that secrets get read from where we store them instead of from GitHub. This couldn’t really be automated on the conversion tool side, since only the end users know where their secrets are and how to access them. All the conversion tool could do in such situations was leave a comment like: “Change these lines so that secrets are being read from where we store them instead of from GitHub. Go to a certain slack channel to find relevant docs about this.”

End-User Value

It’s definitely useful for end users to know where they should look to fix problems, but they still have to do the actual work of fixing the problem. So overall, medium RoI. 

4 – Removing Unnecessary Comments

RoI: Low-medium

Overview

Recall that around 45% of pipelines only partially converted. What this means is that the Importer couldn’t find a GHA equivalent for a Jenkins environment or build step. When this happened, the Importer left a comment in the workflow that looked something like “X Jenkins item was unsupported.” 

As it turns out, the reason why a lot of Jenkins pipelines were unsupported is because that piece of functionality in Jenkins was either already included or not necessary in GHA. So most of those “X was unsupported” comments weren’t too important and would probably be better off removed.

End-User Value

The benefits here to end users was small, as all it really does is declutter the workflows. So overall this had a low return. However, a script to delete strings is also very easy to write. So overall low-medium RoI.

Part 3 – Corrections Tool

Overview

The last part of the implementation is the corrections tool, which would correct the Importer’s workflows. We knew from the start that the tool would have a balance of using Python’s string methods and using LLMs. From the last section, note that out of the 4 categories of action items for the correction tool, the only one that needed AI was “replacing actions with other actions”. So the corrections tool was largely non-AI. It took around 3 weeks to make the corrections tool.

Implementation Architecture

Here’s the bird’s-eye view. The input to the corrections tool was the path to a directory containing workflow yaml files. The following chain of events then happened:

1. The path to each yaml file in that directory was written to an array. Then, for each element in that array:
   1. The contents of the yaml file at that path were read. Then:
      1. Every relevant non-AI correction was made to those file contents, and written back to the original file
      2. Every relevant AI correction was made to those file contents, and written back to the original file

The end result is that every yaml file in the directory inputted to the corrections tool was edited in place.

Implementing the Non-AI Corrections

As mentioned previously, a large part of what needed to happen to the Importer’s yaml files boiled down to “add a comment after a line containing a certain string” or “replace a string with another string”. Python’s String.replace() method was used extensively here. 

Implementing the AI Corrections

Prompt engineering is both science and art. It’s science, because there’s a formula that’s useful for doing prompt engineering:

1. Use the following structure:
   1. Context Setting (let the LLM know what the task is and why it’s doing it)
   2. Specific Steps (Several instructions, each about very specific individual tasks the LLM must do. Too long of a prompt is much less dangerous than too vague of a prompt)
   3. Output Instructions (Instructions about how the LLM should present its output)
2. In the context setting part of the prompt, include any relevant syntax and documentation

Prompt engineering is also art, because experimenting with the prompt and seeing how you feel about the AI’s output is a key part too. There were a few rounds of this process:

1. Start with a very basic prompt
2. See the LLM’s response to it
3. Add detail and context to the prompt to prevent any errors the LLM made

An example of a prompt we’d end up at:

I’m going to provide you with a GitHub Actions workflow yaml file. In the file an action called rtCamp/action-slack-notify@v2.2.1 is being used. Your job is to replace every rtCamp/action-slack-notify@v2.2.1 action with the slack/message-action@v1.0 action, and output the resulting workflow yaml file. For reference, I will provide you with the syntax for the slack/message-action@v1.0 action. I’ll put it between <syntax></syntax> XML tags:

<syntax>

- uses: slack/message-action@v1.0

  with:

   channel: channel-name

   text: text-message

</syntax>

Now, perform the following steps:

1. Find every instance of the rtCamp/action-slack-notify@v2.2.1 action being used.
2. Replace each instance with the slack/message-action@v1.0 action with the aforementioned syntax.
3. Replace channel-name with the value of the SLACK_CHANNEL field in the rtCamp/action-slack-notify@v2.2.1 action.
4. Replace text-message with the value of the SLACK_MESSAGE field in the rtCamp/action-slack-notify@v2.2.1 action.

Also, there are a few things you should absolutely never do:

1. Do not remove any comments from the file
2. Do not alter any other action in the file besides rtCamp/action-slack-notify@v2.2.1

I will now provide you with the workflow file you will be making edits to between <workflow></workflow> XML tags. When providing your output, only provide the contents of a YAML file. Do not describe what you did. Do not provide any justifications for your decisions. Only provide the corrected GitHub Actions yaml file, with nothing else before or after the yaml file in your output. Now, please perform the instructions I gave you:

We examined about 2 dozen of the LLM’s outputs and noticed something stunning — 100% accuracy in performing the requested tasks. A big concern with asking LLMs to do these corrections were hallucinations and other unexpected side effects, so seeing the LLM’s performance was reassuring. Overall, the LLM ended up being really useful for the migration. Even though prompt engineering isn’t trivial, it doesn’t take that long either, and after doing it the LLM could perform as well as any human for these “replace A with B” tasks. LLMs filled a crucial gap for fixing those errors that were too hard to fix with Python’s string methods.

Impact and End User Feedback

Impact

First, consider the process of converting a pipeline from Jenkins to GHA manually:

1. Looking at your Jenkins pipeline as a reference, write a GHA workflow yaml file.
2. Test that workflow yaml and do debugging as needed.
3. Write the documentation for the workflow.
4. Delete the pipeline from its Jenkins instance.

Note how the steps are in descending order of difficulty and time cost. 

The conversion tool lets end users skip step 1 entirely and have a much easier time with step 2. End users are given workflow files that have few, if any, flaws. They can skip straight to debugging, which will be accelerated since the workflows they get are highly accurate.

There are about 242 pipelines that this tool attempted to convert from Jenkins to GitHub Actions. The amount of time it would take a developer to manually convert one Jenkins pipeline to GHA varies. Here are some estimates for averages based on the type of developer:

1. Highly experienced with Jenkins and GHA: 2 hours
   1. So basically, a good chunk of a day
2. Highly experienced with Jenkins but not GHA: 5 hours
   1. These folks would need to learn GHA syntax. So most of a day
3. Not experienced with either: 10 hours
   1. This could take over a day for someone who’s brand new to Jenkins and GHA. This number includes the time it may take others to help them.

Assuming 10% of end users are type 1, 40% are type 2 and 50% are type 3, it gives a total amount of time of about 1700 hours to convert all pipelines to GHA

Suppose that with this tool, half of pipelines run with no edits necessary (this is realistic, since half of pipelines are fully converted), and that the other half may take an average of 3 hours to debug. It would then take about 360 hours to convert all pipelines to GHA.

This tool is thus projected to save over 1,300 hours, 80% of the time it would take to move every Jenkins pipeline to GHA. 

Final Comments

At time of writing, two weeks have passed since the outputs from the conversion tool have been published, and end-user feedback is now starting to arrive. It’s clear that developers at Slack have found the tool useful, but I’ve also heard that there are a few errors in the generated workflows. Even from the very start of my internship, I expected this to happen, but was worried that it might be hard to triage those incidents.

As it turns out, there wasn’t a problem. I made a Canvas doc for this conversion, and whenever end-users of the tool encountered an issue I updated the canvas with details about the problem and its solution. As more users used the tool, I got more feedback, and the canvas became more comprehensive.

If I were to pick the best thing about Slack, both the app and the company, it would be the ease of communication and collaboration. It’s easy to let people know things, to find out who to go to for help, to meet people, to take notes and keep track of info. All these things have made my internship experience more fun, pleasant, and educational than it would’ve been otherwise. And, if you join us, it’d make your time here great too 🙂.

Acknowledgments

Thank you everyone who contributed your knowledge and helped this project get to where it is:

Jerry Shen

Buddhadev Veeramallu

Ellen Wong

Shane Gearon

Sergii Gorbachov

Catherine Li

Nick Matute