Hear directly from Slack's engineers about what we build, why and how we build it, and how you can use it.
Streamlining Security Investigations with Agents
Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI…
Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI agents to optimize our working efficiency and strengthen Slack’s security defenses. This post is the first in a series that will unpack some of the design choices we’ve made and the many things we’ve learnt along the way. The Development Process The Prototype At the end of May 2025 we had a rudimentary prototype of what would grow into our service. Initially, the service was not much more than a 300 word prompt. The prompt consisted of five sections: Orientation: “You are a security analyst that investigates security alerts […]” Manifest: “You have access to the following data sources: […]” Methodology: “Your investigation should follow these steps: […] ” Formatting: “Produce a markdown report of the investigation: […]” Classification: “Choose a response classification from: […]” We implemented a simple “stdio” mode MCP server to safely expose a subset of our data sources through the tool call interface. We repurposed a coding agent CLI as an execution environment for our prototype. The performance of our prototype implementation was highly variable: sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process. We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of data sources. While we did have some success with this approach, ultimately prompts are just guidelines; they’re not an effective method for achieving fine-grained control. The Solution Our solution was to break down the complex investigation process we’d described in the prompt of our prototype into a sequence of model invocations, each with a single, well-defined purpose and output structure. These simple tasks are chained together by our application. Each task was given a structured output format. Structured output is a feature that can be used to restrict a model to using a specific output format defined by a JSON schema. The schema is applied to the last output from the model invocation. Using structured outputs isn’t “free”; if the output format is too complicated for the model, the execution can fail. Structured outputs are also subject to the usual problems of cheating and hallucination. In our initial prototype, we included guidance to “question your evidence”, but had mixed success. With our structured output approach, that guidance had become a separate task in our investigation flow with much more predictable behavior. This approach gave us more precise control at each step of the investigation process. From Prototype to Production While reviewing the literature, two papers particularly influenced our thinking: Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding (Stanford, OpenAI) Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (Microsoft Research) These papers describe prompting techniques that introduce multiple personas in the context of a single model invocation. The idea of modelling the investigation using defined personas was intriguing, but in order to maintain control we needed to represent our personas as independent model invocations. Security tabletop exercises, and how we might adapt their conventions to our application, were also a major source of inspiration during the design process. Our chosen design is built around a team of personas (agents) and the tasks they can perform in the investigation process. Each agent/task pair is modelled with a carefully defined structured output, and our application orchestrates the model invocations, propagating just the right context at each stage. Investigation Loop <img fetchpriority="high" decoding="async" class="wp-image-17657 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?w=1020" alt="Flow diagram illustrating how agents cooperate during security investigations" width="1020" height="1173" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png 2202w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=640,736 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=768,883 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1280,1472 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1336,1536 1336w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1781,2048 1781w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=380,437 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=800,920 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1160,1334 1160w" sizes="(max-width: 1020px) 100vw, 1020px" /> The Director agent poses a question and domain expert agents respond, generating findings. The Critic agent reviews findings for quality and assembles a timeline using the most credible. The Director uses the high-quality findings and timeline to determine how to progress the investigation. Our design has three defined persona categories: Director Agent The Investigation Director. The Director’s responsibility is to progress the investigation from start to finish. The Director interrogates the experts by forming a question, or set of questions, which become the expert’s prompt. The Director uses a journaling tool for planning and organizing the investigation as it progresses. Expert Agent A domain expert. Each domain expert has a unique set of domain knowledge and data sources. The experts’ responsibility is to produce findings from their data sources in response to the Director’s questions. We currently have four experts in our team: Access: Authentication, authorization and perimeter services. Cloud: Infrastructure, compute, orchestration, and networking. Code: Analysis of source code and configuration management. Threat: Threat analysis and intelligence data sources. Critic Agent The Critic is a “meta-expert”. The Critic’s responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we’ve defined. The Critic annotates the experts’ findings with its own analysis and a credibility score for each finding. The Critic’s conclusions are passed back to the Director, closing the loop. The weakly adversarial relationship between the Critic and the expert group helps to mitigate against hallucinations and variability in the interpretation of evidence. Because each agent/task pair is a separate model invocation we can vary all of the inputs, including the model version, output format, prompts, instructions, and tools. One of many ways we’re using this capability is to create a “knowledge pyramid”. Knowledge Pyramid <img decoding="async" class="aligncenter wp-image-17647 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?w=1020" alt="Pyramid diagram illustrating how investigation knowledge flows up from low to high cost models." width="1020" height="786" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png 3200w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=640,493 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=768,592 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1280,987 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1536,1184 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=2048,1579 2048w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=380,293 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=800,617 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1160,894 1160w" sizes="(max-width: 1020px) 100vw, 1020px" /> At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive. Next, the Critic’s review identifies the most interesting findings from that set. During the review process the Critic inspects the experts’ claims and the tool calls and tool results used to support them, which also incurs a significant token overhead. Once the Critic has completed its review, it assembles an up to date investigation timeline, integrating the running investigation timeline and newly gathered findings into a coherent narrative. The condensed timeline, consisting only of the most credible findings, is then passed back to the Director. This design allows us to strategically use low, medium, and high-cost models for the expert, critic, and director functions, respectively. Investigation Flow The investigation process is broken into several phases. Phases allow us to vary the structure of the investigation loop as the investigation proceeds. At the moment, we have three phases, but it is simple to add more. The Director persona is responsible for advancing the phase. <img decoding="async" class="wp-image-17666 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?w=1020" alt="Flow diagram illustrating how the Director progresses the investigation through distinct phases." width="1020" height="1564" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png 1750w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=640,982 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=768,1178 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1280,1963 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1001,1536 1001w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1335,2048 1335w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=380,583 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=800,1227 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1160,1779 1160w" sizes="(max-width: 1020px) 100vw, 1020px" /> Investigations begin in the discovery phase. After each round of investigation the Director decides whether to remain in the current phase or to progress to a new phase. Discovery The first phase of each investigation. The goal in the discovery phase is to ensure that every available data source is examined. The Director reviews the state of the investigation and generates a question that is broadcast to the entire expert team. Director Decision A “meta-phase” in which the Director decides whether to advance to the next investigation phase or continue in the current one. The task’s prompt includes advice on when to advance to each phase. Trace Once the discovery phase has made clear which experts are able to produce relevant findings, the Director transitions the investigation to the trace phase. In the trace phase, the Director chooses a specific expert to question. We also have the flexibility to vary the model invocation parameters by phase, allowing us to use a different model or enhanced token budget. Conclude The Director transitions the investigation to the concluding phase when sufficient information has been gathered to produce the final report. Service Architecture Our prototype used a coding agent CLI as an execution harness, but that wasn’t suitable for a practical implementation. We needed an interface that would let us observe investigations occurring in realtime, view and share past investigations, and launch ad-hoc investigations. Critically, we needed a way of integrating the system into our existing stack, allowing investigations to be triggered by our existing detection tools. The service architecture we created does all of these things and is quite simple. Hub The hub provides the service API and an interface to persistent storage. Besides the usual CRUD-like API, the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost. Worker Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed. Dashboard The Dashboard is used by staff to interact with the service. Running investigations can be observed in real-time, consuming the event stream from the hub. Additionally the dashboard provides management tools, letting us view the details of each model invocation. This capability is invaluable when debugging the system. Example Report We’ve included an edited investigation report which demonstrates the potential of the agents to exhibit novel emergent behavior. In this case, the original alert was raised for a specific command sequence, which we analyze because it can be an indicator of compromise. In the course of investigating the alert, the agents independently discovered a separate credential exposure elsewhere in the process ancestry. <img loading="lazy" decoding="async" class="wp-image-17650 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?w=1020" alt="Tree diagram illustrating how agents navigated the process tree." width="1020" height="746" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png 1317w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=640,468 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=768,562 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=1280,936 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=380,278 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=800,585 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=1160,848 1160w" sizes="auto, (max-width: 1020px) 100vw, 1020px" /> The highlighted leaf process triggered the investigation, but the agents traced the process hierarchy and discovered a different issue in an ancestor process. The text below is a lightly edited version of the report summary from this investigation. Investigation Report: Credential Exposure in Monitoring Workflow [ESCALATE] Summary: While investigating [command sequence], the investigation uncovered a credential exposure elsewhere in the process ancestry chain. Analysis The investigation confirmed that the command execution on [TIMESTAMP] was part of a legitimate monitoring workflow using [diagnostic tool]. The process ancestry shows the expected execution chain. However, critical security concerns were identified: Credential Exposure: A credential was exposed in process command line parameters within the ancestry chain, creating significant security risk. Expert-Critic Contradiction: The expert incorrectly assessed credential handling as secure while the critic correctly identified exposed credentials, indicating analysis blind spots that require attention. What is notable about this result is that the expert did not raise the credential exposure in its findings; the Critic noticed it as part of its meta-analysis of the expert’s work. The Director then chose to pivot the investigation to focus on this issue instead. In the report, the Director highlights both the need to mitigate the security issue, and to follow-up on the expert’s failure to properly identify the risk. We referred the credential exposure to the service owning team to resolve. Conclusion We’re still at an early phase of our journey to streamline security investigations using AI agents, but we’re starting to see meaningful benefits. Our web-based dashboard allows us to launch and watch investigations in real time, and investigations yield interactive, verifiable reports that show how evidence was collected, interpreted, and judged. During our on-call shifts, we’re switching to supervising investigation teams, rather than doing the laborious work of gathering evidence. Unlike static detection rules, our agents often make spontaneous and unprompted discoveries, as we demonstrated in our example report. We’ve seen this occur many times, from highlighting weakness in IAM policies, to identifying problematic code and more. There’s a great deal more to say. We look forward to sharing more details of how our system works in future blog posts. As a preview of some future content from the series: Maintaining alignment and orientation during multi-persona investigations Using artifacts as a communication channel between investigation participants Human in the loop: human / agent collaboration in security investigations Acknowledgements We wanted to give a shout out to all the people that have contributed to this journey: Chris Smith Abhi Rathod Dave Russell Nate Reeves <path stroke="#032d60" stroke-linejoin="round" stroke-width="5" d="m4.112 1c-2.5 6.167-2.4 21.1 18 31.5"/><path stroke="#032d60" stroke-width="5" d="m20.112 18 2.5 14.5-13.5 1.5"/> Interested in taking on interesting projects, making people’s work lives easier, or just building some pretty cool forms? We’re hiring! Apply now
Build better software to build software better
We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until…
We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until far too late. We fixed this problem by combining modern, high-performance build tooling (Bazel) with classic software engineering principles. Here’s how we did it. Thinking About Build (and Code) Performance Imagine a simple application. It has a backend server that provides an API and data storage, and a frontend that presents the user interface. Like many modern applications, the frontend and backend are decoupled; they can be developed and delivered independently. The graph of this build looks like this: <img loading="lazy" decoding="async" width="1642" height="430" class="alignnone size-full wp-image-17605" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png" alt="An example service build graph. It shows a set of Python files building into a backend artifact and a set of TypeScript files building into a frontend artifact." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png 1642w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=640,168 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=768,201 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1280,335 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1536,402 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=380,100 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=800,210 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1160,304 1160w" sizes="auto, (max-width: 1642px) 100vw, 1642px" /> We represent the dependencies between the elements of the build, like source files and deployable artifacts, with arrows, forming a directed acyclic graph. Here, our backend depends on a collection of Python files, meaning that whenever a Python file changes, we need to rebuild the backend. Likewise, we need to rebuild our frontend whenever a TypeScript file changes — but not when a Python file does. Modeling our build as a graph of clearly-defined units of work lets us apply the same kind of performance optimizations we might use to speed up application code: Do less work. Store the results of expensive work you do so that you only have to do it once, trading off memory for time. Share the load. Spread the work you’re doing across more compute resources in parallel, so that it completes more quickly, trading off compute for time. Caching and Parallelization These techniques will look familiar to most engineers, and thinking about them through a code lens helps solidify how they apply to a build system. Take this Python example: def factorial(n): return n * factorial(n-1) if n else 1 Calculating a factorial can be very expensive. If we need to do it often, we’re going to run quite slowly. But intuitively, we know that the factorial of a number doesn’t change. This means we can apply strategy 1 to cache the results of this function. Then it only needs to run once for any given input. @functools.cache def factorial(n): return n * factorial(n-1) if n else 1 The cache stores our inputs (n) and maps them to the corresponding outputs (the return value of factorial()). n is the cache key: it functions like a source file in the build graph, with the output of the function corresponding to the built artifact. Refining our intuition, we can say that this function needs to have a few specific attributes for caching to work. It needs to be hermetic: it only uses the inputs it is explicitly given to produce a specific output value. And it needs to be idempotent: the output for any given set of inputs is always the same. Otherwise, caching is unsound and will have rather surprising effects. Once we have a cache in place, we want to maximize our hit rate: the fraction of times we call factorial() that are answered from the cache, rather than from doing the calculation. The way we define our units of work can help us keep the hit rate high. To illustrate that idea, let’s look at a more complex piece of code: @functools.cache def process_images( images: list[Image], transforms: list[Transform] ) -> list[Image]: .. This function has a cache attached to it, but the cache isn’t very effective. The function takes two collections of inputs: a list of images and a list of transforms to apply to each image. If the caller changes any of those inputs, say by adding one more Image to the list, they also change the cache key. That means we won’t find a result in the cache, and will have to do all of the work from the beginning. In other words, this cache is not very granular, and as a result has a poor hit rate. More effective code might look like this: def process_images( images: list[Image], transforms: list[Transform] ) -> list[Image]: new_images = [] for image in images: new_image = image for transform in transforms: new_image = process_image(new_image, transform) new_images.append(new_image) return new_images @functools.cache def process_image(image: Image, transform: Transform) -> Image: ... We move the caching to a smaller unit of work — the application of a single transform to a single image — and with a correspondingly smaller cache key. We can still retain the higher-level API; we just don’t cache there. When a caller invokes the higher-level API with a set of inputs, we can answer that request by only processing the combinations of Image and Transform that we haven’t seen before. Everything else comes from the cache. Our cache hit rate should both improve, along with our performance. Caching helps us do less work. Now let’s look at sharing the load. What if we fanned out image processing across multiple CPU cores with threads? def process_images_threaded( images: list[Image], transforms: list[Transform] ) -> list[Image]: with ThreadPoolExecutor() as executor: futures = [] for image in images: futures.append(executor.submit(process_images, [image], transforms)) # Returns images in any order! return [future.result() for future in futures.as_completed(futures)] Let’s suss out the qualities we need to be able to run our work in parallel, while also noting some key caveats. Like with caching, we need to rigorously and completely define the inputs and outputs for the work we want to parallelize. We need to be able to move those inputs and outputs across some type of boundary. Here, it’s a thread boundary, but it might as well be a process boundary or a network boundary to some other compute node. Units of work that we run in parallel might complete, or fail, in any order. Our code has to manage those outcomes, and clearly document what guarantees it offers. Here, we’re noting the caveat that images may not return in the same order. Whether or not that’s acceptable depends on the API. The granularity of our units of work also plays a role in how effectively we can parallelize, although there’s a lot of nuance involved. If our units of work are few in number but large in size, we won’t be able to spread them over as many compute resources. The right trade-off between task size and count varies wildly between problems, but it’s something we need to consider as we design our APIs. Transitioning to Build Performance If you’re a software engineer, you’ve probably applied those techniques many times to improve your code’s performance. Let’s apply some of those principles and intuitions to build systems. In a Bazel build, you define targets that form a directed acyclic graph. Each target has three critical elements: The files that are dependencies, or inputs, for this build step. The files that are outputs for this build step. The commands that transform the inputs into the outputs. (It’s a little more complicated, but that’s enough to get the gist for now). Let’s go back to that original sample application. <img loading="lazy" decoding="async" width="1642" height="430" class="alignnone size-full wp-image-17605" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png" alt="An example service build graph. It shows a set of Python files building into a backend artifact and a set of TypeScript files building into a frontend artifact." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png 1642w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=640,168 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=768,201 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1280,335 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1536,402 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=380,100 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=800,210 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1160,304 1160w" sizes="auto, (max-width: 1642px) 100vw, 1642px" /> The Bazel target definitions might look something like this pseudocode: python_build( name = "backend", srcs = ["core/http.py", "lib/options.py", "data/access.py"], outs = ["backend.tgz"], cmd = "python build.py", ) ts_build( name = "frontend", srcs = ["cms/cms.ts", "collab/bridge.ts", "editing/find.ts"], outs = ["frontend.tgz"], cmd = "npm build" ) You can think about a build target like defining (but not invoking!) a function. Notice some themes carrying over from the Python examples above? We exhaustively define the inputs (srcs) and outputs (outs) for each build step. We can think of the srcs of a target (and the transitive srcs of any targets that build them, which we’ll see more later) as the inputs to a function we wish to cache. Because Bazel builds in a sandbox, the cmds that transform inputs to outputs only get to touch the input files we declare. And because our inputs and outputs are simply on-disk files, we solve the problem of moving them across a boundary: they just get copied! Lastly, we have to make Bazel a promise: our build steps’ cmds are in fact idempotent and hermetic. When we do those things, we get some extraordinary capabilities for free. Bazel automatically caches the outcomes of our build actions. When the inputs to a target don’t change, the cached output is used — no build cost! Bazel distributes build actions across as many CPU cores as we allow, or even across multiple machines in a build cluster. Bazel only executes those actions we need for the output artifacts we want. To put it another way, we always run the least amount of work needed for the output we want. Just like in our code examples, we’ll get the most out of Bazel’s magic when we have a well-defined dependency graph, composed of build units that are idempotent, hermetic, and granular. Those qualities allow Bazel’s native caching and parallelization to deliver great speed boosts. That’s plenty of theory. Let’s drill down on the actual problem. Why Quip and Canvas are Harder Here’s the trick: Quip and Canvas are much, much more complex than the simple example we looked at before. Here’s a real diagram we drew to understand how our build worked. Don’t worry about reading all the details — we’ll use a schematic version as we dig into the problem and our solutions. When we analyzed the graph, we discovered some critical flaws that meant we did not have the characteristics we needed to get a speed boost from Bazel: A directed acyclic dependency graph →The graph wasn’t well defined, and in fact contained cycles! Idempotent, hermetic, well-sized units of work. → Build execution units were huge, not all were idempotent, and hermeticity was a challenge because many build steps mutated the working directory. Granular cache keys to keep cache hit rate high. → Our build was so interconnected that our cache hit rate was zero. Imagine that every cached “function” we tried to call had 100 parameters, 2-3 of which always changed. If we had started by throwing Bazel at the build, it would have been ineffective. With a cache hit rate of zero, Bazel’s advanced cache management wouldn’t have helped, and Bazel parallelization would have added little to nothing over the ad-hoc parallelization already present in the build code. To get the Bazel magic, we needed to do some engineering work first. Separating Concerns Our backend code and our build code were deeply intertwined. Without a Bazel-like build system, we’d defaulted to using in-application frameworks to orchestrate the various steps within our build graph. We managed parallelization using Python’s multiprocessing strategy and the async routines built into our core codebase. Python business logic pulled together the Protobuf compilation and built Python and Cython artifacts. Lastly, tools like tsc and webpack, orchestrated by more Python scripts, transformed TypeScript and Less into the unique frontend bundle format used across Slack Canvas and Quip’s desktop and web applications. As we began to unpick that Gordian knot, we zeroed in on how the union of backend and build code distorted the graph of our frontend build. Here’s an easier-to-digest representation of that graph. <img loading="lazy" decoding="async" class="alignnone wp-image-17595 size-full" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png" alt="A graph of a software build. It shows a group of Python files at the top, feeding into a Python application build. The Python build and a set of TypeScript files are then the inputs to a TypeScript build, producing a set of frontend bundles." width="1475" height="906" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png 1475w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=640,393 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=768,472 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=1280,786 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=380,233 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=800,491 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=1160,713 1160w" sizes="auto, (max-width: 1475px) 100vw, 1475px" /> Notice how large the dependency tree is “above” our frontend bundles. It includes not just their TypeScript source and build process, but also the entire built Python backend! Those Python sources and artifacts are transitive sources for each and every frontend bundle. That means that not just a TypeScript change, but also every single Python change, alters the cache key for the frontend (one of those hundred parameters we alluded to earlier), requiring an expensive full rebuild. The key to this build problem is that dependency edge between the Python application and the TypeScript build. <img loading="lazy" decoding="async" width="554" height="525" class="alignnone size-full wp-image-17600" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png" alt="A zoomed-in view of the build graph from above. It focuses on the edges between the Python build and application and the TypeScript build." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png 554w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png?resize=380,360 380w" sizes="auto, (max-width: 554px) 100vw, 554px" /> That edge was costing us an average of 35 minutes per build — more than half the total cost! — because every change was causing a full backend and frontend rebuild, and the frontend rebuild was especially expensive. As we worked through this problem, we realized that the performance cost was not solely a build issue. It was a symptom of our pervasive failure to separate our concerns across the whole application: backend, frontend, and build code. We had couplings: between our backend and our frontend between our Python and TypeScript infrastructures and toolchains between our build system and our application code Beyond making the build performance quantitatively worse, those couplings had qualitative impacts across our software development lifecycle. Engineers couldn’t reason about the blast radius of changes they made to the backend, because it might break the frontend too. Or the build system. Or both. And because our build took an hour, we couldn’t give them early warnings by running builds at the Pull Request level. The lack of health signals, and the difficulty of reasoning through potential consequences, meant that we’d break our build and our main branch far too often, and through no fault of our engineers. Once we understood the problem through the lens of separation of concerns, it became clear we couldn’t succeed with changes to the build system alone. We had to cleanly sever the dependencies between our frontend and backend, our Python and TypeScript, our application and our build. And that meant we had to invest a lot more time than we’d originally planned. Over several months, we painstakingly unraveled the actual requirements of each of our build steps. We rewrote Python build orchestration code in Starlark, the language Bazel uses for build definitions. Starlark is a deliberately constrained language whose limitations aim at ensuring builds meet all the requirements for Bazel to be effective. Building in Starlark helped us enforce a full separation from application code. Where we needed to retain Python scripts, we rewrote them to remove all dependencies save the Python standard library: no links to our backend code, and no additional build dependencies. We left out all of the parallelization code, because Bazel handles that for us. We’ll revisit that paradigm below in thinking about layering. The complexity of the original build code made it challenging to define “correct” behavior. Our build code mostly did not have tests. The only criterion for what was correct was what the existing build system produced under a specific configuration. To reassure ourselves, and to instill confidence in our engineers, we built a tool in Rust to compare an artifact produced by the existing process with one produced by our new code. We used the differences to guide us to points where our new logic wasn’t quite right, and iterated, and iterated more. That work paid off when we could finally draw a new build graph: <img loading="lazy" decoding="async" width="1843" height="947" class="alignnone size-full wp-image-17596" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png" alt="A software build graph. A set of Python source files feed into a Python application build. This build is shown separately from a frontend build, where a set of TypeScript files is shown to be built into frontend bundles." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png 1843w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=640,329 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=768,395 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1280,658 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1536,789 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=380,195 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=800,411 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1160,596 1160w" sizes="auto, (max-width: 1843px) 100vw, 1843px" /> We’d severed all three of those key couplings, and taken off the table any concern that a Python change might break the build or alter the frontend output. Our build logic was colocated with the units it built in BUILD.bazel files, with well-defined Starlark APIs and a clean separation between build code and application code. Our cache hit rate went way up, because Python changes no longer formed part of the cache key for TypeScript builds. The outcome of this work was a substantial reduction in build time: if the frontend was cached, we could build the whole application in as little as 25 minutes. That’s a big improvement, but still not enough! Designing for Composition with Layering Once we severed the backend ←→ frontend coupling, we took a closer look at the frontend build. Why did it take 35 minutes, anyway? As we unraveled the build scripts, we found more separation-of-concerns challenges. Our frontend builder was trying hard to be performant. It took in a big set of inputs (TypeScript source, LESS and CSS source, and a variety of knobs and switches controlled by environment variables and command-line options), calculated the build activity it needed to do, parallelized that activity across a collection of worker processes, and marshalled the output into deployable JavaScript bundles and CSS resources. Here’s a sketch of this part of the build graph. <img loading="lazy" decoding="async" width="1482" height="746" class="alignnone size-full wp-image-17602" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png" alt="A build graph shows TypeScript and LESS source files feeding into a single TypeScript and CSS Build node, along with environment variables and switches. The output of the build node is a set of frontend bundles. The build process interacts with a set of worker processes." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png 1482w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=640,322 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=768,387 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=1280,644 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=380,191 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=800,403 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=1160,584 1160w" sizes="auto, (max-width: 1482px) 100vw, 1482px" /> Like much of what we discovered during this project, this strategy represented a set of reasonable trade-offs when it was written: it was a pragmatic attempt to speed up one piece of the build in the absence of a larger build framework. And it was faster than not parallelizing the same work! There are two key challenges with this implementation. Like our process_image() example above, our cacheable units were too large. We took in all the sources and produced all the bundles. What if just one input file changed? That altered the cache key, and we had to rebuild everything. Or what if we wanted to build exactly one bundle, to satisfy a requirement elsewhere in our build process? We were out of luck; we had to depend on the whole shebang. We’re parallelizing the work across processes, which is good — unless we could be parallelizing across machines, machines with lots more CPU cores. If we have those resources available, we cannot use them here. And we’re actually making Bazel less effective at its core function of parallelizing independent build steps: Bazel and the script’s worker processes are contending for the same set of resources. The script might even be parallelizing work that Bazel already knows it does not need! In both these respects, it was tough for us to compose the build functionality in new ways: either a different form of orchestrating multiple bundle builds, or a different parallelization strategy. This was a layering violation. <img loading="lazy" decoding="async" width="618" height="602" class="alignnone size-full wp-image-17599" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png" alt="A diagram of layered functionality. At the base is the OS, followed by Parallelization and Language Runtime, then Orchestration and App Core, and finally Logic. A red outline cuts out a unit of logic plus the orchestration and parallelization layers." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png 618w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png?resize=380,370 380w" sizes="auto, (max-width: 618px) 100vw, 618px" /> If we draw, as here, layers of capability from the operating system up to the application layer, we can see that the boundary of our builder is cutting across layers. It combines business logic, significant parts of a task orchestration framework, and parallelization. We really just wanted the top layer — the logic — so that we could re-compose it in a new orchestration context. We ended up with a work orchestrator (the builder) inside a work orchestrator (Bazel), and the two layers contending for slices of the same resource pool. To be more effective, we really just needed to do less. We deleted a lot of code. The new version of the frontend builder was much, much simpler. It didn’t parallelize. It had a much smaller “API” interface. It took in one set of source files, and built one output bundle, with TypeScript and CSS processed independently. <img loading="lazy" decoding="async" width="714" height="784" class="alignnone size-full wp-image-17603" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png" alt="A build graph shows parallel tracks, with TypeScript source flowing through a build to produce JavaScript source, and Less and CSS source flowing through a CSS build to form CSS output. The outputs are then combined into a frontend bundle." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png 714w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png?resize=640,703 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png?resize=380,417 380w" sizes="auto, (max-width: 714px) 100vw, 714px" /> This new builder is highly cacheable and highly parallelizable. Each output artifact can be cached independently, keyed only on its direct inputs. A bundle’s TypeScript build and CSS build can run in parallel, both against one another and against the builds of other bundles. And our Bazel logic can make decisions about scope (one bundle? two? all?), rather than trying to manage a build of all of the bundles. See the resonance, again, with our example process_images() API? We’ve created granular, composable units of work, and that’s dramatically improved our ability to parallelize and cache. We’ve also separated the concerns of our business logic and its orchestration, making it possible for us to re-compose the logic within our new Bazel build. This change gave us some really nice outcomes: Because bundle builds and TypeScript and CSS builds can be cached independently of one another, our cache hit rate went up. Given enough resources, Bazel could parallelize all of the bundle builds and CSS compilation steps simultaneously. That bought us a nice reduction in full-rebuild time. As an added benefit, we’re no longer running our own parallelization code. We’ve delegated that responsibility to Bazel. Our build script has a single concern: the business logic around compiling a frontend bundle. That’s a win for maintainability. Outcomes and Takeaways We made our build a lot faster. Faster builds lead to tighter cycle times for engineers, quicker incident resolutions, and more frequent releases. After applying these principles across our whole build graph, we came away with a build that’s as much as six times faster than when we started. <img loading="lazy" decoding="async" width="1133" height="717" class="alignnone size-full wp-image-17597" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png" alt="A diagram shows a "before" build time of 60 minutes for all cases. The "after" time shows three cases, 10 minutes in the best case (cached and parallelized); 12 minutes in the average case (mostly cached and parallelized); and 30 minutes in the worst cases (a cache miss)." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png 1133w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=640,405 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=768,486 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=380,240 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=800,506 800w" sizes="auto, (max-width: 1133px) 100vw, 1133px" /> From a qualitative angle, we took away the key principle that software engineering principles apply to the whole system. And the whole system is more than our application code. It’s also our build code, our release pipeline, the setup strategies for our developer and production environments, and the interrelations between those components. So here’s our pitch to you, whether you’re writing application code, build code, release code, or all of the above: separate concerns. Think about the whole system. Design for composability. When you do, every facet of your application gets stronger — and, as a happy side effect, your build will run a lot faster too.
Advancing Our Chef Infrastructure: Safety Without Disruption
Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating…
Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating the limitations around Chef searches. If you haven’t had a chance to read that post yet, I highly recommend checking it out first to get the full context for this post. At Slack, keeping our service reliable is always the top priority. In my last post, I talked about the first phase of our work to make Chef and EC2 provisioning safer. With that behind us, we started looking at what else we could do to make deploys even safer and more reliable. One idea we explored was moving to Chef Policyfiles. That would have meant replacing roles and environments and asking dozens of teams to change their cookbooks. In the long run, it might have made things safer, but in the short term it would have been a huge effort and added more risk than it solved. So instead, this post is about the path we chose: improving our existing EC2 framework in a way that doesn’t disrupt cookbooks or roles, while still giving us more safety in our Chef deployments. Splitting Chef Environments Previously, each instance had a cron job that triggered a Chef run every few hours on a set schedule. These scheduled runs were primarily for compliance purposes — to ensure our fleet remained in a consistent and defined configuration state. To reduce risk, the timing of these cron jobs was staggered across availability zones, helping us avoid running Chef on all nodes simultaneously. This strategy gave us a buffer: if a bad change was introduced, it would only impact a subset of nodes initially, giving us a chance to catch and fix the issue before it spread. However, this approach had a critical limitation. With a single shared production environment, even if Chef wasn’t running everywhere at once, any newly provisioned nodes would immediately pick up the latest (possibly bad) changes from that shared environment. This became a significant reliability risk, especially during large scale-out events, where dozens or hundreds of nodes could start up with a broken configuration. To address this, we split our single production Chef environment into multiple buckets: prod-1, prod-2, …, prod-6. Service teams could still launch instances as “prod,” but behind the scenes we mapped each instance to one of these more specific environments based on its Availability Zone. That way, new nodes no longer pulled from one global source of truth – they were evenly distributed across isolated environments, which added an extra layer of safety and resilience. At Slack, our base AMIs include a tool called Poptart Bootstrap, which is baked in during the AMI build process. This tool runs via cloud-init at instance boot time and is responsible for tasks such as creating the node’s Chef object, setting up any required DNS entries, and posting a success or failure message to Slack (with customisable channels based on the team that owns the node). To support the environment split, we extended Poptart Bootstrap to include logic that inspects the node’s AZ ID and assigns it to one of the numbered production Chef environments. This change allowed us to stop pointing all production nodes at a single shared Chef environment and instead spread them across multiple environments. Previously, changing the cookbook versions in the single prod environment affected the entire fleet. With this new bucketed approach, we gained the flexibility to update each environment independently. Now, a change to a single environment only impacts nodes in specific AZs, significantly reducing risk and blast radius during deployments. <img loading="lazy" decoding="async" width="438" height="597" class="alignnone size-medium wp-image-17612" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png?w=438" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png 438w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png?resize=380,518 380w" sizes="auto, (max-width: 438px) 100vw, 438px" /> With this approach, we promote the latest cookbook changes to the sandbox environments at the top of the hour. These changes are then managed by a Kubernetes cron job and rolled out to the dev environments at the top of the hour, and finally begin rolling out to production environments starting at 30 minutes past the hour. prod-1 is treated as our canary production environment. It receives updates every hour (assuming there have been new changes merged into the cookbooks) to ensure that changes are regularly exercised in a real production setting – alongside our sandbox and dev environments. This enables us to catch production-impacting issues early and in smaller, safer increments. Changes to prod-2 through prod-6 are rolled out using a release train model. Before a new version is deployed to prod-2, we ensure the previous version has successfully progressed through all production environments up to prod-6. This staggered rollout minimises blast radius and gives us the opportunity to catch regressions earlier in the release train. If we waited for a version to pass through all environments before updating prod-1 – as we do with prod-2 onward – we’d end up testing artifacts with large, cumulative changes. By contrast, updating prod-1 frequently with the latest version allows us to detect issues closer to when they were introduced. Here’s an example showing how artifact version A moves through each environment in this release cycle: Hour X:00 Sandbox ---> Version A (Latest) Dev ---> Version A (Latest) Prod-1 (no change - currently at version Z) Prod-2 (no change - currently at version Z) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour X:30 (Now that Prod-2 to Prod-6 are all on the same version, we’ll begin updating them again) Sandbox (no change - currently at version A) Dev (no change - currently at version A) Prod-1 ---> Version A (dev version) Prod-2 ---> Version A (prod-1 version) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called B. Hour (X + 1):00 Sandbox ---> Version B (Latest) Dev ---> Version B (Latest) Prod-1 (no change - currently at version A) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 1):30 Sandbox (no change - currently at version B) Dev (no change - currently at version B) Prod-1 ---> Version B (dev version) Prod-2 (no change - currently at version A) Prod-3 ---> Version A (prod-2 version) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called C. Hour (X + 2):00 Sandbox ---> Version C (Latest) Dev ---> Version C (Latest) Prod-1 (no change - currently at version B) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 2):30 Sandbox (no change - currently at version C) Dev (no change - currently at version C) Prod-1 ---> Version C (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 ---> Version A (prod-3 version) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called D. Hour (X + 3):00 Sandbox ---> Version D (Latest) Dev ---> Version D (Latest) Prod-1 (no change - currently at version C) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 3):30 Sandbox (no change - currently at version D) Dev (no change - currently at version D) Prod-1 ---> Version D (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 ---> Version A (prod-4 version) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called E. Hour (X + 4):00 Sandbox ---> Version E (Latest) Dev ---> Version E (Latest) Prod-1 (no change - currently at version D) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version Z) Hour (X + 4):30 Sandbox (no change - currently at version E) Dev (no change - currently at version E) Prod-1 ---> Version E (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 ---> Version A (prod-5 version) Someone committed a change, resulting in the creation of a new artifact called F. Hour (X + 5):00 Sandbox ---> Version F (Latest) Dev ---> Version F (Latest) Prod-1 (no change - currently at version E) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version A) Now that Prod-2 to Prod-6 are all on the same version, we’ll begin updating them again and this cycle will continue till the end of time. Hour (X + 5):30 Sandbox (no change - currently at version F) Dev (no change - currently at version F) Prod-1 ---> Version F (dev version) Prod-2 ---> Version F (prod-1 version) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version A) As you can see above, changes now take a bit longer to roll out across all our production nodes. However, this delay provides valuable time between deployments across different availability zones, allowing us to catch and address any issues before problematic changes are fully propagated. By introducing split production environments (e.g., prod-1 to prod-6), we’ve been able to sidestep the old problem of new nodes pulling in a bad config the moment they came online. Now, each node is tied to the environment for its AZ, and changes roll out gradually instead of everywhere at once. That means if something goes wrong in one AZ, we can keep scaling safely in the others while we fix the issue. It’s a much more resilient setup than before – we’ve taken away the single point of failure and built in guardrails that make the whole system safer and more predictable. What changed in the way we trigger Chef? Now that we have multiple production Chef environments, each one receives updates at different times. For example, if a version is currently in the middle of being rolled out, the next version must wait until that rollout is fully completed across all production environments. On the other hand, when there are no updates in progress, a new version can be rolled out to all environments more quickly. Because of this variability, triggering Chef via a fixed cron schedule is no longer practical – we can’t reliably predict when a given Chef environment will receive new changes. As a result, we’ve moved away from scheduled runs and instead built a new service that triggers Chef runs on nodes based on signals. This service ensures Chef only runs when there are actual updates available, improving both safety and efficiency. Chef Summoner <img loading="lazy" decoding="async" width="768" height="768" class="alignnone size-medium wp-image-17614" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?w=640" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=160,160 160w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=640,640 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=110,110 110w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=380,380 380w" sizes="auto, (max-width: 768px) 100vw, 768px" /> If you remember from my previous blog post, we built a service called Chef Librarian that watches for new Chef cookbook artifacts and uploads them to all of our Chef stacks. It also exposes an API endpoint that allows us to promote a specific version of an artifact to a given environment. We recently enhanced Chef Librarian to send a message to an S3 bucket whenever it promotes a version of an artifact to an environment. The contents of that S3 bucket look like this: agunasekara@z-ops-agunasekara-iad-dinosaur:~ >> s3-tree BUCKET_NAME BUCKET_NAME └── chef-run-triggers ├── basalt │ ├── ami-dev │ ├── ami-prod │ ├── dev │ ├── prod │ ├── prod-1 │ ├── prod-2 │ ├── prod-3 │ ├── prod-4 │ ├── prod-5 │ ├── prod-6 │ └── sandbox └── ironstone ├── ami-dev ├── ami-prod ├── dev ├── prod ├── prod-1 ├── prod-2 ├── prod-3 ├── prod-4 ├── prod-5 ├── prod-6 └── sandbox Under the chef-run-triggers key, we maintain a nested structure where each key represents one of our Chef stacks. Within each stack key, there are additional keys for each environment name. Each of these environment keys contains a JSON object with contents similar to the example below: { "Splay": 15, "Timestamp": "2025-07-28T02:02:31.054989714Z", "ManifestRecord": { "version": "20250728.1753666491.0", "chef_shard": "basalt", "datetime": 1753666611, "latest_commit_hash": "XXXXXXXXXXXXXX", "manifest_content": { "base_version": "20250728.1753666491.0", "latest_commit_hash": "XXXXXXXXXXXXXX", "author": "Archie Gunasekara <agunasekara@slack-corp.com>", "cookbook_versions": { "apt": "7.5.23", ... "aws": "9.2.1" }, "site_cookbook_versions": { "apache2": "20250728.1753666491.0", ... "squid": "20250728.1753666491.0" } }, "s3_bucket": "BUCKET_NAME", "s3_key": "20250728.1753666491.0.tar.gz", "ttl": 1756085811, "upload_complete": true } } Next, we built a service called Chef Summoner. This service runs on every node at Slack and is responsible for checking the S3 key corresponding to the node’s Chef stack and environment. If a new version is present in the signal sent out by Chef Librarian, the service reads the configured splay value and schedules a Chef run accordingly. The splay is used to stagger Chef runs so that not all nodes in a given environment and stack try to run Chef at the same time. This helps avoid spikes in load and resource contention. We can also customize the splay depending on our needs – for example, when we trigger a Chef run using a custom signal from Librarian and want to spread the runs out more intentionally. However, if no changes are merged and no new Chef artifacts are built, Librarian has nothing to promote, and no new signals are sent out. Despite that, we still need to ensure Chef runs at least once every 12 hours to maintain compliance and ensure nodes stay in their expected configuration state. So, Chef Summoner will also run Chef at least once every 12 hours, even if no new signals have been received. The Summoner service keeps track of its own state locally – this includes things like the last run time and the artifact version used – so it can compare any new signals with the most recent run and determine whether a new Chef run is required. The overall flow looks like this: <img loading="lazy" decoding="async" width="630" height="1052" class="alignnone size-medium wp-image-17616" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png?w=630" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png 630w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png?resize=380,635 380w" sizes="auto, (max-width: 630px) 100vw, 630px" /> Now that Chef Summoner is the primary mechanism we rely on to trigger Chef runs, it becomes a critical piece of infrastructure. After a node is provisioned, subsequent Chef runs are responsible for keeping Chef Summoner itself up to date with the latest changes. But if we accidentally roll out a broken version of Chef Summoner, it may stop triggering Chef runs altogether – making it impossible to roll out a fixed version using our normal deployment flow. To mitigate this, we bake in a fallback cron job on every node. This cron job checks the local state that Chef Summoner stores (e.g., last run time and artifact version) and ensures Chef has been run at least once every 12 hours. If the cron job detects that Chef Summoner has failed to run Chef in that timeframe, it will trigger a Chef run directly. This gives us a recovery path to push a working version of Chef Summoner back out. In addition to this safety net, we also have tooling that allows us to trigger ad hoc Chef runs across the fleet or a subset of nodes when needed. What’s Next? All of these recent changes to our EC2 ecosystem have made rolling out infrastructure changes significantly safer. Teams no longer have to worry about a bad update impacting the entire fleet. While we’ve come a long way, the platform is still not perfect. One major limitation is that we still can’t easily support service-level deployments. In theory, we could create a dedicated set of Chef environments for each service and promote artifacts individually – but with the hundreds of services we operate at Slack, this quickly becomes unmanageable at scale. With that in mind, we’ve decided to mark our legacy EC2 platform as feature complete and move it into maintenance mode. In its place, we’re building a brand-new EC2 ecosystem called Shipyard, designed specifically for teams that can’t yet move to our container-based platform, Bedrock. Shipyard isn’t just an iteration of our old system – it’s a complete reimagining of how EC2-based services should work. It introduces concepts like service-level deployments, metric-driven rollouts, and fully automated rollbacks when things go wrong. We’re currently building a Shipyard and targeting a soft launch this quarter, with plans to onboard our first two teams for testing and feedback. I’m excited to share more about Shipyard in my next blog post – stay tuned!
Deploy Safety: Reducing customer impact from change
It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing…
It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing our rate of change to meet business needs. Defining the problem System requirements change as businesses evolve whether that be due to changes in customer expectations, load & scale or response to business needs. In this case analysis showed two critical trends: Slack has become more mission critical for our customers, increasing expectations of reliability for the product as a whole and for specific features each company relies on. The increasing majority (73%) of customer facing incidents were triggered by Slack-induced change, particularly code deploys. We use our incident management process heavily and frequently at Slack to ensure we have a strong and rapid response to issues of all shapes and sizes based on impact and severity. A portion of those incidents are customer impacting, many of which showed variable impact depending on which features they used in the product. Change-triggered incidents were occurring in a wide variety of systems and deployment processes. Finally, we also received customer feedback that interruptions became more disruptive after about 10 minutes – something they would treat as a “blip” – and this would continue to reduce with the introduction of Agentforce in 2025. All this was occurring in a software engineering environment with 100s of internal services and many different deployment systems & practices. In the past, our approach to reliability often zeroed in on individual deploy systems or services. This led to manual change processes that bogged down our pace of innovation. The increased time and effort required for changes not only dampened engineering morale but also hindered our ability to swiftly meet business objectives. This led us to set some initial “North Star” goals across all deployment methods for our highest importance services: Reducing impact time from deployments Automated detection & remediation within 10 minutes Manual detection & remediation within 20 minutes Reducing severity of impact Detect problematic deployments prior to reaching 10% of the fleet Maintaining Slack’s development velocity The initial North Star goals evolved and expanded into a Deploy Safety Manifesto that now applies to all Slack’s deployment systems & processes. These goals encourage implementing automated system improvements, safety guardrails and cultural changes. The metric Programs are run to invoke change and change must be measured. Selecting a metric that serves as an imperfect analog has continued to be a constant source of discussion. The North Star goals could be measured against engineering efforts, but were not a direct measure of the end goal: improving customer sentiment of their experience of Slack’s reliability. What we needed was a metric that was a reasonable (though imperfect) analog of customer sentiment. We had the internal data to design such a metric. The Deploy Safety metric: Hours of customer impact from high severity and selected medium severity change-triggered incidents. What does “selected” mean? That seems pretty ambiguous. We found that the incident dataset didn’t exactly match what we needed. Severity levels at Slack convey current or impending customer impact rather than the final impact which often requires careful post-hoc analysis. This means we need to filter medium severity incidents based on a relevant level of customer impact. The on-going discussion around what metric to use relates to the semi-loose connection between: Customer sentiment <-> Program Metric <-> Project Metric They’re all connected, but it’s challenging to know for a specific project how much it is going to move the top line metric. How much does the Program metric reflect individual customer experience? This flow is especially difficult for engineers who prefer a more concrete feedback loop with hard data – i.e., “How much does my work or concept change customer sentiment?” Important criteria for designing the Deploy Safety metric: Measure results Understand what is measured (real vs analog) Consistency in measurement, especially subjective portions Continually validate the measurement matches customer sentiment with the leaders having the direct conversations with customers Which projects to invest in? The two largest unknowns we had at the start of the program were predicting which projects would have the highest likelihood of success given multiple sources of incidents and when each project would deliver impact. Incident data is a trailing dataset meaning that there is a time delay before we know the result. On top of this, customers were experiencing pain right now. This influenced our investment strategy to be: Invest widely initially and bias for action Focus on areas of known pain first Invest further in projects or patterns based on results Curtail investment in the least impactful areas Set a flexible shorter-term roadmap which may change based on results. Our goals directed us to look for projects that would therefore influence one or more the following: Detection of an issue earlier in the deployment process Improved automatic remediation time Improved manual remediation time Reduced issue severity through design of isolation boundaries (blast radius control) We attributed Webapp backend as the largest source of change-triggered incidents. The following is an example investment flow to address our Webapp backend deployments. A quarter of work to engineer automatic metric monitoring Another quarter to confirm customer impact alignment via automatic alerts and manual rollback actions Investing in automatic deployments and rollback Proving success with many automatic rollbacks keeping customer impact below 10 minutes Further investment to monitor additional metrics and invest in manual rollback optimisations Investing in a manual Frontend rollback capability Aligned further investment toward Slack’s centralised deployment orchestration system inspired by ReleaseBot and the AWS Pipelines deployment system to unify the use of metrics-based deployments with automatic remediation beyond Slack Bedrock / Kubernetes to many other deployment systems Achieved success criteria: Webapp backend, frontend, and a portion of infra deployments are now significantly safer and showing continual improvement quarter-over-quarter This pattern of trying something, finding success, then iterating + copying the pattern to other systems continues to serve us well. There have been too many projects to list, some more successful (e.g., faster Mobile App issue detection) and others where the impact hasn’t been as noticeable. In some cases we had reduced impact due to the greater success of automation over manual remediation improvements. It’s very important to note that projects that didn’t have the desired impact are not failures, they’re a critical input to our success through guiding investment and understanding which areas are of greater value. Not all projects will be as impactful, and this is by design. Results <img loading="lazy" decoding="async" class="wp-image-17516 size-medium" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?w=640" alt="Graph showing a 90% decrease in customer impact hours from Feb 2024 to Jan 2025 and highlighting the drop after automatic rollbacks were introduced Apr 2024" width="639" height="602" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png 960w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=640,603 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=768,723 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=380,358 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=800,753 800w" sizes="auto, (max-width: 639px) 100vw, 639px" /> Customer impact by quarter vs progressive targets Please note: our internal metric tracking of “impact” is much more granular and sensitive than disruptions that might qualify for the Slack Status Site. The bar chart above tells quite a story of non-linear progress and the difficulties with using trailing metrics based on waiting for incident occurrence. After the first quarter of work we had already seen improvement, but hadn’t yet deployed any changes other than communicating with the engineering teams about the program. Once projects started to deliver changes, we then had the peak quarter of impact between February and April of 2024. Was this working? There was still confidence that the work we were doing was going to have the desired impact based on the early results from Webapp backend metrics based deploy alerts with manual remediation – the peak of impact in 2024 would have otherwise been even higher. What we needed was automatic instead of manual remediation. Once automatic rollbacks were introduced we observed dramatic improvement in results. We’ve continued to experience a 3-6 month lag time to observe each project’s full impact. Over time the target number has been continually adjusted. Due to the metric requiring trailing incident data, the target was set based on expected potential results through the work delivered the previous quarter. This was notably different from expecting to be able to measure results as soon as work is delivered. We’ve reduced our target number again for 2025 and expect lower customer impact with a focus on mitigating the risk of infrequent spikes through deployment system consistency. Lessons learned This program required continual learning and iteration. We learned many important lessons. Each were critical for a successful program: Ensuring prioritisation and alignment Executive reviews every 4-6 weeks to ensure continued alignment and to seek support where needed High-level priority in company/engineering goals (OKR, V2MOM etc.) Solid support from executive leadership with active project alignment, encouragement, and customer feedback (huge thanks to SVP Milena Talavera, SVP Peter Secor, and VP Cisco Vila) Patience with trailing metrics and faith that you have the right process even when some projects don’t succeed Using a measurement with multiple months of delay from work delivery will need patience. Gather metrics to know if the improvement is at least functioning well (e.g., issue detection) whilst waiting for full results. Faith that you’ve made the best decisions you can with the information you have at the time and the agility to change the path once results are confirmed. Folks will delay adoption of a practice until they’re familiar or comfortable with it They’re worried they’ll make the problem worse if they follow an unknown path even with the reassurance it is better and is recommended. Provide direct training (multiple times if necessary) to many groups to help them become comfortable with the new tooling – “Just roll back!” Continual improvement of manual rollback tooling in response to experience during incidents. Use the tooling often, not just for the infrequent worst case scenarios. Incidents are stressful and we found that without frequent use to build fluency, confidence, and comfort the processes and tools won’t become routine. It would be as if you didn’t build the tool/capability in the first place. Direct outreach to engineering teams is critical The Deploy Safety program team engaged directly with individual teams to understand their systems and processes, provide improvement guidance, and encourage innovation and prioritization. (A huge thanks to Petr Pchelko & Harrison Page for so much of this work.) Not all teams and systems are the same. Some teams know their areas of pain well and have ideas, others want to improve but need additional resources. Keep the top line metric as consistent as possible Pick a metric, be consistent, and make refinements based on validation of results It’s very easy to burn time on deliberating the best metric (there isn’t a perfect one) Maintain consistent, direct communication with engineering staff This is an area we’re working to improve on Management understood prioritisation and results and were well aligned as a group. However, it hasn’t always been clear to general engineering staff revealing an opportunity for better alignment. Into the future Trust is the #1 value for Salesforce and Slack. As reliability is a key portion of that trust we intend to have ongoing investment into Deploy Safety at Slack. Improvements in automatic metrics based deployments & remediation to keep the positive trend going More consistency in our use of Deploy Safe processes for all deployments at Slack to mitigate unexpected or infrequent spikes of customer impact Change of the program’s scope to include migrating remaining manual deploy processes to code-based deploys using Deploy Safe processes Examples of some of the ongoing projects in this space include: Centralised deployment orchestration tooling expansion into other infrastructure deployment patterns: EC2, Terraform and many more Automatic rollbacks for Frontend Metric quality improvements – do we have the right metrics for each system/service? AI metric-based anomaly detection Further rollout of AI generated pre-production tests Acknowledgements Many thanks to all the Deploy Safety team over this first year and a half: Dave Harrington, Sam Bailey, Sreedevi Rai, Petr Pchelko, Harrison Page, Vani Anantha, Matt Jennings, Nathan Steele, Sriganesh Krishnan. And a huge huge thanks to the too many to name teams that have picked up Deploy Safety projects large and small to improve the product experience for our customers.
As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than…
As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than proactive threat detection and prevention. Organizations urgently need solutions that dramatically compress the detection-to-response window to regain a defensive advantage. To tackle this challenge, we’ve developed Anomaly Event Response (AER) – a new proactive defence mechanism inside Slack. By combining real-time monitoring with advanced analytics, AER autonomously identifies high-confidence threat actor behaviours as they emerge on our platform. When suspicious activity is detected, the system automatically terminates the associated user sessions, reducing the security detection and response gap from potential days/hours to mere minutes. The result? A powerful native security capability that disrupts attack chains before they can be fully executed, preventing data exfiltration and system compromise without requiring additional security tools, integration, or human capital. In this post, we’ll explore the technical architecture behind Anomaly Event Response, examining our motivation, detection mechanisms, response capabilities, and the insights we gained along the way. The Shared Responsibility of Securing Slack Slack is where work happens. We’re a central hub for workplace communication and collaboration for tens of millions of users on a weekly basis, generating billions of interactions daily. To ensure our customers’ security, we provide comprehensive audit logs to Enterprise customers that record when entities take an action on the platform. Within the full list includes hundreds of actions that we support, we also provide customers with anomaly audit logs which take a step further by leveraging advanced analytics to flag unusual activities across a workspace including irregular logins, uploading malware, unexpected data transfers, and more. Those interested in learning more about these special audit logs and how they can use them in security investigations will be interested in reading our Slack Audit Logs and Anomalies post. These specialized audit logs serve as an early warning system, highlighting suspicious activity and providing invaluable insight into potential threats that might otherwise remain undetected. That said, they aren’t a solution in and of themselves; they still require review by security personnel or integration with third-party solutions to translate detected behaviour into actionable insights. Integrating your Slack audit logs into a larger security solution offers unparalleled flexibility and customizability, but we also know this isn’t feasible for all customers based on their size, security capabilities, or resources. We created AER to bridge this gap between detection and response with an automated solution that any Enterprise Grid customer can use out-of-the-box, either by itself or as part of their broader security approach. Trust is our number one core value. We believe security is a shared responsibility between us and our customers by empowering them with data and tools to build security solutions while also fostering a secure platform and neutralizing threats. Our Anomaly Event Response solution bridges threat detection and automated response for everyone while still allowing advanced customers to add tailored security measures on top of this tool to address their unique needs. It also continues our ongoing commitment to enhancing user security with innovative automations like our measures against password breaches and cookie hijacking. Design Philosophy When designing this feature, we focused on supporting the most common types of threats that platforms face. Our goal was to prioritize detections that would have the broadest appeal across the platform. We landed on: Accessing Slack from a Tor exit node Excessive downloading as a potential indicator of data exfiltration Data scraping via non-Slack-native automation tools Session fingerprint mismatches Unexpected API call volumes and patterns Unexpected user agents indicating access from non-standard or virtual clients We know every customer uses Slack differently, and every organization is going to have a different risk profile, so we designed AER to let organizations select which detections are important to them and which should simply be logged without any responding action. This same configurability extends to notification preferences. To make changes to your configuration, navigate via Tools and Settings → Organization settings → Security (left-hand sidebar) → Security Settings. From here, there will be an Anomaly Event Response Settings section where you can choose which anomaly events should trigger user session termination and how you would like to be notified, if at all. <img loading="lazy" decoding="async" width="1854" height="2514" class="aligncenter size-large wp-image-17535" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?w=1020" alt="A modal window titled "Anomaly event response" with various checkboxes for detecting anomaly events like "Accessing Slack from a Tor exit node" and "Data scraping," and for sending notifications to "Org Primary Owner" and "Security Admins" via email and Slack." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png 1854w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=640,868 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=768,1041 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1280,1736 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1133,1536 1133w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1510,2048 1510w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=380,515 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=800,1085 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1160,1573 1160w" sizes="auto, (max-width: 1854px) 100vw, 1854px" /> System Architecture & Technical Deep-Dive AER utilizes a multi-tiered architecture we designed for real-time anomaly detection, asynchronous job orchestration, and dynamic notifications. These operations are performed by three core components: the underlying detection engine, a decision framework, and the response orchestrator. <img loading="lazy" decoding="async" width="2545" height="3840" class="aligncenter size-large wp-image-17536" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?w=1020" alt="A flowchart illustrating an "Automated Event Response" process. It starts with "Suspicious User Activity," followed by "Activity Analyzed," then "Anomaly Detected." This leads to "AER Controller," which determines if it's a "Supported Anomaly." If so, "User Sessions Terminated" occurs. From "User Sessions Terminated," two paths emerge: "Audit Logged" (which is "Always" done) and "Customer Notified" (which is "Configurable")." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png 2545w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=640,966 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=768,1159 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1280,1931 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1018,1536 1018w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1357,2048 1357w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=380,573 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=800,1207 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1160,1750 1160w" sizes="auto, (max-width: 2545px) 100vw, 2545px" /> Detection Engine First, AER’s detection engine monitors Slack events at scale of billions per day, using a combination of rule-based heuristics and dynamic thresholds calibrated to each enterprise’s usage patterns. For example, when monitoring excessive downloading activity, what is anomalous for one organization may be the expected baseline for another, so we calculate thresholds for each organization based on historical data. This adaptive approach not only reduces false positives significantly across the customer-base, but it also allows us to constantly fine-tune the sensitivity of our detections and validate their effectiveness. Decision Framework Next, the decision framework evaluates each audit payload created when suspicious behaviour is identified, performing validation checks based on the customer’s AER configuration and internal rules. One of these checks includes analyzing the given payload and checking what preceded it so we can determine if malicious behaviour is persisting through user session terminations on the account while also making sure we’re protecting legitimate users from a continuous termination loop. We achieve this by analyzing the session(s) associated with every audit payload. Once an anomaly is validated and the customer has enabled detection for that event type, an asynchronous job is enqueued that performs the autonomous response. Response Orchestration Finally, AER terminates all active user sessions belonging to the acting user and executes a series of actions designed for transparency and accountability. AER generates a user_sessions_reset_by_anomaly_event_response audit log containing the specific anomaly type that triggered the response, context about the acting user, and the session_id from which the anomaly event originated. During an investigation, this datapoint lets you connect this activity back to the originating anomaly audit log for additional context. Customers should actively monitor for this audit log action so they are aware of when AER has taken autonomous action and if they need to conduct any further investigation into the acting user’s actions across their organization. After generating this audit log, AER queries the customer’s notification preferences to determine how any additional notifications should be routed. While the acting user always receives an email notification informing them of their terminated sessions, the customer’s configuration determines if any additional notifications are routed up to two other contact types: the Org Primary Owner and any users with the Security Admin system role. To reduce notification fatigue at this stage, we built smart notification logic that recognizes when someone is wearing multiple hats in your organization. For example, if the same person is both the Org Primary Owner and has the Security Admin system role, they’ll receive just one notification instead of multiple alerts for the same event. AER efficiently tracks these areas of overlap and ensures everyone gets informed without being overwhelmed by duplicate messages. It’s worth calling out, these contacts can opt to receive an email notification every time AER takes action on a user in their workspace, or they can choose to receive in-Slack DM notifications via Slack Security bot (or both!). <img loading="lazy" decoding="async" width="1068" height="328" class="aligncenter size-large wp-image-17539" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?w=1020" alt="A Slack notification from "Slack Security" stating that "@Matt Brewer has been signed out of all their active sessions in response to a potentially suspicious anomaly event: unexpected_scraping. It provides links to view details in audit logs and learn more."" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png 1068w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=640,197 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=768,236 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=380,117 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=800,246 800w" sizes="auto, (max-width: 1068px) 100vw, 1068px" /> For more information, see the Configure audit log anomaly event responses in Slack article in Slack’s Help Center. Key Findings & Impact Since launching AER in February 2025, its automated interventions have validated the importance of proactive security measures and demonstrated its effectiveness in disrupting suspicious activities in real-time. For context, according to IBM’s most recent Cost of a Data Breach Report, the average cost of a data breach reached $4.88M USD in 2024 with an average incident response time of 277 days for cloud-based collaboration platforms. AER’s ability to reduce this response window is a significant step forward in making a true security impact. By automatically terminating sessions engaging in anomalous activity, AER can prevent potential security incidents that could otherwise go undetected. For our enterprise customers, these benefits translate to enhanced protection of their data while simultaneously reducing security team overhead. As threat actors continue to evolve their techniques, AER provides a scalable, adaptive defence mechanism, ensuring that we keep Slack a secure and trusted platform for collaboration and communication. Conclusion Security should work while you do. Anomaly Event Response continues our tradition of approaching security in innovative ways that prioritize shared responsibility and impact at scale. By closing the gap between detection and response, we’ve neutralized the delay that traditionally favors attackers over defenders. The shared responsibility model we’ve embraced with AER represents the future of enterprise security – where platform providers and customers work together to create multiple layers of defence. By providing advanced security that’s accessible to everyone out of the box while still offering customization for organizations with sophisticated security operations, we’re ensuring that all customers benefit from real-time threat disruption regardless of their size or security resources. And the most exciting part? As the digital threat landscape continues to evolve, so will AER. We’re committed to providing security that works silently and effectively in the background by continuously refining our detection mechanisms, expanding our response capabilities, and incorporating customer feedback to extend the feature with new and exciting capabilities, allowing teams to collaborate with confidence. Slack is where work happens, and that modern collaborative environment requires and deserves modern forward-thinking security.
In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time…
In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time and resources for engineers at Slack. The Problem: Unnecessary Frontend Builds For one of our largest code repositories (a monolithic repository, or monorepo), Slack has a CI/CD pipeline that runs E2E tests before merging code into the main branch. This is critical for ensuring that changes are validated across the entire stack for the Slack application: frontend, backend, database, and the handful of services in between. However, we noticed a bottleneck: building the frontend code was taking longer than expected and occurred too frequently, even when there were no frontend-related changes. Here’s the breakdown: Developer Workflow: A developer makes changes and pushes to a branch. Build Process: The frontend code is built (~5 minutes). Deployment: The build is deployed to a QA environment. Testing: We run over 200 E2E tests, taking another 5 minutes. This entire process took about 10 minutes per run. Half of that time, around 5 minutes, was consumed by frontend builds, even when no frontend changes were involved. <img loading="lazy" decoding="async" width="1661" height="780" class="alignnone size-medium wp-image-17430" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?w=640" alt="Stages of the end to end testing pipeline." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png 1661w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=640,301 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=768,361 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1280,601 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1536,721 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=380,178 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=800,376 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1160,545 1160w" sizes="auto, (max-width: 1661px) 100vw, 1661px" /> Given that hundreds of pull requests (PRs) are merged daily, these redundant builds were not only time-consuming, but costly: Thousands of frontend builds per week, storing nearly a gigabyte of data per build in AWS S3. Half of these builds do not contain frontend changes compared to the last merge into main, causing terabytes of duplicate data. 5 minutes per build, adding unnecessary delays to pipelines (thousands of hours a week). The Solution: Smarter Build Strategy with Cached Frontend Assets To tackle this, we leveraged existing tools to rethink our build strategy. Step 1: Conditional Frontend Builds Our first step was determining whether a fresh frontend build was necessary. We detected changes by utilizing git diff and its 3-dot notation to identify the difference between the latest common commit of the current checked-out branch and main. If changes were detected, we invoke a frontend build job. If no changes were detected, we skipped the build entirely and reused a prebuilt version. Step 2: Prebuilt Assets and Internal CDN When a frontend build is not needed, we locate an existing build from AWS S3. To be efficient, we use a recent frontend build that is still in Production. We delegate the task of serving the prebuilt frontend assets for our E2E tests to an internal CDN. This reduced the need for creating a new build on each PR, while still ensuring we test on current assets. The Challenges: Efficiency at Scale While the approach seemed straightforward, scaling this solution to our monorepo presented a few challenges: Identifying Frontend Changes: Our repository contains over 100,000 tracked files. Determining whether frontend changes were present required efficient file tracking, which git handled in just a couple of seconds. Finding Prebuilt Assets: With hundreds of PRs merged into this repository daily, identifying a prebuilt version that was fresh enough required robust asset management. By using straightforward S3 storage concepts, we were able to balance recency, coherent file naming, and performance to manage our assets. Being Fast: We were able to distinguish if a frontend build was unnecessary and find a recent build artifact in just under 3 seconds on average. <img loading="lazy" decoding="async" width="1661" height="780" class="alignnone size-medium wp-image-17431" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?w=640" alt="End to end pipeline depicting local pre-built frontends at the third stage." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png 1661w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=640,301 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=768,361 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1280,601 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1536,721 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=380,178 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=800,376 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1160,545 1160w" sizes="auto, (max-width: 1661px) 100vw, 1661px" /> The Results: A 60% Drop in Build Frequency and 50% Drop in Build Time Our efforts paid off, delivering remarkable improvements: 60% Reduction in Build Frequency: By intelligently reusing prebuilt frontend assets, we reduced the number of unnecessary frontend builds by over half. Hundreds of Hours Saved Monthly: Cloud compute time and developer wait times are reduced. Several Terabytes of Storage Savings: We reduced our AWS S3 storage by several terabytes each month. These duplicate assets would have otherwise been stored for one year. 50% Build Time Improvement: This was the second major project by the Frontend DevXP Team and its partnering teams. The first project, which upgraded our Webpack setup, reduced the average build from ~10 minutes to ~5 minutes. This project took the average build from ~5 minutes down to just ~2 minutes. With both projects being successful, we reduced our average build time for E2E pipelines from ~10 minutes to ~2 minutes: a huge improvement for the year! <img loading="lazy" decoding="async" width="1658" height="932" class="alignnone size-medium wp-image-17428" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?w=640" alt="January baseline was an average build time of 10 minutes and 6 minutes for testing. By August average build time was 2 minutes, an 80% decrease for the year." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png 1658w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1536,863 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1160,652 1160w" sizes="auto, (max-width: 1658px) 100vw, 1658px" /> Two unexpected outcomes: More Reliable and Trustworthy E2E Results: Our test flakiness, which refers to tests failing intermittently or inconsistently despite no code changes, was significantly reduced. This improvement resulted from the optimized pipeline, decreased likelihood of needing complex frontend builds, and consistent asset delivery. We observed our lowest percentage of test flakiness as a result, based on monthly measurements. Rediscovering Legacy Code: Implementing this optimization required a deep dive into legacy code of multiple systems that hadn’t been significantly modified in a long time. This exploration yielded valuable insights, prompted new questions about the codebase’s behavior, and generated a backlog of tasks for future enhancements. Conclusion: Rethinking Frontend Build Efficiency By strategically utilizing existing tools like git diff and internal CDNs, we managed to save valuable developer time, reduce cloud costs, and improve overall build efficiency. For teams in other companies facing similar bottlenecks in DevOps and DevXP, the lesson is to question what’s truly necessary in your pipeline and optimize accordingly. The improvement from this project seems obvious in hind-sight, but it’s common to overlook inefficiencies in systems that haven’t outright failed. In our case, rethinking how we handled frontend assets turned into a massive win for the organization. Acknowledgments There are a lot of moving parts in a project like this: complex pipelines for building and testing, cloud infrastructure, an internal CDN, intricate build systems for frontend code, and existing custom setups throughout our entire system. It includes code written in Python, JavaScript, Bash, PHP/Hack, Rust, YAML, and Ruby. We achieved this without any downtime! Okay, almost. There was ten minutes of internal downtime for our deployment pipeline, but it was fixed pretty quickly. This work was not possible without contributions from: Anirudh Janga, Josh Cartmell, Arminé Iradian, Anupama Jasthi, Matt Jennings, Zack Weeden, John Long, Issac Gerges, Andrew MacDonald, Vani Anantha and Dave Harrington Interested in taking on interesting projects, making people’s work lives easier, or just building some pretty cool forms? We’re hiring! <path d="M9 17L1 9L9 1" stroke="#1d1c1d" stroke-linecap="round" stroke-linejoin="round" /> Explore Opportunities
How we built enterprise search to be secure and private
Many don’t know that “Slack” is in fact a backronym—it stands for “Searchable Log of all Communication and Knowledge”. And these days, it’s not just a searchable log: with Slack AI, Slack is now an intelligent log, leveraging the latest in generative AI to securely surface powerful, time-saving insights. We built Slack AI from the…
Many don’t know that “Slack” is in fact a backronym—it stands for “Searchable Log of all Communication and Knowledge”. And these days, it’s not just a searchable log: with Slack AI, Slack is now an intelligent log, leveraging the latest in generative AI to securely surface powerful, time-saving insights. We built Slack AI from the ground up to be secure and private following principles that mirror our existing enterprise grade compliance standards: Customer data never leaves Slack’s trust boundary. We do not train large language models (LLMs) on customer data. Slack AI only operates on the data that the user can already see. Slack AI integrates seamlessly with our existing enterprise grade compliance and security offerings. Now, with enterprise search, Slack is more than a log of all content and knowledge inside Slack—it also includes knowledge from your key applications. Users can now surface up-to-date, relevant content that is permissioned to them directly in Slack’s search. We’re starting with Google Drive and GitHub, and you’ll see many more of your connected apps as the year goes on. With these new apps, Slack search and AI Answers are all the more powerful, pulling in context from across key tools to satisfy your queries. We built enterprise search to uphold the same Enterprise-grade security and privacy standards as Slack AI: We never store data from external sources in our databases. External data and permissions are up to date with the external system. Users and admins must explicitly grant Slack access to external sources and may revoke that access at any time. We uphold the principle of least privilege, only requesting the authorizations we need to satisfy search queries. This blog post will explain how these principles guided the architecture of enterprise search. How enterprise search upholds the Slack AI principles First, a refresher: how does Slack AI uphold our security principles? Slack uses AWS to host closed-source large language models (LLMs) in an escrow VPC. This structure ensures that the model provider never has access to Slack customer data and customer data never leaves Slack’s trust boundary—whether it’s Slack messages, enterprise search results, or anything in between. We use Retrieval Augmented Generation (RAG) instead of training LLMs. Using RAG, we supply an LLM with only the content needed to complete the task. This content is permissioned to the user and only available to the LLM at runtime, meaning the LLM doesn’t retain any of your data, ever. To provide a private, permissions-aware AI product, Slack uses the requesting user’s Access Control List (ACL) to ensure that the LLM only receives data the user can already access in Slack. Finally, we re-use all our existing compliance infrastructure (such as Encryption Key Management and International Data Residency) when storing and processing LLM-generated content. And we don’t even store Search Answer summaries—we just show them to the requesting user and immediately discard them. Enterprise search is built atop Slack AI and benefits from many of the innovations we developed for Slack AI. We use the same LLMs in the same escrow VPC; we use RAG to avoid training LLMs on user data; and we don’t store Search Answers in the database (whether or not they contain external content). However, enterprise search adds a new twist. We can now provide permissioned content from external sources to the LLM and in your search results. How enterprise search upholds our security principles We never store data from external sources When developing enterprise search, we decided not to store external source data in our database. Instead, we opted for a federated, real-time approach. Building atop Slack’s app platform, we use public search APIs from our partners to return the most up-to-date, permissioned results for a given user. Note that the Slack client may cache data between reloads to performantly serve product features like filtering and previews. <img loading="lazy" decoding="async" width="2467" height="2292" class="alignnone size-medium wp-image-17462" src="https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?w=640" alt="Architecture of enterprise search" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png 2467w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=640,595 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=768,714 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=1280,1189 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=1536,1427 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=2048,1903 2048w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=380,353 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=800,743 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/03/enterprisesearch.png?resize=1160,1078 1160w" sizes="auto, (max-width: 2467px) 100vw, 2467px" /> External data and permissions are up to date When searching for external data, it’s essential that we only fetch data which the user can access in the external system (this mirrors our Slack AI principle #3, “Slack AI only operates on the data that the user can already see”) and that this data is up to date. Using a real-time instead of an index-based approach helps us uphold this principle. Because we’re always fetching data from external sources in response to a user query, we never risk that data getting stale. There’s nothing stored on our side, so staleness simply isn’t possible. But how do we scope down queries to just data that the querying user can access in the external system? The Slack platform already provides powerful primitives for connecting external systems to Slack, chief among them being OAuth. The OAuth protocol allows a user to securely authorize Slack to take agreed-upon actions on their behalf, like reading files the user can access in the external system. By leveraging OAuth, we ensure that enterprise search can never perform an action the user did not authorize the system to perform in the external system, and that the actions we perform are a subset of those the user could themselves perform. Users must explicitly grant access to external sources We believe that your external data should be yours to control. As such, Slack admins must opt in each external source for use in their organization’s search results and Search Answers. They can also revoke this access (for both search results and Search Answers) at any time. Next, Slack users also explicitly grant access before we integrate any external sources in their search. Users may also revoke access to any source at any time. This level of control is possible due to the OAuth-based approach mentioned above. Principle of Least Privilege An important security principle is that a system should never request more privileges than it requires. For enterprise search, this means that when we connect to an external system, we only request the OAuth scopes which are necessary to satisfy search queries—specifically read scopes. Not only do we adhere to the principle of least privilege, we show admins and end users the scopes we plan to request when they enable an external source for use in enterprise search. This means that admins and end users always know which authorizations Slack requires to integrate with an external source. Conclusion At Salesforce, trust is our #1 value. We’re proud to have built an enterprise search experience that puts security and privacy front and center, building atop the robust security principles already instilled by Slack AI. We’re excited to see how our customers use this powerful new functionality, secure in the knowledge that their external data is always in good hands.
Automated Accessibility Testing at Slack
At Slack, customer love is our first priority and accessibility is a core tenet of customer trust. We have our own Slack Accessibility Standards that product teams follow to guarantee their features are compliant with Web Content Accessibility Guidelines (WCAG). Our dedicated accessibility team supports developers in following these guidelines throughout the development process. We…
At Slack, customer love is our first priority and accessibility is a core tenet of customer trust. We have our own Slack Accessibility Standards that product teams follow to guarantee their features are compliant with Web Content Accessibility Guidelines (WCAG). Our dedicated accessibility team supports developers in following these guidelines throughout the development process. We also frequently collaborate with external manual testers that specialize in accessibility. In 2022, we started to supplement Slack’s accessibility strategy by setting up automated accessibility tests for desktop to catch a subset of accessibility violations throughout the development process. At Slack, we see automated accessibility testing as a valuable addition to our broader testing strategy. This broader strategy also includes involving people with disabilities early in the design process, conducting design and prototype review with these users, and performing manual testing across all of the assistive technologies we support. Automated tools can overlook nuanced accessibility issues that require human judgment, such as screen reader usability. Additionally, these tools can also flag issues that don’t align with the product’s specific design considerations. Despite that, we still felt there would be value in integrating an accessibility testing tool into our test frameworks as part of the overall, comprehensive testing strategy. Ideally, we were hoping to add another layer of support by integrating the accessibility validation directly into our existing frameworks so test owners could easily add checks, or better yet, not have to think about adding checks at all. Exploration and Limitations Unexpected Complexities: Axe, Jest, and React Testing Library (RTL) We chose to work with Axe, a popular and easily configurable accessibility testing tool, for its extensive capabilities and compatibility with our current end-to-end (E2E) test frameworks. Axe checks against a wide variety of accessibility guidelines, most of which correspond to specific success criteria from WCAG, and it does so in a way that minimizes false positives. Initially we explored the possibility of embedding Axe accessibility checks directly into our React Testing Library (RTL) framework. By wrapping RTL’s render method with a custom render function that included the Axe check, we could remove a lot of friction from the developer workflow. However, we immediately encountered an issue related to the way we’ve customized our Jest set up at Slack. Running accessibility checks through a separate Jest configuration worked, but would require developers to write tests specifically for accessibility, which we wanted to avoid. Reworking our custom Jest setup was deemed too tricky and not worth the time and resource investment, so we pivoted to focus on our Playwright framework. The Best Solution for Axe Checks: Playwright With Jest ruled out as a candidate for Axe, we turned to Playwright, the E2E test framework utilized at Slack. Playwright supports accessibility testing with Axe through the @axe-core/playwright package. Axe Core provides most of what you’ll need to filter and customize accessibility checks. It provides an exclusion method right out of the box, to prevent certain rules and selectors from being analyzed. It also comes with a set of accessibility tags to further specify the type of analysis to conduct (‘wcag2a‘, ‘wcag2aa‘, etc.). Our initial goal was to “bake” accessibility checks directly into Playwright’s interaction methods, such as clicks and navigation, to automatically run Axe without requiring test authors to explicitly call it. In working towards that goal, we found that the main challenge with this approach stems from Playwright’s Locator object. The Locator object is designed to simplify interaction with page elements by managing auto-waiting, loading, and ensuring the element is fully interactable before any action is performed. This automatic behavior is integral to Playwright’s ability to maintain stable tests, but it complicated our attempts to embed Axe into the framework. Accessibility checks should run when the entire page or key components are fully rendered, but Playwright’s Locator only ensures the readiness of individual elements, not the overall page. Modifying the Locator could lead to unreliable audits because accessibility issues might go undetected if checks were run at the wrong time. Another option, using deprecated methods like waitForElement to control when accessibility checks are triggered, was also problematic. These older methods are less optimized, causing performance degradation, potential duplication of errors, and conflicts with the abstraction model that Playwright follows. So while embedding Axe checks into Playwright’s core interaction methods seemed ideal, the complexity of Playwright’s internal mechanisms required us to explore some further solutions. <img loading="lazy" decoding="async" width="6960" height="4640" class="alignnone size-medium wp-image-17384" src="https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?w=640" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg 6960w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=640,427 640w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=768,512 768w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=1280,853 1280w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=1536,1024 1536w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=2048,1365 2048w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=380,253 380w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=800,533 800w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/joachim-schnurle-4j732BLw0BI-unsplash.jpg?resize=1160,773 1160w" sizes="auto, (max-width: 6960px) 100vw, 6960px" /> Customizations and Workarounds To circumvent the roadblocks we encountered with embedding accessibility checks into the frameworks, we decided to make some concessions while still prioritizing a simplified developer workflow. We continued to focus on Playwright because it offered more flexibility in how we could selectively hide or apply accessibility checks, allowing us to more easily manage when and where these checks were run. Additionally, Axe Core came with some great customization features, such as filtering rules and using specific accessibility tags. Using the @axe-core/playwright package, we can describe the flow of our accessibility check: Playwright test lands on a page/view Axe analyzes the page Pre-defined exclusions are filtered out Violations and artifacts are saved to a file First, we set up our main function, runAxeAndSaveViolations, and customized the scope using what the AxeBuilder class provides. We wanted to check for compliance with WCAG 2.1, Levels A and AA. constructor(page: Page) { this.page = page; this.defaultTags = ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']; this.bodyText = ''; this.baseFileName = `${test.info().title}-violations`.replace(/\//g, '-'); A11y.filenameCounter = 0; } We created a list of selectors to exclude from our violations report. These fell into two main categories: Known accessibility issues – issues that we are aware of and have already been ticketed Rules that don’t apply – Axe rules outside of the scope of how Slack is designed for accessibility // Exclude selectors for known bugs and elements that we do not consider accessibility issues constants.ACCESSIBILITY.AXE_EXCLUDED_SELECTORS.forEach((excludedSelector) => { axe.exclude(excludedSelector); }); We also wanted to filter for duplication and severity level. We created methods to check for the uniqueness of each violation and filter out duplication. We chose to report only the violations deemed Critical according to the WCAG. Serious, Moderate, and Mild are other possible severity levels that we may add in the future. /** * Filter violations based on criticality, then ensure we * are removing any duplicate violations within a single test file * Please note: this only removes duplicates on a single test, not the entire run */ private filterAndRemoveDuplicateViolations(violations: Violation[]) { return violations .filter((violation) => ['critical'].includes(violation.impact)) .map(this.mapViolation) .filter(this.isUniqueViolation.bind(this)); } We took advantage of the Playwright fixture model. Fixtures are Playwright’s way to build up and teardown state outside of the test itself. Within our framework we’ve created a custom fixture called slack which provides access to all of our API calls, UI views, workflows and utilities related to Slack. Using this fixture, we can access all of these resources directly in our tests without having to go through the setup process every time. We moved our accessibility helper to be part of the pre-existing slack fixture. This allowed us to call it directly in the test spec, minimizing some of the overhead for our test authors. // Run accessibility checks and get the violations await slack.utils.a11y.runAxeAndSaveViolations(); We also took advantage of the ability to customize Playwright’s test.step. We added the custom label “Running accessibility checks in runAxeAndSaveViolations” to make it easier to detect where an accessibility violation has occurred: Test Steps - Before Hooks - apiResponse.json— ../support/api/api.ts:137 - browserContext.waitForEvent— ../support/workflows/login.workflow.ts:273 - Running accessibility checks in runAxeAndSaveViolations— ../support/utils/accessibility.ts:54 Placement of Accessibility Checks in End to End Tests To kick the project off, we set up a test suite that mirrored our suite for testing critical functionality at Slack. We renamed the suite to make it clear it was for accessibility tests, and we set it to run as non-blocking. This meant developers would see the test results, but a failure or violation would not prevent them from merging their code to production. This initial suite encompassed 91 tests in total. Strategically, we considered the placement of accessibility checks within these critical flow tests. In general, we aimed to add an accessibility check for each new view, page, or flow covered in the test. In most cases, this meant placing a check directly after a button click, for example, or a link that leads to navigation. In other scenarios, our accessibility check needed to be placed after signing in as a second user or after a redirect. It was important to make sure the same view wasn’t being analyzed twice in one test, or potentially twice across multiple tests with the same UI flow. Duplication like this would result in unnecessary error messages and saved artifacts, and slow down our tests. We were also careful to place our Axe calls only after the page or view had fully loaded and all content had rendered. With this approach, we needed to be deeply familiar with the application and the context of each test case. <img loading="lazy" decoding="async" class="alignnone wp-image-17386" src="https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?w=640" alt="" width="591" height="443" srcset="https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg 4000w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=640,480 640w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=768,576 768w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=1280,960 1280w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=1536,1152 1536w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=2048,1536 2048w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=380,285 380w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=800,600 800w, https://slack.engineering/wp-content/uploads/sites/7/2024/12/krystian-tambur-k7rZ8wTfABA-unsplash.jpg?resize=1160,870 1160w" sizes="auto, (max-width: 591px) 100vw, 591px" /> Violations Reporting We spent some time iterating on our accessibility violations report. Initially, we created a simple text file to save the results of a local run, storing it in an artifacts folder. A few developers gave us early feedback and requested screenshots of the pages where accessibility violations occurred. To achieve this, we integrated Playwright’s screenshot functionality and began saving these screenshots alongside our text report in the same artifact folder. To make our reports more coherent and readable, we leveraged the Playwright HTML Reporter. This tool not only aggregates test results but also allows us to attach artifacts such as screenshots and violation reports to the HTML output. By configuring the HTML reporter, we were able to display all of our accessibility artifacts, including screenshots and detailed violation reports, in a single test report. Lastly, we wanted our violation error message to be helpful and easy to understand, so we wrote some code to pull out key pieces of information from the violation. We also customized how the violations were displayed in the reports and on the console, by parsing and condensing the error message. Error - [A11Y]: CRITICAL Description: Ensures an element's role supports its ARIA attributes Help: Elements must only use supported ARIA attributes Target selector: #add-channel-tab Fix all of the following: ARIA attribute is not allowed: aria-selected="false" HTML: <button class="c-button-unstyled addTab__brBMy c-tabs__tab js-tab" data-qa="unstyled-button" Environment Setup and Running Tests Once we had integrated our Axe checks and set up our test suite, we needed to determine how and when developers should run them. To streamline the process for developers, we introduced an environment flag, A11Y_ENABLE, to control the activation of accessibility checks within our framework. By default, we set the flag to false, preventing unnecessary runs. This setup allowed us to offer developers the following options: On-Demand Testing: Developers can manually enable the flag when they need to run accessibility checks locally on their branch. Scheduled Runs: Developers can configure periodic runs during off-peak hours. We have a daily regression run configured in Buildkite to pipe accessibility test run results into a Slack alert channel on a daily cadence. CI Integration: Optionally, the flag can be enabled in continuous integration pipelines for thorough testing before merging significant changes. Triage and Ownership Ownership of individual tests and test suites is often a hot topic when it comes to maintaining tests. Once we had added Axe calls to the critical flows in our Playwright E2E tests, we needed to decide who would be responsible for triaging accessibility issues discovered via our automation and who would own the test maintenance for existing tests. At Slack, we enable developers to own test creation and maintenance for their tests. To support developers to better understand the framework changes and new accessibility automation, we created documentation and partnered with the internal Slack accessibility team to come up with a comprehensive triage process that would fit into their existing workflow for triaging accessibility issues. The internal accessibility team at Slack had already established a process for triaging and labeling incoming accessibility issues, using the internal Slack Accessibility Standards as a guideline. To enhance the process, we created a new label for “automated accessibility” so we could track the issues discovered via our automation. To make cataloging these issues easier, we set up a Jira workflow in our alerts channel that would spin up a Jira ticket with a pre-populated template. The ticket is created via the workflow and automatically labeled with automated accessibility and placed in a Jira Epic for triaging. A11Y Automation Bug Ticket Creator - Automatically create JIRA bug tickets for A11Y automation violations Hi there, Would you like to create a new JIRA defect? Button clicked. A new JIRA bug ticket, A11YAUTO-37, was created. What to do next: 1. Please fill out all of the necessary information listed here: https://jira.tinyspeck.com/browse/A11YAUTO-37. 2. Please add this locator to the list of known issues and include the new JIRA bug ticket in the comment. Conducting Audits We perform regular audits of our accessibility Playwright calls to check for duplication of Axe calls, and ensure proper coverage of accessibility checks across tests and test suites. We developed a script and an environment flag specifically to facilitate the auditing process. Audits can be performed either through sandbox test runs (ideal for suite-wide audits) or locally (for specific tests or subsets). When performing an audit, running the script allows us to take a screenshot of every page that performs an Axe call. The screenshots are then saved to a folder and can be easily compared to spot duplicates. This process is more manual than we like, and we are looking into ways to eliminate this step, potentially leaning on AI assistance to perform the audit for us – or have AI add our accessibility calls to each new page/view, thereby eliminating the need to perform any kind of audit at all. What’s Next We plan to continue partnering with the internal accessibility team at Slack to design a small blocking test suite. These tests will be dedicated to the flows of core features within Slack, with a focus on keyboard navigation. We’d also like to explore AI-driven approaches to the post-processing of accessibility test results and look into the option of having AI assistants audit our suites to determine the placement of our accessibility checks, further reducing the manual effort for developers. Closing Thoughts We had to make some unexpected trade-offs in this project, balancing the practical limitations of automated testing tools with the goal of reducing the burden on developers. While we couldn’t integrate accessibility checks completely into our frontend frameworks, we made significant strides towards that goal. We simplified the process for developers to add accessibility checks, ensured test results were easy to interpret, provided clear documentation, and streamlined triage through Slack workflows. In the end, we were able to add test coverage for accessibility in the Slack product, ensuring that our customers that require accessibility features have a consistent experience. Our automated Axe checks have reduced our reliance on manual testing and now complement other essential forms of testing—like manual testing and usability studies. At the moment, developers need to manually add checks, but we’ve laid the groundwork to make the process as straightforward as possible with the possibility for AI-driven creation of accessibility tests. Roadblocks like framework complexity or setup difficulties shouldn’t discourage you from pursuing automation as part of a broader accessibility strategy. Even when it’s not feasible to hide the automated checks completely behind the scenes of the framework, there are ways to make the work impactful by focusing on the developer experience. This project has not only strengthened our overall accessibility testing approach, it’s also reinforced the culture of accessibility that has always been central to Slack. We hope it inspires others to look more closely at how automated accessibility might fit into your testing strategy, even if it requires navigating a few technical hurdles along the way. Thank you to everyone who spent significant time on the editing and revision of this blog post – Courtney Anderson-Clark, Lucy Cheng, Miriam Holland and Sergii Gorbachov. And a massive thank you to the Accessibility Team at Slack, Chanan Walia, Yura Zenevich, Chris Xu and Hye Jung Choi, for your help with everything related to this project, including editing this blog post!