Hear directly from Slack's engineers about what we build, why and how we build it, and how you can use it.
Introduction 🔔 At Slack, notifications are how teams stay in the loop, but they can also become overwhelming when not designed with intention. Our goal was to make staying informed feel effortless. We set out to rebuild one of Slack’s most complicated systems from the ground up by bringing calm, consistency, and clarity to the…
Introduction At Slack, notifications are how teams stay in the loop, but they can also become overwhelming when not designed with intention. Our goal was to make staying informed feel effortless. We set out to rebuild one of Slack’s most complicated systems from the ground up by bringing calm, consistency, and clarity to the experience. Diagnosing the Noise Problem We knew perception of noise in Slack was a universal challenge affecting teams everywhere. Across workspaces, notification overload consistently ranks among the most common frustrations. Research showed that the more channels a person joins, the more likely they are to feel overwhelmed and confused about notification behavior. Internally, the data told a clear story. Issues with notifications are one of the top three drivers of Customer Experience tickets, with users often unsure how to control or understand their settings. The noise problem wasn’t just about volume—it was baked into the architecture itself. Our legacy notification system evolved over years, accumulating complexity that made it nearly impossible for users to understand or control. Four conflicting mental models: Desktop and mobile each had their own preference systems, with different options and behaviors. A “nothing” setting on mobile meant something entirely different from “Off” on desktop. Users couldn’t predict what would happen when they changed a setting. Hidden coupling between preferences: What users were notified about was tightly coupled with how they received notifications. Wanting fewer push notifications meant sacrificing in-app awareness entirely—there was no way to separate the two. Inconsistent state across clients: Settings didn’t reliably sync. Users would configure notifications on desktop only to find mobile behaving completely differently, leading to confusion and duplicate configuration work. Power users left behind: Advanced controls were scattered across multiple menus with no clear hierarchy. Features like “badge all unreads” on mobile were hidden, and there was no unified place to understand all your notification options. These architectural problems directly contributed to the noise users experienced—not just because notifications were frequent, but because users couldn’t confidently control them. Simplifying Notifications at Scale We didn’t just refactor the notifications UI. We completely redesigned how notifications behave across Slack. This project supported a number of notifications improvements to comprehensively address user pain points and confusion: Simpler choices: Channel notifications now have three clear options: All new posts, Mentions, or Mute. Push toggles: Unified on/off options for push notifications across desktop and mobile. Advanced controls: Redesigned settings for power users, including “badge all unreads” on mobile. Global preferences: Modernized desktop and mobile experiences with consistent structure and copy. Sync improvements: Consistent state across clients through simplified preference logic. <img fetchpriority="high" decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17707" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png" alt="Before: four paradigms. After: one unified model with three options." srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-2.png?resize=1160,653 1160w" sizes="(max-width: 1920px) 100vw, 1920px" /> Before: four paradigms. After: one unified model with three options. <img decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17708" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png" alt="Mobile redesign: clearer settings and consistent cross-platform logic." srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-3.png?resize=1160,653 1160w" sizes="(max-width: 1920px) 100vw, 1920px" /> Mobile redesign: clearer settings and consistent cross-platform logic. What “Simple” Really Looked Like Dozens of deep technical threads, many with 100+ replies , guided this project. These weren’t quick bug fixes but rather architecture decisions requiring tight alignment across product, design, frontend, backend, and mobile engineers. We tackled three major technical challenges: The Preference Refactor: Migrating millions of users from four conflicting preference systems to one unified model Modal Makeover and Global Preferences: Separating “what” from “how” with auto-save behavior and consistent cross-platform UI Cross-Platform Parity: Achieving true state consistency between mobile and desktop The Preference Refactor We rebuilt how Slack interprets notification preferences. The old “Off” setting now seamlessly migrates to “Mentions” with push disabled. It sounds simple, but this required deep backend and frontend coordination. With backwards compatibility and the possibility of rollback in mind, we thought it too risky to move people from “off” to “mentions” at the database level. Instead, we used a read time strategy to ensure users had the same experience as before, but using the decoupled push logic. We introduced the new desktop_push_enabled pref which would be the only driver of enabling push notifications. Because this pref did not exist before, we were able to backfill all existing users based on whether they had it previously set to “off” with no interruptions to the current experience. We then did some read time magic to make “off” act as “mentions” but with pushes disabled in the new world (Because that is exactly how it functions today!). In-app notifications and activity are consistent across all clients, but push notifications are further customizable on desktop and mobile. // Prefs before 'desktop': everything | mentions | nothing // Push on desktop 'mobile': everything | mentions |nothing // Push on mobile // Prefs now 'desktop': everything | mentions // Activity on desktop and mobile 'desktop_push_enabled': true | false // Push on desktop 'mobile': everything | mentions | nothing // Push on mobile Impact on noise: This refactor eliminated a major source of confusion. Users who thought they’d turned off all notifications were actually still getting in-app badges—they just didn’t know it. Now, “Mentions” means “notify me about mentions” and the push toggle explicitly controls interruptions. Modal Makeover and Global Preferences The old notification modal forced users to click “Save” after every change, making experimentation unreliable. Users would configure settings, forget to save, and wonder why nothing changed. We introduced auto-save behavior—changes take effect immediately. We decoupled “what” from “how,” giving users independent control over activity and push. And we built cross-platform consistency through reusable React components, replacing legacy mobile-specific UI code. Impact on noise: Users can now fine-tune their notification experience with confidence. Want to see all activity but only get pushed for mentions? Now it’s obvious how to do that. The clearer structure means less trial-and-error and fewer abandoned configuration attempts. <img decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17710" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png" alt="Cleaner, more consistent modal with auto-save behavior." srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-8.png?resize=1160,653 1160w" sizes="(max-width: 1920px) 100vw, 1920px" /> Cleaner, more consistent modal with auto-save behavior. <img loading="lazy" decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17711" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png" alt="Still complex, but far more organized and readable." srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-6.png?resize=1160,653 1160w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /> Still complex, but far more organized and readable. <img loading="lazy" decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17712" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png" alt="Mobile global preferences modernized to match desktop visually and structurally. " srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-5.png?resize=1160,653 1160w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /> Mobile global preferences modernized to match desktop visually and structurally. Cross-Platform Parity Challenge Achieving true parity between mobile and desktop was one of the hardest tasks. The goal was simple: mobile should match desktop by default, with the option to override when needed. The new preference model organizes every option into a clear hierarchy. This isn’t just design polish—it’s the conceptual model that powers how preferences work across all clients: Unified hierarchy: What to notify you about: All new messages, Mentions and DMs (default), or Mute Push notifications: On desktop and mobile (default), desktop only, mobile only, or disabled Advanced: Mobile-specific customization and badge controls This redesign required renaming fields and refactoring client logic for explicit state, eliminating ambiguity and making rollbacks safe. We also rewrote some of the oldest pages in Slack’s iOS app—built before our modern React architecture—to match desktop structure and visuals, ensuring consistency that builds trust across the entire experience. <img loading="lazy" decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17713" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png" alt="This table showcases all the different user preferences we ended adding/updating on the backend " srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-7.png?resize=1160,653 1160w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /> This table showcases all the different user preferences we ended adding/updating on the backend Migration and Rollback Lessons Migrating millions of users without disruption required careful mapping and fallbacks. Key lessons learned: Trust must never break. We added read-time fallbacks so push_enabled: false always means “no push,” even during rollbacks. Tiny schema issues can cause major UX bugs. A malformed field once reset preferences to Mentions until we cleaned data and flushed memcache. Clarity beats cleverness. Removing the sync parameter and storing explicit desktop and mobile values made behavior predictable. Closing Reflection This project wasn’t just a refresh; it was a rebuild of trust. The legacy system created noise through confusion—users couldn’t predict what their settings would do, couldn’t reliably sync state across devices, and couldn’t find the controls they needed. Why it matters Users now control noise with confidence. The unified model means when users set a preference, they know exactly what will happen. No more hidden surprises, no more settings that don’t sync, no more choosing between being uninformed or overwhelmed. Support burden decreased significantly. A unified model means fewer tickets asking “why am I getting notifications?” or “how do I turn off mobile push?” The architecture now matches users’ mental models, making behavior predictable. Teams stay informed without the overwhelm. By separating “what to notify you about” from “how to receive notifications,” users can stay aware of everything happening in a channel while only getting pushed for what truly matters. This is the difference between reactive firefighting and intentional awareness. <img loading="lazy" decoding="async" width="1920" height="1080" class="alignnone size-full wp-image-17706" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png" alt="A unified notifications model across desktop and mobile." srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1-1.png?resize=1160,653 1160w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /> The data proves it: Tracking user engagement from pre-launch through post-launch reveals transformative adoption: Settings engagement increased 5x and sustained for weeks—not one-time curiosity, but active ongoing preference refinement Push notification toggles led to higher usage, with users immediately discovering the decoupled desktop/mobile controls. Advanced visibility options like “badge every unread message” saw significant engagement Better defaults meant fewer workarounds—the percentage of users needing per-channel overrides decreased post-launch Sustained engagement, not a spike—notification settings engagement remained elevated weeks after launch The new default works—the vast majority chose “Mentions and DMs” while “All new messages” and “Mute” served their niche use cases well More importantly, users report feeling more in control of their notification experience—turning Slack from a source of interruption into a tool for intentional focus. What made it work Close collaboration across design, frontend, backend, and mobile Courage to revisit legacy systems instead of patching them Shared alignment on clarity over speed Collective ownership across the Messaging pillar We proved that deep technical simplification can create emotional calm for millions of users. When the system matches how people think, staying informed becomes effortless—and that’s when Slack becomes the calm, focused workspace teams deserve. Acknowledgments Frontend: Frances Coronel, Chris Montrois, Katya Egorova Backend: Shilpa Kannan, Steven Thacher, Yi Chen Che iOS: Evan Hughes, Steven Wu, Sarah Huffman Android: Frank Ding, Matt Pflance XFN Leads: Celia Hunko, Brenda Chang, Annie Lawn, Mala Neti, Tanya Gupta Big shoutout to the cross-functional team that made this project possible. <img loading="lazy" decoding="async" width="1920" height="1080" class="alignnone size-medium wp-image-17704" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?w=640" alt="Notifications Team" srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png 1920w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=1536,864 1536w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/Notifications-2.0-Project-Demo-1.png?resize=1160,653 1160w" sizes="auto, (max-width: 1920px) 100vw, 1920px" /> <path stroke="#032d60" stroke-linejoin="round" stroke-width="5" d="m4.112 1c-2.5 6.167-2.4 21.1 18 31.5"/><path stroke="#032d60" stroke-width="5" d="m20.112 18 2.5 14.5-13.5 1.5"/> Want to help millions of people work more intentionally? Join us at Slack. Apply now
Streamlining Security Investigations with Agents
Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI…
Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI agents to optimize our working efficiency and strengthen Slack’s security defenses. This post is the first in a series that will unpack some of the design choices we’ve made and the many things we’ve learnt along the way. The Development Process The Prototype At the end of May 2025 we had a rudimentary prototype of what would grow into our service. Initially, the service was not much more than a 300 word prompt. The prompt consisted of five sections: Orientation: “You are a security analyst that investigates security alerts […]” Manifest: “You have access to the following data sources: […]” Methodology: “Your investigation should follow these steps: […] ” Formatting: “Produce a markdown report of the investigation: […]” Classification: “Choose a response classification from: […]” We implemented a simple “stdio” mode MCP server to safely expose a subset of our data sources through the tool call interface. We repurposed a coding agent CLI as an execution environment for our prototype. The performance of our prototype implementation was highly variable: sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process. We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of data sources. While we did have some success with this approach, ultimately prompts are just guidelines; they’re not an effective method for achieving fine-grained control. The Solution Our solution was to break down the complex investigation process we’d described in the prompt of our prototype into a sequence of model invocations, each with a single, well-defined purpose and output structure. These simple tasks are chained together by our application. Each task was given a structured output format. Structured output is a feature that can be used to restrict a model to using a specific output format defined by a JSON schema. The schema is applied to the last output from the model invocation. Using structured outputs isn’t “free”; if the output format is too complicated for the model, the execution can fail. Structured outputs are also subject to the usual problems of cheating and hallucination. In our initial prototype, we included guidance to “question your evidence”, but had mixed success. With our structured output approach, that guidance had become a separate task in our investigation flow with much more predictable behavior. This approach gave us more precise control at each step of the investigation process. From Prototype to Production While reviewing the literature, two papers particularly influenced our thinking: Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding (Stanford, OpenAI) Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (Microsoft Research) These papers describe prompting techniques that introduce multiple personas in the context of a single model invocation. The idea of modelling the investigation using defined personas was intriguing, but in order to maintain control we needed to represent our personas as independent model invocations. Security tabletop exercises, and how we might adapt their conventions to our application, were also a major source of inspiration during the design process. Our chosen design is built around a team of personas (agents) and the tasks they can perform in the investigation process. Each agent/task pair is modelled with a carefully defined structured output, and our application orchestrates the model invocations, propagating just the right context at each stage. Investigation Loop <img loading="lazy" decoding="async" class="wp-image-17657 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?w=1020" alt="Flow diagram illustrating how agents cooperate during security investigations" width="1020" height="1173" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png 2202w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=640,736 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=768,883 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1280,1472 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1336,1536 1336w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1781,2048 1781w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=380,437 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=800,920 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_round_simple_vertical_ad2b54.png?resize=1160,1334 1160w" sizes="auto, (max-width: 1020px) 100vw, 1020px" /> The Director agent poses a question and domain expert agents respond, generating findings. The Critic agent reviews findings for quality and assembles a timeline using the most credible. The Director uses the high-quality findings and timeline to determine how to progress the investigation. Our design has three defined persona categories: Director Agent The Investigation Director. The Director’s responsibility is to progress the investigation from start to finish. The Director interrogates the experts by forming a question, or set of questions, which become the expert’s prompt. The Director uses a journaling tool for planning and organizing the investigation as it progresses. Expert Agent A domain expert. Each domain expert has a unique set of domain knowledge and data sources. The experts’ responsibility is to produce findings from their data sources in response to the Director’s questions. We currently have four experts in our team: Access: Authentication, authorization and perimeter services. Cloud: Infrastructure, compute, orchestration, and networking. Code: Analysis of source code and configuration management. Threat: Threat analysis and intelligence data sources. Critic Agent The Critic is a “meta-expert”. The Critic’s responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we’ve defined. The Critic annotates the experts’ findings with its own analysis and a credibility score for each finding. The Critic’s conclusions are passed back to the Director, closing the loop. The weakly adversarial relationship between the Critic and the expert group helps to mitigate against hallucinations and variability in the interpretation of evidence. Because each agent/task pair is a separate model invocation we can vary all of the inputs, including the model version, output format, prompts, instructions, and tools. One of many ways we’re using this capability is to create a “knowledge pyramid”. Knowledge Pyramid <img loading="lazy" decoding="async" class="aligncenter wp-image-17647 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?w=1020" alt="Pyramid diagram illustrating how investigation knowledge flows up from low to high cost models." width="1020" height="786" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png 3200w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=640,493 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=768,592 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1280,987 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1536,1184 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=2048,1579 2048w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=380,293 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=800,617 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/knowledge_pyramid_retina.png?resize=1160,894 1160w" sizes="auto, (max-width: 1020px) 100vw, 1020px" /> At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive. Next, the Critic’s review identifies the most interesting findings from that set. During the review process the Critic inspects the experts’ claims and the tool calls and tool results used to support them, which also incurs a significant token overhead. Once the Critic has completed its review, it assembles an up to date investigation timeline, integrating the running investigation timeline and newly gathered findings into a coherent narrative. The condensed timeline, consisting only of the most credible findings, is then passed back to the Director. This design allows us to strategically use low, medium, and high-cost models for the expert, critic, and director functions, respectively. Investigation Flow The investigation process is broken into several phases. Phases allow us to vary the structure of the investigation loop as the investigation proceeds. At the moment, we have three phases, but it is simple to add more. The Director persona is responsible for advancing the phase. <img loading="lazy" decoding="async" class="wp-image-17666 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?w=1020" alt="Flow diagram illustrating how the Director progresses the investigation through distinct phases." width="1020" height="1564" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png 1750w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=640,982 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=768,1178 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1280,1963 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1001,1536 1001w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1335,2048 1335w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=380,583 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=800,1227 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/investigation_phases_vertical_final_300dpi.png?resize=1160,1779 1160w" sizes="auto, (max-width: 1020px) 100vw, 1020px" /> Investigations begin in the discovery phase. After each round of investigation the Director decides whether to remain in the current phase or to progress to a new phase. Discovery The first phase of each investigation. The goal in the discovery phase is to ensure that every available data source is examined. The Director reviews the state of the investigation and generates a question that is broadcast to the entire expert team. Director Decision A “meta-phase” in which the Director decides whether to advance to the next investigation phase or continue in the current one. The task’s prompt includes advice on when to advance to each phase. Trace Once the discovery phase has made clear which experts are able to produce relevant findings, the Director transitions the investigation to the trace phase. In the trace phase, the Director chooses a specific expert to question. We also have the flexibility to vary the model invocation parameters by phase, allowing us to use a different model or enhanced token budget. Conclude The Director transitions the investigation to the concluding phase when sufficient information has been gathered to produce the final report. Service Architecture Our prototype used a coding agent CLI as an execution harness, but that wasn’t suitable for a practical implementation. We needed an interface that would let us observe investigations occurring in realtime, view and share past investigations, and launch ad-hoc investigations. Critically, we needed a way of integrating the system into our existing stack, allowing investigations to be triggered by our existing detection tools. The service architecture we created does all of these things and is quite simple. Hub The hub provides the service API and an interface to persistent storage. Besides the usual CRUD-like API, the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost. Worker Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed. Dashboard The Dashboard is used by staff to interact with the service. Running investigations can be observed in real-time, consuming the event stream from the hub. Additionally the dashboard provides management tools, letting us view the details of each model invocation. This capability is invaluable when debugging the system. Example Report We’ve included an edited investigation report which demonstrates the potential of the agents to exhibit novel emergent behavior. In this case, the original alert was raised for a specific command sequence, which we analyze because it can be an indicator of compromise. In the course of investigating the alert, the agents independently discovered a separate credential exposure elsewhere in the process ancestry. <img loading="lazy" decoding="async" class="wp-image-17650 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?w=1020" alt="Tree diagram illustrating how agents navigated the process tree." width="1020" height="746" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png 1317w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=640,468 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=768,562 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=1280,936 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=380,278 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=800,585 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/11/process_ancestry_retina.png?resize=1160,848 1160w" sizes="auto, (max-width: 1020px) 100vw, 1020px" /> The highlighted leaf process triggered the investigation, but the agents traced the process hierarchy and discovered a different issue in an ancestor process. The text below is a lightly edited version of the report summary from this investigation. Investigation Report: Credential Exposure in Monitoring Workflow [ESCALATE] Summary: While investigating [command sequence], the investigation uncovered a credential exposure elsewhere in the process ancestry chain. Analysis The investigation confirmed that the command execution on [TIMESTAMP] was part of a legitimate monitoring workflow using [diagnostic tool]. The process ancestry shows the expected execution chain. However, critical security concerns were identified: Credential Exposure: A credential was exposed in process command line parameters within the ancestry chain, creating significant security risk. Expert-Critic Contradiction: The expert incorrectly assessed credential handling as secure while the critic correctly identified exposed credentials, indicating analysis blind spots that require attention. What is notable about this result is that the expert did not raise the credential exposure in its findings; the Critic noticed it as part of its meta-analysis of the expert’s work. The Director then chose to pivot the investigation to focus on this issue instead. In the report, the Director highlights both the need to mitigate the security issue, and to follow-up on the expert’s failure to properly identify the risk. We referred the credential exposure to the service owning team to resolve. Conclusion We’re still at an early phase of our journey to streamline security investigations using AI agents, but we’re starting to see meaningful benefits. Our web-based dashboard allows us to launch and watch investigations in real time, and investigations yield interactive, verifiable reports that show how evidence was collected, interpreted, and judged. During our on-call shifts, we’re switching to supervising investigation teams, rather than doing the laborious work of gathering evidence. Unlike static detection rules, our agents often make spontaneous and unprompted discoveries, as we demonstrated in our example report. We’ve seen this occur many times, from highlighting weakness in IAM policies, to identifying problematic code and more. There’s a great deal more to say. We look forward to sharing more details of how our system works in future blog posts. As a preview of some future content from the series: Maintaining alignment and orientation during multi-persona investigations Using artifacts as a communication channel between investigation participants Human in the loop: human / agent collaboration in security investigations Acknowledgements We wanted to give a shout out to all the people that have contributed to this journey: Chris Smith Abhi Rathod Dave Russell Nate Reeves <path stroke="#032d60" stroke-linejoin="round" stroke-width="5" d="m4.112 1c-2.5 6.167-2.4 21.1 18 31.5"/><path stroke="#032d60" stroke-width="5" d="m20.112 18 2.5 14.5-13.5 1.5"/> Interested in taking on interesting projects, making people’s work lives easier, or just building some pretty cool forms? We’re hiring! Apply now
Background A Voluntary Product Accessibility Template (VPAT) is a document that outlines how well a product aligns with accessibility (a11y) standards. Its primary purpose is to inform customers about a product’s a11y features, enabling them to make informed decisions before purchasing software. At Slack, we conducted a VPAT by a third party a11y vendor in…
Background A Voluntary Product Accessibility Template (VPAT) is a document that outlines how well a product aligns with accessibility (a11y) standards. Its primary purpose is to inform customers about a product’s a11y features, enabling them to make informed decisions before purchasing software. At Slack, we conducted a VPAT by a third party a11y vendor in 2024 following our significant UI change (IA4). During this VPAT process, we identified a11y issues across Android, iOS, and desktop. Of the issues identified on Android, we immediately assigned the obvious shovel-ready issues, such as those with color contrast and missing image labels, to the responsible teams for resolution. To address remaining Android issues, we conducted a comprehensive triage process. In this document, we take a look at the major recurring themes of Android accessibility issues and share resolution strategies and lessons learned in the hopes of helping others make similar improvements. Identified themes [Note: for resolved and for future work.] P1: Error messages are inaccessible P2: Headings are not identified P2: Missing a11y label on EditField P2: Incorrect number of items in list P2: Drag and drop in the workspace switcher is inaccessible P2: Strikethrough information not conveyed to screen reader users P3: Errors indicated by color alone P3: Keyboard Navigation and Focus Keyboard navigability can help customers with motor and visual impairments use Slack on Android. It’s more commonly used on larger form factors (e.g. tablets). We received numerous VPAT tickets related to the keyboard navigation, however, Slack on Android currently does not yet support the large form factor. As a result, we have decided to explore other solutions that align with recommended practices and standards. Issue resolution P1: Error messages are inaccessible. Report If the user enters an invalid input to an edit field and presses “Next” (or Submit), an error message is displayed, but the error is not communicated to users using screen readers. TalkBack should announce that it is in an error state when it receives focus, along with the error message so that users understand why it’s in error. Resolution On Android, errors can be displayed to users in two primary ways: (1) Immediately below the editing field. <img loading="lazy" decoding="async" class="alignnone wp-image-17492" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image5.png?w=552" alt="Error is displayed right below an image edit field." width="290" height="125" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image5.png 552w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image5.png?resize=380,164 380w" sizes="auto, (max-width: 290px) 100vw, 290px" /> (2) Via the SK error banner. <img loading="lazy" decoding="async" class="alignnone wp-image-17493" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png?w=640" alt="Error is shown via SKBanner error type." width="280" height="125" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png 842w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png?resize=640,286 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png?resize=768,343 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png?resize=380,170 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image4.png?resize=800,357 800w" sizes="auto, (max-width: 280px) 100vw, 280px" /> In both cases, TalkBack did not announce errors, so users must swipe through the screen to determine what occurred. For case (1), we modified OutlinedTextField so that the error is announced immediately below the editing field. For case (2), SKBanner for the error type was updated so that when an error occurs, the error is announced to screen reader users. The videos below illustrate the changes before and after using TalkBack for case (2). Before <g fill="none" transform="translate(0 .564)"> <rect height="59" rx="4" width="84"></rect> <path d="M49 29.95L35 36.9V23L49 29.95Z" fill="#fff"></path> </g> After <g fill="none" transform="translate(0 .564)"> <rect height="59" rx="4" width="84"></rect> <path d="M49 29.95L35 36.9V23L49 29.95Z" fill="#fff"></path> </g> P1: Heading is not identified. Report The heading does not convey clear semantic information, making it difficult for screen reader users to quickly grasp the page’s structure and navigate efficiently. Defining headings enables assistive technology to convey the hierarchical organization of the content, enhancing accessibility and providing an alternative navigation method. Resolution We identified and fixed issues with missing headings inside lists (such as in the Preferences page). The vendor also suggested that titles in the top app bar should also be a heading, but we found that this pattern is not consistent with Android standards after testing other apps. Therefore, we closed those tickets. P2: Missing a11y label on EditField Report Some edit fields do not have permanent labels. It is labeled using placeholder text that is removed when the user enters text. Some users with cognitive difficulties may forget the purpose of the edit field without a permanently visible label. Resolution During our recent bug fixes, we encountered a few tricky cases that required collaboration with our design team and screen reader users. Specifically, we focused on our main search field and message input area (AMI). For AMI, space constraints prevented us from implementing an optimal solution. However, for the search field, we opted for a simple yet impactful change by adding an explicit search icon. This minor modification helps users easily identify the purpose of the edit field, significantly enhancing their overall experience. Before After P2: Incorrect number of items in list Report When TalkBack users have Always speak number of list items setting enabled and navigate through a list, TalkBack announces an incorrect number of items. Resolution Our vintage Slack Kit (SK) Bottom sheet was the primary source of the issue. SK divider, although purely decorative, was considered an item within a list. For example, if the bottom sheet had 7 row items, including 2 divider items, TalkBack would announce “7 items in a list” instead of the intended “5 items in a list”. To resolve this, we introduced a new SKListAccessibilityDelegate for SKListAdapter, overwriting the a11y CollectionInfo with the correct number of items. P2: Drag and drop in the workspace switcher is inaccessible. Report Workspaces in the workspace switcher can be rearranged, but it requires the user to select a workspace and drag it to the new location. People who lack the dexterity required to perform this action may have difficulties completing the task, or may not be able to complete the task at all. Resolution In discussions with the design team, we decided to implement an Edit mode in the workspace switcher. When a user enters Edit mode, a six-dot drag handle is displayed for each workspace row so users know that each workspace row has a drag-and-drop capability. Additionally, we introduced custom actions (Move before and after) for TalkBack users so users can move each row item from the TalkBack context menu by three-finger tap on a row item or L or r drawing gestures. Before <img loading="lazy" decoding="async" class="alignnone wp-image-17496" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png?w=640" alt="Before: there is no indication from UI that each workspace is draggable to move around." width="396" height="250" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png 894w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png?resize=640,404 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png?resize=768,485 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png?resize=380,240 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image7.png?resize=800,505 800w" sizes="auto, (max-width: 396px) 100vw, 396px" /> After <img loading="lazy" decoding="async" class="alignnone wp-image-17497" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png?w=640" alt="After: Upon entering Edit mode, each row in the workspace is equipped with a six-dot drag handle. Once the user has completed rearranging the workspaces, they can click the "Done" button located in the top right corner, causing the six-dot drag handles to disappear." width="373" height="250" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png 842w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png?resize=640,429 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png?resize=768,514 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png?resize=380,255 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image1.png?resize=800,536 800w" sizes="auto, (max-width: 373px) 100vw, 373px" /> TalkBack custom actions to move a workspace up and down by one position. <g fill="none" transform="translate(0 .564)"> <rect height="59" rx="4" width="84"></rect> <path d="M49 29.95L35 36.9V23L49 29.95Z" fill="#fff"></path> </g> P2: Strikethrough information not conveyed to screen reader users Report Users have the option to format text as strikethrough, but the strikethrough information is not conveyed to screen reader users. It is visually evident that the text has strikethrough, but it is read as normal text for screen reader users. Resolution During our conversation, we made a decision in consultation with a member of the blind community and agreed that the strikethrough format did not pose an issue. Most screen readers don’t interpret this format because doing so making it verbose and potentially confusing for screen reader users. They may interpret this strikethrough announcement as being part of the message itself. As a general best practice, it’s recommended to avoid strikethrough format for accessibility reasons. Additionally, some sighted users may find it challenging to read. P3: Errors indicated by color alone Report If the user enters an invalid input to an edit field, an error message is displayed below the edit field. The only indication it is an error is that it is written with red text. To users unable to perceive red text, the error text is just another message on the screen and not necessarily an error. Resolution We recently updated the design for displaying errors in edit fields, but there seems to be some inconsistency in its implementation. To address this, we conducted an audit of our usage and updated several screens accordingly. While it’s a relatively minor change, it improves the intuitiveness of understanding error messages. Before <img loading="lazy" decoding="async" class="alignnone wp-image-17499" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image8.png?w=395" alt="Error text with red color is shown below an edit field." width="302" height="225" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image8.png 395w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image8.png?resize=380,283 380w" sizes="auto, (max-width: 302px) 100vw, 302px" /> After <img loading="lazy" decoding="async" class="alignnone wp-image-17498" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image6.png?w=395" alt="Error text with red color and an error icon are shown below an edit field." width="302" height="225" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/image6.png 395w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/image6.png?resize=380,283 380w" sizes="auto, (max-width: 302px) 100vw, 302px" /> Conclusion Through the process of triaging and addressing VPAT issues, we have gained valuable insights. One crucial lesson learned is the importance of immediately announcing errors to screen reader users. This eliminates the need for them to navigate through screens to understand the outcome of their actions. We discovered that seemingly minor UI changes, such as adding visual icons to text, can have a significant impact on our user’s experience. Furthermore, we recognized the need to be more diligent in adding TalkBack custom actions for gestures that may not be easily discoverable, like drag-and-drop or swipe-to-dismiss. Collaborating with our a11y-minded design team greatly facilitated the decision-making process and issue resolution. Although not all Web Content Accessibility Guidelines (WCAG) standards can be directly applied to Android, the industry has long endeavored to establish platform-specific guidelines for mobile native apps. However, significant progress has yet to be made. Our VPAT analysis revealed that certain accessibility issues are not applicable to Android’s native patterns. To gain a deeper understanding, we dedicated time to testing native Google apps and discussing our decisions regarding the applicability of WCAG standards to Android. While the timing of our next VPAT assessment is yet to be determined, we are committed to continuously evaluating and improving the accessibility of Slack across all platforms. We hope you have gained valuable insights from this article. References: WCAG mapping WCAG standard Issue Description 4.1.2 Name, Role, Value (Level A) <img loading="lazy" decoding="async" class="alignnone wp-image-17483" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?w=128" alt=":p1-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Error messages are inaccessible. If the user enters an invalid input to an edit field and presses “Next” (or Submit), an error message is displayed, but the error is not communicated to users using screen readers. TalkBack should announce that it is in an error state when it receives focus, along with the error message so that users understand why it’s in error. 1.3.1 Info and Relationships (Level A) <img loading="lazy" decoding="async" class="alignnone wp-image-17483" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?w=128" alt=":p1-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p1-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Heading is not identified. The heading does not convey clear semantic information, making it difficult for screen reader users to quickly grasp the page’s structure and navigate efficiently. Defining headings enables assistive technology to convey the hierarchical organization of the content, enhancing accessibility and providing an alternative navigation method. 4.1.2 Name, Role, Value (Level A) <img loading="lazy" decoding="async" class="wp-image-17485 alignnone" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?w=128" alt=":p2-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Missing a11y label on EditField Some edit fields do not have permanent labels. It is labeled using placeholder text that is removed when the user enters text. Some users with cognitive difficulties may forget the purpose of the edit field without a permanently visible label. 4.1.2 Name, Role, Value (Level A) <img loading="lazy" decoding="async" class="wp-image-17485 alignnone" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?w=128" alt=":p2-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Incorrect number of items in list When TalkBack users have Always speak number of list items setting enabled and navigate through a list, TalkBack announces an incorrect number of items. 2.5.7 Dragging Movements (Level AA) <img loading="lazy" decoding="async" class="wp-image-17485 alignnone" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?w=128" alt=":p2-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Drag and drop in the workspace switcher is inaccessible. Workspaces in the workspace switcher can be rearranged, but it requires the user to select a workspace and drag it to the new location. People who lack the dexterity required to perform this action may have difficulties completing the task, or may not be able to complete the task at all. 1.3.1 Info and Relationships (Level A) <img loading="lazy" decoding="async" class="wp-image-17485 alignnone" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?w=128" alt=":p2-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p2-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" /> Strikethrough information not conveyed to screen reader users Users have the option to format text as strikethrough, but the strikethrough information is not conveyed to screen reader users. It is visually evident that the text has strikethrough, but it is read as normal text for screen reader users. 1.4.1 Use of Color (Level A) <img loading="lazy" decoding="async" class="alignnone wp-image-17484" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p3-new.png?w=128" alt=":p3-new:" width="20" height="20" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/p3-new.png 128w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p3-new.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/p3-new.png?resize=110,110 110w" sizes="auto, (max-width: 20px) 100vw, 20px" />Errors indicated by color alone If the user enters an invalid input to an edit field, an error message is displayed below the edit field. The only indication it is an error is that it is written with red text. To users unable to perceive red text, the error text is just another message on the screen and not necessarily an error. 2.1.1 Keyboard (Level A) 2.1.2 No Keyboard Trap (Level A) [Not planned] Keyboard navigation and focus
Build better software to build software better
We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until…
We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until far too late. We fixed this problem by combining modern, high-performance build tooling (Bazel) with classic software engineering principles. Here’s how we did it. Thinking About Build (and Code) Performance Imagine a simple application. It has a backend server that provides an API and data storage, and a frontend that presents the user interface. Like many modern applications, the frontend and backend are decoupled; they can be developed and delivered independently. The graph of this build looks like this: <img loading="lazy" decoding="async" width="1642" height="430" class="alignnone size-full wp-image-17605" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png" alt="An example service build graph. It shows a set of Python files building into a backend artifact and a set of TypeScript files building into a frontend artifact." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png 1642w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=640,168 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=768,201 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1280,335 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1536,402 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=380,100 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=800,210 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1160,304 1160w" sizes="auto, (max-width: 1642px) 100vw, 1642px" /> We represent the dependencies between the elements of the build, like source files and deployable artifacts, with arrows, forming a directed acyclic graph. Here, our backend depends on a collection of Python files, meaning that whenever a Python file changes, we need to rebuild the backend. Likewise, we need to rebuild our frontend whenever a TypeScript file changes — but not when a Python file does. Modeling our build as a graph of clearly-defined units of work lets us apply the same kind of performance optimizations we might use to speed up application code: Do less work. Store the results of expensive work you do so that you only have to do it once, trading off memory for time. Share the load. Spread the work you’re doing across more compute resources in parallel, so that it completes more quickly, trading off compute for time. Caching and Parallelization These techniques will look familiar to most engineers, and thinking about them through a code lens helps solidify how they apply to a build system. Take this Python example: def factorial(n): return n * factorial(n-1) if n else 1 Calculating a factorial can be very expensive. If we need to do it often, we’re going to run quite slowly. But intuitively, we know that the factorial of a number doesn’t change. This means we can apply strategy 1 to cache the results of this function. Then it only needs to run once for any given input. @functools.cache def factorial(n): return n * factorial(n-1) if n else 1 The cache stores our inputs (n) and maps them to the corresponding outputs (the return value of factorial()). n is the cache key: it functions like a source file in the build graph, with the output of the function corresponding to the built artifact. Refining our intuition, we can say that this function needs to have a few specific attributes for caching to work. It needs to be hermetic: it only uses the inputs it is explicitly given to produce a specific output value. And it needs to be idempotent: the output for any given set of inputs is always the same. Otherwise, caching is unsound and will have rather surprising effects. Once we have a cache in place, we want to maximize our hit rate: the fraction of times we call factorial() that are answered from the cache, rather than from doing the calculation. The way we define our units of work can help us keep the hit rate high. To illustrate that idea, let’s look at a more complex piece of code: @functools.cache def process_images( images: list[Image], transforms: list[Transform] ) -> list[Image]: .. This function has a cache attached to it, but the cache isn’t very effective. The function takes two collections of inputs: a list of images and a list of transforms to apply to each image. If the caller changes any of those inputs, say by adding one more Image to the list, they also change the cache key. That means we won’t find a result in the cache, and will have to do all of the work from the beginning. In other words, this cache is not very granular, and as a result has a poor hit rate. More effective code might look like this: def process_images( images: list[Image], transforms: list[Transform] ) -> list[Image]: new_images = [] for image in images: new_image = image for transform in transforms: new_image = process_image(new_image, transform) new_images.append(new_image) return new_images @functools.cache def process_image(image: Image, transform: Transform) -> Image: ... We move the caching to a smaller unit of work — the application of a single transform to a single image — and with a correspondingly smaller cache key. We can still retain the higher-level API; we just don’t cache there. When a caller invokes the higher-level API with a set of inputs, we can answer that request by only processing the combinations of Image and Transform that we haven’t seen before. Everything else comes from the cache. Our cache hit rate should both improve, along with our performance. Caching helps us do less work. Now let’s look at sharing the load. What if we fanned out image processing across multiple CPU cores with threads? def process_images_threaded( images: list[Image], transforms: list[Transform] ) -> list[Image]: with ThreadPoolExecutor() as executor: futures = [] for image in images: futures.append(executor.submit(process_images, [image], transforms)) # Returns images in any order! return [future.result() for future in futures.as_completed(futures)] Let’s suss out the qualities we need to be able to run our work in parallel, while also noting some key caveats. Like with caching, we need to rigorously and completely define the inputs and outputs for the work we want to parallelize. We need to be able to move those inputs and outputs across some type of boundary. Here, it’s a thread boundary, but it might as well be a process boundary or a network boundary to some other compute node. Units of work that we run in parallel might complete, or fail, in any order. Our code has to manage those outcomes, and clearly document what guarantees it offers. Here, we’re noting the caveat that images may not return in the same order. Whether or not that’s acceptable depends on the API. The granularity of our units of work also plays a role in how effectively we can parallelize, although there’s a lot of nuance involved. If our units of work are few in number but large in size, we won’t be able to spread them over as many compute resources. The right trade-off between task size and count varies wildly between problems, but it’s something we need to consider as we design our APIs. Transitioning to Build Performance If you’re a software engineer, you’ve probably applied those techniques many times to improve your code’s performance. Let’s apply some of those principles and intuitions to build systems. In a Bazel build, you define targets that form a directed acyclic graph. Each target has three critical elements: The files that are dependencies, or inputs, for this build step. The files that are outputs for this build step. The commands that transform the inputs into the outputs. (It’s a little more complicated, but that’s enough to get the gist for now). Let’s go back to that original sample application. <img loading="lazy" decoding="async" width="1642" height="430" class="alignnone size-full wp-image-17605" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png" alt="An example service build graph. It shows a set of Python files building into a backend artifact and a set of TypeScript files building into a frontend artifact." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png 1642w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=640,168 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=768,201 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1280,335 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1536,402 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=380,100 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=800,210 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1160,304 1160w" sizes="auto, (max-width: 1642px) 100vw, 1642px" /> The Bazel target definitions might look something like this pseudocode: python_build( name = "backend", srcs = ["core/http.py", "lib/options.py", "data/access.py"], outs = ["backend.tgz"], cmd = "python build.py", ) ts_build( name = "frontend", srcs = ["cms/cms.ts", "collab/bridge.ts", "editing/find.ts"], outs = ["frontend.tgz"], cmd = "npm build" ) You can think about a build target like defining (but not invoking!) a function. Notice some themes carrying over from the Python examples above? We exhaustively define the inputs (srcs) and outputs (outs) for each build step. We can think of the srcs of a target (and the transitive srcs of any targets that build them, which we’ll see more later) as the inputs to a function we wish to cache. Because Bazel builds in a sandbox, the cmds that transform inputs to outputs only get to touch the input files we declare. And because our inputs and outputs are simply on-disk files, we solve the problem of moving them across a boundary: they just get copied! Lastly, we have to make Bazel a promise: our build steps’ cmds are in fact idempotent and hermetic. When we do those things, we get some extraordinary capabilities for free. Bazel automatically caches the outcomes of our build actions. When the inputs to a target don’t change, the cached output is used — no build cost! Bazel distributes build actions across as many CPU cores as we allow, or even across multiple machines in a build cluster. Bazel only executes those actions we need for the output artifacts we want. To put it another way, we always run the least amount of work needed for the output we want. Just like in our code examples, we’ll get the most out of Bazel’s magic when we have a well-defined dependency graph, composed of build units that are idempotent, hermetic, and granular. Those qualities allow Bazel’s native caching and parallelization to deliver great speed boosts. That’s plenty of theory. Let’s drill down on the actual problem. Why Quip and Canvas are Harder Here’s the trick: Quip and Canvas are much, much more complex than the simple example we looked at before. Here’s a real diagram we drew to understand how our build worked. Don’t worry about reading all the details — we’ll use a schematic version as we dig into the problem and our solutions. When we analyzed the graph, we discovered some critical flaws that meant we did not have the characteristics we needed to get a speed boost from Bazel: A directed acyclic dependency graph →The graph wasn’t well defined, and in fact contained cycles! Idempotent, hermetic, well-sized units of work. → Build execution units were huge, not all were idempotent, and hermeticity was a challenge because many build steps mutated the working directory. Granular cache keys to keep cache hit rate high. → Our build was so interconnected that our cache hit rate was zero. Imagine that every cached “function” we tried to call had 100 parameters, 2-3 of which always changed. If we had started by throwing Bazel at the build, it would have been ineffective. With a cache hit rate of zero, Bazel’s advanced cache management wouldn’t have helped, and Bazel parallelization would have added little to nothing over the ad-hoc parallelization already present in the build code. To get the Bazel magic, we needed to do some engineering work first. Separating Concerns Our backend code and our build code were deeply intertwined. Without a Bazel-like build system, we’d defaulted to using in-application frameworks to orchestrate the various steps within our build graph. We managed parallelization using Python’s multiprocessing strategy and the async routines built into our core codebase. Python business logic pulled together the Protobuf compilation and built Python and Cython artifacts. Lastly, tools like tsc and webpack, orchestrated by more Python scripts, transformed TypeScript and Less into the unique frontend bundle format used across Slack Canvas and Quip’s desktop and web applications. As we began to unpick that Gordian knot, we zeroed in on how the union of backend and build code distorted the graph of our frontend build. Here’s an easier-to-digest representation of that graph. <img loading="lazy" decoding="async" class="alignnone wp-image-17595 size-full" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png" alt="A graph of a software build. It shows a group of Python files at the top, feeding into a Python application build. The Python build and a set of TypeScript files are then the inputs to a TypeScript build, producing a set of frontend bundles." width="1475" height="906" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png 1475w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=640,393 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=768,472 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=1280,786 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=380,233 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=800,491 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-e1759865828860.png?resize=1160,713 1160w" sizes="auto, (max-width: 1475px) 100vw, 1475px" /> Notice how large the dependency tree is “above” our frontend bundles. It includes not just their TypeScript source and build process, but also the entire built Python backend! Those Python sources and artifacts are transitive sources for each and every frontend bundle. That means that not just a TypeScript change, but also every single Python change, alters the cache key for the frontend (one of those hundred parameters we alluded to earlier), requiring an expensive full rebuild. The key to this build problem is that dependency edge between the Python application and the TypeScript build. <img loading="lazy" decoding="async" width="554" height="525" class="alignnone size-full wp-image-17600" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png" alt="A zoomed-in view of the build graph from above. It focuses on the edges between the Python build and application and the TypeScript build." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png 554w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation-1-e1759866943466.png?resize=380,360 380w" sizes="auto, (max-width: 554px) 100vw, 554px" /> That edge was costing us an average of 35 minutes per build — more than half the total cost! — because every change was causing a full backend and frontend rebuild, and the frontend rebuild was especially expensive. As we worked through this problem, we realized that the performance cost was not solely a build issue. It was a symptom of our pervasive failure to separate our concerns across the whole application: backend, frontend, and build code. We had couplings: between our backend and our frontend between our Python and TypeScript infrastructures and toolchains between our build system and our application code Beyond making the build performance quantitatively worse, those couplings had qualitative impacts across our software development lifecycle. Engineers couldn’t reason about the blast radius of changes they made to the backend, because it might break the frontend too. Or the build system. Or both. And because our build took an hour, we couldn’t give them early warnings by running builds at the Pull Request level. The lack of health signals, and the difficulty of reasoning through potential consequences, meant that we’d break our build and our main branch far too often, and through no fault of our engineers. Once we understood the problem through the lens of separation of concerns, it became clear we couldn’t succeed with changes to the build system alone. We had to cleanly sever the dependencies between our frontend and backend, our Python and TypeScript, our application and our build. And that meant we had to invest a lot more time than we’d originally planned. Over several months, we painstakingly unraveled the actual requirements of each of our build steps. We rewrote Python build orchestration code in Starlark, the language Bazel uses for build definitions. Starlark is a deliberately constrained language whose limitations aim at ensuring builds meet all the requirements for Bazel to be effective. Building in Starlark helped us enforce a full separation from application code. Where we needed to retain Python scripts, we rewrote them to remove all dependencies save the Python standard library: no links to our backend code, and no additional build dependencies. We left out all of the parallelization code, because Bazel handles that for us. We’ll revisit that paradigm below in thinking about layering. The complexity of the original build code made it challenging to define “correct” behavior. Our build code mostly did not have tests. The only criterion for what was correct was what the existing build system produced under a specific configuration. To reassure ourselves, and to instill confidence in our engineers, we built a tool in Rust to compare an artifact produced by the existing process with one produced by our new code. We used the differences to guide us to points where our new logic wasn’t quite right, and iterated, and iterated more. That work paid off when we could finally draw a new build graph: <img loading="lazy" decoding="async" width="1843" height="947" class="alignnone size-full wp-image-17596" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png" alt="A software build graph. A set of Python source files feed into a Python application build. This build is shown separately from a frontend build, where a set of TypeScript files is shown to be built into frontend bundles." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png 1843w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=640,329 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=768,395 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1280,658 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1536,789 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=380,195 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=800,411 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation1-e1759865982755.png?resize=1160,596 1160w" sizes="auto, (max-width: 1843px) 100vw, 1843px" /> We’d severed all three of those key couplings, and taken off the table any concern that a Python change might break the build or alter the frontend output. Our build logic was colocated with the units it built in BUILD.bazel files, with well-defined Starlark APIs and a clean separation between build code and application code. Our cache hit rate went way up, because Python changes no longer formed part of the cache key for TypeScript builds. The outcome of this work was a substantial reduction in build time: if the frontend was cached, we could build the whole application in as little as 25 minutes. That’s a big improvement, but still not enough! Designing for Composition with Layering Once we severed the backend ←→ frontend coupling, we took a closer look at the frontend build. Why did it take 35 minutes, anyway? As we unraveled the build scripts, we found more separation-of-concerns challenges. Our frontend builder was trying hard to be performant. It took in a big set of inputs (TypeScript source, LESS and CSS source, and a variety of knobs and switches controlled by environment variables and command-line options), calculated the build activity it needed to do, parallelized that activity across a collection of worker processes, and marshalled the output into deployable JavaScript bundles and CSS resources. Here’s a sketch of this part of the build graph. <img loading="lazy" decoding="async" width="1482" height="746" class="alignnone size-full wp-image-17602" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png" alt="A build graph shows TypeScript and LESS source files feeding into a single TypeScript and CSS Build node, along with environment variables and switches. The output of the build node is a set of frontend bundles. The build process interacts with a set of worker processes." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png 1482w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=640,322 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=768,387 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=1280,644 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=380,191 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=800,403 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation4-e1759867199624.png?resize=1160,584 1160w" sizes="auto, (max-width: 1482px) 100vw, 1482px" /> Like much of what we discovered during this project, this strategy represented a set of reasonable trade-offs when it was written: it was a pragmatic attempt to speed up one piece of the build in the absence of a larger build framework. And it was faster than not parallelizing the same work! There are two key challenges with this implementation. Like our process_image() example above, our cacheable units were too large. We took in all the sources and produced all the bundles. What if just one input file changed? That altered the cache key, and we had to rebuild everything. Or what if we wanted to build exactly one bundle, to satisfy a requirement elsewhere in our build process? We were out of luck; we had to depend on the whole shebang. We’re parallelizing the work across processes, which is good — unless we could be parallelizing across machines, machines with lots more CPU cores. If we have those resources available, we cannot use them here. And we’re actually making Bazel less effective at its core function of parallelizing independent build steps: Bazel and the script’s worker processes are contending for the same set of resources. The script might even be parallelizing work that Bazel already knows it does not need! In both these respects, it was tough for us to compose the build functionality in new ways: either a different form of orchestrating multiple bundle builds, or a different parallelization strategy. This was a layering violation. <img loading="lazy" decoding="async" width="618" height="602" class="alignnone size-full wp-image-17599" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png" alt="A diagram of layered functionality. At the base is the OS, followed by Parallelization and Language Runtime, then Orchestration and App Core, and finally Logic. A red outline cuts out a unit of logic plus the orchestration and parallelization layers." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png 618w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation3-e1759866526521.png?resize=380,370 380w" sizes="auto, (max-width: 618px) 100vw, 618px" /> If we draw, as here, layers of capability from the operating system up to the application layer, we can see that the boundary of our builder is cutting across layers. It combines business logic, significant parts of a task orchestration framework, and parallelization. We really just wanted the top layer — the logic — so that we could re-compose it in a new orchestration context. We ended up with a work orchestrator (the builder) inside a work orchestrator (Bazel), and the two layers contending for slices of the same resource pool. To be more effective, we really just needed to do less. We deleted a lot of code. The new version of the frontend builder was much, much simpler. It didn’t parallelize. It had a much smaller “API” interface. It took in one set of source files, and built one output bundle, with TypeScript and CSS processed independently. <img loading="lazy" decoding="async" width="714" height="784" class="alignnone size-full wp-image-17603" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png" alt="A build graph shows parallel tracks, with TypeScript source flowing through a build to produce JavaScript source, and Less and CSS source flowing through a CSS build to form CSS output. The outputs are then combined into a frontend bundle." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png 714w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png?resize=640,703 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation5-e1759867349405.png?resize=380,417 380w" sizes="auto, (max-width: 714px) 100vw, 714px" /> This new builder is highly cacheable and highly parallelizable. Each output artifact can be cached independently, keyed only on its direct inputs. A bundle’s TypeScript build and CSS build can run in parallel, both against one another and against the builds of other bundles. And our Bazel logic can make decisions about scope (one bundle? two? all?), rather than trying to manage a build of all of the bundles. See the resonance, again, with our example process_images() API? We’ve created granular, composable units of work, and that’s dramatically improved our ability to parallelize and cache. We’ve also separated the concerns of our business logic and its orchestration, making it possible for us to re-compose the logic within our new Bazel build. This change gave us some really nice outcomes: Because bundle builds and TypeScript and CSS builds can be cached independently of one another, our cache hit rate went up. Given enough resources, Bazel could parallelize all of the bundle builds and CSS compilation steps simultaneously. That bought us a nice reduction in full-rebuild time. As an added benefit, we’re no longer running our own parallelization code. We’ve delegated that responsibility to Bazel. Our build script has a single concern: the business logic around compiling a frontend bundle. That’s a win for maintainability. Outcomes and Takeaways We made our build a lot faster. Faster builds lead to tighter cycle times for engineers, quicker incident resolutions, and more frequent releases. After applying these principles across our whole build graph, we came away with a build that’s as much as six times faster than when we started. <img loading="lazy" decoding="async" width="1133" height="717" class="alignnone size-full wp-image-17597" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png" alt="A diagram shows a "before" build time of 60 minutes for all cases. The "after" time shows three cases, 10 minutes in the best case (cached and parallelized); 12 minutes in the average case (mostly cached and parallelized); and 30 minutes in the worst cases (a cache miss)." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png 1133w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=640,405 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=768,486 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=380,240 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation2-e1759866173684.png?resize=800,506 800w" sizes="auto, (max-width: 1133px) 100vw, 1133px" /> From a qualitative angle, we took away the key principle that software engineering principles apply to the whole system. And the whole system is more than our application code. It’s also our build code, our release pipeline, the setup strategies for our developer and production environments, and the interrelations between those components. So here’s our pitch to you, whether you’re writing application code, build code, release code, or all of the above: separate concerns. Think about the whole system. Design for composability. When you do, every facet of your application gets stronger — and, as a happy side effect, your build will run a lot faster too.
Advancing Our Chef Infrastructure: Safety Without Disruption
Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating…
Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating the limitations around Chef searches. If you haven’t had a chance to read that post yet, I highly recommend checking it out first to get the full context for this post. At Slack, keeping our service reliable is always the top priority. In my last post, I talked about the first phase of our work to make Chef and EC2 provisioning safer. With that behind us, we started looking at what else we could do to make deploys even safer and more reliable. One idea we explored was moving to Chef Policyfiles. That would have meant replacing roles and environments and asking dozens of teams to change their cookbooks. In the long run, it might have made things safer, but in the short term it would have been a huge effort and added more risk than it solved. So instead, this post is about the path we chose: improving our existing EC2 framework in a way that doesn’t disrupt cookbooks or roles, while still giving us more safety in our Chef deployments. Splitting Chef Environments Previously, each instance had a cron job that triggered a Chef run every few hours on a set schedule. These scheduled runs were primarily for compliance purposes — to ensure our fleet remained in a consistent and defined configuration state. To reduce risk, the timing of these cron jobs was staggered across availability zones, helping us avoid running Chef on all nodes simultaneously. This strategy gave us a buffer: if a bad change was introduced, it would only impact a subset of nodes initially, giving us a chance to catch and fix the issue before it spread. However, this approach had a critical limitation. With a single shared production environment, even if Chef wasn’t running everywhere at once, any newly provisioned nodes would immediately pick up the latest (possibly bad) changes from that shared environment. This became a significant reliability risk, especially during large scale-out events, where dozens or hundreds of nodes could start up with a broken configuration. To address this, we split our single production Chef environment into multiple buckets: prod-1, prod-2, …, prod-6. Service teams could still launch instances as “prod,” but behind the scenes we mapped each instance to one of these more specific environments based on its Availability Zone. That way, new nodes no longer pulled from one global source of truth – they were evenly distributed across isolated environments, which added an extra layer of safety and resilience. At Slack, our base AMIs include a tool called Poptart Bootstrap, which is baked in during the AMI build process. This tool runs via cloud-init at instance boot time and is responsible for tasks such as creating the node’s Chef object, setting up any required DNS entries, and posting a success or failure message to Slack (with customisable channels based on the team that owns the node). To support the environment split, we extended Poptart Bootstrap to include logic that inspects the node’s AZ ID and assigns it to one of the numbered production Chef environments. This change allowed us to stop pointing all production nodes at a single shared Chef environment and instead spread them across multiple environments. Previously, changing the cookbook versions in the single prod environment affected the entire fleet. With this new bucketed approach, we gained the flexibility to update each environment independently. Now, a change to a single environment only impacts nodes in specific AZs, significantly reducing risk and blast radius during deployments. <img loading="lazy" decoding="async" width="438" height="597" class="alignnone size-medium wp-image-17612" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png?w=438" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png 438w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/env_flow.png?resize=380,518 380w" sizes="auto, (max-width: 438px) 100vw, 438px" /> With this approach, we promote the latest cookbook changes to the sandbox environments at the top of the hour. These changes are then managed by a Kubernetes cron job and rolled out to the dev environments at the top of the hour, and finally begin rolling out to production environments starting at 30 minutes past the hour. prod-1 is treated as our canary production environment. It receives updates every hour (assuming there have been new changes merged into the cookbooks) to ensure that changes are regularly exercised in a real production setting – alongside our sandbox and dev environments. This enables us to catch production-impacting issues early and in smaller, safer increments. Changes to prod-2 through prod-6 are rolled out using a release train model. Before a new version is deployed to prod-2, we ensure the previous version has successfully progressed through all production environments up to prod-6. This staggered rollout minimises blast radius and gives us the opportunity to catch regressions earlier in the release train. If we waited for a version to pass through all environments before updating prod-1 – as we do with prod-2 onward – we’d end up testing artifacts with large, cumulative changes. By contrast, updating prod-1 frequently with the latest version allows us to detect issues closer to when they were introduced. Here’s an example showing how artifact version A moves through each environment in this release cycle: Hour X:00 Sandbox ---> Version A (Latest) Dev ---> Version A (Latest) Prod-1 (no change - currently at version Z) Prod-2 (no change - currently at version Z) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour X:30 (Now that Prod-2 to Prod-6 are all on the same version, we’ll begin updating them again) Sandbox (no change - currently at version A) Dev (no change - currently at version A) Prod-1 ---> Version A (dev version) Prod-2 ---> Version A (prod-1 version) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called B. Hour (X + 1):00 Sandbox ---> Version B (Latest) Dev ---> Version B (Latest) Prod-1 (no change - currently at version A) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version Z) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 1):30 Sandbox (no change - currently at version B) Dev (no change - currently at version B) Prod-1 ---> Version B (dev version) Prod-2 (no change - currently at version A) Prod-3 ---> Version A (prod-2 version) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called C. Hour (X + 2):00 Sandbox ---> Version C (Latest) Dev ---> Version C (Latest) Prod-1 (no change - currently at version B) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version Z) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 2):30 Sandbox (no change - currently at version C) Dev (no change - currently at version C) Prod-1 ---> Version C (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 ---> Version A (prod-3 version) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called D. Hour (X + 3):00 Sandbox ---> Version D (Latest) Dev ---> Version D (Latest) Prod-1 (no change - currently at version C) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version Z) Prod-6 (no change - currently at version Z) Hour (X + 3):30 Sandbox (no change - currently at version D) Dev (no change - currently at version D) Prod-1 ---> Version D (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 ---> Version A (prod-4 version) Prod-6 (no change - currently at version Z) Someone committed a change, resulting in the creation of a new artifact called E. Hour (X + 4):00 Sandbox ---> Version E (Latest) Dev ---> Version E (Latest) Prod-1 (no change - currently at version D) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version Z) Hour (X + 4):30 Sandbox (no change - currently at version E) Dev (no change - currently at version E) Prod-1 ---> Version E (dev version) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 ---> Version A (prod-5 version) Someone committed a change, resulting in the creation of a new artifact called F. Hour (X + 5):00 Sandbox ---> Version F (Latest) Dev ---> Version F (Latest) Prod-1 (no change - currently at version E) Prod-2 (no change - currently at version A) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version A) Now that Prod-2 to Prod-6 are all on the same version, we’ll begin updating them again and this cycle will continue till the end of time. Hour (X + 5):30 Sandbox (no change - currently at version F) Dev (no change - currently at version F) Prod-1 ---> Version F (dev version) Prod-2 ---> Version F (prod-1 version) Prod-3 (no change - currently at version A) Prod-4 (no change - currently at version A) Prod-5 (no change - currently at version A) Prod-6 (no change - currently at version A) As you can see above, changes now take a bit longer to roll out across all our production nodes. However, this delay provides valuable time between deployments across different availability zones, allowing us to catch and address any issues before problematic changes are fully propagated. By introducing split production environments (e.g., prod-1 to prod-6), we’ve been able to sidestep the old problem of new nodes pulling in a bad config the moment they came online. Now, each node is tied to the environment for its AZ, and changes roll out gradually instead of everywhere at once. That means if something goes wrong in one AZ, we can keep scaling safely in the others while we fix the issue. It’s a much more resilient setup than before – we’ve taken away the single point of failure and built in guardrails that make the whole system safer and more predictable. What changed in the way we trigger Chef? Now that we have multiple production Chef environments, each one receives updates at different times. For example, if a version is currently in the middle of being rolled out, the next version must wait until that rollout is fully completed across all production environments. On the other hand, when there are no updates in progress, a new version can be rolled out to all environments more quickly. Because of this variability, triggering Chef via a fixed cron schedule is no longer practical – we can’t reliably predict when a given Chef environment will receive new changes. As a result, we’ve moved away from scheduled runs and instead built a new service that triggers Chef runs on nodes based on signals. This service ensures Chef only runs when there are actual updates available, improving both safety and efficiency. Chef Summoner <img loading="lazy" decoding="async" width="768" height="768" class="alignnone size-medium wp-image-17614" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?w=640" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=160,160 160w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=640,640 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=80,80 80w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=110,110 110w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_logo.png?resize=380,380 380w" sizes="auto, (max-width: 768px) 100vw, 768px" /> If you remember from my previous blog post, we built a service called Chef Librarian that watches for new Chef cookbook artifacts and uploads them to all of our Chef stacks. It also exposes an API endpoint that allows us to promote a specific version of an artifact to a given environment. We recently enhanced Chef Librarian to send a message to an S3 bucket whenever it promotes a version of an artifact to an environment. The contents of that S3 bucket look like this: agunasekara@z-ops-agunasekara-iad-dinosaur:~ >> s3-tree BUCKET_NAME BUCKET_NAME └── chef-run-triggers ├── basalt │ ├── ami-dev │ ├── ami-prod │ ├── dev │ ├── prod │ ├── prod-1 │ ├── prod-2 │ ├── prod-3 │ ├── prod-4 │ ├── prod-5 │ ├── prod-6 │ └── sandbox └── ironstone ├── ami-dev ├── ami-prod ├── dev ├── prod ├── prod-1 ├── prod-2 ├── prod-3 ├── prod-4 ├── prod-5 ├── prod-6 └── sandbox Under the chef-run-triggers key, we maintain a nested structure where each key represents one of our Chef stacks. Within each stack key, there are additional keys for each environment name. Each of these environment keys contains a JSON object with contents similar to the example below: { "Splay": 15, "Timestamp": "2025-07-28T02:02:31.054989714Z", "ManifestRecord": { "version": "20250728.1753666491.0", "chef_shard": "basalt", "datetime": 1753666611, "latest_commit_hash": "XXXXXXXXXXXXXX", "manifest_content": { "base_version": "20250728.1753666491.0", "latest_commit_hash": "XXXXXXXXXXXXXX", "author": "Archie Gunasekara <agunasekara@slack-corp.com>", "cookbook_versions": { "apt": "7.5.23", ... "aws": "9.2.1" }, "site_cookbook_versions": { "apache2": "20250728.1753666491.0", ... "squid": "20250728.1753666491.0" } }, "s3_bucket": "BUCKET_NAME", "s3_key": "20250728.1753666491.0.tar.gz", "ttl": 1756085811, "upload_complete": true } } Next, we built a service called Chef Summoner. This service runs on every node at Slack and is responsible for checking the S3 key corresponding to the node’s Chef stack and environment. If a new version is present in the signal sent out by Chef Librarian, the service reads the configured splay value and schedules a Chef run accordingly. The splay is used to stagger Chef runs so that not all nodes in a given environment and stack try to run Chef at the same time. This helps avoid spikes in load and resource contention. We can also customize the splay depending on our needs – for example, when we trigger a Chef run using a custom signal from Librarian and want to spread the runs out more intentionally. However, if no changes are merged and no new Chef artifacts are built, Librarian has nothing to promote, and no new signals are sent out. Despite that, we still need to ensure Chef runs at least once every 12 hours to maintain compliance and ensure nodes stay in their expected configuration state. So, Chef Summoner will also run Chef at least once every 12 hours, even if no new signals have been received. The Summoner service keeps track of its own state locally – this includes things like the last run time and the artifact version used – so it can compare any new signals with the most recent run and determine whether a new Chef run is required. The overall flow looks like this: <img loading="lazy" decoding="async" width="630" height="1052" class="alignnone size-medium wp-image-17616" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png?w=630" alt="" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png 630w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/summoner_flow.png?resize=380,635 380w" sizes="auto, (max-width: 630px) 100vw, 630px" /> Now that Chef Summoner is the primary mechanism we rely on to trigger Chef runs, it becomes a critical piece of infrastructure. After a node is provisioned, subsequent Chef runs are responsible for keeping Chef Summoner itself up to date with the latest changes. But if we accidentally roll out a broken version of Chef Summoner, it may stop triggering Chef runs altogether – making it impossible to roll out a fixed version using our normal deployment flow. To mitigate this, we bake in a fallback cron job on every node. This cron job checks the local state that Chef Summoner stores (e.g., last run time and artifact version) and ensures Chef has been run at least once every 12 hours. If the cron job detects that Chef Summoner has failed to run Chef in that timeframe, it will trigger a Chef run directly. This gives us a recovery path to push a working version of Chef Summoner back out. In addition to this safety net, we also have tooling that allows us to trigger ad hoc Chef runs across the fleet or a subset of nodes when needed. What’s Next? All of these recent changes to our EC2 ecosystem have made rolling out infrastructure changes significantly safer. Teams no longer have to worry about a bad update impacting the entire fleet. While we’ve come a long way, the platform is still not perfect. One major limitation is that we still can’t easily support service-level deployments. In theory, we could create a dedicated set of Chef environments for each service and promote artifacts individually – but with the hundreds of services we operate at Slack, this quickly becomes unmanageable at scale. With that in mind, we’ve decided to mark our legacy EC2 platform as feature complete and move it into maintenance mode. In its place, we’re building a brand-new EC2 ecosystem called Shipyard, designed specifically for teams that can’t yet move to our container-based platform, Bedrock. Shipyard isn’t just an iteration of our old system – it’s a complete reimagining of how EC2-based services should work. It introduces concepts like service-level deployments, metric-driven rollouts, and fully automated rollbacks when things go wrong. We’re currently building a Shipyard and targeting a soft launch this quarter, with plans to onboard our first two teams for testing and feedback. I’m excited to share more about Shipyard in my next blog post – stay tuned!
Deploy Safety: Reducing customer impact from change
It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing…
It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing our rate of change to meet business needs. Defining the problem System requirements change as businesses evolve whether that be due to changes in customer expectations, load & scale or response to business needs. In this case analysis showed two critical trends: Slack has become more mission critical for our customers, increasing expectations of reliability for the product as a whole and for specific features each company relies on. The increasing majority (73%) of customer facing incidents were triggered by Slack-induced change, particularly code deploys. We use our incident management process heavily and frequently at Slack to ensure we have a strong and rapid response to issues of all shapes and sizes based on impact and severity. A portion of those incidents are customer impacting, many of which showed variable impact depending on which features they used in the product. Change-triggered incidents were occurring in a wide variety of systems and deployment processes. Finally, we also received customer feedback that interruptions became more disruptive after about 10 minutes – something they would treat as a “blip” – and this would continue to reduce with the introduction of Agentforce in 2025. All this was occurring in a software engineering environment with 100s of internal services and many different deployment systems & practices. In the past, our approach to reliability often zeroed in on individual deploy systems or services. This led to manual change processes that bogged down our pace of innovation. The increased time and effort required for changes not only dampened engineering morale but also hindered our ability to swiftly meet business objectives. This led us to set some initial “North Star” goals across all deployment methods for our highest importance services: Reducing impact time from deployments Automated detection & remediation within 10 minutes Manual detection & remediation within 20 minutes Reducing severity of impact Detect problematic deployments prior to reaching 10% of the fleet Maintaining Slack’s development velocity The initial North Star goals evolved and expanded into a Deploy Safety Manifesto that now applies to all Slack’s deployment systems & processes. These goals encourage implementing automated system improvements, safety guardrails and cultural changes. The metric Programs are run to invoke change and change must be measured. Selecting a metric that serves as an imperfect analog has continued to be a constant source of discussion. The North Star goals could be measured against engineering efforts, but were not a direct measure of the end goal: improving customer sentiment of their experience of Slack’s reliability. What we needed was a metric that was a reasonable (though imperfect) analog of customer sentiment. We had the internal data to design such a metric. The Deploy Safety metric: Hours of customer impact from high severity and selected medium severity change-triggered incidents. What does “selected” mean? That seems pretty ambiguous. We found that the incident dataset didn’t exactly match what we needed. Severity levels at Slack convey current or impending customer impact rather than the final impact which often requires careful post-hoc analysis. This means we need to filter medium severity incidents based on a relevant level of customer impact. The on-going discussion around what metric to use relates to the semi-loose connection between: Customer sentiment <-> Program Metric <-> Project Metric They’re all connected, but it’s challenging to know for a specific project how much it is going to move the top line metric. How much does the Program metric reflect individual customer experience? This flow is especially difficult for engineers who prefer a more concrete feedback loop with hard data – i.e., “How much does my work or concept change customer sentiment?” Important criteria for designing the Deploy Safety metric: Measure results Understand what is measured (real vs analog) Consistency in measurement, especially subjective portions Continually validate the measurement matches customer sentiment with the leaders having the direct conversations with customers Which projects to invest in? The two largest unknowns we had at the start of the program were predicting which projects would have the highest likelihood of success given multiple sources of incidents and when each project would deliver impact. Incident data is a trailing dataset meaning that there is a time delay before we know the result. On top of this, customers were experiencing pain right now. This influenced our investment strategy to be: Invest widely initially and bias for action Focus on areas of known pain first Invest further in projects or patterns based on results Curtail investment in the least impactful areas Set a flexible shorter-term roadmap which may change based on results. Our goals directed us to look for projects that would therefore influence one or more the following: Detection of an issue earlier in the deployment process Improved automatic remediation time Improved manual remediation time Reduced issue severity through design of isolation boundaries (blast radius control) We attributed Webapp backend as the largest source of change-triggered incidents. The following is an example investment flow to address our Webapp backend deployments. A quarter of work to engineer automatic metric monitoring Another quarter to confirm customer impact alignment via automatic alerts and manual rollback actions Investing in automatic deployments and rollback Proving success with many automatic rollbacks keeping customer impact below 10 minutes Further investment to monitor additional metrics and invest in manual rollback optimisations Investing in a manual Frontend rollback capability Aligned further investment toward Slack’s centralised deployment orchestration system inspired by ReleaseBot and the AWS Pipelines deployment system to unify the use of metrics-based deployments with automatic remediation beyond Slack Bedrock / Kubernetes to many other deployment systems Achieved success criteria: Webapp backend, frontend, and a portion of infra deployments are now significantly safer and showing continual improvement quarter-over-quarter This pattern of trying something, finding success, then iterating + copying the pattern to other systems continues to serve us well. There have been too many projects to list, some more successful (e.g., faster Mobile App issue detection) and others where the impact hasn’t been as noticeable. In some cases we had reduced impact due to the greater success of automation over manual remediation improvements. It’s very important to note that projects that didn’t have the desired impact are not failures, they’re a critical input to our success through guiding investment and understanding which areas are of greater value. Not all projects will be as impactful, and this is by design. Results <img loading="lazy" decoding="async" class="wp-image-17516 size-medium" src="https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?w=640" alt="Graph showing a 90% decrease in customer impact hours from Feb 2024 to Jan 2025 and highlighting the drop after automatic rollbacks were introduced Apr 2024" width="639" height="602" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png 960w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=640,603 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=768,723 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=380,358 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/05/Screenshot-2025-04-29-at-4.30.19 pm.png?resize=800,753 800w" sizes="auto, (max-width: 639px) 100vw, 639px" /> Customer impact by quarter vs progressive targets Please note: our internal metric tracking of “impact” is much more granular and sensitive than disruptions that might qualify for the Slack Status Site. The bar chart above tells quite a story of non-linear progress and the difficulties with using trailing metrics based on waiting for incident occurrence. After the first quarter of work we had already seen improvement, but hadn’t yet deployed any changes other than communicating with the engineering teams about the program. Once projects started to deliver changes, we then had the peak quarter of impact between February and April of 2024. Was this working? There was still confidence that the work we were doing was going to have the desired impact based on the early results from Webapp backend metrics based deploy alerts with manual remediation – the peak of impact in 2024 would have otherwise been even higher. What we needed was automatic instead of manual remediation. Once automatic rollbacks were introduced we observed dramatic improvement in results. We’ve continued to experience a 3-6 month lag time to observe each project’s full impact. Over time the target number has been continually adjusted. Due to the metric requiring trailing incident data, the target was set based on expected potential results through the work delivered the previous quarter. This was notably different from expecting to be able to measure results as soon as work is delivered. We’ve reduced our target number again for 2025 and expect lower customer impact with a focus on mitigating the risk of infrequent spikes through deployment system consistency. Lessons learned This program required continual learning and iteration. We learned many important lessons. Each were critical for a successful program: Ensuring prioritisation and alignment Executive reviews every 4-6 weeks to ensure continued alignment and to seek support where needed High-level priority in company/engineering goals (OKR, V2MOM etc.) Solid support from executive leadership with active project alignment, encouragement, and customer feedback (huge thanks to SVP Milena Talavera, SVP Peter Secor, and VP Cisco Vila) Patience with trailing metrics and faith that you have the right process even when some projects don’t succeed Using a measurement with multiple months of delay from work delivery will need patience. Gather metrics to know if the improvement is at least functioning well (e.g., issue detection) whilst waiting for full results. Faith that you’ve made the best decisions you can with the information you have at the time and the agility to change the path once results are confirmed. Folks will delay adoption of a practice until they’re familiar or comfortable with it They’re worried they’ll make the problem worse if they follow an unknown path even with the reassurance it is better and is recommended. Provide direct training (multiple times if necessary) to many groups to help them become comfortable with the new tooling – “Just roll back!” Continual improvement of manual rollback tooling in response to experience during incidents. Use the tooling often, not just for the infrequent worst case scenarios. Incidents are stressful and we found that without frequent use to build fluency, confidence, and comfort the processes and tools won’t become routine. It would be as if you didn’t build the tool/capability in the first place. Direct outreach to engineering teams is critical The Deploy Safety program team engaged directly with individual teams to understand their systems and processes, provide improvement guidance, and encourage innovation and prioritization. (A huge thanks to Petr Pchelko & Harrison Page for so much of this work.) Not all teams and systems are the same. Some teams know their areas of pain well and have ideas, others want to improve but need additional resources. Keep the top line metric as consistent as possible Pick a metric, be consistent, and make refinements based on validation of results It’s very easy to burn time on deliberating the best metric (there isn’t a perfect one) Maintain consistent, direct communication with engineering staff This is an area we’re working to improve on Management understood prioritisation and results and were well aligned as a group. However, it hasn’t always been clear to general engineering staff revealing an opportunity for better alignment. Into the future Trust is the #1 value for Salesforce and Slack. As reliability is a key portion of that trust we intend to have ongoing investment into Deploy Safety at Slack. Improvements in automatic metrics based deployments & remediation to keep the positive trend going More consistency in our use of Deploy Safe processes for all deployments at Slack to mitigate unexpected or infrequent spikes of customer impact Change of the program’s scope to include migrating remaining manual deploy processes to code-based deploys using Deploy Safe processes Examples of some of the ongoing projects in this space include: Centralised deployment orchestration tooling expansion into other infrastructure deployment patterns: EC2, Terraform and many more Automatic rollbacks for Frontend Metric quality improvements – do we have the right metrics for each system/service? AI metric-based anomaly detection Further rollout of AI generated pre-production tests Acknowledgements Many thanks to all the Deploy Safety team over this first year and a half: Dave Harrington, Sam Bailey, Sreedevi Rai, Petr Pchelko, Harrison Page, Vani Anantha, Matt Jennings, Nathan Steele, Sriganesh Krishnan. And a huge huge thanks to the too many to name teams that have picked up Deploy Safety projects large and small to improve the product experience for our customers.
As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than…
As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than proactive threat detection and prevention. Organizations urgently need solutions that dramatically compress the detection-to-response window to regain a defensive advantage. To tackle this challenge, we’ve developed Anomaly Event Response (AER) – a new proactive defence mechanism inside Slack. By combining real-time monitoring with advanced analytics, AER autonomously identifies high-confidence threat actor behaviours as they emerge on our platform. When suspicious activity is detected, the system automatically terminates the associated user sessions, reducing the security detection and response gap from potential days/hours to mere minutes. The result? A powerful native security capability that disrupts attack chains before they can be fully executed, preventing data exfiltration and system compromise without requiring additional security tools, integration, or human capital. In this post, we’ll explore the technical architecture behind Anomaly Event Response, examining our motivation, detection mechanisms, response capabilities, and the insights we gained along the way. The Shared Responsibility of Securing Slack Slack is where work happens. We’re a central hub for workplace communication and collaboration for tens of millions of users on a weekly basis, generating billions of interactions daily. To ensure our customers’ security, we provide comprehensive audit logs to Enterprise customers that record when entities take an action on the platform. Within the full list includes hundreds of actions that we support, we also provide customers with anomaly audit logs which take a step further by leveraging advanced analytics to flag unusual activities across a workspace including irregular logins, uploading malware, unexpected data transfers, and more. Those interested in learning more about these special audit logs and how they can use them in security investigations will be interested in reading our Slack Audit Logs and Anomalies post. These specialized audit logs serve as an early warning system, highlighting suspicious activity and providing invaluable insight into potential threats that might otherwise remain undetected. That said, they aren’t a solution in and of themselves; they still require review by security personnel or integration with third-party solutions to translate detected behaviour into actionable insights. Integrating your Slack audit logs into a larger security solution offers unparalleled flexibility and customizability, but we also know this isn’t feasible for all customers based on their size, security capabilities, or resources. We created AER to bridge this gap between detection and response with an automated solution that any Enterprise Grid customer can use out-of-the-box, either by itself or as part of their broader security approach. Trust is our number one core value. We believe security is a shared responsibility between us and our customers by empowering them with data and tools to build security solutions while also fostering a secure platform and neutralizing threats. Our Anomaly Event Response solution bridges threat detection and automated response for everyone while still allowing advanced customers to add tailored security measures on top of this tool to address their unique needs. It also continues our ongoing commitment to enhancing user security with innovative automations like our measures against password breaches and cookie hijacking. Design Philosophy When designing this feature, we focused on supporting the most common types of threats that platforms face. Our goal was to prioritize detections that would have the broadest appeal across the platform. We landed on: Accessing Slack from a Tor exit node Excessive downloading as a potential indicator of data exfiltration Data scraping via non-Slack-native automation tools Session fingerprint mismatches Unexpected API call volumes and patterns Unexpected user agents indicating access from non-standard or virtual clients We know every customer uses Slack differently, and every organization is going to have a different risk profile, so we designed AER to let organizations select which detections are important to them and which should simply be logged without any responding action. This same configurability extends to notification preferences. To make changes to your configuration, navigate via Tools and Settings → Organization settings → Security (left-hand sidebar) → Security Settings. From here, there will be an Anomaly Event Response Settings section where you can choose which anomaly events should trigger user session termination and how you would like to be notified, if at all. <img loading="lazy" decoding="async" width="1854" height="2514" class="aligncenter size-large wp-image-17535" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?w=1020" alt="A modal window titled "Anomaly event response" with various checkboxes for detecting anomaly events like "Accessing Slack from a Tor exit node" and "Data scraping," and for sending notifications to "Org Primary Owner" and "Security Admins" via email and Slack." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png 1854w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=640,868 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=768,1041 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1280,1736 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1133,1536 1133w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1510,2048 1510w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=380,515 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=800,1085 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Settings-Modal.png?resize=1160,1573 1160w" sizes="auto, (max-width: 1854px) 100vw, 1854px" /> System Architecture & Technical Deep-Dive AER utilizes a multi-tiered architecture we designed for real-time anomaly detection, asynchronous job orchestration, and dynamic notifications. These operations are performed by three core components: the underlying detection engine, a decision framework, and the response orchestrator. <img loading="lazy" decoding="async" width="2545" height="3840" class="aligncenter size-large wp-image-17536" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?w=1020" alt="A flowchart illustrating an "Automated Event Response" process. It starts with "Suspicious User Activity," followed by "Activity Analyzed," then "Anomaly Detected." This leads to "AER Controller," which determines if it's a "Supported Anomaly." If so, "User Sessions Terminated" occurs. From "User Sessions Terminated," two paths emerge: "Audit Logged" (which is "Always" done) and "Customer Notified" (which is "Configurable")." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png 2545w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=640,966 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=768,1159 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1280,1931 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1018,1536 1018w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1357,2048 1357w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=380,573 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=800,1207 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/aer-simple-diagram.png?resize=1160,1750 1160w" sizes="auto, (max-width: 2545px) 100vw, 2545px" /> Detection Engine First, AER’s detection engine monitors Slack events at scale of billions per day, using a combination of rule-based heuristics and dynamic thresholds calibrated to each enterprise’s usage patterns. For example, when monitoring excessive downloading activity, what is anomalous for one organization may be the expected baseline for another, so we calculate thresholds for each organization based on historical data. This adaptive approach not only reduces false positives significantly across the customer-base, but it also allows us to constantly fine-tune the sensitivity of our detections and validate their effectiveness. Decision Framework Next, the decision framework evaluates each audit payload created when suspicious behaviour is identified, performing validation checks based on the customer’s AER configuration and internal rules. One of these checks includes analyzing the given payload and checking what preceded it so we can determine if malicious behaviour is persisting through user session terminations on the account while also making sure we’re protecting legitimate users from a continuous termination loop. We achieve this by analyzing the session(s) associated with every audit payload. Once an anomaly is validated and the customer has enabled detection for that event type, an asynchronous job is enqueued that performs the autonomous response. Response Orchestration Finally, AER terminates all active user sessions belonging to the acting user and executes a series of actions designed for transparency and accountability. AER generates a user_sessions_reset_by_anomaly_event_response audit log containing the specific anomaly type that triggered the response, context about the acting user, and the session_id from which the anomaly event originated. During an investigation, this datapoint lets you connect this activity back to the originating anomaly audit log for additional context. Customers should actively monitor for this audit log action so they are aware of when AER has taken autonomous action and if they need to conduct any further investigation into the acting user’s actions across their organization. After generating this audit log, AER queries the customer’s notification preferences to determine how any additional notifications should be routed. While the acting user always receives an email notification informing them of their terminated sessions, the customer’s configuration determines if any additional notifications are routed up to two other contact types: the Org Primary Owner and any users with the Security Admin system role. To reduce notification fatigue at this stage, we built smart notification logic that recognizes when someone is wearing multiple hats in your organization. For example, if the same person is both the Org Primary Owner and has the Security Admin system role, they’ll receive just one notification instead of multiple alerts for the same event. AER efficiently tracks these areas of overlap and ensures everyone gets informed without being overwhelmed by duplicate messages. It’s worth calling out, these contacts can opt to receive an email notification every time AER takes action on a user in their workspace, or they can choose to receive in-Slack DM notifications via Slack Security bot (or both!). <img loading="lazy" decoding="async" width="1068" height="328" class="aligncenter size-large wp-image-17539" src="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?w=1020" alt="A Slack notification from "Slack Security" stating that "@Matt Brewer has been signed out of all their active sessions in response to a potentially suspicious anomaly event: unexpected_scraping. It provides links to view details in audit logs and learn more."" srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png 1068w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=640,197 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=768,236 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=380,117 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/06/Screenshot-2025-03-25-at-12.10.25 PM.png?resize=800,246 800w" sizes="auto, (max-width: 1068px) 100vw, 1068px" /> For more information, see the Configure audit log anomaly event responses in Slack article in Slack’s Help Center. Key Findings & Impact Since launching AER in February 2025, its automated interventions have validated the importance of proactive security measures and demonstrated its effectiveness in disrupting suspicious activities in real-time. For context, according to IBM’s most recent Cost of a Data Breach Report, the average cost of a data breach reached $4.88M USD in 2024 with an average incident response time of 277 days for cloud-based collaboration platforms. AER’s ability to reduce this response window is a significant step forward in making a true security impact. By automatically terminating sessions engaging in anomalous activity, AER can prevent potential security incidents that could otherwise go undetected. For our enterprise customers, these benefits translate to enhanced protection of their data while simultaneously reducing security team overhead. As threat actors continue to evolve their techniques, AER provides a scalable, adaptive defence mechanism, ensuring that we keep Slack a secure and trusted platform for collaboration and communication. Conclusion Security should work while you do. Anomaly Event Response continues our tradition of approaching security in innovative ways that prioritize shared responsibility and impact at scale. By closing the gap between detection and response, we’ve neutralized the delay that traditionally favors attackers over defenders. The shared responsibility model we’ve embraced with AER represents the future of enterprise security – where platform providers and customers work together to create multiple layers of defence. By providing advanced security that’s accessible to everyone out of the box while still offering customization for organizations with sophisticated security operations, we’re ensuring that all customers benefit from real-time threat disruption regardless of their size or security resources. And the most exciting part? As the digital threat landscape continues to evolve, so will AER. We’re committed to providing security that works silently and effectively in the background by continuously refining our detection mechanisms, expanding our response capabilities, and incorporating customer feedback to extend the feature with new and exciting capabilities, allowing teams to collaborate with confidence. Slack is where work happens, and that modern collaborative environment requires and deserves modern forward-thinking security.
In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time…
In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time and resources for engineers at Slack. The Problem: Unnecessary Frontend Builds For one of our largest code repositories (a monolithic repository, or monorepo), Slack has a CI/CD pipeline that runs E2E tests before merging code into the main branch. This is critical for ensuring that changes are validated across the entire stack for the Slack application: frontend, backend, database, and the handful of services in between. However, we noticed a bottleneck: building the frontend code was taking longer than expected and occurred too frequently, even when there were no frontend-related changes. Here’s the breakdown: Developer Workflow: A developer makes changes and pushes to a branch. Build Process: The frontend code is built (~5 minutes). Deployment: The build is deployed to a QA environment. Testing: We run over 200 E2E tests, taking another 5 minutes. This entire process took about 10 minutes per run. Half of that time, around 5 minutes, was consumed by frontend builds, even when no frontend changes were involved. <img loading="lazy" decoding="async" width="1661" height="780" class="alignnone size-medium wp-image-17430" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?w=640" alt="Stages of the end to end testing pipeline." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png 1661w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=640,301 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=768,361 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1280,601 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1536,721 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=380,178 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=800,376 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1160,545 1160w" sizes="auto, (max-width: 1661px) 100vw, 1661px" /> Given that hundreds of pull requests (PRs) are merged daily, these redundant builds were not only time-consuming, but costly: Thousands of frontend builds per week, storing nearly a gigabyte of data per build in AWS S3. Half of these builds do not contain frontend changes compared to the last merge into main, causing terabytes of duplicate data. 5 minutes per build, adding unnecessary delays to pipelines (thousands of hours a week). The Solution: Smarter Build Strategy with Cached Frontend Assets To tackle this, we leveraged existing tools to rethink our build strategy. Step 1: Conditional Frontend Builds Our first step was determining whether a fresh frontend build was necessary. We detected changes by utilizing git diff and its 3-dot notation to identify the difference between the latest common commit of the current checked-out branch and main. If changes were detected, we invoke a frontend build job. If no changes were detected, we skipped the build entirely and reused a prebuilt version. Step 2: Prebuilt Assets and Internal CDN When a frontend build is not needed, we locate an existing build from AWS S3. To be efficient, we use a recent frontend build that is still in Production. We delegate the task of serving the prebuilt frontend assets for our E2E tests to an internal CDN. This reduced the need for creating a new build on each PR, while still ensuring we test on current assets. The Challenges: Efficiency at Scale While the approach seemed straightforward, scaling this solution to our monorepo presented a few challenges: Identifying Frontend Changes: Our repository contains over 100,000 tracked files. Determining whether frontend changes were present required efficient file tracking, which git handled in just a couple of seconds. Finding Prebuilt Assets: With hundreds of PRs merged into this repository daily, identifying a prebuilt version that was fresh enough required robust asset management. By using straightforward S3 storage concepts, we were able to balance recency, coherent file naming, and performance to manage our assets. Being Fast: We were able to distinguish if a frontend build was unnecessary and find a recent build artifact in just under 3 seconds on average. <img loading="lazy" decoding="async" width="1661" height="780" class="alignnone size-medium wp-image-17431" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?w=640" alt="End to end pipeline depicting local pre-built frontends at the third stage." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png 1661w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=640,301 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=768,361 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1280,601 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1536,721 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=380,178 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=800,376 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image3.png?resize=1160,545 1160w" sizes="auto, (max-width: 1661px) 100vw, 1661px" /> The Results: A 60% Drop in Build Frequency and 50% Drop in Build Time Our efforts paid off, delivering remarkable improvements: 60% Reduction in Build Frequency: By intelligently reusing prebuilt frontend assets, we reduced the number of unnecessary frontend builds by over half. Hundreds of Hours Saved Monthly: Cloud compute time and developer wait times are reduced. Several Terabytes of Storage Savings: We reduced our AWS S3 storage by several terabytes each month. These duplicate assets would have otherwise been stored for one year. 50% Build Time Improvement: This was the second major project by the Frontend DevXP Team and its partnering teams. The first project, which upgraded our Webpack setup, reduced the average build from ~10 minutes to ~5 minutes. This project took the average build from ~5 minutes down to just ~2 minutes. With both projects being successful, we reduced our average build time for E2E pipelines from ~10 minutes to ~2 minutes: a huge improvement for the year! <img loading="lazy" decoding="async" width="1658" height="932" class="alignnone size-medium wp-image-17428" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?w=640" alt="January baseline was an average build time of 10 minutes and 6 minutes for testing. By August average build time was 2 minutes, an 80% decrease for the year." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png 1658w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=640,360 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=768,432 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1280,720 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1536,863 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=380,214 380w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=800,450 800w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image1.png?resize=1160,652 1160w" sizes="auto, (max-width: 1658px) 100vw, 1658px" /> Two unexpected outcomes: More Reliable and Trustworthy E2E Results: Our test flakiness, which refers to tests failing intermittently or inconsistently despite no code changes, was significantly reduced. This improvement resulted from the optimized pipeline, decreased likelihood of needing complex frontend builds, and consistent asset delivery. We observed our lowest percentage of test flakiness as a result, based on monthly measurements. Rediscovering Legacy Code: Implementing this optimization required a deep dive into legacy code of multiple systems that hadn’t been significantly modified in a long time. This exploration yielded valuable insights, prompted new questions about the codebase’s behavior, and generated a backlog of tasks for future enhancements. Conclusion: Rethinking Frontend Build Efficiency By strategically utilizing existing tools like git diff and internal CDNs, we managed to save valuable developer time, reduce cloud costs, and improve overall build efficiency. For teams in other companies facing similar bottlenecks in DevOps and DevXP, the lesson is to question what’s truly necessary in your pipeline and optimize accordingly. The improvement from this project seems obvious in hind-sight, but it’s common to overlook inefficiencies in systems that haven’t outright failed. In our case, rethinking how we handled frontend assets turned into a massive win for the organization. Acknowledgments There are a lot of moving parts in a project like this: complex pipelines for building and testing, cloud infrastructure, an internal CDN, intricate build systems for frontend code, and existing custom setups throughout our entire system. It includes code written in Python, JavaScript, Bash, PHP/Hack, Rust, YAML, and Ruby. We achieved this without any downtime! Okay, almost. There was ten minutes of internal downtime for our deployment pipeline, but it was fixed pretty quickly. This work was not possible without contributions from: Anirudh Janga, Josh Cartmell, Arminé Iradian, Anupama Jasthi, Matt Jennings, Zack Weeden, John Long, Issac Gerges, Andrew MacDonald, Vani Anantha and Dave Harrington Interested in taking on interesting projects, making people’s work lives easier, or just building some pretty cool forms? We’re hiring! <path d="M9 17L1 9L9 1" stroke="#1d1c1d" stroke-linecap="round" stroke-linejoin="round" /> Explore Opportunities