Engineering at Slack
type: channel
description: Hear directly from Slack's engineers about what we build, why and how we build it, and how you can use it.
date: 2025-11-27T00:19:50Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Streamlining Security Investigations with Agents
items: Build better software to build software better
items: Advancing Our Chef Infrastructure: Safety Without Disruption
items: Deploy Safety: Reducing customer impact from change
items: Building Slack’s Anomaly Event Response
items: Optimizing Our E2E Pipeline
items: How we built enterprise search to be secure and private
items: Automated Accessibility Testing at Slack

Streamlining Security Investigations with Agents
type: item
creator: Dominic Marks
date: 2025-12-01T16:00:42Z
subject: Uncategorized
subject: development
subject: security
subject: software-engineering
description: Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI…
encoded: Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI agents to optimize our working efficiency and strengthen Slack’s security defenses. This post is the first in a series that will unpack some of the design choices we’ve made and the many things we’ve learnt along the way. The Development Process The Prototype At the end of May 2025 we had a rudimentary prototype of what would grow into our service. Initially, the service was not much more than a 300 word prompt. The prompt consisted of five sections: Orientation: “You are a security analyst that investigates security alerts […]” Manifest: “You have access to the following data sources: […]” Methodology: “Your investigation should follow these steps: […] ” Formatting: “Produce a markdown report of the investigation: […]” Classification: “Choose a response classification from: […]” We implemented a simple “stdio” mode MCP server to safely expose a subset of our data sources through the tool call interface. We repurposed a coding agent CLI as an execution environment for our prototype. The performance of our prototype implementation was highly variable: sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process. We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of da…

Build better software to build software better
type: item
creator: David Reed (he/his)
date: 2025-11-06T16:00:47Z
subject: Uncategorized
subject: build-performance
subject: caching
subject: ci-cd
subject: developer-experience
subject: devops
subject: python
description: We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until…
encoded: We manage the build pipeline that delivers Quip and Slack Canvas’s backend. A year ago, we were chasing exciting ideas to help engineers ship better code, faster. But we had one huge problem: builds took 60 minutes. With a build that slow, the whole pipeline gets less agile, and feedback doesn’t come to engineers until far too late. We fixed this problem by combining modern, high-performance build tooling (Bazel) with classic software engineering principles. Here’s how we did it. Thinking About Build (and Code) Performance Imagine a simple application. It has a backend server that provides an API and data storage, and a frontend that presents the user interface. Like many modern applications, the frontend and backend are decoupled; they can be developed and delivered independently. The graph of this build looks like this: <img loading="lazy" decoding="async" width="1642" height="430" class="alignnone size-full wp-image-17605" src="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png" alt="An example service build graph. It shows a set of Python files building into a backend artifact and a set of TypeScript files building into a frontend artifact." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png 1642w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=640,168 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=768,201 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1280,335 1280w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=1536,402 1536w, https://slack.engineering/wp-content/uploads/sites/7/2025/10/DevXP-Council-Presentation6-e1759869480617.png?resize=380,100 380w, https:/…

Advancing Our Chef Infrastructure: Safety Without Disruption
type: item
creator: Archie Gunasekara
date: 2025-10-23T18:17:56Z
subject: Uncategorized
subject: aws
description: Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating…
encoded: Last year, I wrote a blog post titled Advancing Our Chef Infrastructure, where we explored the evolution of our Chef infrastructure over the years. We talked about the shift from a single Chef stack to a multi-stack model, and the challenges that came with it – from updating how we handle cookbook uploads to navigating the limitations around Chef searches. If you haven’t had a chance to read that post yet, I highly recommend checking it out first to get the full context for this post. At Slack, keeping our service reliable is always the top priority. In my last post, I talked about the first phase of our work to make Chef and EC2 provisioning safer. With that behind us, we started looking at what else we could do to make deploys even safer and more reliable. One idea we explored was moving to Chef Policyfiles. That would have meant replacing roles and environments and asking dozens of teams to change their cookbooks. In the long run, it might have made things safer, but in the short term it would have been a huge effort and added more risk than it solved. So instead, this post is about the path we chose: improving our existing EC2 framework in a way that doesn’t disrupt cookbooks or roles, while still giving us more safety in our Chef deployments. Splitting Chef Environments Previously, each instance had a cron job that triggered a Chef run every few hours on a set schedule. These scheduled runs were primarily for compliance purposes — to ensure our fleet remained in a consistent and defined configuration state. To reduce risk, the timing of these cron jobs was staggered across availability zones, helping us avoid running Chef on all nodes simultaneously. This strategy gave us a buffer: if a bad change was introduced, it would only impact a subset of nodes initially, giving us a chance to catch and fix the issue before it spread. However, this approach had a critical limitation. With a single shared production environment, even if Chef wasn’t running everywhere at o…

Deploy Safety: Reducing customer impact from change
type: item
creator: Sam Bailey
date: 2025-10-07T16:33:12Z
subject: Uncategorized
subject: automation
subject: ci-cd
subject: deployment
subject: engineering
subject: incident-response
subject: infrastructure
subject: observability
description: It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing…
encoded: It’s mid 2023 and we’ve identified some opportunities to improve our reliability. Fast forward to January 2025. Customer impact hours are reduced from the peak by 90% and continuing to trend downward. We’re a year and half into the Deploy Safety Program at Slack, improving the way we deploy, uplifting our safety culture and continuing our rate of change to meet business needs. Defining the problem System requirements change as businesses evolve whether that be due to changes in customer expectations, load & scale or response to business needs. In this case analysis showed two critical trends: Slack has become more mission critical for our customers, increasing expectations of reliability for the product as a whole and for specific features each company relies on. The increasing majority (73%) of customer facing incidents were triggered by Slack-induced change, particularly code deploys. We use our incident management process heavily and frequently at Slack to ensure we have a strong and rapid response to issues of all shapes and sizes based on impact and severity. A portion of those incidents are customer impacting, many of which showed variable impact depending on which features they used in the product. Change-triggered incidents were occurring in a wide variety of systems and deployment processes. Finally, we also received customer feedback that interruptions became more disruptive after about 10 minutes – something they would treat as a “blip” – and this would continue to reduce with the introduction of Agentforce in 2025. All this was occurring in a software engineering environment with 100s of internal services and many different deployment systems & practices. In the past, our approach to reliability often zeroed in on individual deploy systems or services. This led to manual change processes that bogged down our pace of innovation. The increased time and effort required for changes not only dampened engineering morale but also hindered our ability to swiftly…

Building Slack’s Anomaly Event Response
type: item
creator: Nathan Lehotsky
date: 2025-09-04T10:00:02Z
subject: Uncategorized
subject: incident-response
subject: security
description: As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than…
encoded: As cyberattacks evolve to unprecedented levels of sophistication and speed, the time gap between breach detection and response has never been more critical. Traditional security approaches often operate reactively, identifying compromises only after damage has occurred. This delay grants attackers a tactical advantage, forcing security teams to focus on damage assessment and remediation rather than proactive threat detection and prevention. Organizations urgently need solutions that dramatically compress the detection-to-response window to regain a defensive advantage. To tackle this challenge, we’ve developed Anomaly Event Response (AER) – a new proactive defence mechanism inside Slack. By combining real-time monitoring with advanced analytics, AER autonomously identifies high-confidence threat actor behaviours as they emerge on our platform. When suspicious activity is detected, the system automatically terminates the associated user sessions, reducing the security detection and response gap from potential days/hours to mere minutes. The result? A powerful native security capability that disrupts attack chains before they can be fully executed, preventing data exfiltration and system compromise without requiring additional security tools, integration, or human capital. In this post, we’ll explore the technical architecture behind Anomaly Event Response, examining our motivation, detection mechanisms, response capabilities, and the insights we gained along the way. The Shared Responsibility of Securing Slack Slack is where work happens. We’re a central hub for workplace communication and collaboration for tens of millions of users on a weekly basis, generating billions of interactions daily. To ensure our customers’ security, we provide comprehensive audit logs to Enterprise customers that record when entities take an action on the platform. Within the full list includes hundreds of actions that we support, we also provide customers with anomaly audit logs which ta…

Optimizing Our E2E Pipeline
type: item
creator: Dan Carton
date: 2025-04-14T09:00:30Z
subject: Uncategorized
subject: ci-cd
subject: developer-experience
subject: developer-productivity
subject: devops
subject: frontend
subject: testing
description: In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time…
encoded: In the world of DevOps and Developer Experience (DevXP), speed and efficiency can make a big difference on an engineer’s day-to-day tasks. Today, we’ll dive into how Slack’s DevXP team took some existing tools and used them to optimize an end-to-end (E2E) testing pipeline. This lowered build times and reduced redundant processes, saving both time and resources for engineers at Slack. The Problem: Unnecessary Frontend Builds For one of our largest code repositories (a monolithic repository, or monorepo), Slack has a CI/CD pipeline that runs E2E tests before merging code into the main branch. This is critical for ensuring that changes are validated across the entire stack for the Slack application: frontend, backend, database, and the handful of services in between. However, we noticed a bottleneck: building the frontend code was taking longer than expected and occurred too frequently, even when there were no frontend-related changes. Here’s the breakdown: Developer Workflow: A developer makes changes and pushes to a branch. Build Process: The frontend code is built (~5 minutes). Deployment: The build is deployed to a QA environment. Testing: We run over 200 E2E tests, taking another 5 minutes. This entire process took about 10 minutes per run. Half of that time, around 5 minutes, was consumed by frontend builds, even when no frontend changes were involved. <img loading="lazy" decoding="async" width="1661" height="780" class="alignnone size-medium wp-image-17430" src="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?w=640" alt="Stages of the end to end testing pipeline." srcset="https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png 1661w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=640,301 640w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=768,361 768w, https://slack.engineering/wp-content/uploads/sites/7/2025/04/image4.png?resize=1280,601 1280w, https://slack…

How we built enterprise search to be secure and private
type: item
creator: Ian Hoffman
date: 2025-03-07T01:14:37Z
subject: Uncategorized
description: Many don’t know that “Slack” is in fact a backronym—it stands for “Searchable Log of all Communication and Knowledge”. And these days, it’s not just a searchable log: with Slack AI, Slack is now an intelligent log, leveraging the latest in generative AI to securely surface powerful, time-saving insights. We built Slack AI from the…
encoded: Many don’t know that “Slack” is in fact a backronym—it stands for “Searchable Log of all Communication and Knowledge”. And these days, it’s not just a searchable log: with Slack AI, Slack is now an intelligent log, leveraging the latest in generative AI to securely surface powerful, time-saving insights. We built Slack AI from the ground up to be secure and private following principles that mirror our existing enterprise grade compliance standards: Customer data never leaves Slack’s trust boundary. We do not train large language models (LLMs) on customer data. Slack AI only operates on the data that the user can already see. Slack AI integrates seamlessly with our existing enterprise grade compliance and security offerings. Now, with enterprise search, Slack is more than a log of all content and knowledge inside Slack—it also includes knowledge from your key applications. Users can now surface up-to-date, relevant content that is permissioned to them directly in Slack’s search. We’re starting with Google Drive and GitHub, and you’ll see many more of your connected apps as the year goes on. With these new apps, Slack search and AI Answers are all the more powerful, pulling in context from across key tools to satisfy your queries. We built enterprise search to uphold the same Enterprise-grade security and privacy standards as Slack AI: We never store data from external sources in our databases. External data and permissions are up to date with the external system. Users and admins must explicitly grant Slack access to external sources and may revoke that access at any time. We uphold the principle of least privilege, only requesting the authorizations we need to satisfy search queries. This blog post will explain how these principles guided the architecture of enterprise search. How enterprise search upholds the Slack AI principles First, a refresher: how does Slack AI uphold our security principles? Slack uses AWS to host closed-source large language models (LLMs) in…

Automated Accessibility Testing at Slack
type: item
creator: nstormann
date: 2025-01-07T20:46:20Z
subject: Uncategorized
subject: a11y
subject: accessibility
subject: automation-testing
subject: testing
description: At Slack, customer love is our first priority and accessibility is a core tenet of customer trust. We have our own Slack Accessibility Standards that product teams follow to guarantee their features are compliant with Web Content Accessibility Guidelines (WCAG). Our dedicated accessibility team supports developers in following these guidelines throughout the development process. We…
encoded: At Slack, customer love is our first priority and accessibility is a core tenet of customer trust. We have our own Slack Accessibility Standards that product teams follow to guarantee their features are compliant with Web Content Accessibility Guidelines (WCAG). Our dedicated accessibility team supports developers in following these guidelines throughout the development process. We also frequently collaborate with external manual testers that specialize in accessibility. In 2022, we started to supplement Slack’s accessibility strategy by setting up automated accessibility tests for desktop to catch a subset of accessibility violations throughout the development process. At Slack, we see automated accessibility testing as a valuable addition to our broader testing strategy. This broader strategy also includes involving people with disabilities early in the design process, conducting design and prototype review with these users, and performing manual testing across all of the assistive technologies we support. Automated tools can overlook nuanced accessibility issues that require human judgment, such as screen reader usability. Additionally, these tools can also flag issues that don’t align with the product’s specific design considerations. Despite that, we still felt there would be value in integrating an accessibility testing tool into our test frameworks as part of the overall, comprehensive testing strategy. Ideally, we were hoping to add another layer of support by integrating the accessibility validation directly into our existing frameworks so test owners could easily add checks, or better yet, not have to think about adding checks at all. Exploration and Limitations Unexpected Complexities: Axe, Jest, and React Testing Library (RTL) We chose to work with Axe, a popular and easily configurable accessibility testing tool, for its extensive capabilities and compatibility with our current end-to-end (E2E) test frameworks. Axe checks against a wide variety of acces…