Engineering at Slack
type: channel
description: Hear directly from Slack's engineers about what we build, why and how we build it, and how you can use it.
date: 2026-06-10T18:51:50Z
update period: hourly
update frequency: 1
update base: 2000-01-01T12:00+00:00
generator agent: https://wordpress.org/
items: Agentic Testing: Where Agents Fit in the E2E Testing Stack
items: Slack AI: The Path to Multi-Cloud
items: From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
items: Managing context in long-run agentic applications
items: From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
items: How Slack Rebuilt Notifications 📣
items: Streamlining Security Investigations with Agents
items: Android VPAT journey

Agentic Testing: Where Agents Fit in the E2E Testing Stack
type: item
creator: Sergii Gorbachov
date: 2026-06-11T14:15:28Z
subject: Uncategorized
subject: automation-testing
subject: developer-productivity
subject: innovation
description: Abstract Agent-driven end-to-end (E2E) tests add a new exploratory layer to testing, but should they replace traditional deterministic tests? We ran more than 200 agentic E2E workflows using the Playwright MCP, Playwright CLI, and agent-generated Playwright tests in test workspaces using non-production data to find out how agentic testing could fit into both our and…
encoded: Abstract Agent-driven end-to-end (E2E) tests add a new exploratory layer to testing, but should they replace traditional deterministic tests? We ran more than 200 agentic E2E workflows using the Playwright MCP, Playwright CLI, and agent-generated Playwright tests in test workspaces using non-production data to find out how agentic testing could fit into both our and your testing stacks. 1. From Journeys to Goals Traditional end-to-end tests validate a specific journey through the UI. click → click → type → assert Agent-driven tests instead validate whether a goal can be achieved, often expressed as an instruction (e.g. “send a thread message”): goal → agent adapts → verify result This difference can be summarized simply: Tests enforce journeys. Agents verify goals. Across our agentic test runs, the overall workflow remained consistent (e.g. login → search → result → clear), but the exact sequence of actions varied. In practice, agents took different paths to reach the same outcome: Different input methods (clicking a search suggestion vs pressing Enter) Different navigation patterns (reopening search vs reusing existing state) Additional or skipped steps (extra clicks, snapshots, or intermediate actions) Agents can still validate intermediate steps when needed, but this flexibility comes with tradeoffs in reliability, cost, and execution time, which we explore in the next sections. The Problem Agent-driven E2E testing looks promising, but it raises a real question: can something that costs $15–30 per run and takes over 10 minutes actually fit into modern testing workflows? At first glance, the answer seems like no. But in 200+ runs, we found they are fundamentally different from traditional tests. They can be highly reliable and have a clear place in the testing stack. This is largely due to recent advances in large language models, which enable agents to write code, debug failures, and interact directly with user interfaces. These capabilities introduce a new execu…

Slack AI: The Path to Multi-Cloud
type: item
creator: Shaurya Kethireddy
date: 2026-05-28T14:15:20Z
subject: Uncategorized
subject: aws
subject: backend
subject: cloud-computing
subject: collaboration
subject: engineering
subject: infrastructure
subject: innovation
subject: machine-learning
subject: software-development
description: In early 2023, Slack faced a foundational challenge: serving Large Language Models (LLMs) at enterprise scale with the security, reliability, and performance our customers expect. Over three years, we evolved from basic infrastructure to orchestrating a sophisticated multi-cloud architecture. We didn’t just want shiny new models; we needed a system resilient to regional outages and…
encoded: In early 2023, Slack faced a foundational challenge: serving Large Language Models (LLMs) at enterprise scale with the security, reliability, and performance our customers expect. Over three years, we evolved from basic infrastructure to orchestrating a sophisticated multi-cloud architecture. We didn’t just want shiny new models; we needed a system resilient to regional outages and GPU scarcity. Our journey moved through four distinct phases, shifting from reactive infrastructure management to proactive, multi-vendor orchestration. Phase 1: The SageMaker Era When we built the initial stages of Slack AI, AWS SageMaker was the natural starting point. It was a managed ML Serving platform that offered the key things that we were looking for: Security, FedRamp compliance, model availability and control. We were able to leverage a sophisticated escrow virtual private cloud (VPC) strategy to establish a strict zero-knowledge environment: our data remained private to Slack, and the provider’s proprietary model weights remained inaccessible to us. To maximize uptime for a global user base, we deployed these containers across multiple AWS regions. This required our teams to manage the operational lifecycle, including cross-region IAM roles, balanced routing across model endpoints, proactive capacity planning, and auto-scaling logic. The Operational Reality While SageMaker provided the necessary security, the overhead was immense. We faced three primary taxes: Scaling Latency: Initialization times prevented instantaneous scaling. Hardware Scarcity: Enterprise-grade Nvidia GPUs, such as the A100 (Ampere architecture) and the emerging H100 (Hopper architecture) instances, were often unavailable. Over-Provisioning: Maintaining idle resources to meet peak SLAs. By early 2024, we mitigated these via On-Demand Capacity Reservations (ODCR) and proactive, cron-based scaling. However, this reinforced a hard truth: we were spending too many engineering cycles on plumbing. To scale, we n…

From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
type: item
creator: Mahendran Vasagam
date: 2026-05-05T14:00:01Z
subject: Uncategorized
subject: airflow
subject: aws
subject: big-data
subject: data-engineering
subject: data-infrastructure
subject: security
description: Excerpt By 2024, Slack’s data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We’re talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security…
encoded: Excerpt By 2024, Slack’s data platform had accumulated 700+ SSH-based operators orchestrating critical data pipelines. We’re talking daily search indexing that processed terabytes of data, analytics jobs powering business intelligence, the whole shebang. Every single one of these jobs required direct SSH access to production AWS Elastic MapReduce (EMR) clusters. We had a massive security surface, and we couldn’t move forward on any infrastructure modernization. Not ideal. We needed to eliminate SSH entirely. The solution? Migrate all 700+ jobs to a REST-based architecture. This is the story of how we killed SSH entirely, across 8 data regions, with zero downtime. How We Got Here Slack’s data platform was built around 2017 with a straightforward pattern. Airflow, our data pipeline orchestrator, needed to run jobs on EMR clusters, and SSH was the most direct path. Connect to the EMR master node, execute a command, done. Simple. # The old way - simple, but problematic task = SSHOperator( task_id='run_spark_job', ssh_conn_id='emr_master', command='spark-submit /path/to/job.py', ) This pattern proliferated across the platform. Teams built custom SSH-based operators for different use cases (because hey, if SSH works for Spark, why not everything else). By the time we took stock, we had 700+ jobs in production running everything from MapReduce jobs to AWS CLI commands to custom Python scripts. It worked. But it came with some potential problems. The Real Cost of SSH Potential security risks included: Direct SSH access to compute clusters increases the potential attack surface Key distribution and rotation across orchestration workers adds operational overhead Achieving fine-grained audit granularity typically requires correlating logs across multiple systems Permission management can grow complex, often requiring dedicated security groups and custom configurations Operations were painful: Jobs ran directly on EMR master nodes instead of being distributed, causing resource …

Managing context in long-run agentic applications
type: item
creator: Dominic Marks
date: 2026-04-13T17:17:16Z
subject: Uncategorized
subject: development
subject: security
subject: software-engineering
description: Excerpt In complex, long-running agentic systems, maintaining alignment and coherent reasoning between agents requires careful design. In this second article of our series, we explore these challenges and the mechanisms we built to keep teams of agents working productively over long time spans. We present a range of complementary techniques that balance the conflicting requirements…
encoded: Excerpt In complex, long-running agentic systems, maintaining alignment and coherent reasoning between agents requires careful design. In this second article of our series, we explore these challenges and the mechanisms we built to keep teams of agents working productively over long time spans. We present a range of complementary techniques that balance the conflicting requirements of continuity and creativity. In our first article, we introduced our agentic security investigation service. We described how teams of AI agents collaboratively investigate security alerts. A Director orchestrates the investigation, many specialist Experts gather evidence, and a Critic reviews the Experts’ findings. We suggest you read the series in order. To briefly recap, our investigation process proceeds through a series of defined phases. Each phase implements a distinct set of agent interactions. Within phases, we may have multiple rounds, where each round is one full iteration through the phase. There’s no preset limit on the number of rounds that make up an investigation: investigations continue until concluded by the Director agent. The Challenge of Long-run Coherence Language model APIs are stateless: to provide continuity between requests, the caller must provide the complete message history with each request. Agent frameworks solve the state management problem for users by accumulating message history between API calls. This fills the agent’s context window, which provides a hard limit on how much information the agent can handle. Even approaching an agent’s context window limit can degrade the quality of responses. For short-run applications, no extra context window management is typically required. <img loading="lazy" decoding="async" class="wp-image-17756 size-large" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/pydantic_ai_agentic_loop.png?w=842" alt="High-level overview of how agent frameworks manage context across inference API calls" width="842" hei…

From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus
type: item
creator: Carlo Preciado
date: 2026-03-31T17:00:39Z
subject: Uncategorized
subject: golang
subject: infrastructure
subject: networking
subject: observability
subject: open-source
description: The Problem: Legacy Tooling and Its Limitations Currently, Slack utilizes a hybrid approach to network measurement, incorporating both internal (such as traffic between AWS Availability Zones) and external (monitoring traffic from the public internet into Slack’s infrastructure) solutions. These tools comprise a combination of commercial SaaS offerings and custom-built network testing solutions developed by our…
encoded: The Problem: Legacy Tooling and Its Limitations Currently, Slack utilizes a hybrid approach to network measurement, incorporating both internal (such as traffic between AWS Availability Zones) and external (monitoring traffic from the public internet into Slack’s infrastructure) solutions. These tools comprise a combination of commercial SaaS offerings and custom-built network testing solutions developed by our internal teams over time. This was a suitable enough solution for our needs. When we began rolling out HTTP/3 support on the edge, there was a significant challenge that we encountered: A lack of client-side observability. <img loading="lazy" decoding="async" class="alignnone wp-image-17739 size-medium" src="https://slack.engineering/wp-content/uploads/sites/7/2026/03/quic-diagram.png?w=512" alt="" width="512" height="170" srcset="https://slack.engineering/wp-content/uploads/sites/7/2026/03/quic-diagram.png 512w, https://slack.engineering/wp-content/uploads/sites/7/2026/03/quic-diagram.png?resize=380,126 380w" sizes="auto, (max-width: 512px) 100vw, 512px" /> Since HTTP/3 is built on top of the QUIC transport protocol, it uses UDP instead of the traditional TCP. This fundamental shift to a new transport meant that existing monitoring tools and SaaS solutions were not capable of probing our new HTTP/3 endpoints for metrics. At that time, there was a major gap in the market: None of the SaaS observability tools we investigated supported HTTP/3 probing out of the box. Our internal Prometheus Blackbox Exporter (BBE), a cornerstone of our monitoring, didn’t have native support for QUIC. Without the ability to probe hundreds of thousands of HTTP/3 endpoints in our new infrastructure, we couldn’t get the client-side visibility we needed to monitor regressions to HTTP/2 or accurate round trip measurements. The Intern Who Made It Happen The Open Source Contribution Our intern, Sebastian Feliciano, scoped, implemented, and ultimately open-sourced QUIC support for Promet…

How Slack Rebuilt Notifications 📣
type: item
creator: Frances Coronel
date: 2026-03-19T19:00:54Z
subject: Uncategorized
subject: android
subject: backend
subject: cross-platform
subject: design
subject: frontend
subject: ios
subject: product-launch
subject: push-notification
description: Introduction 🔔 At Slack, notifications are how teams stay in the loop, but they can also become overwhelming when not designed with intention. Our goal was to make staying informed feel effortless. We set out to rebuild one of Slack’s most complicated systems from the ground up by bringing calm, consistency, and clarity to the…
encoded: Introduction At Slack, notifications are how teams stay in the loop, but they can also become overwhelming when not designed with intention. Our goal was to make staying informed feel effortless. We set out to rebuild one of Slack’s most complicated systems from the ground up by bringing calm, consistency, and clarity to the experience. Diagnosing the Noise Problem We knew perception of noise in Slack was a universal challenge affecting teams everywhere. Across workspaces, notification overload consistently ranks among the most common frustrations. Research showed that the more channels a person joins, the more likely they are to feel overwhelmed and confused about notification behavior. Internally, the data told a clear story. Issues with notifications are one of the top three drivers of Customer Experience tickets, with users often unsure how to control or understand their settings. The noise problem wasn’t just about volume—it was baked into the architecture itself. Our legacy notification system evolved over years, accumulating complexity that made it nearly impossible for users to understand or control. Four conflicting mental models: Desktop and mobile each had their own preference systems, with different options and behaviors. A “nothing” setting on mobile meant something entirely different from “Off” on desktop. Users couldn’t predict what would happen when they changed a setting. Hidden coupling between preferences: What users were notified about was tightly coupled with how they received notifications. Wanting fewer push notifications meant sacrificing in-app awareness entirely—there was no way to separate the two. Inconsistent state across clients: Settings didn’t reliably sync. Users would configure notifications on desktop only to find mobile behaving completely differently, leading to confusion and duplicate configuration work. Power users left behind: Advanced controls were scattered across multiple menus with no clear hierarchy. Features like “badge …

Streamlining Security Investigations with Agents
type: item
creator: Dominic Marks
date: 2025-12-01T16:00:42Z
subject: Uncategorized
subject: development
subject: security
subject: software-engineering
description: Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI…
encoded: Slack’s Security Engineering team is responsible for protecting Slack’s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We’re going to show you how we’re using AI agents to optimize our working efficiency and strengthen Slack’s security defenses. This post is the first in a series that will unpack some of the design choices we’ve made and the many things we’ve learnt along the way. The Development Process The Prototype At the end of May 2025 we had a rudimentary prototype of what would grow into our service. Initially, the service was not much more than a 300 word prompt. The prompt consisted of five sections: Orientation: “You are a security analyst that investigates security alerts […]” Manifest: “You have access to the following data sources: […]” Methodology: “Your investigation should follow these steps: […] ” Formatting: “Produce a markdown report of the investigation: […]” Classification: “Choose a response classification from: […]” We implemented a simple “stdio” mode MCP server to safely expose a subset of our data sources through the tool call interface. We repurposed a coding agent CLI as an execution environment for our prototype. The performance of our prototype implementation was highly variable: sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process. We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of da…

Android VPAT journey
type: item
creator: Hye Jung Choi
date: 2025-11-19T18:00:39Z
subject: Uncategorized
subject: a11y
subject: accessibility
subject: android
subject: empathy
subject: ios
subject: ui
description: Background A Voluntary Product Accessibility Template (VPAT) is a document that outlines how well a product aligns with accessibility (a11y) standards. Its primary purpose is to inform customers about a product’s a11y features, enabling them to make informed decisions before purchasing software. At Slack, we conducted a VPAT by a third party a11y vendor in…
encoded: Background A Voluntary Product Accessibility Template (VPAT) is a document that outlines how well a product aligns with accessibility (a11y) standards. Its primary purpose is to inform customers about a product’s a11y features, enabling them to make informed decisions before purchasing software. At Slack, we conducted a VPAT by a third party a11y vendor in 2024 following our significant UI change (IA4). During this VPAT process, we identified a11y issues across Android, iOS, and desktop. Of the issues identified on Android, we immediately assigned the obvious shovel-ready issues, such as those with color contrast and missing image labels, to the responsible teams for resolution. To address remaining Android issues, we conducted a comprehensive triage process. In this document, we take a look at the major recurring themes of Android accessibility issues and share resolution strategies and lessons learned in the hopes of helping others make similar improvements. Identified themes [Note: for resolved and for future work.] P1: Error messages are inaccessible P2: Headings are not identified P2: Missing a11y label on EditField P2: Incorrect number of items in list P2: Drag and drop in the workspace switcher is inaccessible P2: Strikethrough information not conveyed to screen reader users P3: Errors indicated by color alone P3: Keyboard Navigation and Focus Keyboard navigability can help customers with motor and visual impairments use Slack on Android. It’s more commonly used on larger form factors (e.g. tablets). We received numerous VPAT tickets related to the keyboard navigation, however, Slack on Android currently does not yet support the large form factor. As a result, we have decided to explore other solutions that align with recommended practices and standards. Issue resolution P1: Error messages are inaccessible. Report If the user enters an invalid input to an edit field and presses “Next” (or Submit), an error message is displayed, but the error is not communicat…