Background and motivation
In the fast-paced world of software development, having the right tools can make all the difference. At Slack, we’ve been working on a set of AI-powered developer tools that are saving 10,000+ hours of developer time yearly, while meeting our strictest requirements for security, data protection, and compliance. In this post, we’ll describe the areas where generative AI can have the most impact for your developers, both in the short term and the long term.
Resolving escalations in Slack channels
We use channels in Slack as a ticketing mechanism to keep track of escalations and alert on-call engineers. In order to alleviate on-call load we developed a bot in Slack to automatically categorize requests and answer relevant support questions.
The bot operates in two modes to assist engineers:
- Listens for a specific emoji reaction — reacji — and steps in with an answer when appropriate.
- Actively monitors posts in the escalation channel and jumps in when it detects a topic it can assist with.
Classification of posts in Slack
In the active monitoring mode, we use Large Language Models (LLMs) to classify posts and escalation requests into specific categories. This helps the bot to make smart decisions.
Here are the six default categories for posts,
- ANNOUNCEMENT – When a user is announcing a new change to a service, or a new person joins.
- HOW_TO_REQUEST – A user is asking for information, asking how to do something, or generally needs contextual information about a particular topic.
- HUMAN_ACTION – A user is requesting a human to perform an action on their behalf.
- PULL_REQUEST_REVIEW – A user is asking for a pull request review OR has questions about a pull request
- SERVICE_ISSUE_REQUEST – A user is being impacted by a service which is down or not working as expected
- UNKNOWN – Things that are unclear or not specific enough for the LLM to assist on. This can include feedback.
In order to maximize the effectiveness of classification, we provided examples for each category wrapped in <EXAMPLE> tags inside the prompt.
Importing internal knowledge into LLM pipeline
To provide engineers with contextual aware answers, we configured our LLM infrastructure to import data from,
- Technical documentation relevant to the channel from various sources
- Relevant posts inside the escalation channel from history (using message search API), and
- Canvases shared inside the channel (using canvas search API)
We are using Amazon Bedrock Knowledge Bases to vectorize and store data from different internal knowledge sources. This managed service offered a significant advantage by handling much of the underlying infrastructure, offering ease of creation of data sources, seamlessly customizing chunk sizes, managing vector stores, indexes, and providing different search capabilities along with offering on-demand scalable embedding models. This approach significantly streamlined our infrastructure management, reducing scaling and maintenance efforts to a minimum.
Enabling ease-of-use and customization
At Slack no two teams are exactly alike, so we made it easy for engineers to customize the bot for their channel needs. Each channel has its own configuration—adjusting prompts, response lengths, setting filters to pick documentation sections from knowledge bases and ability to respond only when documents surpass a threshold value for similarity scores. These configurations helped minimize hallucinations in respective channels.
Measuring effectiveness
Overall, 30% of interactions were rated five stars by our users, indicating that the bot helped successfully resolve their escalations. The bot is enabled for 25+ channels in our internal workspace and we estimate 3,000+ hours of engineering time saved in this year alone.
Information search and retrieval with contextual awareness
In order to help engineers streamline information discovery from knowledge sources (internal and external), we developed a few different ways of interacting with AI. Developers often work with various platforms such as Slack, an integrated development environment (IDE), browser, terminal and many more tools as part of the development workflow. We found the following tools to be the most effective in assisting our internal teams.
Custom internal AI assistant in Slack
Developers often look for complex information in Slack. Slack AI makes it easier than ever to find relevant information through AI-powered search. To complement this augment the AI search feature effort and to enable conversations, it is now possible to build a custom AI-powered agent or assistant and engage with it via a new dedicated secure UI in that lives right inside Slack (more here). The assistant we built enabled our internal engineers to get contextually aware answers from internal and external knowledge sources. This helped reduce the context switching previously needed for finding information and saved valuable developer time inside Slack. It is currently in beta rollout and constitutes about 14% of all internal LLM traffic.
Web app
To assist engineers looking for information on the web, we developed an interactive web application to have conversations with AI agents and LLMs. This application is built primarily for advanced customization. Agents have access to internal knowledge sources such as technical documentation, making them powerful tools for querying and synthesizing Slack-specific inferences. Users seamlessly customize API parameters to have fine-grained control over the AI’s behavior. Overall, the web app constitutes about 37% of all internal LLM traffic, making it one of the major tools developers actively engage with.
IDE
In our most recent internal developer survey, 65% of respondents said they prefer an IDE as the primary platform to engage with generative-AI for development tasks. Having AI assistance in an IDE reduces the overall context switching otherwise required and saves valuable time spent in translating information between platforms by hand. We are currently exploring in-house implementations and customizing several open source IDE plugins that channel AI traffic to our approved LLM infra. We are actively seeking a potential fit for assisting developers for the following use-cases: developing frontend with Typescript, developing backend with Hack, as well as Android and iOS development.
Measuring effectiveness
The AI assistant inside Slack and the web app have been generally available for five months internally and we are observing 4,000+ monthly chat-turns with 70%+ positive satisfaction.
Assisting engineers in developing AI applications
We are actively partnering with engineering teams internally to embed AI in existing internal tools and to develop new applications.
To set up our infrastructure, we developed a unified LLM platform with the following features,
- We offer access to multiple approved LLMs to ensure we are able to serve all developer needs.
- The defined schema for input and output provides an abstraction layer for applications built on top of it, allowing seamless switching between LLMs to solve complex engineering problems and to develop bespoke LLM pipelines.
- Our derived data system streamlines metrics using a source attribute in the API parameters, allowing a unified view into LLM traffic.
- We offer knowledge base creation, management, and querying through APIs for contextual aware responses.
To assist in AI development efforts, we created the following tools:,
- CLI tools: Command Line Interface tools enable our engineers to experiment in the terminal by making requests to our AI infrastructure with specific parameters.
- Evaluate: A playground environment (as a web application) where developers can experiment with LLMs and fine-tune prompts. This sandbox approach allows for fast iteration, effective prompt engineering and testing of AI interactions.
- Compare: This tool simplifies the selection process by offering side-by-side comparisons of generated text by various LLMs, allowing developers to identify the ideal LLM for their unique project requirements.
While working with 16+ internal teams to create novel AI experiences, we found that having the above tools enabled engineers to experiment, prototype, and rapidly understand the pros and cons of working with generative AI.
The reality check: Limitations and lessons
Here are some of the challenges we noticed as we attempted to integrate generative AI into internal tooling,
- The technical limitations of the current generation of LLMs—such as hallucinations and the inability to effectively solve multi-step problems—severely limited their use in complex technical projects and contributed to lost engineering time.
- Integrating third-party knowledge sources into the context requires a considerable amount of tuning to the overall system.
- Keeping up with the fast-changing landscape of generative -AI is time consuming, and also requires time investment to ensure all of our users come along with us on the journey.
Measuring the impact
We are currently estimating 10,000+ engineering hours saved by the end of this year across all our tooling with generative AI. In computing this metric we accounted for several key factors, including:
- Time spent on information discovery, both internal and external sources (this typically ranges anywhere from few minutes to couple of hours)
- Time required to read internal documentation (based on our study we estimate this to be 30 minutes for medium length documents and up-to 60 minutes for complex documents)
- Time saved by reducing context switching and information translation
- Time needed for comprehension of gathered information (this highly depends on experience level and contextual knowledge)
- Time taken to translate acquired knowledge into action or code
- Time lost to hallucinations (~30% of total interactions)
Conclusion
Developing AI-powered tooling has been a journey of collaboration and continuous learning. We recognized early that the key to success in adopting generative AI lies in balancing the technology with user needs, maintaining a strong focus on security and compliance, and fostering a culture of continuous improvement. As you embark on your own tool development journey, we hope that our journey serves as a roadmap that can be adapted to your unique organizational context.
Acknowledgments
We also wanted to give a shout out to all the people that have contributed to this incredible journey: Ross Harmes, Ryan Katkov, Sridhar Ramakrishnan, Nathan Rockenbach, Dave Harrington, Peter Secor, Sergii Gorbachov, Vani Anantha and our amazing AWS cross functional partners: Chandra Velliachamy, Gene Ting, James Holdsworth, Rachna Chadha and Wendy Leopold
