You know how to design a good API, but how do you evolve that API when it’s time to make changes? We’ve faced this with each major feature release here at Slack over the past few years, most recently while working on the highly anticipated Shared Channels feature. Using shared channels, two organizations connect directly in Slack and work together from the comfort of their own workspaces.
The idea seems simple, but building it wasn’t easy. From a user’s perspective, a shared channel looks and feels just like any other channel. Under the hood, shared channels fundamentally changed the relationship between workspaces, channels, and the web API.
Let’s take a look at how the backend design for channels has evolved over time, and how the release of shared channels turned into an opportunity to build the Conversations API, a unified interface for all types of channels.
The Evolution of Channels
In the beginning when channels had a 1:1 relationship with a workspace, our backend database tables had a close relationship to the web API. Each table had corresponding API endpoints, and each API endpoint had a single corresponding OAuth scope:
But Shared Channels were different. We merged them into the public channels table, because they needed a shared channel ID across workspaces. This was for two reasons:
- Shared channels need to share resources (pins, files, reactions, and more)
- A shared channel can be private for one team and public for another
Here’s where the problem arose for the web API: Using OAuth, customers had explicitly granted permission to public channels (channels:read), private channels (groups:read), or DMs (im:read). But with the merged data in the channels table, we were not able to control data access the same way. Any app installed with channels:read scope was about to be granted access to private shared channels as well.
A New API Design
When we considered what was best for both the Shared Channels feature and our developer ecosystem, we found that there was significant upside in creating a brand new unified interface for all types of channels: the Conversations API.
At the top of our minds was how we were going to maintain backwards compatibility for all the developers who had already built apps using our web API. A new suite of endpoints would allow developers to opt-in when they were ready to update their code for shared channels. A new API would also allow us to address major developer pain points that had emerged over the years — things we really wanted to fix, but couldn’t — because of backwards compatibility.
For the rest of this article, we’ll be looking at the major changes in the new API:
- Smarter OAuth scope resolution
- Simplified arguments
- Better performance for large payloads
- Strict JSON Schema validation
Smarter OAuth Scope Resolution
Previously, developers had to know whether a channel was public or private before calling channels.info or groups.info. With the new API, developers no longer have to know the type of channel before calling conversations.info for any channel type. At the beginning of a request to any Conversations API endpoint, we now compare the type of channel requested with the OAuth scope of the requester’s token. If the requested channel type is not authorized, the developer will get a missing_scope error. Developers can now remove app logic that had been built to map a fragmented model of channels to OAuth scopes and API endpoints.
Simplified Arguments
Our original channel API endpoints had been updated over time as our product evolved, and as a result, dramatic inconsistencies had developed. One endpoint might have a channel argument that required an ID, whereas another endpoint with the same argument might require a channel name instead. We fixed these in the new API so that argument and types are consistent across the whole suite.
To make the Conversations API as easy to use as possible, we also reduced the number of arguments whenever we saw a strong trend in historical requests. For example, all requests to channels.rename used the non-default argument validate=false. As a result, we made validate=false the new default behavior for conversations.rename.
In some cases, we changed the arguments to improve performance and scalability for apps. For example, channels.info used to include a nested list of members, but as larger and larger companies started to use Slack, response payloads grew, in some cases to as much as 290MB for a single channel. In addition, 72% of requests to channels.info specified no_members=true. To help apps scale to large teams, we now exclude members by default for conversations.info. To provide the channel membership data in a more optimal format to developers, we added a new conversations.members endpoint with pagination.
Better Performance for Large Payloads
In the early days, we didn’t anticipate that workspaces would have over 10,000 channels, or that channels would have over 150,000 members. Endpoints that were never intended to be paginated grew to unreasonable proportions! Performance bottlenecks and timeouts hindered developers’ ability to scale their apps, and timeouts for large enterprises became frequent.
Even before Slack released the Conversations API, the Platform team had already started to upgrade older endpoints to a new cursor-based pagination paradigm. However, some older endpoints were using a legacy form of pagination that required suboptimal database queries. We could never fully transition any endpoint that had used the legacy pagination, because we did not want to break any apps that were already using the older request format. For those endpoints, the old and new pagination had to exist side-by-side, which meant that we still experienced undue load on our databases and delivered suboptimal performance to a portion of app users.
In the Conversations API, we committed fully to cursor-based pagination so that apps could scale alongside workspaces — whether or not they used shared channels.
Strict JSON Schema Validation
Previously, Slack backend engineers could update core libraries and accidentally change our downstream API output. App developers would discover bugs before we did when their apps broke from API inconsistencies. Additionally, we had response fields that were assumed false or null when unset, which worked for JavaScript developers but was painful for almost everybody else.
Before Shared Channels, we had already started adding JSON schema validation using the json-schema-rspec gem in our automated RSpec tests to increase reliability in our web API. Since we wanted to make the Conversations API reliable, we designed the endpoints up front and wrote schema validation tests in parallel to feature development. These tests add reliability to every deploy. As a bonus, we used the schema to add Open API specs for these endpoints.
While developing Shared Channels, the Slack engineering team took a step back to think through the best experience for end users and developers. Taking that step back to re-think the design gave us the opportunity to build something we’re proud of.
Evolving APIs is difficult, but we hope you like the changes we’ve made.
Want to use the Conversations API? Read the official docs here and check out the Platform Blog to learn more about building Slack apps. If you liked the teamwork described in this article, come work with us. Finally, stay tuned to the Slack engineering blog for updates on a forthcoming book Designing APIs that Developers Love by Saurabh Sahni, Amir Shevat, and yours truly.
