In Part 1 we introduced the targeted and hermetic UI test approach that helped Slack Android engineers write hundreds of stable UI tests. However, as our UI test suite expanded, we detected a problem. When asked how easy it was to write tests on a scale from 1 (terrible) to 4 (excellent), developers ranked it a 2.5 (a resounding “meh”). In fact, the only category that ranked worse was related to build times.
As one developer put it — “Writing quality UI tests is still a difficult thing to do. It would be great to have some time put into building up a base to work from.” What a great suggestion! In the following Test Week (a quarterly hackathon focused on improving test infrastructure), we set out to address this.
Robots to the Rescue?
As a starting point, we looked at existing solutions that could help us better structure and organize our test code. The robot pattern — a layer of abstraction encapsulating user-facing functionality on the screen — was a good place to start.
By separating out what the test does with the UI from how it interacts with the UI, robots reduce code duplication, make tests more readable, and take some friction out of authoring tests.
However, in the case of targeted and hermetic UI tests, robots only solve part of the problem. What about the other side of the View?
We found that the other side of the view — dealing with setting up hermetic state — is where developers struggled the most.
We previously showed a couple lines of code demonstrating how we feed data to the test via a mock data provider. In practice, it’s rarely just a couple lines, but more like this (except a lot longer in the case of more complex screens):
@Before
@Override
public void setUp() throws Exception {
super.setUp();
User user = User.builder()…/* setters */…build();
when(mockUsersDataProvider.getUser(USER_ID))
.thenReturn(Observable.just(user));
when(mockUserPresenceManager.isUserActive(USER_ID, true))
.thenReturn(true);
when(mockDndInfoDataProvider.getDndInfo(userId))
.thenReturn(Observable.just(new DndInfo(/* params */)));
Channel channel = Channel.builder()…/* setters */…build();
when(mockChannelsDataProvider.getMessagingChannelObservable(CHANNEL_ID))
.thenReturn(Observable.just(channel));
when(mockChannelNameHelper.getDisplayNameObservable(eq(channel)))
.thenReturn(Observable.just(CHANNEL_NAME));
launchActivity();
}We have many more components that need to know about a user and a channel within our app, but this should help illustrate the complexity of our test setup. These components need some minimum state before a View will display properly and give us access to part of the app under test. Before a developer can add a test for their feature, they must swap in test doubles for everything else that’s powering that View — a time-consuming endeavor that often requires sifting through code that was written by others. What’s more, each developer would find themselves in a similar situation, causing potentially every member of the team to go through the same cumbersome test-writing process over and over again.
Our attempts to tuck repetitive setup away into a Base class only got us so far. Tests inheriting from BaseActivityTest needed to set up additional test data, leading to the proliferation of repetitive mocking throughout our codebase. And it turned out this was the most time-consuming part of creating a UI test, prompting us to look for a better way.
Introducing PowerPack
If we consider the data as part of our test — which we should if the test is going to be targeted and hermetic — the UI Test Architecture starts to look a lot more like Diagram 3. Before interacting with Robots, the test has to do some setup on the other end of the spectrum so that our View is “powered” accordingly. Naturally, this starts to look like a test sandwich of sorts, albeit a bit of an imbalanced one since we have both the Models and Tests providing “What” data we’re testing. However, when we take a step back and consider what a Test is actually doing in the setup phase, it’s essentially creating some test models and returning them from the Presenters/Components.
We saw a great opportunity to improve the developer experience. If we look at the UI Test Architecture from a slightly different perspective, with the Test interacting with the Robots on one side of the View and providing the Models on the other side of the View, it made sense to swap in something for the “How” behind the View — similar to Robots in front. In Diagram 4., we introduce the PowerPack pattern to encapsulate the complexity of powering the View.
Part of the complexity comes from figuring out what needs to know about the test data that’s provided, so that the test author can simply think of models and not components. In the case of a User, we introduced a UsersPack that leverages AutoValue to provide a builder interface. By default, the UsersPack is provided with a set of Components that need to know about Users as follows:
public UsersPack.Builder users() {
return UsersPack.builder()
.dndInfoDataProvider(mockDndInfoDataProvider)
.persistentStore(mockPersistentStore)
.usersDataProvider(mockUsersDataProvider)
.userPresenceManager(mockUserPresenceManager);
}And it exposes a pack() method that encapsulates all of the repetitive mocking for all of the Components.
public class UsersPack {
public void pack() {
UsersPack usersPack = build();
UsersDataProvider usersDataProvider = usersPack.usersDataProvider();
UserPresenceManager userPresenceManager = usersPack.userPresenceManager();
DndInfoDataProvider dndInfoDataProvider = usersPack.dndInfoDataProvider();
// More components…
List users = usersPack.users();
for (User user : users) {
when(usersDataProvider.getUser(user.id()))
.thenReturn(Observable.just(user));
when(userPresenceManager.isUserActive(user.id(), true))
.thenReturn(true);
when(dndInfoDataProvider.getDndInfo(user.id()))
.thenReturn(Observable.just(new DndInfo(/* params */)));
// More mocking…
}
}
}In comparison, the ChannelsPack has a lot more Components than we originally illustrated (shown below). As such, we’ll leave it up to your imagination on how much more involved its pack() method is.
public ChannelsPack.Builder channels() {
return ChannelsPack.builder()
.accountManager(mockAccountManager)
.featureFlagStore(mockFeatureFlagStore)
.lastOpenedMsgChannelIdStore(mockLastOpenedMsgChannelIdStore)
.mpdmDisplayNameHelper(mockMpdmDisplayNameHelper)
.persistentStore(mockPersistentStore)
.prefsManager(mockPrefsManager)
.slackApi(mockSlackApi)
.channelListDataProvider(mockChannelListDataProvider)
.channelNameHelper(mockChannelNameHelper)
.channelsPaneDataProvider(mockChannelsPaneDataProvider)
.channelsDataProvider(mockChannelsDataProvider)
.messageCountManager(mockMessageCountManager)
.notificationPrefsDataProvider(mockNotificationPrefsDataProvider)
.notificationPrefsManager(mockNotificationPrefsManager)
.userChannelListDataProvider(mockUserChannelListDataProvider);
}A side benefit of these model-specific Packs is that they can be used by unit tests directly to mock a subset of Components, but their main benefit in UI tests comes from grouping them into an app-wide Pack, which is injected with mocked dependencies and provides them automatically in the model-specific builders. For us, this is a SlackPowerPack, which reduces the complexity in a test setup as follows:
@Before
@Override
public void setUp() throws Exception {
super.setUp();
slackPowerPack = new SlackPowerPack(context.getApplicationContext());
slackPowerPack.users()
.addUser(User.builder()…/* setters */…build())
.addUser(User.builder()…/* setters */…build())
.pack();
slackPowerPack.channels()
.addChannel(Channel.builder()…/* setters */…build())
.addChannel(Channel.builder()…/* setters */…build())
.addChannel(Channel.builder()…/* setters */…build())
.pack();
launchActivity();
}The end result is a lot more clear and concise. The test author approaches a test by thinking of the test data and not the Presenters/Components that need to be mocked out.
It’s worth noting that the complexity in setting up the Presenters/Components with the test data hasn’t been removed — it’s just been abstracted away as seen in the UsersPack.pack() example above. Still, the benefit is that a team of developers only has to take this cumbersome hit once and any changes to Presenter/Component APIs benefit from updates in a single Pack class.
Using the Robot and PowerPack patterns in conjunction, our UI Test Architecture ends up looking like a View sandwich in Diagram 5. The Test manipulates the layer immediately in front of and behind the View, giving us a very targeted and hermetic test where all the complexity in interacting with the View and setting up state is encapsulated and abstracted away. For the test author, just as the Robot pattern provides a fluent API for entering text, tapping buttons, etc., the PowerPack pattern does the same for initializing users, channels, messages, or whatever other test data the app is dependent upon.
With the PowerPack, we saw a significant improvement in the Ease of Testing category of the Developer Experience survey (a jump from 2.5 to 2.92). A friendly Slack developer wrote this — “Ease of test writing: this has improved a lot. It would be nice to take the time to update some of our old tests to use PowerPack/our latest patterns.” And if they stopped there, we would have packed up and gone home. But they also went on to say this — “I would also love an easy way to set up a workspace.”
Which got us thinking…
PowerPack++
Many of the Slack UI tests need a test workspace with channels, users, messages, and other data. We could use the SlackPowerPack builder to set all that up, but what if the setup could be reduced to just one line?
@Before
public void setUp() {
slackPowerPack =
new SlackPowerPack(context.getApplicationContext(), “workspace_config.json”);
launchActivity();
}To accomplish this, we extract the data used to initialize the SlackPowerPack into a json config:
{
“logged_in_user”: String,
“team”: {...},
“users”: [ ],
“channels”: [ ],
“history”: [
{
“channel_id”: String, “messages”: [ ]
}
]
}From there, we can inflate the JSON into model objects and feed them to our PowerPack:
public void loadWorkspaceConfig(String configFilename) throws IllegalStateException {
InputStream inputStream = getClass().getResourceAsStream(configFilename);
InputStreamReader inputStreamReader = new InputStreamReader(inputStream, Charset.forName(“UTF-8”));
WorkspaceConfig workspace = new Gson().fromJson(inputStreamReader, WorkspaceConfig.class);
String userId = workspace.getLoggedInUser();
Team team = workspace.getTeam();
List users = workspace.getUsers();
List channels = workspace.getChannels();
List history = workspace.getHistory();
// Data validation that can throw an IllegalStateException.
accounts()
.userId(loggedInUser.id())
.userName(loggedInUser.name())
.teamId(team.id())
.teamName(team.getName())
.teamDomain(team.getDomain())
.pack();
users()
.addUsers(users)
.pack();
channels()
.addChannels(new ArrayList(channels))
.pack();
if (history != null) {
for (WorkspaceConfig.History channelHistory : history) {
messages()
.channel(channelHistory.getChannelId())
.addMessages(channelHistory.getMessages())
.pack();
}
}
// More packing…
}Beyond PowerPack
Having reduced test setup from a twisted mess of mockito invocations to one line, we got thinking about the next chapter in our testing story.
The JSON config …
… is strangely reminiscent of the data we retrieve from the Slack API. What if, we could charge the PowerPack automatically?
We maintain a small set of end-to-end tests that communicate with a real backend (stay tuned for a blog post about this topic). Using a library like OkReplay, we could capture network traffic from these tests. This API data can then be replayed directly in tests (a technique we use for performance testing, in order to preserve the fidelity of the application under test) or it could be used to auto-charge the SlackPowerPack. Sounds promising — we intend to explore this idea in the near future.
Results
With our developer-driven test approach, we’ve been able to build up a robust test suite that actually resembles a balanced pyramid.
Our unit and UI tests gate every merge to master, ensuring that a developer can merge code with confidence. Equally important, the process of testing exposes gnarly areas of our app and naturally guides us towards writing better, more modular code.
There is still a lot of work to do: building PowerPack++, covering legacy parts of the codebase, continuing to fight flakiness, and taking infrastructure that executes and manages tests to the next level. We look forward to sharing our progress on these initiatives. Thanks for reading!
If this is the kind of work that appeals to you, join our Mobile Developer Experience team. And if, instead, you’d like the idea of authoring UI tests themselves, we are looking for test-friendly Android engineers.
