Developer-driven automated testing is vital to the ongoing health and quality of a codebase. It does, however, come with a noticeable cost. This is especially true of UI testing — an area often avoided by even the most test-friendly of developers. In this two-part blog post, we explore how the Slack Android team made authoring and maintaining UI tests part of the development cycle.
In Part 1, we place UI testing in context of the larger quality landscape and outline an approach that helps produce tests stable enough to run continuously.
Why Developers Should Author UI Tests
Let’s start with acknowledging the fact that many of us have been in the situation where a one-line bug fix required hours of struggle with an automated test. Why go through the effort if you can just merge the code and continue adding more features?
If you have ever refactored legacy code with no test coverage, you may remember the “I too like to live dangerously” feeling as you pulled the trigger to merge. A robust automated test suite (including UI tests) allows a you to move quickly with confidence, rather than feeling like you’re sitting at a high-stakes table in Vegas.
This effect is especially critical as your team grows and complexity rises. Unless you’re building a throwaway prototype, don’t make the mistake of deferring testing to a “better” time. There is no better time than now.
Further, authoring UI tests is a systematic way to exercise the code that you are about to unleash onto other developers and users. From the standpoint of code design, test-driven development is not as much a validation method as a design principle that forces you to break up your code into smaller modules — a natural forcing function for a cleaner, more maintainable codebase.
Ultimately, teams where developers are responsible for testing their own code gain a significant advantage that only grows with time.
Gaining Traction with UI Testing
While most developers agree that unit-testing their code is a good idea, the sentiment is not as positive about UI testing, which is often pushed to dedicated QA teams. A recent small-scale survey distributed to the Android development community suggests that ¾ of respondents write some kind of tests when working on features that involve UI. Of these developers, all did some kind of unit testing, but 37% did not add UI tests for the feature. Here are the top concerns that were cited by respondents when asked — “What prevented you from authoring UI tests?”
We distributed the same survey to the Slack Android team and, to our relief, the results showed a different picture.
What compelled Slack’s Android engineers to write, not only unit tests, but also UI tests? Was it the vast amounts of time they gained from not having to read email? Was it the glory of solving impossible problems? Was it the threat to remove Coconut LaCroix from the fridge? Or, perhaps, we found a UI test strategy that works for us.
Slack’s Approach to UI Testing on Android
Our strategy comes down to reducing the need for UI testing, while keeping the remainder of UI tests stable by making them targeted and hermetic. Let’s dive into these in detail.
Avoid over-testing the UI
No post about testing is complete without the famous geometric shape of testing:
As you move up the pyramid, “larger” tests (i.e. ones that involved interactions between more components) become more difficult to author, maintain and execute. A developer should, therefore, prefer to get the majority of coverage from smaller unit tests, while supplementing the suite with a smaller set of integration and end-to-end tests.
The idea is simple, but in practice, it is complicated — the Android framework invites a developer to bundle business logic with UI interactions in the Activity/Fragment classes. Without adding additional layers to the architecture, you may end up with code that is only testable through UI and, following the path of least resistance, your testing may end up inverted:
Like every other problem in computer science, this can be solved with another layer of indirection. Is it worth introducing this layer for the sake of decoupling business logic from UI? We absolutely believe so. Of the many MV* (MVC, MVP, MVVM, MVI, MVWTF, …) patterns out there, we chose to go with a variant of MVP, with a bi-directional contract between the Presenter (pure business logic) and View (encapsulating simple UI interactions).
Which one should you go with? There are many takes on which pattern works best. The Google Samples Android Architecture repository includes a collection of samples and is a great place to start looking into the subject. With the recent introduction of Android Architecture Components, the Android team is also providing guidance and framework support for more testable architectures. Find a pattern that best fits your code base and go with it.
At this point, an astute (and lazy!) developer might ask — if I can test all complex business logic through unit tests, do I even need UI tests at all? Our answer to that is a resounding yes because it helps you avoid this situation:
By validating whether the UI can be inflated on the actual Android OS and that business and UI layers are glued properly, UI tests fill the middle (integration) layer of the test pyramid. Lastly, you do not want to miss out on the opportunity to watch UI tests execute in parallel. A good rule of thumb is to have at least one test per screen that launches the activity and performs a few simple UI interactions/checks.
Keeping UI Tests Stable
Remember that one of the third most cited concerns from the Android developer survey above was: “Lack of trust in results”. Can UI tests actually be stable or will we have to live in continuous Christmas mode?
In reality, most UI tests end up being unstable because developers forget to follow two best practices in testing: test only what you need to test and keep full control of test state.
Targeted UI Tests
Let’s look at an example. If your test is meant to validate functionality of the settings screen, it should not navigate through the login process, open the navigation drawer, press the overflow button, and only then start the actual test interactions related to settings. That would make the test more difficult to author, slower to execute, and more likely to fail in code paths that are unrelated to the functionality under test.
Instead, the test should launch the settings activity directly. On Android that’s easy to do by including this Junit4 rule in your test class:
@Rule
public ActivityTestRule activityRule =
new ActivityTestRule(SettingsActivity.class);But there’s a catch — if you do this without setting up application state, you’ll likely see a login activity or some other setup screen instead of settings. In our project, we solve this by using a combination of Dagger (a popular dependency injection framework) and Mockito (a framework for easily creating test doubles).
@Module(
includes = AppModule.class,
overrides = true,
library = true
)
final class MockAppModule {
@Provides @Singleton DataStore provideDataStore() {
return mock(DataStore.class);
}
}SlackApp app = (SlackApp) context.getApplicationContext();
app.resetGraph(ObjectGraph.create(new AppModule(app), new MockAppModule());Note: For legacy reasons, we are still on Dagger 1. If you’re starting from scratch, refer to this sample with Dagger 2.
Having the ability to swap in a test double for any object in the graph, we can now simulate a logged-in state by swapping out the component of our application that is responsible for signaling such state.
when(mockAccountManager.hasActiveAccount()).thenReturn(true);
Account account = new Account(ID, TEAM_ID);
when(mockAccountManager.getActiveAccount()).thenReturn(account);This leads us to the next section where we further discuss state.
Hermetic UI Tests
In relation to testing, the word hermetic means that the test is fully isolated from non-deterministic state and interactions. The randomness can creep into the test through data, background operations and interactions with other applications.
Data
We are now able to launch our screen directly without navigating through the rest of the application, but will it even show up given the fact that we did not go through our normal setup flow? That’s a trick question — chances are, the application would simply crash because the data model required for inflating the UI is not there.
There are multiple ways in which test data can be provided to the UI under test:
- API: Most applications communicate with a backend to retrieve data, so you can intercept and replay network traffic during test execution.
- Persistent Storage: Data retrieved from the backend is often stored in local storage, such as an SQLite database. At this layer, you can copy the required database files from a real user session to the application directory to inject data for testing.
- Higher Level of Abstraction: Many application architectures add a layer between local/backend storage and the business-logic components that use the data. During testing, you can swap in a test double for this intermediate layer and use its methods to provide data for the screen under test.
Our recommendation for functional UI testing is to use the “Higher Level of Abstraction” method. We use a layer of classes called data providers, which retrieve data from local and backend sources and provide it in a user-friendly Rx Observable to business logic components. This approach reduces the footprint of what’s being tested, thus making the test even more targeted. It also comes with an important side-benefit of reducing non-determinism caused by background operations used during data fetching, which we’ll talk about shortly.
In the case of the MVP pattern, a reasonable alternative would be to swap in a fake Presenter (an even higher level of abstraction). We believe this approach goes too far. It is true that the Presenter should already be unit-tested, but in a functional UI test, we want to cover the integration boundary between the View and the Presenter.
Background Operations
We’ve now launched a screen and injected an initial set of fake data to be displayed in the UI. This, in itself, is already a win — we can test that the Activity can successfully inflate its view hierarchy. But it is likely that you would want to go further and actually interact with the screen to test a few basic operations. This is where the UI testing story often becomes sad. The reason is multithreading — Android tests run on the instrumentation thread, while the UI is processed on the UI thread, leading to race conditions between test interactions and UI state. Espresso helps by synchronizing all operations with the UI thread and taking care of a few basic background mechanisms like AsyncTasks. But most complex applications use Executors or Rx to schedule background work and Espresso does not handle synchronization with these out-of-the-box.
A common anti-pattern at this point is to sprinkle a test with sleep calls or some other forms of sleep-and-retry mechanisms. If we were to identify the most common source of flaky UI tests, this would be it. If you want developers to believe in UI testing, forget about sleep (in the tests, of course).
Thankfully, there are better ways. One approach would be to use Espresso’s IdlingResource mechanism to synchronize background operations, but this route can be complex and error prone.
A simpler and more resilient approach is to get rid of asynchronous background work altogether. Remember those useful DataProviders that we replaced with test doubles to feed data into our app? We can replace the background work with instantly-returning operations that get processed on the instrumentation test thread:
// Success
when(mockUsersDataProvider.getUser(USER_ID))
.thenReturn(Observable.just(user));
// Failure
when(mockUsersDataProvider.getUser(USER_ID))
.thenReturn(Observable.error(new RuntimeException(“from test”)));Other Applications
One strength of the Android operating system is its ability to facilitate interactions between applications through intents. At the same time, this brings another source of non-determinism to testing. For example, a particular part of your application may send an intent to the photo-gallery app when it needs the user to pick an image. Testing this interaction presents two problems. First, instrumentation UI tests cannot interact with the UI of external processes (a security limitation of the OS). You could work around this by using UiAutomator, but here’s the second, more fundamental problem — interacting with a dependency that you’re not fully in control of breaks the hermetic seal. Thankfully, these are exactly the problems that espresso-intents, a mockito-like framework for Android intents, was designed to solve. Here’s a sample:
Intent data = new Intent();
data.setData(Uri.parse(“android.resource://com.slack/drawable/placeholder”));
intending(not(isInternal()))
.respondWith(new ActivityResult(Activity.RESULT_OK, data));
onView(withText(R.string.label_choose_photo)).perform(click());Some developers go a step further and isolate the current activity under test from other activities within the application. With this approach, if your test triggers a transition to another activity, there is no need to set up the state for the destination UI. The astute (and not lazy!) developer may point out that this amounts to behavior verification (as opposed to the recommended state verification). That is a good point. We leave it up the reader to weigh the benefits of this approach versus the drawbacks.
Mission Accomplished?
With the Targeted and Hermetic approach, our developer-authored UI tests were stable enough to run in CI. We addressed an important concern of trust and got developers to believe in the utility of UI testing. However, there was still a problem — developers did not enjoy the process because it was too cumbersome and time-consuming. In Part 2 of this post, we’ll discuss how we tackled the problem of making UI testing easy, and dare we say, enjoyable.
Interested in joining our test-friendly engineering team? Check out the open Android and Mobile Developer Experience positions here.
