Testing on Android – A practical approach

Testing an application is very important. It may serve a lot of purposes such as be sure some features are correctly working, put in confiance other developers in modifying legacy code, or even development technics like TDD. However testing on Android evolved quite quickly recently,  a lot of new tools have to be considered, and the platform itself make testing on Android quite a challenge.

[Edit] I wont write any part 2 about using emulators for now, since the new official emulator has been released. For now it can not be used in CI since it does not support headless mode yet. More than that, Clean Architecture is taking a huge step forward lately (like Android-CleanArchitecture, or Ribot app). This kind of architecture let us minimize the dependency to the Android framework, and thus allow us to write a lot more pure Java tests.

About writing, debugging, and getting result from tests using AS, the official documentation from Google about testing has been fully rewritten. I encourage you to refer to it.

I may however write posts about specific cases of testing using Clean Architecture, based on RX Android and Dagger 2. [End of edit]

A first (wrong) approach

At University I was writing small Java or C/C++ programs which did not have any UI and thus were very simple to test, using whether input/output, or unit testing through framework like JUnit. When I started to work on Android, I was thinking : “Ok, I know testing can improve my work, especially for my app since I deliver updates often, so lets write some tests for Android”. This was the start of a hell of a journey.

A year ago I was working on this application which needed an authentication of the user to access some features.  I was back then thinking that I need to pass the authentication to be able to test those features. Since the authentication password input was done through a picture of a keyboard generated on a server, I wrote a character recognition feature which was able to enter the correct password by its own, which lets me write tests on features which need this authentication. It may start working, but it also may fail often, because the server went down, or the character recognition did not worked as intended, or even because the server did not respond with the correct values. The execution of such tests was also very long, since it depends on web server (connection latency, server computation), and always started by a character recognition (the authentication). So I stop wrote tests since I was not able to use them to improve my work, to gain value.

To summarize what went wrong :

 

  • Slow tests.
  • Unpredictable results.
  • No valuable for my development team.

Since then, I totally rethought my test approach to have something today, which is actually quick efficient, and precise. I ended up with something that matters. This is this approach I want to share in this article.

My Way

In this article I will mostly talk about my experience with Android testing. So what you will read here should be more considered like a feedback than “this is the way to do it”.

Fork the example !

I strongly recommend you to fork the following project, since all the examples I will present come from it. At the same time, you will be able to play yourself with already set up tests.

This example is an Android application which is able to compute Pi. Of course, Pi is just approximated. The app works as a benchmark application : it can send the result of the computation online (the result is composed by the value computed but also the time needed to compute Pi) and then present a score to the user, which is better if the Pi value is computed is less time than the other result stored online. This is kind of a benchmark application. The server does not exist in fact, but I still use Retrofit…I will come back to that later.

What to test ?

First of all, before writing tests, I end up thinking what does really matters in my application. In Android, I think it is very legitimate to separate two worlds. The Java world, and the Android world. Java world is everything which do not has any dependency to the Android framework. For an application we can represent those two “worlds” as two layers of the application. This separation is from Fernando Cejas :
app-layers

The domain layer

The domain layer is the Java world. For my Github example, the domain layer is represented by the Pi generator interface, implemented here for example. It rely on pur Java objects and do the essential business of the demo app : compute Pi.

It is relevant to separate these two worlds since I can expect that the Java world will behave the same way on the JVM installed on my computed, as on any Android devices since it does not rely on any device specific feature. That means that I should expect the same result when using this layer on my computer and on my Android devices. So if I write tests for this layer, and I run them on my computer, I know the result will be consistant. Consistant is fundamental for tests. More than that, if I can run tests on my computer, they will run as fast as possible (no need to deploy any application whatsoever on a hypothetic device or emulator).

Since a few months, a wonderful feature of Android Studio lets us do that, and I must say, this is one of the feature of Android Studio which drastically improved the quality of the applications I work on, by letting me testing all the business logic of my application directly on the JVM of my computer.

The presentation layer

In the Android world, I want to test features of my applications. With testing the domain layer I know I am confident with the business logic of my application, but whats matter is the application provide the correct services to the final users. The good informations are presented at the right time.

This sounds like UI, functional, instrumentation testing. In fact (for me) the fundamental difference is the dependency of the layer. And in this case, it is Android. Those feature tests are actually specific to the Android and the devices on which we are running the tests. I can not write and run UI tests on the JVM of my computer. I need an emulator, a device or solutions like Roboelectric.

How to test ?

In this part, I will present the tools I use to write and run tests, both for the domain layer and for the presentation layer.

The domain layer under test

As talk before, the official solution from Google is awesome. You can :

  • Write the tests in Android Studio.
  • Run and debug tests from Android Studio.
  • Run tests  from Gradle.
  • Android Studio reports directly the results. HTML reports are also generated (great for CI).

The presentation layer under test

I used two frameworks to test this layer, Robotium, and Espresso. In the demo project, I created an abstract test class to show the implementation of these two frameworks for the same tests. I wrote some UI tests, which let me know if the application has the correct behavior in different use cases.

Espresso

The implementation of the abstract test case with Espresso.

Espresso in use :

  • Clear implementation but not so readable, as the Toast test implementation  (checkSendPIWentWrong()), which is not so readable if you did not wrote it yourself.
  • Write wait conditions is not readable. Check this great blog post for more information about this. In another hand Espresso is already silently able to wait for events like network calls. This is why I do not need to write any wait condition in checkPIComputationWentOK().
  • Intent can be mocked. That means that fake intents can be throw, and through Espresso, we can test if on of these intent was actually throw. This is not possible with Robotium. In the Github example, this let me actually test that a sharing intent as been thrown (testThatDefaultBehaviorIsWorking()).

Robotium

The implementation of the abstract test case with Robotium.

Robotium in use :

  • Clear implementation, very readable. Toast case (checkSendPIWentWrong()) is readable, wait condition API is also very readable (checkPIComputationWentOK()).
  • Do not have mocked intents.
  • Robotium is able to make screenshots. Personally I prefer use Spoon client to do that, since spoon runner will automatically pull the screenshots from the device under test to embedded them inside the HTML report.

Since Espresso embedded the mock intents, I tend to use more and more Espresso, but Robotium is far from being obsolete.

No Randomness

The key to value the result of tests is to be sure that the result is actually correct. A correct result means that if the test fails, it is because the assertion we made was actually not satisfied. If the result is passing test, then it must be because the assertion was fully validated. Those results needs to be systematic and can not suffer randomness.

Imagine you write a test on an activity which needs to do REST transactions, but when executing the tests, the server is down. Then your test will fail, but not because your application actually failed on what you were testing. Now imagine this occurs on your continuous integration server. Developers may be notified by automatic mails that those tests failed and will loose time search what went wrong, and finally will see that actually it was a false positive alert. Working with test is about building with trust your code.

Working with Android framework brings a lot of randomness like :

  • Per device Android implementation.
  • Endpoint (web server) status.
  • Third party libraries still in development process.

If you start writing UI testing without knowing your sources of randomness in your applications, you can quickly end with tests which pass one day and not the other, making you loose time, or even stop testing.

Lets take the example where you want to test if an activity presents correctly data retrieved from a web server, triggered by a click event. In the GitHub example, this is comparable to the part checkSendPIWentOK() from the test testThatDefaultBehaviorIsWorking() from AbstractTestMainActivity :

mock

This diagram shows exactly why randomness is an issue. The external component which is in this case the server may fail in a inconsistent way.

Avoid randomness

To write good tests, you need to avoid randomness, and you can do that with mocking objects and stubs. If you are not familiar with these concepts, I highly recommend you read this article before going further.

Using mock, we are able to manage the behavior of everything which is external to the given component under test. The theoretical solution is to replace the external component by a mocked component. In a pure theoretical way, it is equivalent to consider those mocked external component as input of the test.

Mocking the server, we could represent now the situation with the following diagram :
mock2 (1)

We do not have anymore randomness, and we are now sure that if the test succeed, or fail, it is because what we want to test is correctly implemented, or not.

About mocking strategies

I use two mocking strategies for my tests :

  • One which allow me to set up mock for a given test, for example mock a specific behavior for a web-server to test a specific feature. This mock only live during the test, and is specific to this test. I name it the test scope mock.
  • One which allow me to  mock a component for the whole application, whatever the test is. I call it the application scope mock.

Test scope mock

The perfect example of this kind of mock is the mock web-server I used in AbstractTestMainActivity, using the flavor mockWebServerPi

@Before
    public void setUp() throws Exception {
        mMockWebServer = new MockWebServer();
        try {
            mMockWebServer.start(Integer.parseInt(mActivityRule.getActivity().getString(R.string.port)));
        } catch (IOException e) {
            throw e;
        }
    }

    @After
    public void tearDown() throws Exception {
        mMockWebServer.shutdown();
    }

    @Test
    public void testThatDefaultBehaviorIsWorking() throws Exception {
        mMockWebServer.enqueue(new MockResponse().setBody(AssetsHelper.getStringFromAsset("stubs/rank_ok.json")));
        userAskPIComputation();
        assertTrue("After a Pi computation, user is able to send its result.", checkPIComputationWentOK());
        Spoon.screenshot(mActivityRule.getActivity(), "checkPIComputationWentOK");
        userAskSendPIOnlineForRank();
        assertTrue("After ranking his result online, the user should be able to share his rank.", checkSendPIWentOK());
        Spoon.screenshot(mActivityRule.getActivity(), "checkSendPIWentOK");
        userAskShare();
        assertTrue("After asking for share, the user should be able to choose how he wants to share", checkShareWentOK());
        Spoon.screenshot(mActivityRule.getActivity(), "checkShareWentOK");
    }

    @Test
    public void testThatServerIssueDisplayToast() {
        mMockWebServer.enqueue(new MockResponse().setResponseCode(500));
        userAskPIComputation();
        assertTrue("After a Pi computation, user is able to send its result.", checkPIComputationWentOK());
        Spoon.screenshot(mActivityRule.getActivity(), "checkPIComputationWentOK");
        userAskSendPIOnlineForRank();
        assertTrue("A Toast message should be displayed.", checkSendPIWentWrong());
        Spoon.screenshot(mActivityRule.getActivity(), "checkSendPIWentWrong");
    }

Before each test, the setup() function is call, which create a mock web-server running directly on the device on which the test is running. After each test, in the tearDown() function, the web-server is shutdown. Each test start by configure how the mock web server will behave. The web mock server is specific to each test and have a known behavior. So we can test UI which needs to do web call without any server in this case, eliminating randomness.

Plain Java object can be mocked with Mockito.

Application scope mock

In the GitHub example, I present another way to mock component, using Dagger 2. If you do not know what Dagger is, I strongly recommend you to read this awesome article about it.

So Dagger lets me inject dependencies. You can see that the GitHub example has several flavors. They all have in common than the REST interface is designed using an … interface through Retrofit. The client which implements this interface is injected using dagger, and thus, flavors do not get injected the same client. For example, approximationPi and exactPi flavors get injected a classic RestAdapter (see their DataModule) which will actually do the network request, when DaggerMockedPi get injected a plain Java object where I implement myself the REST interface (see its DataModule). Thus, all the REST calls inside the application will end up on this mock implementation, which let me mock all the REST calls for the whole application.

More than that, we can also see here than the whole service / network stack is bypassed (the HTTP request, the de-serialization of the response) which could be source of randomness.

Conclusion

In this first part, I tried to explain my approach about testing on Android. From my story, I ended up with testing strategies which I think can bring great value in Android development.

Any feedback will be appreciated, on this article or on the Github project !

Cheers (:

Go social !Tweet about this on TwitterShare on Google+Share on FacebookShare on LinkedInEmail this to someone

4 Comments

  1. Awesome post. I’m looking forward to the PART 2.

  2. samson akisanya

    March 11, 2016 at 2:40 pm

    Hi i was trying to run the tests on the project
    sApplicationComponent = DaggerAppComponent.builder().appModule(new AppModule(application)).build();

    • Vincent Brison

      March 27, 2016 at 5:16 pm

      Hi,

      Have you more information ? I am not able to reproduce. If so, please report the issue here.

      Thanks, BR.

Leave a Reply

Your email address will not be published.

*

twenty − one =

© 2017 Vincent Brison

Theme by Anders NorenUp ↑

Fork me on GitHub