Apr 29, 2020
Hello and welcome! We’ll be exploring some of the misconceptions and challenges of UI and app testing and how to overcome them.
User Interface testing (or UI testing) has often been regarded as one of the more straightforward categories of tests, and there’s a lot of truth to that. There’s little in the way of abstraction to deal with, what with the widgets or UI elements being right there in plain sight. There’s also a very clear cut way we intuitively expect a UI to behave. Text inputs are there to type stuff in, buttons are meant to be clicked, labels are there to present information to users, and you don’t have to be a tester or developer to be familiar with these very basic concepts.
But of course, there’s more to this than meets the eye.
Modern apps and websites are back-end driven. That’s just a fancy way of saying, the UI just outputs data or states that are relevant to users based on what the server (or API) tells it. And this is the point where our little dive will begin.
Let’s start with a simple scenario: Text inputs
Typical text input widgets.
We’re given some text input fields in a mobile app that we need to test.
We draft some positive test cases like a minimum and maximum number of characters allowed, a check that only valid email formats are accepted and that only alphanumeric characters can be used for non-email fields.
We also draft a set of negative cases like usage of non-alphanumeric characters, more or less characters than the character limits allow and invalid email formats. Nothing too fancy.
Having gone through all that, our first instinct is usually: “Well, this is pretty comprehensively tested. We know the intended behavior works and that non-intended behavior does not. We’re all set!”
Our first instinct would be wrong, however. To find out why, let’s dive deeper into how widgets work.
Components of a typical widget.
A text input widget will typically have some form of input validation, which we covered quite well with our initial batch of tests. But once the form is submitted, the widget may also have a response handler component (and most widgets usually do) that we have not tested.
To understand how the UI is put into motion by the most common of user interactions, let’s dive in a bit deeper.
Data flow from UI to API and back again.
The UI submits the form which is passed through input validation, and if valid, is forwarded via a HTTP request (POST in our case) to an API. The API, having received the POST request and its body responds to the app. The response handler needs to decide how to handle that response and pass it to the UI.
Remember our tests from a few paragraphs ago? They tell us nothing on how well the response handler is working. The response handler is only concerned with what response the API provides. Our earlier tests will most likely elicit a 200 response from the API, signaling all is well and the form was successfully submitted.
Most common status code ranges used.
With this in mind we can assert that, at best, we have only checked the proverbial happy path. But we still have 4xx and 5xx status ranges we should verify our response handler against before we can confidently state the widget has been comprehensively tested.
Put on your diving suite! We’re diving even deeper.
Order status widget under test
Let’s assume you need to test a widget that shows the status of the last order you placed.
We know the widget will show a price total for the order, the number of the products in the order, an image of one of the products and a progress bar.
We’re also told the progress bar supports 8 states. The kicker is, only one state is testable via normal flow and some states change the widget layout altogether.
Let’s go on to say we need to test that the price, styles and estimated delivery labels in the widget are updated via API response, and are not hard-coded. We’re also interested in verifying the progress bar and that its states update as expected.
There are several problems with our seemingly reasonable requirements, the biggest of which is, the approach we just defined implies the API we’re tied to is not a black box and that we can obtain access to it.
What’s more, we don’t know what services our API is dependent on to generate a state for the widget. Maybe the API we’re using needs a shipping, order and inventory service to provide information it aggregates into a response that our app expects. Or maybe not.
Hypothetical API dependency
The point I’m trying to make is, we shouldn’t be concerned with the whole architectural layout of our system, or with dependencies outside the scope of our app under test. We’re looking for simplicity.
Let’s quickly go back to our first text widget example. We established that to comprehensively test our component, response handler and all, we’d like to elicit a specific response from our API.
Mocking and its benefits
What we need is the response and someone to serve it to the app.
Think of it like this. A PRODUCTION API and a STAGING API can be quite different and we switch the apps we test between them them all the time.
There can be database differences, response structure differences, functionality differences, you name it. The potential in differences between the two is significant, but our app is not picky. So why should we? What we REALLY need is a viable API response our app under test expects and someone to serve that response. Fortunately, there’s a term for that: a mock API.
I won’t go into the specifics of what test doubles are (which is what a mock is) but I will point you to some very handy resources you can check out if you’re interested in learning more.
- Test Double — Martin Fowler
- Test Double — xUnit Patterns
- Mocks Aren’t Stubs — Martin Fowler
- Command Query Separation — Martin Fowler
- Test doubles — fakes, mocks and stubs
There are a lot of tools that provide simple ways of mocking APIs or creating stubs but for the sake of example we’ll use WireMock. Feel free to explore and find what works best for you.
WireMock is essentially a transparent proxy server that can serve stubs (that is, pre-recorded responses) to our app requests. It offers a lot of neat features, can be integrated very easily in automation or manual testing flows and is absolutely free.
Data flow between an app under test and a mock API tool
So what will our mocking tool enable us to do?
Once again, we return to our text input example. A mocking tool will enable us to mock any status code we want. That means we can thoroughly test our widget’s response handler as needed and confidently state that we have comprehensively tested our widget.
What’s more, it enables us to generate custom API responses. That means, we can test complex cases like our last order status widget example. Not only can we set custom status codes, but we can also edit whatever we need in the response to elicit whatever state we want to test in our UI, like responses that change widget string labels or change the progress bar of a widget to whatever state we need it to be.
But wait, there’s more!
Let’s assume, you’re building an automation framework for the app of an external client. The client’s API is a black box and the app you’re testing is not much better. You know that changes to the API will result in undesired behavior in the app you’re testing, behavior ranging from missing widgets, information not getting saved or even crashes. You also need to make sure you app sends data that is interpretable by your client’s API.
There’s actually a name for this sort of test: contract testing.
Contract testing
A contract is an agreement between a client and a provider that outlines expectations, which in turn inform how the interaction between them should work.
Clients have responsibilities like the type of requests made (HTTP request method), what endpoint the request is aimed at and how the request is actually structured (does it need headers, a specific body, query params, cookies, etc.).
The provider isn’t free of responsibilities either. It needs to ensure that it provides responses to exposed endpoints, offers coherent response codes, response bodies and follows a consistent response schema.
We want to be able to check both sides.
So we want to be able to check that our app sends out valid requests to valid endpoints that our API can use on the one hand, and on the other we want our API to respect its part of the contract and never change response types, status codes or responses schema unexpectedly.
Traditionally, contract testing is done somewhere around the unit testing layer but it’s entirely possible to replicate these checks outside the app or API code structure.
Since we’re using WireMock in our example, it’s worth pointing out that the tool is also able to do request matching.
WireMock proxy server capabilities.
This is useful because we can match our app’s outbound API requests to what the API actually expects in a call. We can verify the request specifications as deeply as we want. We can check headers, request body property order, request body type, status code, cookies and whatever else we need.
This takes care of the client side of the contract. But what about the API?
Knowing what endpoints you need from your API is the first step towards ensuring those request responses are predictable and sane all the time.
Standard API tests usually take care of that side of the contract since they deal with response schema validation, status codes, object consistency and so forth.
Barring that, but knowing all the details of your app requests, you can always create your own suite of tests, weather via Postman, some automation library (I personally recommend REST Assured) or simply checking the debug logs of your app call responses (though this is the least efficient way you can possibly use). You can even use WireMock to verify the responses the API feeds back to your app.
And that’s all there is to that. You’ve also got contract testing covered.
If you’d like to know more about contract testing, here are a few interesting resources to check out:
And if you’d like to know more about integrating WireMock in your workflow, feel free to give this article series I wrote on the topic a look:
What are the takeaways?
Having reached the end of our journey, I think we can all agree comprehensive UI testing is far from straightforward, but that doesn’t mean it needs to be daunting or overwhelming in complexity.
We’ve gone through why the traditional notions of UI testing are not even near sufficient to call a test pass complete. Just because it looks like it’s working doesn’t mean all the components are necessarily tested or functional. We’ve also delved deep into how our app and API work in tandem to deliver valid, usable information to our users and how we can use our knowledge of how app and API work to make our work simpler. Not to mention we’ve gone over concepts like mocks and contract testing which we can now use to secure app quality and boost confidence in our work.
I hope our little swim in the murky waters of UIs, Mocks and Contracts have helped give you a deeper understanding of testing as a whole.
Until next time!
