Acceptance Testing for iOS

If you prefer to dive straight in to the code and skip the article then head across to the quick start OCSlimProject docs

In a world of sophisticated and complex mobile Apps, business fortunes are increasingly dependant on App software. How do we get iOS Apps right early, and keep them working as intended?

Acceptance Testing is an essential part of the software development process but often misunderstood.

Conventional wisdom says that Acceptance Testing on mobile is inevitably fragile, slow and complex. Using Fitnesse (a lightweight, open-source testing framework) we can answer the question "Can we submit yet?" within seconds, not days.

Acceptance Testing is critical a step in building software with real business value. Without it, how can we be sure we're building the right thing and that we don't miss something important? Often requirements are written in prose - which is full of ambiguity and open to interpretation. This ambiguity makes it inadequate for explaining the cold hard logic a computer demands.

To fill in the blanks developers are often left making decisions in isolation - decisions that really should be made through collaboration with customers, analysts, and QA. Is it any wonder then that what we build is often not quite what someone thought they were asking for? Things that should be irrefutable turn into post facto time consuming debates about what things should be.

There is a solution for this. It's the crazy idea that we would ask the customer to write their requirements as tests.

"Customers write functional tests so that their confidence in the operation of the program can become part of the program, too. The result is a program that becomes more and more confident over time - it becomes more capable of accepting change, not less" - XP Explained

When written as tests these requirements have no ambiguity, they're right or they're wrong, red or green. If specifications are translated from ambigous statements into irrefutable acceptance tests, and these tests actually run against the latest integration, we would know exactly when we we're done, and that our software was built precisely to the intended design.

Where did it all go wrong?

The history of testing and iOS applications goes like this. A few developers pioneered in the space, often individuals or small teams, building reasonably simple apps - there was no legacy. These apps probably received some cursory testing but were small enough developers could manage that, probably manually. But as the industry exploded armies of developers were being hired to build ever more sophisticated and complex apps. In line with this complexity, quality problems emerged. The need for further QA rigour become the norm (no one seemed to think they should ask those developers why they were producing such low quality work in the first place?)

And so testing cycles consisting generally of people manually pushing at screens became the norm. But then everyone noticed how long this was starting to take, the tests scripts grew, the numbers of testers grew and the quality continued to get worse not better. Some developers thought things like TDD would help, but that seemed like it would slow the developers down. An alternative more preferred management solution was typically "Let's automate it", meaning, "instead of getting humans to prod at screens, we'll get those humans to tell the computers to prod at the screens"!

This approach has been promoted by the emergence of several tools that call themselves acceptance testing frameworks (they're actually little more than wrappers to UIAutomation), Calabash, Frank, KIF, Appium, UIAutomation, all promising the ability to remove the need for an expensive army of manual testers - automating instead with an army of automated simulators. It helped that these tools came with whizz bang terms like BDD and Ruby to impress the boss. Unfortunately this approach is flawed (just the number of competing failed solutions all promising the same thing, should be a smell) and mostly damaging - causing more harm to the development of mobile software than good. No one else tries it like this, why would mobile?

Most iOS developers will likely now have come across the ongoing failed attempts by QA deptmartments attempting to automate testing. Here's why:

  • To automate these tools you have to code them and not simple code, complicated code. More code, more maintenance, more bugs. QA spends most of its time not checking the App but just trying to keep their own code working and in sync with the thing they're testing.

  • Tests are all dependent on an App running on a Simulator and its only way to drive the automation is through UI. Simulators don't play well, they tend to be fussy and break unexpectedly when we use them for things they were never intended to be used for.

  • Because these tests run through the UI and have to navigate your App like a human, change the flow of your app, half your tests might fail, and not tests that check the flow, tests checking something completely unrelated. That happens a few times and people start to lose confidence in these tests.

  • UI is a notoriously hard way to excercise all the possible business logic. Want to test what happens if your API goes belly up? You're going to need that localhost mock webserver that emulates that entire API (oh and keep that it in sync with all the behaviours of your real service).

  • QA can't even start the work until the feature is done, and so are always running behind the actual development.

I could go on, but I'll trust that anyone who's spent anytime in iOS will have witnessed some, if not all of these problems.

So what's the alternative?

The conventional wisdom is that this is just how it is. And unfortunately it's QA teams that also tend to promote conventional wisdom that it's just a slow, fragile and complex process. There's a vested interest in this kind of work and as no one questions it, because of conventional thinking it remains prevalent. That's unfortunate for our platform.

Testing is a multi-faceted process, but let's focus on the functional testing which Automated UI Testing is meant to provide. What do functional tests do and where do they belong?

  • Acceptance Tests are different from Unit Tests. Unit tests check the code is right, Acceptance Tests check that we have the right code. There's an important distinction. You might have the most fabulous bit of code working, but if that code doesn't do what was asked for, it's worthless.

  • Acceptance Tests tests the brains of your application, it excercises the integrations between mutliple components of your system through their API's and sees that together they are exhibiting the expected behaviour.

  • Acceptance Tests are NOT user interface tests, they don't check any of UIKit layer of your application, View Controllers should not be part of your Acceptance Tests. Acceptance Tests work just at the level beneath your UI giving inputs and responding to outputs, just like Controllers in MVC are intended to do.

Testing with Fitnesse

Great software requires collaboration and communication. Fit is a tool for enhancing collaboration in software development. It's an invaluable way to collaborate on complicated problems -and get them right - early in development. - Ward Cunningham

To be able to do what's described above requires a tool. UI testing exists because it seems obvious that it is the only interface available for us to poke at. But there is another interface, a programming interface and that's the one we should be testing. It's fast and it's not subject to the whims of the latest design change or UI or animations. As we're at the API level we're able to easily substitute parts of the system such as databases and networks to create the conditions to be able to excercise any scenario. The most widely accepted tool for writing and running these kinds of tests is Fitnesse. It's created by people who know about this kind of thing so we should pay attention.

Fitnesse has been around for quite some time and a reasonably well accepted standard in the Acceptance Testing world outside of Mobile. It turns out that iOS apps are just made of code like any other and can be tested with Fitnesse. Fitnesse is in itself just a Wiki which takes inputs and passes them to the System Under Test through a layer called Slim (Simple List Invocation Method) "a bare bones RPC system". There are various implementations of Slim for different platforms and fortuantly one for Objective-C also exists and available as a Pod.

You can read lots more about the thinking around Fitnesse and acceptance testing. Here we'll walk you through the basics of getting your iOS App ready to test with Fitnesse.

We're going to use the example from the Slim Decision Table documentation Should I Buy Milk? for our example.

'Should I Buy Milk?'

Assume we have an awesome new idea 'Should I Buy Milk?' it's a sure fire bet to ride this IoT wave all the way to fame and fortune. We know it's going to be a competitive market though so we've chosen to ensure we're building the right thing from the get-go so we're going to use Acceptance Tests and Fitnesse to help us do this. Let's get started.

  • Create a a new iOS App Project called "ShouldIBuyMilk'.

  • Download OCSlimProject

  • Run $ make from within the OCSlimProject folder. This adds the necessary Xcode templates for creating Acceptance Test targets

  • Add a new Target using the 'iOS Acceptance Test' template we just installed under iOS -> Test. This will be our Acceptance Tests app using Swift as the language (Obj-C also works great). I typically just call mine "AcceptanceTests" the name doesn't matter.

  • Set your project up using pod init and add cslim to your AcceptanceTests target in the generated podfile. This is the magic layer between Fitnesse and your App.

      target 'AcceptanceTests' do
      	pod 'OCSlimProject'
  • Perform a Pod install or update and open your newly created ShouldIBuyMilk.xcworkspace generated by CocoaPods.

If everything is setup correctly then Running the App from Xcode you should see the Debug message "getaddrinfo: nodename nor servname provided, or not known" if you see that, that's actually a good thing!

Install and Launch Fitnesse

  • Launch Fitnesse by running the script ./LaunchFitnesse that's generated in your project's root directory
  • Follow the prompt to download Fitnesse
  • You're now ready to run fitnesse, (Fitnesse uses Java so follow any prompts to install Java for OS X).
  • The next step is to add a Test page. Edit your FrontPage and add 'ShouldBuyMilk'.

Writing your first Acceptance Tests

At this point I'd like to introduce our bright Business Analyst who has kindly described the feature they think is going to make for a killer "Should I buy milk?" App. After describing how this works, they've provided a table straight from Excel that we can copy and paste directly into Fitnesse. It clearly describes the various behaviours under which the App should tell the user if they should buy milk or not. (It's a bonus if you can get your BA to write these directly into Fitnesse themselves. You should endevour to deploy Fitnesse within your team in a way that anyone can view and edit it to facilitate this.)

  • Click the ShouldBuyMilk? link you just added and paste in the following table and save.

      |should I buy milk                                              |
      |cash in wallet|credit card|pints of milk remaining|go to store?|
      |0             |no         |0                      |no          |
      |10            |no         |0                      |yes         |
      |0             |yes        |0                      |yes         |
      |10            |yes        |0                      |yes         |
      |0             |no         |1                      |no          |
      |10            |no         |1                      |no          |
      |0             |yes        |1                      |no          |
      |10            |yes        |1                      |nope        |
  • To enable testing of the ShouldBuyMilk? goto Tools > Properties and change the Page Type to "Test".

  • You now see a "Test" button, go ahead and hit this.

Implementing your first Fitnesse Fixture

What you should see when hitting test is the iOS Simulator fire up, showing that Fitnesse is in fact now attempting to talk to your app, but we haven't yet given it what it's expecting to talk to. This is represented by the error "Could not find class ShouldIBuyMilk.".

  • To fix this we go add a class to our Acceptance Tests target.

      class ShouldIBuyMilk: NSObject {
  • Build your Target and Run the test again. "Should I Buy Milk?" should now be green. We now clearly have a Web page talking directly to our iOS app, kinda cool, show it to your friends they'll be impressed!

What we can see from our table is that it's in the format of a decision or truth table using columns of inputs with a final column checking the output denoted by the '?' and each row being a new test case. Succinctly and compactly without any repetition. Fitnesse is telling us it was unable to find these inputs/outputs, let's start with the inputs.

  • Add the input variables "cash in wallet", "credit card", "pints of milk remaining" in their camel cased form in the Fixture class.

      var cashInWallet = ""
      var creditCard = ""
      var pintsOfMilkRemaining = 0
  • Build the Target and Run the Tests again, the warnings in the input columns should be gone.

  • Add the output as a method returning something simple enough to pass the first test.

      func goToStore() -> String {
          return "no"

Congratulations! You have your first running tested features passing green. Notice how this way of development is almost exactly the same as the TDD Red, Green, Refactor. With this specification ready you now are able to turn your attention to developing the feature, using each of the failing tests as a guide for what the next work is to do.

Making the rest of the test pass, I leave as an exercise to the reader. In doing so it's important to remember the fixture class 'ShouldIBuyMilk' is not the actual application code.

Fixtures should only be delegating calls to your real Apps code and returning responses from your Application. They should be extremely lightweight and therefore quick to write and easy to maintain. To afford this it's important that we stay away from the top view layers of the application otherwise we wander into the same problematic areas that UI testing faces.

This however leads us to a question about how to write code that we can easily decouple from the UI and replace with what is in effect and alternative UI such as Fitnesse. For that, the answer is clean architecture and the subject for whole other post.

Join me for a 2 day workshop 'Mastering TDD & BDD for iOS' at CodeNode 22-23rd Sep - Early Bird Pricing ends 26th Aug