Mobile Application Testing: Four Signs That You’re Testing Too Late– And how to Stop

It’s easy to determine that you are testing too late. It is best to recognize the early warning signs and address them before the software quality begins to suffer. Here are four common signs you are testing too late and suggestions for what to do about it.

1. Separating developer and tester
Separating the developer from the tester affects the type of testing that is done because the tester will not have the level of understanding of the application’s internal operation and knowledge of the code as developers do. By separating the developer and tester you compromise your application quality by focusing on only the “black box” testing. With the “black box” testing, testers tends to focus on the usability test without any concern or of plan to deal with the internal operation, logic and the structure of the code itself.

In my observation, most of the organizations that are still following the waterfall methodology tend to lean too much on the “black box” testing which is happening at the system test or the system integration test level. To delivery a high quality application, there needs to be a combination of both “white box” and “black box” testing. The lack of the “white box” testing allows too many critical defects that should have been identified at the unit testing phase of the software development lifecycle to escape into the pre-production/production environment causes project delay and add costs to the overall software development project.

However, on the other hand, by paring the developer with test from the beginning of the project, we will no longer focusing just only the usability test. We will be able to look closely at the internal logic and structure of the code earlier, which helps reduce critical defects escaping into the production environment and minimize the project over all risks in the long run.

Often, test teams and developments are globally distributed, paring the developer with the tester might not be easily accomplished. However, the adoption of the application development platform will allow distributed team members to collaborate in real-time, which make this aspiration feasible.

2. Too much focus on the front-end User Interface (UI) Testing
Mobile architecture has a multi-tier architecture. The test coverage is heavily focused on the UI application front-end. There is not lot of focus on the middle tier and back-end services and protocols in the unit test, factional test and performance. This might be because the process of setting up the test infrastructure to support test execution of the code on mobile device can be time-consuming and costly.

If you find yourself focusing too much on the front-end UI Testing, look for the tools that will allow you to emulate the middle tier and back-end service and protocol using service and protocol virtualization. This will help your test team reduce cost and time on setting up the test infrastructure to support the test execution and setting up a complex environment. Adopt the solution to emulate the middle tier and back-end services and protocols so that the test execution can concentrate on the client tier of the mobile application running on the device itself.

3. Manual UI Testing very little test automation
Manual testing is the most common approach to mobile testing in the industry today. No doubt there are many great advantages to using manual testing. However, manually testing is still the most time-consuming, error-prone and costly technique. In my opinion, we can still use the manual testing as part of the UI testing use cases. However, the test team needs to think about a way to capture the UI test record and use it as part of the sub-sequential and include it as part of the automatic test execution later in the testing lifecycle.

This issue can be easily fixed by leveraging the test management solutions that organize the manual test cases, guide the tester through execution and store the test result. Implementing the testing solution that allows the developers/tester to record the UI interaction of the application and create tests is something that needs to be planned for as well. Once you have your test, then use the Test Management tools to launch the tests and orchestrate the execution of the test on the device automatically. The solution should allow your associated tests as implementations of the test cases and run the test automatically for the test teams.

4. Integrated end-to-end and System Integrated Test is being done later in the software development lifecycle

A red flag is raised, if there is no evidence that the integrated end-to-end system integrated test is being done for the functionality test, regression test and etc. In addition a lack of non-functional and usability testing occurring sufficiently early enough in the lifecycle is also shouldn’t be ignored.

In many organizations, the number one contributing factor that adds cost and creates delay for the test teams is the lack of the capability for them to run their test in the production like environment, the environment that will be available on demand for the test team to run their test whenever they want and how they want it. Furthermore, the complexity of the software and system that we are having to deal with are fully integrated and loaded with tons of dependencies with the various hardware, software and services. Testers are spending a lot of time waiting for the hardware, software and services to be available and accessible so, they can begin their testing efforts.

In order to resolve these testing challenges of testing too late in the software development lifecycle, we need to start adopting service virtualization.

Why wait for the services and software components to become available or accessible, start your testing effort earlier, use stubs and virtual components to model and simulate the real system bahaviour, so you can begin testing. Units that are not yet built can be simulated and tested against early on versus waiting for it to be available and/or accessible. By doing it this way compared to the big bang testing of testing every components all at once, units are introduced into the continuous integration cycle in a prioritized, controlled fashion. Therefore, help mitigate risks through earlier testing, improve quality and reduce defects escaping into the production environment.

Since applications are becoming more complex and highly interconnected with the various systems, processes and infrastructure, there is no doubt that doing any kind of application testing will continue to present challenges for the test team. It is imperative that we keep up with the demand to test, deliver high quality software at a faster pace and be competitive in the market. It is crucial for the test team to adopt a new testing strategy that will allow them to test earlier, test continuously and use technologies to help remove any kind of testing bottlenecks that could compromise the overall software quality.

RTW for iOS (iPhone/iPad) application testing lesson learn and troubleshooting tips

About a month ago, I had an opportunity to work on the proof of concept and proof of technology of the mobile testing on the iOS devices. During the POC/POT, we ran into several issues and it took us some time to setup the environment. For folks that are new to the Rational Test Workbench (RTW) for the iOS platforms, I believe that what have learned during this POC/POT could be very beneficial to you.

How could we use RTW to test iOS applications on the iOS devices (iPad/iPhone)?

Prerequisite before you proceed to Instrumenting iOS applications on an iOS device

  1. Install Rational Test Workbench Eclipse Client and ensure that Rational Test Workbench Eclipse Client and Rational Test Workbench Extension for Mobile are selected

  2. Install Xcode 4.6. or newer on a Macintosh computer running OS X v10.8 Mountain Lion or newer. I personally prefer a newer version Xcode 5, because it comes with a simulator that I can use to run, test and debug my application locally on my Mac.

For those of you who are new to the iOS development and testing, you might not be familiar with Xcode. Xcode is an integrated development (IDE) containing a suite of software development tools developed by Apple for developing software for OS X and iOS. In order to run test and debug your application, you could use either a simulator or use the real device.

  1. Simulator:- It is free and it’s available through iOS SDK and Xcode 5

  2. Real device and if you want to make your apps available in apple’s store:- You will need to join the iOS Developers Program and there is a fee of $99 per yer that you will have to pay.

Before any testing of the mobile application on any of the iOS devices can be done. We must install and push an instrumented iOS application on a mobile device. The application must first be added to the test workbench. I won’t go in much details on how to do this. A step-by-step instructions could be found at Instrumenting iOS applications on an iOS device.

In order to register an iOS application to the test workbench. The test workbench needs to know the application when it received the recording log in order to produce a complete test. The instructions (Instrumenting iOS applications on an iOS device) will guild you through the instrumentation of the application under test by running the script. The instrumentation augments the AUT with code that allow you to record and play back a test. Below is a recap of a couple of issues that we ran into and what we did to resolve it.


Issue #1 If for some reason the script failed to run successfully

Troubleshooting tips

  1. Check to make sure that the current user have the appropriate permissions to run the script. If the permissions are set incorrectly, it could be easily change via a chmod command

  2. On your Macintosh computer, open the Macintosh Terminal application and change to the build-script folder.

    Type the following command to instrument the AUT and send it to the test workbench as an incoming application

    <unpack_dir>/build-script/ <.xcodeproj file>

A lot of people get the step #3 wrong especially when they have to entered the DO NOT include /mobile at the end of the Workbench URL.

Example of a good <workbench URL>

Example of a bad  <workbench URL>


Issue #2 The build script was run successfully but you are unable to record and keep getting the error message of “Cannot Open Page Safari cannot open the page because the address is invalid”


Troubleshooting tips

  1. You would typically get this error when the application installed on the device is not the one setup as part of the RTW. When we instrument the application with the script. The application get a unique uuid number. When you get this error message it means the uuid from the application on the device is not the expected one. Our instinct when something like this failed,  usually what we will do is to remove the application and try to add them to the device again. If you want to remove and add your application again, please make sure that the previous application is removed completed. Make sure that you click the the application on the device long enough to get the cross so you can remove the application properly.

  2. Launch safari and go to the web client (http://<workbench ip>:7878/mobile) through safari, then do managed application again and make sure that the application finished installing

Q&A section

Question #1 This iOS application is not going to be available on the Apple’s store but it will be available internally to my company’s employee. Is it still required for the iOS developer profile to be install on iPhone/iPad?

Answer: It is crucial that the iOS developer profile be installed on your iOS devices. RTW has no control of this at all. When you connect the device (iPhone/iPad) to the Mac it will appear in Xcode organizer. It is important that your device be attached to your provisioning profile.

Question #2 What is the best way to check if the iOS devices is properly configured from an apple license point of view?

Answer:Connect your iOS device to the Macintosh computer, then open Xcode against the application Xcode project and attempt to run the application on the connected device rather than on the iOS simulator.

Question #3 What is the best command line options to capture the log file if the instrumentation failed?

Answer:  sh -x [xcode project] [RTW serveur] >log 2>&1

Also it is a good idea to collect the project.pbxproj file in the .xcodeproj folder and send them  to support or engineering for further investigation and analysis as well.