Part III: Statistical Reports

Statistical Reports (Overall , Step Performance and Resource)

As I mentioned in the Part 2 of my blog on What is new in Rational Test Workbench Eclipse client report v8.6 release. At the end of each test run t he statistical report, live report, and test log are displayed.

If you want to get a statistical report on your device resource of each test run. You have to switch to the performance test runs view tab.  From here, I can quickly view the overall report, step performance report and resource report in details.

  1. Overall Report
    It gives a quick overall run summary of each test run and provide the overall Mobile and Web UI Test health information as as how many of my steps have passed and how many steps have failed, and etc.

If I switch over to my Step Performance tab. I will see the performance report Summary based on the filter that I have configured.

2. Step performance report
It gives me a nice quick summary and breakdown of the response time measurement that it took to complete each step. What is the minimum, maximum and average response time that I get for each step to be completed before it moves to the next

This is very useful for me to help improve my end-user performance experience and help development to better understand what could  potentially be a performance bottleneck in my application based on my tests run. Note: There are a lot more that you can do with the reports that I didn’t write about. There are tons of filters that you can change, modify, add  customize to your report.


Note: There are various configurations that will allow you to customize the look and feel of the graphic of your report as well. IE: Configured your report to use 3D on bar chart and pie chart.

3. Resource Report
This is where I will be able to see my chart report on CPU, Network Received, Network Transmitted, Used Physical memory and Used virtual memory.

To modify the Mobile and Web Performance Counter, all you have to do is right-click on the chart itself, then select “Mobile and Web Performance Counter”

performance_countersThen  select the Counters to add the report. In my case, I want my chart in the report to display CPU, Network Received, Network Transmitted, Used Physical Memory and Used Virtual Memory on my Samsung GT-I9500 device.


By moving the mouse over to each data point on the chart, I will able to see the exact usage information that is happening on the device at that time as well.



Part II: Performance Monitoring (What is new in Rational Test Workbench Eclipse Client v8.6 release)

What I love about these new features, is that now I have options that will allow me to capture the resource response time at the application level and/or the device level that maybe contributing to my applications’ poor performance. This information is extremely useful to help us identify our application’s bottleneck and to better improve end-user overall performance experience.

1.   Resource response time measurement for native, hybrid and web application for Android Device
In RTW Mobile 8.6 release, we can now capture response time measurements for Android application for the native, hybrid and web application. We are  able to look at each step within the mobile test. With the use of the resource counter that Rational Test Workbench provides out of the box ,we can look at what is happening at the application and device level that is  contributing  to poor performance response time in the application.

At the end of each test run. I get a nice report with a visual cue of a response time break down of each step in my test execution. Based on the out of the box reports, I can quickly assess and get an idea of how long each step and/or operation takes to complete before it moves to the next step.

Mobile and Web UI report2.   Resource Monitoring to monitor device resource utilization counters for Android and iOS
The resource Monitoring capability, allow us to monitor the mobile application and device/emulator resource such as

  • CPU
  • Physical and Virtual memory
  • Network In and out traffic
  • Battery Level

Step to capture the resource monitoring data from the actual device
You must kick off the test run from the test workbench(Assuming that you have previously recorded your test and it is ready to be tested)

  1. Put your device in a passive mode
  2. Run Test
  3. Specified which device I want to use to run my test
  4. Check the Resource Monitoring check box then click “Finish” (Note: If this is a WebUI application this option will be grayed out)
  5. Pooling interval is what we use to specify the frequency of the resource data collect
  6. At the end of each test run, you will get a statistical report, live report and test logs will be displayed

test_runBelow is an example of my test log report after each test run

Test_logResource Monitoring known limitations
1.   Counter indicating virtual memory is available only for the application being tested
2.   Application specific network data for Android apps is supported on Android 4.4.2           and above on only mobile devices
3.   Application specific network data for iOS apps is captured based only on the most    commonly used network communication APIs, and not all APIs.” – Release notes – Rational Test Workbench Eclipse Client 8.6


Part I: Mobile Application Testing: What is new in Rational Test Workbench Eclipse Client v8.6 and why you should care?

I could not have been more excited about the new features and capabilities that have recently been released of the Rational Test Workbench Eclipse client 8.6. The following is my quick recap of what I love about this release and things that you need to watch out for.

In case you are new to the Rational Mobile Test Automation and/or Rational Mobile Test Edition v8.6. Why don’t we take a look the Rational Mobile Test Automation Components first

Mobile Test Automation Components

1. RTW Test authoring Eclipse Client
Installed on the user desktop. You will use the RTW Test authoring IDE to initiate the record and play back of the applications from the desktop browser. It is also used for managing devices and applications under test.

Note: In case you are curious about what the installation looks like. I captured all the screen shots of each step of the Rational Test Workbench Eclipse Client v8.6 Installation and mobile extension.

2. Rational Test Workbench Mobile Client
Installed on the mobile devices(Android and iOS). It’s used on your mobile device to test the mobile applications and also to facilitate interactions with the Rational Test Workbench. RTW Mobile Client manages test recording and playback on mobile devices and emulators.

3. Rational Test Workbench Mobile Web Recorder
Installed on the mobile devices. It lives on the devices under test. In order to record and playback the mobile Web UI browserh tests, you will need to download and install the Rational Test Workbench Mobile Web Recorder.

For the iOS, the Rational Test Workbench Mobile Web Recorder can be downloaded from the Apple Store

For Android, the Rational Test Workbench Mobile Web Recorder is packaged with the Rational Test Workbench Mobile Client

Mobile Test Automation v8.6 new features and capabilities
1. Unified web UI Testing
What is really neat about the latest release of RTW Mobile Test Automation version 8.6 is the ability to reuse the test across the platforms both on the RTW mobile and desktop browser.

Back in version 8.5, we can only record a web tests on the RTW browser on the mobile and play back on the RTW browser on the mobile device ONLY.

In version 8.6, you can now playback your web UI tests that you recorded from the mobile on the desktop browsers. On the other hand, if you used the desktop browser to record your web UI tests, you will be able to replay them on the RTW browser on the mobile device as well. The unified web UI testing is definitely something that testers will be able to reap the benefits right away.


    1. For the mobile web UI test, DO NOT use the commercial browser to record your test. Due the security restriction from the browser and the mobile device itself. You have to use the RTW browser (web recorder) that IBM provides through the Apple store to record your mobile web UI test. For Android, the RTW Mobile Web Recorder is packaged with the Rational Test Workbench Mobile Client

    2. Your mobile device supported OS version. In version 8.6, iOS we now support iOS 7.1 and for android we support v4.4

Mobile Application Testing: Four Signs That You’re Testing Too Late– And how to Stop

It’s easy to determine that you are testing too late. It is best to recognize the early warning signs and address them before the software quality begins to suffer. Here are four common signs you are testing too late and suggestions for what to do about it.

1. Separating developer and tester
Separating the developer from the tester affects the type of testing that is done because the tester will not have the level of understanding of the application’s internal operation and knowledge of the code as developers do. By separating the developer and tester you compromise your application quality by focusing on only the “black box” testing. With the “black box” testing, testers tends to focus on the usability test without any concern or of plan to deal with the internal operation, logic and the structure of the code itself.

In my observation, most of the organizations that are still following the waterfall methodology tend to lean too much on the “black box” testing which is happening at the system test or the system integration test level. To delivery a high quality application, there needs to be a combination of both “white box” and “black box” testing. The lack of the “white box” testing allows too many critical defects that should have been identified at the unit testing phase of the software development lifecycle to escape into the pre-production/production environment causes project delay and add costs to the overall software development project.

However, on the other hand, by paring the developer with test from the beginning of the project, we will no longer focusing just only the usability test. We will be able to look closely at the internal logic and structure of the code earlier, which helps reduce critical defects escaping into the production environment and minimize the project over all risks in the long run.

Often, test teams and developments are globally distributed, paring the developer with the tester might not be easily accomplished. However, the adoption of the application development platform will allow distributed team members to collaborate in real-time, which make this aspiration feasible.

2. Too much focus on the front-end User Interface (UI) Testing
Mobile architecture has a multi-tier architecture. The test coverage is heavily focused on the UI application front-end. There is not lot of focus on the middle tier and back-end services and protocols in the unit test, factional test and performance. This might be because the process of setting up the test infrastructure to support test execution of the code on mobile device can be time-consuming and costly.

If you find yourself focusing too much on the front-end UI Testing, look for the tools that will allow you to emulate the middle tier and back-end service and protocol using service and protocol virtualization. This will help your test team reduce cost and time on setting up the test infrastructure to support the test execution and setting up a complex environment. Adopt the solution to emulate the middle tier and back-end services and protocols so that the test execution can concentrate on the client tier of the mobile application running on the device itself.

3. Manual UI Testing very little test automation
Manual testing is the most common approach to mobile testing in the industry today. No doubt there are many great advantages to using manual testing. However, manually testing is still the most time-consuming, error-prone and costly technique. In my opinion, we can still use the manual testing as part of the UI testing use cases. However, the test team needs to think about a way to capture the UI test record and use it as part of the sub-sequential and include it as part of the automatic test execution later in the testing lifecycle.

This issue can be easily fixed by leveraging the test management solutions that organize the manual test cases, guide the tester through execution and store the test result. Implementing the testing solution that allows the developers/tester to record the UI interaction of the application and create tests is something that needs to be planned for as well. Once you have your test, then use the Test Management tools to launch the tests and orchestrate the execution of the test on the device automatically. The solution should allow your associated tests as implementations of the test cases and run the test automatically for the test teams.

4. Integrated end-to-end and System Integrated Test is being done later in the software development lifecycle

A red flag is raised, if there is no evidence that the integrated end-to-end system integrated test is being done for the functionality test, regression test and etc. In addition a lack of non-functional and usability testing occurring sufficiently early enough in the lifecycle is also shouldn’t be ignored.

In many organizations, the number one contributing factor that adds cost and creates delay for the test teams is the lack of the capability for them to run their test in the production like environment, the environment that will be available on demand for the test team to run their test whenever they want and how they want it. Furthermore, the complexity of the software and system that we are having to deal with are fully integrated and loaded with tons of dependencies with the various hardware, software and services. Testers are spending a lot of time waiting for the hardware, software and services to be available and accessible so, they can begin their testing efforts.

In order to resolve these testing challenges of testing too late in the software development lifecycle, we need to start adopting service virtualization.

Why wait for the services and software components to become available or accessible, start your testing effort earlier, use stubs and virtual components to model and simulate the real system bahaviour, so you can begin testing. Units that are not yet built can be simulated and tested against early on versus waiting for it to be available and/or accessible. By doing it this way compared to the big bang testing of testing every components all at once, units are introduced into the continuous integration cycle in a prioritized, controlled fashion. Therefore, help mitigate risks through earlier testing, improve quality and reduce defects escaping into the production environment.

Since applications are becoming more complex and highly interconnected with the various systems, processes and infrastructure, there is no doubt that doing any kind of application testing will continue to present challenges for the test team. It is imperative that we keep up with the demand to test, deliver high quality software at a faster pace and be competitive in the market. It is crucial for the test team to adopt a new testing strategy that will allow them to test earlier, test continuously and use technologies to help remove any kind of testing bottlenecks that could compromise the overall software quality.