Part III: Statistical Reports

Statistical Reports (Overall , Step Performance and Resource)

As I mentioned in the Part 2 of my blog on What is new in Rational Test Workbench Eclipse client report v8.6 release. At the end of each test run t he statistical report, live report, and test log are displayed.

If you want to get a statistical report on your device resource of each test run. You have to switch to the performance test runs view tab.  From here, I can quickly view the overall report, step performance report and resource report in details.

  1. Overall Report
    It gives a quick overall run summary of each test run and provide the overall Mobile and Web UI Test health information as as how many of my steps have passed and how many steps have failed, and etc.

If I switch over to my Step Performance tab. I will see the performance report Summary based on the filter that I have configured.

2. Step performance report
It gives me a nice quick summary and breakdown of the response time measurement that it took to complete each step. What is the minimum, maximum and average response time that I get for each step to be completed before it moves to the next

This is very useful for me to help improve my end-user performance experience and help development to better understand what could  potentially be a performance bottleneck in my application based on my tests run. Note: There are a lot more that you can do with the reports that I didn’t write about. There are tons of filters that you can change, modify, add  customize to your report.

step_Performance

Note: There are various configurations that will allow you to customize the look and feel of the graphic of your report as well. IE: Configured your report to use 3D on bar chart and pie chart.

3. Resource Report
This is where I will be able to see my chart report on CPU, Network Received, Network Transmitted, Used Physical memory and Used virtual memory.

To modify the Mobile and Web Performance Counter, all you have to do is right-click on the chart itself, then select “Mobile and Web Performance Counter”

performance_countersThen  select the Counters to add the report. In my case, I want my chart in the report to display CPU, Network Received, Network Transmitted, Used Physical Memory and Used Virtual Memory on my Samsung GT-I9500 device.

MobileandWebUI_counter_filter

By moving the mouse over to each data point on the chart, I will able to see the exact usage information that is happening on the device at that time as well.

Resource_Performance

Part II: Performance Monitoring (What is new in Rational Test Workbench Eclipse Client v8.6 release)

What I love about these new features, is that now I have options that will allow me to capture the resource response time at the application level and/or the device level that maybe contributing to my applications’ poor performance. This information is extremely useful to help us identify our application’s bottleneck and to better improve end-user overall performance experience.

1.   Resource response time measurement for native, hybrid and web application for Android Device
In RTW Mobile 8.6 release, we can now capture response time measurements for Android application for the native, hybrid and web application. We are  able to look at each step within the mobile test. With the use of the resource counter that Rational Test Workbench provides out of the box ,we can look at what is happening at the application and device level that is  contributing  to poor performance response time in the application.

At the end of each test run. I get a nice report with a visual cue of a response time break down of each step in my test execution. Based on the out of the box reports, I can quickly assess and get an idea of how long each step and/or operation takes to complete before it moves to the next step.

Mobile and Web UI report2.   Resource Monitoring to monitor device resource utilization counters for Android and iOS
The resource Monitoring capability, allow us to monitor the mobile application and device/emulator resource such as

  • CPU
  • Physical and Virtual memory
  • Network In and out traffic
  • Battery Level

Step to capture the resource monitoring data from the actual device
You must kick off the test run from the test workbench(Assuming that you have previously recorded your test and it is ready to be tested)

  1. Put your device in a passive mode
  2. Run Test
  3. Specified which device I want to use to run my test
  4. Check the Resource Monitoring check box then click “Finish” (Note: If this is a WebUI application this option will be grayed out)
  5. Pooling interval is what we use to specify the frequency of the resource data collect
  6. At the end of each test run, you will get a statistical report, live report and test logs will be displayed

test_runBelow is an example of my test log report after each test run

Test_logResource Monitoring known limitations
1.   Counter indicating virtual memory is available only for the application being tested
2.   Application specific network data for Android apps is supported on Android 4.4.2           and above on only mobile devices
3.   Application specific network data for iOS apps is captured based only on the most    commonly used network communication APIs, and not all APIs.” – Release notes – Rational Test Workbench Eclipse Client 8.6

 

Part I: Mobile Application Testing: What is new in Rational Test Workbench Eclipse Client v8.6 and why you should care?

I could not have been more excited about the new features and capabilities that have recently been released of the Rational Test Workbench Eclipse client 8.6. The following is my quick recap of what I love about this release and things that you need to watch out for.

In case you are new to the Rational Mobile Test Automation and/or Rational Mobile Test Edition v8.6. Why don’t we take a look the Rational Mobile Test Automation Components first

Mobile Test Automation Components

1. RTW Test authoring Eclipse Client
Installed on the user desktop. You will use the RTW Test authoring IDE to initiate the record and play back of the applications from the desktop browser. It is also used for managing devices and applications under test.

Note: In case you are curious about what the installation looks like. I captured all the screen shots of each step of the Rational Test Workbench Eclipse Client v8.6 Installation and mobile extension.

2. Rational Test Workbench Mobile Client
Installed on the mobile devices(Android and iOS). It’s used on your mobile device to test the mobile applications and also to facilitate interactions with the Rational Test Workbench. RTW Mobile Client manages test recording and playback on mobile devices and emulators.

3. Rational Test Workbench Mobile Web Recorder
Installed on the mobile devices. It lives on the devices under test. In order to record and playback the mobile Web UI browserh tests, you will need to download and install the Rational Test Workbench Mobile Web Recorder.

For the iOS, the Rational Test Workbench Mobile Web Recorder can be downloaded from the Apple Store

For Android, the Rational Test Workbench Mobile Web Recorder is packaged with the Rational Test Workbench Mobile Client

Mobile Test Automation v8.6 new features and capabilities
1. Unified web UI Testing
What is really neat about the latest release of RTW Mobile Test Automation version 8.6 is the ability to reuse the test across the platforms both on the RTW mobile and desktop browser.

Back in version 8.5, we can only record a web tests on the RTW browser on the mobile and play back on the RTW browser on the mobile device ONLY.

In version 8.6, you can now playback your web UI tests that you recorded from the mobile on the desktop browsers. On the other hand, if you used the desktop browser to record your web UI tests, you will be able to replay them on the RTW browser on the mobile device as well. The unified web UI testing is definitely something that testers will be able to reap the benefits right away.

WATCH OUT FOR!!

    1. For the mobile web UI test, DO NOT use the commercial browser to record your test. Due the security restriction from the browser and the mobile device itself. You have to use the RTW browser (web recorder) that IBM provides through the Apple store to record your mobile web UI test. For Android, the RTW Mobile Web Recorder is packaged with the Rational Test Workbench Mobile Client

    2. Your mobile device supported OS version. In version 8.6, iOS we now support iOS 7.1 and for android we support v4.4

Unable to obtain a valid license key troubleshooting tips and solution

This post covers problems acquiring the Rational Integration Tester (RIT) license. With the administrator privilege, through the IBM Installation Manager, we were able to successfully import the RTW’s product activation and launched the Rational Integration Tester. However, while trying to launch RIT with the other user that doesn’t have the administrative privilege on the machine. The user was unable to obtain a license and it failed with error “Unable to obtain a valid license key; use Installation Manager (Manager Licenses) to specify the location of a floating license server or to import a product Activation Kit

Environment

  • Window 7 Operating System

  • Rational Integration Tester 8.5.1

Symptom

With the administrator privilege, through the IBM Installation Manager, we were able to successfully import the RTW’s product activation and launched the Rational Integration Tester.

However, when we are trying to launch RIT with the other user that doesn’t have the administrative privilege on the machine, the following error message occurred

[CRRIT8371E] Unable to obtain a valid license key; use Installation Manager (Manager Licenses) to specify the location of a floating license server or to import a product Activation Kit

license-errorRoot cause

  • Environment and/or Operating system’s user policy setup might not allow the read/write access Program Data directory

  • When RIT was launched with a windows user, this user doesn’t have the read and write permission to the C:\users\Program Data directory

Troubleshooting tip and work around

  1. Logging in as the user who is unable to acquire the RIT license, and check and make sure that this user is allowed to read and write to the C:\users\Program Data Directory. Underneath the data directory you will see a directory named IBM.

  2. As a work around, please have the infrastructure administrator manually change the permission to allow the read and write access to this directory as well as IBM sub-directory as well.

RTW for iOS (iPhone/iPad) application testing lesson learn and troubleshooting tips

About a month ago, I had an opportunity to work on the proof of concept and proof of technology of the mobile testing on the iOS devices. During the POC/POT, we ran into several issues and it took us some time to setup the environment. For folks that are new to the Rational Test Workbench (RTW) for the iOS platforms, I believe that what have learned during this POC/POT could be very beneficial to you.


How could we use RTW to test iOS applications on the iOS devices (iPad/iPhone)?


Prerequisite before you proceed to Instrumenting iOS applications on an iOS device

  1. Install Rational Test Workbench Eclipse Client and ensure that Rational Test Workbench Eclipse Client and Rational Test Workbench Extension for Mobile are selected

  2. Install Xcode 4.6. or newer on a Macintosh computer running OS X v10.8 Mountain Lion or newer. I personally prefer a newer version Xcode 5, because it comes with a simulator that I can use to run, test and debug my application locally on my Mac.

For those of you who are new to the iOS development and testing, you might not be familiar with Xcode. Xcode is an integrated development (IDE) containing a suite of software development tools developed by Apple for developing software for OS X and iOS. In order to run test and debug your application, you could use either a simulator or use the real device.

  1. Simulator:- It is free and it’s available through iOS SDK and Xcode 5

  2. Real device and if you want to make your apps available in apple’s store:- You will need to join the iOS Developers Program and there is a fee of $99 per yer that you will have to pay.

Before any testing of the mobile application on any of the iOS devices can be done. We must install and push an instrumented iOS application on a mobile device. The application must first be added to the test workbench. I won’t go in much details on how to do this. A step-by-step instructions could be found at Instrumenting iOS applications on an iOS device.

In order to register an iOS application to the test workbench. The test workbench needs to know the application when it received the recording log in order to produce a complete test. The instructions (Instrumenting iOS applications on an iOS device) will guild you through the instrumentation of the application under test by running the rtwBuildXcode.sh script. The instrumentation augments the AUT with code that allow you to record and play back a test. Below is a recap of a couple of issues that we ran into and what we did to resolve it.

 

Issue #1 If for some reason the rtwBuildXcode.sh script failed to run successfully

Troubleshooting tips

  1. Check to make sure that the current user have the appropriate permissions to run the script. If the permissions are set incorrectly, it could be easily change via a chmod command

  2. On your Macintosh computer, open the Macintosh Terminal application and change to the build-script folder.

    Type the following command to instrument the AUT and send it to the test workbench as an incoming application

    <unpack_dir>/build-script/rtwBuildXcode.sh <.xcodeproj file>

A lot of people get the step #3 wrong especially when they have to entered the DO NOT include /mobile at the end of the Workbench URL.

Example of a good <workbench URL>  http://192.0.2.24:7878

Example of a bad  <workbench URL>  http://192.0.2.24:7878/mobile

 

Issue #2 The build script was run successfully but you are unable to record and keep getting the error message of “Cannot Open Page Safari cannot open the page because the address is invalid”

 

Troubleshooting tips

  1. You would typically get this error when the application installed on the device is not the one setup as part of the RTW. When we instrument the application with the rtwBuildXcode.sh script. The application get a unique uuid number. When you get this error message it means the uuid from the application on the device is not the expected one. Our instinct when something like this failed,  usually what we will do is to remove the application and try to add them to the device again. If you want to remove and add your application again, please make sure that the previous application is removed completed. Make sure that you click the the application on the device long enough to get the cross so you can remove the application properly.

  2. Launch safari and go to the web client (http://<workbench ip>:7878/mobile) through safari, then do managed application again and make sure that the application finished installing


Q&A section


Question #1 This iOS application is not going to be available on the Apple’s store but it will be available internally to my company’s employee. Is it still required for the iOS developer profile to be install on iPhone/iPad?

Answer: It is crucial that the iOS developer profile be installed on your iOS devices. RTW has no control of this at all. When you connect the device (iPhone/iPad) to the Mac it will appear in Xcode organizer. It is important that your device be attached to your provisioning profile.

Question #2 What is the best way to check if the iOS devices is properly configured from an apple license point of view?

Answer:Connect your iOS device to the Macintosh computer, then open Xcode against the application Xcode project and attempt to run the application on the connected device rather than on the iOS simulator.

Question #3 What is the best command line options to capture the log file if the instrumentation failed?

Answer:  sh -x rtwBuildcode.sh [xcode project] [RTW serveur] >log 2>&1

Also it is a good idea to collect the project.pbxproj file in the .xcodeproj folder and send them  to support or engineering for further investigation and analysis as well.