Part III: Statistical Reports

Statistical Reports (Overall , Step Performance and Resource)

As I mentioned in the Part 2 of my blog on What is new in Rational Test Workbench Eclipse client report v8.6 release. At the end of each test run t he statistical report, live report, and test log are displayed.

If you want to get a statistical report on your device resource of each test run. You have to switch to the performance test runs view tab.  From here, I can quickly view the overall report, step performance report and resource report in details.

  1. Overall Report
    It gives a quick overall run summary of each test run and provide the overall Mobile and Web UI Test health information as as how many of my steps have passed and how many steps have failed, and etc.

If I switch over to my Step Performance tab. I will see the performance report Summary based on the filter that I have configured.

2. Step performance report
It gives me a nice quick summary and breakdown of the response time measurement that it took to complete each step. What is the minimum, maximum and average response time that I get for each step to be completed before it moves to the next

This is very useful for me to help improve my end-user performance experience and help development to better understand what could  potentially be a performance bottleneck in my application based on my tests run. Note: There are a lot more that you can do with the reports that I didn’t write about. There are tons of filters that you can change, modify, add  customize to your report.

step_Performance

Note: There are various configurations that will allow you to customize the look and feel of the graphic of your report as well. IE: Configured your report to use 3D on bar chart and pie chart.

3. Resource Report
This is where I will be able to see my chart report on CPU, Network Received, Network Transmitted, Used Physical memory and Used virtual memory.

To modify the Mobile and Web Performance Counter, all you have to do is right-click on the chart itself, then select “Mobile and Web Performance Counter”

performance_countersThen  select the Counters to add the report. In my case, I want my chart in the report to display CPU, Network Received, Network Transmitted, Used Physical Memory and Used Virtual Memory on my Samsung GT-I9500 device.

MobileandWebUI_counter_filter

By moving the mouse over to each data point on the chart, I will able to see the exact usage information that is happening on the device at that time as well.

Resource_Performance

Advertisements

Part II: Performance Monitoring (What is new in Rational Test Workbench Eclipse Client v8.6 release)

What I love about these new features, is that now I have options that will allow me to capture the resource response time at the application level and/or the device level that maybe contributing to my applications’ poor performance. This information is extremely useful to help us identify our application’s bottleneck and to better improve end-user overall performance experience.

1.   Resource response time measurement for native, hybrid and web application for Android Device
In RTW Mobile 8.6 release, we can now capture response time measurements for Android application for the native, hybrid and web application. We are  able to look at each step within the mobile test. With the use of the resource counter that Rational Test Workbench provides out of the box ,we can look at what is happening at the application and device level that is  contributing  to poor performance response time in the application.

At the end of each test run. I get a nice report with a visual cue of a response time break down of each step in my test execution. Based on the out of the box reports, I can quickly assess and get an idea of how long each step and/or operation takes to complete before it moves to the next step.

Mobile and Web UI report2.   Resource Monitoring to monitor device resource utilization counters for Android and iOS
The resource Monitoring capability, allow us to monitor the mobile application and device/emulator resource such as

  • CPU
  • Physical and Virtual memory
  • Network In and out traffic
  • Battery Level

Step to capture the resource monitoring data from the actual device
You must kick off the test run from the test workbench(Assuming that you have previously recorded your test and it is ready to be tested)

  1. Put your device in a passive mode
  2. Run Test
  3. Specified which device I want to use to run my test
  4. Check the Resource Monitoring check box then click “Finish” (Note: If this is a WebUI application this option will be grayed out)
  5. Pooling interval is what we use to specify the frequency of the resource data collect
  6. At the end of each test run, you will get a statistical report, live report and test logs will be displayed

test_runBelow is an example of my test log report after each test run

Test_logResource Monitoring known limitations
1.   Counter indicating virtual memory is available only for the application being tested
2.   Application specific network data for Android apps is supported on Android 4.4.2           and above on only mobile devices
3.   Application specific network data for iOS apps is captured based only on the most    commonly used network communication APIs, and not all APIs.” – Release notes – Rational Test Workbench Eclipse Client 8.6