UI Performance matters. With the recently launched Android Vitals initiative Google has started to reduce visibility of Android apps in the Play Store that take too long to launch or are not always smooth to use. Besides, nobody really likes to use software that doesn’t respond quickly or is not fluid in use.
Types of bad UI performance
Before digging in, let’s recap on terminology.
Simply put how long it takes to bring the app to foreground. Here we need to immediately distinguish between three different scenarios – cold, warm and hot starts.
1. In cold start your app is started from scratch. This may be either because the app is starting for the very first time or (more likely) it has been started before, but the system has killed the app to reclaim its allocated memory for other apps to use.
2. In warm starts the app process is already running in the background so the app only needs to create the activity the user is navigating into.
3. In hot starts everything is already in memory, so the OS simply brings for app’s activity into foreground.
Of these 3 scenarios we’re mostly interested in cold and warm start times as hot starts are typically rather quick and trivial actions. Android Vitals advises us that cold starts should take no longer than 5 seconds, warm starts 2 seconds and hot starts 1.5 seconds. Note that these are quite relaxed requirements and you should probably strive for better results especially for warm and hot starts.
Low frame rate, also known as “jank”
The widely accept goal for smooth UI is that it should render at 60 frames per second – always. That means each frame your app makes should take no longer than 16.6 milliseconds to render, and that’s including all the work your doing in your app’s UI thread. If your app can’t keep up with this pace the system will have to skip frames, which causes frame rate to drop (jank), which leads to unhappy users.
Tackling UI performance in Profilence Tau
Today we’re making our performance testing feature available in beta that provides answers to both types of bad performance. We’ve also made it very simple to get started.
How to set up
Here’s an automated test case in Tau:
And here’s an automated performance testing case in Tau:
The only difference is the added
process='com.google.android.apps.photos', so you can convert your entire functional tests into performance tests in minutes. We believe performance should be as important acceptance criteria as app functionality itself therefore we worked hard to make this happen with minimal overhead. The added “process” parameter instructs Tau to monitor performance of this particular process in this test case. You can also have multiple processes set here, separated by semicolons.
What you get
And here’s the results, using the same Google Photos app as an example:
Activity launch times separated into cold and warm starts. Note that it may be that your app displays a tutorial or welcome screen on the first starts so we also show the duration of the activity chain start.
Tau also shows how long it took for the app UI to be fully rendered if your app provides this using reportFullyDrawn() like the Photos app does.
The number of janky frames that took longer than the 16.7 milliseconds to render as well as what we call the average active frame rate for each test iteration. The word active means that Tau will not take into account periods of time where your app was not changing anything on the UI. We believe this is the best way to measure perceived frame rate as any waits or pauses happening inside tests don’t affect the result.
Ideally you should aim for a consistent 60 frames per second and 0 janky frames in your apps – all the time.
We can also drilldown into individual test case level to see the exact frame rendering times. This gives a nice overview of where the jank is happening at – notice the large amount of frames that took longer than 16.7 milliseconds to render.
Great! How should I put this into use?
If you’re already using Tau for stability or regression testing, you should annotate your scripts with the “process” parameter. After all there’s very little cost for doing so, the overhead on CPU on the device is very, very low.
If you’ve got an existing set of tests written for a different UI test automation framework – say, Appium – you could consider either migrating them into Tau by rewriting the tests or simply wrapping your tests inside Tau tests.
In case you are doing performance testing manually or by using a camera-based solution consider switching to automated testing to get much faster feedback from regression testing and better understanding of aging in stability testing.
We’re actively developing two features to take the automation further. First, the ability to launch Systrace automatically when there’s a problem to make the data even more actionable. Second, Tau will soon let you know automatically when performance in a test case has regressed between releases provided you’re using the Tau Cloud or Tau Server services.
Stay tuned for future updates!