The Anatomy of Stability testing for Android OS update

Google releases a new Android version every year. As an end user, do you ever wonder why the rollout of the update for other than Google’s own Pixel phones takes so long? You might be waiting for new security updates and perhaps improvements for battery life? Still, the update rollout plan is unclear – it’s even unclear if your device will ever get the latest update.

A typical end user does not see the chain of events behind the device brand: the anatomy of an Android OS update. In reality, there are multiple complex steps from Google’s official Android release to eventual update rollouts by OEMs. These steps are not typical only for over-the-air SW updates, but for the development of new Android powered products in general. Some time ago Sony published interesting infographics of “The journey of an Android update rollout”. In this post we’ll drill into stability testing activities that should happen on each step during that journey.

Step 1. Google’s Android update

Google develops and updates Android code against their own device configurations, i.e. the Pixel family of phones. In addition to lab tests, Google runs the Android Beta program, which helps them to collect diagnostics data directly from the field from pre-release versions of Android, from volunteer Pixel phone users.

A few weeks before a new Android version is released, Google releases the OS code to chipset vendors: a collection of source files and platform development kits (PDK) – early release to preferred partners.

Google Pixel4

Step 2. Chipset Vendors

A chipset is a complete processing system contained in a single package. It contains multiple processing parts, memory, modems, and other essential bits and pieces manufactured together in a single chip that’s soldered onto the circuit board. Few OEMs produce their own chipsets, but majority source them from chipset vendors, such as Qualcomm or MediaTek.

Before an Android update can go to OEM’s devices each chipset requires a HAL update (Hardware Abstraction i.e. OS compatible drivers) by the chipset vendor. This means that the update rollout roadmap for OEM’s products is highly dependent on the chipset vendor’s chipset specific prioritizations, which are affected by their business drivers and technical capabilities of the chipsets in scope.

On the chipset vendor’s side, a HAL update for each chipset requires specific implementation and dedicated testing efforts, which are done against a selected reference device configuration per chipset.

Breakdown of a mobile device: this imaginary device is powered by a Qualcomm Snapdragon chipset

Step 3. Reference baseline

For a given product configuration, when an OEM starts receiving code drops from the chipset vendor, and drivers for other modules – such as display – are up to date, it’s time to start testing the basic functionality of it: phone calls, web browsing, etc. This is also the phase for re-patching OEM’s own fixes for bugs in the Android OS. In this phase the quality-oriented OEMs turn into their advantage: they invest in testing and stabilizing this software configuration – “the reference baseline” – as good as possible. If something at this point isn’t right it’s easier to blame Google, or the chipset vendor or even the HW module provider. After merging OEM’s own assets with the baseline, the visibility diminishes: in whose code the bug is hiding? Therefore stability and performance test results from this phase are vital as they serve as an important reference once OEMs start customizing the new Android version with their own changes.

In order to follow up the maturity and performance of this baseline, these testing activities should be composed into a repeatable, systematic process, and done for every code drop received from the chipset vendor.

Obviously, the time this stabilization phase takes differs from product configuration to another, and is highly dependent on the quality Google and chipset vendors bring into the table.

In our earlier blog series about Android 11 update stability, we have seen even the final release standing out with poor stability and performance. Therefore, there are challenges expected for the later phases.

Home menu in stock Android
Application menu in stock Android

Step 4. Differentiation

In the differentiation phase, OEMs start working on top of the – hopefully – clean and stable reference baseline, and start adding distinct look-and-feel and experiences: launchers, skins, platform tweaks, company apps and services. The Android One family of devices – focusing on delivering a “pure” version of Android – can mostly skip this phase and thus gain some speed advantage with update rollouts.

Stabilization of the product specific update can finally start: from lab tests and field tests to beta tests of some scale. Data from different testing activities starts flowing in for analysis to drive this iterative process forward. The more efficiently the data is collected and analyzed the better is the understanding of the overall software quality. In this phase it’s advisable to increase volume with stability tests in the labs. Also, beta-testing should be well-organized and efficient: issues detected in the field should be quickly mapped with issues detected in the lab – this results in faster problems solving.

Home menu in Samsung TouchWiz UI
Application menu in Samsung TouchWiz UI

Final quality gates

Before the rollout of the update can start, there are still some final checks remaining, such as conformance standards tests and any tests an operator might require for OEMs.

Summary

Every product configuration (a combination of chipset, hardware modules, Android version, OEM platform code and OEM specific launchers and apps) is unique, and thus updating each is its own project, which contains a lot of dedicated testing and optimization activities. Google largely tests with their Pixel configurations, chipset vendors test with appropriate reference configurations and OEMs use their own product configurations – in none of the phases, for none of the configurations, testing comes for free by somebody else. Test results from each described step are valuable input for the next, but shouldn’t be trusted blindly. Therefore, rollouts of the updates take time.

During the years, we’ve seen ODMs and OEMs who have mistakenly assumed that Google’s tests in the phase 1 are sufficient and thus largely skip their own test processes – effectively delivering an Android update to their customers that is mostly untested in their own devices. It does make for a speedy update rollout, but can be disastrous to quality and reputation.