Android 11 beta watch, part 2

Android 11 beta watch (beta 3 & Final release review)

Summary

Surprisingly not much has changed since beta 2. Crashes in drawer and volume control are still there. But now there’s also modem and messaging crashes and a memory leak in settings. If anything, the final Android 11 release for Pixel 4 is actually worse than beta 2. If you’re using a Pixel 4 you should be aware of these bugs as they might cause some real headaches for you.

How we tested

We designed 100 different test cases that cover common use cases such as making and receiving calls, taking photos and browsing the internet. These test cases were then automated and run for roughly 100 hours continuously for both releases and on both Pixel 3 and Pixel 4. The objective was to find out what kind of stability problems real life end users can expect in long term usage – problems which are likely not detected with typical regression testing. No modifications were done to the hardware or software of the phones apart from setting the developer options on to enable USB connection.

The test set for beta 3 and the final release is the same as for betas 1 and 2.

Key findings

Drawer crashes

This bug was already seen in beta 1. In summary opening the notification drawer or changing the volume using volume keys will occasionally cause the UI to freeze without recovery. Eventually, the Android OS steps in and displays a popup to stop the process.

We’ve found that having more notifications in the drawer (twitter, downloaded files, etc.) seems to increase the odds of the crash. Users that let their notifications pile up in the drawer will likely see an annoying error popup often. Compared to beta 2 this issue is actually more frequent in the final release.

Opening the drawer…
…leads to UI freeze

Modem crashes

In total we saw 6 modem crashes. A modem crash happens in the background but will disrupt all ongoing mobile network connections. For example phone calls will drop, packet based communications like video calls will disconnect and possibly reconnect depending on app behavior.

These crashes were only seen with Pixel 4, not with Pixel 3, leading us to suspect that other devices powered by Qualcomm Snapdragon 855 might be affected as well. This possible issue should be checked by OEMs powering their devices with this chipset.

Don’t let your disk space run out!

Most of the system applications can’t handle low disk space properly. During our testing the device ran low on disk space due to photos and files accumulating in the phone. Ideally Android should make sure there is always enough space left on the internal memory to keep apps operational – even if you could not do much with them. That’s not the case here, as the Photos app is the only one to gracefully handle the scenario by displaying a note that the device is low on space and no further photos can be taken. The launcher is likely the worst offender by entering a mad loop where it crashes, restarts and crashes again all over – for several hours.

Launcher is the heart for the UI and with a broken launcher it is difficult to operate anything with the device. For a non-technical end user, it would be very challenging to overcome such a crash loop which is not necessarily solved even by rebooting the phone.

What is the likelihood of running out of disk space? Kids fill their phones with games; adults with their images and videos. Low cost and mid-range devices typically come with limited amounts of internal storage; good camera phones produce high quality images and videos that take a lot of space.

Crashes in the messaging application

We saw 5 crashes in the Messaging (SMS/MMS) application. The application promptly exits when the user attempts to select a conversation. The reason for the crash is as simple as the application is registering itself for telephony information too many times without ever unregistering.

Crash per day is quite much for one of the most common built-in applications in Android. Even today SMS is still the most widely used data service in the world.

Memory leak in the accessibility settings menu

This is a minor issue but worth mentioning as there is no recovery aside from rebooting the phone. The “Accessibility settings” part of the settings menu leaks a bit of memory every time it is opened. As it lives within a system process (com.android.systemui) it will never be stopped by the Android OS and eventually will cause other apps to fail because of not enough memory will be available to them.

Why is it a minor issue then? Well, how many times are you likely to access that menu? Based on our data, the menu leaks around 7 Mbs of RAM every time accessed. In a device with 6 144 Mbs (6 Gb) of memory it will take roughly 100 times to access the menu to leak 700 Mbs of memory which might start to show. Nonetheless this is caused by sloppy code and could have been very easily fixed if it was detected.

Beta watch conclusions

Overall there were very few changes in stability from beta 1 all the way to the final release. While the camera bugs in beta 1 were fixed, most bugs like with the drawer remain open. Also new bugs like the modem crashes were introduced in beta 3 and messaging crashes in the Final release.

It is odd that Google has not addressed these bugs during the betas and makes one question the usefulness of the beta program. The reason these issues are not fixed can’t be because of lack of data either – our devices in the test lab have sent half a thousand error reports on the drawer bug alone during the past month.

If you have any questions about these results, please do not hesitate to contact us info@profilence.com

You can follow us on LinkedIn or sign up for our newsletter at the bottom of this page.