Does it seem to you that the number of errors in Windows 10 updates has increased over the past two years? If so, do you know why? This question is answered in a recent video by former Microsoft employee (Senior SDET) Jerry Berg, also known as Barnacules.
Berg worked at Microsoft for 15 years, and one of his roles was designing and developing tools and processes to automate testing on Microsoft's operating system. He left the company after the release of Windows 8.1.
According to Berg, Microsoft has significantly changed its testing processes over the past two years. The developer describes in detail how testing was done in late 2014 and early 2015, and how Microsoft's testing processes have changed since then.
In 2014/2015, Microsoft had an entire team dedicated to testing the operating system, builds, updates, drivers, and other code. The team consisted of several subgroups that ran tests and discussed bugs and issues in daily meetings. Testing was done manually by each team and through automated checks. If the specific changes passed the tests, then we could integrate the code into Windows.
The teams ran the tests on “real” hardware in a lab using automated tests. Different systems had different hardware components, such as processors, hard drives, video and sound cards, and other components, so the company could cover a wide range of different system configurations. But this also helped identify bugs that only affected certain hardware.
Microsoft, therefore, stopped using almost the entire Windows testing team as it began to prepare three different systems: Windows, Windows Mobile, and Xbox as a single system. The company began to move most of the testing to virtual machines. This means that testing stopped being done on real systems and many different hardware configurations.
Microsoft employees were then able to use the Windows that was in development on their own computers. The main idea behind this was to be able to debug outside of work hours. Berg says in the video that this is not as widely used as it used to be.
The main source of data, apart from the automated testing systems that exist, comes from telemetry and Windows Insiders. Windows Insider builds are installed on millions of devices and Microsoft collects telemetry data from all of these devices.
If something goes wrong, Microsoft gets information about what and how. One of the issues with collecting telemetry data is that it's just data. The data doesn't accurately explain most errors. If something goes wrong, Microsoft may not be able to discern the relevant bits from the telemetry data. As for Insiders, some report problems and others don't. There are times when bugs seem to have been fixed on some Insider systems, but that doesn't mean they've been fixed for every hardware configuration. Additionally, while Insiders can report bugs, they don't necessarily give Microsoft all the necessary information, which creates huge problems for the developers tasked with fixing these issues.
In 2014/2015, Microsoft's testing team was tasked with analyzing bugs and issues, giving developers the data they needed to resolve them.
Today, Berg says there's only telemetry that developers are trying to analyze to figure out how to fix these issues. And the fixes that are being released for customer devices running Insider Builds can't be properly tested due to a lack of experience. This creates new bugs.
One of the main reasons why Microsoft stopped releasing new feature updates to everyone at once was that undetected issues could affect a large number of the company's customers.
So to avoid mass protests like after the release of Windows 10 version 1809, Windows is released in gradual rollouts and not en masse through Windows Update.
