Critics warn Microsoft it needs to fix broken update process

The latest fiasco to befall Windows 10 - Microsoft two weeks ago pulled the fall feature upgrade from distribution and urged those already with a copy not to install it - has triggered calls for the company to get its act together.

"Microsoft really needs to get a handle on this. They're running out of time," said Chris Goettl, product manager with client security and management vendor Ivanti, when asked his reaction to the show-stopping problem of deleted user files.

"If Microsoft wants to continue down this track of the Windows 10 [rapid release] model, its upgrades and updates must be of better quality than they have been."

Four days after releasing Windows 10 October 2018 Update - also known as 1809 in Microsoft's yymm numeric format - the firm shut down access to the feature upgrade. The problem: On some PCs, the upgrade process erased all the files in the Documents, Pictures, Music and Videos folders.

Although Microsoft restarted testing of 1809 with its Insider beta program participants on 9 October, it has not yet restored access to the upgrade for manual download nor begun to deliver it to customers via Windows Update or WSUS (Windows Server Update Services), the default consumer- and commercial-grade distribution networks.

The tech giant has not set a date when the upgrade process would resume with the debacle the biggest yet in Windows 10's short history.

"That's an extremely unwelcome first for Microsoft, and it raises the question of whether the company is moving too fast and breaking too many things in the process," said long-time Windows-watcher Ed Bott of the decision to shut down 1809's dissemination. Bott also called Microsoft's move "unprecedented."

Although commentary varied about the retraction - and what it meant to Microsoft and customers - there were general themes, none of them really new to the debate over Windows 10 and its maker's OS strategy. It was the tenor of the critiques that seemed different.

Quality? What quality?

Calls for Microsoft to improve the quality of its upgrades and updates have been building for some time. Triggered by long stretches of problem-causing monthly updates - ironically, Microsoft dubs these, including the famous "Patch Tuesday" security releases, "quality" updates - commercial customers have been demanding that Redmond do better.

In July, patch expert Susan Bradley, a computer network and security consultant who also moderates the PatchMangement.org mailing list, implored Microsoft's senior executives to read the findings of a survey of IT professionals she conducted.

That survey outlined how dissatisfied IT personnel were with the Windows patching process, in particular the updates' quality.

"Your customers who are in charge of patching and maintaining systems are not happy with the quality of updates ... and feel that it cannot go on as is," Bradley wrote in an open letter to Microsoft's leaders, including CEO Satya Nadella.

Of the more than 1,000 respondents to her questionnaire, 64 per cent were either "very much not satisfied" or "not satisfied" with the quality of Windows' updates. Those opinions were only reinforced by 1809's stumble.

"Between quality releases and feature releases, they are falling down on quality across the board," Bradley maintained in an email reply to questions this week.

"I keep saying that Surface devices should have the absolute best patching experience and look, again this time 1809 is blocked on some of the Surface devices as they need an Intel driver."

Goettl agreed and ran through a string of buggy updates going back more than a year, ticking off Windows 10's Creators Update, aka 1703, Windows 7, and Windows 10 1803 - the April 2018 Update - as examples of editions with recent flaws. "But losing customer data, that's pretty severe," he said of 1809's deleted files.

Concerns over quality - or the lack of it - in Windows' updates generate more than the immediate impact of crippled PCs or time wasted deploying re-releases.

Low-quality updates spook customers, particularly commercial customers, who become leery of immediately installing security patches or look for ways to skip as many feature upgrades as possible.

"Many users and enterprises have lost trust in the entire updating process," argued Bradley, who also writes for Askwoody.com, the Windows-centric website run by Woody Leonard, a columnist for Computerworld. "No business that I personally know of installs updates immediately."

What happened to testing?

When the talk turns to Windows and the quality of its updates, users inevitably ask about testing. Why didn't testing uncover this bug or that flaw? What's going on with testing? Is Microsoft testing its updates at all?

To many, there is a straight line between Microsoft's 2014 dismissal of a large number of internal testers and the decline in quality.

In response, Microsoft points to Windows Insider, the program that previews feature upgrades - but not the monthly security and non-security updates - to a large pool of Windows 10 volunteers. The 15 million Insiders, the plan goes, report problems, which Microsoft engineers fix before the code reaches production PCs.

While file deletions were reported by some Insiders before 1809 was given a green light, Microsoft implied they were passed over because the company had no inkling of their severity.

"To help us better detect issues like this ... we have added an ability for users to also provide an indication of impact and severity when filing User Initiated Feedback," said John Cable, director of program management in the Windows servicing and delivery group, in an 9 October blog post.

Outsiders were simultaneously aghast that Insider failed to produce a fix and critical of the very concept behind the program.

"This had to come up by the Insiders," Goettl said. "How did [1809] not get held up, how did Microsoft not pause it?"

"The Insider program is a great marketing program, [but] it's not a beta testing program," Bradley charged. "Many, many feedback items have no details, no context, no good bug information, [so] there's no way that Microsoft can take any other action other than to blow them off.

"Currently, the Insider program from this side of the wall looks like [a] hurricane-force firehose that for the most part is a waste of an engineer's resources to cull through.

"I am seeing too many good former beta testers say they no longer do the Insider builds because of the fanboy noise and lack of true beta interaction," she concluded.

Can Microsoft fix this?

It's unclear whether complaints about Windows 10's update strategy have reached a tipping point - that seems unlikely for practical reasons, since enterprises really have no alternative to Windows. But the volume of voices calling for change has certainly increased.

"After the debacle around the Windows 10 October Update/1809, the perception that Microsoft has lost control of its Windows 10 update strategy is even more apparent," wrote Mary Jo Foley, like Bott a long-time Microsoft observer who writes for ZDNet.

Foley's advice to Microsoft: Take a break from larding Windows 10 with new features. "It's time for the company to release at least one - and maybe more - Windows 10 feature updates that focus exclusively on reliability/fundamentals and not on features," Foley urged.

Others had different ideas. "Something like [1809] is going to happen again," said Goettl. "Microsoft needs a better break-glass scenario." By that, he meant the company should have a quick-response plan in place "to get these [systems] back in service" as quickly as possible.

For her part, Bradley suggested Microsoft turn back the clock. "[Microsoft needs] a SUVP external testing program that pulls in more enterprises," she said, referring to the former Software Update Validation Program, which previewed security updates to a small group of invite-only organisations, including enterprises, to test patches before they were issued to the public.

"Insider is great for asking 'So what do you think about this?', [but] it's not good for finding these release bugs."

Goettl knew only that things couldn't continue as they have. "If we stay on the track we're on, it's going to be more painful in the future," he said, predicting more 1809-esque problems to come.