Part of my job as a Windows Insider MVP means working with — and on — Insider Preview releases. Last Wednesday (July 31) the Insider Team dropped its latest Fast Ring + Skip Ahead release (20H1: Build 18950). I run that software on three machines here at my office. One runs on a Dell XPS 2720 (Haswell i7-4770S CPU), another on a Lenovo X220 Tablet (Sandy Bridge i7-2640 M CPU), and the last one on a home-built desktop (Haswell i7-4770K). So far, I’ve upgraded 2 of those 3 machines completely. Things could have gone more smoothly, though. There’s been a bit of the old “two steps forward, and one step back” along the way. Hence, the description as Insider Preview 20H1 upgrade follies. Let me explain further . . .
The top lines from the MS Flight Hub list show that 18950 joined the Fast Build Ranks on 7/31/2019.
[Click image for full-sized view.]
About Those Insider Preview 20H1 Upgrade Follies
It’s traditional for Insiders to whine, moan, fret and complain about unexpected and untoward things they experience during a Preview’s many and frequent upgrade installs. I hope my tone here isn’t that strident or negative. But I will be observing some surprises that have bitten me, sometimes in tender places, during this latest upgrade round. Some of the usual litany for upgrades include:
- lengthy download times (this one wasn’t too bad: under 5 minutes);
- long install times (none of the three completed in under an hour; one took over 2 hours); and
- strange behaviors during and after the upgrade process.
This latter topic describes most of what I’m writing about today, in fact. You can also glean a lot useful information about this release from the forum members’ comments among the 179 comments currently posted at TenForum’s 18950 announcement thread. So far, I haven’t experienced some of what others are reporting, but everything I report here is also covered there, too.
When the Going Gets Weird, the Weird Turn Pro
Older readers (including your humble correspondent) will recognize this heading as a quote from the Father of Gonzo Journalism, Hunter S. Thompson. And indeed, a strong element of this attitude is necessary for those who deliberately tackle beta software as complex as Windows 10. I can’t say I enjoy being bewitched, bothered, and bewildered by the doings of this sometimes tempestuous and capricious operating system. But I can say, I’ve gotten used to it, and have learned to cope with or work around many of its vexations.
First things first: if you’re going to install Insider Previews on PCs to see what happens, be prepared for the worst. I use Macrium Reflect as my backup tool, and I have Rescue Media for all of my Insider Preview test machines. I always have a reasonably current backup of those machines, and the Rescue Media handy, should an upgrade result in an inoperative or overly hinky system. That hasn’t happened to me with 18950, but I was certainly thinking about booting to the Rescue Media and restoring a backup once or twice along the path I’m about to describe.
Weird Thing #1: Looping/Cycling Install Progress
To my way of thinking, the Windows 10 install process falls into two major parts. One, there’s a GUI install piece, which describes the part happens before the first reboot. Actually, this is the part of the upgrade/install process where the old/previous OS is running the Windows installer. It’s handling the “getting ready,” “downloading,” and “installing” activities that Windows Update reports while they’re underway. Sometimes what TenForums members call “cycling” occurs during the GUI portion of the install. Often, the path from these three activities occurs in the order just described: Getting Ready → Downloading → Installing. Cycling occurs when the process goes back to Getting Ready after showing Downloading or Installing. This happened with 18950 on two of my three test machines.
Two, there’s the post-GUI install when the kernel of the new OS takes over and you see a solid-blue screen. It reads “Working on updates,” and shows a percentage of completion. The same screen also quite correctly says “This could take a while.” It usually does.
For all three of my 18950 test machines, the completion percentages bounced around while post-GUI install was underway. A typical sequence is to see a reboot at around 30% complete, another reboot at 64% complete, and another reboot in the 80s (either 82 or 84% on my test machines). This time, things got whacky in the 40-80% range. My X220 Tablet rebooted three times at 72%, 74% and 80%, then showed 64% complete after the last of those three reboots. Decidedly weird!
All three test machines got entirely through the upgrade process and booted into 18950. But the i7-4700K desktop immediately showed alarming symptoms, which led me to restore the previous version. The Settings path for this goes: Settings → Update & Security → Recovery → “Go back to the previous version of Windows 10.” Therein also likes the “next weirdness” about 18950: networking oddities. I am, however, happy to report that my second upgrade attempt completed successfully.
Weird Thing #2: Wireless Networking Shenanigans
The desktop includes an ASUS 802.11 AC-56 PCIe (x1) wireless adapter that I installed in that machine. It’s right above the Spectrum WiFi WAP in our master bedroom closet, in my son’s bedroom. It gets great bandwidth, up to 400-450 Mbps when the LAN is otherwise quiet. But after the 18950 upgrade, the NIC wasn’t working. A quick trip into Device Manager explained “A driver for this device is not installed.” Also, for some reason, Airplane mode was turned on for this PC. I wrassled around with the driver just long enough to understand that something odd was up with this particular Win10 install. Not only was I unable to install the driver, but the machine was running incredibly slowly and fitfully. That’s what decided me to roll back to 18941 (the previous build on that PC) and try again.
Initially, I wrote this off to issues with the specific WiFi NIC on that PC. Then as I was writing this piece, I fired up the X220 Tablet after the GUI install phase had completed, to restart the PC and let phase 2 get underway. I immediately noticed that it, too, had Airplane mode turned on. (Easily turned off through the notification panel.) It also showed the Network Globe icon in the notifications instead of the more typical WiFi symbol. (This also reappeared when Airplane mode was turned off.) Fortunately, this machine had no other networking issues with the upgrade, either during or afterward. I didn’t get that on the Dell XPS 2720 but then, it’s connected via GbE wired Ethernet, not WiFi.
All’s Well That Ends Well
Some Win10 Insider Preview releases are better than others. This one took me about two hours longer to transit than a problem-free upgrade does. But I keep watching what’s going on, and learning what I can along the way. That’s what beta-testing a big, hairy, complex piece of software like Windows 10 is like. Mostly, I enjoy it.
Sometimes, the Windows 10 upgrade installer blocks 1903 upgrade attempts. As reported in the Windows 10 Known Issues list on August 1, users running Intel RST version 126.96.36.1992 through 188.8.131.523 will get a message from the Installer (see following graphic) that tells them to upgrade to version 184.108.40.2064 or newer (today’s current version is 220.127.116.111, dated July 10, 2019 as I write this blog post). According to the MS remediation info, version 18.104.22.1684 is recommended for affected devices. But in certain cases, Intel RST stymies 1903 upgrade once the new driver is installed! Fortunately, there’s an easy fix should that happen.
Sometimes, this error recurs after an IRST driver update.
[Click image for full-sized view.]
If Intel RST Stymies 1903 Upgrade After Current Driver Installed, Do This!
Thanks to alert and savvy users on TenForums, I learned about this problem and its fix at the same time. As documented in the thread entitled “I updated RST drivers, still get the 1903 error,” user Rafaboss explains that deleting or renaming the old driver version named iastora.sys (he suggests iastora-old.sys) will let users take advantage of the “fix” button that the upgrade installer now presents for 1903 when something goes wrong on a previous install attempt. The target directory for this operation is the default Drivers directory (C:\Windows\System32\drivers). Other users correctly observe that you can use the Open Source DriverStore Explorer (RAPR.exe) tool to do the same thing. Either approach will do the trick, and allow the 1903 Upgrade Installer to proceed to a successful completion.
Thus, there’s no need to stay stymied, if Intel RST stymies 1903 upgrade on any of your PCs. ‘Nuff said!
Last November, I wrote about force deleting Windows.old from one of my PCs. I just ran into the same thing again today, and was pleased to put that information to work yet again. As sometimes happens, I’d already run Disk Cleanup (and its more powerful Open Source counterpart, mdiskcleanup.exe from the Comet project at GitHub). After the cleanup, a Cortana app related bit persistently hung on in the Windows.old directory. But when I tried to delete it using normal methods, it resisted. That’s when I discovered that my old trick still kills persistent Windows.old.
What Old Trick Still Kills Persistent Windows.old?
The command comes from the old-fashioned command line, and its syntax is:
rd /s /q %systemdrive% \Windows.old
I ran it using the standard C: moniker for my system drive, so that became rd /s /q C:\Windows.old. Inspecting my C: drive, I discovered that the Windows 10 Upgrade assistant was also still resident on this machine. So, just for grins I also executed rd /s /q C:\Windows10Upgrade. I’m pleased to report that this cheerfully and quietly did away with that folder and its contents as well. The screenshot to the left illustrates the “before” and “after” directory listings, showing I successfully removed Windows10Upgrade from my desktop PC.
[Note: If the screenshot is too small to read, click the image for a full-sized view. Then click the back-arrow in your browser to return to this post’s text.]
The Old Trick Works for Other Root-level Folders, Too
That’s what allowed me to get rid of Windows10Upgrade, as I explained earlier. But you’ll want to be careful using this command. You could also easily do away with necessary parts of the OS, such as Program Files, Users, or even Windows itself. If you can’t recover that stuff from the Recycle Bin, you’ll be left with a broken Windows installation. That’s among the many reasons why I back my production system up with an image backup in Macrium Reflect at 9:00 AM every morning. Don’t mess around with root-level system drive stuff unless you have a current image backup, too!
My Surface Pro 3 (vintage 2014) is still chugging gamely along in my office. I use it as an Insider Preview Slow Ring test machine. Each time I upgrade the OS (about weekly, these days), I clean up Windows.old and make a backup of the newly installed OS. Entirely routine and ho-hum. But this morning, when I went to check my backups on that machine, I found none for the past couple of months. By looking at the dates available on the Insider Flight Hub, I can tell it’s been failing since Build 18908. That’s ultimately how I figured out that protecting backup card proves essential, too. Let me explain . . .
The source of the problem on this dual mSATA USB card was electrical, and my own stupidity.
[Image Source: Newegg SYBA Dual mSATA RAID Adapter]
Why Protecting Backup Card Proves Essential
Operator error can strike in strange ways. First, I have to explain that the backup device on my Surface Pro 3 (SP3) is a dual mSATA USB card. (See preceding image link for details). It’s just a naked circuit card. It’s attached via a short connector cable into a USB3 port at the rear of the Surface Pro 3 dock. By happenstance, I’ve got the SP3 on a baker’s rack next to my office desk. Turns out that the SYBA card somehow moved (the Boss dusts my office regularly) from its usual place on the plastic covered dock base onto the metal shelf on which the dock rests. I can only speculate that at least two of the contacts on the back of the circuit card must’ve made contact with the metal wire shelving. Result: a short that prevented the circuit card from doing its job (and the backup from completing).
Sigh. It’s always something. I could fix the problem permanently by putting the card (which has the same form factor as a SATA laptop drive) in an enclosure. I may very well do so. But for now, the card is back up on the dock and working perfectly (I just made a successful backup). I’ll ponder adding something like the Inatek 2.5″ USB 3.0 Hard Disk Enclosure (FE2002) to my next Newegg order. Heck: it only costs about US$18 and will prevent future backup failures. I probably should have done this in the first place!
DPC is Windows shorthand for “deferred procedure call.” As explained at Resplendence.com (makers of the outstanding, free LatencyMon tool) the Windows thread dispatcher (aka “the scheduler”) handles prioritization of threads scheduled for execution on a given processor. Assume a high-priority thread works its way into the scheduling mix. Then, lower-priority calls for service get deferred and higher-priority threads run sooner. If that results in audio services delay, this can cause “stuttering” in sound output. And that, in fact, is what causes DPC Latency. (Literally, this term measures the amount of time it takes for a deferred procedure call to finally get a turn at the processor — and playback for audio.) That’s why learning that KB4505903 fixes DPC latency issue is good news, especially for those with affected PCs. Here’s a tweet from MS Engineer Pete Brown from Saturday, July 27:
Tweet Proclaims That KB4505903 Fixes DPC Latency Issue
As the preceding tweet capture shows, KB4505903 gets credit for fixing DPC latency spikes in Win10 1903. That’s good news for those with PCs suffering from audio latency issues. If this applies to PCs in your care, use LatencyMon to make before and after measurements. These should show visible (and audible) improvements in audio behavior. LatencyMon is a good tool for troubleshooting PC audio issues. It’s worth adding a copy to your admin toolbox for that reason. Who knows when another Windows 10 version or update may once again introduce audio delays, clicks, pops, or whathaveyou? I’ve dealt with such stuff personally since the Windows 8.x era, and repeatedly with various Win10 versions, too. Better safe than sorry!
For more information on symptoms of and resolution for this issue, see these various TenForums threads:
If you’re anything like me, you’ve got backups upon backups for your Windows 10 systems. Sometimes, it’s useful to identify Win10 backup version info — that is, which version and build the backup contains. As long as you can access the files within the backup set, this turns out to be absurdly easy. That information is readily available via File Explorer, as long as you can access the file in the backup named ntoskrnl.exe. By default this file resides in %windir%\System32. Thus, on my PC it’s in C:\Windows\System32.
How to Identify Win10 Backup Version
First mount the backup, or otherwise navigate in to the backup file set. Access the aforementioned file (namely C:\Windows\System32\ntoskrnl.exe, in my case). Right-click on that entry to produce its properties window, then click on the details tab. You should see something like this:
The Build number appears as the last pair of numbers in the “Product version” string.
Getting from Build Number to Version Info
Personally, I work with this stuff so much I immediately recognize that the build number 18362.239 corresponds to the Windows 10 May 2019 Update, aka 1903. For older build numbers, even I might need a refresher. That’s when the table at Microsoft’s Windows 10 Release Information web page comes in handy. As this table snippet shows, you can map from the build numbers to the version pretty easily.
Careful inspection of this table based on the left-hand side of the build number (e.g. 18362) points right to the version in the leftmost column.
[Click image for full-sized view.]
And presto! That’s all there is to it. Easy-peasey, right?
Just for grins, I’ve held one of the PCs here at Chez Tittel in abeyance. My wife’s mini-ITX PC is built around a Jetway QM77 mobo with a mobile quad core i7 3630. It’s also got 16 GB of DDR3 RAM, and a Samsung EVO 840 250 GB SSD (SATA3). I deliberately left it running 1809 to see how long it would take for WU to offer 1903. This morning, that 1908 MiniITX PC gets 1903 update offer, and now it’s installed. That took a while, didn’t it? Here’s what winver.exe now proudly displays on her desktop:
That took a while, didn’t it?
Why 1809 MiniITX PC Gets 1903 Update Offer Takes So Long?
Let’s do some calendar math. Using the handy-dandy Date Calculator at TimeandDate.com, I determined that it’s 64 days from May 21, 2019 through today. May 21 is the date from Microsoft’s Windows 10 release information page that indicates that’s when the OS version hit the Semi-Annual Channel. Thus, it took just over two months for this particular PC to attain the status of “ready to receive 1903.” This PC does have two USB-attached hard disks (one for backups, one for extended storage). Thus, I believe MS waited for the blocking problem with USB-attached storage cards and media to clear.
Alas though, we’ll never know for sure. MS isn’t transparent about the criteria it uses to queue up systems for feature upgrades nowadays. But given that this system is home to an Ivy Bridge CPU (which were current from Q3 2011 through 2013), this doesn’t represent new technology. Still, it’s been a rock-solid, trouble-free “surf’n’email” machine for “the Boss” for 7 years and longer. Because she really doesn’t care what version of Win10 she’s running — as long as things work to her satisfaction — I was able to conduct this little experiment in patience and fortitude.
How the Upgrade to 1903 Proceeded
It took a good while to download and apply the upgrade. My best guess is the total time involved was just under two hours. At the end of that road, it showed the most current 1903 build (18362.239). So far the machine is running without any obvious hitches. I’m sure if something goes sideways, the Boss will be calling for IT support — me — soon thereafter. And so it goes, here in Windows-World.
Part of my routine system maintenance on Win10 PC is to use the built-in Reliability Monitor. I’ll check my PCs anywhere from once a week to once a month, depending on issues and behavior. Good behavior warrants less frequency; and slowdowns, flakeouts, and blue screens warrant more. I endured a couple of days of slow, balky runtime hijinks from my old Lenovo T520. Then I decided to see what Reliability Monitor had to say. Alas, results showed the reliability index at rock bottom with app issues abounding. “Wow!” I thought to myself “this cries out for repair.” And indeed, an upgrade Win10 repair install saves mangled T520 PC from further mayhem. Here’s what I found when I checked that machine:
Uncontestably, the WORST Reliability Monitor report I’ve ever seen. OUCH!
[Click item for full-sized view.]
How Upgrade Win10 Repair Install Saves Mangled T520 PC
First a couple of hopefully helpful explanations. Point one: a Reliability Index of 1 is as low as the tool goes. I’ve never seen one that low before. Notice that the T520 sported that value for the last 5 days shown in the visual report graph. Point 2: investigation of the causes for the low index values included a large number of built-in Windows Store apps that not only “Stopped working” (as shown in the screenshot) but that also kept firing errors a dozen times a day or more. The day of the massive dip, it dropped from just over 7 points to rock bottom (1) with errors related to application failures, windows failures and miscellaneous failures of a wide variety. Not good, and not work keeping, either.
When Windows gets irretrievably weird, there’s a kind of magic repair that can often restore things to normalcy. By performing an upgrade install of the same version of Windows 10 already running on a PC, one replaces all the OS components and restores all OS-related settings to their defaults. This does, however, leave third-party applications and files alone. That’s why it’s sometimes called an “upgrade repair install” or even an “in-place upgrade” (see the excellent TenForums tutorial on this topic for a more complete description, and step-by-step instructions for performing same).
Making the Repair Install
After looking over the Reliability Monitor report shows above, I immediately made an image backup. Then I visited the Microsoft Download Windows 10 page. There, I used the “Download tool now” button to create fresh Win10 installation media on a USB stick. Here’s what File Explorer shows for the Details on the Setup.exe file I used to perform my in-place repair install/upgrade.
The Build info appears under both the File version and Product version fields — namely, 10.0.18362.1. That’s how I knew I had the right version to perform the repairs necessary for my 1903 installation on the T520. About half an hour later, my repairs were complete. And since then, all I’ve had show up is a few minor odds-n-ends. All are well-known, familiar, and mostly benign. And now, my Reliability Index is somewhere between 8 and 10, which is where it normally lives on this PC. Problem solved!
Repair install completed, Reliability Monitor settles back to normal. Whew!
If you ever find yourself in a similar boat, remember this kind of repair is good at fixing most Windows weirdnesses. It doesn’t take long to apply, and you should be able to tell relatively quickly if it helps — or not.
Yesterday (July 18), MS published financial results for the fourth quarter of its 2019 fiscal year. (MS FY runs from July 1 of the preceding year to June 30 of the current year.) Those results include a variety of interesting tidbits, many of which speak to the company’s future directions and its sources of success. In this connection, it’s also worth noting that MS continues on as one of only a handful of global companies with net worth in excess of $1TRN. With revenue of $33B, and net income and operating income at $12.4B, the kids in Redmond are clearly doing all right. Examined in a bit more detail, though, MS 19Q4 financials offer interesting insights to those willing to tease them out.
As of 7/19/19, MS total market cap stands at $1.06TRN.
[Source: Ycharts.com|Click image for full-sized view.]
Why say: MS 19Q4 Financials Offer Interesting Insights?
The subject nearest and dearest to this blog (and hopefully, its readers) is the Windows OS itself. Microsoft reports “healthy Windows 10 demand” in the Windows OEM Pro segment that resulted in 18% revenue growth. The company explained this also stems from the immanent approach of Windows 7 EOL on January 14, 2020 as well. On the other side of the Windows desktop OS street, however — the non-Pro (home user, end user, non-business user) segment of the PC market — MS reported an 8% drop in revenue. Its explanation: “continued pressure” in the entry-level computing category represents a falling tide that has dropped all prospects and activity. My own personal takeaway from this is a net positive: looks like business Windows users are finally coming aboard the Windows 10 bandwagon, and that we can expect a more orderly transition from 7 to 10 than many of us expected and feared.
Azure, of course, remains the king of the hill, and grew by 64% over the previous quarter. This produced $11.4B in revenue for the company (a 19% increase over the previous quarter). As Mehedi Hassan at Thurrott.com observed “Azure’s 64% growth this quarter is still the lowest in recent times, declining from the 73% growth it experienced in FYQ3.” These are the kind of growth rates that most orgs can only dream about or wish for. MS continues to grapple with them, quarter after quarter. No wonder Azure increasingly steers the overall direction and organization at MS!
What About Surface and Office?
Surface revenue is up by a relatively modest 14% for $1.3B total. MS cites ongoing, strong growth in the commercial segment. Given that it’s been a year or more since the launch of any new Surface devices, or significant refreshes to existing lines, that’s not at all bad. One hopes that Microsoft has more Surface tricks up its sleeve, though, and will deign to share them with the public before calendar year 2019 ends. Not much noise in the rumor mill to that effect, though.
Office also remains a strong and meaningful component in the MS product and business portfolio. The Commercial and cloud-based Office products are up by 14% overall, with Office 365 Commercial revenue up 31%. Office Commercial (boxed Office products) are down as more and more users switch to cloud-based subscriptions instead. On the Consumer side of Office, revenue is up a relatively modest 6%. Office 365 Consumer subscribers now number nearly 35 million. LinkedIn continues up with 25% revenue growth and Dynamics 365 likewise at 45%. Even search is up, at 9%.
Good News Brings More Good News
Given these cheery results, MS tells a positive story more or less across the board. According to YCharts, the company’s net market cap stands at $1.060TRN today.
[NOTE] There’s a fascinating write-up on MS from regular Economist business columnist “Bartleby” in the July 6-12th issue. It’s entitled “Send in the clouds” and is well-worth reading. It explains the profound and surprisingly well-managed culture change at the company that’s allowed it to switch its focus from selling software licenses to things like Windows and Office to a focus on cloud-based services and offerings built around Azure. Check it out!
After recent Windows Defender updates and the latest 1903 Windows 10 CU (KB4507453) Windows 10 may show interesting misbehavior. Symptom: running the sfc /scannow command produces error text rather than a clean bill of health. Reports from TenForums and Bleeping Computer confirm and describe this phenomenon. If you check the CBS.log file that the System File Checker produces, it identifies a Win10 Module named Windows-Defender-Management-Powershell as the culprit. The specific error says “Hashes for file member <filename> do not match.” In fact, what apparently causes recent Win10 updates bollix defender module hashes is proper resynchronization with the Component Store. I’ll depict the fix, then explain it further. [Note: this pathology is directly connected to Windows Defender, so Win10 PCs running some other AV/antimalware package aren’t affected. Interestingly, I did find the same error on current Insider Preview Builds, too.]
On both 1903 and 20H1 (18936.1000) systems, the symptom and the fix are the same, shown here.
[Click image for full-sized view.]
Fixing Win10 Updates Bollix Defender Module Hashes
The preceding PowerShell screen grab shows the symptom, and the multi-step fix. The symptom presents when sfc /scannow reports that “Windows Resource Protect found corrupt files but was unable to fix some of them.” Based on the nature of the actual error, this is a housekeeping problem. Microsoft’s installer clean-up apparently failed to synchronize hashes for those files as compared to the Component Store. Thus, the fix requires two steps, with a third to confirm a successful resolution:
- DISM /online /cleanup-image /restorehealth checks all the Windows files and replaces any it finds out of whack with known, good versions from the Component Store. This creates the situation where all copies of such files match.
- sfc /scannow, now able to work with matching sets of files, can now effect a proper repair because the hash values now match.
- A final iteration of sfc /scannowconfirms that all is well (“Windows Resource Protection did not find any integrity violations.”)
Problem solved. All this said, the fix may be something to entertain those with OCD tendencies rather than the general population. With issue reports abounding, MS should fix this issue soon. That means some upcoming Windows Defender update or Cumulative Update should obliterate this misbehavior. My money’s on a fix via Windows Defender updates, because this issue appears on Win 10 1903 PCs running Build 18362.239, Slow Ring Insider Preview PCs running 18362.1005, and Insider Preview PCs running Build 18936.1000. The only thing all those machines have in common is the same set of Windows Defender updates. Let’s hope it happens sooner, rather than later!
Note Added 1 Day Later (July 18)
This morning, I updated my Surface Pro 3 PC, running a recent Slow Ring Insider Preview (Build 18362.1005). The only new package that came through was Defender stuff (1.237.1331.0). Out of curiosity I next ran sfc /scannow. Sure enough, the same error message (and CBS.log file contents) recurred. Apparently, MS has not yet fixed this issue and the hash mismatch reappears upon obtaining new Defender updates. How hard can this be to fix? Please: get it together, MS!