Microsoft's eye wasn't on open sourcing their OS and describing the deep internals. They still don't want you to develop against the NT API, even though developers certainly do (and Microsoft makes compatibility shims for applications which do, when required).
Tbh, I think Windows' stable ABI/API at all levels + excellent docs is something the open source community could learn from.
Software that is poorly documented, has no stable api, and no way of accepting extensions other than modifying the source, is not very 'free', in the sense, that if I have a good idea, I'll have crazy amounts of trouble getting the change to users even if I manage to decipher and modify the source appropriately.
This approach describes many software projects. If I wanted to change GCC in a way the original authors didn't approve of, I'd have to fork the entire project and lobby the maintainers to include my versions.
If I wanted to change most MS stuff, like Office, I could grab the COM object out of a registry, and override it with mine, just to list one example. Then I could just put the binary on my website, and others would be able to use it.
As for MS not publishing these particular docs - its not like MS doesnt publish tomes on everything, including the kernel. If you're interested in Windows Internals, I recommend the book series of the same name by Pavel Yosifovich, or really anything by him or Mark Rusinovich.
Its also a testament to the solidity of Windows' design that tons of stuff written decades ago is still relevant today.
> If I wanted to change most MS stuff, like Office, I could grab the COM object out of a registry, and override it with mine
This goes back a very long time -- at least to the Windows 3.0 timeframe.
The IBM-only 32-bit OS/2 2.0 came out around the same time as Windows 3.1.
OS/2 2 could run Windows 3 inside what was effectively a VM (a decade before true VMs came to the x86 platform), running the Microsoft code on top of OS/2's built-in DOS emulator.
I remember an IBM person objecting to a journalist saying "so you have access to the Windows source code, and you patch it to run under OS/2?"
Reportedly, the IBM engineer looked a bit pained and said "we don't patch it -- we superclass it in memory to make it a good citizen, well-behaved inside OS/2's protected mode."
(It is over 30Y ago so forgive me if I am not verbatim.)
This was subsequently demonstrated to be true rather than a marketing claim. OS/2 2.0 and 2.1 included a "WinOS2" environment. OS/2 Warp 3 made this an option: it was sold in 2 versions, one with a blue box which contained a Windows 3.1 environment, and one with a red box which did not contain WinOS2 but could be installed on top of an existing Windows 3.1 system and then took over the entire Windows environment, complete with all your installed apps, and ran that inside OS/2.
So you kept all your installed 16-bit apps and settings but got a new 32-bit OS with full memory protection and pre-emptive multitasking as well.
Bear in mind that Windows has not activation mechanism then, so you could copy a complete Windows 3.x installation onto a new PC, change some drivers and it just worked without complaint.
So you could buy a new high-end 486 PC, copy Windows off your old 386, install OS/2 Warp over the top and have a whole new OS with all your apps and their files still running.
This was amazingly radical stuff in the first half of the 1990s.
They invested huge amount of resources to make sure it is backward compatible in NT, to the disagreement of David Cutler. There is a NTVDM extended with WoW for Windows 16-bit, AFAIK. I have a copy of leaked source code of NT 3.5 but I'm not good enough to understand the code. Also probably modern VMs such as DOSBOX do a better job emulating 16-bit stuffs.