Note: This isn’t one of my standard blog entries. I am posting this entry in hopes that the following information makes it into Google’s vast knowledge base and helps some other poor sod out. If you are that sod, feel free to let me know.
I am running a physical Windows Server 2003 machine as a host. On that host, I am running VMWare Server 2.0.2. Inside VMWare Server, I am running 3 virtual machines: a virtual domain controller, a virtual mail server, and a virtual web server. The physical host is not a member of the domain; it resides in a workgroup (WORKGROUP).
To reduce the number of virtual machines, I decided a good solution (ha!) would be to promote the physical host server to become a domain controller. This turned out to be a bad idea.
First, I logged into the physical member server and ran DCPROMO, promoting the server (formerly in WORKGROUP) to a domain controller (DC). When the promotion finished, I was prompted to reboot the physical server. After the server rebooted, I logged in (now with my domain account) and found that none of my virtual machines would launch. Specifically, the VMWare Host Service would not run. The reason the service would not run is because, when I originally installed it (on a non-domain server), it created a local account named “__VMWare_User__”. Unfortunately, once you promote a server to a Domain Controller, all local accounts are removed and you can only use domain accounts.
At this point I assumed (incorrectly) that if you installed VMWare Server on a Windows 2003 Domain Controller, it would create an Active Directory and/or service account to run the VMWare Server services with. (It doesn’t.) What I believe I *should* have done at this point was create a service account named “__VMWare_User__” and granted it local log on rights to the domain controller. [I don’t know if this would have worked or not.] Instead, I (again, incorrectly assuming that attempting to install VMWare Server on a domain controller would create the service account for me) uninstalled VMWare Server 2.0.2 and attempted to reinstall it. After attempting the install I received an error code 25008, informing me (in slightly more technical terms that “this is a bad idea.” I continued past the error and, once the install bar reached 99%, received an error that VMWare Server was unable to create the necessary local accounts and groups. At this point the install failed and ended, and I got to watch the progress bar recede from 99% all the way back down to 0%.
My next course of action was to demote the physical domain controller back down to a member server in a workgroup, reverting back to my old, original configuration. This quickly proved difficult, as I could no longer launch VMWare Server and my virtual domain controller contained all FSMO roles. Running DCPROMO on the physical DC got me to the following prompt, familiar to anyone who has demoted a server before: “Is this the last domain controller in the domain?” When I answered no, the program essentially informed me I was lying, because it knew about the other virtual one. I re-ran DCPROMO and when it asked me “Is this the last domain controller in the domain?” I answered yes, at which point the program said, “Oh really? I don’t believe you, because there was another one around when you promoted this box.” So basically, both yes and no were the wrong answers.
After banging my head against the wall for a while (and bouncing ideas off of my friend’s head) I ended up installing VMWare Player on my main workstation and transferring my virtual domain controller across my LAN to my workstation. At that point, even though I was able to launch up my virtual DC, I still wasn’t able to demote the physical DC — this is because it hadn’t fully registered with DNS yet at that point. The error I got read “DSA operation is unable to proceed because of a DNS lookup”. My knee jerk reaction was that the physical domain controller could not see the virtual one, and so initially I added entries into the local hosts and lmhosts files to try and rectify that. In reality, the problem was that the physical domain controller had not fully registered with my domain’s DNS server yet. A simple “ipconfig /registerdns” on the physical DC, followed by stopped and starting the netlogon service (“sc net stop netlogon” and “sc net start netlogon” for you command line guys) got it registered and fixed the problem. After doing that, DCPROMO ran and I was able to remote the physical DC.
After the demotion was complete I had to reboot the physical box. After the server came back online, I had to move it back into a workgroup. (You don’t want your physical box in a domain when your only DC is virtual and hosted on the same server. My friend talked me out of that one; since no DC will be available for the physical member server at boot time, it’ll take forever to come online.) After moving the box back into WORKGROUP and rebooting it (again), I was finally able to shut down my virtual DC (which was still running on my workstation), move the vmdk files back up to my server across the LAN, and fire the virtual DC back up. Total time invested? Somewhere around 6 hours, not including file copies.
I can’t read the lines (rather I can’t understand 99% of it), but in reading BETWEEN the lines I’m seeing that this was hard and frustrating. When you had one knee-jerk reaction, my reaction would have been to open a window and either throw the machine out it or jump out myself. Sounds like it all worked out for you (I’m guessing at that). I have a very smart son!
If I ever decide to run virtual software in a Windows server environment, I will keep this in mind. I am so used to command line directives and configuration files using various flavors of Linux and various server/desktop setups, I’m not sure I could navigate Window’s GUI server configs. In fact, I’ve never graduated beyond XP. Even getting into the control panel of a typical Win7 or Vista would probably confuse the shite outta me. At least I know who to ask if ever faced with a Windows conundrum.
Along that line of thought Rob could run his VMWare host environment on Linux and then use a VM for the AD server. (:
Thank you for the info. It is strange how quickly “this will make things more efficient/easier to manage” turns into “why didn’t I just leave things alone”. Also a key thing to remember is, do not do anything right before you go out of town or after surgery (that was a big oops for me on both occasions). On the bright side, I am sure that your post will assist others in either not making the same mistake or more likely – how to fix it once they have.
Yeah it’s much better to keep everything virtual. Actually the best way is to run ESXi 4 and dump the OS layer.
Sounds like a lot of what I’ve been dealing with at work too. Our VMs are good but this just reminds me how much of a house of cards all this software can be. One wrong decision can get you in a world of trouble. We’re struggling with getting backups of the VM’s because they are always in use. And if those huge files get corrupted somehow….
As an IT guy since ’97, it’s making me nervous. I am going to long for the days of physical machines. I understand the cost cutting on the hardware side, but it makes server admin stuff that much harder.