VMWare DataCenters and My stupidity == learning new trick

So I decided in a demo environment that I would try to move an ADDC from one datacenter to another- without realizing that you can't. Oh sure, there's a slightly convoluted way of doing so via off-lining the VM, and doing a cold migration between datacenters, but as I found out very quickly- that can be filled with some very nasty gotchas.

Some background on this VM- it's just the test auth for a proof of concept (POC) system, so it does double duty as the RAS host as well. VPN in, do my work, happy clam. Note- this box is the VPN server.

Did I mention it's the VPN server yet? Very important point, and one that makes me damn glad I had an out of bandwidth management setup.

Anyway, I start the preparations to move this VM by... shutting down the VM. Makes sense, it's a cold migration right? Yeah, guess who winds up disconnected? So without a hint of panic, I try to login to the host itself, thinking that AD going offline just popped VCenter. Can't reach the host- very quickly realize I can't reach any of the hosts. Now the panic briefly pops in, however it doesn't last long as I realize I can just log into the console.

And discover another mild gotcha.

The esxi shell is disabled by default.

Mind you, I've not played with 5.1 under the hood that much, so a quick tour of google and I find out how to access the shell once again- F2 at the console, login and navigate to "Troubleshooting Mode Options", enable the shell, and exit out. Finally, I've got shell!

So I login, execute quick vim-cmd vmsvc/getallvms in order to locate the vmid of the vm in question, followed by vim-cmd vmsvc/power.on . No real panic as yet, but that's because I figured I'm already fired anyway, how much worse can it get? Less than 5 minutes later, I'm able to login to the VPN and restart my connection to VCenter.

When I notice the error logs make no mention of VCenter ever having an issue. The only issue on the whole system was the fact that the esxi shell had been enabled on one of the hosts. Then it dawns on me- my original thought of short downtime not having any effect on the network was correct, however in my raging stupidity I'd forgotten where the vpn lived. That issue gets fixed today.

However, in looking around, I had an "AH HA!" moment that I needed to test (in my homelab... even if it's a POC, it's not for my tests- just the customer's). What if I join the new host to the current datacenter, do a live migration to the new host, remove the host and rejoin it to the correct datacenter? One quick test with a host running one half of my local AD (Yes, in the home test lab I run two dc's minimum... I hate rebuilding them.That'll teach me to cut corners).

Removing the host from the datacenter does not delete the VMs on that host. Joining a host with VMs doesn't delete the VMs (but I already knew that)- So what's to keep this from working? As near as I can tell nothing- just make sure the VMs you are trying to move aren't in a cluster, as the cluster will more than likely try to bring them back after you've removed the host.

In conclusion- if you can remove and join hosts from the vmware datacenter, you can do a live migration of VMs between datacenters.It just takes a bit of forethought and planning.

And being smart enough to remember where your VPN lives.

It's been awhile again, I know. Bad poster!

And man have I been busy! The list of items is monstrous indeed- everything from vmware 5.1, server 2012 to NFS. I've been learning user administration (again! seems it changes every few years, go figure), more powershell, more exchange, and getting to play with great toys, like synology's RS2212+ (it's a thing of beauty, and I will be writing about it, and possibly buying one for the house), as well as redeveloping mass vmware deployments of server 2008 (I know, old os, but I had to deploy over 160 VMs in a single weekend)

I'll be detailing everything over the coming weeks, provided I don't get swamped again.

ESEUTIL, Unitrends UEB and my own stupidity

So I just recently got a copy of the Unitrends Enterprise Backup appliance for vmware installed and licensed over the weekend. Now me being me, I just had to try and dive in head first. Couldn't get a lot of things working, but I did get exchange backups working- which is a massive plus in my book.

So, verified the backup, and it cleared my exchange logs- yay! Talk about making me happy. Until a nagging voice at the back of my mind reminded me of recent experiences with backup exec (and the fact that there's no corresponding restore exec). A restore was needed! So I went through the process- but I wanted to restore only a single mailbox. Talk about a nail biter- as I'm going through the options, I created the restore share by selecting my mail db backup, selecting the option of "Next (Select Files/Items)" and...
Waiting. 16GB db plus about 30GB in log files- takes awhile. Cool thing is, this restores to a local samba share, which you then access from your exchange box.

Which is where the stupid on my part comes in- you'll see it soon.
So, I check the db and start trying to bring it to a clean shutdown state- (for those of you who don't know how to do this, check out this blog: ExchangeServerPro, excellent write up!) when I started running into trouble- my db checked the way I expected it to, my log files were good, however during the recovery phase...

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use)

And it doesn't tell me which file. Of course, since I'm such a genius, I decide that it must mean the db is locked... nope, db's not locked.

Oh! right, the samba share is read-only! Why didn't I realize this sooner?
So I move the db to a directory I've got read/write access to, and it's all good, right?

Oh hell no.

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access fi le, the file is locked or in use) exchange 2010

Now, I get a bit irritated and throw handle and process explorer (both from sysinternals, good stuff!) only to find out that db is never locked. In fact, the restore process gets to ~90% before failing without ever opening a log file or the db!

WTF?

Well, turns out that the log directory needs to be read/write accessible too... something I would've figured out almost 10 minutes sooner had I just checked the event logs for the ESE errors...

So, while the samba mount is cool, it's not as useful as I had hoped. But it does make it easier to get at the files and use the standard microsoft tools for manipulating the files (powershell, robocopy, xcopy and all the rest)

Problem solved, files copying, face red with shame.
Hopefully this sheds a bit of light on the problem in case anyone else is as dense as I can be.

NFS in android- why isn't this built in?

This seems like such a no brainer- right now using fileshares with android devices tends to be fairly painful, even with utilities such as astro, es file explorer, etc.

Why? Because none of those let you actually mount filesystems.

You want to view/use a file on a network share, you have to copy it to your device, work on it and then copy it back. That's such a waste, especially considering how long linux has had access to smbmount, nfsmount and the like. And they are so easy to setup. I mean, incredibly easy. But yet for an android device, this is a nightmare of rooting, installing apps (not that I've got anything against CIFS manager mind you, except that you've got to root your device to use it, and it's got to be running CM7), and running a gauntlet of compatibility issues.

NFS/SMB read capabilities are right there at the OS level, literally compiling in NFS support for the kernel is a check box or switch during compile time, and it takes pretty much no effort to supply. Not giving us this ability means that device storage built into our devices is getting wasted needlessly. Filesharing is a common every day solution for a lot of things, and even in the home user market no less. Network attached storage is even an option now on some routers- so the technology is being made easily available.

By including this is in the stock android deployments, we could get much easier use from our devices, and it would be yet another step to making BYOD less of a headache for system's administrators- ie, the data your users are using is still locked on the fileserver, and not on their devices.

Equallogic SAN + VMWare ESXi 5.0 == annoying at times

I know a lot of you will laugh at me for this, but I'm hoping at least a few people find this useful. If you've got monitoring setup correctly on your EQL boxes (and you are running at least 5.0.5 firmware), you might be getting emails like this:
-----

ERROR event from storage array DABIGDISK

subsystem: MgmtExec

    event: 7.4.3

     time: Sat Sep  8 23:37:04 2012

iSCSI login to target 'xxx.xxx.xxx.xxx:3260, iqn.2001-05.com.equallogic:0-8a0906-19f16ee02-6040000003b487d0-vss-control' from initiator 'xxx.xxx.xxx.xxx:58885, iqn.1998-01.com.vmware:FEEKYFEEKY-07e33139' failed for the following reason:

                Initiator tried to bypass the security phase but we cannot.

-----

yup, sorry vmware, no biscuit.

It's an error, and it's kind of a stupid one, but it's not vmware's fault, it's actually a slight flaw (or duh, actually) on the equallogic side. There's two settings that need to be correctly configured, one of which is under Group Discovery -> iSCSI (refer to pic)

As you can see, it's a pretty simple check box. The other location is under VDS/VSS, in which case you simply don't give your vmware boxes permissions to use VDS/VSS unless you've already got the means in place to support it. 

Simple, stupid fix- and I was stupid enough to not even notice it until I actually ran into the headache tonight.

Quick Update, yes I'm alive

Just posting a quick update- two of the projects I've been working on are completed, however (as usual) that's only opened the door to new problems, er... projects.

However, if you've got a moment, check this out! The team over at spiceworks has released version 6, with tons of new features. The next latest being worked on is iLO support, something I'm incredibly excited about!

Anyway, still alive, still busy.
Technically, still posting.

The inherent dangers in prototyping

So I've been working on a few projects, one of which is requiring snmp functionality for a power/ups monitoring solution. One of the headaches I have is finding components- I've got digikey, mouser, newark, jameco and sparkfun... out of all of those, the only one that sensible names things is sparkfun, except that consistency in naming isn't something they're known for. But their site is incredibly easy to navigate, pictures and posts that relate to the items tends to be very useful. Not quite the same with digikey and mouser. I had ordered a ton of parts from digikey just a few days ago, and actually got two sets of headers that were completely wrong. I had misread the specifications, and assumed when I saw 2.54mm across to mean the same thing as 2.54mm spacing... boy was I wrong. One item will partially work, the other (a 2x10 2mm spacing female header) is about completely useless. And I've got 10 of them. And I made the mistake of telling the rather nice customer service lady that my order was fine, and I would be getting the parts used immediately.

Now I find that I've had to re-order three sets of parts, from two different houses (digikey and sparkfun this time) in order to have any hope of meeting my time frame. While I realize this isn't any fault of the distributors, it still doesn't make it easy to know that I've got to really squeeze my debug time in order to make sure I've got time to get it finished before my impending trip to alaska.

In case anyone's interested-
SparkFun- www.sparkfun.com
Digi-Key- www.digikey.com
Mouser- www.mouser.com
newark/element14 - www.newark.com
and jameco- www.jameco.com