Tuesday, March 26, 2013

How to Cable an EMC VNXe for Failover

I really struggled with cabling a new VNXe for failover with vSphere. It's not difficult. There's just not a lot of documentation or explanation out there for it. The best practices docs and howtos treat iSCSI like a 3rd class citizen and assume you'd really rather be using Fiber or NFS. So here's how you really do it and a little explanation about why. We'll go into the details of the vSphere configuration after covering the hardware end of things.

*Disclaimer: I am not an EMC expert, nor am I guaranteeing nothing horrible will happen to your equipment or vSphere environment. Use this guide at your own risk.

At its very core, this is how your network will look with a VNXe for fail over. This assumes you have 2 physical NICs in the host to provide multiple physical hardware paths between the host(s) and the network switches. All cabling is single lines with no trunking.

Note that the VNXe has a single network line from each SP to each switch. This is the most simple way to cable. The SPs are able to handle NIC teaming using LACP trunks but we're going to use vSphere's round robin scheme to get us the same functionality. The requirement of LACP trunks means that you need to rely on a higher end switch, such as a Cisco Catalyst 3560 or an HP Procurve 2910al. Using vSphere's built in capabilities you can use a less switch but be careful about your throughput and switching capacities! You can get burned skimping on switches.

I chose to have each NIC on the SPs set to a different IP address. In this case:

  • SP A eth2 =
  • SP A eth3 = 
  • SP B eth 2 =
  • SP B eth 3 =
Be sure that when you cable the SP NICs that you cable the correct subnets together! We'll end up with a NIC in the 192.168.50.xxx subnet and another NIC in the 192.168.51.xxx subnet on each host. Make sure the 50.xxx NICs are on the same switch and the 51.xxxx NICs are on the other switch.

Moving on to the vSphere end of things. There is an iSCSI requirement that you ensure VMkernels are tied to a single specific port. You can assign multiple ports to a vSwitch but you need to ensure that each VMkernel has the switch failover order overridden to assign a specific physical port to the kernel. This can get confusing and since I don't have a whole lot of vSwitches in my environment I choose to create VMkernels one per vSwitch. VMware advises that the number of vSwitches can impact performance so be aware of how that may impact your environment.

Now that we've created 2 vSwitches, each with their own iSCSI VMkernel we can assign them to the iSCSI software initiator adapter. Configure the Storage Adapters on your ESXi host and view the properties for the iSCSI software adapter. Under the Network Configuration you will have the option to add ports to the initiator. Add both of the iSCSI VMkernels created earlier.

Once you've configured either Dynamic or Static discovery for the your LUNs and rescanned the adapter for devices we can go to configure storage.

Add your storage as you normally would and go to the new datastore's properties. Click on the Manage Paths button in the lower right.

If everything is cabled correctly and operating like it should you should see two paths listed and a Path Selection policy set to Fixed. To set your paths to Active/Active you'll need to choose Round Robin. If you want Active/Standby, choose Fixed. Make sure you hit the Change button before closing the window to make sure your Path Selection policy is saved.

Now that multipath is setup you can move on to testing. You're doing this in a non-production environment, right? Remember that if you pull cables while testing and something isn't right you'll lose Datastore connectivity!.

Go ahead and start pulling cables off the switch to see how the paths are affected. You should see that when you pull one cable from the primary SP for a LUN that the datastore stays online. If you pull the corresponding cable from the other SP you should see a dead link show up in the manage paths interface for storage configuration. Pull the second cable from the primary SP and you should still have connectivity.

Its worth mentioning that some folks online have said it can take up to 60-90 seconds for an SP to fail over to the backup SP. Make sure you take that into account while testing. I seem to have nearly immediate fail over. Your mileage may vary.


The VNXe has a nifty design. There's internal communication links between the SPs. When they detect over the SAN network that an SP NIC is down the network traffic for that particular port is redirected to the corresponding port on the other SP. For instance, If Eth 2 on SP A loses connectivity, the traffic is redirected to SP B Eth 2. Same goes for the other ports. This is why EMC tells you to cable each SP identically. 

The only time you have a true failover scenario is when an SP actually goes down, say for a reboot after an update for instance. Any NIC failures are handled by redirecting traffic.

Because of the massive amounts of traffic generated, you really must NOT do SAN over a network with other general traffic. Not only is that terrible design in a fairly epic fashion, you run the risk of cross SP traffic flooding your network. I had that happen once when I cabled to a port that I forgot to put in the proper VLAN. Not fun.

Friday, March 22, 2013

The Slow IT Movement

If you haven't read about +Rob England 's SlowIT philosophy, I highly recommend that you go over to his blog and read up. He's really on to something here.

When I first read his ideas about slowing down, I completely agreed but didn't fully understand exactly why. His latest post on "The mad competitive scramble" cemented it for me. Its a cultural problem and it is poisonous. I look at it from a purely practitioner's perspective and often the mad dash to the next project is intoxicating or is forced on the IT staff. Since I don't have the 10000' view he has (nor the experience) I didn't fully appreciate how toxic the culture that brings about the scramble he's talking about.

The IT department is just like any other department. It's a tool for the business just like the accounting, HR, or any other departments. The purpose is to get the work done and maintain the viability of the organization.

I've been on the insider and consultant side of things and I cannot count the number of times I've had to reign in users and decision makers. Slow down. Let's look at the value/use cases/purpose/master plan/etc. Decision makers are often very good at seeing the big picture of an organization but seem to be inherently terrible at seeing the big picture of technology. To me it seems that the older folks in those positions are still mystified by tech and the younger decision makers learned all the wrong theories in school about tech. Understanding how tech fits into business isn't rocket science. It just takes a level headed approach and some basic knowledge about how it works.

Be competitive on your own terms (and speed). Learn to be good at customer service. Stop trying to change to compete.

Thursday, March 21, 2013

P2V Server 2008r2 VMware Unable to Update boot.ini

There's not much documentation out there on this so I figured I better do my part and blog the solution so you don't have to pull your hair out like I did.

If you are trying to P2V or V2V Server 2008r2 with VMware vCenter Converter Standalone and you get this dreaded error don't worry. Turns out its a simple fix.


  • P2V/V2V conversion fails at 95-98%
  • New VM in vCenter is not bootable
  • Booting VM to install media to do a repair yields no visibly installed OS
  • Errors include: Warning: Unable to update boot.ini, Error: An error occurred during reconfiguration
  • Convert logs show: Unexpected Exception: converter.fault.ReconfigurationNoSystemVolumeFault
The problem, it turns out, is that the system reserved partition is not online. (VM KB2008012)

The fix is to online the System Reserved partition:
At the command prompt for the machine you are trying to convert
c:\Diskpart> List volume
c:\Diskpart> Select volume 1

Make sure the volume you select is the 100MB system partition. Dead give away is that it is offline.

c:\Diskpart> Online volume
c:\Diskpart> exit

This should bring the System Reserved volume online and now the conversion should work properly.

As a side note, after the conversion you may want to double check that the System Reserved partition is marked to be online at boot (See this Symantec article). You can do this by running this command before you exit diskpart in the above instructions: c:\Diskpart> automount enable

Friday, March 15, 2013

Trust But Verify: It Goes Deeper Than That

Managed Services Part 2

Read Part 1 Here

I continue to struggle with how an MSP fits into the broader strategy of a competent IT department. I've come to the conclusion that many of my issues are trust related. As I think through the list of service providers I've dealt with over the years I can only name three or four that I really trusted. All the others seemed to go to great lengths to destroy any vestige of trust and some even seemed to revel in the fact they were fleecing the customer.

Why does this happen? What is going on that MSPs and service providers in general feel no shame in providing terrible service? I mean, service is in the name of their industry category, right?

There are a few things that jump out at me.

  1. All the businesses I've spent my IT career in are located in an area identified as "Rural". We're a solid 2 hours from the Twin Cities metro area. 
  2. Contracts seem to universally favor the service provider
  3. Many service providers are bigger than the companies they "serve"
  4. Service is hard and service providers are often bare-bones staffed to maximize profits
It's hard to be a service provider. I've been in that boat and I know how much work it is to keep it above water. It doesn't need to be this way though. Good service is hard but it's not difficult to maintain a high level of service if you make it part of your culture.

As I lamented the "just trust us" mentality of MSPs this morning, a friend reminded me of the "Trust but Verify" edict. I think it goes much deeper than that (and so does he). That's really where I start to struggle.

You can monitor and manage an MSP or service provider just like you would an employee or even more than that. You can subject them to strict monitoring and compliance auditing. The fact remains that the worst that happens to a service provider is that you sever your relationship with them. They lose some money. If you have enough clout you might even be able to make sure they lose some reputation. Not much of a consequence for most providers, eh?

They have dedicated sales staff that are professional job seekers. Service providers interview very well. When you are courting a provider you'll never find them providing you references from companies that were unhappy with them. Due diligence is not an easy thing when it comes to service providers or MSPs. Sometimes there isn't much choice available (see #1 in the above list).

The relationship is skewed and very profitable for one party. I don't want to take lightly the qualities an MSP brings to the table. Expertise and knowledge are expensive and just as hard to find as good service is. For many, it's far cheaper and feasible to bring in an MSP than hire staff. It's unimaginable to expect a small business to hire an IT staffer when they only have 10 people on the payroll and technology is not their core business. MSPs fill an important role to be sure!

So I continue to have trust issues with MSPs and service providers. Accountability for the relationship would help a lot with my issues. I look forward to the conversations I'll be having over the next few weeks. Maybe I'll learn to trust again?

Wednesday, March 13, 2013

What is a Partnership?

A vendor came a-courtin' today. I'm probably too critical on vendors when they approach me. It is so frustrating, as someone who's been in the business for a while and qualified to do my job, to see the attitude I'm treated with as a person involved in decision making. All too often potential customers are assumed to be stupid, or at best, ill informed. As a result my defenses go up and I become a top-notch skeptic.

These guys stuck with me through it, though, and in the end a good conversation happened. Hats off to them for being persistent.

This conversation made me think a lot about what a partnership with a vendor looks like. When it comes to managed services I tend to be openly hostile. I've seen very few good experiences with MSPs. No one cares as much about your organization as you do and MSPs, in the end, are making money off your lack of investment in IT (or inability to make an investment). However, there times where its nice to have a vendor to dump some things off on. There is also a certain level of expertise and exposure a vendor has just by being in many environments.

So in the case of an organization where there are already competent IT personnel involved and technology is helping users meet or exceed corporate goals is there room for partnership with an MSP? It was interesting to think about that as the vendor I was talking with was trying to figure out what this sort of relationship would look like. I'm working on bringing nearly everything I can in-house and strategies to cut down the "IT stuff" (defined as tech busywork like password resets, account unlocks, stability-related outages, etc). That doesn't leave much room for the traditional MSP sweet spots of helpdesk and network administration.

So now I have some things to think about. I'd never considered how a vendor fits into my IT strategies. I've never found a vendor that had a broad enough expertise for me to "partner" with. That sounds like a blog post for another day.

Monday, March 11, 2013

Knowing is Half the Battle

A few months ago my family and I went to a friend's house for supper. It was a great chance to visit and see where some of my eggs and produce come from. Our friends run a sustainable ag produce farm and I've been very pleased to have purchased CSA shares from them. Some of the heirloom produce varieties are just so tasty its hard to pass up the opportunity!

Our friends also keep chickens and other fowl. I've really enjoyed the eggs we get from them from time to time. Anyone that's been around a farm where you have free-range birds knows that you are a prime target for predators. Southern Minnesota has seen a pretty healthy increase in the number of coyotes and other wildlife that loves chicken and turkey as much as we do. The reality is that you have to protect your animals.

Now I'm not going to argue rights to this or that, or controlling who has what, constitution, or whatever. That issue has nearly become a dead horse and everyone is holding a stick with which to beat it. What did come out of this visit with friends as a reminder that knowing how to use tools, regardless of their lethal capabilities, is sometimes not only helpful but also life saving.

We hadn't been in the house for more than 30 minutes when my eagle-eyed spouse noticed a rifle propped up in the corner of one room. Our friends don't have children so this wasn't a huge surprise to me; What surprised me was I was the only one in the group there who knew how to handle the weapon. First step when finding a firearm is finding if it is loaded, and this one was. 

I unloaded the gun, left the chamber open and handed it over to my friend who put it in a safer location. 

This seemed like a pretty scary incident to the folks that were there. I've been around and used firearms since I was young so it didn't phase me but everyone's reactions to the situation made an impression on me. It occurred to me how important training is.

Firearms are a part of our culture (for better or worse) and we all have a chance of coming in contact with a weapon at some point in our lives, especially when you live in a more rural area. There are so many deaths and injuries that could be prevented through simple education. My kids will certainly spend some time in a firearm safety class when they're old enough. We'll probably even head out to the farm some day and shoot at pop cans together. I would be terribly remiss as a parent if my children were in my situation at a friend's house and didn't know how to make a weapon safe.

Friday, March 8, 2013

'Round the World

I think I've had more international communication in the last 2-3 weeks than I've had in my entire lifetime. Between Twitter, G+, email, and even phone it's been a lot of fun figuring out time zones and marveling how technology allows us to do this. There are so many incredible people all over who make me think. I love it!

Thanks to everyone who's put up with me. I really enjoy your thoughtfulness and broad minds! I hope I'm returning the favor with good content and ideas.

Thursday, March 7, 2013

Lost in Translation

The longer I continue in my career many things keep getting easier. Experience fills in the knowledge gaps. There's one thing that remains frustratingly difficult though: dealing with non-technical people. Often, when I've sat down with other tech workers and healthy doses of beer, the conversation seems to inevitably turn to non-technical users.

"This job wouldn't be so bad if it weren't for the users."

Experience naturally teaches us how to interact with non-technical users. You step on a lot of toes and hurt some feelings along the way but often the frustration remains even when you can gracefully talk a user back from the ledge and solve their problem. (True story: I've walked into an office once to find a user with gun drawn on his computer. Not even joking.) As technicians we can often look past the technophobia, the lack of understanding, even the anger that users are experiencing. The ignorance and the willful disinterest are much harder to accommodate.

There seems to be an inherent disconnect between non-technical users and IT staff. I'm convinced that much of it has to do with the way our minds work but also in the way we approach technology, our methods for dealing with new information and tools.

I think the best thing we can do to help bridge the gap between user and technical staff is eliminate the silo-ing of employees to some extent. We keep the IT folks in the basement - out of sight, separated off from the good-looking people in sales, no windows. Why do we do this? For one thing, IT staff embrace the separation. We're not usually cuddly or necessarily good with people. The separation is also a symptom of the way business tends to look at technology and IT staff; The viewpoint is that IT is not part of the business but rather a facility we just toss money at.

As an industry, IT needs to get people skills. The helpdesk needs to be social not just in the Social Media sense but in the squishy human contact sense. Bad people skills should not be tolerated any more. Its driving people to use Shadow IT solutions and doing bad things to business.

IT is people.

Tuesday, March 5, 2013

Change From Within

What does change from within look like? It seems to me that many organizations never experience that type of change. That seems to indicate that IT maybe isn't that influential or transformative to most organizations. Maybe our influence has waned after the initial shock of introducing technology into the business world. Perhaps we've assimilated too well for our own careers.

Whatever the reason for the lack of IT driven change I'm excited for it myself. The opportunities available for IT to transform the way we do business are so huge I can't wait to start!

So what does change from within driven by IT look like? Positive change, that is. We've seen and produced plenty of negative change.

Once you've solved the big problems in the server room the next step is to look at what's going on outside of IT. That's where technologists seem to fall on their face. Once you need to look outside of the IT Department we seem to get all awkward and nervous. IT is a unique department in that it is involved in every aspect of a business from building maintenance, to sales, to boardroom, etc. We have a broad viewpoint of the whole organization and it is time to utilize that knowledge and experience to help the business.