“I think you are making an assumption that an enterprise is unable to monitor point-to-point communication on the same VLAN or throughout the network, or can only do so through an IDS placed on every VLAN. Simply not the case, not only can NBA technologies monitor this communication, network management systems like Concord or Network General provide robust protocol analysis throughout the enterprise, both WAN and LAN. Visibility, especially as it pertains to security, within virtual environments is significantly hindered using traditional technologies, which is why VMware is choosing to work with Cisco – what happens to non-Cisco shops?”

No assumptions on what you can and can’t monitor in a network. I was a network engineer in a few major networks for the first half of my career before I crossed into the dark side of selling software. I know very well what can and can’t be seen on a network. My point was most customer don’t implement half of what they could to actually do this monitoring. In addition to that you can still plug in all of the stuff you just talked about (sniffers, protocol analyzers, etc). You have the ability to mirror virtual switch traffic to a physical port of your choosing so all of this stuff still works. That’s been in the VMware ESX server since I’ve been here (version 1.5). I don’t see a lot of customers actually doing that but it’s there. I think a lot more people will go down that road with the new Cisco switch. Most of the non-Cisco shops are current VMware partners so I’m sure it’s just a matter of time before they get around to doing something similar (the SEC and future announcements prohibit me from any more detail). Just as an example though, Checkpoint has a virtual firewall appliance now.

On the management front, I’m not saying that grabbing control of Virtual Center wouldn’t be bad – it would. What I’m saying is that most of those customers also have some sort of lights out management device in their servers that happens to run across that same management VLAN. If you’re on the management VLAN and you happen to have an admin password (something you’d need to take over VC) then you also have the ability to shut down every physical server. My main point was bad guy on internal management VLAN = bad news. It’s no different if you’re physical or virtual. Yes, getting into VC means you have fewer keystrokes to take things down but people who are smart enough and malicious enough to get into an internal management network probably aren’t going to be powering off servers – they’re going to be stealing data. If they were going to power stuff off then you can bet they’ll be going after the physical stuff as well.

Bottom line on the security stuff – yes you need to be concerned, but don’t forget about your basic security defenses that should be in place first. If those basic systems are there (physical security, people not giving out passwords, separation of duties) then you’re going to be just as protected on the virtual side as you will on the physical side.

Two more things to say before signing off:
1) I love the picture for this whole post. Just saw the movie yesterday. Pretty sure I’m dumber after those 2 hours.
2) I hope we get to sit on a panel together at some point. Should be some good conversation.

Thanks for the thoughtful response, there are some things we may need to agree to disagree on, but mostly we agree =)

Comments below:

“I’m just saying most corporate networks aren’t loosing visibility with VM-VM communication since they really didn’t have that visibility implemented in the first place. Never-the-less VMware saw this as a problem. This is why we partnered with Cisco (to start with) to embed your normal IOS-based virtual switch into the environment. http://cisco.com/cdc_content_elements/flash/dataCenter/nexus1000/index.html. Now you get the visibility you were probably wanting but never knew you lost in the first place.”

You do agree that traditional network security mechanisms will need to be augmented – good – that alone means that implementing virtual environments MAY increase management complexity and CAN impact security visibility and control.

I think you are making an assumption that an enterprise is unable to monitor point-to-point communication on the same VLAN or throughout the network, or can only do so through an IDS placed on every VLAN. Simply not the case, not only can NBA technologies monitor this communication, network management systems like Concord or Network General provide robust protocol analysis throughout the enterprise, both WAN and LAN. Visibility, especially as it pertains to security, within virtual environments is significantly hindered using traditional technologies, which is why VMware is choosing to work with Cisco – what happens to non-Cisco shops?

“Yes, attacking the management stack can be a real issue – one that people often overlook in virtual or physical. I’m not saying it’s not an issue in the virtual world, but it’s no worse than the physical world. What happens if you attach your OpenView or Tivoli or IBM, Dell, or HP hardware managers? You still have the same problem.”

The “single point of failure/single point of attack” is actually far worse in the virtual environment, unlike physical systems management technologies, like the ones you listed, virtual environments allow an attacker to take down an entire bank of servers in a single click, many companies do not use HP Openview or Unicenter for command and control, they use them to aggregate information, so attacking one of them will not disrupt an entire data center, it will only disrupt visibility into the data center. That isn’t to say that systems management tools cannot be owned, but again there is a usually one to one relationship fin the physical world for an attack vs. one to many relationship within virtual environments – you own vCenter or an ESX server and you own the enterprise, you own CA Unicenter or HP Openview and you annoy some of the folks in the NOC. What virtual environments have on the backside in a single point of failure also means a single point of continuity, let’s say I bring down a bank of servers that are part of my customer portal (Web server, database, application server, middleware, etc…) in a physical world vs. a virtual world, it would be far easier to return to homeostasis in the virtual world since problem resolution, just like problem creation, have a one to many relationship – that is good!

“I agree that there are over-zealous marketers in the world. I would also agree that there have been plenty of broken virtualization implementations out there. I’ve been in the space for 6 1/2 years which in x86 virtualization is an eternity. I’ve seen plenty of those botched implementations. They got botched for a lot of the reasons you talk about here. Mainly they got botched because there wasn’t a lot of good documentation or best practices or real world experience readily available to the end user.”

This is the real point of my post and I think you sum it up nicely – there are over zealous marketers (many of them), plenty of broken and botched virtualization implementations, many botched for the reasons we talked about and people need to level set, properly plan, and engage the right resources.

Virtualization benefits are not a given and you do not receive “money for nothing”

“you work for VMware and you can only sort of see my thinking that firewalls and physical security won’t cut it – really? How does one stop attacks that move between guest OS systems never traversing the network? How about attack vectors introduced by the virtual layer? The increase in poorly administered and misconfigured systems? The reality is that virtual environments CAN significantly reduce a traditional IT departments visibility and control, and loss of visibility and control will impact an organizations ability to limit the potential of a security incident and to limit the impact once one occurs.”

Yes, there is traffic that will move between VMs without hitting a physical network switch. This happens when 2 VMs are on the same subnet (VLAN). If you’re going between subnets which often demark a security zone then you have to go out to a router and back in – there’s no VM-VM communication. There are things you can do to address VM-VM communication. First, in a VMware environment you can turn this off per VM is you’d like. I haven’t met any customers that go that route but it is possible and in the admin manual. Second, I haven’t seen too many corporate networks that actually implement IDS inside a subnet between every physical system today. Sure there are some out there but most don’t. That doesn’t mean you shouldn’t – quite the opposite. I’m just saying most corporate networks aren’t loosing visibility with VM-VM communication since they really didn’t have that visibility implemented in the first place. Never-the-less VMware saw this as a problem. This is why we partnered with Cisco (to start with) to embed your normal IOS-based virtual switch into the environment. http://cisco.com/cdc_content_elements/flash/dataCenter/nexus1000/index.html. Now you get the visibility you were probably wanting but never knew you lost in the first place.

“One of the biggest problems with virtual environments is that I can now attack a single physical device or vCenter and bring down multiple Guest systems without actually having to attack the guest systems themselves – I now have to deal with a single point of failure and a single point of attack.”

Yes, attacking the management stack can be a real issue – one that people often overlook in virtual or physical. I’m not saying it’s not an issue in the virtual world, but it’s no worse than the physical world. What happens if you attach your OpenView or Tivoli or IBM, Dell, or HP hardware managers? You still have the same problem. Like I tell lots of customers, if you have someone that’s on your internal management network attacking you then you have real serious issues – you shouldn’t blame virtualization or any of the other management vendors for that unless they were the ones that opened the door for that attack in the first place.

“This is where I think marketing hype and reality need to get together and become friends. Claiming a 5 month complete ROI is misleading. I believe that an organization that is looking to implement something new, like a new office or new data center, will experience cost-savings with virtualization, they are going through a refresh – desktop or server – assuming, of course, they take into account the contractors, SI’s, and solution providers involved, the new FTE’s with virtualization specialization, the new eco-system of systems and security management tools needed to manage the systems and the increase in licensing costs.”

This is just something we’re going to have to disagree on. I could sit here and talk about customer after customer I’ve seen with real sub-5 month ROIs. I could point you to VMware’s public case studies or Citrix’s or Microsoft’s or Red Hat’s or SUN’s or anyone else in the virtualization space where customer after customer talks about their near-immediate ROIs. In my mind that’s proof enough. I guess in your mind they’re all blind to their true ROI. Who knows. Bottom line here is it does happen. Not sure how to convince you of that.

“Bottom line: There is no question that virtualization holds a lot of promise for the enterprise, from decreased cost to increased efficiency, but between the ideal and the reality is a chasm of broken promises, mismatched expectations and shady vendors waiting to gobble up your dollars and leave a trail of misery and despair in their wake. Virtualization can improve the efficiency of your operating environment but it requires proper planning, expectation setting and careful deployment.”

I agree that there are over-zealous marketers in the world. I would also agree that there have been plenty of broken virtualization implementations out there. I’ve been in the space for 6 1/2 years which in x86 virtualization is an eternity. I’ve seen plenty of those botched implementations. They got botched for a lot of the reasons you talk about here. Mainly they got botched because there wasn’t a lot of good documentation or best practices or real world experience readily available to the end user. That’s where you and I and other people come in. We need to help these people with real, concrete stuff on how to do this right. The IT industry as a whole needs to come together to help our joint customers. After all, we both sell to the same people and I know of plenty joint customers that we have. Actually the real bottom line is customers need to engage their partners or their customers more and let us help them through the process to make sure they’re successful. I think too many admins try to go it alone. I used to be one of those when I worked in corporate america. Not sure if it was pride or what it was. Usually there’s an eager SE from your vendor just waiting to help you the customer be a rockstar and succeed.

I look forward to more open conversations with you and hopefully together we can sort through the FUD, truly dispel some of these myths, get some good joint best practices out there, and lead this IT space into the next evolution of computing.

Thanks for commenting, I appreciate your time. Note that I said “There is no question that virtualization holds a lot of promise for the enterprise, from decreased cost to increased efficiency…”

Comments below…

“I would have thought the CTO from BigFix (a company I admire) would have put some truth to the myths rather than just perpetuating a lot of them. Some comments below.”

Thank-you I admire the hell out of VMware as well. Remember that the views expressed on this site are personal opinions of the contributors, and may not reflect those of their employers. But just because you don’t agree with my opinions doesn’t mean that they aren’t true.

“Virtualization Reduces Complexity: Yes and no.”

Exactly, that’s what I said, I however focused on the no in this post, while VMware has focused on the yes!

“Patching also gets simplified in a VMware world with VMware Update Manager. The great thing here is if you patch something and it blows up the OS or App you can just revert to the snapshot that Update Manager automatically took before the update was applied. This greatly reduces the complexity in recovering from a bad patch. As the CTO for a patch management company you obviously know that not every patch is successful.”

Yes, of course i know that not all patches are successful, but as for patching being simplified in VMware – this isn’t true at all, what does become easier is rolling back to a pre-patch image, but that doesn’t provide a lot since most patch management systems have fairly mature roll-back capabilities, in fact in terms of patching one now has to worry about patching each guest OS, as well as the virtual layer

Again: One now has to patch each guest OS, as well as the virtual layer, and one does have to actually patch VMware – remember the update on August 12, 2008 that blocked users from starting their corporate servers (here)

“Virtualization Increases Security: I’d agree with you that it doesn’t necessarily make things safer.”

great, so you agree on complexity, and now security =)

“However, you do state that just having firewalls and physical security won’t cut it when you move from physical to virtual. I can sort of see your thinking there”

you work for VMware and you can only sort of see my thinking that firewalls and physical security won’t cut it – really? How does one stop attacks that move between guest OS systems never traversing the network? How about attack vectors introduced by the virtual layer? The increase in poorly administered and misconfigured systems? The reality is that virtual environments CAN significantly reduce a traditional IT departments visibility and control, and loss of visibility and control will impact an organizations ability to limit the potential of a security incident and to limit the impact once one occurs.

“Of course with physical systems couldn’t you copy the data to your laptop and leave as well? If not, then why all of a sudden do you have more access to the VM files? They’re stored in that same physical datacenter. If you now have access to a lot more than you had before then you have some other serious security concerns to worry about outside of being virtual or physical.”

It is easier to copy a file that is the OS, but yes this is true virtual environments are as insecure as physical environments if an IT organization cannot even implement basic controls to limit data leakage. The issue with Virtual environments is that data now moves around in ways that are difficult to monitor – i.e guest OS to guest OS, and most are attempting to monitor this data flow on the network.

“With advancements such as VMSafe security is taking a whole new interesting turn in the virtual world to where VMs will actually become safer to run in than physical machines. “

safer to run, ok this is just BS, yes VMsafe offers interesting capabilities, such as monitoring I/O access while not sitting on the guest OS, but there are so many methods one can use that VMsafe based security technologies would simply be blond to the activity. – also I now have to worry about the security of the virtual layer.

One of the biggest problems with virtual environments is that I can now attack a single physical device or vCenter and bring down multiple Guest systems without actually having to attack the guest systems themselves – I now have to deal with a single point of failure and a single point of attack.

“With that said, virtualization is VERY easy to implement.”

sure it is, it is also easy to install a Windows server, or launch a new Linux disro – implementing isn’t the main thing that requires specialization. A large enterprise, say 30k end-points, 3-4k servers, will need to properly plan how to implement, how to decommission legacy systems, now to handle access controls, how to maintain, manage, and administer, how to properly optimize, how to secure, how to control, and how to report the state of all of this.

“The average payback is less than 5 months. No joke. If it’s longer than that for anyone reading this then you either (a) way overspent upfront, (b) are moving way too slow, or (c) have done the math wrong.”

This is where I think marketing hype and reality need to get together and become friends. Claiming a 5 month complete ROI is misleading. I believe that an organization that is looking to implement something new, like a new office or new data center, will experience cost-savings with virtualization, they are going through a refresh – desktop or server – assuming, of course, they take into account the contractors, SI’s, and solution providers involved, the new FTE’s with virtualization specialization, the new eco-system of systems and security management tools needed to manage the systems and the increase in licensing costs.

“While you did hit on some major myths in virtualization you did nothing to dispel them in which case their not myths to you but rather facts.”

I don’t work for VMware marketing, but I imagine they have an army of resources doing nothing but trying to dispel the myths and propagating the hype – and there is nothing wrong with that. But these are my personal opinions, I never stated they were facts, but they are based on almost 2 decades of IT experience.

Bottom line: There is no question that virtualization holds a lot of promise for the enterprise, from decreased cost to increased efficiency, but between the ideal and the reality is a chasm of broken promises, mismatched expectations and shady vendors waiting to gobble up your dollars and leave a trail of misery and despair in their wake. Virtualization can improve the efficiency of your operating environment but it requires proper planning, expectation setting and careful deployment.

Do you disagree with any of that?

Virtualization Reduces Complexity: Yes and no. Yes since each guest doesn’t need the complexities of NIC teaming, 3rd party HBA failover drivers, backup agents, and soon virus scanning agents. All of this is taken over by the virtualization layer. This means the OS and apps run longer with more reliability. It also means standard OS builds are actually a reality now. Backups get easier and centralized and open the door for actually doing SAN based backups for all of your servers instead of just a handful. Live, file-level based backups can also go on day and night so you don’t have to rely on a narrow backup window. Patching also gets simplified in a VMware world with VMware Update Manager. The great thing here is if you patch something and it blows up the OS or App you can just revert to the snapshot that Update Manager automatically took before the update was applied. This greatly reduces the complexity in recovering from a bad patch. As the CTO for a patch management company you obviously know that not every patch is successful. There are many parts of virtualization that reduce complexity in the environment and it’s only getting better as the industry moves forward.

Virtualization Increases Security: I’d agree with you that it doesn’t necessarily make things safer. However, you do state that just having firewalls and physical security won’t cut it when you move from physical to virtual. I can sort of see your thinking there. VMs are just files and you could copy those to your laptop and leave. I’ve heard that argument before. Of course with physical systems couldn’t you copy the data to your laptop and leave as well? If not, then why all of a sudden do you have more access to the VM files? They’re stored in that same physical datacenter. If you now have access to a lot more than you had before then you have some other serious security concerns to worry about outside of being virtual or physical. With advancements such as VMSafe security is taking a whole new interesting turn in the virtual world to where VMs will actually become safer to run in than physical machines. Imagine if you had a new virus that no one had definitions for yet. With VMSafe (from VMware) you could fingerprint the CPU and memory instructions that looked suspicious, flag it as a bad thing, and then watch for those on every other VM in the environment – all before virus definitions ever created, distributed, and updated. Advantage VMs.

Virtualization Will Not Require Specialization: Everything in IT requires specialization if you want to really fine tune it. Anyone who claims otherwise is a liar or has been in IT for less than 1 day. With that said, virtualization is VERY easy to implement. That’s part of the downfall of a lot of implementations of virtualization. The software is so easy to setup and use and move workloads to that people don’t take the time to learn the new environment they have setup until problems start to creep up. Education is and always will be fundamental in anything you do in life.

Virtualization Will Save You Money Today: It depends. If it doesn’t save you money today it will tomorrow. I’ve been implementing virtualization solutions for 6 1/2 years. I’ve personally helped over 4,000 different companies go virtual. The average payback is less than 5 months. No joke. If it’s longer than that for anyone reading this then you either (a) way overspent upfront, (b) are moving way too slow, or (c) have done the math wrong. In some environments it can save you right up front. I was with a customer that was out of power in their datacenter (not uncommon). It was going to cost them $1.2 MM to run a new power line. They spent $120k in VMware software, $200k in some new hardware (they reused a lot of stuff they had already), and probably used about $350k in man hours from their team to consolidate. They put off that power upgrade for 4 years through this project. That’s immediate payback through virtualization. I could list hundreds of examples like that.

While you did hit on some major myths in virtualization you did nothing to dispel them in which case their not myths to you but rather facts. I personally believe every last one of them is a myth based on my personal experience in this space. Yes, you need to be careful when moving forward with virtualization. That’s why there are thousands of partners out there just waiting to help. Get engaged and get virtual!

DISCLAIMER: I work for VMware as a Principal Systems Engineer. Click on my name for my blog and bio.