Earlier this week, I was working with my friend Mike Poor from Intelguardians on an upcoming webinar that we’re doing together on understanding virtualization security risks, and practical steps any organization can take to mitigate them. As we were capturing our screenshots on a semi-production ESX server, I had several a-ha (or maybe it’s holy crap) moments.

It started as we were talking about the VM admin privileges, and observed the sometimes hilarious nature of privileged access…

• VM Admins have unprecedented amounts of privileged access
• When I started to bug our ESX owner too much for stuff, he put me into a role called VM Power User
• Even VM Power Users have a scary amount of privilege. In fact, I was doing a VM move of an 80GB image, and I kept on getting “Data I/O error” from the VI Client. I logged onto the Service Console, and started to try using SCP or FTP, and kept on trying until the entire ESX server crashed (no pings, no VI client access, nothing). My first reaction was, “Oh crap. I crashed it. They should never have given me this much access!” In reality, however, it was the ESX owner who accidentally messed up a vmKernel port group, and it’s been down since. Which leads me to…
• VM servers can be fragile and prone to errors, which can cause the entire virtualized computing environment to go down, even when run by skilled engineers and administrators. This can be exacerbated when there are “many fingers in the pot,” all working concurrently.

But, as Mike and I shifted our focus from the operational risks to the information security risks, it really hit me that…

• VM Admins really do have a breathtaking amounts of privileged access: consider the number of critical IT services that contain confidential personally identifiable information (PII), and the risk of an unscrupulous admin copying the entire VM datastore. (Forget about backup tapes, which are difficult and cumbersome to carry and hide!)
• This risk also extends to the VM Power Users like me, who was granted this status because I was too much of a pain in the a**for the ESX owner
• From a security oversight perspective, this underscores the need for effective control of access: we must ensure that all privileged VM admins (and powerful accounts) are reconciled, mapped to an authorized staff member and an approval by the appropriate manager.
• And the above, of course, underscores the need to control all the configurations that control access. (See my previous post “If A Virtualization Misconfiguration Or Security Vulnerability Exists Within An “ESX Appliance,” Does It Really Exist?”)
• And of course, because trust is not a control, we must ensure that security is aware of all adds, removes and changes for these classes of accounts.

As Mike put it, virtualization does wonders to solve the IT asset management problem, but creates some huge nightmares for the data containment problem. We both agreed that when dealing with sensitive systems and data, partitioning data to separate ESX clusters with no shared access makes a lot of sense.

“Eh, what’s this? What could go wrong?”

And don’t forget to register the webinar Mike and me present on how to mitigate virtualization security risks here!

Mark Gaydos wrote an article called “Ops or Security: Who’s Responsible for Securing Virtualization?” that hints at the virtualization survey results that we will be publishing next week, exploring how organizations are assigning responsibility for securing the virtualized environment. I had a chance to review some of the results yesterday, and I can attest that there are some very, very interesting findings.

But, I’ve promised Mark and team that I would not scoop the article…

In the meantime, I find the whole debate of who owns the responsibility of securing the virtualization environment so familiar. I had to do some digging, but I finally discovered why. It was actually a blog posting I wrote from 13 years ago. I’m enclosing an excerpt below:

Who will own 10BASE-T security?

July 1, 1992

vs.

The “safe” network vs. the “unsafe” network

As a security professional, I grow increasingly concerned with the advent of the emerging network technology called “Ethernet 10BASE-T.” The flaws of the dominant networking technology now, coaxial 10BASE-2, are well known, as anyone who has accidentally mistaken a 25-ohm resistor with a 50-ohm resister can attest. However, while exciting, 10BASE-T introduces a whole new set of information security and operational risks. Some of the obvious risks are:

• IT will be able to deploy things faster than ever: less control!
• Emerging network switches include “spanning ports” allowing anyone to sniff network traffic: less confidentiality!
• Point to point connections instead of shared bus: less ability to monitor the network!
• IT can utilize existing phone cabling: networks will show up in places we least expect!

Although I ultimately believe that 10BASE-T will eventually replace 10BASE-2, and that it will give information security better control. However, in the transition years, there will be a very dangerous period where the lack of ownership of who owns the 10BASE-T risks will remain ambiguous.

I am working on my presentation on this, which I hope to present at either Black Hat or Phrack.

Gene Kim
gkim@cs.purdue.edu

Gosh, reading what I wrote 16 years ago, I find myself a little embarrassed. It all sounds so shrill and hysterical to me now! What I’ve learned is that as long as someone in IT management owns the controls, whether it’s IT security or IT operations, and the controls are working, even disruptive technologies can help the business achieve its goals, safely and securely.

Earlier this week, I read a blog entry by the Lone Sysadmin about our recently released Tripwire ConfigCheck. He noted that it found some misconfigurations in his VMware ESX server, and then goes on to ask a very interesting question:

Now my question is: is ESX 3.5 an appliance or a host OS? Do I actually want to make the recommended changes? Will it mess up something in the future when a patch from VMware assumes something about my environment that isn’t true because I’ve changed it? Exactly how much do I want to go messing around with things like NTP settings when the recommended way to configure NTP is through VirtualCenter?

I look forward to a time when ESX 3i is on par with ESX 3.5, but in the interim do I change things to gain a little security and run the risk of problems later? Is ESX a Linux distribution or is it an appliance?

I am intrigued by this question, because while it may sound philosophical (e.g., “When a tree falls in a lonely forest, and no animal is near by to hear it, does it make a sound?” ), this is actually a fundamental question about IT controls scoping. You must answer this question first before asking who is responsible for them (the question that esteemed Chris Hoff asks on whether it’s VM admin, security, etc…). Luckily, this has a very precise and logically rigorous answer that everyone in IT management (IT operations and security) should be able to answer.

Q: Do any of the VMs running on the virtual infrastructure provide any critical IT functionality required for IT operations, security or compliance?

A: Most likely yes, if we have any production services running on the VMs, or if there are confidentiality or regulatory requirements for the systems or data.

Q: Does VMware ESX rely on critical functionality provided by underlying host OS?

A: Yes. The correct operation of the VMware ESX server (or virtually any virtualization manager) almost entirely depends on the correct functioning of the underlying OS, including the kernel functionality and the userland utilities. One of the most important functionality by the host OS is the privileged accounts (e.g., root) and file ownerships to restrict access to VM systems and data.

VMware published what these configuration settings are in the “VMware Security Hardening Best Practices” document.

Q: Can you bypass the application controls in the management applications (e.g., VirtualCenter, VI Client)?

A: Yes. Although VMware strongly recommends that you use their management tools for managing their virtual infrastructure, in ESX, many use the Service Console to do work that can’t be done in the management tools, such as automation or hairy break/fix work. But, consider the implications of hitting ‘Enter’ in the screen below…

This would make all the VMware ESX datastores readable, allowing anyone to access the ESX datastore to copy all the VM systems and data. (Why bother hacking the VMs when you can just steal the whole thing?)

Configuring this is so important that it is one of the automated test that ConfigCheck does.

Conclusion

If the answer to the above questions are all “yes,” then we can say conclusively that the services being run on the ESX Server rely on not only the VMware ESX functionality, but also the correctness of the configurations of the underlying host OS.

In other words, it is not enough to call the host OS an “appliance,” and remove them from IT management’s area of responsibility. Misconfigurations and security vulnerabilities create risk, regardless of whether you call the ESX Server a “server” or “appliance.”

I have consistently heard VMware state that their virtualization platform is very secure. Yet I was reading an article by Gregory Ness titled “Microsoft Appears To Be Ready To Start Putting the Squeeze on VMware” in which he states with Microsoft’s recent comments around Hyper-V that:

“This officially puts VMware on notice that any delays in penetrating data centers with virtualization will increase the prospects of Microsoft competition, and possibly force lower margins and longer sales cycles. Microsoft has a formidable presence. They won’t have to have a better product. If they deliver a more secure approach to virtualization, it could also spell trouble for VMware, as that is currently a very significant barrier to the full benefit of virtualization.”

So security is a barrier to the full benefit of virtualization? An interesting proposition. With tools like ConfigCheck that allow an administrator to verify if their VMware environment is configured according to the VMware security hardening policy, I wonder how true this is? It might be.

So the question remains? VMware….. secure or not secure? Or does the IT industry need to gain more experience to determine just how secure the environment truly is? If you have comments, let me know.

I just read an interesting commentary on Microsoft Hyper-V by Richard Adhikari on internetnews.com, in which he likens Microsoft to a cat among pigeons because their Hyper-V release is causing quite a lot of “fluttering about and squawking.”

It will be interesting to see how all this pans out, but I think Richard is right on several points:

1. “Microsoft’s sheer size and power will push it to the forefront.”  This will certainly push adoption in corporate environments and, when you consider that the majority of VMware shops are using VMware to virtualize Windows (and many of the current VM admins are repurposed Windows admins), it makes sense that Hyper-V will gain broad acceptance.
2. Hyper-V’s lack of disaster recovery / business continuity features “will hamper adoption of Hyper-V in the enterprise.”  That is likely true at the Enterprise level, but I think there will also be a lot of adoption by SMB’s and other business that don’t necessarily have the high-availability requirements of a global production datacenter customer.

So, does that mean a bunch of people will be dropping VMware?  Not necessarily.  VMware is a more widely deployed platform, with more mature functionality (especially for the largest IT shops).  VMware has also done a fantastic job of making it easy for staff to become educated and certified on its platforms.  And they have done a great job of becoming more proactive with regard to platform security and stability over the past year or two.

What I think will happen is you’ll see “the great coexistence” emerge in many enterprises, in no small part because Microsoft has made a commitment to manage both VMware and Hyper-V in its management suite, which means you don’t have to make an “either/or” choice.  Microsoft is also being very aggressive in its pricing, and has made some smart licensing changes to make it easier to deploy Hyper-V broadly in the enterprise.

And, contrary to one quote in Adhikari’s article, Microsoft doesn’t intend to keep Hyper-V boxed in as a Windows-centric platform; they are talking the most heterogeneous talk I’ve ever heard from them, and are already supporting both Windows and Suse Linux as VM’s in their first release.  I don’t think they’ll stop there.

This can only be good for customers, as both VMware and Microsoft will have to concentrate not just on technical strengths but on the overall combination of (technology)+(cost savings)+(user experience) to win the hearts and minds of the enterprise.  It’s going to be a fun ride for customers.

While I was at the Gartner IT Operations and Management conference this week in Orlando, I spoke to a particular industry analyst.  This analyst suggested that in a very short time frame, as virtualization gains even greater adoption, that the role of the virtualization administrator will actually disappear.  That as virtualization becomes “virtually” ubiquitous that the need for a dedicated individual to handle virtualization security and management will disappear, as every group will have responsibility for managing the parts of virtualization in their respective areas.

The networking group will need to manage virtualization in their environment.  Operations will have to work with virtualization in the areas they are responsible.  Security will need to be involved with virtualization with the processes they manage.  That every group in IT will have to know the parts of virtualization that touch their areas of responsibility.  It will just become part of the jobs.

What’s the time frame for this to happen?   Six months?  A year?  Two years?  Five years?  Hard to say.  It seems like a fair assumption but it will take time for IT personnel to adopt the technology and permeate knowledge about virtualization throughout their teams and organizations.

If you don’t see this happening anytime soon drop me a comment.  I’d like to hear other thoughts on the topic.

Once upon a time, every vendor published a paper about SOX, and some made very convoluted connections to how they could “do SOX.”  Sometimes, it seemed like the only thing the vendor had to offer that had anything to do with SOX was the white paper.  Is the same thing happening with virtualization?

I’ve run across quite a few vendors who say they “do virtualization” when all it means is that their product will run in a VM.  Does that mean their products are aware that they are running in a VM?  Nope.  It’s what I call “dumb virtualization support” or (more politely) “unaware virtualization support.” 

When you’re looking for vendors to support your virtualization strategy, why not look for “smart virtualization support” or (again, politely) “virtualization aware” tools?  This is particularly important for virtualization security, where there may be nuances that a “dumb” tool just won’t address.

“OK,” you say, “But how can I tell the difference?”  You need to ask some questions.  And I found a good starting list of questions to ask your virtualization vendor, created by Pete Lindstrom on The Burton Group’s Security and Risk Management Blog.

Check ‘em out, and add your own - it will help you get up the curve fast.  And, more importantly, ask the questions and see who gives good answers.  This will help you find out which of your vendors are in the game.

We often hear that you need guard the keys to the kingdom but depending on what you do the kingdom can very different. If you are a Windows administrator then protecting the file system is of utmost importance however if you are a DBA then your world is the database. I could go on and on but I think you get the point.

However, in each case the common denominator to these different environments is getting access which means having a proper user id and password to login to the environment. So in this case the kingdom I am referring to is Active Directory. Now, not ever technology points back to AD but many do including Virtual Center. If I am able to get a valid account in the correct group, then I can cause a great deal of havoc in not only the physical world but the virtual as well.

Edward Haletky wrote an excellent story about adding additional access security to protect your ESX server including TCP wrappers, pam_access, and the packet-filtering firewall. He also wrote that in the polls he conducted on VMware Communities that 70% of the users authenticate using Active Directory. Edward goes onto say he is quite skeptical of the AD authentication model which is why he suggests using the three additional access security controls. While I think this is an excellent technical article I believe this is the not the biggest concern when it comes to authentication via AD.

The largest issue with AD authentication is a lack of monitoring of the environment as well as poor processes when people leave their current role or even the company. When I ask my customers how would they know if a new user was inserted into a new group, the majority say they would not unless they did a periodic audit. That is a bit scary considering that these groups can control access to literally the entire server infrastructure.

In addition to the lack of a monitoring policy of this critical environment, is usually a lack of a process for revoking privileges when a user moves into a new role within an organization or worse when they leave. It is the absence of a well defined process that allows someone to keep their existing authority and add new privileges over time.  At a previous employer, I moved from a database role into more of an account management role (my beginning down the dark path of sales) however I maintained all my previous access rights.  I don’t have extensive surveys to support this but I would say that this is probably more the norm than the exception.  To a degree, this makes sense for a short time after a person transitions into a new role because they may provide some overlay assistance.  Then after this grace period is over, the old access rights should be revoked if they are no longer needed. 

So I do recommend  following Edward’s advice in settting up additional access layers for your ESX server, I would also recommend creatinga process to review access rights on a regular basis as well as monitor your cricital Active Directory groups for membership changes. 

In one of my previous posts I talked about who owns the virtualization initiative in companies.  On this blog Mark Gaydos has talked about who is responsible for securing virtualization.  Chris Hoff responded to that on his blog.  Obviously there is some debate about who owns this today and who will own it tomorrow.  Hopefully all these conversations have attracted some security people, some ops people and some VI Admins.  So I have a question for you that your comments would be much appreciated.  What would you like to see in licensing virtualization applications?

Virtualization changes things for how users will need licenses.  Given how the environment is fluid with VM’s going up and down and moving, the user needs to have the proper licenses available on the spot and not have to go through another purchasing cycle to get the additional licenses.  (There was a great article on this by Chris Wolf in the March/April issue of Virtualization Review magazine - I can’t find the article online to link to it, so if anyone finds it please comment here.)  This answer may be different for different types of products.  A security product buy may want certain licensing and a VI Admin may want something else.  I can get an earful sometimes from our Director of IT about how licensing should work and what providers do it well and who does it poorly, while another good friend of mine who is an IT Director for a PR agency has different strong throughts on this.  SO…here is your chance.  If you were to have your way what would it be? SaaS? Per CPU?  Per hypervisor? Per Socket?  Individual perpetual licenses?  Metered billing?  Subscription? Buckets of licenes?  Something totally different?  I could tell you what I think in this post but I don’t want to sway any of you - not that I could. 

Just ran across this article on The Lone Sysadmin blog along with some comments from readers and figured that I would post here in reference to it seeing as I work for Tripwire and am pretty close the development of the ConfigCheck tool and the full Tripwire Enterprise solutions.

Bob asks the question: “Is ESX a Linux distro or an appliance?” we’ll.. that depends on the version you are running, if you are running ESX 3.0 or 3.5 then it’s really more of a linux distro since it has a service console and a bunch of linux binaries. If you are running the newest ESX 3i (also called ESXi sometimes) then it is really more of an appliance since there is no service console and hence no method of directly connecting to the system like you can with 3.0/3.5. In fact I believe that while you can install it on a standard disk based server (I have, it’s quick and easy) it is really intended to be OEM’d on a chip from vendors such as Dell and HP.

So.. back to the real issue. If you treat ESX 3.0/3.5 like an appliance and take a ‘hands-off’ approach then over time you will have an insecure hypervisor. VirtualCenter has an update manager plug-in that helps you keep the ESX hosts patched so that is one method of keeping it up to date.. at least in-so-far as system binaries go. However there is nothing in VC that addresses incorrect or insecure configurations. That is where ConfigCheck comes in and provides an easy means to spot check your ESX server configurations and provide remediation guidance. This is important since there are settings that are not set securely via a standard ESX installation. Heck.. this is true of most software installations as they generally tend towards more flexible / open access and it is up to the enduser to lock them down, the same is true of ESX.

Another question was: “Will it mess up something in the future when a patch from VMware assumes something about my environment that isn’t true because I’ve changed it?” I don’t think so and that is simply because ConfigCheck is based entirely on the VMware hardening guidelines from VMware themselves. Theoretically a future patch might have an issue since rapid change is the norm in our industry (and right now especially true of VMW) but then again we will always be subject to possibilities like that. In any case I really can’t imagine you going sideways by making sure your ESX systems are configured to the Vendors (VMware’s) own hardening guidelines.

As one person commented on Bob’s blog though, if you are testing servers that are purely in a controlled test/dev environment then you may indeed decide not to fully harden the ESX servers since you do give up some flexibility (the age old security vs convenience issue) but for ESX servers hosting production applications, you will definitely want to fully harden them.