I was reading up on tech trends for 2009, and stumbled across Network World’s piece on hot technology, and found the section Virtualization: Hot Technology for 2009.

For many of us have been dealing with virtualization for years sometimes it doesn’t necessarily feel like it qualifies as new any more.  And, inevitably, the “what’s next” question comes up quite a bit.  According to this article <shock!> the answer is:  more virtualization.

Actually, for some enterprises will bring more advanced, more deployed, or more formalized use of virtualization while for others enterprises 2009 may be when they decide to put their toe in the water.

This is good news (to me, at least) for several reasons:

1. If you’re reading this blog,  virtualization is probably and area you know something about, which means you can add even more value in 2009;
2. The economic uncertainty will likely help accelerate this trend (that’s about the only thing good I can say about the economic uncertainty…);
3. Vendors (not just virtualization providers) will likely introduce more new, exciting (and fun) solutions and tools for us to play work with.

From the article, you can see a number of other areas they’ve tagged as “Hot,” including Cloud Computing.  Worth a few minutes, for sure.

On Network World’s Microsoft Subnet, there is a very solid article on “9 Myths of Microsoft Virtualization -  Busted or Confirmed."  It’s actually an interview with Microsoft’s Mike Neil (general manager of Microsoft virtualization) and it is a fun read.

One of the myths I was very interested in was this one:

No. 3. VMware says that its memory overcommitment feature actually makes its wares cheaper in production environments in terms of total-cost-of ownership than Microsoft’s products (and Xen Server, too). Microsoft (and several users I’ve talked to) say this is a myth … although I’ve also heard that Microsoft is working on a similar feature. Is the "memory overcommitment" a myth and if so, why?

I got even more interested when I read Mr. Neil’s response:

So first off, how many IT pros configure their production servers to overcommit anything? Customers want an SLA and they want to know what resources are being consumed by a VM. Memory costs continue to come down and the number of DIMM sockets are going up, making this argument moot. We are focused on the efficient use of resources and using those resources dynamically — pooling the memory of the whole machine and dynamically balancing the memory between all of the VMs, instead of overcommiting a resource that can lead to bottlenecks. So, you can see the caveats on using overcommit in a production environment. As to Microsoft’s plans for new memory, we don’t look at it as "overcommit" we look at it as "dynamic memory." We want to provide the same benefit without the risk. Watch for future details.

So – what’s the real deal?  Hey you VMware practitioners (read that as “paying customer practitioners):  How big a deal is memory overcommit to you?  And (be straight with me) how much do you actually use it in production?

Computerworld has a good “primer level” quiz about virtualization going on at their site.  It’s a good check on what you know (or think you know) about virtualization.

You can study up on what you missed, as they also provide good information on each of the questions and why each answer is deemed “correct” by the experts.

So – give it a shot, and see if you really do know Jack.  I appear to (at least on this topic).

Reading up on VM sprawl this morning with some help from DABCC, Alan Murphy, and Chris Hoff.  At issue is whether “rogue” VM’s and other unsanctioned VM’s (or ones that leave on laptops, etc.) are a big problem for the enterprise.  These folks are debating the topic like crazy both on their blogs and on Twitter, which is good – it gets you thinking.

When I think of sprawl, I think not only of tools but of compliance, standards, and rules.  To use a parallel in the physical world, I relate it to zoning.  Having had the privilege of living in a variety of cities and traveling to lots more, I have seen how differences in zoning and oversight can make a difference in how cities scale out – both from a management perspective and an aesthetic perspective.

Consider some examples:

• Sprawl cities:  These are cities like Houston, Baton Rouge (my home town), and Los Angeles.  They go on for miles, and you find strange combinations of residential and commercial properties side by side with no rhyme or reason.  This has led to a strange mismatch of “stuff” in the cities due to sprawling development and haphazard zoning.
• “Planned growth” cities:  I happen to live in one of these right now – the Portland, Oregon metro area.  I find it interesting that a liberal,  weird city like Portland has a strictly-enforced urban growth boundary and relatively rigid zoning.  Development is not allowed outside of this boundary, and within the boundary there are clear lines in most areas to segregate commercial and residential development. [side note: It certainly isn’t perfect, but this approach has helped keep Portland home values from dropping as quickly during the current housing debacle, since it’s kept housing inventory growth in check]

What?!?

So what the heck does this have to do with VM sprawl anyway?  I think the common thread is this: once sprawl starts it is difficult to stop, and even more difficult to undo.

For that reason, I suggest that any organization who’s early enough in the virtualization –> production process take a hard look at their VM “zoning” regulations, and try to design policies, controls, standards, etc. that will help them as they scale out and prevent their infrastructure from looking like a landscape of virtual strip malls.

Plan for growth and manageability, and you’ll be much happier in the long run.

Just reading up on a very good set of survey data about virtualization vulnerabilities. 

More than forty per cent of IT directors and managers that have implemented server virtualization may have left their IT networks open to attack because they wrongly believe that security was built in.

No surprise that faulty assumptions and faulty implementations are the leading cause of vulnerabilities, is it?

In addition to good data, the article contains some solid recommendations from the survey source “Clavister”:

1. Re-define the security policy to include the virtualization aspect
2. Use virtual security gateways which run inside the virtual infrastructure
3. Protect the virtual administration center and only allow access to this from a separate network
4. Limit the number of administrators who have access to the virtualization administration tools to a minimum
5. Evaluate and test the security level on a regular basis. Replicating the production environment to a test environment is easy with virtualization and this should be utilized.

I very much agree with this list, as it mirrors my own thinking (and I like my own ideas, of course).  The key is to define what “acceptable and expected” means in your organization (for virtualization and everything else), then audit regularly so you can hold people accountable for doing what they should be doing to keep the business out of trouble.

Last week, working with a prospect on a proof of concept, we got into a pretty interesting discussion on agentless verses agent based technologies. The technical staff are looking at ourselves and a competing vendor to help tighten up their security policy and ability to detect changes in the environment for audit and control purposes.

As all competitors do, the other technology company involved had sold hard on their apparent strengths, the fact that they used agentless technology instead of Tripwire’s agent based approach.

At the core of their argument were the following points

• Significantly faster to implement
• Having multiple agents on servers overload the system
• Scale more easily to cover large numbers of assets

Interesting but let’s break it down a bit to find the truth.

Significantly faster to implement – Most decent IT departments now have a way to centrally distribute software. Agents install silently, without the need for a reboot and can gather data significantly quicker than an agentless technology. I’ve never had customers complain about how long our agents take to deploy.

Having multiple agents on servers overload the system – This is simply FUD. Tripwire have been designing agents since 1992 (when Gene Kim was in short pants) so our ability to do everything we need to without adversely impacting the server is based on 16 years of experience. Our realtime agent only impacts to a maximum of around 2% CPU and our collection agent takes less than a minute to grab all the configurations for a CIS policy on a half decent server. We are dormant a lot of the time as well, literally listening for changes but only impacting less than 1%.

Scale more easily to cover large numbers of assets – This is laughable. Imagine scanning 1000 machines via the network for configurations on servers to run against a policy like CIS. CIS on a Windows 2003 box has around 170 tests that it runs, some are checking the same object like RSoP but most tests are individual files or registry keys. So each time you scan for compliance you are hammering your network and servers for tens of thousands of configurations. We approach this differently, we cache the last known good state on the agent and only transfer changes to that state up to our enterprise server. This drastically reduces the network load and has a huge added advantage – we can scan continuously rather than monthly or weekly. Using an agent, we can identify changes to the compliant state in seconds rather than weeks and this mitigates a whole boat load of risk. But configuration assessment is not the only piece to this, remember the customer also wants to monitor for changes to configuration and binary files.

This lead to a rather interesting discovery due to the fact I just couldn’t figure out how the competitor could hash thousands of files to tell you what had changed without an agent. So I asked the prospect what they had said to him concerning change audit and apparently they install microcode every time a check is run and then delete it after the check has finished. Err hang on, so they install code EVERY TIME. They know they need an agent to capture change data but instead of leaving it installed and getting the advantages of having a resident agent, they constantly install and delete code on the server and, even worse, to accomplish this they need to store a privileged account in their centralized management system. That’s just crazy.

Using an agent, if they are written well and perform as expected, is a far better approach to configuration assessment and change audit. We don’t unduly impact the network or monitored device, we have the ability to collect data more quickly and efficiently, are hugely scalable and can be deployed just as quickly as agentless technology with the added advantage of being more configurable, mitigating risk due to scanning continuously and not impacting the network for huge amounts of data.

That is a challenge that most of us in Sales Engineering have when we either: need to get some training and skills developed around ESX server and VMware Virtual Infrastructure or.. need to test some software that is designed to integrate with VMware VI and ESX server. In addition to those needs (and others I am sure) the challenge of finding actual free systems to run ESX and ESXi servers along with VMware VirtualCenter Server is a big one. The ability to do this could also be very handy if you are studying to take the VCP test and need a sandbox to play in. So.. what better way to do it than in VMware’s own workstation product where you can create snapshots of known good states and roll back easily when needed?

I took a look at a video posted by David Davis How to run ESX Server and VI3 in VMware workstation 6.5 based on a whitepaper from Xtravirt.com (warning: 2mb download on the pdf whitepaper) that gives a very good step by step guide on configuring ESX and ESXi 3.5u2 plus the VI3 server and some iSCSI storage to work in Vmware Workstation 6.5

I have tried to do this before without the help of these resources with marginal success so I am looking forward to seeing how well this works out, especially on the new Dell Latitude E6400 and Dell Precision M4400 laptops we have that are running Vista Enterprise x64 with 8GB of system ram. Theoretically these systems should run a configuration like this very well since the setup David is talking about is on a system with 4GB of system ram.

So in the middle of armageddon (which in Portland, Oregon means about 1 inch of snow), I am perusing the interwebs looking for signs of life and come across this little nugget about the McCain campaign selling off office equipment and other stuff including Blackberries that had not been erased. http://tinyurl.com/a4dw8t

The reporters realized what they had and actually began to call some of the numbers.  If I haven’t said it before, I’ll say it now security people: Over 80 percent of the problems you face are not from the faceless hax0rs trying to get in.  Its from inside your own network.  Folks who retire old servers without wiping hard drives, sell their Blackberries on E-Bay without erasing them, people who write their passwords on a sticky note and stick them to their monitor.

You own internal policies may help or hinder some of these efforts…do you require staff to change their passwords every 60-90 days?  Do you require complex passwords?  What are your current account lockout settings?  Have you adopted a security framework like the Center for Internet Security or ISO27001?  Or have you adopted manufacturer standards like the VMWare Hardening Guidelines?  Any of these things will help you put in place some modicum of security that will allow you to mitigate some of these internal security issues.

Can you imagine if the server someone was retiring was an ESX server and had failed to wipe the drives?  You wouldn’t just be giving someone A server…you would be giving them a BUNCH of servers…like Frank Barrone or our very own Gene Kim likes to say…HOLY CR@P!

Patch Tuesday has come and gone and there were at least 28 Microsoft Office, IE and Windows vulnerabilities addressed.  The IT Mall Ninja’s must be in heaven.  They’ll be sitting in Starbucks twittering with their friends about how insecure Microsoft is on their Linux laptops (all IT Mall Ninjas run Linux on their laptops).  The number 28 does seem rather largish from security standpoint but take into consideration that Yahoo has begun some serious layoffs and that to me has a signifigantly higher potential security impact.  Nothing screams vulnerability like disgruntled soon to be ex-employees with root access…and there are layoffs happening all over the Bay Area apparently…

Can you imagine the havoc that can be wreaked by such a person?  Its been known to happen.  One of the most recent ones was planted at UBS by a disgruntled sysadmin.  Probably a good reason to have good change audit tools in place.  I don’t know…something that can detect if someone installs, deletes or modifies files?  Probably be worse if someone decided to tamper with a hypervisor…why bother taking down individual servers when you can bring down a whole bunch at once by messing with an ESX host…if I were a security officer, I would be paying really close attention to my mission critical virtual infrastructure right now…

Just perusing a solid article from Alan Joch on “6 Tips for Server Virtualization.”  All of his recommendations are right on, but I find them to be ‘necessary but not sufficient.’

One thing he doesn’t explicitly mention is to invest in developing your own standards, policies, practices, and competencies within your organization so you can get real value from your virtualization dollar.

Not to beat a dead horse, but the stuff that will tank your investment usually traces back to a people / process / skills shortfall.  I’ve mentioned the ConfigCheck tool here a bunch of times as a free, easy way to get your v12n (virtualization) staff up the learning curve quickly and confidently.

You can also study the volumes of useful best practice information in books (such as Ed Haletky’s books), VMware’s VMTN communities, the VI:OPS portal, killer blogs like virtualization.info, DABCC, and so forth.

As I’ve noted many times – server virtualization is no silver bullet.  But, used properly – with mad skillz – virtualization will change your business for the better.

(oh – and Joch’s article is very good, otherwise)