Category Archives: Vmware

Hyperconvergence and the Advent of Software Defined Everything (Part 2)

As cravings go, the craving for the perfect morning cup of tea in jolly old England rivals that of the most highly-caffeinated Pacific Northwest latte-addict. So, in the late 1800s, some inventive folks started thinking about what was actually required to get the working man (or woman) out of bed in the morning. An alarm clock, certainly. A lamp of some kind during the darker parts of the year (England being at roughly the same latitude as the State of Washington). And, most importantly, that morning cup of tea. A few patent filings later, the “Teasmade” was born. According to Wikipedia, they reached their peak of popularity in the 1960s and 1970s…although they are now seeing an increase in popularity again, partly as a novelty item. You can buy one on eBay for under $50.

The Teasmade, ladies and gentlemen, is an example of a converged appliance. It integrates multiple components – an alarm clock, a lamp, a teapot – into a pre-engineered solution. And, for it’s time, a pretty clever one, if you don’t mind waking up with a pot of boiling water inches away from your head. The Leatherman multi-tool is another example of a converged appliance. You get pliers, wire cutters, knife blades, phillips-head and flat-head screwdrivers, a can/bottle opener, and, depending on the model, an awl, a file, a saw blade, etc., etc., all in one handy pocket-sized tool. It’s a great invention, and I keep one on my belt constantly when I’m out camping, although it would be of limited use if I had to work on my car.

How does this relate to our IT world? Well, in traditional IT, we have silos of systems and operations management. We typically have separate admin groups for storage, servers, and networking, and each group maintains the architecture and the vendor relationships, and handles purchasing and provisioning for the stuff that group is responsible for. Unfortunately, these groups do not always play nicely together, which can lead to delays in getting new services provisioned at a time when agility is increasingly important to business success.

Converged systems attempt to address this by combining two or more of these components as a pre-engineered solution…components that are chosen and engineered to work well together. One example is the “VCE” system, so called because it is a bundle of VMware, Cisco UCS hardware, and EMC storage.

A “hyperconverged” system takes this concept a step further. It is a modular system from a single vendor that integrates all functions, with a management overlay that allows all the components to be managed via a “single pane of glass.” They are designed to scale by simply adding more modules. They can typically be managed by one team, or, in some cases, one person.

VMware’s EVO:RAIL system, introduced in August of last year, is perhaps the first example of a truly hyperconverged system. VMware has arrangements with several hardware vendors, including Dell, HP, Fujitsu, and even SuperMicro, to build EVO:RAIL on their respective hardware. All vendors’ products include four dual-processor compute nodes with 192 Gb RAM each, one 400 Gb SSD per node (used for caching), and three 1.2 Tb hot-plug disk drives per node, all in a 2U rack-mount chassis with dual hot-plug redundant power supplies. The hardware is bundled with VMWare’s virtualization software, as well as their virtual SAN. The concept is appealing – you plug it in, turn it on, and you’re 15 minutes away from building your first VM. EVO:RAIL can be scaled out to four appliances (today), with plans to increase the number of nodes in the future.

The good news is that it’s fast and simple, it has a small footprint (meaning it enables high-density computing), and places lower demands on power and cooling. Todd Knapp, writing for, says, “Hyperconverged infrastructure is a good fit for companies with branch locations or collocated facilities, as well as small organizations with big infrastructure requirements.”

Andy Warfield (from whom I borrowed the Teasmade example), writing in his blog at, is a bit more specific: “…converged architectures solve a very real and completely niche problem: at small scales, with fairly narrow use cases, converged architectures afford a degree of simplicity that makes a lot of sense. For example, if you have a branch office that needs to run 10 – 20 VMs and that has little or no IT support, it seems like a good idea to keep that hardware install as simple as possible. If you can do everything in a single server appliance, go for it!”

But Andy also points out some not-so-good news:

However, as soon as you move beyond this very small scale of deployment, you enter a situation where rigid convergence makes little or no sense at all. Just as you wouldn’t offer to serve tea to twelve dinner guests by brewing it on your alarm clock, the idea of scaling cookie-cutter converged appliances begs a bit of careful reflection.

If your environment is like many enterprises that I’ve worked with in the past, it has a big mix of server VMs. Some of them are incredibly demanding. Many of them are often idle. All of them consume RAM. The idea that as you scale up these VMs on a single server, that you will simultaneously exhaust memory, CPU, network, and storage capabilities at the exact same time is wishful thinking to the point of clinical delusion…what value is there in an architecture that forces you to scale out, and to replace at end of life, all of your resources in equal proportion?

Moreover, hyperconverged systems are, at the moment, pretty darned expensive. An EVO:RAIL system will cost you well over six figures, and locks you into a single vendor. Unlike most stand-alone SAN products, VMware’s virtual SAN won’t provision storage to physical servers. And EVO:RAIL is, by definition, VMware only, whereas many enterprises have a mixture of hypervisors in their environment. (According to Todd Knapp, saying “We’re a __________ shop” is just another way of saying “We’re more interested in maintaining homogeneity in the network than in taking advantage of innovations in technology.”) Not to mention the internal political problems: Which of those groups we discussed earlier is going to manage the hyperconverged infrastructure? Does it fall under servers, storage, or networking? Are you going to create a new group of admins? Consolidate the groups you have? It could get ugly.

So where does this leave us? Is convergence, or hyperconvergence, a good thing or not? The answer, as it often is in our industry, is “It depends.” In the author’s opinion, Andy Warfield is exactly right in that today’s hyperconverged appliances address fairly narrow use cases. On the other hand, the hardware platforms that have been developed to run these hyperconverged systems, such as the Fujitsu CX400, have broader applicability. Just think for a moment about the things you could do with a 2U rack-mount system that contained four dual-processor server modules with up to 256 Gb of RAM each, and up to 24 hot-plug disk drives (6 per server module).

We’ve built a number of SMB virtualization infrastructures with two rack-mount virtualization hosts and two DataCore SAN nodes, each of which was a separate 2U server with its own power supplies. Now we can do it in ¼ the rack space with a fraction of the power consumption. Or how about two separate Stratus everRun fault-tolerant server pairs in a single 2U package?

Innovation is nearly always a good thing…but it’s amazing how often the best applications turn out not to be the ones the innovators had in mind.

Hyperconvergence and the Advent of Software-Defined Everything (Part 1)

The IT industry is one of those industries that is rife with “buzz words” – convergence, hyperconvergence, software-defined this and that, etc., etc. It can be a challenge for even the most dedicated IT professionals to keep up on all the new trends in technology, not to mention the new terms invented by marketeers who want you to think that the shiny new product they just announced is on the leading edge of what’s new and cool…when in fact it’s merely repackaged existing technology.

What does it really mean to have “software-defined storage” or “software-defined networking”…or even a “software-defined data center? What’s the difference between “converged” and “hyperconverged?” And why should you care? This series of articles will suggest some answers that we hope will be helpful.

First, does “software-defined” simply mean “virtualized?”

No, not as the term is generally used. If you think about it, every piece of equipment in your data center these days has a hardware component and a software component (even if that software component is hard-coded into specialized integrated circuit chips or implemented in firmware). Virtualization is, fundamentally, the abstraction of software and functionality from the underlying hardware. Virtualization enables “software-defined,” but, as the term is generally used, “software defined” implies more than just virtualization – it implies things like policy-driven automation and a simplified management infrastructure.

An efficient IT infrastructure must be balanced properly between compute resources, storage resources, and networking resources. Most readers are familiar with the leading players in server virtualization, with the “big three” being VMware, Microsoft, and Citrix. Each has its own control plane to manage the virtualization hosts, but some cross-platform management is available. vCenter can manage Hyper-V hosts. System Center can manage vSphere and XenServer hosts. It may not be completely transparent yet, but it’s getting there.

What about storage? Enterprise storage is becoming a challenge for businesses of all sizes, due to the sheer volume of new information that is being created – according to some estimates, as much as 15 petabytes of new information world-wide every day. (That’s 15 million billion bytes.) The total amount of digital data that needs to be stored somewhere doubles roughly every two years, yet storage budgets are increasing only 1% - 5% annually. Hence the interest in being able to scale up and out using lower-cost commodity storage systems.

But the problem is often compounded by vendor lock-in. If you have invested in Vendor A’s enterprise SAN product, and now want to bring in an enterprise SAN product from Vendor B because it’s faster/better/less costly, you will probably find that they don’t talk to one another. Want to move Vendor A’s SAN into your Disaster Recovery site, use Vendor B’s SAN in production, and replicate data from one to the other? Sorry…in most cases that’s not going to work.

Part of the promise of software-defined storage is the ability to not only manage the storage hardware from one vendor via your SDS control plane, but also pull in all of the “foreign” storage you may have and manage it all transparently. DataCore, to cite just one example, allows you to do just that. Because the DataCore SAN software is running on a Windows Server platform, it’s capable of aggregating any and all storage that the underlying Windows OS can see into a single storage pool. And if you want to move your aging EMC array into your DR site, and have your shiny, new Compellent production array replicate data to the EMC array (or vice versa), just put DataCore’s SANsymphony-V in front of each of them, and let the DataCore software handle the replication. Want to bring in an all-flash array to handle the most demanding workloads? Great! Bring it in, present it to DataCore, and let DataCore’s auto-tiering feature dynamically move the most frequently-accessed blocks of data to the storage tier that offers the highest performance.

What about software-defined networking? Believe it or not, in 2013 we reached the tipping point where there are now more virtual switch ports than physical ports in the world. Virtual switching technology is built into every major hypervisor. Major players in the network appliance market are making their technology available in virtual appliance form. For example, Watchguard’s virtual firewall appliances can be deployed on both VMware and Hyper-V, and Citrix’s NetScaler VPX appliances can be deployed on VMware, Hyper-V, or XenServer. But again, “software-defined networking” implies the ability to automate changes to the network based on some kind of policy engine.

If you put all of these pieces together, vendor-agnostic virtualization + policy-driven automation + simplified management = software-defined data center. Does the SDDC exist today? Arguably, yes – one could certainly make the case that the VMware vCloud Automation Center, Microsoft’s Azure Pack, Citrix’s CloudStack, and the open-source OpenStack all have many of the characteristics of a software-defined data center.

Whether the SDDC makes business sense today is not as clear. quotes Brad Maltz of Lumenate as saying, “It will take about three years for companies to learn about the software-designed data center concept, and about five to ten years for them to understand and implement it.” Certainly some large enterprises may have the resources – both financial and skill-related – to begin reaping the benefits of this technology sooner, but it will be a challenge for small and medium-sized enterprises to get their arms around it. That, in part, is what is driving vendors to introduce converged and hyperconverged products, and that will be the subject of Part 2 of this series.

Beware of Vendor-Sponsored “Analysis” Reports

Mark Twain allegedly came up with the famous line: "Figures don’t lie, but liars figure." That’s a good thing to keep in mind any time you’re looking through a report that was sponsored ("sponsored" = "paid for") by a vendor that concludes that their product is better than the other guy’s.

Maybe it is better than the other guy’s. But you might want to look closely at what was tested, how it was tested, and whether they were, shall we say, selective in the facts they present.

Case in point: The Tolly Group’s report, released May 27, comparing VMware View 4.6 Premier Edition to Citrix XenDesktop 5 Platinum edition. There are several interesting aspects to this report, which are dealt with in detail in Tal Klein’s blog over on the Citrix Community blog site. Here are a few of the more egregious items:

  • VMware View 4.6 Premier licensing costs less than XenDesktop 5 Platinum. Absolutely true, and absolutely irrelevant. That’s like pointing out that if you load every possible dealer option onto your new car, it’s going to cost more than the basic model. Thank you, Captain Obvious. If you want an "apples-to-apples" comparison, you need to compare VMware View to the XenDesktop VDI Edition. But wait, if you do that, XenDesktop is actually less expensive, and that would be an awkward point to publish in a paper that’s being paid for by VMware.
  • VMware’s PCoIP provides a more consistent multi-media experience than XenDesktop 5. (Over a LAN. Using a single thin client device that did not support any of the Citrix HDX media acceleration features.) Sorry, guys, but once again this is not an apples-to-apples comparison. And did they publish any results of testing across a WAN link? Nope…and for the same reason they didn’t use XenDesktop VDI Edition for their price comparison.
  • It’s easier to upgrade View 4.5 to View 4.6 than it is to upgrade XenDesktop 4 to XenDesktop 5. Once again, both true and irrelevant. It’s easier to give your kitchen a new coat of paint than it is to rip out the cabinets and completely remodel it. Anybody surprised by that? There are significant architectural changes from XenDesktop 4 to XenDesktop 5. It shouldn’t be surprising to anyone that this will involve more effort than a "dot release" upgrade.

I’ve always been skeptical of vendor-sponsored "analysis" reports, and, to be fair, Citrix has used the Tolly Group in the past for its own sponsored reports - but it seems to me that this one is just over the top. Apparently, former Gartner analyst Simon Bramfitt agrees. His pithy assessment of the report: "There are undiscovered tribes lost in the darkest parts of the Amazon jungle that would know exactly what to do if a vendor airdropped a pile of competitive marketing literature authored by the Tolly Group; send it back, and asked [sic] that it be re-printed on more absorbent paper."

What do you think?

IntelliCache and the IOPS Problem

If you’ve been following this blog for any length of time, you know that we’ve written extensively about XenDesktop, and spent a lot of time on best practices and problems to avoid. And one of the biggest problems to avoid is poor storage design resulting in poor VDI performance.

In a nutshell, the problem is that a Windows desktop OS uses disk far differently than a Windows server OS. Thanks to the way Windows uses the swap file, disk writes outnumber disk reads by about 2 to 1. You can build your virtual desktop infrastructure on the latest and greatest server hardware, with tons of processing power and insanely huge amounts of RAM, but if all of the disk I/O for all of those virtual desktops is hitting your SAN, you’ve got a scalability problem on your hands.

Provisioning Services (“PVS”) can help to mitigate this in two ways (assuming for sake of argument that you’re provisioning multiple virtual systems from a common, read-only image): First, if you build your Provisioning Servers correctly, you should be able to serve up most of the OS read operations from the Provisioning Server’s own cache memory. Second, you can use the virtualization host’s local disk storage as the required “write cache” - because all of those write operations have to go somewhere while the virtual system is running.

But XenDesktop 5 introduced a new way to provision desktops called “Machine Creation Services” (“MCS”). We wrote about this in the April edition of our Moose Views newsletter, so if you’re not familiar with all the pros and cons of MCS vs. PVS, I’d encourage you to take a brief time out and read that article. Suffice it to say that, despite all the advantages of MCS, the biggest downside of using MCS to provision pooled desktops was that all of the IOPS hit your SAN storage, which limited the scalability of an MCS-provisioned VDI deployment.

But all of that just changed, with the release of XenDesktop 5 Service Pack 1, which was made available for download a week ago (May 13). With SP1, XenDesktop 5 is now able to take advantage of the “IntelliCache” feature that was introduced as part of XenServer v5.6 Service Pack 2. Using MCS with the combination of XenDesktop 5 SP1 and XenServer SP2…

  • The first time a virtual desktop is booted on a given XenServer, the boot image is cached on that XenServer’s local storage.
  • Subsequent virtual desktops booted on that same XenServer will boot and run from that locally cached image.
  • You can use the XenServer’s local storage for the write cache as well.

The bottom line is that you can move as much as 90% of the IOPS off of the SAN and onto local XenServer storage, removing nearly all of the scalability limitations from an MCS-provisioned environment.

With most of the IOPS for running VMs taking place on local storage, it’s pretty straightforward to figure out how many VMs you can expect to support on a given virtualization host. Dan Feller’s blog post does a great job of walking through the process of calculating the functional IOPS that your local XenServer storage repository should be able to support, and inferring from that number how many light, normal, or power users you should be able to support as a result.

This also means that using XenServer as the hypervisor for your XenDesktop 5 deployment is going to yield a significant performance advantage over any other hypervisor, unless or until the other guys come out with similar local caching features. So, if you’re a VMware shop, my advice is this: Go ahead and virtualize all of the supporting XenDesktop server components on your VSphere infrastructure. Run your XenDesktop 5 VMs on XenServer hosts, and just don’t tell anyone! If you’re asked, just say, “Oh, yeah, these are my XenDesktop host systems - they’re completely separate from our VSphere infrastructure, because we don’t need the (insert favorite VSphere feature) function for these systems.” Your infrastructure will run better, and no one will know but you…

Just Sign the Check Right Here - We’ll Fill In the Amount Later

Back in the old days of minicomputers and mainframes, we used to joke about IBM’s ability to, for all intents and purposes, get the customer to sign a blank check. They were better than anybody I’ve ever seen at getting people to commit to a solution when they really had no idea what the ultimate cost would be - and they were successful because of another cliche (which became a cliche because it was so accurate): “Nobody ever got fired for buying from IBM.” The message was basically, “Yes, we may be more expensive than everybody else, but we’ll take care of you.”

For the most part, those days are long gone, which made it all the more amazing to me to read that VMware is adopting per-VM licensing for most of its management products.

The article nails the basic problem with this licensing approach:

You know how many processors you have on a system, and that’s a fixed number. But the number of VMs on one host — let alone throughout your entire infrastructure — is regularly in flux. How do you plan your purchasing around that? And how do you make sure you don’t violate your licensing terms?

Hey, it’s easy - you just let VMware tell you what to put on your check at the end of the year:

You estimate your needs for the next year and buy licenses to meet those needs. Over the course of those 12 months, vCenter Server calculates the average number of concurrently powered-on VMs running the software. And if you end up needing more licenses to cover what you used, you just reconcile with VMware at the end of the year.

And, before you ask, no, you don’t get money back if you use fewer licenses than you originally purchased.

Sounds to me like a sweet deal - for VMware.

By comparison, the most expensive version of XenServer is $5,000 per server (not per processor, not per VM), and all of the management functionality is included. And the basic version of XenServer, which includes live motion, is free, and still includes the XenCenter distributed management software. (Here’s a helpful comparison chart of which features are included in which version of XenServer.)

A number of years ago, I attended a seminar that discussed the product adoption curve, and how products moved from the “innovation” phase to the “commodity” phase. The inflection point for a particular market was referred to as the “point of most” - where most of the products met most of the needs of most of the customers most of the time. When this point is reached, additional feature innovation no longer justifies a premium price.

The fact is that XenServer and Hyper-V are rapidly achieving feature parity with VMware. If we haven’t reached the “point of most” yet, we certainly will before much more time goes by. So even if you have a substantial investment in VMware already, at some point you have to re-examine what it’s costing every year, don’t you? Or are you OK with just signing a check and letting them fill in the amount later?