Follow Us

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message

SPEC virtualisation benchmark rivals VMware's VMmark

Benchmark could prove an easier, more flexible alternative to VMmark - but it's not cheap

Article comments

So you've rolled up your sleeves and virtualised your data centre or server room to wring more performance out of your existing hardware - and hopefully save a few bucks on power and cooling in the process. The challenge, though, is figuring out whether your virtualised environment is performing as efficiently as it could. Whereas some planners have found it necessary to cobble together homegrown test methodologies for virtualisation workloads, the standard bearer has been VMware's VMmark benchmark.

For some IT admins, though, VMmark hasn't proven sufficiently practical or easy to run as they might've like. Plus, it can be tough to get past the fact that though VMmark is billed a vendor neutral, it comes from a vendor with a vested interest in scoring well in virtualisation tests. But now there's a new benchmark on the block that might prove a more desirable alternative to VMmark: SPEC has unveiled SPECvirt_sc2010, designed to assess the performance as well as power efficiency of various types of workloads in a virtualised environment.

The benchmark, according to SPEC, measures the end-to-end performance of all system components, including the hardware, virtualisation platform, and the virtualised guest operating system and application software. Taking into account that workload types vary from organisation to organisation (or from machine to machine within an organisation), the benchmark supports three types of test apps: a web workload based on SPECweb2005; a Java application server workload based on SPECjAppServer2004; and an IMAP workload based on SPECmail2008.

Related Articles on Techworld

The workloads are injected at different time periods during the benchmark run, representing the spikes experienced in real-world server environments. Additional sets of virtual machines are added until the overall throughput reaches a peak or workloads fail to meet required quality of service criteria. The test takes around three hours to complete at its default settings, according to SPEC.

Beyond measuring raw performance, the new benchmark can measure power performance - comparable to miles per gallon for a vehicle - for a total system or just a server. Those tests draw on SPECpower_ssj2008 for power measurement.

Neutrality at a price

Among SPECvirt_sc2010's top selling points may be the fact that SPEC is a non-profit organisation with representation from an array of tech companies, including VMware, HP, IBM, Intel, and AMD (among others), all of which contributed to the development of the benchmark. Thus, it's easier to assume that the outcome of tests using the benchmark aren't biased toward one vendor or another.

Notably, VMware offers a virtualisation benchmark called VMmark, which the company asserts is vendor-neutral and has garnered a degree of support from the industry, despite initial questions of fairness. VMmark has also earned a reputation for being difficult to work with and for taking too long to run, as well as for onerous test bed hardware requirements. VMmark's system requirements include one server with two CPUs, 6GB of RAM, 80GB of disk space, and a 1Gb NIC; one client PC for every tile, each with two CPUs, 2GB of RAM and 15GB of available disk space, and a private network for connecting clients to the server.

SPECvirt_sc2010's system requirements are fairly vague: "A [server] running virtualisation technology; one or more client systems to act as the controller and/or load drivers for SPECvirt_sc2010; [network] connectivity of at least a 1Gb switch between systems in the testbed," and "the server must include stable and durable storage."

Interested IT planners should also consider the price tag: VMmark is free, whereas SPECvirt_sc2010 costs $3,000 to license.

More information and initial benchmark results are available at SPEC's website.



More from Techworld

More relevant IT news


Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Choose – and Choose Wisely – the Right MSP for Your SMB

End users need a technology partner that provides transparency, enables productivity, delivers...

Download Whitepaper

10 Effective Habits of Indispensable IT Departments

It’s no secret that responsibilities are growing while budgets continue to shrink. Download this...

Download Whitepaper

Gartner Magic Quadrant for Enterprise Information Archiving

Enterprise information archiving is contributing to organisational needs for e-discovery and...

Download Whitepaper

Advancing the state of virtualised backups

Dell Software’s vRanger is a veteran of the virtualisation specific backup market. It was the...

Download Whitepaper

Techworld UK - Technology - Business

Innovation, productivity, agility and profit

Watch this on demand webinar which explores IT innovation, managed print services and business agility.

Techworld Mobile Site

Access Techworld's content on the move

Get the latest news, product reviews and downloads on your mobile device with Techworld's mobile site.

Find out more...

From Wow to How : Making mobile and cloud work for you

On demand Biztech Briefing - Learn how to effectively deliver mobile work styles and cloud services together.

Watch now...

Site Map

* *