Inside Amazon's Cloud: 50,000 server instances per day

Amazon doesn't release a lot of detail about the growth of its Amazon Web Services (AWS) cloud computing operation. But a recen analysis by Guy Rosen found that Amazon is provisioning more than 50,000 EC2 server instances per day. 

Cloud technologist Rosen created this estimate by examining EC2 resource IDs (if you read his post, you'll see how he broke down resource IDs to understand their meaning) and doing a time-series analysis on how much the IDs are incremented per hour.

A 50K/day run rate would imply a yearly total of over 18 million provisioned instances. Rosen admits that his understanding of the resource ID might be incorrect, thereby creating flaws in his analysis; however, even if he's off by an order of magnitude, that would imply a yearly run rate of 1.8 million provisioned instances.

I'm not aware of Amazon announcing its total EC2 statistics, but it has announced S3 stats (S3 is Amazon's storage services.)

In February of this year, Amazon announced S3 contained 40 billion objects. By August, the number was 64 billion objects. This indicates a growth of four billion S3 objects per month, giving a daily growth total of about 133 million new S3 objects per day.

Given the growth in S3, 50,000 EC2 instances being provisioned each day doesn't seem far-fetched at all, making the yearly estimate of 18 million provisioned server instances plausible.

By way of comparison, total server shipments for Q209 were around 1.4 million, according to IDC. Of course comparing server shipments to EC2 provisioned instances is not direct. For one thing, each of the servers shipped in Q2 were very likely going to be virtualised, implying a much larger number of virtual machines being installed, which would be a more appropriate comparison to EC2 instances.

If each server hosts five virtual machines, that would imply a total quarterly VM instance count of 7 million, with a yearly total of 28 million (the number will probably be higher, perhaps significantly so, since the 1.4 million physical servers comes at a time of historic low sales; the yearly total could be significantly higher than 5.6 million, which would therefore raise the total number of virtual machines being hosted as well).

Moreover, while one can confidently state that each physical server represents a true increment to the pool, one cannot make the same claim about EC2 instances. A single Amazon Machine Image (the virtual machine) may be launched multiple times as an EC2 instance, thereby indicating that the true number of individual Amazon servers may be lower, perhaps much lower, than 50K per day. Of course, one could make the same observation about the virtual machines hosted on the physical server count, so the quarterly VM instance count of 7 million might be somewhat lower as well.

Without overstating the accuracy of this analysis, what can we conclude from Rosen's analysis?

People are putting a lot of servers up on Amazon: Whether the real number is 1.8 million or 18 million EC2 instances, it's clear that a lot of computing is being done up on Amazon. And even if many of those instances are "double-dippers" (that is, represent a single AMI that gets launched multiple times), there's still a lot of EC2 instances running on the AWS framework.

People are putting a lot of servers up on Amazon because it's cheap: There's lots of debate about whether cloud computing through an external provider can be less expensive than via an internal data centre. I've addressed this question before in previous posts.