Follow Us

We use cookies to provide you with a better experience. If you continue to use this site, we'll assume you're happy with this. Alternatively, click here to find out how to manage these cookies

hide cookie message

Proceed with caution when adopting Hadoop

Big Data crunching might not be the answer to all your prayers

Article comments

Companies should take a pragmatic approach to implementing Hadoop for their "big data" requirements, a new report released by analyst firm Forrester Research urges.

The guidance is based on the experiences of some early adopters of Hadoop, including Yahoo, AOL, Mozilla and Klout. It cautions companies against getting carried away by the hype surrounding the technology and advocates a staged Hadoop deployment driven purely by business goals.

Avoid Hadoop 'science projects' that lack business value, the report noted. "Be careful not to mistake Hadoop's technological gee whiz factor with a genuine business case that delivers real business value."

Hadoop is an open source technology designed to help companies manage and process extremely large volumes of data. Much of Hadoop's growing popularity lies in its ability to break up large data sets into smaller data blocks that are then distributed across a cluster of commodity hardware for faster processing.

Early adopters have been using Hadoop to store and analyse petabytes of unstructured data that other enterprise data warehouse technologies have not been able to handle as easily. The fact that it is an open source technology with growing vendor support has greatly contributed to its growing allure within enterprises.

"Hadoop is the Linux of enterprise data warehouses," said James Kobielus, a Forrester analyst and one of the co-authors of the report. "It is being adopted and forked and tweaked and optimised," in much the same way that Linux was in the early days, he said.

Hadoop holds significant promise for enterprises, especially in cloud-based environments Kobielus said. "It was architected from the get go to be more than a data warehouse. It was architected from the start to handle huge volumes of complex, unstructured content and not just structured, relational data like conventional relational database technologies are, he said.

Even so, the technology is still a work in progress, he said. Hadoop's relative newness, its immaturity, lack of standards, relative dearth of commercial offerings and skills availability, all pose big challenges for enterprises, he said.

The Forrester report outlines a set of best practices that Kobielus said companies can use to guide their Hadoop deployments.

One recommendation is for companies to align their Hadoop initiatives with a clear big data business strategy, Kobielus said. IT managers need to identify specific business situations where Hadoop's capabilities can deliver clear benefits. Yahoo's business case for using Hadoop for instance, has always been to support analytics on very large data sets for ad placement purposes, Kobielus said.

Companies need to have similar specific goals when implementing Hadoop. In most cases, it is best to start with projects that have near-term benefits and easily tangible impact. Before deploying Hadoop, it is also best to see if other enterprise data warehouse technologies can handle the requirements.

"Explore a big data approach like Hadoop only if your data volumes are likely to scale into the high terabytes or even petabytes," level and include a wide variety of data types, the Forrester report noted. "If you overinvest in data storage, computing and networking capacity, you'll add to your cost overhead without any concomitant business benefit," it warned.

Forrester also recommended that companies use enterprise-grade platforms and tools for powering their Hadoop environments. Where possible, companies should evaluate the functionality, scalability and maturity of the commercial offerings and tools that build on and extend the Apache Hadoop open source distribution.

Some examples include Hadoop distributions from vendors such as Cloudera, EMC Greenplum, IBM and Hortonworks, the report said. It's unwise to attempt to build an in-house Hadoop stack when there are commercial options, Forrester said.

It's also unwise to completely standardise or commit to one vendor's distribution at such an early stage because it is unclear how the market will evolve over the next few years, Kobielus said.

Companies would also do well to build their own Hadoop centres of excellence, Kobielus said. Given the severe shortage of skilled Hadoop professionals, companies should try and develop Hadoop skills in-house as much as they can.

The focus should be on upgrading the skills of database administrators and integration specialists and getting them familiar with technologies such as MapReduce. They should also connect with the broader Hadoop community to keep abreast of best practices and trends, Kobielus said.


More from Techworld

More relevant IT news


Send to a friend

Email this article to a friend or colleague:

PLEASE NOTE: Your name is used only to let the recipient know who sent the story, and in case of transmission error. Both your name and the recipient's name and address will not be used for any other purpose.

Techworld White Papers

Choose – and Choose Wisely – the Right MSP for Your SMB

End users need a technology partner that provides transparency, enables productivity, delivers...

Download Whitepaper

10 Effective Habits of Indispensable IT Departments

It’s no secret that responsibilities are growing while budgets continue to shrink. Download this...

Download Whitepaper

Gartner Magic Quadrant for Enterprise Information Archiving

Enterprise information archiving is contributing to organisational needs for e-discovery and...

Download Whitepaper

Advancing the state of virtualised backups

Dell Software’s vRanger is a veteran of the virtualisation specific backup market. It was the...

Download Whitepaper

Techworld UK - Technology - Business

Innovation, productivity, agility and profit

Watch this on demand webinar which explores IT innovation, managed print services and business agility.

Techworld Mobile Site

Access Techworld's content on the move

Get the latest news, product reviews and downloads on your mobile device with Techworld's mobile site.

Find out more...

From Wow to How : Making mobile and cloud work for you

On demand Biztech Briefing - Learn how to effectively deliver mobile work styles and cloud services together.

Watch now...

Site Map

* *