The future of enterprise search

A startup company called Powerset gained a slew of headlines last week when it launched a beta version of its search engine, which like other offerings employs natural language processing, allowing users to search sets of information in the form of questions.

But the future of search, particularly within enterprises, will go well beyond processing queries or parsing content. Future search systems will get to know the user - and communities of users - as much as the content it crawls, analyses and indexes, observers say.

"Relevance is in the eye of the beholder - what's relevant for me may not be relevant for you. Consequently, what's needed is a profile of the user (interests, vocabulary, previous searches, job title, etc) and a profile of the content (author, subject, date, who's read it, etc). Great search matches the two up," said Guy Creese, an analyst with Burton Group, via email.

"To do that, these profiles need to be equally sophisticated. Enterprise search vendors for a long time have spent a lot of effort on profiling content, but not profiling users. This will change over time, as systems such as Amazon.com make it clear that knowing a lot about the user makes it easier to find and suggest relevant content."

For example, Creese said, if a user was a network engineer and entered "ATM" as a query, a smart search system could rank results for "asynchronous transfer mode" more highly than "automated teller machine."

While many companies have a role to play and products that work, Google is the company to watch in the long term if you want to know where enterprise search is headed, according to analyst Stephen Arnold.

"When you hear the big companies saying, we are doing an enterprise solution and Google isn't a problem, you have to ask yourself, are these guys connected to reality?" he said during a recent speech at the Infonortics Search Engine Meeting in Boston. "Buying into the Wall Street crowd's [contention] that this is an advertising company is crazy."

In the meantime, the search market has fragmented into a few distinct size classes, analysts say: offerings from major vendors like IBM, Oracle and with its recent acquisition of FAST Search & Transfer, Microsoft, larger independents such as Autonomy, and smaller, specialised vendors.