Like it or not, your company is drowning in its own growth. The explosion of corporate content - both in the physical form of documents, records, and data, and in the human form of personal knowledge - has pressed companies into a crisis: Either find a way to tap into and use that knowledge effectively, or watch your company's most vital assets wither on the vine.

Now, more and more companies are turning to search technology for answers.

Alongside this rapid accumulation of data, enterprise search has matured, growing more sophisticated to keep pace with the deluge. From the bud of indexing and retrieval, search tools are blossoming to include capabilities such as taxonomy development, metadata extraction, classification, and personalisation.

Because it is, increasingly, viewed as a way to link users to content and to bolster the return on investment of existing applications and systems, search is emerging as a critical link in the IT infrastructure.

"If you look inside a large corporation, anywhere you swing a stick you can find something that can be made better by search," says Matthew Berk, research director at Jupiter Research.

"We are dependent on information technology for everything employees do. Within an explosion of information, the ability of workers to find the information they need to do their jobs is vital."

Enterprise search tools fall essentially into two camps: external-facing, website search tools for customers and field workers; and tools designed to scour repositories that reside behind the firewall, such as file systems, databases, and business applications.

Natural language technologies in combination with linguistic processing and guided navigation capabilities are being put to use in search for external-facing self-service applications. Suppliers focusing on this space include iPhrase Technologies, Kanisa and InQuira.

Meanwhile, the internally focused enterprise search efforts, lead by suppliers such as Verity, Autonomy, Convera and Fast (Fast Search and Transfer ASA), employ a variety of techniques including concept-based searching, auto categorisation, taxonomy development, summarisation, and personalisation, all in an effort to improve the reach and effectiveness of search.

Categorisation groups content into related groups, whereas taxonomies help structure content by linking similar terms and concepts together. In addition to the larger vendors, a smattering of smaller companies specializes in specific areas such as taxonomy creation and categorisation.

The mixture of all these technologies and techniques is helping drive search toward more of a discovery process that enables workers to unearth content they didn't necessarily know existed.

Specifically, automating the corporate taxonomy development and categorisation processes is helping propel search toward this looser mode of discovery.

Taking search to the next level is "when you don't know the exact terms to query on, or you don't know exactly what is on the network. We want to expose that information", says Andy Feit, Verity's senior vice president of marketing. "Taxonomies can uncover content you may not have known about before looking at it. It lets you expose content to more people."

For external-facing site search, the job of search is not only to solve the query or problem, but also act like a customer service rep who can tell the user about additional services that could prevent the problem in the future, according to Andre Pino, vice president of marketing at iPhrase.

"We are finding people want to shift away from a search-and-find mentality and move toward discovery and managed dialog, especially for external search," Pino says.

In addition, search tools are now employing more than just keywords to unlock information.

Autonomy, for instance, emphasizes the use of pattern-matching techniques that can identify concepts in queries and results. Autonomy's system performs standard Boolean text searches, as well.

Another player in the search space, Endeca, uses a combination of text search and guided navigation to manage the relationships between pieces of content. The Endeca Navigation Engine suggests where users should go next by generating follow-up questions designed to broaden and refine a specific query.

Fast offers Live Analytics technology designed to give on-the-fly data and statistical analysis of content, which enables business performance monitoring, according to Fast officials.

Search spreads its wings

Search is less and less a standalone engine targeting a contained problem. Most large search infrastructure suppliers have a healthy OEM strategy designed to push their technology under the bonnet of a variety of applications, namely content management, portals, CRM, and collaboration.

In addition to exploiting these embedded search capabilities, organisations should also consider a larger search strategy, Berk says. "Search is not just a problem to make individual apps searchable. (Enterprises) need to think about a shared services architecture that can be deployed enterprisewide, and have different line-of-business applications take advantage of (the architecture)," he says.

Aided by the use of XML, open APIs, and web services in search platforms from large suppliers, enterprises can standardise a search offering and stitch the technology throughout the business.

"Enterprise search has been around for a while. Large companies may have 40 different implementations of search. Many are saying, 'Why are we doing this over and over again? Let's standardise on a platform that can be used (in) a variety of ways,' " Berk says.

Getting specific

One of the growing ways to put search to use is through search-derivative applications, in which core search functionality is pressed into service for specific processes such as knowledge management, marketing, SFA, help desk, and training.

"The next generation of search beyond find is taking core language processing technologies - the engines, the neural network, and algorithms - and applying them to different and existing business processes like supply chain and self-service as a way to enhance those processes," says Rob Lancaster, a senior analyst at The Yankee Group.

Furthermore, many enterprises are looking to search tools to solve emerging pain points, such as compliance. In fact, suppliers are rolling out specialised toolsets designed for specific applications of search technology, and they are working with customers on custom deployments.

"You sell what is hot. Right now compliance around information management, Sarbanes-Oxley, records management, and e-mail management is a big issue. Finding that critical piece of information or document can make or break an audit, particularly as enterprise content grows exponentially," Lancaster says.

Many search companies have developed specific modules or added capabilities to existing products to address compliance or to target other applications for search.

Autonomy, for instance, created two divisions within its business to focus on call centre and compliance applications. Audentify, for call centers, and Aungate, for communications compliance, are both based on Autonomy's intelligent data operating layer, but they use search and indexing in different ways. Fast, for its part, plans next year to introduce a series of search-based applications targeting compliance and fraud detection.

Some enterprises, such as independent brokerage firm Linsco Private Ledger, initially eschewed early search technologies because of doubts about accuracy, but came back around as search proved itself as a means to lower TCO and raise the ROI for existing systems.

LPL tapped natural language search supplier iPhrase to add customer-facing search to its password-protected website for LPL brokers/dealers in the field. LPL hoped search technology could alleviate pressure on its call centre, in which 50 people fielded about 50,000 calls per month.

"One big reason we wanted search is that nearly half of the calls from brokers could be answered if they knew where to look on our site. We are wasting our time publishing those documents if people can't find them," says Mike Hamm, an assistant vice president for special projects at LPL.

"We wanted to provide quick access to answers. Content was buried two to five clicks deep in our intranet site. We wanted to bring that information to the forefront."

Rather than build a new knowledge base from scratch, LPL's focus was to exploit content and technology investments already in place, Hamm says.

To ensure accurate results in the specialised financial industry, iPhrase uses an English language dictionary with a financial module on top to ensure that queries for stocks and bonds did not return information about chicken soup and glue.

The iPhrase engine also allowed LPL to tailor the dictionary to custom terms such as "static asset management", which is defined differently throughout the financial industry.

LPL hopes to use the iPhrase search product to enhance LPL's investment research efforts, using cross-selling modules to deliver research related to specific topics, according to Hamm.

Looking toward the future, enterprise search technology will continue to expand beyond its seek-and-find roots, blurring the lines between efforts such as business intelligence and knowledge management in an effort to present a full view of information assets within a company.

With the increased use of technologies that can slice through structured and unstructured data, then identify and analyse patterns in that data, search is bleeding into business intelligence, according to Berk.

"Search is the next business intelligence. Search will replace that layer of Olap and BI that used to fit on top of the database," Berk says. "Search is breaking out of search."

Cathleen Moore writes for InfoWorld

http://www.computerweekly.com/Article126185.htm