A

Information Overload

The world's total yearly production of digital information content amounts to 1.5 billion gigabytes of storage, or 250 megabytes of information for every individual on the planet. Locating key strategic information, such as the latest market predictions for the telecommunications sector, is as easy as accessing the bus timetable for one's local area, at least in theory. However, defects are beginning to appear in this digital fabric, highlighting the problem of "information overload" in everyday life.

Information overload, a great contributor to which is the Internet, refers to the difficulty that users experience in trying to locate and process useful information quickly and easily. The ability to create new information content has far outstripped the ability to process and search it. Moreover, changes in how individuals access information are exacerbating the problem.

In the early stages of the online information age, PC-based Internet access was the primary way of interacting with online information sources. Today there are mobile computing devices such as personal digital assistants (PDAs) and cellular phones. These devices are designed to help people access and manage information. While they provide greater access to online information, they also suffer from significant limitations such as small screen-sizes. A typical cell phone screen is up to 200 times smaller than a standard PC monitor. This imposes limits on one's ability to locate and display the right information quickly at the right time, and therefore the information overload problem itself becomes even more acute. ﾀ Because these access devices are becoming the norm rather than the exception, there is a need to make the next generation of information retrieval tools capable of actively reducing information overload.

Traditionally, the search engine has been the primary tool for information retrieval. Search engines operate by maintaining a comprehensive index of available information. Typically an index represents an information item (such as a web page or a document) in terms of a set of relevant index terms. For example, words that occur frequently in a web page, but that are relatively rare in the web as a whole, are likely to be chosen as relevant index terms for the page.

When an end-user submits query terms to a search engine, they are compared against index terms for relevant items in the search engine index. An overall relevancy score for each information item is computed according to how many of the user's query terms occur in its index, and how important these terms are for that item. The relevant items are then ranked according to their relevancy score before being presented to the user.

Of course there are no guarantees that users will select the same terms for their queries as the search engine has used for its index. This "vocabulary mismatch" problem greatly limits the effectiveness of search engines. In addition, many search engines are incapable of resolving latent ambiguities in query terms. For example, does the query term "jaguar" refer to the cat or the car?

The usefulness of search engines in general is measured by their precision and recall characteristics. Ideally, search engines should have both high precision and high recall.

Precision refers to the proportion of retrieved items that are actually relevant to the user's query. Search engines tend to return very large result lists and many of these results may have been selected because of spurious matches with the query. For example, a user looking for information about Jaguar cars may submit "jaguar" as a query only to be overloaded with irrelevant pages about wild cats among the relevant car-related pages. In this case, precision is low.

The recall of a search engine refers to the proportion of relevant items that are actually retrieved in a search. For example, the "jaguar" search may miss many relevant web pages that focus on the famous E-Type sports car (manufactured by Jaguar in the 1960s) but that fail to mention Jaguar explicitly. In this case, recall is low.

If search engines are limited in their ability to solve the information overload problem by their low precision and recall characteristics, then what does the future hold? What new technologies will provide a solution? One answer lies with recent artificial intelligence (AI) research in the area of personalization. One of the fundamental problems with current search engine technologies is their inability to recognize the motivations and preferences of individual users when carrying out a search. Two users submitting the same query will receive the same results, irrespective of their individual preferences. Personalization ﾀ techniques look at ways of learning about the preferences of individual users over time, by monitoring their online interactions for example, and then use this information to better direct their searches.

For example, a personalization system can learn that a user is interested in cars by noting that the term "automobile" tends to occur frequently in web pages that they visit, and this can be used as an additional query term every time that user searches for information. Thus, the user's "jaguar" query becomes "jaguar automobile" and the search engine is better able to filter out the irrelevant wildlife pages that would otherwise be retrieved.

Personalized information services hold great promise when it comes to relieving the problem of information overload. There are technologies today that are capable of automatically and accurately learning about the information needs and preferences of individual users, and of using this information to guide searches. For example, PTV (www.ptv.ie) is a personalized television listings service based in Dublin, Ireland, that actively learns about the viewing preferences of users (channel, show, and viewing time preferences, for example) in order to compile personalized television guides for them. PTV solves the information overload in the television listings space for PC, PDA, and cellular phone users.

Information overload is a significant problem but personalization techniques provide a real solution at the technology level. However, personalization techniques must also provide a solution at the user level. Due care must be taken to recognize and respect the impact that these new technologies will have on end-users. By their very nature, personalization techniques are designed to learn about the preferences of users automatically—preferences that users may not be willing to reveal to others. Steps are being taken today to define standards for regulating the collection and usage of private user information. With these standards in place, the large-scale deployment of personalization technology may usher in a new information age free from the gridlock of information overload.