Design, query, and evaluate information retrieval systems.

Introduction

Humanity has a never-ending proclivity to generate knowledge. But how does one put the knowledge—the sum total of the collective human experience—to good use? That answer to the question is one of the fundamental responsibilities of the information science profession. Librarians connect their patrons to resources and also function as mediators using those resources during the course of their career to fulfill information needs as they arise. The efficiency of how quickly information needs can be met is dependent on the overall design of the information retrieval (IR) system, the skill in crafting one’s search query, and the pool of information available through the information retrieval system. In the below explication, I will discuss in this competency the design of information retrieval systems, the process in which users parse information into search queries that enable IR systems to retrieve relevant documents, and outline the evaluation of IR systems.

Explication

Design

If one finds themselves involved in designing an IR system, the first step is to determine the primary users and tertiary users of the IR system and then identify the objectives that the IR system needs to meet. One of the best practices when designing IR systems is to conduct user research and use that feedback to determine what features the clientele desire. It is ideal if the design process includes multiple phases soliciting user feedback. With each user trial, program developers can incorporate user suggestions and improve the IR system through an iterative design process.

An IR system processes a search query from a user and then supplies either information that is relevant to the request or the bibliographic records of relevant resources drawn from the collection of surrogates under its purview. The scope of an IR system is often defined by the contents of a single database, but it is not unheard of for an IR system to index multiple sources. Regardless of the substance of the inquiry, the outcome of information retrieval is dependent on how the user interacts with the system and how the information or bibliographic records of resources and materials are organized. The organization of surrogate records is determined by the structure of how the metadata is stored as per the metadata schema. How a document is represented is critical to being able to locate it again.

Take the Machine-Readable Cataloging (MARC) encoding schema as an example of how the structure of metadata can affect the recall of resource discovery. Much time has passed since MARC was established in the late 1960s (by the Library of Congress) (Bolin, 2016, p. 5). As the information in the world increased, and technology progressed, changes were implemented to allow MARC to better catalog this information with greater specificity. These changes to MARC were added in a nonuniform, haphazard manner. The complexity and nonuniformity of how metadata is stored in a MARC environment reduce the recall of relevant bibliographic records. When a single search query should retrieve all relevant records within an IR system’s specified domain, end users—in a MARC environment—would have to conduct multiple searches to retrieve all relevant documents to replicate the same result.

When meeting information needs, subject is a critical attribute when an IR system is engaged in resource retrieval. IR systems can either search the full text of multiple documents or search the subject terms assigned to those documents. Subject terms describe the aboutness of a document. When IR systems index subject terms to identify relevant materials for retrieval, these terms could be either user-generated tags or drawn from a controlled vocabulary.

Search engines and many scholarly databases can scan the entire text of a document to determine subject relevance. Behind any full-text search is an algorithm that identifies words that are used often but filters out words that are used excessively. An example of such a word is the word “to.” Full-text searching is based on the assumption that words in a document—when accounting for frequency and the location of a word (such as a section heading)—do a decent job of representing the aboutness of a body of text. And for the most part, this holds true, but full-text searching does have a few drawbacks.

Documents can include words that are incidental and not representative of their subject matter. In natural language, individual words represent the subject imperfectly due to the meaning of many words stretching across a range of interpretations. The context of the sentence or paragraph surrounding a word narrows the usage of a word to a specific meaning. Full-text searches fail to capture this nuance.

IR systems need not index every word of every document within their purview to determine their aboutness. They can instead focus on retrieving a handful of subject terms. It is not uncommon for IR systems open to the public to use folksonomies to conduct subject searches. Folksonomies are comprised of multiple user-created metadata tags. An example of a widely recognized folksonomy is the tagging system or the Archive of Our Own fanfiction site (Romano, 2021).

Tagging allows users to assign indexable terms to documents, but the complexities of a natural language undermine the effectiveness of search queries. Firstly, discovery tools typically only retrieve exact matches meaning that documents that only use alternative forms such as singular and plural forms of the same word or synonyms (such as juveniles versus children) risk not being identified as relevant—lowering the recall of the discovery tool in question. Secondly, the presence of homographs—words of the same written form but multiple meanings—in a natural language search query could result in the retrieval of irrelevant resources, thereby lowering the search tool’s precision.

It should be noted that there are benefits to using a natural language vocabulary for searching. Many times, users do not need expansive searches to meet their information needs. In such cases, tagging can quickly identify a relevant resource to satisfy their requirements.

Because IR systems have a predisposition to only retrieve direct matches for subject descriptors, information science has implemented controlled vocabularies. In a controlled vocabulary instead of having multiple words represent concept for a single concept, only one term is authorized to denote each concept. If there are synonyms, these should be coded as related terms to the authorized term. The thoroughness of the thesaurus and the IR system’s architecture determine how well the IR system will be able to match a user’s term with an indexable descriptor. The quality of an information retrieval system hinges on its ability to do so. Also, depending on the thoroughness of a controlled vocabulary an IR system will be able to disambiguate (remove uncertainty between deceptively similar terms or attributes) between the intricacies of closely related subjects.

Controlled vocabularies can be problematic when they do not contain terms that quite fit a document that needs to be cataloged. In a similar vein, if a researcher wishes to calibrate their search query to a level of specificity that the vocabulary does not support, then they are out of luck. If a term is not included in the controlled vocabulary, then—from a practical standpoint—the term does not exist. It should be noted that controlled vocabulary can be relatively expensive to create and maintain as such an endeavor requires paying human indexers and human catalogers.

Controlled vocabularies are often structured in a hierarchical arrangement where larger concepts or broader terms (BT) can be subdivided into smaller concepts or narrower terms (NT). If a client is exploring the nature of their information need where subjects are organized in a hierarchical structure, the implementation of a breadcrumb to allow the user to navigate to broader subjects could help them discover relevant material.

Query

From the perspective of users, the main functions of an information retrieval system are search and navigation; but before a user can formulate a search query, they should evaluate which IR system would most likely contain the information they need to satisfy their concerns.

For instance, if one is hoping to look for a scholarly article on a narrow topic, they are probably better off skipping the traditional public library OPAC and going directly to academic databases. Once they have selected an appropriate IR system, the next step is to formulate a search query. Search queries parse a user’s information need into a format—most commonly a string text—that an IR system can use.

IR systems index subject terms to identify relevant materials to retrieve. Subject terms describe the aboutness of an object. One of the certainties in information retrieval is that there will be occasions when end users will use natural language instead of consulting the IR system’s thesaurus to craft their search query using authorized descriptors. This is problematic. After all, multiple words can share the same meaning. An IR system may not be able to meet the user’s information if they are unable to match a user’s term to an indexable term.

When searching library collections, sometimes users know exactly what they are looking for. Therefore, they conduct a known item search. Sometimes patrons are seeking information more broadly. These persons will naturally begin their document retrieval by formulating subject search queries. Sometimes when a patron is conducting a subject search, they are engaged in exploring the information need—itself. Sometimes users have a clear picture of what information they need, but there are occasions when the searcher has not quite discerned what they are looking for. When the objective is unclear, users are functionally exploring their information need.

After IR systems retrieve results there may be various ways that they allow users to organize those retrievals. One common method is to have those results organized by date with the most recent documents presented first. This ranking by publish date can be particularly important if the user is trying to refrain from utilizing material that has been superseded. When results are ranked by relevance, the ranking of retrieved results is determined by computer algorithms. Then the user or an intermediary library staffer determines whether or not the retrievals fulfill the information need.

When an IR system is capable of retaining the search history of a user, researchers can combine previous search queries using Boolean operators to go refine their search results. One widely known example of this is the PubMed database whose advanced search allows medical professionals to combine multiple previous search queries using the AND, OR, and NOT Boolean operators to hone retrievals to a manageable volume.

Should there only be a few results, users will then have to evaluate if the lack of results is due to the composition of the collective resources under the purview of the IR system or if the search can be reworked so that additional documents can be retrieved. If it’s the former, then the researcher should consult another IR system to locate relevant materials.

While many IR systems share common features such as filters and Boolean logic capability, they all have their nuances; a significant component of the expertise of an information professional is their familiarity with these factors as the librarian often plays the role of a mediator between patrons and IR systems. In academic libraries, there is an emphasis on training students (and faculty) on the utilization of their professional or academic field-specific databases and IR systems to conduct research to support their academic and professional careers. When operating in the role of a mediator it is essential that librarians are able to parse a patron’s expressed need into a search query being knowledgeable of the features and nuances of the IR system of the conventions of the field that they are searching. Part of this expertise is gained through experience. As a library assistant, I do not answer many questions about where research databases would be useful (when compared to other sources); but from experience, I know that the ILS is not very good at identifying old movies when the patron cannot remember the title. From experience, I turn to the IMDb database and Internet search engines to identify the film that a patron is hoping to watch. When library staff mediates a patron’s expressed need into a search query, knowledge and experience of the appropriate IR systems will ensure that the patron’s information need is met in a timely manner.

Evaluation

When evaluating an IR system, it helps to break down the IR system into different functions and facets and then measure the different aspects individually. After that, the data can be combined to develop a holistic picture of how well the IR system serves its user population.

The graphical user interface (GUI) is not merely cosmetic. “Dervin’s Sense-Making approach advocates for a user-centric information retrieval system (Savolainen, 2009, p. 1783).” The evaluation of the user interface of an IR system is a determination of whether the interface is intuitive or problematic. From my experience as a library assistant at the Fresno County Public Library, unfamiliarity with computers is a major barrier to the use of IR systems for a significant segment of the public. There is no need to compound that difficulty with a non-intuitive interface. Ideally, users would not need any instruction to operate the interface of an IR system; but no matter how intuitive a specific user interface is, library staffers should always be prepared to assist a patron with operating an information retrieval system.

Librarians are constantly evaluating the usability of IR systems, but “institutional information systems are slow to meet the evolving needs of the public (Savolainen, 2009, p. 1783).” Because of this, non-intuitive features for well-known IR systems can persist over several years. One longstanding example used to be found on the WorldCat homepage. Before their recent update, the “Advanced search” hypertext link’s black text blended in with a dark blue background. The update converted the hypertext link to an icon; but if the older layout still existed, I would have recommended changing the color scheme to provide greater color contrast between the hypertext link and the background. Then the greater color contrast would have boosted the visibility of the union catalog’s advanced search option.

Another measure of a discovery tool’s usability is the presence or lack thereof of appropriate filters. In a related vein, does the IR system offer both a simple search and an advanced search? A comprehensive advanced search allows for greater specification and the application of multiple filters to a single search query. Another aspect of a user interface is how adaptable the interface’s layout is to different window sizes—specifically for mobile devices.

Measuring the speed at which the IR system returns results can also impact the user experience. While I appreciate the scope of the WorldCat union catalog, it is a great resource that allows me to secure interlibrary loans of older and more obscure items, but there have been a few instances where the lag for compiling a list of relevant retrievals irritated the patron and negatively impacted the reference interaction. Additionally, the evaluation of an IR system should determine if there are any accessibility features for the blind, deaf, and differently advantaged.

Fundamental to the performance of an IR system is its ability to match a user’s search term to recognizable descriptors—especially subject terms. After all, multiple words can share the same meaning. The use of a controlled vocabulary can have a material impact on the performance of relevant information retrievals. When evaluating the performance of an IR system to retrieve relevant results, one should measure the precision and recall of results. Precision is a ratio of relevant document retrievals over all documents retrieved and is a measure of accuracy. Recall is a ratio of retrieved relevant documents over all relevant documents with the IR system’s domain and is a measure of thoroughness. An IR system’s ability to accommodate a user who is not carefully crafting their search query from a thesaurus is one measure that IR systems should be evaluated on.

Part of the evaluation of an IR system is the scope of the materials that it indexes. Sometimes a poor information retrieval outcome from an IR system is due to the absence of relevant materials or the poor quality of the discoverable resources. Asking questions such as:

• Are the collections broad or narrow?
• Can one identify any gaps in the collection’s subject coverage.?
• Is the general quality of resources high or substandard? and
• Does the body of surrogate records reference up-to-date resources?

can allow one to assess the scope and content of an IR system.

Evidence

Evidence 1: Evaluation of Reference Resources for Archival Retrieval

As evidence of my knowledge of reference resources in regard to the retrieval of archival materials and my ability to critique the design and effectiveness of these discovery tools to fulfill a search query, I offer this paper evaluating the efficacy of six discovery tools’ ability to locate primary resources (from manuscript and archival repositories). The six reference resources are Discovery from The National Archives (UK), Finding Aids (Library of Congress), WorldCat (OCLC), ArchiveGrid (OCLC), the Mountain West Digital Library (MWDL), and the Online Archive of California (OAC).

This paper is broken up into three sections. The first section discusses the scope and content of the discovery tools. The second section details my experience when searching for “Holocaust AND Internment.” The discussion focuses on the user interface and evaluates the results of my findings. When discussing the user interface, the paper places a particular emphasis on the presence of features that would benefit a researcher looking for primary sources to streamline their search process. Such features include the ability to filter for either archival material or primary sources and the ability to identify and provide archival materials that are open access—available for immediate consumption to a researcher. The third section evaluates the strengths and weaknesses of the six IR systems.

This paper comparing the capabilities of reference resources demonstrates my ability to consult multiple IR systems to assemble a comprehensive accounting of all primary sources or archival material on a particular subject. This paper expresses my knowledge of filters, Boolean logic capability, and other features to manage search results. The report attests to my ability to remain guarded against bias found in collections and an awareness of the features that discovery tools should adopt to be friendly to narrow “mobile-sized screens.”

This proffered evidence to my portfolio is important because it serves as a reminder that IR systems vary widely in their usability as well as the quality and type of resources that they can connect clients to. It behooves information professionals to develop the expertise to direct their clients to the IR system that will satisfy their information needs in the most efficient manner.

Evidence 2: Non-Medline Database Comparison

I am submitting this analysis and comparison of Non-Medline Database medical databases (from INFO 202-12 Medical and Health Sciences Librarianship) as proof of my knowledge and comprehension of medical databases and ability to evaluate their performance to develop recommendations for patrons. This document demonstrates proficiency in the analysis of medical databases and the ability to work in groups toward a common goal and addresses the concepts of information retrieval systems, controlled vocabulary, interfaces, usability, search strategies, and mobile website design. This document discusses the Scopus (Elsevier), Semantic Scholar (Allen Institute for Artificial Intelligence), HAPI, PsycINFO (EBSCO), TOXNET, CINAHL (EBSCO), Licensed Natural Health Products Database (LNHPD), and AgeLine (EBSCO) databases. For each of these databases, the team discussed how the IR system conducts keyword searching and if a controlled vocabulary is involved. This document discusses the best searching conventions in regard to these databases, their interface design, and the scope and content coverage. This proffered evidence to my portfolio is important because it reminds me of the differences between IR systems despite the fact that they may index similar collections of materials.

Evidence 3: Database Design & Subject Analysis-Bibliographic Database

To attest to my insight of database design and subject analysis, I offer this paper detailing a Webdata Pro database that my teammates and I created to allow MLIS students to retrieve data in regard to the subject of information retrieval. The database will store metadata on the title, author source, year, abstract, descriptor, and year range. A definition and instructions were provided for each of these facets. The team entered articles into a prototype of the database to demonstrate how the database would work.

There is an abundance of useful and credible information, but such information can only be put to use if it can be retrieved when it is needed. The bibliographic database outlines the strategies and infrastructure that librarians and information science professionals have adopted for certain types of content available upon demand.

Conclusion

The field of information science is committed to undertaking the task of cataloging all the knowledge that humanity has generated. Professionals in the field of information science accomplish this by organizing the information. Information organization is “The process of describing information resources providing name, title, and subject access to the descriptions, resulting in records that serve as surrogates for the actual items of recorded information and in resources that are logically arranged (Taylory & Joudrey, 2009).” Then an IR system will be able to retrieve the information on an on-demand basis. IR systems can access large and varied collections. The metadata schema—how metadata is structured—plays a significant role in the retrieval of relevant resources. IR systems can determine the subject of materials by indexing the full text of documents. IR systems can also index subject terms specifically assigned to represent the subject of resources. These subject terms can be user-generated tags or be drawn from a controlled vocabulary. Sometimes a poor information retrieval outcome from an IR system is due to the absence of relevant materials or the poor quality of the discoverable resources. The purpose of an IR system is to “match an information need with the information that will satisfy it.”

At the very core of librarianship and information science are information retrieval systems and their companion databases. Information science professionals use a variety of IR systems. It is critical for information science professionals to understand the principles of database and information retrieval system design. Through the courses of INFO 248 Beginning Cataloging, INFO 202 Information Retrieval System Design, INFO 220 Medical Librarianship, and LIBR 247 Vocabulary Design; I have assembled a solid foundation in the principles of Information Retrieval and the central role they play in the field of information science. Through the proffered evidence in this portfolio, the coursework, and my work experience with the Fresno, County Public Library have accumulated a breadth and depth of knowledge on information retrieval systems and their design.

References

Bolin, M. K. (2016). Beginning cataloging and classification. San Jose, CA: San Jose State University.

Romano, A. (2021, February 2). The internet’s most beloved fanfiction site is undergoing a reckoning: A million-plus-word story is holding AO3’s community hostage. Vox. https://www.vox.com/culture/22299017/sexy-times-with-wangxian-ao3-archive-of-our-own-tagging-censorship-abuse

Savolainen, R. (2009, December 9). Everyday life information seeking. In Encyclopedia of library and information sciences, (3rd ed., pp. 1780–1789). Taylor and Francis. http://dx.doi.org/10.1081/E-ELIS3-120043920

Taylory, A. G. & Joudrey, D. N. (2009). The organization of information. (3rd ed.). Libraries Unlimited.