Interaction: Beyond retrieval

What is the role of interaction in the next generation of information systems? In classical information retrieval systems, interaction enables and facilitates the search for relevant documents. Typically, the interface is based on a three-pronged interaction paradigm: formulating keyword queries, scanning results lists, and browsing hypertext structures. This paradigm has been enormously successful. It is the basis for almost every modern, mainstream, large-scale information system, from digital libraries to web search engines and corporate intranets. But is this retrieval-centric approach to interaction a suitable foundation for the next generation of information systems? Is it a satisfactory basis for supporting the full cycle of knowledge discovery, creation, and use? When compared with the physical world, interaction in digital environments is sparse. This is especially true in the realm of cognitive and knowledge work activities. In the physical world, these activities usually involve paper documents. Paper supports a number of simple yet invaluable interactions. It can be cut, folded, rearranged, sorted, and annotated. With paper documents these interactions are easy and obvious, but with digital documents these interactions are often complex and cumbersome, if they are at all possible (Sellen & Harper, 2002). What is more, and critical, is that even simple interactions are an integral part of cognitive and knowledge work activities (Fast & Sedig, 2005; Kirsh, 1997; Marshall, 2003). Interacting with the world is more than a mechanism for accomplishing goals-it is crucial to how we formulate our goals, shape our thoughts, and understand the world. As a result, interaction is an integral part of many cognitively complex activities, such as wayfinding, sense making, decision making, interpreting, modeling, and problem solving (Dourish, 2001; Fast & Sedig, 2005; Mirel, 2005). Designing for these kind of activities requires interaction mechanisms that go beyond the familiar query-search-browse paradigm. This panel will re-examine the concept of interaction, viewing it as an essential ingredient in the next-generation of information systems. In this view, interaction is connected to all the elements of internal and external cognition (including physical artifacts and digital representations), as well as all the modalities, such as vision, sound, and haptics. This approach offers explanations to long-standing questions, such as why information visualization has succeeded in niche markets but not in mainstream applications. More importantly, it advocates a holistic view of the information life cycle, where retrieval is one part of a chain that includes creation and use of information as well. This panel brings together interested researchers to discuss their views on the theoretical and practical issues of interaction in next-generation digital knowledge environments, and share their recent research findings. Of particular interest to the panel is interaction with visual representations, which is of central concern for information visualization and 3D virtual environments. The panel consists of five researchers: Karl Fast will present a descriptive taxonomy of interactions with visual representations for digital library environments; Yan Qu will introduce sense-making as a theoretical framework that may guide the development of interactive systems in knowledge environment; Xia Lin will stress the significance of content representation, and argue for a tight coupling between content representation and interaction; Xiaolong Zhang will summarize the role 3D virtual environments may play in support of integrated information environments. Huahai Yang will discuss the possibility of combining visual representation with other modalities in interacting with information environment, and demonstrate its integration with natural language based retrieval. The modern digital library is a curated and searchable document repository, a descendant of both information retrieval and hypertext. This approach to digital library design has been pragmatic and successful, especially while the field has grappled with fundamental, first-generation research challenges such as digitization, organization, preservation, and access. But digital libraries are entering a new phase in their development, a phase that envisions them as tools that actively support thinking, reasoning, and learning-the digital library as knowledge environment (Larsen, 2004). Transforming digital libraries from document repositories into knowledge environments requires research in several directions. Interaction is one of the most important and promising of these directions, especially interaction with visual representations (Fast & Sedig, 2005). Of course, digital libraries are already interactive. But are they always appropriately interactive for a given set of documents, users, and activities? That is, how does a designer determine which interactions and representations will best allow people to achieve their epistemic goals? Answering this question requires, ultimately, a prescriptive taxonomy of interactions. Developing such a taxonomy is, however, a long-term research goal. This part of the panel will take a step towards that goal by presenting a descriptive taxonomy of interactions with visual representations in digital libraries. The taxonomy is a principled classification of a diverse range of interaction techniques. It aims to serve both as a preliminary framework for designers, and a stepping stone towards a prescriptive taxonomy. This part of the panel draws our attention to a theoretical approach called sense-making - a process where people understand unfamiliar things during accomplishing a task (Dervin, 1983; Russell et al. 1993). Research on sense-making deals with how people seek, organize, understand, and use information in knowledge work context. Sense-making is a complicated, iterative procedure. People often use external representations when the things they are making sense of have rich structure, or become difficult to handle using only internal mental representations. The sense-maker goes through the cyclic processes of searching for representations that reduce the costs of task operations. Interaction with external representation is an important mechanism by which sense-making is made possible. We will share our experience building interactive systems to support sense-making process with functionalities such as web searching, text clustering/classification, annotation, structure comparison, etc, with particular focus on our most recent work in helping people build knowledge representations through interactive information seeking processes. Interaction is an important information seeking process (Marchionini, 1995). Whether searching or browsing, users need to interact with information systems in order to clarify their information needs or filter through information provided by the imperfect retrieval process. Early research on interaction mostly focused on interaction as an advanced technique for interface design. In this presentation, the panel presenter will discuss the entwined relationship of interaction and content representation. Specially, using several of his information visualization prototypes as examples, he will argue that interaction needs to be guided by good content representation. Meanwhile, interaction should also be an integrated part of content representation. In the digital environment, content representation can be generated dynamically from the underlying information. Both content structures and concept relationships can be visualized. Users can select to interact with the content dynamically from different angles. Thus the meaning of structures in content representation will depend on the interaction process. Users might determine different meanings for the same structure. Dynamic content representation should take this into account. Most of current IR-related 3D visualization tools largely focus on the exploitation of the extra dimension in 3D space in visualizing more information attributes, but overlook the advantages of 3D virtual environments in support of an integrated information environment. Information retrieval (IR) becomes an increasingly complicated task. It often involves different query terms (Furnas et al., 1987), different information resources, different searching techniques, and different information artifacts (Bates, 1989). Thus, IR systems should provide tools to support not only those core IR tasks such as query formulation and query-initiated navigation, but also those tasks that may seem peripheral, such as information accumulation, information evaluation, information organization, and so on. Virtual environments, which explicitly put information, users, and user tasks in the same space, may offer opportunities to improve user performances in information retrieval. Research in human-computer interaction (HCI) has provided affluent designs that can support integrated information environments (Furnas & Ranch, 1998; Hearst & Karadi, 1997; Card et al; 1996; Mackinlay et al., 1995), but the potential of these designs has not been fully recognized by IR researchers. For example, the ButterFly interface (Mackinlay et al, 1997) is usually regarded just as a tool to visualize citation networks. The IR community fails to see the features of this visualization technique in supporting information access into different resources, supporting information assessment, and supporting the evolution of IR processes in the same information environment. Although such integrated information environments can also be implemented in 2D, as seen in Furnas & Rausch (1998), current WIMP-based 2D user interfaces lack the flexibility to let people use space freely in managing information and tasks. To support user actions in an integrated information environment, HCI and CS researchers have made great efforts to address interaction issues in 3D. For individual users, various tools are available to enhance user performances in navigation by improving the acquisition of spatial knowledge (Tan et al. 2000; Wear & Fleet 1997; Elvins et al., 1997) and traveling (Hanson et al., 1997; Mackinlay et al., 1990). For collaborative information retrieval, users can find tools to support collaborative browsing (Benford, Snowdon, et al., 1995), social awareness embedded into 3D space (Benford, Bowers, et al., 1995), social navigation (Yang & Olson, 2002), cross-space, cross-scale information sharing (Zhang, 2005; Zhang & Furnas, 2005), and so on. Information visualization should go beyond simply presenting information, and also consider the support of complicated IR activities. This implies the integration of users, information, and technology in IR tools. In HCI research, 3D environments have played an important role in delivering such unified information space. While researchers are keeping exploring new 3D designs to support information retrieval and knowledge exploration, we can also take a more conservative approach by borrowing those beneficial techniques seen in 3D into 2D user interfaces. With a commitment to enhance user interactions in design, we would be able to improve the quality and user experience of IR tools and services. This part of the panel seeks to find possible applications of visual interaction as part of an interactive circle that incorporates other modalities. In particular, we consider the possible combination of natural language processing techniques with visualization. Natural language and direct hand manipulation are two primary means through which people interact with the physical and social environment. These two forms of interaction often tightly integrated in practice. It would be an ideal situation if integration of these two forms of interaction is supported in interactive information systems. This panel presenter argues that advances in natural language processing (NLP) and information visualization have accumulated significant amount of tools for us to start attempting this integration. The strong AI sense of NLP in information retrieval has mostly been considered futile (Jones, 1999). If a literal question-answering application of NLP is not an optimal solution, would NLP still be a viable approach for information retrieval and beyond? Through the experience developing a natural language based search interface for structured data (Li et al, 2005), we show that NLP coupled with visual interaction offers a good combination that maximizes the benefits of both techniques. NLP is suitable for untrained naïve users to express their information needs initially, whereas visual interaction affords fine-tuned adjustments so the user and the system could reach a level of “conceptual agreement” that produces good retrieval results.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.