A mixed reality archive where anyone can be a magician

Museums, archives and other collections have for long strived to employ technical means to facilitate the work of curators, archivists and researchers; and ideally to empower them to contribute, improve, peruse, discover and even re-discover archived materials. Digital technologies, in particular, can be assembled into tools which “embrace interactivity [and] make use of linear and non-linear structures” with reimagined “expressive methods available for displaying and archiving collections”¹.

The last decades have seen a rapid development of mobile information and communication technologies, many aspects of which are also foundational building blocks in the growing field of mixed-reality frameworks and applications. In their origin as tools for communication, we find developments in mobile technologies inevitably enmeshed within the fabric of society and culture - how we learn, communicate and work but also how we collaborate, discover, dream and create. In the best cases, there is a positive synergy between our technology and the way we live.

The way apps are designed nowadays (and even the concept of “app” itself) owes much to the rise of Web 2.0, as to the evolution of smartphones and other mobile devices. Especially since the mid-aughts, an increasing emphasis has been put on user experience (UX). This approach to the design of human-machine interactions finds itself more concerned with human factors, and a context of use that is inescapably social and highly mobile. As we transition into designing UX for mixed-reality environments, a new and exciting space opens up, full of new possibilities. As before, we need to understand this space, to chart the uncharted through exploration and experimentation, with the goal of understanding how specific technologies can facilitate empowering user experiences. In the shift from desktop to mobile to mixed-reality, the body plays an increasingly central role. As does, perhaps more than ever, our innate understanding of a three-dimensional world of objects, people and possibilities for action².

Inherently, a mixed reality application is less of a window into a hyperspace of flat media, and more akin to a museum or exhibition where visitors are encouraged to touch and even play with the artifacts³. Such a space or collection, while sharing characteristics with the physical world which appeal to our evolutionarily-refined capabilities for action and understanding, is also infinitely elastic and malleable, and potentially highly magical: it may easily bend the laws of physics when needed, to be compact or vast, concise or spectacular, directed or playful, on a whim.⁴ As a result, we can reimagine digital archives as magical spaces of play, which any visitor may explore and enjoy regardless of motivation or goal, and potentially find or even create something new upon each visit. These are some of the core ideas which have inspired our work: an experimental archive sharing characteristics with both the physical and the digital, which is partly tool and partly toy, offering the freedom to explore archived materials in a playful way. 

AR[t]chive is a mixed-reality archive of materials relating to the works of artist duo Christa Sommerer and Laurent Mignonneau, who feature prominently in ADA. It appears as a dynamic collection of digital media (such as images, words and 3D objects) procedurally arranged within the physical space surrounding the visitor. The archive is experienced by means of a mixed-reality headset with a clear see-through display, the Microsoft Hololens⁵. In contrast to “fully immersive” virtual reality experiences where users are mostly shut off from the physical world in sight and hearing, AR[t]chive visitors remain able to directly see and hear their surrounding environment. They may move around while exploring the digital collections; and use the layout of the space and real surfaces as affordances to create their own compositions.

Archive contents are arranged in “collections” of media, such as images, videos and 3D models which represent single assets in the archive; but also composite elements such as keywords, books and “globes” representing artworks, which can be expanded into further collections. The archive is effectively navigated by grabbing and expanding items of interest, functionally like following links on a web browser, but performatively more like opening presents or matryoshka dolls. Among the ways archived contents are presented, one may find slightly chaotic swirls of media related to a specific artwork; and three-dimensional word clouds relating to a given topic. Each item displayed in AR[t]chive can be grabbed, rearranged freely, resized and also attached to real surfaces. Using a pinning tool, visitors can connect items with lines to create mind maps or other compositions. The visitor’s surrounding space can be borrowed to confer structure or flow to a composition. Conversely, the digital contents may be used to complement or contextualize actual physical artifacts already present in the space - be it an archive, exhibition space, workshop or classroom, for instance.

Overall, there is no guarantee of how many related items there may be for a single topic or context. A collection that displays beautifully with 100 elements may look deserted with less than 20, so we have to find different strategies to deal with this. As an analogy, two dozen beautiful flowers of different kinds may look great in a thoughtfully arranged bouquet, but be almost lost to view even in a small garden. On the other hand, it is likely unreasonable, if not altogether unfeasible, to split an entire garden in thoughtfully arranged bouquets for easy perusal. Especially in these and similar situations, digital materials and processes can be very malleable and allow us to pick and choose between arrangements, or perhaps ideally, define rules for procedurally constructing displays which are adequate for the context. This can include the visitor’s interests, the layout of the physical space, the archived materials at hand and eventual technical characteristics of the chosen interface. More isn’t always better - human cognition is not limitless, and it is far more capable of handling certain types of noise or clutter than others. Additionally, mobile wearable devices such as the Hololens 2 greatly limit the complexity of 3D scenes that can be rendered, in terms of model details and amount of objects, detail of textures and lighting, among other factors.

The latest addition to AR[t]chive follows the scanning and sorting of printed materials from sources collected by the artists over the years, such as books, exhibition catalogs, conference proceedings, pamphlets and flyers. Thanks to the diligent work of our archivists, approximately 7000 individual scans from 500 sources can now be discovered and perused in the mixed-reality archive. The amount of scanned pages per source varies greatly, as only pages or sections relating to the work of Sommerer and Mignonneau were acquired in this manner. To be displayed in the context of AR[t]chive, the scans had to be processed - converted to lower resolutions and indexed in the app’s local database. This effort serves also as a rehearsal of sorts to the upcoming integration of these same materials in the ADA - or part of that process, at least.

Among the various items in the swirl of media for an artwork, visitors to AR[t]chive may now also find books representing each source referring to it. Each with a distinct cover, they have approximately the same shape as their original counterpart. Like other composite items in AR[t]chive, they can be grabbed and opened up using both hands. In this case, a serpentine ribbon of pages extends away from the visitor, offering a view of all the scans for that source. Page scans can be grabbed, moved or used much like any other image in the archive. Otherwise they remain tied to the collection or source. When their source (the representation of the book) is moved around, the pages will follow, one after another as if tied with an invisible string, behaving much like a gymnastics ribbon would underwater.

AR[t]chive is an experimental tool offering archive visitors an immersive and interactive experience of its contents, testing out in a fairly playful or lighthearted manner some of the potential aspects of a digital archive in mixed reality. Ideally, it may inform the design of magical information spaces, harnessing the best aspects of the latest technologies yet synergizing with our cognition and agency in the real world. A model of a large structure can fit on the palm of your hand, or conversely be made large enough to walk within. Books are not necessarily heavy or inert anymore, they may fly around you when called. You can walk an entire library or just be surrounded by the pages which have caught your interest. Images and words relating to your expedition populate the space, assembling into structures or even spaces of their own, as bricks pieced together into a mind-palace, an immersive mood-board which you can walk through and explore using your whole body.

AR[t]chive has been displayed over two years now as a companion piece to the works of Christa Sommerer and Laurent Mignonneau, integrating the traveling exhibition The Artwork as a Living System at the ZKM Center for Art and Media in Karlsruhe, Germany; OK Offenes Kulturhaus in Linz, Austria; iMal Art Center for Digital Cultures & Technology in Brussels, Belgium; and the Azkuna Contemporary Culture Centre in Bilbao, Spain.

Our work has taken place at Interface Cultures, Department of Media of the University of Art and Industrial design in Linz, Austria, within the framework of a state-funded partnership project with two other Austrian institutions: the Center for Image Science of the University for Continuing Education in Krems, and the Institute of Digital Art, University of Applied Arts in Vienna. The project is titled Lehr- und Forschungsinfrastruktur für Digitale Künste an Hochschulen (in English, “Teaching and Research Infrastructure for Digital Arts at Universities”), abbreviated to LeFo; and is kindly funded by the Austrian Federal Ministry of Education, Science and Research (BMBWF). AR[t]chive was made possible through the contributions of project leads Christa Sommerer and Laurent Mignonneau; researcher and developer Tiago Martins; archivists Lea Schnell, Barbara Jazbec and Julian Stadon; and 3D artists Razieh Kooshki and Vahid Ghaderi. Exhibition planning and set up was assisted by Qian Xu. Thanks to Indiara Di Benedetto for the documental video for AR[t]chive; as well as Nomi Sasaki, Bálint Budai, Sophie Morelli and Tomomi Watanabe (in the photos) for their appearances in audiovisual documentation. Also invaluable were the contributions from our partners and their teams, to whom we would like to extend our sincere thanks.

¹ Oliver Grau, Wendy Coones, Viola Rühse, 2017. Museum and Archive on the Move – Introduction. In Museum and Archive on the Move, De Gruyter, 9-22.

² Ina Wagner, Wolfgang Broll, Giulio Jacucci, Kari Kuutti, Rod Mccall, Ann Morrison, Dieter Schmalstieg, Jean-Jacques Terrin (2009). On the Role of Presence in Mixed Reality. In Presence 18. 249-276.

³ Heaven forbid you treat the artifacts in your traditional museums in such a manner.

⁴ Dieter Bogner, 2017. Museum in Motion? In Museum and Archive on the Move, De Gruyter.

⁵ See: www.microsoft.com/en-us/hololens