The Quantum Archivist

A recent post in the AOTUS blog by David Ferriero entitled  “The Future is in the Palm or Our Hands” called for archivists to think about ways to connect archival collections to potential users through mobile devices. Ferriero was speaking specifically about NARA and its collections, but this idea is of course broadly applicable to all archives and collections.

The great opportunity for archives  in connecting to users through mobile devices comes from one special nature of these devices: they can locate themselves in space, that is, they know where they are. And since they know where they are, we can link digital objects in our collections to those locations and have them pop-up on a mobile device and announce their presence, without the user doing practically anything at all except holding up his smartphone.

The idea of geo-coding locations for historical documents (especially photographs) has been around for some time. I was a part of some work in the late 1990s at Tufts University in collaboration with the Perseus Digital Library to overlay historical resources of London and Boston

"Boston Streets" at Tufts University

onto historical maps. These were large-scale, programming intensive projects that used what we would now consider primitive, web-based GIS display tools to visually display and deliver historical information through a web-browser. They certainly were not optimized for mobile devices, because, of course those devices didn’t really exist then. While these tools were good at showing a visual representation of the location of historical information, we didn’t yet have the ability to do what we could imagine, which was to stand in a particular spot on the earth and connect with the historical record of that particular place.

The advent and general adoption of the Google maps API  made it possible to more easily connect content to maps, and the development of smart phones and web-enabled mobile devices makes it possible to deliver historical documentation to people right where the history happened even though the resources that document that history are stored in our repositories.

How great would it be to stand on the steps of the Lincoln Memorial and hear Martin Luther King Jr.’s “I Have a Dream” speech? Or stand on a street in San Francisco and see photos of that street after the 1906 earthquake? Actually, I don’t know that you CAN’T do this right now. The technology exists, I don’t know if anyone has done it yet.

Of course, there are people already doing this sort of thing. For example, if you are in Philadelphia, you can point your iPhone to http://phillyhistory.org/i/ and be shown historic photos of Philadelphia based on your location. North Carolina State has produced WolfWalk (http://www.lib.ncsu.edu/wolfwalk/) which provides information on the history of approximately 60 major sites on the NCSU campus drawn from resources at the University’s Special Collections. In both cases I need to know that Phillyhistory or WolfWalk exists and what the url is.

What would it take for my Google maps app to list, not only restaurants or barber shops, but historical documents, images, and media related to nearby places?  Well, maybe that’s getting a bit too optimistic, but we can still dream can’t we?

… and your enemies closer. Whether this comes from the Godfather, or Napoleon, or an Ancient Chinese philosopher, it may explain what a fervent believer in open source like me is doing giving a presentation at an ILS vendor’s user group meeting here Chicago.

Image from Wikipedia

Like most academic libraries, we use a combination of tools, applications, and resources to collect and deliver our content. In the past few years, we have made an explicit choice to move toward open source software solutions, at least for our presentation layer.

Why did we do this? There are a number of reasons most of them philosophical and operational rather than economical. Although open source is free (like a puppy), there are many costs associated with development and maintenance. I don’t think the economic argument has a lot of value in terms of decision making, since anything big costs a lot of money. Big products from vendors and big software development projects seem to me to be in the same ballpark cost-wise.

I’m not going to go deeply into the whole argument here, and it is possible to argue any of these points. But my opinion is that given a certain level of technical expertise (that not everyone has or can get) I think the advantage of open source is the ability to be nimble in the face of new demands and serve your user base in much more focused way than vendor solutions can offer. The downside of course is that you have to maintain it all yourself and there is no easy phone call to customer support that you can make to say “just fix it please!”

Which brings me back to Chicago, physically and intellectually. I am part of a panel with two colleagues from our library to talk about harvesting and aggregating metadata–including primary source metadata–into a presentation layer that is usable and useful for researchers.

We will of course talk about the vendor-supplied option that we currently use to harvest and aggregate book and primary source metadata, but I’m going to go another step beyond that to talk about the value of standards-based data exchange and demonstrate not only the vendor-based model, but a few open source based applications that we have developed here at the library because my point is that data aggregation is a matter of policy and practice, not applications.

What I am saying is that aggregated metadata can be used in a variety of ways to support discovery, and that open source applications based on standards that can be re-used and re-purposed for different audiences can go a long way toward serving the needs of our local audiences in ways that “one-size-fits-all” vendor products don’t seem to be doing.

We’ll see what sort of reception this gets in a room full of people who presumably (at least in my mind) are here to hear about the latest product from their vendor and why they should buy it.

Penrose Library's new web site

Recently, the Penrose Library launched a brand new “user-centered” web site. I’m not a big fan of the term “user centered” since I think it is often used as an excuse not to be creative. But what we are trying to do is make available to each group of users the things that they are most interested in right up front. Rather than forcing them to learn how the library is organized administratively, we wanted the site to answer the question: “What do I want to do?” based on a second question:  “Who wants to know?”

Some of this approach was informed by a workshop given by Nancy Fried Foster, library anthropologist at the University of Rochester, that some of us attended a year or so ago. She had recently completed an ethnographic study of undergraduate research behavior at the University of Rochester. Her findings were published in 2007 in a book called “Studying Students: The Undergraduate Research Project at the University of Rochester.”

Other parts of the design were informed by our own observations of user behaviors from Faculty, Students (both graduate and undergraduate) and University staff. If you are interested, there is a short “tour” of the new site, narrated by our Instruction Librarian, Carrie Forbes.

My point is really that, in order to be successful, especially in a library that hopes to teach research and scholarship skills as well as provide information, one size does not fit all, and there should be as many different library experiences as there are groups we wish to serve. Our next step is to extend the granularity of experience down to the individual, and provide each person (or at least each person who is affiliated with DU) with a experience that is tailored to his or her own interests and experience. I mean, if Amazon and L.L. Bean can do it, why can’t a library?

As archivists, we are always trying to find the best way to connect to our user community to give them what they want in the best way possible. The idea of quantum archives is to connect people to the content in as granular a way as possible while preserving the opportunity for them to contextualize the content in ways that they want.   I was recently involved in a conversation where someone who wouldn’t ever consider himself an Archivist put this idea in the most succinct way.

Thought Equity Motion is a for profit stock footage fulfillment and video asset management service that manages the video libraries of some of the biggest media organizations in the world. They happen to be based in Denver and I’ve had a couple of opportunities  over the past few months to talk with Frank Cardello, the EVP for Corporate Development at TEM. TEM has just launched a joint venture with the NCAA called the “NCAA Vault.” Timed to coincide with the beginning of the annual Men’s basketball tournament, the Vault features “ten years of full games and highlights” of the Sweet 16. As a basketball fan I appreciate this opportunity, as an archivist I am even more impressed with how TEM and the NCAA thought about presenting historical information.

A model for archival access?

While I can watch an entire game, I can also use search terms to limit to particular teams, years, and players. There are also some pre-defined categories like “great shots” or  “great finishes.”  Next, but not finally, you have the opportunity to search (using a text-based search box) through the play-by play track of the video footage for a particular moment or play within a game. You can select this clip and share it in other applications.

Frank said that the idea behind this approach was that people initially don’t want to watch the entire game, they want to “experience the moment” and share that moment with others. It was the purpose of the Vault to allow people to experience the moment.

Although he was talking about entertainment consumers, I thought that this was an interesting way to view all types of historical research. Researchers seldom want everything in a collection or a book, but those “moments” that help them prove their points, support their thesis or just inform themselves. This seems to me to be the essence of quantum archives, to reduce archival material to a collection of “moments” that can be used, shared, and re-used both in ways that we define–the pre-defined “great shots”–and the unexpected ways that result from users making their own moment out of a larger whole.

I’d like to coin a new phrase that I think I’ll add to the next version of the Quickstart Guide. It is “Deliver the Moment.” It simply means that we can manage our content according to traditional principles, but always seek to deliver that content in ways that resonate with our users.

I don’t know how scalable this idea is in terms of delivering real-life archival access. The NCAA Vault, for now, focuses on just one sport (Men’s basketball), in a very short time frame (10 years), and over a very limited scope (the last three rounds of the annual tournament). Given these limited parameters it is relatively easy to craft a satisfying user experience based on the principle of delivering the moment. There are plans to add more sports and a greater time span. I’m rooting for them.

Ten years into the twenty-first century we are beginning to see a movement among archivists to move forward into the twentieth century. All this really means is that Archivists are beginning to understand the balance between being and becoming. The idea that constant change, balanced and tempered by a consistent theoretical foundation, might just be the roadmap for the profession is slowly permeating the ethos of more “modern” or forward thinking archivists.

Howard Besser says that most new technology is used at first to mimic the old ways in a new form. “The conceptual steps [of technology development] typically include first trying to replicate core activities that functioned in the analog environment.”  So it stands to reason then that before we could invent new forms of access we had to re-invent the paper finding aid in the form of EAD.

However, it seemed that in doing so we raised the finding aid, rather than access to the material itself, to the level of an objective and actually prevented archivists from providing good service in the Internet environment. Before the age of computers and digitization,  there really was no point in providing highly granular content information to users. You still had to come to the repository and interact with the content in ways that did not disrupt the physical order of the boxes and folders. This filing system approach, while efficient and effective in its time and place, was a barrier to use. Everyone understood this, but no one had any real idea of how to do it better or differently. Thus, archivists became the interpreters of collections, a kind of human finding aid.

As with any bureaucratic system (and I mean this in the most positive possible sense), once it was devised, a class of apparatchiks grew up to tend the system and became vested in its perpetuation. The essential conservative nature of archives also contributed to the idea that the finding aid was sacred and that the Archivist as gatekeeper was the best possible way to provide service. I’ll wager that this access method was never satisfactory to the general user, but then, the only “serious” researchers were supposed to use archives anyway.

I don’t want to say that the parents of EAD didn’t do creative work. But they were working within a context of thinking from which they were unable to break free. It would have been surprising if they had, and if they did, perhaps no one would have listened to them anyway.

My own introduction to formal archival education came just before the introduction of EAD, and since I came from a research and teaching background, my idea of what an archives could or should be was based on user-centered ideas (although I didn’t know that term at the time).  I wanted to get to the “stuff” and draw my own conclusions, after all, that’s what I was there for. Most of my work when I was a classroom teacher centered on teaching with primary sources. It was always surprising what a group of students would see in a set of documents that I had never seen.

When I crossed over from researcher/teacher to service provider, I always believed that keeping the researcher as close to the content as possible was the most important thing an archivist could do. Give them the stuff and get out of the way!

Although it is obviously not quite as simple as that, we do have the capability to do this now. The Internet and digitization makes this all possible. But just how will we do it? By constantly trying new ways to present our archival material.

A couple of years ago, I gave a presentation at the SAA meeting in Chicago that I called “Where Have All the Binders Gone?” That introduced an idea that we should try to manage and provide access as close to the content as possible. This later evolved into the theory of quantum archives and is the inspiration for this blog.

With that statement from George Mitchell as a governing principle, I set out  in 1994 to process 1,000 linear feet of papers from the former Senate Majority Leader. They came in a truck like the ones they use to move households. I had never processed anything on the scale of this collection or of this complexity. It challenged me to think differently about processing and access.

George Mitchell web site, 1999

The first thing we did was to think about a productivity approach to processing, although in a very  paper-based way. We used a primitive, but effective, database system to manage the series and folders and, using the “report writer” function planned to create an electronic finding aid on the College’s gopher. (Anyone remember gopher?)

Well the web exploded onto the scene not too long after we started, and to our good fortune but not surprise, we found that with just a little adjustment to our report templates we could export HTML pages from our database. In the true fashion of reinventing the past in a new technology, we created a finding aid for the collection in a few short weeks that looked suspiciously like a paper finding aid in its construction and organization. We didn’t really know what to do with this new thing, but we knew we had to be there. So we

were on the web and we had pictures, and video! Even then we were exploring the potential of the web for organizing and reorganizing information. We  had a photograph “database” that was really just a categorized alphabetical list of digitized photos. We believed in searching and indexing, but didn’t have the tools in place to be able to do it, so we faked it. Similarly, the “menu” system on the left side of the finding aid is not dynamically generated, but is a set of images hard-coded into every page. We could imagine what we wanted to do, but didn’t have the tools or the expertise to do it.

Somewhat to  my astonishment, more than 10 years later, this finding tool is still available on the web as  part of a larger project to document the former Senator’s career. Take a minute to visit the George J. Mitchell Papers at

Bowdoin College for a look at the past envisioned as the future. Good enough for its time, and a beginning of understanding the power of this new thing called the World Wide Web. It is also a story of attempting, but not completely succeeding, to think out of the box. Even though many of the elements of what would become quantum archives were there for us, we just didn’t have enough experience to see it then.

p.s. Another round of thanks to Eliot Wilczek and Calley Gurley who embraced and supported the experiment. Both of them went on to careers in archives in other institutions. You were wonderful people to work with.

When we were first developing a productivity-based  processing workflow system for the Digital Collections and Archives at Tufts University, we had a whiteboard on which we wrote motivational phrases that reminded us of the things that were important for us to remember. These guiding principles were later codified into what we called the “Quickstart Guide to Becoming a Professional Archivist.“   It had two sections, one on archival principles and one on attitudes about processing. We used the Quickstart Guide as a introductory and training tool for new staff members.

The Guide introduced concepts like “lumpers vs. splitters” and “ruthless efficiency and dogged persistence.” as ideas related to archival processing as well as asking more philosophical questions about the role of the archivist in creating knowledge.

Back then the Quickstart Guide was mostly focused on processing paper records. As time went on and I began to use the Quickstart Guide as a teaching tool, I realized that in the born digital age, processing had changed significantly and that the old Guide was a bit out of touch. For example, the original Guide emphasized that good archival description proceeded from the General to the Specific and moved down that continuum as time and resources allowed. Quantum Archival theory turns that idea on its head, and says that good archival description focuses on specifics first and moves to generalities as time allows.

So I went back and revised it for the digital world. The result is the Quick Start Guide 2.1.

The Quick Start Guide, 2.1

The key change was to emphasize that “management is not access.” That is, the way we manage our collections is not necessarily (or even desirably) the way we want users to access our collections. The ability to separate management from access is one of the key values of digitized and born digital archival content.

The Quick Start Guide remains a central statement of what I consider to be “good” archival attitudes. It is the first thing I teach in my classes.

In 2004, the NSF released the “Report of Blue-Ribbon Advisory Panel on Cyberinfrastructure” that outlined an ambitious program to provide for scientists and science scholars a network of support that went beyond mere bandwidth and computers that would encourage and enable discovery, collaboration, and progress in scientific inquiry.

Not to be outdone, two years later the American Council of Learned Societies’ Commission on Cyberinfrastructure produced, “Our Cultural Commonwealth,” the Humanists’s perspective on supporting scholarship in the digital age.

Since then there have been numerous initiatives relating to cyberinfrastructure in both Higher and K-12 Education, Science, and Humanities scholarship. Yet, to my knowledge nothing has been written concerning the impact or use of cyberinfrastructure on archives or the work that archivists do.

Cyberinfrastructure is “more than a tangible network and means of storage in digitized form, and it is not only discipline-specific software applications and project-specific data collections. It is also the more intangible layer of expertise and the best practices, standards, tools, collections and collaborative environments (italics added) that can be broadly shared across communities of inquiry.” (From “Our Cultural Commonwealth”)

At the heart of the cyberinfrastructure vision is the development of a cultural community that supports peer-to-peer collaboration and new modes of education based upon broad and open access to leadership computing; data and information resources; online instruments and observatories; and visualization and collaboration services. Cyberinfrastructure enables distributed knowledge communities that collaborate and communicate across disciplines, distances and cultures. These research and education communities extend beyond traditional brick-and-mortar facilities, becoming virtual organizations that transcend geographic and institutional boundaries. (from the NSF’s “Cyberinfrastructure Vision for the 21st Century” 2007)

In short, cyberinfrastructure is what underlies the modern academic world of collaborative, interdisciplinary research, teaching, and learning. It is the network of associated technology, middleware, and visualization tools and services that enables interdisciplinary and inter-institutional collaboration and supports the development of innovative teaching and research.

Thinking of it this way, I think an important question we must ask ourselves is how can we integrate primary resources under our stewardship into the cyberinfrastructure of our institutions, our regions, and the world? Alternatively, how do we, as archivists, document and manage digital content that lives everywhere and nowhere at the same time?

Again, I’ll come back to a recurring theme, we need to make it possible to de-contextualize our collection objects so that they can be re-contextualized by scholars or anyone who has a use for them. This is not so different from traditional research, as the research function is all about creating new knowledge from primary resources (what “primary” may mean is another topic altogether). Secondly, we need to project those objects to the places where people are. In an academic setting, that means things like courseware tools or other places where students are encountering the building blocks of their work.

This truism is something we hear about all the time. Recently, I came across an actual example.

We were meeting with the University’s Office of Communications staff, showing them our new “quantum-based” discovery tool for archival content, including thousands of current and historical images of the campus produced by the Office of Communications.  We took great pains in developing this tool to be responsible archivists and present ALL relevant content, including information about content that was not yet digitized.

At the end of the presentation, they had very positive responses to the discovery tool, but wanted to know why in the world we would show them things they couldn’t see or use immediately over the web. For them, working on deadlines and needing instant access to content, the idea of even being bothered to sift through references to analog content was a waste of time.

And they are right, of course. Most, but not all researchers want

FACTS display with "show non-digital" option

what they want, and they want it now. So we changed our interface a bit. Now when you do a search, you are presented first ONLY with content that is available in digital form. And if you want, you can click a box to include non-digital content. We don’t yet have a way to see how many times that box is clicked. I’m sure it would be interesting.

A colleague of mine in the Art Gallery and I have been working on a project to bring to light about 15,000 images taken by the Soviet news photographer Semyon Fridlyand, who was active in middle part of the 20th century. The task of cataloging these images in the traditional way, that is by professionals, was of course very expensive. Since the collection has tremendous breadth, spanning multiple decades and subjects as diverse as war documentary photography, architecture, farming and agricultural technology, and ethnic dress and customs, it seemed impossible that we could assemble the cataloging expertise to do the collection justice. Faced with the choice of digitizing the photos or cataloging the photos, we chose to digitize them with only minimal metadata–not even a title in most cases, just an identification number.

The images are now available in a discovery system

Collection Discovery Tool

that was developed by a couple of first-rate programmers in the DU Library Systems office. The next step is to develop what we are calling a distributed metadata creation system that will make it possible to open cataloging up to nonprofessional volunteer experts. Inspired by the  success of the Steve project, we thought that it was possible to create valuable and valid metadata for our photographs from volunteers. However, we want to take the idea one step beyond simple tagging and develop a system that will create structured metadata–that is metadata that is relevant to particular concepts and “fields” such as “personal name,”  or “geographic location.” This way we can use the metadata to populate faceted search and discovery tools.

We are hoping to make contact with people in the former Soviet Union who were the subjects of these photos and engage them in the process of identifying them. In some cases, these photos are the only surviving visual record of this era for these people and places. Suggested metadata will be reviewed by a “panel of experts” who will work within the application to vet and approve metadata and add it to the “official” metadata for the object.

More postings on this exciting project to come.