Established in 2017, the Archives Unleashed Project began with the goal of unlocking the potential of web archives for scholarly research.

Over the past six years, we’ve engaged with librarians, archivists, technologists, and researchers from a variety of disciplines, including digital humanities, communication and media studies, journalism, computer science, information management, the social sciences and beyond. Thanks to you all!

When we take a broad look at the phases of the Archives Unleashed Project, we’ve seen the shift from a stand-alone Mellon-funded project from 2017–2020 to a collaborative integration with the Internet Archive beginning in 2020. Through those two phases, we’ve evolved our tools and approaches, which address the technical complexities and burdens researchers face when working with web archives.

As the project came to a close at the end of June 2023, our team is looking back at our activities and celebrating the tools that we share (and leave) with the broader community.

Phase I (2017–2020)

The underlying goal of our work has been to lower the barriers and burdens of accessing and using web archives, and we have approached this work through two main activities: Tool Building and Community Engagement.

Building Scalable Analytical Tools

The W/ARC file format is messy and complex and can certainly be intimidating to new scholars. We also appreciate that web archive collections are typically difficult to work with due to their scale. In fact, even when collections are packaged into a more familiar format, such as a CSV, many of them often can’t be used with proprietary tools and software because the data often exceeds the limits of spreadsheet programs. Not every scholar has the means, support, or time to learn extensive computational analysis skills needed to explore this kind of data.

To address this problem, our project has created scalable tools, methodologies, and approaches to help address some of the technical complexities and burdens researchers often face with web archives data.

During the first phase of the project (2017–2020), the team developed open-source tools as well as methodological approaches for conducting scalable analysis. Together these tools presented new opportunities for researchers to investigate and analyze web archives and included:

Archives Unleashed Toolkit (2017 — present)

The Archives Unleashed Toolkit is an open-source platform for exploring and analyzing web archives. The Toolkit provides a set of approaches that allow users to investigate large-scale web archives by creating derivative datasets that can be used to fully explore and extract meaningful insights using Apache Spark.

Extensive documentation, formatted as a cook-book style approach, provides recipes for common use cases so users can confidently and successfully build customized scripts for common analytical tasks to explore text, network, and binary data.

Archives Unleashed Cloud (2018–2020)

The Archives Unleashed Cloud was designed as a one-stop, web-based portal for scholars to ingest their Archive-It collections and execute a number of analyses with the click of a mouse.

From a technical perspective, the Cloud was a Rails application that had a web-based front end, with the Toolkit and a web archive data transfer API called “WASAPI” working in the background. The Cloud provided a connection between a user’s Archive-It account via WASAPI, which provided a pipeline to their Archive-It collections. The Toolkit was used to process web archive collections and generate easy-to-use derivative datasets for further exploration.

The Archives Unleashed Cloud was officially sunsetted on 30 June 2021 and provided the theoretical foundation for work done in Phase II, which eventually led to the ARCH (Archives Research Compute Hub) iteration of 2023. Keep reading to learn more about ARCH!

Over its lifespan, the Archives Unleashed Cloud analyzed just under a petabyte of data and was used by individuals from 59 unique institutions across ten countries.

Warclight (2018–2023)

Warclight was developed as a framework to help with the search and discovery of web archives. As a Project Blacklight-based Rails engine, it supported the discovery of web archives held in the W/ARC formats and allowed faceted full-text search, record view, and other advanced discovery options. Warclight was specifically designed to work with web archive data indexed via the UK Web Archive’s webarchive-discovery project.

As part of the WALK (Web Archive for Longitudinal Knowledge) project, Warclight brought together the archival holdings of a half-dozen Canadian libraries by providing the federated search and access to research derivatives of Canadian web archival collections.

Warclight instances were created for the University of Alberta, University of Toronto, University of Winnipeg, University of Victoria, Simon Fraser University, and Dalhousie University, which indexed their W/ARCs from their Archive-It accounts and provided an interface to interact with this data through a search and discovery layer.

Datathons: Community Building

On the community engagement front, our team hosted a series of datathons for skills development and to foster a sense of belonging and support through collaborative projects. Borrowing from the hacking genre of events often found within the tech industry, as well as an earlier series of events held in 2016 and 2017, Archives Unleashed datathons provided an immersive and uninterrupted period of time for participants to work collaboratively on projects and gain hands-on experience working with web archive data. The datathon series cultivated community formation and empowered scholars to build confidence and the skills needed to work with web archives.

These datathons saw the participation of 73 attendees who were representative of a wide range of professional roles, including librarians, archivists, technologists, digital humanities scholars, graduate/post-graduate researchers, and computer scientists.

Mellon-funded Datathons include:

• University of Toronto (April 2018): Co-hosted with the University of Toronto Libraries.
• Simon Fraser University (November 2018): Co-hosted with Simon Fraser University Library and KEY (SFU’s Big Data initiative).
• George Washington University (March 2019): Co-hosted with George Washington University Library.
• Columbia University (March 2020): Co-hosted with Columbia University Library and the Ivy Plus Libraries Confederation. Due to COVID-19, we transitioned this to a virtual event.

To visit team projects completed, please visit the datathon event pages.

Phase II (2020–2023)

Supported by a second Mellon Foundation grant, the Archives Unleashed Project collaborated closely with the Internet Archive in a highly integrated partnership aimed to enhance the accessibility and usability of web archives for research purposes.

This second phase of Archives Unleashed (2020–2023) saw the maturing of tools and community engagement activities to provide a more robust and interconnected environment for engaging with web archives and more focused and intensive support for research. Much of this was done in collaboration with the Internet Archive’s Archiving and Data Services team.

ARCH (Archives Research Compute Hub)

In partnership with the Internet Archive, our goal was to design a sustainable tool and service that would support web archival research at scale.

On June 26, 2023, we collectively achieved this objective by launching the Archives Research Compute Hub (ARCH), a research and education service that helps users easily build, access, analyze, publish, and preserve web archive datasets at scale.

The road to ARCH was reached by blending and leaning into the expertise of project members throughout the design and development process, which included several components:

Building a Computational Environment. Building what would become the ARCH platform required both enhancing and reimagining the code base to ensure effective and efficient resource allocation and scalability of processing and transformation of W/ARC data. Engineers at the Internet Archive provided leadership in creating the Sparkling data processing library, implementing a new job and queue management system, and building a distributed, high-performance computing cluster. And we can’t forget the building of physical infrastructure, i.e. mounting racks to hold the servers that ARCH now calls home!

Wireframe and Prototyping. Responsive design strategy practices were applied to finesse wireframes and prototypes to ensure the interface aligned with the feedback and requirements outlined by stakeholder groups.

Iterative UX/UI Testing. Five scaffolded iterative rounds of User Experience (UX) and User Interface (UI) testing were conducted to test the efficiency and design of ARCH prototypes. Interviews and surveys were distributed to stakeholder groups, with particular attention on recruiting participants representative of a broad spectrum of skill levels, institutional types, geography, and collection sizes. The collection of qualitative and quantitative feedback evaluated the stability, robustness, and scalability of the ARCH platform. Virtual feedback sessions were also held with 2021–2022 Cohort participants to learn about their experience in piloting the platform.

Documentation. ARCH documentation was created and organized to reflect the wide range of skills of its users. Collaborative writing between Archives Unleashed and web archivists at the Internet Archive has resulted in user documentation (via the ARCH Help Center) that provides onboarding, an overview of the platform’s features, documented use cases of web archives in research, and guidance for using ARCH derivatives with other analysis tools and methods.

Stewardship and Sustainability. In building a platform and service that seamlessly integrates with Internet Archive’s infrastructure, as a project team, we’ve ensured reliable, scalable, and sustainable stewardship.

ARCH Features:

• Intuitive, interactive, user-friendly stand-alone web-based app
• Build custom collections with filtering capabilities
• Transform web archive collections into research objects by generating over a dozen dataset types (e.g., full text, images, hyperlink network graphs, etc.) into widely compatible CSV files
• Download generated datasets directly in-browser or via API
• Research-ready datasets can be used with computational environments and tools (e.g. Jupyter Notebooks, Gephi, Voyant)
• Direct connection between datasets and Google Colab Notebook to conduct initial exploratory analysis
• Review content collection through in-browser visualizations and data previews
• Publish datasets in line with best practices in reproducible research to the Internet Archive. All datasets will be preserved in perpetuity
• Browse extensive documentation to support your use of ARCH, guidance on pairing datasets with computational tools, and sharing research examples and use cases

ARCH provides approachable and powerful opportunities to investigate web archives at scale. Researchers can now easily navigate, search, and make sense of vast collections of web data without requiring extensive technical expertise or specialized tools. This increases accessibility and opens doors for a wider range of researchers to delve into the wealth of information captured in web archives, leading to enhanced scholarly inquiry.

As the Archives Unleashed Project comes to an end, the Internet Archive and Archiving and Data Services team will be stewarding the ARCH platform and service.

To learn more about ARCH, please reach out via the following form.

ARCH: https://webservices.archive.org/pages/arch

ARCH Help Center: https://arch-webservices.zendesk.com/hc/en-us

Archives Unleashed Toolkit (AUT) 1.2.0

As noted earlier, the Archives Unleashed Toolkit is an open-source platform for analyzing web archives using Apache Spark and makes use of Sparkling for parsing W/ARC records. The toolkit provides powerful tools for analytics and data processing.

In November 2022, the Archives Unleashed Toolkit 1.2.0 version was released, which provides a stable and open-source solution for analyzing web archive collections at scale, and marks a major milestone for the Archives Unleashed Project! As of this release, the Toolkit:

• Provides full support of DataFrames (DF) and Resilient Distributed Dataset (RDD) to allow for flexibility in processing and displaying structured, semi-structured, or unstructured data.
• Incorporates Python and Scala implementations to accommodate the programming languages most familiar to scholars working with web archives.
• Presents unlimited combinations in generating standardized datasets that can be used as scholarly objects to explore five functional analysis categories: collection, link, binary, text file analysis, and text extraction.
• Allows users to utilize spark-submit to create over 20 different derivatives of web archive collections.
• Adopts the Sparkling library, which was developed as a data processing library for the Internet Archive, to improve the speed, efficiency, and reliability of parsing of W/ARC records.
• Enhanced user documentation that illustrates dozens of text, network, and binary analytical examples.

The Archives Unleashed Toolkit is open-source and available to anyone interested in exploring and analyzing web archives.

GitHub: https://github.com/archivesunleashed/aut

Documentation: https://aut.docs.archivesunleashed.org/

Understanding that not everyone has access to web archive collections, we’ve collaborated with several institutions to process over two dozen web archive collections through the Toolkit and ARCH. This means that as a scholar, you can access read-to-use derivative datasets and start your research journey. Datasets are available through the Web Archives for Historical Research Group via Zenodo + Borealis.

Computational Notebooks (2019–2023)

Computational notebooks come in all shapes and sizes. Many are familiar with Juptyer and Google Colab. Notebooks provide a space for creating, editing, and sharing documents that can contain live code, equations, visualizations, and text (Driscol).

Notebooks provide a more accessible entry point for researchers interested in exploring and interacting with web archive collections (Ruest). Once a researcher generates (or has access to) derivative datasets, many often ask, “How do I start using these derivatives?” Our goal with the notebooks was to help provide guidance to scholars on what investigating web archive data could look like through examples.

We’ve created both Python-based and Google Colab notebooks which provide a templated “madlib” approach, allowing scholars to work within browsers, filling in the blanks (where appropriate) to apply customized example methods and techniques while investigating web archive collections.

The development of computational notebooks, led by co-investigator Nick Ruest, provided some first steps in investigating these derivations of web archives and entry points or examples of types of analysis that can be conducted.

Notebooks have been integrated into the ARCH platform, providing a seamless transition from collection analysis and generated datasets to using a dataset directly within a Google Colab Notebook.

The Archives Unleashed Project also offers standalone Jupyter notebooks that are publicly available and can be used in conjunction with generated datasets from the Archives Unleashed Toolkit.

Archives Unleashed Cohort Program

As a project, we’ve been committed to community building and engagement because we know projects can’t exist in silos, and a sense of belonging and connection is critical for tools and services to grow and thrive.

Alongside our tool-building efforts, our team expanded on the foundations of our community engagement activities by supporting web archives research in a more intensive way than our datathon series allowed for and was realized through the development of the Archives Unleashed Cohort Program (2021–22 and 2022–23).

The Cohort Program was designed to facilitate intensive research engagement with web archives, foster a community of users, and cultivate a broad spectrum of research use cases across disciplines and fields.

Over two cycles, the program engaged with 50 interdisciplinary researchers from 20 institutions across nine countries. Over the course of each year-long collaboration, teams used web archives as a primary data source to investigate wide-ranging research topics and questions.

Throughout the program, participants were provided technical and academic mentorship to steward research projects through direct one-on-one consultation from Archives Unleashed, connections to field experts, and opportunities for peer-to-peer support.

The Cohort Program has, at a micro level, demonstrated the value in expanding opportunities to access, explore and engage with web archives, all of which are dependent on building connections and relationships.

• Access. Connecting researchers with (web archives) data uncovered opportunities to forge relationships with content creators and collection curators. In partnership with the Internet Archive and in collaboration with specific Archive-It partners, cohort teams were given read-access to web archive collections, which were then analyzed through the ARCH platform.
• Explore. Building community among cohort participants and fostering an environment that embraced skills-building and knowledge-sharing, users were empowered to experiment with new tools and methodologies for investigating web archives.
• Engage. The cohort projects have enhanced the visibility of web archive collections and facilitated meaningful connections with diverse disciplinary communities. As a result, these projects serve as exemplary use cases and showcase the high-quality research potential inherent in working with web archives. Through shared experiences and research insights, cohort participants have highlighted the possibilities, opportunities, and inspiration for those seeking to dive into the world of web archive research.

The scholarly contributions and accomplishments of these ten research teams have been tremendous! If we look at traditional scholarly content, teams have presented at conferences and meetings, led workshops, and written peer-reviewed articles and blog posts. They’ve also published research findings and engaged with the web archive field through creative and unique ways, such as developing computational notebooks for research and teaching, developing tools, contributing a Spanish Wikipedia entry for web archiving, producing a zine publication, and initiating Solr Wayback instances.

We celebrate these researchers and their contributions to the web archiving space. The Cohort Projects Summary Report provides highlights use cases of web archives research for several subjects and disciplines, including crisis communication, health mis/disinformation, commenting systems, digital activism, health mis/dis information, Latin American women’s rights movements, digital labour and motherhood, reconciliation processes, and the COVID-19 Pandemic.

An exclusive summary of the program and research project can also be found in the Archives Unleashed blog post “Celebrating Research Opportunities with Web Archive through Archives Unleashed Cohort Projects.”

To Infinity and Beyond

The Archives Unleashed Project has spent the past six years building innovative tools and fostering a vibrant community. As we come to the end of our project, we proudly unleash our roster of tools, the Archives Unleashed Toolkit, ARCH, and notebooks, alongside co-created resources and datasets to inspire and empower continued research collaborations and knowledge-sharing endeavours.

We are grateful for the support we’ve received and thrilled to have played a part in creating a community of scholars who are deeply passionate about unlocking the rich resources within web archives.

We are not a fan of goodbyes, so we hope to see you soon!

Acknowledgements

The work of the Archives Unleashed Project has been made possible by generous funding from the Mellon Foundation (2017–2023).

We’d also like to acknowledge the financial and in-kind support of Smart Start Labs, the University of Waterloo, York University, Archive-It, Social Sciences and Humanities Research Council of Canada, Compute Canada, and the Ministry of Research, Innovation, and Science.

We are incredibly grateful to our broader community of supporters, peers, and colleagues who have welcomed us into this space, championed our work, and been a source of inspiration.

The Archives Unleashed Project has engaged in highly integrated collaborations throughout its two phases and acknowledges the efforts and contributions of its project members, collaborators, and advisory board members!

Project Team

• Ian Milligan, Primary Investigator, University of Waterloo (Phase I & II)
• Nick Ruest, Co-Investigator, York University (Phase I & II)
• Jimmy Lin, Co-Investigator, University of Waterloo (Phase I & II)
• Jefferson Bailey, Co-Investigator, Internet Archive & Director, Archiving and Data Services, Internet Archive (Phase II)
• Thomas Padilla, Deputy Director, Archiving and Data Services, Internet Archive (Phase II)
• Helge Holzmann, Senior Data Engineer, Archiving and Data Services, Internet Archive (Phase II)
• Samantha Fritz, Project Manager, Archives Unleashed (Phase I & II)
• Kody Willis, Product Operations Manager, Archiving and Data Services, Internet Archive (Phase II)
• Karl-Rainer Blumenthal, Web Archivist, Archiving and Data Services, Internet Archive (Phase II)
• Alex Dempsey, Senior Engineering Manager, Archiving and Data Services, Internet Archive (Phase II)
• Peggy Lee (2021–2022)

Collaborators

• Sarah McTavish, Department of History, University of Waterloo (2018–2021)
• Rebecca MacAlpine, Department of History, University of Waterloo (2019–2020)
• Tobi Adewoye, David R. Cheriton School of Computer Science, University of Waterloo (2019–2020)
• Xiao Han, David R. Cheriton School of Computer Science, University of Waterloo (2019–2020)
• Gursimran Singh, David R. Cheriton School of Computer Science, University of Waterloo (2019–2020)
• Hsiu-Wei Yang, David R. Cheriton School of Computer Science, University of Waterloo (2018–2019)
• Linqing Liu, David R. Cheriton School of Computer Science, University of Waterloo (2018–2019)
• Borui Lin, David R. Cheriton School of Computer Science, University of Waterloo (2018)
• Jeremy Wiebe, Department of History, University of Waterloo (2018–2019)
• Ryan Deschamps, Department of History, University of Waterloo (2017–2019)

Archives Unleashed Advisory Board

2017–2020

• Jefferson Bailey
• Matthew Weber
• Nathalie Casemajor
• Nicholas Worby
• Robert H. McDonald

2020–2023

• Matthew Weber
• Michele Weigle
• Robert H. McDonald
• Jane Winters
• Sylvain Bélanger
• Nicholas Taylor

Additional Reading

Archives Research Comput Hub (ARCH)

• Holzmann, H., Ruest, N., Bailey, J., Dempsey, A., Fritz, S., Lee, P., and Ian Milligan. “ABCDEF: the 6 key features behind scalable, multi-tenant web archive processing with ARCH: archive, big data, concurrent, distributed, efficient, flexible.” In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries (JCDL ‘22). Association for Computing Machinery, New York, NY, USA, 1–11 (2022). [link] [preprint]

Archives Unleashed Toolkit

• Archives Unleashed Toolkit User Documentation
• Nick Ruest, Samantha Fritz, and Ian Milligan. “Creating order from the mess: web archive derivative datasets and notebooks.” Archives and Records, 2022. https://doi.org/10.1080/23257962.2022.2100336
• Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. Proceedings of the 2020 IEEE/ACM Joint Conference on Digital Libraries (JCDL 2020), Wuhan, China.
• Nick Ruest. The Archives Unleashed Toolkit as a Finding Aid Utility. Medium.

Notebooks

• Ruest, N., Fritz, S., and Milligan, I. (2022). Creating Order from the Mess: Web Archive Derivative Datasets and Notebooks. Archives and Records (CJSA). https://doi.org/10.1080/23257962.2022.2100336

Datathons/Community

• Samantha Fritz, Ian Milligan, Nick Ruest, and Jimmy Lin. “Fostering Community Engagement through Datathon Events: The Archives Unleashed Experience”. Digital Humanities Quarterly, Vol. 15, №1, 20201.
• Samantha Fritz, Ian Milligan, Nick Ruest, and Jimmy Lin. “Building community at distance: a datathon during COVID-19.” Digital Library Perspectives, 2020.
• Ian Milligan, Nathalie Casemajor, Samantha Fritz, Jimmy Lin, Nick Ruest, Matthew S. Weber, and Nicholas Worby. “Building Community and Tools for Analyzing Web Archives through Datathons.” Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019).

Archives Unleashed Cloud

• Nick Ruest, Samantha Fritz, Ryan Deschamps, Jimmy Lin, and Ian Milligan. “From archive to analysis: accessing web archives at scale through a cloud-based interface.” International Journal of Digital Humanities, 2021.
• Ryan Deschamps, Samantha Fritz, Jimmy Lin, Ian Milligan, and Nick Ruest. “The Cost of a WARC : Analyzing Web Archives in the Cloud.” Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019).

References

Mike Driscoll. “Jupyter Notebook: An Introduction.” https://realpython.com/jupyter-notebook-introduction/

Nick Ruest, Samantha Fritz & Ian Milligan (2022) Creating order from the mess: web archive derivative datasets and notebooks, Archives and Records, 43:3, 316–331, DOI: 10.1080/23257962.2022.2100336

Unleashing Web Archives: A Final Letter to Our Community was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

The program has seen two cycles of year-long intensive collaborations to support research with web archives as a primary data source and scholarly objects.

Research teams were provided with technical and academic mentorship to steward projects through direct one-on-one consultation from the Archives Unleashed team, members of the Archiving and Data Services team, connections to field experts, and opportunities for peer-to-peer support.

The program also saw teams publish their research through a number of scholarly avenues — from journal publications to conferences to community engagement and advocacy work. As these projects offer high-quality use cases for working with web archive data, we provide a summary of the cohort program and feature the research of these interdisciplinary and international project teams.

View their project profiles:

• 2021–2022 Cohorts: https://archivesunleashed.org/cohorts2021-2022/
• 2022–2023 Cohorts: https://archivesunleashed.org/cohorts2022-2023/

Project Summaries Report: https://bit.ly/AUCohortProjects

About Cohort Researchers

Map featuring the geographic representation of cohort teams

From 2021–2023, the program hosted ten research projects involving an international and interdisciplinary group of scholars. Their projects spanned a broad range of topics, from perspectives, experiences, and responses to the COVID-19 Pandemic, to the role of new media in feminist-based landscapes and queer identity and discourse formation in online spaces to cultural practices and reconciliation processes.

Word cloud of self-expressed topics and themes studied by cohort teams

Our research teams have self-identified over two dozen topics and themes their projects explored and studied, including :

#COVID19 #Pandemic #Multilingualism #CrisisCommunication #RiskCommunication #BigData #CommentingSystems #OnlineComments #NewsWebsites #DigitalActivism #LatinAmericanFeminism #CyberFeminism #CounterArchives #DigitalLabor #Motherhood #Blogging #Feminism #Domesticity #DisInformation #MisInformation #CulturalPractices #ReconciliationProcesses #HistoricalRedress #HumanRightsDefense

Methods + Tools

When collaborating with researchers interested in investigating web archive collections, we’ve found one of the first questions asked is what types of tools or methods should be used to explore web archival data.

The answer: it really depends on your research question(s) or what you are asking of the data. As cohort participants discovered, their questions often have to be reimagined based on the content of a web archive collection.

That said, all teams began their research process by using the ARCH (Archives Research Compute Hub) platform. ARCH transforms web archival collections, specifically the WARC files, into accessible scholarly objects that can be used with a myriad of tools and methods for further research. These include domain frequency statistics, hyperlink network graphs, extracted full-text, and metadata about binary objects within a collection.

After generating datasets with ARCH, cohort teams used mixed-methods to conduct text and network analysis. Interestingly to note, while several teams engaged in similar analysis methods, their approaches to, for instance, topic modelling, had a wide range of techniques and tools applied!

Visual summary of analysis methods and tools used by cohort researchers

Text Analysis

Many teams began their investigations by exploring the text of their web archive collections. Since ARCH provides a dataset encompassing the full text of a web archive, it became an invaluable starting place to understand the types of content present. Cohorts used a wide range of methods, techniques, and tools and faced the challenge of dealing with scale.

Mixing both distant and close reading was crucial given that even smaller collections have thousands, if not hundreds of thousands, of lines of content, which makes it impossible to conduct manual coding or reading of all the content.

We saw many topic modelling and classification techniques used to engage with the data from a macro level, as well as the use of programming languages and computational notebooks to investigate recurring themes and patterns to draw out stories from the data.

Network Analysis

Looking at the interconnectivity within a web archive, teams used network analysis to understand how actors, bodies, and groups within web archive collections were linked, how information flowed, and what entities were central or influential to conversations.

Image + Binary Analysis

A few of our cohorts were interested in using information from binary files found within web archive collections, such as images, PDFs, videos, spreadsheets, word documents, and the like — essentially any non-textual-based objects.

Web Archive Collections

In partnership with members of the Internet Archive’s Archiving and Data Services team, cohorts were granted access to conduct analysis on web archive collections, many of which were curated by Archive-It Partners.

Establishing connections between cohorts and collection curators was important for several reasons:

1. Highlighting the use of institutional web archive collections in scholarly inquiry & research;
2. Provides a way for cohorts to gain contextualized knowledge about a collection — i.e. what scoping rules were applied for inclusion/exclusion criteria; and
3. Facilitated collaborations and open opportunities to contribute content to web archives of underrepresented communities and voices

Collectively, cohort teams have explored 44 web archive collections from 24 Archive-It Partners and collection curators. We thank all curators and institutions for their care in preserving cultural heritage through these web archive collections. The cohorts used collections by the following institutions and individuals:

• Archive-It, Archiving and Data Services team
• Archive Team
• Brigham Young University
• Brock University
• Columbia University Libraries
• Duke University
• International Internet Preservation Consortium
• Library and Archives Canada
• Mark Graham
• McGill University
• National Museum of Women in the Arts
• New York University
• Nick Ruest
• San Francisco Public Library
• San Jose State University, School of Information
• Schlesinger Library
• Smith College
• Temple University Special Collections
• University of Manitoba
• University of Michigan Library
• University of Michigan School of Information
• University of Saskatchewan
• University of Texas at Austin Libraries
• University of Texas at San Antonio Libraries Special Collections

Celebrating Project Accomplishments

It has been a pleasure to collaborate with these teams and witness their growth, successes, and boundless creativity in meeting challenges!

The Archives Unleashed Cohort roster includes ten projects by fifty researchers from twenty institutions.

In collaboration with cohort teams, we’ve created a project summary document that highlights the use cases of working with web archival data and shares experiences in meeting challenges that big data presents.

Cohort Project Summaries: https://bit.ly/AUCohortProjects

Here we share the description for each project and celebrate their achievements!

Crisis Communication in the Niagara Region during the COVID-19 Pandemic by Tim Ribaric, David Sharron, Cal Murgu, Karen Louise Smith, and Duncan Koerber

Using web archives collected by Brock University, this project will examine how organizations in the Niagara region have responded to government COVID-19 mandates. Analysis will focus on investigating three types of entities: local government, non-profit organizations, and major private entities. Findings from this research aim to inform future crisis communication organizational planning, specifically at the local and municipal level. The project will also create several open computational notebooks to support teaching, learning, and research.

• Project Website and blog: https://brockdsl.github.io/archives_unleashed/
• Journal article currently under peer review with the Canadian Journal of Communication
• Paper presented at International Association for Media and Communication Research 2023 conference
• Press Coverage: “Brock examining Niagara’s crisis communications during COVID-19”, St. Catharines Standard

AWAC2 — Analysing Web Archives of the COVID Crisis through the IIPC Novel Coronavirus dataset by Valérie Schafer, Karin De Wild, Frédéric Clavert, Niels Brügger, Susan Aasman, Sophie Gebeil, Joshgun Sirajzade

Investigating transnational events through web archive collections, the AWAC2 team will focus on a distant reading of the IIPC COVID-19 web archival collection to understand actors, content types and interconnectivity throughout it.

• The AWAC2 team has presented results at 10+ seminars, international conferences, and meetings, including the WARCnet final conference, IIPC (2022 & 2023), and RESAW
• “Studying Women and the COVID-19 Crisis through the IIPC Coronavirus Collection” IIPC, Netpreserve Blog
• “Analysing Web Archives of the Covid-19 Crisis through the IIPC collaborative collection: early findings and further research question” IIPC, Netpreserve Blog
• Two published articles use the AWAC2 project as a case study within a broader approach to web archives’ challenges
• 1 article under review
• Drafting chapter for publication in 2023 related to work on Women, Gender and COVID-19

Mapping and tracking the development of online commenting systems on news websites between 1996–2021 by Anne Helmond, Johannes Paßmann, Robert Jansma, Luca Hammer, and Lisa Gerzen

This project aims to reconstruct a history of online commenting by examining the role of commenting technologies in the popularisation of commenting practices. It will do so by examining the distribution and evolution of commenting technologies on the top 25 Dutch, German, and world news websites from 1996–2021 to understand how they have shaped the practices of users. This will allow them to explore the interplay between technologies and practices of the past and to investigate histories of natively-born technologies and practices.

• Two presentations at the Association of Internet Researchers Conference, 2022, Dublin, Ireland [Slides].
• WARCnet closing event, 2022, in Aarhus, Denmark [Slides].
• Internet Archive, 2021, online event [Slides].
• Internet Archive, 2022, Canada.
• Workshops on ‘online technography’ using web archives at the University of Siegen.
• Development of Jupyter Notebooks for internal use.
• Development of ‘the Technograph’ tool (in-progress), a visualization interface on the dataset to assist in the analysis.

Everything Old is New Again: A Comparative Analysis of Feminist Media Tactics between the 2nd- to 4th Waves by Shana MacDonald, Aynur Kadir, Brianna Wiens, Sid Heeg

Project members will explore web archive collections to conduct a comparative analysis of the history of feminist media practices across interdisciplinary multi-media sources. The team expects to produce a timeline of issue responses from different historical moments and map different feminist media practices over this timeline to determine overlaps. The project’s key outcome will be to recover earlier feminist media practices and contextualize them in the digital present.

• “Approaches to Archiving Feminist Memes.” Preserving Digital Born Media by Women: methods for decolonial & feminist futures (Panel). Film and Media Studies Association of Canada annual conference, May 2023
• “Activists Archiving the Internet: Social Justice Informed Approaches to Digitally Born Content.” Panel with Nick Ruest, Brianna Wiens, Shawn Walker, Mina Momeni. Shaking Up the Archive, Queen Margaret University, Edinburgh, June 23–25, 2023. Accepted February 2023.
• “Reconceptualizing Internet Archives: Feminist Memes as Repertoire” Canadian Association for Theatre Research, Halifax, June 9–12, 2023. Accepted February 2023.
• “From Placards to Memes: The Utopic Refusals of Feminist Media Techno-Imaginaries,” co-author Brianna Wiens, Feminist Encounters (Invited for special issue Sept 2023)

Viral health misinformation from Geocities to COVID-19 by Shawn Walker, Michael Simeone, Kristy Roschke, and Anna Muldoon

This project will examine and compare two case studies of health misinformation: HIV mis/disinformation circulating on Geocities in the mid-1990s to early 2000s with the role of official COVID-19 Dashboards in COVID mis/disinformation. This work contributes to our understanding of current and historical health misinformation as well as the connections between them, and will also garner insights into how historical narratives of health misinformation have been recycled and repurposed.

• Viral health misinformation: From GeoCities to COVID-19, presented at the Association of Internet Researchers, Dublin
• Paper under review

Latin American Women’s Rights Movements: Tracing Online Presence through Language, Time and Space by Sylvia Fernandez, Rosario Rogel-Salazar, Verónica Benítez-Pérez, Alan Colín-Arce, Abraham García-Monroy, Hejin Shin

In analyzing web archives related to human rights and feminist movements, the project will develop a historical analysis of the websites from women’s rights movements in Mexico and Latin America, particularly those focusing on eradicating femicides and gender violence. The team will also study how these movements relate to women’s rights movements globally, specifically by comparing language expression.

• Built Web Archive Collection, as part of Huellas Incómodas Project: https://idrhku.org/huellasincomodas/webarchive
• Spanish Wikipedia entry for web archiving https://es.wikipedia.org/wiki/Archivado_web
• DH Workshop hosted at the Autonomous University of the State of Mexico, 2023, organized with a grant from the Science Council of the State of Mexico https://idrhku.org/huellasincomodas/investigaciondigital
• (Forthcoming Article) Rosario Rogel-Salazar, Abraham García, Alan Colin-Arce, Verónica Benítez-Pérez. Preserving the memory of Latin American feminist movements on digital and web counterarchives
• Received additional $100,000 MXN grant from the Science Council of the State of Mexico under the program Research Funding for Women Scientists.

Historicizing Aughts-Era Mormon Mommy Blogging Media Landscapes by Emily Edwards, Robin Hershkowitz, and Lauren Andrikanich

In the era of post-blogging, or microblogging, this project will explore early aughts Mormon mommy blogging culture, mediations of marriage, mothering, and feminine domesticity to historicize this period in relation to contemporary manifestations and trends of mommy influencing on social media platforms.

This project will identify connections and transitions to new media ecologies and new iterations of racialized, gendered domestic ideologies that share historic genealogies to digital media practices and structures pioneered by Mormon mommy bloggers.

• Abstract accepted, “Digital Pioneers: Mormon Mommy Bloggers and Digital Domestics,” for a special issue of Internet Histories on “Gender and the Internet/Web History” forthcoming 2025
• Zine “Sliding Data: Feminist Methodological Pathways” accepted for publication in DIY Methods Low-Carbon Research Methods Initiative forthcoming 2023
• Submission as part of a panel on “Digital scholarship and the web: Exploring new sources and emerging research methods” for the Digital Library Federation Forum Conference 2023

Using Web Archives for Mapping the Use of Cultural Practices in Postconflict Societies and During Reconciliation Processes by Ricardo Velasco Trujillo and Luis Gomez

Employing computational methods such as contextual search, data mining, and web scraping, amongst others, this project aims at making an initial assessment and map out the use of cultural practices in different reconciliation processes and across several human rights organizations working actively in post-conflict societies.

• Digital publication in Cultural Ecologies of Memory (CEM) Platform:
• Interactive Project Results: http://www.culturalecologies.com/case/DataMining

Querying Queer Web Archives by Filipa Calado, Corey Clawson, Di Yoong, and Lisa Rhody

In studying queer online spaces, project investigators will explore investments in concepts like utopia, play, radicalism, normativity, religion, and conversion and how they affect queer identity and discourse formation over time.

• Yoong, D., Calado, F., & Clawson, C., (2023, May 3). Querying Queer Web Archives [Conference presentation]. IIPC WAC 2023, Hilversum, The Netherlands.

Web Archiving and the Saskatchewan COVID Archive: Expanding Coverage to Capture Social Media, Medical Misinformation, and Radicalization by Jim Clifford, Derek Cameron, Erika Dyck, Craig Harkema, Patrick Chasse, and Tim Hutchinson

Using web archives collected by the University of Saskatchewan, this project seeks to map and connect conversations and experiences of the COVID-19 pandemic with the province of Saskatchewan. The research team will develop timelines and knowledge trees to provide an opportunity to learn about the causal relationship between social media and this important shift in public health policy in Saskatchewan.

• Nazeem Muhajarine et al., “Capturing and Documenting the Wider Health Impacts of the COVID-19 Pandemic Through the Remember Rebuild Saskatchewan Initiative: Protocol for a Mixed Methods Interdisciplinary Project,” JMIR Research Protocols 12, no. 1 (June 6, 2023): e46643, https://doi.org/10.2196/46643.
• Derek Cameron, “Archiving Twitter During the Upheaval’” ActiveHistory.ca, https://activehistory.ca/2022/11/archiving-twitter-during-the-upheaval/

Conclusion

It has been a wonderful two years of deep collaboration with our Cohort teams. At the end of our collaborations, saying goodbye is always tough: but we’ve universally been so proud of the excellent work that we have seen. Our goal was to help support web archive-based research that would make disciplinary and societal impact… we hope you agree that we’ve been successful on that front!

Celebrating Research Opportunities with Web Archive through Archives Unleashed Cohort Projects was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

We are excited to announce the public availability of ARCH (Archives Research Compute Hub), a new research and education service that helps users easily build, access, and analyze digital collections computationally at scale. ARCH represents a combination of the Internet Archive’s experience supporting computational research for more than a decade by providing large-scale data to researchers and dataset-oriented service integrations like ARS (Archive-it Research Services) and a collaboration with the Archives Unleashed project of the University of Waterloo and York University. Development of ARCH was generously supported by the Mellon Foundation.

What does ARCH do?

ARCH helps users easily conduct and support computational research with digital collections at scale — e.g., text and data mining, data science, digital scholarship, machine learning, and more. Users can build custom research collections relevant to a wide range of subjects, generate and access research-ready datasets from collections, and analyze those datasets. In line with best practices in reproducibility, ARCH supports open publication and preservation of user-generated datasets. ARCH is currently optimized for working with tens of thousands of web archive collections, covering a broad range of subjects, events, and timeframes, and the platform is actively expanding to include digitized text and image collections. ARCH also works with various portions of the overall Wayback Machine global web archive totaling 50+ PB going back to 1996, representing an extensive archive of contemporary history and communication.

Who is ARCH for?

ARCH is for any user that seeks an accessible approach to working with digital collections computationally at scale. Possible users include but are not limited to researchers exploring disciplinary questions, educators seeking to foster computational methods in the classroom, journalists tracking changes in web-based communication over time, to librarians and archivists seeking to support the development of computational literacies across disciplines. Recent research efforts making use of ARCH include but are not limited to analysis of COVID-19 crisis communications, health misinformation, Latin American women’s rights movements, and post-conflict societies during reconciliation.

What are core ARCH features?

Build: Leverage ARCH capabilities to build custom research collections that are well scoped for specific research and education purposes.

Access: Generate more than a dozen different research-ready datasets (e.g., full text, images, pdfs, graph data, and more) from digital collections with the click of a button. Download generated datasets directly in-browser or via API.

Analyze: Easily work with research-ready datasets in interactive computational environments and applications like Jupyter Notebooks, Google CoLab, Gephi, and Voyant and produce in-browser visualizations.

Publish and Preserve: Openly publish datasets in line with best practices in reproducible research. All published datasets will be preserved in perpetuity.

Support: Make use of synchronous and asynchronous technical support, online trainings, and extensive help center documentation.

How can I learn more about ARCH?

To learn more about ARCH please reach out via the following form.

Build, Access, Analyze: Introducing ARCH (Archives Research Compute Hub) was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dates can be really hard to decipher in web archives. As a result, we tend to rely on the crawl date, which is a pretty easy thing to grab out of a WARC since it is a mandatory field. While this is something we’ve always had in aut, it’s not the date of a response or creation of a website, but instead is the date on which the crawl occurred. Not ideal for a lot of research questions I’ve seen teams ask of web collections they are exploring.

Embarrassingly my response to these questions had always been, “dates are hard in web archives,” and then shrug. I’d then point folks to one of the many great projects out of the ODU’s Web Science and Digital Libraries Research Group such as Carbon Dating The Web, as potential path forward.

Even more embarrassing, an easy solution has been right in front of my face this entire time, sitting in the HTTP headers of WARC `response` records. Sometimes, but not all the time, a resource in a WARC record HTTP header has a value for Last-Modified. This date should come from the file’s attributes, e.g. ls -lt filename. The header field is pretty easy to grab with Sparkling, and then incorporate it into the DataFrames produced by aut.

Normally, the date should be in this format: Tue, 15 Nov 1994 12:45:26 GMT (RFC1123). But of course, there are always exceptions. So we wrote a fuzzy parser utility to extract the date and format to YYYYMMDDHHMMSS so that it mirrors the same format as crawl_date.

This field is useful for web archive collections that are only crawled once. For example, in situations like Geocities permanently shutting down or the excellent work being done by the Saving Ukrainian Cultural Heritage Online project to preserve Ukrainian cultural heritage.

This functionality is now available as of the 1.2.0 release of the Archives Unleashed Toolkit, and will be available in ARCH in the coming days.

Speaking of ARCH, if you’re interested in piloting it out, please fill out this form.

Here it is in action:

import io.archivesunleashed._

val data = "/sample-data/geocities/GEOCITIES-20091027143300–00114-ia400112.us.archive.org.warc.gz"

RecordLoader.loadArchives(data, sc)
 .all()
 .select($"crawl_date", $"last_modified_date", $"mime_type_web_server")
 .show(20, false)

+--------------+------------------+--------------------+
|crawl_date    |last_modified_date|mime_type_web_server|
+--------------+------------------+--------------------+
|20091027143300|                  |text/html           |
|20091027143259|20000923233454    |image/jpeg          |
|20091027143259|20020913163029    |image/jpeg          |
|20091027143300|20020211154553    |image/jpeg          |
|20091027143259|19980919164703    |image/jpeg          |
|20091027143259|20080125150303    |text/html           |
|20091027143300|20010921224658    |image/gif           |
|20091027143258|20081009015203    |image/jpeg          |
|20091027143300|                  |text/html           |
|20091027143259|20020416145103    |image/jpeg          |
|20091027143300|20090223022835    |text/html           |
|20091027143300|20030928090558    |image/jpeg          |
|20091027143300|20091027143300    |text/html           |
|20091027143300|20021203212451    |text/html           |
|20091027143300|                  |text/html           |
|20091027143300|20040530033010    |image/bmp           |
|20091027143300|                  |text/html           |
|20091027143259|20090223022352    |text/html           |
|20091027143300|                  |text/html           |
|20091027143300|20010608202736    |text/html           |
+--------------+------------------+--------------------+
only showing top 20 rows

There are a few other possibilities for trying to figure out the date of a resource. Since we already use Apache Tika to identify language of a content, and mime_type_tika, we could also potentially use it to try and identify some dates from a given resource in a WARC. It could be a future project if there is a need for it.

I’ve also take the time to update the Geocities dataset again. The updated dataset includes last_modified_date columns for the file format, web pages, and text files derivatives. It also includes crawl_date format fix for domain graph derivative (YYYYMMDD instead of YYYYMMDDHHMMSS).

Here are the 20 most frequent years resources were last modified in the dataset:

Finally, our preliminary research showed reliable values coming from the Last-Modified headers, so we’d love to hear feedback from users on this new feature as they process their data.

AUT & Last Date Modified was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the course of this program, we have collectively hosted and collaborated with forty-five researchers from 21 unique institutions across the globe.

Following a successful and energy-packed year, we are pleased to introduce the five teams participating in the cohort program’s second round. These research teams will investigate a diverse range of topics by exploring web archive collections curated by Archive-It partners.

Latin American Women’s Rights Movements: Tracing Online Presence through Language, Time and Space

• Sylvia Fernandez, University of Texas at San Antonio
• Rosario Rogel-Salazar, Universidad Autónoma del Estado de México, Toluca
• Verónica Benítez-Pérez, Universidad Autónoma del Estado de México, Toluca
• Alan Colín-Arce, Universidad Autónoma del Estado de México, Toluca
• Abraham García-Monroy, Universidad Autónoma del Estado de México, Toluca

The lack of effective responses to femicides and gender-based violence towards indigenous women and women of colour is not only legible across Mexico but comparatively in Latin America, the Caribbean, the United States, Canada, and other parts of the globe.

Throughout the years, women have formed collectives and organizations in public and digital spaces to fight against impunity and to demand justice and rights. This has led to the creation of many online sites, social media profiles, and digital activism.

By engaging web archive collections curated around human rights and feminist movements, project collaborators will examine how these sites have shifted over time and explore language expressions used by women’s rights movements in Mexico and Latin America.

Ultimately, web archives will provide a critical perspective to understanding how the movements against femicides and gender-based violence created a digital identity publicly and in the online domain.

Historicizing Aughts-Era Mormon Mommy Blogging Media Landscapes

• Emily Edwards, St. Francis College
• Robin Hershkowitz, Bowling Green State University

In the era of post-blogging, or microblogging, this project will explore early aughts Mormon mommy blogging culture, mediations of marriage, mothering, and feminine domesticity to historicize this period in relation to contemporary manifestations and trends of mommy influencing on social media platforms.

Feminist scholars have emphasized that practices of mommy blogging and social media influencing are forms of affective digital labour (Mäkinen 2020). This project also emphasizes that these mediated practices of Mormon mommy blogging in the early aughts not only are forms of socially reproductive labour but are ideologically productive as well (Jarrett 2015). Project collaborators will identify connections and transitions to new media ecologies and new iterations of racialized, gendered domestic ideologies that share historic genealogies to digital media practices and structures pioneered by Mormon mommy bloggers.

Web Archiving and the Saskatchewan COVID Archive: Expanding Coverage to Capture Social Media, Medical Misinformation, and Radicalization

• Jim Clifford, University of Saskatchewan
• Derek Cameron, University of Saskatchewan
• Erika Dyck, University of Saskatchewan
• Craig Harkema, University of Saskatchewan
• Patrick Chasse, University of Saskatchewan
• Tim Hutchinson, University of Saskatchewan

The Saskatchewan COVID Archive project began in April 2020 to ensure the capture and preservation of sites that reflected the conversations around and experiences of the COVID-19 pandemic in Saskatchewan. Using this rich dataset, the project will examine how the pandemic response unexpectedly fostered social movements that challenged mainstream ideas about science, public health, and individual responsibility.

Using web archives collected by the University of Saskatchewan, this project seeks to map and connect conversations and experiences of the COVID-19 pandemic with the province of Saskatchewan. The research team will develop timelines and knowledge trees to provide an opportunity to learn about the causal relationship between social media and this important shift in public health policy in Saskatchewan.

Querying Queer Web Archives

• Filipa Calado, The Graduate Center, CUNY
• Corey Clawson, Rutgers University — Newark
• Di Yoong, The Graduate Center, CUNY
• Lisa Rhody, The Graduate Center, CUNY

Studying web archives from a variety of angles that pertain to contemporary subfields and areas of interest in Queer Studies, this team will investigate how queerness interacts with, is informed by, and forms web spaces. In particular, the project will focus on how queerness has been theorized across various web communities and what web materials and formats can teach us about the creation of queer community. In studying queer online spaces, project investigators will explore investments in concepts like utopia, play, radicalism, normativity, religion, and conversion and how they affect queer identity and discourse formation over time.

Using Web Archives for Mapping the Use of Cultural Practices in Postconflict Societies and During Reconciliation Processes

• Ricardo Velasco Trujillo, University of Minnesota Twin Cities
• Luis Gomez, The Australian Academy of Science

Despite the growing interest in the role of cultural practices during reconciliation processes, in societies trying to confront the legacies of troubled pasts and widespread human rights violations, there is a broad gap in statistical knowledge and a lack of reliable data regarding what type of cultural practices are used, and their correlation in the effectiveness of reconciliation processes across different geographical and cultural contexts.

Employing computational methods such as contextual search, data mining, and web scraping, amongst others, this project aims at making an initial assessment and mapping out the use of cultural practices in different reconciliation processes and across several human rights organizations working actively in post-conflict societies.

Stay up to date with Cohorts by visiting the program page: https://archivesunleashed.org/cohorts2022-2023/

Web Archives Research: Return of the Cohorts was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

Since 2017, the Archives Unleashed Project has been making petabytes of historical content accessible for exploration and investigation, by lowering the barriers to web archives.

While the nearly three-decade legacy of collection and preservation speaks for itself — literally through the volume of data, the Internet Archive’s Archive-It alone now has petabytes of data — the scarcity of accessible, usable, and approachable tools to deal with web archives at scale has been felt acutely by scholars across disciplines.

To address this challenge, Archives Unleashed focuses on building open-source tools and approaches that lower the barriers to working with digital cultural artifacts held in the W/ARC file format.

The Archives Unleashed Toolkit (AUT) is an open-source Apache Spark library for processing and conducting analysis on web archives (W/ARCs), which has provided the foundations for web archive analytical platforms including the Archives Unleashed Cloud (2018–2020) and the Archives Research Compute Hub (ARCH).

As a contribution to the web archiving community, the Toolkit significantly improves the interaction with web archive data, as it offers large-scale processing required by W/ARCs while opening an opportunity to interdisciplinary scholars by reducing (though not completely eliminating — you still need to use the command line!) the technical burden.

Presenting AUT 1.1.0

Our team is proud to present the 1.1.0 release of the Archives Unleashed Toolkit!

This release is the result of 5 years of development (and a lot of hard work) and is a major milestone for the Archives Unleashed Project and its contributors.

The drive toward making web archives more accessible is directly reflected through the Toolkit’s evolution. The concept of the Toolkit initially grew out of the analytics functions of Warcbase (Apache HBase and OpenWayback). In 2017 the team made significant changes to its technical foundation by creating an Apache Spark library. Most recently, the team integrated the Internet Archive’s Sparkling for parsing W/ARC records.

Under the direction of our co-investigator Nick Ruest, along with extensive contributions from project investigators Ian Milligan and Jimmy Lin, and Helge Holzmann of the Internet Archive, the Archives Unleashed Toolkit has seen 1,018 commits, including 44 enhancements, 27 bug fixes, and 157 closed issues.

Let’s review the significant developments and features that contribute to the Archives Unleashed Toolkit 1.1.0.

• Full support of DataFrames (DF) and Resilient Distributed Dataset (RDD). The Toolkit started with RDD due to the seamless way it could process structured and unstructured data. After Apache Spark’s adoption of DataFrames and consultation with our user community and scholars at large, development pivoted to include DataFrames because it provides an easily readable visual presentation of data (in a tabular format). As reflected in the user documentation, RDD and DF are supported.
• Python and Scala implementations to accommodate the programming languages most familiar to scholars working with web archives.
• Extensive user documentation covering dozens of text, network, and binary analytical examples. The cookbook-style documentation provides examples of how users can engage with various scripts to explore web archive collections.
• Creation of standard derivatives that can be run with spark-submit. This provides a single script to submit and launch the processing of a wide variety of jobs — i.e. run graph, binary information, or text extraction jobs — making a user’s time more efficient within the Toolkit.
• Adoption of Sparkling, developed by Helge Holzmann, is a data processing library for Apache Spark. The use of Sparkling has drastically affected the speed of data processing and the efficiency of loading, parsing, and storing W/ARC.
• Toolkit reaches parity with ARCH! A standardized set of over a dozen web archive research derivatives can be generated through the Toolkit. To reflect the additional functionality designs and developments reached through the project’s second phase, the Toolkit has been updated to generate the same derivatives available through ARCH, including a new set of derivatives for extracting text file content (css, html, js, json, plain text, and xml).

If you liked the Toolkit…Check out our Notebooks!

The Toolkit provides powerful options for investigating web archive collections including over a dozen unique extraction and filtering jobs. For instance, users can generate derivatives that include, all the text of a webpage, extract the most frequent image URLs in a web collection, export link structure data to Gephi by writing to the GEXF format, and filter based on dates or exact URLs among a myriad of other combinations.

Dataset derivatives created through the Toolkit, act as a bridge to other projects we’ve undertaken to assist scholars, including a variety of example notebooks, including an introduction to working with ARCH derivatives for librarians, archivists, or researchers new to web archive derivatives. The notebooks provide a space for creating, editing, and sharing documents containing live code, equations, visualizations, and text. We believe that notebooks offer a meaningful experience for scholars, as they provide a more accessible entry point for exploring and interacting with web archive collections.

…Or GeoCities Derivatives

As a team dedicated to the principles of open access and usability, we are conscious of the roadblocks scholars face, especially around accessing data. In collaboration with institutions and collaborators throughout North America, the Toolkit has been used to generate derivatives on dozens of web archive collections to expand access, use and visibility. We see two major outcomes of collaboratively producing derivatives: they provide increased visibility of an institution’s web archive collection holdings, and they lower the barriers researchers face in gaining direct access to collections and in turn using them for scholarly inquiry.

Most recently, AUT 1.1.0 has been used to update the GeoCities Web Archive Derivative Datasets, in alignment with the derivatives produced by Archives Research Compute Hub (ARCH). This means that researchers can dive straight into the 18 CSV formatted files that cover domain, image, webgraph, text files, binary information and more!

Additionally, over three-dozen collections have been processed through the Toolkit with generated derivatives openly available via the Web Archives for Historical Research Group on Zenodo and Borealis (previously known as Scholars Portal Dataverse). If you are a researcher in need of derivatives, check out the full list of Archives Unleashed Datasets.

Many Thanks from the Archives Unleashed Project

We would like to thank our project collaborators who have contributed their time to code development and review: Helge Holzmann, Jeremy Wiebe, Ryan Deschamps, Alice Zhou, Gursimran Singh, Sarah McTavish, Jinfeng Rao, Youngbin Kim, Ryan Wolniak, Boris Lin, Clemens Neudecker, Vicky Zhu, Zack Wang, Billy Jin, Mat Kelly, Andy Jackson, and David Portabella.

Many thanks also go out to our community of users, testers, and supporters whose invaluable feedback inspires our continued improvement.

The Archives Unleashed Project and Toolkit work is primarily supported by the Andrew W. Mellon Foundation. Other financial and in-kind support comes from the Social Sciences and Humanities Research Council, Compute Canada, the Ontario Ministry of Research, Innovation, and Science, York University Libraries, Start Smart Labs, and the Faculty of Arts and David R. Cheriton School of Computer Science at the University of Waterloo.

Further Reading

If you would like a more in-depth look at the project, please check out the following articles:

• Nick Ruest, Jimmy Lin, Ian Milligan, and Samantha Fritz. The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. Proceedings of the 2020 IEEE/ACM Joint Conference on Digital Libraries (JCDL ‘0), Wuhan, China. [link] [preprint]
• Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017. [link]
• Helge Holzmann, Nick Ruest, Jefferson Bailey, Alex Dempsey, Samantha Fritz, Peggy Lee, and Ian Milligan. “ABCDEF: the 6 key features behind scalable, multi-tenant web archive processing with ARCH: archive, big data, concurrent, distributed, efficient, flexible.” In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries (JCDL ‘22). Association for Computing Machinery, New York, NY, USA, 1–11 (2022). [link] [preprint]
• Tobi Adewoye, Xiao Han, Nick Ruest, Ian Milligan, Samantha Fritz, and Jimmy Lin. “Content-Based Exploration of Archival Images Using Neural Networks.” In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL ’20). Association for Computing Machinery, New York, NY, USA, 489–490 (2020). [link] [preprint] [poster demo] [video summary]
• Ryan Deschamps, Nick Ruest, Jimmy Lin, Samantha Fritz, and Ian Milligan. “The Archives Unleashed Notebook: Madlibs for Jumpstarting Scholarly Exploration of Web Archives.” Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019). [preprint]
• Hsiu-Wei Yang, Linqing Liu, Ian Milligan, Nick Ruest, and Jimmy Lin. “Scalable Content-Based Analysis of Images in Web Archives with TensorFlow and the Archives Unleashed Toolkit.” Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019). [preprint]
• Ryan Deschamps, Samantha Fritz, Jimmy Lin, Ian Milligan, and Nick Ruest. “The Cost of a WARC : Analyzing Web Archives in the Cloud.” Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019). [link] [preprint]

Archives Unleashed Toolkit 1.0.0: A Sparkling New Way to Explore Web Archives was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

Web archives present opportunities to explore and enhance our understanding of our digital cultural heritage. After a quarter-century of intentional collection and curation, they offer a breadth of coverage on topics fast outpacing traditional analogue material. Yet, web archives maintain a mysterious quality.

Over the lifespan of the Archives Unleashed project, we’ve heard so often, “how do you do web archival research?”, “where do you start”? Even the primary step of selecting a topic of study can be overwhelming. Initially, we ran a series of datathons to train scholars and provide a hands-on approach to working with archived web content. However, while datathon events contributed to skill-building, their short (two-day) intensive model meant a loss of momentum for projects, with teams unable to go beyond exploratory discovery, and a missed opportunity for comprehensive analyses or publication development.

In addition to our collaboration with members from Internet Archive’s Archive-It team to develop a platform for generating web archive derivatives (ARCH), our combined teams seek to support and facilitate research engagement with web archives.

Building off past successes of community building and engagement, the inaugural Archives Unleashed Cohort Program was launched in July 2021.

Five teams were selected to take part in a year-long collaboration and provided with additional resources, support and mentorship to conduct focused research while using web archives as scholarly objects.

These teams, spanning across North America and Europe, have selected a wide range of topics to study, including crisis communication, health misinformation, pandemic discourse, comparative feminism media activism, and the development of online commenting systems. They truly highlight the innovative applications of web archive research within and adjacent to the digital humanities field.

It’s been truly inspiring to see the creative ways teams have approached web archive collections, from the research questions they are asking to the methodologies employed. As such, we are delighted to share highlights from cohort projects and celebrate their successes with collection curators and the broader #webarchiving and research communities.

Archives Unleashed Cohort Project Roundup

AWAC2 Analysing Web Archives of the COVID Crisis through the IIPC Novel Coronavirus dataset

Project Members: Valérie Schafer (University of Luxembourg), Frédéric Clavert, (University of Luxembourg), Karin De Wild (Leiden University), Niels Brügger, Aarhus University, Susan Aasman (University of Groningen), Sophie Gebeil (University of Aix-Marseille)

The AWAC2 (Analysing Web Archives of the COVID Crisis through the IIPC Novel Coronavirus Dataset) project was launched by a team from the WARCnet network. The aim is to investigate a transnational event through web archive collections by performing a distant reading of the IIPC COVID-19 web archive collection to shed light on actors, content types and interconnectivity.

The AWAC2 team has so far conducted a preliminary investigation of the corpus content using text-mining software such as IRaMuTeQ, Jupyter Notebooks, Python libraries such as gensim, and the network analysis application Gephi. The main aim of these analyses is to shed light on what is in the IIPC COVID-19 corpus. The use of simple distant reading techniques — such as crawl distribution by language by day or crawl distribution by domain by day — improved our understanding of the content of this fascinating corpus. One of the methodological questions that arose from the preliminary investigations is the issue of duplicates and versions, which is now being investigated.

In parallel, our colleague (and the project initiator) Valérie Schafer published a post on the IIPC blog] describing the project and asked the web archiving community to vote for a theme that we should investigate in more detail — going beyond the broad big data analyses that we will perform on the corpus. The community voted for us to take a closer look at *Women, Gender and COVID-19* within the collection (domestic violence, care, homeschooling, etc.). We will start exploring this topic soon.

Crisis Communication in the Niagara Region during the COVID-19 Pandemic

Project members: Tim Ribaric, David Sharron, Cal Murgu, Karen Louise Smith, Duncan Koerber (Brock University)

Tim Ribaric and team are exploring COVID-19 from a localized perspective, by examining how organizations in the Niagara region have responded to provincial and municipal government COVID-19 mandates. When the pandemic began the Archives at Brock University quickly started harvesting content from different sources in the Niagara region. These sources were community groups, regional governments, local health units, and various other local news sources. The end result was the COVID 19 Niagara Archive. Further, the goals of the project are to serve both the pursuit of research as well as to contribute to various teaching and learning applications within the University.

In terms of research, an analysis of the web archive content is currently underway using a framework known in media studies as crisis communication. In other words, an examination of the content and tone of the information that was shared in the community during the various inflection points of the crisis. This investigation is completed in part by making use of a search tool called SolrWayBack, an easy-to-use platform for indexing and searching large-scale web archival collections. Additional strides in this investigation are being made using computational notebooks hosted on the Google Colab environment. By carefully extracting full-text derivatives from the complete archive using targeted domains and analyzing using notebooks several subsets of the initial dataset have been created. With these datasets in hand, we can now proceed with a more targeted exploration.

With respect to support of teaching and learning, this project has created several curricular and co-curricular opportunities that have introduced web archives via computational notebooks as an emergent source of information that can be used in varying contexts. By collocating derivatives of the full archive and presenting them in fastidiously crafted computation notebooks, learners can interact with different components of the data without needing to understand coding. We’ve introduced this style of learning in one of our co-investigator’s classrooms, as well as unique co-curricular programming through the Brock Library. We believe that web archive collections represent a rich source of information that have the potential to inform various academic disciplines. This project is helping move this notion forward in the Brock community by demonstrating exactly what can be accomplished through examinations of web archives, both in research and in the classroom.

Regular updates are provided through the project website: https://brockdsl.github.io/archives_unleashed/

Mapping and tracking the development of online commenting systems on news websites between 1996–2021

Project members: Anne Helmond (University of Amsterdam/University of Siegen), Johannes Paßmann, Robert Jansma (University of Siegen), Luca Hammer (University of Siegen), Lisa Gerzen (Ruhr University Bochum). Contributors: Dave Wahl (University of Amsterdam), Steffen Reinhard (Ruhr University Bochum), and Theresa Schulte (University of Siegen).

This project traces the adaptation and distribution of online commenting technologies on news websites. Early accounts of online comments conceptualized them as blurring the lines between the producers and consumers of online texts as internet users could now “talk back” to mass media by providing feedback on the editorial content of professional media organizations (Bruns, 2005). Whilst these ideas emphasize the “democratizing” potential of online comments, more recently commenting is seen as a problematic practice that belongs to the “bottom half of the web” (Reagle, 2016). As online comments have become more toxic and have expanded in scale and scope they increasingly require moderation, leading many news websites to shut down their commenting sections (Gillespie, 2018; Wired, 2015).

In collaboration with the Internet Archive and the Archives Unleashed Project, we have created three longitudinal datasets consisting of the top 50 international, German, and Dutch archived news websites between 1996–2021. The new ARCH interface (Ruest et al., 2020) provides access to these newly created datasets in derivative formats and makes the (large) amount of data more manageable and turns it into a more accessible format for researchers. Using these derivatives we aim to trace the dynamics of news websites implementing, changing, and shutting down their commenting systems to understand the evolution and distribution of these technologies. Such changes in technologies also serve as key starting points for qualitative inquiries to understand commenting practices by interviewing the developers of commenting systems and webmasters implementing them (Paßmann 2021).

To detect the presence of a commenting system we are searching for the code patterns of commenting technologies in the archived HTML code of these news websites (cf. Helmond, 2017; Nielsen, 2019; Owens and Thomas, 2019). We are creating a database of known commenting technologies and their recognition patterns. Using Jupyter Notebooks, the recognition patterns can be used to search for commenting technologies across our three archived news datasets or any other web archive. Initial findings show that not all online comments are well preserved, however, there are still patterns in the archived HTML code showing which commenting technology websites were used. A key contribution of this project is to operationalize an approach for the large-scale analysis of archived HTML code which may also be used to study the evolution and distribution of other key web technologies such as content management systems, analytics, advertising, social buttons, and trackers.

Everything Old is New Again: A Comparative Analysis of Feminist Media Tactics between the 2nd- to 4th Waves

Project Members: Shana MacDonald (University of Waterloo), Aynur Kadir (University of Waterloo), Brianna Wiens (York University), Sid Heeg (University of Waterloo)

The Everything Old is New Again: A Comparative Analysis of Feminist Media Tactics project is run by the Feminist Think Tank (FTT) research lab at the University of Waterloo. The project aims to explore the shifting conversations and practices that have shaped feminist media over the last four decades. To explore these shifts and flows the FTT team has been working with several collections within the Internet Archive including:

• Capturing Women’s Voices
• Contemporary Culture Collection
• Contemporary Women Artists on the Web
• Fales Library New York Feminist Art Institute
• Guerrilla Girls Broadband archives
• Sallie Bingham Center for Women’s History and Culture
• Schlesinger Library’s #MeToo Web Archives collection
• Schlesinger Library Series
• Tamiment-Wagner: Feminism and Women’s Movements
• Tamimnet-Wagner: Other Left Activism

The team is focusing on two specific research questions. The first uses a text-based analytic approach to determine the presence of a set of feminist ‘keywords’ across the collections. The goal is to see if the prominence of each word shifts in its usage over time in order to determine the different priorities of feminist media activism in different eras. The second exploration seeks to determine image content within the collections to see if particular images emerge as dominant across the collection. This will help to map and better understand the role of visual culture in furthering feminist principles at different moments in the historical trajectory of the study.

To this point, we’ve been spending time getting comfortable with the format of web archives (which is a new endeavour for our team!). It has helped us clarify research questions and shed light on the different kinds of insights offered by large web archival collections which contrast, but also compliment, our smaller scale FTT collection of feminist social media tactics (memes, hashtags, etc.) that we’ve been gathering over the last eight years. Working with the ARCH (Archive Research Compute Hub) platform gives us the chance to put these different approaches to digitally-born artifacts from our commitment to feminist data and feminist archiving principles which seeks to examine the existence of power embedded within our technologies, interfaces, platforms, and even in our research questions and frames.

Viral health misinformation from Geocities to COVID-19

Project members: Shawn Walker, Michael Simeone, Kristy Roschke, Anna Muldoon, Major Brown (Arizona State University)

Shawn Walker and colleagues from Arizona State University are exploring and comparing two case studies of health misinformation: HIV mis/disinformation circulating on Geocities in the mid-1990s to early 2000s with the role of official COVID-19 Dashboards in COVID mis/disinformation.

Combining web archives and Twitter data, the team is focused on investigating the IIPC Novel Coronavirus (COVID-19) Archive-It collection, Geocities derivatives collection by Nick Ruest, and Twitter’s COVID-19 stream.

Jupyter notebooks and other computational methods have been critical for experimenting and establishing the scaffolding necessary to conduct large-scale analysis. Currently, the team is engaging with the plain text of the Geocities data, and exploring methods such as coding for keyword and sentiment analysis, and topic modelling to better understand text reuse within the larger corpus.

This work contributes to our understanding of current and historical health misinformation as well as the connections between them, and will also garner insights into how historical narratives of health misinformation have been recycled and repurposed.

Stay Connected!

We look forward to sharing additional highlights as cohorts progress and reach milestones over the coming months. Stay connected with our project and community:

• Follow us on Twitter
• Join our Slack group
• Participate on Github
• Subscribe to our newsletter
• Share our news with colleagues and friends

Many Thanks to Tim Ribaric, Cal Murgu, Valérie Schafer, Frédéric Clavert, Anne Helmond, and Shana MacDonald for their contributions to this piece!

Research Applications with Web Archives: Collaboration Among Archives Unleashed Cohorts was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Usability is about people and how they understand and use things, not about technology.” — Steve Krug

Earlier this spring, we shared our roadmap for building a robust cloud-based interface to support web archival research at scale and enhance access to web archives. Months of development have resulted in our fully functional prototype: ARCH (Archive Research Compute Hub). As we move towards an official launch in Spring 2022, we are working with users to ensure it meets all of their web archive analysis needs!

How does ARCH Work?

ARCH allows users to delve into the rich data within web archival collections for further research. Users can generate and download over a dozen datasets within the interface, including domain frequency statistics, hyperlink network graphs, extracted full-text, and metadata about binary objects within a collection. ARCH also provides several in-browser visualizations that present a glimpse into collection content.

The design process for ARCH has involved a variety of interconnected stages from sketching a wireframe, to connecting back-end processes with a user interface design, and conducting a multi-staged user testing process to continually assess user sentiment and impact with functionality and interface improvements.

This user testing has been a vital component to understanding the user journey, intuitive workflows, and varying expertise and research needs for working with web archives at scale.

Stages of ARCH UX Testing

At its core, User Experience (UX) testing seeks to understand the impressions, experience, and feelings a user expresses while interacting with a product prototype. These insights are critical because it brings the creators and developers into closer alignment with their end-users.

Conducting UX Testing for ARCH has allowed our team to understand research behaviours and the user journey while assessing what works well, what challenges arise, and identifying needs that aren’t being met.

Testing encompassed five main stages:

1. Define Objectives. To provide scoping goals for our UX Testing we determine evaluation criteria, methods, and testing protocols of each stage. The purpose of testing was to engage with selected Archives Unleashed / Archive-It users who would provide feedback and input on their initial ARCH impressions. Ultimately these insights informed issues that needed to be addressed regarding usability, workflow, and functionality.
2. Recruit. We then identified and engaged with Archives Unleashed users and Archive-It partners. This stage also meant thematically grouping users based on their relationship to the Archives Unleashed Project (e.g. Concept Design Interviewees, Advisory Board members, and “Power Users”). There was also a conscious effort to ensure recruitment reflected a diverse range of institutional categories, as set by Archive-It (e.g. University & Colleges, National Institutions, Public Libraries & Local Governments, etc.).
3. Test. We primarily tested ARCH through remote surveys, which collected qualitative and quantitative data to determine satisfaction on several key indicators including intuitiveness, ease of navigation, terminology, visualizations, processing time, and application to user research.
4. Analyze. A variety of methods were used to extract qualitative and quantitative data collections. Statistical descriptors were used to provide a user profile that identified geographic and institutional representation, professional role, and comfort level in using data analytical software and tools. A five-point Likert scale was used to measure satisfaction of intuitiveness, navigation, terminology, visualizations, processing time, and application. To address qualitative feedback, thematic coding was applied to comments in the areas of user interface design choices, workflow, language, documentation, outputs, feature requests, and errors encountered.
5. Implement Findings. Results were shared with the team, and feedback was translated into GitHub tickets to provide action-based tasks for future development cycles. Implementing user suggestions also provides a base for future iterative UX testing rounds.
6. Verify with Users. As a multi-stage UX testing process, each subsequent round of testing served as another opportunity to review and refine impressions of prior development and improvements — improving our accuracy and capacity to match user needs at each stage

A Snapshot of ARCH UX Round 3 Results

The most recent round of UX testing was conducted throughout August and September 2021, connecting with past concept design interviewees, our project advisory board members, and selected Archives Unleashed and Archive-It “power” users. Participants shared their experience and feedback through a survey.

Tester Profiles

Profiling participants revealed users were primarily from North America and representative of two main institution categories: colleges/universities and national libraries. In addition, testers can be categorized into four main professional roles: researcher/professor, librarian/archivist, technologist, and managerial/leadership.

The survey asked testers to identify their comfort level in using data analytical software and tools to help assess their experience and feelings with the technical aspects of analyzing data. The majority of participants in this group reflected high confidence, while 25% of respondents described themselves as slightly comfortable, meaning they can use tools and software but need assistance.

Areas of Satisfaction

A second area of quantitative questioning used a five-point Likert scale to measure the satisfaction of intuitiveness, navigation, terminology, visualizations, processing time, and application.

Overall, testers identified a positive ARCH experience, noting the benefits in being able to access collections and conduct initial analysis. Using a combination of a heat map visualization and averaging satisfaction scores, it was easy to identify the highest and lowest scores, while also diagnosing areas for improvement. Statements that incurred few neutral or disagreement scores were further reflected in open comments.

Constructive Feedback

Open-ended survey comments offered a chance for testers to express their thoughts, feelings, and experience of using ARCH using their own words.

Overall, users were impressed with the new interface, noting that the integration between Archives Unleashed and Archive-It provides a familiar and dedicated environment for working with web archives, and offers opportunities for new research use cases.

Testers also appreciated the quick processing speeds, the variety of dataset and output options, and that no technical set-up was required (e.g. running a Spark shell).

Analysis of 94 comments identified six themes that conveyed detailed suggestions for improvements and considerations. These areas included interface design choices, workflow, language, accessibility, documentation, and output usability. Participants also identified errors encountered.

The majority of suggestions related to improving the UI workflow and navigation, which was supplemented by changes to language, design choices, and visualization features to improve recommendations around accessibility and usability.

Implementing UX Feedback

Our team uses GitHub to manage development and version control of the source code for the project’s ARCH platform, and uses tickets to monitor improvements and roadmap. As a result, 27 GitHub tickets were created from the open comments to provide direction for actionable items and team discussions.

Lessons Learned

Carrying out UX testing has afforded an opportunity to learn more about our users and carry lessons learned forward into future testing cycles.

Here are some of our insights:

1. People are genuinely excited to help out! When you build a community, you also build a system of support, encouragement, and connection. In reaching out to individuals who are familiar with our project — both by following progress and using our tools — we found there was general enthusiasm and responsiveness to UX testing!
2. Keep it simple for the user. Understanding time commitments can be a big ask, we purposefully kept our survey short and simple. We were conscious to ensure UX testing didn’t become a burdensome task, but rather presented an exciting opportunity for participants to impact platform development directly.
3. Responses were higher for interviews than surveys. It was a bit of a (pleasant) surprise that participants were willing to schedule a longer call to discuss thoughts and impressions. In comparison, we had a 41% response rate and very thoughtful input via a feedback form. We had anticipated the asynchronous element of the survey form would have yielded a high response rate, but we also are very aware of how the element of human-interaction and the opportunity to discuss development with members of the product team may have felt more approachable and held a more personal, appealing touch.
4. No matter how much you prepare, there will always be glitches. As they say, prepare for everything! During our UX testing, we had some unforeseen technical issues, which took our prototype offline. As a team, we triaged this issue, with some members focusing on the technical aspect while others connected with testers. Although all errors were remedied, this did cause a disruption to our testing process. This experience did reinforce within our team that no matter what issues arise, communication is critical!

Next Steps

Our final two rounds of UX testing will be conducted early in 2022. Participants will have a chance to interact with the latest version of ARCH, with implementations from our most recent testing. These final rounds will also provide an opportunity for a stress test of ARCH’s to monitor the back-end for any areas that are overwhelmed or in cases where processes fail.

We look forward to sharing ARCH with the public in the Spring of 2022!

ARCH UX Testing: Designing for the Users was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

This spring, we mark a few special anniversaries for the Archives Unleashed Project! It’s been four years since the start of the project and our third year of running the Archives Unleashed Cloud.

With the support of the Andrew W. Mellon Foundation, our focus has been on developing open-source analysis tools to help make web archives more accessible. One of our primary goals has been “to make petabytes of historical internet content accessible.” Thanks to our community of Cloud users and collaborators, we’ve been able to make that goal a reality.

As we sunset the Cloud at the end of June 2021, we reflect on what Archives Unleashed has accomplished with the Cloud.

Building the Cloud

The Archives Unleashed Cloud was developed to support access and use of web archives for scholarly inquiry. The Cloud presents an accessible and user-friendly interface, while simplifying the process of conducting analysis on web archive collections.

The first commit to the Archives Unleashed Cloud repository was on 24 October 2017. The proceeding weeks focused on designing and building the site’s basic framework as well as functions for conducting analysis.

The Cloud was built as an open-source platform, running on Ruby on Rails and Apache Spark. The majority of back-end development occurred between November 2017 and April 2018, with a focus on three main areas:

1. Harness the power of the Toolkit. The Cloud hosts the “canonical instance” of the Toolkit, meaning it uses the Toolkit as the underlying code-base to drive analytical functions. In providing an accessible user interface, researchers could focus on research questions and conduct analysis in as few as three clicks.
2. Hook up WASAPI. The Web Archiving Systems API (WASAPI), is a “standardized mechanism to export and import web archive data between systems”. Using WASPI created a connection between platform and data source; the platform being the Archives Unleashed Cloud, and the data source being a user’s web archive collections from Archive-It.
3. Build background jobs. Adopting Apache Spark provided a pathway for building processing jobs that resulted in creating derivative files and in-browser visualizations.

As you can imagine, we did a lot of testing! Special attention was paid to performance testing to validate, understand, and improve responsiveness, scalability, speed and resource usage.

In May 2018, the Archives Unleashed Cloud was soft launched.

Achievements

The Archives Unleashed Cloud ultimately provided new opportunities to engage with web archival research and explore web archive collections. Our team is extremely proud to reflect on the work completed and achievements met. So let’s take a look!

Reach and Impact

The Archives Unleashed Cloud has been adopted across 60 unique institutions in 10 countries, though the majority of our users were from North America, Europe, Australia and New Zealand. It has been used among a diverse range of institutions and organizations, including:

• Universities/College Libraries
• National Libraries
• International & Non-Profit Organizations
• Public Libraries
• Government Organizations
• Art Archives

On the research end, the Cloud has been used to analyze collections covering a wide range of topics, from journalism and politics to health and national sporting events. We’ve also seen the Cloud used within the classroom setting, as instructors have promoted experiential learning and understanding of web archives. Similarly, there are several cases of graduate students using the Cloud for exploratory research, and even identifying how it fits within scholarly workflows that incorporate multiple (and interrelated) tools.

The Cloud provided a scalable solution for the analysis of web archives and a robust data processing scaffolding. Over its lifespan, the Cloud has processed just under a petabyte of W/ARC data across 1500 collections. That’s a lot of web archive data!

Collaborations

Engaging with institutions has been important for the growth of our user community, and in bridging the gap between researchers and data.

Project co-investigator Nick Ruest worked with several institutions to support access by creating and sharing derivative data of over 30 web archive collections. Using the Archives Unleashed Toolkit and Cloud, derivatives were generated for select collections curated by Columbia University, Ivey Plus Library Confederation, and the Bibliothèque et Archives nationales du Québec. These derivative datasets have been made publicly available through Web Archives for Historical Research Group communities on Zenodo and Dataverse with a citable DOI.

This initiative had two goals in mind:

1. To help institutions promote and increase the visibility of their web archive collections; and
2. Continue to lower barriers of access and use, which in this case, provides a starting point for researchers who don’t have immediate access to WARCs while encouraging exploration of web archival data.

Early on in the Cloud’s development, our project connected with six Canadian research universities to test analysis functionality and processes. From this experience, we were able to test scalability, efficiency, as well as identify and troubleshoot errors on a variety of collection sizes and topics.

Learning Resources

Our users noted that they often had difficulties knowing where to start once they had web archive data to explore. In response, and to provide a bit of inspiration, our team developed learning guides that walked through how to use Cloud derivatives with external tools. In total, half a dozen guides were created and addressed common approaches to both network and textual analysis, while incorporating tools like Gephi and Voyant. These resources have provided a starting point for exploring web archives and promoted confidence in scholars new to web archive research.

The Next Evolution

While the Archives Unleashed Cloud is sunsetting, it isn’t entirely going away. The conceptual framework has been reimagined to continue the support of access to and use of web archives, while also addressing long-term sustainability plans.

We will see the analytical functions from the Cloud brought into alignment within the Archive-It environment. This integration (2020–2023) will allow for an end-to-end service for working with web archives at scale: from collection curation to analysis. The collaboration between Archives Unleashed and Archive-It ultimately seeks to broaden and enhance the accessibility and usability of web archives.

Our team would like to express gratitude to everyone who has supported this project. We truly appreciate the enthusiastic response we’ve received from the community and look forward to sharing our next evolution.

The codebase for the Cloud is available through our Github repository: https://github.com/archivesunleashed/auk.

Additional Reading

Archives Unleashed has published several articles, posts, and materials addressing the concept, design, and impacts of the Cloud.

Journal Articles / Conference Papers

• From archive to analysis: accessing web archives at scale through a cloud-based interface. International Journal of Digital Humanities, 2021.
• The Archives Unleashed Project: Technology, Process, and Community to Improve Scholarly Access to Web Archives. arXiv:2001.05399, January 2020.
• The Cost of a WARC : Analyzing Web Archives in the Cloud. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, Vol. 19 (2019).

Invited Talks

• “Lowering the Barrier to Access: The Archives Unleashed Cloud Project.” The Web That Was: Archives, Traces, Reflections, June 2019, Amsterdam, Netherlands. [slides]
• “Project Sustainability and Research Platforms : The Archives Unleashed Cloud Project.” International Internet Preservation Consortium Web Archiving Conference, June 2019, Zagreb, Croatia. [slides]

Medium Posts

• Cloud-hosted web archive data: The winding path to web archive collections as data
• Learning to Use Large-Scale Methods to Explore Web Archives
• Archives Unleashed Cloud: A Powerful Web Archive Analysis Portal
• On Cloud Number 9: Accessibility, Usability, and Functionality with the Archives Unleashed Cloud (AUK)

So Long, Farewell to the Archives Unleashed Cloud was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.

Earlier this year, the Archives Unleashed Project announced the Cohort Program, which aims to support web archives research by providing resource support and mentorship. Starting in July, cohorts will engage in collaborative activities and conduct focused research that explores a variety of web archive collections.

We are pleased to introduce the five teams that will make up our inaugural Cohort program.

AWAC2 — Analysing Web Archives of the COVID Crisis through the IIPC Novel Coronavirus dataset

Valérie Schafer, University of Luxembourg (LU)

Karin De Wild, Leiden University (NL)

Frédéric Clavert, University of Luxembourg (LU)

Niels Brügger, Aarhus University (DK)

Susan Aasman, University of Groningen (NL)

Sophie Gebeil, University of Aix-Marseille (FR)

Investigating transnational events through web archive collections, the AWAC2 team will focus on a distant reading of the IIPC COVID-19 web archival collection to understand actors, content types and interconnectivity throughout it.

Everything Old is New Again: A Comparative Analysis of Feminist Media Tactics between the 2nd- to 4th Waves

Shana MacDonald, University of Waterloo

Aynur Kadir, University of Waterloo

Brianna Wiens, York University

Sid Heeg, University of Waterloo

Project members will explore web archive collections to conduct a comparative analysis of the history of feminist media practices across interdisciplinary multi-media sources. The team expects to produce a timeline of issue responses from different historical moments and map different feminist media practices over this timeline to determine overlaps. The project’s key outcome will be to recover earlier feminist media practices and contextualize them in the digital present.

Mapping and tracking the development of online commenting systems on news websites between 1996–2021

Anne Helmond, University of Amsterdam/University of Siegen

Johannes Paßmann, University of Siegen

Robert Jansma, University of Siegen

Luca Hammer, University of Siegen

Lisa Gerzen, University of Siegen

This project aims to reconstruct a history of online commenting by examining the role of commenting technologies in the popularisation of commenting practices. It will do so by examining the distribution and evolution of commenting technologies on the top 25 Dutch, German, and world news websites from 1996–2021, to understand how they have shaped the practices of users. This will allow them to explore the interplay between technologies and practices of the past and to investigate histories of natively-born technologies and practices.

Crisis Communication in the Niagara Region during the COVID-19 Pandemic

Tim Ribaric, Brock University

David Sharron, Brock University

Cal Murgu, Brock University

Karen Louise Smith, Brock University

Duncan Koerber, Brock University

Using web archives collected by Brock University, this project will examine how organizations in the Niagara region have responded to government COVID-19 mandates. Analysis will focus on investigating three types of entities: local government, non-profit organizations, and major private entities. Findings from this research aim to inform future crisis communication organizational planning, specifically at the local and municipal level. The project will also create several open computational notebooks to support teaching, learning, and research.

Viral health misinformation from Geocities to COVID-19

Shawn Walker, Arizona State University

Michael Simeone, Arizona State University

Kristy Roschke, Arizona State University

Anna Muldoon, Arizona State University

This project will examine and compare two case studies of health misinformation: HIV mis/disinformation circulating on Geocities in the mid-1990s to early 2000s with the role of official COVID-19 Dashboards in COVID mis/disinformation. This work contributes to our understanding of current and historical health misinformation as well as the connections between them, and will also garner insights into how historical narratives of health misinformation have been recycled and repurposed.

Introducing Archives Unleashed Cohorts was originally published in Archives Unleashed on Medium, where people are continuing the conversation by highlighting and responding to this story.