Web Archiving: Preserving the Ephemeral
Less than a year into Donald J. Trump’s tenure as the 45th president of the United States of America, federal agencies such as the Environmental Protection Agency (EPA) systematically purged references to climate change from their online resources [1]. The move, intended to align the organization’s web presence with the Trump administration’s agenda of undoing environmental protections, removed links to databases containing historical temperature and emissions records [2].
A mirror of EPA’s website before the modifications containing 46 gigabytes worth of compressed data was made publicly available by aj10017, a voluntary member of the Internet Archive [3]. The Internet Archive is a nonprofit building a digital corpus of web pages and other cultural artifacts. Web archiving aims to collect, store, and preserve the World Wide Web despite its transient nature.
The controversy involving the EPA is a microcosm of the role web archiving has in permitting knowledge stakeholders to operate. It encapsulates how webpages can disappear or be altered without warning or trace, leading to the erasure of data the public needs to be able to inform themselves to hold democratic institutions accountable [4]. Furthermore, scientists and historians build upon the research of those before them. When primary sources are lost, doubt can be cast on their findings as they cannot be reproduced [5]. Multipliers such as policymakers and educators are then hampered in enacting effective change [5, 6].
Technologies employed by the Internet Archive and similar initiatives are a field of active research. A suite of tools, formats, protocols, and technologies have been developed for web archival that this essay will explore. The imperative of safeguarding our digital legacy will be demonstrated by delving into these multifaceted approaches, starting with the basic architecture of web crawlers, critical for the historical accuracy of impermanent web content, in Section 2. This continuity of knowledge is enabled by a standardized file format, whose specification is covered in Section 3 and was used to store the previously mentioned EPA records for governance oversight. Limitations arising from the complex demands of archiving dynamic content are discussed in Section 4. Section 5 shows how the advent of social media platforms restricting open access further aggravated the challenge of this task. Section 6 then concludes this essay with a brief overview and outlook.
Web Crawlers
Web crawlers are automated programs designed to traverse the World Wide Web and capture web pages, media files, and other forms of digital content. One of the most prominent applications of web crawling in the context of web archiving is executed by the Internet Archive. Their open-source web crawler, Heritrix, maps out the internet from a starting set of seed URIs. A snapshot is taken for each web page before recursively following its hyperlinks to find additional resources [7].
Primary components of Heritrix’s implementation include the Frontier [8] and the Fetcher [9]. The Frontier manages the queue of URIs to be explored. The hyperlinks discovered when scanning the page are at the front of the queue. They are released for probing in order; the first element is dequeued at appropriate times to ensure politeness policies are followed. These guidelines are the wishes of webmasters who want to limit or prevent bot requests as they may overwhelm their servers.
Directives are kept in a robots.txt file that tells crawlers like Heritrix how to behave. “Crawl-delay” is widely recognized and respected as it specifies how long to wait between successive actions but is not part of the robots.txt specification. Interestingly, despite being a de-facto Internet standard, the Internet Engineering Task Force has yet to ratify this robot exclusion protocol [10].
The Fetcher is responsible for downloading the web content to produce a memento, i.e., the original resource as it exists at the time of the snapshot. It retrieves web pages by sending HTTP requests and saves responses in the “Web ARChive” (WARC) format without running JavaScript code or rendering visuals [6].
Users usually interact with the Internet Archive through the Wayback Machine [11] to avoid technical hurdles in configuring and decompressing huge backups such as the EPA’s. It allows them to re-experience archived web pages by replaying mementos stored as WARC files.
The Wayback Machine has indexed the personal homepage of chess grandmaster Bobby Fischer [12]. The hundreds of snapshots from October 2001 until he died in 2008 preserve tournament game commentary aired on radio interviews and the promotion of a chess variant he invented and is played officially today. Furthermore, it hosts a letter from the USA’s Treasury Department outlining his ban from playing the 1992 World Championship match, which he played and won nonetheless — leading to a warrant for his arrest being issued. Scrolling through the years on the Wayback Machine showcases his tragic demise to mental illness and paranoia. Combined with the other artifacts, the record-keeping of this unfortunate process from such a private perspective provides an invaluable insight into an individual’s life with historical and geopolitical significance.
The WARC Format
Unlike the Robot Exclusion Protocol, which is not officially specified by any norms body, the WARC format is specified in ISO 28500 [13]. It builds upon the ARC format the Internet Archive initially used to store web crawls. WARC is intended as the sole standard format for web archives but has grown beyond that to keep digital or digitized materials in general.
From a technical perspective, WARC is a container for concatenated data objects in a single file, forming multiple resource records. Each record contains a WARC header and the corresponding payload whose type is arbitrary. The header describes the associated content through key-value pairs, such as fields for the kind of record, the timestamp of capture, and the content length [13]. The payload contains the actual archived content: for an HTTP request, it includes query parameters sent to the contacted website, with the response record holding HTML markup and media. Hence, the information and its surrounding context are preserved for future analyses. Doing so is crucial as the nature of an artifact is essential to its preservation; the ephemeral dimensions of a web page must be documented as they constitute part of the work [14, 15]. Bobby Fischer’s homepage has a unique 2000s web page feel and aesthetic retrievable by the Wayback Machine that plain text alone cannot convey.
A best practice to facilitate the management of WARC files is to ensure WARC file sizes take at most 1 GB of space. These can be grouped into more extensive collections for bulk operations. Storage space should be kept low by GZIPping archives [13]. Moreover, WARC files are often accompanied by an index file, permitting faster seek times and random access, thereby obviating the need to scan the entire archive for specific information. Additional metadata in the WARC file, such as checksums on top of institutional practices regarding naming conventions, prevent duplicate aggregate WARC files from being created [13].
Archiving the Dynamic Web
With the advent of dynamic web technologies, archiving web content has become a more intricate problem than simply saving static HTML pages as web applications replace them. This JavaScript-fueled development messes with the reasonable expectation that mementos should function as they did when they were initially archived [16]. Extracted URIs must be rewritten by the replay software to display mementos accurately so that the hostname points to the archive rather than the original server [6].
For example, the JavaScript code archived during a Heritrix crawl may contain functions that download specific fonts for the user interface. The font will show during replay in the Wayback Machine if this URI is detected, parsed, and replaced correctly. However, if the link is programmatically generated, these URIs either do not load or resolve to an existing live resource, questioning the archive’s fidelity. These so-called zombies can create temporal inconsistencies, such as showing a 2012 advertisement for an American presidential candidate debate between Barack Obama and Mitt Romney in a CNN memento of the 2008 campaign [16].
The situation is exacerbated by the widespread adoption of Ajax, which dynamically inserts content into a website from a database or server given user inputs or backend events [16] — only the necessary component refreshes, such as a chat box, upon a new incoming message. The top-level URI stays constant despite the additions to the application, making such web elements elusive targets for traditional crawling and capturing tools like Heritrix.
New approaches have been developed that focus on simulating user input with headless browsing technologies while recording network traffic, addressing these issues to construct reproducible web corpora [17]. Despite the advances in tooling, the everevolving nature of the web makes it a moving target for archivists. Algorithms and techniques must continuously adapt to the newest advancements in web technologies, such as the rise of social media.
Archiving Social Media
The contemporary social media landscape is increasingly characterized by a shift from archival practices toward ephemeral communications. Platforms like Snapchat catalyzed this transition by enabling users to share media visible only once for 10 seconds. The “stories” feature, i.e., posts that disappear after 24 hours, was subsequently emulated by Meta’s family of applications [18] and other messaging platforms like Telegram and Signal. Archiving such ephemeral content presents challenges that amplify the complexities of preserving dynamic web data discussed in the previous section.
As data became an asset, social media platforms have become increasingly insular, favoring access through proprietary mobile applications rather than web browsers. For the minority that accommodates web-based access, imposing stringent API restrictions and other access barriers is becoming the norm. For instance, Pinterest restricts image viewing to registered users, while Quora requires account registration to access threads and follow-up responses fully. Such practices impede content discovery by search engines, rendering results invisible as pages are banished to the deep web. Google users started appending “Reddit” to their queries to filter out locked-down results.
Get Daniel Petri’s stories in your inbox
Join Medium for free to get updates from this writer.
Nevertheless, even Reddit — once a bastion for open, text-based forum-like communication — tightened its API policies in response to large language models such as ChatGPT using their data for training [19]. The change resulted in a knock-on effect that forced third-party applications to shutter. Protests ensued in which subreddit moderators temporarily closed their communities against these policies but ultimately failed [20]. Blogging- and microblogging platforms, such as Medium, Substack, and X, have also followed this trajectory of increased restrictiveness.
In light of these developments, systematic archiving methodologies for social media must be fostered. The historical precedent set by the collapse of MySpace, Vine, and Orkut, which once had massive user bases, shows that current platforms will become obsolete, too. Therefore, the necessity for archiving extends beyond social media and encapsulates the need for an open internet more broadly, given its ever-evolving nature.
Conclusion
Throughout this essay, instances in which web archiving created actual value have been provided. In maintaining public records, as in the EPA example, and preserving personal histories, as seen in Bobby Fischer’s page, the importance of this field in preserving our digital heritage was underscored. The Heritrix crawler and the WARC format are the backbone for the Internet Archive, the Wayback Machine, and web archiving in general. These technologies, however, could be better, especially when it comes to dynamic content: the execution of old code embedded in mementos may retrieve modern information that is incongruous with the snapshot’s timestamp. Furthermore, as social media emphasizes ephemeral, locked-down media, digital archivists’ struggles worsen. This trend is unlikely to reverse, given that the clear incentive structure can withstand considerable backlash, as seen with Redditors’ pressure campaign against API fees. The need for archiving, however, remains, with vast swathes of narratives already being lost after the disappearance of social media giants like MySpace. Ensuring an internet record is as complete and accurate as possible is not just a technical challenge but a cultural imperative currently being cast aside.
References
[1] L. Friedman. “E.P.A. Scrubs a Climate Website of ‘Climate Change’.”
[2] C. Davenport. “How Much Has ‘Climate Change’ Been Scrubbed From Federal Websites? A Lot.”
[3] “Environmental Protection Agency Mirror.” Internet Archive.
[5] P. Buneman, S. Khanna, K. Tajima, and W.-C. Tan, “Archiving scientific data,” ACM Trans. Database Syst., vol. 29, no. 1, pp. 2–42, 2004, doi: 10.1145/974750.974752.
[6] J. Berlin, M. Kelly, M. L. Nelson, and M. C. Weigle, “To Re-experience the Web: A Framework for the Transformation and Replay of Archived Web Pages,” ACM Trans. Web, vol. 17, no. 4, p. Article 28, 2023, doi: 10.1145/3589206.
[7] “Heritrix3.” Internet Archive.
[8] “Heritrix3 Documentation: Frontier.” Internet Archive.
[9] “Heritrix 3 FetchHTTP API.” Internet Archive.
[11] M. Kelly and M. Weigle, “WARCreate: create wayback-consumable WARC files from any webpage,” presented at the Proceedings of the 12th ACM/IEEECS joint conference on Digital Libraries, Washington, DC, USA, 2012. [Online].
[12] R. J. Fischer. “Personal Homepage.”
[13] WARC file format, ISO, 2017. [Online].
[14] V. Pignagnoli and R. Sapino, “Introduction: Ephemerality in the digital age — creating, preserving, and sharing practices across media,” Neohelicon, vol. 48, no. 1, pp. 1–6, 2021/06/01 2021, doi: 10.1007/s11059–021–00591–0.
[15] P. Bootz, “The ephemerae of digital literature,” Neohelicon, vol. 48, no. 1, pp. 7- 22, 2021/06/01 2021, doi: 10.1007/s11059–021–00583–0.
[16] J. F. Brunelle, M. Kelly, M. C. Weigle, and M. L. Nelson, “The impact of JavaScript on archivability,” International Journal on Digital Libraries, vol. 17, no. 2, pp. 95–117, 2016/06/01 2016, doi: 10.1007/s00799–015–0140–8.
[17] J. Kiesel, F. Kneist, M. Alshomary, B. Stein, M. Hagen, and M. Potthast, “Reproducible Web Corpora: Interactive Archiving with Automatic Quality Assessment,” J. Data and Information Quality, vol. 10, no. 4, p. Article 17, 2018, doi: 10.1145/3239574.
[18] M. Isaac. “Instagram Takes a Page From Snapchat, and Takes Aim at It, Too.”
[19] M. Isaac. “Reddit Wants to Get Paid for Helping to Teach Big A.I. Systems.”
[20] M. M. Kircher. “What’s Going On With Reddit?”
All links were last accessed on December 8th, 2023.
