Since 2014 the Documenting the Now project has worked to develop tools and cultivate a community of practice for collecting social media. Documenting the Now began as a collective memory project amidst the protests following the murder of Mike Brown in Ferguson. Informed by early work studying the political and social significance of social media (Freelon, McIlwain & Clark, 2016) the Documenting the Now project hosted workshops, a Slack forum for discussing how to collect social media data, and developed several specialized tools specifically for Twitter, including:

• twarc: for collecting data from the Twitter v1.1, v2 and Enterprise APIs
• Catalog: a clearinghouse for sharing tweet id datasets
• Hydrator: a desktop application for converting tweet id datasets into JSON and CSV
• DocNow: a web application for collecting Twitter data and obtaining consent from content creators

According to Google Scholar, twarc has been cited over 500 times as a tool for collecting and hydrating Twitter datasets. It has been starred 1,300 times and forked 250 times on GitHub as it has been part of other software tools such as the Social Feed Manager. Numerous guides have been independently prepared by data scientists, Software Carpentry, and even Twitter themselves as part of their Twitter Research outreach. Enhanced access under Twitter’s Academic Research Product Track enabled searching the full archive of public tweets back to Twitter’s beginning in 2006.

After the acquisition of Twitter by Elon Musk, the changes to the Twitter API have resulted in the complete dismantling of DocNow’s tools. While it still technically functions (if you still have API keys), twarc is basically useless unless you want to pay $100/month to retrieve 10,000 tweets ($0.01 per tweet), or $5000/month for up to 1 million tweets. These prices and the new policies around deletion of Twitter data are designed for commercial use, and to terminate academic research and study of the platform. The DocNow application, which was built around Academic Research Product Track quotas and historical search no longer functions with the new API regime. Twitter has revoked the keys for Hydrator and the instance we have been running at community.docnow.io.

The truth is that the writing has been on the wall for access to corporate social media platform APIs for some time (Freelon, 2018 ; Bruns, 2019). Twitter and Reddit have been anomalies more than examples of the norm when it comes to access to platform data. Scraping practices are widespread, but are fraught with fragility due to interface changes, lack of documentation, and also present major reproducibility and provenance challenges. While social media APIs have typically been layered on top of the web, their key management, authentication, URL routes, and data representations have required the development of specialized tooling, that also has fragmented methods and diminished cooperation amongst projects.

With these challenges, and the shuttering of API access except for the most powerful, we believe that the social media and web research community needs to embrace the use of general-purpose web archiving technology and practices to preserve and study social media platforms and the web. As divergent as they are most platforms still have some kind of web surface as an interface. Service providers such as the Internet Archive’s Wayback Machine, and tools such as those provided by the Webrecorder project need support from the research community. Tools for analyzing web archive data such as those by the Archives Unleashed Project and specialized methodological approaches to using WARC data in research (Proferes, 2019) need to be further developed and socialized. Expertise in using WARC data needs to become more commonplace through curriculum development, training, and community workshops.

In addition to supporting existing tools and service providers, we should foster a more diverse community of people who can use web and social media archiving tools to create data collections. A more diverse set of users beyond academic researchers and the handful of archivists at academic and national libraries, will allow us to grow the number of collections available for research. A more diverse set of users of these tools also creates the possibility to envision new types of tools and practices for collecting this data, brings new meanings to what is valuable as web data, and builds a more inclusive historical record. Documenting the Now and the Archiving the Black Web project will be working more towards this community-building and education approach in the coming years.

Some aspects of traditional archiving practices also present a way forward, for example, working with individuals to donate their collections (or social media data), respecting an individual’s right to consent to have their information collected, building publicly accessible digital repositories of archival materials, and collaborative collection development, are useful practices and can help us imagine new ways forward for working with social media data in this post open API era.

No matter the path, the preservation of social media and the web will require not only our willingness to collaborate and share but also our recognition that its preservation is our responsibility. These platforms will not protect us in the physical world or the worlds we’ve created online.

References

Bruns, A. (2019). After the `APIcalypse’: Social media platforms and their fight against critical scholarly research. Information, Communication & Society. https://www.tandfonline.com/doi/abs/10.1080/1369118X.2019.1637447

Freelon, D. (2018). Computational research in the post-API age. Political Communication. https://www.tandfonline.com/doi/abs/10.1080/10584609.2018.1477506

Freelon, D., McIlwain, C. D., & Clark, M. D. (2016). Beyond the Hashtags: #Ferguson, #BlackLivesMatter, and the struggle for online justice. Center for Social Media and Impact. http://www.cmsimpact.org/sites/default/files/beyond _the _hashtags _2016.pdf

Proferes, N., & Summers, E. (2019). Algorithms and agenda-setting in Wikileaks’ #Podestaemails release. Information, Communication & Society, 22(11), 1630–1645. https://doi.org/10.1080/1369118X.2019.1626469

Looking Forwards was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today the Documenting the Now project announces the release of DocNow, an application for appraising, collecting, and gathering consent for Twitter content. DocNow reimagines the relationship between content creators and social media analysts (archivists and researchers) by addressing two of the most challenging issues of social media archiving practice — the issues of consent and appraisal. The Documenting the Now Project is happy to release version 1.0 of our open-source tool freely for anyone to use. Read all about the app, what it does, and how to engage with the DocNow project team for support and providing feedback.

Over the last seven years, Documenting the Now has helped to foster an environment where a more broad sector of cultural memory workers can learn about web archiving tools and practices and can become involved with web archiving networks. This has largely been achieved by practicing transparency and inviting people who have traditionally been left out of established web content archiving networks into the project to participate, namely students, activists, and archivists who represent marginalized communities and who work in community-centered organizations, HBCUs, public libraries, community-based archives, and tribal libraries and archives

Documenting the Now was a response to the need among scholars, activists, archivists, and other memory workers for new tools that would provide easily-accessible and user-friendly means to collect, visualize, analyze, and preserve web and social media content to better document public events. In addition, it aimed to respond to questions and concerns related to ethics, safety, intellectual property, and access issues for the collection, preservation, and dissemination of Twitter data in particular.

Documenting the Now has also developed community-centered web and social media archiving tools that both prioritize care for content creators and robust functionality for users:

• Twarc — a command-line tool and Python library for collecting tweet data from Twitter’s official API
• Hydrator — a desktop application for turning Tweet ID datasets back into tweet data to use in your research
• Social Humans — a label system to specify the terms of consent for social media content
• The Catalog — a community-sourced clearinghouse to access and share tweet identifier datasets

In continuing to support and develop tools that embody ethical practices for social media archiving, the DocNow app joins this suite of tools. DocNow is an application for appraising, collecting, and gathering consent for Twitter content and includes several new features including:

• Trends tab to view trending topics across the globe in real-time
• Explore tab to view content by users, media, URLs, and related hashtags all on one screen
• Live testing and refining of collecting parameters on recent tweets
• Tweets per hour calculator to easily identify Twitter bot accounts
• Search and Collect tweets back in time via Search API and forwards with Stream API
• Activate toggle to start collecting tweets and send a notification tweet to encourage transparency and communication in Twitter data collection
• Collections tab to share information about your collection with the public
• “Find Me” and Insights Overview features to specify and gather consent using Social Humans labels
• Download Tweet ID archive for sharing following Twitter’s terms of service

The DocNow app also works in concert with other Documenting the Now tools, creating for users, a 4-step social media archiving journey:

Step 1: Collect content with the DocNow App by activating a search. Set collection limits and explore insights as your collection grows.

Step 2: Download your archive from the DocNow App, which includes a Tweet Viewer, Tweet IDs, and media files.

Step 3: Hydrate your Tweet IDs from the archive’s tweets.csv file back into full tweets using DocNow’s Hydrator desktop application.

Step 4: Describe your collection and share your Tweet IDs with other researchers by adding them to the DocNow Catalog.

Ways to Use DocNow

There are 3 different ways to use DocNow including joining the community instance, running DocNow locally on a computer, and installing an instance of DocNow in the cloud. The Community Instance is a great way to get familiar with the tool before committing to running an instance but those with development skills may want to administer their own instance of DocNow locally or in the cloud.

1. Join Documenting the Now’s hosted community instance
2. Run DocNow locally on your machine
3. Install your own instance in the cloud

For help with installation and getting started, the Documenting the Now team will host community conversations. Dates will be announced soon! More information about the DocNow App can be found at https://www.docnow.io/docnow-app/.

Documenting the Now is seeking community input on all of our features as we continue to develop DocNow. Please join our slack channel by going to our website at docnow.io or email us at info@docnow.io. We welcome all feedback!

Introducing DocNow 1.0 was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today Twitter released a new API endpoint that allows you to fetch the number of tweets that match a query (like a hashtag) over time.

… we’re excited to launch two new endpoints, recent Tweet counts and full-archive Tweet counts, to the Twitter API v2. These endpoints are valuable for a number of reasons, but most commonly, to understand the size of the conversation, or the amount of data a query will return, prior to submitting a search request.

Thanks to some light coordination between Twitter and software developer Igor Brigadir a new version of the twarc utility was released today with a new command for collecting data from the counts endpoint.

This is significant news for researchers because these aggregate statistics can be critical for data gathering activities, and these numbers were previously really only known to Twitter themselves.

For example, it’s now possible to use a single twarc command to get the number of #blacklivesmatter tweets per day since the very first tweet, and save it as a CSV file, that you can then use to create a graph of the hashtag’s usage over time:

This admittedly dull graph is really quite remarkable, because it clearly shows that on May 28, 2020 there was a huge increase in the number of #blacklivesmatter tweets. There’s really no need to look any closer at the data to realize what happened that day:

Video: George Floyd Protests Escalate in Minneapolis

The results are a bit surprising however because the total number of #blacklivesmatter tweets is 71,190,571. That’s a large number, but it’s not as big as you might expect.

Counting tweets that match #blacklivesmatter or #blm doesn’t change the shape of the graph too much, but it does result in almost five times as many tweets being returned: 302,401,621. It’s important to remember here that these counts do not include tweets that have since been deleted or protected. Depending on what you are counting this can be a significant factor when interpreting counts for particular events.

Once installed and configured the twarc command for collecting these counts is pretty simple. Here is the command for retrieving the raw hourly count data (JSON) from the API for blacklivesmatter over the last seven days:

$ twarc2 counts "blacklivesmatter OR blm" counts.jsonl

To collect the daily counts from the full archive and write the data as CSV instead of JSON a few more options are needed.

$ twarc2 counts "blacklivesmatter OR blm" counts.csv --archive --csv --granularity day 

For an example of how the simple graph above was created take a look at this Jupyter notebook.

Finally as a counterpoint consider this graph of tweets mentioning #maga:

There are a few peaks and valleys there, but zooming in on the largest of them yields this graph:

That peak is January 6, 2021.

Counting Tweets was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

From its early adopters in 2007 at SXSW to its use today by journalists around the world, Twitter has always been a social media platform for documenting events. In 2014 the Documenting the Now project itself got started around efforts to document the protests and activism following the murder of Michael Brown in Ferguson, Missouri. Twitter remains one of the most accessible sources of data for breaking events of various kinds.

The combination of Twitter’s new V2 API and their Academic Research Product Track means that Twitter is still an excellent source of information about events. But figuring out the best practices to document events can be challenging. So we were really pleased to see Ryan Gallagher’s short thread about the 4 steps he uses for using Twitter’s API to document events.

Ryan J. Gallagher on Twitter: "The Twitter API v2 for academics makes it a lot easier to collect complete event dataI currently have a 4 step process that I finally successfully ran end to end on a real event this week, so I thought it might be useful to share1/ / Twitter"

The Twitter API v2 for academics makes it a lot easier to collect complete event dataI currently have a 4 step process that I finally successfully ran end to end on a real event this week, so I thought it might be useful to share1/

This post is simply a gloss on Ryan’s thread which provides examples of how to perform each step with the new functionality we released in twarc2. But put on your snorkel and diving gear because we are headed to the command line. (Aside: f you are new to the command line and are looking for some help look for the links to the documentation and our Slack channel at the end of this post).

1. Collect Live Tweets

Ryan J. Gallagher on Twitter: "1. Try to use the filter stream to get as much of the data as possibleI think the 1% threshold still applies here (you can only stream all filtered tweets if they're <1% of all tweets at any time). If this is really trouble for you, you can just use the next step / Twitter"

1. Try to use the filter stream to get as much of the data as possibleI think the 1% threshold still applies here (you can only stream all filtered tweets if they're <1% of all tweets at any time).

If the event is ongoing and you know a few relevant hashtags or keywords you can use the filter stream API to collect tweets from the filter stream API as they happen. We did this recently for the Kenmure Street Protests where the hashtags #kenmurestreet and #kenmurest were being used. First we added the two keywords to our streaming rules:

$ twarc2 stream-rules add kenmurest
$ twarc2 stream-rules add kenmurestreet

And then started streaming the tweets to a file tweets.jsonl:

% twarc2 stream > tweets.jsonl

You can leave this running as long as you want to collect tweets: hours, days, months … we’ve run a twarc job for over a year before. The hard thing is keeping the computer turned on and connected to the Internet. Most of the time you are only interested in a few days or weeks which was the case for the #kenmurestreet data collection.

2. Collect Past Tweets

Ryan J. Gallagher on Twitter: "Also the completeness of your data will vary based on how quickly you're able to run the full archive searchNote: this is where you need to have the academic track / Twitter"

Also the completeness of your data will vary based on how quickly you're able to run the full archive searchNote: this is where you need to have the academic track

Unless you are planning to document an event before it happens you often need to go back and get older tweets that were already sent. If you note down when you start collecting the live tweets you can use this time in combination with when the event began to scope the data collection, for example:

$ twarc2 search 'kenmurest OR kenmurestreet' --start-time 2021-05-13 --end-time 2021-05-21 > tweets.jsonl

You can collect tweets from the last 7 days, but as Ryan says you will need access to the Academic Research product track to search further back. If you have access you can simply add the --archive option, but be careful to limit how much you are collecting with a --start-time or --limit (discussed below) or else you can use up your monthly quota!

$ twarc2 search --archive 'kenmurest OR kenmurestreet' --start-time 2021-01-01 --end-time 2021-05-21 > tweets.jsonl

3. Collect Conversations

Ryan J. Gallagher on Twitter: "3. Use all of the conversation IDs of all the tweets from the streaming and the search to get all their reply threadsThis is new with v2! There's no excuse anymore not to actually have the conversations that happen around events in your data / Twitter"

3. Use all of the conversation IDs of all the tweets from the streaming and the search to get all their reply threadsThis is new with v2! There's no excuse anymore not to actually have the conversations that happen around events in your data

Collecting hashtags and keywords can be a useful entry point into tweets about an event, but it will not collect responses to those tweets and all the threaded conversations that can happen. You can use theconversations command to collect all the conversation threads that were referenced in your initially collected data:

$ twarc2 conversations tweets.jsonl > conversations.jsonl

4. Collect Timelines

Ryan J. Gallagher on Twitter: "4. (optional) Collect the user timelines of those in your datasetThis is so that you have contextual data on what people were saying prior to a particular event. When we focus just on event data, we lose the bigger pictureBut a warning: / Twitter"

4. (optional) Collect the user timelines of those in your datasetThis is so that you have contextual data on what people were saying prior to a particular event. When we focus just on event data, we lose the bigger pictureBut a warning:

It can also be useful to see what users were doing while they were documenting an event. Collecting the user timelines of users can be one way of seeing what some of this ambient content andactivity might be.

$ twarc2 timelines tweets.jsonl > timelines.jsonl

If the event didn’t happen in the last 7 days you will need to use the historical archive:

$ twarc2 timelines --archive tweets.jsonl > timelines.jsonl

As Ryan notes depending on how many tweets you have collected you could end up putting a big dent in your monthly quota. You can use --limit and --conversation-limit to control how much data is returned. For example if you only wanted to collect the last 100 tweets of each user and no more than 25,000 timeline tweets in total:

% twarc2 timelines tweets.jsonl --limit 25000 --conversation-limit 100 > timelines.jsonl

5. Limits

Ryan J. Gallagher on Twitter: "One potential issue is that there isn't a way around with v2 is the 10,000,000 tweets/month cap. So this pipeline is only going to work on moderate sized events. Nothing something large or ongoing like the presidential election or national vaccine rollouts / Twitter"

One potential issue is that there isn't a way around with v2 is the 10,000,000 tweets/month cap. So this pipeline is only going to work on moderate sized events. Nothing something large or ongoing like the presidential election or national vaccine rollouts

Speaking of limits you can actually apply --limit to any stage of the process. So if you wanted to search for tweets, but no more than 50,000 you could have:

$ twarc2 search kenmurest --limit 50000 > search.jsonl

Or if you wanted to limit the number of conversation tweets by total tweets:

$ twarc2 conversations tweets.jsonl --limit 50000 > conversations.jsonl

You can also control how many tweets per conversation to collect. So for example if you only wanted to collect the first 100 tweets in each conversation:

$ twarc2 conversations tweets.jsonl --conversation-limit 100 > conversations.jsonl

Now What?

twarc will log what activities it is performing and when to a file named twarc.log This log can be useful for remembering what you’ve done along the way. Once you have collected the data you will no doubt wonder what to do with all the JSON data. While this is largely a topic for another post a lot of people want to turn the data into CSV for examination in Google Sheets or as an R or Pandas dataframe. To do that you can install the twarc-csv plugin and:

twarc2 csv tweets.jsonl > tweets.csv

If you have any questions about this please check out the twarc2 documentation or join us in the Documenting the Now Slack. Documenting the Now’s Ed Summers also did a recent presentation about the Twitter V2 API and twarc2 at the University of Maryland Social Data Science Center:

Documenting Events was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

twarc has been redesigned from the ground up to work with the new Twitter v2 API and their Academic Research track. Many thanks for the code and design contributions of Betsy Alpert, Igor Brigadir, Sam Hames, Jeff Sauer, and Daniel Verdeer that have made twarc2 possible, as well as early feedback from Dan Kerchner, Shane Lin, Miles McCain, 李荣蓬, David Thiel, Melanie Walsh and Laura Wrubel. Extra special thanks to the Institute for Future Environments at Queensland University of Technology for supporting Betsy and Sam in their work, and for the continued support of the Mellon Foundation.

Update 2021–05–04: If you are interested in watching a discussion of how to install and use twarc2 on the TwitterDev Twitch check out this recording.

Back in August of last year Twitter announced early access to their new v2 API, and their plans to sunset the v1.1 API that has been active for almost the last 10 years. Over the lifetime of their v1.1 API Twitter has become deeply embedded in the media landscape. As magazines, newspapers and television have moved onto the web they have increasingly adopted tweets as a mechanism for citing politicians, celebrities and organizations, while also using them to document current events, generate leads and gather feedback for evolving stories. As a result Twitter has also become a popular object of study for humanities and social science researchers looking to understand the world as reflected, refracted and distorted by/in social media.

On the surface the v2 API update seems pretty insignificant since the shape of a tweet, its parts, properties and affordances, aren’t changing at all. Tweets with 280 characters of text, images and video will continue to be posted, retweeted and quoted. However behind the scenes the representation of a tweet as data, and the quotas that control the rates at which this data can flow between apps and other third party services will be greatly transformed.

Needless to say, v2 represents a big change for the Documenting the Now project. Along with community members we’ve developed and maintained open source tools like twarc that talk directly to the Twitter API to help users to search for and collect live tweets that match criteria like hashtags, names and geographic locations. Today we’re excited to announce the release of twarc v2 which has been designed from the ground up to work with the v2 API and Twitter’s new Academic Research track.

Clearly it’s extremely problematic having a multi-national corporation act as a gatekeeper for who counts as an academic researcher, and what constitutes academic research. We need look no further than the recent experiences of Timnit Gebru and Margaret Mitchell at Google for an example of what happens when research questions run up against the business objectives of capital. We only know their stories because Gebru and Mitchell’s bravely took a principled approach, where many researchers would have knowingly or unknowingly shaped their research to better fit the needs of the company.

So it is important for us that twarc still be usable by people with and without access to the Academic Research Track. But we have heard from many users that the Academic Research Track presents new opportunities for Twitter data collection that are essential for researchers interested in the observability of social media platforms. Twitter is making a good faith effort to work with the academic research community, and we thought twarc should support it, even if big challenges lie ahead.

So why are people interested in the Academic Research Track? Once your application has been approved you are able to collect data from the full history of Tweets, at no cost. This is a massive improvement over the v1.1 access which was limited to a one week window and researchers had to pay for access. Access to the full archive means it’s now possible to study events that have happened in the past back to the beginning of Twitter in 2006. If you do create any historical datasets we’d love for you to share the tweet identifier datasets in The Catalog.

However this opening up of access on the one hand comes with a simultaneous contraction in terms of how much data can be collected at one time. The remainder of this post describes some of the details and the design decisions we have made with twarc2 to address them. If you would prefer to watch a quick introduction to using twarc v2 please check out this short video:

Installation

If you are familiar with installing twarc nothing is changed. You still install (or upgrade) with pip as you did before:

$ pip install --upgrade twarc

In fact you will still have full access to the v1.1 API just as you did before. So the old commands will continue to work as they did¹

$ twarc search blacklivesmatter > tweets.jsonl

twarc2 was designed to let you to continue to use Twitter’s v1.1 API undisturbed until it is finally turned off by Twitter, at which point the functionality will be removed from twarc. All the support for the v2 API is mediated by a new command line utility twarc2. For example to search for blacklivesmatter tweets and write them to a file tweets.jsonl *

$ twarc2 search blacklivesmatter > tweets.jsonl

All the usual twarc functionality such as searching for tweets, collecting live tweets from the streaming API endpoint, requesting user timelines and user metadata are all still there, twarc2 --help gives you the details. But while the interface looks the same there's quite a bit different going on behind the scenes.

Representation

Truth be told, there is no shortage of open source libraries and tools for interacting with the Twitter API. In the past twarc has made a bit of a name for itself by catering to a niche group of users who want a reliable, programmable way to collect the canonical JSON representation of a tweet. JavaScript Object Notation (JSON) is the language of Web APIs, and Twitter has kept its JSON representation of a tweet relatively stable over the years. Rather than making lots of decisions about the many ways you might want to collect, model and analyze tweets twarc has tried to do one thing and do it well (data collection) and get out of the way so that you can use (or create) the tools for putting this data to use.

But the JSON representation of a tweet in the Twitter v2 API is completely burst apart. The v2 base representation of a tweet is extremely lean and minimal, and just includes the text of the tweet its identifier and a handful of other things. All the details about the user who created the tweet, embedded media, and more are not included. Fortunately this information is still available, but the user needs to craft their API request to request tweets using a set of expansions that tell the Twitter API what additional entities to include. In addition for each expansion there are a set of field options to include that control what of these expansions is returned.

So rather than there being a single JSON representation of a tweet API users now have the ability to shape the data based on what they need, much like how GraphQL APIs work. This kind of makes you wonder why Twitter didn’t make their GraphQL API available. For specific use cases this customizability is very useful, but the mutability of the representation of a tweet presents challenges when collecting data for future use. If you didn’t request the right expansions or fields when collecting the data then you won’t be able to analyze that data later when doing your research.

To solve for this twarc2 has been designed to collect the richest possible representation for a tweet, by requesting all possible expansions and field combinations for tweets. See the expansions module for the details if you are interested. This takes a significant burden off of users to digest the API documentation, and craft the correct API requests themselves. In addition the twarc community will be monitoring the Twitter API documentation going forward to incorporate new expansions and fields as they will inevitably be added in the future.

Flattening

This is diving into the weeds a little bit, but it’s worth noting here that Twitter’s introduction of expansions allows data that was once duplicated across multiple tweets (such as user information, media, retweets, etc) to be included once per response from the API. This means that instead of seeing information about the user who created a tweet in the context of their tweet the user will be referenced using an identifier, and this identifier will map to user metadata in the outer envelope of the response.

It makes sense why Twitter have introduced expansions since it means in a set of 100 tweets from a given user the user information will just be included once rather than repeated 100 times, which means less data, less network traffic and less money. It’s even more significant when consider the large number of possible expansions. However this pass by-reference rather than by-value presents some challenges for stream based processing which expects each tweet to be self-contained.

For this reason we’ve introduce the idea of flattening the response data when persisting the JSON to disk. This means that tools and data pipelines that expect to operate on a stream of tweets can continue to do so. Since the representation of a tweet is so dependent on how data is requested we’ve taken the opportunity to introduce a small stanza of twarc specific metadata using the __twarc prefix.

This metadata records what API endpoint the data was requested from, and when. This information is critically important when interpreting the data, because some information about a tweet like its retweet and quote counts are constantly changing.

Data Flows

As mentioned above you can still collect tweets from the search and streaming API endpoints in a way that seems quite similar to the v1 API. The big changes however are the quotas associated with these endpoints which govern how much can be collected. These quotas control how many requests can be sent to Twitter in 15 minute intervals.

In fact these quotas are not much changed, but what’s new are app wide quotas that constrain how many tweets a given application (app) can collect every month. An app in this context is a piece of software (e.g. your twarc software) identified by unique API keys set up in the Twitter Developer Portal. The standard API access sets a 500,000 tweet per month limit. This is a huge change considering there were no monthly app limits before. If you get approved for the Academic Research track your app quota is increased to 10 million per month. This is markedly better but the achievable data volume is still nothing like the v1.1 API, as these graphs attempt to illustrate:

twarc2 will still observe the same rate limits, but once you’ve collected your portion for the month there’s not much that can be done, for that app at least.

Apart from the quotas Twitter’s streaming endpoint in v2 is substantially changed which impacts how users interact with twarc. Previously twarc users would be able to create up to to two connections to the filter stream API. This could be done by simply:

twarc filter obama > obama.jsonl

However in the Twitter v2 API only apps can connect to the filter stream, and they can only connect once. At first this seems like a major limitation but rather than creating a connection per query the v2 API allows you to build a set of rules for tweets to match, which in turns controls what tweets are included in the stream. This means you can collect for multiple types of queries at the same time, and the tweets will come back with a piece of metadata indicating what rule caused its inclusion.

This translates into a markedly different set of interactions at the command line for collecting from the stream where you first need to set your stream rules and then open a connection to fetch it.

twarc2 stream-rules add blacklivesmatter
twarc2 stream > tweets.jsonl

One useful side effect of this is that you can update the stream (add and remove rules) while the stream is in motion:

twarc2 stream-rules add blm

While you are limited by the API quota in terms of how many tweets you can collect, tweets are not “dropped on the floor” when the volume gets too high. Once upon a time the v1.1 filter stream was rumored to be rate limited when your stream exceeds 1% of the total volume of new tweets.

Plugins

In addition to twarc helping you collect tweets the GitHub repository has also been a place to collect a set of utilities for working with the data. For example there are scripts for extracting and unshortening urls, identifying suspended/deleted content, extracting videos, buiding wordclouds, putting tweets on maps, displaying network graph visualizations, counting hashtags, and more. These utilities all work like Unix filters where the input is a stream of tweets and the output varies depending on what the utility is doing, e.g. a Gephi file for a network visualization, or a folder of mp4 files for video extraction.

While this has worked well in general the kitchen sink approach has been difficult to manage from a configuration management perspective. Users have to download these scripts manually from GitHub or by cloning the repository. For some users this is fine, but it’s a bit of a barrier to entry for users who have just installed twarc with pip.

Furthermore these plugins often have their own dependencies which twarc itself does not. This lets twarc can stay pretty lean, and things like youtube_dl, NetworkX or Pandas can be installed by people that want to use utilities that need them. But since there is no way to install the utilities there isn’t a way to ensure that the dependencies are installed, which can lead to users needing to diagnose missing libraries themselves.

Finally the plugins have typically lacked their own tests. twarc’s test suite has really helped us track changes to the Twitter API and to make sure that it continues to operate properly as new functionality has been added. But nothing like this has existed for the utilities. We’ve noticed that over time some of them need updating. Also their command line arguments have drifted over time which can lead to some inconsistencies in how they are used.

So with twarc2 we’ve introduced the idea of plugins which extend the functionality of the twarc2 command, are distributed on PyPI separately from twarc, and exist in their own GitHub repositories where they can be developed and tested independently of twarc itself. This is all achieved through twarc2’s use of the click library and specifically click-plugins. So now if you would like to convert your collected tweets to CSV you can install the twarc-csv:

$ pip install twarc-csv
$ twarc2 search covid19 > covid19.jsonl
$ twarc2 csv covid19.jsonl > covid19.csv

Or if you want to extract embedded and referenced videos from tweets you can install twarc-videos which will write all the videos to a directory:

$ pip install twarc-videos
$ twarc2 videos covid19.jsonl --download-dir covid19-videos

You can write these plugins yourself and release them as needed. Check out the plugin reference implementation tweet-ids for a simple example to adapt. We’re still in the process of porting some of the most useful utilities over and would love to see ideas for new plugins. Check out the current list of twarc2 plugins and use the twarc issue tracker on GitHub to join the discussion.

You may notice from the list of plugins that twarc now (finally) has documentation on ReadTheDocs external from the documentation that was previously only available on GitHub. We got by with GitHub’s rendering of Markdown documents for a while, but GitHub’s boilerplate designed for developers can prove to be quite confusing for users who aren’t used to selectively ignoring it. ReadTheDocs allows us to manage the command line and API documentation for twarc, and to showcase the work that has gone into the Spanish, Japanese, Portuguese, Swedish, Swahili and Chinese translations.

Feedback

Thanks for reading this far! We hope you will give twarc2 a try. Let us know what you think either in comments here, in the DocNow Slack or over on GitHub.

✨ ✨ Happy twarcing! ✨ ✨ ✨

1. Windows users will want to indicate the output file using a second argument rather than redirecting output with >. See this page for details.

twarc2 was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Trump’s tweets are gone from twitter.com but still exist spectrally in various states all over the web. After profiting off of their distribution Twitter now has a responsibility to provide meaningful access to the Trump tweets as a read only archive.

So Trump’s Twitter account is gone. Finally. It’s strange to have had to wait until the waning days of his presidency to achieve this very small and simple act of holding him accountable to Twitter’s community guidelines…just like any other user of the platform.

Better late than never, especially since his misinformation and lies can continue to spread after he has left office.

But isn’t it painful to imagine what the last four years (or more) could have looked like if Twitter and the media at large had recognized their responsibility and acted sooner?

When Twitter suspended Trump’s account they didn’t simply freeze it and prevent him from sending more hateful messages. They flipped a switch that made all the tweets he has ever sent disappear from the web.

These are tweets that had real material consequences in the world. As despicable as Trump’s utterances have been, a complete and authentic record of them having existed is important for the history books, and for holding him to account.

Twitter's suspension of Donald Trump's account has also removed all of his thousands of tweets sent over the years. I personally find it useful as a reporter to be able to search through his tweets. They are an important part of the historical record. Where do they live now?

Where indeed? One hopes that they will end up in the National Archives (more on that in a moment). But depending on how you look at it, Trump’s tweets are everywhere.

Twitter removed Trump’s tweets from public view at twitter.com. But fortunately, as Shawn Jones notes, embedded tweets like the one above persist the tweet text into the HTML document itself. When a tweet is deleted from twitter.com the text stays behind elsewhere on the web like a residue, as evidence (that can be faked) of what was said and when.

It’s difficult to say whether this graceful degradation was an intentional design decision to make their content more resilient, or it was simply a function of Twitter wanting their content to begin rendering before their JavaScript had loaded and had a chance to emboss the page. But design intent isn’t really what matters here.

What does matter is the way this form of social media content degrades in the web commons. Kari Kraus calls this process “spectral wear”, where digital media “help mitigate privacy and surveillance concerns through figurative rather than quantitative displays, reflect and document patterns of use, and promote an ethics of care.” This spectral wear is a direct result of tweet embed practices that Twitter itself promulgates while simultaneously forbidding it Developer Terms of Service:

If Twitter Content is deleted, gains protected status, or is otherwise suspended, withheld, modified, or removed from the Twitter Applications (including removal of location information), you will make all reasonable efforts to delete or modify such Twitter Content (as applicable) as soon as possible.

Fortunately for history there has probably never been a more heavily copied social media content than Donald Trump’s tweets. We aren’t immediately dependent on twitter.com to make this content available because of the other other places on the web where it exists. What does this copying activity look like?

I intentionally used copied instead of archived above because the various representations of Trump’s tweets vary in terms of their coverage, and how they are being cared for.

Given their complicity in bringing Trump’s messages of division and hatred to a worldwide audience, while profiting off of them, Twitter now have a responsibility to provide as best a representation of this record for the public, and for history.

We know that the Trump administration have been collecting the @realDonaldTrump Twitter account, and plan to make it available on the web as part of their responsibilities under the Presidential Records Act:

The National Archives will receive, preserve, and provide public access to all official Trump Administration social media content, including deleted posts from @realDonaldTrump and @POTUS. The White House has been using an archiving tool that captures and preserves all content, in accordance with the Presidential Records Act and in consultation with National Archives officials. These records will be turned over to the National Archives beginning on January 20, 2021, and the President’s accounts will then be made available online at NARA’s newly established trumplibrary.gov website.

NARA is the logical place for these records to go. But it is unclear what shape these archival records will take. Sure the Library of Congress has (or had) its Twitter archive. It’s not at all clear if they are still adding to it. But even if they are LC probably hasn’t felt obligated to collect the records of an official from the Executive Branch, since they are firmly lodged in the Legislative. Then again they collect animated GIFs so, maybe?

Reading between the lines it appears that a third party service is being used to collect the social media content: possibly one of the several e-discovery tools like ArchiveSocial or Hanzo. It also looks like the Trump Administration themselves have entered into this contract, and at the end of its term (i.e. now) will extract their data and deliver it to NARA. Given their past behavior it’s not difficult to imagine the Trump administration not living up to this agreement in substantial ways.

This current process is a slight departure from the approach taken by the Obama administration. Obama initiated a process where platforms [migrate official accounts] to new accounts that were then managed going forward by NARA (Acker & Kriesberg, 2017).

We can see that this practice being used again on January 20, 2021 when Biden became President. But what is different is that Barack Obama retained ownership of his personal account @barackobama, which he continues to use. Arguably this is good because Trump used his Twitter account in a very different manner than Obama. This process is complicated further by the fact that Trump’s account has been removed from the web. Nevertheless, NARA has announced that they will be archiving Trump’s now deleted (or hidden) personal account.

A number of Trump administration officials, including President Trump, used personal accounts when conducting government business. The National Archives will make the social media content from those designated accounts publicly available as soon as possible.

The question remains, what representation should be used, and what is Twitter’s role in providing it?

Meanwhile there are online collections like The Trump Twitter Archive, the New York Times’ Complete List of Trump’s Twitter Insults, Propublica’s Politwoops and countless GitHub repositories of data which have collected Trump’s tweets. These tweets are used in a multitude of ways including things as absurd as a source for conducting trades on the stock market.

But seeing these tweets as they appeared in the browser, with associated metrics and comments is important. Of course you can go view the account in the Wayback Machine and browse around. But what if we wanted a list of all the Trump tweets? How many times were these tweets actually archived? How complete is the list?

After some experiments with the Internet Archive’s API it’s possible to get a picture of how the tweets from the @realDonaldTrump account have been archived there. There are a few wrinkles because a given tweet can have many different URL forms (e.g. tracking parameters in the URL query string). In addition just because there was a request to archive a URL for something that looks like a realDonaldTrump tweet URL doesn’t mean it resulted in a successful response. Success here means a 200 OK from twitter.com when resolving the URL. Factoring these issues into the analysis it appears the Wayback Machine contains (at least) 16,043,553 snapshots of Trump’s tweets:

https://twittter.com/realDonaldTrump/status/{tweet-id}

Of these millions of snapshots there appear to be 57,292 unique tweets. This roughly correlates with the 59K total tweets suggested by the last profile snapshots of the account. The maximum number of times in one day that his tweets were archived was 71,837 times on February 10, 2020. Here’s what the archive snapshots of Trump’s tweets look like over time (snapshots per week).

It is relatively easy to use the CSV export from the Trump Twitter Archive project to see what tweets they know about that the Internet Archive does not and vice-versa (for the details see the Jupyter notebook and SQLite database here).

It looks like there are 526 tweet IDs in the Trump Twitter Archive that are missing from the Internet Archive. But further examination shows that many of these are retweets, which in Twitter’s web interface, have sometimes redirected back to the original tweet (a 302 Found instead of a 200 OK). After removing these retweets to specifically look at Trump’s own tweets there are only 7 tweets in the Trump Twitter Archive that are missing from the Internet Archive. Of these 4 are in fact retweets that have been misclassified as tweets by the Trump Twitter Archive.

One of the three remaining is this one, which is identified in the Trump Twitter Archive as deleted, and wasn’t collected quick enough by the Internet Archive before it was deleted:

Roger Stone was targeted by an illegal Witch Hunt tha never should have taken place. It is the other side that are criminals, including the fact that Biden and Obama illegally spied on my campaign — AND GOT CAUGHT!”

Sure enough, over at the Politwoops project you can see that this tweet was deleted 47 seconds after it was sent:

Turning the table it’s also possible to look at what tweets are in the Internet Archive but not in the Trump Twitter Archive. It turns out that there are 3,592 tweet identifiers in the Wayback machine for Trump’s tweets which do not appear in the Trump Twitter Archive. Looking a bit closer we can see that some are clearly wrong, because the id itself is too small a number, or too large. And then looking at some of the snapshots it appears that they often don’t resolve, and simply display a “Something went wrong” message:

Yes, something definitely went wrong (in more ways than one). Just spot checking a few there also appear to be some legit tweets in the Wayback that are not in the Trump Twitter Archive like this one:

Notice how the media will not play? These tweets appear to be in the Internet Archive, but it’s not at all certain what their state is. It would take some heavy manual curation work to sort through these tweet IDs to see which ones are legit, and which ones aren’t. But if you are interested here’s an editable Google Sheet of the tweets in Wayback that don’t appear in The Trump Archive. The situation gets even more complicated because there appear to be 67 deleted realDonaldTrump tweets in Politwoops that are not in the Internet Archive.

Finally, here is a list of the top ten archived (at Internet Archive) tweets. The counts here reflect all the variations for a given tweet URL. So they will very likely not match the count you see in the Wayback Machine, which is for the specific URL (no query paramters).

• Thank you Alabama! #Trump2016#SuperTuesday 65,489
• MAKE AMERICA GREAT AGAIN! 65,360
• Thank you Georgia!#SuperTuesday #Trump2016 65,358
• Such a beautiful and important evening! The forgotten man and woman will never be forgotten again. We will all come together as never before 55,426
• Watching the returns at 9:45pm. #ElectionNight #MAGA🇺🇸 https://t.co/HfuJeRZbod 54,889
• #ElectionDay https://t.co/MXrAxYnTjY https://t.co/FZhOncih21 54,291
• I will be watching the election results from Trump Tower in Manhattan with my family and friends. Very exciting! 54,263
• I love you North Carolina- thank you for your amazing support! Get out and https://t.co/HfihPERFgZ tomorrow!Watch:… https://t.co/jZzfqUZNYh 54,252
• Still time to #VoteTrump! #iVoted #ElectionNight https://t.co/UZtYAY1Ba6 54,224
• Watching my beautiful wife, Melania, speak about our love of country and family. We will make you all very proud…. https://t.co/DiKmSnTlC2 54100

The point of this rambling data spelunking, if you’ve made it this far, is to highlight the degree to which Trump’s tweets have been archived (or collected), and how the completeness and quality of those representations is very fluid and difficult to ascertain. Hopefully Twitter is working with NARA to provide as complete a picture as possible of what Trump said on Twitter.

As much as we would like to forget, we must not.

References

Acker, A., & Kriesberg, A. (2017). Tweets may be archived: Civic engagement, digital preservation and obama white house social media data. Proceedings of the Association for Information Science and Technology, 54(1), 1–9.

Kraus, K. (2019). The care of enchanted things. In M. K. Gold & L. F. Klein (Eds.), Debates in the digital humanities 2019. Retrieved from https://www.jstor.org/stable/10.5749/j.ctvg251hk.17

Thanks to Jess Ogden, Shawn Walker, Adam Kriesberg and Derek Willis who read versions of this post and provided feedback/ideas. All the mistakes are my own.

Originally published at https://inkdroid.org.

Trump’s Tweets was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Ilya Kreymer recently asked a question over in Documenting the Now Slack about Twitter’s API data and whether it includes metadata for when the tweet has been labeled as disinformation. This structured data is important for building tools that help trace how disinformation is propagating in Twitter and social media more generally. It also can provide a view into how Twitter themselves are working to combat the problem.

I’ve looked for the disinformation label in Twitter API JSON in the past and not seen it. But I figured it couldn’t hurt to look again so, I used this tweet as an example. It’s a snap to fetch the JSON data for a tweet with twarc:

$ twarc tweet 1297495295266357248 > tweet.json

I’ve included the retrieved data below. I don’t see anything related to the label, do you?

I also took the opportunity to look at Twitter’s new v2 API and see if the twet looks any different there. Twarc doesn’t support the v2 API yet so I hand rolled a little program to talk to the v2 Twitter API:

More about the options in a moment. This program fetched this representation of the same tweet:

Again, I still don’t see any information about the content warning, but maybe I wasn’t squinting right? I did see that according to the data that Donald Trump is in the class of Person who are “Named people in the world like Nelson Mandela”. I mean yes, but NO.

So about the options. Unlike the v1.1 API when using the v2 API you need to indicate in the request what fields you would like to have returned in the response. There are a set of names for the types of fields, such as media.fields, tweet.fields, place.fields, poll.fields and user.fields. Each of these field types has an enumerated set of associated values like duration_minutes for a poll, or context_annotations for a tweet, etc. Think of this as some kind of strange alternative to GraphQL.

There are lots of these enumerated values to choose from so I started by simply requesting all of them. Interestingly this failed, and the error message I received indicated that I had requested field values that required elevated privileges. Once I removed these from the request I was able to get back the JSON I pasted above.

The little error message actually provided a small glimpse of what data Twitter does and doesn’t provide to “regular” developer accounts through the v2 API. For example I had to remove non_public_metrics from media_fields and tweet_fields because the following fields required additional permissions:

• non_public_metrics.impression_count
• non_public_metrics.url_link_clicks
• non_public_metrics.user_profile_clicks

I guess that makes sense since they are non-public. Similarly I had to remove organic_metrics from media.fields and tweet.fields because the following fields required additional permissions:

• organic_metrics.impression_count
• organic_metrics.like_count
• organic_metrics.reply_count
• organic_metrics.retweet_count
• organic_metrics.url_link_clicks
• organic_metrics.user_profile_clicks

And finally I had to remove promoted_metrics from media.fields and tweet.fields because the following fields required additional permissions:

• promoted_metrics.impression_count
• promoted_metrics.like_count
• promoted_metrics.reply_count
• promoted_metrics.retweet_count
• promoted_metrics.url_link_clicks
• promoted_metrics.user_profile_clicks

I guess these are metrics for advertisers who are paying to have their tweets slipped into our timelines. Many of these seem to be present in the v1.1 Metrics API but it’s interesting that the new API is folding that functionality in to the representation of tweets. I did notice that the public_metrics counts were all zero. As much as I might wish this to be true for this particular tweet I know it’s probably just a bug that Twitter are working on.

I think this highlights one of the major disadvantages to collecting the JSON representation of a tweet. Some information is passed by reference, such as URLs for images and videos. Those need to be fetched in order to understand a tweet.

But some information is simply not included, such as this disinformation label, which must exist somewhere in Twitter’s infrastructure. The same is true for reply threads to a tweet — although this might get easier with the v2 API’s support for threads. And lastly the JSON representation changes. It has been remarkably stable for the past 10 years or so, with small things added here and there (e.g. 280 characters instead of 140). v2 marks a significant change. Where stream based processing of tweets will require substantial changes to keep working.

Returning to the topic of disinformation, for the moment it might be useful to add support toa scraping tool like twint to see if these disinformation labels could be pulled out of the page and serialized into the JSON and CSV it generates. Scraping is against Twitter’s terms of service, but this information is so important for public discourse the research and archival communities are left with little other choice.

Originally published at https://inkdroid.org.

Disinformation Metadata was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dianne Weinthal is a recent graduate of the UCLA MLIS program, and has been working with a few members of her cohort on criteria to evaluate, circulate, and create social media in service of safe social action.

Their first platform focus has been on Instagram. We thought this could be a good opportunity for us in the Documenting the Now community to gather in an informal call to talk about the thinking that went into the work they have done so far. This is especially important as so much of the Documenting the Now’s work to date has focused on Twitter. While we expect some issues to remain constant across social media archiving platforms, others will differ since the affordances and algorithmic effects of these platforms are varied.

Their draft document they have been working on can be found at:

https://docs.google.com/document/d/1L2WoH4-5u2Ryf78AJabO486cURE5cWyZdbTIpKqSIvM/edit?usp=sharing

We will be meeting on August 11 at 9–10AM PT. If you would like to participate please email info@docnow.io with the subject “Instagram” and we will email you the Zoom link to join.

Instagram, Social Action and Archives was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Friday, June 19th, Documenting the Now hosted a live streamed conversation with WITNESS, The Blackivists, Texas After Violence Project, and Project STAND. We are so thankful to the 800+ attendees and to our amazing panelists: Tracy Drake, Raquel Flores-Clemons, Erin Glasco, Stacie Williams, Skyla Hearn, Steven Booth, Gabriel Solis, Yvonne Ng, Lae’l Hughes-Watkins, and Jessica Ballard. Also much thanks to the Documenting the Now team members who helped put the event together: Alexandra Dolan-Mescal (who also designed the social media graphic), Francis Kayiwa, and Ed Summers.

Below is a recording of the conversation or you can hop on over to YouTube.

This is the original event description:

The recent wave of protests sparked by the death of George Floyd, Breonna Taylor, and so many others, have been inspiring and have renewed calls to dismantle the racist systems that inflict so much harm on Black people and other marginalized people. While the protests seemed to have ignited a shift in public opinion towards finally doing what is right, the violent reaction from law enforcement to the protesters has been disturbing and yet an unsurprising aspect of the recent events. As archivists, activists, and other memory workers attempt to document these protests, it is vital to consider how protestors can be protected from further harm by police while they are taking part in actions or when information about them is found in archives at a later date. In this conversation, Documenting the Now, WITNESS, Texas After Violence Project, Blackivists, and Project STAND will discuss actions that protestors and memory workers can take and resources they can use to safely and ethically document the extreme police violence we are seeing towards protestors.

These are some of the resources that were share by our panelists during the conversation:

Documenting the Now

• DocNow Digital Blackness Symposium: Panel- The Ferguson Effect on Local Activism and Community Memory
• Documenting the Now White Paper on Ethics and Social Media Archiving
• Archivists Supporting Activists
• Join DocNow Slack
• Documenting Activism Now

The Blackivists

• Five Tips for Organizers, Protestors, and Anyone Documenting Movements
• The Blackivists on Documenting Movements

Texas After Violence Project

• Documenting Narratives of Violence: Mitigating Risks
• Documenting Narratives of Violence: Trauma-Informed Interviewing

WITNESS

• Filming Police Violence in the United States resources
• People’s Database for Community-Based Police Accountability: A Berkeley Copwatch / WITNESS Initiative
• Profiling the Police (El Grito de Sunset Park & WITNESS)
• How To Film Police (US)
• I Shot A Video Of Police Brutality During A Protest. Should I Post It Online?
• Eyes on ICE: Documenting Abuses Against Immigrant Communities resources
• Real Or Rumor: How To Verify Online Reports Of ICE Raids
• Legal Video Advocacy resources
• Plano De Coleta — Operações Policiais E Militares Em Favelas Ou Áreas Periféricas (Portuguese)
• Activists’ Guide to Archiving Video
• Video as Evidence Field Guide

Project STAND

• Archiving Student Activism Toolkit
• S.A.V.E Method
• College and University Responses/Reaction
• Presentation-Prezi Link

Archiving Protests, Protecting Activists was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve been following our work for the past few years you may have occasionally seen us mention something called the Catalog. The Catalog is a clearinghouse for Twitter datasets, that has been built by a small but growing community of researchers, archivists and librarians who recognize that social media platforms like Twitter provide essential documentary evidence of our times.

For a variety of reasons Twitter’s Terms of Service don’t allow data collected from their APIs to be published on the public web. But they do allow, and even encourage, researchers to share Tweet Identifier (ID) datasets, which can then be reconstituted (or hydrated) as data using Twitter’s API. We’ve developed a desktop application called the Hydrator, which makes it easy to perform this process of turning a dataset of tweet IDs (which is essentially just a list of numbers), back into tweet JSON, and then even transform it into CSV for research and analysis.

Data archiving is important for one’s memory work and reproducibility of research, but we think its important to further the agency of content creators who are using social media to document their lives. Consent to have ones content be archived is a difficult thing to ascertain when collecting “public” tweets from thousands (or millions) of users. However, when a user decides to delete a tweet, or take their account private, we think it’s important to honor that decision. It’s far from perfect (some deleted tweets need to be remembered), but sharing tweet ID datasets is one way of simultaneously pursuing the goals of data publishing for research, and ethical archiving. P.S. Look for more approaches for navigating consent later this year.

So, the Catalog is simply a descriptive list of these tweet ID datasets, with pointers to the repositories around the web where the data resides. After a few years of modest but steady submissions the Catalog now contains 113 datasets that, taken together, comprise 2.5 billion tweets. While this is a large number the Documenting the Now community aims for quality over quantity. Each dataset has been assembled by a librarian, archivist or researcher for a specific purpose. When it comes to social media and the web it’s just not possible to preserve everything, so our decisions about what to preserve and why become that much more important.

Today we are releasing a new version of the Catalog to support the increased number of datasets it contains, and also to make it a bit easier to explore and submit them. The first version of the Catalog was a very simple static website built using the Jekyll publishing framework. At the time we wanted the Catalog to be a static site to make it easier to maintain and sustain over time. Once deployed static sites have many less moving parts than a typical web application. But as the number of datasets grew, and more and more people wanted to submit them, we knew we had to make some changes.

Version 2 of the Catalog is still a static site, but it uses the Gatsby framework which allows the website to function more like an web application, while still operating as a simple static site on the server side. This means we still have the sustainability wins of running a static site, while allowing users to filter and sort the datasets, and also then visit a distinct page for each dataset.

Furthermore, the netlify-cms JavaScript library allows the Catalog to offer administrative forms for easily adding and editing datasets, which then seamlessly commits the change back to the docnow/catalog GitHub repository. This means we still have a static site that is easy to maintain over time, where all the content lives in a versioned Git repository. But we are no longer asking users to hand edit data files and submit pull requests in order to share information about their datasets.

Having a form now provides an opportunity to learn more from contributors about their engagement with ethical aspects of the data collecting work. This work has been informed specifically by Alexandra Dolan-Mescal’s Social Humans project, who also designed the new Catalog. The new add dataset form has this additional set of questions:

• Did you publicly share (on Twitter) that you were collecting this content?
• Do content creators have the option to opt out of your collecting or remove their content after collection has taken place?
• Do you have an easy-to-find way for content creators to reach out to you or your organization about your collecting project?
• Have you analyzed the tweet dataset for potential threats to those whose content has been collected?
• Is this tweet dataset part of a larger collecting effort of topical materials, including oral histories, web archiving, or physical materials?

And finally, for the data nerds out there, one nice side effect of using Gatsby and netlify-cms is that each dataset in the Catalog is now stored as a distinct Markdown document. Markdown lets you to compose rich textual documents, with lists, links, images and other formatting, while also expressing structured metadata. The netlify-cms JavaScript library makes it easy for users to edit these documents without them even knowing it. For example, here is the Markdown file for a dataset contributed by Laura Wrubel and Dan Kerchner from George Washington University:

George Washington University is currently still in the process of updating this dataset, as the Coronavirus pandemic continues to spread around the world. The new Catalog allows them and others to easily edit the description as they release more data.

We hope that you will get a chance to give the new Catalog a try, either for publishing a new dataset, or finding a new dataset to work with. If you get a chance to use it, have ideas, or notice problems please don’t hesitate to send us an email at info@docnow.io, or to add issues to the issue tracker at GitHub. We’ve got some more things planned for integrating the Catalog better with the Hydrator, and the DocNow application, so look for those later this year.

Introducing The Catalog v2 was originally published in Documenting DocNow on Medium, where people are continuing the conversation by highlighting and responding to this story.