<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Justin Littman on Medium]]></title>
        <description><![CDATA[Stories by Justin Littman on Medium]]></description>
        <link>https://medium.com/@justinlittman_38536?source=rss-21197675409a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*STujW-xn0BKV9N6C.</url>
            <title>Stories by Justin Littman on Medium</title>
            <link>https://medium.com/@justinlittman_38536?source=rss-21197675409a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:30:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@justinlittman_38536/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Twarc-Cloud: Twitter data collection in the cloud]]></title>
            <link>https://medium.com/@justinlittman_38536/twarc-cloud-twitter-data-collection-in-the-cloud-d1cac85f57a5?source=rss-21197675409a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1cac85f57a5</guid>
            <category><![CDATA[twitter-data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Justin Littman]]></dc:creator>
            <pubDate>Thu, 14 Mar 2019 10:54:40 GMT</pubDate>
            <atom:updated>2019-03-14T10:54:40.791Z</atom:updated>
            <content:encoded><![CDATA[<p>During my tenure at George Washington University Libraries, I was a member of the team that developed <a href="https://gwu-libraries.github.io/sfm-ui/">Social Feed Manager</a> (SFM). SFM is open source software that harvests social media data from Twitter, Tumblr, Flickr, and Sina Weibo. It is intended to empower researchers, faculty, students, and archivists to collect, manage, and export social media data.</p><p>While SFM was a huge success in most respects and well received by the community, it is not widely adopted. There are around a dozen deployments in academic institutions of which I’m aware. In my experience, the reason for SFM’s deployment deficit is evident: while we worked hard to <a href="https://sfm.readthedocs.io/en/latest/install.html">simplify deployment</a> by using Docker (only 3 commands to bring up SFM!), at the end of the day needing a server and/or a sysadmin is a formidable barrier in most academic institutions.</p><p>Twarc-Cloud is an attempt to address this barrier by eliminating the need for a server and/or sysadmin by relying on cloud-based services, in particular, Amazon Web Services (AWS). This has motivated Twarc-Cloud’s design principles:</p><ul><li>Serverless, so no server to maintain or pay for when not in use.</li><li>Use as few AWS services as possible to reduce complexity and cost.</li></ul><p>Twarc-Cloud collects tweets using DocNow’s <a href="https://github.com/docnow/twarc">Twarc</a> running as Fargate Elastic Container Service (ECS) tasks. Tweets are written to S3 buckets as gzip-compressed, newline-delimited JSON. Twarc-Cloud is managed using a command-line application. Thus, there is no web server, database, message queue, etc.</p><p>An additional benefit of using these AWS services is scalability becomes a non-issue. You can collect as many tweets as you want and AWS will automagically scale the compute and storage resources.</p><p>Using a cloud provider immediately begs the question of cost. Since Twarc-Cloud is stingy in its use of AWS services and only uses serverless resources, the cost is incredibly cheap:</p><ul><li>Fargate ECS: $0.012345 per harvest per hour</li><li>S3: $0.023 per GB per month</li></ul><p>Twarc-Cloud supports collecting user timelines, searches, and filter streams from Twitter’s API. User timeline and search harvests can be run as needed or according to a schedule. Filter streams are run continuously (i.e., they are either on or off). Filter streams run as an ECS service so that if there is a fault such as a network hiccup, collecting will be restarted.</p><p>I’d characterize Twarc-Cloud as situated between Twarc (command-line) and SFM (web-based). While Twarc is easy to get installed and running, it doesn’t provide scheduling or uninterrupted collecting. And, as mentioned, while Twarc-Cloud doesn’t require a server like SFM, it lacks a web interface and the enterprise-like features of SFM such as user/group management.</p><p>Here’s a whirlwind tour of using Twarc-Cloud.</p><p>A collection is specified by a <em>collection.json</em> file. Here’s an example for collecting some user timelines:</p><pre>{<br> “id”: “dem_candidates”,<br> “keys”: {<br>   “consumer_key”: “mBbq9ruEcnggfQzgTHUhr8eKn0”,<br>   “access_token”: “2875189485-cf3rCYq59k1adfdhV88fyShQZ0rUsbZszp1”<br> },<br> “type”: “user_timeline”,<br> “users”: {<br>   “216776631”: {<br>     “screen_name”: “BernieSanders”<br>   },<br>   “357606935”: {<br>     “screen_name”: “ewarren”<br>   }<br> },<br> “delete_users_for”: [“protected”, “suspended”, “not_found”],<br> “timestamp”: “2019–03–14T01:20:37.851912”<br>}</pre><p>Twarc-Cloud provides a number of commands to make creating a <em>collection.json</em> easier. For example:</p><pre>$ python twarc_cloud.py collection-config screennames <a href="http://twitter.com/KamalaHarris">@KamalaHarris</a><br>Getting users ids for screen names. This may take some time …<br>Added screen names to collection.json.</pre><p>A collection can then be added and scheduled:</p><pre>$ python twarc_cloud.py collection add<br>Collection added.<br>Don’t forget to start or schedule the collection.</pre><pre>$ python twarc_cloud.py collection schedule dem_candidates “rate(7 days)”<br>Scheduled</pre><p>Twarc-Cloud will now run a harvest for this collection every 7 days.</p><p>Harvests can be monitored:</p><pre>$ python twarc_cloud.py harvest list<br>tweets_mentioning_trump =&gt; Bucket: twarc-cloud2. Status: RUNNING<br>dem_candidates =&gt; Bucket: twarc-cloud2. Status: RUNNING</pre><pre>$ python twarc_cloud.py harvest last dem_candidates<br>dem_candidates =&gt; Bucket: twarc-cloud2. Harvest timestamp: 2019–03–14T01:29:18.684991. Tweets: 9,641. Files: 1 (3M)<br>No user changes.</pre><p>For user timelines, not only are a user’s tweets collected, but so is information on the user’s account. If any changes are noticed for a user, these are recorded.</p><p>And when you’re ready, a collection can be downloaded:</p><pre>$ python twarc_cloud.py collection download dem_candidates<br>Collection downloaded to download/twarc-cloud2/collections/dem_candidates</pre><p>Lastly, Twarc-Cloud keeps track over every change that is made to a collection:</p><pre>$ python twarc_cloud.py collection-config changes dem_candidates<br>users -&gt; 216776631 -&gt; screen_name changed from None to BernieSanders on 2019–03–14T01:27:14.420473<br>users -&gt; 357606935 -&gt; screen_name changed from None to ewarren on 2019–03–14T01:27:14.420473<br>users -&gt; 30354991 -&gt; screen_name changed from None to KamalaHarris on 2019–03–14T01:27:14.420473</pre><p>For more details, check out the <a href="https://twarc-cloud.readthedocs.io/en/latest/quickstart.html">quickstart</a> in Twarc-Cloud’s <a href="https://twarc-cloud.readthedocs.io/en/latest/">docs</a>.</p><p>Some additional possibilities:</p><ul><li>Twarc-Cloud is a potential step in the direction of Twitter data collection as a service (similar to Internet Archive’s Archive-It for web archiving) that conforms with Twitter’s Developer Policies. If you’re interested in exploring this, please reach out.</li><li>Having your Twitter data stored on S3 opens up all sorts of analytic possibilities, including EMR (Hadoop), SageMaker (machine learning), Comprehend (text analysis), and Elasticsearch (fulltext indexing).</li></ul><p>Twarc-Cloud builds upon the work of the teams at GW Libraries and DocNow. I am greatly appreciative of their contributions to this field, as well as my colleagues at Stanford University Libraries’ Digital Library Systems and Services for schooling me on AWS.</p><p>As always, feedback, discussion, tickets, and pull requests are welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1cac85f57a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter’s Developer Policies for Researchers, Archivists, and Librarians]]></title>
            <link>https://medium.com/on-archivy/twitters-developer-policies-for-researchers-archivists-and-librarians-63e9ba0433b2?source=rss-21197675409a------2</link>
            <guid isPermaLink="false">https://medium.com/p/63e9ba0433b2</guid>
            <category><![CDATA[social-media]]></category>
            <dc:creator><![CDATA[Justin Littman]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 21:17:05 GMT</pubDate>
            <atom:updated>2019-01-08T19:01:46.063Z</atom:updated>
            <content:encoded><![CDATA[<p>I have long maintained that one of the most significant barriers to Twitter research and archiving are <a href="https://developer.twitter.com/en/developer-terms/policy">Twitter’s Developer Policies</a>. This barrier takes the form of not only the restrictions contained in the policies, but the ambiguity of the documents themselves.</p><p>In addition to just being poorly worded, my read of the policies is that they are written primarily for Twitter’s business partners. As such, it is unclear how or if Twitter intends them to apply to research and archiving.</p><p>In my work supporting research and archiving using the Twitter API, my approach is to try to suss out the spirit of the policies (assuming Twitter’s good intentions), and balance it against the best interests of Twitter, the societal value of the research and archiving, and the agency of the content creators. It is my experience that colleagues in the research, library, and archives community do the same, though as you see below, the results can be quite varied.</p><p>In the rest of this post, I will unpack some key portions of Twitter’s Developer Policies that are relevant to research and archiving and offer my interpretation. In many cases, my interpretation is “I have no idea what this means.”</p><p>Agreeing to Twitter’s Developer Policies is required to get a developer account; a developer account is required to access the API. (Applying for a developer account recently switched from a <em>pro forma</em> registration to requiring an applicant to describe her application / use case for Twitter approval. However, we’ll leave that for another blog post.)</p><p>Just to be clear, I am not a lawyer and this is not a legal reading of the document. It is my recommendation that you be wary about sharing Twitter’s policies with your institution’s counsel, as their reading is likely to be problematic for your work.</p><p>Also, just to be clear, the <a href="https://gwu-libraries.github.io/sfm-ui/resources/social_media_research_ethical_and_privacy_guidelines.pdf">ethical obligations and considerations</a> of a social media researcher or archivist extend beyond any ethical considerations codified in Twitter policies.</p><p>Starting with the Developer Policy:</p><blockquote>I.A.5. Remember, Twitter may suspend or revoke access to the Twitter API if we believe you are in violation of this Policy. Do not apply for or register additional API tokens if Twitter has suspended your account. Instead, contact us.</blockquote><p>Losing access to the API is the primary penalty you will face for violating Twitter’s Developer Policies. This can be extremely frustrating because:</p><ul><li>Based on complaints in the Twitter Developer Forum, it seems really difficult to get a non-automated response when you contact Twitter support.</li><li>If you are offering a service to a community (e.g., a university campus), this will disable your service for everyone.</li></ul><p>Researchers and archivists are frequently concerned that they will face legal action if terms are violated; I’m not aware of Twitter suing anyone in academia or anything else similarly serious. (If you know of any cases, let me know and I’ll add here.)</p><p>While discussing implications for breaching Twitter policies, I should point out that it doesn’t matter how we read the policies, all that matters is how Twitter reads their own policies. Thus, finding a “loophole” in the wording does not mean Twitter cannot take action. This is why I advocate a focus on the spirit of the policies, rather than a legalistic parsing of the wording.</p><blockquote>I.B.3. Do not modify, translate or delete a portion of the Twitter Content.</blockquote><p>I read this as applying when you are using the API to update content. (For archiving and research, we are typically only retrieving content.)</p><blockquote>I.B.7. Do not (and do not allow others to) aggregate, cache, or store location data and other geographic information contained in the Twitter Content, except as part of a Tweet or Periscope Broadcast. Any use of location data or geographic information on a standalone basis is prohibited.</blockquote><p>The meaning of this is unclear and is typically ignored for Twitter research with geolocation data.</p><blockquote>I.C.1. Get the user’s express consent before you do any of the following:</blockquote><blockquote>b. Republish Twitter Content accessed by means other than via the Twitter API or other Twitter tools.</blockquote><p>I’m not sure what this means. Regardless, where possible, it is recommended that researchers get the consent from a tweet’s author before republishing it in a scholarly publication. (See, for example, the guidance in <a href="http://journals.sagepub.com/doi/pdf/10.1177/0038038517708140">this article</a>.)</p><p>It’s worth pointing out that it is extremely difficult to contact users on Twitter without an existing relationship. If your account isn’t followed by a user, the best you can do is @ mention him. I worked with a research team that reached out to some Twitter users to participate in a study. The response rate: 0%.</p><p>It continues:</p><blockquote>d. Store non-public Twitter Content such as Direct Messages or other private or confidential information.</blockquote><blockquote>e. Share or publish protected Twitter Content, private or confidential information.</blockquote><p>For Twitter research and archiving, we don’t typically deal with non-public or protected Twitter content. With limited exceptions, this isn’t available from the API methods that are used to acquire tweets.</p><blockquote>I.C.3. If Twitter Content is deleted, gains protected status, or is otherwise suspended, withheld, modified, or removed from the Twitter Service (including removal of location information), you will make all reasonable efforts to delete or modify such Twitter Content (as applicable) as soon as reasonably possible, and in any case within 24 hours after a request to do so by Twitter or by a Twitter user with regard to their Twitter Content, unless otherwise prohibited by applicable law or regulation, and with the express written permission of Twitter.</blockquote><p>These terms invoke tremendous angst in researchers and archivists. I expect that this policy is written for Twitter’s business partners who have access to Enterprise API methods that provide notification when content is deleted or protected. Social media analytic platforms such as <a href="https://www.crimsonhexagon.com/">Crimson-Hexagon</a> do adhere to this policy. Those of us using the Basic API do not have access to a mechanism to deal with deleted tweets at scale, and even if we did the technical infrastructure required to make use of it would be prohibitive.</p><p>The spirit of this term suggests that researchers and archivists have obligations to be mindful of deleted tweets; however, these terms does not spell out that obligation in a relevant way. I’m not aware of any non-commercial parties that collect Twitter data that actively purge deleted tweets from their datasets. As explained below, the prohibitions on sharing datasets of complete tweets offers a significant “right to be forgotten” for users deleting tweets.</p><blockquote>I.C.4. If your Service will display Twitter Content to the public or to end users of your Service, and you do not use Twitter Kit or Twitter for Websites to do so, then you must use the Twitter API to retrieve the most current version of the Twitter Content for such display. If Twitter Content ceases to be available through the Twitter API, you may not display such Twitter Content and must remove it from non-display portions of your Service as soon as reasonably possible.</blockquote><p>I take this term as not applying to datasets collected for research and archiving, as they are not typically displayed on a website. For those displaying tweets on a website, using <a href="https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-oembed.html">oEmbed</a> is any easy way to conform with this restriction as it makes sure that the tweets are displayed according to Twitter rules and deleted tweets are not displayed. (This is the approach that I used for <a href="https://tweetsets.library.gwu.edu/">TweetSets</a> to display sample tweets from a dataset.)</p><blockquote>F. Be a Good Partner to Twitter</blockquote><p>Agree!</p><blockquote>1. Follow the guidelines for using Tweets in broadcast if you display Tweets offline and the guidelines for using Periscope Broadcasts in a broadcast if you display Periscope Broadcasts offline.</blockquote><blockquote>2. If you provide Twitter Content to third parties, including downloadable datasets of Twitter Content or an API that returns Twitter Content, you will only distribute or allow download of Tweet IDs, Direct Message IDs, and/or User IDs.</blockquote><p>This is a key term that impacts the sharing of Twitter datasets for research and archiving. Each tweet has a unique id called a “tweet id.” Given a tweet id, a user can retrieve the complete tweet from Twitter’s API (called “hydrating”) unless that tweet has been deleted or the author’s account has been deleted, suspended, or protected.</p><p>This term requires that when sharing datasets with “third parties” (more on third parties shortly) only the tweet ids be exchanged (not the text of the tweet or other tweet metadata). (Here’s an example of GW Libraries sharing <a href="https://doi.org/10.7910/DVN/PDI7IN">280 million tweet ids for tweets related to the 2016 U.S. election</a>.) The person who receives the tweet ids can then hydrate the dataset to get the complete tweets.</p><p>This has a number of key implications:</p><ul><li>It adds friction to exchanging large Twitter datasets. <a href="https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup">Hydrating</a> is limited to 360,000 tweets per hour.</li><li>Because tweets are regularly deleted and accounts are deleted, protected, or suspended, the hydrated dataset is likely to be missing tweets in the original dataset. While this provides a “right to be forgotten” for users, it is an impediment to reproducible research. It also complicates research on topics like bots or misinformation.</li></ul><p>Essential to this term is determining who is and is not a “third party” in the context of archiving and research, since third parties can only be given tweet ids and presumably, first and second parties can be given the complete tweets. Twitter’s policies do not spell out what is a third (or other) party. Thus, this term has been interpreted in a variety of ways, ranging from the narrow (anyone who is not you is third party) to more liberally. At George Washington (GW) University Libraries, we (unofficially) interpreted this to allow sharing Twitter datasets that we collected with anyone affiliated with GW (including students, faculty, and other researchers) and their collaborators. (What constitutes a “collaborator” is, of course, ambiguous.) If someone from outside GW contacts the library about a dataset, only the tweet ids are shared.</p><p>[Clarification: Until summer of 2018, I was a member of the <a href="http://go.gwu.edu/sfm">Social Feed Manager</a> team at GW Libraries. I’m now at Stanford Libraries. Nothing in this post represents the official positions of either of these organizations.]</p><p>I’ve talked with others who have considered the same approach but applying it to a larger organization such as a consortium. Thus, you can imagine an organization like <a href="https://www.icpsr.umich.edu/icpsrweb/">ICPSR</a> making Twitter datasets available to researchers at its member institutions. (I’m not suggesting this is their policy). This would allow for broader sharing of Twitter datasets, though limiting distribution to an academic research context.</p><p>In addition, one question that regularly arises and is not addressed by this term is whether the exchange of aggregated data or analytics derived from datasets is allowed (e.g., tweet counts by user or top hashtags).</p><p>One of my criticisms of this policy is that it treats all tweeters the same, regardless of whether they are a private citizen, bot, politician, celebrity, or government. I’d suggest that the protections that are afforded to private citizens should not be fully extended to the tweets of public figures and institutions. This should include both the handling of deleted tweets and the ability to share those tweets in an academic research context (and possibly beyond). As an example, at GW Libraries we collected the <a href="https://doi.org/10.7910/DVN/2N3HHD">Twitter accounts of U.S. government agencies</a>; a dataset containing the complete tweets should be publicly shareable. It is also unclear how Twitter policy interacts (collides?) with public records laws or the works of the US government being in the public domain.</p><p>Similarly, unaccounted for by this term is the ability for a Twitter user to donate her tweets to an archive and for that archive to distribute those tweets. This is a particularly important use case for community-based archives or archives of social movements. (For more on this, see the work of the <a href="https://www.docnow.io/">Documenting the Now</a> project.)</p><blockquote>I.F.2.a. You may, however, provide export via non-automated means (e.g., download of spreadsheets or PDF files, or use of a “save as” button) of up to 50,000 public Tweet Objects and/or User Objects per user of your Service, per day.</blockquote><p>This is further explained in the “Redistribution of Twitter Content” section of the <a href="https://developer.twitter.com/en/developer-terms/more-on-restricted-use-cases">Restricted Uses of the Twitter APIs</a> policy:</p><blockquote>We permit limited redistribution of hydrated Twitter content via non-automated means. If you choose to share hydrated Twitter content with another party in this way, you may only share up to 50,000 hydrated public Tweet Objects and/or User Objects per recipient, per day, and should not make this data publicly available (for example, as an attachment to a blog post or in a public Github repository).</blockquote><p>I believe this term is primarily intended to limit the export of complete tweets from social media analytic services such as Crimson-Hexagon or Sprout Social. However, it can also be read as allowing the sharing of small datasets, but not the public posting of those datasets.</p><blockquote>I.F.2.b. Any Twitter Content provided to third parties remains subject to this Policy, and those third parties must agree to the Twitter Terms of Service, Privacy Policy, Developer Agreement, and Developer Policy before receiving such downloads.</blockquote><p>So make sure that anyone you share datasets with is aware of Twitter’s policies.</p><blockquote>I.F.2.b.i. You may not distribute more than 1,500,000 Tweet IDs to any entity (inclusive of multiple individual users associated with a single entity) within any given 30 day period, unless you are doing so on behalf of an academic institution and for the sole purpose of non-commercial research or you have received the express written permission of Twitter.</blockquote><p>When Twitter originally announced this policy, there was no exclusion for academic uses. However, in response to <a href="https://gwu-libraries.github.io/sfm-ui/posts/2017-05-18-twitter-policy-change">concerns raised by the library and academic communities</a>, they added this exclusion.</p><blockquote>I.F.2.b.ii. You may not distribute Tweet IDs for the purposes of (a) enabling any entity to store and analyze Tweets for a period exceeding 30 days unless you are doing so on behalf of an academic institution and for the sole purpose of non-commercial research or you have received the express written permission of Twitter, or (b) enabling any entity to circumvent any other limitations or restrictions on the distribution of Twitter Content as contained in this Policy, the Twitter Developer Agreement, or any other agreement with Twitter.</blockquote><p>I’m not sure what this means but have comfortably ignored it because of the exclusion for academic uses.</p><p>We now turn to the <a href="https://developer.twitter.com/en/developer-terms/agreement.html">Developer Agreement</a>.</p><blockquote>I.B. License from Twitter. Subject to the terms and conditions in this Agreement (as a condition to the grant below), Twitter hereby grants you and you accept a non-exclusive, royalty free, non-transferable, non-sublicensable, revocable license solely to:</blockquote><blockquote>1. Use the Twitter API to integrate Twitter Content into your Services or conduct analysis of such Twitter Content;</blockquote><p>This seems to allow for academic research; archiving is a bit less clear.</p><blockquote>II.B. Rate Limits. You will not attempt to exceed or circumvent limitations on access, calls and use of the Twitter API (“Rate Limits”), or otherwise use the Twitter API in a manner that exceeds reasonable request volume, constitutes excessive or abusive usage, or otherwise fails to comply or is inconsistent with any part of this Agreement. If you exceed or Twitter reasonably believes that you have attempted to circumvent Rate Limits, controls to limit use of the Twitter APIs or the terms and conditions of this Agreement, then your ability to use the Licensed Materials may be temporarily suspended or permanently blocked. Twitter may monitor your use of the Twitter API to improve the Twitter Services and to ensure your compliance with this Agreement and the Developer Terms.</blockquote><p>Twitter imposes <a href="https://developer.twitter.com/en/docs/basics/rate-limits">rate limits</a> on how often you can make calls to the API with a specific set of credentials. In general, these are reasonably generous and allow collecting at scale, as long as your software is designed to take the limits into account and you have some patience. In our use of Social Feed Manager at GW Libraries, we encouraged each researcher to create her own credentials so that the collecting can be spread across them.</p><p>The rate limit that is particularly problematic for collecting at scale is the restriction that only one filter stream can run at a time for a set of credentials. To collect on a number of topics concurrently, it is desirable to run multiple filter streams, which would require multiple credentials.</p><p>It is tempting to create “dummy” Twitter accounts to circumvent rate limits. However, I would suggest that doing so at any significant scale violates the spirit of this term.</p><p>In my particular case, I have a handful of Twitter accounts that I use of various purposes. I do create a credential for each account and use them to collect multiple filter streams concurrently.</p><blockquote>II.C. Geographic Data. Your license to use Twitter Content in this Agreement does not allow you to (and you will not allow others to) aggregate, cache, or store location data and other geographic information contained in the Twitter Content, except in conjunction with the Twitter Content to which it is attached. Your license only allows you to use such location data and geographic information to identify the location tagged by the Twitter Content. Any use of location data or geographic information on a standalone basis or beyond the license granted herein is a breach of this Agreement.</blockquote><p>I’m unclear if and how this applies to academic research that involves geolocation.</p><blockquote>IV.A. Ownership. The Licensed Material is licensed, not sold, and Twitter retains and reserves all rights not expressly granted in this Agreement. You expressly acknowledge that Twitter, its licensors and its end users retain all worldwide right, title and interest in and to the Licensed Material, including all rights in patents, trademarks, trade names, copyrights, trade secrets, know-how, data (including all applications therefor), and all proprietary rights under the laws of the United States, any other jurisdiction or any treaty (“IP Rights”). You agree not to do anything inconsistent with such ownership, including without limitation, challenging Twitter’s ownership of the Twitter Marks, challenging the validity of the licenses granted herein, or otherwise copying or exploiting the Twitter Marks during or after the termination of this Agreement, except as specifically authorized herein. If you acquire any rights in the Twitter Marks or any confusingly similar marks, by operation of law or otherwise, you will, at no expense to Twitter, immediately assign such rights to Twitter.</blockquote><p>Content creators actually retain the copyright for their tweets and, presumably, related content such as images. I believe there was some legal ruling that 140 characters was too short to copyright, but I’m not sure if that changed when a tweet went to 240 characters. Regardless, you should keep in mind that tweets are (usually) produced by people, whose agency should be considered. Untangling copyright considerations is one area that would definitely benefit from some legal expertise.</p><blockquote>VII.A. User Protection. Twitter Content, and information derived from Twitter Content, may not be used by, or knowingly displayed, distributed, or otherwise made available to:</blockquote><blockquote>1. any public sector entity (or any entities providing services to such entities) for surveillance purposes, including but not limited to:</blockquote><blockquote>a. investigating or tracking Twitter’s users or their Twitter Content; and,</blockquote><blockquote>b. tracking, alerting, or other monitoring of sensitive events (including but not limited to protests, rallies, or community organizing meetings);</blockquote><blockquote>2. any public sector entity (or any entities providing services to such entities) whose primary function or mission includes conducting surveillance or gathering intelligence;</blockquote><blockquote>3. any entity for the purposes of conducting or providing surveillance, analyses or research that isolates a group of individuals or any single individual for any unlawful or discriminatory purpose or in a manner that would be inconsistent with our users’ reasonable expectations of privacy;</blockquote><p>These are terms that Twitter actively enforces. Those doing research that is funded by a government or a government contractor, especially in the defense or homeland security area, should be mindful of these terms.</p><p>The list continues:</p><blockquote>4. any entity to target, segment, or profile individuals based on health (including pregnancy), negative financial status or condition, political affiliation or beliefs, racial or ethnic origin, religious or philosophical affiliation or beliefs, sex life or sexual orientation, trade union membership, data relating to any alleged or actual commission of a crime, or any other sensitive categories of personal information prohibited by law;</blockquote><blockquote>5. any entity that you reasonably believe will use such data to violate the Universal Declaration of Human Rights (located at <a href="http://www.un.org/en/documents/udhr/),">http://www.un.org/en/documents/udhr/),</a> including without limitation Articles 12, 18, or 19.</blockquote><p>See also the “Sensitive Information” section of Twitter’s <a href="https://developer.twitter.com/en/developer-terms/more-on-restricted-use-cases">Restricted Use Cases policy</a>.</p><p>It is unclear whether the prohibition on “any entity to target, segment, or profile individuals” on a variety of protected classes or behaviors is intended to apply to research and archiving, though I expect in the wake of Cambridge Analytica, Twitter intends it to apply. There is a wide array of valuable research that is conducted in these areas and an outright ban could potentially have negative impact on society and those groups themselves. Typically research ethics require heightened scrutiny and review of research in these areas, but not an outright ban.</p><p>Also problematic about this term is there is some indication from the case of <a href="https://sifter.texifter.com/">Sifter</a> that Twitter expects a service to police its use by users to stop prohibited use cases. But again, this is unclear.</p><p>Weighing in on prohibited use cases is a relatively new part of Twitter’s policies. My general experience is that researchers either ignore these terms or interpret them as not applying to their work. Instead, they take guidance from the research ethics of their fields or policies of their institutions.</p><p>My observation has been that Twitter is aware of and sympathetic to the unique needs for research and archiving. That being said, it is notoriously difficult to get clarification on Twitter policies and, in general, the clarification is usually “no”. As a result, few bother to ask.</p><p>Because of the difficulty of getting clarity from Twitter, the ambiguity of Twitter’s policies, and our desire to do our work, most of us performing research and archiving with Twitter data make our own private interpretation. I say “private” because we are afraid to “say out loud” what that interpretation is for fear of Twitter clamping down. Thus, important discussions about research methodology, data sharing, and ethics do not occur and actual practices for Twitter research and archiving vary greatly.</p><p>In libraries and archives, it is almost impossible to have an official policy for Twitter data or for libraries to discuss and share model policies. As my colleague, Laura Wrubel explains:</p><blockquote>I see this when I go to conferences and there is a lot of discussion and interest in this topic, but still little in the way of shareable policies and practices. One of the effects is extreme caution, which leads to no involvement in social media work by librarians and archivists, which is a risk in itself.</blockquote><p>The result is that much work with Twitter data occurs “in the shadows” where the interests of Twitter, researchers, archivists, librarians, and users would benefit from greater discussion and transparency. I’d take it even further and suggest that this is one of the factors that lead to research ethics failures. And it is clearly leading to gaps in the historical record.</p><p>I hope that Twitter takes steps to clarify its policies as they apply to research and archiving. And, I would further suggest that if they reached out to the relevant communities they would find good partners willing to discuss those policies.</p><p>And for those of you doing or considering research and archiving with Twitter data, but find Twitter’s policies confusing and daunting, “you are not alone.” I would encourage you to assume good intentions on Twitter’s behalf, try to identify the spirit of Twitter’s policies until they provide adequate clarification, look to the ethical principles of your discipline, and start working to turn this into a public dialogue about how to be good partners.</p><p>Did I miss any part of Twitter’s policies that you think are salient? Disagree with my reading? Have relevant examples? Let me know.</p><p><em>Thanks to Laura Wrubel for reviewing this post.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63e9ba0433b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/on-archivy/twitters-developer-policies-for-researchers-archivists-and-librarians-63e9ba0433b2">Twitter’s Developer Policies for Researchers, Archivists, and Librarians</a> was originally published in <a href="https://medium.com/on-archivy">On Archivy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Islamic State Extremists Are Using the Internet Archive to Deliver Propaganda]]></title>
            <link>https://medium.com/on-archivy/islamic-state-extremists-are-using-the-internet-archive-to-deliver-propaganda-a132597dd16?source=rss-21197675409a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a132597dd16</guid>
            <category><![CDATA[archive]]></category>
            <category><![CDATA[twitter-data]]></category>
            <category><![CDATA[isis]]></category>
            <dc:creator><![CDATA[Justin Littman]]></dc:creator>
            <pubDate>Mon, 22 Jan 2018 14:03:08 GMT</pubDate>
            <atom:updated>2018-01-22T20:47:41.825Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<p>The <a href="https://archive.org/">Internet Archive</a> (IA) is best known for the Wayback Machine, a massive web archive. In actuality, the scope of IA is much broader, also including television broadcasts, ebooks, and even vintage video games. Less well known is that anyone can upload anything to the Internet Archive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfwA_IU1c5ky646zauU00A.png" /></figure><p>So, for example, George Washington (GW) Libraries has uploaded digitized copies of <a href="https://archive.org/search.php?query=subject%3A%22George+Washington+University--Students--Yearbooks%22">The Cherry Tree</a>, GW’s yearbook. The Internet Archive serves the uploaded files. While intended for archival purposes, in essence, IA services as a web delivery platform.</p><p><strong>ISIS sympathizers are using the Internet Archive as a web delivery platform for extremist propaganda, posing a threat to the archival mission of IA.</strong></p><p>Let me back up …</p><p>GW’s <a href="https://extremism.gwu.edu/">Program on Extremism</a> (POE) studies violent and non-violent extremism. One area of focus for the POE is the Islamic State of Iraq and Syria (ISIS). To this end, the POE has collaborated with GW Libraries’ <a href="https://gwu-libraries.github.io/sfm-ui/">Social Feed Manager</a> team to collect the tweets of ISIS sympathizers. POE researchers also collect the postings of ISIS sympathizers on Telegram, but using a different methodology.</p><p>Last October, the POE released “<a href="https://extremism.gwu.edu/sites/extremism.gwu.edu/files/DigitalDecayFinal_0.pdf">Digital Decay: Tracing Change Over Time Among English-Language Islamic State Sympathizers on Twitter</a>”, a report summarizing their finding from the dataset of tweets by ISIS sympathizers, authored by Audrey Alexander. The report noted the Internet Archive was one of the top “file sharing services” that was referenced in links within the tweets. In presenting the results, it was also noted that the Internet Archive was showing up towards the top based on initial research with Telegram posts. I tweeted about this:</p><h3>Justin Littman on Twitter</h3><p>Interesting. @internetarchive shows up in list of top file sharing sites uses by ISIS supporters on Telegram in study by @gwupoe&#39;s @_bCliff.</p><p>At the time I merely assumed that some ISIS-related content was being swept up in web captures and then being referenced in the tweets of ISIS sympathizers.</p><p>More recently, Bennett Clifford, one of the POE researchers tweeted about updated finding from Telegram. Here he noted that IA had become the top file sharing site.</p><h3>bennett clifford on Twitter</h3><p>A few key inferences, in comparison to first three months - More talk about IS affiliates, less talk about IS non-military activities, governance, etc. - Internet Archive overtakes YouTube as top filesharing site - Seemingly much greater number of public channels in past 3 months https://t.co/FF4KT8H7TZ</p><p>[For more information, see this <a href="https://extremism.gwu.edu/sites/extremism.gwu.edu/files/Telegram%20Tracker%20Fall%202017%20%285%29.pdf">overview of the Telegram Tracker</a>.]</p><p>This time my interest was piqued. Bennett shared some screen shots from Telegram, which contained some links to IA. Here’s one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*ptK8XUQch7-OhDmLJ3MfdA.png" /></figure><p>Bennett provided some additional context on this post. It “contains a video release from the al-Hayat Media Center, one of the principal ISIS media outlets, usually responsible for releasing video content. When major releases are disseminated through Telegram, they are shared using a number of platforms and links so when one link is removed for violating content restrictions, another one is available. IA is one of the sites used frequently to host major media releases, which may explain why it shows up so often on Telegram.”</p><p>The screen shots showed my earlier assumption was entirely incorrect: <strong>ISIS sympathizers were linking to uploaded content, not web archives</strong>.</p><p>For additional evidence (and with the permission of the POE), I returned to the Twitter data collected with Social Feed Manager that was used to create the “Digital Decay” report. The dataset is “845,646 tweets produced by 1,782 English-language pro-IS accounts from February 15, 2016 to May 1, 2017.” I unshortened all of the URLs (using Twarc’s <a href="https://github.com/DocNow/twarc/blob/master/utils/unshrtn.py">unshrtn.py</a> script), extracted only the URLs for “archive.org”, and normalized the URLs (for example, https://ia800207.us.archive.org/27/items/naba20/naba20.pdf was normalized to https://archive.org/details/naba20). After normalization, there were 513 unique IA URLs. The top URLs and the number of tweets that referenced them are:</p><ul><li><a href="https://archive.org/details/BeautifulPearlsInTheFiqhOfTheIslamicKhilafah_201604">https://archive.org/details/BeautifulPearlsInTheFiqhOfTheIslamicKhilafah_201604</a> (37)</li><li><a href="https://archive.org/details/SangPourSangBloodForBloodComp">https://archive.org/details/SangPourSangBloodForBloodComp</a> (22)</li><li><a href="https://archive.org/details/gharib_20160708_1550">https://archive.org/details/gharib_20160708_1550</a> (20)</li><li><a href="https://archive.org/details/TawheedByShaykhAhmadJibril">https://archive.org/details/TawheedByShaykhAhmadJibril</a> (20)</li><li><a href="https://archive.org/details/AnNaba45">https://archive.org/details/AnNaba45</a> (14)</li><li><a href="https://archive.org/details/7D5DX3">https://archive.org/details/7D5DX3</a> (14)</li><li><a href="https://archive.org/details/geztg478_mail_Maa">https://archive.org/details/geztg478_mail_Maa</a> (13)</li><li><a href="https://archive.org/details/fasaekfek">https://archive.org/details/fasaekfek</a> (13)</li><li><a href="https://archive.org/details/1bjhdt06032016">https://archive.org/details/1bjhdt06032016</a> (13)</li><li><a href="https://archive.org/details/nasron4">https://archive.org/details/nasron4</a> (12)</li><li><a href="https://archive.org/details/CollectionOfExcerpts">https://archive.org/details/CollectionOfExcerpts</a> (12)</li><li><a href="https://archive.org/details/26_Jumada">https://archive.org/details/26_Jumada</a> (12)</li><li><a href="https://archive.org/details/ftbeno">https://archive.org/details/ftbeno</a> (10)</li><li><a href="https://archive.org/details/bgfezthbfd_mail_Sawn">https://archive.org/details/bgfezthbfd_mail_Sawn</a> (10)</li><li><a href="https://archive.org/details/TerjemahDabiq14SampaiHal.43">https://archive.org/details/TerjemahDabiq14SampaiHal.43</a> (10)</li><li><a href="https://archive.org/details/Jaza2aWifa8a">https://archive.org/details/Jaza2aWifa8a</a> (10)</li><li><a href="https://archive.org/details/Adnani_Raed">https://archive.org/details/Adnani_Raed</a> (10)</li><li><a href="https://archive.org/details/1jz07032016">https://archive.org/details/1jz07032016</a> (10)</li></ul><p>IA URLs constitute 1,654 of 129,042 non-unique, non-Twitter URLs in the dataset.</p><p>A few important considerations for contextualizing this claim:</p><ul><li>I have not evaluated the content itself to determine if it is extremist in nature. Much of the content is in Arabic and I have no subject matter expertise.</li><li>I don’t know anything about who is uploading content to IA and whether they have any sort of a relationship with the authors of the tweets.</li><li>Not all content that is used as propaganda by ISIS sympathizers is extremist in nature. In fact, many of the other URLs in the dataset link to news stories in mainstream Western media that happen to support the extremist narrative.</li><li>IA’s website does provide statistics on the number of views of each item. However, I haven’t evaluated the view numbers and I’m unclear how the number of views relates to downloads.</li><li>IA does take down uploaded files due to content issues and, as shown below, some content is geo-restricted and/or requires login. I don’t know whether there is an official policy, how the content review / takedown process currently operates, or how often content is removed. (I couldn’t find any information on their website, but might have missed it.)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/1*PuzLbgdMu-QjlT7MEb_rOg.png" /></figure><p>However, regardless of the above, it is deeply concerning that (1) ISIS propaganda is referencing content that is being served by the Internet Archive and (2) the role of IA as a web delivery platform for extremist propaganda seems to be increasing. As explained by Samantha Weirman and Audrey Alexander in the forthcoming “Hyperlinked Sympathizers: URLS and the Islamic State,” the use of file sharing services such as IA allow extremists to “separate the content producers, disseminators, and consumers from the material itself.” Further, it allows extremists to continue to spread propaganda as other publishing platforms are blocked. Clearly, the role of IA in providing a web delivery platform for extremist propaganda demands further consideration, investigation, and discussion.</p><p>Full disclosure: I am one of the Internet Archive’s biggest advocates. If this issue is not properly addressed by IA, I see it as a potential existential threat. The recent experiences of Facebook, Twitter, Reddit, and the like have made it clear that a web delivery platform cannot absolve itself of responsibility for what is published on it and the misuse of IA for spreading extremism is only likely to increase, especially as these other platforms clamp down. As evidence of how seriously this is being taken, Google, Facebook, and Twitter were <a href="https://www.recode.net/2018/1/9/16871158/us-congress-facebook-google-twitter-testify-online-extremism">hauled before the Senate Commerce Committee</a> on January 17 to address extremist content on their site.</p><p>These are all murky, uncomfortable waters for an archive; clearly IA has taken some measure to perform appraisal on the content it collects. And there may be legitimate reasons for archiving this content that counterbalance the harm from its availability as propaganda. However, I would urge that this demands further debate and both that IA’s efforts be more public and that these efforts be scaled to address this growing threat.</p><p>Note: This post was shared with the Internet Archive in advance of publication.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a132597dd16" width="1" height="1" alt=""><hr><p><a href="https://medium.com/on-archivy/islamic-state-extremists-are-using-the-internet-archive-to-deliver-propaganda-a132597dd16">Islamic State Extremists Are Using the Internet Archive to Deliver Propaganda</a> was originally published in <a href="https://medium.com/on-archivy">On Archivy</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>