<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeanna Schoonmaker on Medium]]></title>
        <description><![CDATA[Stories by Jeanna Schoonmaker on Medium]]></description>
        <link>https://medium.com/@jeanna-schoonmaker?source=rss-56cf328b663------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*MyQSwcdpTtNqCab9h8G3EA.jpeg</url>
            <title>Stories by Jeanna Schoonmaker on Medium</title>
            <link>https://medium.com/@jeanna-schoonmaker?source=rss-56cf328b663------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:16:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeanna-schoonmaker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I Built an App for Creating Interactive LinkedIn Network Graphs]]></title>
            <link>https://jeanna-schoonmaker.medium.com/how-i-built-an-app-for-creating-interactive-linkedin-network-graphs-451140a498cf?source=rss-56cf328b663------2</link>
            <guid isPermaLink="false">https://medium.com/p/451140a498cf</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[network-graph]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[networkx]]></category>
            <category><![CDATA[pyvis]]></category>
            <dc:creator><![CDATA[Jeanna Schoonmaker]]></dc:creator>
            <pubDate>Mon, 20 Dec 2021 00:20:11 GMT</pubDate>
            <atom:updated>2021-12-20T00:20:11.878Z</atom:updated>
            <content:encoded><![CDATA[<p>Using Networkx, Plotly, Pyvis, and Streamlit sharing to create and share network graphs from LinkedIn connections data</p><figure><img alt="Jeanna’s network graph of her LinkedIn connections featuring a black background and several colorful nodes with the biggest company connections in blue and smaller connections in purples and pink." src="https://cdn-images-1.medium.com/max/784/1*h-2RVSpK8wSdW2Kaukp8wg.png" /><figcaption>Network graph by author</figcaption></figure><p><strong><em>tl:dr </em></strong>— if you want to go directly to the app and make your own LinkedIn connection graphs, click here: <a href="https://share.streamlit.io/jschoonmaker/linkedin_network/main/li.py">https://share.streamlit.io/jschoonmaker/linkedin_network/main/li.py</a></p><h4>Network Graphs</h4><p>In my work as a data scientist, I primarily work with data that is in tabular format, meaning it is stored in rows and columns and generally part of a relational database system. Even when working with images or text, the eventual goal is often to put the metadata into a table for further analysis. Most data science tutorials emphasize pandas for good reason — it’s a tool data scientists use often for working with tabular data.</p><p>However, I was interested in learning more about network graphs and how they represent the relationship between entities in the graph by using nodes and edges.</p><figure><img alt="Network graph with edges (links) shown in green between nodes (vertices) shown as blue circles" src="https://cdn-images-1.medium.com/max/420/1*nrO0esCNshIpgCLc5RZfFA.png" /><figcaption>Example of nodes and edges in a network graph, from Math Insight <a href="http://mathinsight.org/network_introduction">http://mathinsight.org/network_introduction</a></figcaption></figure><p>I also wanted to learn more about how to create these graphs and customize them according to their features. Since concepts like nodes and edges were a little fuzzy for me, I started with a simple network that was intuitive to work with — my LinkedIn connections.</p><p>I found a great <a href="https://medium.com/bitgrit-data-science-publication/visualize-your-linkedin-network-with-python-59a213786c4">tutorial</a> online from <a href="https://medium.com/u/9a24cc840494">Benedict Neo</a> and <a href="https://bschoeneweis.github.io/visualizing-your-linkedin-connections-using-python">another</a> from <a href="https://medium.com/u/a05c94e778e5">Bradley Schoeneweis</a> that clearly explained how to download a csv of my LinkedIn connections, how to clean and filter that data, and how to use it to create a network graph. After testing out the tutorials on my own data, I decided to build a streamlit app that could help others learn about and visualize their connections as network graphs as well.</p><h4>Streamlit Features</h4><p>Streamlit is an open-source app framework that allows for quick and easy app development and, with the streamlit sharing option, app deployment. (see more from streamlit at <a href="https://medium.com/streamlit">https://medium.com/streamlit</a>)</p><p>My goal for my streamlit app was to make it possible for anyone to create their own network graphs and to include multiple <a href="https://docs.streamlit.io/">streamlit</a> features such as:</p><ol><li>An <a href="https://docs.streamlit.io/library/api-reference/layout/st.expander">expander option</a> so instructions could be shared but then minimized, keeping the look of the app clean</li><li>A <a href="https://docs.streamlit.io/library/api-reference/layout/st.sidebar">sidebar</a> for customizing the graphs</li><li>A <a href="https://docs.streamlit.io/knowledge-base/using-streamlit/record-screencast">screencast</a> <a href="https://docs.streamlit.io/library/api-reference/media/st.video">video</a> showing users how to navigate the app</li><li>An option to <a href="https://docs.streamlit.io/library/api-reference/widgets/st.download_button">download the graphs</a> the app created</li></ol><p>The links in each of the steps above will give you instructions on how to implement the features in your own streamlit app, or you can look at the github of this project to see them in action: <a href="https://github.com/jschoonmaker/linkedin_network/blob/main/li.py">https://github.com/jschoonmaker/linkedin_network/blob/main/li.py</a></p><p>My app takes a LinkedIn connections csv file as an input and returns:</p><ol><li>Customizable Plotly bar graphs showing the top connections by company and by title, along with the dates most connections were made on LinkedIn</li><li>Customizable Pyvis Networkx graphs showing weighted nodes and annotated hover text that provides more detailed data</li></ol><p>Let’s dig in to how to add this functionality to a streamlit app.</p><h4>Plotly Bar Graphs</h4><p>When the streamlit app first launches, it needs a csv file of LinkedIn connections to be uploaded before any graphs can be created. Originally, before a csv file was uploaded, an error would show on the app saying the df referenced in the code didn’t exist.</p><p>In order to both have a cleaner look and to avoid the streamlit program trying to run before the user had uploaded a file, I added this to the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebd184a29f25977cc8d1f7c48c6be8c0/href">https://medium.com/media/ebd184a29f25977cc8d1f7c48c6be8c0/href</a></iframe><p>Streamlit loads plotly graphs easily, so that feature worked by simply adding an st.write() wrapper around the plotly graph code, but I also wanted to add the option of changing the color scheme of the plotly graphs.</p><p>This code creates a drop down menu in the sidebar that lets the user choose a color theme. That color theme is then returned in a function, and that function is called as part of the bar graph parameters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/665bde59840b55d148a215e072008d92/href">https://medium.com/media/665bde59840b55d148a215e072008d92/href</a></iframe><h4>Pyvis Networkx Graphs</h4><p>For the network graphs, I used a similar approach for the color options, but also added a slider that applied the minimum number of connections that needed to exist for that company or job title to be included in the network graph. This ensured that enough connections are returned to show a good graph, but that the graph won’t be overpowered by including <strong>every</strong> connection.</p><p>In order to include this option, I created a slider in the sidebar with numbers from 1 to 20 and a default option of 3, then I used the variable option assigned to the user output of the slider as the filter for the dataframe the network graph was built from:</p><pre>count_option = st.sidebar.slider(&#39;Choose the minimum number of connections at company for them to be included:&#39;, 1, 20, 3)</pre><pre>df_company_reduced = df_company.loc[df_company[&#39;count&#39;]&gt;=count_option]</pre><p>Next, the edges and nodes needed to be added to the graph by iterating through the dataframe created from the connections csv. The count of connections determines the size of the node, as well as the color. The title of each node is collected from the list of positions and count associated with each company:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60beb587e594a8670fe43fece2ebc360/href">https://medium.com/media/60beb587e594a8670fe43fece2ebc360/href</a></iframe><p>Finally, it’s time to create the network graph visualization. I originally tried to use plotly to build the network graphs so that all graphs in the app referenced the same color scheme, but ultimately I found it too difficult to get plotly working with all of the networkx features. If you have any expertise in this area and want to share, please add links to suggested resources in the comments! Since I couldn’t get plotly working with the network graphs, I used pyvis instead.</p><p>Pyvis takes the collected Networkx graph data, applies any size and color specifications along with either the ‘repulsion’ or ‘hrepulsion’ option based on whether the user chose a ‘packed’ or ‘spoke’ graph, then creates the graph as an interactive html object that can be moved and manipulated by dragging a node to another part of the graph.</p><p>In order to display the html object that Pyvis returns in streamlit, components.html can be used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30fae86d3bfe0aab3a8894e66722177e/href">https://medium.com/media/30fae86d3bfe0aab3a8894e66722177e/href</a></iframe><p>Finally, by uploading all of the project code into its own github directory and following <a href="https://docs.streamlit.io/streamlit-cloud/get-started/deploy-an-app">these directions</a>, the streamlit app can be deployed and anyone who clicks on the link will be able to access and use your app.</p><p>And that’s it! This was a great project to not only learn more about network graphs and the python modules that can be used to create and visualize them, but also a good opportunity to learn more about the options for building, deploying, and sharing a streamlit app.</p><p>Click here to see my streamlit app in action: <a href="https://share.streamlit.io/jschoonmaker/linkedin_network/main/li.py">https://share.streamlit.io/jschoonmaker/linkedin_network/main/li.py</a></p><p>or click here to see the github repo where all the code is stored: <a href="https://github.com/JSchoonmaker/LinkedIn_Network">https://github.com/JSchoonmaker/LinkedIn_Network</a></p><p>Happy building!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=451140a498cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CryptoDataScience Part 3: Two Methods for Anomaly Detection on the CryptoPunk Blockchain]]></title>
            <link>https://jeanna-schoonmaker.medium.com/cryptodatascience-part-3-two-methods-for-anomaly-detection-on-the-cryptopunk-blockchain-1a0c29bf654e?source=rss-56cf328b663------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a0c29bf654e</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[isolation-forests]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jeanna Schoonmaker]]></dc:creator>
            <pubDate>Wed, 10 Nov 2021 13:13:07 GMT</pubDate>
            <atom:updated>2021-11-10T13:31:04.571Z</atom:updated>
            <content:encoded><![CDATA[<h4>Comparing Human in the Loop and Unsupervised Machine Learning Methods</h4><p>A record-setting CryptoPunks sale occurred recently — on Oct 30, 2021, Punk 9998 sold for over 500 million dollars.</p><p>Or did it?</p><p>According to <a href="https://news.slashdot.org/story/21/10/29/2155246/an-nft-just-sold-for-532-million-but-didnt-really-sell-at-all">news.slashdot.org</a>:</p><blockquote>Larva Labs, which created the CryptoPunks, <a href="https://twitter.com/larvalabs/status/1453903818308083720">said on Twitter</a> that “<strong>someone bought this punk from themself with borrowed money and repaid the loan in the same transaction.</strong>” Evidently, this isn’t the first time this has happened. “Some recent large bids were done the same way. The ether is offered and removed in a single transaction. So, while technically briefly valid, the bid can never be accepted. We’ll add filtering to avoid generating <a href="https://twitter.com/cryptopunksbot/status/1453872415591829504">notifications</a> for these kinds of transactions in the future.”</blockquote><blockquote>In conventional, regulated securities markets, <strong>this would be called wash trading, which is banned on grounds that trading with yourself can artificially inflate prices and suggest more demand than really exists.</strong></blockquote><p>In this case, wash trading was obvious. Are suspicious transactions always so easy to spot? That’s what we’ll be exploring in today’s post.</p><p>My CryptoDataScience posts thus far have involved:</p><h4>Part 1 — <a href="https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70">Accessing blockchain data</a></h4><h4>Part 2 — <a href="https://jeanna-schoonmaker.medium.com/cryptodatascience-part-2-exploratory-data-analysis-on-cryptopunks-transactions-910e6dcb14bd">Exploring blockchain data</a></h4><p>all of which leads up to</p><h4>Part 3 — Sciencing blockchain data</h4><p>Now that we have determined how to access and explore blockchain data, I’ll be examining the CryptoPunks transaction data more closely to look for patterns of buying and selling behavior. This is our ‘human in the loop’ standard for determining which transactions look odd.</p><p>Following that, I’ll run an unsupervised machine learning anomaly detection algorithm on the data to see which transactions are caught in the algorithm’s net.</p><h4>What are we looking for?</h4><p>First, let’s define what we’re looking for in the blockchain transactions.</p><p>From Investopedia — the definition of wash trading is:</p><blockquote>Wash trading is a process whereby a trader buys and sells a security for the express purpose of feeding misleading information to the market. In some situations, wash trades are executed by a trader and a broker who are colluding with each other, and other times wash trades are executed by investors acting as both the buyer and the seller of the security. Wash trading is illegal under U.S. law.</blockquote><p><a href="https://www.investopedia.com/terms/w/washtrading.asp">Wash Trading: What It Is and How It Works, With Examples</a></p><h4>What does the data show about the possibility of wash trading in CryptoPunks transactions?</h4><p>As mentioned in the definition and example above, wash trading is when one person “bids up” their own asset with a secondary account in an attempt to establish increased demand and justify a higher asking price for their asset.</p><p>Other than accounts that choose to name themselves or publicly post their wallet address, ownership of crypto wallets is pseudonymous (meaning not fully anonymous — wallet owners are identifiable by the wallet address, but their identity, location, etc. is unknown) and a person can sign up for as many wallets as they want.</p><p>Thus, it is difficult to PROVE wash trading unless a person admits to owning several wallets that are shown to be buying from each other — or unless a person buys and sells a Punk from the same address, as occurred with Punk 9998.</p><p>However, it <em>is</em> possible to see patterns in the data. If the transactions identified within those patterns are also identified as anomalies by a machine learning algorithm, it strongly suggests that the transactions are suspicious and at least worth an eyebrow raise, even if nothing has been proven.</p><p>An example of wash trading in blockchain NFT transactions could look something like this:</p><ol><li>Person 1 ‘claims’ the original Punk #6276 in its initial offering.</li><li>Person 1 sells Punk #6276 to Buyer A for $50.</li><li>Buyer A sells Punk #6276 to Buyer B for $100.</li><li>Buyer B sells Punk #6276 to Buyer C for $200.</li><li>Buyer C sells Punk #6276 to Buyer A for $350.</li><li>Buyer A sells Punk #6276 to New Buyer for $600.</li></ol><p>And imagine steps 2–6 all happen within a few days or weeks.</p><p>This is a plausible turn of events — perhaps a little strange that Buyer A rebought the Punk after already selling it once, but maybe buyer’s remorse played a part?</p><p>However, what crosses the line from strange buying behavior into wash trading is if Buyer A, B, and C are all the same person or a group of people colluding together to buy and sell the Punk to each other in an effort to raise its perceived value, both because of increased volume of trades, and because of a steady increase in the asset price.</p><p>Buyer A/B/C has paid the original cost of the Punk ($50 in this example) and the gas fees required to ‘write’ the purchase transactions to the blockchain (let’s say $1 per transaction, so $4 total) but has sold the Punk for $600. The cryptocurrency amounts per transaction are transferred from the wallet of Buyer A to Buyer B to Buyer C, but since they are the same person, it’s the equivalent of transferring funds from one account to another, albeit with a small transaction fee. The wash trader has spent $54 and been paid $600 from New Buyer, for a net gain of $546.</p><p>Now multiply that by several Punks, and multiply the cryptocurrency value by up to 10x, and it becomes clear why fraudulent transactions occur.</p><p>Let’s dig in and see what we find.</p><h3>The Human In the Loop Approach</h3><p>Plotting a histogram of the CryptoPunks data over time shows 3 distinct time periods when counts of Punk sales transactions were elevated — fall of 2020, spring 2021, and summer/fall 2021.</p><figure><img alt="Histogram of CryptoPunks sales transactions over time" src="https://cdn-images-1.medium.com/max/849/1*6Cqg2Cyc3umRl7QfCuZbgg.png" /></figure><p>We also learned during the EDA that the scale of transactions changed significantly between 2020 and 2021, going from sales worth $140k per Punk up to sales worth millions. Because the anomaly detection algorithm we will use depends on looking for outliers in the data, I will split the data into 3 subsets representing the distinct time periods with heavier sales traffic.</p><p>Today’s post will center on the first dataset which includes 2017 to 2020, but will mainly focus on the fall of 2020.</p><p>As noted at the end of <a href="https://jeanna-schoonmaker.medium.com/cryptodatascience-part-2-exploratory-data-analysis-on-cryptopunks-transactions-910e6dcb14bd">CryptoDataScience Part 2</a>, there are a few users/addresses that turn up many times in our CryptoPunks datasets as frequent sellers and buyers.</p><p>Next, I will take it a step further and identify the accounts that bought and sold the SAME Punk NFT more than once. Similar to the Buyer A/B/C example, while doing so might not be a concrete indicator of wash trading, it justifies another look.</p><p>Let’s create a subset of the 2017–2020 data with only the From, To, and ID fields — remember that ‘ID’ in this data refers to the CryptoPunk id, which is a unique identifier for each of the 10k Punks that were minted.</p><pre>wash_check = df_punks[[&#39;From&#39;, &#39;To&#39;, &#39;ID&#39;]].copy()</pre><p>Next, we’ll check for transactions where the same seller sold a Punk multiple times OR the same buyer bought a Punk multiple times, and we’ll put those Punk IDs into a list for further analysis.</p><p>Keep in mind, this will only identify accounts that transact on the same Punk multiple times. Due to the pseudonymous nature of crypto wallets, it is possible that there are suspicious trades that occur that aren’t detectable through this process.</p><p>As mentioned earlier, it’s also possible <em>none</em> of these transactions are fraudulent— however, the ‘human in the loop’ baseline in this case is identifying accounts that repeatedly bought and sold the same asset. This merits delving into the transaction details for more information.</p><pre>dupes = wash_check[wash_check.duplicated([&#39;From&#39;, &#39;ID&#39;])]<br>suss_ids = dupes[&#39;ID&#39;].tolist()</pre><pre>dupes2 = wash_check[wash_check.duplicated([&#39;To&#39;, &#39;ID&#39;])]suss_ids2 = dupes2[&#39;ID&#39;].tolist()</pre><pre>all_suss = suss_ids + suss_ids2</pre><pre>print(&#39;The number of transactions we want to examine further is:&#39;)<br>len(all_suss)</pre><p>which gives us:</p><pre>The number of transactions we want to examine further is:</pre><pre>135</pre><p>135 transactions that meet the criteria, involving 74 punks. There were 7,554 transactions in this dataset, so roughly 1.8% of the transactions merit further analysis. Keep this number in mind for later on.</p><p>Now that we have a list of Punk IDs with odd transactions, let’s create a new dataframe that shows all sales transactions associated with those IDs.</p><pre>boolean_check = df_punks.ID.isin(all_suss)</pre><pre>df_p = df_punks[boolean_check]</pre><pre>df_p.sort_values(by=[&#39;ID&#39;, &#39;Txn&#39;, &#39;Crypto&#39;]).head()</pre><p>Looking through the Punks transactions, it seems some names/wallet addresses show up often, just as was noted during exploratory data analysis. Let’s get a list of all accounts showing in the From/To fields of the specified transactions and then filter it to accounts that show up at least 10 times.</p><pre>accounts = df_p[&#39;From&#39;].to_list() + df_p[&#39;To&#39;].to_list()<br>most_common = Counter(accounts).most_common()<br>most_common = [x for x in most_common if x[1]&gt;10]</pre><p>gives us</p><pre>[(‘Pranksy’, 165), (‘0x5aaeb9’, 32), (‘Hemba’, 26), (‘MrNFT’, 24), (‘0x00d7c9’, 21), (‘Carlini8’, 20), (‘natealex’, 18), (‘Dude_Nak\x85’, 18), (‘0x11c9a7’, 17), (‘evkort’, 16), (‘ross_VRO\x85’, 14), (‘Goop’, 14), (‘NeonNFT’, 12), (‘Zieg’, 11)]</pre><p>Let’s take a closer look at a few of the transactions that involve the top account listed.</p><p>NOTE: After a google search, it seems Pranksy is an active NFT collector and early adopter of many NFTs. Analyzing this transaction history is not meant to be an accusation of Pranksy or any other accounts mentioned in the analysis.</p><figure><img alt="dataframe showing sales transaction for CryptoPunk 3260" src="https://cdn-images-1.medium.com/max/1024/1*uLJR_ANx4WHT1wUwhNoM0g.png" /></figure><p>Let’s look at Punk 3260’s transaction history, step by step</p><p>a. Starting at the top of the table, you can see that Hemba was Punk 3260’s first owner. On 9/22/20, Pranksy purchased 3260 for 1.8 Eth (‘Crypto’), which was equivalent to $613.</p><p>b. Two days later on 9/24, Pranksy sold 3260 to 0x5aaeb9 for 2.19 Eth or $757.</p><p>c. On that same day, 9/24, 0x5aaeb9 sold 3260 to 0xef764b for 2.29 Eth/$792.</p><p>d. Four days later, on 9/28/20, 0xef764b sold 3260 back to Pranksy for 3.99 Eth, or $1427.</p><p>e. Later that day, Pranksy sold 3260 to 0x13816f for 4.79 Eth, or $1714.</p><p>See additional examples below, both of which also include sales to and from Pranksy to 0x5aaeb9.</p><figure><img alt="Dataframe showing sales transaction for CryptoPunk 2880" src="https://cdn-images-1.medium.com/max/1024/1*wzQCFF0l22QSdFcvwnfgVw.png" /></figure><figure><img alt="dataframe showing sales transaction for CryptoPunk 6755" src="https://cdn-images-1.medium.com/max/1024/1*o3cT-JZbmfCvo787z6bIZw.png" /></figure><p>To be fair, while Pranksy was a part of several of the transactions we are analyzing, there were also transactions that don’t register as anomalous, like this one:</p><figure><img alt="dataframe showing sales transaction for CryptoPunk 3379" src="https://cdn-images-1.medium.com/max/1024/1*3Rx1wvgW9yTrNF8dH2bgsA.png" /></figure><p>There are also transactions which met the ‘human in the loop’ standards for further review that involve other accounts, such as the one below:</p><figure><img alt="dataframe showing sales transaction of CryptoPunk 1845" src="https://cdn-images-1.medium.com/max/1024/1*DNA0DGUO5lpMsDQH-qROEw.png" /></figure><p>One transaction not shown in the above data (because it was not a sale) is when the Punk was “<a href="https://dappradar.com/blog/dapp-review-of-the-week-wrapped-cryptopunks">wrapped</a>” between the first and 2nd row, which is why the ‘To’ and ‘From’ fields don’t match between those rows. This transaction is also one of the few where the Punk was sold for $500 LESS than it had been previously — in fact, a couple of the transactions for this Punk lose value, which is an oddity in itself.</p><p>The strangest Punk buying and selling behavior that I saw when exploring the datasets, though, was on Punk 6276 from the 2021 dataset. Despite a previous sale of $40k, this Punk has been repeatedly sold and transferred…for $0. The transactions are a little jumbled due to multiple transactions on the same date, but the long list of transfers and sales back and forth to the same addresses for $0 is definitely strange:</p><figure><img alt="dataframe of sales transactions for CryptoPunk 6276" src="https://cdn-images-1.medium.com/max/1024/1*idG1jwTjIctHAQ_7iCfbfQ.png" /></figure><p>Several examples fit our criteria of odd transactions that might suggest wash trading is occurring when using a ‘human in the loop’ approach to labeling.</p><p>Next, let’s train an unsupervised machine learning model to identify anomalous transactions.</p><h3>The Isolation Forest Approach</h3><p>The <a href="https://en.wikipedia.org/wiki/Isolation_forest">Isolation Forest algorithm</a> takes a unique approach to finding anomalies — instead of creating a profile of what a ‘normal’ transaction looks like in a dataset, and classifying any transactions outside of the norm as outliers, it instead looks for the outliers from the beginning, which results in a faster model that uses less memory.</p><p>Because we do not have any labeled data or concrete examples of fraudulent transactions, we need an unsupervised machine learning method.</p><p>I will be using the <a href="https://pyod.readthedocs.io/en/latest/index.html">PyOD</a> toolkit to access the IForest model. My thanks to <a href="https://www.justintodata.com/unsupervised-anomaly-detection-on-bank-transactions-outliers/#:~:text=%20A%20use%20case%20of%20Unsupervised%20Learning%20with,a%20Machine%20Learning%20Algorithm.%20How%20do...%20More%20">JustintoData</a> for their tutorial using IForest on financial transactions!</p><p>Two additional preprocessing steps we need to do before training the IForest model:</p><ol><li>Converting the dataframe’s date field into the index.</li></ol><pre>df_punks = df_punks.set_index(&#39;Txn&#39;)</pre><pre>df_punks = df_punks.sort_index()</pre><p>2. Creating a count of transactions within a 5 day window per Punk.</p><pre>df_punks[&#39;count_5days&#39;] = df_punks.groupby(&#39;ID&#39;)[&#39;USD&#39;].transform(lambda s: s.rolling(timedelta(days=5)).count())</pre><p>Next, we need to give the model a parameter for how many anomalous transactions we expect it to find. Because we don’t have labeled data, this parameter keeps the model to a reasonable output.</p><p>Remember earlier in the human in the loop section when I said to keep the amount 1.8% of transactions in mind? That’s the same range we will use now. We’ll instruct the Isolation Forest model to expect 1.8% of the transactions to be outliers, and then we will fit the model on the 5 day count of transactions along with the amount of ether (‘Crypto’) spent on the transaction.</p><pre>anomaly_proportion = 0.018</pre><pre>clf_name = &#39;Anomaly Detection - Isolation Forest&#39;<br>clf = IForest(contamination=anomaly_proportion)<br>X = df_punks[[&#39;count_5days&#39;, &#39;Crypto&#39;]]<br>clf.fit(X)</pre><p>We can now get the prediction scores and outlier labels from the model and add it to our dataframe. If the transaction is labeled 1, that indicates the model identified it as an outlier, and in the raw outlier scores, the higher the number is, the greater the anomaly it was found to be.</p><pre>df_punks[&#39;y_pred&#39;] = clf.labels_</pre><pre>df_punks[&#39;y_scores&#39;] = clf.decision_scores_</pre><p>You may recognize a few of the accounts on this list:</p><figure><img alt="dataframe showing anomalous transactions" src="https://cdn-images-1.medium.com/max/1024/1*yO7DcPUekubUB9ZIN1vIig.png" /></figure><p>Next, we’ll create a visualization of the inliers and outliers of our Isolation Forest model, along with the boundaries the model established between normal and anomalous transactions.</p><figure><img alt="Isolation Forest visualization" src="https://cdn-images-1.medium.com/max/1024/1*c3g9AzDq2-DK3dzGOW9r3Q.png" /></figure><p>Note that the model was trained on the price per transaction along with the count of transactions per 5 day window, which is reflected in how the model identified outliers. In our human in the loop approach, I paid little attention to the price of a transaction simply because the volatility of the NFT market and the lack of established valuations means determining a “normal” amount to pay for an NFT is very difficult to do.</p><p>The Isolation Forest approach also identifies specific transactions, where our human in the loop approach looked at an entire Punk’s sales history. Still, many of the transactions identified by the IForest algorithm involved Punks that were also identified by our human in the loop approach.</p><p>Now that we’ve scienced the data and used a couple of methods for identifying outliers and possible wash trading in the CryptoPunks transactions — what did I learn from all this?</p><h3>Takeaways from my CryptoDataScience adventure:</h3><ol><li><strong>The NFT/web3/blockchain space is still very new.</strong> Even during the few weeks while I was working on this project, major news stories occurred — the <a href="https://news.slashdot.org/story/21/10/29/2155246/an-nft-just-sold-for-532-million-but-didnt-really-sell-at-all">$500 million sale</a> of a CryptoPunk that never was, the <a href="https://www.nme.com/news/gaming-news/nft-owners-in-the-us-to-start-paying-taxes-on-their-investments-3090185">newly passed bill that has tax and reporting implications for NFTs</a>, and the <a href="https://marketrealist.com/p/world-of-women-nft-reese-witherspoon/">continued interest in NFTs by celebrities</a>, just to name a few.</li><li><strong>There’s amazing potential,</strong> especially (in my opinion) for data science and analytics on blockchain data. The data is always stored on chain, but that doesn’t make it easy to access or clean. Those who find ways to add insights will likely have <a href="https://thegraph.com/en/">plenty of opportunities</a> to do so, and <a href="https://www.flipsidecrypto.com/bounty-program">get paid while doing it</a>.</li><li><strong>Caveat emptor</strong>. The deregulated, decentralized, pseudonymous nature of blockchain transactions — including NFTs (a least for now) — means high volatility and high risk. It is very much a gamble, and chances are there is fraudulent activity occurring. If you plan to wade in, keep that in mind.</li><li><strong>The hype cycle for web3 and blockchain is in full effect</strong>. Even though I <em>just</em> mentioned the potential in this space myself, it is also worth noting that almost everyone* who advocates for this tech has a financial incentive to do so because they own eth or bitcoin or other altcoins and benefit when others buy in and increase scarcity and hype. This makes honest valuations tough to come by. It doesn’t mean people don’t honestly believe in the tech, but it does mean they likely benefit financially when you buy into it too. <em>(* as of writing, I do not own a crypto wallet or any NFTs)</em></li><li><strong>Outcomes from data projects are often ambiguous, and crypto data projects are no exception</strong>. It would be amazing if every data science project resulted in a crystal clear answer for how to easily implement or identify needed changes. Despite their outlier status and suspicious sales patterns, I can’t immediately identify clearly fraudulent behavior in CryptoPunks transactions. And there are <a href="https://www.okex.com/academy/en/top-nft-cryptopunks">experts in the space who don’t see evidence of wash trading in the NFT market</a>, though<a href="https://www.fxstreet.com/cryptocurrencies/news/analysts-fear-nfts-are-being-used-for-wash-trading-202108271200"> there are also some who do</a>. So was it worth it? Yes, because…</li><li><strong>The real treasure is the lessons learned along the way.</strong> Okay, a wallet full of CryptoPunks would also qualify as treasure, but given that I don’t own any of those, I’ll take the lessons learned instead. This was a side project for me, unrelated to my job, but exploring it gave me the chance to learn a lot about previously unknown (to me) tech, and even join a DAO myself: <a href="https://www.linkedin.com/company/charliedao/about/">https://www.linkedin.com/company/charliedao/about/</a>. If you work in data and are interested in this space, I encourage you to jump in and explore!</li><li><strong>Possible next steps</strong> could include doing the same 2 methods described in this post on the other CryptoPunks datasets from 2021. It would also be interesting to go beyond CryptoPunks and explore other NFTs, or explore other blockchain data related to DeFi (decentralized finance) or DAOs (decentralized autonomous organizations).</li><li><strong>Data science rules</strong>. The end.</li></ol><p><em>You can find all code used (plus bonus odd transactions not shared here for the sake of space) at this </em><a href="https://github.com/JSchoonmaker/blockchain_projects/blob/main/Anomaly_Detection_in_CryptoPunks_Transactions.ipynb"><em>github link</em></a><em>. My thanks to </em><a href="https://github.com/Omni-Analytics-Group/eth-data-science-course"><em>Omni Analytics</em></a><em> and </em><a href="https://medium.com/u/228de692dad0"><em>Bojan Tunguz</em></a><em> for the </em><a href="https://www.kaggle.com/tunguz/cryptopunks"><em>datasets</em></a><em> I used for this project.</em></p><p>Revisit Part 1 here: <a href="https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70">https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70</a></p><p>Or Part 2 here: <a href="https://jeanna-schoonmaker.medium.com/cryptodatascience-part-2-exploratory-data-analysis-on-cryptopunks-transactions-910e6dcb14bd">https://jeanna-schoonmaker.medium.com/cryptodatascience-part-2-exploratory-data-analysis-on-cryptopunks-transactions-910e6dcb14bd</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a0c29bf654e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CryptoDataScience Part 2: Exploratory Data Analysis on CryptoPunks Transactions]]></title>
            <link>https://jeanna-schoonmaker.medium.com/cryptodatascience-part-2-exploratory-data-analysis-on-cryptopunks-transactions-910e6dcb14bd?source=rss-56cf328b663------2</link>
            <guid isPermaLink="false">https://medium.com/p/910e6dcb14bd</guid>
            <category><![CDATA[crypto]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nft]]></category>
            <dc:creator><![CDATA[Jeanna Schoonmaker]]></dc:creator>
            <pubDate>Wed, 03 Nov 2021 02:50:40 GMT</pubDate>
            <atom:updated>2021-11-03T03:07:08.217Z</atom:updated>
            <content:encoded><![CDATA[<p>Cleaning Blockchain Data for Data Science and Analysis</p><p>gm!</p><p>In my <a href="https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70">previous story about crypto data science</a>, I discussed several possible sources for accessing and downloading blockchain data. In Part 2, I will be exploring the data from two of the data sources mentioned in Part 1 in order to clean the data, analyze it, and prep it for being used in machine learning processes, which will be Part 3.</p><p>The two data sources I’m using for this article include the punks.csv dataset from the Omni Analytics course Learning Data Science on the Ethereum Blockchain found here: <a href="https://github.com/Omni-Analytics-Group/eth-data-science-course">https://github.com/Omni-Analytics-Group/eth-data-science-course</a> and the API from Flipside Crypto created by querying the Ethereum blockchain using the code shown in Part 1.</p><p>First, let’s start with the easy part — loading data from the csv.</p><h3>CSV Data</h3><pre>punks_url = &#39;https://raw.githubusercontent.com/Omni-Analytics-Group/eth-data-science-course/master/courses/Module%201/punks.csv&#39;</pre><pre>df = pd.read_csv(punks_url, encoding=&#39;unicode_escape&#39;)</pre><pre>df.head()</pre><p>will give us this:</p><figure><img alt="5 rows of a dataset showing CryptoPunks transactions" src="https://cdn-images-1.medium.com/max/1024/1*2JdV0wE3E_467dedib03gg.png" /></figure><p>Let’s dig into the data a little more.</p><figure><img alt="Table of metadata collected about the dataframe from the command df.info" src="https://cdn-images-1.medium.com/max/548/1*4JncWUmeapMWz-JpJjk5eA.png" /></figure><p>It looks like the “Txn” field has our date info, but is in string format. Let’s change it to datetime format and then take a look at the max and min dates in this dataset.</p><figure><img alt="Code showing converting a column to datetime, then finding the min and max dates within that column" src="https://cdn-images-1.medium.com/max/787/1*vai_YwYdYqVJb7NSX28nBQ.png" /></figure><p>Let’s explore further.</p><figure><img alt="df.describe and the resulting data table showing statistics associated with the fields" src="https://cdn-images-1.medium.com/max/1024/1*VNPUFYPpdMmmpO2r265V3Q.png" /></figure><h4>Initial thoughts after first pass at this data:</h4><ol><li>There are 17554 transactions in this dataset, and 10019 of them occurred on Jun 23, 2017 when the punks were first made available. <strong>More than half of our transactions in this dataset are from the initial ‘claims’ of the punks.</strong></li><li>The “From” and “To” fields indicate the number of unique users who transacted in this dataset. <strong>Interesting that with 10,000 punks, there are only 682 unique users on the selling side of the transaction, with 1524 users on the buying/claiming side of the transaction.</strong></li><li>The “Sex,” “Type,” and “Skin” columns refer to descriptive Punks attributes. The ID is the Punk identifier. “Slots” and “Rank” offer additional data about the sales of the Punks.</li><li>Ethereum addresses are 40 character hexadecimal strings. However, in this dataset, the “From” and “To” columns, which show the users buying and selling the Punks, have been shortened to 8 characters. Some addresses also use a name instead of the shortened hex string.</li><li>The column titled “Crypto” is referring to the cryptocurrency Ether. <strong>With a mean of 2.39 Eth/857.31 USD and a standard deviation of 6.12 Eth/2833.27 USD, we have a lot of variation in our data.</strong> Let’s take a look at it in a scatterplot.</li></ol><figure><img alt="Scatterplot graph of CryptoPunks transactions through 12/30/2020" src="https://cdn-images-1.medium.com/max/1024/1*zzF8PYePeJ3KVTcM_E-nOg.png" /></figure><p>There was scarce but steady CryptoPunks sales activity until 2020, but summer through fall of 2020 became much more active. There also appear to be a few outliers in our data.</p><p>One more quick analysis of this data — with only 682 users selling and 1524 users responsible for 17554 transactions, let’s see which users are responsible for the most buying and selling in this dataset.</p><figure><img alt="Data showing top users and sellers of CryptoPunks through 12/30/2020" src="https://cdn-images-1.medium.com/max/772/1*kDvipaifeG5I-tb8v8r3GQ.png" /></figure><p>It is great to be looking at understandable blockchain data, and pd.read_csv is one of the easiest ways to start exploring.</p><p>But what if you need to access raw blockchain data and clean it yourself?</p><p>Now let’s bring in 2021’s data from Flipside Crypto using the process I introduced in <a href="https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70">Part 1.</a></p><h3>Flipside Crypto Data</h3><p>Here is the code to bring in data from Flipside Crypto’s API:</p><pre>url = (&#39;https://api.flipsidecrypto.com/api/v2/queries/99990404-5727-40cb-989a-f9d66738bdca/data/latest&#39;)</pre><pre>response = requests.get(url)</pre><pre>print(f&quot;Request returned {response.status_code} : &#39;{response.reason}&#39;&quot;)</pre><pre>payload = response.json()</pre><p>Using json_normalize helps to unpack the json response, then let’s look at the first few rows of data.</p><pre>df_flip = pd.json_normalize(payload)</pre><pre>df_flip.head()</pre><figure><img alt="First few rows of transaction data retrieved from Flipside Crypto about CryptoPunks transactions" src="https://cdn-images-1.medium.com/max/1024/1*ps5CAoGrTogST4YoxVnlEA.png" /></figure><p>Looking interesting already, but we still have a nested json result in our EVENT_INPUTS column. We’ll use flat-table to unpack that, then take another look.</p><pre>df_flip[&#39;EVENT_INPUTS&#39;] = df_flip[&#39;EVENT_INPUTS&#39;].apply(lambda x:json.loads(x))</pre><pre>df_flip = flat_table.normalize(df_flip)</pre><pre>df_flip.head()</pre><figure><img alt="First few rows of modified data table with more unnested json data now visible" src="https://cdn-images-1.medium.com/max/1024/1*VK-iFv0AsX_wdo1uX8lmYw.png" /></figure><p>We can now see the EVENT_INPUTS values in their own columns based on the key names.</p><p>Unfortunately, we can also see that the EVENT_INPUTS.value (which is the amount paid for a transaction) has a negative value in a field which should not have negative values.</p><p><em>While I originally planned for this part to have EDA on datasets from both a csv and from the Flipside Crypto API, the data from Flipside is not accurate right now. I have submitted a ticket to Flipside to determine what has gone wrong. In the meantime I will share my thoughts and then move on with a different data source.</em></p><h4>Thoughts from this dataset:</h4><ol><li>The csv was much more nicely formatted. <strong>As is generally the case, raw data is messy, noisy, and likely needs quite a bit of work before it is understandable and/or useful.</strong></li><li>I see several columns we don’t need — as mentioned in Part 1, the contract address refers to the original address of the CryptoPunks contract, so it will be the same value for every column and can be dropped. Same with the “PunkBought” event_name. N<strong>ext steps will include dropping unnecessary columns.</strong></li><li>The To and From addresses in this dataset contain the entire hexadecimal string instead of just the first 8 characters. It does not include any named accounts, so <strong>we will need to reconcile named accounts with hex strings if we plan to join the csv dataset with this one.</strong></li><li>The EVENT_INPUTS.value field looks like our cryptocurrency amount for the transaction — but why is the number so large? Because <strong>it shows the transaction amount in wei </strong>(see here for more info: <a href="https://www.investopedia.com/terms/w/wei.asp">https://www.investopedia.com/terms/w/wei.asp</a>) <strong>which we will convert to Ether by dividing by 1e18.</strong></li><li>A key difference from our csv dataset — <strong>we don’t have the USD conversion in this dataset, so we will need to add that in ourselves</strong>.</li><li>Finally — data projects often don’t go as planned. <strong>Being able to pivot and augment or use alternate sources is an important skill when working with data!</strong></li></ol><p>As an alternate data source, I’ll be using the CryptoPunks transaction data from Kaggle, also mentioned in <a href="https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70">Part 1</a>. You can either download the file to your local computer, then upload it to your google drive to access it in Google Colab (as I’ve done below), or you can use it directly in Kaggle at <a href="https://www.kaggle.com/tunguz/cryptopunks">https://www.kaggle.com/tunguz/cryptopunks</a></p><figure><img alt="Table showing Punks transaction data" src="https://cdn-images-1.medium.com/max/1024/1*IOPDb-HV8yGnNXpj-3Akiw.png" /></figure><p>I am only interested in Punks sales and Punks transfers, and since our other csv has data through 12/30/20, we’ll filter this one to start where that one left off.</p><pre>df_alt[&#39;date&#39;] = pd.to_datetime(df_alt[&#39;date&#39;]).dt.date</pre><pre>start_date = startdate = pd.to_datetime(&#39;2020-12-30&#39;).date()</pre><pre>df_alt = df_alt.loc[(df_alt[&#39;txn_type&#39;] == &#39;Sold&#39;) | (df_alt[&#39;txn_type&#39;] == &#39;Transfer&#39;)].copy()</pre><pre>df_alt = df_alt.loc[df_alt[&#39;date&#39;] &gt; start_date].copy()</pre><figure><img alt="df_alt.describe() results" src="https://cdn-images-1.medium.com/max/1024/1*3wknTlrLbbov53ISiBY5Ng.png" /></figure><p>Interesting to note — in the <strong>3.5 year</strong> period from Jun 2017 thru Dec 2020, there were <strong>7554 sale transactions of CryptoPunks to 1524 unique users.</strong></p><p>In the <strong>9.5 month</strong> time frame from 12/31/20 to 10/7/21, there were over <strong>11,400 transactions of Punks to 5256 unique users</strong>.</p><p>The max price paid for a Punk through 12/30/20 was 189.99 Eth, worth roughly $140k at the time.</p><p>The max price paid for a Punk from 12/31/20 to Oct 2021 was 4200 Eth, worth roughly — well, I’m not sure yet. We need to put a USD-Eth conversion rate in our dataset!</p><p>Many thanks to this blog post that walks through how to access conversion rates and create a dataframe: <a href="https://blog.patricktriest.com/analyzing-cryptocurrencies-python/">https://blog.patricktriest.com/analyzing-cryptocurrencies-python/</a>. I will follow his process, so see his post for explanation and code, or see it in my Github <a href="https://github.com/JSchoonmaker/blockchain_projects/blob/main/CryptoPunks_EDA_Part_2.ipynb">here</a>.</p><p>Here’s how our dataframe of conversion rates looks:</p><figure><img alt="daily dataframe of Ether to USD conversion rates" src="https://cdn-images-1.medium.com/max/1024/1*4B_rCnTzG6fmwXF_NUiVvA.png" /></figure><p>Now that we have the conversion rate table, next steps include:</p><ul><li>Moving the date from being an index column to a regular date column</li><li>Renaming the ‘close’ column as our conversion rate column</li><li>Joining the conversion rate table to the df_alt Punks dataframe</li></ul><pre>crypto_price_df.reset_index(inplace=True)<br>crypto_price_df[&#39;date_join&#39;] = crypto_price_df[&#39;date&#39;].dt.date</pre><pre>rates = crypto_price_df[[&#39;date_join&#39;, &#39;close&#39;]].copy()<br>rates = rates.rename(columns={&#39;date_join&#39;: &#39;date&#39;, &#39;close&#39;: &#39;eth_usd_conversion_rate&#39;})</pre><pre>df_alt2 = pd.merge(df_alt, rates, on=&#39;date&#39;, how=&#39;left&#39;)</pre><p>Next we’ll multiply the Eth amount by the conversion rate to get USD for each transaction.</p><pre>df_alt2[&#39;USD&#39;] = df_alt2[&#39;eth&#39;] * df_alt2[&#39;eth_usd_conversion_rate&#39;]<br>df_alt2[&#39;USD&#39;] = pd.to_numeric(df_alt2[&#39;USD&#39;])<br>df_alt2.fillna(0, inplace=True)<br>df_alt2[&#39;type_str&#39;] = df_alt2[&#39;type&#39;].apply(lambda x: &#39;,&#39;.join(map(str, x)))</pre><p>which gives us</p><figure><img alt="df_alt2 describe info" src="https://cdn-images-1.medium.com/max/418/1*9dD7u0GSQpTLcBxGFbWY2A.png" /></figure><p>The minimum transaction amount in this dataset is 0 USD, but the max is a whopping $7,658,401!</p><p>Let’s do another scatterplot.</p><pre>import matplotlib.ticker as ticker<br>sns.set(style=&quot;darkgrid&quot;)<br>fig, ax = plt.subplots(figsize=(20, 8))</pre><pre>g = sns.scatterplot(x=df_alt2[&#39;date&#39;], y=df_alt2[&#39;USD&#39;], hue=df_alt2[&#39;type_str&#39;]).set(title=&#39;2021 Punks Purchases&#39;)<br>ax.yaxis.set_major_formatter(ticker.EngFormatter())</pre><figure><img alt="Scatterplot of 2021 Punks Purchases" src="https://cdn-images-1.medium.com/max/1024/1*xJQjJaMiktUjxbhcvekbaQ.png" /></figure><p>The biggest difference between our previous scatterplot on 2017–2020 data and this one is the scale of the y axis.</p><p>Our previous scatterplot’s y axis topped out at 140k.</p><p>This scatterplot’s y axis is now scaled to millions!</p><p>Finally, let’s take a look at this dataset’s top sellers and buyers:</p><figure><img alt="List of users with the most selling transactions in 2021 Punks data" src="https://cdn-images-1.medium.com/max/581/1*S_NuM-bA_WSf3VCLdOY5SQ.png" /></figure><figure><img alt="list of most common buyers in 2021 CryptoPunks dataset" src="https://cdn-images-1.medium.com/max/569/1*2jM-3M2ep59NljTPBpo1zQ.png" /></figure><p>Do any of these users look familiar from the top buyers and sellers in the 2017–2020 dataset…?</p><p>Now that we have loaded, cleaned, and explored our data, we’re ready to move forward with the final step in our data science project. More to come in Part 3!</p><p><em>All code used for this story can be found here: </em><a href="https://github.com/JSchoonmaker/blockchain_projects/blob/main/CryptoPunks_EDA_Part_2.ipynb"><em>https://github.com/JSchoonmaker/blockchain_projects/blob/main/CryptoPunks_EDA_Part_2.ipynb</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=910e6dcb14bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crypto Data Science: Comparing Five Options for Accessing Blockchain Data]]></title>
            <link>https://jeanna-schoonmaker.medium.com/crypto-data-science-comparing-five-options-for-accessing-blockchain-data-619f5f4e2f70?source=rss-56cf328b663------2</link>
            <guid isPermaLink="false">https://medium.com/p/619f5f4e2f70</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[nft]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Jeanna Schoonmaker]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 04:14:15 GMT</pubDate>
            <atom:updated>2021-10-21T00:34:27.959Z</atom:updated>
            <content:encoded><![CDATA[<p>Part 1 in a series of posts about data science and machine learning on blockchain data</p><p><em>If you are new to learning about blockchain, I recommend reading an intro like this: </em><a href="https://www.geeksforgeeks.org/blockchain-technology-introduction/">https://www.geeksforgeeks.org/blockchain-technology-introduction/</a> <em>My post assumes the reader has a basic understanding of how blockchains function.</em></p><p>gm!</p><p>(that’s ‘good morning’ in crypto circles)</p><p>On any given day, I see dozens of tweets about blockchain, NFTs, web3, crypto, etc. Many will address the potential, the tech, and the hype. Others will point out the scams, the lack of regulations, and well, the hype. Like a lot of hot topics right now, the polarization of opinions seems stark. You’re either all in on the crypto train, or you’re dismissing it as a Star Registry-esque grift.</p><h3>The Why</h3><p>So why explore blockchain data?</p><p>As exciting as hype cycles are, and as promising as tech twitter makes blockchain seem, W. Edward Deming said:</p><blockquote>“In God we trust.</blockquote><blockquote>All others must bring data.”</blockquote><p>That goes for both the pro and anti crypto camps, in my opinion. Reading about emerging tech whose selling points include data that is readily available and immutable was enough to get me interested, but the critics of crypto may be right. Let’s dig into the data and see what we find. We’ll start with a well known NFT, or non-fungible token, example — transactions for CryptoPunks, which are on the Ethereum blockchain.</p><p>Please note, I am new to working with crypto data, so I am not representing any of my research or recommendations here as an expert opinion. Feel free to add a comment about options I missed, things that could have been done better, or new things to try!</p><h3>The Where</h3><p>With ease of access to blockchain data being a major talking point of the blockchain system, I was surprised to find out that getting a csv of a subset of transactions from a blockchain was not as easy as I expected.*</p><p>It is an option to install web3.py and query a blockchain directly, but as of 10/18/21 when writing this, <a href="https://bitinfocharts.com/ethereum/">bitinfocharts.com</a> shows that the Ethereum blockchain size is over 345 GB. It seemed unnecessary to download the entire Ethereum blockchain when I only wanted to look at a small subset of transactions that have occurred on it.</p><p>Instead, I explored 5 services that provide options for querying Ethereum data. They are:</p><ol><li><a href="https://etherscan.io/">Etherscan.io</a></li><li><a href="https://thegraph.com/en/">Thegraph.com</a></li><li><a href="https://dune.xyz/home">Dune Analytics</a></li><li><a href="https://graphql.bitquery.io/ide">Bitquery.io</a></li><li><a href="https://www.flipsidecrypto.com/">Flipside Crypto</a></li></ol><p>My criteria for using a service to query the data are pretty simple and encompass what I feel most data scientists would find reasonable for a personal project:</p><ul><li><strong>No crypto wallet required</strong>. Much like the annoyance of needing to enter a credit card number when signing up for a free trial, I had no interest in going through the steps to set up a wallet. Perhaps someday having a crypto wallet (or multiple wallets) will be as ubiquitous as having an email address. For now, though, I simply wanted to access data.</li><li><strong>It should be free</strong>. Writing to a blockchain has a cost, but reading/viewing a blockchain is free, and since I am new to learning about this data space, I didn’t want to rack up a lot of charges without even knowing if I’d get the data I wanted.</li><li><strong>It should use a language most data scientists know, like SQL</strong>.</li><li><strong>It should output the data into either a csv or an API </strong>for further use in machine learning tasks.</li><li>While not required, <strong>a ‘playground’ to test and see the results from queries is super useful</strong>, especially when you’re new and don’t know what you’re doing (hi!). Connecting to an API only to find out your dataset is empty is beyond frustrating.</li></ul><p>So let’s review our options and how they stack up to the criteria.</p><p><a href="https://etherscan.io/token/0xb47e3cd837ddf8e4c57f05d70ab865de6e193bbb">Etherscan.io</a> is a powerful search platform for crypto transactions that I used several times while checking info for this post, but in order to access NFT-specific transactions via their API, you have to sign up for a Pro account, which has a cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vfX86QqZg9L7XWlxKwmx0w.png" /><figcaption>Screenshot of Etherscan.io’s CryptoPunks page (image by author)</figcaption></figure><p><a href="https://graphql.bitquery.io/ide/SL2wWtYlhv">Bitquery.io</a> uses a graphql IDE, which is a very intriguing option that I’d like to learn more about in future projects — but for the purposes of this project, there were enough new pieces in play without needing to adopt a new language syntax as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*828il01a71Vxjt1qrIXiKw.png" /><figcaption>Screenshot of Bitquery.io’s graphql IDE (image by author)</figcaption></figure><p><a href="https://dune.xyz/rantum/NFT-Sales-Overview-by-Project">Dune.xyz</a> is a terrific analytics platform for blockchain data, with many publicly shared queries by people who write much better SQL than I do. The output of Dune’s queries can be graphs or dashboards, but only paid users can output to csvs, and no APIs are offered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lXtXDuKUX2Ia0sJJ4zlBrw.png" /><figcaption>Screenshot of Dune.xyz dashboard by user RantumBits (image by author)</figcaption></figure><p>Flipside Crypto and Thegraph.com were both worth exploring further for querying blockchain data.</p><p><a href="https://thegraph.com/hosted-service/">Thegraph.com</a> does require a crypto wallet if you are accessing their Subgraphs service, but if you are accessing legacy data via their Hosted Service, you do not need to connect a wallet. According to their docs, while the Hosted Service will eventually no longer be supported, <a href="https://thegraph.com/docs/hostedservice/what-is-hosted-service">it will still be around for a while</a>. Thegraph actually uses graphql like bitquery.io, but their example queries made their service accessible despite that.</p><p><a href="https://www.flipsidecrypto.com/">Flipside Crypto</a> requires logging in with an email address and password, and is set up to provide bounties (usually in the form of crypto tokens) for doing analytics tasks on blockchain data. However, I’m most impressed by how they allow you to either download a csv of query results OR access the results data via an API.</p><h3>The What</h3><p>What will we be using Flipside Crypto and Thegraph.com to explore? As mentioned earlier, we will be querying the Ethereum blockchain for CryptoPunks sales transactions.</p><figure><img alt="Cryptopunk image of female punk with brown hair and glasses" src="https://cdn-images-1.medium.com/max/336/1*InyquezW1HMNJ6Se2rUlFw.png" /><figcaption>CryptoPunk 8771. I think there’s a bit of a resemblance. (img from <a href="https://www.larvalabs.com/cryptopunks/details/8771">CryptoPunks: Details for Punk #8771 (larvalabs.com)</a>)</figcaption></figure><p>CryptoPunks are 24x24 pixel portraits that were originally offered by Larva Labs for free in June 2017. The only cost associated with them was the gas fee needed to write the transaction to the blockchain and ‘claim’ one of the 10,000 Punks. At the time, the gas fee was as low as 11 cents.</p><p>Following their inauspicious beginnings, CryptoPunks exploded in popularity in late 2020 and 2021. Prices for Punks now range from 30k to as much as 10 million dollars. <a href="https://techcrunch.com/2021/04/08/the-cult-of-cryptopunks/">(https://techcrunch.com/2021/04/08/the-cult-of-cryptopunks)</a></p><p>From my reading, CryptoPunks seem to be the OG NFTs that paved the way for CryptoKitties, BoredApeYachtClubs, and many other NFTs that have followed.</p><h3>The How</h3><p>Now that we know what we’re looking at, why, and where to find it, how do we access the data?</p><h4>Flipside Crypto</h4><ol><li>Log in to your account on app.flipsidecrypto.com, then click on the ‘New’ button in the upper right hand corner of your screen and then click on ‘New Query.’</li><li>On the left hand side of your screen, there is a dropdown menu that lets you choose from the blockchains available to query on Flipside.</li><li>Click on Ethereum to see the tables you can query within that blockchain.</li><li>Click on the small table avi next to one of the table names to get a list of columns within that table.</li><li>Type your SQL query in the middle window.</li><li>Click Run, and get immediate feedback on whether your query worked or not!</li></ol><p>To begin, I recommend trying an easy query just to get a better idea of what the fields are and what you might want to filter by. Here’s an example of when I selected 3 columns (contract_name, contract_address, and block_timestamp) from the ethereum.events_emitted table, limiting my results to 10:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rxr5s9SEKk4_BByGY2ugtw.png" /><figcaption>Query results from the Ethereum blockchain by Flipside Crypto (img by author)</figcaption></figure><p>Contract_address is a reference to the 42 character hexadecimal address location of the actual token contract that manages the logic for the tokens. For Cryptopunks, the contract address is ‘0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB’. That address will not change and will be referenced every time a CryptoPunks transaction occurs. Contract addresses can be found by using Etherscan.io to look up an NFT or token by name, or by googling the token.</p><blockquote>Flipside Crypto note — while contract addresses are not case sensitive elsewhere, on Flipside all contract addresses are in lower case. Make sure you either modify any contract addresses you copy/paste in or wrap them in the lower() SQL command to save yourself from empty result dataset frustration!</blockquote><p>Now let’s actually query for CryptoPunks sales transactions. ALL transactions (including bids offered, bids withdrawn, transfers, etc) get recorded on the blockchain, but I am only interested in purchases, so I filtered by the event_name ‘PunkBought.’</p><p>There are much better ways to write the SQL query so that you pull in less unnecessary data and return only what you need in your final dataset, but since this is an exploratory project, I selected everything from these transactions and will later decide what to keep and what to drop.</p><p>My SQL query for Flipside Crypto:</p><pre>select *<br>from ethereum.events_emitted<br>WHERE contract_address = lower(&#39;0xb47e3cd837dDF8e4c57F05d70Ab865de6e193BBB&#39;)<br>  AND event_name = &#39;PunkBought&#39;<br>  AND TX_SUCCEEDED = &#39;TRUE&#39;<br>LIMIT 100000</pre><p>When my results displayed, I clicked on the ‘API’ button, verified that it contained the data I expected, then copied and pasted the API’s url to use in my code, which we’ll see after the next section.</p><h4>Thegraph.com</h4><p>Using this url: <a href="https://thegraph.com/hosted-service/subgraph/itsjerryokolo/cryptopunks?query=Example%20query">https://thegraph.com/hosted-service/subgraph/itsjerryokolo/cryptopunks?query=Example%20query</a>, we can see an example of how to query the specific blockchain transactions that are related to CryptoPunks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVk4huSge_vQd12tY7JvhQ.png" /><figcaption>TheGraph.com’s Hosted Service Query Playground (img by author)</figcaption></figure><ol><li>On the right hand panel, there is a list of the schema available in this subgraph.</li><li>Click on the ‘Sale’ schema to see the columns available from that table.</li><li>Using the example query in the left hand panel as a guide, rewrite the query to look at all of the available information in the Sale schema (see query below).</li><li>Then hit the purple ‘play’ button over the middle panel, and your results will show up in that middle panel. Note — Thegraph limits the results returned to 1000, so this dataset will be incomplete and we will either need to download multiple versions of it or figure out another way to access more of the data from Thegraph.com.</li></ol><pre>{sales(first: 1000){<br>    id<br>    to {<br>      id<br>    }<br>    amount<br>    from {<br>      id<br>    }<br>    nft {<br>      id<br>    }<br>    timestamp<br>    type<br>    contract {<br>      id<br>    }}}</pre><p>Despite the fact that we are pulling the same information from the same blockchain, you can see there are much different approaches to accessing the data depending on which service you use.</p><p>Now that we have working queries for both services, let’s connect to them using a Google Colab notebook and pull the data into dataframes for exploratory analysis and eventually, anomaly detection. (spoiler alert!)</p><p>Using Flipside Crypto:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5353e3f84678356ac37ef8342ad3c17/href">https://medium.com/media/b5353e3f84678356ac37ef8342ad3c17/href</a></iframe><p>Using Thegraph.com:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f51d195e2951ea8b9ec38a14754de93f/href">https://medium.com/media/f51d195e2951ea8b9ec38a14754de93f/href</a></iframe><p>That’s it for Part 1. Part 2 will explore the CryptoPunks transaction data itself. See you then!</p><p>gn.</p><p><em>* NOTE: There are some already-existing csvs of Cryptopunks data available, including one that was added while I was researching this post: </em><a href="https://www.kaggle.com/tunguz/cryptopunks"><em>https://www.kaggle.com/tunguz/cryptopunks</em></a></p><p><em>You can also find cryptopunks transaction data through 12/30/2020 in this dataset provided by Omni Analytics for their Learning Data Science on the Ethereum Blockchain: </em><a href="https://github.com/Omni-Analytics-Group/eth-data-science-course"><em>https://github.com/Omni-Analytics-Group/eth-data-science-course</em></a></p><p><em>Despite the immutability of blocks already written to the chain, new blocks are added constantly, so be sure to check how often datasets are updated if the recency of your blockchain data matters.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=619f5f4e2f70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing 4 methods for pdf text extraction in python]]></title>
            <link>https://medium.com/social-impact-analytics/comparing-4-methods-for-pdf-text-extraction-in-python-fd34531034f?source=rss-56cf328b663------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd34531034f</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[social-impact]]></category>
            <category><![CDATA[pdf]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nonprofit]]></category>
            <dc:creator><![CDATA[Jeanna Schoonmaker]]></dc:creator>
            <pubDate>Wed, 24 Mar 2021 16:46:11 GMT</pubDate>
            <atom:updated>2021-03-24T16:46:11.714Z</atom:updated>
            <content:encoded><![CDATA[<p>Accuracy and processing time for PyPdf2, PdfMiner.six, Grobid, and PyMuPdf</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFXIoB2tmyCdKmdzlvKl8Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@imandrewpons?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Andrew Pons</a> on <a href="https://medium.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><em>In comparing 4 python packages for pdf text extraction, PyMuPdf was found to be an optimum choice due to its low Levenshtein distance, high cosine and tf-idf similarity, and fast processing time, though all 4 packages performed very well in general and Grobid produced the cleanest text output. All code provided at github link at the end of the article.</em></p><p>At Social Impact Analytics Institute, we are working to gather information and find patterns in the collected data about social issues. Many articles and primary sources of information are stored as pdfs.</p><p>Despite Portable Document Format, or pdf, being one of the most common formats for document storage, it is not standardized. Pdfs can vary from being scanned copies of old documents to being computer-generated articles, which affects how well a program can “read” the text within a pdf. Since SIAi’s text data will be used for NLP, sentiment analysis, and further data exploration, it is critical that the extracted text be as accurate as possible. Since we anticipate needing to process thousands of pdfs, it’s also important that our process be time-friendly. We compared 4 open-source methods in python for text extraction from pdfs with these guidelines in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*sT5ufsS7eoT8dm58GvEb-A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*5pAYmoW2UP5b8KR3WGKtIQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dh-lPy0R0XCnQ8k96D9bhQ.jpeg" /></figure><p>Three of the packages tested — PyPdf2, PdfMiner.six, and PyMuPdf — can be pip installed. Grobid, which stands for “GeneRation Of BIbliographic Data,” requires additional installation steps, which are clearly laid out in its documentation: <a href="https://grobid.readthedocs.io/en/latest/Install-Grobid/">https://grobid.readthedocs.io/en/latest/Install-Grobid/</a> Grobid uses machine learning to process pdfs into xml files which can then be processed for text extraction. This is a multi-step task, which makes it a bit of an outlier in this comparison, but we feel its inclusion is warranted due to the nature of its output as discussed below.</p><p>We used a file called “Threat Assessment in the Campus Setting,” which can be found at this link: <a href="https://schoolshooters.info/sites/default/files/2014-NaBITA-Whitepaper-Text-with-Graphics.pdf">https://schoolshooters.info/sites/default/files/2014-NaBITA-Whitepaper-Text-with-Graphics.pdf</a> to compare these methods. This file seemed representative of a typical pdf article due to its use of graphics, footnotes, bulleted lists, etc (thumbnails of sample pages shown below). The file was converted to .txt type to use as a baseline for comparison. Pdf and txt versions of this file, as well as the xml version created through the Grobid process, can be found in the github link at the end of this article.</p><figure><img alt="Thumbnails of a pdf document" src="https://cdn-images-1.medium.com/max/1024/1*kwcXPPhKKIKXp79hzAYjMQ.png" /></figure><p>Each python package was evaluated by comparing its text output with the baseline .txt file. The time it took each process to run inside a jupyter notebook cell was also measured and compared. As can be seen in the following code, some options have multiple steps for the pdf to go through, while others are more straightforward. In each case, we referred to the documentation for the python package to ensure we used the correct steps to extract text.</p><p><strong>PyMuPdf text processing code:</strong></p><pre>import sys, fitz<br>fname = &#39;./2014-NaBITA-Whitepaper-Text-with-Graphics.pdf&#39;<br>doc = fitz.open(fname)  <br>text = &#39;&#39;<br>for page in doc:  <br>    text += page.getText() <br><br>pymupdf_test = text</pre><p><strong>Pdfminer.six text processing code:</strong></p><pre>def pdf_to_txt(path):<br>    from io import StringIO<br><br>    from pdfminer.converter import TextConverter<br>    from pdfminer.layout import LAParams<br>    from pdfminer.pdfdocument import PDFDocument<br>    from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter<br>    from pdfminer.pdfpage import PDFPage<br>    from pdfminer.pdfparser import PDFParser<br><br>    output_string = StringIO()<br>    with open(path, &#39;rb&#39;) as in_file:<br>        parser = PDFParser(in_file)<br>        doc = PDFDocument(parser)<br>        rsrcmgr = PDFResourceManager()<br>        device = TextConverter(rsrcmgr, output_string, laparams=LAParams())<br>        interpreter = PDFPageInterpreter(rsrcmgr, device)<br>        for page in PDFPage.create_pages(doc):<br>            interpreter.process_page(page)<br>    text = str(output_string.getvalue())<br>    return text<br><br>file = &#39;./2014-NaBITA-Whitepaper-Text-with-Graphics.pdf&#39;<br><br>pdfminersix_test = pdf_to_txt(file)</pre><p>The text outputs from each package were then compared with the baseline .txt file and used to calculate the <a href="https://en.wikipedia.org/wiki/Levenshtein_distance">Levenshtein distance</a>, <a href="https://en.wikipedia.org/wiki/Cosine_similarity">Cosine similarity</a>, and <a href="https://www.jeremyjordan.me/identifying-related-bodies-of-text-using-tf-idf-vectorization/">Tf-idf similarity</a>.</p><p>An in-depth explanation of each of the measurements can be found at the links provided, but in general:</p><ol><li>Levenshtein distance is measured by counting each character difference between one string and another, with a desired output of zero. This would mean there are zero differences between the characters and spacing in one string and another.</li><li>Our Cosine similarity process uses a word count to create a vector for each string, then measures the cosine of the angle between vectors to determine how similar word counts between the two strings are.</li><li>Tf-idf similarity measures for the similarity in frequency of words between strings. In both cosine and tf-idf similarity, the desired output is 1.0, meaning the strings are the same in word count and word frequency.</li></ol><p>A note about Levenshtein distances and Grobid — when the .txt file was created from our source article, it kept ALL text from the original article in place, including page numbers, headers, footers, etc., exactly in the place they were found on the page. The process of converting the file to .txt, as well as the process to extract text from pdfs using most of these python methods, doesn’t classify the different parts of the document. Instead, it keeps everything, which results in page numbers inserted into the middle of sentences, as shown by the page number “2” being included in the sentence here (emphasis added):</p><blockquote>“NaBITA has come to realize over the last five years that this essential function can be more accurately, affordably and accessibly provided <strong>within and by 2 the</strong> team through the SIVRA-35.”</blockquote><p>While Grobid’s Levenshtein distance looks high in comparison to the other text extraction methods, it is worth noting that Grobid’s machine learning and computer vision approach to categorizing the text from a document into an xml output separates the page numbers, headers, and footers from the text, and doesn’t include them in the text output. This makes the Grobid Levenshtein distance high when compared to the .txt file, but also means <strong>the Grobid text output was actually the cleanest because it did not include page numbers and other noise</strong>. Despite the complexity associated with installing and processing pdfs using Grobid, its clean text output makes it a very good option when working with pdfs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BFoi9AWXCakCZrzFl4DRDA.png" /></figure><p>In the results table, the baseline_string row compared the txt file with itself, resulting in a perfect score for Levenshtein distance, cosine and tf-idf similarity, which makes sense since the two documents being compared in this case were identical. For processed pdf files, PdfMiner.six had the lowest Levenshtein distance. PdfMiner.six and PyMuPdf have identical tf-idf and cosine similarity scores, despite PyMuPdf having a slightly higher Levenshtein distance. This likely means the difference between the two Levenshtein distances is based on spaces or characters, but that both methods extracted the words within the text similarly well. However, Pdfminer.six took nearly 2.5 seconds to run while PyMuPdf processed the pdf in 42 milliseconds. While 2.5 seconds does not sound like much, when multiplied by hundreds or thousands of documents, it would be tough to justify using a process that takes 6 times as long to complete when other measurements are similar.</p><p>Ultimately, all of the methods tested produced very accurate text results in relatively short amounts of time and would work well for most use cases. The full code for all steps described in these tests can be found here: <a href="https://github.com/JSchoonmaker/PDF-Text-Extraction">https://github.com/JSchoonmaker/PDF-Text-Extraction</a></p><p>Once you’ve extracted text from pdfs, an important next step could include spell-checking — check out SIAi team member Elisa Ponte’s article about python packages for spell checking here: <a href="https://medium.com/social-impact-analytics/spell-checking-computer-extracted-text-from-pdfs-9390c05bb2e5">https://medium.com/social-impact-analytics/spell-checking-computer-extracted-text-from-pdfs-9390c05bb2e5</a></p><p>Visit <a href="http://www.siainsitute.org/">www.siainsitute.org</a> to learn more about the research we are doing at Social Impact Analytics Institute.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd34531034f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/social-impact-analytics/comparing-4-methods-for-pdf-text-extraction-in-python-fd34531034f">Comparing 4 methods for pdf text extraction in python</a> was originally published in <a href="https://medium.com/social-impact-analytics">Social Impact Analytics</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>