<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kaan Erkoç on Medium]]></title>
        <description><![CDATA[Stories by Kaan Erkoç on Medium]]></description>
        <link>https://medium.com/@mkaanerkoc?source=rss-58dd91dc1574------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5GWuZc5NLIsYx25OFKv6Og.png</url>
            <title>Stories by Kaan Erkoç on Medium</title>
            <link>https://medium.com/@mkaanerkoc?source=rss-58dd91dc1574------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 10:40:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mkaanerkoc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The most important tool for NFT valuation, the Jaccard distance]]></title>
            <link>https://medium.com/coinmonks/the-most-important-tool-for-nft-valuation-the-jaccard-distance-ea1848689f34?source=rss-58dd91dc1574------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea1848689f34</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[scipy]]></category>
            <category><![CDATA[nft]]></category>
            <category><![CDATA[rarity]]></category>
            <category><![CDATA[jaccard-index]]></category>
            <dc:creator><![CDATA[Kaan Erkoç]]></dc:creator>
            <pubDate>Fri, 18 Nov 2022 22:35:35 GMT</pubDate>
            <atom:updated>2022-11-22T15:45:30.245Z</atom:updated>
            <content:encoded><![CDATA[<p>Non-fungible tokens, or NFTs, entered our lives at the speed of light, and obviously, 2021 was an awesome year for NFTs. The concept of digital ownership enables innovative solutions for broad use cases, such as gaming, art, sports, entertainment, media, and real estate. As a result of that, the NFT hype reached staggering proportions in 2021 and 2022.</p><blockquote><a href="https://blog.liquid.com/what-is-nft">Non-fungible tokens (NFTs)</a> make it possible to tokenize and prove ownership of real-world or digital assets through <a href="https://blog.liquid.com/what-are-smart-contracts">smart contracts</a>. <a href="https://blog.liquid.com/nft-use-cases">[1]</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*drzyYZLiuQJ2QYRyrcZV2A.png" /><figcaption>Google trends results of ‘NFT’ <a href="https://trends.google.com/trends/explore?date=2020-01-01%202022-11-13&amp;q=NFT">[2]</a></figcaption></figure><p>An <strong>NFT collection</strong> is an assortment of digital assets released by an artist, which may range from a single token to many. It contains a limited number of individual NFTs which all conform to the same art style, but with slight variations and differences that may contradict each other yet still be appealing.</p><p>A great example of this would be the <a href="https://opensea.io/collection/boredapeyachtclub">Bored Ape Yacht Club</a>, an NFT collection that’s one of the most popular and valuable around. This specific collection is made up of 10,000 unique non-fungible tokens on the Ethereum blockchain depicting simian avatars with various characteristics. For instance, only 5% of Bored Apes have red fur and 3% sport biker vest. The more <strong>scarce</strong> a Bored Ape’s features, the <strong>higher</strong> price it tends to fetch on the market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9obU2T_aNXEphQiHJoDxA.jpeg" /><figcaption>BAYC, Bored Ape Yacht Club banner</figcaption></figure><p>In essence, what makes an NFT relatively valuable in the collection is how <strong>unique</strong> the <strong>attributes </strong>it has. It basically means that any metric which gauges the <strong>uniqueness</strong> or <strong>rareness</strong> of an NFT can easily determine whether it is overpriced, underpriced, or fairly priced.</p><p>However<strong>, rarity </strong>does not have a single, straightforward mathematical definition. There are different approaches like distribution-based ones and distance-based ones. Personally, I think each of the different methods has its own pros and cons. Because of the exact same reasons, the most known NFT marketplace, OpenSea, created its own protocol for rarity calculation. <a href="https://support.opensea.io/hc/en-us/articles/9517355517843-What-is-OpenRarity-">OpenRarity</a> aims to bring an industry-wide definition of NFT rarity.</p><p>Based on our past experiences with quite a broad of NFT collections, we’ve decided to use <strong>Jaccard distance</strong> as our instrument since it is the most reliable way to calculate rarity scores.</p><p>In this article, we’ll explore the Jaccard distance, how it’s used in NFT rarity calculation and some of the advantages and disadvantages of using this metric.</p><p>You can find the notebook and the data I used in this article in my Github account.</p><p><a href="https://github.com/mkaanerkoc/nft-rarity">GitHub - mkaanerkoc/nft-rarity: Experimental work for NFT rarity calculation</a></p><h3>Jaccard distance</h3><p>The Jaccard distance is a simple and effective way to measure the similarity of two sets of data. It is based on the concept of set intersection and union and can be used to compare any two sets of data, no matter their size or complexity.</p><p>To use the Jaccard distance, simply compare the two sets of data and calculate the ratio of items that are common in both sets to the total number of items of sets. This ratio is called the Jaccard index and can be expressed as a percentage or a decimal value. The closer the index is to 1 (or 100%), the more similar the two sets of data are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*gHAI9egqfrKwCB0csgcsow.png" /><figcaption>Jaccard index formula</figcaption></figure><p>As an example, let’s say there are three MMORPG characters with the following types of equipment on them:</p><p><strong>Character 1, Thorr Moddon</strong></p><ul><li>Weapon: Iron Sword</li><li>Body Armor: Worn Chain</li><li>Shoulder: Ogre</li><li>Amulet: Copper</li></ul><p><strong>Character 2, Marmy Onrett</strong></p><ul><li>Weapon: Iron Sword</li><li>Body Armor: Worn Chain</li><li>Shoulder: Barbaric</li><li>Amulet: Copper</li></ul><p><strong>Character 3, Sassayl Delron</strong></p><ul><li>Weapon: Mithril Axe</li><li>Body Armor: Draconic</li><li>Shoulder: Barbaric</li><li>Amulet: Copper</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*eu3cmW8QqnKZtjU2jI45Kg.png" /><figcaption>Venn diagram of characters</figcaption></figure><p>By just looking at their equipment can you guess which character is rarer than the others?</p><p>To answer that question we first need to calculate the pair-wise Jaccard distances of each character.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdgRK3105hksaBhTe950Vw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QB5Es-q6SDlqOeqS2xDJiw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dYkQ3DGbeHWgyeD0wGABSQ.png" /></figure><p>It seems like the Jaccard index drops significantly whenever Sassayl Delron is taken into calculation. This implies that Sassayl Delron has fewer common features or attributes than other characters. It does not yet tell us the order of rarity, but at least we can use this example as a ground-truth reference. Note that every equipment type, like Weapon and Body Armor, is taken into account equally as they are added together with the weight of 1.</p><h3>Implementation</h3><h4>Jaccard distance with Scipy</h4><p>Even though the Jaccard index formula looks easy to implement from scratch, we will use the <strong>Scipy</strong> package for calculating the Jaccard distance. If it is already implemented, why bother ourselves to implement the same functionality, unless it is needed, right?</p><p>Let’s have a look at the functions that we will use;</p><p><strong>scipy.spatial.distance.pdist</strong></p><p>Pairwise distances between observations in n-dimensional space.[3]</p><p>Parameters:</p><ul><li><strong>X, </strong>array_like: An m by n array of m original observations in an n-dimensional space.</li><li><strong>metric, </strong>str, or function, optional. The distance metric to use. The distance function can be ‘braycurtis’, ‘canberra’, ‘chebyshev’, ‘cityblock’, ‘correlation’, ‘cosine’, ‘dice’, ‘euclidean’, ‘hamming’, ‘<strong>jaccard</strong>’, ‘jensenshannon’, ‘kulczynski1’, ‘mahalanobis’, ‘matching’, ‘minkowski’, ‘rogerstanimoto’, ‘russellrao’, ‘seuclidean’, ‘sokalmichener’, ‘sokalsneath’, ‘sqeuclidean’, ‘yule’.</li></ul><p>Returns:</p><ul><li><strong>Y, </strong>ndarray. Returns a condensed distance matrix Y. For each and (where ), where m is the number of original observations. The metric dist(u=X[i], v=X[j]) is computed and stored in entry m * i + j - ((i + 2) * (i + 1)) // 2.</li></ul><p><strong>scipy.spatial.distance.squareform</strong></p><p>Convert a vector-form distance vector to a square-form distance matrix, and vice-versa. [4]</p><p>Parameters:</p><ul><li><strong>X, </strong>array_like. Either a condensed or redundant distance matrix.</li></ul><p>Returns</p><ul><li><strong>Y, </strong>ndarray: If a condensed distance matrix is passed, a redundant one is returned, or if a redundant one is passed, a condensed distance matrix is returned.</li></ul><p>Before we apply these methods to the whole NFT collection I would like to demonstrate a quick example with the above characters first. In that way, it is much easier to validate whether we are using the scipy functions correctly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/577c9ad557178b70af668015963b5775/href">https://medium.com/media/577c9ad557178b70af668015963b5775/href</a></iframe><p>Notice that I used a specific format for describing the name and attributes of each character. This is just for making our example complaint with the NFT metadata format that we will use later on. In this way, we can use the same functions without any modifications. 🤗</p><p>The workflow of calculating Jaccard distance consists of 4 steps;</p><ul><li>Convert collection metadata into pandas.DataFrame</li><li>Calculate one-hot encoded attributes DataFrame</li><li>Calculate the Jaccard distance matrix</li><li>Calculate the mean of Jaccard distances of each NFT and sort them.</li></ul><p>The below function creates pandas.DataFrame from collection metadata. It should be noted that it assigns a unique ID for every &lt;trait_type, value&gt; pair.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf84553613b49ee6a518b939a6c68a51/href">https://medium.com/media/cf84553613b49ee6a518b939a6c68a51/href</a></iframe><p>And this is what our DataFrame looks like after the first step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3NTPin8yQcOkqzljXsqoLA.png" /></figure><p>The Scipy <strong>pdist </strong>function expects input arguments for Jaccard distance in boolean vector format. Boolean vectors are also known as <a href="https://en.wikipedia.org/wiki/One-hot">one-hot encoded</a> feature vectors in the Machine learning field. The function below calculates one-hot encoded attribute data frame;</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/379e1ab113d0cc4716dd99d39d4b9394/href">https://medium.com/media/379e1ab113d0cc4716dd99d39d4b9394/href</a></iframe><p>Let’s have a look at the output;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rccFj2vFN_ppNrk6TdPrpg.png" /><figcaption>one-hot encoded attributes of characters</figcaption></figure><p>In total, we have 7 different unique attributes across 3 game characters. ID#0, which is Thorr Moddon, has attributes Worn Chain, Iron Sword, and Ogre amulet, so only these attribute columns are <strong>True</strong> whereas the rest is <strong>False. </strong>You can also validate the other characters by yourself.</p><p>As we have our character&#39;s attributes in the one-hot encoded format we can proceed with the Jaccard distance matrix calculation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vl1-XrWTDdySgqXDEtebGg.png" /><figcaption>Jaccard distance matrix calculation</figcaption></figure><p><strong>pdist</strong> function returns condensed distance matrix. Therefore we used the <strong>squareform</strong> function to convert it back to a redundant distance matrix to easily visualize distances.</p><p>Let’s compare the results with our ground-truth reference.</p><ul><li>Distance between #0 (Thorr Moddon) and #1 (Marmy Onrett) is 0.6 ✅</li><li>Distance between #1 (Marmy Onrett) and #2 (Sassayl Delron) is 0.33 ✅</li><li>Distance between #0 (Thorr Moddon) and #2 (Sassayl Delron) is 0.14 ✅</li></ul><p>Congratulations! We successfully used scipy functions to calculate Jaccard distances and verified them against our ground-truth reference. The next and last step is to sort the characters based on their Jaccard distances from other characters.</p><p>To sort characters concerning their rareness, we need to find a single scalar metric. Since we calculate pair-wise distances, we can just calculate the mean of pair distances for each character. <strong>Please note that the less the Jaccard score, the rarer the character is.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OoWmBE_pM8eWfYrzkcsy3A.png" /><figcaption>rarity scores of each character</figcaption></figure><p>And normalize rarity scores</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PiVsX-xuJ5yRu85N0Fmi2w.png" /><figcaption>normalized rarity scores</figcaption></figure><p>The above results tell us the rarest character is Sassayl Delron. The second rare one is Thorr Moddon and the least rare is Marmy Onrett. By just looking at their Venn diagram, can you address why the rarity calculation is producing sensible results?</p><h4>Applying Jaccard Distance for NFT collections</h4><p>As we verified that our algorithm generates usable results for rarity scoring, we can indeed scale up with real NFT collection data.</p><p>NFT collection data is stored at remote servers and accessible via protocols such as HTTP/HTTPS or IPFS most of the time. Each NFT item in the collection has a unique ID that we are going to use for requesting metadata.</p><p>I chose the <a href="https://opensea.io/collection/boredapeyachtclub">Bored Ape Yacht Club</a>, <a href="https://opensea.io/collection/proof-moonbirds">MoonBirds</a>, and <a href="https://opensea.io/collection/azuki">Azuki</a> collections as examples for the rest of the article.</p><p>Those who do not want to deal with fetching the NFT metadata can directly proceed with the .json files in my GitHub repository.</p><p>Fetching the NFT collection data starts with finding the Base URL that consecutively stores all the NFTs metadata. To find that, the collection’s smart contract is the best trustful source of truth. We can find the collection’s smart contract address on OpenSea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D45VcpMO50wckcXG2ca2Tg.png" /><figcaption>BAYC collection’s OpenSea page</figcaption></figure><p>Once we get the etherscan.io page loaded with the collection’s smart contract, we can check the <strong>Read Contract</strong> to find the base URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q71Q7bBfR-sHndpnJ0iszQ.png" /><figcaption>BAYC collection smart contract</figcaption></figure><p>Most of the time there is a method called <strong>tokenURI </strong>or <strong>baseURL </strong>which returns a base URL for the connection. In BAYC&#39;s case, it is <strong>baseURI.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h35MY1Ez8aHZlFcFMeOWuw.png" /><figcaption>read contract of the BAYC collection</figcaption></figure><p>The Moonbirds collection is a little bit different but essentially the same. It does not have any function that returns the base URL. Instead, it has a function that takes an NFT ID as a parameter and returns the whole URL to the corresponding NFT metadata. We can use this function and get the prefix as the base URL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YwxG-buJ_P8hGgw3IzolIQ.png" /><figcaption>read contract of the Moonbirds collection</figcaption></figure><p><strong>HTTPS vs. IPFS</strong></p><p>You might also notice that BAYC is deployed on an IPFS server whereas Moonbirds is deployed on HTTPS. IPFS is Web3 way of storing and accessing data. It is completely distributed over the network and instead of accessing data via its address as we do in HTTPS, we request the data from the network by its hash. It is analogically closer to Torrent. Because of that difference, the way we read data from IPFS is not straightforward as reading from HTTPS servers.</p><p>Here comes Infura for our help! Infura is a gateway between our applications and the Ethereum network. It is certainly a better alternative to running a local Ethereum node to communicate with the network. For more detailed information about IPFS and Infura please check the articles below.</p><ul><li><a href="https://blog.infura.io/post/an-introduction-to-ipfs">An Introduction to IPFS</a></li><li><a href="https://medium.com/what-is-infura/what-is-infura-59dbdd778455">What is Infura</a></li></ul><p><strong>Bulk requesting</strong></p><p>Each NFT collection has thousands of NFTs inside. And yet we need to get every item’s metadata in order to calculate rarity scores successfully. If you have missing data for several items, your rarity analysis will not take them into account, and there is a high chance that you might end up with crappy rarity scores.</p><p>Sending thousands of HTTP requests is not something we would like to do sequentially. We can leverage asynchronous mechanisms here to improve our software’s efficiency. I explained this in my other article and I won&#39;t cover the same topics again. I recommend you read this one first if you really want to fetch NFT collection data by yourself.</p><p><a href="https://medium.com/@mkaanerkoc/how-to-send-bulk-http-requests-with-aiohttp-asyncio-in-python-62aee008e057">How to send bulk HTTP requests with aiohttp &amp; asyncio in Python</a></p><p>I used the <strong>get_collection_attributes_async </strong>function from the notebook I used in the article above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a89b03b29cfecce88b8aa92d2947a9a/href">https://medium.com/media/2a89b03b29cfecce88b8aa92d2947a9a/href</a></iframe><p><strong>Rarity Calculation &amp; Analysis Pipeline</strong></p><p>I abstracted every step that we’ve done so far to calculate rarity scores into high-level functions, <strong>calculate_jaccard_score,</strong> and <strong>rarity_score_pipeline.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab41763165519b95df05c072106af347/href">https://medium.com/media/ab41763165519b95df05c072106af347/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/835ac6fcdce157aed8b3a57898b7bd4f/href">https://medium.com/media/835ac6fcdce157aed8b3a57898b7bd4f/href</a></iframe><p>Let’s execute our pipeline for BAYC, Moonbirds, and Azuki collections.</p><p><strong>Analyzing the BAYC collection</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nwmkDmiBskb8TfmW4nxONw.png" /><figcaption>rarity results of the BAYC collection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DGrVYELf1l4-5IYnGtWRgw.png" /><figcaption>IDs of the top 10 rare NFTs in the BAYC collection</figcaption></figure><p><strong>Analyzing the Moonbirds collection</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o0sa5S67l8P5VwQZjVHLIQ.png" /><figcaption>rarity results of the Moonbirds collection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3Dm1xPfm21IKRkOZsVpGA.png" /><figcaption>IDs of the top 10 rate NFTs in the Moonbirds collection</figcaption></figure><p><strong>Analyzing the Azuki collection</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3LqSJZ8VvMrmcOL8W4011g.png" /><figcaption>rarity results of the Azuki collection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TxxHd2Fn5Lh5VwWxnkvpHQ.png" /><figcaption>IDs of the top 10 rare NFTs in the Azuki collection</figcaption></figure><p><strong>Results</strong></p><p>Awesome! Our prototype code worked perfectly for different NFT collections. It looks like each collection has different rarity distributions. BAYC has a positive skewed normal distribution whereas Moonbirds has a negative skewed and Azuki has tired-camel-shaped distribution. (I just made this up). Before buying an NFT you better check where the NFT you want to make an offer happens to be in the collection’s rarity distribution. For instance, you would not want to buy an NFT which happens to be in the very middle of the curve. ( or in the camel body part of Azuki distribution, lol )</p><h3>Conclusions</h3><p>Jaccard distance is yet another powerful tool that I am using for trading NFTs. It provides a quick and efficient analysis of NFT price fairness. There are many websites which are providing rarity scores and most of them use different algorithms to calculate scores such as Double Sort, Trait count, and Percentage weighting. Therefore it is perfectly fine to find minor differences in the ordering. But overall you should see similar results.</p><p>The only downside of Jaccard distance that I recognize so far is being so sensitive to missing data. It might generate completely different results even if you have 5 missing NFT metadata in the 10.000 size collection. According to my experience, percentage-based rarity calculations are more robust to missing data. However, missing data is dangerous for rarity calculation in any case. So make sure your data fetching mechanism is able to recover missing data.</p><p>Another thing I realized during my experiments, I think it can be improved by applying different weights to different attribute types. For example, having rare armor might be better than having a rare amulet. In our implementation for today, we assume that all the attribute types equally contributed to overall rarity.</p><p>If you like this content and want to learn more about other rarity calculation algorithms please let me know via email or comments. I would love to write further articles on rarity analysis.</p><p>Thank you for reading the article. If you have any questions/suggestions please bombard the comments section. Until next time, keep your head ringing!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDjZM3ll6nyWAj_X9DHu6w.png" /><figcaption>“fluffy gangster monsters rapping at hiphop concert, crowd gone crazy, neon lights, realistic photo”</figcaption></figure><p><a href="https://github.com/mkaanerkoc/nft-rarity/blob/main/jaccard_distance.ipynb">nft-rarity/jaccard_distance.ipynb at main · mkaanerkoc/nft-rarity</a></p><h3>References</h3><ol><li><a href="https://blog.liquid.com/nft-use-cases">https://blog.liquid.com/nft-use-cases</a></li><li><a href="https://trends.google.com/trends/explore?date=2020-01-01%202022-11-13&amp;q=NFT">https://trends.google.com/trends/explore?date=2020-01-01%202022-11-13&amp;q=NFT</a></li><li><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html">https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html</a></li><li><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.squareform.html">https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.squareform.html</a></li></ol><blockquote>New to trading? Try <a href="https://medium.com/coinmonks/crypto-trading-bot-c2ffce8acb2a">crypto trading bots</a> or <a href="https://medium.com/coinmonks/top-10-crypto-copy-trading-platforms-for-beginners-d0c37c7d698c">copy trading</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea1848689f34" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/the-most-important-tool-for-nft-valuation-the-jaccard-distance-ea1848689f34">The most important tool for NFT valuation, the Jaccard distance</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Financial market clustering: Part3, k-means clustering]]></title>
            <link>https://medium.com/@mkaanerkoc/financial-market-clustering-part3-k-means-clustering-5b84d65a00cb?source=rss-58dd91dc1574------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b84d65a00cb</guid>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cryotocurrency]]></category>
            <dc:creator><![CDATA[Kaan Erkoç]]></dc:creator>
            <pubDate>Sun, 06 Nov 2022 13:48:43 GMT</pubDate>
            <atom:updated>2022-11-06T14:37:15.894Z</atom:updated>
            <content:encoded><![CDATA[<p>So far in this series, we accomplished the very first steps which play an important role in our work. We successfully get the online price data and use the B-splines to parameterize the features that we want to use for clustering. If you have come across this article randomly and have no idea about our past work, no worries, you can find it below.</p><p><a href="https://medium.com/@mkaanerkoc/financial-markets-clustering-part-2-b-spline-fitting-c12d2f8d36c">Financial markets clustering: Part 2, B-spline fitting</a></p><p>In this article, part 3 of the series, we will dive into the world of <strong>clustering</strong>. Again with the same mindset of keeping the problem small and solving them incrementally, we will extend our toolkit with <strong>k-means clustering</strong>. Next chapter, we will use our new tool for clustering the market hierarchically. But not to make today’s context bigger, we will be focusing on clustering the market with a single level.</p><p>You can directly access the Jupyter notebook for this work in my Github account.</p><p><a href="https://github.com/mkaanerkoc/clustering/blob/main/clustering.ipynb">clustering/clustering.ipynb at main · mkaanerkoc/clustering</a></p><h3><strong>Clustering</strong></h3><p>Have you ever found yourself struggling when you need to set up your room after moving into a new house? It indeed seems like a lot of work in the beginning. But your brain will soon recognize some patterns and categories which guide you on which items need to be placed together in your cabinet, your desk, or your bookshelf.</p><p>But how does your brain recognize these items and how does it decides to chunk them? In short, your brain just calculates the <strong>similarity</strong> between items. That similarity might be either their type or their use cases, but in any way what your brain is doing under the hood is calculating similarities between your preciouses. The items that have higher similarity are chunked together in your brain, as they should be in your room as well.</p><p>In the machine learning world, <strong>clustering </strong>relies on the category of <strong>unsupervised learning </strong>as we let the algorithm find similar groups by itself. If we design a model which assigns the items into particular classes that are known in the data set then it would be called <strong>classification.</strong></p><p>There are several different types of clustering. The most important ones are the following[1];</p><ul><li><strong>Centroid-based clustering</strong> organizes the data into non-hierarchical clusters, in contrast to hierarchical clustering defined below. k-means is the most widely-used centroid-based clustering algorithm.</li><li><strong>Density-based clustering</strong> connects areas of high example density into clusters. This allows for arbitrary-shaped distributions as long as dense areas can be connected. These algorithms have difficulty with data of varying densities and high dimensions.</li><li><strong>Distribution-based clustering</strong> approach assumes data is composed of distributions, such as <a href="https://wikipedia.org/wiki/Normal_distribution"><strong>Gaussian distributions</strong></a>. As the distance from the distribution’s center increases, the probability that a point belongs to the distribution decreases</li><li><strong>Hierarchical clustering</strong> creates a tree of clusters. Hierarchical clustering, not surprisingly, is well suited to hierarchical data, such as taxonomies.</li></ul><p>The clustering approach that we will be following is a sort of hybrid approach that uses the k-means algorithm in a hierarchical manner. The reason why I choose this approach is the intuitional thought I have that tells me it would make sense to cluster the market hierarchically starting from high time frames towards low time frames with a simple clustering mechanism on each level. But as I explained above, we will be focusing only k-means clustering algorithm without hierarchies in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VzV1xBZpU6Lw3xMXTa0ZsQ.jpeg" /><figcaption>A multi-wavelength view of galaxy clusters [2]</figcaption></figure><h3><strong>Work</strong></h3><p>In today’s article, we will scale up our work to analyze the whole pairs that are available in the Binance spot exchange. Not only scaling up but we are also going to discover the k-means algorithm to cluster the market with respect to their price action. Hopefully, we can get some insights into current market dynamics.</p><p>Let’s have a brief look at what we will be implementing today.</p><p>First, we need to get candle stick data for all available pairs in the Binance. To do that, we need to know which pairs are <strong>actively</strong> traded at the moment. After we get the whole active pair list, we can request price data.</p><p>As we previously do, preprocessing the price data is needed to convert it into pandas.DataFrame. Once we get price data in the DataFrame format we can proceed with <strong>feature calculation, normalization, </strong>and <strong>B-spline fitting</strong>.</p><p>After we successfully get the coefficients of B-spline curves we can move on to <strong>clustering the pairs</strong> by use of the k-means algorithm. As the last step, we will calculate <strong>cluster errors</strong> to have quantitative results to evaluate our model.</p><h4>Getting Started</h4><p>We start today’s work by importing our dependencies. At this time we have two extra dependencies. We will be using the <strong>sklearn</strong> package for the k-means algorithm and the <strong>seaborn</strong> package for nicer visualizations. We can easily install these via;</p><pre>!pip install sklearn<br>!pip install seaborn</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29111f9a9cb016df7da46e978b6bce04/href">https://medium.com/media/29111f9a9cb016df7da46e978b6bce04/href</a></iframe><h4>Fetching price data for all USDT pairs</h4><p>Unfortunately, Binance API does not support requesting price data of multiple pairs at once. Hence we need to fetch them individually, it also means that we need to send many requests to Binance. Sending bulk HTTP requests is one of the best use cases for asynchronous programming in Python. Luckily, our big-hearted friend <strong>python-binance </strong>module supports the asynchronous client which makes our life easier when dealing with async requests.</p><p>Initializing <strong>AsyncClient</strong> class is the same as initializing the synchronous <strong>Client</strong> class. Only the class name is different.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b73ab0c1e095dc26d76f2a4422f89517/href">https://medium.com/media/b73ab0c1e095dc26d76f2a4422f89517/href</a></iframe><p><strong>Getting active USDT pairs</strong></p><p>Binance API has an endpoint that provides detailed information about exchange and pairs. This endpoint can be accessed with the <strong>get_exchange_info() </strong>method<strong> </strong>of the Client/AsyncClient classes. The endpoint returns a lot of information that might be useful for live-trading sessions such as allowed order types, minimum order size, etc. but we are only interested in the <strong>status </strong>field. Note that some of the stablecoin pairs are also excluded from our target pairs list.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90247d7837442fdb6f1d33c43e632b31/href">https://medium.com/media/90247d7837442fdb6f1d33c43e632b31/href</a></iframe><p>After we get actively traded USDT pairs we can proceed with price data fetching. Here I leveraged <strong>asynchronous</strong> mechanisms to improve our software’s efficiency. The way I use chunking and asynchronous programming is written in more detail in my other article. I strongly recommend to you read this one if you want to build a decent knowledge of asynchronous HTTP requests in Python.</p><p><a href="https://medium.com/@mkaanerkoc/how-to-send-bulk-http-requests-with-aiohttp-asyncio-in-python-62aee008e057">How to send bulk HTTP requests with aiohttp &amp; asyncio in Python</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7bd87e3812a48d658208286c707b4d9e/href">https://medium.com/media/7bd87e3812a48d658208286c707b4d9e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffe94399fdc6d0962ae500d9b1516768/href">https://medium.com/media/ffe94399fdc6d0962ae500d9b1516768/href</a></iframe><h4>Feature calculation and Normalization</h4><p>As we already did in the previous chapter, the features such as EMA25, EMA50, and EMA100 needed to be calculated for every pair. But this time we have to do one extra step which is <strong>essential</strong> to make our clustering algorithm work.</p><p><strong>Normalization</strong> is quite an important consept in data science. In essence, it addresses the possible problems when having data with different scales and different distribution parameters. For example, the BTCUSDT pair has been trading between 19k and 21k as I am writing this article and the ETHUSDT pair has been trading between 1.4k and 1.6k. Imagine you fit these two pairs with B-Spline, you will have completely different coefficients even though their price action in the last month is nearly the same. If we normalize their scale, BTCUSDT would be trading between 1.0 and 1.10 and ETHUSDT would be between 1.0 and 1.42. Fitting the normalized data surely would give better results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55f5578cca5167d0d266d306259b023e/href">https://medium.com/media/55f5578cca5167d0d266d306259b023e/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005297aace317fca6655aadb80742bb4/href">https://medium.com/media/005297aace317fca6655aadb80742bb4/href</a></iframe><p>Let’s have a look at our final <strong>market_prices </strong>DataFrame. You should be able to see the same output as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMh7QKtrcnEf1BTDe6GrSQ.png" /></figure><h4>Fitting with B-Spline</h4><p>B-spline parametrization will be almost the same as we did in the previous chapter. To encapsulate the parameters and results for using them afterward I created a class called <strong>BSplineFitter. </strong>It also provides functionalities for generating a curve using B-spline coefficients.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0047bf8f0a53fac7d1adb0d93bda83f0/href">https://medium.com/media/0047bf8f0a53fac7d1adb0d93bda83f0/href</a></iframe><p>BSplineFitter class accepts only one argument in its constructor which is the feature tag. This argument is used for getting parameters from the <strong>parameters </strong>dictionary. Even though we will be using a single parameter set in this example, I did not see any harm to have a variety of parameters and try out them separately.</p><p><strong>fit()</strong> method accepts only one argument as well, which is the <strong>market_prices</strong> DataFrame we constructed. It groups by symbol names and apply b-spline fitting to each group. I used the <strong>close_50_ema</strong> feature for this article. In the next article, we will use all the features we have and try to cluster them in a hierarchical manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uGCdB3JzEOdzKTJFN3WMdA.png" /><figcaption>b-spline coefficients for each pair</figcaption></figure><p>Let’s plot some of the results we got. I chose BTC, ETH, LTC, and FTM for visualization.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10ac35e19d5711c296b9ee258cee355f/href">https://medium.com/media/10ac35e19d5711c296b9ee258cee355f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KDbtvLl0J8aDkDY4Ht8RWA.png" /><figcaption>fitting results for BTC, ETH, LTC, and FTM</figcaption></figure><p>It looks like our BSplineFitter class is working as expected and generating some useful results. Did you also notice how the normalization pulls down all different scales into the same level? By visually checking we can already see that BTC and LTC are looking more similar than others. But can we also find these similarities without any human effort required? That we will find out in the next step, fingers are crossed.</p><h4>Clustering B-Spline coefficients with K-means</h4><p>Fill your coffee cup up. Here comes the heart of this article. We are ready to use the k-means algorithm to discover clusters of the symbols.</p><p>To have a better <a href="https://en.wikipedia.org/wiki/Separation_of_concerns">separation of concerns</a> in the code base, I wrote two classes responsible for the clustering process. We definitely should consider good software design principles, even though what we are doing here is completely experimental work. Who knows maybe we can get decent results and we will decide to put our work in the production pipeline. It is always nice to have modular and reusable code. I tried to visualize the relationship between those classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*-6y_6qChT5ozhIWMjppUCg.png" /><figcaption>diagrams of classes in this article</figcaption></figure><p><strong>Cluster</strong> class is instantiated from clustering results. It is responsible for error calculation and visualization for a dedicated cluster.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/934b2dbeb050d63ffb719ba6cac73282/href">https://medium.com/media/934b2dbeb050d63ffb719ba6cac73282/href</a></iframe><p><strong>ClusterFinder </strong>class is responsible for finding the clusters as its name indicates. It receives <strong>BSplineFitter </strong>object as a constructor parameter and it uses its fitting results for the clustering. To follow the scientific python naming convention I again chose <strong>fit() </strong>as a name for the clustering method.</p><p>Coefficients are retrieved from CurveFitter object and they are converted into numpy.array before they passed down to kmeans.fit() method. Let’s have a look at what we got as a result of this method.</p><p>KMeans.fit() method returns a fitted KMeans object. Therefore you can find fitting results stored in its attributes. [3]</p><ul><li><strong>cluster_centers_</strong><em>ndarray of shape (n_clusters, n_features): </em>Coordinates of cluster centers. If the algorithm stops before fully converging (see tol and max_iter), these will not be consistent with labels_.</li><li><strong>labels_, </strong><em>ndarray of shape (n_samples,): </em>Labels of each point</li><li><strong>inertia_, </strong><em>float: </em>Sum of squared distances of samples to their closest cluster center, weighted by the sample weights if provided.</li><li><strong>n_iter_, </strong><em>int: </em>Number of iterations run.</li><li><strong>n_features_in_, </strong><em>int: </em>Number of features seen during <a href="https://scikit-learn.org/stable/glossary.html#term-fit">fit</a>. <em>New in version 0.24.</em></li><li><strong>feature_names_in_</strong><em>ndarray of shape (</em><em>n_features_in_,)</em>Names of features seen during <a href="https://scikit-learn.org/stable/glossary.html#term-fit">fit</a>. Defined only when X has feature names that are all strings. <em>New in version 1.0.</em></li></ul><p>After the fit operation was finished, I created a new DataFrame called <strong>symbols_in_clusters</strong> which matches the cluster_id and symbols. Later on that, information per cluster is calculated and assigned to a new instance of the <strong>Cluster </strong>object.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48f3c86699d28c55b815b4f6e96b3024/href">https://medium.com/media/48f3c86699d28c55b815b4f6e96b3024/href</a></iframe><p>So far, so good. Let’s try out our new tools!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eANStfUYFPBaErp9SJ4mYw.png" /><figcaption>fitting ClusterFinder class</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCbs7aswKlHY39SYh4UkzA.png" /><figcaption>representative curve of each cluster</figcaption></figure><p>Voilà! We clustered pairs in 32 different groups and plotted the representative curves for each group. It looks like you would not want to be long in clusters 7, 12, or 24. But indeed it would be quite a nice portfolio if you have long positions in clusters like 4, 22, and 23.</p><p>Let’s have a detailed look at these clusters. Here we will use Cluster class for visualizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wkvtN_7gOfgkGbP_xwYAig.png" /><figcaption>cluster #7 symbols</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hCxd3EhrGAc0EqmHQT5Bg.png" /><figcaption>cluster #12 symbols</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WbR1tIi2cUqlmBrFPAVZrw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_dgnV7eIOe2GQAP9qjHp8A.png" /><figcaption>cluster #24 symbols</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1AmJ0DdP5lsxvoVuc1Wiw.png" /><figcaption>cluster #4 symbols</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9ztfXqs1rSgcUak8ND0TQ.png" /><figcaption>cluster #22</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JyI1GuBh9lLoOur7qdanLQ.png" /><figcaption>cluster #23</figcaption></figure><p>Congratulations! At the first sight, it looks like we successfully cluster the market into different dynamics. By looking at only cluster centers we can get a grasp of whole market characteristics. The mini-bull season that started at the end of the October can easily be detected. However, there is still 1/3 of the market going sideways as it was captured at clusters 1, 11, and 16. In the next articles of this series, we will look at how these clusters are performing over time.</p><h4>The Feedback Loop</h4><p>It seems like all the work we have done so far gives us what we wanted. But how confident we are about the performance of our model? Can we just say good or bad by visually checking the cluster plots? How we can quantitatively compare clustering results with different parameters applied such as the number of clusters, sampling period, and the feature vector we used for clustering? Indeed, trying to evaluate results by visually checking is not a good practice. We need to have KPI items to do that.</p><p>If you recall attributes of the KMeans class, there is one field called <strong>inertia_</strong> which shows the sum of squared distances of samples to their closest cluster center. We can use it as an overview KPI.</p><p>While having an overview KPI is okay, I would like to calculate symbol errors individually and want to see their distributions. <strong>Cluster </strong>class is responsible for calculating the <a href="https://en.wikipedia.org/wiki/Euclidean_distance">Euclidean distance</a> between each symbol and the center curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCh4g5S_cWPpMxzxEU6KCQ.png" /><figcaption>error distribution for 32 clusters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EzD7M0v2uUU7S1WVjzfc7w.png" /><figcaption>boxplot of errors</figcaption></figure><p>Let’s check how much having fewer clusters, 12 in this case, will affect our KPIs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5r7A0RczDDjyFqFMgJZTcw.png" /><figcaption>clustering into 12 groups</figcaption></figure><p>As it seems from the results, having fewer clusters resulted in more outliers. Even though the mean of the errors did not change much, the 95% percentile and inertia increased disquietingly. Additionally, it makes perfect sense to have more outliers in fewer groups of clusters, by intuitively thinking.</p><h3>Conclusion</h3><p>We put great effort to get there, now it is time to celebrate our newest tool. I strongly recommend you check out the Jupyter notebook for this article and experiment with our new tool by yourself. You can play with the parameters, time scales, and feature vectors depending on your use case. You will also recognize that not all the pairs in the clusters are looking like they are belonging there. But this will be the next problem that we gonna try to solve in the next article! Until next time, take care!</p><p>Jupyter notebook for this article can be found in my Github account.</p><p><a href="https://github.com/mkaanerkoc/clustering/blob/main/clustering.ipynb">clustering/clustering.ipynb at main · mkaanerkoc/clustering</a></p><h3>References</h3><ol><li><a href="https://developers.google.com/machine-learning/clustering/clustering-algorithms">https://developers.google.com/machine-learning/clustering/clustering-algorithms</a></li><li><a href="https://www.cosmos.esa.int/web/space-science-faculty/research/galaxy-clusters">https://www.cosmos.esa.int/web/space-science-faculty/research/galaxy-clusters</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans.fit">https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans.fit</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b84d65a00cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to send bulk HTTP requests with aiohttp & asyncio in Python]]></title>
            <link>https://medium.com/@mkaanerkoc/how-to-send-bulk-http-requests-with-aiohttp-asyncio-in-python-62aee008e057?source=rss-58dd91dc1574------2</link>
            <guid isPermaLink="false">https://medium.com/p/62aee008e057</guid>
            <category><![CDATA[http-request]]></category>
            <category><![CDATA[asynchronous]]></category>
            <category><![CDATA[nft]]></category>
            <category><![CDATA[asyncio]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Kaan Erkoç]]></dc:creator>
            <pubDate>Sun, 23 Oct 2022 17:17:13 GMT</pubDate>
            <atom:updated>2022-10-23T17:17:13.588Z</atom:updated>
            <content:encoded><![CDATA[<h3>Background</h3><p>We live in a world that is connected way more than before and generates data every year more than all generated data before. The availability of that amount of <strong>services</strong> and <strong>data</strong> makes it possible for everyone to develop applications that we could not even think of one decade ago.</p><p>As a direct result of these facts, the necessity of <strong>communicating with remote</strong> computers has become crucial in our software applications to either execute a function or fetch data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0O6IRz8rdgh84_MHxRHCYQ.jpeg" /><figcaption>a soldier releases homing bird to send a message during WW1</figcaption></figure><p>Remote communication, by nature, requires <strong>sending</strong> and <strong>receiving</strong> messages over protocols such as TCP/IP. I want to take your attention to <strong>receiving</strong> part first. As a receiving counterpart, you don’t have much control over the things that you wait to receive. You might choose either <strong>wait</strong> or <strong>do other things</strong> until the response has been received. I guess none of us would be in the favor of waiting unless it is the only available option. In the computer science field, we call this situation<strong> </strong><a href="https://en.wikipedia.org/wiki/I/O_bound"><strong>I/O bound</strong>.</a></p><p>There are several mechanisms to not make our software I/O bound. The most used ones are multithread programming and asynchronous programming. Both are used for doing other things while waiting for I/O.</p><p>In the diagram below, it is clearly shown how the CPU is being utilized in different mechanisms. <strong>Option A </strong>shows inefficient, single-threaded software which waits for responses almost %90 of its lifetime. <strong>Option B </strong>shows how multiple threads can be used for each waiting task. Lastly, <strong>Option C </strong>shows<strong> </strong>how it would be<strong> </strong>done in single-threaded asynchronous coroutines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2vEQA81jHPM-P1jqyiRDw.png" /><figcaption>Different techniques of concurrency in Python. Source: Adapted from Anderson 2019 <a href="https://realpython.com/python-concurrency/">*</a></figcaption></figure><p>There are several reasons for me to choose option C over option B. In the first place, it is very difficult to write thread-safe code. It is not easy to manage inter-thread communication. You might also find yourself dealing with callbacks a lot, as it is shown in the diagram ( Box 1, 2, 3 ). However, option C is working in a single thread. So it is much easier to manage memory and thus write the code. Also, it takes less amount of system resources since it is working in a single thread. It makes the asynchronous approach perform better on the scale.</p><p>I recommend you read the article below to understand the differences in more detail.</p><p><a href="https://itnext.io/practical-guide-to-async-threading-multiprocessing-958e57d7bbb8">Practical Guide to Async, Threading &amp; Multiprocessing</a></p><h3>Implementation</h3><h4>Problem</h4><p>In this post, we will be using <strong>aiohttp </strong>and<strong> asyncio</strong> packages in Python to make our software do other things while waiting for the response. Instead of working on a toy problem, I would like to use one of our NFT projects as a real-world example here.</p><p>I and my buddy <a href="https://medium.com/u/706428417f70">Rasih Onur Olgun</a> developed software that calculates the rarity score of NFTs in the given collection. Each NFT in a collection is identified with a unique NFT id and their metadata is stored either in IPFS or another server. As you can already guess, it requires fetching thousands of NFT metadata. That’s where we are going to use <strong>aiohttp</strong> &amp; <strong>asyncio </strong>to improve efficiency.</p><p>As I always do, I am going to use Jupyter notebooks for this work again.</p><p>If you are interested in the complete Jupyter notebook, you can find it in my Github account.</p><p><a href="https://github.com/mkaanerkoc/nft-rarity/blob/main/asyncio_example.ipynb">nft-rarity/asyncio_example.ipynb at main · mkaanerkoc/nft-rarity</a></p><h4>Dependencies</h4><p><strong>aiohttp</strong> is a Python module that supports both HTTP and WebSocket protocols as server and client. Yet it is implemented with <strong>async </strong>functions. Therefore it will save you from dealing with callback hell.</p><p><strong>requests</strong> is another module that we will use to send HTTP requests but only for demonstrating <strong>Option A</strong> and comparing with <strong>Option C. </strong>If you are not interested in this practice, you can skip it.</p><p>Let’s install them.</p><pre>!pip install aiohttp<br>!pip install requests</pre><p>and importing necessary modules.</p><pre>import asyncio<br>import aiohttp<br>import requests<br>import time<br>import json<br>from itertools import islice</pre><h4>Bulk requests and chunking</h4><p>In theory, we should be able to send as many requests as our network allows. But in the practical world, there is always a limit to the number of requests that you can send at a single time, either because of server firewall or server capacity. In this manner, we are going to divide our whole NFT items into smaller chunks and request them serially.</p><p>Let’s say we want to fetch data of 5000 NFT items. It would be nice if we can send 5000 HTTP requests in one shot but unfortunately, we need to divide that into smaller chunks. I chose a chunk size of 250 due to my past experiences. So it means there will be 20 chunks with a size of 250. We will use an iterable that chunks input list and returns chunks on each iteration.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/603626dcdbfe237c45f9fb1c7272df0b/href">https://medium.com/media/603626dcdbfe237c45f9fb1c7272df0b/href</a></iframe><h4>Asynchronous Code</h4><p>Asynchronous implementation is hierarchically separated into three functions.</p><p>The first one is the top-level function which is responsible for creating a TCP session and chunking over the whole NFT list. The <strong>aiohttp</strong> library has a very nice feature that creates a TCP session once and uses it for sending all the requests. It surely increases performance by getting rid of session creation overhead for every single request. It is also implemented as a context manager, thus you don’t need to care about ending the session when you exit from <strong>with</strong> block.</p><p>The second function is where the magic happens. This function is responsible for creating <a href="https://docs.python.org/3/library/asyncio-task.html">Futures</a> for each NFT in the chunk. Each future is an awaitable object that represents the eventual result of an asynchronous operation. <strong>asyncio.gather </strong>function waits for all futures to be finished and returns their results as a list. This is actually where all the speed increase is gained.</p><p>The last function, and also the lowest level function, is I/O bound HTTP GET request function. It simply sends the GET request and <strong>await</strong>s the response.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/655742c7b0fafe0f7f85c1f95853bde1/href">https://medium.com/media/655742c7b0fafe0f7f85c1f95853bde1/href</a></iframe><h4>Synchronous Code</h4><p>The complexity of code required for serially reading data is much lower as we don’t need to create chunks and deal with the sessions. All we need to do is loop over the URLs and send HTTP requests one by one with the <strong>requests</strong> library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c05687cade08eaa551f801f47fffe7f3/href">https://medium.com/media/c05687cade08eaa551f801f47fffe7f3/href</a></iframe><h4>Performance</h4><p>I chose the <a href="https://opensea.io/collection/cool-cats-nft">CoolCats</a> collection as an example for this article for no reason. The collection data is stored at ‘<a href="https://api.coolcatsnft.com/cat&#39;">https://api.coolcatsnft.com/cat/{nft_id}&#39;</a>. For instance, <a href="https://api.coolcatsnft.com/cat/231">https://api.coolcatsnft.com/cat/231</a> returns metadata of NFT#231.</p><p>When it comes to talking about performance, one must keep the track of it. We will use a quite handy context manager to track the amount of time a function has spent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d09f232015df81cde768d2d7003ad3b5/href">https://medium.com/media/d09f232015df81cde768d2d7003ad3b5/href</a></iframe><p>I fetched NFT metadata between 0 and 2000 IDs with both async and sync functions to compare results. As it is expected, the async function overperforms the sync one. What was your initial expectation over performances? For 2000 HTTP requests the difference is huge, 50x.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXwXV7CWEAHsdGXcodWccg.png" /><figcaption>performance comparison between async and sync functions.</figcaption></figure><h3>Conclusion</h3><p>Concurrency, multi-threading, and asynchronous programming are challenging topics in computer science. And of course, it is not possible to cover details in a single article.</p><p>I always find the <strong>async</strong> features of Python quite important and useful. They are certainly one of the tools that every software developer should have in their toolkit.</p><p>I hope that the article gives you enough base knowledge to start writing asynchronous code.</p><h3>References</h3><ol><li><a href="https://en.wikipedia.org/wiki/I/O_bound">https://en.wikipedia.org/wiki/I/O_bound</a></li><li><a href="https://realpython.com/python-concurrency/">https://realpython.com/python-concurrency/</a></li><li><a href="https://docs.python.org/3/library/asyncio-task.html">https://docs.python.org/3/library/asyncio-task.html</a></li><li><a href="https://itnext.io/practical-guide-to-async-threading-multiprocessing-958e57d7bbb8">https://itnext.io/practical-guide-to-async-threading-multiprocessing-958e57d7bbb8</a></li><li><a href="https://stackoverflow.com/questions/4024056/threads-vs-async">https://stackoverflow.com/questions/4024056/threads-vs-async</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62aee008e057" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Financial markets clustering: Part 2, B-spline fitting]]></title>
            <link>https://medium.com/@mkaanerkoc/financial-markets-clustering-part-2-b-spline-fitting-c12d2f8d36c?source=rss-58dd91dc1574------2</link>
            <guid isPermaLink="false">https://medium.com/p/c12d2f8d36c</guid>
            <category><![CDATA[quantitative-finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[binance]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Kaan Erkoç]]></dc:creator>
            <pubDate>Mon, 17 Oct 2022 09:52:23 GMT</pubDate>
            <atom:updated>2022-11-07T11:08:17.880Z</atom:updated>
            <content:encoded><![CDATA[<h3>Financial market clustering: Part 2, B-spline fitting</h3><h3>Introduction</h3><p>In the first article of the market clustering series, I wrote about the rationales behind my motivation to use such a market clustering system which I can use its valuable insights in my trading decisions.</p><p>If you haven’t checked that article yet, I strongly recommend you to read Part 1 first to understand the work that is being done in this article and in the upcoming articles.</p><p><a href="https://medium.com/@mkaanerkoc/financial-markets-clustering-series-part-1-d243a12538bf">Financial markets clustering series, Part 1</a></p><p>Today we will see how to fetch price data from Binance, post-process it, resample for different time scales, apply EMA filters and finally fit with b-splines as we discussed in the previous chapter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*ALW_pekwJ9BU-8cSnCNtQw.png" /><figcaption>generative art with splines</figcaption></figure><h3>Getting started</h3><p>I prefer Jupyter notebooks for such experimental work because of their handy documentation and visualization capabilities. Jupyter package comes with the Anaconda toolkit but it can also be installed separately with the following command.</p><pre>pip install jupyter</pre><p>After installing Jupyter, let’s create a new directory called /<strong>clustering</strong> under the home path to store our work and initialize a notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPm0mk2qqEU4r0KaV5nAmQ.png" /><figcaption>Running a new instance of Jupyter notebook.</figcaption></figure><p>Now your default browser should automatically open a new tab with the Jupyter browser. You can click the ‘New’ button to create a new notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EkOdKD2cTWRa1EivABhQtg.png" /><figcaption>Creation of a new Jupyter notebook from the Jupyter browser.</figcaption></figure><h3>External Dependencies</h3><p>To retrieve price data from Binance exchange I am using the famous python package, <a href="https://python-binance.readthedocs.io/en/latest/"><strong>python-binance</strong></a>. It is a nice written module and it supports all the API functionality provided by Binance, as far as I know. I am not going to touch on details on how to get API keys from Binance and set up this library, but if you don’t know how to do it, you can check the article below.</p><p><a href="https://algotrading101.com/learn/binance-python-api-guide/">Binance Python API - A Step-by-Step Guide - AlgoTrading101 Blog</a></p><p>I am using the <a href="https://pypi.org/project/stockstats/"><strong>stockstats</strong></a><strong> </strong>package to calculate various price indicators. There are several different modules to calculate price indicators in the PIP archive but I’ve found <strong>stockstats</strong> is the most useful and intuitive among the others.</p><p>Finally, I will be using the <strong>scipy</strong> package for b-spline fitting and of course <strong>matplotlib</strong> for visualizations.</p><pre>!pip install python-binance<br>!pip install stockstats<br>!pip install scipy<br>!pip install matplotlib</pre><h3>Initialising the Binance client</h3><p>We first need to provide the API key and API secret to initialize the Binance client instance. It also should be noted that having keys and secrets inside of your code is of course not a good practice in terms of security.</p><p>It is good practice to have this type of sensitive information in environment variables. For more detailed information you can check the below article.</p><p><a href="https://networkdirection.net/python/resources/env-variable/">https://networkdirection.net/python/resources/env-variable/</a></p><p>But since this tutorial only cares about getting things done, I will use keys in my code directly.</p><pre>from binance import Client</pre><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>from stockstats import wrap, unwrap<br>from scipy import interpolate</pre><pre>api_key = “** Provide your API Key **”<br>api_secret = “** Provide your API Secret **”</pre><pre>client = Client(api_key, api_secret)</pre><h3>Checking the connection</h3><p>Let’s check whether our client instance working properly and Binance servers are active. According to python-binance documentation, the return value of <strong>get_system_status </strong>method<strong> </strong>should seems like below;</p><pre>status = client.get_system_status()<br>print(status)</pre><pre>-&gt; {&#39;status&#39;: 0, &#39;msg&#39;: &#39;normal&#39;}</pre><h3>Fetching the prices</h3><p>Client class has a nice method called <strong>get_historical_klines</strong> for fetching the prices. It looks like a simple wrapper around the API call, however, it sends multiple requests to Binance API and aggregates the responses to fetch more than 500 data points when it is needed. Because Binance API only allows us to request 500 data points at a single request. And another thing that I like about this function is the format flexibility of datetime arguments.</p><p><strong>BTCUSDT</strong> symbol prices started from 3 months ago will be fetched in 1 Hour intervals.</p><pre>btcusdt_raw = client.get_historical_klines(“BTCUSDT”, Client.KLINE_INTERVAL_1HOUR, “90 day ago UTC”)</pre><h3>Converting the prices into pandas.DataFrame</h3><p>Unfortunately, price data is not received in our favorite format, which is pandas.DataFrame. We need to translate it into DataFrame first. Column names are set according to Binance API documentation.</p><p><a href="https://binance-docs.github.io/apidocs/spot/en/#kline-candlestick-data">Binance API Documentation</a></p><pre>btcusdt = pd.DataFrame(btcusdt_raw,<br> dtype=’float’,<br> columns=[‘open_time’, ‘open’, ‘high’, ‘low’, ‘close’, ‘base_volume’, ‘close_time’, ‘quote_volume’, ‘trade_count’, ‘taker_buy_vol’, ‘taker_buy_quote_vol’, ‘unused’])</pre><pre>btcusdt.info()</pre><pre>-&gt;<br>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 2160 entries, 0 to 2159<br>Data columns (total 12 columns):<br> #   Column               Non-Null Count  Dtype  <br>---  ------               --------------  -----  <br> 0   open_time            2160 non-null   float64<br> 1   open                 2160 non-null   float64<br> 2   high                 2160 non-null   float64<br> 3   low                  2160 non-null   float64<br> 4   close                2160 non-null   float64<br> 5   base_volume          2160 non-null   float64<br> 6   close_time           2160 non-null   float64<br> 7   quote_volume         2160 non-null   float64<br> 8   trade_count          2160 non-null   float64<br> 9   taker_buy_vol        2160 non-null   float64<br> 10  taker_buy_quote_vol  2160 non-null   float64<br> 11  unused               2160 non-null   float64<br>dtypes: float64(12)<br>memory usage: 202.6 KB</pre><h3>Post processing DataFrame</h3><p>Even though we got the price data in our best data structure format, it still needs some polishing before we continue.</p><p>First, we need to convert epoch timestamps to DateTime objects to set this as TimeIndex to our data frame. This operation is done by the ‘pandas.to_datetime’ function. the unit argument is set as ‘ms’ as Binance API timestamps in the ‘ms’.</p><p>The second operation is just dropping the columns that we don’t need. However, this step is application specific.</p><p>And lastly, I set the ‘time’ column as an index to our data frame.</p><pre>btcusdt[‘time’] = pd.to_datetime(btcusdt[‘open_time’], unit=’ms’)</pre><pre>btcusdt.drop(columns=[‘open_time’, ‘close_time’, ‘taker_buy_vol’, <br> ‘taker_buy_quote_vol’, ‘base_volume’, ‘trade_count’, ‘unused’], <br> inplace= True) # drop the unnecessary columns.</pre><pre>btcusdt.rename(columns={‘quote_volume’:’volume’}, inplace=True)</pre><pre>btcusdt.set_index(‘time’, inplace=True)</pre><pre>btcusdt.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*quD-33qq0wP3vQmN1Bya-w.png" /><figcaption>post-processed BTCUSDT price data</figcaption></figure><h3>Resampling the financial data</h3><p>Resampling the financial time series data is one of the crucial steps that we use often. Although it is not crucial in the very step of the series, I decided to put <strong>resample</strong> function here as it is more related to price data processing. We will be using this function in the next chapters.</p><pre>def resample(stock_price_df, new_period, offset=0):<br> ohlc = {<br>   ‘open’: ‘first’,<br>   ‘high’: ‘max’,<br>   ‘low’: ‘min’,<br>   ‘close’: ‘last’,<br>   ‘volume’: ‘sum’<br> }</pre><pre>return stock_price_df.resample(new_period, offset).apply(ohlc)</pre><pre>btcusdt_4h = resample(btcusdt, ‘4H’)<br>btcusdt_12h = resample(btcusdt, ‘12H’)</pre><h3>Plotting</h3><p>Finally, we can plot the <strong>close</strong> prices. How spiky the prices have been in the last months, huh.</p><pre>fig, ax = plt.subplots()</pre><pre>btcusdt_4h[&#39;close&#39;].plot(ax=ax)<br>btcusdt_12h[&#39;close&#39;].plot(ax=ax)<br>ax.legend([&#39;BTCUSDT 4H&#39;, &#39;BTCUSDT 12H&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*N1g26MGMLR1vWJOXPQX5PQ.png" /><figcaption>BTCUSDT prices with different time intervals</figcaption></figure><h3>Calculating EMA indicators</h3><p>As I explained at the very beginning of this article we will use the <strong>stockstats</strong> by <a href="https://medium.com/u/f499ca0b1f85">Cedric Zhuang</a> to calculate EMA indicators. <strong>stockstats</strong> is a python module based on pandas.DataFrame and yet provides an incredibly well-thought protocol to calculate indicator values <a href="https://en.wikipedia.org/wiki/Lazy_evaluation">lazily</a>.</p><p>All you need to do is wrap your DataFrame and comply with the following protocol.</p><pre>price_data_df[{candle_part}_{args}_{indicator_name}]</pre><p>For example, if you want to get EMA of close prices with a period of 16 you can easily access them via the <strong>close_16_ema </strong>column name.</p><pre>btcusdt = wrap(btcusdt)</pre><pre>fig, ax = plt.subplots()</pre><pre>btcusdt[‘close’].plot(ax=ax)<br>btcusdt[‘close_50_ema’].plot(ax=ax)<br>btcusdt[‘close_100_ema’].plot(ax=ax)<br>btcusdt[‘close_200_ema’].plot(ax=ax)</pre><pre>ax.legend([‘close’, ‘EMA 50’, ‘EMA 100’, ‘EMA 200’])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*FzgpKk0ElKm1U2D2SaaA1Q.png" /><figcaption>EMA50–100–200 with close prices</figcaption></figure><h3>B-Spline fitting</h3><p>All up to this point is the preparation phase for getting us ready for the most important and fun step. I hope you have not lost your will and you can pay attention to the rest.</p><p><strong>Splines</strong> are one of the piecewise, non-linear regression interfaces which are used in engineering applications frequently. They don’t bear the disadvantages of polynomial regressions such as overfitting and instability and yet they are continuous between the pieces. Doesn’t it sound great?</p><p>Piecewise regressions are fundamentally polynomial regressions but data is split into multiple regions via <strong>knots </strong>and each region between the knots is fit with local basis functions instead of fitting whole data with global functions. If <strong>continuity</strong> and <strong>smoothness</strong> constraints are applied to these local functions you can craft splines.</p><p>The reason I have chosen B-splines is that basis functions are quite local and any changes in the controlling points only affect the curve of this corresponding segment.</p><p>Even though it is quite a challenging mathematical method, it is only a few lines of code with a great <strong>Scipy</strong> package. There is only one function and one class that we need to grasp every detail about them. <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.splrep.html#scipy.interpolate.splrep"><strong>interpolate.splrep</strong></a> and <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.BSpline.html?highlight=bspline#scipy.interpolate.BSpline"><strong>interpolate.BSpline</strong></a><strong>. </strong>In<strong> </strong>essence, splrep is used for calculating coefficients of b-splines from 1-D curves, whereas BSpline is a class for constructing b-splines from coefficients.</p><p>The most important point here to comprehend is that knot positions will be defined by us, not calculated splrep. Because it won’t be possible to calculate the similarity between different time series based on their B-splines if their knot positions are different ( small displacements are actually okay, but let’s stay safe ). Letting the splrep do that will give different knot positions for time series data that are behaving differently. Therefore we will subsample the time axis in a 1:96 ( 1 data point in 4 days) ratio to use as fixed knot positions.</p><p>For now, I will use only <strong>close_25_ema</strong> values for simplicity. We can easily calculate knot positions and then B-spline coefficients as follows;</p><pre>PADDING = 8<br>SR = 96</pre><pre>x_axis = np.arange(0, len(btcusdt.index))<br>knot_positions = np.arange(PADDING, len(btcusdt.index) — PADDING, SR)</pre><pre>coefficients = interpolate.splrep(x_axis, btcusdt[‘close_25_ema’], t=knot_positions, k=5)<br>bspline_fit = interpolate.BSpline(*coefficients)(x_axis)</pre><p>Yikes! It is done. Let’s explore what our B-splines look like;</p><pre>fig, ax = plt.subplots()</pre><pre>ax.scatter(btcusdt.iloc[coefficients[0]].index, <br>           btcusdt.iloc[coefficients[0]][&#39;close_50_ema&#39;], <br>           marker=&#39;x&#39;, <br>           color=&#39;r&#39;)</pre><pre>ax.plot(btcusdt[&#39;close_25_ema&#39;])<br>ax.plot(btcusdt.index, bspline_fit)</pre><pre>ax.legend([&#39;knots&#39;, &#39;close_25_ema&#39;, &#39;b-spline&#39;])<br>plt.xticks(rotation=45)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*IEp_Vr7LHlLBvzpNkFwdQA.png" /><figcaption>knot positions, close prices, and b-spline fit results</figcaption></figure><p>Great! It looks like we successfully reduce a huge amount of data but still have a grasp of its high-level dynamics. The original data set has 2160 data points where we can just use 36 b-spline coefficients to represent it which is %1.2, as the knots are fixed.</p><h3>Conclusion</h3><p>As can be observed from the graph, B-splines provide great fitting performance even though knot positions are subsampled in a 1:72 ratio. That amount of data reduction of course comes with some local fitting errors which need to be addressed as well.</p><p>The notebook that I used for this article can be found on my GitHub profile. I strongly recommend you execute the Jupyter notebook by yourself and play with B-spline parameters to discover this great tool better.</p><p><a href="https://github.com/mkaanerkoc/clustering/blob/main/bspline_fit.ipynb">https://github.com/mkaanerkoc/clustering/blob/main/bspline_fit.ipynb</a></p><p>In the next chapter, we will extend the work we have done in this post by doing the same things but for all active symbols in Binance. Later on, we can proceed with clustering with the <a href="https://en.wikipedia.org/wiki/K-means_clustering">k-means</a> algorithm.</p><p>You can want to continue with Part3, you can check it out!</p><p><a href="https://medium.com/@mkaanerkoc/financial-market-clustering-part3-k-means-clustering-5b84d65a00cb">Financial market clustering: Part3, k-means clustering</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c12d2f8d36c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Financial markets clustering: Part 1, Intuitional buildup]]></title>
            <link>https://medium.com/trading-data-analysis/financial-markets-clustering-series-part-1-d243a12538bf?source=rss-58dd91dc1574------2</link>
            <guid isPermaLink="false">https://medium.com/p/d243a12538bf</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[quantitative-strategies]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[quantitative-finance]]></category>
            <dc:creator><![CDATA[Kaan Erkoç]]></dc:creator>
            <pubDate>Sat, 08 Oct 2022 11:39:55 GMT</pubDate>
            <atom:updated>2022-11-29T15:56:19.306Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>We all know that trading is not an easy way to make consistent income, even though it sounds easy. Many of us are putting incredible effort into crafting trading strategies and making money out of them. There are tremendous different approaches followed by retail traders and investment companies.</p><p>I have been passionate about trading systems for a couple of years since I find the finance engineering domain quite challenging and competitive. After investing a remarkable amount of my time to educate myself, I am finally feeling encouraged enough to share my humble thoughts about markets and how to leverage systematic approaches to gain some edge in this zero-sum game.</p><p>In this series, you will read about my way of seeing the market and how to easily get a grasp of it. And of course, it will contain implementation details and code in Python. This series aims to start with intuition and extend the ideas as we go further.</p><h3>Prediction Game</h3><p>Isn’t it all about predicting what will be the next price action? We analyze the price data to determine entry points, take-profit levels, and stop-loss points before we get into the position, right?</p><p>Personally, I’ve been finding that approach straight but not scalable due to reasons that I will explain in the next chapters.</p><p>As humans, we can only track a few open positions at the same time due to our cognitive capacity. So many of us, retail traders, prefer to keep less than 10 positions open at any given time. But now another question pops up here, and I think it is a way more important question to answer than the entry, and exit levels question. <strong>How do we find the symbols that we trade with?</strong></p><h3>Symbol Selection</h3><p>Last time I checked, almost 350 USDT pairs were traded in the Binance spot market. Any experienced retail trader can easily realize that it is just too much data to track the price actions of each symbol. Even a trader can follow price actions, but it is really hard to synthesize price and market dynamics and do it regularly.</p><p>If you are a lazy person and not a full-time trader like me, you can only do it for a couple of days, honestly. You will lose your motivation to do all the work and <strong>you cannot control your emotions</strong> all the time while analyzing the market, especially when you have open positions. It is a burdensome process and I don’t think it’s worth spending your precious time and giving up your health to earn some pennies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jyl_NJCFEQrPiPKtv-1ZRQ.png" /><figcaption>Crab fisher uses undersea radars to trace their hunt</figcaption></figure><p>I had been there.</p><h3>Lazy Price Summary</h3><p>My view of seeing price actions started to change when I started using multiple EMA filters together with multiple time frames.</p><p>EMA filters, or any other moving average filters, are deadly simple yet powerful smoothing filters that can clean out the noise from time series data.</p><p>But the real magic starts when we bring multiple EMA filters with different periods.</p><p>Many of us might come across a strategy called <strong>EMA crossover</strong> consisting of at least two EMA filters with different periods. The one with the shorter period is called ‘fast’ EMA and the other with the longer period is called ‘slow’ EMA. Essentially, it uses these fast and slow EMA’s to grasp price trends.</p><p>But why limit ourselves to 2? We can use more EMA filters to trace price actions at a more granular level. For example, I have been using 5 EMA’s with periods of 16, 32, 54, 72, and 96. And I have to admit that my eyes have evolved to see price action as thermodynamic flows already 🦅</p><figure><img alt="MKRUSDT pair with EMA16,32,54,72,96 plotted in 1H, 4H, 12H, 48H timeframes" src="https://cdn-images-1.medium.com/max/1024/0*07vZDUPD0sIRqkmV.jpg" /><figcaption>MKRUSDT pair with EMA16,32,54,72,96 plotted in 1H, 4H, 12H, 48H timeframes</figcaption></figure><p>As you can see, it can also be used to identify trend phases, retracements, and trend reversals. Especially when it is used with multiple time frames.</p><p>But, the thing that I like most about this approach is being capable of representing price action in a nice summary that is easy to remember and it does not require any human knowledge/effort to calculate.</p><h3>Representing of curves</h3><p>The idea was nice at first sight, but it still requires too much human effort, at least for me. We must be able to represent these EMA curves with functions that we can use in our market discovery algorithm. You can make sure that I’ve spent an incredible amount of time here coming up with a nice solution. After finishing my research, I was in favor of using TCN deep autoencoder networks to compress the EMA curves and represent them with latent space, but I chose a more classical approach due to practical reasons for now. Those who want to get familiar with TCN and autoencoder architectures can have a look at these papers.</p><p><a href="https://arxiv.org/abs/1608.08242">https://arxiv.org/abs/1608.08242</a></p><p><a href="http://www.gm.fh-koeln.de/ciopwebpub/Thill20a.d/bioma2020-tcn.pdf">http://www.gm.fh-koeln.de/ciopwebpub/Thill20a.d/bioma2020-tcn.pdf</a></p><p><a href="https://arxiv.org/abs/2102.01331">https://arxiv.org/abs/2102.01331</a></p><p>I choose a relatively much simpler way to represent EMA curves with fewer data points. It is called <strong>b-splines</strong>. I love them a lot, to be honest.</p><p>Splines are one of the piecewise regression techniques used frequently in engineering applications to represent spatial data. They are piecewise, which means they have nicer stability features over high-level polynomial regressions. If you want to learn about data fitting principles and enjoy your free time, I strongly recommend Dr. Nathan Kutz’s great video series on his channel.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBqZXS3n75l0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBqZXS3n75l0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBqZXS3n75l0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcdf3c221ff38fbb808c6465dcf70570/href">https://medium.com/media/dcdf3c221ff38fbb808c6465dcf70570/href</a></iframe><p>We are going to use b-splines to fit EMA curves on the price data to represent them with fewer data points. And then we are going to use it for every symbol that is currently being traded in the Binance spot market. Later we do, we will aim to cluster the market with these price representations to grasp market dynamics at the moment.</p><h3>Conclusion</h3><p>I always think it’s important to have a fundamental understanding of the underlying challenge. I hope this was a good introduction to the work we will be implementing in the next chapters. As a beginner-level writer, it is sometimes hard to convey my thoughts in a clear and structured way. We will get our hands dirty in the following chapters. Stay tuned.</p><p>Any feedback and comments are welcomed. Thanks for your time reading this, I hope you enjoyed it.</p><p>You can continue reading on part 2.</p><p><a href="https://medium.com/@mkaanerkoc/financial-markets-clustering-part-2-b-spline-fitting-c12d2f8d36c">Financial markets clustering: Part 2, B-spline fitting</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d243a12538bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/trading-data-analysis/financial-markets-clustering-series-part-1-d243a12538bf">Financial markets clustering: Part 1, Intuitional buildup</a> was originally published in <a href="https://medium.com/trading-data-analysis">Trading Data Analysis</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>