<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by M on Medium]]></title>
        <description><![CDATA[Stories by M on Medium]]></description>
        <link>https://medium.com/@mattodon?source=rss-736dc59c2a3c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by M on Medium</title>
            <link>https://medium.com/@mattodon?source=rss-736dc59c2a3c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:15:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mattodon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Creating a Distributed Computer Cluster with Python and Dask]]></title>
            <link>https://medium.com/data-science/creating-a-distributed-computer-cluster-with-python-and-dask-5dd7e5cc5b2f?source=rss-736dc59c2a3c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dd7e5cc5b2f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[finance]]></category>
            <dc:creator><![CDATA[M]]></dc:creator>
            <pubDate>Fri, 22 May 2020 07:19:54 GMT</pubDate>
            <atom:updated>2020-05-23T13:39:36.447Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>How to set-up a distributed computer cluster on your home network and use it to calculate a large correlation matrix.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ngm-TiHKhUx56OeE" /><figcaption>Photo by <a href="https://unsplash.com/@tvick?utm_source=medium&amp;utm_medium=referral">Taylor Vick</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Calculating a correlation matrix can very quickly consume a vast amount of computational resources. Fortunately, correlation (and covariance) calculations can be intelligently split into multiple processes and distributed across a number of computers.</p><p>In this article, we will use Dask for Python to manage the parallel computation of a large correlation matrix across a number of computers on a Local Area Network.</p><h3>What is a Correlation Matrix</h3><p>A correlation matrix shows the linear statistical relationship between two variables. Famously, correlation does not imply causation, but we still regularly make use of it as part of our efforts to understand the datasets we work with.</p><p>If you just want to go ahead and compute a correlation matrix on your own home cluster of computers then skip this section, but if you are interested in how to actually calculate correlation mathematically then read on.</p><p>One easy way to visualise the steps is to recreate this in Excel, so I will show you in Excel how to calculate the correlation and covariance between three sets of Foreign Exchange data.</p><p>First, we take our three sets of time-series data, arrange it by columns for each currency pair, and then take the mean (sum of values, divided by number of values) of each set of FX rates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*QdgCdDYJADA1xApQUKjM4Q.png" /></figure><p>Then for each time series data point, we calculate the difference of that point from the mean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*KeQeAGC8qbeovkv7onyXqA.png" /></figure><p>We then square each of those differences which will give us our variance for each of the datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOf26g10hzPZpiWLeaESrA.png" /></figure><p>We then calculate the sum of the range of squared values, divided by our sample size minus one, giving us our Variance, 0.0013% for GBPUSD for example as seen in row 3. We subtract one from the count of our data points as this is a sample of our dataset (we do not have every single historical datapoint) rather than the entire population. This is known as the “sample variance”. If we had the entire population we would not subtract one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*gnj9xRrvPiQ9Yv1zAPLfSw.png" /></figure><p>We can also calculate the Standard Deviation at this stage, which is simply the square root of the variance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*maR9SVBJY1T75D-6hF9Vcg.png" /></figure><p>We can calculate the Covariance between two given datasets by multiplying the two “Difference from mean” values at each point in time. The example below shows the single point-in-time covariance calculation for GBPUSD and JPYUSD:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ELcdTmL-05B7oAaWpz4zBQ.png" /></figure><p>We have called GBPUSD dataset A, JPYUSD dataset B, and so on. With four individual datasets we then have 6 combinations for our covariance calculation: AB, AC, AD, BC, BD, and CD. All of these are calculated in the same way, by multiplying the differences from the means of each dataset at each point in time.</p><p>We then take the mean of these values, in the same way as we did for the variance — subtracting one from the denominator — to get the covariance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yYPO65q51vhe9PiXQXVotg.png" /></figure><p>And finally, we can calculate the correlation of these two datasets by dividing the covariance by the product of the two standard deviations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDmzn2AW4kJWMZu5R9ZtNw.png" /></figure><p>Mathematically, we can represent this as :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*Y7k-Uhd86QdOv-CcEVlv8Q.png" /></figure><p>This may look daunting but it is just what we calculated already. Looking at the very right-hand equation, the top of the fraction is saying “take the difference of each data point in dataset X from the mean of dataset X, and multiply that by the equivalent calculation in dataset Y” and then sum all of those values together. The x with the subscript i just means a given point of dataset x, whilst x̅ means the average for the dataset. The same goes for the y-values.</p><p>The denominator is saying, again, for every point in X, subtract the mean value of X from that point and then square the value you get. Sum all of those together and do the same for Y. Now multiply those two values together and take the square root of that number. The result is the product of the standard deviations of the two datasets.</p><p>This is exactly what we have just calculated for ourselves and it’s exactly what we are doing today, except we are using a home computer cluster to calculate correlations like this across a large number of financial instruments for a significant number of points in time.</p><h3><strong>Setting up the Environment</strong></h3><p><strong><em>The Network</em></strong></p><p>The first step is to establish your computation environment on every computer that will form part of the cluster. You will need to make sure the computers can see each other on the Local Network (through firewall exceptions). You will also need to note the IP address of each computer.</p><p>On Windows you can go to a Command Prompt and type:</p><pre>ipconfig</pre><p>This will give you your IP address on the Local Network. On a Linux machine you can type:</p><pre>hostname -I</pre><p>Or on OSX the below should work:</p><pre>ipconfig getifaddr en0</pre><p>You should note each of the IP addresses on each of your machines as you will need this to set-up the cluster. You can also check that each computer can see the others by running the command:</p><pre>ping &lt;Target IP Address&gt;</pre><p>If this responds with data then the two computers can see each other, if not you will need to double-check the IP address and ensure that a firewall is not blocked communication.</p><p>We are going to use three computers as part of our small cluster, 192.168.1.1, 192.168.2 and 192.168.1.3.</p><p><strong><em>Python and Dask</em></strong></p><p>One very important point to note is that <strong>each Python environment must be identical on each computer</strong>. If there are inconsistencies in library versions this can cause problems.</p><p>If it is easier, use Anaconda or a similar tool to create a new Python environment with matching versions of the libraries you will be using such as Numpy and Pandas.</p><p>You will also need to install Dask itself on each of the computers. You can do this using Conda:</p><pre>conda install dask distributed -c conda-forge</pre><p>Or you can use Pip:</p><pre>python -m pip install dask distributed --upgrade</pre><p>You should now have consistent Python installations, with matching library versions, across all of the computers in your cluster.</p><h3>Sourcing the Data</h3><p>In this example, I am using a large dataset of daily equities data. This contains the daily close prices of ~7,000 US equities from 2000 to 2020. This would be extremely time-consuming to compute on any one of the personal computers that form part of my cluster. When I tried the calculation initially the system crashed after a few hours due to hitting a memory limit when writing to the page file.</p><p>These data are stored in a large SQL table which we are going to access sequentially for each stock and then merge. This proved significantly faster than performing the manipulations in Python.</p><p>We will also be using a different implementation of a MySQL Database Connector. The standard MySQL connector library is relatively slow when dealing with large datasets because it itself is written in Python. The C implementation of this connector is must faster and you can find the installation details here:</p><p><a href="https://mysqlclient.readthedocs.io/user_guide.html">MySQLdb User&#39;s Guide - MySQLdb 1.2.4b4 documentation</a></p><p>Once MySQLdb is installed you can begin to query the data. I am firstly extracting a list of all unique, chronologically-ordered dates from the table. This will be the golden source of each timestep in the overall database.</p><p>I am then extracting, stock-by-stock, the time series data, and left-joining this to the golden time series source. I then add this individual dataframe to a list of dataframes, one for each stock that has been extracted. I defer the merge operation to combine each stock time series into a single dataset until later.</p><p>This query approach can itself be parallelised, where a single query to extract all data from the table could not be.</p><p>This leaves us with a list of datasets with a common time-series index which we can then work with.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e84a1114967929a590e6b7dfe94118c5/href">https://medium.com/media/e84a1114967929a590e6b7dfe94118c5/href</a></iframe><p>Here we are extracting our list of stocks, building our golden source for the time-series, and then iterating through the list extracting the time series data for each stock in this common format.</p><p>The query where we extract the stock data:</p><pre>SELECT date, close as {} FROM history where stock = &#39;{}&#39;</pre><p>takes the closing price of the stock and names that column with the ticker for that stock. This means that instead of having 7,000 columns each with the name “close” we will have 7,000 columns each named with the relevant ticker code and mapped to a common time-series. This will be much easier to deal with.</p><h3>Calculating the Correlation Matrix</h3><p>Now for the good bit. We are going to get the data in its final format and begin to calculate the correlation matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/abbfd6fe70d66fc767ecb2637c772c84/href">https://medium.com/media/abbfd6fe70d66fc767ecb2637c772c84/href</a></iframe><p>Here we are setting the date index for each of the dataframes in our list and merging them together. Because we named each column after the relevant stock ticker, we now have one large table which has a common list of dates as an index and a labeled column for each stock. We’re going to simplify the table where we can by dropping and column with all “NaN” values — this is where there is no relevant data.</p><p>This is the exact format we need to calculate our correlation matrix so we are going to:</p><pre>import dask.dataframe as dd</pre><p>and create a Dask dataframe</p><pre>merged = dd.from_pandas(merged, 20)</pre><p>This is the time when you will need to make an important design decision that will significantly impact the speed of processing the correlation matrix.</p><p>Here we are converting the Pandas dataframe into a Dask dataframe and we also have to specify a number, here “20”, for the number of partitions in the dataset.</p><p>Each partition will be treated as an independent unit for the purposes of parallel calculation. If you select “1” for example, the correlation matrix will be calculated by a single thread on a single CPU and you will achieve no parallelisation benefit.</p><p>On the other end of the spectrum if you set this number too high your performance will suffer as there is an overhead to loading and processing each task.</p><p>Initially, I set this equal to the number of threads across the computers in my mini-cluster. However, I soon found that one of the machines was much slower and the two faster machines were sat idle waiting for this one to complete. To mitigate this I increased the number of processes so that the faster computers could continue to work on later tasks whilst the slower one continued to work on the initially allocated processes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51af185bbe6974ab5830446d4427cf86/href">https://medium.com/media/51af185bbe6974ab5830446d4427cf86/href</a></iframe><p>After making this decision, we convert the merged dataframe to Float 16 from Float 32 datatype as the extra precision is unnecessary and will slow our next calculations.</p><p>We then run the critical line:</p><pre>merged = merged.corr().compute()</pre><p>There are two functions here, <em>.corr()</em> which calculates our correlation matrix, and <em>.compute()</em> which sends it to Dask to calculate.</p><p>After this has processed, we will receive a correlation matrix, showing the correlation of each stock with each other. Below is a simplified illustration of what this could look like (with made-up data as we haven’t calculated the matrix</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34daf7b1d08479d3b21798e0380081d1/href">https://medium.com/media/34daf7b1d08479d3b21798e0380081d1/href</a></iframe><p>You will notice there is a lot of repetition and unnecessary data here. The correlation of AAPL with APPL is 1. The correlation of AMZN with AMZN is 1. This is going to be the case across the diagonal of your entire matrix.</p><p>Similarly, there is duplication. The correlation of AMZN with AAPL is clearly the same as the correlation of AAPL with AMZN (it’s the same thing!).</p><p>There is no point in duplicating all of this data, particularly for such a large dataset, when you come to save this back in your dataset. So let’s get rid of this duplication.</p><p>You will notice that the unnecessary data values form half of the dataset, across a diagonal shape. So let’s mask out that diagonal before we save down this dataset.</p><pre>corrs = merged.mask(np.tril(np.ones(merged.shape)).astype(np.bool))</pre><p>This line does exactly that. It creates a matrix of ones in the shape of our correlation matrix. Then, it uses Numpy to return a copy of the data with the given diagonal zeroed-out using the tril() function. Then it turns these zeroes into a False condition by casting it to the type bool. Finally it uses Pandas to apply this as a mask to the dataset and saves it as our new correlation matrix.</p><p>We can then use convert this simplified matrix into a table which we can then upload back into our database.</p><pre>levels = merged.columns.nlevels</pre><pre>...</pre><pre>df1 = corrs.stack(list(range(levels))).reset_index()<br>del df1[&quot;level_0&quot;]<br>del df1[&quot;level_2&quot;]<br>df1.columns = [&quot;Stock 1&quot;, &quot;Stock 2&quot;, &quot;Correlation&quot;] <br>print(&quot;Uploading SQL&quot;)<br>df1.to_sql(&quot;correlations&quot;, con=engine, if_exists=&#39;append&#39;, chunksize=1000, index=False) <br>print(&quot;Process Complete&quot;)</pre><p>Here we convert this back into a table for our database by using the stack() function. This gives us a table that looks something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/189655ec1d989f5c5cb264d12f6db737/href">https://medium.com/media/189655ec1d989f5c5cb264d12f6db737/href</a></iframe><p>You can see there is no duplication of values or redundant 1s showing a stock’s correlation with itself. This can then be uploaded back into the database and the process is complete.</p><h3>Distributed Computation of the Matrix</h3><p>There is one final, crucial line we must add to our code but first we need to create our cluster.</p><p>On each computer you will need to go to a Python terminal. If you are using Anaconda you can use the GUI to launch a terminal, or you can go to Anaconda Prompt and select your environment using:</p><pre>conda activate &lt;environment&gt;</pre><p>You will now need to decide which computer will act as a scheduler and manage the distribution of the tasks and which will act as workers. It is perfectly possible for a computer to act as both scheduler and worker.</p><p>On the scheduler, go ahead and enter the following into the Python terminal:</p><pre>dask-scheduler</pre><p>This will launch the scheduler and manage the workflow around your cluster. Make a note of the local network IP address that you have done this on, in our case, we will use 192.168.1.1.</p><p>We will want this machine to also act as a worker, so we will open another Python terminal and type:</p><pre>dask-worker 192.168.1.1:8786</pre><p>This should show that we have established a connection between the worker and the scheduler.</p><p>Now we need to repeat this command on each of the other computers in our cluster, with each pointing towards the scheduler hosted at 192.168.1.1.</p><p>I mentioned earlier than one of the computers was slower than the others. It did, however, have twice as much RAM which can be handy for assembling the correlation matrix at the end. Instead of removing it from the cluster entirely, I decided to limit the number of processes it could run by restricting the number of threads available to Dask. You can do this by appending the following to your Dask-worker instruction:</p><pre>dask-worker 192.168.1.1:8786 --nprocs 1--nthreads 1</pre><p>Now your cluster is up and running, you can add the crucial line to your code. Add the following near the top of your Python file (or in any case, above your call to .corr().calculate():</p><pre>client = _get_global_client() or Client(&#39;192.168.1.1:8786&#39;)</pre><p>Make sure you replace the IP address with that of your scheduler. This will point Dask towards the scheduler to manage the computation.</p><p>Now, you can run your program.</p><h3>Monitoring the Computation</h3><p>Dask provides an excellent Dashboard for monitoring the progress of your computation.</p><p>Go to 192.168.1.1:8787 (or whichever IP address you have established your scheduler on using the port 8787).</p><p>This will show you the Dask Dashboard so you can track progress and monitor the utilisation of the cores and memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_YDRuliP9XH373m8nwhA9w.png" /><figcaption><em>The memory and CPU usage of the cluster</em></figcaption></figure><p>The “Workers” page above shows you the nodes in your cluster and their current levels of utilisation. You can use this to track which are doing the most work and make sure this remains in an acceptable range. It is worth noting the CPU utilisation percentage is based on each core, so you can have 200% CPU utilisation if two cores are used, or 800% for eight cores etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gAOkgWGCp6QjgrVefKpmGg.png" /><figcaption>The Dask Dashboard status screen</figcaption></figure><p>The status screen shows the breakdown of the tasks being worked on. The screenshot above shows that we have converted 4 of the 20 partitions from Pandas dataframes into Dask dataframes, with the others being worked on across the cluster. We can also see that we have already stored over 4 GB of data, with plenty more to come.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U6Q_AoS0vBBc7-pEEpbNzw.png" /><figcaption>The Dask Dashboard graph screen</figcaption></figure><p>The Graph screen shows graphically how the task has been divided up. We can see that each partition is being handled separately, so there are 20 partitions as expected. Each partition goes through three stages, stage 0 where it is converted from Pandas to Dask, stage 1 where the correlation matrix for that particular partition is calculated, and stage 3 where all the partitions are combined and the cross-correlations are calculated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6iL_OlsyR_RzLJ_kBnGgSA.png" /><figcaption>The Dask Dashboard System screen</figcaption></figure><p>The System screen shows the current system utilisation of resources, including CPU, Memory, and Bandwidth. This shows you how much computational power is being drawn by the cluster at any given time and is useful for seeing the current strain and load on the network.</p><p>You can use these screens to watch your calculation progress and you will gradually see the various Tasks turn green as they finish calculating. The results will then be propagated across the network to a single node which will then compile the results and return them to your Python program.</p><p>As you have already coded it to do, the resultant matrix will then be slimmed down to its relevant diagonal part, tabulated, and uploaded to your database.</p><h3>Conclusion</h3><p>You have now successfully set-up a computer cluster and used it to calculate a correlation matrix, simplify your results, and store them away for further analysis.</p><p>You can use these core components to do many more interesting things with your cluster, from machine learning to statistical analysis.</p><p>Let me know how you get on.</p><p><em>You can follow me on Twitter </em><a href="https://twitter.com/mgrint"><em>@mgrint</em></a><em> or see more from me at </em><a href="https://grint.tech/"><em>https://grint.tech</em></a><em>. Email me at </em><a href="/service/mailto:matt@grint.tech"><em>matt@grint.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dd7e5cc5b2f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/creating-a-distributed-computer-cluster-with-python-and-dask-5dd7e5cc5b2f">Creating a Distributed Computer Cluster with Python and Dask</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are Hedge Funds Buying?]]></title>
            <link>https://mattodon.medium.com/what-are-hedge-funds-buying-8c24444ad56?source=rss-736dc59c2a3c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c24444ad56</guid>
            <category><![CDATA[investing]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[M]]></dc:creator>
            <pubDate>Tue, 28 Apr 2020 06:02:19 GMT</pubDate>
            <atom:updated>2020-04-28T09:02:48.529Z</atom:updated>
            <content:encoded><![CDATA[<h3>What are Hedge Funds Buying? Extracting 13F data with Python.</h3><p><em>Part I — How to retrieve the stock holdings of Hedge Funds automatically using Python and store these in an SQL database.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dl4ETYgvJ1iKhOrB" /><figcaption>Photo by <a href="https://unsplash.com/@m_b_m?utm_source=medium&amp;utm_medium=referral">M. B. M.</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3><strong>Introduction</strong></h3><p>Hedge Fund and other investment managers with over $100m in Assets under Management (AUM) are required to file a quarterly document in the US showing the securities that they hold. These documents, known as a 13F filing, are keenly watched by markets to understand what top hedge funds are thinking and how they are allocating their money.</p><p>Unfortunately for most investors, simply mimicking the investments you see in a 13F is unlikely to yield you much better than the market average, and could be significantly worse. There are several reasons for this. Firstly, the form isn’t filed until 45 days after the end of the quarter, meaning plenty of investments could have changed in the intervening time. The anticipated price movements could also have already occurred by this point.</p><p>Secondly, even when you have the holdings data it is sometimes hard to know what an asset manager’s net position is. If a fund reports a long position in the shares, a put option and a call option they could be net long, net short or more likely have a limited pay-off in a given circumstance (e.g. a collar, providing returns if the price moves between 2% and 10%). There could also be unlisted positions, such as short positions in a convertible bond, that would require longs in the underlying security whilst the manager maintains a net short exposure — or an investor could be partially hedged via Swaps or CDS.</p><p>Thirdly, the 13F does not require short positions to be listed so again it can give an incomplete or potentially misleading view of the asset manager’s net exposure to given security.</p><p>Furthermore, the analysis is point-in-time, and without knowing what the holding times are for various assets it is difficult to know whether they are trading in and out of the stocks on a day-by-day basis, or whether these are long-term conviction purchases.</p><p>Finally, the frequency of the reporting means that piggybacking on these trades is difficult. Knowing when a manager has exited a position could take up to a quarter of a year after it actually occurs.</p><p>That being said, 13Fs are very useful for understanding thematically what the market is looking at. You can also see where Hedge Funds are clustering their trades (Big Tech) and get a sense of flows into and out of sectors. Depending on the investor you can also sometimes infer some characteristics of the company. For example, Elliot Management are activist investors who may be proposing changes to the corporate structure. Berkshire Hathaway is likely to hold their investment for many years. AQR are likely to have identified some fundamental value factors at play in their long positions.</p><h3>Getting the holdings with Python</h3><p>We’re going to be using Python to get the 13F, primarily because Python is a useful tool for the analysis we will be doing in Part II.</p><p>Fortunately, there are already many tools for Python that make sourcing this data very simple so let’s not reinvent the wheel and start by loading these packages.</p><pre>from sec_edgar_downloader import Downloader</pre><p>This package is called SEC Edgar Downloader. Edgar is the regulatory filing system to which the 13F is submitted.</p><p>We’ll use this package to form the core of our get_files() function:</p><pre>def get_files(filing, CIK):<br>    save_path = &#39;C:\\13F\\&#39;<br>    dl = Downloader(save_path)<br>    dl.get(filing, CIK)<br>    CIK = CIK.lstrip(&quot;0&quot;)<br>    files = os.listdir(&quot;C:\\13F\\sec_edgar_filings\\&quot; + CIK + &quot;\\&quot; + filing)<br>    data = [parse(file, CIK) for file in sorted(files)]<br>    try:<br>        return pd.concat(data)<br>    except ValueError:<br>        print(&quot;All Values are None&quot;)<br>        return None</pre><p>Here we save the13F files to our local drive in the form of .txt documents. We pass two parameters, firstly the name of the filing — “13F-HR”, HR standing for Holdings Report. Secondly, we pass a CIK which is a Central Index Key identifying the asset manager.</p><p>Then we use the SEC Edgar Downloader to download the relevant 13F text files. Once this is done, we can call our next function <strong>parse()</strong> using list comprehension. This line:</p><pre>data = [parse(file, CIK) for file in sorted(files)]</pre><p>sorts all files by name (which also sorts chronologically based on the naming convention) and passes the file to our next function.</p><h3>Parsing the Data</h3><p>To parse the data we are going to make use of a great tool called BeautifulSoup.</p><pre>from bs4 import BeautifulSoup</pre><p>This will allow us to parse the XML format of the 13F filings. The XML format is customised for the SEC document but Beautiful Soup can parse the files.</p><pre>def parse(file, CIK):<br>    xml = open(&#39;C:\\13F\\sec_edgar_filings\\&#39; + CIK + &#39;\\13F-HR\\&#39; + file).read()<br>    soup = BeautifulSoup(xml, &#39;lxml&#39;)<br>    try:<br>        date = datetime.datetime.strptime(soup.find(&#39;signaturedate&#39;).contents[0], &#39;%m-%d-%Y&#39;)<br>        name = soup.find(&#39;name&#39;).contents[0]<br>    except:<br>        return<br>    cols = [&#39;nameOfIssuer&#39;, &#39;cusip&#39;, &#39;value&#39;, &#39;sshPrnamt&#39;, &#39;sshPrnamtType&#39;, &#39;putCall&#39;]<br>    data = []<br>    print(&quot;Processing &quot; + name + &quot; (&quot; + CIK + &quot;) for date &quot; + str(date))<br>    for infotable in soup.find_all([&#39;ns1:infotable&#39;, &#39;infotable&#39;]):<br>        row = []<br>        for col in cols:<br>            data = infotable.find([col.lower(), &#39;ns1:&#39; + col.lower()])<br>            row.append(data.text.strip() if d else &#39;NaN&#39;)<br>        row.append(date)<br>        row.append(CIK)<br>        row.append(name)<br>        data.append(row)<br>    df = pd.DataFrame(data)<br>    cols.append(&#39;date&#39;)<br>    cols.append(&#39;fund_cik&#39;)<br>    cols.append(&#39;fund&#39;)<br>    try:<br>        df.columns = cols<br>        return df<br>    except:<br>        return</pre><p>Here we are doing several things to parse our data, extract the content that we want, and get it all into a neat DataFrame. Firstly, we are opening the file and parsing it with Beautiful Soup. This gives us a variable — <em>soup</em> —<em> </em>which we can use to traverse the document.</p><p>The first thing we are going to extract is the date of the document via a field called <em>signaturedate</em> and convert this into a DateTime object. We are also going to extract the name of the asset management through the appropriately named <em>name </em>field.</p><p>If we can’t extract the name and date we are simply going to terminate the function. The signature date field was introduced in 2013 so if you are looking for holdings data before this, simply remove the (overly wide) Try: and Except: block that I have put in and change your data parser. There is a text field at the top of the document that you can use, outside of BeautifulSoup.</p><p>The actual data we are looking for is held in a series of <em>info_table tags, </em>so we are going to use Beautiful Soup to find all those and loop through them. Each time we loop through we are going to capture the variables of interest to us, here I am taking:</p><pre>cols = [‘nameOfIssuer’, ‘cusip’, ‘value’, ‘sshPrnamt’, ‘sshPrnamtType’, ‘putCall’]</pre><p>As we loop through and add each row to our result, I am also adding the date, fund name, and the CIK to each row so that we can use these in our lookup table later.</p><p>Finally, a DataFrame is returned which contains all the rows and columns we have selected. This is returned to our previous function which concatenates all the results into a single DataFrame and returns this.</p><h3>Storing the Data</h3><p>Now that we have extracted the data that we want it would be a good idea to store it somewhere. I’ve decided to store it in an SQL database but a NoSQL solution might be preferable for your purposes.</p><pre>form = &#39;13F-HR&#39;<br>ciks = [&#39;1037389&#39;]</pre><pre>i = 1<br># Get files from Edgar<br>for cik in ciks:<br>    # Provide feedback<br>    print(&quot;----- &quot; + str(i) + &quot; of &quot; + str(len(ciks)) + &quot; -----&quot;)<br>    i += 1<br><br>    # Download files<br>    df = get_files(form, cik)<br><br>    if df is not None:<br><br>        # Perform cleaning<br>        df[&#39;value&#39;] = df[&#39;value&#39;].astype(float)<br>        df[&#39;sshPrnamt&#39;] = df[&#39;sshPrnamt&#39;].astype(int)<br>        df[&#39;date&#39;] = df[&#39;date&#39;].apply(get_qend)<br><br>        # Upload to Database<br>        engine = create_engine(&quot;mysql+pymysql://{user}:{pw}@localhost/{db}&quot;<br>                               .format(user=&quot;<strong>USERNAME</strong>&quot;,<br>                                       pw=&quot;<strong>PASSWORD</strong>&quot;,<br>                                       db=&quot;<strong>DATABASE</strong>&quot;))<br>        try:<br>            df.to_sql(&#39;holdings&#39;, con=engine, if_exists=&#39;append&#39;, chunksize=1000, index=False)<br>        except:<br>            pass<br>        print(&quot;Uploaded holdings to sql&quot;)</pre><p>Here is the main function of the file which calls our <strong>get_files()</strong> function, cleans the data a little, and then uploads it to our SQL database.</p><p>We declare our information of interest in the form of a variable <em>form</em> which in this case is 13F-HR, and a list of CIKs for the funds we wish to investigate. Here I have used 1037389 for Renaissance Technologies as they seem like pretty smart guys.</p><p>We loop through the list providing a bit of feedback to the console on how far we have got, in case you decide to add several CIKs and want to check on progress. You can find CIKs here: <a href="https://www.sec.gov/edgar/searchedgar/cik.htm">https://www.sec.gov/edgar/searchedgar/cik.htm</a></p><p>We convert the two numeric data fields into a float and an int respectively and apply a custom function called <strong>get_qend</strong> to our date. This function takes the date and returns the end date of the prior quarter, which is the reference date for the 13F.</p><p>There are few date functions used for various purposes which I’ve included below for your reference.</p><pre># Get prior quarter<br>def get_prior_quarter(q):<br>    prior = q - 1<br>    if prior == 0:<br>        prior = 4<br>    return prior<br><br><br># Get current quarter<br>def get_quarter(date):<br>    return (date.month - 1) / 3 + 1<br><br># Get prior weekday for a given date<br>def prev_weekday(date):<br>    date -= timedelta(days=1)<br>    while date.weekday() &gt; 4:  <br>        date -= timedelta(days=1)<br>    return date<br><br># Get prior quarter end for a given date<br>def get_qend(date):<br>    quarter = get_prior_quarter(int(get_quarter(date)))<br>    month = quarter * 3<br>    year = date.year<br>    day = calendar.monthrange(year, month)[1]<br>    quarterEnd = datetime.date(year, month, day)<br>    if quarterEnd.weekday() &gt; 4:<br>        return prev_weekday(quarterEnd)<br>    else:<br>        return quarterEnd</pre><p>Finally, we use SQLAlchemy to connect to our database using <strong>create_engine</strong>. You will need to pass your own username, password, and database table name to this function.</p><p>Then we convert our DataFrame to SQL automatically using SQLAlchemy and upload the data to our database. In this case, I am using index=false to prevent it from creating a numeric index as I am using a composite primary key across several fields for uniquely identifying the data.</p><p>Once you have run this your 13F data should be safely stored in your database, ready for analysis. I’ll show you some of the analysis you can do in Part II.</p><h3>Conclusion</h3><p>Whilst the 13F isn’t going to give you the portfolio of your dreams, it can provide some interesting insights into thematic allocations of leading hedge funds.</p><p>We’ve shown how to automatically retrieve the data for any number of Hedge Funds, clean the data and extract the fields that you are interested in, and then store this in a relational database.</p><p>In Part II I will show some of the analyses that you can do on these holdings and apply some data science techniques to see if we can find any interesting characteristics of these Hedge Funds.</p><p><em>You can follow me on Twitter </em><a href="https://twitter.com/mgrint"><em>@mgrint</em></a><em> or read more of my articles at </em><a href="https://grint.tech/"><em>https://grint.tech</em></a><em>. Email me at </em><a href="/service/mailto:matt@grint.tech"><em>matt@grint.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c24444ad56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sending push notifications to Android with PHP and FCM]]></title>
            <link>https://mattodon.medium.com/sending-push-notifications-to-android-with-php-and-fcm-ca34cdf2eb51?source=rss-736dc59c2a3c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca34cdf2eb51</guid>
            <category><![CDATA[android-development]]></category>
            <category><![CDATA[php]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[android]]></category>
            <dc:creator><![CDATA[M]]></dc:creator>
            <pubDate>Sat, 25 Apr 2020 05:35:06 GMT</pubDate>
            <atom:updated>2020-04-25T07:45:00.182Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Send timely alerts to users even when your app isn’t running. In this example, we will send a daily notification showing the most-read story of the day on the Novella app using Firebase Cloud Messaging.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rxz6mb97F51zJxeZ" /><figcaption>Photo by <a href="https://unsplash.com/@brunoreyna?utm_source=medium&amp;utm_medium=referral">Bruno Reyna</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We are going to be using the Firebase SDK for PHP which you can download here: <a href="https://github.com/kreait/firebase-php/">https://github.com/kreait/firebase-php/</a>. The first step will be to install this using Composer.</p><p><strong>Installing Firebase SDK</strong></p><p>The easiest way to install the Firebase SDK on your server is using Composer. You will need to log-in to your server using an SSH client; the leading free client is PuTTY which you can download here: <a href="https://www.chiark.greenend.org.uk/~sgtatham/putty/">https://www.chiark.greenend.org.uk/~sgtatham/putty/</a></p><p>Once you are connected to your server using your log-in credentials you will need to install Composer if it is not already set-up. Below is the recommended install script for Composer to be used in your SSH terminal or by saving as a batch script in your chosen directory. Navigate to your website directory and enter the below, or run as a batch script.</p><pre>php -r &quot;copy(&#39;https://getcomposer.org/installer&#39;, &#39;composer-setup.php&#39;);&quot;<br>php -r &quot;if (hash_file(&#39;sha384&#39;, &#39;composer-setup.php&#39;) === &#39;e0012edf3e80b6978849f5eff0d4b4e4c79ff1609dd1e613307e16318854d24ae64f26d17af3ef0bf7cfb710ca74755a&#39;) { echo &#39;Installer verified&#39;; } else { echo &#39;Installer corrupt&#39;; unlink(&#39;composer-setup.php&#39;); } echo PHP_EOL;&quot;<br>php composer-setup.php<br>php -r &quot;unlink(&#39;composer-setup.php&#39;);&quot;</pre><p>Once Composer is installed, you can use it to automatically install the Firebase SDK as below:</p><pre>composer require kreait/firebase-php</pre><p>This will install all the components you need to use Firebase on your server.</p><p><strong>Setting up Firebase Account</strong></p><p>If you don’t have a Firebase Account you will need to create one. If you already use Firebase in your project you can skip to the next section.</p><p>Within Android Studio you can click Tools-&gt;Firebase and follow the semi-automated process to set up your account. Otherwise, you can visit <a href="https://console.firebase.google.com/">https://console.firebase.google.com/</a> and register your app on this website.</p><p>After entering your project details you will need to provide your app’s SHA fingerprint. You can get this easily from Gradle by clicking the ‘Gradle’ button on the right-hand side of Android Studio. Then click on your app’s name to open the sub-folders, then Tasks-&gt;android-&gt;SigningReport.</p><p>Running this will show your SHA-1and SHA-256 fingerprint in the build console at the bottom of the screen. On the Firebase Console, go to your project settings and scroll to “SHA certificate fingerprints” and add one of these fingerprints. SHA-256 is a much more secure hashing algorithm than SHA-1 so it makes sense to use that.</p><p>Now, from the Firebase Console click on Service Accounts and Generate New Private Key. You will need to keep this safe and secure as it provides admin access to your Firebase set-up. Upload this to your server in a folder that cannot be accessed by HTTP, e.g. one level above your webpage’s main directory.</p><p><strong>Getting user’s device tokens</strong></p><p>Before we can send a message, we need to know who we are sending it to. For this, we are going to simply store device tokens in an SQL database however you will likely want to capture additional user information for a more targeted approach. Within your app put the below code to capture a device token from a user.</p><pre>// Set up Firebase Cloud Messaging<br>FirebaseInstanceId.getInstance().<em>instanceId</em>.addOnSuccessListener(<br>    this<br>) <strong>{ </strong>instanceIdResult: InstanceIdResult <strong>-&gt;<br>    </strong>val newToken = instanceIdResult.<em>token<br>    </em>updateToken(newToken)<br><br><strong>}</strong></pre><p>The above code loads the token from Firebase and when it is ready uses the callback to the <strong>updateToken()</strong> function.</p><p>Within updateToken() you will want to send the token to your server, below I have sent the token along with the user’s identifying details from Firebase Auth.</p><pre>fun updateToken(token: String){<br>    <em>user</em>?.getIdToken(true)<br>        ?.addOnCompleteListener <strong>{ </strong>task <strong>-&gt;<br>            </strong>if (task.<em>isSuccessful</em>) {<br>                val idToken: String? = task.<em>result</em>?.<em>token<br>                </em>val uid = FirebaseAuth.getInstance().<em>currentUser</em>?.<em>uid<br>                </em>Fuel.post(<br>                    &quot;./saveToken.php&quot;,<br>                    <em>listOf</em>(&quot;token&quot; <em>to </em>token, &quot;uid&quot; <em>to </em>uid, &quot;idToken&quot; <em>to </em>idToken)<br>                ).responseString <strong>{ </strong>request, response, result <strong>-&gt;<br>                    </strong>Log.e(<br>                        TAG,<br>                        &quot;Sent token to server with response: ${result.component1()}&quot;<br>                    )<br>                <strong>}<br>            </strong>} else {<br>                Log.e(<br>                        TAG,<br>                        &quot;Error sending token: ${task.getException()}&quot;<br>                    )<br>            }<br>        <strong>}<br></strong>}</pre><p>You can then store this token in a database to use later for sending messages to that user.</p><p>Here is an example SQL query but you will also want to perform user validation if you are updating details for a logged-in user.</p><pre>$stmt = $conn-&gt;prepare(&quot;INSERT INTO fcm_tokens (user_id, token) VALUES (:uid, :token) ON DUPLICATE KEY UPDATE token=:token&quot;);<br>$stmt-&gt;bindParam(&#39;:uid&#39;, $uid);<br>$stmt-&gt;bindParam(&#39;:token&#39;, $token);<br>$stmt-&gt;execute();</pre><p>If you are using FirebaseAuth to manage your users, you can use the below code to validate the credentials (uid and idToken) sent with the <strong>updateToken() request.</strong></p><pre>use Kreait\Firebase\Factory;<br>use Kreait\Firebase\ServiceAccount;<br><br>$serviceAccount = ServiceAccount::<em>fromJsonFile</em>(&#39;<strong>../YOUR PRIVATE KEY.json</strong>&#39;);<br>$firebase = (new Factory)<br>    -&gt;withServiceAccount($serviceAccount)<br>    -&gt;create();<br><br><br>$idTokenString = $_REQUEST[&quot;idToken&quot;];<br>$uid = $_REQUEST[&quot;uid&quot;];<br><br><br><br>try {<br>    $verifiedIdToken = $firebase-&gt;getAuth()-&gt;verifyIdToken($idTokenString);<br>    if ($verifiedIdToken-&gt;getClaim(&#39;sub&#39;) != $uid) {<br>        die(&quot;Error UID mismatch&quot;);<br>    }<br>} catch (InvalidToken $e) {<br>    die($e-&gt;getMessage());<br>}<br><br>$uid = $verifiedIdToken-&gt;getClaim(&#39;sub&#39;);<br>$user = $firebase-&gt;getAuth()-&gt;getUser($uid);</pre><p><strong>Sending a user message</strong></p><p>Finally, we can actually send a user message. For this tutorial, we are going to take the most read story from the last 24 hours from the Novella app and broadcast it to all users. In practice, you will probably want to create separate groups for different readers to make sure you only send relevant content.</p><p>Firstly, create your new PHP file on your server and load the Firebase SDK. Make sure this file is somewhere that requires authentication to access, otherwise any user could visit this page and send a notification to your users!</p><pre>require <em>__DIR__ </em>. &#39;/vendor/autoload.php&#39;;<br><br>use Kreait\Firebase\Factory;<br>use Kreait\Firebase\ServiceAccount;<br>use Kreait\Firebase\Messaging\CloudMessage;<br>use Kreait\Firebase\Messaging\Notification;<br><br><br>$serviceAccount = ServiceAccount::<em>fromJsonFile</em>(&#39;../<strong>YOUR PRIVATE KEY</strong>.json&#39;);<br>$firebase = (new Factory)<br>    -&gt;withServiceAccount($serviceAccount)<br>    -&gt;create();</pre><p>Next, I am selecting the data from my database that I want to send to users, namely the number one most-read story of the past 24 hours:</p><pre>$stmt = $conn-&gt;prepare(&quot;SELECT s.title, s.id, s.author, <em>count</em>(action) as actions FROM actions a JOIN stories s ON a.story=s.id WHERE DATE(a.date)=DATE(NOW() - INTERVAL 1 DAY) GROUP BY story ORDER BY actions DESC LIMIT 1&quot;);<br>$stmt-&gt;execute();<br><br>$result = $stmt-&gt;fetch(PDO::<em>FETCH_ASSOC</em>);</pre><p>Now we use this information to build our notification. I am providing a title, message, and some data for this notification — we can use the data to load the correct story within the app.</p><pre>$messaging = $firebase-&gt;getMessaging();<br>// Format Notification<br>$title = &quot;Most Read Story: &quot; . $result[&quot;title&quot;];<br>$message = &quot;Click here to read the most read story on Novella today.&quot;;<br>$data = [<br>    &#39;story&#39; =&gt; $result[&quot;id&quot;]<br>];<br><br>$notification = Notification::<em>fromArray</em>([<br>    &#39;title&#39; =&gt; $title,<br>    &#39;body&#39; =&gt; $message,<br>    &#39;data&#39; =&gt; $data<br>]);</pre><p>Now we have built the message, all that remains to be done is to retrieve our users’ device tokens and send them the message.</p><pre>$stmt = $conn-&gt;prepare(&quot;SELECT token FROM fcm_tokens&quot;);<br>$stmt-&gt;execute();<br><br>$result = $stmt-&gt;fetchAll(PDO::<em>FETCH_ASSOC</em>);<br><br>foreach ($result as $row) {<br>    $message = CloudMessage::<em>withTarget</em>(&#39;token&#39;, $row[<em>token</em>])<br>        -&gt;withNotification($notification);<br>    $messaging-&gt;send($message);<br>}</pre><p>When you now visit this page with your browser it will send the notification with your chosen message!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIaSOacTwf5pbWwcYLiRAw.jpeg" /><figcaption>The finished notification.</figcaption></figure><p><strong>Automating the sending of messages</strong></p><p>You don’t want to have to visit this page every time you want to send a message. There are many ways to automate this but an easy one is to set up a Cron Job on your server. Depending on which host or platform you are using this will be done slightly differently but fundamentally you will want to set up a Cron Job to run an HTTP request every day (or whatever your chosen period of time is). The Cron Job should visit the web page you have just created and you should provide the authentication details so that it can load the page.</p><p><strong>Conclusion</strong></p><p>That is the notification process complete. You have now automated a daily notification to all your users sharing a piece of timely and relevant information. You can further customise this by tailoring the message for each user, or dividing your users into groups to receive different pieces of information at different times — perhaps based on their time zone or interests.</p><p><em>Signup for notifications about the Novella app at </em><a href="https://novella.kickoffpages.com/"><em>https://novella.kickoffpages.com/</em></a><em>.</em></p><p><em>You can follow me on Twitter </em><a href="https://twitter.com/mgrint"><em>@mgrint</em></a><em> or read more of my articles at </em><a href="https://grint.tech/"><em>https://grint.tech</em></a><em>. Email me at </em><a href="/service/mailto:matt@grint.tech"><em>matt@grint.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca34cdf2eb51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto-saving drafts in Kotlin for Android]]></title>
            <link>https://mattodon.medium.com/auto-saving-drafts-in-kotlin-for-android-5d9b2a3d3bc1?source=rss-736dc59c2a3c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d9b2a3d3bc1</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[kotlin]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[android]]></category>
            <dc:creator><![CDATA[M]]></dc:creator>
            <pubDate>Fri, 17 Apr 2020 05:47:14 GMT</pubDate>
            <atom:updated>2020-04-17T05:47:14.473Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Automatically saving content as it’s written is an essential part of the user experience. It’s used everywhere from Medium to Microsoft Word. Here’s how to implement this in Kotlin for Android.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V9Gp7oqQZ21pG2Sy" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Plan</h3><p>To create a seamless user experience, we want to save content as user types. Rather than saving after every character that is typed, it would be best to wait for a pause in the typing and save each time the typing stops.</p><p>To do this we need three things. Firstly, a user content form. Secondly, a way of determining when the user is typing and thirdly a way of sending the content to our server for the draft to be stored in our database.</p><h3>Creating the content form</h3><p>The styling of the content form is very much a design choice, but fundamentally it will be an EditText component. This component will allow the user to type long-form text. I’ve used the example below:</p><pre>&lt;EditText<br>    android:id=&quot;@+id/edit_story&quot;<br>    android:layout_width=&quot;wrap_content&quot;<br>    android:layout_height=&quot;match_parent&quot;<br>    android:layout_marginHorizontal=&quot;24dp&quot;<br>    android:background=&quot;@android:color/transparent&quot;<br>    android:layout_marginVertical=&quot;24dp&quot;<br>    android:hint=&quot;Start writing your story..&quot;<br>    android:inputType=&quot;textCapSentences|textAutoCorrect|textMultiLine&quot;<br>    app:fontFamily=&quot;@font/belgrano&quot; /&gt;</pre><h3>Creating our SaveDraft function</h3><p>To save the draft, we need to send the data to our server to be stored in a database table. To keep it simple, we will send the full content every time but if your users are writing particularly large documents then it may be beneficial to optimize this by only sending additions to the existing text.</p><pre>fun saveDraft(text: String){<br>    <em>user</em>?.getIdToken(true)<br>        ?.addOnCompleteListener <strong>{ </strong>task <strong>-&gt;<br>            </strong>if (task.<em>isSuccessful</em>) {<br>                val idToken: String? = task.<em>result</em>?.<em>token<br>                </em>val uid = FirebaseAuth.getInstance().<em>currentUser</em>?.<em>uid<br>                </em>Fuel.post(&quot;./saveDraft.php&quot;, <em>listOf</em>(&quot;uid&quot; <em>to </em>uid, &quot;idToken&quot; <em>to </em>idToken, &quot;creationDate&quot; <em>to </em>creationDate, &quot;text&quot; <em>to </em>text, &quot;title&quot; <em>to title</em>)).responseString <strong>{ </strong>request, response, result <strong>-&gt;<br>                    </strong>val (saveResult, error) = result<br>                    if (saveResult == &quot;Success&quot;){<br>                        Snackbar.make(findViewById&lt;EditText&gt;(R.id.<em>edit_story</em>), &quot;Saved&quot;, Snackbar.<em>LENGTH_LONG</em>).show()<br>                    }<br>                    else<br>                    {<br>                        Log.e(<em>TAG</em>, &quot;Failed to save user story.&quot;)<br>                    }<br>                <strong>}<br>            </strong>} else {<br>                Log.e(<em>TAG</em>, &quot;Failed to generate user token&quot;)<br>            }<br>        <strong>}<br></strong>}</pre><p>Here we are using FirebaseAuth for our authentication and sending the details to the server to be validated along with the user draft. We are also using Fuel to manage our HTTPS request to send the data to the server. We are sending 5 variables to the server:</p><ul><li><em>uid </em>and <em>idToken </em>are authentication variables for FirebaseAuth</li><li><em>creationDate</em> is used as an identifier for the draft so that a user can maintain multiple drafts at the same time. We can set this with val creationDate = LocalDateTime.now()</li><li><em>text </em>and <em>title</em> are the values for the content that we are saving.</li></ul><h3>How to check if the user has stopped typing?</h3><p>To figure out when a pause occurs in the user’s typing, we need to flip this question around and detect when the user starts typing. We can then set a timer which will save our draft after a given time, say one second. If we detect that the user presses another key within that one second then we cancel our action to save the draft. This prevents sending data after every keystroke, which is wasteful but still saves the content after each flurry of activity.</p><p>To do this we need to create a Handler to run our SaveDraft function after a short delay. We can then initiate this every time the user presses a key and cancel it if another keystroke follows within this short time period.</p><pre>val saveDraftHandler = Handler()</pre><p>Once we have created this handler, we then need to detect user keystrokes and use this to initiate and cancel our handler.</p><p>We do this using a TextWatcher component. This can be added to an EditText using a lambda function as below:</p><pre>findViewById&lt;EditText&gt;(R.id.<em>edit_story</em>).addTextChangedListener(object : TextWatcher {<br>    override fun afterTextChanged(p0: Editable?) {<br><br>        if (p0.<em>toString</em>() != &quot;&quot;) {<br>            findViewById&lt;EditText&gt;(R.id.<em>edit_story</em>).<em>hint </em>= &quot;&quot;<br>            saveDraftHandler.removeCallbacksAndMessages(null);<br>            saveDraftHandler.postDelayed({ saveDraft(p0.<em>toString</em>()) }, 1000)<br>        }<br>    }<br><br><br>    override fun beforeTextChanged(p0: CharSequence?, p1: Int, p2: Int, p3: Int) {<br>    }<br><br>    override fun onTextChanged(p0: CharSequence?, p1: Int, p2: Int, p3: Int) {<br>    }<br><br>})</pre><p>The TextWatcher object is created and added as a Listener for when the text changes within the EditText. We must override the three functions which comprise a TextWatcher but the main one we are interested in for now is <strong>afterTextChanged()</strong>. This is called after every keystroke the user makes within that text box.</p><p>Within this function we are using <strong>removeCallbacksAndMessages()</strong> to cancel the Handler to prevent it running the saveDraft activity if it is currently scheduled.</p><p>Then we are rescheduling the activity with <strong>postDelayed()</strong> to run the saveDraft activity after 1000ms.</p><p>If the user is typing a stream of text then the handler keeps getting cancelled and rescheduled until the user eventually stops typing for more than a second, at which point the saveDraft activity is run and the file is saved.</p><p>We have implemented the Runnable within the <strong>postDelayed()</strong> function as a lambda which in turn calls our <strong>saveDraft()</strong> function. We could have built the <strong>saveDraft() </strong>functionality directly into this lambda, however then we wouldn’t have been able to call <strong>saveDraft()</strong> in other contexts so we have decomposed these elements.</p><h3>Conclusion</h3><p>Creating an auto-saving form in Kotlin is very simple using a TextWatcher and the postDelayed functionality of a Handler. To improve this and make it more efficient, one could reduce the amount of data transmitted by checking if this is simply appending text to the previous draft.</p><p>You could do this by storing lastDraft = p0.text and running the comparison if(lastDraft == p0.text.take(lastDraft.length) to check if the two start in the same way. This would give you something like the below:</p><pre>if(lastDraft == text.<em>take</em>(lastDraft.length)){<br>    val appendText = text.<em>drop</em>(lastDraft.length)<br>}</pre><p>More advanced solutions are possible for optimising the data transfer, let me know if you come up with an interesting one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d9b2a3d3bc1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting clicks or scrolls in Kotlin for Android]]></title>
            <link>https://mattodon.medium.com/detecting-clicks-or-scrolls-in-kotlin-for-android-9c2cf0aab7fa?source=rss-736dc59c2a3c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c2cf0aab7fa</guid>
            <category><![CDATA[ui]]></category>
            <category><![CDATA[kotlin]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[android]]></category>
            <dc:creator><![CDATA[M]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 06:35:15 GMT</pubDate>
            <atom:updated>2020-04-25T07:45:32.818Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Build a responsive UI that shows or hides the toolbar in response to user clicks whilst allowing undisturbed user scrolling.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LmKQIv-_vZu5yLD2" /><figcaption>Photo by <a href="https://unsplash.com/@clemhlrdt?utm_source=medium&amp;utm_medium=referral">Clément H</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Kotlin has a fantastic ability to detect user gestures on Android systems. This is important for creating smooth, functional user interfaces that respond in a consistent and intuitive way. However, it can sometimes be difficult to differentiate between similar user actions. For example, when a user scrolls down a window of text this is detected as both a click and a scroll activity. This article will look at how to detect whether a user is clicking the screen, or whether they are scrolling, so that the toolbar can be toggled by user clicks but remain hidden whilst the user scrolls.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIDVUsiB6wN8z1WB9CNGqQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TeaLW3i5Fk6GF9GVV8W1Vg.jpeg" /><figcaption>Screenshots showing the toolbar in hidden and visible modes</figcaption></figure><p>The screenshots above show the toolbar being displayed and hidden. We would like to toggle this behaviour in response to a user click, whilst allowing the user to scroll down the text without displaying the toolbar. To do this we need to follow a few steps:</p><ol><li>Set a listener to detect when a user is touching the ScrollView which contains the text</li><li>Identify whether the user is clicking the screen or scrolling up/down</li><li>If the user is clicking toggle the toolbar, otherwise, ignore the action</li></ol><p>We are going to decompose this into two functions. Firstly a <strong>setHidingToolbar() </strong>function which detects the clicks or scrolls on a given a view. Secondly, a <strong>toggleToolbar()</strong> function which animates the showing and hiding of a given toolbar. These functions will be created in such a way that you can use them on multiple toolbars across your app.</p><h3><strong>setHidingToolbar()</strong></h3><pre>fun setHidingToolbar(view: View, toolbar: Toolbar) {<br>    view.setOnTouchListener <strong>{ </strong>v, event <strong>-&gt;<br>        </strong>when (event.<em>action</em>) {<br>            MotionEvent.<em>ACTION_DOWN </em>-&gt; {<br>                <em>x </em>= event.<em>x<br>                y </em>= event.<em>y<br>            </em>}<br><br>            MotionEvent.<em>ACTION_UP </em>-&gt; {<br>                if (<em>abs</em>(<em>x </em>- event.<em>x</em>) &lt; <em>SCROLL_THRESHOLD </em>|| <em>abs</em>(<em>y </em>- event.<em>y</em>) &lt; <em>SCROLL_THRESHOLD</em>){<br>                    <em>toggleToolbar</em>(toolbar)<br>                }<br>            }<br>        }<br>        v.performClick()<br>        v.onTouchEvent(event) ?: true<br>    <strong>}<br></strong>}</pre><p>The code above takes a view, which is the UI component on which you would like to detect user clicks. In this case, it is a ScrollView. This function also takes as a parameter a toolbar so that we can generalise this function for multiple toolbars across your project.</p><p>First, we set an <strong>onTouchListener </strong>to this view so that whenever the user touches or interacts with this view our function will be called. When the user touches the screen, the <strong>MotionEvent.ACTION_DOWN</strong> will immediately be called, however at this stage we don’t know whether they are looking to scroll the screen or whether they are tapping it to display the toolbar.</p><p>There are two ways of resolving this problem. Either we could see if the user leaves their finger on the screen for a given period of time, or we could see if the user moves their finger across the screen. I have had more luck with the latter as different users touch the screen at different speeds and it can be hard to differentiate between short scrolls (perhaps to read the line above or below the current window) and genuine clicks.</p><p>So secondly, we need to determine how far across the screen a user has scrolled and see if that is far enough to count as a scroll or whether we should consider it a click. We store the location of the initial <strong>ACTION_DOWN</strong> in the form of x and y coordinates.</p><p>When the user lifts their finger off the screen, the <strong>ACTION_UP</strong> motion event will be called. We can then check to see if the user’s finger has moved more than our given tolerance threshold, in this case, the constant <strong>SCROLL_THRESHOLD </strong>which I have set to 10px. If the absolute distance travelled, <strong>abs()</strong>, in either the x or y direction is less than 10px between the user pressing their finger to the screen and lifting it off again then we detect a click and call our second function <strong>toggleToolbar().</strong></p><p>The final piece of this function is to call <strong>performClick()</strong> and pass the remainder of the event handling to <strong>onTouchEvent()</strong>. The <strong>performClick()</strong> component is important for accessibility reasons.</p><h3>toggleToolbar()</h3><p>We would also like to generalise this function in order to be able to show or hide any given toolbar. As such we will need a way to keep track of whether any given toolbar is currently showing and then a method to animate the movement either onto or off the screen. This looks much nicer than simply calling .show() or .hide().</p><pre>fun toggleToolbar(toolbar: Toolbar) {<br>    if (<em>showingTaskbar</em>.getOrDefault(toolbar, false)) {<br><br>        // Toggle tracker for displaying toolbar<br>        <em>showingTaskbar</em>[toolbar] = false<br><br>        // Extract the height of the action bar<br>        val tv = TypedValue()<br>        var actionBarHeight = 0<br>        if (toolbar.<em>context</em>.<em>theme</em>.resolveAttribute(android.R.attr.<em>actionBarSize</em>, tv, true)) {<br>            actionBarHeight = TypedValue.complexToDimensionPixelSize(tv.data, toolbar.<em>context</em>.<em>resources</em>.<em>displayMetrics</em>)<br>        }<br><br>        // Initiate the animation<br>        toolbar.animate().translationY(-actionBarHeight.toFloat()).setInterpolator(AccelerateInterpolator(2f)).start()<br>    } else {<br>        // Toggle tracker for displaying toolbar<br>        <em>showingTaskbar</em>[toolbar] = true<br><br>        // Initiate the animation<br>        toolbar.animate().translationY(0f).setInterpolator(DecelerateInterpolator(2f)).start()<br>    }<br>}</pre><p>This function accepts a toolbar as a parameter and keeps track of its toolbars, and their current visibility, in a HashMap. We are hiding all our toolbars by default so we first check to see if our HashMap has a value for the toolbar and if not we know it is currently hidden and take action to show the toolbar. This moves the toolbar from above the top of the screen down into its usual position and sets its value in the HashMap.</p><p>When the function is called again, we check our HashMap for a value and know that the toolbar is currently displaying. We then find the size of the Toolbar using the <strong>resolveAttribute </strong>function. For this to work, your toolbars must be sized with the XML <strong>android:layout_height=”?attr/actionBarSize”.</strong></p><p>We then move the toolbar off-screen by the negative of its size, meaning it is sitting just above the visible screen.</p><p>This is all animated by the inbuilt <strong>animate() </strong>function using <strong>translationY()</strong> which simply corresponds to a movement in the Y (vertical) direction. The Interpolators we are using make speed up the movement as the toolbar disappears and slow it down as the toolbar reappears. This looks better than a simple, linear movement in one direction or another.</p><p>This provides all the functionality we need to detect whether a user is clicking the screen or performing some other action and then toggling the toolbar in response. The final thing to do is to call the function with:</p><pre><em>setHidingToolbar</em>(scrollView as View, findViewById(R.id.<em>my_toolbar</em>))</pre><p>and watch it work! The full code is included below for your reference.</p><h3>Full Code</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b52f683e9637c60ac910be0d1671e62/href">https://medium.com/media/6b52f683e9637c60ac910be0d1671e62/href</a></iframe><h3>Conclusion</h3><p>It is easy to detect user actions within Android using Kotlin and the granular actions than can be captured can be assessed to delineate different user intentions. In this case, we have detected when a user is tapping the screen, rather than scrolling or highlighting, and used this to toggle the toolbar in an animated way.</p><p><em>You can follow me on Twitter </em><a href="https://twitter.com/mgrint"><em>@mgrint</em></a><em> or read more of my articles at </em><a href="https://grint.tech/"><em>https://grint.tech</em></a><em>. Email me at </em><a href="/service/mailto:matt@grint.tech"><em>matt@grint.tech</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c2cf0aab7fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>