<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by German Gensetskiy on Medium]]></title>
        <description><![CDATA[Stories by German Gensetskiy on Medium]]></description>
        <link>https://medium.com/@ignis2497?source=rss-1f0336a570e1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mI0jaF5k4xc2vNjKatkcCA.jpeg</url>
            <title>Stories by German Gensetskiy on Medium</title>
            <link>https://medium.com/@ignis2497?source=rss-1f0336a570e1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:06:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ignis2497/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[[Tutorial] How to use SSH tunnel to expose a local server to the internet]]></title>
            <link>https://medium.com/gowombat/tutorial-how-to-use-ssh-tunnel-to-expose-a-local-server-to-the-internet-4e975e1965e5?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e975e1965e5</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ssh]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[servers]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Fri, 07 Feb 2020 08:58:31 GMT</pubDate>
            <atom:updated>2020-02-07T08:58:31.666Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAzoWacpKoytGf99f7Rxcg.jpeg" /></figure><p>We have a local server in our office. And there is no access to that server outside of the local network. We decided to deploy one of our internal tools on the virtual machine inside that local server. And use one of the remotely hosted servers to gain access to that tool through the internet.</p><p>It’s a good and free alternative to tools like Ngrok or Serveo.</p><h3>TL;DR Daemon service for Tunnel</h3><p>On the <strong>local</strong> server. Create a file in the /etc/systemd/system/ with the name liketool-tunnel.service</p><pre>[Unit]<br>Description=AutoSSH tunnel to remote server<br>After=network.target</pre><pre>[Service]<br>Environment=&quot;AUTOSSH_GATETIME=0&quot;<br>ExecStart=/usr/bin/autossh -M 20000 -N -i /home/tool/.ssh/id_rsa user@[IP] -R 8080:localhost:80 -C</pre><pre>[Install]<br>WantedBy=multi-user.target</pre><p>On the <strong>remote</strong> server — add your public SSH key to the authorized_keys</p><h3>Remote server</h3><h4>Giving access</h4><p>To be able to create SSH tunnel to that remote server you need to give your local server SSH access. You need to generate a key (or just use the existing one if you have) using ssh-keygen command. After it is done add you public key from local server to the ~/.ssh/authorized_keys on the remote server.</p><p>You can test if it works by trying to SSH on the remote server from the local server (caution, possible word “server” overdose): ssh username@[IP]</p><h3>Local server</h3><h4>SSH Tunnel</h4><p>Finally, the SSH tunnel itself. Use a command like that:</p><pre>ssh -N -i /home/tool/.ssh/id_rsa user@[IP] -R 8080:localhost:80 -C</pre><p>So, let’s go through all the arguments:</p><ul><li>-N: Do not execute a remote command. It’s useful for just forwarding ports.</li><li>-i /home/tool/.ssh/id_rsa: Which identity (key) to use.</li><li>user@[IP]: user — username on the remote server, IP — ip of the remote server</li><li>-R 8080:localhost:80: This allows anyone on the remote server to connect to TCP port 8080 on the remote server. The connection will then tunneled back to the client host, and the client then makes a TCP connection to port 80 on the localhost.</li><li>-C: Enables data compression</li></ul><p>Now you can try to test it by opening your URL in the browser. Or sending some requests.<em> Don’t forget to launch your application ;)</em></p><h4><a href="https://linux.die.net/man/1/autossh">AutoSSH</a></h4><p>Also, we need to make a tunnel persistent — make sure the tunnel will always run. For example, once your ssh connection times out (By server-side timeout), your tunnel should be re-established automatically.</p><p>There is a tool that will help us — AutoSSH.</p><blockquote>Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.</blockquote><p>The command I come up with is next:</p><pre>autossh -M 20000 -N -i /home/tool/.ssh/id_rsa user@[IP] -R 8080:localhost:80 -C</pre><p>Only things that added are autossh instead of ssh and argument -M 20000, which means that AutoSSH uses port 20000 to check the connection.</p><h4>Service</h4><p>To make sure that the tunnel will be established automatically after server reboot and to have control over the process, we need to set up a systemd service.</p><p>Create a file in the /etc/systemd/system/ with the name liketool-tunnel.service (you can change tool to something more descriptive ;) )</p><pre>[Unit]<br>Description=AutoSSH tunnel to remote server<br>After=network.target</pre><pre>[Service]<br>Environment=&quot;AUTOSSH_GATETIME=0&quot;<br>ExecStart=/usr/bin/autossh -M 20000 -N -i /home/tool/.ssh/id_rsa user@[IP] -R 8080:localhost:80 -C</pre><pre>[Install]<br>WantedBy=multi-user.target</pre><p><em>P.s. Don’t forget to change [IP] in the command.</em></p><p>After the file it’s done, run:</p><pre>sudo systemctl daemon-reload<br>sudo systemctl start tool-tunnel  # To start your service<br>sudo systemctl enable tool-tunnel  # To make it start automatically </pre><p>Now you can check it with sudo systemctl status tool-tunnel.</p><h4>Done!</h4><h3>If you don’t want to use Nginx</h3><p>By default, OpenSSH only allows connecting to remote forwarded ports from the server host. However, you can change that behavior.</p><p>For that, you need to change the GatewayPorts option in the server configuration file <a href="https://www.ssh.com/ssh/sshd_config/"><strong>sshd_config</strong></a><strong> </strong>(Usually this file is /etc/ssh/sshd_config). Simply cahnge it to yes:</p><pre>GatewayPorts yes</pre><h3>If you want to use Nginx</h3><h4>Remote server</h4><p>First, we need to prepare the remote server to handle requests and redirect them to port that used in the SSH tunnel. We used <a href="https://nginx.org/">Nginx</a> to set it up.</p><p>In /etc/nginx/sites-available/ you need to create a file that contains config for your site.</p><p>The contents are pretty simple:</p><pre>server {<br>    listen 80;<br>    server_name tool.gowombat.team;<br>    access_log /var/log/nginx/tool-proxy.log;<br>    error_log /var/log/nginx/tool-proxy.error.log;</pre><pre>client_max_body_size 32m;</pre><pre>location / {<br>        proxy_pass <a href="http://127.0.0.1:8080/">http://127.0.0.1:8080</a>;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Real-IP $remote_addr;<br>    }<br>}</pre><p>After you done it — you need to add it to /etc/nginx/sites-enabled/, best way — using symlink:</p><pre>ln -s /etc/nginx/sites-available/tool /etc/nginx/sites-enabled/</pre><h4>Local server</h4><p>On the local server, we’re used Nginx as well to serve applications and static files. But if you have something simple or something that doesn’t need static files, you can skip that part.</p><p>The config file needs to be put in the same directory as before.</p><pre>server {    <br>    listen 80;<br>    server_name tool.gowombat.team;<br>    access_log /var/log/nginx/tool.log;<br>    error_log /var/log/nginx/tool.error.log;</pre><pre>client_max_body_size 32m;</pre><pre>location /static/ {<br>        alias /home/tool/tool-backend/collected_static/;<br>    }</pre><pre>location / {<br>        include proxy_params;<br>        proxy_pass <a href="http://unix/home/aht/aht-backend/aht_backend.sock">http://unix:/home/tool/tool-backend/tool_backend.sock</a>;<br>    }</pre><pre>}</pre><p>Thanks for reading! I hope you liked it.</p><p>German Gensetskiy under the support of the “<a href="https://gowombat.team/">Go Wombat Team</a>”.</p><p>Feel free to contact me if you have any questions: Ignis2497@gmail.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e975e1965e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/tutorial-how-to-use-ssh-tunnel-to-expose-a-local-server-to-the-internet-4e975e1965e5">[Tutorial] How to use SSH tunnel to expose a local server to the internet</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-hosted Open-Source Cheap alternative for Google Maps API services]]></title>
            <link>https://medium.com/gowombat/self-hosted-open-source-cheap-alternative-for-google-maps-api-services-9a2c2002952b?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a2c2002952b</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[alternative]]></category>
            <category><![CDATA[cheap]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[google-maps]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Thu, 19 Dec 2019 10:02:15 GMT</pubDate>
            <atom:updated>2019-12-19T13:11:03.745Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Jyh5BJgR9Qj8iSQq" /></figure><p>When you need a map to embed it on your site, or you need to make an address autocomplete — the first thing you probably think about is Google Maps.<br>While this is the most straightforward and perfect solution, it can be quite expensive. And we’re, at Go Wombat Team, always trying to find cost-effective solutions for our clients.</p><p>There is an alternative called OpenStreetMap. It will require more effort to set up, but you will have full control over it, and the price you need to pay — is only the price for the server. For applications with a big load, it can be a great benefit. But let’s investigate.</p><h3>Price comparison</h3><h4>Google Maps API</h4><p>Google maps geocoding API costs 5$ per 1000 requests until it’s 100,000 requests, then 4$ until 500,000. So let’s say we have a maximum of 500,000 requests per month — that will cost $2100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*QEVU6C2zQwmTXbjSuRrbIg.png" /><figcaption>Google Maps Services price calculator</figcaption></figure><h4>DigitalOcean Server</h4><p>The cheapest server on Digital Ocean will cost $5 per month. But since we have 500,000 requests per month — probably it will be not enough. So an average server with 8 GB RAM, 4 CPUs, and 160 GB SSD disk — will cost $40. And even the top high-end server will cost $960. It will have 192 GB, 32 CPUs, and 3.75 TB SSD disk!</p><h3>Tools to use</h3><p>We used <a href="http://photon.komoot.de/">Photon</a> for search-as-you-type with multilingual support and geocoding. Photon is an open-source geocoder built for <a href="http://www.osm.org/">OpenStreetMap</a> data. It is based on <a href="http://elasticsearch.org/">elasticsearch</a> — an efficient, robust, and highly scalable search platform.</p><p>Alternatively, you can use <a href="https://wiki.openstreetmap.org/wiki/Nominatim">Nominatim</a> for geocoding as well. But we found Photon simpler to set-up and use.</p><h3>OSM Tile Server</h3><p>So, if you want to go fully self-hosted, you will also need to render a map on your service. For that purpose, you will need to set-up an <a href="https://wiki.openstreetmap.org/wiki/Tile_servers">OSM Tile Server</a>.</p><p>The main <strong>problem</strong> here, that’s from the different sources — the data for Tile Server may take up to 3 TB of space. And Volume on DigitalOcean for 3 TB will cost $300 per month.</p><h3>Cheapest solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*FdpLpw2jbP6dWeYjGjyI5A.png" /><figcaption>Pricing for Google Maps Embed API</figcaption></figure><p>Google Maps Embed is free so that you can use it as a map provider. And set-up a Photon for geocoding and autocomplete.</p><p>Thanks for reading! I hope you liked it.</p><p>German Gensetskiy under the support of the “<a href="https://gowombat.team/">Go Wombat Team</a>”.</p><p>Feel free to contact me if you have any questions: Ignis2497@gmail.com</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a2c2002952b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/self-hosted-open-source-cheap-alternative-for-google-maps-api-services-9a2c2002952b">Self-hosted Open-Source Cheap alternative for Google Maps API services</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to properly use serverless technologies and AWS Lambdas in our opinion]]></title>
            <link>https://medium.com/gowombat/how-to-properly-use-serverless-technologies-and-aws-lambdas-in-our-opinion-960152510966?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/960152510966</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[serverless]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Fri, 27 Sep 2019 11:22:05 GMT</pubDate>
            <atom:updated>2019-09-27T11:22:05.594Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MCKcELa-XKi0uOaR.png" /></figure><p>Recently I read and shared with team a topic about serverless technologies usage in terms of web API: <a href="http://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/">http://einaregilsson.com/serverless-15-percent-slower-and-eight-times-more-expensive/</a></p><p>In short — the author did research to decide if he needs to use serverless technologies for his API instead of the old setup. He found out that Lambda is a little bit slower than EC2 Instance, but that’s not the main problem. The main problem was in the pricing, and actually it was in the price for API Gateway — not the Lambda itself.</p><p>I liked the research the author did and his approach. But it reminded us of one of our successful usage of lambdas.</p><p>Lambda function is a great asset for small independent functions (that’s all in the name). And especially it’s good when you need to use different APIs or scrap some web-pages. In that case, you can invoke as many lambdas as you need in parallel. Starting from 1 instance to a million without changing the infrastructure code. It just works and that’s it.</p><p>Each lambda will be launched on separate VPS with its IP address — which makes it hard to block your scrapper. <br>Because the only way to block it will be banning the whole pull of the AWS IP addresses, which is probably a bad decision.</p><h3>Example</h3><p>Words are cool, but what about some code example? Sure! Here we go.</p><p>In our example, we will work with cryptocurrency exchange markets to retrieve the latest info about the rate of exchange between two cryptocurrencies.</p><p>We’ll use <a href="https://github.com/ccxt/ccxt/wiki/Manual#exchanges">CCXT API</a> for that, it supports 127 different cryptocurrency exchange markets and has a Python implementation.</p><p>First of all, we need to create a simple lambda function. Its responsibility is to retrieve data about specified tickers from the specified market.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/231104201d2a2c49537b64f611f90146/href">https://medium.com/media/231104201d2a2c49537b64f611f90146/href</a></iframe><p>When Lambda is ready — the next step is to run it. We will retrieve data from many sources and we need to do it as fast as possible because things can change very fast. For that purpose, we will make our code asynchronous and will invoke lambdas in parallel.</p><p>To invoke AWS Lambda we need to use Boto — AWS SDK for Python. In our case, the asynchronous version of Boto — aioboto3.</p><pre>client = aioboto3.client(<br>    <strong>&#39;lambda&#39;</strong>,<br>    region_name=<strong>&#39;[YOUR_LAMBDA_REGION]&#39;</strong>,<br>    aws_access_key_id=<strong>&#39;[YOUR_AWS_KEY_ID]&#39;</strong>,<br>    aws_secret_access_key=<strong>&#39;[YOUR_AWS_KEY]&#39;<br></strong>)</pre><pre>response = <strong>await</strong> client.invoke(<br>    FunctionName=<strong>&#39;[YOUR_LAMBDA_FUNCTION_NAME_OR_ARN]&#39;</strong>,<br>    InvocationType=<strong>&#39;RequestResponse&#39;</strong>,<br>    LogType=<strong>&#39;Tail&#39;</strong>,<br>    Payload=json.dumps(<strong>[PAYLOAD]</strong>),<br>)</pre><p>Here is a simple example of how to invoke a lambda, you can always find more info in the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke">Boto3 docs</a>.</p><p>And here is the code of the runner I ended up with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efc98a9e3c31b0bb4827e4e03275eb9c/href">https://medium.com/media/efc98a9e3c31b0bb4827e4e03275eb9c/href</a></iframe><p>The main logic here goes in methods __invoke_lambda and invoke.</p><p>__invoke_lambda uses code same as above to run lambda and retrieve its response. It’s an asynchronous method.</p><p>invoke maps __invoke_lambda coroutine to each payload, and <a href="https://docs.python.org/3/library/asyncio-task.html#asyncio.gather">asyncio.</a><a href="https://docs.python.org/3/library/asyncio-task.html#asyncio.gather"><strong>gather</strong></a><strong> </strong>schedules them as parallel tasks.</p><h4>Bonus</h4><p>If you don’t want to write or use asynchronous code — here is the same using the Python multiprocessing module:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/740f29fbefd283921dd7baea65728d21/href">https://medium.com/media/740f29fbefd283921dd7baea65728d21/href</a></iframe><p>Thanks for reading! Hope you liked it.</p><p>German Gensetskiy under the support of <a href="https://gowombat.team/">Go Wombat Team</a>.</p><p>P.S. In case of any troubles I will be happy to hear your feedback. You can reach me out by the email: <a href="/service/mailto:Ignis2497@gmail.com">Ignis2497@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=960152510966" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/how-to-properly-use-serverless-technologies-and-aws-lambdas-in-our-opinion-960152510966">How to properly use serverless technologies and AWS Lambdas in our opinion</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A simple example of using Spark in Databricks with Python and PySpark.]]></title>
            <link>https://medium.com/gowombat/a-simple-example-of-using-spark-in-databricks-with-python-and-pyspark-90c2754775b8?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/90c2754775b8</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Tue, 28 May 2019 12:31:05 GMT</pubDate>
            <atom:updated>2020-02-07T09:02:54.018Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*v_V9-ox4m485jGjC.png" /></figure><blockquote>Fresh new tutorial: <a href="https://medium.com/gowombat/tutorial-how-to-use-ssh-tunnel-to-expose-a-local-server-to-the-internet-4e975e1965e5">A free alternative to tools like Ngrok and Serveo</a></blockquote><p><a href="https://spark.apache.org/"><strong>Apache Spark</strong></a> is an open-source distributed general-purpose <a href="https://en.wikipedia.org/wiki/Cluster_computing">cluster-computing</a> <a href="https://en.wikipedia.org/wiki/Software_framework">framework</a>. And setting up a cluster using just bare metal machines can be quite complicated and expensive. Therefore cloud solutions are coming to help us. It’s really useful especially when you’re just a developer who needs to do some experiments with data.</p><p><a href="https://databricks.com/"><strong>Databricks</strong></a> provides a very fast and simple way to set up and use a cluster. It’s also has a <a href="https://community.cloud.databricks.com/">community</a> version that you can use for free (that’s the one I will use in this tutorial). There is also <a href="https://azure.microsoft.com/services/databricks/">Azure Databricks</a> and <a href="https://databricks.com/aws">AWS Databricks</a> for usage besides development.</p><p>Create an account and let’s begin.</p><h3>Setting up the cluster</h3><p>So, as I said, setting up a cluster in Databricks is easy as heck. Just click “New Cluster” on the home page or open “Clusters” tab in the sidebar and click “Create Cluster”. You will see a form where you need to choose a name for your cluster and some other settings. For this tutorial — all of the settings except for name you can leave with default values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*VBaQEXWZi2IFLoGgYnfQHg.png" /><figcaption>Example of cluster creation</figcaption></figure><h3>Preparing the data</h3><p>For this example, I chose the data from the Google Play Store:</p><p><a href="https://www.kaggle.com/lava18/google-play-store-apps">Google Play Store Apps</a></p><p>After downloading CSV with the data from Kaggle you need to upload it to the <a href="https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html">DBFS (Databricks File System)</a>. When you uploaded the file, Databricks will offer you to “Create Table in Notebook”. Let’s accept the proposal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*4IJ-ded6AGjK6h1aEYP0xw.png" /><figcaption>Example of uploading data to DBFS</figcaption></figure><p>You will end up with a notebook like that. Note: for that dataset, you’ll need to set first_row_is_header = “true”, it is false by default.</p><p><a href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/833936658545166/1841095518689174/880569066821071/latest.html">2019-05-27 - DBFS Example - Databricks</a></p><h3>Data preparation</h3><p>When reading CSV files into dataframes, Spark performs the operation in an eager mode, meaning that all of the data is loaded into memory before the next step begins execution, while a lazy approach is used when reading files in the <a href="https://parquet.apache.org/">parquet</a> format. Generally, you want to avoid eager operations when working with Spark.</p><p>To do that, we first need to remove spaces from columns names. I decided to convert it to snake case:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bcbbcf3b666969264923ffea06f929e/href">https://medium.com/media/1bcbbcf3b666969264923ffea06f929e/href</a></iframe><p>After converting the names we can save our dataframe to Databricks table:</p><pre>df.write.format(&quot;parquet&quot;).saveAsTable(TABLE_NAME)</pre><p>To load that table to dataframe then, use read.table:</p><pre>df = spark.read.table(TABLE_NAME)</pre><h3>Processing the data</h3><p>Let’s find an average rating for each category and try to understand which apps customers are happy to use.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fignisor%2Fembed%2Fpreview%2FrgKKKB%3Fheight%3D600%26slug-hash%3DrgKKKB%26default-tabs%3Dhtml%2Cresult%26host%3Dhttps%3A%2F%2Fcodepen.io&amp;url=https%3A%2F%2Fcodepen.io%2Fignisor%2Fpen%2FrgKKKB&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fm.cdpn.io%2Fscreenshot-coming-soon-small.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e39fb8700616bd5f459824abc4c2aa5/href">https://medium.com/media/7e39fb8700616bd5f459824abc4c2aa5/href</a></iframe><p>Don’t worry, I will describe all we have done above now.</p><p>UDF (@udf(‘[output type]’) decorator) — User defined functions. PySpark UDFs work in a way similar to the pandas’ .map() and .apply(). The only difference is that with PySpark UDF you have to specify the output data type. All the types supported by PySpark <a href="http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=types#module-pyspark.sql.types">can be found here</a>.</p><p>rename_category function — that’s a simple function to rename categories to a little bit more human-readable names. Wrapped as UDF function.</p><p>display — databricks’ helper to simply display dataframe as a table or plot a graph of it.</p><p>select(rename_category(‘category’).alias(‘category’), ‘rating’) — same as in SQL selects columns you specify from the data table. Here we’re selecting columns category and rating from our table. Also, we use our UDF rename_category to rename our categories. alias is used on the renamed result to make the column have the same name.</p><p>where(df.rating != ‘NaN’) — filters out rows in which rating is NaN.</p><p>groupBy(‘category’) — grouping as in SQL query, to aggregate data based on the groups. So we grouping ratings of the apps by their category, so then we will be able to find an average rating for each category.</p><p>agg(avg(‘rating’).alias(‘avg_rating’)) — apply aggregation on the grouped data. avg — calculate an average. alias is used in the same way, to give a column with aggregated data new name. By default, it would be avg(rating).</p><p>where(col(‘avg_rating’).isNotNull() &amp; (col(‘avg_rating’) &lt;= 5.0)) — filter out rows with the Null rating, or rating bigger than 5 (because there are only 5 stars in the Google Play Store). That is required because dataset has some invalid data.</p><p>sort(‘avg_rating’) — simply sort data by average rating.</p><p>As a result, we will be able to see a bar chart in the databricks, with average rating per category.</p><p>After getting an average rating, I decided to check an average downloads amount for each category as well. There is not a big difference in data processing, but one tricky part.</p><p>Because Google Play Store does not show the actual amount of downloads, but only shows the amount in a format like 10,000+ (more than 10,000 downloads). We can’t just simply apply aggregation on installs column since it’s a string.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fignisor%2Fembed%2Fpreview%2FRmBobr%3Fheight%3D600%26slug-hash%3DRmBobr%26default-tabs%3Dhtml%2Cresult%26host%3Dhttps%3A%2F%2Fcodepen.io&amp;url=https%3A%2F%2Fcodepen.io%2Fignisor%2Fpen%2FRmBobr&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fm.cdpn.io%2Fscreenshot-coming-soon-small.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fbb0055cf778b5537f0ff21679c0de4/href">https://medium.com/media/3fbb0055cf778b5537f0ff21679c0de4/href</a></iframe><p>So, to convert installs from string format to the integers we can use UDF. A simple function convert_installs that removes symbols+, , and converts it to an integer. If the resulting number is not an integer simply returns None.</p><h3>Results</h3><p>The resulting notebook I got you can find here:</p><p><a href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/833936658545166/3886044108360537/880569066821071/latest.html">Google Play Store Databricks example - Databricks</a></p><p>Thanks for reading! Hope you liked it.</p><p>German Gensetskiy under the support of <a href="https://gowombat.team/">Go Wombat Team</a>.</p><p>P.S. In case of any troubles I will be happy to hear your feedback. You can reach me out by the email: <a href="/service/mailto:Ignis2497@gmail.com">Ignis2497@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90c2754775b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/a-simple-example-of-using-spark-in-databricks-with-python-and-pyspark-90c2754775b8">A simple example of using Spark in Databricks with Python and PySpark.</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First impressions about Uber’s Ludwig. A simple machine learning tool. Or not?]]></title>
            <link>https://medium.com/gowombat/first-impressions-about-ubers-ludwig-a-simple-machine-learning-tool-or-not-714962bbbedc?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/714962bbbedc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ludwig]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[uber]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Fri, 03 May 2019 08:16:06 GMT</pubDate>
            <atom:updated>2019-05-03T08:22:09.458Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/993/0*mPhq7kewR_pkwLzw.png" /><figcaption>Ludwig logo</figcaption></figure><p>Not so long ago Uber presented a promising tool intended to simplify work with deep machine learning algorithms.</p><blockquote><a href="https://uber.github.io/ludwig/">Ludwig</a> is a toolbox that allows to train and test deep learning models without the need to write code.</blockquote><p>And so, we were very interested to check that out. Is it a machine that will take our jobs or a useful tool that we can work with.</p><p>The interesting part — is that Ludwig provides a Python API, so it must be simple to integrate the model with your applications.</p><h3>Task</h3><p>As a simple task for our experiment, we decided to do an image classification — is it a person on the image or not. For a better understanding of simplicity (or complexity) of the tool, we asked some of our teammates, that never used to work with deep machine learning before, to do the same.</p><h3>Data</h3><p>Well, despite the fact that Ludwig must make your work with deep learning algorithms simple — you still need to find and prepare some data to work with.</p><p>So, I started to look for some open datasets with people on it. I found two that seemed suitable to me. A <a href="http://vision.stanford.edu/Datasets/40actions.html">“Stanford 40 Actions”</a> and <a href="http://pascal.inrialpes.fr/data/human/">“INRIA Person Dataset”</a>. After looking through them I decided to use <a href="http://vision.stanford.edu/Datasets/40actions.html">Stanford’s</a> dataset as positive (where the person present) images and for negative one — <a href="http://pascal.inrialpes.fr/data/human/">INRIA</a> part with negative (where are no persons) images.</p><p>Ludwig works with CSV annotations. For images, you just need to put a path in your CSV file and specify type: image in the model config file. To annotate my dataset I used a simple python script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ce68e80c4e2f5db9594729306111fca/href">https://medium.com/media/0ce68e80c4e2f5db9594729306111fca/href</a></iframe><h3>Setting up “Ludwig”</h3><p>Well, that part is mostly pretty simple. I just started to go through <a href="https://uber.github.io/ludwig/getting_started/">“Getting started”</a> part of the official Ludwig’s documentation. Actually, if you just want to have a simple CPU enabled the version of Ludwig in your environment, it’s all about pip install ludwig.</p><p>But, if you want to have a GPU enabled one you will need to clone Ludwig from GitHub repository and change tensorflow to tensorflow-gpu in requirements.txt and in setup.py.</p><p>It actually would be pretty useful if there was a package ludwig-gpu. And I decided — why not to make a simple one for myself. Since pip allows us to install packages just from git repository very easy, I simply forked the repo and changed the requirements. After that, all I need is to do pip install git+<a href="https://github.com/Ignisor/ludwig-gpu.git">https://github.com/Ignisor/ludwig-gpu.git</a>.</p><p>And of course, if you’re planning to use GPU you need to install all of the <a href="https://www.tensorflow.org/">Tensorflow</a> requirements. It’s all described <a href="https://www.tensorflow.org/install/gpu#software_requirements">here</a>.</p><h3>Model parameters</h3><p>Ludwig works with model parameters in YAML format. There is a plenty amount of <a href="https://uber.github.io/ludwig/examples/">examples for most of the use cases</a>. I decided to keep it simple and just use an example provided for image classification.</p><pre><strong>input_features</strong>:<br>    -<br>        <strong>name</strong>: image_path<br>        <strong>type</strong>: image<br>        <strong>encoder</strong>: stacked_cnn<br><br><strong>output_features</strong>:<br>    -<br>        <strong>name</strong>: class<br>        <strong>type</strong>: category</pre><p>But when I started to train I got this error:</p><pre>ValueError: could not broadcast input array from shape (424,300,3) into shape (300,354,3)</pre><p>Not really descriptive one as for a user that not familiar with tensors. That one happens because images have different resolution. Fortunately, Ludwig, has some preprocessing built in:</p><pre><strong>input_features</strong>:<br>    -<br>        <strong>name</strong>: image_path<br>        <strong>type</strong>: image<br>        <strong>encoder</strong>: stacked_cnn<br>        <strong>preprocessing</strong>:<br>            <strong>resize_method</strong>: interpolate<br>            <strong>height</strong>: 400<br>            <strong>width</strong>: 300<br><br><strong>output_features</strong>:<br>    -<br>        <strong>name</strong>: class<br>        <strong>type</strong>: category</pre><p>But even now, one more problem happened to me. When I start training — My PC just freezes. The problem for me was that default batch size is 128 and my machine quickly ran out of memory. And the fix is simple:</p><pre><strong>input_features</strong>:<br>    -<br>        <strong>name</strong>: image_path<br>        <strong>type</strong>: image<br>        <strong>encoder</strong>: stacked_cnn<br>        <strong>preprocessing</strong>:<br>            <strong>resize_method</strong>: interpolate<br>            <strong>height</strong>: 400<br>            <strong>width</strong>: 300<br><br><strong>output_features</strong>:<br>    -<br>        <strong>name</strong>: class<br>        <strong>type</strong>: category<br><br><strong>training</strong>:<br>    <strong>batch_size</strong>: 16</pre><h3>Training</h3><p>To start training your model — all you need is just one command:</p><pre>ludwig experiment -mdf model_definition.yaml -g 0 --data_csv humans_data.csv</pre><p>About parameters:</p><ul><li>-mdf — Model definition file. YAML file describing the model.</li><li>-g — List of GPUs to use.</li><li>--data_csv — path to file with CSV that describes the data.</li></ul><p>Why it’s experiment instead of train? This command combines training and prediction into a single handy command. It will train the model and then check the accuracy.</p><h3>Example and visual check using Python API</h3><p>To present results you can use Jupyter Notebook or a simple python script. Using a Ludwig’s Python API.</p><p>API is quite simple to use:</p><pre>model = LudwigModel.load(&#39;model/&#39;)<br>result = model.predict(data_csv=&#39;predict.csv&#39;)</pre><p>But there was one trouble. Preprocessing was not working because of some minor bug. So, it’s open-source, why not to fix it ;)</p><p><a href="https://github.com/uber/ludwig/pull/300">Resizing image before switching channels by Ignisor · Pull Request #300 · uber/ludwig</a></p><p>Here is what I’ve got, not perfect, but good enough to understand how to work with Ludwig.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1720ba297c8a21b31df926664c8e8150/href">https://medium.com/media/1720ba297c8a21b31df926664c8e8150/href</a></iframe><p>Thanks for reading! Hope you liked it.</p><p>German Gensetskiy under the support of <a href="https://gowombat.team/">Go Wombat Team</a>.</p><p>P.S. In case of any troubles I will be happy to hear your feedback. You can reach me out by the email: <a href="/service/mailto:Ignis2497@gmail.com">Ignis2497@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=714962bbbedc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/first-impressions-about-ubers-ludwig-a-simple-machine-learning-tool-or-not-714962bbbedc">First impressions about Uber’s Ludwig. A simple machine learning tool. Or not?</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[IoT] Simple IoT with free cloud solutions (tutorial)]]></title>
            <link>https://medium.com/gowombat/iot-simple-iot-with-free-cloud-solutions-tutorial-76ac5cf6e5a0?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/76ac5cf6e5a0</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Sun, 24 Feb 2019 10:49:00 GMT</pubDate>
            <atom:updated>2020-04-13T06:55:12.993Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*jXU2WqzZwQX6-1KC.png" /></figure><h3>Before we start</h3><p>The resulting dashboard can be found <a href="http://freeboard-cucumbers.s3-website.eu-north-1.amazonaws.com/#source=cucumbers.json"><strong>here</strong></a><strong> ;)</strong></p><h3>Introduction</h3><p>I love automation. And I always wanted to automatize the watering of my plants (because I always forget about them). And yes, there are plenty of different turnkey solutions, but that’s not a developer way.</p><p>As a first step, I decided to implement the monitoring of soil moisture. But to monitor some value — I need to see that value somehow. And also important to see the historical change of that value. Therefore I need some dashboard and server to host it and some data storage. And that’s cost money. But I don’t need the full power and storage of even the smallest and cheapest server on Digital Ocean for example. Ideally, I don’t want to spend money each month at all for that. So, why not to just use a free plan of plenty of different cloud solutions for everything?</p><p>And I was able to achieve that and now I’m ready to share it with you.</p><h3>Weapon of choice</h3><p><strong>MQTT broker — </strong><a href="https://www.cloudmqtt.com/"><strong>CloudMQTT</strong></a><br>Very straight and really simple to set up cloud MQTT solution. Has plenty of plans for different purposes and in case you will need to scale. Also has a free plan named “Cute Cat” that allows you to have 5 devices and supports 10 Kbit/s. Pretty enough for an experiment or PoC.</p><p>[Update] Currently CloudMQTT says that free plan is “out of stock”. You can <a href="https://elements.heroku.com/addons/cloudmqtt">try it on Heroku</a> or use an alternative like <a href="https://myqtthub.com/en">MyQttHub</a>.</p><p><strong>Backend and Database — </strong><a href="https://www.back4app.com/"><strong>Back4App</strong></a><strong><br></strong>Backend as a Service with a pretty user-friendly interface. Provides you a storage and REST API. It will give you 10 requests per second and 10 000 requests per month. It gives you around 13 requests per hour. So be careful with testing ;)</p><p><strong>Dashboard — </strong><a href="https://freeboard.io/"><strong>Freeboard</strong></a><strong><br></strong>Simple but powerful dashboard. It provides you a cloud solution but there are no free plans. But Freeboard is open-source so we can host it by ourselves. Since it will be self-hosted — no limitations on what you can do.</p><p><strong>Server — </strong><a href="https://www.heroku.com/"><strong>Heroku</strong></a><strong><br></strong>Popular Platform as a Service. Has a free plan and gives you the possibility to scale up later with what you need. Great way to host something simple like freeboard.</p><h3>Structure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5nbIrp_kgHMu0CfjzcDwA.png" /><figcaption>Project structure</figcaption></figure><p>So, the device with a sensor (soil moisture sensor in our case) will publish its measurements using the MQTT protocol. MQTT to HTTP server will listen to the topic where the measurements are published and will resend them through HTTP API to the Back4App. Back4App will store all the history of measurements and provide us an endpoint to retrieve that data. Freeboard will use that API to load the last 50 measurements and build sparkline based on it.</p><p>Since our device will work with MQTT and Back4App has only REST API, we need one more layer between them — a simple MQTT to HTTP translator that will listen our MQTT topic with sensors data and send it to the Back4App API.</p><p><em>In that particular case using MQTT is not required, the device simply can send an HTTP request directly to Back4App. But it will be simpler to add more devices and implement communication between devices using MQTT broker.</em></p><h3>First step: MQTT broker</h3><p>Setting it up using <a href="https://www.cloudmqtt.com/">CloudMQTT</a> is as simple as possible. Go to the web page, click on the button <em>“Get a managed IoT broker today”. </em>Choose a free plan <em>“Cute Cat”</em> and Sign Up. After that, you will need to choose a name and region for your project and then you will have all the credentials you need to work with it.</p><p>Here is a small example of how to connect and publish a message using MicroPython:</p><pre><strong>import </strong>machine<br><strong>from </strong>umqtt.simple <strong>import </strong>MQTTClient<br></pre><pre>mqtt = MQTTClient(<br>    ubinascii.hexlify(machine.unique_id()), <br>    <strong>&quot;m24.cloudmqtt.com&quot;</strong>, <br>    port=27730, <br>    user=<strong>&quot;[username]&quot;</strong>, <br>    password=<strong>&quot;[password]&quot;</strong>, <br>    ssl=<strong>True<br></strong>)</pre><pre>mqtt.publish(<strong>&#39;sensors/moisture&#39;</strong>, str(moisture_value).encode())</pre><h3>Second step: Backend</h3><p><a href="https://www.back4app.com/">Back4App</a> gives you a way to rapidly set up your database and start working with it. Just sign up, click on Build new app and choose a name. After that, you will see a dashboard and a small tutorial will be started. When you pass the tutorial you can create a new clear App or use that one if you want.</p><p>In the App, you’ll need to create a new class named moisture (actually you can name it whatever you want but in next steps, I will use that name).</p><p>By default, any class you create has the next columns: objectId, updatedAt, createdAt and ACL. It’s useful for as to have a column so we don’t need to take care of the time of the measurement.</p><p>We need to add one more column named value with type Number to store the actual value of soil moisture. And we’re mostly done after that!</p><h4>Security (optional)</h4><p>It’s not so important for some personal experiment to set up the security, so you can skip that part if you don’t want to make your dashboard public to show it to someone.</p><p>So, if you don’t want a stranger to be able using your API key from Freeboard (where you can’t hide it) to mess with your data (e.g. deleting everything just because he can). You’ll need to restrict write rights for the moisture table and create a User that will have write privileges.</p><p>The class for the users already exists in App by default. So we need to create a new record here. Click on “Add a row” and fill out next fields: email, password, username, emailVerified. Just like that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzsV3UVb6sgIiBL6ZhUy_A.png" /><figcaption>Example of user creation</figcaption></figure><p>After filling out all of the required fields with valid values object must be created automatically and objectId will be assigned to it.</p><p>When the user created you need to set up security on the moisture table. Click on the icon with shield and in the dialog window remove the checkmark from theWrite column, in the field below enter the username and press enter. The checkmark on the Write field will be already set and the user’s ID will be shown instead of the user’s name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/1*4iLGOoMWiQOOP25u6O2JtQ.png" /><figcaption>Example of permissions settings</figcaption></figure><p>The last step is to Log In with the created user to retrieve its session token. Just send the GET request with username and password of the user specified in parameters. From the response, you will need a sessionToken. You can find an example <a href="https://dashboard.back4app.com/apidocs/KyjOOhzG63RIQMnFzg2O5galmvCGJhcfivlfAzKC#logging-in">here</a>.</p><h3>Third step: MQTT to HTTP bridge</h3><p>For that part, I decided to make a simple python script using <a href="https://pypi.org/project/paho-mqtt/">Paho MQTT</a> and <a href="https://pypi.org/project/requests/">Requests</a> libraries. It’s simply listening to thesensors/moisture topic and pushes each value that is published there.</p><p>Let’s begin with object creation with Back4App API. Using requests it’s super simple:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FBack4App-Medium-example%3Flite%3Dtrue&amp;url=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FBack4App-Medium-example%3Flite%3Dtrue&amp;image=https%3A%2F%2Frepl.it%2Fpublic%2Fimages%2Freplit-logo-800x600.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=repl" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a8fb7fa35fedb6dcb79c205ae4c2f8a/href">https://medium.com/media/9a8fb7fa35fedb6dcb79c205ae4c2f8a/href</a></iframe><p><em>All of the keys can be found in the Back4App dashboard, in the </em><em>App Settings, </em><em>Security &amp; Keys.</em></p><p>After we ready to create new objects the next step is to subscribe to our MQTT topic with measurements data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FPaho-MQTT-Medium-example%3Flite%3Dtrue&amp;url=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FPaho-MQTT-Medium-example%3Flite%3Dtrue&amp;image=https%3A%2F%2Frepl.it%2Fpublic%2Fimages%2Freplit-logo-800x600.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=repl" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf4d8ea4c9bc52e9b2cf1c6fdf86537c/href">https://medium.com/media/bf4d8ea4c9bc52e9b2cf1c6fdf86537c/href</a></iframe><p>In the example above when any new message will be published to the topic sensors/moisture the function on_message will be called. You can try it out by publishing messages using WebSocket UI in your CloudMQTT console.</p><p>So, after we’ve figured out how to work with both parts all we need is to add an object creation to the on_message function:</p><pre><strong>def </strong>on_message(client, obj, msg):<br>    print(<strong>f&#39;Received {msg.topic} (QoS: {msg.qos}): {msg.payload}&#39;</strong>)<br>    create_object(msg.payload.decode())</pre><h4>Deployment</h4><p>Last part of that step is to deploy our small app on Heroku. First of all, you need to create an account on Heroku (if you don’t have one already). After that simply create a new app, chose a name and region for it.</p><p>Also, we will need to specify a Procfile for Heroku so it will know how too run our application. Simply create a file named Procfile (no extension needed) and put it there:</p><pre>server: python -u run.py</pre><p>After it’s done we’re ready to deploy our app:</p><ul><li>If you haven’t already, log in to your Heroku account and follow the prompts to create a new SSH public key.<br>$ heroku login</li><li>Initialize a git repository in the folder with your app<br>$ git init</li><li>Add Heroku’s remote<br>$ heroku git:remote -a app-name</li><li>Deploy your changes<br>$ git add .<br>$ git commit -am “initial commit”<br>$ git push heroku master</li></ul><p><em>Basically, almost the same instructions can be found in the “Deploy” tab of your app dashboard.</em></p><p>After deployment is done you can test it using WebSocket UI in your CloudMQTT console.</p><h3>Fourth step: Dashboard</h3><p>We will use <a href="https://freeboard.io/">Freeboard</a> to simply create a dashboard for our small system. First of all, clone it from <a href="https://github.com/Freeboard/freeboard">Freeboard’s GitHub</a>. Then from the folder Git just created run npm install and then run grunt.</p><p>Unfortunately, for our setup, we can’t simply use Freeboard as is. That’s because of the default sparkline widget in it designed to work with real-time data, but we need to draw a historical data from our database.</p><p>Therefore we will need to make a simple plugin with sparkline that able to show an array of historical data. You can find my simple implementation of it in <a href="https://github.com/Ignisor/freeboard-heroku/blob/master/plugins/historic.sparkline.js">GitHub</a>. You can simply download a file, add it to the folder plugins in your Freeboard clone and then add the path to the plugin in the index.html :</p><pre>&lt;<strong>script type=&quot;text/javascript&quot;</strong>&gt;<br>    head.<strong>js</strong>(<strong>&quot;js/freeboard_plugins.min.js&quot;</strong>,<br>            <strong>&quot;plugins/historic.sparkline.js&quot;</strong>,  <em>// &lt; Right there<br>            // *** Load more plugins here ***</em></pre><h4>Set up your dashboard</h4><p><strong>Important: </strong>don’t refresh the page because all of the changes exist only in your browser until you save them.</p><p>Now you can simply open index.html using your browser. There you will find an empty and ready to go Freeboard interface. You can play with it as you want, while I will describe to you how to prepare it for our system.</p><p>And the first step here is to add a data source. In the “Datasources” panel click on ADD. In the pop up select a type JSON and choose the name you like. In the URL field specify Back4App endpoint to retrieve moisture.</p><pre><a href="https://parseapi.back4app.com/classes/moisture?limit=50&amp;order=-createdAt">https://parseapi.back4app.com/classes/moisture?limit=50&amp;order=-createdAt</a></pre><p>I also added here parameters to retrieve only the last 50 records. <br>order=-createdAt — order rows by creation date in reversed order (the last one will go first).<br>limit=50 — retrieve only 50 rows.</p><p>Method must be GET and two headers need to be added:<br>X-Parse-Application-Id: [Application ID key for Back4App]<br>X-Parse-JavaScript-Key: [JavaScript key for Back4App]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*LJdxiqfJd0d92BK_eBUSvA.png" /><figcaption>Example of data source set up</figcaption></figure><p><em>All of the keys can be found in the Back4App dashboard, in the </em><em>App Settings, </em><em>Security &amp; Keys.</em></p><p>The next thing we need to add is out historical sparkline. Click on ADD PANE to add a pane (wow). Those panes basically a “shelves” for your widgets. After pane is added — click on the plus icon in its corner. In type select the Historical Sparkline. Choose the title for it and add our data source datasources[“moisture”][“results”].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*ouGvEVZ0-MyrHm0B4yyi1Q.png" /><figcaption>Example of historical sparkline setup</figcaption></figure><p>If everything goes well you must be able to see a small graph with your data (hope you have some random numbers in the database, right?).</p><p>I also wanted to add an average value for moisture. Here comes another cool part of the Freeboard — you can use JavaScript function as a value for the widget.</p><p>To make it happen, create a new widget with type Text. And near the value field — click on the .JS EDITOR. Put that simple code in there:</p><pre>var total = 0;</pre><pre>for(var i=0; i &lt; datasources[&quot;moisture&quot;][&quot;results&quot;].length; i++) { <br>    total += datasources[&quot;moisture&quot;][&quot;results&quot;][i][&quot;value&quot;]; <br>}</pre><pre>return Math.round(<br>    total / datasources[&quot;moisture&quot;][&quot;results&quot;].length<br>);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*i8EADw6nxZ46hOtlSvEk-g.png" /><figcaption>Example of the average value widget setup</figcaption></figure><h4>Persist your dashboard</h4><p>We don’t want to lose all our hard work here just after we close the browser. So we need to save our dashboard.</p><p>Click on the SAVE FREEBOARD in the heading panel and chose the way of JSON file formatting (I prefer PRETTY version). Save your dashboard to the root directory of your clone of Freeboard, we will need it later to be accessible on the server.</p><h4>Freeboard on Heroku</h4><p>After we are done with setting up the dashboard — we might be wanting to make it accessible from anywhere and not just locally. Deploying Freeboard on the Heroku from scratch for the first time can be tricky. But I already figured it out and will give you some simple instructions ;)</p><p>You will need to create a new Heroku app for that part.</p><p>First of all, we need something to serve ours index.html. For that, we need to add some packages to the NPM: npm install connect serve-static --save.</p><p>After that, we need to create a simple server.js file with the next content:</p><pre><strong>var </strong><em>connect </em>= require(<strong>&#39;connect&#39;</strong>);<br><strong>var </strong><em>serveStatic </em>= require(<strong>&#39;serve-static&#39;</strong>);</pre><pre><strong>const </strong>PORT = process.env.PORT || 8000;</pre><pre><em>connect</em>().use(<em>serveStatic</em>(__dirname)).listen(PORT, <strong>function</strong>(){<br>    <strong><em>console</em></strong>.log(<strong>&#39;Server running...&#39;</strong>);<br>});</pre><p>The last step is to specify Procfile:</p><pre>web: node server.js</pre><p>And that’s all. After it’s done commit all the changes and push it to the Heroku remote (the same process that was done in the <strong>MQTT to HTTP</strong> setup).</p><h4>Load your board by default</h4><p>That’s why we needed to add the dashboard JSON to the repository. To load some JSON with dashboard you just need to add that small part to the end of the URL to your Freeboard #source=dashboard.json. E.g.</p><pre><a href="http://freeboard-cucumbers.s3-website.eu-north-1.amazonaws.com/#source=cucumbers.json">http://freeboard-cucumbers.s3-website.eu-north-1.amazonaws.com/<strong>#source=cucumbers.json</strong></a></pre><h3>Congratulations!</h3><p>If everything was smooth you now must have a working prototype of your cloud-based IoT infrastructure. The one I ended up with you can find here:</p><p><a href="http://freeboard-cucumbers.s3-website.eu-north-1.amazonaws.com/#source=cucumbers.json">http://freeboard-cucumbers.s3-website.eu-north-1.amazonaws.com/#source=cucumbers.json</a></p><p>Thanks for reading! Hope you liked it.</p><p>German Gensetskiy under the support of Go Wombat Team.</p><p>P.S. In case of any problems with following the tutorial I will be happy to hear your feedback. You can reach me out by the email: <a href="/service/mailto:Ignis2497@gmail.com">Ignis2497@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76ac5cf6e5a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/iot-simple-iot-with-free-cloud-solutions-tutorial-76ac5cf6e5a0">[IoT] Simple IoT with free cloud solutions (tutorial)</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[IoT] LoRa with Micropython on the ESP8266 and ESP32]]></title>
            <link>https://medium.com/gowombat/iot-lora-with-micropython-on-the-esp8266-and-esp32-59d1a4b507ca?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/59d1a4b507ca</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[embedded-systems]]></category>
            <category><![CDATA[iot]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 12:31:58 GMT</pubDate>
            <atom:updated>2019-01-18T12:33:24.006Z</atom:updated>
            <content:encoded><![CDATA[<h3>[IoT] LoRa with MicroPython on the ESP8266 and ESP32</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8DpavixCubeZGkYJ.jpg" /></figure><p>LoRa (Long Range) is a patented digital wireless data communication technology with low power consumption. That’s really important features for IoT applications.</p><p>That technology has great potential and we were interested in better understanding it so we decided to try it out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sDiNsOEM6yFElBYq-U_kkA.jpeg" /><figcaption>LoRa Modules</figcaption></figure><p>And we’re ordered different modules from Aliexpress… We had E32–868T20D 868MHz, ESP32 with SX1278 433MHZ, Ra-02 AiThinker 433MHz RFM96W 433 MHz.</p><blockquote>Not that we needed all that for the testing, but once you get locked into a serious LoRa collection, the tendency is to push it as far as you can.</blockquote><p>Next step was in understanding how to use it with micropython. Because we love Python and therefore we love Micropython ❤. We found out that there is exists an <a href="http://sx127x_driver_for_micropython_on_esp8266/">SX127x driver for ESP8266, ESP32 and Raspberry Pi</a>. And a clean version for ESP32 named <a href="https://github.com/lemariva/uPyLora">uPyLora</a>.</p><pre><strong>import </strong>time<br><br><strong>from </strong>lora.sx127x <strong>import </strong>SX127x<br><strong>from </strong>lora.controller_esp32 <strong>import </strong>ESP32Controller<br><br><br>controller = ESP32Controller()<br>lora = controller.add_transceiver(<br>    SX127x(name=<strong>&#39;LoRa&#39;</strong>),<br>    pin_id_ss=ESP32Controller.PIN_ID_FOR_LORA_SS,<br>    pin_id_RxDone=ESP32Controller.PIN_ID_FOR_LORA_DIO0<br>)<br><br>counter = 0<br>print(<strong>&quot;LoRa Sender&quot;</strong>)<br><br><strong>while True</strong>:<br>    payload = <strong>&#39;Hello ({0})&#39;</strong>.format(counter)<br>    print(<strong>&quot;Sending packet: \n{}\n&quot;</strong>.format(payload))<br><br>    lora.println(payload)<br><br>    counter += 1<br>    time.sleep(1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*6r0P5WTaYBdwNUQV6ffImQ.png" /><figcaption>Picocom output</figcaption></figure><p>And with ESP32 it worked well.</p><p>But unfortunately, on ESP8266 that code throws memory error. The simplest way was to slightly optimize that driver by removing constant variables that were used only once and placing their values straight into the code.</p><p>Unfortunately, we have only one antenna that was included with ESP32 we ordered. That test was done on a low range, I tried to move away with module without antenna — and maximal communication range was around 1.5–2 meters. So make sure to order one for each module or check that there will be pin to attach at least a simple wire.</p><p>Thanks for reading! Hope you’ll find that info helpful.</p><p><a href="https://github.com/Ignisor"><em>German Gensetskiy</em></a><em> under the support of </em><a href="https://gowombat.team/"><em>Go Wombat Team</em></a><em>.</em></p><p>And that’s helped.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59d1a4b507ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/iot-lora-with-micropython-on-the-esp8266-and-esp32-59d1a4b507ca">[IoT] LoRa with Micropython on the ESP8266 and ESP32</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using WikiData API to get the sense of words]]></title>
            <link>https://medium.com/gowombat/using-wikidata-api-to-get-the-sense-of-words-4db597227e8c?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/4db597227e8c</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[wikipedia]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Thu, 20 Dec 2018 14:46:37 GMT</pubDate>
            <atom:updated>2018-12-20T14:46:37.281Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7iD_9NDJQ9Km1BjV" /><figcaption>Photo by <a href="https://unsplash.com/@impatrickt?utm_source=medium&amp;utm_medium=referral">Patrick Tomasso</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In one of our current projects, we needed to get the main title of some entity. E.g. if we have “US” we need to know that this is “United States of America”.</p><p>Such a problem can be represented as an NLP (Natural Language Processing) problem. There was a possible solution like training some Word2Vec neural network model to make the semantics analysis of those words and find the semantic similarity of the words.</p><p>However, it pulls along problems like collecting and preparing a dataset, looking for the best model, training it and comparing the accuracy of different approaches.</p><p>In modern words, when everything goes very fast, especially the development process — sometimes you just don’t have enough time to do all that stuff. It is an important skill for a data scientist or even a developer to look and find an alternative which is a simpler and faster way to solve the problem.</p><p>And <em>free knowledge base with 53,282,301 data items</em> that is constantly expanding sounds like the easier and more stable solution. After all, there is not any artificial neural network that is better than human brains.</p><blockquote>Wikidata acts as central storage for the structured data of its Wikimedia sister projects including Wikipedia, Wikivoyage, Wikisource, and others.</blockquote><p>So, like any other Wikimedia project, it has an API that we can easily use. Especially, we interested in wbsearchentitiesaction. That endpoint provides us a label and description for the entity.</p><p>You can check the example below:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FWikiData-API%3Flite%3Dtrue&amp;url=https%3A%2F%2Frepl.it%2F%40Giermanlessless%2FWikiData-API%3Flite%3Dtrue&amp;image=https%3A%2F%2Frepl.it%2Fpublic%2Fimages%2Freplit-logo-800x600.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=repl" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b739e0647f0f44a9aee6ddb311280bb/href">https://medium.com/media/2b739e0647f0f44a9aee6ddb311280bb/href</a></iframe><p>Thanks for reading! Hope you’ll find it helpful.</p><p><a href="https://github.com/Ignisor"><em>German Gensetskiy</em></a><em> under the support of </em><a href="https://gowombat.team/"><em>Go Wombat Team</em></a><em>.<br>Special thanks to </em><a href="https://www.linkedin.com/in/stas-dragun-005451109/"><em>Stas Dragun</em></a><em> for helping me finding Wikidata.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4db597227e8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/using-wikidata-api-to-get-the-sense-of-words-4db597227e8c">Using WikiData API to get the sense of words</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[[GIT] Tags for versioning and GitLab CI]]></title>
            <link>https://medium.com/gowombat/git-tags-for-versioning-and-gitlab-ci-9c093c171a10?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c093c171a10</guid>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[regex]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Fri, 07 Dec 2018 10:29:27 GMT</pubDate>
            <atom:updated>2018-12-14T10:39:09.104Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugK1KImGZDpTNrXU_XxDLg.png" /></figure><p>In one of my current projects, i decided to use versions for my stage environment. I found that way quite useful and therefore i would like to share it with you.</p><p>So in the current setup we have 3 environments:</p><ul><li>development (for front-end developers and testing)</li><li>stage (for proper QA and demos)</li><li>production (for actual usage)</li></ul><p>Development and production environments are automatically updated from the corresponding branches (dev and master).</p><pre><strong>deploy:dev</strong>:<br>  <strong>stage</strong>: deploy<br>  <strong>only</strong>:<br>    - dev</pre><p>But for staging — there are version tags. Commonly those tags are created from the dev branch and are automatically updated using GitLab CI. It supports regular expressions and I created a simple one for versions in format v(float) e.g. v1.3</p><pre><strong>deploy:stage</strong>:<br>  <strong>stage</strong>: deploy<br>  <strong>only</strong>:<br>    - /^v[0-9]+\.[0-9]+$/</pre><p><a href="https://regex101.com/r/O08jjy/3">https://regex101.com/r/O08jjy/3</a> — here you can check examples and explanations of used regular expression.</p><p>Thanks for reading! Hope you’ll find it helpful.</p><p><a href="https://github.com/Ignisor"><em>Ignisor</em></a><em> under support of </em><a href="https://gowombat.team/"><em>Go Wombat Team</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c093c171a10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/git-tags-for-versioning-and-gitlab-ci-9c093c171a10">[GIT] Tags for versioning and GitLab CI</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple “Face ID” for your PC]]></title>
            <link>https://medium.com/gowombat/simple-face-id-for-your-pc-780168b95321?source=rss-1f0336a570e1------2</link>
            <guid isPermaLink="false">https://medium.com/p/780168b95321</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[face-recognition]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[German Gensetskiy]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 12:20:41 GMT</pubDate>
            <atom:updated>2018-12-14T10:40:12.611Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eNnjCbo99K-a6n4Tt_FTMA.png" /></figure><p>At our office, locking screen is a habit that you need to develop quickly. Because if you will leave your computer unlocked, someone will have fun and change your wallpapers or alias you `sudo` to something ;)</p><p>And one day, i started to think, why can’t i automate it? And here i come to the Face Recognition python library. It’s really easy to set up and use.</p><p>But first things first. We need to check if it is possible to lock screen from python and how to do it.</p><h3>Locking Screen</h3><p>I use Linux Mint with Cinnamon desktop environment. And fortunately, in Cinnamon case it’s pretty easy to lock or unlock the screen using screensaver command.</p><pre><strong>cinnamon-screensaver-command --activate  # to lock the screen<br>cinnamon-screensaver-command --deactivate  # to unlock the screen</strong></pre><p>And it’s not a big deal to run terminal command from python:</p><pre><strong>from </strong>subprocess <strong>import </strong>call</pre><pre>LOCK_ARGS = {<br>    <strong>True</strong>: <strong>&#39;--activate&#39;</strong>,<br>    <strong>False</strong>: <strong>&#39;--deactivate&#39;</strong>,<br>}<br></pre><pre><strong>def </strong>lock_screen(lock):<br>    call((<strong>&#39;cinnamon-screensaver-command&#39;</strong>, LOCK_ARGS[lock]))<br></pre><pre>lock_screen(<strong>True</strong>)  <em># will lock the screen<br></em>lock_screen(<strong>False</strong>)  <em># will unlock the screen</em></pre><h3>Setting up face_recognition</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6THTj-E65jvdScDsXIjhHw.jpeg" /></figure><p>The next step is to recognize your lovely face. We will use <a href="https://github.com/ageitgey/face_recognition">face-recognition</a> library. You can find many good examples in the repository, I believe <a href="https://github.com/ageitgey/face_recognition/blob/master/examples/facerec_from_webcam_faster.py">that one</a> is useful for us.</p><p>It uses OpenCV to capture stream from camera. Also I decided to use constitutional neural network to locate faces in the frame. To have better accuracy.</p><pre><strong>from </strong>threading <strong>import </strong>Timer</pre><pre><strong>import </strong>cv2<br><strong>import </strong>face_recognition<br></pre><pre><strong>def </strong>load_user_encoding():<br>    user_image = face_recognition.load_image_file(os.path.join(BASE_DIR, <strong>&#39;user.jpg&#39;</strong>))<br>    user_image_face_encoding = face_recognition.face_encodings(user_image, num_jitters=10)[0]<br><br>    <strong>return </strong>user_image_face_encoding<br></pre><pre><strong>def </strong>find_user_in_frame(frame, user_encoding):<br>    face_locations = face_recognition.face_locations(frame, model=<strong>&#39;cnn&#39;</strong>)<br>    face_encodings = face_recognition.face_encodings(frame, face_locations, num_jitters=2)<br><br>    <strong>for </strong>face_encoding <strong>in </strong>face_encodings:<br>        matches = face_recognition.compare_faces((user_encoding, ), face_encoding, tolerance=0.9)<br><br>        <strong>return </strong>any(matches)<br></pre><pre><strong>if </strong>__name__ == <strong>&#39;__main__&#39;</strong>:<br>    user_encoding = load_user_encoding()<br>    video_capture = cv2.VideoCapture(0)  <em># get a reference to webcam #0 (the default one)<br><br>    </em>lock_timer = <strong>None<br>    </strong>process_this_frame = <strong>True<br>    <br>    while True</strong>:<br>        ret, frame = video_capture.read()<br>        small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25)<br>        rgb_small_frame = small_frame[:, :, ::-1]<br><br>        <strong>if </strong>process_this_frame:<br>            user_found = find_user_in_frame(rgb_small_frame, user_encoding)<br><br>            <strong>if </strong>user_found:<br>                print(<strong>&#39;user found&#39;</strong>)<br>                lock_screen(<strong>False</strong>)<br><br>                <strong>if </strong>lock_timer <strong>is not None</strong>:  <em># cancel lock timer if it exists<br>                    </em>lock_timer.cancel()<br>                    lock_timer = <strong>None<br>            else</strong>:<br>                print(<strong>&#39;user not found&#39;</strong>)<br><br>                <strong>if </strong>lock_timer <strong>is None</strong>:  <em># start timer if it&#39;s not started already<br>                    </em>lock_timer = Timer(5, lock_screen, (<strong>True</strong>,))<br>                    lock_timer.start()<br><br>        process_this_frame = <strong>not </strong>process_this_frame</pre><p>As you can see I used threading.Timer to lock screen after 5 seconds in case the user is not found. I recommend to wait a bit before locking the screen because sometimes it can not recognize your face on some frames. Or you can just turn away for a moment.</p><h3>Optimizing</h3><p>With that solution it has a nasty delay for reading a frame and bad frameskip. So i decided to optimize it and move the recognition process to separate process using multiprocessing</p><p>First of all we need to rewrite our function for finding user so it will be able to called as Process with Pipe instead of return:</p><pre><strong>def </strong>find_user_in_frame(conn, frame, user_encoding):<br>    face_locations = face_recognition.face_locations(frame, model=<strong>&#39;cnn&#39;</strong>)<br>    face_encodings = face_recognition.face_encodings(frame, face_locations, num_jitters=2)<br><br>    found_user = <strong>False<br>    for </strong>face_encoding <strong>in </strong>face_encodings:<br>        matches = face_recognition.compare_faces((user_encoding, ), face_encoding, tolerance=0.9)<br><br>        found_user = any(matches)<br>        <strong>if </strong>found_user:<br>            <strong>break<br><br>    </strong>conn.send(found_user)</pre><p>And after that we need to call that function using multiprocessing.Process in main loop:</p><pre><strong>if </strong>__name__ == <strong>&#39;__main__&#39;</strong>:<br>    user_encoding = load_user_encoding()<br>    video_capture = cv2.VideoCapture(0)  <em># get a reference to webcam #0 (the default one)<br><br>    </em>lock_timer = <strong>None<br><br>    </strong>parent_conn, child_conn = Pipe()<br>    find_user_process = <strong>None<br>    while True</strong>:<br>        ret, frame = video_capture.read()<br><br>        small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25)<br><br>        rgb_small_frame = small_frame[:, :, ::-1]<br><br>        <em># if process of finding user is not working - start new one<br>        </em><strong>if </strong>find_user_process <strong>is None</strong>:<br>            find_user_process = Process(target=find_user_in_frame, args=(child_conn, rgb_small_frame, user_encoding))<br>            find_user_process.start()<br>        <em># if process of finding user is already working - check is it done<br>        </em><strong>elif </strong>find_user_process <strong>is not None and not </strong>find_user_process.is_alive():<br>            user_found = parent_conn.recv()<br>            find_user_process = <strong>None<br><br>            if </strong>user_found:<br>                print(<strong>&#39;user found&#39;</strong>)<br>                lock_screen(<strong>False</strong>)<br>                <strong>if </strong>lock_timer <strong>is not None</strong>:<br>                    lock_timer.cancel()<br>                    lock_timer = <strong>None<br>            else</strong>:<br>                print(<strong>&#39;user not found&#39;</strong>)<br>                <strong>if </strong>lock_timer <strong>is None</strong>:<br>                    lock_timer = Timer(LOCK_TIMEOUT, lock_screen, (<strong>True</strong>,))<br>                    lock_timer.start()</pre><p>Now it works more smooth and the delay is minimal.</p><p>Thanks for reading! Hope it was interesting and you enjoyed it.</p><p>Source code can be found on github: <br><a href="https://github.com/Ignisor/face-screenlock">https://github.com/Ignisor/face-screenlock</a></p><p><a href="https://github.com/Ignisor"><em>Ignisor</em></a><em> under support of </em><a href="https://gowombat.team/"><em>Go Wombat Team</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=780168b95321" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/simple-face-id-for-your-pc-780168b95321">Simple “Face ID” for your PC</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>