<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Artem Hoholiuk on Medium]]></title>
        <description><![CDATA[Stories by Artem Hoholiuk on Medium]]></description>
        <link>https://medium.com/@mr.shrrgn?source=rss-4695036aca0d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*H4W5tbrvtygnLAVA.jpg</url>
            <title>Stories by Artem Hoholiuk on Medium</title>
            <link>https://medium.com/@mr.shrrgn?source=rss-4695036aca0d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:06:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mr.shrrgn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to know that file was fully downloaded or not with Python and Nginx]]></title>
            <link>https://medium.com/gowombat/how-to-know-that-file-was-fully-downloaded-or-not-with-python-and-nginx-fc7f168a887?source=rss-4695036aca0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc7f168a887</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nginx]]></category>
            <dc:creator><![CDATA[Artem Hoholiuk]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 14:11:30 GMT</pubDate>
            <atom:updated>2020-02-13T14:30:15.297Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*7c8bXK6wFdg6Tgd2ahKLRA.jpeg" /></figure><h3>Introduction</h3><p>A few weeks ago, I got an interesting task on my current project: to find out if a file was completely downloaded from a server and write information about this event in the database by using API. I started looking for a solution and found one on <a href="https://stackoverflow.com/questions/33855350/how-to-notify-if-a-resource-completely-download-in-nginx">StackOverflow</a>. It suggested using the Nginx server. For API service, I used Django and Django Rest Framework. After starting implementing the described solution I realized it had many disadvantages. I’m gonna show my solution to the task.</p><p>First of all, you need to create a few microservices:</p><ul><li>web (for managing API requests);</li><li>nginx (as you understand it will be an Nginx web-server);</li><li>logger (a part responsible for gathering and sending logs to web-part).</li></ul><p>Also, it is good practice to use docker containers for this purpose.</p><h3>Nginx part</h3><p>Now it is time to write the nginx.conf file. We will store all the information about a request, so specify the name of the label, in our case, <strong>‘download’</strong>. Then, describe the format of your future log file: an IP, time of a request, a request, a status code, a number of bytes which were sent, the name of the file. To know more, check Nginx <a href="http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format">documentation</a>. <em>Traffic</em> is the main parameter because, in the API, you must compare its value with actual file size and find out if the file was downloaded completely (a little trick).</p><pre>log_format download &#39;{&quot;remote_addr&quot; : &quot;$remote_addr&quot;,&#39; <br>       &#39; &quot;time&quot;:&quot;$time_local&quot;,&#39; <br>       &#39; &quot;request&quot;:&quot;$request&quot;, &#39;<br>       &#39; &quot;status&quot;:&quot;$status&quot;,  &#39;<br>       &#39; &quot;traffic&quot;:<strong>$</strong>body_bytes_sent, &#39;<br>       &#39; &quot;uri&quot;: &quot;$uri&quot; }&#39;;</pre><p><strong>/media </strong>is a directory in the Django app which stores files, so we add an<strong> alias</strong> to get files from its, that’s why you need to add a <strong>proxy_pass </strong>to the web application.<strong> </strong>After this, just add a location for the log file.</p><pre>location /media {<br>     proxy_pass http://web:8000;<br>     default_type application/octet-stream;<br>     alias /web/media/;<br>     access_log /var/log/nginx/download.checker.log download;<br>}</pre><p>Everything else is like in an ordinary Nginx config. Full the config:</p><pre>worker_processes auto;<br><br>events {}<br><br>http {<br><br>  log_format download &#39;{&quot;remote_addr&quot; : &quot;$remote_addr&quot;,&#39; <br>       &#39; &quot;time&quot;:&quot;$time_local&quot;,&#39; <br>       &#39; &quot;request&quot;:&quot;$request&quot;, &#39;<br>       &#39; &quot;status&quot;:&quot;$status&quot;,  &#39;<br>       &#39; &quot;traffic&quot;:<strong>$</strong>body_bytes_sent, &#39;<br>       &#39; &quot;uri&quot;: &quot;$uri&quot; }&#39;;<br><br>  server {<br>    listen 80; # the port your site will be served on<br>    charset     utf-8;<br><br>    client_max_body_size 700M;  # max upload size<br>    <br>    # configs for the Django app<br>    location / {<br>      proxy_pass  http://web:8000;<br>      proxy_redirect  off;<br>      proxy_set_header Host <strong>$</strong>host;<br>      proxy_set_header X-Forwarded-For <strong>$</strong>proxy_add_x_forwarded_for;<br>    }<br><br>    location /media {<br>      proxy_pass http://web:8000;<br>      default_type application/octet-stream;<br>      alias /web/media/;<br>      access_log /var/log/nginx/download.checker.log download;<br>    }<br>  }<br>}</pre><p>In the same directory, add the simple Dockerfile for the Nginx:</p><pre>FROM nginx<br><br>WORKDIR /nginx_logs<br>COPY nginx.conf /etc/nginx<br><br>EXPOSE 80</pre><h3>logger part</h3><p>Now, the turn of the logger service. Create in the root the <strong>logger</strong> directory, add to it <strong>logs</strong> directory, and <strong>download.py</strong> file inside. You need to read data in real-time from the log file. Use the <strong>Popen </strong>class from the subprocess python module:</p><pre>log = subprocess.Popen(<br>    [&#39;tail&#39;, &#39;-F&#39;, &#39;download.checker.log&#39;],<br>    stdout=subprocess.PIPE,<br>    stderr=subprocess.PIPE<br>)<br><br>while True:<br>    send_logs(log)</pre><p><strong>send_logs</strong> function sends data about a request to web application, that’s why you need to install <strong>requests</strong> module for this. All example is uploaded below:</p><pre>import subprocess<br>import json<br>import requests<br>import logging<br><br><br>def send_stats(log: subprocess.Popen):<br>    data = json.loads(log.stdout.readline(), encoding=&quot;utf-8&quot;)<br>    payload = {<br>        &quot;traffic&quot;: data[&quot;traffic&quot;],<br>        &quot;file&quot;: data[&quot;uri&quot;].split(&quot;/&quot;)[-1],<br>        # you can add your fields<br>    }<br><br>    try:<br>        response = requests.post(<br>           url=&quot;http://web:8000/api/v1/stats&quot;, <br>           json=payload<br>        )<br>        response.raise_for_status()<br>    except Exception as e:<br>        logging.error(e)<br>    else:<br>        logging.info(f&quot;Log was sent to web API {payload}&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    log = subprocess.Popen(<br>        [&#39;tail&#39;, &#39;-F&#39;, &#39;download.checker.log&#39;],<br>        stdout=subprocess.PIPE,<br>        stderr=subprocess.PIPE<br>    )<br><br>    while True:<br>        send_stats(log)</pre><p>And docker file. Nothing ordinary here, but it deserves special attention because it uses the slim variant of Python image, which saves significant resources.</p><pre>FROM python:3.8-slim-buster<br><br>WORKDIR /logs<br>RUN pip install requests<br>COPY /logs /logs<br><br>CMD [&quot;python&quot;, &quot;download.py&quot;]</pre><h3>docker-compose file</h3><p>On the final step, specify docker-compose file with shared volume. Why do you need it? It is simple. Because your log file is in the Nginx container and logic for sending data in the other. So, create a special volume <strong>shared_logs</strong> and connect containers.</p><pre>version: &#39;3.7&#39;<br>services:<br><br>  web:<br>    # your application with API<br><br>  nginx:<br>    build:<br>      context: ./nginx<br>    links:<br>      - web<br>    volumes:<br>      - &quot;shared_logs:/var/log/nginx/&quot;<br>    ports:<br>      - &quot;80:80&quot;<br><br>  logger:<br>    build:<br>      context: ./logger<br>    links:<br>      - web<br>    volumes:<br>      - &quot;shared_logs:/logs&quot;<br><br>volumes:<br>  shared_logs:</pre><p><strong>Conclusion</strong></p><p>As you see, it is a simple task, but it may take a long time if you don’t know where to start. I hope you liked my article. Thank you for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc7f168a887" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/how-to-know-that-file-was-fully-downloaded-or-not-with-python-and-nginx-fc7f168a887">How to know that file was fully downloaded or not with Python and Nginx</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detect groups of targets on an image using deep learning and OpenCV]]></title>
            <link>https://medium.com/gowombat/detect-groups-of-targets-on-an-image-using-deep-learning-and-opencv-150409dff20?source=rss-4695036aca0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/150409dff20</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Artem Hoholiuk]]></dc:creator>
            <pubDate>Mon, 20 May 2019 09:24:53 GMT</pubDate>
            <atom:updated>2019-05-20T09:24:53.118Z</atom:updated>
            <content:encoded><![CDATA[<p>In the detection of objects and people, sometimes the question arises of identifying groups of people in a separate frame for subsequent processing. There are many variations of solutions to this problem, but in this article I will describe an approach that allows one to achieve the desired result by connecting deep learning with computer vision.</p><p>First you need to understand what is considered a group. It was decided to treat a group of people as a group of three or more people visually located nearby.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gYn_WCY7NOPqsHuJziO9hQ.jpeg" /></figure><h3>Tools</h3><p>First of all, the Python programming language will be used for this, since many implementations of machine learning algorithms are written on it. Also, it is simple and elegant. It is necessary to process the video through frames, which can be obtained through the <a href="https://opencv.org/">OpenCV</a> library. Each frame, in this case, will be considered as a separate image like in <a href="https://docs.opencv.org/3.0-beta/doc/py_tutorials/py_gui/py_video_display/py_video_display.html">OpenCV tutorial</a>.</p><p>Next was the choice of what to use to determine AI targets in the image, because the method should not be cumbersome and fast. The choice was made to use <a href="https://pjreddie.com/darknet/yolo/">YOLO</a>. This system works out quite quickly, even on a CPU. It has many implementations and pretrained models.</p><h3>How YOLO works?</h3><p>Prior detection systems repurpose classifiers or localizers to perform detection. They apply the model to an image at multiple locations and scales. High scoring regions of the image are considered detections.</p><p>Were applied a single neural network to the full image. This network divides the image into regions and predicts bounding boxes and probabilities for each region. These bounding boxes are weighted by the predicted probabilities.</p><p>The <a href="https://pjreddie.com/darknet/">Darknet</a> model has several advantages over classifier-based systems. It looks at the whole image at test time so its predictions are informed by global context in the image. It also makes predictions with a single network evaluation unlike systems like <a href="https://github.com/rbgirshick/rcnn">R-CNN</a> which require thousands evaluations for a single image. This makes it extremely fast, more than 1000x faster than R-CNN and 100x faster than <a href="https://github.com/rbgirshick/fast-rcnn">Fast R-CNN</a>.</p><h3>Deep Learning Backend</h3><p>The model itself is easily loaded via the <a href="https://gluon-cv.mxnet.io/">GluonCV</a> (a deep learning toolkit for computer vision), which uses the <a href="https://mxnet.incubator.apache.org/">MXNet</a> deep learning library as the backend. The library also has good documentation and has many implemented algorithms.</p><p>In order for the video frame to work with the MXNet library, you need to turn numpy array in mxnet array.</p><pre>right_frame = mxnet.nd.array(frame)</pre><h3>Processing and finding groups</h3><p>Next, the implementation of YOLO was chosen <em>yolo3_darknet_coco</em>, pre-trained on COCO dataset. In this set there is already a class “person” which made it possible not to waste time on training the neural network.</p><pre>net = model_zoo.get_model(<br>     &#39;yolo3_darknet53_coco&#39;, <br>     pretrained=True<br>)</pre><pre>x, img = data.transforms.presets.yolo.transform_test(<br>    right_frame, <br>    short=512, <br>    max_size=1024<br>)</pre><pre>class_IDs, scores, bounding_boxes = net(x)</pre><p>As a result of processing, arrays containing the names of the classes (class_IDs), the probability of the resulting class (scores), and an array of bounding boxes were obtained (bounding_boxes). After receiving the bounding boxes array, you can find groups through intersecting player boxes using the coordinates of the extreme points of the rectangle.</p><p>Further, the following actions were performed: if the bounding boxes of two targets are intersected, add this pair of values to the array <strong>target_intersection</strong>. Then, looping over , we check if there is a target in both arrays. If exists, then create or update a group of them.</p><pre>target_intersection = [</pre><pre>   sorted((k1,k2))</pre><pre>   for k1,v1 in targets.items()</pre><pre>   for k2, v2 in targets.items()</pre><pre>   if k1 != k2 and bounding_box_intersection(v1, v2)</pre><pre>]</pre><pre>groups = []</pre><pre>for pair in target_intersection:</pre><pre>   found_group = False</pre><pre>   for group in groups:</pre><pre>      if set(pair).intersection(group):</pre><pre>         group.update(pair)</pre><pre>         found_group = True</pre><pre>         break</pre><pre>   if not found_group:</pre><pre>      groups.append(set(pair))</pre><h3>Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*EZyelXM8TJZu0QqeAtU8qg.jpeg" /></figure><p>In the picture, the AI targets are highlighted through the bounding boxes along with the name of the class and the probability of getting to this class. The program result:</p><pre>groups = find_groups(persons)</pre><pre>print(f&quot;Result = {groups}&quot;)</pre><pre>Result = [6,5]</pre><p>Result = [6, 5] means the two groups of people with six in first and five in second. If result was equal to [] it means that no groups of people on an image.</p><h3>Conclusion</h3><p>Were designed concept for getting groups of AI targets on a frame using deep learning and OpenCV. This method is fast and accurate and also can be used for video surveillance systems, video montage, etc.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=150409dff20" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gowombat/detect-groups-of-targets-on-an-image-using-deep-learning-and-opencv-150409dff20">Detect groups of targets on an image using deep learning and OpenCV</a> was originally published in <a href="https://medium.com/gowombat">Go Wombat</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>