<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by mustafa can uslu on Medium]]></title>
        <description><![CDATA[Stories by mustafa can uslu on Medium]]></description>
        <link>https://medium.com/@pathintegral?source=rss-5c28904105f3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0MMvmbHPCZbkxG3yjFsjzQ.jpeg</url>
            <title>Stories by mustafa can uslu on Medium</title>
            <link>https://medium.com/@pathintegral?source=rss-5c28904105f3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:05:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pathintegral/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Üretken Yapay Zeka: İlham mı alıyor, taklit mi ediyor?]]></title>
            <link>https://medium.com/life-turknet/%C3%BCretken-yapay-zeka-i%CC%87lham-m%C4%B1-al%C4%B1yor-taklit-mi-ediyor-5c721865bf1f?source=rss-5c28904105f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c721865bf1f</guid>
            <category><![CDATA[tolkien]]></category>
            <category><![CDATA[carl-sagan]]></category>
            <category><![CDATA[bilimkurgu]]></category>
            <category><![CDATA[yapay-zeka]]></category>
            <dc:creator><![CDATA[mustafa can uslu]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 11:22:05 GMT</pubDate>
            <atom:updated>2025-10-01T14:43:58.703Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nd23ER1QJ4SM6FA-" /></figure><p>Merhaba Başka Bi’Bülten okuyucuları,</p><p>Bu hafta bülteni devralan benim. İnsanlık tarihinin yeniliklerle dolu geçmişine bakarak bugünü nasıl şekillendirdiğimizi ve üretken yapay zekânın bu hikayedeki yerini birlikte keşfedeceğiz. Ayrıca, bu teknolojiyi daha empatik ve anlayışlı bir hale getirme yolunda karşılaştığımız zorluklara da değineceğiz.</p><p>Hazırsanız, geçmişten ilham alıp geleceğe uzanan bu yolculuğa başlayalım!</p><blockquote><em>Carl Sagan’ın “Bugünü anlamak için geçmişi bilmek gerekir” sözü, insanlık tarihindeki yeniliklerin temelinde geçmişten ilham alınıp dönüştürülmüş fikirlerin yattığı gerçeği hatırlatıyor. Hem bugün hayatlarımızı kolaylaştıran teknolojiler üreten hem de yarınları tasarlamaya çalışan bir yapay zekâ mühendisi olarak, bu sözün benim de düşünme şeklimi fazlasıyla etkilediğini düşünüyorum. Bir yandan da yapay zekâ ile birlikte hayatlarımız hiç olmadığı kadar hızla dönüşürken, “yeni bir şeyler üretmiyor, var olanlardan ilham alarak onları dönüştürüyor” eleştirilerini ve aksi yöndeki beklentileri biraz enteresan buluyorum. Zira, yaratıcılığın doğası tam da bu çağrışımlar ve dönüşümler üzerine kurulu değil midir sizce de?</em></blockquote><p>Tarih boyunca çağ atlatan yenilikler, büyük ölçüde önceki birikimlerin üzerine inşa edilmiştir. Leonardo da Vinci’nin çizimlerinden esinlenilerek geliştirilen modern hava taşıtları ya da Jules Verne’in denizaltı konseptinin gerçek dünya mühendisliğine ilham vermesi buna somut örneklerdir. Aynı şekilde, bugün kullandığımız çok sayıda teknoloji, bilim-kurgu eserlerinde uzun yıllar öncesinde hayal edilmiştir. Akıllı telefonların temelini oluşturan dokunmatik ekranlar ve sesli asistanlar, 20. yüzyıl bilim-kurgu filmlerinin vazgeçilmez ekipmanlarıydı. Bu fikirler, dönemlerinde birer fantezi öğesi olarak görülse de zamanla gerçekliğe dönüştürüldü.</p><p>Türevlerinden farklı olarak üretken yapay zekâ da beslendiği verilerden öğrenerek yeni içerikler oluşturan bir teknoloji olarak bugün hayatımızda önemli bir yer tutuyor. Bir yapay zekâ modeli, mevcut verilerden öğrenerek yeni bir şiir, görsel ya da melodi oluşturduğunda, aslında insanlık tarihindeki yaratıcı sürecin benzer bir şeklini takip ediyor. Örneğin, JRR Tolkien bugün bile benzersiz nitelikteki Orta-dünya evrenini yaratırken; edebi metin olarak var olan destanları referans almış ve kendi tecrübelerini de bu destanlarla harmanlamıştır. Elflerin Kelt mitolojisinde, ejderhaların Beowulf destanında, cücelerin İskandinav mitolojisinde benzerlerini bulmak mümkündür. Mordor’un ya da Isengard’ın karanlığı ise sanayi devrimi sonrasında Tolkien’in de bizzat tecrübe ettiği makineleşmenin ve 1. Dünya Savaşı’nın izlerini taşır.</p><p>Üretken yapay zekâyı eleştirenlerin göz ardı ettiği nokta, bu teknolojinin insan yaratıcılığıyla iç içe çalıştığı gerçeğidir. Bir sanatçı, tasarımcı ya da mühendis, yapay zekâ aracılığıyla yeni fikirler keşfedebilir, bu fikirleri kendi bakış açısıyla geliştirerek eşsiz bir sonuç ortaya koyabilir. Bu ortak yaratım süreci, insanoğlunun yaratıcılık potansiyelini daha da ileriye taşımasına çok daha büyük katkılar sunabilir.</p><p>Bugün üretken yapay zekâ tarafından üretilen eserler, kaçınılmaz olarak geçmişten esinlenmiş olabilir; ancak bu durum, şu an kullandığımız çoğu teknolojinin ortaya çıkış hikayesinden farklı değildir. Carl Sagan’ın sözüne son bir kez daha odaklanacak olursak: geçmişe bakarak öğrendiğimiz sürece, bugün yarını yaratacak fikirlerin tohumlarını ekiyoruz. Haliyle üretken yapay zeka da, bu yaratıcı sürecin modern bir yansıması olarak, geleceği şekillendirmede önemli bir rol oynuyor.</p><p>Tüm bu tartışmanın ardından, bence elimizde hala daha çözülmemiş bir problem bulunuyor:</p><p>Müşterimizin karşısına koyduğumuz üretken yapay zekâlı ürünlerimizi gerçekten de empati yapabilir hale nasıl getireceğiz? Yıllar boyunca soğuk, tepkisiz, duygusuz robotlar olarak beyaz perdeye yansıttığımız robotları şimdi anlayışlı şeyler haline getirebilir miyiz sizce?</p><p>Yaşayıp göreceğiz ve hep birlikte bu sorunun cevabını vereceğiz.</p><p>Görüşmek üzere!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/0*Db-SSznzq_UH3kL5" /></figure><p><a href="https://www.linkedin.com/in/mustafa-can-uslu/">Mustafa Can Uslu</a><br>AI Lead</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c721865bf1f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/life-turknet/%C3%BCretken-yapay-zeka-i%CC%87lham-m%C4%B1-al%C4%B1yor-taklit-mi-ediyor-5c721865bf1f">Üretken Yapay Zeka: İlham mı alıyor, taklit mi ediyor?</a> was originally published in <a href="https://medium.com/life-turknet">Life@Turknet</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to build your -own- KYC?]]></title>
            <link>https://medium.com/turknettech/how-to-build-your-own-kyc-375beaee29d8?source=rss-5c28904105f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/375beaee29d8</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[id-verification]]></category>
            <category><![CDATA[kyc]]></category>
            <category><![CDATA[yolo]]></category>
            <category><![CDATA[paddleocr]]></category>
            <dc:creator><![CDATA[mustafa can uslu]]></dc:creator>
            <pubDate>Fri, 08 Nov 2024 05:56:58 GMT</pubDate>
            <atom:updated>2024-11-08T05:56:58.385Z</atom:updated>
            <content:encoded><![CDATA[<p>Nowadays, you are required to verify that “you are you” while registering for a service. One of the most well-known options is <strong><em>verifying your ID card via cameras</em></strong>, in addition to the liveness test, signature verification, etc. In this post, I’ll guide you in building the know-your-customer solution for ID card verification.</p><p>So, we need a web page, an API, an identity classifier model, a few field detection models, and an <strong>OCR model</strong> to satisfy that. By integrating those components, a platform-independent ID verification service will be ready to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tXF6RTPjFIMcxu8vTfzrNQ.png" /><figcaption>Verification cycle, step-by-step.</figcaption></figure><p>Let’s start with the web page. Consider an empty HTML page with just a video element that will access to camera and some javascript code to stream camera input to the <strong>video</strong> element. Save the following code as <strong>index.html</strong> and open it via browser to see a full-page version of yourself.</p><pre>&lt;!DOCTYPE html&gt;<br>&lt;html&gt;<br>  &lt;head&gt;<br>    &lt;title&gt;Real-Time ID Verification&lt;/title&gt;<br>  &lt;/head&gt;<br>  &lt;body&gt;<br>    &lt;video id=&quot;video&quot; autoplay style=&quot;width:100vw; height:100vh;&quot;&gt;&lt;/video&gt;<br>  &lt;/body&gt;<br>  &lt;script&gt;<br>    window.onload = function () {<br>    // Access the device camera and stream to video element<br>    navigator.mediaDevices.getUserMedia({ video: true })<br>      .then(function (stream) {<br>        document.getElementById(&#39;video&#39;).srcObject = stream;<br>      })<br>      .catch(function (error) {<br>        console.log(&quot;Error accessing webcam: &quot;, error);<br>      });<br>    };<br>  &lt;/script&gt;<br>&lt;/html&gt;</pre><p>Now, let’s build a web application that will serve the index.html and process incoming images as an API. <strong>FastAPI</strong> is my go-to solution for such projects.</p><pre>from fastapi import FastAPI, Request<br>from fastapi.templating import Jinja2Templates<br><br>app = FastAPI()<br>templates = Jinja2Templates(directory=&quot;templates&quot;)<br><br>@app.get(&quot;/&quot;)<br>async def index(request: Request):<br>    return templates.TemplateResponse(&quot;index.html&quot;, {&quot;request&quot;: request})</pre><p>Let’s organize the file/folder structure as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*HwV_fOkRAbrqU73sjdgE8Q.png" /><figcaption>Folder structure for the project.</figcaption></figure><p>Now, let’s start the project via <strong>uvicorn</strong>:</p><pre>uvicorn main:app --host 0.0.0.0 --port 8000 --reload</pre><p>You can check if everything is fine by visiting <a href="http://localhost:8000."><strong>http://localhost:8000</strong></a>. Similar to opening index.html in a web browser, you should see yourself in a full-screen video input.</p><p>Just before diving into the rabbit hole, now it’s time to make the stream active from the web page to the API. It’s handled by a few line-length JavaScript codes. Since I’m not a front-end guru and feel cool by playing with jQuery, I used <strong>$.ajax</strong> utility for streaming inputs to API.</p><p>In addition to that, accessing the webcam of a computer or mobile phone’s camera might be tricky. Not all the devices are supported as far as I saw. Yet, there are lots of JS libraries for such needs. If you packed this project up with an Ionic-like framework, you may utilize NFC-like features too.</p><pre>function classify_card_type() {<br>    let video = document.getElementById(&#39;video&#39;);<br>    let canvas = document.createElement(&#39;canvas&#39;);<br>    canvas.width = video.videoWidth;<br>    canvas.height = video.videoHeight;<br>    canvas.getContext(&#39;2d&#39;).drawImage(video, 0, 0);<br><br>    canvas.toBlob(function (blob) {<br>        let formData = new FormData();<br>        formData.append(&#39;video_frame&#39;, blob);<br><br>        $.ajax({<br>            url: &#39;/classify&#39;,<br>            type: &#39;POST&#39;,<br>            data: formData,<br>            contentType: false,<br>            processData: false,<br>            success: function (response) {<br>                if (response.data.confidence &gt;= 0.999) {<br>                    is_classified = true;<br>                    card_type = response.data.card_type;<br>                }<br>            },<br>            error: function (error) {<br>                console.log(&#39;Error: &#39;, error);<br>            }<br>        });<br>    }, &#39;image/jpeg&#39;);<br>}</pre><p>Let’s implement <strong>/classify</strong> and <strong>/verify</strong> endpoints to FastAPI and stream incoming frames to FastAPI.</p><pre>@app.post(&quot;/classify&quot;)<br>async def classify(request: Request):<br>    &quot;&quot;&quot;<br>    Classifies a document based on an uploaded video frame (image).<br><br>    This function processes an uploaded image file, attempts to classify the document<br>    type using a YOLO-based classification model, and returns the classification results<br>    along with associated confidence and document fields.<br>    &quot;&quot;&quot;<br>    try:<br>        form_data = await request.form()<br>    except Exception as e:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;An error occured while obtaining the request.&quot;, &quot;detail&quot;: str(e)}<br><br>    if &#39;video_frame&#39; not in form_data:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;Invalid request data.&quot;}<br><br>    # Parse the payload.<br>    video_frame = form_data.get(&#39;video_frame&#39;)<br><br>    # Decode image.<br>    try:<br>        contents = await video_frame.read()<br>        nparr = np.frombuffer(contents, np.uint8)<br>        img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)<br>    except Exception as e:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;An error occured while decoding the image.&quot;, &quot;detail&quot;: str(e)}<br><br>    # Classify image.<br>    result = classify_image(img)<br>    confidence = result.get(&#39;confidence&#39;, 0)<br>    card_type = result.get(&#39;name&#39;, &#39;&#39;)<br><br>    return {&quot;success&quot;: True, &quot;data&quot;: {&quot;confidence&quot;: confidence, &quot;card_type&quot;: card_type}}</pre><p>As it seems in the code itself, <strong>/classify</strong> endpoint is responsible for classifying incoming images into ID card types. To satisfy that, a classifier model is trained for various card types such as ID cards, driving licenses, temporary work/residence permit cards, passports, etc. We are using this model for<strong> two basic reasons</strong>:</p><p><strong>First</strong>, each card type contains different metadata. For example, you can’t find mother’s or father’s name on a driver’s license whereas they exist on ID cards. If you are required to verify those, you should not continue with driver’s license and prompt a message that indicates showing ID card instead of driver’s license to the customer.</p><p><strong>Second</strong>, if the customer opens the camera and does not show their ID to the camera; this will cause an unnecessary usage of resources on the server side. You may add some rules to eliminate that non-efficient situation such as if 5 consequent frames do not contain an ID card then disable camera input.</p><p><strong>classify_image </strong>function takes the image as input and returns the YOLO card classifier model output as a dictionary with <strong>name</strong> and <strong>confidence</strong> keys. This is also a good opportunity for implementing various business models. If the card type is not supported you will be able to prompt the customer that “<strong><em>a supported card is required to continue</em></strong>”.</p><p>In our case, both pages of the ID card (front/back) are required for verification. So, when you start the process by verifying the front page first and then the back page. Also, to verify something you need a reference too. We are using the data supplied by the customer via the sales funnel as the reference. You’ll see those parameters as <strong><em>document_side</em></strong><em> </em>and <strong><em>payload</em></strong> in /verify endpoint.</p><pre>@app.post(&quot;/verify&quot;)<br>async def detect_fields(request: Request):<br>    &quot;&quot;&quot;<br>    Detects and verifies fields in a document image.<br><br>    This function processes an uploaded video frame (image) of a document,<br>    verifies the detected fields against provided information, and returns<br>    the results of the field detection and verification process.<br>    &quot;&quot;&quot;<br>    try:<br>        form_data = await request.form()<br>    except Exception as e:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;An error occured while obtaining the request.&quot;, &quot;detail&quot;: str(e)}<br><br>    if &#39;video_frame&#39; not in form_data or &#39;document_side&#39; not in form_data or &#39;payload&#39; not in form_data:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;Invalid request data.&quot;}<br><br>    # Parse payload.<br>    try:<br>        video_frame = form_data.get(&#39;video_frame&#39;)<br>        document_side = form_data.get(&#39;document_side&#39;)<br>        payload = form_data.get(&#39;payload&#39;)<br>        payload = json.loads(payload)<br>    except Exception as e:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;An error occured while parsing the request data.&quot;, &quot;detail&quot;: str(e)}<br><br>    # Decode image.<br>    try:<br>        contents = await video_frame.read()<br>        nparr = np.frombuffer(contents, np.uint8)<br>        img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)<br>    except Exception as e:<br>        return {&quot;success&quot;: False, &quot;message&quot;: &quot;An error occured while decoding the image.&quot;, &quot;detail&quot;: str(e)}<br><br>    # Verify image.<br>    document_fields = verify_image(<br>        img, document_side, payload)<br><br>    return {&quot;success&quot;: True, &quot;data&quot;: document_fields}</pre><p><strong>verify_image</strong> function takes the image, document_side, and payload as input and returns document_fields as output. Field detection models are utilized in this step. To simplify, I continued with a single type of card but if you are required to verify different types, you just need to implement the <strong>card_type</strong> parameter to verify_image function which comes from /classify endpoint output.</p><p>After successfully completing field detection, it’s time to extract text from each field. <strong>PaddleOCR</strong> is the best free OCR solution for me for a while. It’s presented as SOTA for generic usage. So, we utilize PaddleOCR to extract texts in this step.</p><p>Finally, we can review the output of the OCR model and check if they are the same as the payload. If any field is not, then let the customer know that too. Sometimes, fields are occluded by fingers or lightened/darkened because of light source positioning. In such cases, it’s better to tell the customer “<strong><em>Field X is not verified please make it visible to the camera”</em></strong>.</p><p>To visualize what we’ve done so far and what the complete picture is, let me help you with the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YYtctKPQmIKbHuOT1LMJPQ.png" /></figure><p>Overall, there are lots of possible improvements and implementations for the project up to your requirements. I tried to mention just the core functionality and the rest is yours. Before finishing this guide, what I learned from this project and what should be considered to implement such a project is listed below.</p><ul><li>Keep the video input size as small as possible. Since most computer cameras are not good for capturing, using ~640x480 inputs yields better performance in terms of accuracy and speed.</li><li>Train your classifier and field detection model for low-res images. As you may already know YOLO is blazing fast. If you continue with low-res images, you may convert your service from near-real-time to real-time with ease.</li><li>You may increase/decrease the frequency of streaming for optimal experience. Classification ends up to 0.5 seconds on CPUs. Field detection and OCR end up to 1 second with by-default choices. Moreover, if there are N customers that trying to verify their ID cards simultaneously you may increase the frequency to 2 seconds.</li><li>Since neither endpoint is state-aware, you should add some procedures to the front-end side. Turn the camera off after consecutive same answers for a while.</li><li>Guide the customer for better results. Use model outputs as directives and prompt them to the customer for successful verification.</li></ul><p>Soon, I’ll update this post with a GitHub repository with all the components. Until then, stay safe and stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375beaee29d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/turknettech/how-to-build-your-own-kyc-375beaee29d8">How to build your -own- KYC?</a> was originally published in <a href="https://medium.com/turknettech">Turknet Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ghost in the Fiber-Optics]]></title>
            <link>https://medium.com/@pathintegral/ghost-in-the-fiber-optics-9b5a78aade6e?source=rss-5c28904105f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b5a78aade6e</guid>
            <category><![CDATA[grafana]]></category>
            <category><![CDATA[influxdata]]></category>
            <category><![CDATA[snmpwalk]]></category>
            <category><![CDATA[pysnmp]]></category>
            <dc:creator><![CDATA[mustafa can uslu]]></dc:creator>
            <pubDate>Sat, 27 Jul 2024 21:20:23 GMT</pubDate>
            <atom:updated>2024-07-27T21:20:23.448Z</atom:updated>
            <content:encoded><![CDATA[<p>Right after joining the NOC Tools team as a data scientist/data engineer role almost one year ago, I learned that monitoring would be an essential part of the roadmap. The goal for 2021 was to monitor each device’s health and performance on the field, on the exchange point, or in the customers’ offices and homes across the whole of Turkey. Of course, there is an ongoing monitoring system that handles all those operations and sends alerts to respondents. Yet, most of the parts of that system consist of outdated tools and legacy codes. So, I took the responsibility for every single part of which “data” is coming through or going out and managed to replace oldies with newbies and also goodies.</p><p>One of the particular tasks I’ve done was collecting traffic stats of users from BNGs (Broadband Network Gateway). Most of the collectors can obtain traffic stats per user. However, in some cases, you would not like to let your devices to respond SNMP calls of collectors because of performance or privacy-related issues. So, that drives me to build a custom SNMP collector.</p><blockquote><em>Simple Network Management Protocol (SNMP) is an Internet Standard protocol for collecting and organizing information about managed devices on IP networks and for modifying that information to change device behavior.</em></blockquote><blockquote><a href="https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol"><em>https://en.wikipedia.org/wiki/Simple_Network_Management_Protocol</em></a><em>, Last access: 14.11.2021</em></blockquote><p>In the early days, I learned that the network teams were using 2 methods to collect traffic stats and those are <strong>Pure SNMP</strong> or <strong>SNMP with Telegraf</strong>. <br>– <strong>Pure SNMP:</strong> It’s pure because we are just sending a few SNMP GET requests.<br>– <strong>SNMP with Telegraf:</strong> End-to-end solution for collecting both historical and real-time traffic stats. Please, check the link for further details <a href="https://www.influxdata.com/blog/monitor-your-snmp-devices-with-telegraf/">(https://www.influxdata.com/blog/monitor-your-snmp-devices-with-telegraf/</a>)</p><p>Since Telegraf wasn’t suitable for our needs, we decided to continue with pure SNMP calls. Those can be listed as:</p><ul><li>Get the list of all sessions on the devices.</li><li>Get the ifindex of the user by using the user’s OID.</li><li>Get traffic stats (ifOutOctets, ifInOctets) of the user by using ifindex.</li></ul><p>Before falling into the rabbit hole, I’d like to share some insights about the former state. What I took over is a few bash scripts that use files as data storage and the runtime of the total process was more than 10 minutes whereas it should be 2–3 minutes. I/O operations and sequential calls for data collection seemed “the usual suspects” of unacceptable runtimes to me. So, I was focused on parallelization and handling I/O operations.</p><p><strong>snmpbulkwalk</strong> was used to send requests and collect outputs. You may run it via the command below and get the output as follow.</p><pre># snmpbulkwalk documentation<br>snmpbulkwalk -v SNMP_VERSION \<br>             -u USERNAME \<br>             -l SECURITY_LEVEL \<br>             -a AUTH_PROTOCOL \<br>             -A AUTH_PROTOCOL_PASSPHRASE \<br>             -x PRIVACY_PROTOCOL \<br>             -X PRIVACY_PROTOCOL_PASSPHRASE \<br>             IP_ADDRESS \<br>             OID<br>-----<br># example usage of snmpbulkwalk command to get list of all sessions on device<br>snmpbulkwalk -v 3 -u XXX -l authPriv -a md5 -A XXX -x des -X XXX X.X.X.X 1.3.6.1.4.1.9.9.150.1.1.3.1.2</pre><pre># output<br>SNMPv2-SMI::enterprises.9.9.150.1.1.3.1.2.3246813 = STRING: &quot;USER1@ISP.com&quot; <br>SNMPv2-SMI::enterprises.9.9.150.1.1.3.1.2.3246933 = STRING: &quot;USER2@ISP.com&quot; <br>SNMPv2-SMI::enterprises.9.9.150.1.1.3.1.2.3247113 = STRING: &quot;USER3@ISP.com&quot;<br>...</pre><p>There are some interesting points on both command and output. Let’s focus on the command first. As you can see, I’m using <strong>SNMP_VERSION = 3</strong> which is far more secure than previous versions (Please check Cisco’s documentation about why to use SNMP v3:<a href="https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/snmp/configuration/xe-3se/3850/snmp-xe-3se-3850-book/nm-snmp-snmpv3.pdf"> https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/snmp/configuration/xe-3se/3850/snmp-xe-3se-3850-book/nm-snmp-snmpv3.pdf</a>). Also, we are using a series of numbers split by dots, like an IP address from an exoplanet, which is OID (Object Identifier).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*xdc_P1Qb2RYx_3oe.png" /></figure><pre># example usage of snmpbulkwalk to get ifindex of a particular session<br>snmpbulkwalk -v 3 -u XXX -l authPriv -a md5 -A XXX -x des -X XXX X.X.X.X 1.3.6.1.4.1.9.9.150.1.1.3.1.8.3246813</pre><pre># output<br>SNMPv2-SMI::enterprises.9.9.150.1.1.3.1.8.3246813 = INTEGER: 5421</pre><p>To get ifindexes, you just need to change the OID of the previous command. IfIndex (Interface Index, please check for further details: <a href="https://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/28420-ifIndex-Persistence.html">https://www.cisco.com/c/en/us/support/docs/ip/simple-network-management-protocol-snmp/28420-ifIndex-Persistence.html</a>) is used to specify the interface (might be physical or virtual, in our case virtual) of a particular session. So, we can say that <strong>USER1@ISP.com</strong>‘s session is on the interface whose index is <strong>5421</strong> at the X.X.X.X device.</p><pre># example usage of snmpbulkwalk to get traffic data of a particular interface<br>snmpbulkwalk -v 3 -u XXX -l authPriv -a md5 -A XXX -x des -X XXX X.X.X.X IF-MIB::ifInOctets.5421<br>snmpbulkwalk -v 3 -u XXX -l authPriv -a md5 -A XXX -x des -X XXX X.X.X.X IF-MIB::ifOutOctets.5421</pre><pre># output<br>IF-MIB::ifInOctets.5421 = Counter32: 578684682<br>IF-MIB::ifOutOctets.5421 = Counter32: 1582316104</pre><p>Finally, we have what we needed.. at least one of them. Now, the goal is to convert this set of commands to a python service and apply it to thousands of sessions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sDb6V6ENlo_8B8gk" /></figure><p>Adapting those commands to python was relatively easy with the help of the pysnmp library and its well-written documentation. Especially, examples that cover both snmpwalk / snmpbulkwalk command pysnmp code together are decreased the development time significantly.</p><p>As I mentioned earlier, I planned a parallelization structure as follows:</p><ul><li><strong>Parallelization of the 1st step:</strong> Each CPU core is going to fetch all sessions on a BNG.</li><li><strong>Parallelization of the 2nd step:</strong> Each CPU core knows OID and IP address of relevant BNG is going to get ifindex.</li><li><strong>Parallelization of the 3rd step:</strong> Each CPU core knows the ifindex and IP address of relevant BNG is going to get traffic stats.</li></ul><p>Unfortunately, all the given efforts in terms of parallelization didn’t end up with a crystal-clear solution. As can be seen from the diagram below, Step 2 has to wait until the completion of Step 1 and Step 3 waits for both Step 1 and Step 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*ZBTrFJf_xvbZyvqz.png" /></figure><p>After deploying this setup on an 8-core server, gathering traffic stats of 6k sessions from 600k total sessions on 40 BNGs finished in 2–3 minutes. On the contrary, with a fully sequential run, it was about 30 minutes.</p><p>I prepared a benchmark to visualize the overall performance. Durations represent more than I mentioned because test runs are completed on a different setup and that causes a network delay. In the production setup, those durations decreased to almost a quarter of the shown values. It can be seen in the second table.<br><strong>num_of_bng:</strong> # of BNG<br><strong>get_list_of_sessions: </strong>Duration to list all sessions in a BNG.<br><strong>get_traffic_data:</strong> Duration to get ifIndex and traffic stats for selected sessions.<br><strong>total_duration:</strong> Total duration of the process.<br><strong>num_of_sessions:</strong> # of selected sessions to extract traffic stats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*PFVsoKsZTkHknf0g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*19JFexCi2hnHSBp0.png" /></figure><p>At the end of the day, all the extracted stats were inserted into InfluxDB and visualized to help the network operations team in <strong>Grafana</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b5a78aade6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A brief data engineering on Netflow]]></title>
            <link>https://medium.com/@pathintegral/a-brief-data-engineering-on-netflow-acdfc6c8dbaa?source=rss-5c28904105f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/acdfc6c8dbaa</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[goflow]]></category>
            <category><![CDATA[ksqldb]]></category>
            <category><![CDATA[ipfix]]></category>
            <category><![CDATA[netflow]]></category>
            <dc:creator><![CDATA[mustafa can uslu]]></dc:creator>
            <pubDate>Sat, 27 Jul 2024 21:06:17 GMT</pubDate>
            <atom:updated>2024-07-27T21:06:17.843Z</atom:updated>
            <content:encoded><![CDATA[<p>There has been a phenomenon of how to decide if the data is big or not. I’ve seen TBs of image data in terms of size, and GBs of text data in terms of count. Yet, <strong>Netflow</strong> is whole a different story… For the first time, I felt like operating Nebuchadnezzar as Tank does in the Matrix movie. Please, think about the incoming data of ~1 million customers’ requests, and referrals for a second, and then buckle up to dive into the rabbit hole.</p><figure><img alt="Opentelemetry – ITNEXT" src="https://cdn-images-1.medium.com/proxy/0*RM18YeovA9Vye65o" /></figure><blockquote><strong><em>NetFlow</em></strong><em> is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface.</em><strong><em>NetFlow</em></strong><em> is introduced by Cisco in 1996 to collect IP network traffic as incoming or outgoing.</em></blockquote><blockquote><a href="https://en.wikipedia.org/wiki/NetFlow"><em>https://en.wikipedia.org/wiki/NetFlow</em></a></blockquote><p>We can extract the source/destination of traffic, network protocol, duration, usage in terms of bytes, and some more details from Netflow data. Due to the increasing need, lots of new Netflow versions are released. Nowadays, “<strong>Netflow v9/IPFIX</strong>” is the most popular one all around the world. While Netflow is designed to work with Cisco routers, IPFIX became a more universal solution and works perfectly for all types of network devices. To see all the fields and representations of IPFIX, please check the <a href="https://www.iana.org/assignments/ipfix/ipfix.xhtml">IANA’s IPFIX Entities page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oXobI8wNlZu3oeF7.png" /></figure><p>Just before the technical aspect, I’d like to mention CDR data too. Since Netflow contains IP addresses as an identifier, we need to find a way to add usernames for output. This can be done with the help of CDR. CDR tells us which IP address is occupied by which customer in which time range. To sum up, what I need to do is “get the IP address from Netflow, and join CDR by considering traffic direction and time conditions”.</p><p>There are lots of open-source and proprietary tools for collecting Netflow data. As you know I’m an open-source volunteer, so my by-default choices became the ones that can be found on GitHub. When I joined the project, <a href="https://github.com/phaag/nfdump"><strong>nfdump</strong></a> was used. It’s another open-source alternative yet it doesn’t support <strong>streaming data architecture</strong> (I’ll discuss that in incoming paragraphs) which is crucial to overcome matrix-like incoming data problems. nfdump collects data for given intervals (1 minute, 5 minutes, etc.) and divides it into small packages which require another tool to extract raw data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*EO0E5KzSG5bf4DTw.png" /></figure><p>To work with a similar aspect, CDR data was requested with intervals too. So, I prepared a benchmark to understand what are the reasons behind the congestion in preparing output data. Benchmark and report indicated working with the quantized data is the main issue. Finally, I realized that changing tools themselves will not save us and we need to make a radical change to adapt streaming data architecture.</p><p>The new stack arises with the following tools:<br>* Goflow2: <a href="https://github.com/netsampler/goflow2">https://github.com/netsampler/goflow2</a><br>* Apache Kafka: <a href="https://kafka.apache.org/">https://kafka.apache.org/</a><br>* KSQLDB: <a href="https://ksqldb.io/">https://ksqldb.io/</a><br>* Freeradius (not in my expertise, but I’ll mention why it’s good)<br>* of course some magical Python scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qFifSf-niHYtISPS.png" /></figure><p>Then, I proposed the architecture above. What I was thinking is basically, keeping the code in that ecosystem minimal and decreasing the need for human intervention to as low as possible. Also, the CDR part is going to be replaced with Freeradius and then we’ll get rid of another code, soon. I’m not the enemy of “codes”, yet I want to be as quick as possible and not to make any more delays. I chose KSQLDB for almost the same reason. KSQLDB helps us to join two Kafka topics just like SQL joins. Despite my first try at such a project, KSQLDB became the hearth of this architecture.</p><p>As you can guess, designing such an architecture needs also reliability. All the choices mentioned before have a common feature. <strong>They all are scalable.</strong> So, you can run as many as goflow2 simultaneously. Just like Kafka and KSQLDB.</p><p>Let me introduce you to the stack, one by one. <strong>Goflow2</strong> is a fork of <a href="https://github.com/cloudflare/goflow"><strong>Cloudflare’s Goflow</strong></a> project. It’s a Netflow/sFlow/IPFIX collector written via Go language. You can start as many as goflow2 instances in a server. What makes goflow2 a better option among all the other collectors? Transport options of course. You can ship incoming netflow data to lots of tools, such as Kafka, Elasticsearch, etc.</p><p>Now, it’s time for most data engineering parts of the architecture.. and those are Kafka and KSQLDB. <a href="https://kafka.apache.org/">Kafka</a> stores and redirects data through the topics. It’s probably the most commonly used tool all around the world for streaming data architectures. It’s worth that check it out when you need solutions for “data pipelines”, “streaming analytics” and more. <a href="https://ksqldb.io/">KSQLDB</a> plays a mission-critical role here. You can process Kafka topics in real-time with the native Kafka support and all you need to do is write some SQL queries. It’s super easy to create a new stream for merged data.. just organize your ordinary SQL query “SELECT * FROM” with some “WHERE blah blah” conditions and put “CREATE STREAM” at first.</p><pre>CREATE STREAM s3 AS<br>  SELECT s1.c1, s2.c2<br>  FROM s1<br>  WHERE s1.c1 &gt; 10<br>  JOIN s2 WITHIN 5 MINUTES<br>  ON s1.c1 = s2.c1 EMIT CHANGES;</pre><p>Previously, we had a pull-based method to get CDR data from accounting servers. As a part of modernization, accounting servers and applications are renewed with Freeradius. Once again I’d like to mention that, I don’t have any experience with those tools or concepts directly. Yet, I’m glad for being able to consume CDR data directly from Kafka topics with the help of some open-source plugins that can ship data on Freeradius to a given Kafka topic.</p><p>For logging, health-checking, and garbage collecting purposes I’ve sprinkled some Python scripts all over the architecture. Logging configurations are set to ERROR level and transferred to Elasticsearch just after parsing via Logstash.</p><p>To sum up, the architecture and tools mentioned above solved many issues on our side. I bypassed the installation and integration details since they all have pretty user-friendly documentation and also lots of items to configure for specific cases.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acdfc6c8dbaa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RRDtool — InfluxDB — Grafana / Read — Write — Monitor]]></title>
            <link>https://medium.com/@pathintegral/rrdtool-influxdb-grafana-read-write-monitor-9777fb04df59?source=rss-5c28904105f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9777fb04df59</guid>
            <category><![CDATA[rrdtool]]></category>
            <category><![CDATA[influxdb]]></category>
            <category><![CDATA[grafana]]></category>
            <dc:creator><![CDATA[mustafa can uslu]]></dc:creator>
            <pubDate>Sat, 27 Jul 2024 21:02:49 GMT</pubDate>
            <atom:updated>2024-07-27T21:13:43.190Z</atom:updated>
            <content:encoded><![CDATA[<p>During my first few weeks of working at a TELCO company, I was tasked with handling RRD files. RRD’s (round-robin database) are data storage files containing time-series data with ranging resolutions. Because of the way data is stored in RRD files, the system storage footprint is always constant. I was responsible for reading the data stored in these RRD files, writing them into our company’s InfluxDB and monitoring the data values through Grafana.</p><p>Due to the nature of the formatting, directly opening an RRD file will not yield any legible results, as it contains binarized and encrypted text. Utilizing RRDtool, we’re able to read RRD files in a more understandable format. We install this tool via:</p><pre>sudo apt install rrdtool</pre><p>After installation, we’re able to get the dump of the RRD as an XML and write it as an output:</p><pre>rrdtool dump [FILENAME].rrd &gt; test.xml</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*EK6EwbWuLFD2QYKh.png" /></figure><p>As shown on the right, the contents of the RRD file include definitions, values. The various time resolutions (300 seconds, 1800 seconds, 7200 seconds, etc.) are the time intervals, which are usually denoted as steps in most documentation. What this means is that if our step value is 300, then there are 300 seconds between each measurement.</p><p>The RRD file contains <strong>&lt;cf&gt;</strong> tags, which stand for consolidation function and will denote the operation that was made on the time-series data. The included functions are: AVERAGE, MIN, MAX, and LAST. For more on the consolidation functions, please refer to this <a href="https://www.loriotpro.com/Products/On-line_Documentation_V5/LoriotProDoc_EN/V22-RRD_Collector_RRD_Manager/V22-A1_Introduction_RRD_EN.htm">source</a>.</p><p>The <strong>&lt;ds&gt;</strong> tag represents the data source of the file. In the XML I’ve shown, we can see that there are 5 &lt;ds&gt; tags. When using a data source, more properties need to be specified. Further details can be found in the official RRDtool <a href="https://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html">documentation</a>.</p><p>And finally, we have the time-series data packed inside the database tag as follows:</p><pre>&lt;!-- 2020-12-16 07:30:00 +03 / 1608093000 --&gt;<br>&lt;row&gt;<br>&lt;v&gt;0.0000000000e+00&lt;/v&gt;<br>&lt;v&gt;1.1793301000e+01&lt;/v&gt;<br>&lt;v&gt;1.4160830000e+01&lt;/v&gt;<br>&lt;v&gt;1.3353590000e+00&lt;/v&gt;<br>&lt;v&gt;9.5392500000e+00&lt;/v&gt;<br>&lt;/row&gt;</pre><p>There are a limited number of lines in each resolution. For example, after 30 minutes have passed; the last line of the 1800 second step database will be replaced with a new entry. The newly inserted line is equivalent to 6 lines of the 300-second step database.</p><p>In my case, I needed to extract the last hour’s values of the AVERAGE cf in the 300-second step. In order to parse the RRD file to retrieve these values, I came up with two methods.</p><p>My first method was to parse the XML output itself, and the second was to use the Python binding, <strong>python-rrdtool</strong>. After experimenting and testing with both, using python-rrdtool ended up being much faster. As it yields far better results, I recommend that anyone wishing to parse RRD files use python-rrdtool.</p><p>The other two tools I utilized were InfluxDB and Grafana.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*eTgXL18QyfsuDs05.png" /></figure><p>Grafana is an open-source tool, used mainly for analytics and visualization. It has incredible flexibility and is compatible with almost any time-series database. When building a monitoring stack, Grafana as the visualization component is always my go-to choice.</p><p>InfluxDB is an open-source time-series database that aims to handle large amounts of write and query loads. Written in the programming language Go, InfluxDB is widely used for purposes such as monitoring real-time analytics, IoT sensor data, application metrics.</p><p>I was able to set up both of these open-source tools with relative ease thanks to Docker. To obtain their images, simply use the following lines:</p><pre>docker pull grafana/grafana<br>docker pull influxdb</pre><p>After obtaining the images, we run the containers as follows (default options shown below):</p><pre>docker run -d --name=grafana -p 3000:3000 grafana/grafana<br>docker run -d --name=influxdb -p 8086:8086 influxdb</pre><p>Now that both of the containers are up and ready. It’s time to create a new database to insert metrics in. To achieve that, please follow the codes below.</p><pre>docker exec -it influxdb bash<br>--&gt;<br>influx<br>-&gt;<br>CREATE DATABASE pings;<br>USE pings;<br>CREATE USER data WITH PASSWORD &#39;data&#39;;<br>GRANT ALL ON pings TO data;</pre><p>Initialization is completed. I have a table named as <strong>pings</strong>. I can reach it with the credentials that username:<strong>data</strong>and password:<strong>data</strong>. Now, I may use the Python script that reads RRD file and insert the extracted values into InfluxDB.</p><p>I implemented other components to this stack, but due to confidentiality reasons, I’m unable to share the code itself as of now. But I am able to offer some tips that will definitely make your life easier when faced with these problems that I ran into.</p><ol><li>The RRD file doesn’t provide an explicit timestamp when parsing via Python. Therefore, you need to extract the starting or ending timestamp from the XML data. From there, you’re able to create a range by incrementing the extracted timestamp by the time step. (i.e. if your step is 300, then add/subtract 300 seconds from your timestamp incrementally)</li><li>Convert timestamps to UTC to produce a better visualization and support various data source locations. After inserting into InfluxDB, you’re able to set timezones on Grafana to prevent any detrimental changes to your data.</li><li>Write your insertion code as a function in order to use it frequently as needed.</li></ol><p>After inserting successfully, let’s visualize our data points from InfluxDB on a Grafana dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/0*4w1pP6zgeRafMzVZ.png" /></figure><p>Navigate to the location of your Grafana interface on your browser. Using the default ports, it should be at <a href="http://localhost:3000/"><strong>http://localhost:3000</strong></a></p><p>You can see <strong>Configuration</strong> on the left sidebar.</p><p>On the <strong>Configuration</strong> sidebar on the left, click on <strong>Data Sources</strong> and choose <strong>InfluxDB</strong>. As mentioned before, you’ll be able to see all the different databases that Grafana supports.</p><p>Fill in the required fields (if you initialized using the same information as me, then the following should be the same).</p><p><strong>HTTP &gt; URL:</strong> <a href="http://172.17.0.3:8086/">http://172.17.0.3:8086</a> (This URL gathered from Docker container itself. You can attach to the terminal via<strong> docker exec -it influxdb bash</strong> and check the IP address with <strong>hostname -I</strong>)</p><p><strong>InfluxDB Details &gt; Database:</strong> pings<br><strong>InfluxDB Details &gt; User:</strong> data<br><strong>InfluxDB Details &gt; Password:</strong> data</p><p>Hit <strong>Save &amp; Test</strong>. If everything went well, there shouldn’t be any errors.</p><p>We’re almost done with our RRDtool — InfluxDB — Grafana stack. Lastly, create a dashboard by going to <strong>Dashboards &gt; Manage</strong> from the left sidebar and add a panel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/0*JFeueQ_hlZdwmojy.png" /></figure><p>Select your query parameters (don’t forget to indicate your timezone)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/0*HXTSjOPqDzyTTlwk.png" /></figure><p>As you can see, the data comes from RRD files processed by a Python script and is then inserted to InfluxDB. From there, the InfluxDB data is shown on our Grafana dashboard we just made. If you want streamed data, you’ll need to set a cronjob to periodically run your Python script or daemonize your script via some time-related rules and Supervisord.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*-H0hp_fQODEy-9UY.png" /></figure><p>There is definitely room for improvement, but fundamentally this is how I approached my monitoring stack.</p><p>Additionally, I incorporated <strong>Tableau </strong>into this stack to generate reports and further visualization. I plan to write more about this in my next blog post, which may include more details from this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9777fb04df59" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>