<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Igor Gorbenko on Medium]]></title>
        <description><![CDATA[Stories by Igor Gorbenko on Medium]]></description>
        <link>https://medium.com/@igorvgorbenko?source=rss-b79c227003cc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Sk3gCUSCeogsWWsUziEY4g.jpeg</url>
            <title>Stories by Igor Gorbenko on Medium</title>
            <link>https://medium.com/@igorvgorbenko?source=rss-b79c227003cc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:46:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@igorvgorbenko/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your AI Product Is Already Unsafe. The Incident Just Hasn’t Happened Yet]]></title>
            <link>https://medium.com/@igorvgorbenko/your-ai-product-is-already-unsafe-the-incident-just-hasnt-happened-yet-f3559e3ed9d9?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3559e3ed9d9</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Wed, 15 Jul 2026 14:32:04 GMT</pubDate>
            <atom:updated>2026-07-15T14:32:04.860Z</atom:updated>
            <content:encoded><![CDATA[<p>Why security and compliance must be built into AI from day one</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gGEDVP6oES5mFHzVUm8SgA.png" /></figure><p>In February 2024, a Canadian <a href="https://www.cbsnews.com/news/aircanada-chatbot-discount-customer/">tribunal ordered Air Canada to pay a passenger $812.02(CAD)</a> after the airline’s chatbot provided incorrect information about its bereavement fare policy. Air Canada attempted to distance itself from the chatbot’s response. The tribunal rejected that argument: the chatbot was part of the airline’s website, so Air Canada remained responsible for the information it published.</p><p>In August 2025, researchers used a prompt of roughly 400 characters to make Lenovo’s Lena chatbot generate malicious HTML. Because the AI-generated output was not properly validated or sanitized, the code was executed as cross-site scripting, or XSS. During a controlled security test, the researchers were able to intercept active session cookies, including cookies belonging to support agents who opened the compromised conversation. Lenovo confirmed the vulnerability and said it had taken steps to address it.</p><p>Then, in May 2025, the Italian data protection authority fined Luka Inc., the company behind the AI companion Replika, €5 million for GDPR violations. The issues included the absence of an appropriate legal basis for processing personal data, deficiencies in its privacy policy, and inadequate age-verification measures.</p><p>Three different failures:</p><ol><li>An incorrect promise.</li><li>A vulnerability that enabled session interception through AI-generated output.</li><li>And a violation of data protection law.</li></ol><blockquote>What connects them is not “bad AI.”</blockquote><blockquote>It is the absence of built-in boundaries defining what the system may say, what data it may process, and what actions it may initiate.</blockquote><h3>The Trap Almost Every Founder Falls Into</h3><p>The startup logic is understandable. At first glance, it even sounds rational:</p><blockquote>“First we find product-market fit. Everything else comes later. Security is a problem for companies that already have something to lose.”</blockquote><p>With AI, that logic breaks down in three places.</p><h4>First, you have something to lose from the very first user</h4><p>In an AI product, data moves through a complex processing chain. Prompts, documents, and conversations may be sent to the model, external providers, logging systems, and observability platforms.</p><p>A data leak does not require scale.</p><p>It can happen on launch day.</p><h4><strong>Second</strong>, AI introduces categories of attack that do not appear in most traditional security checklists</h4><ul><li>prompt injection;</li><li>training-data poisoning;</li><li>jailbreaks;</li><li>system-prompt extraction;</li><li>and inaccurate outputs with legal or financial consequences.</li></ul><p>A security checklist created before the widespread adoption of large language models is no longer sufficient.</p><h4><strong>Third</strong>, regulators are no longer waiting</h4><p>The EU AI Act is being introduced in stages. Prohibitions on certain AI practices and AI-literacy requirements have applied since February 2025. Rules for general-purpose AI have applied since August 2025.</p><p>On 2 August 2026, a significant portion of the remaining provisions begins to apply, including several transparency requirements.</p><p>Later deadlines apply to high-risk systems. Rules for certain systems used in biometrics, critical infrastructure, education, recruitment, and other sensitive areas are scheduled to apply from 2 December 2027.</p><p><a href="https://www.hklaw.com/en/insights/publications/2026/04/us-companies-face-eu-ai-acts-possible-august-2026-compliance-deadline">For AI embedded in regulated products, the relevant deadline is 2 August 2028.</a> Depending on the violation, penalties <a href="https://artificialintelligenceact.eu/article/99/">may reach tens of millions of euros or be calculated as a percentage of a company’s global annual turnover</a>. The central requirements of the EU AI Act are no longer a distant-future concern.</p><h3>The Numbers Your Board Should See</h3><p>IBM’s <a href="https://www.ibm.com/think/insights/data-matters/cost-of-a-data-breach">Cost of a Data Breach Report 2025</a> examined shadow-AI risks separately for the first time.</p><p>Among the 600 organizations studied that had already experienced a data breach, one in five reported an incident involving unauthorized AI use.</p><p>In organizations with high levels of shadow AI, the average cost of a breach <a href="https://www.deepinspect.ai/blog/shadow-ai-breach-cost">was $670,000 higher </a>than in companies with little or no shadow-AI activity.</p><ul><li>Personal data was compromised in 65% of shadow-AI-related incidents.</li><li>Among organizations that reported security incidents involving AI models or applications, 97% lacked proper access controls.</li><li>And 63% of organizations that had experienced a breach either had no AI governance policy or were still developing one.</li></ul><h3>“Build It In From Day One” Is Cheaper, Not More Expensive</h3><p>This may sound counterintuitive, so let’s be honest about it.</p><blockquote>Fixing an architectural flaw in Production usually costs significantly more than preventing it during the design phase.</blockquote><p>With AI, however, there is an additional problem: retrofitting controls may reduce future risk, but <strong>it cannot undo data processing that has already taken place</strong>.</p><p>If a model was trained on data the company had no legal right to use, a simple patch may not be enough. The organization may need to remove datasets, replace the model, or retrain it.</p><p>If user prompts were sent to an external provider for months without the necessary contractual or legal basis, signing new documents will not make the previous processing lawful.</p><p>If the system prompt was the only line of defence, the solution is not better wording.</p><p>The architecture itself must change.</p><p>Security debt can sometimes be paid down gradually.</p><p>The consequences of unlawful data processing and completed security incidents cannot.</p><p>That is why basic constraints are cheaper to establish in advance.</p><p>Minimizing logs, classifying data, masking personally identifiable information, and separating what the model knows from what the model is allowed to do are all easier before launch than after customers, integrations, and historical data have accumulated.</p><h3>What “From Day One” Actually Means</h3><p>You do not need a CISO during the MVP stage.</p><p>You need five habits, each of which can be introduced in a basic form during the first few sprints.</p><h4>1. Create a Data Map Before Writing the First Line of Code</h4><blockquote>Identify what data enters the model, where it goes, where it is stored, and which third parties participate in processing it.</blockquote><p>For an MVP, a single page may be enough.</p><p>When a regulator or enterprise customer asks — and eventually they will — you will have an answer instead of a panic-driven investigation.</p><h4>2. Treat the Model as an Untrusted Component</h4><p>Design the system as though the LLM were an intern who had read the entire internet:</p><p>Useful, fast, and occasionally impressive — but not someone you would allow to access company funds or a production database without supervision.</p><blockquote>Everything the model “decides” should be validated at the system boundary.</blockquote><p>The Air Canada case demonstrates why there must be a controlled boundary between generating an answer and making a statement on behalf of the company.</p><p>From the customer’s perspective, the chatbot was part of the airline’s website. The tribunal therefore refused to treat it as an independent entity whose statements were somehow separate from the company.</p><h4>3. Put Adversarial Testing Into the Test Plan From the First Release</h4><p>The Lenovo case exposed two failures at once.</p><p>First, the model could be manipulated into generating malicious output.</p><p>Second, a traditional sanitization flaw turned that output into executable XSS.</p><blockquote>Build your own library of adversarial prompts and run it in CI as a regression-test suite.</blockquote><p>The first version can be assembled quickly, but it must evolve with the product and run whenever you change:</p><ul><li>the model;</li><li>system prompts;</li><li>connected tools;</li><li>data sources;</li><li>or response-processing logic.</li></ul><p>Testing should not stop at prompt injection.</p><p>You need to test the entire response-processing chain, including output formats, escaping, allowed content types, and the actions available to the client application.</p><h4>4. Create an AI Policy for Your Own Team</h4><p>IBM found that 63% of organizations that had experienced a breach either had no AI governance policy or were still developing one.</p><blockquote>To begin, a one-page policy is enough.</blockquote><p>It should explain:</p><ul><li>what employees may send to external AI tools;</li><li>what they must never send;</li><li>and which tools are approved.</li></ul><p>You do not want your own marketing team to turn the company into another shadow-AI statistic.</p><h4>5. Treat Compliance as a Product Feature, Not a Tax</h4><p>SOC 2 readiness, data protection measures, and GDPR preparedness are already being evaluated by many enterprise customers during early security reviews and due diligence.</p><p>Startups that already have credible answers can close deals while competitors are still forming committees.</p><p>Security is not what prevents you from selling.</p><p>Security is what helps you sell.</p><h3>The Objection You Are Probably Formulating Right Now</h3><blockquote>“But what about speed? While we are writing policies, our competitor is shipping features.”</blockquote><p>Basic controls can begin in the first sprint.</p><p>Maturity will take longer, but that becomes a managed evolution of the system rather than an emergency rebuild after an incident.</p><p>Speed without brakes is not speed.</p><p>It is free fall — and free fall feels exciting right until the moment of impact.</p><h3>Final Takeaway</h3><p>AI products do not fail because the models are inherently bad.</p><p>They fail because there is a gap between “the demo works” and “this system can be trusted with real data.”</p><p>Most teams discover that gap while they are already in the air.</p><p>At the same time, a basic security foundation can be established in weeks, not years.</p><p>Security from day one is not perfectionism.</p><p>It is not paranoia.</p><p>It is the only way to build an AI product capable of surviving its own success.</p><p>Your product is already talking to users.</p><blockquote>The only question is: who else is listening?</blockquote><p>If you found this useful, leave a 👏 and follow me. I write about building Data/AI products you can confidently show to both customers and regulators.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3559e3ed9d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An AI Copilot for Auditors and CISOs]]></title>
            <link>https://medium.com/@igorvgorbenko/an-ai-copilot-for-auditors-and-cisos-8514a2540449?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8514a2540449</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[audit]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Sun, 14 Jun 2026 10:05:56 GMT</pubDate>
            <atom:updated>2026-06-14T10:05:56.672Z</atom:updated>
            <content:encoded><![CDATA[<p>Why the Future of Compliance AI Is Reasoning over Organizational Knowledge</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xvs210Y3xQ_4m0GA6u4YMQ.png" /></figure><p>Compliance complexity has exploded. A decade ago, organizations managed a handful of frameworks. Today they must navigate PCI DSS, ISO 27001, GDPR, DORA, NIS2, SWIFT CSP, and many others — often simultaneously.</p><p>Yet the process remains largely unchanged. Auditors still collect evidence manually, maintain spreadsheets, and reconcile controls across disconnected systems. CISOs still spend weeks assembling a compliance picture from scattered data sources.</p><p>This approach no longer scales. As regulatory requirements grow, manual compliance becomes increasingly expensive, slow, and outdated.</p><p>The industry’s answer has been AI. But most organizations are solving the wrong problem.</p><h3>Why Most Compliance AI Projects Fail</h3><p>If you look under the hood of the overwhelming majority of solutions sold today under the “AI for compliance” label, they share the same architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JaJ06A3aPBt-u3P7GUHb9g.png" /></figure><p>This is classic RAG (Retrieval-Augmented Generation): documents are split into chunks, turned into vectors, stored in an index, and at answer time the LLM pulls the most similar chunks and phrases a response. For a use case like “find me the paragraph about password policy in the security policy,” this works beautifully.</p><p>But compliance is not a text-search problem. And here is why such projects hit a ceiling time after time:</p><h4>Documents Capture Intent. Graphs Capture Reality</h4><p>An access-management policy describes how things <em>should</em> be. It says nothing about how they actually are: which accounts truly hold privileged access, when permissions were last reviewed, and whether there are “orphaned” accounts of departed employees. RAG over the policy will hand you an elegant quote that has nothing to do with the actual risk.</p><h4>Embedding Relevance Is Not Evidential Connectivity</h4><p>Vector search finds semantically similar text. But compliance with PCI DSS 8.4.2 (multi-factor authentication for all access into the CDE) is proven not by a similar paragraph but by a concrete chain: the IdP configuration + the list of in-scope CDE systems + the MFA test log + the date of the last check. None of these pieces of evidence is semantically “similar” to the text of the requirement.</p><h4>Documents Go Stale. A Live Knowledge Graph Stays Current</h4><p>A vulnerability scan from last Tuesday, sitting as a PDF in a vector database, becomes misinformation a week later. The “chat with documents” architecture has no notion of time, versioning, or the lifecycle of evidence.</p><h4>Hallucinations Are Expensive in Compliance</h4><p>When an LLM “fills in” a plausible but non-existent control, that’s harmless in marketing copy. In a report for a SWIFT assessor or a PCI QSA, <strong>it is a falsification</strong> of the evidence base.</p><blockquote>The root of the problem is simple: document-oriented AI automates reading, whereas auditors and CISOs are not occupied with reading but with reasoning about the organization&#39;s state.</blockquote><h3>Auditors Do Not Work with Documents</h3><p>To understand what AI is actually required, we must first understand how auditors and CISOs work. Documents are only containers. The real work revolves around entities and their relationships.</p><ul><li><strong>Requirements</strong> define obligations.</li><li><strong>Controls</strong> implement requirements.</li><li><strong>Policies</strong> formalize intent and governance.</li><li><strong>Evidence</strong> proves that controls operate effectively.</li><li><strong>Assets</strong> are the systems and data those controls protect.</li><li><strong>Vulnerabilities</strong> expose weaknesses in assets.</li><li><strong>Findings</strong> capture gaps between expected and actual state.</li><li><strong>Remediation tasks</strong> close those gaps.</li></ul><p>These entities do not exist in isolation. They form a connected network of relationships.</p><p>The auditor’s daily work is navigating that network:</p><p><strong>Requirement → Control → Asset → Evidence → Vulnerability → Finding → Remediation</strong></p><p>For example:</p><p><em>“Requirement 8.4.2 → Which control implements it? → Which assets are covered? → What evidence proves effectiveness? → Is that evidence current? → Are there vulnerabilities undermining the control? → If a finding exists, who owns remediation and when is it due?”</em></p><p>None of these questions are document searches. They are graph traversals.</p><p>This is why “chat with a PDF” reaches a hard limit. Documents store information, but auditors reason about relationships. The problem is not finding text — it is understanding how requirements, controls, assets, evidence, risks, and findings connect to one another.</p><h3>A Day in the Life of an Auditor</h3><blockquote>Consider a typical PCI DSS assessment. A payment provider has 400 servers, roughly 60 of them within the Cardholder Data Environment (CDE).</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYMcz-uXxXXJol4tqq7suA.png" /></figure><h4><strong>Days 1–2. Scoping</strong></h4><p>The auditor starts with a simple question: <em>where does cardholder data actually flow?</em></p><p>The company provides a dataflow diagram and a CMDB export. The CMDB shows 60 in-scope systems; the diagram shows 54. After several emails and follow-ups, it turns out that three systems were recently added and three were decommissioned. A day and a half is spent reconciling two versions of the same asset inventory.</p><h4><strong>Days 3–8. Evidence </strong>Collection</h4><p>The auditor works through roughly 300 requirements.</p><p>For PCI DSS 8.4.2 (MFA), evidence comes from three different teams: IdP configuration, system inventory, and testing records. Each arrives in a different format — screenshots, CSV exports, PDFs. The auditor manually organizes the evidence and updates tracking spreadsheets.</p><p>Repeat this process hundreds of times.</p><h4><strong>Days 9–11. Testing and </strong>Sampling</h4><p>The auditor samples 10 systems and asks:</p><ul><li><em>Is the control applicable?</em></li><li><em>Is evidence available?</em></li><li><em>Is it current?</em></li><li><em>Are there vulnerabilities that undermine it?</em></li></ul><p>A critical CVE appears on one sampled server. To understand its compliance impact, the auditor must manually correlate data from the vulnerability scanner, CMDB, and GRC system. The answer takes about an hour to find.</p><h4><strong>Days 12–14. Findings</strong></h4><p>A review reveals that centralized logging is enabled on only 57 of 60 systems. The three missing systems are the same replicas overlooked during scoping.</p><p>The auditor creates a finding, links it to the requirement, recommends remediation, waits for evidence of the fix, and verifies it manually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OojZ6ZT-3AyQks4SRqkcKw.png" /></figure><h4>The Real Problem</h4><p>At no point was the auditor simply “reading documents.” They were continuously connecting assets, requirements, controls, evidence, vulnerabilities, findings, and remediation tasks across multiple systems.</p><p>In other words, they were manually traversing a graph that does not physically exist.</p><p>A live Compliance Knowledge Graph changes this entirely. Asset discrepancies are detected automatically. Evidence is linked to requirements. Vulnerabilities are already connected to affected controls and assets. Findings emerge as soon as control coverage breaks.</p><p>Two weeks of manually rebuilding relationships become a process of reviewing and validating a graph that is continuously maintained.</p><h3>The Concept of an AI Copilot for Auditors and CISOs</h3><p>The difference between a chatbot and a copilot is not cosmetic — <strong>it is architectural.</strong></p><p>Traditional document-centric <strong>chatbots</strong> are primarily reactive. They answer questions using documents as their source of knowledge.</p><p>A <strong>copilot</strong> is context-aware. It operates on top of a Compliance Knowledge Graph representing the organization and works with entities such as assets, controls, risks, findings, and evidence.</p><blockquote><strong>The copilot does not make compliance decisions</strong>. Instead, it gathers context, correlates data across systems, identifies gaps, and prepares recommendations. The auditor or CISO reviews the results and makes the final decision.</blockquote><p>This is not simply a safety measure — it reflects the reality of regulated environments. Compliance responsibility remains with the human expert.</p><p>The value of the copilot is straightforward: it automates the tedious work of collecting, correlating, and analyzing information, reducing weeks of manual effort to minutes while leaving judgment and accountability where they belong.</p><h3>Building a Compliance Knowledge Graph</h3><p>The foundation of a copilot is not a vector database of documents but a structured model of the organization’s knowledge. Documents are merely one of the sources from which entities and relationships are extracted.</p><p>The minimal relationship model looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aTWKW0zJlT9XxLejOqUQMg.png" /></figure><p>A knowledge graph enables questions that traditional document-based systems cannot answer efficiently.</p><p>For example:</p><ul><li>Which PCI DSS requirements depend on controls whose evidence is older than 90 days?</li><li>Which in-scope assets have critical vulnerabilities that undermine compliance controls?</li><li>If a server is decommissioned, which requirements and standards lose coverage?</li></ul><p>These are not text-search problems. They are relationship problems.</p><p>A second advantage is <strong>describe once, comply many</strong>. A single control, once implemented and evidenced, can satisfy requirements across PCI DSS, ISO 27001, DORA, and other frameworks simultaneously. There is no need to recreate the same mapping for every standard.</p><p>A knowledge graph transforms compliance from a collection of documents and spreadsheets into a unified model of the organization — one that can be queried, analyzed, and continuously maintained.</p><blockquote>An important point is that the knowledge graph remains valuable even without AI. It serves as a structured, continuously updated model of the organization’s compliance state. The LLM does not create this knowledge — it simply makes it easier to query, analyze, and explain. The graph is the system of record; AI is the interface.</blockquote><h3>Enterprise AI Copilot Reference Architecture</h3><p>A full-fledged copilot is a multi-layer system in which the LLM occupies just one layer out of seven. Below is a reference architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BB7jSza5Mkb7k-nIIxmOsA.png" /></figure><h4><strong>Layer 1 — Data Ingestion</strong></h4><p>Compliance data lives across CMDBs, scanners, SIEMs, IAM systems, tickets, cloud platforms, and documents. Without direct access to these sources, a copilot cannot reason about reality.</p><h4><strong>Layer 2 — Document Processing</strong></h4><p>Documents are transformed into structured entities, relationships, controls, evidence, and vulnerabilities.</p><h4><strong>Layer 3 — </strong>Compliance Knowledge Graph (Core)</h4><p>A unified, versioned knowledge graph representing the organization’s compliance state. This is the foundation of the entire system.</p><h4>Layer 4 — Private LLM</h4><p>A self-hosted reasoning model operating within the organization’s security perimeter.</p><h4>Layer 5 — Retrieval</h4><p>Hybrid retrieval combining graph queries, structured filters, and vector search. Graph traversal provides facts; vectors provide semantic matching.</p><h4>Layer 6 — Reasoning and Orchestration</h4><p>Plans tasks, invokes tools, applies rules, validates facts, and synthesizes conclusions grounded in the graph.</p><h4>Layer 7 — User Interaction</h4><p>Dashboards, workbenches, and chat interfaces. Every conclusion must remain fully traceable to requirements, controls, evidence, and source systems.</p><h3>Practical Use Cases</h3><p>The value becomes clear in real-world tasks:</p><h4>Audit Findings</h4><p>Automatically identifies missing or stale evidence and drafts findings for auditor review.</p><h4>Evidence Gap Analysis</h4><p>Shows which controls lack evidence, where evidence is outdated, and where coverage is incomplete.</p><h4>Compliance Gap Assessment</h4><p>Maps new regulations to existing controls and highlights only the true compliance gaps.</p><h4>Control Coverage Analysis</h4><p>Answers which requirements depend on a control and what coverage is lost if that control fails.</p><h4>Vulnerability Impact Analysis</h4><p>Links vulnerabilities to assets, controls, requirements, and findings in a single workflow.</p><h4>Executive Reporting</h4><p>Generates real-time readiness, risk, and compliance status directly from the live knowledge graph.</p><h4>Audit Interview Assistance</h4><p>Provides auditors with relevant requirements, evidence, open questions, and prior observations during interviews.</p><h3>Why Private AI Matters</h3><p>For regulated industries, the question is not how powerful an AI model is, but where it runs and where data goes.</p><p>Compliance data often contains some of the organization’s most sensitive operational and security information: assets, vulnerabilities, findings, controls, and security gaps. Exposing this information outside the trusted perimeter is often restricted by regulation and considered unacceptable from a security standpoint.</p><p>Additional constraints come from GDPR, data-sovereignty laws, banking secrecy requirements, SWIFT CSP, and PCI DSS, all of which impose strict controls on data handling, access, and auditability.</p><p>As a result, most enterprises prefer private-cloud or on-premise AI deployments. Models operate within the organization’s perimeter, data never leaves the trusted zone, prompts and responses remain under organizational control, and security controls such as logging, guardrails, and DLP can be enforced consistently.</p><blockquote>For compliance AI, private deployment is not a preference — it is a requirement.</blockquote><h3>The Role of LLMs in Compliance</h3><p>One of the most common misconceptions about compliance AI is that the LLM itself is the product.</p><p><strong>It is not.</strong></p><blockquote>In a copilot architecture, the LLM is just one layer — and one of the most replaceable layers. Models evolve rapidly; today’s leader may be obsolete in six months. A well-designed system can swap models without changing the rest of the architecture.</blockquote><p>The real asset is organizational knowledge: the graph of requirements, controls, assets, evidence, findings, and their relationships. That knowledge is built over years and represents the organization’s true intellectual property.</p><p>The value comes from the structure and the links, not from the model. The ability to understand that a vulnerability impacts a control, which supports multiple requirements across several standards, comes from the knowledge graph. The LLM simply explains and reasons over that information.</p><p>Its role is straightforward: translate user intent into queries, orchestrate analysis, synthesize results, and generate human-readable explanations.</p><ul><li>The graph organizes and relates the facts</li><li>The LLM reasons over them</li></ul><p>For architects, the implication is clear: invest in the ontology, entity extraction, and knowledge model. Models will change. Your knowledge graph is what creates long-term value.</p><h3>Continuous Compliance: The Future State</h3><p>Today’s compliance model is periodic and reactive. Auditors arrive once or twice a year, and organizations rush to prove compliance for a point in time that has already passed.</p><blockquote>The future is <strong>continuous compliance</strong> — a living compliance model that updates in real time.</blockquote><p>When something changes — a new vulnerability appears, evidence expires, a configuration drifts, or a control fails — the Compliance Knowledge Graph updates immediately and triggers the appropriate response: risk recalculation, finding creation, owner notification, or remediation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M42MI5IcIFsTmg9n8T1WHw.png" /></figure><p>In this model:</p><ul><li><strong>Compliance becomes a live metric</strong>, not an annual report.</li><li><strong>Vulnerabilities automatically affect risk assessments</strong> and compliance status.</li><li><strong>Evidence freshness is monitored continuously</strong>, with gaps identified before an audit.</li><li><strong>Control failures are detected as they occur</strong>, not months later.</li><li><strong>Audits become validation exercises rather than discovery exercises.</strong></li></ul><p>This is the evolution of compliance: from a reactive, document-centric process to a proactive, continuously monitored system built on live organizational data.</p><h3>Conclusion</h3><p>Most of today’s attempts to “apply AI to compliance” solve the wrong problem. They automate the reading of documents, whereas auditors and CISOs are occupied with reasoning about the state of the organization. “Chat with a PDF” is a local maximum whose ceiling is set by a fundamental mismatch between what documents hold and how security and compliance professionals actually work.</p><p>The real breakthrough lies not in a smarter model but in the right architecture: a structured model of organizational knowledge at the core, hybrid retrieval and reasoning on top of it, a private LLM as a replaceable interpretation engine, and a human in the decision loop. Models come and go. The Compliance Knowledge Graph of controls, evidence, assets, and risks remains — and that is the asset.</p><blockquote>The future of compliance AI is not a smarter chatbot. The future of compliance AI is a reasoning layer built on top of organizational knowledge, controls, evidence, risks, and security data.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8514a2540449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How We Built a Route Safety Engine Using CatBoost, Open Data, and ClickHouse]]></title>
            <link>https://python.plainenglish.io/how-we-built-a-route-safety-engine-using-catboost-open-data-and-clickhouse-5f39a046b222?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f39a046b222</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clickhouse]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Mon, 26 May 2025 05:25:17 GMT</pubDate>
            <atom:updated>2025-05-27T05:59:23.135Z</atom:updated>
            <content:encoded><![CDATA[<h4>From accident prediction to real-time risk-aware routing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*437r0fe3hc7oFTTU" /></figure><p><strong>Road safety isn’t just a policy issue anymore — it’s a data problem.</strong></p><p>As cities grow denser and traffic incidents become more frequent, traditional measures like signage and patrols are no longer enough to keep people safe. In the chaotic pulse of a modern megacity, we need faster, smarter, and more adaptive solutions.</p><p><strong>Imagine a system that doesn’t just show you how to get from A to B — but how to get there safely.</strong></p><p>A system that analyzes real-time conditions, accident history, and behavioral patterns to recommend routes that actively minimize risk. This kind of intelligence could become a vital tool for drivers, delivery fleets, tourists, and urban planners alike.</p><p>In this article, I’ll walk you through how to build such a system — a <strong>data-driven safety engine</strong> powered by open datasets and a machine learning model built with <strong>CatBoost</strong>. You’ll see how to train a model on traffic incident data, integrate it into real-world applications using <strong>ClickHouse</strong>, and visualize risk-aware routing in action.</p><p><strong>This project is a step toward building smarter cities — where data and AI make everyday movement not just more efficient, but genuinely safer.</strong></p><h3>Why Traditional Approaches Aren’t Enough</h3><p>Current road safety systems rely on:</p><ul><li><strong>Signs and signals</strong>: Static visual indicators, often outdated or insufficient</li><li><strong>Patrols</strong>: Limited by manpower and area coverage</li><li><strong>Public accident maps</strong>: Useful but require manual analysis and don’t offer recommendations</li></ul><p>These methods have clear limitations:</p><ul><li>They don’t account for real-time or contextual factors</li><li>They can’t adapt to changing conditions on the fly</li><li>They offer no personalized suggestions</li></ul><h3>The Case for a Data Product</h3><h4>Open Data Availability</h4><p>With platforms like <a href="https://www.dubaipulse.gov.ae/"><strong>Dubai Pulse</strong></a>, we now have access to vast datasets, including detailed traffic accident records — covering <strong>location, time, and incident characteristics</strong>. This provides a strong foundation for risk analysis and prediction.</p><h4>Advances in Technology</h4><p>Modern machine learning algorithms like <strong>CatBoost</strong> can handle large volumes of data and capture complex feature interactions. At the same time, columnar databases such as <strong>ClickHouse</strong> play a vital role in analytics. While ClickHouse isn’t designed for real-time prediction, it’s excellent for hypothesis testing and retrospective model evaluation.</p><p>For instance, a trained model can be deployed in ClickHouse via the catboostEvaluate function, allowing analysts to assess route risks and test data-driven safety hypotheses.</p><h4>Rising Demand for Safety</h4><p>Users are increasingly aware of road safety and are often willing to take slightly longer routes if they offer <strong>greater safety</strong>. This creates growing demand for solutions that combine <strong>convenience with peace of mind</strong>.</p><h3>Example Use Case</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1MLth_86zZrelcJW" /></figure><p>Maria, a young mother, is planning a trip across the city with her children. After entering the start and end points into the app, she receives not just the fastest route — but one that’s been carefully analyzed for safety. The app takes into account accident history, seasonal and temporal patterns, and correlates them with other relevant features to suggest a safer alternative — even if it’s slightly longer.</p><p>Now, Maria can travel with confidence, knowing her family is taking a route that minimizes risk.</p><h3>Data Sources</h3><h4>Source of Data</h4><p>The model was built using open datasets from <a href="https://www.dubaipulse.gov.ae/"><strong>Dubai Pulse</strong></a>, which provides access to a wide variety of municipal data, including detailed traffic accident statistics. It’s a goldmine for analysts and data enthusiasts interested in building practical solutions.</p><h4>Dataset Description:</h4><p><strong>Timeframe</strong>: Several months of incident data</p><p><strong>Content</strong>:</p><ul><li>Geolocation of accidents</li><li>Date and time of incidents</li><li>Descriptions in Arabic, including severity levels</li></ul><p>This information forms the basis for feature engineering — helping the model identify and predict high-risk areas along a route.</p><h3>Feature Engineering for the Model</h3><h4>Date and time processing</h4><p>The dataset is already a cleaned and structured collection, ready for analysis. While the incident descriptions contain rich information (e.g., severity of the crash, type of vehicle involved, etc.), this particular study focuses on analyzing the <strong>seasonal patterns of traffic accidents in specific geolocations</strong>.</p><p>To support this, we extract key features from each incident’s timestamp:</p><pre>df[&#39;acci_time&#39;] = pd.to_datetime(df[&#39;acci_time&#39;], dayfirst=True)<br><br>df[&#39;day&#39;] = df[&#39;acci_time&#39;].dt.day<br>df[&#39;month&#39;] = df[&#39;acci_time&#39;].dt.month<br>df[&#39;year&#39;] = df[&#39;acci_time&#39;].dt.year<br>df[&#39;hour&#39;] = df[&#39;acci_time&#39;].dt.hour<br>df[&#39;day_of_week&#39;] = pd.to_datetime(df[[&#39;year&#39;, &#39;month&#39;, &#39;day&#39;]]).dt.weekday</pre><h4>Translating Accident Types and Extracting Severity</h4><p>The incident descriptions are originally written in Arabic. To make them accessible for analysis, we use <strong>GoogleTranslator</strong> to translate all unique descriptions into English and normalize the severity levels into two categories: <em>minor</em> and <em>severe</em>.</p><pre>acci_name = df[&#39;acci_name&#39;].unique()<br>acci_name_translated = {}<br><br>for acci in acci_name:<br>    acci_name_translated[acci] = GoogleTranslator(source=&#39;ar&#39;, target=&#39;en&#39;).translate(acci)<br><br>severity = {<br>    &#39;baligh&#39;: &#39;severe&#39;,<br>    &#39;eloquent&#39;: &#39;severe&#39;,<br>    &#39;minor&#39;: &#39;minor&#39;,<br>    &#39;severe&#39;: &#39;severe&#39;,<br>    &#39;simple&#39;: &#39;minor&#39;<br>}</pre><p>As previously mentioned, the textual description of an incident contains a lot of valuable insights. For instance, we can <strong>enrich the dataset by identifying specific accident types</strong>, such as those involving motorcycles:</p><pre>df[&#39;acci_name_en&#39;] = df[&#39;acci_name&#39;].map(<br>    lambda x: acci_name_formatted[x][&#39;description&#39;] if x in acci_name_formatted else None<br>)<br><br>df[&#39;severity&#39;] = df[&#39;acci_name&#39;].map(<br>    lambda x: acci_name_formatted[x][&#39;severity&#39;] if x in acci_name_formatted else None<br>)<br><br>df[&#39;is_motorcycle&#39;] = df[&#39;acci_name_en&#39;].apply(<br>    lambda x: 1 if &#39;motorcycle&#39; in x.lower() or &#39;bike&#39; in x.lower() else 0<br>)</pre><h4>Geocoding with Geohash</h4><p>Instead of using raw latitude and longitude coordinates, we apply <strong>geohashing</strong> — a widely-used method of encoding geographic locations into strings of a defined length.</p><blockquote>The more characters in the geohash, the higher the precision.</blockquote><p>For this project, we use geohash_7, which provides a <strong>balanced level of precision</strong> while also grouping nearby incidents together for approximate spatial analysis.</p><pre>def to_geohash(lat, lon, precision=7):<br>    return geohash2.encode(lat, lon, precision=precision)<br><br>df[&#39;geohash_7&#39;] = df.apply(<br>    lambda row: to_geohash(row[&#39;acci_x&#39;], row[&#39;acci_y&#39;], precision=7),<br>    axis=1<br>)</pre><h4>Selecting Data for the Model</h4><p>For the first model version, it’s important to balance data volume with computational efficiency. Since this is a <strong>Proof of Concept (PoC)</strong> phase, we focus on incidents from the <strong>last 5 months</strong>, which is sufficient to capture meaningful patterns for model training.</p><pre>df_sub = df[<br>    (df[&#39;year&#39;].isin([2025])) &amp;<br>    (df[&#39;month&#39;].isin([1, 2, 3, 4, 5]))<br>]</pre><blockquote>The more historical data is available, the more accurate the model can become. However, during early prototyping, it’s crucial to optimize resource usage to ensure fast iteration and delivery of a working first version.</blockquote><h3>Visualizing Incidents on the Map</h3><p>In-depth analytics and advanced visualizations are beyond the scope of this article. Our primary goal is to demonstrate how a <strong>scalable, real-world solution</strong> can be built using a minimal dataset.</p><p>As a simple example, we’ll visualize <strong>severe motorcycle-related incidents</strong> on an interactive map. This helps highlight <strong>key hotspots</strong> where motorcycle accidents occur most frequently, showcasing the potential for further analysis and safety improvements.</p><pre>import folium<br>from folium.plugins import MarkerCluster<br><br>dubai_map = folium.Map(location=[25.276987, 55.296249], zoom_start=12)<br><br>marker_cluster = MarkerCluster(options={&#39;spiderfyDistanceMultiplier&#39;: 1.5}).add_to(dubai_map)<br>df_moto_severe = df_sub[(df_sub[&#39;is_motorcycle&#39;] == 1) &amp;<br>                        (df_sub[&#39;severity&#39;] == &#39;severe&#39;)]<br><br>for _, row in df_moto_severe.iterrows():<br>    popup_text = (<br>        f&quot;Time: {row[&#39;acci_time&#39;]}&lt;br&gt;&quot;<br>        f&quot;Type incident: {row[&#39;acci_name_en&#39;]}&lt;br&gt;&quot;<br>    )<br><br>    folium.CircleMarker(<br>        location=[row[&#39;acci_x&#39;], row[&#39;acci_y&#39;]],<br>        radius=5,<br>        color=&#39;blue&#39;,<br>        fill=True,<br>        fill_color=&#39;blue&#39;,<br>        fill_opacity=0.6,<br>        popup=folium.Popup(popup_text, max_width=300)<br>    ).add_to(marker_cluster)<br><br>folium.LayerControl().add_to(dubai_map)<br><br>dubai_map</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nb3Iru2nnEMTiVzUq2iWfg.png" /></figure><h3>Building the Route Risk Prediction Model</h3><h4>Model Selection</h4><p>To predict high-risk segments along a route, we selected the <a href="https://catboost.ai/docs/en/"><strong>CatBoost</strong></a> algorithm for several key reasons:</p><ul><li><strong>Native categorical feature support</strong>: CatBoost handles categorical variables like day of the week or geohashes directly, eliminating the need for complex preprocessing.</li><li><strong>High accuracy</strong>: The model performs well on <strong>imbalanced datasets</strong>, such as accident data where dangerous segments make up only a small fraction of the total.</li><li><strong>Resistance to overfitting</strong>: Thanks to regularization mechanisms, CatBoost remains stable even with smaller datasets.</li><li><strong>Ease of use</strong>: The model requires minimal hyperparameter tuning and works well out of the box.</li></ul><h4>Model Objective</h4><p>The primary goal is to predict the <strong>risk probability</strong> for each segment along a route.</p><p>For every route point, the model should:</p><ul><li>Estimate the likelihood of an incident occurring.</li><li>Highlight the safest path based on aggregated risk estimates.</li></ul><h4>Data Generation</h4><p>To train the model, we need both <strong>incident (positive)</strong> and <strong>non-incident (negative)</strong> data points. Since we only have accident data, we generate negative (safe) samples synthetically.</p><p><strong>Generation process</strong>:</p><ul><li>For each day and geohash, we generate combinations of spatial and temporal features such as day of week, hour, and month.</li><li>Accident cases are excluded to leave only safe segments.</li><li>For each day, we generate <strong>10 times more safe points than accident points</strong> to balance the dataset.</li></ul><p><strong>Result:</strong></p><ul><li>Accident records are labeled as target = 1</li><li>Synthetic safe points are labeled as target = 0</li></ul><p>This creates a <strong>balanced dataset</strong> that improves model learning quality.</p><h4>Feature Enrichment Using Statistics</h4><p>To further enhance the dataset, we calculate accident density and probability for each geohash:</p><ul><li>accidents_count: number of incidents in that area</li><li>total_count: total number of records (accident + safe)</li><li>accident_probability: likelihood of an accident in a given geohash</li></ul><pre>geohash_stats = df_combined.groupby(&#39;geohash_7&#39;).agg(<br>    accidents_count=(&#39;target&#39;, &#39;sum&#39;),<br>    total_count=(&#39;target&#39;, &#39;count&#39;)<br>).reset_index()<br><br>geohash_stats[&#39;accident_probability&#39;] = (<br>    geohash_stats[&#39;accidents_count&#39;] / geohash_stats[&#39;total_count&#39;]<br>)<br><br>df_combined = df_combined.merge(geohash_stats, on=&#39;geohash_7&#39;, how=&#39;left&#39;)</pre><h4>Model Training</h4><p>Here is the code used to train the model:</p><pre>import numpy as np<br>from catboost import CatBoostClassifier<br>from sklearn.model_selection import train_test_split<br>from sklearn.metrics import classification_report, roc_auc_score<br><br>selected_features = [&#39;accident_probability&#39;, &#39;day&#39;, &#39;hour&#39;, &#39;day_of_week&#39;, &#39;geohash_7&#39;, &#39;accidents_count&#39;]<br><br>X = df_combined[selected_features]<br>y = df_combined[&#39;target&#39;]<br><br>categorical_features_indices = np.where(X.dtypes != float)[0]<br><br>X_train, X_temp, y_train, y_temp = train_test_split(X, y, train_size=0.8, random_state=42)<br>X_validation, X_test, y_validation, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, shuffle=False)<br><br>model = CatBoostClassifier(<br>    iterations=3000,<br>    learning_rate=0.01,<br>    depth=4,<br>    random_seed=42,<br>    eval_metric=&#39;AUC&#39;,<br>    cat_features=categorical_features_indices,<br>    class_weights=[1, 10],<br>    l2_leaf_reg=3,<br>    early_stopping_rounds=50,<br>    verbose=100,<br>    task_type=&quot;CPU&quot;<br>)<br><br>model.fit(X_train, y_train, eval_set=(X_validation, y_validation))<br><br>y_val_pred = model.predict(X_validation)<br>y_val_pred_proba = model.predict_proba(X_validation)[:, 1]<br>print(&quot;Validation Classification Report:\n&quot;, classification_report(y_validation, y_val_pred))<br>print(&quot;AUC-ROC on the validation subset:&quot;, roc_auc_score(y_validation, y_val_pred_proba))<br><br>y_test_pred = model.predict(X_test)<br>y_test_pred_proba = model.predict_proba(X_test)[:, 1]<br>print(&quot;Test Classification Report:\n&quot;, classification_report(y_test, y_test_pred))<br>print(&quot;AUC-ROC on the test subset:&quot;, roc_auc_score(y_test, y_test_pred_proba))</pre><h4>Training Details</h4><p>We use AUC as the primary evaluation metric, with a tree depth of 4 and up to 3000 iterations. The dataset is split into:</p><ul><li>80% for training</li><li>10% for validation</li><li>10% for testing</li></ul><pre>Validation Classification Report:<br>               precision    recall  f1-score   support<br><br>           0       0.98      0.88      0.92     14907<br>           1       0.71      0.94      0.81      4825<br><br>    accuracy                           0.89     19732<br>   macro avg       0.84      0.91      0.87     19732<br>weighted avg       0.91      0.89      0.90     19732<br><br>AUC-ROC on the validation subset: 0.9564305255624597<br>Test Classification Report:<br>               precision    recall  f1-score   support<br><br>           0       0.98      0.88      0.92     14753<br>           1       0.72      0.94      0.82      4979<br><br>    accuracy                           0.89     19732<br>   macro avg       0.85      0.91      0.87     19732<br>weighted avg       0.91      0.89      0.90     19732<br><br>AUC-ROC on the test subset: 0.957064441208216</pre><h4>Summary of Results</h4><p>The model was trained on a <strong>balanced dataset</strong> that included both high-risk and safe segments. We evaluated its performance using standard classification metrics:</p><ul><li><strong>Precision</strong>: Proportion of truly dangerous segments among those predicted as dangerous</li><li><strong>Recall</strong>: Proportion of real dangerous segments correctly identified by the model</li><li><strong>AUC-ROC</strong>: Measures the model’s ability to distinguish between safe and risky areas. Scores close to 1 indicate high accuracy</li></ul><h4>✅ Final Results:</h4><p><strong>Overall accuracy</strong> on test set: 89<strong>%</strong></p><p><strong>AUC-ROC</strong>: <strong>0.957</strong> — the model distinguishes well between safe and dangerous points</p><p>For dangerous segments (Class 1):</p><ul><li><strong>Precision</strong>: 72%</li><li><strong>Recall</strong>: 94%</li><li><strong>F1-Score</strong>: 82%</li></ul><p>The model demonstrated high performance and stability across both validation and test sets. With AUC-ROC consistently above 0.95, the model is a <strong>reliable tool for analyzing routes and recommending safer paths</strong>.</p><h3>Integrating the Model into the Product</h3><p>The route risk analysis model can be integrated into a product in two primary ways:</p><ul><li>As part of a <strong>user-facing service</strong> that predicts high-risk areas along a route in real time.</li><li>As a tool for <strong>advanced analytics</strong> using the ClickHouse database.</li></ul><p>Let’s explore both.</p><h3>Primary Use Case: Real-Time Risk-Aware Routing Service</h3><p>The model is embedded into a backend service that provides users with safe route recommendations.</p><p><strong>User Input:</strong></p><ul><li>The user provides the start and end points of their trip, along with the intended departure time.</li><li>This data is used to build a route, which is divided into sequential segments.</li></ul><p><strong>Route Segment Analysis:</strong></p><p>For each point along the route (interpolated every ~250 meters):</p><ul><li>The system calculates temporal and spatial features (hour of day, day of week, geohash).</li><li>These features are passed to the trained CatBoost model to estimate accident risk (predicted_risk).</li></ul><p><strong>Visualization and Recommendation:</strong></p><ul><li>High-risk zones are highlighted on the map.</li><li>The system recommends a <strong>safer alternative</strong> route to minimize collision risk.</li></ul><h4>Example Implementation (using osmnx, networkx, and Folium):</h4><p>The system builds and interpolates a route, then displays it on a map with risk markers:</p><ul><li><strong>Red markers</strong>: High accident probability.</li><li><strong>Orange markers</strong>: Medium risk.</li><li><strong>Unmarked</strong>: Low or negligible risk.</li></ul><p>The user can visualize the entire route and decide whether to follow the suggested path or choose an alternative.</p><h4>Demo Scenario</h4><p>Let’s imagine a user planning a trip across Dubai on <strong>August 15, 2026, at 5:00 PM</strong>, starting in <strong>Dubai Marina</strong>, passing through <strong>JVC</strong>, continuing to <strong>Town Square</strong>, and ending in <strong>Deira</strong>, driving at an average speed of <strong>60 km/h</strong>.</p><h4>Route Analysis &amp; Visualization Code:</h4><pre>start_coords = (25.0806, 55.1389)  # Dubai Marina<br>town_square_coords = (25.0117, 55.2888)  # Town Square<br>jvc_coords = (25.0621, 55.2095)  # JVC<br>end_coords = (25.2711, 55.3075)  # Deira<br>start_time = datetime(2026, 8, 15, 17, 0)<br>speed_kmph = 60<br>step_km = 0.25<br><br>G = ox.graph_from_place(&quot;Dubai, UAE&quot;, network_type=&quot;drive&quot;)<br><br>route_coords1 = get_route_coords(G, start_coords, jvc_coords)<br>route_coords2 = get_route_coords(G, jvc_coords, town_square_coords)<br>route_coords3 = get_route_coords(G, town_square_coords, end_coords)<br><br>route_coords = route_coords1 + route_coords2[1:] + route_coords3[1:]<br>route_coords = interpolate_route(route_coords, step_km=step_km)<br><br>timestamps = [start_time]<br>for i in range(len(route_coords) - 1):<br>    dist_km = ox.distance.great_circle(<br>        route_coords[i][0], route_coords[i][1],<br>        route_coords[i + 1][0], route_coords[i + 1][1]<br>    ) / 1000<br>    time_delta = timedelta(hours=dist_km / speed_kmph)<br>    timestamps.append(timestamps[-1] + time_delta)<br><br>route_data = pd.DataFrame({<br>    &#39;month&#39;: [ts.month for ts in timestamps],<br>    &#39;day&#39;: [ts.day for ts in timestamps],<br>    &#39;day_of_week&#39;: [ts.weekday() for ts in timestamps],<br>    &#39;hour&#39;: [ts.hour for ts in timestamps],<br>    &#39;latitude&#39;: [lat for lat, lon in route_coords],<br>    &#39;longitude&#39;: [lon for lat, lon in route_coords],<br>    &#39;accident_probability&#39;: 0.0<br>})<br><br>route_data[&#39;geohash_7&#39;] = route_data.apply(<br>    lambda row: geohash2.encode(row[&#39;latitude&#39;], row[&#39;longitude&#39;], precision=7), axis=1<br>)<br><br>route_data = prepare_route_data(route_data)<br><br>predicted_risk = []<br>for i in range(len(route_data)):<br>    point_data = route_data.iloc[[i]][selected_features]<br>    risk = model.predict_proba(point_data)[:, 1][0]<br>    predicted_risk.append(risk)<br><br>route_data[&#39;predicted_risk&#39;] = predicted_risk</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3cT3sTh1Y5IvV6lQ" /></figure><p>This process results in a <strong>fully evaluated route</strong>, with each segment tagged by predicted risk level. The map visualization can be enhanced using Folium to show the user a complete picture of potential hazards.</p><h3>Using ClickHouse for Advanced Analytics</h3><p><strong>CatBoost</strong> is a machine learning model optimized for working with tabular data — particularly categorical features — making it a powerful tool for analyzing routes and identifying risk zones. Its <strong>native integration with ClickHouse</strong> allows models to be exported in a compatible format and executed directly within ClickHouse queries using the <a href="https://clickhouse.com/docs/sql-reference/functions/other-functions#catboostevaluate">catboostEvaluate</a> function.</p><p>While the primary purpose of the model is to serve real-time user requests, integrating with ClickHouse unlocks <strong>advanced analytical capabilities</strong> by providing fast, large-scale data querying and risk evaluation.</p><h4>Setting Up ClickHouse for CatBoost Integration</h4><p>To run CatBoost models inside ClickHouse, you need to not only train and export the model, but also properly configure your ClickHouse environment.</p><ul><li><strong>Configure the Library Bridge</strong></li></ul><p>In your ClickHouse config file (config.xml or config.d/custom.xml), add the following block if not already present:</p><pre>&lt;library_bridge&gt;<br>    &lt;port&gt;9019&lt;/port&gt; &lt;!-- Port used to interact with external libraries --&gt;<br>&lt;/library_bridge&gt;</pre><ul><li><strong>Specify the Path to the CatBoost Library</strong></li></ul><p>Ensure that the libcatboostmodel.so file (provided by CatBoost) is present on the server, and specify its path:</p><pre>&lt;catboost_lib_path&gt;<br>    /etc/clickhouse-server/libs/libcatboostmodel.so<br>&lt;/catboost_lib_path&gt;</pre><ul><li><strong>Restart ClickHouse</strong></li></ul><p>Apply the changes by restarting the ClickHouse server.</p><h4>Exporting and Uploading the Model</h4><p>After training, export your CatBoost model in cbm format:</p><pre>model.save_model(&quot;catboost_model.cbm&quot;, format=&quot;cbm&quot;)</pre><p>Then upload the model to the ClickHouse server:</p><pre>docker cp catboost_model.cbm df27bb24a64e:/etc/clickhouse-server/models/catboost_model.cbm</pre><h4>Using the Model in SQL Queries</h4><p>Once everything is set up, you can run inference directly from ClickHouse using the catboostEvaluate function:</p><pre> SELECT<br>    catboostEvaluate(&#39;/etc/clickhouse-server/models/catboost_model.cbm&#39;,<br>        accident_probability, day, hour, day_of_week, geohash_7, month) AS raw_score,<br>    1 / (1 + exp(-raw_score)) AS probability<br>FROM routes<br>ORDER BY 2 DESC<br>LIMIT 10;</pre><p>Example Output:</p><pre>┌────────────raw_score─┬─────────probability─┐<br>│ -0.34231973168361335 │ 0.41524609678476404 │<br>│ -0.5038888848931045  │ 0.3766272025834066  │<br>│ -0.5944779763289947  │ 0.35560805950430163 │<br>│ -0.6212236275700299  │ 0.3495032076181987  │<br>└──────────────────────┴─────────────────────┘</pre><p>CatBoost returns <strong>raw scores</strong>, which can be transformed into probabilities using the sigmoid function:</p><pre>1 / (1 + exp(-raw_score))</pre><p>This maps the output to the [0, 1] range, making it intuitive to interpret as the likelihood of an incident.</p><h4>Why This Integration Matters</h4><p>By combining CatBoost and ClickHouse, the system becomes useful not only for individual users but also for <strong>large-scale, city-level applications</strong>, such as traffic safety management.</p><p>Thanks to the <strong>native CatBoost — ClickHouse integration</strong>, even <strong>complex analytics</strong> can be executed with <strong>high efficiency and minimal performance overhead</strong>, making this architecture ideal for smart cities and operational dashboards.</p><h3>Potential Use Cases</h3><p>A route risk analysis system powered by machine learning offers a wide range of practical applications. Below are key use cases across different user groups.</p><h4>For Individual Users: Personalized Route Recommendations for Drivers</h4><p><strong>Problem: </strong>Drivers often choose routes without full visibility into their relative safety — particularly when driving through unfamiliar neighborhoods or during high-risk periods like nighttime or rush hour.</p><p><strong>Solution:</strong></p><ul><li><strong>Personalized Safe Routing</strong>: The system provides drivers with safer route alternatives by minimizing exposure to high-risk zones.</li><li><strong>Real-Time Risk Awareness</strong>: High-risk segments are flagged on the map in real time.</li><li><strong>User Experience</strong>: Seamlessly integrates with popular navigation apps (e.g., Google Maps, Waze), enhancing them with real-time risk insights. Routes are visualized with clearly marked high-risk areas, while safer alternatives are intelligently suggested to guide users with confidence.</li></ul><p><strong>Example</strong>: A user selects a route from Dubai Marina to Deira. The system evaluates multiple options, identifies the safest path, and marks high-risk segments — providing safety insights to the driver before departure.</p><h4>For City Services: Optimizing Public Transport and Emergency Routes</h4><p><strong>Problem: </strong>The effectiveness of municipal services — public transit, emergency response — depends on both speed and safety. Traditional routing may overlook accident-prone areas.</p><p><strong>Solution:</strong></p><ul><li><strong>Optimized Public Transit Planning</strong>: Leverage historical safety data to design bus and taxi routes that avoid high-risk areas. Adapt routes based on time-specific accident patterns to enhance passenger safety during peak hours.</li><li><strong>Support for Emergency Services</strong>: Quickly compute the safest and fastest routes for emergency responders — including ambulances, fire trucks, and police — ensuring minimal response times without compromising safety.</li></ul><p><strong>Example</strong>: The city’s transit authority analyzes risk data and reroutes a bus line to avoid intersections with frequent peak-hour accidents.</p><h3>Scalability Potential</h3><h4>Expandability to Other Cities and Regions:</h4><p>The system can be adapted to any city or country, provided there is access to local accident data, weather records, and relevant infrastructure details.</p><p><strong>Example</strong>: In tourist-heavy cities, it could help ensure safer walking and driving routes.</p><h4>Data Enrichment Opportunities:</h4><ul><li><strong>Weather Data</strong>: Incorporating rain, fog, or extreme heat can improve model accuracy.</li><li><strong>Road Work Data</strong>: Factoring in temporary closures and detours allows for more relevant routing.</li><li><strong>Social &amp; Crime Data</strong>: Especially useful for pedestrians and tourists navigating unfamiliar or high-crime areas.</li><li><strong>IoT Device Integration</strong>: Leverage smart city infrastructure — traffic cameras, road sensors — to assess real-time road conditions.</li></ul><h4>Integration with Other Services:</h4><ul><li>Mapping APIs (e.g., Google Maps, Yandex Maps) for real-time, dynamic route visualizations.</li><li>Mobile apps for <strong>tourists</strong>, <strong>delivery services</strong>, or <strong>ride-sharing platforms</strong>.</li></ul><h3>Conclusion: Safer Cities Start with Smarter Data</h3><p>This project illustrates a powerful truth: with the right data, thoughtful modeling, and accessible technology, we can make cities not only smarter — but safer.</p><p>By combining open accident data, geospatial analytics, and CatBoost-based risk modeling, we’ve built a system that does more than just map routes — it understands them. It helps drivers make informed decisions, supports city services in planning safer infrastructure, and opens the door to a future where AI actively reduces human risk on the road.</p><p>But this is just the beginning.</p><p>As the system evolves — with richer datasets, IoT integration, and global scalability — it has the potential to become a <strong>core layer of urban intelligence</strong>, helping cities proactively protect their citizens.</p><p>Safe routing is no longer just a matter of speed or distance.</p><p>It’s a matter of trust — in the systems that guide us, and the data that shapes our choices. And that trust isn’t given — it’s built.</p><p><strong>Built with data. Strengthened by AI. Driven by human vision.</strong></p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/@InPlainEnglish"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a> | <a href="https://differ.blog/inplainenglish"><strong>Differ</strong></a> | <a href="https://twitch.tv/inplainenglish"><strong>Twitch</strong></a></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f39a046b222" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/how-we-built-a-route-safety-engine-using-catboost-open-data-and-clickhouse-5f39a046b222">How We Built a Route Safety Engine Using CatBoost, Open Data, and ClickHouse</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Harmony in Data: The Music Recommendation System with Milvus]]></title>
            <link>https://python.plainenglish.io/harmony-in-data-the-music-recommendation-system-with-milvus-c9711609ed36?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9711609ed36</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Mon, 08 Apr 2024 07:33:56 GMT</pubDate>
            <atom:updated>2024-04-09T01:05:18.376Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Art of Personalized Music Recommendations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TjoyytNp0d803Tjgau0xhQ.png" /><figcaption>The imagination of a neural network</figcaption></figure><p>Music is not just sounds.</p><p>It’s a way to express your emotions, mood, and feelings. In a world where each of us is looking for our own special path, music becomes a faithful companion, helping to find harmony and inspiration. But how can we ensure that every user finds exactly the music that resonates in their heart? How can we ensure that in the sea of musical tracks, everyone can find their unique wave?</p><p>In this article, I want to share with you an approach to creating a music recommendation system that focuses on caring for users. A system that uses advanced technologies and algorithms, but at the same time does not forget about the main thing — human feelings and preferences. Special attention is paid to the use of the <a href="https://milvus.io/">Milvus</a> database, which represents a robust solution for managing vector data and allows for the efficient processing of large volumes of information necessary for accurate music recommendations.</p><p>Join me as we explore how Milvus and intelligent data analysis can create a service that not only recommends songs but also tailors them to the unique characteristics of each user, bringing them joy and harmony through music!</p><h3>Vector representation of data</h3><p>Vector representation is a method of representing objects, like song lyrics, as a set of numerical values. These values encapsulate the content and essence of the text. They can capture various textual attributes, such as the prevalence of certain words or their association with particular themes.</p><p>In music track recommendation systems, vector representations of song lyrics facilitate the assessment of similarity between tracks. When two songs share lyrical similarities, their vector representations will be proximate in the vector space. This proximity enables the recommendation system to grasp the thematic nuances of the song lyrics and provide users with more precise recommendations, suggesting songs that echo similar themes.</p><h3>Milvus database</h3><h4>Main features and advantages</h4><p>Milvus is a powerful vector database tailored for processing and searching extensive vector data. It stands out for its high performance and scalability, rendering it perfect for machine learning, deep learning, similarity search tasks, and recommendation systems. Key features and advantages include:</p><ul><li><strong>Scalability</strong>: Handles billions of vectors and trillions of queries, ideal for large-scale projects and recommendation systems requiring efficient data management.</li><li><strong>High Performance</strong>: Optimized algorithms ensure fast similarity searches across large datasets, facilitating swift and precise recommendations.</li><li><strong>Flexibility</strong>: Supports various <a href="https://milvus.io/docs/index.md">indexing algorithms such as IVF_FLAT, IVF_SQ8, HNSW</a>, etc., allowing customization based on specific requirements.</li><li><strong>Ease of Use</strong>: Offers user-friendly APIs for multiple programming languages, enabling easy integration with existing projects.</li><li><strong>Data Management</strong>: Allows partitioning based on given keys, enhancing search performance and simplifying data management.</li><li><strong>Reliability</strong>: Features data replication and fault tolerance mechanisms to maintain data availability.</li><li><strong>Data Type Compatibility</strong>: Accommodates various vector types, catering to diverse data requirements in recommendation systems.</li></ul><h4>Application in Recommendation Systems:</h4><ul><li><strong>Storing Vector Representations:</strong> Enables storage of vector representations for items (e.g., products, movies, music tracks) and users, which can be derived from machine learning methods like word2vec, BERT, or other vectorization algorithms.</li><li><strong>Personalization</strong>: Updates vector representations in real-time based on user behavior and preferences, ensuring tailored recommendations.</li><li><strong>Integration and Experimentation</strong>: Easily integrates with other system components and supports A/B testing to refine algorithms and strategies.</li></ul><p>By harnessing the capabilities of Milvus, recommendation systems can deliver tailored, precise, and scalable suggestions, thereby elevating user experience and satisfaction. The integration of Milvus in recommendation tasks facilitates the development of high-performance, customized systems that can adapt to the evolving needs of businesses and users.</p><p>Let’s delve into the aforementioned techniques and construct a recommendation system that operates in real-time. This system will present users with music tracks that closely align with their preferences and past interactions on the platform. We’ll delve into crucial elements of constructing such a system, encompassing the gathering and processing of user preference data, the analysis of musical compositions to generate their vector representations, and the deployment of machine learning algorithms for personalized recommendations. Furthermore, we’ll explore strategies to ensure the system’s scalability and performance, enabling it to effectively cater to a vast number of online users.</p><h3>Architecture of the System Under Development</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcuXDYzkn9yS2xpcjNIfNQ.png" /></figure><p>The primary resource for our recommendations is the <a href="https://www.kaggle.com/datasets/notshrirang/spotify-million-song-dataset">Kaggle Spotify Million Song Dataset</a>, which offers a wealth of data on artists, song titles, and lyrics. While advanced recommendation systems can delve into various parameters such as genre, style, tempo, theme, duration, and release year, the characteristics we’ve selected provide a solid foundation for understanding the core principles of recommendation systems. This example can be further enriched with additional parameters to explore more nuanced aspects of music recommendations.</p><p>Let’s consider the architecture of the developed system:</p><h4><strong>Create Song Embeddings</strong></h4><p>Using a language model, we generate a vector representation for each song’s text and store it in the songs_collection in Milvus.</p><h4><strong>Create Artist Embedding</strong></h4><p>We calculate an average vector for all songs by an artist based on their song vectors and save it in the artists_collection.</p><h4><strong>Prepare Cold Start</strong></h4><p>To onboard new users, a pre-selected sample of songs is needed, ideally grouped by clustering songs or artists. This can also include information about popular recommendations previously issued to other users or those similar to the new user. There are various strategies for creating an initial recommendation, ranging from surveying the user’s preferences to applying heuristics and other methods.</p><h4><strong>Get Online Recommendation</strong></h4><p>Directly providing recommendations to the user based on their platform usage. Business requirements for a real system might include:</p><p><strong>For new users </strong>— Cold Start and feedback collection from the received suggestions.</p><p><strong>For current users</strong>:</p><ul><li>50% of recommendations from a model searching similar artists to those the user liked;</li><li>30% of recommendations from a model searching by song themes similar to those the user liked;</li><li>20% of recommendations from the Cold Start model to diversify the suggestions and offer unexpected options that the user might also enjoy;</li></ul><p>Limitations on the repetition of recommendation issuance, using Redis with TTL keys can be used to cache issuance results, which provides filtering and issuance of only new songs for the user.</p><h4>Impression</h4><p>Gathering user feedback on the received recommendations helps us understand their interests and refine recommendation accuracy over time. In large systems, this can extend beyond direct reactions like likes or dislikes to include listening duration, frequency of listens to the same songs/artists, etc. In our case, we’ll focus on song likes, with information published in Kafka.</p><h4>Create User Embeddings</h4><p>A real-time Kafka consumer builds a user vector based on their reactions to recommendations and stores the data in the users_collection and users_artists_collection in Milvus.</p><h4>PostgreSQL database</h4><p>Stores all events occurring within the system, such as recommendations and impressions. This data enables the Data Science team to conduct further analysis on the quality of recommendations, user responses, and the outcomes of A/B testing. Data is fed into PostgreSQL using Kafka Connect or a custom Kafka Consumer.</p><h3>Data preparation</h3><h4>Reading the dataset and creating collections in Milvus</h4><p>The Milvus database can be independently deployed on your workstation or you can opt for <a href="https://cloud.zilliz.com/">Zilliz’s cloud</a> solution, which offers $100 in free credits each month.</p><blockquote>You can find the full code for data preparation in <a href="https://github.com/igorgorbenko/songs_recommendation/blob/main/data_preparation.ipynb">my Jupyter Notebook</a></blockquote><p>Let’s read the saved dataset with the source data and put it into a Pandas DataFrame while removing line breaks:</p><pre>df = pd.read_csv(&quot;./dataset/Spotify Million Song Dataset_exported.csv&quot;)<br>df = df.dropna()<br><br>cleaned_df = df.loc[(df[&quot;text&quot;].str.len() &gt; 50)]<br>cleaned_df[&quot;text&quot;] = cleaned_df[&quot;text&quot;].str.replace(&quot;\n&quot;, &quot; &quot;)</pre><p>Let’s create the first collection in Milvus to store song embeddings</p><pre>def create_songs_collection():<br>    if utility.has_collection(MILVUS_SONGS_COLLECTION):<br>        utility.drop_collection(MILVUS_SONGS_COLLECTION)<br><br>    # object should be inserted in the format of (artist, song, embedding embedding)<br>    fields = [<br>        FieldSchema(name=&quot;id&quot;, dtype=DataType.INT64, is_primary=True, auto_id=True),<br>        FieldSchema(name=&quot;artist&quot;, dtype=DataType.VARCHAR, max_length=100),<br>        FieldSchema(name=&quot;song&quot;, dtype=DataType.VARCHAR, max_length=100),<br>        FieldSchema(name=&quot;embedding&quot;, dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)<br>    ]<br><br>    schema = CollectionSchema(fields=fields)<br>    collection = Collection(name=MILVUS_SONGS_COLLECTION, schema=schema)<br><br>    collection.create_index(field_name=&quot;embedding&quot;,<br>                            index_params={<br>                                &quot;index_type&quot;: &quot;IVF_FLAT&quot;,<br>                                &quot;metric_type&quot;: &quot;L2&quot;,<br>                                &quot;params&quot;: {&quot;nlist&quot;: 128},<br>                            })<br>    collection.load()<br>    return collection<br><br>songs_collection = create_songs_collection()</pre><p>In the code snippet above, we create a collection designed to store song information, featuring an auto-incrementing id field that acts as the primary key. The embedding field holds the vectorized representation of each song.</p><p>It’s crucial to note that the dimensionality of this field is rigidly determined by the model employed for generating the embeddings, ensuring consistent data for future applications.</p><blockquote>An <a href="https://milvus.io/docs/v2.0.x/index.md#IVF_FLAT">IVF_FLAT</a> index with the L2 metric (Euclidean distance) is created for the embedding field. The <em>nlist</em> parameter is set to 128. The index is used to speed up the search for similar vectors in the collection. Such an index provides a balance between search speed and accuracy of results, making it a suitable choice for many nearest neighbor search tasks in large sets of vector data.</blockquote><h4>Song vectorization and creation of the artist vector</h4><p>To generate vector representations of song lyrics, we will apply the <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2"><strong>sentence-transformers/all-MiniLM-L6-v2</strong></a> model. This model is a reduced version of the BERT model, optimized for smaller size and higher processing speed while maintaining comparable quality of embeddings. With these characteristics, the<strong> </strong>all-MiniLM-L6-v2<strong> </strong>model is an ideal choice for tasks where both the quality of vector representations and the requirements for resources and processing speed are important.</p><p>Example code for generating and inserting vector representations into a Milvus collection:</p><pre>transformer = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br><br><br>def embed_insert(data, collection):<br>    try:<br>        texts = np.array(data[2])<br>        embeddings = transformer.encode(texts)<br><br>        insert_data = [<br>            {&quot;artist&quot;: artist, &quot;song&quot;: song, &quot;embedding&quot;: embedding.tolist()}<br>            for artist, song, embedding in zip(data[0], data[1], embeddings)<br>        ]<br><br>        collection.insert(insert_data)<br>        return {f&quot;{artist}-{song}&quot;: embedding.tolist() for artist, song, embedding in zip(data[0], data[1], embeddings)}<br>    except Exception as e:<br>        print(f&quot;Error during embedding and insertion: {e}&quot;)<br>        return None<br><br><br>data_batch = [[], [], []]<br>song_vectors = {} <br><br>for index, row in cleaned_df.iterrows():<br>    data_batch[0].append(row[&quot;artist&quot;])<br>    data_batch[1].append(row[&quot;song&quot;])<br>    data_batch[2].append(row[&quot;text&quot;])<br><br>    if len(data_batch[0]) % BATCH_SIZE == 0:<br>        batch_vectors = embed_insert(data_batch, songs_collection)<br>        if batch_vectors is not None:<br>            song_vectors.update(batch_vectors)<br>        data_batch = [[], [], []]<br><br>if len(data_batch[0]) != 0:<br>    batch_vectors = embed_insert(data_batch, songs_collection)<br>    if batch_vectors is not None:<br>        song_vectors.update(batch_vectors)<br><br>try:<br>    songs_collection.flush()<br>    print(&quot;Flushed changes to the songs collection.&quot;)<br>except Exception as e:<br>    print(f&quot;Error during flushing the songs collection: {e}&quot;)</pre><p>Based on the vectors constructed for each song of each artist, it is possible to calculate the<strong> average vector for each artist</strong>. This average vector characterizes the <strong>overall style and direction</strong> of the artist’s creative work:</p><pre># Calculate the average vector for each artist<br>from collections import defaultdict<br><br>start_time = time.time()<br>artist_vectors = defaultdict(list)<br><br>for artist, song, embedding in zip(cleaned_df[&quot;artist&quot;], cleaned_df[&quot;song&quot;], song_vectors.values()):<br>    artist_vectors[artist].append(embedding)<br><br>artist_avg_vectors = {artist: np.mean(vectors, axis=0).tolist() for artist, vectors in artist_vectors.items()}<br><br>end_time = time.time()<br>execution_time = end_time - start_time<br>print(f&quot;The average vector calculation elapsed: {execution_time} seconds&quot;)</pre><p>Similarly to the example with the song collection, let’s create a collection for artists with their vectors</p><pre>def create_artist_collection():<br>    if utility.has_collection(MILVUS_ARTISTS_COLLECTION):<br>        utility.drop_collection(MILVUS_ARTISTS_COLLECTION)<br><br>    # object should be inserted in the format of (artist, embedding embedding)<br>    fields = [<br>        FieldSchema(name=&quot;id&quot;, dtype=DataType.INT64, is_primary=True, auto_id=True),<br>        FieldSchema(name=&quot;artist&quot;, dtype=DataType.VARCHAR, max_length=100),<br>        FieldSchema(name=&quot;embedding&quot;, dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)<br>    ]<br><br>    schema = CollectionSchema(fields=fields)<br>    collection = Collection(name=MILVUS_ARTISTS_COLLECTION, schema=schema)<br><br>    index_params = {<br>        &quot;index_type&quot;: &quot;IVF_FLAT&quot;,<br>        &quot;metric_type&quot;: &quot;L2&quot;,<br>        &quot;params&quot;: {&quot;nlist&quot;: 128},<br>    }<br>    collection.create_index(field_name=&quot;embedding&quot;, index_params=index_params)<br>    collection.load()<br>    return collection<br><br><br>collection_artists = create_artist_collection()<br><br><br>data_batch = [[], []]<br><br>for key, value in artist_avg_vectors.items():<br>    data_batch[0].append(key)<br>    data_batch[1].append(value)<br><br>    if len(data_batch[0]) % BATCH_SIZE == 0:<br>        collection_artists.insert(data_batch)<br>        data_batch = [[], []]<br><br>if len(data_batch[0]) != 0:<br>    collection_artists.insert(data_batch)<br><br>try:<br>    collection_artists.flush()<br>    print(&quot;Flushed changes to the artist collection&quot;)<br>except Exception as e:<br>    print(f&quot;Error during flushing the artist collection: {e}&quot;)</pre><h3>Testing and verifying the quality of vectorization</h3><h4>Search by the user query</h4><p>Now that our primary collections are prepared and filled, let’s assess the quality of the embeddings and the efficacy of content-based search. We’ll initiate a test query to identify songs with themes similar to the user’s interests, such as:</p><pre>search_terms = (&quot;I&#39;m looking for a song about the epic battles, &quot; <br>                &quot;heroes and warriors of the True metal with hearts of steel&quot;)</pre><p>Code for searching a song based on a user’s query:</p><pre>from sklearn.metrics.pairwise import cosine_similarity<br>from tabulate import tabulate<br><br><br>def search_and_evaluate(search_terms, collection, top_k=10):<br>    query_embedding = transformer.encode(search_terms)<br>    query_embedding = np.array(query_embedding)<br><br>    search_results = collection.search(<br>        data=query_embedding,<br>        anns_field=&quot;embedding&quot;,<br>        param={&quot;metric_type&quot;: &quot;L2&quot;, &quot;params&quot;: {&quot;nprobe&quot;: 20}},<br>        limit=top_k,<br>        output_fields=[&quot;artist&quot;, &quot;song&quot;, &quot;embedding&quot;]<br>    )<br><br>    results = []<br>    similarities = []<br>    for hit in search_results[0]:<br>        artist = hit.entity.get(&#39;artist&#39;)<br>        song = hit.entity.get(&#39;song&#39;)<br>        similarity = cosine_similarity(query_embedding, np.array([hit.entity.get(&#39;embedding&#39;)]))[0][0]<br>        similarities.append(similarity)<br>        results.append([artist, song, f&quot;{similarity:.2f}&quot;])<br><br>    print(tabulate(results, headers=[&quot;Artist&quot;, &quot;Song&quot;, &quot;Similarity&quot;]))<br>    average_similarity = np.mean(similarities)<br>    print(f&quot;Average similarity: {average_similarity:.2f}&quot;)<br>    return search_results, average_similarity<br><br>results = search_and_evaluate([search_terms], songs_collection, top_k=20)</pre><p>Results:</p><pre>Artist          Song                              Similarity<br>--------------  ------------------------------  ------------<br>Manowar         Army Of Immortals                       0.51<br>Manowar         The Sons Of Odin                        0.49<br>Manowar         Heart Of Steel                          0.48<br>Manowar         Call To Arms                            0.48<br>W.A.S.P.        The Rock Rolls On                       0.46<br>Manowar         Warriors Of The World                   0.46<br>Michael Bolton  I&#39;m Not Made Of Steel                   0.46<br>W.A.S.P.        War Cry                                 0.46<br>George Michael  I Knew You Were Waiting For Me          0.46<br>Chicago         Glory Of Love                           0.46<br>Ziggy Marley    Still The Storms                        0.45<br>Manowar         Metal Daze                              0.45<br>The Script      Superheroes                             0.44<br>Manowar         The Lord Of Steel                       0.44<br>Peter Cetera    Glory Of Love                           0.44<br>Manowar         Blood Of The Kings                      0.44<br>Bon Jovi        Nobody&#39;s Hero                           0.43<br>Linkin Park     Until It Breaks                         0.43<br>Manowar         Brothers Of Metal                       0.43<br>Slayer          Fight Till Death                        0.43<br><br>Average similarity: 0.46</pre><p>The result obtained matches the query we posed earlier. If desired, one can compare the similarity of the song lyrics from the sample with the request “I’m looking for a song about the epic battles, heroes, and warriors of the True metal with hearts of steel” to confirm the relevance of the suggested sample. And who knows, you might just find the anthem for your next heroic adventure or at least a track to headbang to! :)</p><h4>Visualization of artist clusters</h4><p>As we observed earlier, the embeddings are crafted to enable efficient searching for songs similar to those in user queries. Now, let’s explore the artist collection. Were the average vectors for the performers accurately constructed?</p><p>To address these questions, we will normalize and cluster the artists’ average vectors and then visualize the resulting clusters in a two-dimensional vector space:</p><pre>import numpy as np<br>from adjustText import adjust_text<br>from sklearn.manifold import TSNE<br>from sklearn.cluster import KMeans<br>import matplotlib.pyplot as plt<br><br>artist_vectors_list = list(artist_avg_vectors.values())<br>artist_vectors_np = np.array(artist_vectors_list)<br><br>tsne = TSNE(n_components=2, random_state=1)<br>reduced_vectors = tsne.fit_transform(artist_vectors_np)<br><br>n_clusters = 35<br><br>kmeans = KMeans(n_clusters=n_clusters, random_state=1)<br>kmeans.fit(reduced_vectors)<br>labels = kmeans.labels_<br><br>plt.figure(figsize=(22, 15))<br><br>colors = [<br>    &#39;firebrick&#39;, &#39;indigo&#39;, &#39;darkorange&#39;,<br>    &#39;purple&#39;, &#39;darkblue&#39;, &#39;darkred&#39;, &#39;darkslateblue&#39;, &#39;darkolivegreen&#39;,<br>    &#39;darkgoldenrod&#39;, &#39;darkcyan&#39;, &#39;darkgreen&#39;, &#39;darkmagenta&#39;, &#39;darkslategray&#39;, &#39;teal&#39;,<br>    &#39;saddlebrown&#39;, &#39;midnightblue&#39;, &#39;deeppink&#39;, &#39;forestgreen&#39;, &#39;mediumvioletred&#39;,<br>    &#39;chocolate&#39;, &#39;steelblue&#39;, &#39;darkkhaki&#39;, &#39;mediumblue&#39;, &#39;tomato&#39;,<br>    &#39;darkolivegreen&#39;, &#39;sienna&#39;, &#39;royalblue&#39;, &#39;crimson&#39;, &#39;seagreen&#39;,<br>    &#39;cadetblue&#39;, &#39;maroon&#39;, &#39;darkblue&#39;, &#39;darkorange&#39;, &#39;darkgoldenrod&#39;, &#39;navy&#39;<br>]<br><br>assert len(colors) &gt;= n_clusters, &quot;There are not enough colors for all clusters&quot;<br><br>texts = []<br><br>for i in range(n_clusters):<br>    cluster_color = colors[i]<br><br>    cluster_indices = [index for index, label in enumerate(labels) if label == i]<br>    cluster_points = reduced_vectors[cluster_indices]<br>    scatter = plt.scatter(cluster_points[:, 0], cluster_points[:, 1], label=f&#39;Cluster {i + 1}&#39;, color=colors[i])<br><br>    cluster_center = np.mean(cluster_points, axis=0)<br><br>    artists_in_cluster = [list(artist_avg_vectors.keys())[index] for index in cluster_indices]<br>    artists_text = &#39;\n&#39;.join(artists_in_cluster[:10])<br><br>    text = plt.text(<br>        cluster_center[0],<br>        cluster_center[1],<br>        f&#39;{i + 1}: {artists_text}&#39;,<br>        ha=&#39;center&#39;,<br>        fontsize=11,<br>        color=colors[i],<br>        bbox=dict(facecolor=&#39;white&#39;, alpha=0.85, edgecolor=colors[i], boxstyle=&#39;round,pad=0.2&#39;)  <br>    )<br>    texts.append(text)<br><br>adjust_text(texts, arrowprops=dict(arrowstyle=&#39;-&gt;&#39;, color=&#39;red&#39;))<br><br>plt.title(&#39;Artist Clusters in 2D&#39;)<br>plt.xlabel(&#39;t-SNE Feature 1&#39;)<br>plt.ylabel(&#39;t-SNE Feature 2&#39;)<br>plt.show()</pre><p>In the chart below, each cluster is represented by points of the same color, and the center of each cluster is marked with the cluster number and the names of the first 10 artists of that cluster. This allows us to visually assess how well the average vectors of the artists are grouped and represent different styles or genres of music</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9KiJ6L1JKorYLiNq1rHpsg.png" /><figcaption>Clusters of the artists</figcaption></figure><p>Pay attention to <strong>cluster 18 </strong>— it includes artists performing extreme genres of heavy music, such as<em> Black Sabbath, Death, Dream Theater, Helloween, Iron Maiden, Judas Priest, King Diamond, Manowar, Marilyn Manson, and Megadeth.</em> This indicates that the averaging algorithm and the vectorization model effectively recognize and group artists with similar musical characteristics.</p><p>Also, on the chart, one can notice distinct <strong>clusters 4 and 29</strong>, which include rap and hip-hop artists with similar themes in their work, such as <em>Drake, Eminem, Everlast, Flo-Rida, Mc Hammer, and Nicki Minaj</em>. This highlights the algorithm’s ability to distinguish musical genres based on textual data.</p><p>The used model and averaging algorithm have shown good results in forming vectors for the recommendation system. Now we can move on to the next stage — building the Core part and implementing the main logic of issuing recommendations.</p><h3>Creating the engine of the recommendation system</h3><p>Let’s write the code for a Flask application with two methods:</p><ul><li>Getting recommendations by user ID — the recommend method (GET)</li><li>Publishing the user’s reaction to the received recommendation — the impression method (POST)</li></ul><blockquote>Also, we’ll develop a simple web form to display the results of the recommendation system and place it in templates/recommendation_form.html</blockquote><pre>from flask import Flask, request, render_template, jsonify<br>from recommendations.recommender import Recommender<br><br>app = Flask(__name__)<br>RECOMMENDER = Recommender()<br><br><br>@app.route(&#39;/&#39;)<br>def index():<br>    return render_template(&#39;recommendation_form.html&#39;)<br><br><br>@app.route(&#39;/recommend&#39;, methods=[&#39;GET&#39;])<br>def recommend():<br>    user_id = request.args.get(&#39;user_id&#39;)<br>    if not user_id:<br>        return &quot;User ID is required&quot;, 400<br><br>    request_id, recommendations, user_vector = RECOMMENDER.get_recommendations(user_id)<br>    return jsonify({&#39;request_id&#39;: request_id,<br>                    &#39;recommendations&#39;: recommendations,<br>                    &#39;user_vector&#39;: user_vector})<br><br><br>@app.route(&#39;/impression&#39;, methods=[&#39;POST&#39;])<br>def impression():<br>    data = request.json<br>    request_id = data.get(&#39;request_id&#39;)<br>    user_id = data.get(&#39;user_id&#39;)<br>    song_id = data.get(&#39;song_id&#39;)<br>    artist = data.get(&#39;artist&#39;)<br>    is_like = data.get(&#39;is_like&#39;)<br><br>    if not user_id:<br>        return &quot;User ID is required&quot;, 400<br>    if not song_id:<br>        return &quot;Song ID is required&quot;, 400<br>    if artist is None:<br>        return &quot;Artist is required&quot;, 400<br>    if is_like is None:<br>        return &quot;Impression (is_like) is required&quot;, 400<br><br>    result = RECOMMENDER.put_impression(request_id, user_id, song_id, artist, is_like)<br>    return jsonify(result)<br><br><br>if __name__ == &#39;__main__&#39;:<br>    app.run(debug=True)</pre><h4>The Recommender class</h4><p>The Recommender class implements the core logic of the recommendation system and includes two main methods:</p><ul><li>get_recommendations: In this method, a unique request identifier (request_id) is generated, which is used for analytics of the system&#39;s operation. The user&#39;s interest vector for songs (user_vector) and the user&#39;s interest vector for artists (user_artist_vector) are searched. Additionally, song identifiers that have already been shown to the user are extracted from Redis. If the user already has an established interest profile, they will receive recommendations for two songs similar to their previously liked tracks, three songs by artists akin to those they’ve shown interest in, and one song from the Cold Start collection to add diversity to their interest vector. For new users, a cold start will be offered based on the most popular songs from each artist cluster. The recommendation results are published in the Kafka topic events with the event name recommend.</li></ul><pre>{<br>  &quot;event_name&quot;: &quot;recommend&quot;,<br>  &quot;request_id&quot;: &quot;b0b709f3-0994-4afb-8282-70fa45cbb348&quot;,<br>  &quot;ts&quot;: &quot;2024-03-29T07:47:24.378756&quot;,<br>  &quot;user_id&quot;: 1005,<br>  &quot;recommendations&quot;: {<br>    &quot;online_user&quot;: {<br>      &quot;song_ids&quot;: [448571723228979100, 448571723228955840],<br>      &quot;artists&quot;: [&quot;Allman Brothers Band&quot;, &quot;Alison Krauss&quot;],<br>      &quot;songs&quot;: [&quot;Worried Down With The Blues&quot;, &quot;Foolish Heart&quot;]<br>    },<br>    &quot;online_user_artists&quot;: {<br>      &quot;song_ids&quot;: [448571723228980900, 448571723228977400, 448571723228963650],<br>      &quot;artists&quot;: [&quot;Black Sabbath&quot;, &quot;Yngwie Malmsteen&quot;, &quot;Incubus&quot;],<br>      &quot;songs&quot;: [&quot;Eternal Idol&quot;, &quot;Sacrifice&quot;, &quot;Let&#39;s Go Crazy&quot;]<br>    },<br>    &quot;coldstart&quot;: {<br>      &quot;song_ids&quot;: [448571723229008830],<br>      &quot;artists&quot;: [&quot;Tom Lehrer&quot;],<br>      &quot;songs&quot;: [&quot;Wernher Von Braun&quot;]<br>    }<br>  }<br>}</pre><ul><li>put_impression. In this method, a new message is prepared and published to the Kafka topic events with the event name impression for further processing of user reactions to recommendations. The event contains information about whether the user liked the recommended song.</li></ul><pre>class Recommender:<br>    def __init__(self):<br>        self.kafka_producer = KafkaEventProducer(bootstrap_servers=BOOTSTRAP_SERVERS, topic=EVENTS_TOPIC_NAME)<br>        logger.info(&quot;Recommender initialized&quot;)<br><br>    def put_impression(self, request_id, user_id, song_id, artist, is_like):<br>        try:<br>            # Send the impression event to Kafka<br>            event = {<br>                &#39;event_name&#39;: &#39;impression&#39;,<br>                &#39;request_id&#39;: request_id,<br>                &#39;is_like&#39;: is_like,<br>                &#39;ts&#39;: datetime.now(timezone.utc).strftime(&#39;%Y-%m-%dT%H:%M:%S.%f%z&#39;),<br>                &#39;user_id&#39;: int(user_id),<br>                &#39;song_id&#39;: int(song_id),<br>                &#39;artist&#39;: artist<br>            }<br>            self.kafka_producer.produce_event(event)<br>            logger.info(f&quot;Published impression event for user {user_id} and &quot;<br>                        f&quot;song {song_id} by artist {artist} (like == {is_like})&quot;)<br>        except Exception as e:<br>            logger.error(e)<br>            return {&#39;code&#39;: 500, &#39;message&#39;: str(e)}<br>        return {&#39;code&#39;: 200}<br><br>    def get_recommendations(self, user_id):<br>        request_id = str(uuid.uuid4())<br>        logger.info(&quot;The process has started&quot;)<br>        user_vector = Utils.get_vector(USERS_COLLECTION, f&quot;user_id == {user_id}&quot;)<br>        user_artist_vector = Utils.get_vector(USERS_ARTISTS_COLLECTION, f&quot;user_id == {user_id}&quot;)<br>        recommendations = {}<br><br>        seen_songs = {int(song_id.decode(&#39;utf-8&#39;)) for song_id in REDIS_CLIENT.smembers(f&quot;user:{user_id}:seen_songs&quot;)}<br><br>        if user_vector:<br>            logger.info(f&quot;User {user_id} has a user_vector&quot;)<br>            song_ids, artists, songs = self.find_songs(user_vector, limit=100)<br>            filtered_songs = [(song_id, artist, song) for song_id, artist, song in zip(song_ids, artists, songs) if<br>                              song_id not in seen_songs]<br>            shuffle(filtered_songs)<br>            selected_songs = filtered_songs[:min(2, len(filtered_songs))]<br><br>            recommendations[&#39;online_user&#39;] = {<br>                &#39;song_ids&#39;: [song[0] for song in selected_songs],<br>                &#39;artists&#39;: [song[1] for song in selected_songs],<br>                &#39;songs&#39;: [song[2] for song in selected_songs]<br>            }<br><br>            if user_artist_vector:<br>                artist_names = self.find_artists(user_artist_vector, limit=20)<br>                song_ids, artists, songs = self.get_songs_by_artists(artist_names, seen_songs=seen_songs)<br><br>                all_songs = list(zip(song_ids, artists, songs))<br>                shuffle(all_songs)<br>                selected_songs = all_songs[:min(3, len(all_songs))]<br><br>                recommendations[&#39;online_user_artists&#39;] = {<br>                    &#39;song_ids&#39;: [song[0] for song in selected_songs],<br>                    &#39;artists&#39;: [song[1] for song in selected_songs],<br>                    &#39;songs&#39;: [song[2] for song in selected_songs]<br>                }<br><br>            cold_start_song = sample(COLD_START_SONGS, 1)<br>            cold_start_song_ids, cold_start_artists, cold_start_songs = Recommender.get_songs_description(cold_start_song)<br>            logger.info(f&#39;Cold Start songs retrieved for User: {user_id}&#39;)<br>            recommendations[&#39;coldstart&#39;] = {&#39;song_ids&#39;: cold_start_song_ids,<br>                                            &#39;artists&#39;: cold_start_artists,<br>                                            &#39;songs&#39;: cold_start_songs}<br><br>        else:<br>            recommended_songs = sample(COLD_START_SONGS, 5)<br>            song_ids, artists, songs = self.get_songs_description(recommended_songs)<br>            recommendations[&#39;coldstart&#39;] = {&#39;song_ids&#39;: song_ids,<br>                                            &#39;artists&#39;: artists,<br>                                            &#39;songs&#39;: songs}<br><br>        formatted_vector = self.format_vector(user_vector) if user_vector is not None else &#39;None&#39;<br>        self.publish_recommendation_info(request_id, user_id, recommendations)<br>        return request_id, recommendations, formatted_vector</pre><h4>Processing User Reactions</h4><p>The KafkaEventConsumer class is designed to analyze user reactions to received song recommendations. It is used to read impression events from the Kafka events topic and update the user&#39;s interest vector in case of a positive reaction to the recommended song.</p><p>Key methods and functions:</p><ul><li>process_impression_event: The method for processing the impression event. It analyzes whether the user liked the song and, in case of a positive reaction, updates the user&#39;s interest vector in the corresponding Milvus collections.</li><li>update_vector: The method for updating the user&#39;s interest vector. If the vector already exists, it is updated by averaging it with the new vector. If the vector does not exist, it is created in the corresponding Milvus collection.</li></ul><p>How it works:</p><ul><li>The Kafka consumer (KafkaEventConsumer) reads events from the specified topic.</li><li>For each impression event, it checks whether the user liked the song (is_like).</li><li>If the song is liked, the user’s interest vector (user_vector) and the user&#39;s interest vector for artists (user_artist_vector) are updated in the corresponding Milvus collections. The vectors are updated by averaging them with the song vector (song_vector) and the artist vector (artist_vector).</li><li>Regardless of the user’s reaction, the song identifier is added to a Redis set to track the songs already shown to the user.</li><li>The updated user vectors are used in the subsequent issuance of more accurate recommendations.</li></ul><pre>class KafkaEventConsumer:<br>    def __init__(self, bootstrap_servers, group_id, topic):<br>        self.config = {<br>            &#39;bootstrap.servers&#39;: bootstrap_servers,<br>            &#39;group.id&#39;: group_id,<br>            &#39;auto.offset.reset&#39;: &#39;earliest&#39;<br>        }<br>        self.topic = topic<br>        self.consumer = Consumer(self.config)<br>        self.consumer.subscribe([self.topic])<br><br>    def process_impression_event(self, event_data):<br>        user_id = event_data[&#39;user_id&#39;]<br>        song_id = event_data[&#39;song_id&#39;]<br>        artist = event_data[&#39;artist&#39;]<br>        is_like = event_data[&#39;is_like&#39;]<br><br>        if is_like:<br>            user_vector = Utils.get_vector(USERS_COLLECTION, f&quot;user_id == {user_id}&quot;)<br>            song_vector = Utils.get_vector(SONGS_COLLECTION, f&quot;id == {song_id}&quot;)<br>            artist_vector = Utils.get_vector(ARTISTS_COLLECTION, f&quot;artist == &#39;{artist}&#39;&quot;)<br><br>            self.update_vector(USERS_COLLECTION, user_id, user_vector, song_vector, &quot;user&quot;)<br><br>            user_artist_vector = Utils.get_vector(USERS_ARTISTS_COLLECTION, f&quot;user_id == {user_id}&quot;)<br>            self.update_vector(USERS_ARTISTS_COLLECTION, user_id, user_artist_vector, artist_vector, &quot;user-artist&quot;)<br><br>        REDIS_CLIENT.sadd(f&quot;user:{user_id}:seen_songs&quot;, song_id)<br>        logger.info(f&quot;The song {song_id} for User {user_id} has been saved to Redis&quot;)<br><br>    def update_vector(self, collection, user_id, existing_vector, new_vector, vector_type):<br>        if new_vector is not None:<br>            if existing_vector is not None:<br>                updated_vector = np.mean([np.array(existing_vector), np.array(new_vector)], axis=0).astype(np.float32)<br>                collection.upsert([[int(user_id)], [updated_vector]])<br>                logger.info(f&#39;Updated {vector_type} vector for user {user_id}&#39;)<br>            else:<br>                new_vector = np.array(new_vector).astype(np.float32)<br>                collection.insert([[int(user_id)], [new_vector]])<br>                logger.info(f&#39;Inserted new {vector_type} vector for user {user_id}&#39;)<br>        else:<br>            logger.warning(f&#39;Vector for {vector_type} not found&#39;)<br><br>    def consume_events(self):<br>        try:<br>            while True:<br>                msg = self.consumer.poll(timeout=1.0)<br><br>                if msg is None:<br>                    continue<br>                if msg.error():<br>                    if msg.error().code() == KafkaError._PARTITION_EOF:<br>                        continue<br>                    else:<br>                        logger.error(msg.error())<br>                        break<br><br>                event = json.loads(msg.value().decode(&#39;utf-8&#39;))<br>                if event[&#39;event_name&#39;] == &#39;impression&#39;:<br>                    self.process_impression_event(event)<br><br>        except KeyboardInterrupt:<br>            pass<br>        finally:<br>            self.consumer.close()</pre><h3>Demonstration of the system’s operation</h3><p>Let’s explore how the system works using the example of a new user who likes Heavy Metal music, etc.</p><p>The user with ID 1030 is new to our system, and there’s no information about their interests yet, so they receive recommendations from the Cold Start. Suppose that from all the recommendations, they liked the song “Parasite” by Venom.</p><p>Clicking the Like button on the test web form generates a call to the put_impression method of the Recommender class, followed by the publication of the event in Kafka and processing by the Kafka Consumer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FAvjB2qWANGE57Mk08_K0Q.png" /></figure><p>During the next request for the list of recommendations, we see the user’s vector and the response from three models: <strong>Cold Start, online_user, and online_user_artist</strong>. Let’s assume that from all the recommendations, the user liked the song “Only The Good Die Young” by Iron Maiden:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8xdVtJ1tC23RoegRQgi2A.png" /></figure><p>After that, the user’s vectors will be updated again, and the recommendations will appear as shown in the slide below. Let’s imagine that the user now liked the song “Guilty As Hell” by Black Sabbath:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-nZtHDIij-T8sNElSfQnlA.png" /></figure><p>Subsequent recommendations will primarily feature bands that play in heavy styles with similar themes in their songs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-LrpS8N7wiIDkYN2CDx8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fc474sjoKh9z3azdvT56sA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_IRQ8Q02YN26vK0xwzHIA.png" /></figure><h3>Conclusion</h3><p>In this article, we explored a music recommendation system based on the use of the Milvus vector database. We demonstrated how to build a basic recommendation system relying solely on one feature — song lyrics. This approach serves as a template that offers extensive opportunities for further expansion and improvement.</p><p>The development of recommendation systems is a dynamically evolving field, where each new enhancement or added feature can significantly improve the quality and accuracy of suggestions. Caching results, more detailed filtering and ranking of recommendations, and incorporating additional song parameters such as genre, theme, tempo, and loudness can greatly enrich the user experience.</p><p>Onboarding users and collecting information about their preferences will allow the system to better adapt to individual tastes and offer more personalized recommendations. Perfecting these aspects can transform a basic recommendation system into a powerful tool capable of accurately determining users’ musical preferences and offering them exactly the compositions they would like to hear.</p><p>I sincerely hope you found this article insightful and enjoyable. I’d love to hear your thoughts, ideas, or any feedback you might have. Your input is invaluable, not just for the growth of this project, but also for the broader conversation around music recommendation systems. Let’s continue this journey together, with the aim of making the world of music more personalized and harmonious for every listener.</p><p>You can find the entire source code of the project in <a href="https://github.com/igorgorbenko/songs_recommendation">my GitHub repository</a>.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9711609ed36" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/harmony-in-data-the-music-recommendation-system-with-milvus-c9711609ed36">Harmony in Data: The Music Recommendation System with Milvus</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real Programmers Also Do Documentation: Automating Documentation Through GitHub and ChatGPT]]></title>
            <link>https://python.plainenglish.io/real-programmers-also-do-documentation-automating-documentation-through-github-and-chatgpt-242a7efcb830?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/242a7efcb830</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[gi̇thub]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Mon, 11 Mar 2024 07:19:07 GMT</pubDate>
            <atom:updated>2024-04-09T00:40:58.159Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*9nPIddYwbolhKo9umMp0PA.png" /><figcaption>The imagination of a neural network</figcaption></figure><blockquote>“A system without documentation is like a house without a door. It doesn’t matter how well the house has been constructed if you can’t enter the house. And if a program cannot be used, it doesn’t matter how efficient or elegant the program is”, says Bill Inmon in <a href="https://www.linkedin.com/pulse/real-programmers-dont-do-documentation-bill-inmon/?trackingId=%2BDMaLjXFfCvii%2BSVXJ5pZg%3D%3D">his LinkedIn article.</a></blockquote><p>It’s difficult to argue against this statement.</p><p>In practice, programmers often encounter a huge number of poorly documented systems and projects, and they sometimes spend considerable effort and time studying the features and understanding how the system or program handed over by a predecessor is structured.</p><p>Over many years in IT, I have encountered only a few cases where active development went hand in hand with equally active updating of project documentation. Documentation preparation is often perceived by programmers as a routine, uninteresting, and even burdensome task. Technical specialists sometimes find it difficult to express their thoughts concisely and clearly, which only exacerbates the process of working on the system description. <strong>Most programmers would much rather write code than update a page in the corporate knowledge base</strong>.</p><h3>Why is this important?</h3><p>Documentation plays a crucial role in the software development lifecycle. It not only facilitates understanding of the code for new team members but is also an integral part of maintaining and updating existing systems. Well-structured documentation includes descriptions of system architecture, functions, interfaces, and algorithms, allowing developers to quickly navigate the code and efficiently implement changes.</p><p>Without adequate documentation, even the most experienced programmers can struggle to comprehend the system’s logic or the principles of its component interactions. This can lead to increased development and testing time, as well as a higher risk of errors when modifying the code.</p><blockquote>But is there a way to overcome this problem? How can we assist programmers in creating clear and comprehensible descriptions of their work with minimal effort?</blockquote><h3>Automation as a Solution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GRH9ZK4xtpNYe2pne9RPXA.png" /></figure><p>To address the issue of missing or outdated documentation, it is proposed to automate the process of its creation and updating using <a href="https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions">GitHub Actions</a> and the <a href="https://platform.openai.com/docs/api-reference/introduction">ChatGPT API</a>. This approach allows for automatic generation and updating of documentation following changes in the code, thereby ensuring its relevance and reducing the burden on developers.</p><blockquote>GitHub Actions is a powerful platform for automating tasks, which can be integrated into the <a href="https://docs.github.com/en/actions/using-workflows/about-workflows">Git Flow workflow</a>. Configured GitHub Workflow processes can automatically perform operations in response to various events, manually or on a schedule.</blockquote><blockquote>The core of this solution involves a workflow that is triggered when changes are merged into the DEV branch. This automated process scans the modified files and utilizes the ChatGPT API to generate up-to-date documentation that accurately reflects the latest code modifications. Upon creating a release and integrating changes from the DEV branch into the MAIN branch, the documentation is automatically updated and subsequently published on GitHub Pages, ensuring that it remains current and accessible.</blockquote><p>Although there are numerous ready-made tools for automatic documentation generation, such as <a href="https://www.sphinx-doc.org/en/master/">Sphinx for Python</a>, Javadoc for Java, and <a href="https://pkg.go.dev/golang.org/x/tools/cmd/godoc">godoc for Go</a>, which analyze comments and directory structures, this article will focus on utilizing ChatGPT. Thanks to its versatility, ChatGPT enables the generation of more informative descriptions by taking into account the context of the code.</p><h3>Implementation of the solution</h3><h4>Registering an account with OpenAI and receiving API Keys</h4><p>After completing the simple registration process on the OpenAI website, you can create your API Key, which looks like “sk-*****”. Save the obtained API key in a secure place, as it will be used to authenticate your requests to the ChatGPT API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sdx53NLQOrAEdrGWsY9JQ.png" /></figure><h4>Example code for using the ChatGPT API</h4><p>Below is an example code of the prepare_documentation function, where a call is made to the <a href="https://platform.openai.com/docs/models/gpt-3-5-turbo">gpt-3.5-turbo</a> ChatGPT model, which takes a task description specified in the system_content variable. The input to this function is the code text for which a description needs to be prepared and documentation created, which will be hosted on GitHub Pages.</p><p>To limit ChatGPT&#39;s imagination when generating documentation and focus its work exclusively on the provided code, we specify this during model setup:</p><pre>system_content = &quot;&quot;&quot;<br>    Your task is to assist in generating documentation for code <br>    from a repository. <br>    Based on the code provided, you need to return the documentation <br>    that is ready for publication on GitHub Pages. <br>    You will only use the provided code using the provided data <br>    and nothing else.&quot;&quot;&quot;</pre><p>Full text of the function prepare_documendation:</p><pre>import openai<br><br><br>def prepare_documentation(code_snippet):<br>    system_content = &quot;&quot;&quot;<br>        Your task is to assist in generating documentation for code <br>        from a repository. <br>        Based on the code provided, you need to return the documentation <br>        that is ready for publication on GitHub Pages. <br>        You will only use the provided code using the provided data <br>        and nothing else.&quot;&quot;&quot;<br><br>    user_query = f&quot;&quot;&quot;<br>        Code snippet:<br>        ---<br>        {code_snippet}<br>        --- <br>        Provide a documentation for this code suitable for GitHub Pages.&quot;&quot;&quot;<br>    <br>    response = openai.chat.completions.create(<br>        model=&quot;gpt-3.5-turbo&quot;,<br>        messages=[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: system_content},<br>                  {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_query}],<br>        temperature=0.7,<br>        top_p=1<br>    )<br><br>    return response.choices[0].message.content</pre><p>The get_changed_files function is used to retrieve a list of files that need to be checked for changes:</p><pre>def get_changed_files():<br>    result = subprocess.run([&#39;git&#39;, &#39;rev-parse&#39;, &#39;--verify&#39;, &#39;HEAD~1&#39;], stderr=subprocess.PIPE)<br>    if result.returncode != 0:<br>        result = subprocess.run([&#39;git&#39;, &#39;ls-files&#39;], stdout=subprocess.PIPE)<br>    else:<br>        result = subprocess.run([&#39;git&#39;, &#39;diff&#39;, &#39;--name-only&#39;, &#39;HEAD~1&#39;], stdout=subprocess.PIPE)<br>    changed_files = result.stdout.decode(&#39;utf-8&#39;).splitlines()<br>    return changed_files</pre><p>The get_changed_files() function is used to retrieve a list of files that have been changed in the most recent commit. Here&#39;s what happens in the function:</p><ol><li>The function first attempts to verify the existence of a previous commit using git rev-parse --verify HEAD~1. This command checks if there is a commit before the current HEAD.</li><li>If there is no previous commit (which would be the case for a newly initialized repository), the return code of the command will be non-zero. In this case, the function uses git ls-files to list all files in the repository, as all files would be considered &quot;changed&quot; in the first commit.</li><li>If there is a previous commit, the function proceeds to use git diff --name-only HEAD~1 to list the names of files that have changed between the previous commit and the current HEAD.</li><li>The output of the git command is then decoded from bytes to a UTF-8 string and split into individual lines to create a list of changed file names.</li><li>Finally, the function returns the list of changed files.</li></ol><h4>Setting up a GitHub repository</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FPABuIji_biAqBXzyyUesQ.png" /></figure><p>In a new GitHub repository, you can create a structure similar to the one shown in the picture above:</p><ul><li>generate_documentation.py: This script contains the logic for scanning the project&#39;s source code directory ./src. The script will be executed every time a Pull Request from a feature branch is successfully merged into the dev branch.</li><li>In the ./src directory, I placed dummy code that simulates playing musical instruments. Every time there is a change in the code within this directory, the documentation in the ./docs directory will be updated.</li><li>To automate all this, we will create a file ./.github/workflows/generate_documentation.yml with the description of the GitHub Workflow logic.</li></ul><h4>GitHub Workflow Sample Code</h4><pre>name: Generate Documentation<br><br>on:<br>  pull_request:<br>    types: [closed]<br>    paths:<br>      - &#39;src/**&#39;<br><br>jobs:<br>  build:<br>    if: github.event.pull_request.merged == true &amp;&amp; github.ref == &#39;refs/heads/dev&#39;<br>    runs-on: ubuntu-latest<br><br>    steps:<br>    - uses: actions/checkout@v2<br><br>    - name: Set up Python<br>      uses: actions/setup-python@v2<br>      with:<br>        python-version: &#39;3.x&#39;<br><br>    - name: Install dependencies<br>      run: |<br>        python -m pip install --upgrade pip<br>        pip install openai<br><br>    - name: Generate Documentation<br>      env:<br>        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}<br>      run: |<br>        python generate_documentation.py<br><br>    - name: Setup SSH<br>      run: |<br>        mkdir -p ~/.ssh<br>        echo &quot;${{ secrets.SSH_PRIVATE_KEY }}&quot; &gt; ~/.ssh/id_rsa<br>        chmod 600 ~/.ssh/id_rsa<br>        ssh-keyscan github.com &gt;&gt; ~/.ssh/known_hosts<br><br>    - name: Commit and Push Documentation<br>      run: |<br>        git config --global user.name &#39;GitHub Actions&#39;<br>        git config --global user.email &#39;actions@github.com&#39;<br>        if git status --porcelain | grep &#39;docs/&#39;; then<br>          git add docs/<br>          git commit -m &quot;Update documentation&quot;<br>          git push git@github.com:igorgorbenko/gpt_doc_generator.git dev<br>        else<br>          echo &quot;No changes to commit.&quot;<br>        fi</pre><p>Let’s examine an example of the generate_documentation.yml file:</p><ul><li>Every time code is merged from child branches into the dev branch, if changes were made in the ./src directory, the GitHub Workflow will be triggered.</li><li>The ChatGPT API key (OPENAI_API_KEY) is read from the repository secrets and placed in the environment variables where the GitHub Workflow is running.</li><li>The generate_documentation.py script creates an updated index.md file in the ./docs directory.</li><li>The private key (SSH_PRIVATE_KEY) is read from the repository secrets, allowing GitHub to make changes to the dev branch after the documentation update process is completed.</li></ul><h4>Adding keys to the repository</h4><p>To complete the setup, you need to perform the following actions: add two keys — <strong>OPENAI_API_KEY</strong> and <strong>SSH_PRIVATE_KEY</strong> — to the secrets section of your GitHub repository, and also place the corresponding public key in the Deploy keys section to ensure access to the repository. To do this, go to the settings (Settings) of your repository, select the “Secrets and variables” tab, then go to the “Actions” subsection and add the necessary Repository Secrets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hAl7Bu5YaOUY-xAkT_65jw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nDIQxUobJL3eQg2zF9Jh3g.png" /></figure><h4>Setting up GitHub Pages</h4><p>In the GitHub Pages section, you need to specify the branch and the directory from which the documentation will be taken for the main page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m2HfkiTAGP7AD_Eb80ZNgA.png" /></figure><h3>Demo</h3><ul><li>After setting up the GitHub Workflow, every push to the dev branch of your repository will automatically trigger the documentation generation process.</li><li>You can check the results of the workflow in the “Actions” section of your GitHub repository, where the logs of each step of your workflow will be displayed.</li><li>The generated documentation will be available at the link <a href="https://[username].github.io/[repository_name]/">https://[username].github.io/[repository_name]/</a> after merging the dev branch into the main branch.</li></ul><p>In the feature branch, let’s add another guitarist named Kerry to the rock band, commit the code, and create a Pull Request into the dev branch:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wMXhXb2NJwXbl8NawHigxw.png" /></figure><p>The Actions tab displays the execution process for each GitHub Workflow step</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A1Vd6pURPB6u8BO4BLus4Q.png" /></figure><p>A commit appeared in the list of commits, which was added automatically — “Update documentation”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SZTd-DJ6J1xvEGp6dj1Aug.png" /></figure><p>To complete the process, you need to merge the dev and main branches and check that the project documentation has been updated with the new guitarist. After the dev branch is merged into main, GitHub will initiate the build and deployment process of the updated documentation to GitHub Pages – known as &quot;pages build and deployment&quot;.</p><p>You can do this by creating a Pull Request from the dev branch to the main branch and then merging it. Once merged, GitHub Pages will automatically update to reflect the changes in the documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lf5p5PlDCCsYxFKYUgiqFw.png" /></figure><p>The automatically generated documentation for the project looks like this and is available at <a href="https://[username].github.io/[repository_name]/">https://[username].github.io/[repository_name]/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bMtTzuXfBeKIyZeRUxlnJA.png" /></figure><p>You can find the link to the source code for the article on <a href="https://github.com/igorgorbenko/gpt_doc_generator">my GitHub</a>. Documentation automatically <a href="https://igorgorbenko.github.io/gpt_doc_generator/">generated using ChatGPT is here</a></p><h3>Conclusion</h3><p>In this review article, we explored one way to address the pressing issue of missing or outdated documentation in software development. The proposed method, combining automation through GitHub Actions and the capabilities of the ChatGPT API, opens new horizons in approaches to maintaining and updating documentation.</p><p>For very large projects with an extensive codebase, where processing the entire volume of changes can become problematic, it’s important to consider optimization options. This may include breaking down the code into smaller segments for phased processing, incremental documentation generation, caching results, and limiting the frequency and volume of API requests. Such approaches can help reduce the system load, lower costs, and provide a more efficient and economical solution for keeping documentation up to date.</p><p>It should be noted that the proposed solution is just a framework that requires further refinement before it is ready for implementation in a production environment. Depending on the specific requirements of the project and the characteristics of the codebase, additional configuration and fine-tuning of the responses generated by the ChatGPT model may be necessary to ensure maximum relevance and accuracy of the documentation.</p><p>I am open to discussions and ready to answer all your questions. Your feedback and comments on the article will help improve and deepen the proposed solutions, making them more versatile and applicable in various development scenarios. Let’s work together to promote a culture of documentation automation, making this process not only less labor-intensive but also more effective for projects of any scale!</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=242a7efcb830" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/real-programmers-also-do-documentation-automating-documentation-through-github-and-chatgpt-242a7efcb830">Real Programmers Also Do Documentation: Automating Documentation Through GitHub and ChatGPT</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MLOps: Synergy of Data Science, Engineering, and Operations]]></title>
            <link>https://aws.plainenglish.io/mlops-synergy-of-data-science-engineering-and-operations-c1f046505c56?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1f046505c56</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 10:31:23 GMT</pubDate>
            <atom:updated>2024-04-09T13:32:34.150Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1pSEOjUMGTlu9CUO8c-osw.png" /><figcaption>The imagination of a neural network</figcaption></figure><p><strong>“People love to be cared for” </strong>— a guiding phrase said to me by an engineering manager many years ago when I was still a student and was working part-time developing websites during the summer holidays.</p><p>And he was right in some way. People love care, and it doesn’t matter who expresses it — other people or machines and algorithms. The world is evolving, and ML is actively being integrated into all modern systems and applications. We all appreciate personalized recommendations and “smart” services that adapt to our preferences. It’s nice to realize that behind the scenes of your favorite mobile app with interesting video plots, someone is constantly working to make sure you get personalized recommendations, carefully selected especially for you according to your interests. Great, isn’t it?</p><p>But how do you ensure that these algorithms are up-to-date, the data they use are fresh, and the recommendations users receive are relevant? How can the research of data scientists bring value to the business, instead of just remaining in Jupyter Notebook? How does the data accumulated by the company work for it and make users happier and more loyal? How to assess the quality of the recommendations given by ML systems and, accordingly, the impact of data products on the company’s business? MLOps answers all these questions.</p><p>MLOps is a set of practices combining machine learning (<strong>ML</strong>), development (<strong>Dev</strong>), and operations (<strong>Ops</strong>) to provide efficient and sustainable development and deployment of machine learning models. In the modern business environment, where data is becoming a key asset, MLOps plays a crucial role, ensuring the fast, efficient, and reliable creation and deployment of models. In this article, we explore the structure of MLOps, divided into four main areas of responsibility: <strong>Data Engineering, Data Science, ML Engineering, and DevOps.</strong></p><h3>Principles and benefits of MLOps</h3><p>MLOps represents an innovative methodology aimed at the integration and automation of machine learning processes from start to finish. The main goal of MLOps is to eliminate barriers between development, testing, and operations to ensure fast and efficient delivery of machine learning models into production.</p><p>The main principles of MLOps include:</p><ul><li><strong>Automation and Standardization</strong>: Automating workflows and standardizing practices to accelerate development and reduce errors.</li><li><strong>Continuous Integration and Delivery (CI/CD)</strong>: Continuous testing and delivery of changes to ensure quality and stability.</li><li><strong>Monitoring and Optimization:</strong> Constant monitoring and optimization of models to maintain their relevance and efficiency.</li><li><strong>Collaboration</strong>: Integrating development teams, DevOps, and data science for knowledge sharing and synergy.</li></ul><blockquote>The advantages of MLOps are numerous and multifunctional — from improving efficiency and quality to flexibility and synergy. These advantages make MLOps an integral part of modern organizations striving to maximize the potential of machine learning to achieve their business goals and meet client needs.</blockquote><h4><strong>Enhancement of Productivity and Efficiency</strong></h4><p>One of the main advantages of implementing MLOps is a substantial increase in productivity and efficiency. The MLOps methodology allows for the automation and optimization of numerous processes related to the development, testing, and deployment of machine learning models. That leads to a reduction in the time required for developing and deploying models and enables data specialists and developers to use their time and resources more efficiently.</p><h4><strong>Improvement in the Quality of Models and Solutions</strong></h4><p>MLOps also aids in enhancing the quality of machine learning models and the solutions developed based on them. Standardization of processes, continuous testing, and monitoring of models within MLOps ensure high accuracy and reliability of the developed systems. That minimizes the risk of errors and allows for the quick detection and correction of problems, ultimately leading to the creation of higher-quality products and services.</p><h4><strong>Flexibility and Scalability</strong></h4><p>The application of MLOps ensures high flexibility and scalability of machine learning systems. Organizations can quickly and easily adapt their models and systems to changing requirements and conditions and can scale them in accordance with growing volumes of data and business needs. This is especially important in a dynamically changing and competitive business environment where the ability to quickly respond to changes is a key to success.</p><h4><strong>Synergy among Various Fields</strong></h4><p>MLOps fosters synergy between various fields such as data science, software development, and IT operations. That ensures closer interaction between different teams and departments, leading to the exchange of knowledge, ideas, and best practices, and allowing for the creation of innovative solutions.</p><h3>Areas of Responsibility in MLOps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MEPKctDgZtL1oJiFQpjwUg.gif" /></figure><p>In the MLOps methodology, four main areas of responsibility are identified, each playing a crucial role in the development and deployment of machine learning systems.</p><h4>Data Engineering</h4><p>Data Engineering plays a key role, providing quality and accessible data for the entire machine learning process. Without timely delivery of data, machine learning models will be ineffective and unreliable. Data specialists work closely with Data Scientists to provide the necessary data and with DevOps to ensure efficient storage and access to data.</p><blockquote><strong>Example</strong>: Data Engineers can work with Data Scientists to develop optimized data processing pipelines that accelerate the data preparation process.</blockquote><p>Tasks and Responsibilities:</p><ul><li><em>Development and support of data collection and storage systems.</em></li><li><em>Ensuring the quality, availability, and security of data.</em></li><li><em>Creation and optimization of data processing pipelines.</em></li></ul><p>In this area, there are three key blocks:</p><ul><li><strong>Data Collecting</strong>: Gathering data sources necessary for training and validating models.</li><li><strong>Data Acquisition</strong>: Preparation and processing of data for further analysis.</li><li><strong>Feature Store/Data Store</strong>: Storing data and features used in models.</li></ul><h4>Data Science</h4><p>The Data Science area focuses on the development and validation of machine learning models that can be used for prediction and analysis. Collaboration with Data Engineering and ML Engineering ensures that models are developed and trained based on high-quality data and are integrated into product systems seamlessly.</p><blockquote><strong>Example</strong>: Data Scientists can collaborate with ML Engineers to determine the optimal parameters and algorithms for training models, improving their accuracy and efficiency.</blockquote><p>Tasks and Responsibilities:</p><ul><li><em>Conducting research and analysis of data.</em></li><li><em>Development and validation of machine learning models.</em></li><li><em>Interpretation of results and conclusions.</em></li></ul><p>Blocks:</p><ul><li><strong>Data Analysis:</strong> identifying patterns and insights in data.</li><li><strong>Feature Engineering:</strong> Creating new features to enhance models.</li><li><strong>Model Training</strong>: Training models based on preprocessed data and features.</li><li><strong>Model Evaluation</strong>: Assessing the quality and efficiency of models.</li></ul><h4>ML Engineering</h4><p>Provides a bridge between model development and its deployment, focusing on optimization, versioning, and monitoring of models. Close interaction with Data Scientists and DevOps allows for quick and efficient integration of models into production and ensures their stable operation.</p><blockquote><strong>Example:</strong> ML Engineers can work with DevOps to create automated CI/CD pipelines, simplifying the process of deploying models and updates.</blockquote><p>Tasks and Responsibilities:</p><ul><li><em>Development and optimization of systems for training and deploying models.</em></li><li><em>Versioning and managing models and code.</em></li><li><em>Monitoring and optimizing the performance of models.</em></li></ul><p>Blocks:</p><ul><li><strong>Trained Model / Support Code</strong>: Development and support of trained models and accompanying code.</li><li><strong>Version Contro</strong>l: Managing versions of models and code.</li><li><strong>Train Process:</strong> Managing the processes of training models.</li><li><strong>Unit Tests:</strong> Testing models and code for correctness and stability.</li></ul><h4>DevOps</h4><p>DevOps focuses on the infrastructure needed for deploying and operating models and on automating the respective processes. Interaction with ML Engineering ensures smooth deployment and monitoring of models, and collaboration with Data Engineering provides efficient data management.</p><blockquote><strong>Example:</strong> DevOps can collaborate with Data Engineering to optimize data storage, improving system performance and reducing storage costs.</blockquote><p>Tasks and Responsibilities:</p><ul><li><em>Development and support of infrastructure for deploying and operating models.</em></li><li><em>Automation of deployment processes and monitoring.</em></li><li><em>Ensuring the reliability and availability of services.</em></li></ul><p>Blocks:</p><ul><li><strong>Model Registry</strong>: Storing and managing model versions.</li><li><strong>Build Process:</strong> Automating build and deployment processes.</li><li><strong>Deployment Process</strong>: Managing the processes of deploying models into production.</li><li><strong>Prediction Service:</strong> Providing services for obtaining inference from models.</li><li><strong>Monitoring/Logging</strong>: Monitoring performance and logging system events.</li></ul><h3>The Importance of Integrating Data Science Work</h3><p>The integration of Data Scientists’ work is critically important as it transforms scientific research into concrete business solutions. These specialists extract knowledge from data, and when their work is integrated with developer teams and business processes, it allows companies to utilize this knowledge more effectively for making informed decisions and developing innovative products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*nRR06ytKAGLoqhR7VopSVg.png" /><figcaption>The imagination of a neural network</figcaption></figure><blockquote>Taking care of the Data Science team and their close integration with other departments creates conditions for developing exceptional innovative solutions, which in turn bring significant value to the business.</blockquote><ul><li><strong>Speed of Implementation</strong>: Integration accelerates the process of transforming research into real products and solutions, ensuring promptness in implementing ideas.</li><li><strong>Quality Solutions</strong>: Collaborative work of the Data Science team, developers, and business analysts leads to the creation of more accurate and effective solutions that precisely reflect the needs and requirements of the business.</li><li><strong>Strategic Decision-Making:</strong> Integration improves the understanding and use of data for strategy formation and data-driven business decision-making.</li></ul><p>Thus, the integration of the work of Data Scientists is not just a way to enhance product development and the decision-making process, but it also serves as a catalyst for innovation and the creation of a competitive advantage. It allows organizations to understand their data more deeply, bring ideas to life faster, and develop products and services that best satisfy the needs and expectations of customers.</p><h3>A Real-World MLOps Example: Anomaly detection system</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*42StZYEtn9mUga0hzShqyg.jpeg" /></figure><blockquote>The Data Science team embarked on a mission replete with complexities and aspirations. Their goal was not just to detect anomalies and unearth uncommon patterns in the data but to refine these detections, reducing false positives and enhancing adaptability to newly emerging anomaly types. As the product landscape evolved, so did the need to perpetually expand and update the roster of features under control.</blockquote><p>Moreover, beyond the analytical challenges, there was an imperative for clear communication: results needed not only to be visualized but also articulated in a manner that offered actionable insights to clients about the anomalies identified.</p><p>Looking back, our initial attempts at detecting anomalies in user behavior were fraught with difficulties. We contended with an array of challenges, from delayed detections to outright misses, primarily due to the absence of a coherent and integrated process. Our operations were marred by segmented workflows, an over-reliance on manual processes, and constrained data processing capacities, all of which collectively delayed our reaction to potential threats. The lack of a consolidated infrastructure birthed data silos, obscuring the holistic view of user behaviors and patterns. As the intricacy and sheer volume of data surged, our conventional methodologies began to falter, stymieing our pace of innovation and adaptability.</p><p>Enter MLOps, utilizing the AWS cloud, which ushered in a transformative shift in our approach.</p><h4><strong>Main System Flow</strong></h4><ul><li>Regularly, the main application’s backend, located in the client’s environment, dispatches incremental raw data <strong>(1)</strong> for anomaly detection.</li><li>The Amazon CloudWatch Rules service acts as a scheduler, initiating a Lambda function <strong>(2)</strong>. Depending on the data volume and the feature set (which varies by client subscription), this function launches an Amazon EMR cluster equipped with Apache Spark.</li><li>Amazon EMR <strong>(3)</strong> stands as the core part of the system, overseeing all computational tasks.</li><li>Post-calculation, the results are stashed in an S3 bucket. Later, this data is uploaded to Amazon Redshift <strong>(4)</strong>, a potent columnar database adept at managing vast data arrays. Clients can then access analytical reports, based on Redshift data, via a Web UI.</li></ul><h4><strong>Unit Tests</strong></h4><ul><li>The Data Science team consistently fine-tunes the models, while the ML engineers amplify the training regimen and augment new features. Utilizing Amazon CloudWatch rules with Lambda, a test Amazon EMR cluster is activated daily <strong>(5)</strong>.</li><li>Spark applications operate on predefined test datasets daily <strong>(6)</strong>, navigating all scenarios. This produces comprehensive developer reports, highlighting potential disruptions in logic or instances where a new model version underperforms its predecessor.</li><li>Should all tests conclude without a hitch, we stand poised to craft a new code build for subsequent integration into the production cycle <strong>(7)</strong>.</li></ul><h4><strong>Model Training</strong></h4><ul><li>The system is constantly updated with fresh client data. To keep models current, their training on revised data is recurrent. Amazon CloudWatch rules routinely propel a Lambda <strong>(8)</strong>, setting up the Amazon EMR cluster.</li><li>The historical data is used to train a fresh model <strong>(9)</strong>. Simultaneously, model quality is ascertained against test data (a step not depicted in the diagram) to ensure the new iteration maintains or elevates the quality of inferences.</li><li>Concluding this phase, the newly minted model is deposited in the S3 bucket <strong>(10)</strong>, ready for usage in the main system flow.</li></ul><h4><strong>Team Contributions</strong></h4><ul><li><strong>The DevOps team</strong> set up the CI/CD pipelines, managed the Unit Tests, and addressed other infrastructure-related issues.</li><li><strong>The backend developers</strong> designed a regular data extraction process from the client’s environment and also developed the backend for an analytical tool that the client interacts with.</li><li><strong>Data engineers</strong> prepared data processing pipelines required for model training, established a data repository, and created an API to access system output for analysis.</li><li><strong>ML engineers </strong>developed cyclic model training processes, training process testing, and inference extraction.</li><li>On the foundation of the prepared platform, <strong>the data science team</strong> can build models, test hypotheses, run experiments, improve models, and develop new concepts for anomaly detection.</li></ul><p>The transition to an MLOps-focused approach has provided a structured, efficient, and agile framework. This not only ensures rapid and accurate anomaly detection but also fosters an environment of continuous improvement and innovation. Reflecting on a system I designed and enhanced years ago, it’s evident how much has evolved. With the advent of MLOps, we’d transitioned from disjointed efforts to a more streamlined and efficient approach, maximizing the value we delivered to our customers.</p><h3>Conclusion</h3><p>MLOps is a crucial approach in applying machine learning in business, synthesizing data science, development, and operations to create sustainable, scalable, and efficient solutions. This discipline is of colossal importance as it enables organizations to quickly adapt to changing market conditions, ensuring continuous optimization of processes and products.</p><p>In today’s world, where data is becoming the new gold, integrating the work of Data Scientists through MLOps becomes critically important. Close collaboration between data scientists, developers, and infrastructure specialists ensures not only the quality and accuracy of models but also their relevance and compliance with business requirements. That allows organizations to create innovative products and services that meet the real needs and expectations of users, truly converting scientific research into a competitive advantage.</p><p>In the future, MLOps is likely to continue evolving, adapting to new technological trends and business models, and making significant contributions to the development of machine learning technologies, considering the growing importance of issues related to ethics, transparency, and security in AI.</p><p>I hope that this article will not only help you better understand the significance of MLOps but also inspire the integration of its principles in your organization, opening up new horizons and opportunities for your business and society.</p><p>May your journey in the world of MLOps be fascinating and fruitful!</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1f046505c56" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/mlops-synergy-of-data-science-engineering-and-operations-c1f046505c56">MLOps: Synergy of Data Science, Engineering, and Operations</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Geospatial Data Analysis in ClickHouse: Polygons, Geohashes, and H3 Indexing]]></title>
            <link>https://medium.com/@igorvgorbenko/geospatial-data-analysis-in-clickhouse-polygons-geohashes-and-h3-indexing-2f55ff100fbe?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f55ff100fbe</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[geo]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 08:30:17 GMT</pubDate>
            <atom:updated>2023-08-17T08:30:17.726Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F8_zR1F0yK-mCiBy" /><figcaption>Photo by <a href="https://unsplash.com/@nicnut?utm_source=medium&amp;utm_medium=referral">Nicola Nuttall</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In an era of intensive business growth, understanding geospatial data becomes critically important for a company’s success. Modern business, especially in sectors like logistics and sales, places immense value on geospatial data. Analyzing the geographical distribution of customers can optimize delivery routes, suggest directions for expansion, and shape marketing strategies. In logistics, geospatial data helps reduce delivery time and costs, while in sales, it identifies potential growth points and determines markets for new products.</p><p>Geodata also finds applications in numerous other fields: from analyzing regions with a high risk of natural disasters for insurers to creating interactive weather forecast maps. However, despite its numerous applications, the primary challenge remains the efficient processing of large data volumes. In this article, we will explore three methods of processing geodata in ClickHouse: <strong>polygon</strong> search, the use of <strong>geohashes</strong>, and <strong>H3 indexing</strong>, comparing them based on parameters such as memory usage, processed bytes, and execution time.</p><p>Welcome to the world of geospatial analysis, where the significance of boundaries is determined less by their coordinates and more by the expertise in processing and interpreting them!</p><h3>Overview of Geohashes and H3 Indexing</h3><h4>Geohashes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zDkLe21xBOwSCwom-9bNbQ.png" /><figcaption>The geohash map</figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Geohash">Geohash</a> is a geocoding system that encodes geographic coordinates into a short string of characters. The geohashing system was proposed by Gustavo Niemeyer in 2008 and quickly gained traction due to its simplicity and efficiency.</p><p>The geohashing method works by recursively dividing space into squares. Initially, the world is divided into 32 large squares (4x8), each assigned a unique character from a set of 32 characters: numbers from 0 to 9 and specific letters from A to Z, excluding “A”, “I”, “L”, and “O”. These squares are subsequently divided into 32 smaller ones, each receiving the next character in sequence. Thus, the length of a geohash determines its precision: the longer the geohash, the finer the area it denotes.</p><p>In the realm of databases like ClickHouse, geohashes play a vital role. They are typically stored as strings and can be instrumental in searching and grouping data by location. For example, one could use a geohash to swiftly pinpoint all data within a distinct geographical zone.</p><blockquote>A key advantage of geohashes is their inherent hierarchical structure, enabling streamlined searches for associated “parent” geohashes.</blockquote><blockquote>For instance, given a geohash “gcpuvp” of 6-character precision, its “parent” geohash “gcp” of 3-character precision can be readily derived by trimming the trailing characters. Such a feature proves invaluable when conducting scaling operations and grouped queries</blockquote><h4>H3 Indexing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6tjjAe3oh39EEQJ8ezrwbw.png" /><figcaption>H3 indexing</figcaption></figure><p>H3 is a geocoding system developed by <a href="https://www.uber.com/en-AE/blog/h3/">Uber</a> that represents geographic space as a grid of hexagons. Each hexagon in the grid possesses a unique H3 index, and hexagons within the same level are of uniform size.</p><p>H3 indexes divide the Earth’s surface into hexagons of different nested levels, creating grids of varying precision. Similar to geohashes, a longer H3 index indicates greater precision.</p><p>In contrast to the geohash, an H3 index can be represented as an integer value (UInt64), which means that each H3 index will consistently occupy a fixed 8 bytes. Geohashes, being strings, can vary in memory consumption:</p><ul><li>An <strong>8-character geohash</strong> like <em>“udtsfn9w”</em> will take up <strong>9 bytes</strong> on disk: 8 bytes for the string characters and 1 byte for its length.</li><li>A <strong>2-character geohash</strong> like <em>“ud”</em> will occupy <strong>3 bytes</strong> on disk: 2 bytes for storing the string characters and 1 byte for storing its length.</li></ul><blockquote>Despite these differences, ClickHouse employs various optimizations when dealing with numerical data types such as UInt64. This is primarily because numerical operations usually execute faster than string operations, given that low-level hardware instructions can be leveraged for numerical tasks.</blockquote><h3>Comparison Methodology</h3><h4>Initial Preparation</h4><ol><li><strong>Environment:</strong> We will use the free version of ClickHouse in the cloud, available for 30 days on the official website. A brief instruction on its use can be found <a href="https://clickhouse.com/docs/en/cloud-quick-start">[here]</a></li><li><strong>Dataset:</strong> The primary dataset selected for analysis is the built-in <em>“Cell Towers”</em> dataset, which is <strong>1.1 GB</strong> in size. It can be <a href="https://clickhouse.com/docs/en/getting-started/example-datasets/cell-towers#load-the-sample-data">easily imported</a> into your ClickHouse instance</li><li><strong>Comparison Methods:</strong></li></ol><ul><li><em>Filtering by polygon only</em></li><li><em>Filtering using geohashes and a polygon</em></li><li><em>Filtering by H3 indexes and a polygon</em></li></ul><p>4. <strong>Metrics</strong>:</p><ul><li><em>Query execution time</em></li><li><em>Number of scanned rows</em></li><li><em>The volume of scanned bytes</em></li><li><em>Memory used for the operation</em></li></ul><p>5. <strong>Location Types:</strong> Analysis will be carried out in three different geographical areas:</p><ul><li><em>Nevsky Prospect in St. Petersburg</em> — small scale</li><li><em>Istanbul, Turkey </em>— medium scale</li><li><em>Spain</em> — large scale</li></ul><p>6. <strong>Data Preparation:</strong> Considering the differences in dimensions between geohashes and H3, our goal is to ensure roughly the same number of values for filtering in each method. It’s necessary to create a table (or set of tables) with a specific sorting key:</p><ul><li><em>cell_towers_polygon</em>: ORDER BY (lon, lat, radio, mcc, net, created)</li><li><em>cell_towers_geo_hash_*</em>: ORDER BY (geo_hash_*, radio, mcc, net, created)</li></ul><p>Extracting the parent index from an H3 numeric value is not straightforward, so for the sake of experiment clarity, we will create three separate tables: <em>cell_towers_h3_10, cell_towers_h3_4, cell_towers_h3_3</em></p><blockquote><strong>Why is extracting the parent index from a child index in H3 not straightforward?</strong></blockquote><blockquote><strong>Complex Conversion:</strong> Unlike some systems where you can simply truncate or modify characters to derive the parent index (as is possible with geohash), H3 requires you to transform the child index into geographic coordinates, determine the coordinates of the parent hexagon, and then convert these back to an index.</blockquote><blockquote><strong>Binary Representation:</strong> H3 employs a binary representation for its indexing, which necessitates specialized algorithms and frameworks to extract a parent index.</blockquote><blockquote><strong>Hexagonal Boundary Constraints:</strong> Owing to the inherent nature of hexagonal division, child hexagons don’t always align flawlessly with the boundaries of their parent hexagons.</blockquote><h3>Examples of creating tables and preparing data</h3><p>Creating the table <em>geo.cell_towers_polygon</em>:</p><pre>CREATE DATABASE geo  <br><br>CREATE TABLE geo.cell_towers_polygon<br>(<br>    `radio` Enum8(&#39;&#39; = 0, &#39;CDMA&#39; = 1, &#39;GSM&#39; = 2, &#39;LTE&#39; = 3, &#39;NR&#39; = 4, &#39;UMTS&#39; = 5),<br>    `mcc` UInt16,<br>    `net` UInt16,<br>    `area` UInt16,<br>    `cell` UInt64,<br>    `unit` Int16,<br>    `lon` Float64,<br>    `lat` Float64,<br>    `range` UInt32,<br>    `samples` UInt32,<br>    `changeable` UInt8,<br>    `created` DateTime,<br>    `updated` DateTime,<br>    `averageSignal` UInt8<br>)<br>ENGINE = ReplicatedMergeTree(&#39;/clickhouse/tables/{uuid}/{shard}&#39;, &#39;{replica}&#39;)<br>ORDER BY (lon, lat, radio, mcc, net, created)<br>SETTINGS index_granularity = 8192;<br><br>INSERT INTO geo.cell_towers_polygon<br>SELECT * FROM default.cell_towers</pre><p>Creating the table <em>geo.cell_towers_geo_hash</em>:</p><pre>CREATE TABLE geo.cell_towers_geo_hash<br>(<br>    `radio` Enum8(&#39;&#39; = 0, &#39;CDMA&#39; = 1, &#39;GSM&#39; = 2, &#39;LTE&#39; = 3, &#39;NR&#39; = 4, &#39;UMTS&#39; = 5),<br>    `mcc` UInt16,<br>    `net` UInt16,<br>    `area` UInt16,<br>    `cell` UInt64,<br>    `unit` Int16,<br>    `lon` Float64,<br>    `lat` Float64,<br>    `range` UInt32,<br>    `samples` UInt32,<br>    `changeable` UInt8,<br>    `created` DateTime,<br>    `updated` DateTime,<br>    `averageSignal` UInt8,<br>    `geo_hash_9` String<br>  )<br>ENGINE = ReplicatedMergeTree(&#39;/clickhouse/tables/{uuid}/{shard}&#39;, &#39;{replica}&#39;)<br>ORDER BY (geo_hash_9, radio, mcc, net, created)<br>SETTINGS index_granularity = 8192;<br>  <br>INSERT INTO geo.cell_towers_geo_hash<br>SELECT *, geohashEncode(lon, lat, 9) FROM default.cell_towers</pre><p>Creating the table <em>geo.cell_towers_h3_*</em>:</p><pre>CREATE TABLE geo.cell_towers_h3_10<br>(<br>    `radio` Enum8(&#39;&#39; = 0, &#39;CDMA&#39; = 1, &#39;GSM&#39; = 2, &#39;LTE&#39; = 3, &#39;NR&#39; = 4, &#39;UMTS&#39; = 5),<br>    `mcc` UInt16,<br>    `net` UInt16,<br>    `area` UInt16,<br>    `cell` UInt64,<br>    `unit` Int16,<br>    `lon` Float64,<br>    `lat` Float64,<br>    `range` UInt32,<br>    `samples` UInt32,<br>    `changeable` UInt8,<br>    `created` DateTime,<br>    `updated` DateTime,<br>    `averageSignal` UInt8,<br>    `h3_10` UInt64<br>)<br>ENGINE = ReplicatedMergeTree(&#39;/clickhouse/tables/{uuid}/{shard}&#39;, &#39;{replica}&#39;)<br>ORDER BY (h3_10, radio, mcc, net, created)<br>SETTINGS index_granularity = 8192  <br><br>INSERT INTO geo.cell_towers_h3_10<br>SELECT *, geoToH3(lon, lat, 10) FROM default.cell_towers<br><br>-- Creating the table geo.cell_towers_h3_4:<br><br>CREATE TABLE geo.cell_towers_h3_4<br>...<br><br>ENGINE = ReplicatedMergeTree(&#39;/clickhouse/tables/{uuid}/{shard}&#39;, &#39;{replica}&#39;)<br>ORDER BY (h3_4, radio, mcc, net, created)<br>SETTINGS index_granularity = 8192;<br><br><br>-- Inserting datatint в geo.cell_towers_h3_4:<br>INSERT INTO geo.cell_towers_h3_4<br>SELECT *, geoToH3(lon, lat, 4) FROM default.cell_towers;<br><br>-- The same fro geo.cell_towers_h3_3<br><br>...</pre><h3>Source data for queries and expected results</h3><p>Having successfully created and populated five tables, we can now proceed to run data retrieval queries. As previously mentioned, we’re dealing with three locations. The images below depict the polygons corresponding to these areas, as well as the geohashes and H3 indexes encompassing them. We selected the precision of the geo-indexes for each example such that their count was roughly equivalent, ensuring at the same time that the area outside the polygon was kept to a minimum:</p><ul><li><strong>Nevsky Prospect, St. Petersburg</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGAGO1sHVG3HhON7wt525Q.png" /><figcaption>The polygon area and the area of geohashes with a precision of 7 are highlighted in red</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MtpzQ-WdivbARJZ0dBPUw.png" /><figcaption>The polygon area is highlighted in red and the visualization of H3 indices with a precision of 10</figcaption></figure><ul><li><strong>Istanbul, Turkey</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3UyLT0nMWGJMXOLKbR0ug.png" /><figcaption>The polygon area and the area of geohashes with a precision of 4 are highlighted in red</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6wCZCB3UpKSzc1TCwXFmw.png" /><figcaption>The polygon area is highlighted in red and the visualization of H3 indices with a precision of 4</figcaption></figure><ul><li><strong>Spain</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSHOh9CS8aF2jZV37f4ekA.png" /><figcaption>The polygon area and the area of geohashes with a precision of 3 are highlighted in red</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fY-Zc-XqXHXnB4CQMKlE-g.png" /><figcaption>The polygon area is highlighted in red and the visualization of H3 indices with a precision of 3</figcaption></figure><p>Let’s run three queries for each of the areas and gather the metrics</p><h3>Metrics collection</h3><h4>Getting results for the small scale (Nevsky Prospect, St. Petersburg)</h4><p>Executing the query with filtering by the polygon only on the table <em>geo.cell_towers_polygon</em>:</p><pre>select radio, count()<br>from geo.cell_towers_polygon<br>where pointInPolygon((lon, lat), <br>    [(30.31283445840512, 59.93731653223952), <br>    (30.312501800021835, 59.93659901654749), <br>    (30.359919789647535, 59.930538163165636),  <br>    (30.384620591702515, 59.92268781981602), <br>    (30.38561525487222, 59.924017109105904), <br>    (30.360950045625515, 59.93158921046111), <br>    (30.31283445840512, 59.93731653223952)])<br>group by 1<br><br>Query id: 0e53775a-ee47-4174-91af-7b26d9176620<br><br>┌─radio─┬─count()─┐<br>│ GSM   │     457 │<br>│ LTE   │     284 │<br>│ UMTS  │    1165 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.870 sec. <br>Processed 43.28 million rows, 692.51 MB (49.77 million rows/s., 796.38 MB/s.)</pre><p>Now the same location, but using geohashes and the table <em>geo.cell_towers_geo_hash:</em></p><pre>select radio, count()<br>from geo.cell_towers_geo_hash<br>where pointInPolygon((lon, lat),<br>    [(30.31283445840512, 59.93731653223952),<br>    (30.312501800021835, 59.93659901654749),<br>    (30.359919789647535, 59.930538163165636),<br>    (30.384620591702515, 59.92268781981602),<br>    (30.38561525487222, 59.924017109105904),<br>    (30.360950045625515, 59.93158921046111),<br>    (30.31283445840512, 59.93731653223952)])<br>    and (<br>        startsWith(geo_hash_9, &#39;udtsgkt&#39;) or startsWith(geo_hash_9, &#39;udtsgks&#39;) or<br>        startsWith(geo_hash_9, &#39;udtsgkv&#39;) or startsWith(geo_hash_9, &#39;udtsgkg&#39;) or<br>        startsWith(geo_hash_9, &#39;udtsgku&#39;) or startsWith(geo_hash_9, &#39;udtsgkf&#39;) or<br>        startsWith(geo_hash_9, &#39;udtsgkc&#39;) or startsWith(geo_hash_9, &#39;udtsgkb&#39;) or<br>…<br>  group by 1<br><br>Query id: c4f940fe-f241-46c0-a2af-da51b41a80d0<br><br>┌─radio─┬─count()─┐<br>│ GSM   │     457 │<br>│ LTE   │     284 │<br>│ UMTS  │    1165 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.476 sec. <br>Processed 49.15 thousand rows, 1.56 MB (103.32 thousand rows/s., 3.29 MB/s.)</pre><p>And finally, filtering by the polygon and by the H3 index from the data located in the table <em>geo.cell_towers_h3_10:</em></p><pre>select radio, count()<br>from geo.cell_towers_h3_10<br>where pointInPolygon((lon, lat), <br>    [(30.31283445840512, 59.93731653223952), <br>    (30.312501800021835, 59.93659901654749), <br>    (30.359919789647535, 59.930538163165636),  <br>    (30.384620591702515, 59.92268781981602), <br>    (30.38561525487222, 59.924017109105904), <br>    (30.360950045625515, 59.93158921046111), <br>    (30.31283445840512, 59.93731653223952)])<br>    and h3_10 in (<br>                  621796229703827455,<br>                  621796229703958527,<br>                  621796229703073791,<br>                  621796229703991295,<br>                  621796229703368703,<br>                  621796229703204863,<br>  …<br>)<br>group by 1<br><br>Query id: f30a858d-f260-4a65-a46e-e7a899f5c7ed<br><br>┌─radio─┬─count()─┐<br>│ GSM   │     457 │<br>│ LTE   │     284 │<br>│ UMTS  │    1165 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.561 sec. <br>Processed 40.96 thousand rows, 950.27 KB (72.98 thousand rows/s., 1.69 MB/s.)</pre><p>Now let’s prepare benchmarks for other locations</p><h4>Getting results for a medium scale (Istanbul, Turkey)</h4><p>Executing the query with filtering by the polygon only on the table <em>geo.cell_towers_polygon</em>:</p><pre>select radio, count() <br>from geo.cell_towers_polygon<br>where pointInPolygon((lon, lat), <br>    [(29.06883411162758, 41.191306335710976),<br>    (28.794830240269164, 41.12493965554057),<br>    (28.593044443531397, 41.014446342512),<br>    (28.57286586385851, 40.92062181325289),<br>    (29.31310112872967, 40.78727440681351),<br>    (29.34708610502321, 40.827467532876454),<br>    (29.42142824066238, 40.910188631382056),<br>    (29.439482759317826, 40.99681430788482),<br>    (29.145300308285357, 41.1593307823311),<br>    (29.06883411162758, 41.191306335710976)])<br>group by 1<br><br>Query id: 645ab109-d9a1-4c06-a180-cdacd20ae388<br><br>┌─radio─┬─count()─┐<br>│ GSM   │   21706 │<br>│ LTE   │   27956 │<br>│ UMTS  │  161787 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.841 sec. <br>Processed 43.28 million rows, 692.79 MB (51.45 million rows/s., 823.57 MB/s.)</pre><p>The same location, but using geohashes and the table <em>geo.cell_towers_geo_hash:</em></p><pre>select radio, count() <br>from geo.cell_towers_geo_hash<br>where pointInPolygon((lon, lat), <br>    [(29.06883411162758, 41.191306335710976),<br>    (28.794830240269164, 41.12493965554057),<br>    (28.593044443531397, 41.014446342512),<br>    (28.57286586385851, 40.92062181325289),<br>    (29.31310112872967, 40.78727440681351),<br>    (29.34708610502321, 40.827467532876454),<br>    (29.42142824066238, 40.910188631382056),<br>    (29.439482759317826, 40.99681430788482),<br>    (29.145300308285357, 41.1593307823311),<br>    (29.06883411162758, 41.191306335710976)])<br>    and (<br>        startsWith(geo_hash_9, &#39;sxkb&#39;) or <br>        startsWith(geo_hash_9, &#39;sxk8&#39;) or<br>        startsWith(geo_hash_9, &#39;sxk2&#39;) or<br>        startsWith(geo_hash_9, &#39;sxk9&#39;) or<br>        startsWith(geo_hash_9, &#39;sxkc&#39;) or<br>        startsWith(geo_hash_9, &#39;sxk3&#39;) or<br>        startsWith(geo_hash_9, &#39;sxkd&#39;)<br>    )<br>group by 1<br><br>Query id: 6dc4e689-548b-4fb5-9086-76eb9ba17365<br><br>┌─radio─┬─count()─┐<br>│ GSM   │   21706 │<br>│ LTE   │   27956 │<br>│ UMTS  │  161787 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.439 sec. <br>Processed 262.14 thousand rows, 8.86 MB (596.94 thousand rows/s., 20.18 MB/s.)</pre><p>Filtering by the polygon and by the H3 index from the data located in the table <em>geo.cell_towers_h3_4:</em></p><pre>select radio, count()<br>from geo.cell_towers_h3_4<br>where pointInPolygon((lon, lat), <br>    [(29.06883411162758, 41.191306335710976),<br>    (28.794830240269164, 41.12493965554057),<br>    (28.593044443531397, 41.014446342512),<br>    (28.57286586385851, 40.92062181325289),<br>    (29.31310112872967, 40.78727440681351),<br>    (29.34708610502321, 40.827467532876454),<br>    (29.42142824066238, 40.910188631382056),<br>    (29.439482759317826, 40.99681430788482),<br>    (29.145300308285357, 41.1593307823311),<br>    (29.06883411162758, 41.191306335710976)])<br>    and h3_4 in (<br>        595016746188931071,<br>        595016737598996479,<br>        595016754778865663,<br>        595016789138604031,<br>        595016780548669439,<br>        595016771958734847)<br>group by 1<br><br>Query id: 8c611b22-e7bb-4c59-9022-b429f9ab3c77<br><br>┌─radio─┬─count()─┐<br>│ GSM   │   21706 │<br>│ LTE   │   27956 │<br>│ UMTS  │  161787 │<br>└───────┴─────────┘<br><br>3 rows in set. Elapsed: 0.470 sec. <br>Processed 294.91 thousand rows, 7.24 MB (626.82 thousand rows/s., 15.40 MB/s.)</pre><h4>Getting results for a large scale (Spain)</h4><p>Executing the query with filtering by the polygon only on the table <em>geo.cell_towers_polygon</em>:</p><pre>select radio, count()<br>from geo.cell_towers_polygon<br>where pointInPolygon((lon, lat), <br>    [(-9.283159105703874, 44.223199067959), <br>    (-10.52662776366833, 36.22870543241301), <br>    (-2.3285489819528493, 35.83063297648961),  <br>    (3.730790053677566, 42.33834347424349), <br>    (-2.1493839318864048, 43.800646326588236), <br>    (-9.283159105703874, 44.223199067959)])<br>group by 1<br><br>Query id: 21d5f933-a963-47d0-b034-82f64308352f<br><br>┌─radio─┬─count()─┐<br>│ GSM   │  236917 │<br>│ LTE   │  173629 │<br>│ NR    │      39 │<br>│ UMTS  │  648953 │<br>└───────┴─────────┘<br><br>4 rows in set. Elapsed: 0.870 sec. <br>Processed 43.28 million rows, 696.81 MB (49.76 million rows/s., 801.22 MB/s.)</pre><p>The same location, but using geohashes and the table <em>geo.cell_towers_geo_hash:</em></p><pre>select radio, count()<br>from geo.cell_towers_geo_hash<br>where pointInPolygon((lon, lat), <br>    [(-9.283159105703874, 44.223199067959), <br>    (-10.52662776366833, 36.22870543241301), <br>    (-2.3285489819528493, 35.83063297648961),  <br>    (3.730790053677566, 42.33834347424349), <br>    (-2.1493839318864048, 43.800646326588236), <br>    (-9.283159105703874, 44.223199067959)])<br>    and (<br>        startsWith(geo_hash_9, &#39;eyk&#39;) or startsWith(geo_hash_9, &#39;ey7&#39;) or<br>        startsWith(geo_hash_9, &#39;eye&#39;) or startsWith(geo_hash_9, &#39;eys&#39;) or<br>        startsWith(geo_hash_9, &#39;eyt&#39;) or startsWith(geo_hash_9, &#39;eyw&#39;) or<br>  …<br>)<br>group by 1<br><br>Query id: 12884d65-2192-4624-92ba-ff1480543b11<br><br>┌─radio─┬─count()─┐<br>│ GSM   │  236917 │<br>│ LTE   │  173629 │<br>│ NR    │      39 │<br>│ UMTS  │  648953 │<br>└───────┴─────────┘<br><br>4 rows in set. Elapsed: 0.516 sec. <br>Processed 1.35 million rows, 45.41 MB (2.62 million rows/s., 88.06 MB/s.)</pre><p>Filtering by the polygon and by the H3 index from the data located in the table <em>geo.cell_towers_h3_3:</em></p><pre>select radio, count() <br>from geo.cell_towers_h3_3<br>where pointInPolygon((lon, lat), <br>    [(-9.283159105703874, 44.223199067959), <br>    (-10.52662776366833, 36.22870543241301), <br>    (-2.3285489819528493, 35.83063297648961),  <br>    (3.730790053677566, 42.33834347424349), <br>    (-2.1493839318864048, 43.800646326588236), <br>    (-9.283159105703874, 44.223199067959)])<br>    and h3_3 in (<br>        590976161215938559,<br>        590976436093845503,<br>        590976023776985087, <br>…<br>)<br>group by 1<br><br>Query id: 2d87ca3e-e109-4df9-920b-ff509c9fd37f<br><br>┌─radio─┬─count()─┐<br>│ GSM   │  236917 │<br>│ LTE   │  173629 │<br>│ NR    │      39 │<br>│ UMTS  │  648953 │<br>└───────┴─────────┘<br><br>4 rows in set. Elapsed: 0.445 sec. <br>Processed 1.29 million rows, 32.21 MB (2.91 million rows/s., 72.33 MB/s.)</pre><h3>Comparison of the received results</h3><p>Based on the data taken from the system table <em>system.query_log</em>, we will build charts for the main query metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-kXVoQklHi3N3gjFpbn1Lw.jpeg" /><figcaption>Query execution time for each location using different types of filtering</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nsZCktsRj11ScyM0qbGubg.jpeg" /><figcaption>Number of rows read during query execution</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOM8Mvwem4OT9mCQuQ7hHQ.jpeg" /><figcaption>The volume of bytes read during query execution</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oLf48rjn9naxaHLPHoszUQ.jpeg" /><figcaption>Memory utilization during query execution</figcaption></figure><h3>Conclusion</h3><p>Based on the research and collected metrics, we can draw the following conclusions:</p><h4>Basic parameters</h4><p><strong>Query Performance</strong></p><ul><li>Filtering solely <strong>by polygon</strong> criteria results in longer execution times compared to other geofiltering methods, as shown in the <em>query_duration_ms</em> chart. This is likely attributed to polygonal queries being inherently more complex due to the intricacies of geometric shape processing. Moreover, ClickHouse does not take into account the data&#39;s sorting by <strong>longitude</strong> and <strong>latitude</strong>, leading to a <strong>full scan</strong> of all the data.</li><li>Using <strong>H3 indexes</strong> for filtering proves to be the <strong>most expedient</strong> method. It’s optimized for the rapid execution of geospatial queries, making it a preferable choice for real-world applications where response time is paramount. However, it’s essential to remember that we created separate tables for each H3 index. This introduces the <strong>overhead of maintaining</strong> additional entities and requires additional disk space.</li></ul><p><strong>Data Volume Processed</strong></p><ul><li>Polygon filtering consistently processes the highest amount of data, as indicated by the <em>read_rows</em> and <em>read_bytes</em> metrics. This likely contributes to its <strong>slower performance</strong>.</li><li>While the <strong>H3 method</strong> is efficient, it processes data volumes roughly equivalent to those of <strong>geohash.</strong></li></ul><p><strong>Resource Utilization</strong></p><ul><li>In certain tested locations, such as <strong>Spain</strong>, the <strong>geohash</strong> method consumes the <strong>most memory</strong>, as indicated by the <em>memory_usage</em> metric. This could suggest the need for optimization, especially when dealing with larger datasets or complex geospatial queries.</li><li>While the H3 method is notably fast, it’s also memory-intensive.</li><li>In contrast, even though filtering <strong>by polygon</strong> involves <strong>full table scans</strong>, it is <strong>more resource-efficient</strong> in terms of memory consumption.</li></ul><h4><strong>Additional Considerations</strong></h4><ul><li><strong>Cost and Maintenance. </strong>When choosing a geofiltering method, consider not only its performance but also the associated maintenance costs and system scalability. Intense memory or disk space usage can introduce added financial and technical burdens.</li><li><strong>Flexibility.</strong> Geohash provides flexibility by allowing operations at various levels of precision without the need to create new tables. This can simplify scaling efforts and ensure adaptability for diverse tasks.</li><li><strong>Data Security.</strong> Always prioritize data security when handling geodata. Ensure that the method you choose complies with all necessary security and confidentiality requirements.</li></ul><p>In conclusion, the optimal geofiltering approach is determined by your application’s specific demands, available resources, and performance goals. <strong>H3 indexes</strong> excel for speed-driven applications, <strong>polygons</strong> cater to detailed geometric filtering, while <strong>geohash</strong> provides a balanced alternative, notable for its parent index searches without additional table requirements. Given the intricacies of geofiltering, method selection must be tailored to the application’s precise needs and resource constraints.</p><p>I hope the insights I’ve shared are valuable to you. Wishing you every success in your research pursuits! See you on our next journey through the world of data!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f55ff100fbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Inner Workings of Python Set and Collision Handling]]></title>
            <link>https://python.plainenglish.io/the-inner-workings-of-python-set-and-collision-handling-65577227c462?source=rss-b79c227003cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/65577227c462</guid>
            <category><![CDATA[collision]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[hash]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Igor Gorbenko]]></dc:creator>
            <pubDate>Sat, 22 Jul 2023 07:02:27 GMT</pubDate>
            <atom:updated>2024-04-09T00:40:46.920Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zDA6py9TbKHBJlVj" /><figcaption>Photo by <a href="https://unsplash.com/@rubaitulazad?utm_source=medium&amp;utm_medium=referral">Rubaitul Azad</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Python is a powerful and flexible programming language that offers a variety of structures for data processing and organization. One of these data structures is the set. Sets in Python provide fast element lookup, thanks to a hash table. But what happens when <strong>different values</strong> have <strong>the same hash</strong>? Let’s find out together!</p><h3>Internal Structure of Python Set</h3><p>As mentioned earlier, sets in Python are hash tables. In the official Python implementation, a mechanism for handling collisions in hash tables uses open addressing with arbitrary probing.</p><p>The process is as follows: when a collision occurs (that is, two different keys have the same hash value), Python doesn’t just move to the next available slot in the hash table but applies arbitrary probing to determine the next slot for checking.</p><p>The method determines the next slot in the hash table, which will be checked when looking for a place for a new element if a collision occurs. Instead of just going to the next slot in the table, Python calculates a new index based on the original hash value of the element. This is done to minimize the likelihood of further collisions (the <strong>set_lookkey</strong> function in the file <a href="https://github.com/python/cpython/blob/main/Objects/setobject.c">Objects/setobject.c</a>:</p><pre>i = (i * 5 + 1 + perturb) &amp; mask;</pre><p>Here, i is the current index, perturb is the original hash value, shifted right on each iteration (<em>perturb &gt;&gt;= PERTURB_SHIFT</em>), and <em>mask</em> is a mask used to limit the index within the size of the table.</p><p>This process is repeated until a free slot or key is found, allowing for effective collision resolution.</p><h4>Creating Collisions in Python Set</h4><blockquote>A collision in the context of hash tables and sets occurs when two different values have the same hash</blockquote><p>Although the <a href="https://docs.python.org/3/library/functions.html#hash">hashing function</a> in Python is designed to minimize the likelihood of collisions, we can manually create such a situation. To do this, let’s define our own class of objects and override its <em>hash</em> and <em>eq</em> methods to see how they behave in the event of collisions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c1685964f9329e7e39c3d5cb03b4f64/href">https://medium.com/media/5c1685964f9329e7e39c3d5cb03b4f64/href</a></iframe><p>Now, let’s create two instances of this class (<em>coll_obj_1</em> and <em>coll_obj_2</em>) with the value of <em>“1”</em> and <em>“2”</em>, respectively, and put them in a set <em>coll_set</em></p><pre>coll_obj_1 = HashCollisionObject(1)<br>coll_obj_2 = HashCollisionObject(2)<br><br>print(&quot;Adding elements to the set:&quot;)<br>coll_set = set()<br>coll_set.add(coll_obj_1)<br>coll_set.add(coll_obj_2)<br><br>print(&quot;\nChecking for element presence in the set:&quot;)<br>print(&quot;coll_obj_1 exists in the coll_set&quot; if coll_obj_1 in coll_set else &quot;coll_obj1 does not exist in the coll_set&quot;)<br>print(&quot;coll_obj_2 exists in the coll_set&quot; if coll_obj_2 in coll_set else &quot;coll_obj2 does not exist in the coll_set&quot;)</pre><p>As a result, we get a set consisting of two objects: <strong>despite identical hash values, both objects are present in it.</strong></p><blockquote>This is possible because Python set, in addition to comparing hashes, also conducts object comparison by value</blockquote><pre>Adding elements to the set:<br>Calling __hash__ for 1<br>Calling __hash__ for 2<br>Calling __eq__ for 1 and 2, result: False<br><br>Checking for element presence in the set:<br>Calling __hash__ for 1<br>coll_obj_1 exists in the coll_set<br>Calling __hash__ for 2<br>Calling __eq__ for 1 and 2, result: False<br>coll_obj_2 exists in the coll_set</pre><h3>Benchmarks</h3><p>Now, let’s look at the performance of element inserting and searching in a set when hash collisions occur. For this, let’s slightly adjust the code of <em>HashCollisionObject</em>: we’ll make the probability of collision occurrence in this object about 10% and create another class in which there won’t be an implementation of an artificial collision. The built-in <em>hash() </em>function ensures a fairly low probability of collision occurrence:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f87e59a0efae47a4010bb3c75e041bb/href">https://medium.com/media/6f87e59a0efae47a4010bb3c75e041bb/href</a></iframe><p>Now let’s create sets of various sizes of two types (with collisions and without) and calculate the average time to insert and find an element in each of these types of sets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TZCj2RXxPyCkyZKpYYAog.png" /><figcaption>Average Insert Time</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A7rm6UdRYlBSv-Npfg05iw.png" /><figcaption>Average Search Time</figcaption></figure><p>As illustrated in the above graphs, the presence of even ~10% hash collision in the set results in a linear degradation of the performance of this data structure. We observe a decline in performance both in terms of inserting new elements and searching for an element within the set</p><h3>Conclusion</h3><p>Python has strong and flexible data structures that help handle and process data. Understanding the inner workings of these data structures can help to use them effectively and correctly in your programs.</p><p>Python sets represent a data structure intended for efficient handling of unique elements. However, our exploration has shown that hash collisions, even though rare with Python’s native hash function, can impact this efficiency, leading to a linear degradation in performance for operations like element insertion and searching.</p><p>Creating such collisions artificially has given us a chance to better understand the mechanisms at play within Python’s set implementation, and underscored the importance of effective hash functions for the performance of our data structures.</p><p>I sincerely thank you for taking the time to read this article and I hope you found the insights valuable and applicable to your Python programming journey. Please feel free to share your thoughts and comments as your feedback is greatly appreciated.</p><p>Happy coding!</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a><strong> | </strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a><strong> | </strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a><strong> | </strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a><strong> | </strong><a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://stackademic.com/"><strong>Stackademic</strong></a><strong> | </strong><a href="https://cofeed.app/"><strong>CoFeed</strong></a><strong> | </strong><a href="https://venturemagazine.net/"><strong>Venture</strong></a><strong> | </strong><a href="https://blog.cubed.run/"><strong>Cubed</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65577227c462" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/the-inner-workings-of-python-set-and-collision-handling-65577227c462">The Inner Workings of Python Set and Collision Handling</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>