<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mauro Krikorian on Medium]]></title>
        <description><![CDATA[Stories by Mauro Krikorian on Medium]]></description>
        <link>https://medium.com/@maurok?source=rss-f4f95113ee79------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JPP9uiTMR4KYTpFP1o3buA.jpeg</url>
            <title>Stories by Mauro Krikorian on Medium</title>
            <link>https://medium.com/@maurok?source=rss-f4f95113ee79------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 22:36:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maurok/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Coding Assistants to a Governed Delivery System]]></title>
            <link>https://medium.com/@maurok/from-coding-assistants-to-a-governed-delivery-system-4af4046cf252?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/4af4046cf252</guid>
            <category><![CDATA[engineering-excellence]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-software-factory]]></category>
            <category><![CDATA[sdlc-innovation]]></category>
            <category><![CDATA[software-development-ai]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 13:57:44 GMT</pubDate>
            <atom:updated>2026-11-15T16:10:00.028Z</atom:updated>
            <content:encoded><![CDATA[<p>What we learned building an AI Software Factory — and what the measurements actually show</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GrGkSXuuFCArrxbpKVN4og.png" /><figcaption>SOUTHWORKS AI Software Factory in a Nutshell</figcaption></figure><p>I wrote this white paper, not as a market commentary or a prediction — but as a first-hand account of an ongoing effort my team and I started back in 2025, when we set out to answer a question our customers kept raising in different words: AI was making individual developers faster, so why was delivery not getting faster with them? Every architectural decision, failure, and open gap in it comes from our own delivery work across real client engagements. More than a half-year later, I think we have an answer worth publishing — along with the architecture, the tradeoffs, and the numbers behind it.</p><h3>The first wave solved a narrow problem</h3><p>Coding assistants help people produce code faster. Enterprise delivery is larger than that. Requirements have to be interpreted, repositories have to evolve together, approvals have to be captured, failures have to be recovered from — and every step has to be visible and auditable. Every tool we tried could write code. None could hold state across a wait, resume after a failure, or carry a lesson into the next work item. Those turned out to be the delivery lifecycle, not the periphery.</p><p>One principle organizes everything we built: <strong>AI agents should reason, and workflows should coordinate</strong>. Workflows own state, approvals, retries, and recovery. Agents handle planning, implementation, and problem solving inside controlled execution environments. That separation is what made the system reliable. Human involvement is what made it adoptable.</p><h3>The shift is economic as much as technical</h3><p>A development center sells hours, and the buyer carries the risk that those hours produce less than expected. A delivery line inverts that: the unit being bought is a reviewed, merged change, and its cost is a measurement rather than a forecast. On standard, recurring work — roughly two-thirds of a typical backlog — that measurement is an order of magnitude in our favor on speed, and considerably more than that on cost.</p><p>I resisted the temptation to publish a single headline multiplier. The paper gives a full chapter to it instead: what each ratio is, which baseline it is measured against, the population it was computed over, why speed and cost figures should never be multiplied together, and why the honest whole-backlog number is smaller than the per-feature one. It also covers what happens to review capacity and throughput over the first quarters — and names the one dimension we have not measured yet.</p><h3>The parts most people underestimate</h3><p><strong>Governance is what unlocked adoption, not what slowed it down.</strong> Full autonomy was never the goal. Approval is a workflow state rather than a side conversation: a plan can be approved, sent back, or escalated, and a reviewer’s comment becomes a remediation input rather than a note someone has to act on by hand. Automated quality gates run before any person is involved, which is what keeps reviewer attention on judgment instead of on defects a machine should have caught.</p><p><strong>Failure is a route, not a full stop.</strong> A failed check or a requested change is classified and routed — implementation-level fixes back to implementation, design-level problems back to planning — which is what makes imperfect first-pass output still worth having.</p><p><strong>Learning is the part that compounds.</strong> The feedback humans give at the gates becomes reusable, reviewed guidance in a version-controlled knowledge base. A team improves as its individuals learn, so that improvement lasts as long as their tenure. A delivery line improves by writing down what it learned, so the guidance outlives the project, the engineers, and the review thread it was said in.</p><p><strong>Where it runs is part of the design.</strong> The control plane, the execution sandboxes, the datastores, the telemetry, and the knowledge base all sit inside the client’s own tenancy. Nothing is routed to a vendor-hosted service — which also means none of the numbers in the paper depend on us to be believed. They are read from audit tables in the client’s environment, and the organizations that produced them can re-derive any of them without our involvement.</p><h3>We also wrote down what we cannot yet prove</h3><p>Planning still varies across runs on identical work, and closing that gap is our first priority. We can route any stage to any approved model, but we cannot argue with complete certainty which model belongs where — providers, versions, and models keep changing underneath a running system, so the right choice is not a fixed answer but one that depends on the scenario, the context, the enterprise, and the cost budget it has to fit. And everything we measured is speed, cost, and volume — a defect-rate comparison against a conventional baseline is the measure that would settle the quality question outright, and it is the next thing we intend to put beyond dispute. The paper states all of that plainly, along with the risks we manage, because a model that only describes what works is of little use to anyone deciding whether to adopt it.</p><h3>What is in the full paper</h3><ul><li>The three grades of AI adoption we passed through, and the ceiling each one hit.</li><li>Six architecture phases, including the two decisions that shaped everything downstream — and the ones we had to reverse.</li><li>The anatomy of a delivery line, its eleven stages, and where it deliberately stops.</li><li>What it actually takes to stand one up: eight setup activities, and the two most often compressed.</li><li>The full measurement chapter — definitions, baselines, populations, and what the ratios do not cover.</li><li>What happened when we deliberately killed a running process, fifteen lessons learned, and a seven-step guide to where a team should start on Monday.</li></ul><p>If there is one line to take away, it is this: AI-enabled software engineering is an orchestration discipline before it is a generation capability. The gains we can measure came from the orchestration, governance, and learning built around the agents — not from the agents alone.</p><p><a href="http://swrks.co/AI-Software-Factory-wp"><strong>Download the full white paper </strong></a><strong>— <em>Building an AI Software Factory: From Coding Assistants to Governed Delivery Systems.</em></strong></p><p><strong>Wanna see it running? Click </strong><a href="https://www.youtube.com/playlist?list=PLcoHLzadi9Fg"><strong>here</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af4046cf252" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The $9 Trillion Problem Hiding in Your Team’s Chat History]]></title>
            <link>https://medium.com/@maurok/the-9-trillion-problem-hiding-in-your-teams-chat-history-9cbdba37aac7?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cbdba37aac7</guid>
            <category><![CDATA[microsoft-teams]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[knowledge-management]]></category>
            <category><![CDATA[azure-ai]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 08:33:25 GMT</pubDate>
            <atom:updated>2026-07-24T08:34:49.251Z</atom:updated>
            <content:encoded><![CDATA[<p>Every day, critical delivery knowledge is created across your team’s collaboration and chat tools — and <strong>quietly lost</strong>. This is how a source-backed AI assistant turns those everyday conversations into <strong>living documentation</strong>: answers in seconds, decisions preserved, and knowledge that <strong>never goes stale</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6zvFEnzbrtHvNuj1" /><figcaption>Photo by <a href="https://unsplash.com/@vixxer?utm_source=medium&amp;utm_medium=referral">ün LIU</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>EXECUTIVE DEBRIEF</strong></p><p><strong>The problem:</strong> Undocumented knowledge costs organizations an estimated <strong>$9 trillion a year</strong>, drains up to <strong>25% of revenue</strong>, and has teams spending <strong>one day out of every five</strong> just searching for answers that already exist. <em>(Deloitte 2026, Bloomfire 2025, Atlassian 2025)</em></p><p><strong>The move:</strong> Not another chatbot, but a source-backed AI assistant that lives inside Microsoft Teams — answering in seconds, escalating to a human when evidence is thin, and re-indexing every answer so knowledge compounds instead of disappearing. This is AI that <strong>really works</strong> in daily operations. <strong>Here’s how.</strong></p><p>The real productivity breakthrough from AI does not come from adding another assistant to the enterprise stack. It comes from helping teams preserve the knowledge they already create every day but rarely have time to document properly: architecture decisions, troubleshooting paths, design trade-offs, operational agreements, and implementation context shared across conversations.</p><p>This is the story of a Microsoft Teams integration designed around that problem: turning daily project conversations into a source-backed, reusable knowledge base. The goal was not to ask teams to stop delivery work and write perfect documentation. It was to make the knowledge produced during delivery easier to capture, retrieve, validate, and reuse.</p><p><strong>When AI captures knowledge as fast as teams create it, documentation stops being a task, and starts being a by-product of getting work done.</strong></p><h3>The Business Problem: Documentation Cannot Keep Up with Delivery</h3><p>Every project creates a growing body of knowledge. Some of it is captured in documents, but much of it first appears in meetings, chat threads, troubleshooting discussions, and quick answers shared between team members. The issue is rarely that the knowledge does not exist. More often, it exists inside conversations before anyone has time to turn it into architecture notes, troubleshooting guidance, or formal design documentation.</p><p>For leaders, this is more than a documentation gap. It becomes a delivery risk. When the reasoning behind an architecture choice, a service dependency, or a troubleshooting resolution stays buried in a conversation, the team pays for it later through repeated questions, inconsistent answers, slower onboarding, and duplicated investigation.</p><p><strong>THE COST OF KNOWLEDGE THAT STAYS IN CONVERSATIONS</strong></p><p>Knowledge that is never captured or reused represents an estimated <strong>$9 trillion</strong> in lost value each year — a “knowledge exodus” as expertise and hard-won context leave organizations undocumented. <em>(</em><a href="https://www.deloitte.com/us/en/insights/topics/talent/knowledge-management-plan.html"><em>Deloitte Insights, The $9 Trillion Knowledge Exodus, 2026</em></a><em>)</em></p><p>Knowledge inefficiency — duplicated effort, repeated mistakes, and slow retrieval — can cost organizations as much as <strong>25% of annual revenue</strong> when past decisions and lessons learned are hard to access. <em>(</em><a href="https://bloomfire.com/resources/value-report"><em>Bloomfire, The Value of Enterprise Intelligence, 2025</em></a><em>)</em></p><p>Executives and teams alike spend about <strong>a quarter of the workweek — roughly one day out of every five — </strong>searching for information that already exists inside their organization. <em>(</em><a href="https://www.atlassian.com/blog/state-of-teams-2025"><em>Atlassian, State of Teams, 2025)</em></a></p><p>The challenge was not to create another documentation process. The challenge was to make everyday collaboration produce reusable project memory without adding friction to the team’s workflow.</p><h3>A Delivery Scenario: Turning AKS Conversations into Living Documentation</h3><p>Picture a production Azure Kubernetes Service environment with several services under active development. Every day, the team makes decisions that keep the platform healthy: DevOps configuration, node pool allocation, add-on setup, service-specific customization, and workload dependencies. Each carries real risk — a wrong node pool or scaling choice degrades performance, a missed add-on or dependency breaks service accessibility, and an overlooked control can open a compliance gap. These are high-stakes calls, and they must be made fast and accurately.</p><p>The reasoning behind these decisions usually starts as a conversation — why a service needs a dedicated node pool, what add-on configuration was agreed, which dependency blocked a deployment, what monitoring threshold matters, or what constraint must be respected before scaling. Left in scattered threads and notes, that context goes stale, and the next person risks repeating a mistake the team already solved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*94W67neFBHgy6PcGx5qE0w.png" /><figcaption><em>Figure 1. A production-critical Friday-afternoon scale-up question answered in seconds — full context in hand, right people to confirm with, and the judgement call still with the engineer.</em></figcaption></figure><p>With the assistant inside Microsoft Teams, that reasoning becomes reusable memory. Anyone can ask what was agreed about a node pool, dependency, add-on, or scaling constraint and get a source-backed answer in the thread — or a human escalation when the evidence is thin. Because every validated answer is re-indexed, the knowledge stays accurate and current instead of degrading over time, so the next production decision is faster and safer.</p><p><strong>Before:</strong> An engineer prepares to scale a payment service under load. The reasoning behind its dedicated node pool — a compliance isolation requirement — lives in an old Teams thread no one remembers. Confirming the constraint, the ceiling, and who owns the dependency takes half a day of pinging colleagues, and quietly resurfaces the next time someone joins.</p><p><strong>After:</strong> They ask in the channel and get a source-backed answer instantly — the isolation requirement, the practical scaling ceiling, and the blocking cache dependency — with the original author cited and the right people already tagged. Re-indexed on the way through, so the next person gets it in seconds.</p><p>For the team, this creates a better feature-shipping rhythm: fewer blocked conversations, faster alignment on environment decisions, and less time spent rediscovering previous agreements. The assistant does not replace documentation or engineering judgment. It helps turn day-to-day collaboration into living documentation that supports delivery while the project is still moving.</p><h3>The Strategic Choice: Put AI Inside the Flow of Work</h3><p>The most important design choice was not the model. It was the placement. The assistant needed to live directly inside Microsoft Teams, where questions, follow-ups, agreements, and decisions already happen.</p><p>That matters because adoption friction is one of the biggest reasons productivity tools fail to deliver their promise. If the team has to leave the conversation, open another system, and search manually, the value is reduced before the answer is found. By keeping the interaction inside Teams, the assistant turns the existing collaboration channel into the front door for project knowledge.</p><p>Questions are asked where the work is happening. AI-generated and human-generated answers return to the same thread. The answer remains visible to the team, so the next person does not have to restart the same investigation.</p><p><em>That single decision is where much of the productivity gain comes from: the assistant meets people where they already are, instead of adding one more tool to check.</em></p><h3>From Conversations to Reusable Project Memory</h3><p>The Teams experience is supported by a retrieval pipeline that turns existing project material into usable delivery context. Documents are uploaded to Azure Blob Storage, indexed through Azure AI Search, and broken into smaller chunks so the assistant can retrieve the most relevant passages instead of forcing the team to search through entire files or long conversation histories.</p><p>This is what makes the solution useful for delivery work. The assistant is not relying on unsupported memory. It retrieves approved context, detects similar or duplicate content through embeddings, and uses vector search to find the right information even when people ask the same question in different words.</p><p>For the team, the flow is simple: chat (as you usually do) in Microsoft Teams, receive a source-backed answer in the same thread, and keep the resolution visible for future use. Over time, validated answers are added back into the index, so project memory improves as work continues.</p><p><strong>1 · Ask / Need</strong></p><p>A question or need is identified in Microsoft Teams</p><p><strong>2 · Classify &amp; Retrieve</strong></p><p>Foundry classifies intent; Azure AI Search retrieves matching chunks</p><p><strong>3 · Answer or Escalate</strong></p><p>A source-backed answer is posted, or a human is escalated when evidence is thin</p><p><strong>4 · Re-index</strong></p><p>Every validated answer is re-indexed for the next person</p><p>The diagram below traces the full path of a conversation — from Teams, through Foundry orchestration and Azure AI Search retrieval, to a source-backed answer (or human escalation) posted back in the same thread. Here’s what happens under the hood between the moment a question is identified in Teams and the moment a source-backed answer lands back in the same thread.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eU7gJpRXttWyT_JRHmuEQA.png" /><figcaption><em>Figure 2. End-to-end flow — from a question typed in Microsoft Teams, through Foundry orchestration and Azure AI Search retrieval, to a source-backed answer (or human escalation) posted back in the same thread and re-indexed for the next person.</em></figcaption></figure><h3>The Operating Model: Proactive Answers, Controlled by Evidence</h3><p>For executives, the operating model matters for two reasons: it must scale economically as usage grows, and it must remain traceable enough to trust. Azure AI Foundry provides the orchestration layer that makes this possible, coordinating classification, retrieval, and response generation so the team sees a single assistant rather than a set of moving parts.</p><p>The supporting technology choices reinforce those two priorities. Rather than defaulting to the largest available model, the team deliberately selected a lighter, cost-efficient model for high-volume classification and response generation — trading a marginal amount of raw capability for unit economics that hold up as usage scales across many conversations. Azure AI Search provides source-backed retrieval, and Application Insights captures how each answer was produced. The result is an assistant that is affordable to run frequently and transparent enough to rely on.</p><h3>Governance by Design: Capture Knowledge Without Guessing</h3><p>A documentation-helper integration should know when not to write the record on its own. When the assistant does not have enough relevant context, it does not guess. It escalates the question to a human. Users can also explicitly request human assistance, and the system detects that intent.</p><p>When it escalates, the question is handed to a human — a support engineer or subject-matter expert — who answers directly in the Teams thread, and that human response becomes the authoritative record for the issue. The alerting and follow-up that ensure no escalation is missed are handled by the monitoring loop described next.</p><p>This human-in-the-loop model is not a fallback weakness; it is the governance mechanism that protects trust. AI helps capture and reuse knowledge when the information exists, while human expertise remains involved when judgment, confirmation, or missing context is required.</p><h3>Monitoring the Knowledge Loop Until Every Question Is Resolved</h3><p>If governance decides <em>when</em> a human should step in, monitoring makes sure that hand-off is never dropped. The integration includes an automated monitoring process that keeps unanswered conversations visible until they are resolved. Every 30 minutes, it reviews open questions that have not yet received either an AI-generated or human-generated response, so a missed question becomes a tracked one rather than a lost one.</p><p>If intervention is required, notifications are sent to a dedicated Slack channel. To avoid unnecessary noise, the monitoring service uses exponential backoff, sending additional alerts after progressively longer intervals until a valid answer is detected, the AI later answers with sufficient confidence, or the maximum number of retry attempts is reached.</p><p>Once a valid response is found, the incident closes automatically and future alerts for that conversation are suppressed. The result is a practical safety net: open questions remain visible, subject-matter experts are brought in when needed, and resolved answers can continue strengthening the shared knowledge base.</p><h3>The Productivity Outcome: Living Documentation, Faster Delivery</h3><p>The upside is the mirror image of the problem: if knowledge workers lose roughly <strong>1 day per week</strong> hunting for information that already exists, then reclaiming even a fraction of that time — by answering in seconds instead of hours — turns directly into <strong>capacity for delivery</strong>. Every question resolved once and reused many times compounds that gain across the team.</p><p>As a live AI integration, this plays out in practice inside the Azure Kubernetes Service scenario described earlier: questions that once took hours of searching — why a service runs on a dedicated node pool, which dependency blocked a deployment, or what scaling constraint must be respected — are answered directly in the Microsoft Teams thread in seconds, and every validated resolution is re-indexed so it never has to be investigated twice.</p><p>This integration does not replace formal documentation. It helps teams reuse what they already know, preserve the context created during delivery, and keep project knowledge from disappearing into scattered conversations.</p><h4>For leaders, three points are worth remembering:</h4><p>· <strong>One channel.</strong> The assistant lives inside Microsoft Teams, so knowledge is captured and reused where the team already works.</p><p>· <strong>Two-tier responses.</strong> AI answers when it has the evidence, and escalates to a human when it does not.</p><p>· <strong>Zero knowledge lost.</strong> Every validated answer is re-indexed, so each resolved question makes the next one faster.</p><p>The takeaway for executives is simple: AI productivity is not measured by how impressive the assistant sounds, but by whether teams capture knowledge as work happens, resolve uncertainty earlier, and keep delivery moving with confidence.</p><h3>What’s Next: From Shared Memory to Auto-Drafted Documentation</h3><p>Because every validated answer is already source-backed and re-indexed, the knowledge base becomes more than a place to look things up — it becomes structured, evidence-backed raw material for formal documentation. A natural next step is a dedicated documentation agent that draws on this accumulated project memory to draft formal artifacts — architecture notes, troubleshooting runbooks, onboarding guides, and design-decision records — with subject-matter experts reviewing and approving rather than writing from a blank page. In that model, the everyday conversations captured today become the first draft of tomorrow’s official documentation, closing the loop between delivery work and the records it should produce.</p><h3>The Road Ahead: Extending the Baseline</h3><p>From this baseline, several improvements can extend the platform’s value as adoption grows:</p><p>· <strong>Broader source coverage.</strong> Extend ingestion beyond Microsoft Teams to Azure DevOps and GitHub, SharePoint, wikis, and meeting transcripts, so more delivery knowledge becomes searchable.</p><p>· <strong>Trust and freshness signals.</strong> Add answer confidence scoring and stale-knowledge flagging, so living documentation stays current and reliable as decisions evolve.</p><p>· <strong>Leadership analytics.</strong> Turn execution traces into a knowledge-gap view — most-asked topics, unanswered questions, and time-to-answer trends — showing leaders where to invest.</p><p>· <strong>Governance and scale.</strong> Introduce role- and project-based access control and treat model and embedding selection as ongoing optimization, ready for multi-client rollout.</p><p>· <strong>Agentic expansion.</strong> Move from answering to acting — specialized agents for documentation, onboarding, and compliance, coordinated by Foundry, with human approval gating each action.</p><p><strong>WHERE TO START</strong></p><p>If you are weighing this pattern for your own delivery organization, three questions are a useful starting point: where does your team lose the most time re-finding answers, which decisions are most costly to lose, and where would a source-backed assistant fit into the tools they already use? We would be glad to walk through how this approach could apply to your teams.</p><p>I’d like to thank the team behind this live AI integration — Facundo Báez, Jerónimo Colombo, Matías Lera, and Javier Macchiavello — for their contribution across the architecture implementation, cross-team integration, and operational hand-off activities that helped turn the scenario into a practical capability for improving any operations team productivity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cbdba37aac7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stocks Streaming: Real-Time Intelligence on Microsoft Fabric]]></title>
            <link>https://medium.com/@maurok/stocks-streaming-real-time-intelligence-on-microsoft-fabric-8b9c4417186a?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b9c4417186a</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 10:45:43 GMT</pubDate>
            <atom:updated>2026-07-14T10:49:16.064Z</atom:updated>
            <content:encoded><![CDATA[<h4>A technical story about a real-time stocks intelligence platform on Microsoft Fabric, where market movement, contextual enrichment, AI-assisted reasoning, alerting, and analytics operate as one connected experience.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RmncrxFKivRhoYpQ" /><figcaption>Photo by <a href="https://unsplash.com/@dkfra19?utm_source=medium&amp;utm_medium=referral">Dimitri Karastelev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Problem: Turning Market Movement into Real-Time Insight</h3><p>The problem starts with the difference between reporting and action. Financial customers do not only need a historical view of market activity; they need to know what is happening now, understand when a movement matters, and see enough context to decide whether it is worth attention. In this platform, that need is expressed as a real-time data problem: ingest market data as it moves, detect meaningful price movement, enrich that movement with context, and turn it into a personalized recommendation.</p><p>So what should the platform do when a stock drops beyond a meaningful threshold? It has to determine whether that movement is a buy-the-dip opportunity for any customer, identify who should be notified, and explain why the signal matters. Everything in the implementation supports that path, from raw market signals to customer-facing output.</p><p>That is where the engineering story changes. Collecting prices or rendering a dashboard is not enough. The platform has to keep market data moving, separate fast signals from slower context, reason over the assembled evidence, and record the outcome so it can both act and preserve an audit trail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QPt1sIDcrkrvWuCwHk87JQ.png" /><figcaption>End-to-end architecture view showing how market data enters through producer paths, lands in the shared Fabric real-time foundation, and supports downstream decisioning and serving experiences.</figcaption></figure><h3>The Architecture: A Fabric-Centered Real-Time Spine</h3><p>The architecture is organized as a connected real-time platform with a shared spine. Market data enters through four producer paths that reflect different data shapes: price snapshots, trade executions, company profile data, and company news. Each path feeds the same Fabric real-time architecture, and the shared analytical store becomes the place where raw events, curated data, enriched context, and alert decisions come together.</p><p>But once the data is flowing, not every path should move at the same speed. The price path acts as the hot path: quotes and trades are refined close to ingestion and monitored for sharp movement. The news path runs on a slower cadence because articles require heavier preparation and sentiment enrichment. The decision path ties both together by combining price movement, trading context, customer exposure, and recent sentiment before deciding whether an alert should proceed.</p><p>This is the first major design choice: the system does not force every data shape into the same tempo. Trades behave like a true live stream, quotes are frequent snapshots, company profiles change more slowly, and news becomes useful as contextual enrichment. Treating those inputs according to their natural pace keeps the real-time path focused while still giving the decision layer the context it needs.</p><p>From that same foundation, the platform serves multiple audiences: live monitoring, analytical reporting, and natural-language exploration. The important architectural point is not any single surface, but that one curated and enriched data foundation supports operational visibility, analysis, and ad-hoc questions without creating separate pipelines for each experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QPt1sIDcrkrvWuCwHk87JQ.png" /><figcaption>Architecture spine showing the four producer paths — quotes, trades, company profiles, and news — feeding the shared Microsoft Fabric real-time foundation.</figcaption></figure><h3>Implementation Flow: From Market Signal to Decision</h3><p>The implementation begins by accepting that market data arrives in different forms. Price snapshots, trade executions, profile information, and news articles are collected through separate producer paths because they do not change at the same speed and do not require the same treatment. The first step in the journey is therefore not transformation; it is respecting the shape and tempo of the incoming data.</p><p>From there, each input has to become usable. Fast-moving price data follows a lightweight curation path: it needs mechanical cleanup, readable fields, and a structure that supports immediate monitoring. News takes a different route because it can arrive more than once, needs to be aligned into time windows, and becomes more valuable after sentiment enrichment.</p><p>That split becomes one of the central technical decisions in the platform. Raw-to-curated is not treated as a single pattern. Price data is refined close to ingestion so the hot path stays fast. News is processed on a schedule because the work is heavier, more stateful, and tied to enrichment rather than immediate detection.</p><p>But is a price move enough to become a useful recommendation? Not on its own. The platform still needs recent price behavior, trading volume, customer relevance, and news sentiment, so the decision is based on a broader picture than the live tick itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_boPpuQKcEnWvGtKC50vPw.png" /><figcaption>News curation flow showing how contextual data is deduplicated, aligned into time windows, and prepared on a cadence before it supports the decision layer.</figcaption></figure><p>At this point, the implementation moves from data movement to decisioning. The platform detects a sharp move on the live price path, applies a sequence of checks before invoking AI-assisted reasoning, and emits a decision event whether the alert is sent or suppressed. That design keeps action and audit connected: the same platform that triggers the customer notification also records why a signal did or did not become an alert.</p><p>The result is a technical flow with clear stages: ingest market data, curate each shape according to its tempo, enrich the slower context where it adds value, detect meaningful movement, reason over the assembled evidence, act through a notification, and preserve the decision for later inspection.</p><p>This structure turns the platform from a data pipeline into a real-time intelligence loop. The system does not only ingest and display stock data; it detects, enriches, reasons, acts, and audits within one connected implementation.</p><h3>The Decisions That Shaped the Platform</h3><p><strong>Decision 1: Start with real-time response, not delayed reporting.</strong> The platform is built around the need to know what is happening now, identify when a market movement matters, and give the customer enough context to act. That makes the problem real-time from the start.</p><p><strong>Decision 2: Separate the inputs by shape and tempo.</strong> The platform treats quotes, trades, company profiles, and news as different data shapes. Trades drive the live alerting path, quotes provide frequent snapshots, profiles act as slower reference data, and news feeds a contextual enrichment path.</p><p><strong>Decision 3: Use one shared real-time foundation.</strong> All producer paths feed a shared Fabric architecture where raw events, curated records, enriched context, and alert decisions can be read by the decision loop, dashboards, reports, and natural-language experiences from the same underlying store.</p><p><strong>Decision 4: Match curation to the work each data shape needs.</strong> Price data uses a fast, lightweight curation path because it is high-frequency and needs only mechanical cleanup. News uses a scheduled enrichment path because it requires deduplication, time alignment, and sentiment analysis before it becomes useful to the decision layer.</p><p><strong>Decision 5: Use AI differently depending on the job.</strong> Sentiment enrichment is applied as a repeatable batch-style step over many news items. The buy-the-dip decision is handled as a single, context-specific reasoning step after the platform has assembled the relevant price history, volume, customer context, and recent sentiment.</p><p><strong>Decision 6: Emit every decision, not only successful alerts.</strong> The platform records both sent and suppressed decisions. A validated opportunity can trigger a customer notification, while a suppressed event still records why the alert did not proceed. That makes the decision loop useful for action, deduplication, and audit.</p><h3>Design Trade-offs: Keeping the Right Path Fast</h3><p>The first constraint is that the incoming data does not share one shape or one clock. Trades, quotes, profiles, and news each move differently, so the architecture avoids pushing every source through the same processing pattern.</p><p>The answer is to split the flow by tempo. The price path stays close to real time and remains focused on sharp market movement. News runs on a cadence because it requires heavier preparation and sentiment enrichment. That separation keeps the fast path focused while the slower path adds the context that helps explain why the movement may matter.</p><p>The second constraint is avoiding duplicate contextual data and unnecessary reprocessing in the news path. The platform addresses this by processing only new records, checking against already curated data, and aligning news into time windows so it can later be compared with price movement.</p><p>Even then, not every sharp price drop should become a customer alert. The platform does not notify on a raw movement alone. It checks whether the symbol was already alerted recently, whether trading volume supports the move, whether there are relevant customers, and whether the assembled context supports a buy-the-dip recommendation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKfw4Hcr1f81RGIyq1lEDA.png" /><figcaption>Alerting loop showing how a sharp price movement triggers validation checks, AI-assisted buy-the-dip reasoning, and sent or suppressed decision events.</figcaption></figure><p>And what happens when the signal does not become an alert? The platform still preserves the decision trail. Instead of dropping non-opportunities silently, it publishes a decision event for both sent and suppressed outcomes. That choice turns alerting into a loop: past decisions help prevent repeated alerts, and every near-miss has a recorded reason.</p><p>Together, these choices make the implementation cohesive. Fast data stays on the hot path, slower context is prepared on its own cadence, AI is applied where reasoning or enrichment is needed, and decision events connect action with audit.</p><h3>From Price Movement to Decision Loop</h3><p>The journey starts with a live market signal. A price moves sharply, but that movement cannot stand alone. It has to be captured on the near real-time path, placed where it can be monitored and queried, and connected with additional context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUWZy27VNx_G3hz30iEwPg.png" /><figcaption>Live monitoring view showing current market activity from the shared real-time foundation.</figcaption></figure><p>What does the platform need before it can act? Recent trading activity, customer relevance, historical movement, and recent news sentiment. Because those inputs arrive at different speeds, they are prepared through different paths before meeting in the decision layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RNLReQfNzebbhWpsgvtcLg.png" /><figcaption>Analytical reporting view showing how price movement, volume, and enriched news sentiment can be explored together.</figcaption></figure><p>The price path shows what is happening right now by watching live movement and detecting sharp drops. The news path helps explain the movement by preparing articles, enriching them with sentiment, and making that context available when the decision needs it.</p><p>Only when those paths meet does the platform have enough context to decide. When a meaningful movement is detected, it assembles the available evidence, applies checks that avoid repeated or low-quality alerts, and uses AI-assisted reasoning to decide whether the signal should become a customer-facing recommendation.</p><p>If the recommendation is strong enough, the platform sends a customer notification that combines fixed, factual market details with generated explanation text. If the recommendation does not pass the checks, the platform still records the decision and the reason it stopped. Either way, the decision becomes part of the platform’s history.</p><p>That history matters because the platform is not only reacting; it is remembering. Previous alert decisions help suppress repeated notifications, while the full decision trail supports later inspection. This is what turns the alerting path from a one-way script into a decision loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oj7GUdIpg8KbBC_ZaN5UjA.png" /><figcaption>Natural-language exploration view showing how users can ask ad-hoc questions over the same curated and enriched foundation.</figcaption></figure><p>The full implementation therefore moves through a clear sequence: capture live market movement, refine the fast and slow data paths differently, enrich the context, detect a meaningful condition, reason over the assembled evidence, notify when appropriate, and preserve the outcome for audit and future decisions.</p><h3>Conclusion: A Real-Time Pattern That Acts and Remembers</h3><p>The Stocks Streaming platform shows how live market data can become a real-time intelligence experience when the architecture connects ingestion, curation, enrichment, reasoning, action, and audit. It is not just a dashboard over changing prices; it is a platform that can detect a meaningful movement, assemble the context around it, decide whether it matters, and explain the outcome.</p><p>The technical approach works because each part has a defined role. Real-time inputs feed the hot path. Slower context is prepared on its own cadence. AI enriches news and reasons over a single triggered event. Alerts are sent only after the platform checks the available evidence, and every decision is recorded. That is the path from raw stock movement to a real-time intelligence platform on Microsoft Fabric.</p><p>I want to give special thanks to the people that drove this idea into a real-implementation and built the platform: Pablo Costantini, Leandro Martínez, Horacio Monsalvo, and Santiago Moreno — thank you guys!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b9c4417186a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Data Platforms to Intelligent Operations: Building AI-Native Systems with Microsoft Fabric IQ]]></title>
            <link>https://medium.com/@maurok/from-data-platforms-to-intelligent-operations-building-ai-native-systems-with-microsoft-fabric-iq-bac13c2926d9?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac13c2926d9</guid>
            <category><![CDATA[microsoft-fabric]]></category>
            <category><![CDATA[intelligent-operations]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[fabriciq]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 09:31:45 GMT</pubDate>
            <atom:updated>2026-07-14T09:31:45.531Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>AgriOps showcases how SOUTHWORKS uses Microsoft Fabric and Fabric IQ to turn fragmented data, real-time signals, ML models, and AI agents into an intelligent operating model for faster decisions, proactive response, and continuously improving business outcomes.</em></h4><h3>The Shift to AI-Native Operations</h3><p>Organizations are entering a new data reality: data is no longer just a source for retrospective reporting; it has become the operating layer for real-time decisions, AI-enabled workflows, and continuous optimization.</p><p>AI has raised the bar, but 2026 evidence shows the challenge has shifted from adoption to execution. Gartner forecasts that 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from less than 5% in 2025 — making trusted data, governance, and operational context critical to real business value.</p><p>AgriOps — a Southworks R&amp;D scenario for intelligent farm operations that fully leverages Fabric-native capabilities, and it’s designed to show how real-time telemetry, historical data, ML models, AI agents, dashboards, and natural language access can work together as one operating system– puts that shift into practice by unifying real-time and historical signals, applying ML where prediction matters, embedding AI agents where monitoring and response are needed, and making insights accessible through natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Btx88vNIVO3Gz9yD" /><figcaption>Photo by <a href="https://unsplash.com/@antilumen?utm_source=medium&amp;utm_medium=referral">Robert Wiedemann</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The Integration Gap Holding AI Back</h3><p>Most organizations are no longer asking whether to use AI; they are trying to make it work reliably inside complex operations. In 2026, that means unifying data, business meaning, governance, and action — not simply adding another AI tool on top of disconnected systems.</p><p>Recent 2026 research reinforces the execution gap: one enterprise data and analytics survey found that 99% of leaders still struggle to define consistent business metrics across tools and departments, 87% want more visibility into how AI uses and interprets data, and nearly 80% of data teams spend more than half their time preparing data instead of generating insight.</p><p>Microsoft’s 2026 Fabric IQ update frames the same problem from a platform perspective: AI agents and real-time applications need a shared context layer so people and agents operate from a consistent, always-current understanding of the business.</p><p>In a scenario like AgriOps, this means real-time telemetry, historical sensor records, financial datasets, weather inputs, ML features, dashboards, alerts, and natural language access must operate as one system.</p><p>AgriOps was designed to test that pattern: moving from disconnected data and isolated AI experiments to a Fabric-native architecture where data engineering, real-time intelligence, ML, Fabric IQ agents, and business experiences are connected from day one.</p><h3>SOUTHWORKS’ R&amp;D Model: From Real-World Complexity to AI-Native Architecture</h3><p>SOUTHWORKS approached AgriOps as an R&amp;D initiative with a clear goal: create a compelling, realistic scenario that applies state-of-the-art data platforms and techniques to connect data, AI, and operational workflows into one coherent architecture for a high-complexity environment.</p><p>Agriculture was chosen because it mirrors challenges seen across many industries: distributed operations, real-time telemetry, fragmented history, predictive risk, field-level decisions, and users who need insights without becoming data specialists.</p><p>The scenario connects Microsoft Fabric for unified data engineering and analytics, Fabric IQ for AI agents and natural language experiences, Eventhouse for real-time intelligence, OneLake for a governed data foundation, Power BI for decision support, and ML pipelines for predictive intelligence.</p><h3>Fabric IQ: The Embedded Intelligence Layer</h3><p>Fabric IQ turns the platform from a passive data repository into an active decision-support system. Microsoft announced Fabric IQ general availability in June 2026, positioning it as the shared context layer for AI agents and real-time applications.</p><p>· <strong>Operations Agent</strong> — an AI-driven monitoring system that continuously evaluates live telemetry against business rules defined in natural language, generating proactive alerts and recommendations without requiring code.</p><p>· <strong>Data Agent</strong> — a natural language interface that allows users to query data across the platform, removing dependency on dashboards or technical intermediaries.</p><h3>Our Take on Operational ML: Predictive, Explainable, Continuously Learning</h3><p>Beyond Fabric IQ, AgriOps uses a closed-loop ML architecture designed for operational decisions. A forward model predicts current-season financial risk using only data available during the season, while a retrospective model evaluates closed financial periods and creates labels for future training.</p><p>The daily inference pipeline pulls IoT readings from Eventhouse, aggregates them by farm, combines them with the latest known financial data, and produces three outputs: distress flag, expected margin, and risk score. Delta Time Travel compares each farm against the previous day to classify risk as stable, improving, or worsening.</p><p>At season end, actual financials feed the retrospective model, new labels are created, and the best-performing models are registered in MLflow. Power BI exposes the outputs through business, model performance, and current-season views — keeping ML traceable, monitored, and connected to operational action.</p><p><strong>Model-driven improvements enabled by Fabric</strong></p><ul><li>Engineered 50+ Gold-layer ML features across financial, farm profile, sensor-derived, and climate categories.</li><li>Generated 3 daily predictions per farm: distress flag, expected margin, and risk score.</li><li>Used an 80/20 train-validation split for model training.</li><li>Compared Logistic Regression, Random Forest, and XGBoost across the prediction tasks.</li><li>Exposed ML outputs through 3 Power BI views: business, model performance, and current season.</li></ul><p><strong>Operational ML improvements</strong></p><ul><li>Shifted ML from offline experimentation to a daily operational workflow.</li><li>Connected live telemetry, historical financials, and Gold-layer features into one predictive loop.</li><li>Compared each farm against the previous day using Delta Time Travel to detect stable, improving, or worsening risk.</li><li>Registered and versioned the best-performing models in MLflow for traceability.</li><li>Closed the learning loop with retrospective labels that feed the next training cycle.</li></ul><h3>Why SOUTHWORKS Is Built for Microsoft Fabric Innovation</h3><p>AgriOps demonstrates how SOUTHWORKS turns emerging platform capabilities into practical, production-oriented patterns: ingestion, storage, analytics, AI agents, and ML connected into one operating model where business outcomes drive the technical design.</p><h3>What This Unlocks: AI-Native Operations in Practice</h3><p>The platform highlights five practical outcomes that matter beyond the demo:</p><ul><li><strong>AI-driven operations, not just analytics</strong> — continuous monitoring and intelligent alerting transform data into immediate action, reducing reliance on manual oversight.</li><li><strong>Embedded intelligence through Fabric IQ</strong> — AI agents operate directly on live data, enabling proactive decision-making without custom engineering layers.</li><li><strong>Predictive, self-improving systems</strong> — integrated ML pipelines anticipate risk and continuously refine predictions through feedback loops.</li><li><strong>Democratized access to insights</strong> — natural language interaction removes technical barriers and enables business users to engage directly with data.</li><li><strong>Accelerated time-to-value</strong> — by embedding AI and ML within Fabric, organizations can move from data ingestion to actionable intelligence without building and maintaining complex integration layers.</li></ul><h3>From R&amp;D Showcase to Repeatable Enterprise Pattern</h3><p>AgriOps is not just an agricultural scenario; it is a repeatable pattern for data-rich operations that need faster visibility, earlier risk detection, and trusted AI-enabled action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*mbeVD0ZpyBmGhGdjylpB6A.png" /><figcaption>Intelligent Operations Dashboard — Daily Predictions, Risk Signals, and Operational KPIs Driving Proactive Farm Management.</figcaption></figure><ul><li><strong>Unified operating foundation</strong> — real-time, historical, financial, and environmental data connected in one Fabric-native architecture.</li><li><strong>Actionable intelligence</strong> — dashboards, alerts, agents, and ML predictions tied to live operational conditions.</li><li><strong>Continuous improvement</strong> — retrospective labels, MLflow versioning, and retraining cycles that keep models improving.</li><li><strong>Transferable model</strong> — a pattern that can extend to logistics, energy, manufacturing, retail operations, field services, and other complex environments.</li></ul><h3>Closing Thoughts</h3><p>AgriOps brings together the core themes behind SOUTHWORKS’ R&amp;D model: start with a complex real-world scenario, apply Microsoft Fabric capabilities with purpose, and prove how data, AI, and operational workflows can work as one system. The result is a practical showcase of Fabric across ingestion, OneLake, Eventhouse, Medallion architecture, Power BI, Fabric IQ agents, and closed-loop ML — not as isolated tools, but as an integrated foundation for faster insight, proactive response, and continuous improvement.</p><p>For organizations modernizing their data estate or operationalizing AI, the lesson is clear: durable impact comes from connecting platform architecture to business execution. AgriOps shows how a Fabric-native approach can move teams beyond experimentation and toward intelligent operations that are governed, explainable, and ready to scale.</p><p><em>I’d like to wrap up this post by thanking the team that participated in this initiative — Facundo Báez, Carolina Bruscantini, Derek Fernández, Alejo Galetto, and Vanesa Oshiro — for their collective contribution in shaping the scenario, validating what is possible, and exploring how to leverage Fabric IQ to its fullest.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac13c2926d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a fast and efficient AI Assistant with LangChain and OpenAI]]></title>
            <link>https://medium.com/southworks/building-a-fast-and-efficient-ai-assistant-with-langchain-and-openai-d1bd115ab0b9?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1bd115ab0b9</guid>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[langchain-agents]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 11:47:27 GMT</pubDate>
            <atom:updated>2024-06-10T11:47:27.058Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Short Intro</h3><p>In an era where digital transformation is reshaping industries, the financial sector is no exception. We’ve recently embarked on a pivotal project to implement an AI assistant for one of the largest banks in one of the countries where SOUTHWORKS operates. This initiative reflects a strategic move by this big bank to leverage AI technologies and deliver enhanced solutions for their customers.</p><p>The bank’s primary objective was to explore innovative solutions that harness the power of AI to improve customer experiences. Our team collaborated closely with theirs, examining several business cases to identify the most critical and impactful one. Through this process, we pinpointed the area causing the most significant pain points for their customers and them, making it our focal point for AI enhancement.</p><p>By addressing this critical business case, the bank stands to significantly elevate its reputation. The targeted enhancements are expected to boost customer satisfaction, and in the competitive banking industry, such improvements can differentiate the bank from its competitors, fostering stronger customer relationships and loyalty, in addition to attracting new customers through innovation.</p><p>The implementation aligns with stringent government regulations on financial information sharing and openness, ensuring compliance while driving innovation. This balance of regulatory adherence and cutting-edge technology positions the bank as a leader in the financial sector.</p><h3>The Business Case</h3><p>Unrecognized transactions on customers’ bank statements are a common issue that frequently leads to increased support calls. When customers identify transactions they do not recognize, they often reach out to the bank’s support team for clarification. Support agents then have to go through multiple data sources to link and verify records, a process that can be time-consuming and prone to errors. This cumbersome procedure not only impacts the efficiency of support operations but also contributes to customer frustration and dissatisfaction.</p><p>In many instances, if the support agent cannot resolve the issue quickly, a formal claim must be filed to follow up and consolidate all relevant information. This additional step not only delays resolution but also increases the administrative burden on the bank. Financial regulations mandate that the number of claims remains below certain thresholds, which adds pressure on the bank to resolve unrecognized transaction issues promptly and efficiently, and more important avoid them from even happening. Failure to comply with these regulatory requirements can result in penalties and damage to the bank’s reputation.</p><p>To address these challenges, a fast and smart solution for identifying and contextualizing unrecognized transactions is crucial. Implementing an AI-powered assistant can significantly streamline this process by providing both customers and support agents with instant access to transaction details and context. This technology can quickly analyze transaction data, cross-referencing it with multiple sources to provide accurate and comprehensive information. By doing so, it reduces the need for filing claims and ensures that issues are resolved promptly, keeping the number of claims within regulatory limits.</p><p>The adoption of this AI assistant not only improves operational efficiency but also enhances customer satisfaction and loyalty. Customers benefit from quicker resolutions and a more seamless banking experience, while support agents can focus on more complex issues rather than spending excessive time on routine transaction inquiries. Ultimately, this smart solution helps the bank maintain compliance with financial regulations, while it enhances its reputation and strengthens its competitive edge in the financial services industry.</p><h3>Our Solution</h3><p>Leveraging the capabilities of LangChain and Azure Open AI, we developed an advanced AI agent specifically designed to tackle this issue. This agent, crafted in Python, incorporates a wide array of tools to enhance LLM’s capabilities, ensuring comprehensive and accurate transaction analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*9hkprVNoa_RVXQuf_zQIZg.png" /></figure><p>In a multi-cloud environment where compute and data sources are physically isolated and deployed, our solution maintains optimal performance and security. The AI assistant can operate seamlessly across these isolated systems, effectively managing the complex data landscape typical of large financial institutions. By enabling real-time querying over a few Data Lake sources, the assistant utilizes various Retrieval-Augmented Generation (RAG) techniques to ground its responses with precise and relevant information. This ability to instantly cross-reference multiple data sources ensures that both customers and support agents receive accurate and comprehensive information regarding unrecognized transactions.</p><p>Furthermore, the AI assistant employs the latest OpenAI models, including GPT-Turbo and GPT-4, to deliver sophisticated and contextually relevant financial insights. These models enhance the assistant’s ability to understand and respond to complex queries, providing users with detailed explanations and solutions. By quickly analyzing transaction data and offering clear context, the AI assistant reduces the need for filing claims and ensures that issues are resolved promptly.</p><p>Ultimately, the implementation of this AI-powered assistant transforms the way unrecognized transactions are handled. Customers benefit from quicker resolutions and a more seamless banking experience, while support agents can focus on more complex issues, reducing their administrative burden.</p><p>Sounds amazing, right? But…</p><h3>Problems along the way</h3><p>As we progressed through the project, we encountered several issues that added layers of complexity to our implementation. Initially, our scope included a manageable number of data sources. However, as different teams within the bank continued to bring more data sources into the picture, the complexity of information retrieval and grounding queries increased significantly. Each new data source required integration, validation, and adjustment, which complicated the AI assistant’s prompt engineering and data handling processes.</p><p>The initial approach of using a single <a href="https://arxiv.org/abs/2210.03629">ReAct (Reason &amp; Act) agent</a> to respond to all types of questions became increasingly cumbersome and slow over time. This methodology, which relies on a cycle of interactions between the LangChain ReAct agent and the underlying LLM model, struggled to keep up with the growing complexity. As the data’s schema and quality varied widely, the original prompts became laden with detailed instructions to handle specific use cases. This increased prompt complexity led to the model “unlearning” certain tasks and hallucinating more frequently, producing inaccurate or fabricated responses.</p><p>One significant issue we encountered was LLM hallucinations. On multiple occasions, the language model responded with seemingly proper answers that were actually based on incorrect or invented data. For instance, when users asked for a summary of expenses, the LLM sometimes generated a summary without actually processing any real data, resulting in a fabricated expense report. These hallucinations were particularly problematic because they undermined the trust in the AI assistant’s reliability.</p><p>In some use cases, these hallucinations were easy to detect because the LLM’s responses did not correlate with any accessed data. Re-iterating the user’s question often resolved the issue temporarily, but this was not a sustainable solution. The need to continually adjust and refine prompts to mitigate hallucinations and ensure accurate data retrieval highlighted the limitations of our initial approach and underscored the need for more robust and scalable solutions. As a result, we ended up exploring more advanced techniques and possibly different architectural models to enhance the accuracy and reliability of the AI assistant.</p><p>Have you ever heard about the old and known ‘divide and conquer’ technique? 😏</p><h3>Refactoring for Performance and Efficiency</h3><p>After considering different options and architectures, that I’ll share in another article as I want to keep this one short, we ended up implementing a multi-agent system to better manage the increasing complexity. This refactoring aimed to improve the speed and accuracy of the AI assistant’s responses, thereby enhancing the overall user experience.</p><p>This new approach involves first identifying the type of inquiry and then routing it to a more specialized agent. By leveraging a mix of GPT-Turbo and GPT-4 models, we can tailor responses more effectively while ensuring that each agent is optimized for specific tasks. By creating specific agents for a subset of use cases, we managed to keep the prompts smaller and the LLMs interactions faster and more reliable. This modular approach allows each agent to handle its designated tasks more efficiently, reducing the risk of hallucinations and improving response times.</p><p>Before moving forward allow me sharing a high-level diagram of our multi-agent architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*f1J09PAea-FfSMmxzVLxWw.png" /></figure><p>In our multi-agent system, we employ different OpenAI models to validate and detect the user inquiry initially. These models are responsible for identifying the proper use case and routing the query to the appropriate specialized agent. This initial validation step ensures that each inquiry is directed to an agent best suited to handle it, streamlining the entire process and enhancing accuracy. By routing inquiries correctly from the start, we minimize unnecessary interactions with the LLMs and ensure that users receive precise and relevant responses.</p><p>The need for the ReAct model diminishes in this refactored approach, as each use case is now tailored to a specific data query. When necessary, the LLMs can still be asked to generate the required data query, but many use cases do not require further LLMs interaction once the initial inquiry is detected. For example, a “balance inquiry” can be directly mapped to a specific data query and a templated text response, eliminating the need for additional processing. This targeted handling of use cases not only speeds up response times but also reduces the cognitive load on the LLMs, further minimizing the risk of hallucinations.</p><h3>Conclusions</h3><p>Large prompts were one of the significant challenges we faced. Depending on the underlying LLM model, extensive prompts with numerous instructions were often ignored, took longer to process, and tended to cause more hallucinations compared to smaller, more focused prompts. Additionally, maintaining large and complex prompts proved to be cumbersome.</p><p>The multi-agent approach allowed us to manage the complexity of data sources and queries more effectively. By assigning specialized agents to handle specific tasks, we ensure that the AI assistant remains responsive and accurate, even as the scope of the project expands.</p><p>At the end of the day, this new implementation after the refactor not only addresses the initial problems but also provides a scalable framework that can adapt to future requirements, ultimately delivering a more robust and reliable AI-powered assistant to solve your business case.</p><p>This collaboration between SOUTHWORKS and this top-5 bank on building this Personal Finance AI assistant underscores the transformative impact of AI in financial services. By addressing key pain points, enhancing customer satisfaction, and ensuring regulatory compliance, the bank is set to achieve significant gains in reputation, customer loyalty, and market expansion.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1bd115ab0b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/building-a-fast-and-efficient-ai-assistant-with-langchain-and-openai-d1bd115ab0b9">Building a fast and efficient AI Assistant with LangChain and OpenAI</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fraud Detection applying Unsupervised Learning techniques]]></title>
            <link>https://medium.com/southworks/fraud-detection-applying-unsupervised-learning-techniques-4ae6f71b266f?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ae6f71b266f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Tue, 29 Jun 2021 16:24:24 GMT</pubDate>
            <atom:updated>2021-06-29T16:24:24.791Z</atom:updated>
            <content:encoded><![CDATA[<p>Within these series I will be sharing with you the result of research and development work we conducted with SOUTHWORKS’ teams during some periods of time over 2020–2021.</p><p>As there were lot of people involved through different stages, and not to forget mentioning anyone, I want to thank everybody who was involved into different development projects, researching spikes, generating documentation, and so forth and so on along the way.</p><p>You can refer to these two previous posts which are setting the ground for the concepts and techniques I’m going to discuss in this article:</p><ul><li><a href="https://medium.com/southworks/a-short-intro-glimpse-into-the-machine-learning-world-6c4b96452236">A short-intro glimpse into the Machine Learning world</a></li><li><a href="https://medium.com/southworks/a-deeper-view-into-unsupervised-learning-concepts-and-techniques-7c404a3bc87b">A deeper view into Unsupervised Learning concepts and techniques</a></li></ul><p>For this particular post, I want to give special thanks to the people that helped me to wrap it up: Pablo Costantini, Derek Fernandez &amp; Ignacio Grayeb — thank you guys!</p><h3>First.. A little bit of introductory theory!</h3><p>Also known as outlier detection, anomaly detection is a data mining process used to determine types of anomalies found in a data set and to determine details about their occurrences. In simpler words, anomaly detection deals with the identification of unusual patterns/behavior that doesn’t conform to the usual trend.</p><p>An outlier is nothing but a data point that differs significantly from other data points in the given dataset.</p><p>Outliers can come in different flavors, depending on the environment: <strong>point outliers</strong>, <strong>contextual outliers</strong>, or <strong>collective outliers</strong>. Point outliers are single data points that lay far from the rest of the distribution. Contextual outliers can be noise in data, such as punctuation symbols when realizing text analysis or background noise signal when doing speech recognition. Collective outliers can be subsets of novelties in data such as a signal that may indicate the discovery of new phenomena.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ZuMtQWQ5cfoRnZSq" /><figcaption>Source: <a href="https://www.google.com/search?q=anomaly+detection&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwj2ypeW-4fxAhWqBmMBHRvdCpkQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p><strong>Anomaly detection is the process of finding the outliers in the data</strong>, i.e. points that are significantly different from the majority of the other data points.</p><p>Outliers can be of two kinds: <strong>univariate</strong> and <strong>multivariate</strong>. Univariate outliers can be found when looking at a distribution of values in a single feature space. Multivariate outliers can be found in a n-dimensional space (of n-features). <strong>Looking at distributions in n-dimensional spaces can be very difficult for the human brain, thus we need to train a model to do it for us.</strong></p><p>Large, real-world datasets may have very complicated patterns that are difficult to detect by just looking at the data. That’s why the study of anomaly detection is an extremely important application of Machine Learning.</p><p>Anomaly detection is based on 3 principles:</p><ul><li><strong>Most of the real world data significantly deviate from normal,</strong></li><li><strong>Dataset can have multiple anomalies, </strong>so we need an algorithm that can detect multiple anomalies,</li><li><strong>Data points can depend on a lot of features</strong>, thus we require an algorithm that can overcome the course of the dimensionality.</li></ul><h3>Use-cases</h3><p>Anomaly detection has wide applications across industries. In this section, we’ll present some of the most popular use cases. However, you should have in mind that this list is not exhaustive:</p><p><strong>Banking</strong>. Anomaly detection can help finding abnormally high deposits. Every account holder generally has certain patterns of depositing money into their account. If there is an outlier to this pattern the bank needs to be able to detect and analyze it; e.g. for money laundering.</p><p><strong>Finances</strong>. Finding the pattern of fraudulent purchases. Every person generally has certain patterns of purchases which they make. If there is an outlier to this pattern the bank needs to detect it in order to analyze it for potential fraud.</p><p><strong>Manufacturing</strong>. Abnormal machine behavior can be monitored for cost control. Many companies continuously monitor the input and output parameters of the machines they own. It is a well-known fact that before failure a machine shows abnormal behavior in terms of these input or output parameters. A machine needs to be constantly monitored for anomalous behavior from the perspective of preventive maintenance.</p><p><strong>Networking</strong>. Detecting intrusion into networks. Any network exposed to the outside world faces this threat. Intrusions can be detected early on using monitoring for anomalous activity in the network.</p><p><strong>Healthcare</strong>. Detecting fraudulent insurance claims and payments.</p><h3>Techniques</h3><h4><strong>Isolation Forest</strong></h4><p>It is an unsupervised learning algorithm that identifies anomalies by isolating outliers in the data — based on the <a href="https://blog.paperspace.com/decision-trees/">Decision Tree</a> algorithm. It does so by randomly selecting a feature from the given set of features and then randomly selecting a split value between the max and min values of that feature. This random partitioning of features will produce shorter paths in trees for the anomalous data points, thus distinguishing them from the rest of the data.</p><p>In general, the first step is to construct a profile of what’s “normal”, and then report anything that cannot be considered normal as anomalous. However, the isolation forest algorithm does not work on this principle; it does not first define “normal” behavior, and it does not calculate point-based distances.</p><p>As you might expect from the name, Isolation Forest instead works by isolating anomalies <strong>explicitly isolating anomalous points in the dataset.</strong></p><p>The Isolation Forest algorithm is based on the principle that <em>anomalies are observations that are few and different, which should make them easier to identify</em>. Isolation Forest uses an ensemble of Isolation Trees for the given data points to isolate anomalies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h9i7fPJ5cu-mHYsv.png" /><figcaption>Source: <a href="https://www.google.com/search?q=isolation+forest+anomaly+detection&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwie7dzD-4fxAhXQAmMBHXXbClAQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>Isolation Forest recursively generates partitions on the dataset by randomly selecting a feature and then randomly selecting a split value for the feature. Presumably the anomalies need fewer random partitions to be isolated compared to “normal” points in the dataset, so the anomalies will be the points which have a smaller path length in the tree, being the path length the number of edges traversed from the root node.</p><p>Using Isolation Forest, we can not only detect anomalies faster but also <strong>require less memory</strong> compared to other algorithms since, as abnormal data points mostly have a lot shorter tree paths than normal data points, trees in the isolation forest do not need to have a large depth.</p><h4><strong>One-Class Support Vector Machine</strong></h4><p>A <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.675.575&amp;rep=rep1&amp;type=pdf">support vector machine</a> is another effective technique for detecting anomalies. A SVM is typically associated with supervised learning, but there are extensions such as <a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.OneClassSVM.html">One-Class SVM</a> that can be used to identify anomalies as an unsupervised problems (in which training data is not labeled).</p><p>As we’ve just presented it, a One-Class Support Vector Machine is an unsupervised learning algorithm that is trained only on the <em>normal</em> data, it learns the boundaries of these points and is therefore able to classify any points that lie outside the boundary as <strong>outliers</strong>.</p><p>In a Support Vector Machine, data points are represented as points in space in such a way that points from different categories are separated by a plane. It’s possible to think of this like a line through data points that separates data of different classes.</p><p>New data-points are mapped into the same space and their location relative to the plane is used to predict which categories each point belongs, with the plane being referred to as the decision boundary. In the case where the decision boundary needs to be non-linear (i.e. where classes cannot be separated by a straight line), SVMs also have the ability to project space through a non-linear function (kernels), lifting the data to a space with a higher dimension where a linear decision boundary <em>does</em> separate classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/0*zc14U-DFfp1HDuHI" /><figcaption>Source: <a href="https://www.google.com/search?q=One-Class+Support+Vector%C2%A0Machine&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjK7dnQ-4fxAhUHmRQKHWegAV8Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><h4>Neural Networks with LSTM autoencoders</h4><p>These types of networks excel at finding complex relationships in multivariate time series data. This is the way to go when we want to detect anomalies in time series</p><p>Long Short-Term Memory (LSTM) networks are a sub-type of the more general Recurrent Neural Networks (RNN). A key attribute of recurrent neural networks is their <em>ability to persist information</em>, or cell state, for using it later on. This makes them particularly well suited for analysis of temporal data that evolves over time. LSTM networks are used in tasks such as speech recognition, text translation and here, in the <strong>analysis of sequential data streams for anomaly detection</strong>.</p><p>One way goes uses LSTMs to build a <strong>prediction model </strong>—<strong> </strong>i.e., given current and past values, predict the next few steps in the time-series. Then, error in prediction gives an indication of anomaly. Another way is to directly use LSTM as a <strong>classifier </strong>with two classes, normal and anomalous.</p><p>The prediction model based approach is better when anomalous instances are not easily available, whereas a classifier based approach is more suitable when there are sufficient labelled instances of both normal and anomalous instances.</p><p>There is another way where <strong>LSTM autoencoders and encoder-decoder frameworks</strong> have been used as <strong>reconstruction models </strong>— where some form of reconstruction error is used as a measure of anomaly. On the idea behind such models the autoencoder is trained to reconstruct the normal time-series and it is assumed that such a model would do badly to reconstruct the anomalous time-series having not seen them during training. In simpler words, the system looks at the previous values over hours or days and predicts the behavior for the next minute. If the actual value a minute later is within a standard deviation, then there is no problem. If it is more, it is an anomaly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A9eiKgljlRazFuRm.png" /><figcaption>Source: <a href="https://www.google.com/search?q=LSTM+autoencoders+frameworks&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjDq4D--4fxAhUZEWMBHStfBv0Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>These models have been used for anomaly detection in electrocardiograms (ECG), engines, power demand, network (failures/intrusions), and novelty detection in music, etc.</p><h4>Local Outlier Factor</h4><p>LOF is a score that indicates how likely it is that a certain data point is an outlier (or anomaly). It detects anomalous points computing the local density deviation of a given data point with respect to its neighbors.</p><p>There are two important parameters to consider:</p><ol><li>Number of neighbors</li><li>Contamination</li></ol><p>LOF calculates the distance of the {<em>number of neighbors}-</em>th nearest point (if the number of neighbors is three, for example, it calculates the distance between the current point and the third nearest point) and then for each point it computes the density of points inside the calculated distance for that point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/0*JRcpWyqnmROl1DGD.png" /><figcaption>Source: <a href="https://www.google.com/search?q=local+outlier+factor&amp;hl=en-US&amp;tbm=isch&amp;source=lnms&amp;sa=X&amp;ved=2ahUKEwjG07eM_IfxAhXlAmMBHUQUAPQQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329&amp;dpr=2">Google Images</a></figcaption></figure><p>The density of each point will then be compared to the densities of their <em>k</em> neighbors. Considering a data point <em>a</em>, the LOF is basically the average ratio of the densities of the neighbors of <em>a</em> to the density of <em>a</em>. If the ratio is greater than <em>1</em>, the density of point <em>a</em> is on average smaller than the density of its neighbors and, thus, from point <em>a</em>, we have to travel longer distances to get to the next point or cluster of points than from <em>a</em>’s neighbors to their next neighbors. Keep in mind that the neighbors of a point <em>a </em>may not consider <em>a </em>a neighbor as they have points in their reach which are a lot closer.</p><p>The contamination parameter determines the proportion of the most isolated points (points that have the highest LOF scores) to be predicted as anomalies.</p><p>The <a href="https://scikit-learn.org/stable/">scikit-learn</a> Python library has its own implementation of this algorithm, which has two different methods determined by a parameter named <em>novelty</em>. These methods are:</p><ul><li><strong>Novelty Detection</strong>: The model is fitted with clear data (no outliers) and then, when the predict function is called with another dataset, the model can predict if each point is an outlier or not.</li><li><strong>Outlier Detection</strong>: The model is fitted and predictions are run over the same dataset, determining if each sample is an outlier. However, it is not possible to run predictions with a dataset other than the one used to fit the model.</li></ul><h4>Gaussian mixture</h4><p>A <a href="https://brilliant.org/wiki/gaussian-mixture-model/">Gaussian mixture model</a> is a probabilistic model that assumes that the probability distributions for all points in a dataset can be obtained as a weighted sum of Gaussian (or normal) distributions with unknown parameters.</p><p>These models attempt to find different subpopulations within an overall population, without knowing which subpopulation a data point belongs to (thus being an unsupervised learning technique). The goal is to estimate the parameters for each normal distribution component in the dataset.</p><p>There are different techniques to implement Gaussian mixture models, the most common of which is <strong>expectation-maximization (EM)</strong>. We will focus on this technique.</p><p>The EM method is most commonly used when the number of components <em>k </em>is known. Expectation-maximization is a statistical algorithm that involves an iterative process to determine which points came from which Gaussian component. This process consists of two steps:</p><ul><li><strong>Expectation</strong>: This step consists of calculating the expectation of the component assignments for each data point given their parameters (component weight, mean and standard deviation).</li><li><strong>Maximization</strong>: The second step has the goal of maximizing the expectations calculated in the previous step, updating the model’s parameters.</li></ul><p>The algorithm iterates repeating these two steps until it converges, resulting in a maximum likelihood estimate. The animation below shows a two-component bivariate example adjusting its parameters over multiple iterations until it converges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*_TrRWKUgyaxy85Jz.gif" /><figcaption>Source: <a href="https://www.google.com/search?q=gaussian+mixture+clustering&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwiOqaOi_IfxAhX-DmMBHdQbDfsQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>An extension of this algorithm, <a href="https://scikit-learn.org/stable/modules/mixture.html#estimation-algorithm-variational-inference">variational inference</a>, adds regularization by integrating information from prior distributions.</p><h4>Other methods</h4><ul><li><em>Density-based anomaly detection with K-Nearest Neighbors:</em> It assumes that normal data-points occur around a dense neighborhood and abnormalities are far away. The nearest set of data-points are evaluated using a score, which could be Euclidean distance or a similar measure dependent on the type of the data (categorical or numerical). <strong>K-nearest neighbor</strong> is a simple, non-parametric lazy learning technique used to classify data based on similarities in distance metrics such as Euclidean, Manhattan, Minkowski, or Hamming distance.</li><li><em>Clustering-based anomaly detection with K-Means</em>: It assumes that data-points that are similar tend to belong to similar groups or clusters, as determined by their distance from local centroids. <strong>K-means</strong> is a widely used clustering algorithm. It creates <em>k</em> similar clusters of data points. Data instances that fall outside of these groups could potentially be marked as anomalies.</li></ul><h3>Credit card fraud detection — Hands on</h3><p>This scenario has four different techniques evaluating their results on a credit card fraud detection dataset. The dataset used is labelled although labels are only being used to evaluate results — not to fit models.</p><p>For this scenario the following four techniques were applied:</p><ul><li>Isolation Forest</li><li>Autoencoders</li><li>Local Outlier Factor</li><li>Gaussian Mixture</li></ul><p>Implementation in Python is available in a <a href="https://github.com/southworks/machine-learning-unsupervised/blob/main/anomaly-detection/fraud_detection.ipynb">Jupyter notebook</a> in our GitHub repository.</p><h4>The dataset</h4><p>The chosen <a href="https://www.kaggle.com/shubhamjoshi2130of/abstract-data-set-for-credit-card-fraud-detection?select=creditcardcsvpresent.csv">dataset</a>, that is available on Kaggle, contains raw data corresponding to credit card transactions. It has 3075 entries with 11 features and an additional column that indicates whether each transaction was fraudulent or not. For model fitting, this last column was dropped and only used to validate results. Around 15% of the entries are labelled as frauds.</p><p>An initial cleansing of the dataset was performed to have a basic version to work with. At this stage, columns that lacked data or that didn’t provide any useful information were discarded, and some minor tweaks were applied — like turning all string values to numbers. The resulting dataset had 9 features out of the original 11. With this minimally preprocessed dataset, each technique was executed in order to chose the one that performed best.</p><p>Later on, more preprocessing was done within the dataset, looking for a positive impact on the best technique’s performance. The number of features was reduced down to 6, while additional transformations were applied after analyzing the data, by looking at the distributions for each feature and finding correlations between them:</p><ul><li>Three columns that were related to credit card charge-backs were merged. These columns contained different measures that were related, thus the process simplified them into a single binary feature that indicates whether there was a charge-back or not.</li><li>The original dataset has a column to indicate whether the transaction was with a foreign country and another one that determines if that country is high-risk. A combination of these two columns into a single categorical one with three possible values was done: local transactions (0), foreign transactions with a low-risk country (1) and foreign transactions with a high-risk country (2).</li><li>Columns related to transaction amounts were normalized to narrow down their ranges.</li><li>Taking a look at correlation values between features, the weight of features more closely correlated to fraudulent cases was increased.</li></ul><h4>Results</h4><p>Each technique’s algorithm has a sensibility parameter that has an impact on the amount of cases that it considers to be outliers. In a real-world scenario, cases marked as outliers (frauds) by an algorithm would then be audited by subject matter experts to make the final decision on whether they are actual frauds or not.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46342b04eda6153f07cb61cda4d871d8/href">https://medium.com/media/46342b04eda6153f07cb61cda4d871d8/href</a></iframe><p>Once you can ‘trust’ on algorithm’s accuracy, the goal would be to catch all fraud cases by auditing as little transactions as possible. In this dataset, about 15% of the transactions are fraudulent, so the “ideal algorithm” would mark for further analysis only that subset which will be the 100% of the frauds within the dataset.</p><p>If we compare algorithms performance from that standpoint, and evaluate the efficiency of each algorithm by the ability to submit the smallest and more accurate subset for an auditor to validate, can observe some algorithm like Autoencoder or Gaussian Mixture not doing well at all — as a matter of fact, the last one start to fall in accuracy while getting the subset for validation bigger and bigger.</p><p>The curves within the plots below, show the ratio of marked frauds as a function of the ratio of cases required to be audited for each algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GRnVtfP3JPtyLkVH.png" /></figure><h3>Isolation Forest and One-class SVM — Hands on</h3><p>A more simpler, and different, scenario using other of the techniques mentioned through this post was designed. This is a simple spike for detecting abnormal values within a dataset consisting of 2D data points normally distributed, containing a portion of uniformly distributed outliers.</p><p>Two different techniques, along with their Python implementation, are described:</p><ul><li>Isolation Forest</li><li>One-class Support Vector Machine</li></ul><p>The spike can be found within our repository in a <a href="https://github.com/southworks/machine-learning-unsupervised/blob/main/anomaly-detection/anomaly_detection.ipynb">Jupyter notebook</a>.</p><h4>The dataset</h4><p>These are two small, independent datasets consisting on 200 <em>normally-distributed</em> data-points, which will represent the normal values, and 50 <em>uniformly-distributed </em>data-points, with higher-highs and lower-lows compared to normal data-points.</p><p>Data-points are distributed in the following way:</p><p><strong>Base clusters (normally distributed)</strong></p><ul><li>(x, y) / {x ∈ [1, 3], y ∈ [1, 3]}</li><li>(x, y) / {x ∈ [-3, -1], y ∈ [-3, -1]}</li></ul><p><strong>Outliers (uniformly distributed)</strong></p><ul><li>(x, y) / {x ∈ [-4, 4], y ∈ [-4, 4]}</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*aX_wa8xsFyt4sMw_" /></figure><h4>Results</h4><p>After fitting both Isolation Forest and One-class Support Vector Machine on the training set, predictions were ran over the test set in order to test how both algorithms extrapolate their knowledge.</p><p><strong>Isolation Forest</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/0*6PJYmOITRRRYoeeG.png" /></figure><p>Metrics report that for the spike, this algorithm achieved:</p><ul><li>Accuracy: 92% (46/50 anomalies detected)</li><li>False detections: 0% (0 total)</li></ul><p><strong>One-class Support Vector Machine</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*95VCZu5Jtn2ehduW.png" /></figure><p>Metrics report that for the spike, this algorithm achieved:</p><ul><li>Accuracy: 78% (39/50 anomalies detected — 18 false detections)</li><li>False detections: 31.5% (18 total)</li></ul><h3>Wrapping up</h3><p>As you have seen through the article there are many techniques within the unsupervised learning paradigm, and within them, some more aligned to detecting anomalies in a dataset.</p><p>As detecting a fraud is finding anomalies in the dataset at the end of the day, is that these techniques are a perfect fit for this scenario. Depending on the attributes your dataset contains and the different pre-processing techniques applied to curate and get the input ready for analysis, some are of them are going to perform better than others — as you have seen within the example showcased above.</p><p>It is always a good idea to try many different pre-processing approaches and algorithms, that you will need to fine-tune, in order to understand which one delivers to you the best results. Although for some scenarios, like the ones we showcased above, there are algorithms that are known to perform better than others, a big part of the job is the work your data scientist does within the initial raw dataset to get it ready for the training and validation jobs ahead.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ae6f71b266f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/fraud-detection-applying-unsupervised-learning-techniques-4ae6f71b266f">Fraud Detection applying Unsupervised Learning techniques</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A deeper view into Unsupervised Learning concepts and techniques]]></title>
            <link>https://medium.com/southworks/a-deeper-view-into-unsupervised-learning-concepts-and-techniques-7c404a3bc87b?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c404a3bc87b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Wed, 23 Jun 2021 14:17:11 GMT</pubDate>
            <atom:updated>2021-06-23T14:17:11.617Z</atom:updated>
            <content:encoded><![CDATA[<p>Within these series I will be sharing with you the result of research and development work we conducted with SOUTHWORKS’ teams during some periods of time over 2020–2021.</p><p>As there were lot of people involved through different stages, and not to forget mentioning anyone, I want to thank everybody who was involved into different development projects, researching spikes, generating documentation, and so forth and so on along the way.</p><p>Please do not forget to go read the introductory post to the machine learning world <a href="https://medium.com/southworks/a-short-intro-glimpse-into-the-machine-learning-world-6c4b96452236">here</a>.</p><p>For this particular post, I want to give special thanks to the people that helped me to wrap it up: Pablo Costantini, Derek Fernandez &amp; Horacio Monsalvo — thank you guys!</p><p>You can find some interesting spikes and stuff at this <a href="https://github.com/southworks/machine-learning-unsupervised">repo</a>, go check it out!</p><h3>So.. Unsupervised Learning, what is it?</h3><p><strong>Unsupervised learning</strong> is a type of machine learning that looks for previously undetected patterns in a dataset with <strong>no pre-existing labels</strong> and with a <strong>minimum of human supervision</strong>.</p><p>It allows you to create general systems that can be trained with very little data. In contrast to supervised learning that usually makes use of human-labeled data, unsupervised learning — also known as self-organization — allows for modeling of probability densities over inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MH1u4NU4Sk8QsJSV.png" /><figcaption>Source: <a href="https://www.google.com/search?q=unsupervised+machine+learning&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwidx73R-YfxAhXj4uAKHefbBjAQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>If you recall on what I surfaced on my <a href="https://medium.com/southworks/a-short-intro-glimpse-into-the-machine-learning-world-6c4b96452236">previous post</a>, learning implies that model parameters are estimated in some way. Generally, the problem is reduced to minimize a cost function (or maximize some other metric) that depends directly on the parameters.</p><p>Ideally, cost functions are convex functions (as shown below) and the absolute minimum can be easily obtained following the descendant gradient. However, <strong>most practical cost functions are non-convex</strong> having multiple minimums, some local and some absolute. Non-convex functions are harder to optimize since algorithms cannot ensure finding a close solution.</p><p>If you remember a little bit from math courses, most of this implies calculating first and second derivatives, among other calculations — so yes, hard math underlies beneath all these concepts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*CR8YmCE2-An3Q_6o.png" /><figcaption>Source: <a href="https://www.google.com/search?q=convex+non-convex&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjW27XI_IfxAhUt8uAKHSSgCOgQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>Further on, unsupervised learning algorithms consist generally of iterative methods that depend on the initial conditions. These are great for this problems because after getting stuck in a local area early on, it can escape those to find the absolute minimum.</p><p>So, algorithms are mostly driven by the following steps:</p><ol><li>Initialize parameters, either randomly or using some criteria.</li><li>Recalculate parameters following the proposed method.</li><li>Calculate metric, generally some likelihood-related value.</li><li>If metric increased its value from the past iteration, stop. If not, go to point 2.</li></ol><h3>Application domains</h3><p>As previously explained, unsupervised learning is very powerful when having a really huge unlabeled set of data that needs to be processed in an efficient manner.</p><p>Unsupervised learning use cases can be grouped into two categories: <strong>exploratory analysis </strong>and<strong> dimensionality reduction</strong>.</p><p>Unsupervised learning is very useful in <strong>exploratory analysis</strong> because it can <strong>automatically identify patterns in data, and correlate them</strong>. For example, if an analyst were trying to segment consumers, unsupervised clustering methods would be a great starting point. In situations where it is either impossible or impractical for a human to propose trends in the data, unsupervised learning can provide initial insights that can then be used to test individual hypotheses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rD1f9_tVZXB6YARDhxY3pQ.png" /><figcaption>Source: <a href="https://www.google.com/search?q=clustering&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwi9tMyLoYjxAhXUAmMBHR6LDw0Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p><strong>Dimensionality reduction</strong>, which refers to the methods used to <strong>represent data using a smaller set of features</strong>, can be accomplished through unsupervised methods such as implementing auto-encoders, which are a type of artificial neural networks (ANN). The aim of an auto-encoder is to learn a representation, or encoding for a set of data, by training the network to ignore signal “noise”. Along with the reduction side, a reconstruction side is learnt, where the auto-encoder tries to generate from the reduced encoding a representation as close as possible to its original input.</p><p>In representation learning, the goal is to learn relationships between individual features, allowing you to represent the data using the latent features that interrelate its initial features. This sparse latent structure is often represented using far fewer features than you have started with, so further data processing can be done much less intensive while eliminating redundant features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pepF9H7w5gQlAB7t.png" /><figcaption>Source: <a href="https://www.google.com/search?q=autoencoder+neural+network&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwj0kfDq-YfxAhWF5-AKHXVZAbEQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>Just to mention some few real-life applications:</p><ul><li><strong>Healthcare:</strong> A subgroup of cancer patients grouped by their gene expression measurements.</li><li><strong>E-Commerce: </strong>Groups of shoppers based on their browsing and purchasing histories.</li><li><strong>Ratings:</strong> Movie group by the rating given by movies viewers.</li><li><strong>Image/Video segmentation:</strong> Image or video clustering analysis to divide them in groups based on similarities.</li></ul><h3>Use cases</h3><h4>Clustering</h4><p>Clustering is a Machine Learning technique that involves the grouping of data points. Given a set of data points, you can use a clustering algorithm to classify each data point into a specific group (cluster).</p><p>In theory, data points that are in the same group should have similar properties or features, while data points in different groups should have highly dissimilar ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZaiZSLXBMMXFEUo6.jpg" /><figcaption>Source: <a href="https://www.google.com/search?q=unsupervised+clustering&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwib3POB-ofxAhVF8uAKHf5cB14Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>Just to mention some few real-life applications:</p><ul><li><strong>Marketing:</strong> It can be used to characterize &amp; discover customer segments for marketing purposes.</li><li><strong>Biology:</strong> It can be used for classification among different species of plants and animals.</li><li><strong>Libraries:</strong> It is used for grouping different books on the basis of topics and information.</li><li><strong>Insurance:</strong> It is used to acknowledge the customers, their policies and identifying the frauds.</li><li><strong>City Planning: </strong>It is used to make groups of houses and to study their values based on their geographical locations and other factors present.</li><li><strong>Earthquake studies: </strong>By learning the earthquake-affected areas we can determine the dangerous zones.</li></ul><p>Clustering also involves several different techniques. Below you can find the 2 most-used techniques:</p><ul><li><strong>K-Means </strong>is an iterative algorithm that tries to partition the dataset into <em>K</em> pre-defined, distinct, non-overlapping subgroups or clusters, where <em>each data point belongs to only one group</em>. The algorithm tries to make the intra-cluster data points as similar as possible while also keeping the clusters as different (far) as possible. It assigns data points to a cluster such that the <strong>sum of the</strong> <strong>squared distance between the data points and the cluster’s centroid</strong> (arithmetic mean of all the data points that belong to that cluster) <strong>is</strong> <strong>minimized</strong>. The less variation we have within clusters, the more homogeneous the data points are within the same cluster.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*AWkNqiMA9D2HVWgh" /><figcaption>Source: <a href="https://www.google.com/search?q=k-means+clustering&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwj9z_yb-ofxAhUSohQKHV9nBtoQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><ul><li><strong>Mean Shift </strong>is a sliding-window-based algorithm that attempts to find dense areas of data points. It is a centroid-based algorithm, meaning that the goal is to locate the center points of each group/class, which works by updating candidates for center points to be the the mean of the points within the sliding-window. These candidate windows are then filtered in a post-processing stage to eliminate near-duplicates, forming the final set of center points and their corresponding groups. In contrast to K-Means, <strong>there is no need to select the number of clusters as mean-shift automatically discovers this.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/0*Nj2nKP3dxh3t1G97.gif" /><figcaption>Source: <a href="https://www.google.com/search?q=k-means+clustering&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwj9z_yb-ofxAhUSohQKHV9nBtoQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><h4>Anomaly detection</h4><p>Also referred to as outlier detection, <strong>anomaly detection</strong> is simply the mode of detecting and identifying anomalous data in any data-based event or observation that differs majorly from the rest of the data.</p><p>Anomalies in a big dataset may follow very complicated patterns, which are difficult to detect “by eye” in the great majority of cases. This is the reason why the field of anomaly detection is well suited for the application of Machine Learning techniques.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*zaACxN9ZmSEZCb41.jpg" /><figcaption>Source: <a href="https://www.google.com/search?q=anomaly+detection&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjhycnk-ofxAhUM0uAKHSZ3CykQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>It has several real-life applications, such as:</p><ul><li><strong>Banking:</strong> Every account holder generally has certain patterns of depositing money into their account. If there is an outlier to this pattern the bank needs to be able to detect and analyze it, e.g. for money laundering.</li><li><strong>Finances:</strong> Finding the pattern of fraudulent purchases.</li><li><strong>Healthcare:</strong> Detecting fraudulent insurance claims and payments.</li><li><strong>Manufacturing:</strong> A machine needs to be constantly monitored for anomalous behavior from the perspective of preventive maintenance.</li><li><strong>Networking:</strong> Detecting intrusion into networks.</li></ul><p>And there are also several techniques for performing this task, each having its own fundamentals. Here will list two of the most used ones:</p><ul><li><strong>Isolation Forest </strong>works on the principle of isolating anomalies, instead of the most common techniques of profiling normal points. This is due to abnormal points being easier and less computationally expensive to isolated them compared to normal points.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/0*YKRk66U0Y5sKPXiV.jpg" /><figcaption>Source: <a href="https://www.google.com/search?q=isolation+forest&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjPhP2fnYjxAhXJ8OAKHSpdDt4Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><ul><li><strong>Artificial Neural Networks with auto-encoders: </strong>The aim of using auto-encoders is to learn a representation (encoding) for a set of data, typically for dimensionality reduction. Along with the reduction side, a reconstructing side is learnt, where the auto-encoder tries to generate from the reduced encoding a representation as close as possible to its original input.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*AKM16kpEAtvEIKyM.jpg" /><figcaption>Source: <a href="https://www.google.com/search?q=ann+with+autoencoders&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjL0vWcnojxAhUTxBQKHe-eC5EQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>The idea is to <strong>use the auto-encoder network to compress features to a lower-dimensional representation, which captures the correlations and interactions between the various variables</strong>. The auto-encoder network is then trained on data representing the “normal” operating state, with the goal of first compressing and then reconstructing the input variables. During the dimensionality reduction, the network learns the interactions between the various variables and should be able to re-construct them back to the original variables at the output. As this happens, one will start to see an increased error in the networks re-construction of the input variables. Thus, outliers can be detected by monitoring the re-construction error.</p><blockquote>This approach is usually used for detecting abnormal patterns in real-time signals.</blockquote><h3>Wrapping up</h3><p>As part of the series, I’m going to be publishing more deep dives, with customer samples in some cases, into each of the use cases and real-life applications described here. Sooner, posts on clustering and anomaly detection will be shared as part of the R&amp;D work SOUTHWORKS is engaged into.</p><p>As you have learn through the article, the main goal of unsupervised learning is to help you understand and better leverage your data for further analysis and processing as part of a bigger business intelligence pipeline. A technique that might be called <em>semi-supervised learning</em> is closely related to the unsupervised learning paradigm as a research focus on pre-training a model (called <em>discriminator</em> or <em>encoder</em>), leveraging the underlying structure or distribution within the data.</p><p>For instance, using unsupervised learning helps into discovering and learning the structure and patterns in the input variables, and correlating them. Later on, this correlated data can be used as inputs for a supervised learning algorithm as training data, using the obtained model to make predictions of unseen inputs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c404a3bc87b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/a-deeper-view-into-unsupervised-learning-concepts-and-techniques-7c404a3bc87b">A deeper view into Unsupervised Learning concepts and techniques</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A short-intro glimpse into the Machine Learning world]]></title>
            <link>https://medium.com/southworks/a-short-intro-glimpse-into-the-machine-learning-world-6c4b96452236?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c4b96452236</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Mon, 07 Jun 2021 14:40:57 GMT</pubDate>
            <atom:updated>2021-07-02T08:48:47.398Z</atom:updated>
            <content:encoded><![CDATA[<h3>A short-glimpse into the Machine Learning world</h3><p>Before diving into cool features like self-driving cars, predicting future outcomes for sports betting, autonomous retail stores, improved e-commerce user experience by automatically tagging, organizing and visually searching content, or even some image/video segmentation, or voice recognition, let’s set a common ground of terms and concepts to understand what are things like artificial intelligence (AI), machine learning (ML) and deep learning (DL) that, most of times, are used indistinctly.</p><p>As with this I’m starting a series of posts of super interesting work we have started sometime ago, I want to thank the team that worked with me at the very beginning into putting the ball in motion: Pablo Costantini, Derek Fernandez, Juan Manso &amp; Horacio Monsalvo — thank you very much for your contributions.</p><h3>AI, ML, DP — What’s the difference?</h3><p>Let’s start by talking about the difference between these terms that are commonly mentioned when you see a self-driving car or a spam email blocked properly (or not). They <em>are</em> related but they <em>are not</em> the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FWQzMMhzCrwuMx-k.png" /><figcaption>Source: <a href="https://www.google.com/search?q=deep+learning&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwim4tOm-IfxAhUCcBQKHd5VBPoQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><h3>Artificial Intelligence (AI)</h3><p>Artificial intelligence can be summed up as the ability of machines to perform, in an automated manner, intellectual tasks that are normally performed by humans.</p><p>As shown on the figure above, AI involves ML and DL, but there are also some approaches that don’t require any learning by the machine. For instance, if you enter some hard-coded rules into a hardware or software to perform a task, the machine can successfully do its job without any learning (if all the edge cases are considered of course). This type of AI is called <em>symbolic AI</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*JowDbAN3oD7fg8ht.png" /></figure><p>Arguably, one of the first machines to perform AI was the vending machine. Think about it for a moment and forget about Siri or smart-ads. At that time asking a machine for a specific refreshment by clicking a button and entering a coin, and then the machine giving it to you, was like sci-fi. Before vending machines, you needed to interact with a human being to get your favorite refreshment, later on you could get it from a machine.</p><p>The example is pretty silly, but the idea is there. You could also program a machine to play chess and everybody can agree that the machine is “<em>thinking</em>”, but all it is doing is following some predefined rules. Just like the vending machine!</p><p>Now, if you let the machine set those rules on their own, then we are talking about machine learning.</p><h3>Machine Learning (ML)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*llCvBVvzkNMuUl-e.png" /><figcaption>Source: <a href="https://www.google.com/search?q=machine+learning&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjPrbjQ-IfxAhUKjhQKHQIMBLQQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>With Machine Learning, we enter into a whole new paradigm. Instead of defining the rules by ourselves, we want to simply show the system a sequence of data inputs and answers, so it can generate rules that best fit our given set of inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/0*pSfdyLSSHBX5RAiQ.png" /></figure><p>Then, we can say that the machine is <em>trained</em> rather than <em>programmed</em> to do a specific task.</p><p>To understand a little bit better on how it ‘<em>learns</em>’, let’s take a look at the following example solved on the next figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lEe-oMd0HFygewHf.png" /></figure><p>Our system needs to establish a rule to determine if a given dot in space corresponds to the black dotted group, or the white one.</p><p>If we would have to create such rule <em>by hand</em>, we could take the approach number 2 where we draw a couple of straight lines and choose black for one side, and white for the other. That rule will have very good results, but if we leave our system <em>to learn</em> from the raw data, it could internally come up with a different representation such as the number 3.</p><p><strong>So in a single sentence, machine learning systems loop finding the best possible representation for data to output a specific task. <em>Learning </em>is the process of automatically search for better representations.</strong></p><p>There are three widely known <em>learning paradigms</em>, each one having its own strengths and flaws:</p><ul><li>Supervised learning</li><li>Unsupervised learning</li><li>Reinforcement learning</li></ul><p>Let’s delve a bit deeper into each one of them in the sections ahead.</p><h4><strong>Supervised learning</strong></h4><p>Supervised learning is a <em>machine learning task in which a function maps the input to output data using the provided input-output pairs</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4Ha3scpBShDJKs8C.png" /><figcaption>Source: <a href="https://www.google.com/search?q=supervised+learning&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwivt8_l-IfxAhWGmBQKHUsICC4Q_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>In this type of learning, you need to give both the <em>inputs</em> and <em>outputs</em> (usually in the form of labels) to the computer for it to learn from it. What the computer does is that it generates a function based on this data, which can be anything like a simple line, to a complex convex function, depending on the data provided.</p><p>When it comes to Supervised learning and its practical uses, we can mention the two most popular ones:</p><p><strong>1. Classification</strong>: The input of a classification algorithm is a set of labeled examples. The output of a binary classification algorithm is a classifier, which you can use to predict the class of new unlabeled instances. Classification could be either Binary or Multi-class, depending on the number of labels. Examples of these could be:</p><ul><li><strong>Binary</strong></li></ul><p>- Understanding sentiment of Twitter comments as either “positive” or “negative”.</p><p>- Diagnosing whether a patient has a certain disease or not.</p><p>- Making a decision to mark an email as “spam” or not.</p><p>- Determining if a photo contains a particular item or not, such as a dog or fruit.</p><ul><li><strong>Multi-class</strong></li></ul><p>- Determining the breed of a dog as a “Siberian Husky”, “Golden Retriever”, “Poodle”, etc.</p><p>- Understanding movie reviews as “positive”, “neutral”, or “negative”.</p><p>- Categorizing hotel reviews as “location”, “price”, “cleanliness”.</p><p><strong>2. Regression</strong>: Learning task that is used to predict the value of the label from a set of related features. The label can be of any real value and is not from a finite set of values as in classification tasks. Regression algorithms model the dependency of the label on its related features to determine how the label will change as the values of the features are varied. The input of a regression algorithm is a set of examples with labels of known values. The output of a regression algorithm is a function, which you can use to predict the label value for any new set of input features. Examples of regression scenarios include:</p><p>- Predicting house prices based on house attributes such as number of bedrooms, location, or size.</p><p>- Predicting future stock prices based on historical data and current market trends.</p><p>- Predicting sales of a product based on advertising budgets.</p><h4><strong>Unsupervised learning</strong></h4><p>In this learning paradigm, the computer is provided with just the <em>inputs</em> to develop a learning pattern. It is basically learning from no results, but inferring them instead.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ztNghUuaHoZ2mcQS.png" /><figcaption>Source: <a href="https://www.google.com/search?q=unsupervised+learning&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjko5f0-IfxAhUEA2MBHRz5AVoQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>This means that the computer has to recognize a pattern in the given input, and to develop a learning algorithm accordingly. So observing this behavior, we can conclude that <em>the machine learns through observation &amp; finds structures in data.</em></p><p>Some real-life examples of unsupervised learning involve:</p><p><strong>1. Clustering</strong>: Learning task that is used to group instances of data into clusters that contain similar characteristics. Clustering can also be used to identify relationships in a dataset that you might not logically derive by browsing or simple observation. The <em>inputs</em> and <em>outputs</em> of a clustering algorithm depends on the methodology chosen. You can take a distribution, centroid, connectivity, or density-based approach. Examples of clustering scenarios include:</p><p>- Understanding segments of hotel guests based on habits and characteristics of hotel choices.</p><p>- Identifying customer segments and demographics to help build targeted advertising campaigns.</p><p>- Categorizing inventory based on manufacturing metrics.</p><p><strong>2. Association</strong>: Rule-based machine learning method for discovering interesting relations between variables in large databases.</p><p>- Detecting complimentary goods by customers’ purchasing behavior. For instance, results for a real-life research on the sales data of a supermarket would indicate that if a customer buys onions and potatoes together, they are likely to also buy hamburger meat.</p><h4><strong>Reinforcement learning</strong></h4><p>Reinforcement Learning is a type of Machine Learning that allows machines and software agents to automatically determine the ideal behavior within a specific context, in order to maximize its performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/0*sWUZn0KMl4AIDBnO.png" /><figcaption>Source: <a href="https://www.google.com/search?q=reinforcement+learning&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwj8ucaA-YfxAhUR_RQKHZvhDS8Q_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>It differs from supervised learning in not needing labeled <em>input</em>/<em>output</em> <em>pairs</em> be presented, and also it does not need sub-optimal actions to be explicitly corrected. Instead, <em>the focus is on finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge)</em>.</p><p>There is an excellent analogy to explain this type of learning paradigm: training a dog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*jm1bp13Rvh_FtksJ.png" /></figure><p>This learning paradigm is like a dog trainer, which teaches the dog how to respond to specific signs, such as a whistle, a clap, or anything else. Whenever the dog responds correctly, the trainer gives a reward to the dog, which can be a bone or a biscuit.</p><p>A variety of different problems can be solved using Reinforcement Learning. Because its agents can learn without expert supervision, the type of problems that are best suited for this paradigm are complex problems where it appears to be no obvious or easily programmable solution. Two of the main ones are:</p><p><strong>1. Game theory</strong>: Determining the best move to make in a game often depends on a number of different factors, hence the number of possible states that can exist in a particular game is usually very large.</p><p><strong>2. Control theory</strong>: Such as elevator scheduling. Again, it is not obvious what strategies would provide the best, most timely elevator service. For control problems such as this, Reinforcement learning agents can be left to learn in a simulated environment and eventually they will come up with good controlling policies.</p><p>Independently of the paradigm, most machine learning algorithms are bound with math statistics and such, but some branches of ML such as Deep Learning, can come short in comparison with the others and be tagged as more empirical, engineering driven approach.</p><h3>Deep Learning (DL)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*tKr86-Ubuqy2m511.png" /><figcaption>Source: <a href="https://www.google.com/search?q=deep+learning&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwjLn7-N-YfxAhWM2BQKHVZeCIgQ_AUoAnoECAEQBA&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>Instead of having a model that simply adjusts its parameters to represent data correctly, we can create a successive set of <em>layers</em> that turn input data into multiple representations (highlighting one feature or another) that are feed to the next layer to represent it in better forms, obtaining higher level representations of data. This simple and power concept is the core of most deep learning models, where <em>deep</em> refers to the depth of the network, the amount of hidden layers.</p><p>To picture this, we are going to follow the next example: from a handwritten number image (taken from <a href="https://en.wikipedia.org/wiki/MNIST_database">MNIST dataset</a>), the system should output the correct number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sQBbY-0wm2563KNx.png" /><figcaption>Source: <a href="https://www.google.com/search?q=deep+learning+MNIST&amp;hl=en-US&amp;source=lnms&amp;tbm=isch&amp;sa=X&amp;ved=2ahUKEwiGoPqc-YfxAhVZ5eAKHfD9CGsQ_AUoAXoECAEQAw&amp;biw=2560&amp;bih=1329">Google Images</a></figcaption></figure><p>At first, we could guess what the first layer is trying to highlight on its outputted representation, mainly noise filtering, edge detection and such. But then, the representation increases its level of abstraction and we are unable to correctly guess what’s being emphasized. On this particular case, what’s being calculated is the correlation of a neighborhood of pixels with each other, but it could vary depending on models architecture and task at hand.</p><p>Thus, we can conclude that deep learning models are simply a multi-stage way to learn data representation.</p><h4>How does Deep Learning work?</h4><p>To correctly train deep learning models (this also applies to most ML models), we generally need to have 3 components well defined:</p><ul><li><em>Lots of Input data</em></li><li><em>Expected output value given a certain input</em></li><li><em>Measurement of models performance to determine whether it is improving or not</em></li></ul><p>A model is composed of layers, which are then composed by <em>nodes</em> or <em>neurons</em> (hence why they are called <em>neural</em> networks). Each neuron has many inputs; it could be the actual input of the network or multiple neurons outputs, or anything else depending on your architecture. Thus, neurons need to have some way to control the inputs and highlight the most important ones.</p><p>Each link of input and output is associated with a <strong>weight</strong>. To avoid getting too mathematical, we will say that neurons take the sum of the weighted inputs and output a value with range from 0 to 1 (in some cases its just binary). Those weights are the actual <em>parameters</em> that create the previously illustrated high level representation of the input.</p><p>On each learning loop, the network tweaks weights’ values of each layer — in some way — until it settles for the right ones</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JZr89E8ez0t1lq-C.png" /></figure><p>How do we know that these weights are correct? Well, we need to define a <em>loss function</em> that will take Y&#39; predictions and the true Y targets and will output a score to determine whether the network is working fine or not, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w-h-m2vVzjIWYhe1.png" /></figure><p>So, how does the network adjust itself based on the result of the loss function? Like every system, to be controlled correctly we need to measure its outcome and use it as a <em>feedback signal</em> to correct itself as shown on the next image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VcEogxKtZufUtKB8.png" /></figure><p>The <em>optimizer</em> implements an algorithm of <em>back-propagation</em> that “points” the right direction for the weights to be changed and lower the loss score. On the image below, we see an example where the optimizer is depicted as the black arrows pointing to the correct direction to get to the minimum loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*ebq0WzuhxZg3nyzS.png" /></figure><p>Thus, <strong>the weights of a trained network are the ones that minimize the loss function</strong>.</p><h3>Wrapping up</h3><p>As you have seen, there are a lot of different paradigms and approaches of what is called Machine Learning, and now you understand the difference between AI and ML, being the last one a subset of the previous.</p><p>In our next series of posts we are going to be showing you examples, spikes and other stuff we have been working on lately to dive a little bit deeper into each of the paradigms described in this article and concrete, although not unreachable, real life applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c4b96452236" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/a-short-intro-glimpse-into-the-machine-learning-world-6c4b96452236">A short-intro glimpse into the Machine Learning world</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a ‘Mental Health Monitoring’ App with AWS AI Services — Part III]]></title>
            <link>https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-iii-bc828ad6c39?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc828ad6c39</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Tue, 25 May 2021 10:05:45 GMT</pubDate>
            <atom:updated>2021-06-29T11:50:30.916Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building a ‘Mental Health Monitoring’ App with AWS AI Services — Part III</h3><p>In this article, we are going to wrap up the series that we started time ago on how to build a Mobile App that monitors participants feelings daily using Amazon Web Services Platform. The <a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-i-fa7566c6a3a5">application</a> allows managers to create groups and invite participants to register into these each of these groups. Once done, these participants are being asked about daily feelings (check-in points) and an AI automatically analyzes participants shared messages — creating time-series graphs on participants feeling evolution and notifying a manager in case an immediate action needs to take place. As sentiment analysis is a central piece of the whole thing, in this article we discuss on how to use AWS SageMaker for sentiment analysis using advanced machine learning techniques.</p><p>We will be seeing how to tackle into two different approaches: using SageMaker’s built-in <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/blazingtext.html">BlazingText algorithm</a> and a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) provided by the <a href="https://chainer.org/">Chainer framework</a>. In the process, we will show how to prepare the dataset (using as baseline a dataset of Twitter tweets that have already been labeled as positive or negative) and then build, train and deploy a <em>model</em> that can effectively categorize unseen inputs.</p><p>Finally, we will configure a SageMaker endpoint to provide the service for replacing the Lex/Comprehend bot solution shown in the previous articles (<a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-i-fa7566c6a3a5">Part I</a> &amp; <a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-ii-97134d553242">Part II</a>) in order to have a custom trained model for the specific scenario we have at hand, while keep improving and improving it.</p><p>Before moving forward I want to mention the group of people that collaborated on creating this article, the reference implementation linked to it and/or provided valuable feedback to allow me sharing it with you: Raul Benitti, Javier Caniparoli, Pablo Costantini &amp; Mauro Della Vecchia.</p><h3>SageMaker introduction</h3><p>AWS SageMaker is a cloud service that provides both a toolset for implementing machine learning flows, and resources to host trained models exposing them as directly consumable services.</p><p>It allows developing, testing and training algorithms using Jupyter Notebooks or to bring in custom solutions in the form of docker images, prepare the datasets, train models using custom algorithms (or built-in ones) and deploy endpoints that can be used to consume these models.</p><p>We used SageMaker to tackle the next three main tasks:</p><ul><li>Dataset preparation</li><li>Training a model</li><li>Deploying the model</li></ul><h3>Dataset preparation</h3><p>Text Preprocessing is an important step for Natural Language Processing (NLP) tasks. The transformations done in this step are targeted to bringing the text into a more digestible, predictable and analyzable representation that allows the ML algorithms to perform better.</p><p>To clean and format the dataset, we followed the next steps:</p><p>1) Remove unnecessary data: the dataset contains columns we do not need (e.g.: username of the user who wrote the tweet, date-time, id and flag). For our purposes, we only need text, and the sentiment label (positive or negative).</p><p>2) Separate columns in two lists, one for the text and one for sentiment labels. Both list should have the same length, with each tweet and associated sentiment label stored at the same position in each corresponding list.</p><p>3) Process the list of tweets removing information that can affect training: Not all tasks need the same level of preprocessing. For some tasks, you can get away without it or do it at a minimum. However, in some cases the dataset can be so noisy that, if not preprocessed enough, the training turns out to be garbage-in-garbage-out.</p><ul><li>Lower casing: Lowercasing ALL text, although commonly overlooked, is one of the simplest and most effective forms of text preprocessing. It is applicable to most text mining and NLP problems and can help in cases where the dataset is not very large and significantly helps with consistency of the expected output. For example, models might treat a word which is in the beginning of a sentence with a capital letter different from the same word which appears later in the sentence but without any capital letter and this might reduce the model accuracy.</li><li>Noise removal: is about removing characters, digits and pieces of text that can interfere with the text analysis and can produce results that are inconsistent in the downstream tasks. Noise removal is one of the most essential text preprocessing steps and is highly domain dependent. Some ways to reduce it include punctuation removal, special character removal, numbers removal, html formatting removal, domain specific keyword removal (e.g. ‘RT’ for retweets), source code removal, header removal, etc. It all depends on which domain you are working in and what entails noise for the task. For example, in the tweets’ dataset noise could be all special characters except hashtags as it signifies concepts that can characterize a tweet:</li><li><em>Replacing URLs</em>: Links starting with “http” or “https” or “www” are replaced by “URL”.</li><li><em>Replacing Emojis</em>: Replace emojis by using a pre-defined dictionary containing emojis along with their meaning. (eg: “:)” to “EMOJIsmile”)</li><li><em>Replacing Usernames</em>: Replace @Usernames with the “USER” word. (eg: “@Kaggle” to “USER”)</li><li><em>Removing Non-Alphabets</em>: Replacing characters except Digits and Alphabets with a space.</li><li><em>Removing Consecutive letters</em>: 4 or more consecutive letters are replaced by 2 letters. (eg: “Heyyyy” to “Heyy”)</li><li><em>Removing Short Words</em>: Words with length less than 2 are removed.</li><li>Removing Stopwords: Stop words are the commonly used words that do not add much meaning to a sentence. They can be safely ignored without sacrificing the meaning of the sentence. Examples of stop words in English are “a”, “the”, “is”, “are”, etc. The intuition behind removing stop words is that we remove noise and can focus on the meaningful words instead.</li><li>Normalization: Text normalization is the process of transforming text into a canonical (standard) form. For example, the word “gooood” and “gud” can be transformed to the canonical form “good”. Text normalization is important for noisy texts such as social media comments, text messages and comments to blog posts where abbreviations, misspellings and use of out-of-vocabulary words (oov) are prevalent.</li></ul><p>4) Tokenization and Word Embedding: Tokenization consists of breaking up the text into smaller chunks or segments called tokens (a letter, a word, or sub-words). This chunks of information can be then represented as discrete elements in the form of binary vectors with a dimension for each token. Word Embedding can then be used to compress these high dimensional vectors into smaller dimensions while extracting information about the relation between the tokens (i.e., learning that words like <em>one</em>, <em>two</em> and <em>three</em> are semantically close).</p><p>5) Split the dataset: Training algorithms usually need input datasets to be split into <em>training</em> and <em>testing</em> datasets, with the training dataset including all the possible patterns that define the problem. By general rule, 70% of the available data is allocated for training while the remaining 30% is used for validation and testing. Besides selecting the right ratio, it is also important that the resulting datasets are representative for the domain (the dataset should maintain the probability distribution of the domain). For Chainer, we used <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train_test_split function</a>. BlazingText, on the other hand, does not require partitioning.</p><p>6) Upload data: We upload the resulting datasets to an Amazon S3 bucket to be used as input for the training jobs.</p><h3>Training a model</h3><p>As we mention before we used both the built-in BlazingText algorithm, and an algorithm provided by the Chainer Framework.</p><h4>Training a model with a custom algorithm</h4><p>SageMaker offers a complete set of examples and templates to use both built-in as external frameworks (these are provided as Jupiter Notebooks).</p><p>In this example we will cover the steps we followed to train our RNN with LSTM model with Chainer.</p><p><strong>Implement the training function</strong></p><p>This requires providing a training script that the SageMaker platform can run. The script will be responsible for loading the data from the S3 location, configuring the training job with hyperparameters, training the model and saving its artifacts for deployment. The most important thing to keep in mind is to follow the interfaces required by SageMaker and the algorithm’s framework. To do so, the script must define some standard functions (a generic implementation will be used when a function definition is missing):</p><ul><li><em>model_fn</em>(…): Specifies how to load a model from the artifacts files.</li><li><em>input_fn</em>(…): Parses the input data into an object suitable for predict_fn.</li><li><em>predict_fn</em>(…): Receives data returned by <em>input_fn</em> and then makes a prediction on it using the model returned by <em>model_fn</em>.</li><li><em>output_fn</em>(…): This is called with the return value of <em>predict_fn</em> as input and is used to serialize the predictions back to the client.</li></ul><p>We recommend taking a look at each particular case in <a href="https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-inference-container.html">Adapting Your Own Inference Container</a>.</p><p>This script has access to different SageMaker environment variables that should be used to complete the task. For example, the artifacts generated should be store in the location specified by the SM_MODEL_DIR variable. Please refer to <a href="https://sagemaker.readthedocs.io/en/stable/frameworks/chainer/using_chainer.html#prepare-a-chainer-training-script">Using Chainer with the SageMaker Python SDK</a> and the <a href="https://github.com/aws/SageMaker-containers#list-of-provided-environment-variables-by-SageMaker-containers">official documentation</a> for further details.</p><p><strong>Hyperparameters</strong></p><p>Typically, the training scripts receive hyperparameters that can be used to define the number of epochs, timeout, batch-size, learning-rate, etc.</p><p>For further details please refer to the official SageMaker documentation:</p><ul><li><a href="https://sagemaker.readthedocs.io/en/stable/frameworks/chainer/index.html">Chainer framework on SageMaker</a></li><li><a href="https://sagemaker.readthedocs.io/en/stable/frameworks/chainer/using_chainer.html#train-a-model-with-chainer">Training a model with Chainer</a></li><li><a href="https://docs.aws.amazon.com/sagemaker/latest/dg/frameworks.html">SageMaker available frameworks</a></li></ul><p><strong>Creating an estimator</strong></p><p>Once the training script is done, you use it to create an estimator. In this step you also determine the training instance where the training will take place (count and type, the execution role, both python and framework’s version as well as the training hyperparameters).</p><p><strong>Executing training job</strong></p><p>Finally, with the estimator created the only thing you need to do is start the training job. You can do it by calling the fit method on the estimator.</p><h4>Training a model with a built-in algorithm</h4><p>The advantage of using built-in algorithms is that you do not need to code anything related to the algorithm itself. You can create a training job directly from the AWS console, set up all the configuration needed for our use case and after a couple of minutes we will have a model artifact stored in S3 bucket. Then, you just need to configure the endpoint to make predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pDX_Z80neFXYPpSrNljbZg.png" /></figure><h3>Deploying the model</h3><p>The training job only creates the artifacts for our model. In order to use it , you need to deploy this model to an endpoint. This endpoint is a service running in a hidden EC2 instance. You do not need to worry about setting this server up, only need to choose things like instance type, auto-scaling groups, how many servers do you want, etc. After the deployment is done, you can see the deployed endpoint in the SageMaker console.</p><p>The advantages of using SageMaker for model deployment can be summarized in:</p><ol><li>Host the ML model in an endpoint: You can call this endpoint as you call any other endpoint, you can also call this model from a Lambda function. <em>Note</em>: Amazon hosts the SageMaker model on a physical server that runs 24 hours a day. The server keeps running all the time like EC2, but you don’t configure and manage it, just like Lambda.</li><li>Hosting of multiple models: You can host multiple models in the same endpoint. This is fairly complex, but there are <a href="https://github.com/aws/amazon-sagemaker-examples/tree/master/sagemaker_endpoints/a_b_testing">starter codes in Github by AWS</a>.</li><li>All your logs get easily stored in CloudWatch logs: You do not need to create your own logging pipeline.</li><li>Simple monitoring: You can monitor the load on your machines and scale the machines based on demand.</li></ol><h4>Model endpoint</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hizMGOiqumgBxWtN7_KBgg.png" /></figure><p>Once the model is deployed, you can copy the URL and try calling it using POSTMAN or any similar REST client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JThS0WpnD7859N-q9bjb1w.png" /></figure><p>You can look the status/health of the endpoint with the panel called Monitor provided by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*WfIKJhW0w1QlbrzW5r1Cbg.png" /></figure><p>You can see the logs in Cloud Watch and look if there is any trouble.</p><p>Once you tested that the endpoint is working as you expected, you should add security configurations.</p><h4>Securing the endpoint</h4><p>All SageMaker endpoints are secured behind AWS signature and cannot be accessed freely through the web unless they are exposed, for example, with API Gateway.</p><p>Internally, you can add more granularity to the endpoint by using IAM roles and policies, by using a VPC, or both.</p><p>One way of doing it is by allowing access to certain SageMaker operations (SageMaker:InvokeEndpoint) only from specific Lambda functions. When you create an AWS Lambda function, you must create a special IAM role that has permission to execute the function. This role is called the execution role. When you set up a Lambda function, you must specify the IAM role you created as the corresponding execution role.</p><p>The execution role provides the Lambda function with the credentials it needs to run and to invoke other web services (in our case, SageMaker:InvokeEndpoint). As a result, you don&#39;t need to provide credentials to the Node.js code you write within a Lambda function.</p><p>If it is needed to call the endpoint using a client like Postman, you should include in the headers the params <em>AccessKey</em> and <em>SecretKey</em>.</p><h3>Updating our solution using SageMaker’s models</h3><p>Now we want to use our SageMaker models in the old step function that predicts sentiments as positive or negative. So, we just updated that part in the whole solution, as you can see below, keeping the results going to the next step in the pipeline as we already explained in <a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-ii-97134d553242">Part II</a>.</p><p>When our model in SageMaker cannot determine if sentiment is positive or negative, we tag it as <em>unrecognized</em> and send it to the unrecognized workflow that stores this result in an S3 bucket with the same format needed to retrain the model. After a user classifies each message as positive or negative we can retrain the model with this new dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvXvc8FyF4wB4wUhTpziXw.png" /></figure><h3>Performance comparison</h3><p>After testing the different models, we got the following inference metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*xBkivSZcT2myrJyVsKTIhA.png" /></figure><blockquote>As depicted in the table above, trained models perform better than Lex/Comprehend for our scenario.</blockquote><p>In addition, we executed load testing and found that using trained models has a better performance than the Lex/Comprehend tools used in <a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-ii-97134d553242">Part II</a> (we tested the BlazingText and Chainer model using 2 different types of AWS instances, <em>ml-t2-medium</em> and <em>ml-m4-xlarge</em>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyjZy5c24YvqvxnwoiMe_g.png" /><figcaption>Values show milliseconds</figcaption></figure><p>Lex and Comprehend has a limit of concurrent calls up to 2 for the <em>$LATEST</em> bot, a single instance starts to fail when more than a hundred concurrent requests are made. This could be mitigated by increasing the limit (a call to Amazon is required), or spinning up multiple instances and balancing the request among them (which could be fairly hard to implement).</p><p>Using a single Lex bot caused problems when trying to send more than 100 concurrent request. We assumed that this is due to the <a href="https://docs.aws.amazon.com/lex/latest/dg/gl-limits.html">quotas set for the number of concurrent operations for PostText</a>. We can see this below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fg93Loz8nLGhSaiRQHBlVQ.png" /></figure><h3>Conclusions</h3><p>During the series of articles we saw our platform evolve in many different ways, and as part of it, two different approaches to get sentiment analysis where used. First, we used Lex and Comprehend as it provided an easier approach when considering retraining although more limited — you can easily retrain the bot to detect new <em>intents</em>. Now, with custom models specifically trained for the scenario of our interest, we showed you we more accurate predictions focused on making the platform more robust and reliable in terms of its main goals.</p><p>After performing all the steps required by a typical machine learning pipeline, from data preparation to model deployment, we were able to get a comparison between the old and new approach. As we have seen, using SageMaker to train and deploy a model with built-in algorithms (like BlazingText) provided a more accurate prediction service. Although re-training these models could be a bit more cumbersome, as you must collect enough data and perform the process using the AWS console.</p><p>Cost/Benefit wise, Lex charges per message processed while SageMaker endpoints charge per hour of uptime. While the smallest instance of SageMaker costs around $0.065 per hour and can process as many requests as possible (we saw that it can easily handle 1000), Lex charges $0.0020 per text input. If we have more than 25k messages per month, SageMaker will provide us with a better choice both for cost and performance (this in addition to Lex and Comprehend having scalability issues). So even when using the smallest instances, SageMaker endpoints have good support for at least a thousand simultaneous requests, being both faster and more resilient than the Lex/Comprehend counterpart.</p><p>That’s all folks! Hope that you enjoyed the whole series.</p><p>If you are interested in the technical details of the techniques mentioned in this article, please refer to our <a href="https://github.com/southworks/aws-ml-sagemaker-checkin-app">reference implementation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc828ad6c39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/building-a-mental-health-monitoring-app-with-aws-ai-services-part-iii-bc828ad6c39">Building a ‘Mental Health Monitoring’ App with AWS AI Services — Part III</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Health-Care: A Secure Electronic Medical Record Scenario over Blockchain]]></title>
            <link>https://medium.com/southworks/health-care-a-secure-electronic-medical-record-scenario-over-blockchain-cf4d78f6d30d?source=rss-f4f95113ee79------2</link>
            <guid isPermaLink="false">https://medium.com/p/cf4d78f6d30d</guid>
            <category><![CDATA[hyperledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[healthcare]]></category>
            <category><![CDATA[hyperledger-fabric]]></category>
            <dc:creator><![CDATA[Mauro Krikorian]]></dc:creator>
            <pubDate>Fri, 07 May 2021 13:55:31 GMT</pubDate>
            <atom:updated>2021-05-07T13:55:31.635Z</atom:updated>
            <content:encoded><![CDATA[<p>Nowadays businesses often need a system-of-record platform with auditing capability to keep track of critical data such as internal compliance, regulatory data (e.g., GDPR) or complete history of assets. For this kind of use cases, blockchain technology offers many advantages compared to traditional database systems. Blockchain based systems provide their consumers with trust in the integrity and validity of the data stored thanks to its immutability properties and consensus mechanisms (to learn more about Blockchain and its basic concepts we recommend reading our <a href="https://medium.com/southworks/an-introduction-to-smart-contracts-552b88e4ebc5">Basic Concepts</a> article, and to get acquaintance with <a href="https://medium.com/southworks/short-intro-to-hyperledger-fabric-service-351f8e0040c8">Hyperledger Fabric Concepts</a> that will be heavily used through this article).</p><p>Particularly, and in several scenarios, the HealthCare Industry seems to have issues regarding the cooperation between different organizations in terms of client privacy and sensitive data which is heavily regulated.</p><p>We strongly believe that this case is a perfect fit for Smart Contracts, as they allow all parties involved to agree on the implementation of a system that guarantees that required regulations are followed while information exchange protocols can be agreed on top of this technology. During this article we focus on describing a scenario with a custom solution using blockchain technology.</p><p>Before we get into the thing, I don’t want to skip mentioning the great team of software engineers that contributed and provided feedback into putting this together: Matias Diaz, Luciano Pereira &amp; Emanuel Vecchio— thank you guys, you made it happen!</p><h3>Our Scenario</h3><p>What we will show you ahead consists of a system of record implementation that allows different Medical Provider organizations to store their patient’s Electronic Medical Records<strong>[1]</strong> (EMRs) and share them between each other as needed.</p><p>Each medical provider<strong>[2]</strong> has three type of actors, each one playing a different role in the network:</p><ol><li>Healthcenter: The Medical provider employee in charge of the patient admission process.</li><li>Physician<strong>[3]</strong>: The medical practitioner working for a medical provider. This actor will treat patients.</li><li>Patient: The person that is admitted to a medical provider for a medical treatment or consultation.</li></ol><p>Regarding smart contracts implementation, we conceptually divide them by actors in the network.</p><ol><li>Healthcenter: This will contain the operations needed to create a patient’s EMR and to authorize the sharing of an EMR to a different Medical provider.</li><li>Physician: This actor will be able to access the EMRs that belong to the medical provider they work for. Additionally, they will be able to append additional information to the EMRs about a medical visit.</li><li>Patient: This actor will be able to download his EMR and allow a medical provider to access his EMR stored at a different medical provider.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*GdvPHNm2tQNem6JuTVLVFQ.png" /></figure><h3>Medical Record Workflows</h3><p>As we have already defined the different entities and their roles in our scenario, we now want to detail the 2 workflows we chose to implement for our medical records business. Our record system has two workflows: “<strong>Managing an EMR</strong>” and “<strong>Sharing and EMR</strong>”.</p><h4>Managing an EMR</h4><p>This one is the main workflow. It involves EMR’s creation and edition process. Each entity is in charge of specific operations to create or update an EMR as is showing in the following flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*Ep-YYfa1cocPOJdCTjmmJg.png" /></figure><ol><li>A patient arrives to a Medical Provider for the first time.</li><li>The Medical Provider’s administration signs up the patient and creates their EMR. The Medical Provider becomes the owner of that EMR.</li><li>The patient visits a Medical Provider’s Physician.</li><li>The Physician reads the Patient’s EMR before attending the patient.</li><li>After the consultation, the Physician adds a new medical note to the EMR.</li><li>Finally, the Patient can download their updated EMR.</li></ol><h4>Sharing an EMR</h4><p>This is a workflow between Medical Providers. It allows EMRs to be shared from one Medical Provider to another, after the patient’s authorization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*qzukQujSMgVgx7JB2dv6UQ.png" /></figure><ol><li>A Medical Provider tries to get an EMR belonging to another Medical Provider.</li><li>The Requesting Medical Provider cannot read the external EMR because it does not have permission on that EMR.</li><li>The Medical Provider requests permission on that EMR.</li><li>The owner Medical Provider approves the request.</li><li>The owner Patient approves the request.</li><li>A Medical Provider tries to get the EMR.</li><li>The Requesting Medical Provider reads the external EMR.</li></ol><h3>The Solution — High-level Architecture</h3><h4>Overview</h4><p>We implemented our solution using the <a href="https://www.hyperledger.org/use/fabric">Hyperledger Fabric</a> platform. We think this platform is the most fitting for our scenario since it was originally designed for enterprise use and it focus on permissioned systems.</p><p>For the smart contract development, Hyperledger Fabric offers three different programming languages: Golang, Java, and Node.js. We decided to use Node.js for our reference implementation since it is the most widely used language between the three.</p><p>We also created shell scripts that utilize <a href="https://docs.docker.com/compose/">Docker-Compose</a> and the Hyperledger Fabric binaries and Docker images to easily create and manage a Blockchain network where medical provider organizations can be added as members. These scripts also provide the entry point that allow medical providers to interact with the network and execute its smart contracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*gcJyQsoCpXyN8VMS7_677w.png" /></figure><h4>Managing an EMR</h4><p>EMR managing process consist of one Health Provider organization creating patient’s EMR (at first visit, if does not exist before). Later, physicians from that Medical Provider are allowed to read and append medical entries to patient’s EMR — this process create the history captured within the EMR. That instance of the EMR belongs to the Medical Provider which creates the record and cannot be accessed nor read by any other institution in the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_uGwMKR60oWM5cHbo7Tsng.png" /><figcaption>Manage Patient’s EMR</figcaption></figure><p>The diagram above shows the following:</p><ol><li>A Medical Provider 1 administration employee uses their signed certificate to communicate with the organization’s peer node as an authenticated health center user. They ask the peer to invoke the EMR creation operation installed in the peer’s chaincode. This operation is endorsed by all other Medical Provider peers and in case it is successful, the new EMR gets stored in the peers’ ledgers (although it is only accessible by users belonging to the Medical Provider 1 organization).</li><li>A Medical Provider 1 physician uses their signed certificate to invoke the chaincode operation to add a note to the patient’s EMR. If the operation is successfully endorsed by all Medical Providers, the updated EMR gets stored in the peers’ ledgers.</li><li>A Medical Provider 1 patient uses their signed certificate to invoke the chaincode operation that retrieves their own EMR.</li><li>A Medical Provider 2 physician uses their certificate to communicate with the organization’s peer node and tries to read an EMR from a patient belonging to the Medical Provider 1 organization. The chaincode itself uses the physician’s certificate to determine the organization they belong to and restrict they access to the EMR.</li></ol><h4>Sharing an EMR</h4><p>EMR sharing process consist of one Health Provider organization giving another organization access to read one of their patient’s EMRs (with patient consent). As stated before, organizations own the EMRs, but once shared (for reading purposes as they still control appending operations) they do not control what recipients do with this information. Additionally, when sharing an EMR with another organization in the network, it allows that organization’s physicians to access it.</p><p>Since patient’s EMRs are often updated with information about new medical appointments and treatments, when sharing an EMR between organizations we evaluated two alternatives. The first one was to give full access to the EMR, allowing the recipient to access the latest EMR version and see updates in real time. The second one was to provide the recipient with a current snapshot of the EMR, requiring the process to be repeated if the receiver organization needs new information.</p><p>We decided to go with the second option and share an EMR snapshot since it gives the sharer organization more control over the information being shared, allowing for more privacy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jelHgJV88YWCjtwRlqWxqQ.png" /><figcaption>Patient’s EMR Sharing</figcaption></figure><p>The diagram above shows the following:</p><ol><li>A Medical Provider 1 administration employee uses their signed certificate to invoke a chaincode operation that signals that the organization gives the Medical Provider 2 organization permission to read an EMR belonging to a specific patient (the permissions are stored in the ledger).</li><li>A Medical Provider 1 patient uses their signed certificate to invoke a chaincode operation that signals that they give the Medical Provider 2 organization permission to read their EMR (which is owned by the Medical Provider 1).</li><li>A Medical Provider 2 physician uses their certificate to communicate with the organization’s peer node and tries to read an EMR from a patient belonging to the Medical Provider 1 organization. The chaincode retrieves the permissions from the peer’s ledger. Since both the health center user and the patient user have given the physician’s organization permission to read the EMR, the physician can access it.</li></ol><h4>Network Components &amp; Entities</h4><p>We created the following diagram in order to give you an idea of the high-level picture of the different actors (virtual or human) that interact with each other within the platform (it has all components and entities structured as a hierarchy of relations):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*ShTr_Xk1UWd4C5HnlgYyKw.png" /></figure><h3>Implemented Operations</h3><p>Our solution’s chaincode is composed of different methods or “smart contracts” that manipulate the ledger’s data.</p><p>The Hyperledger Fabric response objects contain a JSON object, together with a status field that follows the HTTP status codes. For more information you can check the <a href="https://hyperledger.github.io/fabric-chaincode-node/release-2.2/api/fabric-shim.Response.html">Fabric SDK documentation</a>.</p><h4>Healthcenter</h4><p>These operations can only be executed by an authenticated healthcenter user:</p><ul><li><em>CreateEmr</em>: Creates a EMR for a specific patient and stores it in the ledger.</li><li><em>CheckIsPatientRegistered</em>: Checks if a patient is already registered in the organization.</li><li><em>AuthorizeEmrReading</em>: Authorizes another Health Provider to read the current version of a specific patient’s EMR.</li></ul><h4>Physician</h4><p>These operations can only be executed by an authenticated physician user:</p><ul><li><em>ReadEmr</em>: Retrieves an EMR by id.</li><li><em>GetEmrByPatientId</em>: Retrieves an EMR by its patient id.</li><li><em>AddEmrNote</em>: Adds a note to an existing EMR.</li><li><em>GetSharedEMR</em>: Retrieves an EMR belonging to a patient from another organization only if the owner organization and the patient have given access permission.</li></ul><h4>Patient</h4><p>These operations can only be executed by an authenticated patient user:</p><ul><li><em>GetOwnEMr</em>: Retrieves the patient’s own EMR. The chaincode gets the patient id from the user’s certificate.</li><li><em>GetMedicalVisitCount</em>: Returns the number of times the patient has visited a specific Health Provider.</li><li><em>ApproveEmrSharing</em>: Authorizes another Health Provider to read the patient’s current EMR version.</li><li><em>RemoveEmrSharing</em>: Removes authorization for a Health Provider to read the patient’s EMR.</li></ul><h3>Summary</h3><p>As we have seen through this article Blockchain technology can be applied to many different scenarios and industries and not only be tied to crypto-currencies (as most of people think).</p><p>Blockchains, as distributed ledger technology, offering transparency to network members as well as immutability of the data being recorded within them are a perfect fit for several domains. In addition, different flavors of them which add different layers of security and privacy of transactions are the perfect fit and way to cover a wide set of use cases and requirements (from business solution &amp; compliance to technical perspective).</p><p>In this case, the Health Industry can leverage this technology to handle sensitive medical data &amp; records of their customers with highest levels of confidentiality, resiliency, flexibility, and scalability, while being compliant with most regulations in the sector. The scenario described in the article is just a glimpse of what it can be achieved with these amazing technologies.</p><p>We hope you liked it! Don’t forget to visit our <a href="https://github.com/southworks/blockchain-hyperledger-healthcare">reference implementation</a> if you are curious into technical details or just want to dig deeper into decisions being made during the construction of this sample.</p><h4>Glossary</h4><p><strong>[1]</strong> Electronic Medical Record: (aka EMR) A digital medical record<strong>[4]</strong> maintained by a specific medical provider.</p><p><strong>[2] </strong>Medical Provider: A medical institution that is responsible for creating and maintaining an Electronic Medical Record.</p><p><strong>[3] </strong>Physician: A health-care professional licensed to practice medicine.</p><p><strong>[4]</strong> Medical Record: Documentation of a single patient’s medical history and care across time within one medical provider’s jurisdiction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cf4d78f6d30d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/southworks/health-care-a-secure-electronic-medical-record-scenario-over-blockchain-cf4d78f6d30d">Health-Care: A Secure Electronic Medical Record Scenario over Blockchain</a> was originally published in <a href="https://medium.com/southworks">SOUTHWORKS</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>