<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Chernenkov on Medium]]></title>
        <description><![CDATA[Stories by Daniel Chernenkov on Medium]]></description>
        <link>https://medium.com/@danielckv?source=rss-2e573e0a0400------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*mvZ7_PysY5N3BtMbXYMkeQ.png</url>
            <title>Stories by Daniel Chernenkov on Medium</title>
            <link>https://medium.com/@danielckv?source=rss-2e573e0a0400------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:23:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danielckv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Imperative: Why LLMs Are Not Intelligent]]></title>
            <link>https://danielckv.medium.com/the-imperative-why-llms-are-not-intelligent-249845f5a7a7?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/249845f5a7a7</guid>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 19:55:35 GMT</pubDate>
            <atom:updated>2025-09-30T19:55:35.053Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*ltwpxnBwE0LhkXfnV1UjMQ.png" /></figure><h3>The Paradox of AI Sophistication</h3><p>The explosive growth of artificial intelligence has been largely driven by the public-facing capabilities of LLMs. <br>These models exhibit a remarkable fluency and coherence in generating human-like text, often giving the impression of an encyclopedic knowledge base and genuine intelligence. <br>However, a deeper, more rigorous analysis reveals a fundamental paradox: the impressive surface-level sophistication of a standalone LLM belies an inherent naiveté. These models are not reasoning agents or true knowledge repositories; they are, at their core, probabilistic text predictors.</p><p>This is my own opinion, an article posits that the pathway to building robust, reliable, and truly sophisticated AI systems does not lie in the pursuit of larger, more complex black-box models. Instead, it resides in the architectural patterns that augment and interface with them.</p><p>By “wrapping” LLMs with external systems — a process critically enabled by hidden vector embeddings — it is possible to transcend their core limitations, providing them with access to real-time information, factual grounding, and verifiable context. This is the paradigm of Retrieval-Augmented Generation (RAG).</p><p>The architectural shift to Retrieval-Augmented Generation (RAG) solved the immediate crises of factual grounding and knowledge freshness. However, true intelligence is more than accurate recall; it is the ability to reason, strategize, and act within a dynamic environment. The next evolution of AI moves beyond passive information synthesis and into the realm of active problem-solving. This requires two fundamental advancements: a simulated world for AI to test its reasoning and a next-generation cognitive architecture that enables it to learn, adapt, and self-correct.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*7rj34TYWzXmXJOBlao17hA.png" /></figure><h4>The Proving Ground: AI Simulation and Digital Worlds</h4><p>The most advanced RAG systems, particularly Agentic RAG, give an AI the ability to use tools. But to what end? An agent that can browse the web or run code is useful, but its potential is truly unlocked when it can operate within a high-fidelity simulation of a real-world system. This is the concept of the AI Proving Ground — a risk-free environment where an AI can test hypotheses, learn from consequences, and develop complex strategies without impacting the real world.</p><h3><strong>Digital Twins as a High-Fidelity Sandbox</strong></h3><p>A Digital Twin is a virtual, dynamic model of a physical object, process, or entire ecosystem.1 Imagine a complete digital replica of your company’s supply chain, a factory floor, or even a customer market. By connecting an AI agent to this Digital Twin, you transform it from a mere data processor into a strategic simulator.<br>The AI can be tasked with complex goals, such as:</p><ul><li><strong>“What is the most effective way to re-route our logistics to minimize disruption if a major port in Southeast Asia closes for two weeks?”</strong></li><li><strong>“Run 10,000 simulations of our new product launch against different competitor response strategies and identify the top three marketing budget allocations.”</strong></li></ul><p>In this sandbox, the AI agent doesn’t just retrieve static documents; it interacts with a living system. It can make a decision (e.g., “increase production at the Mexico facility”), observe the cascading effects on inventory, shipping costs, and delivery times within the simulation, and learn from the outcome. This iterative process of action and observation is how true, experience-based learning occurs.</p><h3>Multi-Agent Systems and Emergent Strategy</h3><p>The next level of simulation involves not one, but multiple AI agents operating within the same digital world. These Multi-Agent Systems (MAS) can be programmed to collaborate or compete, mimicking the complex dynamics of a real market.2 For example, a company could create AI agents representing itself, its top three competitors, and various customer segments.</p><p>The LLM acts as the “brain” for each agent, defining its goals, personality, and decision-making logic based on its unique persona (“You are a risk-averse competitor focused on market share,” or “You are a price-sensitive customer segment”).3 When the simulation runs, these agents interact, negotiate, and strategize independently. The result is not a predictable outcome, but an <strong>emergent strategy</strong> — a complex pattern of behavior that arises organically from the interactions. This allows businesses to discover unforeseen vulnerabilities and opportunities in their plans that no human team could have anticipated.</p><h4>Next Frontier: Blueprints for Next-Gen AI Architectures</h4><p>As the tasks we give AI become more complex, the underlying architecture must also evolve. The “naive” RAG pipeline is a foundational first step, but the next generation of AI solutions will be built on more dynamic, reflective, and specialized frameworks.</p><h4><strong>The Rise of the Specialist: Mixture-of-Experts (MoE) Architectures:</strong></h4><p>The “one model to rule them all” approach is becoming obsolete. Just as a CEO relies on a board of specialists (a CFO, a CTO, a CMO), a next-generation AI system will function as a Mixture-of-Experts. This is an architecture where a primary routing model analyzes an incoming task and delegates it to a smaller, specialized AI agent.</p><ul><li>A query requiring financial data analysis is routed to an agent equipped with Python data science libraries.</li><li>A query about legal compliance is sent to an agent with access to a fine-tuned legal model and a vector database of regulatory documents.</li><li>A creative brainstorming request is handled by a highly fluent, general-purpose generative model.</li></ul><p>This approach is not only more efficient — using smaller, optimized models for specific tasks is cheaper and faster — but also far more effective. It allows each component of the system to excel at its specific function, leading to higher-quality, more reliable outputs.</p><p><strong>Adaptive and Proactive Data-Gathering</strong></p><p>Current RAG systems are reactive; they retrieve information only when a query is made. The next frontier is an AI that proactively and adaptively perceives its information environment. This system would:</p><ul><li><strong>Monitor data streams in real-time:</strong> It would constantly watch for updates in specified knowledge sources (e.g., news feeds, internal sales data, regulatory alerts) and update its vector knowledge base automatically.</li><li><strong>Learn what information is valuable:</strong> Based on user interactions and feedback, the system would learn which data sources are most frequently useful or lead to the best outcomes, prioritizing them in future retrievals.</li><li><strong>Ask clarifying questions:</strong> If a query is ambiguous, the system would initiate a dialogue to understand the user’s intent <em>before</em> beginning the retrieval and generation process, preventing wasted computation and inaccurate answers.</li></ul><p><strong>The Reflective Loop: Building Self-Correcting Systems</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*TyIH2ZqwJb0yjkEZMoymbg.png" /></figure><p>Perhaps the most crucial leap towards true intelligence is building systems capable of self-reflection and correction. Current LLMs generate a response in a single, forward pass. A next-generation architecture will operate in a Reflective Loop:</p><ol><li><strong>Plan:</strong> The AI first deconstructs the problem and forms a multi-step plan to address it.</li><li><strong>Act:</strong> It executes the plan, retrieving information, using tools, and generating an initial draft of the response.</li><li><strong>Reflect &amp; Critique:</strong> The AI then pauses and critically reviews its own work. It asks questions like: “Does this answer fully address the user’s prompt? Have I cited the strongest possible sources? Is there any internal contradiction in my logic? What are the weaknesses of this argument?”</li><li><strong>Refine:</strong> Based on its self-critique, the AI refines its answer, performs additional retrieval steps if necessary, and generates a final, improved output.</li></ol><p>This iterative process, much like how a human expert works, dramatically increases the quality, accuracy, and nuance of the final output.8 It moves the AI from being a fast but sometimes-flawed generator to a deliberate and reliable reasoning partner. By integrating these simulation environments and next-generation architectures, we finally move beyond the illusion of intelligence and begin to build systems that demonstrate the real thing.</p><p><em>Ultimately</em>, an LLM is a powerful tool — a magnificent engine for language — but it is not a self-contained vehicle for intelligence. Its true power is unlocked only when it is “wrapped with different interfaces” and integrated into a system that provides it with the verifiable context and external knowledge it was never designed to have. The true sophistication of AI lies not in the core model, but in the systems and interfaces that enable its full potential.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=249845f5a7a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Genesis of Self-Learning AI: How Vector Insights are Powering Truly Autonomous Agents]]></title>
            <link>https://danielckv.medium.com/the-genesis-of-self-learning-ai-how-vector-insights-are-powering-truly-autonomous-agents-568a99682423?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/568a99682423</guid>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Wed, 07 May 2025 07:05:32 GMT</pubDate>
            <atom:updated>2025-05-07T07:19:25.037Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bg8h1I0ENshzcKH54qKOqw.jpeg" /><figcaption>The <strong>Self-Generative AI Agents brain</strong></figcaption></figure><p>The promise of Artificial Intelligence has always been a system that can not only execute tasks but also learn, adapt, and even innovate.</p><blockquote>AI agents have excelled at specific, pre-defined operations.</blockquote><p>Give them a clear set of tools and explicit instructions, and they perform admirably. But what happens when the environment changes, a new tool emerges, or a problem requires a novel combination of existing capabilities? Traditionally, this meant human intervention — reprogramming, retraining, re-engineering.</p><p>I believe we’re on the cusp of a significant shift. I’ve been deeply immersed in developing a framework for <strong>Self-Generative AI Agents</strong> — systems that don’t just follow orders, but actively learn <em>how</em> to act, <em>what</em> tools to use, and even <em>generate new code</em> to achieve their goals. The secret sauce? Harnessing the power of <strong>Vector Insights</strong> drawn from an ever-evolving knowledge base.</p><p>This isn’t just about smarter prompting; it’s about building an AI that continuously improves its operational capabilities through experience.</p><h4><strong>The Bottleneck of Static Agency</strong></h4><p>Traditional AI agents, even those powered by LLMs, often operate with a fixed understanding of their capabilities. They might have a list of available tools (APIs, functions) and use an LLM to decide which one to call based on a prompt. While powerful, this has limitations:</p><ol><li><strong>Brittleness:</strong> If a tool’s API changes or a new, more efficient tool becomes available, the agent often needs manual updating.</li><li><strong>Limited Discovery:</strong> They typically can’t discover novel ways to combine tools or write new helper functions on the fly unless explicitly trained or programmed for very specific scenarios.</li><li><strong>Static Knowledge:</strong> Their understanding of “how to do things” is largely frozen at the time of their last update or training.</li></ol><p><strong><em>The question I posed to myself was: how can we build agents that transcend these limitations and exhibit a more organic form of learning and adaptation?</em></strong></p><h4><strong>The Core Innovation: Vector-Powered Cognition</strong></h4><p>The breakthrough lies in moving beyond static tool lists and towards a dynamic, “living” knowledge base, queried through <strong>vector embeddings</strong>.</p><p>Imagine a specialized database — not just storing structured data, but a rich tapestry of:</p><ul><li><strong>Code Snippets &amp; Libraries:</strong> Examples of efficient, reusable code for various tasks.</li><li><strong>API Documentation &amp; SDK Usage:</strong> Detailed information on how to interact with external tools and services.</li><li><strong>Past Action Sequences:</strong> Records of successful (and importantly, failed) attempts to achieve goals, including the context, actions taken, and outcomes.</li><li><strong>User Feedback &amp; Goal Clarifications:</strong> How humans have described tasks or corrected previous agent attempts.</li><li><strong>Tool Interaction Logs:</strong> Raw data on how tools behave, their outputs, and error messages.</li></ul><p>Each piece of this information is converted into a high-dimensional vector embedding.</p><p>These embeddings capture the semantic meaning and context. When an agent faces a new task, instead of just looking up a predefined action, it queries this vector database. The system retrieves the most semantically similar and relevant pieces of knowledge — these are the <strong>“Vector Insights.”</strong> These insights become the raw material for decision-making and action generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KvHbo8x7RcPTf3NjhJlEgw.png" /></figure><h4><strong>Introducing the “</strong>Agentic <strong>AI </strong>Swarm<strong> Framework”</strong></h4><p>To bring this concept to life, I’ve been developing an internal architecture I call the <strong>“</strong>AASF<strong>”</strong> <br>This framework isn’t a single monolithic agent but a collaborative ecosystem of specialized agents:</p><p><strong>The Orchestrator (or Planner Agent):</strong></p><ol><li><strong>Role:</strong> The central nervous system. It receives the high-level user goal (e.g., “Analyze last month’s sales data and identify underperforming product categories”).</li><li><strong>Action:</strong> It performs an initial query against the Vector Database to gather high-level strategies, potential tools, or relevant past experiences. It then breaks down the goal into manageable sub-tasks.</li></ol><p><strong>The Action Generator Agent:</strong></p><ol><li><strong>Role:</strong> The creative engine. This agent takes a sub-task from the Orchestrator and the initial Vector Insights.</li><li><strong>Action:</strong> It queries the Vector Database for more specific, actionable insights (e.g., “find Python code for connecting to Salesforce API,” “retrieve examples of data aggregation for sales analysis”). Crucially, it can use these insights to:</li></ol><p><strong>Select and parameterize existing tools/APIs.</strong></p><ul><li><strong>Generate new code snippets</strong> (e.g., a Python script) by synthesizing relevant code examples from the database, potentially with the help of an LLM fine-tuned for code generation based on these insights.</li><li><strong>Formulate complex queries</strong> for other data systems.</li><li>This agent is where the “self-generative” aspect truly shines, as it can create novel solutions not explicitly pre-programmed.</li></ul><p><strong>Executor Agents (e.g., Coder Agent, Tool User Agent):</strong></p><ul><li><strong>Role:</strong> The hands and feet. These are specialized agents responsible for carrying out the concrete actions proposed by the Action Generator.</li><li><strong>Coder Agent:</strong> Executes generated code, handles debugging (with feedback), and manages code environments.</li><li><strong>Tool User Agent:</strong> Interacts with external APIs, services, or even command-line interfaces, managing authentication and parsing responses.</li></ul><p><strong>The Feedback &amp; Learning Loop:</strong></p><ul><li><strong>Role:</strong> The mechanism for continuous improvement.</li><li><strong>Action:</strong> After an Executor Agent attempts an action, the outcome (success, failure, output data, error messages, performance metrics) is captured. This outcome, along with the context and the action taken, is processed, vectorized, and <strong>fed back into the Vector Database.</strong></li><li>This creates a virtuous cycle: successful strategies are reinforced, failures provide learning opportunities (e.g., “this API call structure failed, but this similar one succeeded”), and the system’s knowledge base becomes richer and more nuanced over time.</li></ul><h4><strong>Agent-to-Agent (A2A) Communication: The Symphony of Collaboration</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IE2PL0C_b-H23pUVQaKMrw.png" /></figure><p>A key design principle within the Framework is robust <strong>Agent-to-Agent (A2A) communication</strong>. Agents don’t operate in isolation. The Orchestrator delegates to the Action Generator, which might consult the Vector Database multiple times. The Action Generator then passes executable plans to the Coder or Tool User Agents. If a Coder Agent encounters an issue executing a script, it might send feedback to the Action Generator, which could then re-query the Vector Database for alternative solutions or try to refine the code. This internal dialogue and delegation make the system more resilient and adaptable.</p><p><strong>A Glimpse into the Workflow: An Example</strong></p><p>Let’s say the goal is: “Summarize recent customer reviews for Product X and identify key sentiment trends, then draft an email to the product team.”</p><ol><li><strong>Orchestrator:</strong> Receives goal. Queries Vector DB for “customer review analysis,” “sentiment analysis tools,” “email drafting.” Breaks it down:</li></ol><ul><li>Fetch reviews for Product X.</li><li>Perform sentiment analysis.</li><li>Aggregate trends.</li><li>Draft email.</li></ul><ol><li><strong>Action Generator (for “Fetch reviews”):</strong> Queries Vector DB for “how to fetch reviews Product X.” Finds insights related to an internal CRM API. Generates an API call.</li><li><strong>Tool User Agent:</strong> Executes the API call, retrieves reviews. Reports success to Orchestrator.</li><li><strong>Action Generator (for “Perform sentiment analysis”):</strong> Queries Vector DB for “sentiment analysis Python code,” “using sentiment library X with text data.” Finds relevant code snippets and library documentation. Generates a Python script.</li><li><strong>Coder Agent:</strong> Executes the Python script on the reviews. Reports results (sentiments, trends) to Orchestrator.</li><li><strong>Action Generator (for “Draft email”):</strong> Queries Vector DB for “email templates for product teams,” “summarizing sentiment data in emails.” Finds relevant examples. Generates a draft email incorporating the identified trends.</li><li><strong>Feedback Loop:</strong> All steps, their outcomes, and the generated artifacts (code, API calls, email draft) are vectorized and stored, making the system smarter for the next similar task. If the first attempt at sentiment analysis failed, that failure and any error messages would also be stored, helping the Action Generator make a better choice next time.</li></ol><h3><strong>Why This Paradigm Shift Matters</strong></h3><p>The implications of self-generative, learning agents are profound:</p><ul><li><strong>True Adaptability:</strong> Agents can learn to use new tools or adapt to changes in existing ones simply by ingesting their documentation or observing successful interactions.</li><li><strong>Hyper-Automation of Complex Workflows:</strong> Tasks that currently require intricate human oversight or complex, brittle scripts can be automated more robustly.</li><li><strong>Emergent Capabilities:</strong> The system can potentially discover novel solutions or efficient code patterns that weren’t explicitly designed by humans.</li><li><strong>Reduced Development Overhead:</strong> Instead of meticulously programming every possible action, developers focus on curating the knowledge base and refining the learning mechanisms.</li><li><strong>Personalized &amp; Evolving Assistance:</strong> Agents can adapt to individual user preferences and learn from their specific feedback and data.</li></ul><h3><strong>The Road Ahead: Challenges and Opportunities</strong></h3><p>This is an exciting frontier, but it’s not without its challenges:</p><ul><li><strong>Scalability:</strong> Managing and efficiently querying massive vector databases.</li><li><strong>Explainability &amp; Debuggability:</strong> Understanding <em>why</em> an agent made a particular decision can be complex.</li><li><strong>Safety and Control:</strong> Ensuring agents operate within ethical and safe boundaries, especially when generating code or interacting with critical systems.</li><li><strong>Quality of Knowledge:</strong> The “garbage in, garbage out” principle applies. Curating a high-quality, relevant knowledge base is crucial.</li><li><strong>Refinement of Feedback:</strong> Designing effective mechanisms to translate raw outcomes into useful, vectorized learning signals.</li></ul><p>Despite these challenges, the potential is immense.</p><p>We’re moving towards AI systems that are not just tools, but dynamic collaborators capable of genuine learning and autonomous problem-solving.</p><p>The “SwarmAI Agentic Framework” is one exploration of this future, and I’m thrilled to be part of this journey into creating more intelligent, adaptive, and ultimately, more helpful AI.</p><p><strong>What are your thoughts? How do you see self-learning agents transforming industries? I’d love to hear your perspectives in the comments below!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=568a99682423" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking Your Local LLMs: A Deep Dive into MCP On-Premise]]></title>
            <link>https://danielckv.medium.com/unlocking-your-local-llms-a-deep-dive-into-mcp-on-premise-f8def899520c?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8def899520c</guid>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Sat, 12 Apr 2025 23:19:17 GMT</pubDate>
            <atom:updated>2025-04-13T02:49:18.038Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bY_FXHydrZGWmquDhVaUnQ.png" /></figure><h3>Bridging the Gap Between LLMs and the Real World</h3><p>LLMs have demonstrated remarkable capabilities in understanding and generating human language. However, their inherent design often isolates them from the dynamic, real-time information and specific tools necessary to solve complex, real-world problems effectively. Standard LLMs lack direct access to current data streams, private knowledge repositories, or the ability to interact with external APIs and applications. Historically, bridging this gap required developing bespoke, often proprietary integrations, leading to fragmented ecosystems and significant development overhead.</p><p>To address this fundamental challenge, the Model Context Protocol (MCP) has emerged as a crucial open standard.</p><p>Developed initially by Anthropic and subsequently open-sourced to encourage broad adoption, MCP aims to standardize how applications provide essential context — data, tools, and capabilities — to LLMs.</p><p>It functions as a universal connector, much like a USB-C port for LLM context, facilitating seamless communication between AI models and the external world.</p><p>Simultaneously, the landscape of AI deployment is shifting. The rise of local AI execution frameworks, prominently OLLAMA, empowers users to run powerful LLMs directly on their own hardware. This approach offers significant advantages in terms of data privacy, security, cost control, and the ability to utilize custom or fine-tuned models. This trend brings a critical question to the forefront: How can these increasingly capable local models be connected, in a standardized and efficient manner, to the diverse external context they need to operate effectively?</p><p>This report provides an expert-level deep dive into the Model Context Protocol. It explores MCP’s architecture, operational workflow, and its practical integration with local OLLAMA instances. Furthermore, it examines the benefits and potential use cases of this combination, discusses the conceptual simplification of MCP server creation, analyzes current challenges, and considers the future trajectory of this important standard.</p><h3>Demystifying the Model Context Protocol (MCP)</h3><h4>Core Purpose &amp; Vision</h4><p>The fundamental objective of the Model Context Protocol (MCP) is to enable seamless and, critically, <em>standardized</em> integration between applications utilizing LLMs (referred to as Hosts) and the external data sources or functional tools these models need. The emphasis on standardization is paramount. By defining a common language and interaction pattern, MCP aims to dismantle the silos created by custom integrations, fostering interoperability across different LLM applications and context providers. This standardization directly combats vendor lock-in, allowing developers greater flexibility in choosing both LLM providers and context sources.</p><h4>Architecture: The Client-Server Model</h4><p>MCP employs a classic client-server architecture to facilitate communication. The protocol relies on(<a href="https://www.jsonrpc.org/">https://www.jsonrpc.org/</a>) 2.0 messages for structuring requests and responses exchanged between the participating entities. These connections are stateful, meaning context is maintained throughout an interaction session.</p><p>Communication primarily occurs via two transport models 6:</p><ol><li>STDIO (Standard Input/Output): Communication happens through the standard input and output streams of processes. This model is particularly well-suited for local integrations where the client and server run on the same machine.</li><li>SSE (Server-Sent Events): This model uses HTTP-based Server-Sent Events, enabling communication over networks, suitable for scenarios where the client and server are distributed.</li></ol><h4>Key Players in the Ecosystem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EpioH4qKyOVA_fE3kXfnMw.png" /></figure><p><strong>The MCP ecosystem involves three primary roles:</strong></p><ul><li>MCP Hosts: These are the applications that leverage LLMs and require access to external context or capabilities via MCP. Examples include Integrated Development Environments (IDEs) like Cursor or VS Code, dedicated AI chat interfaces such as Claude Desktop, AI agent frameworks like Semantic Kernel, or any custom-built application needing to enrich its LLM interactions.</li><li>MCP Clients: Residing within the Host application, MCP Clients are responsible for implementing the protocol specifics. They establish and manage the one-to-one, stateful connection with an MCP Server.4 Clients can be implemented using official SDKs (available in TypeScript, Python, Java, etc. 3), dedicated libraries like mcp-go or McpDotNet.Extensions.AI, or through integrations within larger AI frameworks such as Google&#39;s Genkit, LangChain, or Microsoft&#39;s Semantic Kernel.</li><li>MCP Servers: These are typically lightweight programs designed to expose specific functionalities or data sources through the standardized MCP interface.3 An MCP server acts as a gateway or adapter, translating between the MCP protocol and the underlying resource, which could be a local data source (like a file system or database) or a remote service (like a web API).4 A growing number of pre-built and community-maintained servers exist for various functions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g7j7W0iIW4LtLc1xx5PL8Q.png" /></figure><h4>The Interaction Flow (Request -&gt; Context -&gt; Response)</h4><p>A typical interaction within the MCP framework follows these steps:</p><ol><li>Request Initiation: The process begins when an MCP Host application, via its integrated MCP Client, needs external context or needs to perform an action. It formulates and sends a JSON-RPC request to a specific, connected MCP Server.</li><li>Negotiation (Capability Discovery): Upon establishing a connection, an important negotiation phase occurs where the client and server exchange information about their capabilities and supported protocol versions.This allows the client (and thus the Host/LLM) to dynamically discover what features (Resources, Prompts, Tools) the server offers.</li><li>Server Processing: The target MCP Server receives the request. It parses the JSON-RPC message, identifies the requested operation (e.g., retrieve a resource, execute a tool), and interacts with its designated backend system. This could involve reading local files, querying a database 6, calling a third-party API (like GitHub or Brave Search), executing a script, or interacting with a local service like the OLLAMA API.</li><li>Response Formulation: Once the server has retrieved the necessary data or completed the requested action, it packages the result (or an error) into a structured JSON-RPC response, adhering to the MCP specification.</li><li>Return to Host/LLM: The response is sent back from the Server to the Client. The Client relays this information to the Host application. The Host can then use the provided context (data, tool execution results) to inform the LLM’s reasoning process or present the information directly to the user.</li></ol><h4>Core Capabilities (Server Features): What Servers Offer</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTRzA8clc0XF3efuy_GARA.png" /></figure><p>MCP servers can expose several distinct types of capabilities to clients:</p><ul><li>Resources: These represent contextual information or data that can be supplied to the LLM or user. This could include the content of local files, database records, or specific data structures relevant to the server’s domain.</li><li>Prompts: These are pre-defined, templated messages or structured workflows designed to guide user interaction with the server’s services. They can simplify common tasks or encapsulate complex multi-step interactions.</li><li>Tools: Perhaps the most powerful feature, Tools represent executable functions that the <em>LLM itself</em> can choose to invoke.<br>When an LLM determines it needs to perform an action (like searching the web, querying a database, sending an email, or manipulating files), it can request the execution of a specific tool offered by an MCP server.2 This mechanism provides a standardized way for LLMs to interact with external systems and take actions, moving beyond simple text generation. This is conceptually similar to function calling in various LLM APIs but standardized under the MCP umbrella.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6GWLCEoCSX9wDZZ9lc7Ow.png" /></figure><h3>Client Features and Supporting Utilities</h3><p>While servers primarily offer capabilities, clients can also possess features. Notably, the Sampling feature allows for server-initiated agentic behaviors, potentially enabling more complex, recursive interactions where the server can trigger further LLM processing.</p><p>Beyond these core features, the MCP specification thoughtfully includes definitions for essential supporting utilities needed for building robust applications. These include mechanisms for Configuration, standardized Progress Tracking for long-running operations, Cancellation support, structured Error Reporting, and guidelines for Logging.</p><p>The inclusion of these elements underscores that MCP is designed not merely as a communication channel but as a comprehensive framework. It provides the necessary building blocks — negotiation, distinct feature types like Tools, operational utilities, and security considerations — to construct reliable, manageable, and sophisticated context-aware AI systems and complex agentic behaviors.</p><h3>Security &amp; Trust</h3><p>Recognizing the power inherent in allowing AI models to access data and execute actions, the MCP specification places significant emphasis on security and trust principles. It outlines guidelines covering user consent and control, data privacy, tool safety (preventing misuse), and controls for LLM sampling behavior. Implementers are strongly advised to build robust user consent flows, provide clear documentation about server capabilities, implement appropriate access controls, adhere to general security best practices, and carefully consider the privacy implications of their server designs. This focus is crucial given the potential risks associated with granting LLMs access to sensitive systems or the ability to perform actions.</p><h3>Powering Local &amp; Private AI: MCP Meets OLLAMA</h3><h4>The Strategic Synergy</h4><p>The combination of the Model Context Protocol (MCP) and OLLAMA represents a particularly powerful synergy for the future of AI applications. MCP furnishes the <em>standardized interface</em> for context and tool integration, while OLLAMA delivers the <em>local, private, and customizable LLM execution engine</em>.</p><p>This pairing unlocks several compelling benefits:</p><ul><li>Enhanced Data Privacy: By running LLMs locally with OLLAMA and potentially connecting to local data sources via MCP servers, sensitive data can remain entirely within the user’s infrastructure, addressing critical privacy concerns.</li><li>Full Control: Users retain complete control over the LLM models used, the versions deployed, and the underlying hardware infrastructure.</li><li>Cost Efficiency: Running models locally can potentially reduce reliance on expensive cloud-based LLM APIs.</li><li>Customization: Enables the use of specialized, open-source, or fine-tuned LLMs tailored to specific tasks or domains.</li><li>Flexibility: Avoids vendor lock-in at both the protocol level (thanks to MCP) and the model execution level.</li></ul><h4>Integration Architecture Explained</h4><p>Integrating MCP with OLLAMA typically involves an MCP server acting as an intermediary or bridge. The common interaction flow looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YFLOlW6tVAf6qrNrCV_Wyg.png" /></figure><p>In this setup, the MCP server’s role is crucial. It might directly wrap the OLLAMA REST API, exposing OLLAMA functionalities (like running models, managing them, or using the chat endpoint) as standard MCP tools.</p><p>Alternatively, a more sophisticated MCP bridge might connect to <em>multiple</em> other MCP servers (e.g., for filesystem access, web search, specific APIs) and route tool execution requests either to those servers or to the OLLAMA LLM for generation, managing the overall interaction flow.</p><p>Several specific implementations facilitate this integration:</p><ul><li>@NightTrek/Ollama-mcp: A dedicated MCP server (built with Node.js) that provides comprehensive access to the OLLAMA API via MCP tools. It allows pulling, running, listing, creating, and managing OLLAMA models, and importantly, offers an OpenAI-compatible chat completion endpoint accessible via MCP.</li><li>ollama-mcp-bridge (by patruff): A TypeScript project acting as a bridge. It connects an OLLAMA-served LLM to a suite of configured MCP servers (e.g., filesystem, Brave search, GitHub, Gmail, Google Drive, Flux image generation, memory). This effectively allows local LLMs to utilize a range of tools similar to those available in cloud platforms like Claude, but entirely locally.</li><li>mcp-cli / mcphost: Command-line tools that can act as MCP hosts. They can be configured to use OLLAMA as the backend LLM provider, allowing users to interact with various MCP servers using a local model.</li><li>Dolphin MCP: A Python library and CLI tool designed to connect to multiple MCP servers simultaneously and make their tools accessible to various LLM backends, including OLLAMA.</li><li>Framework Integrations: Major AI development frameworks are incorporating MCP support. Microsoft’s Semantic Kernel has examples of connecting to MCP servers and converting MCP tools into Kernel functions, which could then be used with an OLLAMA backend. Google’s Genkit, LangChain, Dify, and others also offer potential pathways for combining OLLAMA and MCP.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n_PlYwzJ8beK-5y5ZG3lag.png" /></figure><h3>Getting Started (Conceptual Steps)</h3><p>Setting up an MCP-OLLAMA integration involves these general steps:</p><ol><li>Install and Run OLLAMA: Download the OLLAMA application for your operating system. Use the ollama pull &lt;model_name&gt; command to download the desired LLMs (e.g., llama3.2, qwen2.5:7b, phi4-mini). Ensure the OLLAMA API server (usually on http://localhost:11434) is running and accessible. If running OLLAMA in Docker or on a separate machine from the MCP components, you may need to configure the OLLAMA_HOST environment variable to allow network access (e.g., OLLAMA_HOST=&quot;0.0.0.0&quot;).</li><li>Set Up MCP Server(s): Choose an integration approach:</li></ol><ul><li>Dedicated OLLAMA Server: Install and configure a server like @NightTrek/Ollama-mcp. This typically involves installing Node.js, cloning the repository or using npm/yarn, and potentially configuring the path to your OLLAMA API endpoint in a configuration file or environment variable.</li><li>Bridge: Install a bridge like ollama-mcp-bridge. This requires installing the bridge itself and <em>also</em> installing and configuring the individual MCP servers it will connect to (e.g., @modelcontextprotocol/server-filesystem, @modelcontextprotocol/server-brave-search). Configuration usually involves editing a JSON file (like bridge_config.json) to define the paths/commands for each server and specify the OLLAMA model to use.</li><li>Generic Servers + OLLAMA Client: Install standard MCP servers (e.g., for SQLite database access 6, filesystem operations, or making web requests via curl). Then, use an MCP client or host application configured to use OLLAMA as its LLM backend.</li></ul><p>3. Configure MCP Client/Host: Set up your chosen Host application (e.g., Claude Desktop, a custom script using an MCP SDK, mcp-cli, Semantic Kernel) to connect to the MCP server(s) established in the previous step. This often involves modifying a configuration file (like claude_desktop_config.json or bridge_config.json ) or providing command-line arguments to specify the server&#39;s command or network address (URL for SSE, command/args for STDIO).2 If the client/host supports multiple LLM backends, ensure it&#39;s configured to use OLLAMA.</p><h3>Example Workflow: Local LLM Using an MCP Tool</h3><p>Consider a scenario where a user interacts with a development assistant application running the qwen2.5:7b-instruct model locally via OLLAMA. The application is connected to a Filesystem MCP server via an MCP bridge. The user asks: <em>&quot;Create a new directory named &#39;project-alpha&#39; in my workspace.&quot;</em></p><p>The workflow would proceed as follows:</p><ol><li>Host Request: The development assistant (Host) receives the user’s request.</li><li>LLM Tool Identification: The Host (potentially via the bridge) sends the request to the OLLAMA LLM (qwen2.5). The LLM analyzes the request and determines that the &quot;create directory&quot; action is needed. It identifies the appropriate tool provided by the connected Filesystem MCP server (e.g., a tool named createDirectory or similar).</li><li>MCP Tool Call Formulation: The bridge or MCP client formats a JSON-RPC request to invoke the identified tool on the Filesystem MCP server. This request would include the tool name (createDirectory) and the necessary arguments (e.g., {&quot;path&quot;: &quot;workspace/project-alpha&quot;}).</li><li>Request to Server: The formatted request is sent to the running Filesystem MCP Server.</li><li>Server Action: The Filesystem MCP Server receives the request, validates it, and executes the underlying filesystem command to create the directory project-alpha within its allowed workspace path.</li><li>Server Response: The Filesystem MCP Server sends back a JSON-RPC response indicating the success (or failure) of the operation.</li><li>Result to LLM/Host: The MCP client/bridge receives the response. It might pass the success confirmation back to the OLLAMA LLM, or directly inform the Host application.</li><li>Final User Response: The Host application informs the user that the directory ‘project-alpha’ has been successfully created.</li></ol><p>Invoking such tools programmatically often involves using functions provided by the MCP client library or bridge, abstracting the underlying JSON-RPC communication. For example, code might look conceptually similar to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkshEzlZym0HJADfy6TX2A.png" /></figure><h4>The Critical Role of LLM Tool-Using Capabilities</h4><p>It is crucial to understand that the effectiveness of any MCP integration, particularly one involving OLLAMA, is not solely determined by the protocol implementation or the server setup. It heavily depends on the inherent capabilities of the chosen local LLM.</p><p>MCP defines the <em>standard</em> for how tools are described and invoked, but the <em>decision</em> to use a tool, selecting the <em>correct</em> tool for the task, and formulating the <em>appropriate arguments</em> based on the user’s request, primarily falls to the LLM itself.</p><p>Different LLMs possess varying degrees of proficiency in tool use or function calling.</p><p>Models specifically trained or fine-tuned for these tasks (like Llama 3.2 2, Qwen2.5, or phi4-mini adapted for function calling) are more likely to successfully leverage MCP tools.</p><p>Smaller or less capable models might struggle to reliably infer the correct tool or parameters from ambiguous natural language prompts.</p><p>In such cases, users might need to be more explicit in their prompts, potentially even mentioning the desired tool name, to guide the LLM effectively.</p><p>OLLAMA itself provides underlying support for tool usage that MCP bridges can leverage, but the quality of that usage hinges on the model loaded into OLLAMA.</p><p>Therefore, selecting an OLLAMA model with strong tool-calling abilities is essential for fully realizing the potential of MCP’s “Tool” feature in a local environment.</p><h4>Simplifying Development: The “MCP Server Generator” Concept</h4><p>While the MCP specification and available SDKs provide the foundation for building MCP servers, the process still demands a degree of development effort. Developers need to understand the JSON-RPC protocol, handle negotiation, structure tool definitions correctly, and manage the server lifecycle.</p><p>Lowering this barrier to entry is crucial for encouraging wider adoption and fostering a richer ecosystem of available MCP servers, particularly for developers who simply want to expose an existing function, script, or API endpoint via the MCP standard without becoming protocol experts.</p><p>Addressing the user’s query about simplifying server generation, one can conceptualize a tool — perhaps a command-line utility or a simple graphical interface — designed specifically to abstract away the boilerplate code involved in creating a basic MCP server.</p><p><strong>Such a conceptual “MCP Server Generator” might operate as follows:</strong></p><ul><li>Input: The user would provide the core details of the tool or capability they wish to expose:</li><li>A unique name for the tool (e.g., get_weather, run_sql_query).</li><li>A clear description of the tool’s purpose and functionality (this is crucial for the LLM to understand when to use it).</li><li>Definitions of required input parameters (name, data type, description).</li><li>A definition of the expected output structure or data type.</li><li>The actual implementation logic: This could be a reference to a Python function, a shell command template, configuration details for calling an external API (URL, method, authentication), or similar executable logic.</li><li>The desired transport mechanism (STDIO for local use or SSE for network access).</li><li>Process: The generator tool would take these inputs and leverage the official MCP SDKs (e.g., in TypeScript or Python 3) to automatically generate the necessary server code. This generated code would handle:</li><li>Setting up the chosen transport listener (STDIO or SSE).</li><li>Implementing the JSON-RPC message handling.</li><li>Managing the MCP initialization and capability negotiation handshake.</li><li>Correctly formatting the tool definition based on user input, making it discoverable by MCP clients.</li><li>Wrapping the user-provided execution logic within the MCP request/response cycle.</li><li>Implementing basic error handling and reporting as defined by the protocol.</li><li>Output: The tool would produce a ready-to-run MCP server package. This could be a standalone script (e.g., server.js, server.py), potentially including necessary dependencies, or even a Dockerfile for easy containerization and deployment.</li></ul><p>The primary benefit of such a tool would be a significant reduction in the development time and complexity required to create simple MCP servers. It would empower more developers to contribute to the MCP ecosystem by easily wrapping existing utilities and data sources, encouraging experimentation and accelerating the availability of diverse, standardized context providers for LLMs.</p><h4>Why MCP Matters: Benefits and Use Cases Revisited</h4><p>MCP offers compelling advantages for developers building the next generation of AI-powered applications. Its core benefits stem from its focus on standardization and interoperability:</p><ul><li>Standardization: MCP acts as that “USB-C” port for LLM context, providing a uniform interface between LLM applications and external capabilities. This drastically reduces the complexity associated with building and maintaining numerous custom integrations. A single database vendor, for instance, could implement one MCP server usable by many different MCP-compliant clients.</li><li>Flexibility &amp; Interoperability: Developers gain the freedom to easily switch between different LLM providers (e.g., Anthropic, OpenAI, or local models via OLLAMA) or swap out different MCP servers providing similar functionality, often without requiring significant changes to the host application code. Adopting MCP allows applications to join and benefit from a growing ecosystem of interoperable tools and data sources.</li><li>Enhanced Security &amp; Privacy: MCP facilitates architectures where sensitive data can remain within a user’s or organization’s infrastructure, especially when paired with local execution engines like OLLAMA. The protocol specification itself incorporates security considerations and guidelines, promoting safer integration practices.</li><li>Enabling Sophistication: By providing standardized access to external tools and data, MCP is a key enabler for building more complex and capable AI systems. It facilitates the development of sophisticated agents that can perform multi-step tasks, execute intricate workflows involving external actions, and leverage dynamic context. It also supports the creation and reuse of prompt templates and workflows.</li></ul><p>These benefits translate into a wide range of practical use cases, particularly powerful when leveraging local OLLAMA models:</p><ul><li>Hyper-Contextual Development Environments: IDEs like VS Code, Cursor, or Roo Code can use MCP to connect local OLLAMA models to the user’s workspace. This allows the LLM to access local file systems, understand the codebase, interact with linters or debuggers, and provide highly relevant coding assistance, all while keeping the code private.</li><li>Private Enterprise Knowledge Assistants: Organizations can build internal assistants powered by OLLAMA models running on-premises. Custom MCP servers can provide secure, standardized access to internal databases, proprietary APIs, document repositories, and employee directories, enabling employees to query company knowledge without exposing sensitive data to external services.</li><li>Secure Research &amp; Analysis Tools: Researchers working with sensitive datasets (e.g., medical records, financial data) can use local OLLAMA models for analysis. Secure MCP servers, running within the same trusted environment, can expose controlled access to these datasets or specific analysis functions, ensuring compliance and data confidentiality.</li><li>Local Agentic Systems for Automation: Developers can build autonomous agents using frameworks like mcp-agent or bridges like ollama-mcp-bridge. These agents, powered by local OLLAMA models, can leverage a suite of MCP tools (e.g., web search, file manipulation, email composition, API calls) to perform complex, multi-step tasks like generating reports, managing schedules, or automating workflows, all locally.</li><li>Cross-Platform Tool Provisioning: A software vendor (e.g., providing database services or specialized APIs) can create a single MCP server for their product. This server can then be used by any MCP-compliant client application (IDEs, chatbots, custom tools), simplifying integration for their customers across different platforms.</li></ul><h4>Navigating the Landscape: Challenges and Considerations</h4><p>Despite its significant potential, the adoption and implementation of MCP involve several challenges and considerations:</p><ul><li>Ecosystem Maturity: While the MCP ecosystem is actively growing, with an increasing number of clients and servers becoming available, it is still relatively young. Achieving widespread adoption requires a broader range of robust, well-documented, and easily discoverable servers covering diverse functionalities. The protocol specification itself may continue to evolve, requiring implementers to stay updated.</li><li>Implementation Complexity: Although SDKs exist to aid development, building reliable and secure MCP servers requires non-trivial technical expertise. Developers need to correctly handle the protocol nuances, manage state, implement error handling, and ensure efficient interaction with backend resources.<br>Debugging the interactions between Host, Client, Server, and the underlying LLM can also be complex, although tools like the MCP Inspector exist to help. The need for a conceptual “Server Generator” (discussed in Section IV) highlights this existing complexity gap.</li><li>Security Implementation is Crucial: The MCP specification provides security guidelines, but the ultimate responsibility for secure implementation lies with the developers of both clients and servers. Given that MCP servers can potentially access sensitive data or execute arbitrary code, robust security practices are paramount. This includes proper authentication, authorization, input validation (to prevent injection attacks), rate limiting, and careful consideration of the permissions granted to the server process. <br>Poorly secured MCP servers represent a significant potential vulnerability, potentially undermining the privacy benefits sought, especially in local deployments.<br>Security in MCP is therefore a shared responsibility: the protocol provides a framework, but developers must build secure components, and host applications must manage connections and user consent diligently.</li><li>LLM Tool Use Reliability: As previously discussed, the success of MCP’s “Tool” feature heavily relies on the underlying LLM’s ability to reliably determine when to use a tool, select the correct one, and provide appropriate arguments. <br>This can be inconsistent across different models, particularly smaller or less capable ones. <br>Achieving reliable tool use may require careful prompt engineering, providing very explicit instructions, or even fine-tuning models specifically for tool interaction.</li><li>Discovery and Configuration: While MCP standardizes the communication <em>protocol</em>, the mechanisms for discovering available MCP servers and configuring clients to connect to them are less standardized. Currently, configuration often relies on manual editing of JSON files or providing command-line arguments. More streamlined discovery and configuration mechanisms could further simplify adoption.</li></ul><h3>Context-Aware: Conclusion and Call to Action</h3><p>The Model Context Protocol stands as a vital open standard poised to significantly reshape how AI models interact with external data and tools. By championing standardization, MCP fosters interoperability, enhances developer flexibility, and provides a robust framework for building more powerful, secure, and truly context-aware AI applications. Its synergy with local AI execution frameworks like OLLAMA is particularly compelling, offering a pathway to create sophisticated AI capabilities while maintaining data privacy and user control.</p><p>The future of MCP appears promising. Continued growth in adoption is likely, leading to a richer and more diverse ecosystem of clients and servers.</p><p>We can anticipate advancements in developer tooling, potentially including tools that simplify server creation as conceptualized earlier, making the protocol more accessible. Deeper integration of MCP capabilities into mainstream LLM development frameworks and platforms seems probable. Furthermore, an ongoing focus on refining security best practices and ensuring robust implementations will be critical as MCP deployments become more widespread and handle increasingly sensitive operations.</p><p>Professionals involved in AI development, software architecture, and LLM integration are encouraged to explore the Model Context Protocol further. Valuable resources include:</p><ul><li>Official GitHub Repository: <a href="https://github.com/modelcontextprotocol">https://github.com/modelcontextprotocol</a> (Includes SDKs, specification links, sample servers)</li><li>MCP Documentation Website: <a href="https://modelcontextprotocol.io/">https://modelcontextprotocol.io</a> (Provides guides, tutorials, and overviews)</li><li>MCP Specification: <a href="https://spec.modelcontextprotocol.io/">https://spec.modelcontextprotocol.io</a> (Details the protocol architecture, features, and requirements)</li></ul><p>Exploring existing MCP server implementations and investigating the various approaches for integrating MCP with OLLAMA can provide practical insights into its current capabilities and potential applications.</p><p><strong>The development of context-aware AI is a rapidly evolving field.</strong></p><p>Standards like MCP play a crucial role in shaping this evolution towards more open, interoperable, and capable systems. Engaging with the protocol, experimenting with its implementations, and participating in the community discussions will be key to unlocking its full potential.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8def899520c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beyond Normalization: Connecting the Unstructured Web with Identity Queries and AI Agents]]></title>
            <link>https://danielckv.medium.com/beyond-normalization-connecting-the-unstructured-web-with-identity-queries-and-ai-agents-855595b92944?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/855595b92944</guid>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Sat, 14 Dec 2024 20:16:28 GMT</pubDate>
            <atom:updated>2024-12-14T20:21:39.404Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZrSXPyqq1ZM1Ca862nsf0Q.jpeg" /></figure><p>The internet is a sprawling, chaotic mess of information. We have structured data neatly organized in databases, and then we have the vast majority of the web: unstructured data in the form of images, text documents, and videos. Traditional search engines rely heavily on keywords and indexing to make sense of this chaos, but what if we could understand the underlying <em>identity</em> of data, regardless of its format or structure?</p><p>This is the driving force behind my exploration into <strong>Identity Queries</strong>. Instead of relying on surface-level similarities, this approach uses AI to understand the core meaning and relationships within data, allowing us to connect seemingly disparate pieces of information in a way that was previously impossible. Imagine a system that can seamlessly link the text extracted from an image using OCR to a corresponding entry in a CSV file, purely based on their shared identity.</p><p><strong>Embeddings: The Foundation of Understanding</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OToiw5iq2Hr_zUMEuFWhw.png" /></figure><p>At the heart of this approach lies the concept of <strong>embeddings</strong>. Large language models (LLMs) like GPT-3 can convert text, images, and even code into numerical vectors that capture their semantic meaning. These embeddings allow us to compare and connect data points in a multi-dimensional space, where proximity signifies semantic similarity.</p><pre>import openai<br><br># Assuming you have your OpenAI API key set up<br>openai.api_key = &quot;YOUR_API_KEY&quot;<br><br>def get_embedding(text, model=&quot;text-embedding-ada-002&quot;):<br>  text = text.replace(&quot;\n&quot;, &quot; &quot;)<br>  return openai.Embedding.create(input = [text], model=model)[&#39;data&#39;][0][&#39;embedding&#39;]<br><br># Example usage<br>text1 = &quot;This is a sentence about a cat.&quot;<br>text2 = &quot;The feline was playing with a ball of yarn.&quot;<br>embedding1 = get_embedding(text1)<br>embedding2 = get_embedding(text2)<br><br># Calculate cosine similarity between the embeddings<br>from sklearn.metrics.pairwise import cosine_similarity<br>similarity = cosine_similarity([embedding1], [embedding2])<br><br>print(f&quot;Similarity between sentences: {similarity[0][0]}&quot;)</pre><p>This simple example demonstrates how we can use embeddings to quantify the semantic similarity between two sentences. By applying this technique to various data types, we can create a web of interconnected information that transcends traditional boundaries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kk1reIldOWEYUldp39hAiQ.png" /></figure><h4><strong>Identity Queries: Connecting the Dots</strong></h4><p>Now, let’s imagine we have an image of a receipt with the text <em>“Acme Corp — Invoice #1234”</em> extracted using OCR. In a separate CSV file, we have a record with the same invoice number and details about the transaction. An Identity Query engine can connect these two pieces of information by:</p><ol><li><strong>Generating embeddings:</strong> Creating embeddings for both the OCR text and the CSV entry.</li><li><strong>Identifying key entities:</strong> Using named entity recognition (NER) to extract key information like “Acme Corp” and “Invoice #1234.”</li><li><strong>Mapping relationships:</strong> Understanding the relationship between these entities (e.g., “Acme Corp” issued “Invoice #1234”).</li><li><strong>Connecting based on identity:</strong> Matching the embeddings and entity relationships to establish a connection between the image and the CSV record.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g_XXrZdPAfa_Ya77jFcPjg.png" /></figure><h3><strong>AI Agents: Deepening the Insights</strong></h3><p>But the real power of this approach comes from combining it with <strong>AI agents</strong>. These agents can be assigned to specific entities or topics, and are responsible for:</p><ul><li><strong>Gathering information:</strong> Crawling the web and collecting relevant data from various sources.</li><li><strong>Analyzing and summarizing:</strong> Extracting key insights and summarizing information from the collected data.</li><li><strong>Building knowledge graphs:</strong> Constructing a comprehensive knowledge graph that represents the relationships between entities and concepts.</li><li><strong>Generating insights:</strong> Providing valuable insights and answering complex questions based on the accumulated knowledge.</li></ul><p>For example, an AI agent assigned to “Acme Corp” could gather information about the company from its website, news articles, and social media. It could then analyze this information to understand the company’s financial performance, its products and services, and its reputation. This information can be used to enrich the knowledge graph and provide valuable context to the connected data points.</p><p><strong>Real-World Applications</strong></p><p>The potential applications of this technology are vast:</p><ul><li><strong>Enhanced due diligence:</strong> Connecting disparate pieces of information about individuals and organizations for thorough background checks.</li><li><strong>Investigative journalism:</strong> Uncovering hidden connections and patterns in large datasets to expose corruption or wrongdoing.</li><li><strong>Customer relationship management:</strong> Building a comprehensive view of customers by connecting their interactions across various channels.</li><li><strong>Market research:</strong> Analyzing trends and patterns by connecting data from social media, news articles, and customer reviews.</li></ul><p><strong>Example: Connecting News Articles and Financial Data</strong></p><p>Let’s say we want to analyze the impact of news articles on a company’s stock price. We can use an Identity Query engine to:</p><ol><li><strong>Extract entities:</strong> Identify companies and their stock symbols mentioned in news articles.</li><li><strong>Connect to financial data:</strong> Link these entities to historical stock price data from a CSV file.</li><li><strong>Analyze with AI agents:</strong> Deploy AI agents to analyze the sentiment of news articles and correlate it with stock price fluctuations.</li></ol><p>This could reveal valuable insights into how public perception influences market behavior, helping investors make informed decisions.</p><p><strong>The Future of Information Access</strong></p><p>Identity Queries and AI agents represent a paradigm shift in how we access and understand information. By moving beyond keyword-based search and embracing the underlying identity of data, we can unlock a new level of interconnectedness and insight. This approach has the potential to revolutionize various fields, from business and finance to research and journalism, ultimately leading to a more informed and connected world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=855595b92944" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating Cloud & AI Architecture: A CTO’s Journey Through Data-Driven Environments]]></title>
            <link>https://danielckv.medium.com/navigating-cloud-ai-architecture-a-ctos-journey-through-data-driven-environments-5b417257345b?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/5b417257345b</guid>
            <category><![CDATA[team-management]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Fri, 15 Dec 2023 09:51:30 GMT</pubDate>
            <atom:updated>2023-12-15T09:51:30.365Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k3q6hlzC69GGjtRE" /><figcaption>Photo by <a href="https://unsplash.com/@matthewhenry?utm_source=medium&amp;utm_medium=referral">Matthew Henry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the fast-paced world of technology, aligning cloud architecture with system architecture presents unique challenges and opportunities.</p><p>As the Chief Technology Officer of various companies, my journey through this landscape has offered valuable lessons worth sharing.</p><h3>Understanding the System Context</h3><p>The first step in this alignment is deeply understanding the system’s context, scope, and objectives. This understanding guides every architectural decision, ensuring that the chosen cloud service and deployment models align perfectly with the system’s unique requirements and constraints.</p><h3>AI and Big Data Technologies in Cloud Architecture</h3><p>My experience as a CTO has also involved integrating advanced AI solutions and big data technologies to manage large-scale data loads effectively.</p><p>This aspect of our cloud architecture has been critical in companies like RapidAPI, Odo Security, and DermaDetect, where handling vast amounts of data efficiently is a key requirement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TLQv7RpeJSp_tthl" /></figure><p><strong>AI for Data Management and Analysis</strong></p><p>We utilize AI algorithms for scraping and analyzing large data sets. These AI systems are designed to process and interpret massive amounts of unstructured data, extracting valuable insights and automating complex tasks. This capability is crucial in sectors like healthcare and security, where data-driven decisions can have significant impacts.</p><h3>Adopting Effective Strategies</h3><p>Effective strategies are key to addressing alignment challenges. This involves prioritizing cloud architecture goals and principles in harmony with the system’s vision. Implementing a systematic and iterative process is essential, with stakeholder involvement at each phase.</p><p><strong>Real-World Examples</strong></p><p>Looking at industry giants like Netflix, Spotify, and Airbnb, we see varied approaches to cloud architecture, each tailored to specific business needs. Netflix’s microservices-based architecture on AWS, Spotify’s hybrid approach on GCP and AWS, and Airbnb’s multi-cloud strategy on AWS and GCP illustrate different paths to achieving scalability, flexibility, and performance.</p><p><strong>Leveraging Big Data Technologies</strong></p><p>To support these AI systems, we’ve integrated big data technologies like Apache Cassandra, Hadoop, and Spark. Cassandra offers a highly scalable and reliable database system, ideal for managing large volumes of data across distributed systems. Hadoop’s ecosystem provides a robust framework for data storage and processing, while Spark enhances our data processing capabilities with its in-memory computation feature.</p><p>This combination of AI and big data technologies enables us to handle and analyze data loads from diverse sources efficiently. It ensures our cloud architecture can support high-volume data analysis and real-time decision-making, essential in today’s data-intensive business environments.</p><h3>My Approach: A Synthesis of Diverse Tech Stacks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jmnXRkODR1p3JC-e" /></figure><p>In my role as a CTO across various companies like <a href="https://rapidapi.com/">RapidAPI</a>, <a href="https://en.globes.co.il/en/article-check-point-buys-israeli-cloud-based-co-odo-security-1001343006">Odo Security</a>, and <a href="https://dermadetect.com/">DermaDetect</a>, <a href="https://iturintel.com/">Iturintel</a>, I’ve developed a comprehensive approach to cloud architecture, integrating diverse technologies and methodologies.</p><p>Our architecture leverages Kubernetes for container orchestration, ensuring scalability and resilience. gRPC facilitates efficient microservices communication, while Kafka addresses real-time data processing needs. ZeroMQ is another integral component, providing high-speed, low-latency messaging capabilities.</p><p>Furthermore, the adoption of the Command Query Responsibility Segregation (CQRS) pattern has been pivotal. By separating read and write operations, we’ve achieved enhanced performance and scalability, crucial for handling complex systems across different domains.</p><p>This eclectic mix of technologies reflects a commitment to building robust, adaptable architectures. Whether it’s managing APIs, enhancing security solutions, or developing healthcare applications, this approach ensures our architecture is versatile and ready for future advancements and challenges.</p><p><strong>Continuous Learning and Adaptation</strong></p><p>The ever-evolving landscape of technology necessitates continuous learning and adaptation.</p><p>We regularly update our architecture to incorporate the latest trends and technologies. Encouraging innovation and staying engaged with the tech community has also been crucial for our growth and adaptation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5b417257345b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Kubernetes Operator Pattern]]></title>
            <link>https://danielckv.medium.com/the-kubernetes-operator-pattern-8c89e4f86085?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c89e4f86085</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[cloud-architecture]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 14:00:40 GMT</pubDate>
            <atom:updated>2023-01-13T14:00:40.311Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YDrgGf9WgFW8vANhzOE41Q.png" /></figure><p>Kubernetes Operator is a powerful tool for managing and scaling applications on the Kubernetes platform. It allows developers to easily create, configure, and manage applications within a Kubernetes cluster.</p><p>One of the key benefits of using Kubernetes Operator is its ability to automate tasks that would otherwise have to be done manually. For example, an operator can automatically scale an application based on resource usage, or automatically update the application to the latest version. This can save a lot of time and effort for developers, and also ensures that applications are running smoothly and efficiently.</p><p>Another benefit of Kubernetes Operator is its ability to manage the entire application lifecycle. This includes creating and configuring the application, scaling it up or down as needed, and performing updates and upgrades. This makes it easier for developers to focus on writing code and creating new features, rather than managing the infrastructure.</p><p>Kubernetes Operator is also highly customizable, allowing developers to tailor it to their specific needs. This includes creating custom resources and controllers, as well as integrating with other tools and services.</p><p>Overall, Kubernetes Operator is an incredibly powerful tool for managing and scaling applications on the Kubernetes platform. It can save time and effort for developers, and also ensures that applications are running smoothly and efficiently. It’s an essential tool for anyone working with Kubernetes and looking to take their applications to the next level.</p><pre>package main<br><br>import (<br> &quot;fmt&quot;<br> &quot;time&quot;<br><br> &quot;github.com/operator-framework/operator-sdk/pkg/sdk&quot;<br> sdkVersion &quot;github.com/operator-framework/operator-sdk/version&quot;<br> &quot;github.com/sirupsen/logrus&quot;<br> _ &quot;k8s.io/client-go/plugin/pkg/client/auth/gcp&quot;<br>)<br><br>func main() {<br> sdk.ExposeMetricsPort()<br><br> resource := &quot;example.com/v1alpha1&quot;<br> kind := &quot;Example&quot;<br> namespace, err := sdk.GetWatchNamespace()<br> if err != nil {<br>  logrus.Fatalf(&quot;Failed to get watch namespace: %v&quot;, err)<br> }<br> resyncPeriod := time.Duration(5) * time.Second<br> logrus.Infof(&quot;Watching %s, %s, %s, %d&quot;, resource, kind, namespace, resyncPeriod)<br> sdk.Watch(resource, kind, namespace, resyncPeriod)<br> sdk.Handle(sdk.NewHandler())<br> sdk.Run(nil)<br>}</pre><p>This code sets up a basic operator that watches for changes to a specific resource, in this case “example.com/v1alpha1”, and a specific kind, in this case “Example”. It also sets up a resync period of 5 seconds, so the operator will check for changes every 5 seconds.</p><p>It’s important to note that this is just a basic example and depending on your use case, you might need to add more functionality to your operator. Additionally, this example is written in Go, but you can also write Kubernetes Operators using other languages such as Python.</p><p>I recommend you to check the Kubernetes Operator Framework documentation for more detailed and specific examples: <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">https://kubernetes.io/docs/concepts/extend-kubernetes/operator/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c89e4f86085" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training models with Kubernetes & Kubeflow]]></title>
            <link>https://danielckv.medium.com/training-models-with-kubernetes-kubeflow-4f7e6dfe893c?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f7e6dfe893c</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubeflow]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Tue, 14 Dec 2021 10:01:34 GMT</pubDate>
            <atom:updated>2021-12-23T14:24:28.321Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kubeflow Intro — Machine Learning on Kubernetes</h3><p>If you’ve been following the tech news, it’s a little hard to miss the buzz around Kubernetes and Kubeflow. <br>In this short guide, we are going to discuss some of Kubernetes’s and Kubeflow’s basic concepts, what sort of problems they solve, and how you can use them in your development endeavors.</p><p><a href="https://bigabid.com/">Bigabid</a> based on huge data-sets with multiple structures that grows each day and becomes to be a bottleneck for our training processes of <strong>machine learning</strong> models.<br>As the data grows — as also the time for our models to be built, as part of our solution we have started to look around for new way to distribute our building blocks for Machine Learning models and give us the ability to build highly available, scalable software that’s also easy to manage and update.</p><p><a href="https://www.youtube.com/watch?v=HBxyLnEzyhw">Kubeflow: Machine Learning on Kubernetes — YouTube</a></p><p><a href="https://www.youtube.com/watch?v=cTZArDgbIWw">Introduction to Kubeflow — YouTube</a></p><p><strong>What is Kubernetes</strong></p><p>Kubernetes, also known as K8S, is a portable, extensible, open-source platform for managing containerized workloads and services. It is used for automating deployment, scaling, and managing containerized applications and has a large, continually growing ecosystem. Google open-sourced the project in 2014, and it quickly became one of the most promising projects in the recent wave of progress in the DevOps scene.<br>To fully understand its utility, let’s take a short trip down memory lane to the early days of deployment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yF8L2pNmVBUUQxsAR5Onrw.png" /></figure><p>In the <strong>traditional deployment age</strong>, organizations used to run applications via physical servers, which made it impossible to define resource boundaries. If multiple apps run on a physical server, there will be instances where one of them would take up most of the resources and cause the others to perform poorly. A solution would have been running each application on separate physical servers — but such a move would generate huge costs.</p><p>Then, in the <strong>virtualized deployment age, </strong>a solution that allows running multiple Virtual Machines via the same server’s CPU was introduced. Virtualization enabled isolating applications between Virtual Machines and didn’t allow applications to access another application’s information freely. This led to improved use of resources, better scalability, reduced hardware costs, and the list could go on.</p><p>Then came the <strong>container deployment age</strong>. Containers resemble VMs, but they have the option to share the Operating System between the applications — making them lightweight. A container has its memory, file system, CPU share and is portable across OS distributions and clouds. Of course, containers have a wide array of extra benefits:</p><ul><li>They allow continuous development, integration, and deployment;</li><li>They allow agile application creation and deployment;</li><li>They isolate resources and improve efficiency;</li><li>They allow breaking down applications into independent pieces that can be managed and deployed dynamically;</li><li>They allow cloud and OS distribution portability.</li></ul><p>Kubernetes provides developers with a framework that enables them to run distributed systems smoothly, handling the scaling and failover, providing deployment patterns, and streamlining the process from start to finish. Some of the areas where Kubernetes is extremely useful are:</p><ul><li>Automating rollouts and rollbacks;</li><li>Orchestrating the storage;</li><li>Load balancing and service discovery;</li><li>Automating bin packing;</li><li>Self-healing containers;</li><li>Secret and configuration management;</li></ul><h3><strong>What is Kubeflow and what is it used for</strong></h3><p>Kubeflow was initially created to provide a more straightforward way to run TensorFlow jobs on Kubernetes, and it was based on a TensorFlow Extended pipeline. Then, it was extended to support various architectures and clouds to be used as a machine learning pipeline framework. It’s the machine learning toolkit for Kubernetes, in fewer words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*Y31q_cXZIJnjxTMBuY5VkQ.png" /></figure><p>Kubeflow was launched as a response to two huge IT trends that were beginning to earn the spotlight: cloud-native architectures and data science/machine learning. Kubeflow runs on Kubernetes clusters locally or in the cloud, harnessing the power of training machine learning models on multiple devices and speeding up the process.</p><p>Kubeflow’s tools can help a developer/engineer easily build machine learning models, analyze their performance, refine hyper-parameters, deploy models to production, and manage to compute power.</p><p>Kubeflow is comprised of three main features that simplify machine learning: composability, scalability, and portability. Of course, developers can build containerized machine learning pipelines on Kubernetes without Kubeflow — but it helps standardize the process and make it more efficient. Kubeflow also makes it easier to configure implementations to use hardware accelerators without tweaking the code.</p><p>Kubeflow can be used for the multi-cloud framework, monitoring tools, workflow management, documentation, and model deployment.</p><p>Kubeflow integrates Tensorboard, which provides all the tools needed to visualize the machine learning process and prevent failure. The ability to monitor the training model process enables developers and engineers to refine the model’s parameters in real-time, save resources, accelerate building time via rapid iteration.</p><figure><img alt="Image source: What is Kubeflow? Machine Learning Basics with Kubeflow — BMC Software | Blogs" src="https://cdn-images-1.medium.com/max/347/1*e_MbdUv3aVj9ISE15tAfmw.png" /></figure><p>There are multiple manners to deploy a model via Kubeflow, starting with the KFserving tool, which runs on multiple machine learning frameworks like PyTorch, SciKit Learn, and TensorFlow.</p><p><strong>How Kubeflow elevates an ML workflow</strong></p><p>Constructing, training and deploying ML systems is an iterative process consisting of several stages. One needs to evaluate the output of various stages of the ML workflow and apply changes to the model and parameters when necessary. The diagram below showcases how Kubeflow contributes to each stage:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*BhW5t1qOPV2esgGJYZBN4g.png" /></figure><p>The next diagram shows an actual example of a specific ML workflow that can be used to train and serve a model trained on the MNIST dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ICe5a0Fx5Qaosjak0x_e6g.png" /></figure><p><strong>Summary</strong></p><p>In the race to build highly available, scalable software that’s also easy to manage and update, containers are quickly becoming a viable option for more and more companies across many different types of industries. <br>As more organizations look toward container-driven infrastructure, open-source tools like Kubernetes (K8s) have rapidly become industry standards for automating containerized applications’ deployment, management, and scaling.</p><p>This onslaught of interest in K8s has spawned the creation of newer platforms like Kubeflow, which aims to make the process of managing K8s even easier by abstracting most of the work away from users and letting them focus on their data science projects.</p><p><em>Sources:</em></p><ul><li><a href="https://kubernetes.io/docs/home/"><em>Kubernetes Documentation | Kubernetes</em></a></li><li><a href="https://www.kubeflow.org/docs/"><em>Documentation | Kubeflow</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f7e6dfe893c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The FHIR Protocol]]></title>
            <link>https://danielckv.medium.com/the-fhir-protocol-1ec16499b82d?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ec16499b82d</guid>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Mon, 13 Dec 2021 14:16:35 GMT</pubDate>
            <atom:updated>2021-12-13T14:16:35.708Z</atom:updated>
            <content:encoded><![CDATA[<h4>Fast Healthcare Interoperability Resources (FHIR) for Health Care technology providers.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*BOZ4SdskhscDUs8w.png" /><figcaption>FHIR logo</figcaption></figure><p>In the IT world, talking about open standards and protocols is common. However, when it comes to healthcare, progress in this regard has been slower — at least until the arrival of the FHIR.</p><p>In <a href="https://dermadetect.com/">DermaDetect</a>, most of the data that populated and digitalised for the patient privacy used on the FHIR protocol.</p><p>The Fast Healthcare Interoperability Resources (FHIR) developed by HL7 International, marks a milestone that allows interoperability between different systems, which is extremely important to improve the quality of health service.</p><p>In this article, we will discuss what FHIR is and why it is vital to the future of healthcare.</p><h3>What is FHIR?</h3><p>The Fast Healthcare Interoperability Resources (FHIR) is a specification created to facilitate the transparent exchange of healthcare information between different platforms. In that sense, you can think of FHIR as a protocol that standardizes the exchange of healthcare information, which in turn enables applications, platforms, and healthcare providers to share patient data regardless of the software they use.</p><p>When explaining the concept of FHIR, it is common to use the simile of the HTTP protocol, since it allows communication between different systems easily and quickly.</p><h3>How FHIR works?</h3><p>One aspect that makes FHIR unique is its internet-based approach to connecting different discrete elements. In FHIR terms, “elements” or “resources” have a unique identifier (tag) that works in a similar way to a URL of a web page.</p><p>The benefits of such an approach are extraordinary.</p><p>As with web pages, said “URL” or identifier allows the information to be accessed from any type of device, be it a tablet, a phone, a PC, or any other kind of terminal. Moreover, the resources created using FHIR are based on very easy to handle structures such as XML, JSON, HTTP, Atom, or OAuth. So not only is the data easily accessible, but also the resources created by this standard can be interpreted by any type of system, application, or technology. In other words, FHIR is designed from the ground up to be highly compatible with the platforms currently in use, and even in cases where it is not compatible, its standardized representation of data and its coherent structure allow it to be extremely easy to port to other platforms.</p><h3>What are the key benefits of FHIR?</h3><p>The benefits that FHIR offers are innumerable. In fact, we have already mentioned a few. Here is a list that brings together the main advantages of the FHIR framework.</p><p>● Standardizes the structure and content of the data for easy manipulation on any type of platform</p><p>● It feeds the information directly into the workflow which speeds up data entry</p><p>● Helps to make medical data available easily and conveniently</p><p>● Assign each resource a unique identifier to facilitate its access from any type of device</p><p>● Offers built-in traceability mechanisms to HL7 RIM and other content models</p><p>● Reduces the complexity of healthcare data manipulation and storage</p><p>● Allows the creation of documents that contain a dataset or collection of related data</p><p>● Its design allows developers to use the FHIR API to create applications that handle information more efficiently and quickly.</p><h3>What makes FHIR unique?</h3><p>Contrary to what some may think, the FHIR protocol was not created to replace Electronic Health Records (EHR). As its name indicates, FHIR emerges as a solution to achieve better interoperability between different EHRs. In that sense, FHIR is unique since it allows to achieve a large number of objectives.</p><p>● It allows using a standardized protocol that allows communication between different EHRs</p><p>● Associates a unique identifier to each patient, which is invaluable for physicians as this allows them to access all kinds of information on the medical history of said patient</p><p>● The FHIR standard is created as an API that gives you unprecedented flexibility and growth.</p><p>● As part of its API, FHIR has implemented powerful authorization mechanisms based on the OAuth standard which allows organizations to control data access</p><h3>A quick overlook of FHIR protocol</h3><p>Up to this point, we have talked about what FHIR is, its many benefits, and how it can help the future of medical information management. However, what does this protocol look like? Is it complex or easy to handle?</p><p>Below we present an example of a medical algorithm for assessment and treatment of overweight and obesity, taken directly from the <a href="https://www.hl7.org/fhir/plandefinition-protocol-example.json.html">HL7 FHIR page</a>. In the screenshot below you can get a general idea of the protocol display (raw JSON view).</p><p>As can be seen in the <a href="https://www.hl7.org/fhir/plandefinition-protocol-example.json.html">source file</a>, all the information management is done following the <a href="https://www.hl7.org/fhir/json.html">JSON Format Specification</a>, which is widely known and accepted in the industry. Each property is clearly identified which makes it incredibly easy to understand the logic that the protocol follows.</p><p>One aspect that makes the FHIR protocol extremely flexible is that it does not limit developers to a specific format. Are you more familiar with XML? No problem. In <a href="https://www.hl7.org/fhir/plandefinition-protocol-example.xml.html">this link</a>, you can see the same example using XML.</p><p>Here is another example, this time the screenshot shows patient data visualized as a table:</p><p>The XML code for defining the patient is quite simple:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*7m6ep4y7FP-_NyIF6MhJYA.png" /></figure><p>Or if you prefer JSON, that’s only a matter of formatting it accordingly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*nOVkgtvOEeOLvv-t1aH5vg.png" /></figure><p>FHIR also lets you group information by defining a <strong>resource type</strong> called “Bundle”. <br><a href="https://simplifier.net/Patientexamplesfromt/Bundle-example/~json">Here</a> is an example of the code that bundles the information of 13 patients.</p><p>For more information on the FHIR specification and its Resource Description Framework Representation (RDF), we invite you to visit the <a href="https://www.hl7.org/fhir/rdf.html">official documentation</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ec16499b82d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to CQRS — In Microservices]]></title>
            <link>https://danielckv.medium.com/introduction-to-cqrs-in-microservices-70e4759d9ecc?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/70e4759d9ecc</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cto]]></category>
            <category><![CDATA[cqrs]]></category>
            <category><![CDATA[software-architecture]]></category>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 12:30:57 GMT</pubDate>
            <atom:updated>2021-12-15T17:34:09.643Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to CQRS — In Microservices</h3><figure><img alt="CQRS communication model" src="https://cdn-images-1.medium.com/max/1024/1*kxq4beVj0tLZkBGZWAYd2g.png" /></figure><p>Are you dealing with an application that involves complex business logic?</p><p>If that’s the case, then Command and Query Responsibility Segregation (CQRS) architectural pattern might be the solution you were looking for.</p><p>In recent years the focus on improving user experience has been notable -<br>The way users perceive applications has changed enormously. <br>Nowadays, users expect applications to be lighting fast, especially when performing queries.</p><p>The problem is that as applications become more complex, the handling of detailed queries and validation rules also grows in complexity. In this type of scenario, the conventional <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">Create, Read, Update, and Delete (CRUD)</a> data model could become cumbersome to implement and maintain.</p><p>In this article, <strong>Introduction to CQRS</strong>, we will explore the principles behind the CQRS pattern, as well as its pros and cons, to assist you in deciding whether or not the CQRS pattern is right for your project.</p><h3>How CQRS works</h3><p>The best way to illustrate how CQRS works is with an example. Let’s say you are designing a simple todo application — Following the traditional CRUD data model, you may implement a service in charge of creating, listing, updating, and deleting tasks. A simple solution that reads/updates the data store.</p><p><strong>The Command and Query Responsibility Segregation (CQRS) pattern propose separating the write data model from the read data model</strong>. <strong>This separation of responsibilities would provide the flexibility to decide whether the read and write services should coexist in the same data store or be managed in completely different databases.</strong></p><p>In the CQRS context, <strong>commands are methods whose sole purpose is performing an action</strong>. Simply put, commands in our example would be responsible for creating, updating, and deleting tasks (write operations). In the CQRS architecture, commands cannot return data, since that functionality is unique to queries. In that regard, <strong>queries are methods that are only able to read and return data without modifying it</strong>. Therefore, in our example queries would perform read-only operations such as listing tasks or returning the status of a particular task.</p><p>The implications of CQRS architecture to microservices is huge. In essence, CQRS takes microservices to the next level by splitting CRUD operations into two independent models that can be managed separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*de6XnFji7h-ciacWiHDCfw.png" /><figcaption>CQRS Pattern</figcaption></figure><h4>— How is exactly the CQRS pattern related to microservices</h4><p>Without a doubt, <a href="https://en.wikipedia.org/wiki/Microservices">microservices</a> has changed the way applications are conceived. The monolithic model has transitioned to a decentralized architecture where each service can be scaled and optimized independently. CQRS adds another layer of scalability, performance, and flexibility to microservices by allowing granular read-write optimizations at the database level.</p><p>● Different users normally require unique data representations associated with their roles that may include complex queries. Thanks to CQRS, the read side can be optimized to address such requirements.</p><p>● Business logic usually involves complex validations to ensure that each user is granted with proper permissions. Since CQRS separates read and write sides, the complexity of domain logic can be isolated and optimized more efficiently.</p><p>● Separation of the CRUD data model into specialized and independent services is ideal since it allows taking full advantage of microservices architecture benefits.</p><p>Combining microservices architecture with the CQRS pattern brings many advantages but also has some shortcomings that are discussed next.</p><h3>CQRS benefits in a nutshell</h3><p><strong>Facilitates distributing dev workload</strong>. By separating read and write concerns large teams can manage application development more efficiently. Depending on business logic complexity, developer managers can assign more resources where they are needed the most.</p><p><strong>Easier maintenance</strong>. The flexibility of redistributing the team at will makes maintenance and bug fixing more efficient since resources can be allocated on demand.</p><p><strong>Optimized database design</strong>. Arguably one of the most appealing advantages of the CQRS pattern is the ability to optimize the read side and therefore enhance query performance significantly. Thanks to CQRS, the read model can be optimized for queries, giving the users the freedom to manipulate data views, create detailed filters, custom projections, and more without hindering overall application performance. Similarly, the write side can be designed without the limitations associated with read-intensive applications, meaning better use of database resources.</p><p><strong>Easier scalability</strong>. Another key benefit of using CQRS in microservices has to do with the ability to scale read and write services independently. Since each application is unique, it’s difficult to predict how it could evolve over time. CQRS read/write separation has the advantage of splitting the data management which allows different scalation strategies. In most cases that means the read side can be scaled faster and more efficiently, preserving application performance and user experience.</p><p><strong>Enhanced security</strong>. Nowadays, any technology that helps to harden overall security is welcome. That’s the case of CQRS. By separating write and read operations, it is easier to isolate and decide which domains entities have access to a particular dataset and which ones will have the permissions to write or modify data.</p><h3>CQRS key disadvantages</h3><p><strong>Added complexity</strong>. Although splitting read and write into separate models seems logical, truth is it adds complexity to application design. The level of complexity will depend on the application itself. However, similar to a microservices architecture, this complexity is compensated by the flexibility, performance, and scalability that CQRS offers.</p><p><strong>Eventual consistency</strong>. Separating read and write databases does not come without challenges. Keeping the read model updated is one of them. That explains why using <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">Event Sourcing</a> and CQRS together is so common. Event Sourcing solves the consistency issue but also adds another layer of complexity to the design.</p><h3><em>CQRS vs CRUD, which is better?</em></h3><p>The short answer is, it depends. The CQRS pattern <strong>is not recommended</strong> <strong>when the application business logic is straightforward</strong>. Since CQRS adds complexity to the application design, it has no sense “forcing” the CQRS pattern when the application can work using CRUD-style operations without any scalability issue.</p><p>On the other hand, CQRS could bring substantial benefits in the following scenarios:</p><p><strong>Overly difficult business logic</strong>. The CQRS pattern can help to reduce business logic complexity by isolating read and write concerns.</p><p><strong>Large collaborative applications</strong>. In scenarios where a massive amount of users access data in parallel, the ability to isolate, scale, and fine-tune query operations make CQRS a viable solution.</p><p><strong>Scenarios needing high read and write performance</strong>. Applications requiring fine-tuning both, the reading performance and the writing performance, can take advantage of the CQRS pattern since it allows independent scalation of each model.</p><p><strong>Applications that integrate with Event-driven architecture</strong>. As said earlier, CQRS and Event Sourcing are often used together since they complement each other perfectly.</p><p>Keep in mind that, in complex applications with multiple bounded contexts, nothing prevents you from implementing the CRUD data model in some contexts, and the CQRS pattern in others, leveraging the strengths of each architecture.</p><h3>Wrapping Up</h3><p>Implementing CQRS in your application can maximize its performance, scalability, and security. However, you should carefully consider whether your application is really worth the additional effort involved in such an implementation since, in many cases, the solution offered by the CRUD model is more than enough.</p><p>Does this dilemma sound familiar?</p><p>Deciding whether or not implementing the CQRS pattern leads to challenges similar to those you face when deciding between monolithic applications vs. microservices-based applications. In the end, you will have to ponder if the benefits justify the time and energy required.</p><p>—</p><p><em>Daniel Chernenkov, a software engineering manager @ </em><a href="https://bigabid.com/"><em>Bigabid</em></a><em> and technical advisor for startups.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70e4759d9ecc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is RBAC in Kubernetes?]]></title>
            <link>https://danielckv.medium.com/what-is-rbac-in-kubernetes-c54457eff2dc?source=rss-2e573e0a0400------2</link>
            <guid isPermaLink="false">https://medium.com/p/c54457eff2dc</guid>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Daniel Chernenkov]]></dc:creator>
            <pubDate>Fri, 08 Nov 2019 07:18:56 GMT</pubDate>
            <atom:updated>2020-06-07T14:23:24.752Z</atom:updated>
            <content:encoded><![CDATA[<p>RBAC stands for <strong>Role-Based Access Control</strong>. It is an approach that is used for restricting access to users and applications on the system/network. RBAC is used by Kubernetes for authorization, for example giving access to a user, adding/removing permissions and setting up rules, etc. So basically, it adds security to a Kubernetes cluster. RBAC in Kubernetes is the way that you restrict who can access what within the cluster.</p><h3>Why RBAC?</h3><p>Let’s see why you’re interested in Access Control. Well obviously, you have multiple people on your teams and they’re accessing your Kubernetes cluster. Each of them needs to have some amount of security from one another. There could be cases, where a member in your team is interfering with the other members, work accidentally, for example, one developer might accidentally delete someone else’s work or gain visibility into a secret project that’s being developed. In such cases, having Role-Based Access Control is mandatory.</p><p>When you think about the differences between developers and operators cluster administrators, it becomes even more clear that there are different capabilities that they need to be associated with different types of roles within a Kubernetes cluster. That is where the notion of RBAC or Role-Based Access Control comes into play.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*j8UMSaN5G7KekoOgLEVmsg.png" /><figcaption>RBAC schema model</figcaption></figure><p>RBAC got introduced from Kubernetes 1.8, it uses <strong>rbac.authorization.k8s.io</strong> API group for creating authorization policies. To enable RBAC manually for a Kubernetes cluster you need to start the cluster with the flag<strong> — authorization-mode=RBAC</strong></p><p>By default, RBAC is enabled in Kubernetes. RBAC in Kubernetes infrastructure is implemented through Role, ClusterRole, RoleBinding, and ClusterRoleBinding. Let me tell you what they are with examples.</p><h3>Role</h3><p><strong>Role</strong> in Kubernetes Role-Based Access Control defines a notion of a verb like get or list and a set of nouns like pod volumes etc. So, a role defines what you can do to a set of resources. It contains a set of rules which define a set of permission.</p><p>I encourage people to check out all the different roles that are predefined inside of Kubernetes. There’s a lot of them that can give you an idea and you can even use to build up your own sort of access control and take a deep look into how you want to manage the different roles inside of your cluster so that you can provide a secure cluster where to developers or two operators can’t step on each other’s toes see each other’s Secrets or otherwise interfere with one another.</p><p>Here is a simple example for Roles, where the user has permission to just get and list the pods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*LzxKsPv8I_KloeCSVDA5XQ.png" /></figure><h3>ClusterRole</h3><p>Roles are used to assigning resources for a namespace, but if you need to assign resources on a cluster level, you need to use <strong>ClusterRole</strong>. It is similar to Roles, but it can grant permissions that are cluster-scoped such as giving resource permissions across all namespaces in the cluster.</p><p>Below YAML code defines ClusterRole which grants all then permission to cluster-admin.</p><pre>apiVersion: rbac.authorization.k8s.io/v1</pre><pre>kind: ClusterRole</pre><pre>metadata:</pre><pre>    annotations:</pre><pre>       rbac.authorization.kubernetes.io/autoupdate: “true”</pre><pre>       creationTimestamp: null</pre><pre>       labels:</pre><pre>         kubernetes.io/bootstrapping: rbac-defaults</pre><pre>         name: cluster-admin</pre><pre>  rules:</pre><pre>      - apiGroups:</pre><pre>      - ‘*’</pre><pre>      resources:</pre><pre>       - ‘*’</pre><pre>       verbs:</pre><pre>       - ‘*’</pre><h3>RoleBinding</h3><p><strong>RoleBinding</strong> is used for granting permission to a Subject in a Kubernetes cluster. Subjects are nothing but a set of users, services or groups trying to access Kubernetes API. It defines what operations a user, service or group can perform. It provides privileges within the context of a particular namespace.</p><p>Below is an example of RoleBinding where <strong>roleRef</strong> is being used to bind user John with the role I created above — ElasticSearch.</p><pre>apiVersion: rbac.authorization.k8s.io/v1</pre><pre>kind: RoleBinding</pre><pre>metadata:</pre><pre>    name: elasticsearch</pre><pre>roleRef:</pre><pre>    apiGroup: rbac.authorization.k8s.io</pre><pre>    kind: Role</pre><pre>    name: elasticsearch</pre><pre>subjects:</pre><pre>- kind: User</pre><pre>  name: John</pre><pre>  namespace: default</pre><h3>ClusterRoleBinding</h3><p><strong>ClusterRoleBinding</strong> is used to grant permission to a subject on a cluster-level in all the namespaces.</p><p>It can provide you with permissions for cluster resources and it can also provide you with permissions for resources within any namespace within a cluster. Obviously clusterRoleBindings are very powerful and you want to be careful with how you apply them because they apply not only to any existing namespaces but to any future namespaces that you might create as well.</p><p>Below is an example of ClusterRoleBinding where <strong>roleRef</strong> is being used to bind group masters with the <strong>ClusterRole</strong> I created above — cluster-admin.</p><pre>apiVersion: rbac.authorization.k8s.io/v1</pre><pre>kind: ClusterRoleBinding</pre><pre>metadata:</pre><pre>   name: cluster-admin</pre><pre>roleRef:</pre><pre>    apiGroup: rbac.authorization.k8s.io</pre><pre>    kind: ClusterRole</pre><pre>    name: cluster-admin</pre><pre>subjects:</pre><pre> - apiGroup: rbac.authorization.k8s.io</pre><pre>   kind: Group</pre><pre>   name: masters</pre><p>­­</p><p>The main idea behind <strong>Role-Based Access Control</strong> is to provide access to resources to those users who require and deny access to resources from users who do not require it. This increases security in a Kubernetes Infrastructure.</p><p>If you have any question or need a specific consultancy about Kubernetes, I’m available at <a href="/service/mailto:contact@danielckv.com">contact@danielckv.com </a>(https://danielckv.com)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c54457eff2dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>