<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DATA AI MOTION on Medium]]></title>
        <description><![CDATA[Stories by DATA AI MOTION on Medium]]></description>
        <link>https://medium.com/@aidatamotion?source=rss-a9f5ecc0b31e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tnEIHLkpoJKHHYZfJDdhqw.png</url>
            <title>Stories by DATA AI MOTION on Medium</title>
            <link>https://medium.com/@aidatamotion?source=rss-a9f5ecc0b31e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:53:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aidatamotion/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Semantic Layers: Building an Intelligent Data Architecture That Explains “Why”]]></title>
            <link>https://medium.com/@aidatamotion/beyond-semantic-layers-building-an-intelligent-data-architecture-that-explains-why-eae606592fdd?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/eae606592fdd</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Wed, 04 Feb 2026 12:10:10 GMT</pubDate>
            <atom:updated>2026-02-04T13:22:02.382Z</atom:updated>
            <content:encoded><![CDATA[<p>How combining semantic layers with domain ontologies creates AI-ready data infrastructure</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mr_Tr3Kh9asT8OzDdP63tQ.jpeg" /></figure><p>If you’ve worked with modern data stacks, you’ve likely encountered the semantic layer. Tools like dbt Metrics, Cube, and Looker promise to solve the “single source of truth” problem-ensuring everyone calculates revenue the same way.</p><p>But here’s the uncomfortable truth: <strong>semantic layers answer “what” and “how,” but they can’t answer “why.”</strong></p><p>Ask your semantic layer “What is total revenue by borough?” and you’ll get precise numbers. Ask “Why is Manhattan revenue 5x higher than Brooklyn?” and you’ll get… silence. The calculation layer has no concept of business districts, tourist zones, or the fact that Manhattan has limited parking which drives taxi demand</p><p>In this tutorial, I’ll show you how to build an <strong>Intelligent Semantic Layer</strong> -a 3-layer architecture that combines:</p><ol><li><strong>Technical Layer</strong> (OpenMetadata) → What data exists</li><li><strong>Semantic Layer</strong> (boring-semantic-layer + PostgreSQL) → How to compute metrics</li><li><strong>Ontology Layer</strong> (OWL/RDF) → Why patterns exist and what they mean</li></ol><p>The result? An AI agent that doesn’t just compute metrics-it explains them.</p><pre><br>User: &quot;Why are tips lower in Brooklyn compared to Manhattan?&quot; <br>Agent: Brooklyn tips average $2.10 vs Manhattan&#39;s $3.45. Several factors: <br>  1. Payment method effect: Cash tips are NOT recorded in the data. Brooklyn has higher cash usage (~40% vs 30% in Manhattan). <br>  2. Trip demographics: Manhattan has more business/tourist travelers who tip higher (expense accounts, unfamiliar with local norms). <br>  3. Zone type: Manhattan&#39;s business districts see consistent 15-20% tips while Brooklyn&#39;s residential zones see routine local trips. <br><br>Important: Lower recorded tips ≠ lower actual tips due to the cash ga</pre><p>This <strong>isn’t </strong>prompt engineering or RAG over documentation. It’s a structured knowledge architecture that gives AI agents real domain understanding.</p><p>Before diving in, let’s be honest about when this approach is overkill.</p><p><strong>A semantic layer alone is sufficient when:</strong></p><ul><li>You just need consistent metric definitions across BI tools</li><li>Your analysts know the domain deeply and don’t need explanations</li><li>Questions are purely quantitative (“show me revenue by month”)</li><li>You’re building dashboards for known, recurring questions</li></ul><p><strong>You need the ontology layer when:</strong></p><ul><li>Users ask “why” questions that require domain knowledge</li><li>You’re building AI/LLM interfaces that need business context</li><li>Analysts are exploring unfamiliar data domains</li><li>You need to encode business rules that affect interpretation (e.g., “cash tips aren’t recorded”)</li><li>Different metrics require different analytical approaches based on context</li></ul><p>The key insight from <a href="https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic">Prukalpa’s analysis on Metadata Weekly</a>: <strong>“Meaning isn’t the same as measurement.”</strong> Semantic layers excel at measurement. Ontologies provide meaning.</p><h3>The Architecture: Three Layers Working Together</h3><p>Here’s what we’re building:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*fu_rsyXJZ9BvOSrNRuqyqg.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c4aa067f8d5b6d18e7c679b95a9aac5/href">https://medium.com/media/7c4aa067f8d5b6d18e7c679b95a9aac5/href</a></iframe><p>Let’s build each layer.</p><h4>Step 1: Set Up the Foundation</h4><p>We’ll use the NYC Yellow Taxi dataset-2.76 million trips from January 2024. This is the same dataset Simon Späti uses in his <a href="https://motherduck.com/blog/semantic-layer-duckdb-tutorial/">excellent DuckDB semantic layer tutorial</a>, which makes for a nice comparison.</p><pre>intelligent-semantic-layer/<br>├── docker-compose.yml          # PostgreSQL + OpenMetadata<br>├── semantic_model.yml          # Metric definitions (YAML)<br>├── ontology/<br>│   └── nyc_taxi.ttl           # Domain ontology (OWL/RDF)<br>├── src/<br>│   ├── semantic_layer.py      # Computation layer<br>│   ├── ontology_layer.py      # Knowledge layer<br>│   └── agent.py               # AI agent with tools<br>└── smantic-layer ├── docker-compose.yml # PostgreSQL + OpenMetadata ├── semantic_model.yml # Metric definitions (YAML) ├── ontology/ │ └── nyc_taxi.ttl # Domain ontology (OWL/RDF) ├── src/ │ ├── semantic_layer.py # Computation layer │ ├── ontology_layer.py # Knowledge layer │ └── agent.py # AI agent with tools └── scripts/ └── chat.py # Interactive CL# Clone the repo git clone https://github.com/metazense/intelligent-semantic-layer.git cd intelligent-semantic-layer # Install dependencies (using uv) uv sync # Start PostgreSQL with taxi data docker-compose up -d # Load the data uv run python scripts/load_data.py</pre><h4>Start the Infrastructure</h4><pre># Clone the repo<br>git clone https://github.com/metazense/intelligent-semantic-layer.git<br>cd intelligent-semantic-layer<br><br># Install dependencies (using uv)<br>uv sync<br><br># Start PostgreSQL with taxi data<br>docker-compose up -d<br><br># Load the data<br>uv run python scripts/load_data.py</pre><p>You now have 2.76M taxi trips in PostgreSQL on port 5433.</p><h4>Step 2: The Semantic Layer (The “What” and “How”)</h4><p>The semantic layer defines metrics and dimensions in a declarative YAML file. This is similar to what you’d find in dbt Metrics, Cube, or Looker-a single source of truth for calculations.</p><pre>### semantic_model.yml<br>trips:<br>  table: trips_tbl<br>  time_dimension: pickup_datetime<br><br>  dimensions:<br>    pickup_datetime: _.pickup_datetime<br>    passenger_count: _.passenger_count<br>    trip_distance: _.trip_distance<br>    vendor_id: _.vendor_id<br>    rate_code_id: _.rate_code_id<br>    payment_type: _.payment_type<br><br>  measures:<br>    # Volume metrics<br>    trip_count: _.count()<br><br>    # Revenue metrics<br>    total_revenue: (_.fare_amount + _.tip_amount).sum()<br>    total_fare: _.fare_amount.sum()<br>    total_tips: _.tip_amount.sum()<br>    avg_fare: _.fare_amount.mean()<br>    avg_tip: _.tip_amount.mean()<br><br>    # Payment analysis<br>    credit_card_rate: (_.payment_type == 1).mean()<br><br>    # Airport trips (rate_code 2=JFK, 3=Newark)<br>    airport_trips: ((_.rate_code_id == 2) | (_.rate_code_id == 3)).sum()<br><br>  joins:<br>    pickup_zone:<br>      model: zones<br>      type: one<br>      with: _.pickup_location_id</pre><p>This gives us <strong>23 measures</strong> and <strong>12 dimensions</strong> that compute consistently every time.</p><h4>Using the Semantic Layer</h4><pre>from semantic_layer import SemanticLayer<br><br>sl = SemanticLayer()<br><br># Query revenue by borough<br>result = sl.query(<br>    dimensions=[&quot;pickup_zone.borough&quot;],<br>    measures=[&quot;trip_count&quot;, &quot;total_revenue&quot;, &quot;avg_tip&quot;],<br>    order_by=[(&quot;total_revenue&quot;, &quot;desc&quot;)]<br>)<br><br>for row in result[&quot;data&quot;]:<br>    print(f&quot;{row[&#39;pickup_zone_borough&#39;]}: ${row[&#39;total_revenue&#39;]:,.2f}&quot;)nx: $2,345,678.90 Staten Island: $234,567.12</pre><p><strong>Output</strong></p><pre>Manhattan: $44,075,886.52<br>Queens: $15,707,095.75<br>Brooklyn: $8,567,890.23<br>Bronx: $2,345,678.90<br>Staten Island: $234,567.12</pre><p><strong>This is exactly what traditional semantic layers do well.</strong> We have consistent metrics computed from a single definition. No more “my revenue doesn’t match your revenue” debates.</p><p>But watch what happens when we ask “why”:</p><pre># Traditional semantic layer response to &quot;why&quot;<br>sl.explain(&quot;Why is Manhattan revenue higher?&quot;)<br># &gt;&gt;&gt; AttributeError: &#39;SemanticLayer&#39; object has no attribute &#39;explainTraditional semantic layer response to &quot;why&quot; sl.explain(&quot;Why is Manhattan revenue higher?&quot;) # &gt;&gt;&gt; AttributeError: &#39;SemanticLayer&#39; object has no attribute &#39;explain&#39;</pre><p>The semantic layer has no concept of <em>why</em>. It knows <em>what</em> to calculate and <em>how</em>, but not <em>what it means</em>.</p><h4>Step 3: The Ontology Layer (The “Why”)</h4><p>This is where we diverge from traditional semantic layer tutorials. We’re going to add a <strong>domain ontology</strong> -a formal representation of business knowledge that an AI agent can query for context and reasoning.</p><p>Think of the ontology as encoding everything a domain expert knows that isn’t in the data:</p><ul><li><strong>Classifications</strong>: What types of trips exist? (Airport, Commute, Long-distance)</li><li><strong>Zone semantics</strong>: What makes a zone special? (Business district, Tourist area, Transit hub)</li><li><strong>Business rules</strong>: Why does this metric behave this way? (Cash tips aren’t recorded)</li><li><strong>Inference rules</strong>: How should I interpret this pattern? (Low tips in outer boroughs ≠ bad service)</li></ul><pre>@prefix : &lt;http://example.org/nyc-taxi#&gt; .<br>@prefix owl: &lt;http://www.w3.org/2002/07/owl#&gt; .<br>@prefix rdfs: &lt;http://www.w3.org/2000/01/rdf-schema#&gt; .<br><br># =============================================================================<br># CORE CLASSES<br># =============================================================================<br><br>:Trip a owl:Class ;<br>    rdfs:label &quot;Trip&quot; ;<br>    rdfs:comment &quot;A completed taxi trip from pickup to dropoff&quot; ;<br>    :mapsToTable &quot;trips&quot; .<br><br>:Borough a owl:Class ;<br>    rdfs:label &quot;Borough&quot; ;<br>    rdfs:comment &quot;One of the 5 NYC administrative districts&quot; .<br><br>:PaymentType a owl:Class ;<br>    rdfs:label &quot;Payment Type&quot; ;<br>    rdfs:comment &quot;Method of payment for the trip&quot; .<br><br># =============================================================================<br># BOROUGH INSTANCES (with business context!)<br># =============================================================================<br><br>:Manhattan a :Borough ;<br>    rdfs:label &quot;Manhattan&quot; ;<br>    :boroughCode &quot;Manhattan&quot; ;<br>    :businessContext &quot;Dominates taxi activity with ~90% of all trips.<br>                      High business/tourist density, limited parking drives demand.&quot; .<br><br>:Brooklyn a :Borough ;<br>    rdfs:label &quot;Brooklyn&quot; ;<br>    :boroughCode &quot;Brooklyn&quot; ;<br>    :businessContext &quot;Primarily residential. Lower taxi usage, shorter trips,<br>                      more local transportation alternatives.&quot; .<br><br>:Queens a :Borough ;<br>    rdfs:label &quot;Queens&quot; ;<br>    :boroughCode &quot;Queens&quot; ;<br>    :businessContext &quot;Contains JFK and LaGuardia airports. High airport trip<br>                      volume with flat-rate fares.&quot; .<br><br># =============================================================================<br># PAYMENT TYPES (with critical business rules!)<br># =============================================================================<br><br>:CreditCard a :PaymentType ;<br>    rdfs:label &quot;Credit Card&quot; ;<br>    :paymentCode 1 ;<br>    :businessContext &quot;Tips ARE recorded. ~70% of all trips.&quot; .<br><br>:Cash a :PaymentType ;<br>    rdfs:label &quot;Cash&quot; ;<br>    :paymentCode 2 ;<br>    :businessContext &quot;CRITICAL: Tips are NOT recorded in data.<br>                      Tip analysis should exclude cash trips.&quot; .<br><br># =============================================================================<br># TRIP TYPE CLASSIFICATIONS<br># =============================================================================<br><br>:AirportTrip a owl:Class ;<br>    rdfs:subClassOf :Trip ;<br>    rdfs:label &quot;Airport Trip&quot; ;<br>    :classificationCondition &quot;rate_code_id IN (2, 3)&quot; ;<br>    :businessContext &quot;JFK has flat $52 rate to Manhattan. Higher tips expected.<br>                      Predictable revenue.&quot; ;<br>    :avgFareRange &quot;$40-$70&quot; .<br><br>:CommuteTrip a owl:Class ;<br>    rdfs:subClassOf :Trip ;<br>    rdfs:label &quot;Commute Trip&quot; ;<br>    :classificationCondition &quot;hour IN (7,8,9,17,18,19) AND weekday&quot; ;<br>    :businessContext &quot;Rush hour patterns. Longer durations due to traffic.<br>                      Regular riders, predictable demand.&quot; .<br><br>:LongDistanceTrip a owl:Class ;<br>    rdfs:subClassOf :Trip ;<br>    rdfs:label &quot;Long Distance Trip&quot; ;<br>    :classificationCondition &quot;trip_distance &gt; 10&quot; ;<br>    :businessContext &quot;Higher revenue per trip. Often airport or outer borough.&quot; .<br><br># =============================================================================<br># ZONE TYPE CLASSIFICATIONS<br># =============================================================================<br><br>:BusinessDistrict a owl:Class ;<br>    rdfs:subClassOf :Location ;<br>    rdfs:label &quot;Business District&quot; ;<br>    :businessContext &quot;High weekday demand during business hours.<br>                      Expense account travelers. Good tip rates (15-20%).&quot; ;<br>    :exampleZones &quot;Midtown, Financial District, Flatiron&quot; .<br><br>:TouristZone a owl:Class ;<br>    rdfs:subClassOf :Location ;<br>    rdfs:label &quot;Tourist Zone&quot; ;<br>    :businessContext &quot;Consistent demand across week. Higher tips from visitors.<br>                      Often unfamiliar with local tipping norms (tip more).&quot; ;<br>    :exampleZones &quot;Times Square, Central Park, SoHo&quot; .<br><br>:ResidentialZone a owl:Class ;<br>    rdfs:subClassOf :Location ;<br>    rdfs:label &quot;Residential Zone&quot; ;<br>    :businessContext &quot;Lower taxi volume. Morning trips TO business districts,<br>                      evening trips FROM. Regular local riders (tip less).&quot; .<br><br># =============================================================================<br># INFERENCE RULES (the real magic)<br># =============================================================================<br><br>:CashTipDataGap a :InferenceRule ;<br>    rdfs:label &quot;Cash Tip Data Gap&quot; ;<br>    :condition &quot;payment_type = 2&quot; ;<br>    :inference &quot;Tip amount is unknown (recorded as $0)&quot; ;<br>    :businessContext &quot;Cash tips given directly to driver, not captured in data.<br>                      ALWAYS exclude cash payments from tip analysis.&quot; .<br><br>:LowTipNotServiceIssue a :InferenceRule ;<br>    rdfs:label &quot;Low Tip Interpretation&quot; ;<br>    :condition &quot;avg_tip &lt; $2.50 AND borough NOT IN (&#39;Manhattan&#39;)&quot; ;<br>    :inference &quot;Low tip reflects demographics, NOT service quality&quot; ;<br>    :businessContext &quot;Outer borough trips have lower recorded tips due to:<br>                      1) Higher cash payment rates (tips not recorded)<br>                      2) Local residents vs tourists (different tipping norms)<br>                      3) Shorter, routine trips (lower baseline)&quot; .<br><br>:ManhattanDominance a :InferenceRule ;<br>    rdfs:label &quot;Manhattan Dominance Pattern&quot; ;<br>    :inference &quot;Manhattan will dominate any volume or revenue metric&quot; ;<br>    :businessContext &quot;~90% of yellow taxi trips start in Manhattan due to:<br>                      - Business district concentration<br>                      - Tourist activity<br>                      - Limited parking (drives taxi demand)<br>                      - Density of destinations&quot; .<br><br>:TipAnalysisRule a :InferenceRule ;<br>    rdfs:label &quot;Tip Analysis Rule&quot; ;<br>    :appliesTo :tipAmount ;<br>    :condition &quot;payment_type != 2&quot; ;<br>    :reasoning &quot;When analyzing tips, ALWAYS filter to credit card payments.<br>                Cash tips exist but aren&#39;t recorded.&quot; .</pre><p>This is <strong>55 classes</strong> encoding domain knowledge that no YAML metric definition can capture.</p><h3>Using the Ontology Layer</h3><pre>from ontology_layer import OntologyLayer<br><br>ontology = OntologyLayer()<br><br># Get business context for a concept<br>context = ontology.get_concept_context(&quot;Brooklyn&quot;)<br>print(context[&quot;businessContext&quot;])<br># &gt;&gt;&gt; &quot;Primarily residential. Lower taxi usage, shorter trips...&quot;<br><br># Classify a trip<br>classifications = ontology.classify_trip(<br>    rate_code_id=2,<br>    trip_distance=15,<br>    pickup_hour=8,<br>    day_of_week=2  # Tuesday<br>)<br>print(classifications)<br># &gt;&gt;&gt; [&quot;AirportTrip&quot;, &quot;LongDistanceTrip&quot;, &quot;CommuteTrip&quot;]<br><br># Get relevant rules for an analysis type<br>rules = ontology.get_context_for_analysis(&quot;tips&quot;)<br>print(rules[&quot;critical_rules&quot;])<br># &gt;&gt;&gt; [&quot;CashTipDataGap: Cash tips not recorded - exclude payment_type=2&quot;]</pre><p>Now we have knowledge, not just calculations.</p><h3>Step 4: The AI Agent (Bringing It Together)</h3><p>The final piece is an AI agent that can use both layers as tools. We’re using Azure OpenAI with function calling, but this works with any LLM that supports tools.</p><h4>The Agent’s Tools</h4><pre>TOOLS = [<br>    {<br>        &quot;name&quot;: &quot;query_metrics&quot;,<br>        &quot;description&quot;: &quot;Query aggregated metrics from NYC taxi data&quot;,<br>        &quot;parameters&quot;: {<br>            &quot;dimensions&quot;: [&quot;pickup_zone.borough&quot;],<br>            &quot;measures&quot;: [&quot;trip_count&quot;, &quot;total_revenue&quot;, &quot;avg_tip&quot;]<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;get_context&quot;,<br>        &quot;description&quot;: &quot;Get business context from the ontology&quot;,<br>        &quot;parameters&quot;: {<br>            &quot;concept&quot;: &quot;Manhattan&quot;  # or &quot;CreditCard&quot;, &quot;AirportTrip&quot;, etc.<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;explain_metric&quot;,<br>        &quot;description&quot;: &quot;Get explanation for why a metric has a certain value&quot;,<br>        &quot;parameters&quot;: {<br>            &quot;metric&quot;: &quot;avg_tip&quot;,<br>            &quot;context&quot;: {&quot;borough&quot;: &quot;Brooklyn&quot;}<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;classify_trip&quot;,<br>        &quot;description&quot;: &quot;Classify a trip using ontology rules&quot;,<br>        &quot;parameters&quot;: {<br>            &quot;rate_code_id&quot;: 2,<br>            &quot;trip_distance&quot;: 15,<br>            &quot;pickup_hour&quot;: 8<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;get_analysis_context&quot;,<br>        &quot;description&quot;: &quot;Get relevant rules for a type of analysis&quot;,<br>        &quot;parameters&quot;: {<br>            &quot;analysis_type&quot;: &quot;tips&quot;  # or &quot;revenue&quot;, &quot;demand&quot;, &quot;airport&quot;<br>        }<br>    }<br>]</pre><h4>The Agent’s System Prompt</h4><pre>SYSTEM_PROMPT = &quot;&quot;&quot;You are an expert NYC taxi data analyst with access to:<br><br>1. SEMANTIC LAYER (query_metrics tool):<br>   - Computes actual metrics from 2.76M taxi trips<br>   - Dimensions: pickup_zone.borough, zone_name, payment_type, etc.<br>   - Measures: trip_count, total_revenue, avg_fare, avg_tip, etc.<br><br>2. ONTOLOGY (get_context, explain_metric tools):<br>   - Business knowledge about NYC taxi domain<br>   - Zone classifications (Airport, BusinessDistrict, Tourist, Residential)<br>   - Trip classifications (AirportTrip, CommuteTrip, LongDistance)<br>   - Inference rules explaining WHY metrics vary<br><br>CRITICAL RULES:<br>1. Always use query_metrics to get ACTUAL DATA - never make up numbers<br>2. Use get_context to understand WHY patterns exist<br>3. Cash tips (payment_type=2) are NOT recorded - note this in tip analysis<br>4. Low tips in outer boroughs reflect demographics, NOT service quality<br>5. Manhattan dominates (~90% of trips) due to business/tourist density<br><br>When answering:<br>1. First query the data to get actual numbers<br>2. Then get ontology context to explain the patterns<br>3. Provide clear answer with both DATA and EXPLANATIuery_metrics Args: {&quot;dimensions&quot;: [&quot;pickup_zone.borough&quot;], &quot;measures&quot;: [&quot;avg_tip&quot;, &quot;credit_card_rate&quot;]} Result: Manhattan: $3.45 avg tip, 70% credit card Brooklyn: $2.10 avg tip, 58% credit card --- Iteration 2 --- Tool: get_context Args: {&quot;concept&quot;: &quot;Brooklyn&quot;} Result: {&quot;businessContext&quot;: &quot;Primarily residential. Lower taxi usage...&quot;} --- Iteration 3 --- Tool: get_context Args: {&quot;concept&quot;: &quot;Cash&quot;} Result: {&quot;businessContext&quot;: &quot;CRITICAL: Tips are NOT recorded...&quot;} --- Iteration 4 --- Tool: get_analysis_context Args: {&quot;analysis_type&quot;: &quot;tips&quot;} Result: {&quot;critical_rules&quot;: [&quot;CashTipDataGap&quot;, &quot;LowTipNotServiceIssue&quot;]}</pre><h4>The Agent in Action</h4><pre>from agent import NYCTaxiAgent<br><br>agent = NYCTaxiAgent()<br><br>response = agent.ask(&quot;Why are tips lower in Brooklyn compared to Manhattan?&quot;)<br>print(response.answer)<br>```<br><br>**Agent&#39;s reasoning process:**<br><br>```<br>--- Iteration 1 ---<br>Tool: query_metrics<br>Args: {&quot;dimensions&quot;: [&quot;pickup_zone.borough&quot;], &quot;measures&quot;: [&quot;avg_tip&quot;, &quot;credit_card_rate&quot;]}<br>Result: Manhattan: $3.45 avg tip, 70% credit card<br>        Brooklyn: $2.10 avg tip, 58% credit card<br><br>--- Iteration 2 ---<br>Tool: get_context<br>Args: {&quot;concept&quot;: &quot;Brooklyn&quot;}<br>Result: {&quot;businessContext&quot;: &quot;Primarily residential. Lower taxi usage...&quot;}<br><br>--- Iteration 3 ---<br>Tool: get_context<br>Args: {&quot;concept&quot;: &quot;Cash&quot;}<br>Result: {&quot;businessContext&quot;: &quot;CRITICAL: Tips are NOT recorded...&quot;}<br><br>--- Iteration 4 ---<br>Tool: get_analysis_context<br>Args: {&quot;analysis_type&quot;: &quot;tips&quot;}<br>Result: {&quot;critical_rules&quot;: [&quot;CashTipDataGap&quot;, &quot;LowTipNotServiceIssue&quot;]}</pre><h4><strong>Final answer:</strong></h4><blockquote><em>Brooklyn tips average $2.10 vs Manhattan’s $3.45. This difference has several explanations:</em></blockquote><ul><li><strong>Payment method effect</strong>: Cash tips are NOT recorded in the data. Brooklyn has higher cash usage (42% vs 30% in Manhattan), meaning more tips are invisible.</li><li><strong>Trip demographics</strong>: Manhattan has more business travelers and tourists who tip higher (expense accounts, unfamiliar with norms). Brooklyn is primarily residential with routine local trips.</li><li><strong>Zone characteristics</strong>: Manhattan’s business districts (Midtown, Financial District) see consistent 15–20% tip rates. Brooklyn’s residential zones see shorter, cheaper neighborhood trips.</li></ul><blockquote><strong><em>Important</em></strong><em>: Lower recorded tips ≠ lower actual tips. The cash payment gap means Brooklyn’s true tipping may be similar to Manhattan’s.</em></blockquote><p>This is fundamentally different from what a semantic layer alone can provide. The agent <strong>computed</strong> the numbers from the semantic layer and <strong>explained</strong> them using the ontology.</p><h3>The Comparison: What Each Approach Can Answer</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58b4ef6d998fb58925c8bc6f48937791/href">https://medium.com/media/58b4ef6d998fb58925c8bc6f48937791/href</a></iframe><h3>The YAML Problem (And How Ontologies Solve It)</h3><p>As Prukalpa Sankar notes in her <a href="https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic">Metadata Weekly article</a>, semantic layers attempt to “encode business meaning in YAML files.” But YAML captures calculations, not understanding.</p><p>Here’s the same concept in YAML vs OWL:</p><pre>### YAML (Semantic Layer)<br>```yaml<br>measures:<br>  avg_tip: _.tip_amount.mean()<br>```</pre><p>This tells us <em>how</em> to calculate. Nothing more.</p><p><strong>OWL (Ontology)</strong></p><pre>```turtle<br>:tipAmount a owl:DatatypeProperty ;<br>    rdfs:label &quot;tip amount&quot; ;<br>    rdfs:comment &quot;Tip paid (only recorded for credit card)&quot; ;<br>    :mapsToColumn &quot;trips.tip_amount&quot; ;<br>    :unit &quot;USD&quot; ;<br>    :businessRule &quot;Only available for credit card payments (payment_type = 1)&quot; .<br><br>:CashTipDataGap a :InferenceRule ;<br>    :condition &quot;payment_type = 2&quot; ;<br>    :inference &quot;Tip amount is unknown (recorded as $0)&quot; ;<br>    :reasoning &quot;Cash tips given directly to driver, not captured in data.&quot; .<br><br>:TipAnalysisRule a :InferenceRule ;<br>    :appliesTo :tipAmount ;<br>    :condition &quot;payment_type != 2&quot; ;<br>    :reasoning &quot;When analyzing tips, ALWAYS filter to credit card payments.&quot; .<br>```</pre><p>This tells us <em>what it means</em>, <em>when it’s valid</em>, and <em>how to interpret it</em>.</p><h4>When to Invest in This Approach</h4><p>Building an ontology is more work than writing YAML. It requires domain expertise and knowledge engineering skills. Here’s when it’s worth it:</p><p><strong>High value:</strong></p><ul><li>Building AI/LLM interfaces to your data</li><li>Analysts exploring unfamiliar domains</li><li>Complex domains with many business rules (healthcare, finance, logistics)</li><li>Self-service analytics where users need guidance</li><li>Data products where explanations matter</li></ul><p><strong>Lower value:</strong></p><ul><li>Internal dashboards for domain experts</li><li>Simple aggregation queries</li><li>Well-understood, stable metrics</li><li>Small teams with shared context</li></ul><p>The <a href="https://www.ebi.ac.uk/ols/ontologies/go">life sciences industry</a> has invested in ontologies for decades because they enable discovery and reasoning impossible through metrics alone. As AI becomes the primary consumer of our data infrastructure, other industries will need the same investment.</p><p><strong>Try It Yourself</strong><br>The complete implementation is open source:</p><pre>git clone https://github.com/metazense/intelligent-semantic-layer.git<br>cd intelligent-semantic-layer<br><br># Install and run<br>uv sync<br>docker-compose up -d<br>uv run python scripts/load_data.py<br><br># Start chatting<br>uv run python scripts/chat.py </pre><p>Example questions to try:</p><ul><li>“What is the total revenue by borough?”</li><li>“Why is Manhattan revenue so much higher than other boroughs?”</li><li>“Why are tips lower in Brooklyn compared to Manhattan?”</li><li>“Classify a trip from JFK to Midtown at 8 AM on Tuesday”</li><li>“What should I know before analyzing tip patterns?”</li></ul><h4>What’s Next: The Context-Aware Future</h4><p>The semantic layer was built for BI dashboards-human consumption through visualizations. But AI agents don’t need dashboards. They need <strong>context and meaning</strong>.</p><p>I believe we’re heading toward what Prukalpa calls “context-aware semantic layers”-systems that combine:</p><ol><li><strong>Metric governance</strong> (semantic layers) → Consistent calculations</li><li><strong>Domain knowledge</strong> (ontologies) → Business understanding</li><li><strong>Decision context</strong> (context graphs) → Why decisions were made</li></ol><p>This isn’t just about better analytics. It’s about building data infrastructure that AI can actually reason with.</p><p>The semantic layer answered “what” and “how.” The ontology layer answers “why.” The next layer will answer “what should we do about it.”</p><p><em>The code for this tutorial is available at </em><a href="https://github.com/metazense/intelligent-semantic-layer"><em>github.com/metazense/intelligent-semantic-layer</em></a><em>. Built with boring-semantic-layer, rdflib, and Azure OpenAI. Inspired by Simon Späti’s </em><a href="https://motherduck.com/blog/semantic-layer-duckdb-tutorial/"><em>DuckDB semantic layer tutorial</em></a><em> and Prukalpa Sankar’s </em><a href="https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic"><em>analysis of ontologies and semantic layers</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eae606592fdd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Metrics to Meaning: Why AI Agents Need More Than a Semantic Layer]]></title>
            <link>https://medium.com/@aidatamotion/from-metrics-to-meaning-why-ai-agents-need-more-than-a-semantic-layer-ec8715b8fc63?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec8715b8fc63</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Tue, 27 Jan 2026 11:51:01 GMT</pubDate>
            <atom:updated>2026-01-27T12:32:35.149Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical guide to building unified data architectures for the agentic era</h4><h3>WHAT: The Problem We’re Solving</h3><p>Modern data teams face a familiar challenge: <strong>metrics sprawl</strong>. Revenue gets calculated differently in the finance dashboard than in the sales report. “Active users” means one thing to product and another to marketing. Every analyst writes their own SQL, and definitions drift.</p><p>The <strong>semantic layer</strong> emerged as the solution — a single source of truth for metric definitions. Define “revenue” once, query it everywhere, get consistent numbers.</p><p>But a new challenge has arrived: <strong>AI agents</strong>.</p><p>When you ask an AI agent, <em>Why did tips decrease in Brooklyn last month?</em>, it needs more than a consistent number. It needs to understand:</p><ul><li>What Brooklyn <strong>is</strong> (a borough in NYC, primarily residential)</li><li>What tips <strong>mean </strong>(indicator of satisfaction, varies by trip type and demographics)</li><li>How trips <strong>relate</strong> to zones, providers, and time patterns</li><li>What rules govern the domain (airport trips tip differently than local rides)</li></ul><p><strong>The semantic layer tells you WHAT the number is. But AI needs to understand WHY</strong>.</p><p>This is the gap we’re addressing: building data architectures that serve both <strong>human analysts</strong> (who need consistent metrics) and <strong>AI agent</strong>s (who need contextual understanding and reasoning capabilities).</p><h3>WHY: A Semantic Layer Alone Isn’t Enough</h3><h4>The Metrics-First Paradigm</h4><p>Semantic layers were designed for the BI era. Their job is straightforward:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0d3a3dcfd36814bf7a451789233eb324/href">https://medium.com/media/0d3a3dcfd36814bf7a451789233eb324/href</a></iframe><p>This works beautifully for dashboards. An analyst asks, “What’s our revenue by borough?” The semantic layer translates that into SQL, executes it, returns numbers. Consistent. Governed. Reliable.</p><h4><strong>Where It Falls Short</strong></h4><p>But consider what happens when an AI agent receives this question:</p><p><strong>Why are tips lower in Brooklyn than Manhattan?</strong></p><p>A semantic layer can return:</p><ul><li><em>Manhattan: $4.20 average tip</em></li><li><em>Brooklyn: $2.80 average tip</em></li></ul><p>But it <strong>cannot</strong> explain:</p><ul><li>Brooklyn is primarily residential with local commuters</li><li>Manhattan has more tourists and business travelers who tip generously</li><li>Airport trips (common in Manhattan-bound travel) have higher tip norms</li><li>Lower tips in Brooklyn reflect demographics, not service quality</li></ul><p>The semantic layer knows <strong>how to calculate</strong>. It doesn’t know <strong>what things mean</strong>.</p><h4>The AI Reasoning Gap</h4><p>As the <a href="https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic">Metadata Weekly article</a> puts it: <em>Semantic layers were designed for humans to consume through BI tools… Ontologies are designed for system and domain understanding</em>.</p><p>AI agents don’t just retrieve numbers — they <strong>reason</strong>. They need to:</p><ol><li><strong>Understand concepts</strong>: What is a “borough”? What is a “shared ride”</li><li><strong>Navigate relationships</strong>: Trips connect to zones, zones belong to boroughs, providers compete with each other</li><li>3. <strong>Apply rules</strong>: If a trip starts at JFK, it’s an airport trip with different characteristics</li><li><strong>Generate explanations</strong>: Connect data patterns to domain knowledge</li></ol><p>A <em>YAML</em> file with metric calculations cannot encode this knowledge. <strong>Meaning isn’t the same as measurement</strong>.</p><h3>WHY: A Unified Approach Is Better</h3><p>The solution isn’t to abandon semantic layers — they solve a real problem. The solution is to <strong>augment</strong> them with ontological knowledge.</p><h4><strong>What Each Layer Contributes</strong></h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eebd41eb8c8aeb854376e0146f791515/href">https://medium.com/media/eebd41eb8c8aeb854376e0146f791515/href</a></iframe><h4>The Unified Vision</h4><p>Microsoft Fabric IQ demonstrates this approach: agents don’t query databases directly — they query an <strong>ontology</strong> that has <strong>semantic bindings</strong> to the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e487330040d043b248de600b8d987660/href">https://medium.com/media/e487330040d043b248de600b8d987660/href</a></iframe><p>Now an AI agent can:<br>1. <strong>Query</strong>: Get the numbers ($4.20 vs $2.80)<br>2. <strong>Contextualize</strong>: Understand why they differ<br>3. <strong>Explain</strong>: Generate meaningful answers<br>4. <strong>Avoid misinterpretation</strong>: Know that low tips don’t indicate poor service</p><h3>CONCEPTUAL FOUNDATIONS</h3><h4>What Is a Semantic Layer?</h4><p>A <strong>semantic layer</strong> is an abstraction between raw data and consumers. It defines business metrics in one place so they’re calculated consistently everywhere.</p><p><strong>Core idea</strong>: Instead of every analyst writing <em>`SELECT SUM(base_fare) FROM trips</em>`, they query <em>`total_revenue`</em> and the semantic layer handles the translation.</p><p><strong>Components:</strong><br>- <strong>Dimensions</strong>: Things you group by (borough, zone, date)<br>- <strong>Measures</strong>: Things you calculate (trip_count, avg_fare)<br>- <strong>Joins</strong>: How tables connect (trips → zones via LocationID)</p><p><strong>Optimized for</strong>: Measurement consistency, metric governance, SQL abstraction:</p><p><strong>Example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d28c7d3adf57ec1551bb678e49e78236/href">https://medium.com/media/d28c7d3adf57ec1551bb678e49e78236/href</a></iframe><h4>What Is an Ontology?</h4><p>An <strong>ontology</strong> is a formal representation of knowledge in a domain. It defines what <strong>things</strong> <em>are</em>, how they <em>relate</em>, and what <em>rules</em> govern them.</p><p><strong>Core idea</strong>: Beyond storing data, capture the <strong>meaning </strong>of concepts so machines can reason about them.</p><p><strong>Components</strong>:</p><ul><li><strong>Classes </strong>: Types of things (Trip, Zone, Borough, Provider)</li><li><strong>Properties</strong>: Attributes and relationships (startsAt, locatedIn, competesWith)</li><li><strong>Instances</strong>: Specific entities (Manhattan, JFK Airport, Uber)</li><li><strong>Rules</strong>: Inference logic (if pickup is airport → AirportTrip)</li></ul><p><strong>Optimized for</strong>: Knowledge representation, machine reasoning, contextual understanding</p><p><strong>Example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e993e4f149768d572ba9471713fcd08/href">https://medium.com/media/2e993e4f149768d572ba9471713fcd08/href</a></iframe><h4>The Key Distinction</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7195719861988c9427029145d883c09f/href">https://medium.com/media/7195719861988c9427029145d883c09f/href</a></iframe><p>The semantic layer tells you that <em>Brooklyn’s average tip</em> is $2.80.</p><p>The ontology tells you that <em>Brooklyn is a residential borough </em>where trips are mostly local, demographics differ from Manhattan’s tourist-heavy traffic, and therefore <strong>lower tips are expected and don’t indicate service problems</strong>.</p><p><strong>HOW: Practical Implementation with NYC Taxi Data</strong></p><p>Let’s make this concrete using the NYC Taxi dataset — 20 million ride-hailing trips with fares, tips, locations, and timestamps.</p><h4>Layer 1: Physical Data</h4><p>Raw data in DuckDB:</p><pre>trips_tbl (20M rows)<br>├── pickup_datetime<br>├── PULocationID (foreign key)<br>├── trip_miles<br>├── base_passenger_fare<br>├── tips<br>├── shared_match_flag (&#39;Y&#39;/&#39;N&#39;)<br>└── …<br>taxi_zones_tbl (265 rows)<br>├── LocationID<br>├── Borough<br>├── Zone (name)<br>└── service_zone</pre><h4>Layer 2: Semantic Layer (Metrics)</h4><p>Define consistent calculations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06e6fb26bd45a9c3c4a0f776ec8214b6/href">https://medium.com/media/06e6fb26bd45a9c3c4a0f776ec8214b6/href</a></iframe><p><strong>Result</strong>: Any system can query <em>`trip_count`</em> by <em>`pickup_zone.borough`</em> and get consistent numbers.</p><h4>Layer 3: Ontology (Knowledge)</h4><p>Add meaning, relationships, and rules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ebe4440b02a3ee20ee0133b81638cfc/href">https://medium.com/media/4ebe4440b02a3ee20ee0133b81638cfc/href</a></iframe><h4>Layer 4: AI Agent</h4><p>An agent that queries the unified ontology:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/245e6a442d70bb573766994cccebe378/href">https://medium.com/media/245e6a442d70bb573766994cccebe378/href</a></iframe><p><strong>Example interaction:</strong></p><pre>User: &quot;Why are tips lower in Brooklyn than Manhattan?&quot;<br>Agent calls:<br> → query_metrics(dimensions=[&quot;pickup_zone.borough&quot;], measures=[&quot;avg_tips&quot;])<br> → get_context(&quot;Borough&quot;, &quot;Brooklyn&quot;)<br> → get_context(&quot;Borough&quot;, &quot;Manhattan&quot;)<br> → get_measure_info(&quot;avg_tips&quot;)<br>Agent responds:<br> &quot;Brooklyn tips average $2.80 compared to Manhattan&#39;s $4.20.<br>This $1.40 difference reflects:<br>1. Trip composition: Manhattan has more airport transfers and tourist<br> trips where 15–20% tips are standard. Brooklyn trips are mostly<br> local and utilitarian.<br>2. Demographics: Manhattan&#39;s business travelers often expense rides<br> and tip generously. Brooklyn has more cost-conscious local riders.<br>3. Trip distance: Shorter Brooklyn trips yield smaller absolute tips<br> even at similar percentage rates.<br>Important: This is NOT a service quality issue. The ontology explicitly<br> notes that &#39;low tips in outer boroughs reflect demographics, not service.&#39;<br> Compare tips within similar trip types for meaningful analysis.&quot;</pre><p>The agent didn’t just return numbers — it <strong>explained</strong> them using ontology knowledge.</p><h4>Answering Complex Questions</h4><p>The unified approach enables questions a semantic layer alone cannot answer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f024372b2e38f83e823984d41bdd002d/href">https://medium.com/media/f024372b2e38f83e823984d41bdd002d/href</a></iframe><h3>ARCHITECTURE SUMMARY</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUihILyQeWuVt2tiGvt61Q.png" /></figure><h3>KEY TAKEAWAYS</h3><p>1.<strong> Semantic layers solve metric consistency </strong>— essential for reliable analytics, but limited to calculations.</p><p>2. <strong>Ontologies encode domain knowledge</strong> — what things mean, how they relate, and why patterns exist.</p><p>3. <strong>AI agents need both</strong> — consistent numbers AND contextual understanding to reason and explain.</p><p>4. <strong>The unified approach</strong> combines semantic bindings (data → metrics) with ontological knowledge (concepts, relationships, rules).</p><p>5. <strong>Agents query the ontology, not SQL </strong>— they work with business concepts like <em>avg_tips</em> and <strong>Borough</strong>, not raw tables and columns.</p><p>6. <strong>This is where the industry is heading </strong>— Microsoft Fabric, Palantir’s context graphs, and similar platforms are building unified semantic-ontological layers for AI-native architectures.</p><p>— -</p><h3>CONCLUSION</h3><p>The semantic layer was the right abstraction for the BI era: consistent metrics, governed access, SQL hidden from analysts.</p><p>But AI agents aren’t analysts. They don’t just retrieve numbers — they reason about them. They need to understand that Brooklyn is residential, that airport trips tip differently, that low metrics can be normal rather than problematic.</p><p><strong>Meaning isn’t the same as measurement</strong>.</p><p>By unifying semantic layers (for consistent calculation) with ontologies (for domain knowledge), we can build data architectures that serve both:</p><ul><li><strong>Human analysts </strong>who need reliable, governed metrics</li><li><strong>AI agents </strong>who need contextual understanding and reasoning capabilities</li></ul><p>The future of data infrastructure isn’t just knowing WHAT the numbers are — it’s understanding WHY they are what they are.</p><h3>REFERENCES</h3><p>1. [Why Semantic Layers Matter — and How to Build One with DuckDB](<a href="https://motherduck.com/blog/semantic-layer-duckdb-tutorial/">https://motherduck.com/blog/semantic-layer-duckdb-tutorial/</a>) — MotherDuck Tutorial</p><p>2. [Ontologies, Context Graphs, and Semantic Layers: What AI Actually Needs in 2025](<a href="https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic">https://metadataweekly.substack.com/p/ontologies-context-graphs-and-semantic</a>) — Metadata Weekly</p><p>3. [boring-semantic-layer](<a href="https://github.com/boringdata/boring-semantic-layer">https://github.com/boringdata/boring-semantic-layer</a>) — Python library used in implementation</p><p>4. [NYC TLC Trip Record Data](<a href="https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page">https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page</a>) — Source dataset</p><p>— -</p><p><em>This article accompanies a working implementation demonstrating these concepts. </em><a href="https://github.com/ai4data/semantic-plus-ontology">The code in Github</a><em> includes a semantic layer (nyc_taxi.yml), an ontology (ontology/*.ttl), a unified version (unified_ontology.yml), and an AI agent (agent.py) powered by an LLM.</em>*</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec8715b8fc63" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MCP: Hype vs. Reality in AI Agent Tooling]]></title>
            <link>https://medium.com/@aidatamotion/mcp-hype-vs-reality-in-ai-agent-tooling-66384dbe7014?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/66384dbe7014</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[mcp-protocol]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Thu, 13 Mar 2025 22:53:10 GMT</pubDate>
            <atom:updated>2025-03-13T23:17:57.888Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*8zBceVjKAXAxiY3U" /></figure><p>The rise of AI agents promises a new era of automation, where intelligent systems can interact with tools and services to streamline workflows, fetch information, and perform tasks autonomously. A crucial aspect of this revolution is the standardization of how these agents interact with the world around them. One protocol seeking to define this interaction is MCP (Model Context Protocol).</p><p>But before we fully embrace MCP, we need to ask tough questions: Is it truly the best approach? Is it over-engineered? Are the security implications being adequately addressed? As with any new technology, a critical voice is essential to ensure we’re building the right foundation for the future.</p><p><strong>The Two Critical Concerns: Persistent Connections and the Sampling Dilemma</strong></p><p>MCP introduces two key challenges that cause many to question its real-world applicability: the unnecessary burden of stateful connections and the security vulnerabilities inherent in its “Sampling” feature. These problems raise concerns about both efficiency and the potential for malicious exploitation.</p><ol><li><strong>Persistent Connections: An Unnecessary Burden of State</strong></li></ol><p>A defining characteristic of MCP is its reliance on a <em>persistent connection</em> between the AI agent and the tool it interacts with. Think of it like keeping a phone line constantly open to a pizza place, even when you’re just browsing the menu. This is simply not needed.</p><p>In contrast, traditional REST APIs operate on a stateless model: each request is independent, like making a single, self-contained phone call to place your pizza order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*BVcRYpetYGQ7B_Ybnx3hRg.png" /></figure><p>This persistent connection introduces significant overhead:</p><ul><li><strong>Increased Server Load:</strong> Maintaining numerous persistent connections puts a strain on server resources.</li><li><strong>Scaling Inefficiencies:</strong> Scaling becomes more complex and resource-intensive.</li><li><strong>Serverless Incompatibility:</strong> It fundamentally clashes with the benefits of serverless architectures like AWS Lambda, Google Cloud Functions, and Azure Functions, where stateless execution is a core principle.</li></ul><p>Modern architectures are trying to move toward stateless, but MCP is holding these implementations back.</p><p><strong>2 . Sampling: Reversal of Control is a Potential Security Nightmare</strong></p><p>Even more concerning is MCP’s “Sampling” feature. As the official documentation explains, Sampling allows the <em>server</em> (the tool adapter) to <em>request</em> that the <em>agent</em> (the AI agent) run Large Language Model (LLM) operations.</p><p>Here’s how it might work in practice:</p><ol><li>The AI agent wants a tool to generate an image with certain specifications</li><li>The tool has a function to ask the user’s computer to generate code.</li><li>The agent has to use it</li></ol><p>The problem? This allows the tool to reverse a level of control and can cause a big security issue.</p><p><strong>The Security Risk:</strong></p><ul><li>If the tool becomes compromised from dependency poisioning, it could exfiltrate code from the agent.</li><li>The agent has no way of identifying whether this is real or a malicious attack</li></ul><ol><li>This is just one example of a security risk that occurs from this technique.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*smbvDS3WkADno_7fqtkZqg.png" /></figure><p>I find this incredibly dangerous and unnecessary, especially in an Enterprise Setting</p><p><strong>The Sampling Scenario: A Software Developer and a Compromised Tool</strong></p><p>Let’s illustrate the potential dangers with a concrete example. Imagine an AI agent assisting a software developer with a “Code Quality Checker” tool:</p><ol><li><strong>Without MCP:</strong> The agent sends a request to the API. Simple, stateless, and efficient.</li><li><strong>With Stateful MCP and Sampling:</strong></li></ol><ul><li>The agent connects to the “Code Quality Checker.”</li><li>During the code analysis, the server requests the agent to perform LLM operations for code commenting.</li><li>The server is compromised. It gets access to user API keys.</li></ul><p>The persistent connection and the “Sampling” feature are the issues I am seeing in this technology.</p><p><strong>A Call for Simplicity: Rethinking the Approach to AI Agent Tool Interaction</strong></p><p>The problems with MCP highlight a critical need to rethink this approach. Instead of complex, stateful protocols, we should consider:</p><ul><li><strong>Standardized OpenAPI Descriptions:</strong> Use OpenAPI specifications to describe tool capabilities in a consistent, machine-readable format. This allows agents to easily understand how to interact with tools.</li><li><strong>Lightweight JSON Metadata:</strong> Employ lightweight JSON files (like agents.json by Wildcard AI) to provide additional metadata about tools and their purpose.</li><li><strong>Stateless Interactions:</strong> Prioritize stateless interactions over persistent connections. Keep things simple and modular.</li></ul><p>This has allowed new technologies like Cloud Functions to flourish because they do not have to deal with high complexities.</p><p><strong>Conclusion: Embracing the KISS (Keep It Simple, Stupid) Principle</strong></p><p>MCP, in its current form, appears to be over-engineered for the majority of AI agent tool interaction scenarios. Its stateful nature, combined with the security risks of the “Sampling” feature, creates unnecessary complexity. A simpler, stateless approach that leverages existing standards will provide a more robust, scalable, and secure foundation for the future of AI agent integration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66384dbe7014" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlock your data: The data preparation process]]></title>
            <link>https://medium.com/@aidatamotion/unlock-your-data-the-data-preparation-process-a7360286a7ba?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7360286a7ba</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Sun, 21 Jan 2024 13:59:30 GMT</pubDate>
            <atom:updated>2024-01-24T10:46:23.930Z</atom:updated>
            <content:encoded><![CDATA[<p>The <a href="https://medium.com/@trefoilanalytics/unlock-your-data-data-sourcing-step-ab57f14aa771">data sourcing process</a> concludes when the data is transferred to the raw data zone, also known as the landing zone. This zone contains unprocessed copies of data “as-is” from various source systems. Following this, another sub-process begins: the data preparation process. Most modern data architecture approaches are currently based on what is known as the data lake medallion architecture. Briefly, medallion architecture necessitates dividing the Data Lake into three primary areas: Bronze, Silver, and Gold. In addition to these three layers, a fourth area called the Landing Zone is sometimes required. These areas are depicted in the image below. The Landing Zone, Bronze Layer, and Silver Layer are typically utilized to prepare data for different business uses and generally do not require an understanding of business-level aggregates. On the other hand, the Gold Layer contains transformed, aggregated, and modeled data that necessitates business comprehension. This is why the Gold Layer is considered part of the data and analytics delivery segment of the data supply chain, as opposed to the data onboarding phase. The Gold Layer will not be discussed during the preparation process phase, but it will be mentioned later when we explore the data and analytics delivery segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F63Fpv_dxj595-lbkbYj_Q.png" /><figcaption>Date medallion architecture</figcaption></figure><p>The preparation process involves several steps, and we’ll discuss each of them in detail. We’ll also talk about the data management capabilities required to execute these steps and how AI can be used to automate the process.</p><h4>Describing the preparation process</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dpzbhKV8gvn1sSA0lIsGSQ.png" /><figcaption>Data Preparation process</figcaption></figure><p>The data preparation process involves several steps to ensure that the data is clean, well-structured, and ready for integration and analysis. This process is carried out by the data engineering team and the data quality team. It consists of the following steps:</p><p>1. <strong>Validate the (meta)-data requirement:</strong> The first step in the data preparation process is to validate the data requirement from the sourcing process. This includes ensuring that the (meta) data received is according to the data requirement. Metadata is validated to ensure the integrity, quality, and usability of the data. Metadata provides contextual information about the data, which is crucial for data management and governance.</p><p>2. <strong>Enrich the data with metadata: </strong>Once the data has been validated, the data engineering team enriches the data with some metadata. This step involves segregating the data by source system, dataset, and time-based hierarchies, converting it into a standardized file format such as Parquet or Delta Tables, and storing it in the Bronze Layer.</p><p>3. <strong>Prepare for and store the data in the <em>Silver Layer</em>:</strong> The next step is to perform various data pipelines checks and transformations to ensure that the data is clean and well-structured. This includes executing DQ ppipelines checks, fixing any DQ pipeline issues, normalizing the data, ensuring that business keys are unique, de-duplicating data, anonymizing data if needed, augmenting the data with surrogate keys, establishing foreign key relationships, enforcing data types, and ensuring that the file format has ACID capabilities and a transaction log. Once these tasks are completed, the data is stored in the <em>Silver Layer</em> using a Delta Lake. At this point data is made available for the data quality team to DW check and repair for business compliance</p><p>4. <strong>Data quality checks execution and error repair:</strong> The data quality team executes data quality checks to identify and fix any errors or inconsistencies in the data. This involves repairing any issues that are found and storing the cleansed data in the <em>Silver Layer</em>. The Data quality team is also responsible for optimizing the DQ score and creating a DQ KPI score.</p><p>5. <strong>Handling missing requirements:</strong> In case any process requirements are missing, such as missing metadata or data quality requirements, the data engineering team together with the data quality team will work to address these gaps. This may involve reverse engineering the missing requirements using advanced methods such as Generative AI to suggest the value of the missing requirements.</p><p>By following this data preparation process, the data engineering team and the data quality team ensure that the data is clean, well-structured, and ready for integration and analysis, enabling data users to derive valuable insights from the data.</p><h4>People involved in the preparation process</h4><p><strong>Matching the preparation process with the DM capabilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*mn49mSQQ4RWnfEN2-nYlWg.png" /></figure><ol><li><em>Validate the (meta)-data requirement</em>: ad-hoc, self-build manual solutions.</li><li><em>Enrich the data with metadata: </em>ad-hoc, self-build manual solutions.</li><li><em>Prepare for and store the data in the Silver Layer: </em>ad-hoc, self-build manual solutions.</li><li><em>Data quality checks execution and error repair: </em>ad-hoc, self-build manual solutions.</li><li><em>Handling missing requirements: </em>ad-hoc, self-build manual solutions</li></ol><h4>Potential AI solutions</h4><ul><li><em>Validate the (meta)-data requirement</em>: LLMs can learn and validate expected metadata schemas and properties. If invalid, the LLM suggests more probable valid schemas, properties, or key-values.</li><li><em>Enrich the data with metadata: </em>LLMs can propose potential linkages between technical metadata found in existing databases and the entities in the business metadata, possibly clustering tables related to the same concept such as customer, market, or product. This aids in identifying tables for priority integration.</li><li><em>Prepare for and store the data in the Silver Layer: </em>LLMs can propose automatic correction of misspellings, semantics data transformation, extracting and repairing syntactic pattern, taxonomy corrections.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*A_4zXrslEaqJH83bpkyclA.png" /></figure><p>Other manual and time-consuming activities like normalizing the data, ensuring that business keys are unique, de-duplicating data, anonymizing data if needed, augmenting the data with surrogate keys, establishing foreign key relationships or enforcing data types.</p><ul><li><em>Data quality checks execution and error repair: </em>see point 3.</li><li><em>Handling missing requirements: </em>Any suggestions<em>? </em>LLMs?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7360286a7ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlock your data: data sourcing step]]></title>
            <link>https://medium.com/@aidatamotion/unlock-your-data-data-sourcing-step-ab57f14aa771?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab57f14aa771</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 19:25:41 GMT</pubDate>
            <atom:updated>2024-01-10T14:51:09.332Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*sq4WlO2Xi-iT3Ki9zGF7jA.png" /><figcaption>Data sourcing process</figcaption></figure><p>The data sourcing In the <a href="https://medium.com/@hzmarrou/data-management-2-0-ai-and-automation-for-next-gen-data-management-frameworks-introduction-2a00ae67ccaf">previous article</a>, we started to unravel the complex web that is the data management process — the supply chain connecting data source and consuming. In the forthcoming articles, we’re going to delve deeper into four pivotal topics that orbit this process.<br>For each topic, we’ll dissect and elaborate on the sub-steps involved in every stage, giving you a comprehensive understanding of the tasks at hand. Then, we’ll spotlight the key players responsible for executing these steps, using the RACI (Responsible, Accountable, Consulted, and Informed) model. In addition, we’ll examine the data management capabilities needed to carry out the process effectively, spotlighting the essential skills and resources required. Finally, we’ll look at how innovative Data &amp; AI solutions can boost these capabilities and make the process smarter and more data-informed.</p><p>So, let’s start with the data sourcing process.</p><h3>The data sourcing process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aIydxdMO-hwktrKG.png" /><figcaption>Data sourcing process</figcaption></figure><h4>Describing the sourcing process</h4><p>he data sourcing process is the step in the data value chain that involves identifying and collecting data from various sources to make it available for further processing and consumption. This process can be quite complex, depending on the number of sources, the types of data, and the specific requirements of the data platform.</p><p>Here is a general overview of the data sourcing process:</p><p>1. <strong>Identify data need</strong>: Begin by determining the data needs of the organization, project, use case or a portfolio of use case. This includes understanding the types of data required, the level of detail, and the specific attributes needed for analysis.</p><p>2. <strong>Identify potential data sources</strong>: Next, identify the possible sources of data that can meet your requirements. This may include internal sources such as databases, applications, and spreadsheets, as well as external sources such as public data sets, partner data, or purchased data.</p><p>3. <strong>Evaluate and select the golden source:</strong> The golden source is the most accurate, reliable, and complete source of data for a particular data set. Evaluate each potential source based on factors such as data quality, timeliness, and coverage, and select the best one(s) to serve as the golden source.</p><p>4. <strong>Establish data ownership:</strong> Identify the data owner(s) responsible for maintaining and managing the selected data sources. Data owners are critical for ensuring the quality and security of the data throughout its lifecycle.</p><p>5. <strong>Define metadata:</strong> Metadata is the information that describes the data, such as data definitions, data types, and relationships between data elements. Documenting metadata is essential for understanding, integrating, and managing the data when unlocked from the source. It is also essential to map the business metadata to the technical metadata coupling the data elements to the corresponding data attributes (columns) in the actual databases.</p><p>6. <strong>Registering and classifying the data</strong> involves creating a structured data catalog that organizes and categorizes collected information. This step facilitates easier discovery, understanding, and access to data for users. It includes metadata documentation, data categorization, data lineage tracking, access control, and version control, all of which contribute to maintaining data quality, reliability, and security. By effectively managing data in this manner, organizations can support informed decision-making and ensure data is accessible and useful for their specific needs.</p><p><strong>7.</strong> <strong>Determine DQ requirements: </strong>Refers to the process of identifying and defining the criteria and standards that your data must meet to ensure that it can be effectively used for decision-making, reporting, and analysis.</p><p>8. <strong>Establish data extraction methods:</strong> Determine the appropriate methods for extracting data from the source applications. This may involve using APIs, direct database connections, or file-based extraction processes.</p><h3>Key players involved in the sourcing process</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*p6gTWIRDOUiaZg0EtDjsig.png" /><figcaption>RACI Matrix for the sourcing process</figcaption></figure><p>The sourcing process is the very first step in the data value chain, its holds liberating the data from the source, it involves the following key players, including the <strong>data user, the data owner, data stewards, the engineering team,</strong> and the <strong>data quality team</strong>.</p><h4>Mapping the Cloud Data Management Capabilities to the sourcing process</h4><p>To successfully execute the sourcing step in the data management process, it is essential to possess specific data management capabilities. These capabilities are abilities that a data management team or function possesses to achieve the specific purpose or function.</p><p>Starting with building capabilities as per the Cloud Data Management Capability (CDMC) framework, or any other data management framework, can be beneficial. However, we believe these frameworks are primarily designed to provide control mechanisms to ensure compliance with internal and external policies, and they offer limited enablement capabilities. Enablement capabilities are crucial for automating and speeding up the data management process, unlocking data’s full potential, and promoting a data-driven culture. Let’s look at how the steps in the sourcing process map to the potential capabilities in the CDMC framework:</p><ul><li><em>Identifying data needs:</em> Capabilities not present!</li><li><em>Identifying potential data sources:</em> Capabilities not present!</li><li><em>Evaluating and selecting the golden source:</em> Capabilities not present!</li><li><em>Establishing data ownership:</em> Addressed by CDMC 1.0 Governance and Accountability.</li><li><em>Defining metadata</em>: Capabilities not present!</li><li><em>Registering and classifying data:</em> Addressed by CDMC 2.0 Cataloguing and Classification.</li><li><em>Determining DQ requirements:</em> Capabilities not present!</li><li><em>Establishing data extraction methods:</em> Capabilities not present!</li></ul><p><em>Enablement capabilities</em> are central to fully leveraging the potential of data and fostering a data-centric culture. Unlike control, enablement focuses on developing capabilities that encourage strategic and efficient use of data to swiftly achieve business goals. This includes helping to find the right data for a specific use case, identifying suitable data sources, suggesting metadata when it’s lacking, proposing Data Quality (DQ) rules when they’re not present, and more.</p><p>We maintain that Artificial Intelligence (AI) can be used to both enhance existing capabilities and expedite the data management process.</p><h4>Potential AI solutions</h4><ul><li><em>Identifying data needs</em>: Using metadata descriptions of available datasets and previous data use cases, Large Language Models (LLMs) can analyze this information and recommend suitable datasets for the current use case. See our medium post here</li><li><em>Identifying potential data sources</em>: Same as Identifying data need!</li><li><em>Evaluating and selecting the golden source</em>: (LLMs)can help identify golden sources by analyzing metadata and data lineage, determining if an application is the first storage point for data, improving accuracy in the onboarding process to data platforms or marketplaces.</li><li><em>Defining metadata</em>: LLM can help to extract business terms from technical term and the data domain where the data belong.</li><li><em>Registering and classifying data</em>: Addressed by CDMC 2.0 Cataloguing and Classification.</li><li><em>Determining DQ requirements: </em>From (multi-columns) profiling to DQ rules</li><li><em>Establishing data extraction methods: </em>Eng. Capability!</li></ul><p>The data sourcing process concludes when the data is transferred to the raw data zone, also known as the landing zone. This zone contains unprocessed copies of data “as-is” from various source systems. Following this, another sub-process begins: the data preparation process. Most modern data architecture approaches are currently based on what is known as the data lake medallion architecture. In the following article we will deep dive into this step.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab57f14aa771" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlock your data: AI and automation for next-gen data management frameworks. Introduction]]></title>
            <link>https://medium.com/@aidatamotion/data-management-2-0-ai-and-automation-for-next-gen-data-management-frameworks-introduction-2a00ae67ccaf?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a00ae67ccaf</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 18:43:58 GMT</pubDate>
            <atom:updated>2024-01-09T10:37:21.427Z</atom:updated>
            <content:encoded><![CDATA[<p>In our rapidly evolving data-centric world, effective data management has become the golden key that unlocks the true potential of large-scale data analysis and AI applications. It’s not only about managing data, but also about ensuring smooth and trusted access to it across your organization, enabling everyone to find the answers they need. But, this journey has its own set of challenges. Roadblocks such as data availability, accessibility, quality, security, usability, and harmonization of scattered datasets often emerge due to our dependence on outdated, manual methods of managing data.</p><p>At Trefoil Analytics, we’re charting a new course. Guided by the principles of DAMA-DMBOK 2, the Cloud Data Management Capabilities (CDMC) framework, and CDQ AG, we’ve designed a cutting-edge data management framework. This isn’t just another system; it’s a solution powered by advanced AI capabilities, crafted to boost every aspect of data management. It’s our way of overcoming the traditional hurdles and fully unlocking the power of your data assets.</p><p>Over the coming weeks, we will share a series of articles detailing our unique approach to data management. Look forward to content that explores how to build applications within these categories. As always, we welcome your engagement and feedback!</p><h3>What is data management?</h3><p>Data management encompasses the processes of creating, collecting, preparing, storing, identifying, accessing, processing, and monitoring the organization’s data, ensuring its readiness for consumption. As the generation and consumption of data reach unparalleled levels, data management solutions become indispensable in making sense of the vast data pools.</p><h3>The Importance of data management</h3><p>Data management is a vital initial step for implementing large-scale data analysis that yields valuable insights, enhancing customer experiences and boosting revenue. Effective data management enables individuals across an organization to locate and access trustworthy data for their inquiries. Key benefits of a robust data management solution include:</p><h4>Visibility</h4><p>Data management enhances the visibility of your organization’s data assets, simplifying the process of locating and accessing accurate data for analysis. Improved data visibility promotes better organization and productivity, enabling employees to find the necessary data to excel in their roles.</p><h4>Reliability</h4><p>By establishing processes and policies for data usage, data management minimizes potential errors and fosters trust in the data used for decision-making throughout your organization. Access to reliable, current data allows companies to adapt more efficiently to market fluctuations and customer demands.</p><h4>Security</h4><p>Protecting your organization and its employees from data losses, theft, and breaches, data management employs authentication and encryption tools. Robust data security ensures that critical company information is backed up and retrievable in case the primary source becomes inaccessible. Moreover, security is increasingly crucial when handling personally identifiable information that must be managed in compliance with consumer protection laws.</p><h4>Scalability</h4><p>Data management facilitates the effective scaling of data and usage instances by implementing repeatable processes to maintain up-to-date data and metadata. With easily replicable processes, your organization can sidestep unnecessary duplication costs, such as employees conducting repetitive research or re-running costly queries without cause.</p><h3>The challenge of data management</h3><p>Organizations are confronted with a multitude of data management challenges, including data quality monitoring, data integration, metadata gathering processes, data modeling, data classification, and the identification of key and critical data elements, all while aligning with business goals and determining the scope for managing data. Traditionally, these activities have relied heavily on human intervention, such as expert data model building or rigid coding of data quality rules, resulting in delays within the data value chain.</p><p>In recent years, numerous initiatives have been developed to automate these processes using advanced technology, tools, and techniques, such as machine learning algorithms, generative AI, and data integration platforms. These state-of-the-art solutions can substantially reduce manual intervention, streamline processes, enhance flexibility, improve efficiency, and ultimately drive informed decision-making based on reliable and comprehensive data insights. Trefoil Analytics is dedicated to building upon existing data management frameworks and utilizing advanced tools to provide our clients with user-friendly solutions that accelerate their data management processes, thereby expediting the time to value for their data initiatives. By partnering with us, organizations can effectively tackle data management challenges and unlock the full potential of their data to achieve growth and success.</p><h3>Methodology for developing data management 2.0</h3><p>Trefoil Data Management 2.0 is a data management solution that is built upon two popular frameworks, the Cloud Data Management Capabilities framework (CDMC) and the CDQ Data Excellence Model. By combining these two models, Trefoil aims to bring a business focus to data management and make the data management model cloud proof. At the core of these two frameworks are data management capabilities and business capabilities. However, while these frameworks provide information on <strong>what </strong>capabilities an organization needs to develop to achieve data excellence and business value, they do not provide information on how to develop them. Trefoil Data Management 2.0 provides the <strong>“how”</strong> part of the equation by describing the different data management processes needed to achieve data excellence and business value, which utilize the data management capabilities outlined in the two frameworks. <strong>Additionally</strong>, Trefoil data management 2.0 is powered by advanced methods based on machine learning and generative AI, which help to automate data management processes and keep data up-to-date and relevant.</p><h4>The cloud data Management capabilities framework</h4><p>The CDMC framework defines the best practice capabilities for managing and controlling data in cloud environments. It includes controls that address six main areas: <em>governance and accountability</em>, <em>cataloging and classification</em>, <em>accessibility and usage</em>, <em>protection and privacy</em>, the <em>data life cycle</em>, and <em>data and technical architecture</em>. Across these areas, the CDMC identifies 14 sub-capabilities necessary for an effective program.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*i6jHfI1ZrHw6AxvL8svH-Q.png" /><figcaption><em>Overview of the CDMC Key Controls Framework (Source: EDM Council, CDMC Working Group)</em></figcaption></figure><h4>The CDQ data excellence model</h4><p>The data excellence model is a data management framework that offers support and guidance for practitioners in the implementation of data management by defining major design areas, while at the same time supporting the transformation into a digital and data-driven company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*usC8GZXZt6epzRAusutpcQ.png" /><figcaption>Overview of the CDQ Data Excellence Model</figcaption></figure><p>The structure of the data excellence model builds on the principles of performance management and the logic of management cycles. The reference model specifies design areas of data management in three categories: <strong>goals</strong>, <strong>enablers</strong>, and <strong>results</strong>, which are interlinked in a continuous improvement cycle.</p><p><strong>Goals</strong> break down the overall aim and purpose of data management by outlining necessary business capabilities and data management capabilities and explicating them in the form of a data strategy.</p><p><strong>Enablers</strong> help to achieve the goals specified regarding six design areas<em>: people, roles and responsibilities</em>; <em>performance management</em>; <em>processes and methods</em>; <em>data architecture</em>; <em>data lifecycle</em>; and <em>data applications</em>.</p><p><strong>Results </strong>indicate to what extent the goals are achieved in terms of two quantifiable aspects: data excellence and business value; and</p><p><strong>Continuous improvement</strong> allows adjustment of <strong>goals</strong> and <strong>enablers</strong>, ensuring the dynamic nature of the model.</p><h4>The Trefoil Data Management Framework</h4><p>The concepts of “capability” within both the CDMC and CDQ frameworks define the essential skills and abilities an organization needs to master to accomplish the desired results of a data management program. These results are specifically data excellence and the generation of business value. However, the capability alone does not provide the methodology for achieving these outcomes. This is where the business process or value stream becomes relevant.</p><p>Clarifying the business process or value stream often requires a more comprehensive explanation than merely defining the capability. It includes outlining the necessary tasks in sequence, identifying the responsible parties, and highlighting crucial decisions that need to be made, including the rationale behind these decisions and the decision-makers involved.</p><p>In the subsequent sections, we will offer a thorough and customized blueprint for building your data management value stream and start to cultivate the required capabilities for executing the process and achieving the expected results.</p><p>The Trefoil Data Management Framework (TDMF) is a combination of the CDMC and CDQ data management frameworks. These frameworks outline the essential skills and abilities that a company should develop to manage their data effectively at scale. Additionally, the TDMF presents a detailed, <em>generic data management process</em>, which explains <em>how</em> to manage data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*1BkVLpTaUFo5965T5_0cMw.png" /><figcaption>Combining the CDMC and the CDQ data management capabilities</figcaption></figure><h3>The data value chain</h3><h4>Overview of the Enterprise data types</h4><p>In an enterprise, there are various types of data, each serving different purposes and playing crucial roles in decision-making and operations, as depicted in Figure 1. <em>Reference data</em> includes controlled vocabularies, flat lists of values, and classification hierarchies that provide context and standardization. <em>Master data</em> consists of primary and secondary data elements that represent core business entities, such as customers, products, and suppliers. <em>Transactional data</em> captures the details of business activities, like sales and purchases. <em>Classical analytical data </em>comprises simple and advanced pre-defined structures/queries used for reporting and analysis. <em>Observational data</em> is collected through sensors, IoT devices, or surveys, providing insights into real-world events and behaviors. <em>Media data</em> encompasses multimedia content, such as images, videos, and audio files. <em>Advanced analytical and AI data</em> involves training data labels, feature stores, predictive models, performance metrics, and source code used in machine learning and artificial intelligence applications. Lastly, <em>metadata</em> describes various aspects of other data sets, including structural, technical, business, operational, and social information, to enable better understanding and management of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*gNlafZrtZmW-Maw6u7HQKA.png" /><figcaption>Enterprise data types of overviews.</figcaption></figure><h4>Supply chain connecting data source and data consuming</h4><p>The data value chain consists of a series of interconnected steps that transform raw data into actionable insights. It starts with <em>creating and maintaining</em> data, which involves generating or collecting information from various sources and ensuring its accuracy and relevance. <em>Sourcing </em>identifies appropriate sources and extracts the required data, while the <em>preparation</em> step involves cleaning and transforming data into a suitable format. <em>Storing </em>focuses on securely saving the processed data in the data platform, being a Lakehouse or a Datawarehouse. <em>Identifying </em>ensures that relevant data sets are discovered and cataloged for easy retrieval. <em>Accessing</em> grants authorize users’ access to the data, and <em>processing</em> involves integrating and manipulating the data to create data products ready for use and share in the business. <em>Monitoring</em> maintains data quality, integrity, and security throughout the chain. Finally, the <em>using</em> stage interprets and applies the insights generated from the data to inform decision-making processes and create value for the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*7NL-uNgxEsx46SRSydU_9g.png" /><figcaption>Data supply chain</figcaption></figure><h4>High level stakeholder (incomplete) map for the data supply chain</h4><p>The data value chain comprises various roles, each contributing to the effective management and utilization of data within an organization. <em>Data Creators</em>, either internal or external, are responsible for generating and maintaining data according to the agreed-upon definitions with <em>Data Owners</em>. <em>Data Owners</em>, typically senior employees within the organization, hold accountability for data definition within their specific areas of responsibility and have the authority to verify data accuracy. <em>Data Users</em> are employees who utilize data for specific purposes and are responsible for setting data requirements. D<em>ata Consumers</em>, which can be internal or external parties, use the data as intended by the Data Owners and Data Users. The <em>Business Application Owner</em> oversees the core functionality and interfaces of an application, ensuring its proper maintenance, access control, and service delivery. The (Golden) <em>Source Application</em> is the application where data is created and provided, serving as the original source of data. Lastly, the <em>Consuming Application</em> is where data is stored and integrated for specific usage, allowing for seamless access and retrieval within the organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*-Nk_RtiC7PWNxHC8AyorAA.png" /><figcaption>High level stakeholder map for the data supply chain</figcaption></figure><p>In the upcoming artilces, we will delve deeply into each of the individual process steps in the data value chain, clarifying the sub-processes required for their execution. Additionally, we will identify the key stakeholders necessary for governing and executing these steps, as well as outline the technology, tools, AI techniques and resources available for people to implement the process.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a00ae67ccaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A detailed look into five core application of Large Language Models]]></title>
            <link>https://medium.com/@aidatamotion/exploring-the-potential-of-large-language-models-a-detailed-look-into-five-core-application-1cc4b007261d?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cc4b007261d</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 16:04:22 GMT</pubDate>
            <atom:updated>2024-01-09T08:47:36.034Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*ADWLMzgbaIyGFM974AEX8g.png" /><figcaption>Five Core Application of Large Language Models</figcaption></figure><p>In our day-to-day interactions with clients and AI experts, we are frequently asked if we can simplify and narrow down the fundamental use cases for Large Language Models (LLMs).</p><p>In this article, we will delve into the realm of software applications that employ large language models. We will discuss our perspective on the varying categories of these applications, noting that most LLM-powered applications fall into one of five main categories. We will explain these categories, provide examples of business applications, and discuss some architectural design considerations for implementing these apps. This article is designed to assist developers in determining which category their LLM-based applications fit into. While some applications may straddle multiple categories, most predominantly fit into one of these five.</p><h4><strong>1. NATURAL LANGUAGE PROCESSING (NLP) TASKS</strong></h4><p>Topping the list is the category of traditional Natural Language Processing (NLP) tasks. Here, LLMs have proven their superiority in tasks like named entity recognition, sentiment analysis, and data extraction. They serve as the Swiss Army Knife of the data world, with the power to convert unstructured data into structured data, a crucial tool for these tasks.<br>One prominent business use of LLM in this context is summarizing call center conversations and extracting structured data such as the caller’s first and last name, reason for the call, problem being addressed, sentiment of the caller, and storing it in a structured format. Another high-value application is extracting entities, their definitions, and their relationships from enterprise documents, aiding in the creation of an Enterprise Semantic Graph, which is essential for accelerating a company’s data journey.</p><p>The deployment of such applications closely resembles the deployment of other NLP models like Bert or Spacy. Developers can utilize existing inference pipelines and replace existing endpoints of older, smaller models like Bert or Spacy with new LLM endpoints.</p><h4>2. CONVERSATIONAL AGENTS OR CHATBOTS</h4><p>The second category is conversational agents or chatbots. These have been a part of the tech scene for over a decade, but recent advancements in LLMs, such as GPTs, have made these agents more viable and effective. From virtual friends to customer service representatives, these chatbots are designed to mimic human interaction and are widely used in customer support and appointment setting among other areas. Historically, chatbots were limited to specific, closed domains and relied heavily on rule-based approaches. However, the introduction of GPT models, such as ChatGPT, has opened up the world of open-domain conversations, leading to more interactive and engaging conversations with users.</p><p>One prominent business example is customer service, where these engaging chatbots are used to interact with customers in a friendly way, answering general questions or serving as a wrapper on a company’s website to swiftly extract information from the site and answer questions directly instead of letting the customer browse the entire website to find an answer. A good example of such an application is Bing Chat, which combines chatting and browsing experiences.</p><p>The typical architecture for such a chatbot used in enterprise applications breaks down a monolithic application (the chatbot) into separate services for each logical part. For a chatbot, these services could include a front-end service, a conversation plus storage service, a service for monitoring prompts (guardrails) and making requests to the LLM , and databases for storing conversations and embeddings. The data flow in this architecture begins with the front-end, which sends a request to the conversation service. This service retrieves and stores the conversation, then sends a request to the LLM service. The LLM service returns an answer, which is stored and sent back to the front-end. Each of these services can be containerized using Docker, providing each service with its own operating system resources and environment. An orchestrator like Kubernetes is used to manage these containers, maintaining the applications’ health and can start and stop services as necessary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yxemORjbw5bVbrb5q2PkVA.png" /><figcaption>High level architecture of a chatbot that can answer general questions about the company, restaurant, such as operating hours, menu options, and health protocols. Heavy internal knowledge base is not part of the archi.</figcaption></figure><h4>3. THE COPILOTS</h4><p>The third category is the “Copilots and Duets,” inspired by Microsoft and Google’s versions. These are assistants that utilize customer data and knowledge to help users achieve their goals. There are two types of copilots: those embedded in software or SaaS applications such as email server software, GitHub, etc., and those used for education and guidance. The former is favored by large companies with extensive customer data, while the latter is suitable for startups venturing into the learning and education sector.</p><p>Prominent examples include Github Copilot, M365 Copilot, or Google Duet. AI is embedded across Google Workspace apps, such as Google Sheets, Docs, Slides, Gmail, and Meet. It can assist with tasks like creating documents, spreadsheets, presentations, emails, and meeting notes. It can also answer questions, provide suggestions, and adjust the tone and style of your writing.<br>In addition to these AI assistants offered by tech giants, copilots are emerging as AI assistants integrated within existing platforms to better assist users in navigating an educational or e-commerce platform. Examples include Personalized AI-Powered Tutors such as Khanmigo, Duolingo Max, Jasper AI, or HarveyAI.</p><p>While there isn’t much public information on the architecture of MS copilots, much of the Copilot mechanism depends on the Microsoft Search feature in Microsoft 365. This feature will build a semantic index based on data being indexed from Microsoft 365 and can also include third-party data connected via Graph. Custom federated data sources can also feed into the Search Engine from data sources such as Dynamics or Azure Cognitive Search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EazwQ8aV2Lw9JsN1jZx3IA.png" /><figcaption><a href="https://techcommunity.microsoft.com/t5/copilot-for-microsoft-365/microsoft-365-copilot-design-and-architecture/m-p/3859228">Microsoft 365 Copilot design and architecture</a></figcaption></figure><p>The data flow in the architecture can be broken down step-by-step as follows:<br><strong>User Prompt:</strong> An end-user queries the system with a question, such as “How much vacation am I entitled to during the summer?” This input is received by the Azure OpenAI service, specifically the Copilot feature.<br><strong>Prompt Processing:</strong> Copilot interprets the user’s prompt. It doesn’t necessarily alter it with its own system message but understands the intent behind the question. This understanding can involve processing natural language and determining the type of information that is being requested.<br><strong>Microsoft Search Integration:</strong> Copilot, using its integration with Microsoft Search, triggers a search query across the user’s accessible Microsoft 365 data. This is where Semantic Search comes in — it uses the power of AI to understand the meaning behind the words in the query, rather than just the words themselves. It may search through documents, emails, or other data sources available within the organization’s Microsoft 365 environment.<br><strong>Access Control: </strong>The search is conducted, respecting the permissions and access levels of the user. Microsoft Search ensures that users can only see results that they have the rights to view, maintaining data security and privacy.<br><strong>Search Results: </strong>The search results are then retrieved, which might include documents, emails, or other forms of data related to the query about vacation entitlement.<br><strong>Azure OpenAI Service: </strong>These results are passed back to the Azure OpenAI service, which then uses its AI models to synthesize the information and generate a coherent and relevant response based on the data retrieved.<br><strong>Response Delivery</strong>: Finally, the Azure OpenAI service delivers the response to the user, providing a synthesized answer like “You are entitled to X number of days,” and it may reference the sources from which the information was drawn to support the answer.<br>The key here is the seamless integration between natural language processing through the Azure OpenAI service, the data retrieval and semantic understanding capabilities of Microsoft Search, and the access control provided by the organization’s Microsoft 365 environment. This architecture ensures that users receive accurate, contextually relevant, and secure information in response to their queries.</p><h4>“CHAT WITH DATA” OR RETRIEVAL AUGMENTED GENERATION (RAG)</h4><p>Large Language Models (LLMs) exhibit significant capabilities but also face challenges such as hallucination, outdated knowledge, and non-transparent, untraceable reasoning processes. Augmented Generation (RAG) has emerged as a promising solution to these issues by incorporating real-time data from external databases into LLM responses. This enhances the accuracy and credibility of the models, particularly for knowledge-intensive tasks, and allows for continuous knowledge updates and integration of domain-specific information. RAG synergistically merges the intrinsic knowledge of LLMs with the vast, dynamic repositories of external databases. Many startups have emerged in the past year offering RAG solutions to other companies. However, they face challenges in this arena, as basic, naive RAG systems offer little competitive advantage, and high-quality RAG systems require customization for specific topics.<br>Business examples of LLM use in this context include deploying a chatbot for customer service with an internal knowledge base as context for responses, implementing a RAG system as an AI contract manager within the procurement department, or employing a RAG system as an AI assistant to manage and understand metadata, among other applications.<br>The architecture of RAG systems is well established and looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*XEpXOffvimafQZF8vnqJwA.png" /><figcaption><em>Inspired from Emerging Architectures for LLM Applications(</em><a href="https://a16z.com/emerging-architectures-for-llm-applications/"><em>Link</em></a><em>)</em></figcaption></figure><p>The architecture for a Retrieval Augmented Generation (RAG) application of generative AI depicted in the image outlines how different components interact to provide the final output to the user. Here’s a natural language explanation of the data flow within such an application:</p><p><strong>Contextual Data</strong>: This is the initial data that provides context for the application. It could come from various sources and is used to inform and condition the outputs generated by the AI. This data is processed through data pipelines to ensure it is in a suitable format for use.</p><p><strong>Data Pipelines</strong>: These are systems that prepare and transform the contextual data, ensuring it is ready for processing by the AI models. This might involve cleaning, normalizing, and augmenting the data.</p><p><strong>Embedding Model</strong>: The processed data is then passed through an embedding model. This model converts the data into a mathematical representation (vectors) that the AI can understand and work with efficiently.</p><p><strong>Vector Store</strong>: The vectors created by the embedding model are stored in a vector store, which can be queried later to retrieve information when needed.</p><p><strong>Playground/WebApp Hosting</strong>: A playground/WebApp Hosting environment, where the AI models run and interact with the user, should be avilable to different type of users to proocess the prompts and examples</p><p><strong>Develop GenAI powered app</strong>: Developers use programming languages and tools (LangChain, Semantic Kernel, LLamaindex, etc) to develop the generative AI application. They integrate various APIs and use AI frameworks to harness the AI’s capabilities.</p><p><strong>LLM Cache:</strong> To improve performance, the application may cache certain outputs or data using systems like GPTCache or SQLite, so that frequent requests can be served faster.</p><p><strong>APIs/Plugins:</strong> The application may interface with external APIs or plugins to extend functionality, such as accessing proprietary APIs from companies like Zapier API, Wolfram|Alpha APIs, Weather APIs, etc.</p><p><strong>Storage:</strong> The application stores prompts and the AI’s completions to possibly use them for future queries or to improve the model through learning.</p><p><strong>Validate/Evaluate: </strong>There is a validation and evaluation step where outputs from the AI are assessed for quality and correctness using tools like Nemo or Guardrails.</p><p><strong>Output: </strong>The final output, which is the AI-generated content based on the prompts and the context, is then returned to the users.</p><h4>AUTONOMOUS AGENTS</h4><p>Autonomous agents are not a new concept. They have been successfully implemented in various fields, providing solutions to complex problems requiring decentralized control, cooperation, and autonomy. Success stories include:<br><strong>Traffic Management:</strong> Autonomous agents have been used to optimize traffic flow and reduce congestion in cities. For example, Amsterdam implemented a multi-agent system where each traffic light is considered an intelligent agent that communicates with others to optimize traffic flow. <strong>Supply Chain Management:</strong> Companies like Procter &amp; Gamble and Unilever use autonomous agents to manage their supply chains. These systems coordinate activities like procurement, production, and distribution, improving efficiency and reducing costs.<br><strong>Power Grid Management: </strong>Autonomous agents have been applied to manage power grids, enabling dynamic pricing and load balancing. One example is the PowerMatcher, a technology that balances supply and demand of electricity in the grid, which has been successfully used in the Netherlands.</p><p><strong>Distributed Data Mining:</strong> Autonomous agents have been used in distributed data mining where large amounts of data need to be processed. This is particularly useful in fields like bioinformatics, where data from different sources needs to be analyzed collectively.<br><br>The novelty with GenAI is that these agents can be powered by a Large Language Model (LLM) and their resoning capacity (ReAct, Chain of Thoughts, etc), where the LLM functions as the agent’s brain, complemented by several key components:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*5-NrUYN0ZolGNLWRMM-lDw.png" /><figcaption>Overview of a LLM-powered autonomous agent system <a href="https://lilianweng.github.io/posts/2023-06-23-agent/">(Link)</a></figcaption></figure><p><strong>Planning</strong>: The agent breaks down large tasks into smaller, manageable subgoals, enabling efficient handling of complex tasks. The agent can self-critique and self-reflect on past actions, learn from mistakes, and refine them for future steps, thereby improving the quality of final results.</p><p><strong>Memory: </strong>I would consider all the in-context learning (See Prompt Engineering) as utilizing the short-term memory of the model to learn. Long-term memory provides the agent with the capability to retain and recall (infinite) information over extended periods, often by leveraging an external vector store and fast retrieval.</p><p><strong>Tool use:</strong> The agent learns to call external APIs for extra information that is missing from the model weights (often hard to change after pre-training), including current information, code execution capability, access to proprietary information sources, and more.</p><p>These examples highlight the flexibility and effectiveness of autonomous agents in managing complex tasks across different sectors. Finally, we have autonomous agents, designed to perform tasks independently using reasoning and decision-making capabilities. Much like a team of specialized agents, these autonomous agents need to be highly modular, with multiple mini agents performing specific tasks. An interesting feature of these agents is their ability to self-reflect and improve their performance, marking them as a promising area for future development.</p><p>In conclusion, the five categories of LLM applications are traditional NLP tasks apps, conversational agents, copilots and duets, chat with data apps, and autonomous agents. Each category has its unique characteristics and challenges, and understanding these can guide developers in their journey of app development. Stay tuned for future content that will delve into how to build applications within these categories. As always, reader engagement and feedback are welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cc4b007261d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embracing the Future of Enterprise AI with Foundation Models II]]></title>
            <link>https://medium.com/@aidatamotion/embracing-the-future-of-enterprise-ai-with-foundation-models-ii-4bef1097bd4a?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bef1097bd4a</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Tue, 16 May 2023 16:02:13 GMT</pubDate>
            <atom:updated>2024-01-09T10:52:55.681Z</atom:updated>
            <content:encoded><![CDATA[<h3>Embracing the future of enterprise AI with foundation models II</h3><p><a href="https://medium.com/@hzmarrou/embracing-the-future-of-enterprise-ai-with-foundation-models-i-f5bc659acf3e">In Part I of this series</a>, we explored the need for a <strong><em>paradigm shift</em></strong> from developing individual AI models for specific tasks to adopting Enterprise Foundation Models that can serve multiple tasks. We also discussed the limitations and failure modes of the current approach to building AI and ML models in enterprises. In this second part, we will delve into the steps for initiating and implementing this paradigm shift within your organization.</p><h3><strong>Centralize all NLP and NLU Activities</strong></h3><p>To begin the process of adopting an Enterprise Foundation Model, it is crucial to centralize all NLP and NLU activities within your organization. This can be achieved by:</p><ul><li><strong>Consolidating existing NLP and NLU models</strong> through a central service offered through robust APIs such as Azure OpenAI, OpenAI, Cohere, or similar. These central APIs are able to serve most of NLP services such as language translation; text summarization; topic classification; entity resolution, etc</li><li><strong>Centralizing access to Foundation Models</strong> to serve different teams in the business, streamlining collaboration and resource allocation.</li><li>Establishing a <strong>central Foundation Models team</strong> responsible for assessing the security, ethicality, legality, and robustness of these models, educating users on their proper use, and maintaining frameworks, such as LangChain and Semantic kernel, for building and integrating software applications around the foundation model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*hNRvxYKN2G8ZLvW4xvYfyw.png" /><figcaption>Figure 1: Illustration of the current way of doing NLP vs the upcoming way of doing NLP and NLU.</figcaption></figure><h3>Establish a Model Marketplace</h3><p>Creating a Model Marketplace, see figure 2, is the next step in adopting an Enterprise Foundation Model. This marketplace should consist of a model base or a model registry, a feature store, and AutoML tools.</p><ul><li><strong>Model base or model registry:</strong> A model base or registry serves as a centralized storage facility for existing ML models, complete with a catalog that enables users to search for models that can predict their desired targets. This comprises both model metadata and model objects. Any model that is published in the registry must be accompanied by a Model Card, ensuring transparency and accountability in the model’s usage.</li><li><strong>Feature Store:</strong> A centralized repository that stores a catalog of features and labels, making it easy to create training or scoring sets. The feature store serves to quickly build and deploy a not-yet-existing model or retrain an existing model using a model recipe.</li><li><strong>AutoML tools: </strong>such as Auto-SKLearn and MLBox automate the model creation process, through a mode lrecipe, for specific tasks where a model does not yet exist or when an existing model needs to be retrained. This approach allows for the creation of new versions of current ML models, making the process more efficient and effective.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*Uh4di0w5d2lrX7gceUE_SA.png" /><figcaption>Figure 2: Illustration of the Model Market Place</figcaption></figure><p>The Model Marketplace should support REST APIs for integration with the Enterprise Foundation models.</p><h3>Stacking the Model Marketplace on top of the Foundation models</h3><p>At this stage, you should have:</p><ul><li>Consolidated all NLP models in a central Enterprise Foundation Model dedicated to serving all NLP/U tasks in the company.</li><li>Created a central Model Marketplace to store the smaller AI models that didn’t fall under the foundation model umbrella.</li><li>Non-AI components responsible for tasks such as user interface, data storage, and interaction with other systems.</li></ul><p>The next step is to let the foundation model act as a coordination connector between external components, i.e., the Model Marketplace and non-AI components. The foundation model will invoke the ML models from the model marketplace and other tools to solve tasks that are not possible with only the Enterprise Foundation Model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*WMkZALzh7csmvfH748pvyw.png" /><figcaption>Figure 3: Illustration of the integration of the foundation models with other Enterprise AI and non-AI components</figcaption></figure><p>Examples of such Large language models that have utilized this concept for great purposes are BabyAGI and AutoGPT. Both LLMs recursively call themselves, with BabyAGI training and evaluating various AI agents in a simulated environment, and AutoGPT using GPT-4 and GPT-3.5 via API to create full projects by iterating on its own prompts.</p><h3>Choosing ML Models from the Model Marketplace</h3><p>The concept of leveraging the power of Foundation Models, or LLMs, to perform tasks beyond their intended use (prompt &gt; LLM &gt; Answer) is rapidly evolving. It is critical to exercise caution before adopting these next-level LLMs. In the early stages, it is recommended to limit the Foundation Model’s access to the Model Marketplace (MMP), particularly by starting with ML models in de MMP that pose minimal integration and business risk. These models could include those that are exclusively designed for internal use, such as extracting metadata or performing data quality checks. By gradually expanding the scope of the Foundation Model’s capabilities, organizations can avoid the potential risks and complications associated with rapid adoption.</p><h3>Architecture evolution</h3><p><strong>Present Architecture: </strong>Numerous ML models and non-ML components — The existing architecture of ML systems typically consists of both ML components (i.e., ML models) and non-ML components. These elements coexist within the ML system’s structure and interact to enable proper functioning. ML models handle data processing and inference, while non-ML components deal with tasks such as user interfaces, data storage, and interactions with other systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*hHOZxWQPWILjol-mw8MUaw.png" /><figcaption>Figure 4: ML Architecture evolution: from” Many AI models + many non-AI components” to ”foundation model as a coordination connector with Few ML Models and many non-AI components”</figcaption></figure><p><strong>Upcoming Architecture</strong> — Enterprise Foundation model as a coordination link: 1 foundation model, fewer small ML models, and numerous non-ML components — In this proposed architecture, the foundation model serves as a link between external elements, including smaller ML models or non-ML components. The foundation model’s primary function in this capacity is coordination. It actively calls APIs provided by third-party applications to perform tasks, such as having GPT decide when to call certain tools for specific problems. In this structure, prompt engineering plays a crucial role in the early stages to guide foundation models in generating high-quality responses. However, as the foundation model evolves, the need for prompt engineering will gradually diminish and eventually disappear.</p><h3>Challenges of LLM applications for production</h3><p>Throughout computer history, engineers have primarily used programming languages to write instructions. These languages tend to be highly precise, and any ambiguity can lead to frustration or even animosity among developers (consider dynamic typing in Python or JavaScript). In contrast, prompt engineering uses natural languages, offering more flexibility but potentially resulting in a negative developer experience.</p><p>This flexibility originates from two aspects: the way users create instructions and the manner in which LLMs respond. Firstly, user-defined prompts’ flexibility can cause silent failures. For example, if a user unintentionally modifies code by inserting or deleting characters, an error is likely to occur. However, if a prompt is accidentally altered, it may still function but produce drastically different results.</p><p>While user-defined prompt flexibility might be irritating, the ambiguity in LLM-generated responses can be a major issue. It presents two challenges:</p><ul><li><strong>Ambiguous output format</strong>: downstream applications relying on LLMs require specific output formats for parsing. Although prompts can be designed to clearly indicate the desired format, there’s no guarantee that outputs will consistently adhere to it.</li><li><strong>Inconsistent user experience: </strong>users expect a degree of consistency when using applications. For instance, it would be frustrating if an insurance company provided varying quotes each time you visited their website. LLMs are stochastic, meaning they cannot guarantee identical outputs for the same input.</li></ul><p>Setting the temperature to 0 can force an LLM to produce consistent responses, but it doesn’t necessarily foster trust in the system. It’s akin to a teacher providing consistent scores only when sitting in a particular room, while giving wildly different scores in other rooms.</p><p>To address this ambiguity issue, OpenAI is actively working on enhancing their models’ reliability, providing a notebook with helpful tips. Some individuals who have worked with LLMs for an extended period have embraced the ambiguity and adjusted their workflows accordingly. Although it requires a different mindset than developing deterministic programs, it is not an insurmountable challenge.</p><p>Applying rigorous engineering principles can help mitigate this ambiguity. The remainder of this post will explore how to make prompt engineering more systematic, if not entirely deterministic.</p><h3><strong>Conclusion</strong></h3><p>In conclusion, Foundation Models are poised to become the backbone of future AI systems. By adopting an Enterprise Foundation Model, organizations can streamline their AI and ML efforts, which can lead to significant improvements in efficiency and effectiveness. Centralizing NLP and NLU activities, establishing a Model Marketplace, and stacking it on top of the Foundation models can help create a powerful AI infrastructure that benefits the entire organization.<br>In the next part of this article, we will delve deeper into how you can implement these concepts and ideas using various tools and services provided by Microsoft Azure, including Azure OpenAI, Azure ML Service, and Azure Feature Store.</p><p><em>Disclaimer: The views and opinions expressed in this article are those of the author and do not necessarily reflect the opinions or positions of any entities author represents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bef1097bd4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embracing the Future of Enterprise AI with Foundation Models I]]></title>
            <link>https://medium.com/@aidatamotion/embracing-the-future-of-enterprise-ai-with-foundation-models-i-f5bc659acf3e?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5bc659acf3e</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Tue, 16 May 2023 16:01:54 GMT</pubDate>
            <atom:updated>2024-01-10T19:23:34.241Z</atom:updated>
            <content:encoded><![CDATA[<h3>Embracing the future of Enterprise AI with foundation models I</h3><p>As AI continues to transform the business landscape, enterprises are searching for ways to capitalize on the potential of Machine Learning applications. Yet despite the billions of dollars invested in AI development over the past decade, many organizations have been unable to successfully deploy and scale AI models, resulting in a lack of meaningful ROI. This is even though hundreds of thousands — and potentially millions — of data scientists and ML engineers have been trained to develop models using thousands of different technologies and tools. In addition, numerous books and resources have been published to help ML teams accelerate the deployment and consumption of these models, while many operating models have been tested to shift from pilot projects to enterprise-level AI.<br>Despite all these efforts, many organizations still view AI as expensive, risky, and complex, leading to a lower priority for AI initiatives. In fact, many data scientists are leaving their jobs to pursue other career paths, such as data engineering or software engineering, or becoming consultants. Clearly, the current approach to AI in the enterprise environment is not delivering the desired results.</p><p>So, what’s the solution? The recent progress in Large Language Models (LLMs) like <strong>GPT-4</strong>, <strong>LaMDA</strong> or <strong>LLaMA</strong> and the release of chatbots like ChatGPT and Bard, have caught the attention of many worldwide. As a result, there is a growing trend towards using Large Language Models and more generally Foundation Models as the fundamental building blocks for most future AI systems. To make the most of this trend, AI leaders should develop what I call “<strong>Enterprise Foundation Models (EFM)</strong>” that uses models trained on both structured and unstructured data, see part II for the explanation. These models should fulfill a wide range of tasks and be available through APIs to easily integrate with other systems and applications. By building <strong>EFMs </strong>on top of LLMs and making them available through APIs, organizations can streamline their AI development and deployment processes, reduce costs, and improve ROI. Overall, leveraging the power of LLMs has the potential to accelerate the deployment of AI models across the enterprise.</p><figure><img alt="Foundation Model (FM) was coined by Stanford researchers to introduce a new category of ML models. They defined FMs as models trained on broad data that can be adapted to a wide range of downstream tasks." src="https://cdn-images-1.medium.com/max/940/1*SqfpcL9V-tC33e0WYwiTuA.png" /><figcaption>The term Foundation Model (FM) was coined by Stanford researchers to introduce a new category of ML models. They defined FMs as <strong>models trained on broad data that can be adapted to a wide range of downstream tasks.</strong></figcaption></figure><h3>The current state of Enterprise Machine Learning and AI</h3><p>Machine Learning (ML) and AI have been gaining significant traction in the enterprise world over the past decade. However, non-digital native organizations such as banks, airlines, and manufacturers are still struggling in adopting these technologies on a scale. Some organizations are still in the <strong>early stages </strong>of exploring what AI is and what it can bring to the organization. Others have progressed beyond exploration and are experimenting with <strong>Proofs of Concept (POCs)</strong> and pilots. Finally, some organizations are moving from POC/pilot to <strong>implementing AI solutions</strong> in production. However, even at this stage, keeping AI solutions valuable still requires significant organizational work. <br>All these hurdles at different stage of AI lifecycle can be attributed to the complexity of the AI life cycle and decision-making process, which requires careful consideration from discovery to implementation for each <strong>unique use case</strong>, see figure 2. The process involves multiple dimensions, including data, processes, organization, and technology, which must interact seamlessly to achieve the desired outcome. This requires significant investment in resources, capabilities and the williness the change. Deploying a new ML model can take more than six months, and even longer for organizations with complex data environments or legacy systems. Moreover, organizations must have the necessary operating model, infrastructure, tools, and skills to support the implementation and management of AI solutions. Developing a machine learning application is a complex undertaking, as shown in the accompanying image, and the model’s relevance may become outdated within months or even a year of development.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*j8-lnq5UrBaxIbc4uaeCRQ.png" /><figcaption>Figure 2, source : <a href="https://ml-ops.org/content/end-to-end-ml-workflow">https://ml-ops.org/content/end-to-end-ml-workflow</a></figcaption></figure><h3>Current State of Enterprise AI</h3><p>Let’s delve into the limitations that come with the current approach of developing and deploying Machine Learning models by examining their characteristics from multiple perspectives including data, process, organization, and technology.</p><p><strong>Data</strong>: The first step in any machine learning or data science project is to identify and collect the necessary data sources. Nowadays, companies routinely accumulate large amounts of data about their customers, sales, employees, and other relevant factors. In the most more advanced organizations, these data sources are combined into a centralized data platform that can be accessed by the machine learning team once a data sharing agreement has been established, usually on a <strong><em>case-by-case</em></strong> basis. In cases where such a data platform does not exist, the machine learning team will have to manually retrieve the required data sets from various applications where they are stored. In such cases, data access management is also agreed upon between the data user and the application owner.</p><p><strong>Process</strong>: Developing and deploying machine learning (ML) applications is a complex and multifaceted process. In the more advanced teams, this process begins by defining a portfolio of AI use cases, assessing their value and complexity, categorizing them based on their AI risk and required governance level, and finally prioritizing them for execution. Once the AI use case portfolio is defined, both overall portfolio management and individual use case management begin in a traditional project management manner. However, experience has shown that only a small fraction of the use cases in the portfolio reaches the production phase, and even fewer are used for decision-making. In addition to the process of defining and executing the portfolio of use cases, many organizations, especially those that are highly regulated, have mandatory processes for developing and deploying data-related applications. This process includes passing through a series of checks, such as reviews by privacy and legal offices, governance bodies like the architecture board, model risk validation, and others. Successful organizations, have a high agility, use design thinking techniques to rapidly experiment, test assumptions and create a steep funnel for shorter disinvestments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*RWq4QWZcjrEFTWVtVpI0cA.png" /><figcaption>Figure 3: Portfolio of AI use cases are managed on a case-by-case basis.</figcaption></figure><p><strong>Organization</strong>: Developing effective Machine Learning applications necessitates well-structured teams. As ML is an emerging field, the creation of ML-powered applications demands close collaboration between business and IT departments. This evolving landscape presents a distinctive challenge for organizations striving to integrate AI experts and teams into their organizational structure. There are several models available for organizing data science teams, varying from individual data scientists working within specific business units to centralized Data &amp; IT teams supporting the entire organization. Each of these models possesses its own advantages and disadvantages and is accompanied by distinct challenges.<br>Organizations must evaluate which approach works best for their existing structures and culture. Properly organizing ML teams is critical to developing successful applications that leverage ML, and organizations must carefully consider which model will enable them to achieve their goals.<br><strong>Technology</strong>: In the AI technology market, there are various technologies specifically designed for enterprise AI, and each has its own advantages and disadvantages. Some of the commonly used technologies include Azure ML service, SageMaker, DataRobot, and Google Cloud AI Platform. These technologies provide a variety of tools and features to streamline the process of developing and scaling machine learning models, including data preparation, model training, and model deployment, which are widely employed by data scientists and ML engineers. In addition to utilizing AI engineering technologies, organizations must also possess specialized knowledge and expertise in AI, which encompasses a thorough understanding of the different machine learning and deep learning algorithms and their specific purposes.</p><h3>Failure modes of this approach to Enterprise AI</h3><p>The approach of developing and deploying AI models through a portfolio and <em>case-by-case basis</em>, as shown in figure 3, can be effective for organizations with a simpler decision-making process, quick access to new data sources and technology choices, and the ability to bring about organizational changes. This approach is ideal for ML applications that offer significant returns on investment, like the recommendation engines of Netflix, Spotify, and other e-commerce firms. Nevertheless, it may not be a good fit for legacy companies with intricate internal procedures, fixed technology and architecture, and ML-powered applications that do not scale enough to justify investment in optimizing internal processes. Unfortunately, more than 70% of AI initiatives fail to make it to the production stage, and even successful initiatives struggle to cover the costs of investment, including the maintenance team’s salaries, according to various surveys.<br>This approach frequently fails due to various pressure points, including issues with data, processes, organization, and technology.<br><strong>Data:</strong> The process of requesting and accessing data for each individual use case, eventually after establishing a data contract or data sharing agreement between the data science team (data user) and data owner, can be both cumbersome and risky. Firstly, identifying the required data sets for a particular use case can be a time-consuming process. Secondly, miscommunication between the data user and data owner may hinder their ability to agree on the use of the data. Data owners may be hesitant to grant permission for ML use cases, while the data science team may struggle to provide assurances that the data will be handled responsibly. Moreover, this approach carries inherent risks since the identified and collected data may not be suitable for training a useful model. This can lead to wasted time and resources, and ultimately, an unsuccessful AI initiative. <br><strong>Process: </strong>At present, delivering an ML portfolio on a <em>case-by-case</em> basis entails numerous steps, each of which necessitates the involvement of individuals with a diverse range of skills, including business, research, engineering, legal, and so on. This complexity in team collaboration often leads to communication challenges, resulting in project delays and occasional setbacks in realizing the use case objectives.<br><strong>Organization</strong>: There is a structural tension between business and data science teams on one side and IT and data science teams on the other side. Business goals prioritize stability, meeting production plans, and keeping costs low. In contrast, data science teams are designed to disrupt the business, finding bigger improvements, changing decision-making processes, and creating new products. This can create tension, as data science represents the antithesis of stability.<br>Data scientists may also clash with IT organizations over their use of external tools and data sources. While IT prioritizes security and reliability, data scientists may require access to a wide range of external resources to perform their work effectively. This can cause tension, especially if data scientists are scattered throughout the organization and both blood types are in a increasing conflict: IT restricting more and Data Scientists complying less by using more and more ‘shadow IT’.<br>These challenges contrast with the exploratory and disruptive nature of data science work, which can result in a high failure rate. Some companies have even established informal rules such as “Never ask the data science team for help” and “Be aware that the data scientists are like cowboys” due to the toxic tension between the two groups.<br><strong>Technology</strong>: The complexity and immaturity of the technology landscape make it difficult to successfully deliver cases. Although many technology vendors claim that their ML solutions can solve technical challenges, the reality is that even with the most advanced and user-friendly solutions, Machine Learning engineering requires immense effort to set up the ML platform and the software needed to drive the platform. The way I lovingly describe most ML developer stacks these days is that they are “like DIY craft kits, with the instructions and 70% of the parts missing”. Most importantly, we haven’t converged yet on a dominant design for ML Platforms, which leads to a proliferation of differently scoped and shaped systems with ill-defined interfaces within the same Enterprise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*ZjeXQR4y2trwhWOpCRsbdQ.png" /><figcaption>Figure 4: ML Platforms usually come with lots of pieces missing and little to no assembly instructions (source image: <a href="https://tinyurl.com/3b5nev6h">https://tinyurl.com/3b5nev6h</a> )</figcaption></figure><p>In addition to the complexity of the technology, delivering AI use cases requires highly skilled AI engineers. These engineers must possess a deep understanding of machine learning algorithms, data science, and software engineering, who are currently in short supply. The scarcity of these professionals makes it difficult to deliver.</p><p>A negative spiral when all these pressure points start interacting: the right talent to cope with these challenges is highly educated, in hard-skills and soft-skills, to be able to steer, program, communicate and collaborate in a way that is necessary. If one of the pressure points starts to really disallow talent to make a difference, i.e. make impact and or change the organization for the good they leave. Extraordinary high salaries for data engineers and data scientists can keep organizations withstand the war on talent for some time but ROI is even further away. When good people start to leave, situations worsens and problems complexify creating a negative spiral and the ML endeavors may never result in the golden mountains that were promised.</p><p>In conclusion, it is evident that the current approach of addressing and developing ML solutions on a <em>case-by-case</em> basis or at a portfolio level, with <em>models that handle one task</em>, is no longer a sustainable approach. With the increasing complexity of data, processes organization and technology, it is essential to have another vision for Enterprise AI that is more comprehensive and integrated. This new approach should focus on creating AI models that can handle multiple tasks and operate across the entire enterprise. By doing so, businesses can leverage the power of AI to drive growth, enhance customer experiences, and streamline operations from one place. It is time to adopt a more holistic approach to AI that can support the ever-evolving demands of modern enterprises.</p><h3>Lessons learnt from the NLP progress.</h3><p>In the past, NLP machine learning use cases required different models for various tasks like classification, question answering, market intelligence or speech recognition. This process was time-consuming and required a lot of effort, resources, and data to train. However, the introduction of new techniques such as transformers and transfer learning has changed the game.<br>Nowadays, we can train a large transformer model that can serve as a “generalist” for various NLP tasks. By adding a few neural network layers to adapt the embeddings from the pre-trained models that can be adapted to a wide range of downstream tasks. This approach has made it easier and more efficient to take advantage of pre-trained models for language processing. ChatGPT is a great example of how this approach is developing and is living proof of the progress that has been made in NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*nwpoiOM7zAUM0xeXhTeZ3g.png" /><figcaption>Figure 5: The emergence of Large Language Models (LLMs) will significantly alter the way data science is conducted.</figcaption></figure><p>We no longer require separate datasets, processes, teams that , or technologies for each NLP use case. Instead, all that is needed is to make these Foundation Models (FMs) securely available within your organization and to understand how to communicate effectively with them. This communication method is commonly known as “prompt engineering” in the contemporary context. This novel approach to NLP in enterprises has the potential to revolutionize the way we handle language processing tasks.</p><h3>The Future of Enterprise AI</h3><p>The future of Enterprise AI embraces the same principle of Foundation Models, develop Enterprise Foundation Models that can be used for a variety of tasks such marketing campaign optimization, churn prediction, sentiment analysis, fraud detection or similar.</p><p>What is the solution to the failure modes and characteristics that we have discussed? In my opinion, a <em>paradigm shift </em>is necessary to address them. This shift must address the root cause of these failure modes, which can be summed up in the fact that the current approach of addressing AI on a case-by-case basis or at a portfolio level, with models that handle only one task and managing their development, deployment, and maintenance as any other project or portfolio of projects by different teams that are expected to master all aspects of the ML lifecycle, which is almost impossible in large companies. Additionally, this paradigm shift should be inspired by the recent advancements in Foundation Models (FMs) and built upon them, and by the vital techniques that were used to develop them, to create enterprise AI models based on both structured and unstructured data. Moreover, the emergence of agents and multi-agent modeling has proven to be a very useful technology in augmenting Foundation Models to perform tasks that were not originally intended for them. Popular emerging frameworks for these tasks are LangChain and Semantic Kernel.</p><p>Foundation Models were intentionally developed as general AI models, allowing them to circumvent the root cause of failure modes that arise from the current AI development process. To achieve the future state of Enterprise AI, I propose adopting the following principles:</p><ul><li><em>Develop Enterprise Foundation Models instead of individual models.</em></li><li><em>Establish one central, highly skilled cross-functional AI A-team, rather than multiple data science teams.</em></li><li><em>Users and contributors to AI models, rather than AI Developers and maintainers of AI Models</em></li></ul><h4>Foundation Enterprise Models instead of individual models</h4><p>By focusing on the development of Large Enterprise Models, organizations can establish versatile AI solutions adept at handling an extensive array of tasks, including marketing campaign optimization, churn prediction, sentiment analysis, and fraud detection. This approach promotes efficiency and minimizes the necessity for multiple specialized models, thereby streamlining maintenance, fostering knowledge sharing, and enhancing adaptability and scalability across diverse business units. Crucially, these models are readily accessible via APIs for immediate utilization.</p><h4>Establish one central, highly skilled cross-functional AI A-team, rather than multiple data science teams.</h4><p>Although the central team is very well skilled team in training and deploying machine learning models, its primary role of the central team is not to develop foundation models themselves. instead, its focus is on selecting the appropriate foundation model, such as <strong>GPT-4</strong>, <strong>LLaMA</strong>, or others, and building upon them. Consequently, the central team’s main responsibility does not involve training and deploying new models; rather, it focuses on stacking various existing models on top of base models, such as GPT-4 or similar, to create a larger, customized foundation model tailored to the enterprise. This approach is further discussed in other parts of this series. Additionally, this team is tasked with introducing new frameworks that facilitate the integration of these foundation models into existing business processes and educating different users on achieving success. The central team also plays a crucial role in constructing the model marketplace, see Part IIof this series.</p><h3>Users and contributors to AI models, rather than AI Developers and maintainers of AI Models</h3><p>The principle of having one central team developing Enterprise Foundation AI models at the enterprise level, by stacking external foundation models such GPT4 and small AI internal models, along with users and contributors to these models, brings about a transformative shift in the AI development landscape. By consolidating AI expertise within a central team, enterprises can leverage the skills of current data scientists and AI engineers to build a robust and versatile Enterprise Foundation Model. Simultaneously, other data scientists can transition into the roles of users and remote contributors, providing valuable input and expertise to further enhance and fine-tune the large models. <br>This approach requires upskilling users of Large Enterprise Models to ensure they can effectively communicate with the central team and understand how to integrate these models into their existing processes. This empowers them to harness the full potential of the AI models, thereby creating more efficient and productive workflows across the organization. <br>The rise of job roles such as AI Systems Integrators and Prompt Engineers is a testament to this paradigm shift. These professionals specialize in integrating AI models with existing systems and crafting effective prompts that enable AI models to deliver the desired output. This new breed of professionals exemplifies the evolving role of data scientists and AI engineers, as they transition from being solely developers and maintainers of AI models to becoming users and contributors who actively participate in shaping the future of AI-driven enterprises.</p><h3>Conclusion</h3><p>Considering the high rate of failure in traditional data science over the past decade and all failure modes discussed before, it is essential to embrace a new paradigm for leveraging AI systems in enterprise settings. Foundation models offer great potential as the fundamental building blocks for future AI systems, as demonstrated by NLP teams adopting them for quick time-to-market models. In Part 2, we will delve into the practical steps organizations can take to integrate Foundation models into their existing architecture.</p><p><em>Disclaimer: The views and opinions expressed in this article are those of the author and do not necessarily reflect the opinions or positions of any entities author represents.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5bc659acf3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A blue print for a data strategy VI:
Foster enablers: Technology]]></title>
            <link>https://medium.com/@aidatamotion/a-reference-guide-for-designing-and-implementing-a-data-strategy-vi-foster-enablers-technology-1338bc215bdf?source=rss-a9f5ecc0b31e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1338bc215bdf</guid>
            <dc:creator><![CDATA[DATA AI MOTION]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 15:51:12 GMT</pubDate>
            <atom:updated>2023-01-08T17:18:23.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*JJG2Ib0Ya5Trb7AGBV6N9w.png" /></figure><p>In this part of this series about designing and implementing a data strategy we will explain the key concepts related to technology why it is required for a successful data strategy. <br>Technology is a key enabler for a data strategy as it provides the tools and infrastructure necessary to acquire, manage, share, analyze, and govern data effectively. The demand for better data management tools is increasing across organizations, and technology must evolve to support the diverse needs of all users. This includes both IT experts and non-technical users who require self-service analytics capabilities.</p><h3>Build stronger tool sets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*UQBZ6T0kSrZPHQsP0xjsbw.png" /></figure><p>To build stronger tool sets, organizations need to develop capabilities in data management, analytics, and governance. Data management tools, such as data sharing and publishing apps, data integration and quality tools, metadata management solutions, and database and data lake systems, ensure that data flows to each use case and is managed effectively. Analytics tools enable a range of use cases, including traditional enterprise reporting, self-service data preparation and visualization, and advanced data modeling and simulation. Governance tools, including master data management, access policy controls, and data stewardship solutions, ensure that data is used and managed appropriately.</p><h3>Empower different user types</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*H5BVQpSZPAYEAmHB1Whxzg.png" /></figure><p>Empowering different user types, including data consumers, explorers, experts, and innovators, is crucial to ensuring that the data strategy meets the needs of all stakeholders. This requires providing the necessary tools and infrastructure for each group to access and use data in the most effective way possible.</p><h3>A cloud first strategy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/343/1*4bUF32zlM5vktR0aOwznPg.png" /></figure><p>A cloud first strategy can also be a key enabler for a data strategy, as it allows organizations to consume software, computing, and technology “as needed” and can provide greater scalability and cost-effectiveness. This can be achieved through the use of infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS) offerings.</p><p>Overall, technology is essential for enabling a successful data strategy, as it provides the tools and infrastructure necessary to acquire, manage, share, analyze, and govern data effectively. By building stronger tool sets, empowering different user types, and adopting a cloud first approach, organizations can effectively support the diverse data needs of all stakeholders.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1338bc215bdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>