<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Meroxa - Medium]]></title>
        <description><![CDATA[Meroxa is a code-first stream processing data application platform that empowers developers to build streaming apps using code. - Medium]]></description>
        <link>https://medium.com/meroxa?source=rss----b452e8fce8c7---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Meroxa - Medium</title>
            <link>https://medium.com/meroxa?source=rss----b452e8fce8c7---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:10:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/meroxa" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Solution Use Case: Accelerating AI/ML Success with Meroxa and Databricks]]></title>
            <link>https://medium.com/meroxa/solution-use-case-accelerating-ai-ml-success-with-meroxa-and-databricks-e37f822315cf?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e37f822315cf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[meroxa]]></category>
            <category><![CDATA[use-cases]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Fri, 17 Jan 2025 05:31:06 GMT</pubDate>
            <atom:updated>2025-01-17T05:31:06.530Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oQf_Wf7M-h2uz4vIUCR3MQ.png" /></figure><h3>Overview</h3><p>Organizations facing real-time data challenges can achieve <strong>up to 25% cost savings</strong> on data pipeline management while accelerating model training, improving prediction accuracy, and enhancing operational efficiency. By integrating <strong>Meroxa</strong> for seamless data movement with <strong>Databricks</strong> for scalable data processing and analytics, organizations transform their data infrastructure to meet the demands of modern AI/ML workflows.</p><h3>The Challenge: Delayed Data Access and Siloed Systems</h3><p>AI/ML models rely on timely, high-quality data to deliver accurate predictions and drive meaningful business outcomes. However, many organizations encounter the following issues:</p><ul><li><strong>Delayed Data Access</strong>: Data from critical systems — such as customer interactions, transaction logs, or marketing campaign metrics — is often processed in nightly batches. This delay results in models trained on outdated data, reducing relevance and predictive accuracy.</li><li><strong>Siloed Systems</strong>: Data resides in disparate sources like Postgres databases, Kafka event streams, and third-party platforms. Integrating these sources involves manual workflows and complex ETL processes that introduce delays and potential errors.</li><li><strong>Slow Model Development</strong>: Preparing data for ML workflows is time-consuming, often taking <strong>2–3 days per iteration</strong>, slowing experimentation and innovation.</li><li><strong>Business Impact</strong>: The lack of real-time insights impacts customer engagement and revenue. For example, abandoned carts increase, conversion rates stagnate, and opportunities for personalization are missed.</li></ul><h3>The Solution: Integration of Meroxa and Databricks</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SgRpdUc533vmD7dF.png" /></figure><p>To overcome these challenges, the combination of <strong>Meroxa</strong> and <strong>Databricks</strong> offers a modern, automated solution for real-time data ingestion, processing, and analytics.</p><ol><li><strong>Real-Time Data Ingestion with Meroxa</strong></li></ol><ul><li>Meroxa enables seamless, real-time ingestion of data from Postgres, Kafka, and APIs into an integrated pipeline.</li><li>Its <strong>developer-friendly platform</strong> allows engineering teams to build pipelines in hours rather than days.</li><li><strong>Key Benefit</strong>: Reduce data latency from <strong>24 hours to under 30 seconds</strong>, ensuring immediate availability for ML models.</li></ul><p><strong>2. Unified Data Processing in Databricks</strong></p><ul><li>Data from multiple sources is consolidated into <strong>Delta Lake</strong>, ensuring consistency and enabling low-latency querying.</li><li>Databricks’ scalable environment processes <strong>billions of daily events</strong> efficiently, even during peak loads.</li><li>Feature engineering is streamlined, supporting the creation of <strong>50+ model features</strong> without manual intervention.</li></ul><p><strong>3. End-to-End Pipeline Automation</strong></p><ul><li>Integration between Meroxa and Databricks automates the entire data pipeline, eliminating manual ETL processes.</li><li>Real-time monitoring and observability tools help reduce troubleshooting time by <strong>40%</strong>, ensuring data reliability.</li></ul><h3>The Results: Faster Insights and Enhanced Predictions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*abKShaN0eedL2ncp.png" /></figure><p>Organizations implementing the Meroxa-Databricks solution realize measurable outcomes, including:</p><ol><li><strong>Accelerated Model Training</strong></li></ol><ul><li>ML training cycles shrink from <strong>48 hours to 6 hours</strong>, enabling faster deployment and iteration of AI models.</li><li>Teams can deploy <strong>10% more models per quarter</strong>, enhancing agility and innovation.</li></ul><p><strong>2. Improved Prediction Accuracy</strong></p><ul><li>Access to real-time, high-quality data improves model accuracy by <strong>23%</strong>, boosting customer engagement.</li><li>Applications like product recommendations experience a <strong>35% increase in click-through rates (CTR)</strong>.</li></ul><p><strong>3. Operational Efficiency Gains</strong></p><ul><li>Automated workflows save <strong>30+ hours per week</strong> for engineering teams, allowing them to focus on strategic initiatives.</li><li>Integration costs decrease by <strong>25%</strong> compared to batch-based ETL processes.</li></ul><p><strong>4. Scalability for Growth</strong></p><ul><li>The system seamlessly scales to handle <strong>2x data volume growth</strong> without additional infrastructure investment.</li><li>Adding new data sources is streamlined, requiring less than a week for integration.</li></ul><p><strong>5. Business Impact</strong></p><ul><li>Conversion rates increase by <strong>15%</strong>, and abandoned cart rates drop by <strong>12%</strong>, driving immediate ROI.</li><li>Revenue from personalized insights grows by millions annually due to enhanced prediction accuracy and real-time availability.</li></ul><h3>Key Benefits</h3><ul><li><strong>Meroxa</strong>: Provides real-time, reliable data ingestion with developer-focused tools, reducing latency and manual intervention.</li><li><strong>Databricks</strong>: Delivers scalable, unified data processing and analytics, enabling organizations to build and deploy AI models efficiently.</li><li><strong>Synergy</strong>: Together, they create a powerful, automated pipeline solution that supports rapid AI/ML workflows, real-time insights, and business scalability.</li></ul><h3>Conclusion</h3><p>The Meroxa and Databricks integration transforms how organizations approach AI/ML workflows. By eliminating data silos, reducing latency, and automating pipelines, this solution delivers faster, more accurate insights that drive tangible business outcomes.</p><p>Ready to unlock your data’s full potential? <a href="https://meroxa.com/contact/sales/">Get started with Meroxa today</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e37f822315cf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/solution-use-case-accelerating-ai-ml-success-with-meroxa-and-databricks-e37f822315cf">Solution Use Case: Accelerating AI/ML Success with Meroxa and Databricks</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Bigger Isn’t Always Better: The Case for Ditching LLMs in Favor of Tiny Models Powered by…]]></title>
            <link>https://medium.com/meroxa/why-bigger-isnt-always-better-the-case-for-ditching-llms-in-favor-of-tiny-models-powered-by-2a30f7b404b2?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/2a30f7b404b2</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[meroxa]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[realtime]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 02:47:40 GMT</pubDate>
            <atom:updated>2024-12-20T02:47:40.080Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Bigger Isn’t Always Better: The Case for Ditching LLMs in Favor of Tiny Models Powered by Real-Time Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJMlQ1EWlIza6eRBzoRqZg.png" /></figure><p>As the CEO of Meroxa, I’ve had a front-row seat to the AI revolution sweeping through the enterprise technology. Companies that just came to grips with having to become a data company are now scrambling to leverage AI to optimize huge parts of their business. While large language models (LLMs) like GPT-4, Claude, Llama, and Gemini have captured the public imagination, I’m increasingly convinced that the future of practical AI applications lies in a different direction: tiny, specialized language models powered by real-time data streams.</p><h3>The Hidden Costs of Large Language Models</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YrD5L31iqt6eyExu.png" /></figure><p>Let’s be frank: LLMs are impressive, but they come with significant drawbacks. Training these models requires massive computational resources, with costs running into millions of dollars. They consume enormous amounts of energy, making them environmentally questionable. And despite their size, they still struggle with hallucinations — those confident but incorrect responses that can wreak havoc in business applications.</p><p>But perhaps most importantly, LLMs are fundamentally disconnected from your business’s current reality. They’re trained on historical internet data, not your organization’s live, operational data. This disconnect creates a critical gap between AI capabilities and business needs.</p><h3>The Tiny Model Advantage</h3><p>This is where tiny language models shine. By “tiny,” I mean models that are:</p><ul><li>Trained on specific domains rather than attempting to know everything</li><li>Updated continuously with real-time data streams</li><li>Optimized for specific business tasks rather than general-purpose conversation</li></ul><p>The advantages are compelling:</p><h3>1. Reduced Hallucinations Through Real-Time Data</h3><p>Tiny models trained on current, streaming data are less likely to hallucinate because they’re working with fresh, relevant information. When your model is continuously updated with real-time data from your actual business operations, it doesn’t need to “fill in the gaps” with potentially incorrect information.</p><h3>2. Dramatic Cost Reduction</h3><p>The economics are straightforward. Training a tiny model on a specific domain requires:</p><ul><li>Significantly less computational power</li><li>Smaller training datasets</li><li>Shorter training times</li><li>Lower ongoing operational costs</li></ul><p>We’ve seen organizations reduce their AI training costs by 90% or more by switching to domain-specific tiny models.</p><h3>3. Improved Relevancy and Accuracy</h3><p>When your model is focused on a specific domain and continuously updated with real-time data, it becomes remarkably accurate within its scope. Instead of being “okay” at everything, it becomes excellent at what matters to your business.</p><h3>Real-World Applications</h3><p>Consider a few scenarios where tiny models excel:</p><p><strong>Customer Support</strong>: Instead of using a general-purpose LLM, deploy a tiny model trained specifically on your product documentation, support tickets, and real-time customer interactions. The model stays current with product updates and emerging issues.</p><p><strong>Financial Services</strong>: Rather than relying on an LLM’s outdated knowledge, use a tiny model that continuously learns from market data, transaction patterns, and regulatory updates.</p><p><strong>Supply Chain Operations</strong>: Deploy models that understand your specific inventory, logistics, and supplier relationships, updated in real-time as conditions change.</p><h3>The Hybrid Approach</h3><p>This isn’t to say that LLMs don’t have their place. A hybrid approach often works best:</p><ul><li>Use LLMs for broad, creative tasks where general knowledge is valuable</li><li>Deploy tiny models for specific, business-critical operations where accuracy and currentness are paramount</li><li>Leverage both in combination where appropriate</li></ul><h3>The Critical Role of Data Streams</h3><p>Here’s where the rubber meets the road: tiny models are only as good as the data they’re trained on. The key to success is having robust, reliable data streams that can:</p><ul><li>Capture real-time business events</li><li>Clean and prepare data automatically</li><li>Feed models continuously for training and updates</li></ul><p>This is why at Meroxa, we’ve focused on building the infrastructure that makes this possible. Our platform enables organizations to create and manage the real-time data streams that power these next-generation AI systems.</p><h3>Reference Architecture</h3><p>To make this concrete, let’s look at a reference architecture for implementing tiny language models with real-time data streams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/0*BpryeqY9VOOJu9Y-.png" /></figure><p>This architecture shows how Meroxa serves as the foundation for real-time data processing that powers tiny language models. Let’s break down the key components:</p><ol><li><strong>Data Ingestion</strong>: Meroxa handles real-time data capture from various sources, ensuring no valuable information is lost.</li><li><strong>Stream Processing</strong>: Our Turbine engine processes and transforms data in real-time, preparing it for model consumption.</li><li><strong>Data Storage</strong>: A multi-tiered approach combines historical data for training with hot data for real-time inference.</li><li><strong>ML Pipeline</strong>: Continuous training and evaluation ensure models stay current and accurate.</li><li><strong>Monitoring</strong>: Comprehensive monitoring helps detect data drift and trigger model updates when needed.</li></ol><p>The beauty of this architecture is its ability to maintain model freshness while managing computational resources efficiently.</p><h3>Getting Started</h3><p>The path to implementing tiny models in your organization starts with your data infrastructure. Here’s what you need:</p><ol><li>Identify the specific domains where AI could add value</li><li>Map out your data sources and streams</li><li>Set up real-time data pipelines (this is where Meroxa comes in)</li><li>Start small with a focused model in one domain</li><li>Measure results and iterate</li></ol><h3>The Path Forward</h3><p>As AI continues to evolve, the winners won’t be those with the biggest models, but those with the most relevant ones. The combination of tiny models and real-time data streams represents a more sustainable, efficient, and effective approach to enterprise AI.</p><p>Ready to explore how tiny models could transform your organization? Let’s talk about how Meroxa can help you build the real-time data infrastructure that makes it possible. <a href="https://meroxa.com/contact/sales/">Sign up</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a30f7b404b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/why-bigger-isnt-always-better-the-case-for-ditching-llms-in-favor-of-tiny-models-powered-by-2a30f7b404b2">Why Bigger Isn’t Always Better: The Case for Ditching LLMs in Favor of Tiny Models Powered by…</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Government IT Modernization with Meroxa: Accelerate Your Digital Transformation]]></title>
            <link>https://medium.com/meroxa/government-it-modernization-with-meroxa-accelerate-your-digital-transformation-375347cf56da?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/375347cf56da</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[government]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <category><![CDATA[accelerator]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 17:31:56 GMT</pubDate>
            <atom:updated>2024-12-16T17:35:04.749Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RISwcK0yrpARHn4r.png" /></figure><p><strong>Empower Your Government to Work Smarter and Faster</strong></p><p>Constituents today expect government services to be as seamless and user-friendly as the digital experiences they enjoy in the private sector. But with legacy systems, siloed data, and increasing demands, many agencies face challenges in meeting those expectations.</p><p>At Meroxa, we bridge the gap between traditional government systems and modern digital agility. With our <strong>real-time data integration and movement platform</strong>, we enable governments to modernize IT infrastructure without sacrificing security, transparency, or cost efficiency. Whether you’re transforming local services, scaling state-level programs, or overhauling federal systems, Meroxa equips you to work at the speed of innovation.</p><h3>Our Government IT Solutions</h3><ol><li><strong>Real-Time Data, Real-Time Results</strong></li></ol><ul><li>Enable real-time insights and decision-making by <strong>reducing data latency by up to 60%</strong>.</li><li>Deliver on-demand services with <strong>seamless data synchronization</strong> across departments and platforms.</li></ul><p><strong>2. Modernization Without Overhauls</strong></p><ul><li>Extend the value of legacy systems with <strong>modern integrations</strong> that don’t require costly replacements.</li><li>Simplify IT transformations with our <strong>low-code platform</strong>, reducing operational complexity.</li></ul><p><strong>3. Scalability and Efficiency</strong></p><ul><li>Dynamically scale your infrastructure to handle high-volume workloads like benefits processing, public safety initiatives, and more.</li><li>Save up to <strong>30% in operational costs</strong> by automating manual workflows and optimizing resource use.</li></ul><p><strong>4. Data Security and Transparency</strong></p><ul><li>Protect sensitive constituent data with <strong>end-to-end encryption</strong> and robust access controls.</li><li>Maintain compliance with strict audit trails and data lineage features that provide visibility into every transaction.</li></ul><h3>The Challenges of Government IT Modernization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TPMHKtFwhx-hivv3.png" /></figure><p>Transforming government IT isn’t just about technology — it’s about navigating complex challenges:</p><ul><li><strong>Legacy Systems:</strong> Many agencies still rely on outdated systems that weren’t designed to support modern demands.</li><li><strong>Siloed Data:</strong> Departments often operate in isolation, leading to inefficiencies and fragmented constituent experiences.</li><li><strong>Budget Constraints:</strong> Governments need solutions that balance cost, performance, and scalability.</li></ul><p>Meroxa helps governments overcome these hurdles by enabling <strong>real-time data movement</strong>, modern integrations, and flexible infrastructure upgrades — all without the need for costly, full-scale replacements.</p><h3>How Meroxa Powers Government IT Transformation</h3><h3>1. Real-Time Data Integration</h3><p>Governments generate and rely on massive amounts of data — but disconnected systems create bottlenecks. Meroxa unifies data flows, enabling real-time communication between legacy systems, new applications, and external platforms.</p><p><strong>Key Features:</strong></p><ul><li>Seamlessly integrate databases, CRMs, and analytics tools across departments.</li><li>Power faster decision-making with real-time insights into mission-critical operations.</li><li>Ensure scalability with <strong>dynamic pipeline management</strong> that adjusts to peak workloads.</li></ul><p><strong>Example:</strong> A state health department reduced benefits processing times by <strong>50%</strong> by unifying data across welfare, healthcare, and child services systems.</p><h3>2. Cost-Effective Modernization</h3><p>Replacing aging systems outright isn’t always feasible. With Meroxa, governments can extend the functionality of legacy systems while embracing cutting-edge solutions.</p><p><strong>Key Features:</strong></p><ul><li><strong>Low-code platform:</strong> Simplify complex integrations with intuitive, developer-friendly tools.</li><li>Automate data workflows to eliminate manual errors and inefficiencies.</li><li>Enable faster rollouts of digital services while staying within budget.</li></ul><p><strong>Example:</strong> A national child support system managing over $360 million annually saved <strong>20% in operational costs</strong> by integrating existing databases with Meroxa’s platform.</p><h3>3. Better Constituent Services</h3><p>Modern constituents expect fast, personalized, and secure access to government services. Meroxa helps agencies deliver:</p><p><strong>Key Features:</strong></p><ul><li>Real-time personalization for services like benefits processing, permit applications, and public inquiries.</li><li>Streamlined interagency collaboration to reduce delays and improve outcomes.</li><li>AI-powered insights to predict and address constituent needs proactively.</li></ul><p><strong>Example:</strong> In Israel, welfare application processing times were reduced from months to hours using Meroxa’s real-time data platform.</p><h3>4. Transparent and Secure Operations</h3><p>Governments handle sensitive data daily, making security and transparency paramount. Meroxa ensures:</p><p><strong>Key Features:</strong></p><ul><li><strong>End-to-end encryption:</strong> Protect data at every step of the pipeline.</li><li>Detailed audit logs and data lineage tracking for compliance and governance.</li><li>Secure integration with monitoring tools like Splunk and OpenTelemetry.</li></ul><p><strong>Example:</strong> A federal agency managing public safety programs maintained <strong>100% compliance</strong> with strict data governance regulations by leveraging Meroxa’s security features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XyBEWNxKHysWVo65.png" /></figure><h3>Reimagine Government IT with Meroxa</h3><p>Modernizing IT infrastructure is no longer optional — it’s essential to meet the needs of today’s constituents. With Meroxa, governments can:</p><ul><li>Deliver faster, more efficient services at scale.</li><li>Achieve digital transformation without abandoning legacy investments.</li><li>Securely manage and share data across systems, agencies, and platforms.</li></ul><h3>How Can Meroxa Help You?</h3><p><strong>In the U.S.:</strong></p><p>Transform federal, state, and local government services with real-time data integration.</p><p><strong>Globally:</strong></p><p>From smart cities to social services, Meroxa empowers governments around the world to reimagine how they serve their citizens.</p><p><strong>Contact Us:</strong></p><p>Ready to modernize your IT infrastructure? Speak to a Meroxa expert today and take the first step toward a more agile, efficient government.</p><p><a href="https://meroxa.com/contact/sales/"><strong>Let’s Connect →</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375347cf56da" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/government-it-modernization-with-meroxa-accelerate-your-digital-transformation-375347cf56da">Government IT Modernization with Meroxa: Accelerate Your Digital Transformation</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meroxa’s Conduit Platform: Real-Time Data Movement at Scale, with Proven Performance]]></title>
            <link>https://medium.com/meroxa/meroxas-conduit-platform-real-time-data-movement-at-scale-with-proven-performance-ddd696d3d771?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/ddd696d3d771</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[conduit]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[meroxa]]></category>
            <category><![CDATA[streaming]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Thu, 05 Dec 2024 01:39:11 GMT</pubDate>
            <atom:updated>2024-12-05T01:39:11.603Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8_PIboJb5IdTwoA0-6fQg.png" /></figure><p>In the state of big data real-time movement isn’t just a luxury — it’s a necessity. Tools like Fivetran, while useful for batch processing, can’t deliver the performance required for real-time operations at scale. Meroxa’s Conduit Platform stands apart, providing <strong>real-time data streaming</strong> with unmatched scalability, all while empowering businesses to own and manage their data lakes and warehouses.</p><p>But how does Meroxa’s performance stack up? Let’s explore the numbers.</p><h3>Real-Time vs. Batch: The Meroxa Advantage</h3><p>Real-time processing isn’t just faster — it’s transformative. Meroxa’s Conduit Platform is optimized to deliver <strong>up to 90% faster data delivery</strong> compared to batch-based systems like Fivetran. This means businesses can act on insights almost instantaneously rather than waiting minutes or hours for batch updates.</p><h3>Performance Highlights:</h3><ul><li><strong>99.9% data delivery reliability</strong> across real-time pipelines, even during peak loads.</li><li><strong>80% reduction in data latency</strong>, enabling near-instantaneous responses to critical events.</li><li><strong>4x throughput capacity</strong> compared to traditional ETL tools, supporting millions of events per second.</li></ul><h3>Bring Your Own Data Lake or Warehouse (BYO)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xkhqQDJPigWPtePC.png" /></figure><p>Unlike Fivetran’s managed approach, which often locks businesses into proprietary systems, Meroxa’s Conduit Platform embraces flexibility. With Meroxa, you can leverage your existing data lake or warehouse infrastructure — whether that’s AWS S3, Snowflake, BigQuery, or others — while enjoying the performance benefits of real-time streaming.</p><h3>Benefits of the BYO Approach:</h3><ol><li><strong>Cost Savings</strong>: Businesses report <strong>up to 60% lower infrastructure costs</strong> by eliminating the need for redundant storage and managed services.</li><li><strong>Full Control</strong>: Maintain ownership of your data, ensuring compliance, security, and flexibility.</li><li><strong>Seamless Integration</strong>: Build pipelines tailored to your unique architecture, avoiding the limitations of one-size-fits-all solutions.</li><li><strong>Scalable Performance</strong>: Stream data directly to your data lake or warehouse, ensuring it’s ready for analysis with <strong>50% faster ingestion times</strong> compared to batch systems.</li></ol><h3>Scalability and Flexibility: Built for Enterprise Loads</h3><p>Meroxa’s Conduit Platform is built to handle the most demanding use cases. With <strong>elastic scaling</strong>, the platform easily supports high-throughput environments without sacrificing performance.</p><h3>Key Metrics:</h3><ul><li><strong>Handles up to 10 million events per second</strong>, ensuring seamless scalability for even the largest organizations.</li><li><strong>50% faster pipeline deployment</strong>, reducing time-to-market for data integration projects.</li><li><strong>Zero downtime</strong> during scaling events, guaranteeing uninterrupted operations.</li></ul><h3>Real-World Use Cases with Proven Results</h3><p><strong>Real-Time Fraud Detection</strong></p><p>A fintech company leveraged Meroxa to stream transactional data in real time to its data lake for fraud detection. The result? <strong>90% faster anomaly detection</strong>, reducing fraud losses by millions annually.</p><p><strong>Dynamic Inventory Management</strong></p><p>An e-commerce platform used Meroxa’s Conduit Platform to synchronize inventory data across warehouses in real time. This enabled <strong>95% accuracy in stock levels</strong>, minimizing missed sales and overstocking.</p><p><strong>Personalized Customer Engagement</strong></p><p>A streaming service processed user behavior data with Conduit, achieving <strong>85% faster recommendation updates</strong>, leading to a <strong>30% increase in customer engagement</strong> and retention rates.</p><h3>Meroxa vs. Fivetran: A Performance Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NXr_3krf27LYvsQS.png" /></figure><h3>Unlock Real-Time Data Movement at Scale</h3><p>Meroxa’s Conduit Platform offers unmatched real-time data movement performance, giving you the speed, flexibility, and scalability to stay ahead of the competition. Whether you’re optimizing fraud detection, inventory management, or customer engagement, Conduit delivers the power you need to make data-driven decisions faster and more efficiently.</p><p><strong>Experience performance at scale and control your data destiny — choose Meroxa.</strong></p><p><a href="https://meroxa.com/contact/sales/">Sign up today!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddd696d3d771" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/meroxas-conduit-platform-real-time-data-movement-at-scale-with-proven-performance-ddd696d3d771">Meroxa’s Conduit Platform: Real-Time Data Movement at Scale, with Proven Performance</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stale Data is Killing Your AI Models: Why Real Time Data is the Best Path Forward]]></title>
            <link>https://medium.com/meroxa/stale-data-is-killing-your-ai-models-why-real-time-data-is-the-best-path-forward-b485c58bfd78?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/b485c58bfd78</guid>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[ai-model]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Mon, 25 Nov 2024 19:30:10 GMT</pubDate>
            <atom:updated>2024-11-25T19:30:09.994Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frmNXqmVzTx54XtYBP_8SQ.png" /></figure><p>As we navigate the explosive growth of AI adoption across industries, one challenge remains persistently thorny: ensuring our AI models remain accurate, reliable, and cost-effective to maintain. At Meroxa, we’ve observed a clear pattern emerge — organizations that leverage real-time data for their AI models consistently outperform those relying on static, historical datasets.</p><h3>The Hidden Cost of Stale Data</h3><p>Most organizations today train their AI models on historical data dumps, typically refreshed weekly or monthly. While this approach might have sufficed in the past, it’s becoming increasingly inadequate in our fast-paced digital environment. Here’s what we’re seeing in the field:</p><ul><li>Models trained on outdated data are more prone to hallucinations, especially in dynamic domains like finance, e-commerce, and social media</li><li>Companies spend millions retraining models that have drifted from reality</li><li>Time-to-market for AI features is hampered by lengthy data preparation and training cycles</li></ul><h3>Real-Time Data: The Antidote to AI Hallucinations</h3><p>When AI models have access to real-time data streams, they maintain a closer connection to reality. At Meroxa, we’ve helped numerous organizations implement real-time data pipelines for their AI systems, and the results are compelling:</p><p>Our financial services clients report a 40% reduction in model hallucinations after implementing real-time data feeds. The reason is simple — when models can continuously learn from current market conditions, customer behaviors, and emerging patterns, they’re less likely to generate responses based on outdated assumptions.</p><h3>The Economic Argument for Real-Time Data</h3><p>The financial benefits of real-time data integration extend beyond improved accuracy. We’re seeing organizations achieve:</p><ol><li>Reduced Training Costs: Instead of massive, periodic retraining sessions, models can be fine-tuned incrementally with fresh data, requiring significantly less computational resources.</li><li>Faster Time-to-Market: Real-time data pipelines eliminate the need for time-consuming ETL processes and data preparation, allowing teams to deploy and iterate on models more rapidly.</li><li>Lower Infrastructure Costs: By processing data incrementally rather than in large batches, organizations can maintain smaller, more efficient infrastructure footprints.</li></ol><h3>From Theory to Practice: Implementing Real-Time Data Pipelines</h3><p>The benefits of real-time data are clear, but implementation has traditionally been a significant hurdle. This is where modern data infrastructure platforms come into play. At Meroxa, we’ve built our platform specifically to address these challenges, offering:</p><ul><li>Seamless integration with existing data sources that support the vector datatype</li><li>Built-in stream processing to automate data preparation</li><li>Automatic scaling to handle varying data volumes</li><li>Enterprise-grade security and compliance features</li></ul><h3>The Future is Real-Time</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvKnVndq38QB2RXPrdryUQ.png" /></figure><p>As AI continues to evolve and become more deeply embedded in business operations, the importance of real-time data will only grow. Organizations that invest in robust real-time data infrastructure today will be better positioned to:</p><ul><li>Deploy more accurate and reliable AI models</li><li>Respond faster to changing market conditions</li><li>Reduce their overall AI infrastructure costs</li><li>Stay ahead of competitors in AI-driven innovation</li></ul><h3>Getting Started</h3><p>The shift to real-time data doesn’t have to be overwhelming. Start by identifying one critical AI model in your organization that would benefit from fresher data. Consider the current refresh rate, the cost of retraining, and the impact of model drift on your business outcomes.</p><p>At Meroxa, we’ve helped organizations across industries make this transition successfully. Whether you’re just starting your AI journey or looking to optimize existing models, we have the expertise and technology to help you implement real-time data pipelines that drive better AI outcomes. Remember, in the world of AI, your models are only as good as the data they learn from. Make sure that data is as fresh and relevant as possible.</p><p><em>Want to learn more about implementing real-time data pipelines for your AI infrastructure? Visit </em><a href="http://meroxa.com/"><em>meroxa.com</em></a><em> or reach out to our team for a demo.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b485c58bfd78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/stale-data-is-killing-your-ai-models-why-real-time-data-is-the-best-path-forward-b485c58bfd78">Stale Data is Killing Your AI Models: Why Real Time Data is the Best Path Forward</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Case Study: Streamlining Data Flow for The Hotels Network with Meroxa]]></title>
            <link>https://medium.com/meroxa/case-study-streamlining-data-flow-for-the-hotels-network-with-meroxa-e176e1fce120?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e176e1fce120</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[meroxa]]></category>
            <category><![CDATA[real-time-data]]></category>
            <category><![CDATA[case-study]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Sun, 17 Nov 2024 17:57:07 GMT</pubDate>
            <atom:updated>2024-11-17T17:57:07.597Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNPjLSsYO4obdOWmYzKYJA.png" /></figure><h3>Client Overview</h3><p>The Hotels Network (THN) is a leading technology company in the hospitality sector, offering innovative tools to enhance guest experiences and optimize revenue. The Hotels Network (THN) approached us to help address a critical challenge — managing the data silo between their sales and support teams. This disconnect was hindering their ability to operate efficiently and gain a unified view of customer interactions. By streamlining their data flows and integrating their systems, we were able to bridge the gap between sales and support, ultimately increasing THN’s operational efficiency and enhancing its ability to deliver a seamless customer experience.</p><p>In the face of rising costs and operational complexity; THN needed a more streamlined solution that would support and scale their daily volume efficiently without compromising security.</p><h3>Challenges</h3><p>The Hotels Network (THN) faced significant challenges due to fragmented insights and disconnected data streams between their sales and support teams. This <strong>siloed data</strong> made it difficult to achieve a unified view of customer interactions and hindered efficient communication. Their <strong>complex architecture</strong>, built on multiple data streaming services, added layers of cost and complexity, making it hard to manage and scale effectively. On top of this, the organization needed to handle <strong>high volumes</strong> of data, processing around 100 daily events, each with numerous data elements, which created operational bottlenecks and inefficiencies. These challenges underscored the need for a streamlined, integrated approach to data management that could support both high throughput and seamless cross-functional insights.</p><h3>Solution Overview</h3><p>Meroxa provided THN with a comprehensive data integration solution using their</p><p><strong>Conduit Platform</strong></p><p>to streamline the flow of data between Salesforce and Redpanda. By creating a unified pipeline, Meroxa simplified the architecture, reducing operational costs and providing real-time updates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YC5T88kSg5_xM45F.png" /></figure><h3>Key Components of the Solution</h3><ol><li><strong>Salesforce Integration</strong></li></ol><ul><li>Meroxa set up a Salesforce trigger and platform event configuration to publish key events in real-time.</li><li>This ensured that customer interactions and property data were kept up-to-date without manual interventions, improving the overall flow of data from the sales to the support teams.</li><li>We’ve configured the Salesforce platform event to publish different event types, and Meroxa was able to handle them as expected using multiple Topics</li><li>Meroxa was able to adapt the JSON format to be published in Redpanda to the specifications of the customer’s team. This helped us to keep a consistent JSON notation with other integrated systems.</li></ul><p><strong>2. Redpanda Cluster Setup</strong></p><ul><li>Meroxa configured a Redpanda cluster, managing topics with secure Access Control Lists (ACL) and authentication mechanisms to protect data and ensure seamless, secure connections between the systems.</li></ul><p><strong>3. Meroxa Conduit Platform</strong></p><ul><li>The entire pipeline was managed using Meroxa’s platform, which consumed Salesforce events and streamed them into Redpanda. Secrets management for secure credentials and connection monitoring was handled through Meroxa, providing a centralized and reliable data flow infrastructure.</li><li>The solution was designed to support THN’s current data volume and had the flexibility to scale as their business grows.</li></ul><h3>Implementation Timeline</h3><ul><li><strong>Phase 1</strong>: Initial consultation and requirements gathering.</li><li><strong>Phase 2</strong>: Salesforce and Redpanda integration setup (1 week).</li><li><strong>Phase 3</strong>: Testing and troubleshooting (2 weeks).</li><li><strong>Phase 4</strong>: Full deployment and production (within 1 month from project start).</li></ul><p><strong><em>“The Meroxa team worked with us to design, build &amp; deploy an efficient low-code solution to connect our Salesforce org with our internal backend system via the Redpanda streaming platform.”</em></strong></p><p><strong>David Sanchez Carmonan</strong></p><p><em>Senior GTM Systems Manager</em></p><h3>Results and Benefits</h3><ol><li><strong>Improved Data Flow</strong></li></ol><ul><li>With the consolidated data pipeline, THN eliminated siloed data between Salesforce and Redpanda, achieving a unified, seamless data flow.</li><li><strong>Data updates are now processed in real-time</strong>, enabling faster response times and improved service levels.</li></ul><p><strong>2. Reduced Operational Costs</strong></p><ul><li>By streamlining its architecture and eliminating the need for multiple data streaming services, THN reduced operational complexity and cut costs by up to <strong>30%</strong>.</li></ul><p><strong>3. Real-Time Insights</strong></p><ul><li>Real-time data streaming from Salesforce into Redpanda allowed THN to make informed decisions quickly, enhancing customer support with more timely responses to customer interactions.</li></ul><p><strong>4. Scalability</strong></p><ul><li>The solution provided by Meroxa offers a flexible and scalable platform, allowing THN to handle current data volumes and easily expand as their needs grow.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YOdTh4qOU2lCUb8A.png" /></figure><p><strong><em>“By publishing event messages in Salesforce’s Apex to Meroxa, we’ve enabled our Client Success team to speed up the onboarding process for new clients and a more reduced number of data entry issues”.</em></strong></p><p><strong>David Sanchez Carmonan</strong></p><p><em>Senior GTM Systems Manager</em></p><h3>Conclusion</h3><p>Meroxa’s Conduit Platform has transformed THN’s data architecture by providing a streamlined, scalable, and cost-effective solution to manage their data flow between Salesforce and Redpanda. With real-time insights, reduced complexity, and lower operational costs, THN is now positioned to enhance its customer service capabilities and optimize its operations for future growth.</p><p>By consolidating multiple services into a single, efficient pipeline, THN can continue to innovate and scale, with a future-proof infrastructure capable of adapting to their evolving needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e176e1fce120" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/case-study-streamlining-data-flow-for-the-hotels-network-with-meroxa-e176e1fce120">Case Study: Streamlining Data Flow for The Hotels Network with Meroxa</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conduit Operator v0.02 with Schema Registry Support!]]></title>
            <link>https://medium.com/meroxa/conduit-operator-v0-02-with-schema-registry-support-e8f978928e68?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e8f978928e68</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[conduit]]></category>
            <category><![CDATA[data-migration]]></category>
            <dc:creator><![CDATA[dion keeton]]></dc:creator>
            <pubDate>Mon, 04 Nov 2024 19:19:08 GMT</pubDate>
            <atom:updated>2024-11-04T19:19:08.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSvE6IJ6mqDDnN0k8V4RNg.png" /></figure><p>We are thrilled to announce the release of <strong>Conduit Operator v0.0.2</strong>, designed to simplify the management and orchestration of Conduit instances within Kubernetes.</p><h3>What Is the Conduit Operator?</h3><p>The <strong>Conduit Operator</strong> extends the Kubernetes API, allowing users to manage Conduit instances as custom resources. These resources define how each Conduit pipeline is provisioned and managed throughout its lifecycle, giving you full control over your data flow while leveraging Kubernetes-native features like scaling, monitoring, and logging.</p><p>Conduit pipelines can be declared using YAML configuration, much like how any other Kubernetes resources are configured and deployed. This flexibility allows you to integrate your data streaming processes into existing DevOps workflows and infrastructure management tools with ease.</p><h3>A Glimpse into Conduit Custom Resources</h3><p>Conduit pipelines are represented as Kubernetes custom resources, where each pipeline runs as its own distinct Conduit instance. Below is a basic example of how a Conduit pipeline is defined:</p><pre>apiVersion: operator.conduit.io/v1alpha<br>kind: Conduit<br>metadata:<br>  name: conduit-generator<br>spec:<br>  running: true<br>  name: generator.log<br>  description: generator pipeline<br>  connectors:<br>    - name: source-connector<br>      type: source<br>      plugin: builtin:generator<br>      settings:<br>        - name: format.type<br>          value: structured<br>        - name: format.options.id<br>          value: &quot;int&quot;<br>        - name: format.options.name<br>          value: &quot;string&quot;<br>        - name: format.options.company<br>          value: &quot;string&quot;<br>        - name: format.options.trial<br>          value: &quot;bool&quot;<br>        - name: recordCount<br>          value: &quot;3&quot;<br>    - name: destination-connector<br>      type: destination<br>      plugin: builtin:log</pre><p>This configuration provides a declarative way to manage data pipelines, reducing the manual overhead typically required to build and manage streaming architectures.</p><h3>Streamlining Connector Management</h3><p>Using standalone connector with the Conduit Operator is simplified by allowing them to be hot loaded through GitHub repositories rather than included into the instance image.</p><pre># github.com/conduitio/conduit-connector-generator will be build and loaded<br># by the operator.<br>- name: source-connector<br>  type: source<br>  plugin: conduitio/conduit-connector-generator<br>  pluginVersion: v0.8.0<br>  settings:<br>    ...</pre><p>The operator automatically provisions and manages the necessary resources to run the connectors. These connectors, sourced from organizations like <strong>conduitio</strong>, <strong>meroxa</strong>, and <strong>conduitio-labs</strong>, provide out-of-the-box integrations with popular systems.</p><h3>Schema Support for Enhanced Data Handling</h3><p>As of Conduit v0.11.0, the platform supports schema registries, which allows connectors to encode and decode data using predefined schemas. This enables more robust data management and ensures compatibility between different data systems.</p><p>An example configuration for utilizing a schema registry looks like this:</p><pre>apiVersion: operator.conduit.io/v1alpha<br>kind: Conduit<br>metadata:<br>  name: conduit-generator-schema-registry<br>spec:<br>  schemaRegistry:<br>    url: http://apicurio:8080/apis/ccompat/v7<br>    basicAuthUser:<br>      - value: &lt;schemaUser&gt;<br>    basicAuthPassword:<br>      - secretRef:<br>        key: schema-registry-password<br>        name: schema-registry-secret</pre><p>This level of flexibility is essential for businesses dealing with large-scale data integrations, as it allows multiple Conduit instances to share a schema registry across different environments and scale pipelines independently.</p><h3>Deploying Conduit Operator</h3><p>Deployment of the Conduit Operator can be done via Helm, a popular Kubernetes package manager. By using Helm charts, you can easily manage deployments, scaling, and updates of Conduit instances within your Kubernetes clusters.</p><p>To deploy the operator, you can simply run:</p><pre>helm repo add conduit https://conduitio.github.io/conduit-operator<br>helm install conduit-operator \<br>    conduit/conduit-operator --create-namespace -n conduit-operator</pre><h3>Monitoring and Scaling with Kubernetes</h3><p>One of the key advantages of the Conduit Operator is its integration with Kubernetes-native features. For example, you can add annotations to your Conduit instances to automatically scrape metrics using Prometheus:</p><p>This is achieved by customizing the Helm value file when deploying the operator. Future work will allow for these annotations to be placed directly on the Conduit resource.</p><pre># Create values.yaml using these settings<br>controller:<br>  conduitMetadata:<br>    podAnnotations:<br>      prometheus.io/scrape: true<br>      prometheus.io/path: /metrics<br>      prometheus.io/port: 8080<br>      <br># Install or upgrade the operator via helm<br>helm install conduit-operator \<br>    conduit/conduit-operator --create-namespace -n conduit-operator \<br>    -f values.yaml</pre><p>This seamless integration enables robust monitoring and scaling options, ensuring your data pipelines are optimized for performance and reliability.</p><h3>Why Use Conduit Platform?</h3><p>While the <strong>Conduit Operator</strong> offers a robust solution for managing data pipelines within Kubernetes, the <strong>Conduit Platform</strong> takes this further by providing a <strong>low-code experience</strong> and additional <strong>enterprise features</strong>. With the Conduit Platform, you can easily build, monitor, and scale complex data pipelines with minimal manual effort.</p><p>Key advantages of using the Conduit Platform include:</p><ul><li><strong>Low-Code Interface</strong>: Quickly configure and manage pipelines without extensive coding.</li><li><strong>Enterprise Features</strong>: Enhanced security, monitoring, and scaling options tailored for large-scale enterprise needs.</li><li><strong>Streamlined Workflows</strong>: Easily connect disparate data sources and sinks, optimizing data flow across your infrastructure.</li></ul><p>Whether you’re looking to deploy individual instances with Conduit Operator or scale enterprise-wide with the Conduit Platform, Meroxa provides the tools and flexibility to manage your data pipelines efficiently.</p><h3>Conclusion</h3><p>The <strong>Conduit Operator</strong> simplifies data pipeline management in Kubernetes environments, enabling you to easily manage data streams. For businesses looking to scale, integrate complex data systems, and optimize their operations, the <strong>Conduit Platform</strong> provides a powerful low-code solution that expands on the capabilities of the Conduit Operator.</p><p>Get started with Conduit Operator on <a href="https://github.com/ConduitIO/conduit-operator">GitHub</a> and take your data pipeline management to the next level with the Conduit Platform for a low-code, enterprise-ready experience. Also check out our <a href="https://conduit.io/docs/scaling/conduit-operator">documentation</a>.</p><p>Looking for managed platform solutions? Check out our Conduit Platform by <a href="https://https//meroxa.com/contact/sales/">requesting a demo</a>. Let’s build the future of data integration together!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8f978928e68" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/conduit-operator-v0-02-with-schema-registry-support-e8f978928e68">Conduit Operator v0.02 with Schema Registry Support!</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spire Maritime AIS Source Data Integration now Generally Available]]></title>
            <link>https://medium.com/meroxa/spire-maritime-ais-source-data-integration-now-generally-available-e5db2205bd1a?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/e5db2205bd1a</guid>
            <category><![CDATA[maritime-ai]]></category>
            <category><![CDATA[spire]]></category>
            <category><![CDATA[spire-ais]]></category>
            <category><![CDATA[meroxa]]></category>
            <category><![CDATA[spire-maritime-ais]]></category>
            <dc:creator><![CDATA[Sara Menefee]]></dc:creator>
            <pubDate>Wed, 24 May 2023 19:17:37 GMT</pubDate>
            <atom:updated>2023-05-24T22:09:16.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*KTwRbwm685FQT0Ro.png" /></figure><p>We are excited to announce general availability of the source data integration with <strong>Spire Maritime AIS </strong>. Meroxa customers can stream maritime activity from the <strong>Spire Maritime 2.0 GraphQL API </strong>, transform that data using function code, and deliver to any downstream destination in real-time.</p><p>This is a first of its kind source data integration with Spire Maritime AIS that works natively with a stream-processing data platform.</p><p><a href="https://spire.com/maritime/"><strong>Spire Maritime AIS </strong></a>delivers real-time global maritime activity information by using a constellation of satellites and terrestrial sensors that track and transmit vessel and ship signals to provide their location, routes, and movements.</p><p>Organizations worldwide analyze and process insights from the Spire AIS APIs and TCP stream to help with global logistics, collision avoidance, surveillance, fishery management, and environmental monitoring.</p><p><a href="https://meroxa.com/"><strong>Meroxa </strong></a>is a Stream Processing Data Application Platform as a Service that enables developers to build and run stream-processing data applications that respond to real-time data and events while managing all of the underlying infrastructure required to scale stream-processing workloads.</p><p>The Meroxa platform manages the underlying infrastructure required to scale stream-processing data applications and was designed to work natively with our own in-house designed, easy-to-use application framework called Turbine. Turbine enables developers to quickly build using popular programming languages, such as Python, JavaScript, Ruby, and Go, without needing to write domain-specific code.</p><h3>Getting started with Spire Maritime AIS on Meroxa</h3><p>To stream maritime activity from Spire Maritime AIS on the Meroxa Platform, you must have an existing Spire AIS client account. If you are not already a client and wish to purchase Spire AIS products, you should <a href="https://spire.com/talk-to-sales/">contact the Spire AIS team </a>directly.</p><p>Once you have a unique API token, login to your <a href="https://auth.meroxa.io/login?state=hKFo2SBVdHNFUFNGLUtoNE4yOVNZSGV5VTZjRDJsRWJoVWJWeaFupWxvZ2luo3RpZNkgQ3loN3NNY2NKSk5EU19RZ3gtQ0c2WDRwZlk4YVRFem6jY2lk2SBUeTJQeUxiZGFoNnBJcVJaaXEzdXhod0Exdmh2ZzZDNg&amp;client=Ty2PyLbdah6pIqRZiq3uxhwA1vhvg6C6&amp;protocol=oauth2&amp;redirect_uri=https%3A%2F%2Fdashboard.meroxa.io%2Fcallback&amp;audience=https%3A%2F%2Fapi.meroxa.io%2Fv1&amp;scope=openid+profile+email+user&amp;response_type=code&amp;response_mode=query&amp;nonce=U2dGN3FNTmQtTENaOWYyVm1uTXNlXzNvemFLN08zYTEtMzNjR3E1Q3NCVA%3D%3D&amp;code_challenge=4kMrf0dlxTltWmY3GwdwQ9F8fCpjCP-m6UG5s8cURi0&amp;code_challenge_method=S256&amp;auth0Client=eyJuYW1lIjoiYXV0aDAtc3BhLWpzIiwidmVyc2lvbiI6IjEuMTQuMCJ9&amp;mode=login">Meroxa account </a>and <a href="https://dashboard.meroxa.io/resources/new?type=spire_maritime_ais">create a Spire Maritime AIS resource </a>. Don’t have a Meroxa account? to get started.</p><h3>Create a Spire Maritime AIS Resource</h3><p>In order for a Turbine streaming application to securely connect with the <strong>Spire Maritime 2.0 GraphQL API</strong> as a source, a Resource must be created.</p><p>Resources are used by the Meroxa platform to abstract sensitive credentials away from the Turbine application code. In this section, we’ll guide you through the steps on how to create <strong>Spire Maritime AIS </strong>Resources.</p><p>As mentioned earlier, we require a unique <strong>Spire Maritime 2.0 GraphQL API token</strong> to create a Meroxa resource. This can be acquired by contacting a representative at Spire AIS. Once you have received your API token, login to your <a href="https://auth.meroxa.io/login?state=hKFo2SBVdHNFUFNGLUtoNE4yOVNZSGV5VTZjRDJsRWJoVWJWeaFupWxvZ2luo3RpZNkgQ3loN3NNY2NKSk5EU19RZ3gtQ0c2WDRwZlk4YVRFem6jY2lk2SBUeTJQeUxiZGFoNnBJcVJaaXEzdXhod0Exdmh2ZzZDNg&amp;client=Ty2PyLbdah6pIqRZiq3uxhwA1vhvg6C6&amp;protocol=oauth2&amp;redirect_uri=https%3A%2F%2Fdashboard.meroxa.io%2Fcallback&amp;audience=https%3A%2F%2Fapi.meroxa.io%2Fv1&amp;scope=openid+profile+email+user&amp;response_type=code&amp;response_mode=query&amp;nonce=U2dGN3FNTmQtTENaOWYyVm1uTXNlXzNvemFLN08zYTEtMzNjR3E1Q3NCVA%3D%3D&amp;code_challenge=4kMrf0dlxTltWmY3GwdwQ9F8fCpjCP-m6UG5s8cURi0&amp;code_challenge_method=S256&amp;auth0Client=eyJuYW1lIjoiYXV0aDAtc3BhLWpzIiwidmVyc2lvbiI6IjEuMTQuMCJ9&amp;mode=login">Meroxa account </a>and create a <strong>Spire Maritime AIS</strong> resource in one of two ways:</p><p><strong>Meroxa CLI</strong></p><p>In the following example, we create a Spire Maritime AIS resource named <strong>my-spire-ais </strong>. Resources names may contain lowercase letters, numbers, underscores, and hyphens. We recommend that you choose something easy to identify as this will be used to refer to your Spire Maritime AIS resource when writing your Turbine application code.</p><p>Using the by running the following command:</p><pre>meroxa resource create my-spire-ais \ --type spire_maritime_ais \ --token $SPIRE_MARITIME_AIS_API_TOKEN</pre><p>Replace the <strong>$SPIRE_MARITIME_AIS_API_TOKEN </strong>placeholder in the example command above with the API token provided by the Spire AIS team. When you’re ready, simply hit return and wait for confirmation through the Meroxa CLI that the resource has been successfully created.</p><p><strong>Meroxa Dashboard</strong></p><p>Below are the steps required to create a Spire Maritime AIS resource using the <a href="https://dashboard.meroxa.io/resources/new?type=spire_maritime_ais">Meroxa Dashboard </a>:</p><p>Resources can be updated in the Meroxa dashboard by going to the tab in the dashboard and clicking on the <strong>Spire Maritime </strong>resource you’d like to update. You can update</p><p>We do not display credentials, such as the API token, in any of our interfaces. However, if you need to update the API token at any time, you can do so in the Meroxa Dashboard or the Meroxa CLI.</p><p>A notification in the dashboard will appear once your Spire Maritime AIS resource has been successfully created.</p><h3>Using Spire Maritime AIS as a Source with Turbine</h3><p>Now that a Spire Maritime AIS resource has been created, you can use the Turbine application framework to stream and transform data in real-time directly from the <strong>Spire Maritime 2.0 GraphQL API</strong> to any destination. To do this, you must have the Meroxa CLI installed.</p><p>In the following examples, we will demonstrate how to do this with JavaScript using TurbineJs.</p><p>First, initialize a Turbine streaming app by running the following command in the :</p><pre>meroxa app init my-spire-app --lang javascript</pre><p>A printed confirmation will let you know when you have successfully initialized your Turbine streaming app, meaning the application project files will be created in your current local directory. You may also include a <strong>— path </strong>argument at the end of the command to provide an alternative local path.</p><p>Next, run a command to get to the root of the Turbine application project:</p><pre>cd my-spire-app</pre><p>Within the project directory, you will find an <strong>app.js </strong>file. Open this with your preferred code editor. There you will see self-documented boilerplate code with a custom function written in JavaScript to execute against the example data record set provided in the fixtures directory.</p><p>To use the Spire Maritime AIS resource, directly pass its name ( <strong>my-spire-ais </strong>) as the only argument to the method.</p><p>To represent the source data stream, a method is used. Because there is no concept of a collection of data with the <strong>Spire Maritime 2.0 GraphQL API</strong>, simply pass through an empty string:</p><pre>exports.App = class App { <br>  async run(turbine) { <br>    let source = await turbine.resources(&quot;my-spire-ais&quot;); <br>    let records = await source.records(&quot;*&quot;); <br>   } <br>};</pre><p>That’s all it takes to get real-time maritime data streaming into your Turbine application.</p><p>There are a couple of additional configurations that can be defined in your Turbine application code.Source Configurations</p><p>The following source configurations are supported:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*QxyZ1WyNh3jXrLVa-NvgEw.png" /></figure><h3>What’s Next?</h3><p>All that is left is for you to write function code to transform your Spire Maritime AIS stream data and event records to a downstream set of data stores, databases, or third-party APIs. Imagine what you can do with the power of Spire Maritime AIS at your fingertips.</p><p>Need potential ideas for Turbine streaming apps? Check out our example <a href="https://github.com/meroxa/turbine-examples">Turbine app examples </a>to get started. But don’t let these examples hinder you. There is no limit to what you and your team can achieve using Spire Maritime AIS and the power of the Meroxa platform.</p><p>We can’t wait to see what you build! 🚀</p><p>As always, if you need help, have questions, or just want to chat:</p><ul><li>Don’t have a Meroxa account? <a href="https://meetings.hubspot.com/haller/get_started">Schedule an onboarding session</a> with our team.</li><li>Have a technical question? Reach out via email at <a href="/service/mailto:support@meroxa.com">support@meroxa.com</a>.</li><li><a href="https://discord.meroxa.com/">Join our Discord</a> community.</li></ul><p><em>Originally published at </em><a href="https://meroxa.com/blog/spire-maritime-ais-source/"><em>https://meroxa.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5db2205bd1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/spire-maritime-ais-source-data-integration-now-generally-available-e5db2205bd1a">Spire Maritime AIS Source Data Integration now Generally Available</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Announcing the Oracle Database Source Data Integration]]></title>
            <link>https://medium.com/meroxa/announcing-the-oracle-database-source-data-integration-1a84375d8b6?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/1a84375d8b6</guid>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[data-migration]]></category>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[oracle-database]]></category>
            <dc:creator><![CDATA[Sara Menefee]]></dc:creator>
            <pubDate>Wed, 15 Mar 2023 23:22:24 GMT</pubDate>
            <atom:updated>2023-03-29T22:10:43.872Z</atom:updated>
            <content:encoded><![CDATA[<p>We are excited to announce that the Meroxa Platform now supports data integrations for Oracle Databases.</p><p>Oracle Database, also known as Oracle or Oracle DB, is a relational database management system (RDBMS) developed by <a href="https://www.oracle.com/corporate/">Oracle Corporation</a>. It is one of the most widely used databases in the world by large enterprise companies that require robust and dependable database solutions to store, process, and access data at a massive scale.</p><h3>Oracle Database as a Source</h3><p>Meroxa’s Turbine application framework lets you write code naturally by using Meroxa Functions to alter incoming data records and events from an Oracle Database source data stream before arriving at any downstream destinations, whether another database or system. The Turbine application framework supports programming languages such as JavaScript, Python, Ruby, and Go.</p><p>When you deploy a Turbine streaming application with an Oracle Database source, the Meroxa Platform takes an initial snapshot of the source table. Once the snapshot is complete, it begins tracking new data records and events, including <strong>INSERT</strong>, <strong>UPDATE</strong>, and <strong>DELETE</strong> operations, and pushes them into the data stream.</p><h3>Real-Time Data Streaming with Change Data Capture</h3><p>Using Change Data Capture (CDC), we can process Oracle Database data record events in real-time. We do this by creating a tracking table and a database trigger to track event records.</p><p>The tracking table and trigger name have the same names as a source table with the prefix <strong>MEROXA</strong>. The tracking table has all the same columns as the source table plus three additional columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*lCHIbt5YuZ1YJrFK8_glMg.png" /></figure><p>An event record will be written in a tracking table when data is added, changed, or deleted from an Oracle Database table. The queries retrieving these event records from the tracking table are similar to those used in Snapshot mode but with <strong>CONNECTOR_TRACKING_ID</strong>as the ordering column.</p><p>An Ack method will collect the <strong>CONNECTOR_TRACKING_ID</strong> of the event records successfully applied and are later removed from the tracking table every 5 seconds or when the connection is closed.</p><h4>Things to be aware of…</h4><ul><li>Changes sometimes need to be made to columns in an Oracle Database table. The changes must also be applied to the tracking table when this happens by your Oracle Database administrator.</li><li>All tracking information only exists within the Oracle Database. Upon deletion of the tracking table, the tracking process will restart from the beginning by initiating a new snapshot of the table, which could lead to unintended replication of data downstream.</li></ul><h3>Creating an Oracle Database Resource on the Meroxa Platform</h3><p>Customers can create Resources for Oracle Databases using the Meroxa CLI or Dashboard. You must have a Meroxa account and be logged in to your account to get started.</p><h4>Meroxa CLI</h4><p>In the following example, we create an Oracle Database Resource named <strong>my-oracle-db</strong>. Resource names may contain lowercase letters, numbers, underscores, and hyphens. Use this name to reference your Oracle Database when writing your Turbine application code.</p><p>Using the , run the following command:</p><pre>$ meroxa resource create my-oracle-db --type oracle --url oracle://user:password@host.com:1571/database</pre><h4>Meroxa Dashboard</h4><p>Below are the steps required to create an Oracle Database Resource using the <a href="https://dashboard.meroxa.io/resources/new?type=oracle">Meroxa Dashboard </a>:</p><ol><li>Navigate to the <strong>Resources</strong> tab.</li><li>Click the <strong>Add a Resource</strong> button.</li><li>Search for <strong>Oracle DB</strong> using the search bar.</li><li>Click the <strong>Add Resource</strong> button for <strong>Oracle DB</strong>.</li><li>Confirm you are on the Add a resource form with Oracle DB selected.</li><li>Provide a valid <strong>Connection URL</strong> for your Oracle Database instance (.e.g., oracle://user: password@host.com :1571/database-name)</li><li>Click the <strong>Save</strong> button.</li></ol><p>A notification in the dashboard will appear once your Oracle Database Resource has been successfully created.</p><h3>Using Oracle Database as a Source with Turbine</h3><p>Using Turbine, customers can use any Turbine-supported language such as JavaScript, Python, Ruby, or Go to stream and transform business-critical data from an Oracle Database table to any destination.</p><p>The following example demonstrates how to do this with TurbinePy using Python.</p><p>First, initialize your Turbine streaming app by running the following command in the Meroxa CLI:</p><pre>$ meroxa app init my-first-app --lang python</pre><p>You should receive confirmation from the Meroxa CLI that you’ve initialized your Turbine streaming app, meaning the application files have been created locally in the current directory.</p><p>From this point, run the following command to get to the root of the application code.</p><pre>$ cd my-first-app</pre><p>Within your Turbine streaming app you will discover a <strong>main.py</strong> file. Open this with your preferred code editor. You will see self-documented boilerplate code with a custom function written against the example data record set provided in a fixtures directory.</p><p>Look for the following code with the <strong>resources</strong> and <strong>records</strong> methods:</p><pre>source = await turbine.resources(&quot;source_name&quot;) <br>records = await source.records(&quot;collection_name&quot;, {})</pre><p>The <strong>resources</strong> method is used to specify a Resource on the Meroxa Platform. Replace <strong>source_name</strong> with the name of your Oracle Database Resource. In the following example, we’ll use the name we used when creating the Oracle Database Resource <strong>my-oracle-db</strong>.</p><pre>source = await turbine.resources(&quot;my-oracle-db&quot;)</pre><p>The <strong>records</strong> method is used to specify the respective table you wish to use as the source of data. In the following example, there is a table called <strong>transactions</strong>. Replace <strong>collection_name</strong> with the name of your Oracle Database table.</p><p>In addition, you will need to indicate which field will be used for ordering rows. This column must contain unique values that can be used for sorting otherwise, the snapshot will not work properly. In the following example, we will use the <strong>id</strong> column.</p><pre>records = await source.records(&quot;transactions&quot;,{&quot;orderingColumn&quot;:&quot;id&quot;})</pre><p>There are a few additional configurations for Oracle Database Source data integrations that can be defined in your Turbine application code. In the following example, we want to change the <strong>batchSize</strong> from its default value of <strong>1000</strong> to <strong>2000</strong>. We do this by including another key value pairing in the configuration which is the second argument of the <strong>records</strong> method.</p><pre>records = await source.records(&quot;transactions&quot;,{&quot;orderingColumn&quot;:&quot;id&quot;,&quot;batchSize&quot;:&quot;2000&quot;})</pre><p>Below is a list of the supported configurations for Oracle Database sources.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*Xo4JYUCFQtlBoo-Zuoj_fA.png" /></figure><p>You’re ready to start streaming with an Oracle Database as a source for your Turbine streaming app!</p><h3>What’s next?</h3><p>All that’s left for you to do is to write function code to transform the streaming data and event records to a downstream set of Resources or third-party APIs. Need ideas for a Turbine app using Oracle Database as a source? Check out <a href="https://github.com/meroxa/turbine-examples">our example Turbine apps </a>to get started. But don’t let this example hinder you. The sky’s the limit for what you and your team can achieve.</p><p>We can’t wait to see what you build! 🚀</p><p>As always,</p><ul><li>If you need help or have questions, please reach out at <a href="/service/mailto:support@meroxa.com">support@meroxa.com</a>.</li><li>Join our <a href="https://discord.meroxa.com/">Discord</a> community.</li><li>Follow us on <a href="https://twitter.com/meroxadata">Twitter</a>.</li></ul><p><em>Originally published at </em><a href="https://meroxa.com/blog/oracle-database-data-integration/"><em>https://meroxa.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a84375d8b6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/announcing-the-oracle-database-source-data-integration-1a84375d8b6">Announcing the Oracle Database Source Data Integration</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Conduit uses Buf to work with Protobuf]]></title>
            <link>https://medium.com/meroxa/how-conduit-uses-buf-to-work-with-protobuf-f6cf720ab765?source=rss----b452e8fce8c7---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6cf720ab765</guid>
            <category><![CDATA[conduit]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[protobuf]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[buf]]></category>
            <dc:creator><![CDATA[Lovro Mažgon]]></dc:creator>
            <pubDate>Thu, 07 Jul 2022 17:29:14 GMT</pubDate>
            <atom:updated>2022-07-11T18:05:04.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k_TKexcrqeLzEM8wSaV66g.png" /></figure><p><a href="https://github.com/conduitio/conduit">Conduit</a>, our Kafka Connect alternative written in Go, uses Protobuf on two fronts:</p><ol><li>To define the gRPC API,</li><li>It is the protocol used for communicating with standalone connectors.</li></ol><p>However, we started facing challenges when working with Protobuf that impacted our developer experience, so we began looking for ways to resolve these problems. Continue reading to learn more about the challenges we faced and how we resolved them with <a href="https://buf.build/">Buf</a>.</p><h3>What is Protobuf?</h3><p><a href="https://developers.google.com/protocol-buffers">Protobuf</a> is the short-hand term for “protocol buffers”, a data format with an accompanying interface definition language. You can imagine it as XML or JSON, the difference being that the same data encoded with Protobuf generally results in a smaller memory footprint and better (de)serialization performance. Protobuf is commonly used as the data format in <a href="https://grpc.io/">gRPC</a>.</p><h3>Challenges working with Protobuf</h3><p>While Protobuf solves a <a href="https://developers.google.com/protocol-buffers/docs/overview#solve">whole set of problems</a> it also introduces some challenges. These are the ones we ran into:</p><ul><li><strong>Managing Tools</strong>: To use Protobuf, you need to write a .proto file that describes the data structure you intend to serialize. Once you have a Protobuf file, you can run the Protobuf compiler to generate code in any of the supported languages. This in turn means you need to make sure you have the correct version of the compiler, as well as the correct version of any plugins you might need. Managing these tools quickly becomes a problem when multiple developers are involved since they need to ensure their environments are configured the same way.</li><li><strong>Managing Dependencies</strong>: Protobuf files can import dependencies that need to be provided to the compiler at compile time. Developers are left on their own to figure out how to find existing Protobuf definitions, manage the dependencies, and ensure they are up to date.</li><li><strong>Evolving the Schema</strong>: Data structures evolve, and so do Protobuf files. When you need to change the data structures in a Protobuf file there are <a href="https://developers.google.com/protocol-buffers/docs/proto3#updating">rules</a> you have to follow to ensure the new schema is backwards compatible. These rules are not enforced and are easy to miss.</li></ul><h3>What is Buf and how are we leveraging it?</h3><p><a href="https://buf.build/">Buf</a> is a set of tools that aim to alleviate the challenges when working with Protobuf. We leverage the following tools to solve the above problems when developing Conduit:</p><ul><li><a href="https://buf.build/product/cli/">Buf CLI</a> comes with a built-in Protobuf compiler, linter, breaking change detection and formatter.</li><li>Buf provides <a href="https://docs.buf.build/ci-cd/github-actions">Github Actions</a> for setting up Buf, running the linter, detecting changes, and pushing Protobufs to their schema registry using the Github CI/CD system.</li><li><a href="https://buf.build/explore">Buf Schema Registry</a> is an online registry where you can push your Protobuf schemas. It will automatically generate a nice UI to browse the documentation of your schema, make it easily available to consumers to import as a dependency, and even automatically generate code so that consumers can entirely skip the compilation step (currently only available for Go).</li></ul><h3>Conduit Connector Protocol</h3><p>Conduit has the ability to run connectors as plugins that don’t have to be included in the Conduit binary. Standalone connectors are invoked by Conduit and run in their own process that communicates with Conduit through gRPC (see <a href="https://github.com/ConduitIO/conduit/blob/main/docs/architecture-decision-records/20220121-conduit-plugin-architecture.md">this document</a> for more information). The gRPC service definitions and data structures are defined in the Github repository <a href="https://github.com/ConduitIO/conduit-connector-protocol">ConduitIO/conduit-connector-protocol</a> which is using Buf to manage Protobuf definitions. Here we will describe how we structured our workflow.</p><h4>CI Actions</h4><p>We use Github Actions provided by Buf to lint our proto files, detect breaking changes and upload them to the Buf Schema Registry. You can find the full workflow file <a href="https://github.com/ConduitIO/conduit-connector-protocol/blob/main/.github/workflows/buf.yml">here</a>.</p><p>Let’s first look at the validate job that contains the first two steps.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FiNucVdw2IHNVbD0c2OFc%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FiNucVdw2IHNVbD0c2OFc&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9dcf59931f381fdbf9a2da199d9e1f1/href">https://medium.com/media/c9dcf59931f381fdbf9a2da199d9e1f1/href</a></iframe><p>First, we need to do some setup — we check out the repository (<a href="https://github.com/actions/checkout">actions/checkout</a>) and install the latest Buf CLI (<a href="https://github.com/bufbuild/buf-setup-action">bufbuild/buf-setup-action</a>). After that, we are ready to call the lint action (<a href="https://github.com/bufbuild/buf-lint-action">bufbuild/buf-lint-action</a>) that ensures our proto files follow the defined style guide.</p><p>After the lint is successful, we execute an action ensuring the new schema is backwards compatible with the old one. We achieve this by first fetching the main branch and executing the breaking action (<a href="https://github.com/bufbuild/buf-breaking-action">bufbuild/buf-breaking-action</a>) against the current content of the main branch.</p><p>If the validate job succeeds and the action is being executed on a commit to the main branch, then we trigger the job push.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FJI21UMyQI2oqH7tcNmli%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FJI21UMyQI2oqH7tcNmli&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/01884bfb256ff66fd2eda012adce4324/href">https://medium.com/media/01884bfb256ff66fd2eda012adce4324/href</a></iframe><p>You’ll notice this job also starts with the checkout and Buf setup actions, followed by the push action (<a href="https://github.com/bufbuild/buf-push-action">bufbuild/buf-push-action</a>) that takes a secret token to authenticate with the Buf Schema Registry and pushes the new Protobuf definitions.</p><p>These Github Actions result in a workflow that doesn’t rely on the developer having their local environment set up correctly, as the CI/CD is the single place where all Protobuf files are validated. Additionally, we don’t need to share secrets between developers, the CI/CD takes care of pushing schemas to the registry.</p><h4>Schema Registry</h4><p>We use the Buf Schema Registry to host the Protobuf definitions and get a UI for our <a href="https://buf.build/conduitio/conduit-connector-protocol/docs/main:connector.v1">docs</a>. The registry also tracks old versions of the same schema file so anyone referencing an older version can keep using it or update to the new version using buf mod update.</p><h4>Remote Code Generation</h4><p>Pushing our Protobuf definitions to the Buf Schema Registry opens up the possibility of using <a href="https://docs.buf.build/bsr/remote-generation/overview">remote code generation</a>. The registry will take care of generating the Go code for us and expose it as a go module, ready to be imported. This feature allows us to entirely skip the manual compiling step and simply import the compiled code as a dependency.</p><p>For instance, to fetch the latest Conduit connector protocol code we can invoke this command:</p><pre>go get go.buf.build/protocolbuffers/go/conduitio/conduit-connector-protocol</pre><p>Every time we update the Protobuf definitions and push them to the registry, the code will be remotely generated and ready to be used in any dependent code.</p><h4>Local Development</h4><p>Our workflow heavily leans on hosted services like Github Actions and the Buf Schema Registry, so the natural question is how can we do local development? The answer are go mod <a href="https://go.dev/ref/mod#go-mod-file-replace">replace directives</a>.</p><p>To switch to locally generated Protobuf code we follow the following steps:</p><ul><li>buf generate — executing this in the proto folder will compile the proto files and generate Go code locally in the folder internal</li><li>go mod init github.com/conduitio/conduit-connector-protocol/internal — executing this in folder internal will initialize a (temporary) Go module in the newly generated Go code</li><li>go mod edit -replace go.buf.build/library/go-grpc/conduitio/conduit-connector-protocol=./internal — executing this at the root of the repository will replace any references to the remotely generated code with the locally generated code (similarly we can do this for other repositories that depend on remotely generated code)</li></ul><h3>Conclusion</h3><p><a href="https://buf.build/">Buf</a> is a great tool that allows us to streamline the management of our Protobuf files, ensures we follow code guidelines and don’t unknowingly introduce breaking changes. It solves these problems in an elegant way and enhances the developer experience.</p><p>You know what else enhances the developer experience? <a href="https://github.com/conduitio/conduit">Conduit</a>! We’re still very much in the early stages and rely on the feedback of our community to steer the project in the right direction. Try it out… if you like it join the <a href="https://discord.com/invite/pN24QPca6b">discussion</a> and show us some love!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6cf720ab765" width="1" height="1" alt=""><hr><p><a href="https://medium.com/meroxa/how-conduit-uses-buf-to-work-with-protobuf-f6cf720ab765">How Conduit uses Buf to work with Protobuf</a> was originally published in <a href="https://medium.com/meroxa">Meroxa</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>