<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Open xG HyperCore - Medium]]></title>
        <description><![CDATA[Approaches and Design Blueprints for AI Stacks and Application Platforms using open source software with Hybrid-Cloud. - Medium]]></description>
        <link>https://medium.com/open-5g-hypercore?source=rss----bd5258a5ea5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Open xG HyperCore - Medium</title>
            <link>https://medium.com/open-5g-hypercore?source=rss----bd5258a5ea5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:31:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/open-5g-hypercore" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Episode-XXIV Artificially Intelligent Platform Interface]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xxiv-artificially-intelligent-platform-interface-667f44dcecf1?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/667f44dcecf1</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 23:38:40 GMT</pubDate>
            <atom:updated>2024-11-17T02:58:28.494Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors: </strong>Fatih Nar, Shujaur Mufti, Ian Hood, Ather Chaudhry, David Kypuros, Arun Thomas</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gsJSfYh_VmwplTTb" /></figure><p>Remember the last time you called your telecom provider and heard, “Your call is important to us, please hold”? The telecom industry finally realizes that nobody likes to wait — not for calls, data, or innovation. The industry is undergoing a transformative shift, evolving from traditional connectivity providers to sophisticated platform enablers.</p><p>With the demand for Network-as-a-Service (NaaS) growing and the rise of standardized network APIs, telcos are poised to redefine the landscape. However, progress has been hindered by networks lacking openness and APIs that are often static and complex, making widespread adoption challenging. But now, with the rise of the Open RAN &amp; API 1st movement, networks are starting to open up, creating new opportunities.</p><p>In this next phase, artificial intelligence (AI), open-source technologies, and platform programmability are crucial in taking API capabilities to the next level. Modern application platforms, particularly those with advanced AI-driven interfaces, enable telcos to manage, scale, and optimize network services in real time. By integrating dynamic programmability and automated resource management, these platforms offer telcos enhanced control, flexibility, and scalability, ready to meet the evolving needs of enterprises.</p><p><strong>The Role of AI in the API Ecosystem</strong></p><p>The Artificially Intelligent Platform Interface (AI-PI) introduces a transformative paradigm for managing APIs. Traditional API frameworks rely on static configurations, manual scaling, and limited automation, which fall short of 5G, edge computing, and IoT drive real-time, mission-critical services. This is where AI-driven orchestration redefines network management.</p><p>With AI-enabled capabilities, telcos can automate network configuration, scaling, and management of API-exposed services. Key features include:</p><ul><li><strong>Predictive Scaling</strong>: AI-driven insights help telcos predict network traffic spikes and automatically scale network resources to optimize services such as video streaming, gaming, and IoT.</li><li><strong>Real-time Configuration Management</strong>: AI-PI can dynamically adjust security policies, resource limits, and traffic routing based on real-time network conditions, enhancing both reliability and user experience.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z05Rn4NHPDH9h6Gf6xzdYA.png" /><figcaption>Figure-1 3GPP Network API Evolution Path</figcaption></figure><p><strong>Commercialization Opportunities</strong></p><p>AI-PI holds significant commercial potential, enabling telcos to offer a range of API-driven services for enterprises to seamlessly integrate network functionality. Notable use cases include:</p><ul><li><strong>IoT &amp; Edge Computing</strong>: APIs enable low-latency services critical for industries such as healthcare and manufacturing. AI-powered orchestration dynamically allocates resources based on sensor activity, optimizing real-time analytics.</li><li><strong>5G Network Slicing</strong>: Enterprises can request AI-managed network slices on demand, catering to applications like real-time video conferencing and autonomous vehicle communications.</li></ul><p>In terms of commercialization, telcos can monetize these services through pay-per-use or subscription models, creating new revenue streams while optimizing operational costs by automating service level adjustments and reducing over-provisioning.</p><p>Having AI-PI as an embedded capability, pre-integrated with operational support systems (OSS) and business support systems (BSS), provides powerful, out-of-the-box NaaS functionalities (e.g., Red Hat OpenShift’s AI-PI Agent for 5G Core -&gt; Figure-2 Below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnuhyxx-pfzsJOsIwqP1qg.png" /><figcaption>Figure-2 Red Hat AI-PI in Action for 5G Core</figcaption></figure><p><strong>Enhancing the End-user Experience</strong></p><p>Integrating AI-driven orchestration into network APIs significantly improves the end-user experience by intelligently managing East-West (E-W) and North-South (N-S) traffic, optimizing microservices performance, and ensuring seamless load balancing. For users, this means:</p><ul><li><strong>Zero Downtime</strong>: Predictive scaling and traffic management ensure smooth application performance even during peak usage.</li><li><strong>Enhanced Security</strong>: AI-enabled configuration management allows APIs to automatically adapt security settings based on real-time threat assessments, enhancing protection without impacting performance.</li></ul><p>This approach not only boosts service performance but also empowers developers to focus on innovation, accelerating time-to-market for new features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ET-J5h2cqRr7FwXP" /><figcaption>Figure-3 B2X (Business to All -X- Capabilities with AI-PI)</figcaption></figure><h3>Value Proposition-Based Examples</h3><ul><li><strong>Dynamic Resource Allocation for NaaS</strong>: AI-PI allocates network resources based on real-time demand, ensuring optimal service quality during high-demand events without over-provisioning, resulting in cost savings and improved user experience.</li><li><strong>Automated Troubleshooting</strong>: Embedded AI monitors network health, identifies issues, and initiates self-healing, escalating to ticketing systems only when necessary, enhancing network reliability.</li><li><strong>On-Demand Network Slicing</strong>: Enterprises can request custom network slices for specific applications, with AI-PI managing resources dynamically, unlocking premium service revenue opportunities.</li><li><strong>Adaptive Security Response</strong>: AI-PI proactively responds to network threats, adjusting policies and isolating impacted areas, maintaining uninterrupted service, especially valuable to enterprise clients.</li><li><strong>Edge Resource Optimization for IoT</strong>: AI-PI dynamically manages edge resources for latency-sensitive applications, ensuring that critical systems remain responsive in smart factories or healthcare facilities.</li></ul><h3>API to Agent Walk: Embedded Platform AI Agent</h3><p>The embedded Platform AI Agent adds a transformative layer, enabling intelligent workflows within the platform. Moving beyond static configurations, this AI middleware supports complex task automation through domain-specific services and customizable alerting. The AI Agent streamlines network operations by automating workflows based on real-time data, with limited need for manual oversight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/0*ea1WVD1R1zjnEBdZ" /><figcaption>Figure-4 New AI-PI Agentic Architecture</figcaption></figure><p>The modular design (see Figure-4) of this agentic AI infrastructure includes “Agentic Primitive Resources” — foundational building blocks equipped with memory and contextual databases for adaptive decision-making. This architecture enables telcos to create specialized workflows tailored for domains such as telecom, IoT, healthcare, and finance.</p><p>Through Southbound API Plugins, telcos can integrate with enterprise-level AI services, large language models, and predictive analytics, enabling the platform to autonomously manage interactions and configurations. This intelligent, middleware-powered approach optimizes performance and resource allocation across various environments, enhancing scalability and resilience in cloud-native operations.</p><p><strong>The Role of Hyperscalers</strong></p><p>The API ecosystem extends beyond telcos, with hyperscalers playing pivotal roles. Hyperscalers like AWS and Google Cloud enable telcos to optimize multi-cloud and edge environments for API delivery. Similarly, application platform providers like Red Hat Openshift offer APIs to provide abilities for real-time platform programmability. AI-PI empowers these providers to offer intelligent, on-demand scaling of communication services based on real-time demand.</p><p>By embracing AI-powered programmability, application platforms enable telcos, hyperscalers, and platform providers to deliver more scalable, intelligent services, driving the next phase of API economy growth.</p><p><strong>Looking Ahead: The Future of Network APIs and NaaS</strong></p><p>The future of Network APIs lies in AI-driven automation and the evolution of NaaS. With dynamic, intelligent services, telcos can provide:</p><ul><li><strong>Intelligent Network Orchestration</strong>: AI will optimize network resource allocation based on real-time data, enabling highly adaptable workloads.</li><li><strong>Advanced E-W/N-S Traffic Control</strong>: As applications become more distributed, efficient traffic management will be essential. AI-driven platforms will automate load balancing, traffic prioritization, and resource scaling, supporting seamless operations across cloud environments.</li></ul><p>This evolution will enhance network scalability, create new revenue models, and meet the demands of real-time, mission-critical applications.</p><blockquote><strong>Conclusion: AI-PI in the API Revolution</strong></blockquote><blockquote>As telcos transform their networks into open platforms, AI-driven programmability will become central to API innovation. With AI-PI, telcos can offer advanced orchestration, dynamic scaling, and real-time application management, fundamentally enhancing how network services are delivered and consumed.</blockquote><blockquote>In this new era of AI-driven Network APIs, Red Hat OpenShift with Operator Framework is leading the charge, equipping telcos to create smarter, more adaptable networks, shaping the future of digital communications. Through this AI-powered approach, telcos can unlock new revenue streams and provide agile, secure, and scalable services for the coming 6G, distributed edge realizations (finally), and Massive-IoT age.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=667f44dcecf1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xxiv-artificially-intelligent-platform-interface-667f44dcecf1">Episode-XXIV Artificially Intelligent Platform Interface</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XXII: AI Accelerators’ Performance vs Sustainability]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xxii-ai-accelerators-performance-vs-sustainability-256244f83b1b?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/256244f83b1b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[power-grid-system]]></category>
            <category><![CDATA[accelerator]]></category>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Fri, 07 Jun 2024 04:47:08 GMT</pubDate>
            <atom:updated>2024-06-18T00:56:01.620Z</atom:updated>
            <content:encoded><![CDATA[<p>Author: Fatih E. NAR</p><blockquote>In recent years, AI accelerators have become the superheroes of artificial intelligence, saving the day in everything from sophisticated data centers to edge computing devices. With their parallel processing superpowers, they speed up and streamline large datasets and complex algorithms. But, like any hero, they have their kryptonite: performance degradation over time, especially under heavy usage.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*udqXKvQ3BE1UkdnR_kY0CQ.png" /><figcaption>Figure-1 Burned-End BBQ</figcaption></figure><p><strong>Heat: The Arch-Nemesis</strong></p><p>Heat is the villain in our AI accelerator saga. These components, when processing massive workloads, turn into tiny furnaces. If not properly managed, they face several deadly consequences:</p><p>&gt;&gt; Thermal Throttling: Many AI accelerators have built-in thermal throttling mechanisms to avoid a meltdown. When they get too hot, they slow down to cool off, like taking a breather during a workout, leading to a noticeable performance drop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*muwjvjEayPi-qJC4oRwhoQ.png" /><figcaption>Figure-2 AI Accelerators in Texas Summer (<a href="https://gist.github.com/fenar/8242203c2f9e196f48b4143fcb5562e4">Link</a>)</figcaption></figure><p>&gt;&gt; Component Degradation: Persistent high temperatures can wear down semiconductor materials faster than you can say “silly-con”. Over time, this can slow processing speeds and increase the chance of failure.</p><p>&gt;&gt; Increased Power Consumption: High temperatures can cause leakage currents in transistors, leading to higher power consumption and even more heat. This creates a vicious cycle that is harder to escape than an on-demand TV show binge (Slow Horses is My Last Redemption 😎).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*hbUSol3EUOnC1lRFmK9Ctg.png" /><figcaption>Figure-3 Platform Observability Capabilities for GPU Observability (<a href="https://gist.github.com/fenar/081b22237b5eaa5398dafa70a80fdff1">Link</a>)</figcaption></figure><blockquote>Site note: In order to have an idea about the utilization levels, power consumption and temperatures of the accelerators, it is crucial to have a proper observability solution in place, not only for observing their state and also plug-in such information points to a autonomous operational workflow for corrective actions.</blockquote><p><strong>Load-Driven Distortion: Stress on the Circuit Superheroes</strong></p><p>AI accelerators handle immense computational loads, and this stress can lead to:</p><p>1. Microcracks: Constant thermal expansion, contraction, and heavy loads can cause tiny cracks in the silicon. These microcracks can mess up electrical pathways, causing performance degradation and potential failure.</p><p>2. Electromigration: High currents can cause electromigration, in which metal atoms in the interconnect decide to go on a road trip. This thins conductive pathways and increases resistance, eventually leading to circuit failure.</p><blockquote>Higher currents in denser circuits accelerate electromigration, causing faster degradation of the chip’s interconnects.</blockquote><p>3. Intra-Short Circuits: Stress and distortion can cause unintended conductive paths within the chip, leading to immediate failures or pesky intermittent issues that degrade performance over time.</p><p>4. Higher Power Density: Packing more transistors into a smaller space increases the heat dissipation challenge, impacting the accelerator’s lifespan. Smaller nodes like 4Nm and 4NmP generate more heat per unit area, increasing thermal design power (TDP).</p><p><em>Side note: Thermal design power (TDP) measures the maximum heat a computer component, such as a processor or graphics card, can dissipate under normal operating conditions. TDP can also estimate the maximum power the GPU will draw under typical, high-load conditions. It’s not the absolute maximum power the GPU could ever draw but rather a guideline for system builders and consumers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*w9LGAmlWqVw8L8v68esYFg.png" /><figcaption>Figure-4 Setting TDP Cap for NVIDIA GPU</figcaption></figure><blockquote>Increased density can create hot spots on the chip, leading to uneven thermal expansion and stress.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*fyM_EMQksLtNQQS5-VQMLg.png" /><figcaption>Figure-5 4N Evolution Brings Increase in Thermal Design Power (TDP)</figcaption></figure><p><strong>Other Mischievous Factors</strong></p><ol><li>Manufacturing Variability: Like snowflakes, no two chips are exactly the same. Minor imperfections can become significant under heavy use, leading to performance variability and a shorter lifespan.</li></ol><blockquote>-&gt; There is a reason “Founders Editions” are more popular than OEM ones.</blockquote><p>2. Electrostatic Discharge (ESD): ESD events are like mini lightning strikes that can zap AI accelerators. Effective grounding and ESD protection measures are crucial to protecting these sensitive components.</p><p>3. Software Efficiency: Poorly optimized algorithms are the fast food of AI accelerators. They cause excessive strain on the hardware, speeding up wear and reducing lifespan. Endless epochs with unimaginable batch sizes and chunks of steps would torch your accelerators.</p><blockquote>It is not necessarily true that, all longer trainings with bigger datasets make AI models better!</blockquote><p><strong>Power Demand and Investment: The Bigger Picture</strong></p><p>With the growing footprint of AI accelerators, the demand for power (to run &amp; cool’m) from the grid reaches ultimate levels nationwide, as a result -&gt; the global data center power demand is set to more than double by 2030.</p><p>This power demand growth will require utility investments of $50 billion in new power generation capacity, with a 60/40 split between gas and renewables, driving ~3.3 bcf/d incremental natural gas demand by 2030.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*ablW-2PQ_UEwrpBq" /><figcaption>Figure-6 Datacenter Power Consumption Projections (Ref: <a href="https://www.goldmansachs.com/intelligence/pages/gs-research/generational-growth-ai-data-centers-and-the-coming-us-power-surge/report.pdf">Link</a>)</figcaption></figure><p>Goldman Sachs forecasts a 15% CAGR in data center power demand from 2023–2030, driving data centers to make up 8% of total US power demand by 2030, up from about 3%.</p><blockquote>This will necessitate approximately 47 GW of incremental power generation capacity, with about 60% gas and 40% renewable sources, requiring significant capital investment in US power generation capacity through 2030.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/0*Il0TUMk0uLSX77Qk" /><figcaption>Figure-7 Power Profiles &amp; AI’s Role in Consumption (Ref: <a href="https://www.goldmansachs.com/intelligence/pages/gs-research/generational-growth-ai-data-centers-and-the-coming-us-power-surge/report.pdf">Link</a>)</figcaption></figure><p><strong>Final Thoughts: The Road -42- Ahead</strong></p><p>Understanding the factors that impact the lifespan and performance of AI accelerators is crucial for businesses and our lives, which depend increasingly on AI enrichments. We may extend the life of these critical components by addressing heat issues, managing load-driven stress, and considering other factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/0*VBlOQBxc7G0DNUkd" /><figcaption>Figure-8 Elon’s Prediction (Not all comes true as we all know)</figcaption></figure><p>Moreover, AI-driven data center growth highlights investment opportunities in utilities, renewable generation, and industrial sectors. Balancing the benefits of advanced node designs with their impact on lifespan will be crucial for the sustainable development of high-performance AI accelerators and the supporting power infrastructure.</p><p>So, while AI accelerators continue to revolutionize technology, making our lives easier and saving us time at work and in our daily lives, managing their performance and impact on the power grid will keep us all on our toes.</p><blockquote>PS: Do not buy used gpus from unknown sellers for low prices, unless you want a hot summer with no power. 🔥</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=256244f83b1b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xxii-ai-accelerators-performance-vs-sustainability-256244f83b1b">Episode XXII: AI Accelerators’ Performance vs Sustainability</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XXI: Avoid AI Blindness for Business Vision and See the Rainbow]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xxi-avoid-ai-blindness-for-business-vision-and-see-the-rainbow-954eaa0dfa80?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/954eaa0dfa80</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Thu, 09 May 2024 17:24:06 GMT</pubDate>
            <atom:updated>2025-01-27T03:38:00.321Z</atom:updated>
            <content:encoded><![CDATA[<p>Author: Arun Thomas, Fatih E. NAR</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dm5MgHM76axiHU5zD9ebTw.png" /><figcaption>Rainbow Seeking Businessman</figcaption></figure><p>As we navigate the age of Artificial Intelligence (AI), the conversation often skews towards a sci-fi screenplay where robots take over the world — or at least our jobs. But before you start preparing for a machine-led apocalypse, let’s refocus the lens. In the bustling software-defined world, the real story isn’t about AI elbowing out humans; it’s about AI handing us a paintbrush to color our business landscape with efficiency, innovation, and a splash of creativity.</p><p><strong>Upskilling: Attention Is All Your Employee Need</strong></p><p>Remember when Google revolutionized AI with the idea that “Attention Is All You Need” for transformer architectures? Well, your employees need a bit of that attention too. In the frenzy of finding the next big AI prodigy, we often overlook the gold mine of talent already within our ranks. Data scientists with expertise in an enterprise domain (say Telco 😁) are as rare as a peaceful day on social media. So, why not turn inward and invest in the people who already speak your company’s language? Providing focused training and upskilling opportunities can transform them into the AI-savvy professionals you’re searching for. It’s not just about feeding new software into the old system; it’s about enriching the substrate of your workforce with new capabilities and insights -&gt; isn’t this a good culture (that you have been ignoring) or what?!.</p><p><strong>Data Challenges: Like Herding Cats, But Harder</strong></p><p>If you think gathering and cleaning tons of relevant, trustworthy data sounds challenging, you’re spot on — it’s like herding cats, if the cats were on a redbull buzz. The integrity of AI’s insights hinges on the quality of data it’s fed. Garbage in, garbage out, as they say. Ensuring the data is not just big, but also clean &amp; relevant and fresh, is crucial. No one wants to base business decisions on data that’s as sketchy as an old-looking three-dollar bill.</p><p><strong>Model Precision and Reliability: Not Just a Numbers Game</strong></p><p>AI models can be finicky. Their performance can swing wildly depending on the computational power at hand and the biases lurking in the data shadows. So, before you take your AI model’s word as gospel, remember to check its heritage. Think of it as a reality check for your digital prophet. Ensuring your AI’s predictions are stable and reliable across different scenarios is like making sure your umbrella holds up in a storm — not just when you buy it in the store at sunny day.</p><p><strong>The Bright Future with AI: More Human Than Ever</strong></p><p>AI isn’t here to steal jobs but to amplify human potential, speed up progress, and increase efficiency. We’re looking at a future where humans can explore new realms of creativity and innovation, guided by AI’s assistance. It’s about using AI to find the pot of gold at the end of the data rainbow, freeing us up to do what humans do best — think, dream, and create (and make a good coffee, because humans still run on caffeinated dreams).</p><p>The bottom line is; the journey with AI in the realm of business and beyond should be less about fearing a robot takeover and more about embracing the collaborative dance between human ingenuity and artificial intelligence. By integrating AI with a dose of humor and a focus on human potential, businesses can not only survive but thrive, painting a future that’s as bright as it is smart. So, let’s strap on our new digital tool belts and get to work — with a smile. 🤠</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=954eaa0dfa80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xxi-avoid-ai-blindness-for-business-vision-and-see-the-rainbow-954eaa0dfa80">Episode XXI: Avoid AI Blindness for Business Vision and See the Rainbow</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XX: Open Source, The Wild West of Innovation, Not a Free Buffet!]]></title>
            <link>https://medium.com/open-5g-hypercore/open-source-the-wild-west-of-innovation-not-a-free-buffet-88b77e1d7d49?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/88b77e1d7d49</guid>
            <category><![CDATA[open-source-software]]></category>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Thu, 28 Mar 2024 13:46:01 GMT</pubDate>
            <atom:updated>2024-05-09T17:06:18.122Z</atom:updated>
            <content:encoded><![CDATA[<p>Author: Fatih E. NAR</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6IrBH4k6Y4ov9CVA" /><figcaption>Open Source Sheriff</figcaption></figure><p>Ah, the vast, untamed wilderness of technology and software development, where the open-source movement rides high like a valiant cowboy of innovation, collaboration, and communal prosperity. This bustling town of code-sharing is built on the noble ideal that by tossing our knowledge and expertise into the communal pot, we can cook up technological advancements far beyond the reach of the lone gunslinger working under the cloak of proprietary secrecy.</p><blockquote>But, hold your horses! It seems not everyone got the memo on what open source is all about.</blockquote><p>Sometimes, there’s a bit of a mix-up, or worse, a downright heist attempt by those looking to rustle up some code without tipping their hat in return. This opens up a barn door of discussion on the true spirit of open source and a gentle reminder to not mistake the communal water trough for an all-you-can-eat buffet.</p><p>Now, about those rules of the range. Open source isn’t the Wild West without a sheriff. We’ve got ourselves a variety of licenses, like the revered General Public License (GPL) version 3, which is akin to the law of the land for ensuring that software remains as free as a bird and as accessible as the saloon on a Friday night. It allows folks to modify, share, and even make a few bucks off their work, so long as they pass that freedom on like a passed hat at a hoedown. This ain’t just about being neighborly; it’s about fostering a cycle of continuous improvement and shared bounty, ensuring that both the creators and users of software can ride off into the sunset, better off than when they rode in.</p><p>Lately, there’s been some chatter by the campfire about some big outfit trying to stake their claim on open-source territory without playing by the communal rules. Imagine walking into a potluck, taking the biggest piece of pie, and not even nodding at the baker. That’s the kind of scenario sparking debates faster than lightning on a dry prairie about how we balance the give and take of open-source software. Open source is an invitation to a barn-raising, not a signal to swipe the barn doors for your own homestead.</p><p>The magic of open source is in its posse of developers, contributors, and dreamers. It’s a community pledge to keep the software prairie wide, wild, and welcoming for all who wish to explore and contribute to it. This ethos has led to inventions and innovations that are the backbone of our digital frontier today. It’s a land where respect is earned by those who contribute, not by those who only take, reminding us that even the smallest contribution to the commons is worth its weight in gold.</p><p>As the open-source frontier expands, let’s tip our hats to the values that have made it a land of opportunity: innovation, collaboration, and shared prosperity. Let’s continue to protect these principles, ensuring that the open-source world remains a place where everyone, from the solo developer to the biggest tech outfits, can come together to build something truly spectacular.</p><blockquote>After all, in the grand tapestry of open source, every stitch counts, and it’s the collective pattern that makes it a masterpiece!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88b77e1d7d49" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/open-source-the-wild-west-of-innovation-not-a-free-buffet-88b77e1d7d49">Episode XX: Open Source, The Wild West of Innovation, Not a Free Buffet!</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XIX: Novel Approach for High-Performing Multi-Cloud Networking]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xix-novel-approach-for-high-performing-multi-cloud-networking-cd2698982b66?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/cd2698982b66</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Wed, 13 Dec 2023 23:48:41 GMT</pubDate>
            <atom:updated>2024-02-21T14:36:45.164Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors:</strong>[MIT/Steinwurf: Muriel Medard , Peter Bech , Morten Pedersen] [Red Hat: Fatih E. NAR, Doug Smith, Ali Bokhari, Azhar Sayeed] [Fatih Baltaci CTO at DDosify]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LR2-s7uWLyBe4l4f" /></figure><p><strong>1.0 Introduction</strong></p><p>In the dynamic landscape of enterprise networking, reliability stands as the cornerstone of success. Timely and dependable data delivery is a critical key performance indicator (KPI) for networks, while any deviation from this, referred to as erasures, can impose significant costs on businesses.</p><p>Data erasures occur due to unrecoverable errors in end-to-end transmission and/or excessive jitter (high variations in network performance consistency) in delivery. The associated costs encompass revenue losses, wastage of network resources in futile service attempts, and regulatory penalties for non-compliance.</p><p>Traditional methods to enhance reliability often resort to data repetition, which can introduce undesirable latency when applied reactively or incur excessive costs through upfront overprovisioning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K3fcGkRy-BOfB6QB" /><figcaption>Figure-1(CY2023) Latency Map — Worldwide (Ref: <a href="https://radar.cloudflare.com/quality/?dateRange=52w">Link-1</a>)</figcaption></figure><p>Before we unveil our approach, let’s go through some fundamentals we would build our solution on:</p><p><strong>Kubernetes Networking:</strong></p><p>Kubernetes (K8s) emerges as a leading container orchestration platform, empowering efficient deployment, scaling, and management of containerized applications, driving modern microservices-based business logic.</p><p>At the heart of Kubernetes lies its networking framework, which is pivotal in enabling seamless communication between containers and the outside world. The Container Networking Interface (CNI) specification defines how container runtimes harness various network plugins to endow containers with essential networking capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/0*ZTvaX60pixdCRNsA.png" /><figcaption>Figure-2 Network Interfaces for a K8s Pod</figcaption></figure><p>CNI plugins configure container network interfaces, assign IP addresses, set up routes, and offer networking customization for Kubernetes pods. Among these, the Multus CNI plugin stands out, enabling the attachment of multiple network interfaces to a single pod. This capability facilitates intricate networking configurations such as traffic isolation and network segmentation. Multus CNI is the linchpin for high-performance and network-centric workloads in Kubernetes, faithfully adhering to the Kubernetes Network Plumbing Working Group’s standards for multi-networking within Kubernetes.</p><p><strong>Forward Erasure Correction:</strong></p><p>Forward Erasure Correction (FEC) is used in data communication to enhance reliability and stability, particularly in networks where data retransmission is not feasible or efficient. It’s a technique used to manage the failure of timely packet delivery. The sender adds redundant data to its message, thus allowing the receiver to detect and correct errors without asking the sender for additional data. FEC is especially useful when retransmissions are costly or impossible, such as in broadcasting or communication with long round-trip times between endpoints (e.g., multi-hop routing with packet loss). Here’s how FEC contributes to network stability:</p><blockquote>Erasure Correction Without Retransmission: In a typical communication system without FEC, the receiver must ask the sender to retransmit the data if data is corrupted during transmission. This retransmission can lead to increased traffic on the network, potentially causing congestion and instability. By allowing the receiver to correct erasures, FEC reduces the need for retransmissions.</blockquote><blockquote>Improved Efficiency in Unstable Conditions: In environments where the signal quality is poor or highly variable (like in wireless networks), FEC can significantly improve the data transmission efficiency. It helps maintain a stable connection even in the presence of interference or signal degradation from phenomena such as fading or blockages.</blockquote><blockquote>Bandwidth Utilization: Although FEC increases the amount of data transmitted a priori, that extra data is used to repair erasures, so it is not wasted redundancy. It can lead to more efficient bandwidth use than adding repair post facto, for instance, by repeating packets after erasures occur. This is because the overhead caused by retransmissions in a network without FEC can be much greater when end-to-end reliability mechanisms are used over broadcast or multi-hop networks.</blockquote><blockquote>Latency Reduction: In real-time applications like video streaming, online gaming, and autonomous vehicles, waiting for retransmitted data can lead to unacceptable delays. FEC can reduce this latency by correcting erasures on the fly, thus maintaining a smoother and more stable experience.</blockquote><blockquote>Scalability in Broadcast Applications: In broadcast applications (like satellite TV), providing individual retransmissions for each receiver is not feasible. FEC allows a single data stream to be reliably received by many users simultaneously, regardless of individual errors in reception.</blockquote><p>In this article, we present a new approach to Kubernetes network implementation that capitalizes on the structural maturity and control offered by Multus CNI, together with incorporating a new FEC algorithm via an overlay tunnel mesh to steer us to reduced jitter &amp; latency and amplified bandwidth for containerized workloads (🤞).</p><p><strong>2.0 Solution</strong></p><p><strong>2.1 RLNC &amp; Steinwurf RAMP</strong></p><p>Random Linear Network Coding (RLNC) revolutionizes data packet management by converting them into algebraic components representing portions of multiple packets. This approach fundamentally shifts the paradigm of data delivery, offering a stark contrast to conventional FEC algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/0*oIgEWGRgm_6GxsTm.png" /><figcaption>Figure-3 RLNC FEC vs Traditional FEC</figcaption></figure><p>In traditional scenarios without RLNC, there’s a stringent requirement to send or receive data packets in a specific sequence for the data to be intelligible. However, RLNC breaks away from this limitation. With RLNC, the process involves gathering a sufficient number of RLNC-encoded packets, which, when combined, solve a particular algebraic equation. This transformation results in an FEC algorithm that’s more adaptable, intelligent, and significantly more efficient in data handling.</p><p>By taking the superiority of RLNC FEC into action, Steinwurf has built the RAMP solution, which can upgrade network capabilities and be applied at the application layer to dramatically improve the whole network’s reliability, latency, and bandwidth usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/0*v_9RnVSctH6mNCzB.jpg" /><figcaption>Figure-4 FEC Solution Approaches</figcaption></figure><p>RAMP is uniquely designed and implemented as an RLNC FEC approach that does not require a trade-off in any of the three dimensions (latency, reliability, and bandwidth). RAMP has outperformed all available alternatives across traffic types and network fabric patterns.</p><p>In addition, the flexibility provided by RAMP makes it possible to combine retransmissions and multi-paths in ways never before possible to form a latency-aware adaptive transport protocol.</p><p><strong>Note-1:</strong> You can find more details on Steinwurf RAMP solution with FAQ here: [<a href="https://www.steinwurf.com/faqs">Link-2</a>]</p><p><strong>2.2 RAMP CNI</strong></p><p>Our minimum viable product (MVP) implementation is a reference concept implementation to run the Steinwurf RAMP solution as a sidecar to a pod (similar to Istio with Envoy however, we do NOT butcher the network performance instead -&gt; we aim to boost it 🚀):</p><blockquote>This approach allowed us to integrate our new FEC implementation into our networking stack without modifying the immutable host OS or kernel. The sidecar container can leverage the FEC capabilities, correcting errors for the data transmitted through the overlay tunnel.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ocX2qqWJ32sUeZUe" /><figcaption>Figure-5 Solution &amp; Benchmarking Architecture</figcaption></figure><p>In our MVP implementation, Steinwurf RAMP runs as a background process in a sidecar container. Multus CNI creates connectivity over a secondary network, and the Steinwurf RAMP process creates a TUN device over which it asserts its network optimization technology.</p><p>Limitations: In the current MVP, one of the major limitations is service discovery. Therefore, all endpoints must be specified in advance.</p><p>Future work: A fully automated solution (<a href="https://sdk.operatorframework.io/docs/overview/operator-capabilities/">Level-IV/V Operator Implementation</a>), including installation, configuration, and service discovery, would provide a full mesh between endpoints and automate the discovery of the application endpoints.</p><p><strong>Note-2:</strong> You can find our testbed implementation git-repo here: [<a href="https://github.com/dougbtv/k8s-plumbr-demo">Link-3</a>]</p><p><strong>3.0 Benchmarking</strong></p><p>With the above-mentioned setup in place, several tests were run to determine the ability of RAMP CNI to recover from packet erasures on the network. The first goal was to see how video quality improves when RAMP CNI is applied to the packet stream.</p><p>For that purpose, a video streaming server was deployed in the Mumbai (India) cluster, and a video client in the Ohio (USA) cluster (~ 8,088 miles / 13,017 km separation) was used to stream the video from the Mumbai origin server.</p><p><strong>Note-3:</strong> You can find our video streaming test app git-repo here: [<a href="https://github.com/fenar/fnr-vlc">Link-4</a>]</p><p>Once the video stream was established, packet loss was introduced by the long-distance network path. Visual observations clearly showed that the video quality suffered significantly when losses were happening but was only lightly affected when RAMP CNI was enabled on the same lossy network path. In the interest of having quantifiable data, the following set of test cases was also executed:</p><ol><li>Identify the impact on UDP traffic when a packet loss happens (up to %10) on the end-to-end network path:</li><li>Use the iperf3 tool to run a throughput test with UDP traffic between the Ohio and Mumbai clusters</li><li>Repeat the test with different packet sizes.</li></ol><p>The result of this test case, as shown in the figure below, shows that RAMP CNI could recover effectively from such a significant level of packet erasure, and the test tool did not observe any packet loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C2TdYyE_a-Cti17v" /><figcaption>Figure-6 Benchmark Results — UDP</figcaption></figure><ol><li>Identify the impact on TCP traffic when a packet loss happens (up to %10) on the end-to-end network path:</li><li>Use the iperf3 tool to run a throughput test with TCP traffic between the Ohio and Mumbai clusters</li><li>Repeat the test with different packet sizes</li></ol><p>The result of this test case, as shown in the figure below, shows that in the absence of RAMP CNI, the TCP throughput was significantly reduced because of the packet loss. However, with RAMP CNI in the network path, the TCP throughput significantly improved, and the receiver received 96% of the traffic sent by the sender.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Q-7N8jhDKjSnbHM" /><figcaption>Figure-7 Benchmark Results — TCP</figcaption></figure><ol><li>Identify the impact on jitter for HTTP traffic when a packet loss happens (up to %10) on the end-to-end network path:</li><li>Install the speed test application in a pod in the Mumbai cluster</li><li>Using a browser in the Ohio cluster, run a speed test against the speed test server in the Mumbai cluster</li></ol><p><strong>Note-4:</strong>You can find our ping/jitter test app git-repo here: [<a href="https://github.com/fenar/5gspeedtest">Link-5</a>]</p><p>The result of this test case, as shown in the figure below, reflects a 500 times improvement in jitter when the RAMP CNI is enabled on the traffic path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k7_YNTjVb3OEqtzK" /><figcaption>Figure-8 Benchmark Results — Jitter</figcaption></figure><p><strong>Note-5:</strong> For custom multi-origin latency measurements towards custom target(s), we kindly recommend using <a href="https://ddosify.com/">DDosify</a> as easy to use &amp; price/performance outcomes. You can also deploy DDosify on-premise as self-hosted with their k8s helm charts: [<a href="https://github.com/ddosify/ddosify">Link-6</a>]</p><p><strong>4.0 Observations &amp; Closure</strong></p><p>The integration of Steinwurf’s RAMP solution with Kubernetes networking, utilizing Multus CNI, presents a significant advancement in network performance for containerized applications with key business benefits:</p><ul><li><strong>Enhanced Network Performance:</strong> The solution ensures faster data transmission and reduced latency, which is crucial for real-time applications and improving operational efficiency.</li><li><strong>Reliable Data Transmission:</strong> Steinwurf RAMP provides dependable data transmission, which is essential for maintaining data integrity in distributed systems.</li><li><strong>Ease of Configuration:</strong> The RAMP CNI offers a user-friendly design that simplifies the setup and management, reducing the learning curve and operational complexity.</li><li><strong>Adaptable to Various Conditions:</strong> The solution’s versatility allows it to perform consistently across different network conditions and data formats, ensuring stable performance in diverse environments.</li></ul><blockquote>Our novel approach represents a significant advancement in the realm of distributed application networking, particularly in the context of executing multi-cloud strategies for large enterprises. It equips businesses with superior performance, heightened reliability, and simplified usability within containerized network landscapes.</blockquote><blockquote>As this solution continues to evolve, it is poised to further refine and enhance networking across multiple geographies, ushering in significant improvements in both efficiency and performance at scale.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd2698982b66" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xix-novel-approach-for-high-performing-multi-cloud-networking-cd2698982b66">Episode XIX: Novel Approach for High-Performing Multi-Cloud Networking</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XVIII 5G Open Telemetry (OTel)]]></title>
            <link>https://medium.com/open-5g-hypercore/5g-open-telemetry-otel-bccf100e753f?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/bccf100e753f</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Tue, 14 Nov 2023 16:30:51 GMT</pubDate>
            <atom:updated>2023-11-14T17:22:40.649Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors:</strong> [<strong>Red Hat:</strong> Fatih Nar, Jose Gomez-Selles, Paul Lancaster, Sean Cohen] [<strong>I2I-Systems: </strong>Özlem Demir, Omer Dogan, Hasan Köse] [<strong>Intel:</strong> John Browne] [<strong>DDosify:</strong> Fatih Baltaci]</p><p><strong>1.0 Introduction</strong></p><p>In our <a href="https://medium.com/open-5g-hypercore/episode-xvi-the-eye-of-the-bee-holder-a8e81be2dfa2">preceding discussion</a>, we emphasized the pivotal role of data in AIOps, mainly focusing on utilizing OpenTelemetry (OTel) data derived from 5G Cloud Native Network Functions (CNF).</p><blockquote><em>In this concise study, we will demonstrate how to harness the power of OTel by seamlessly integrating existing instruments into the CNF code. By following this guide, our audience will gain valuable insights into efficient data collection and processing, enabling them to implement observability and visualize their 5G stack performance.</em></blockquote><p><strong>2.0 Background</strong></p><p>OpenTelemetry (OTel) encompasses a suite of open-source projects dedicated to offering a cohesive and standardized methodology for observability within contemporary software applications. It enables developers to instrument, collect, and export telemetry data such as traces, metrics, and logs from their applications. By integrating OTel, developers can gain insights into distributed systems’ performance, behavior, and health, making it easier to troubleshoot issues, optimize performance, and enhance overall application reliability.</p><p>Key points about OTel:</p><pre>&gt; Observability Standardization: OTel aims to standardize how developers instrument and collect telemetry data from applications, integrating with various monitoring and observability tools more accessible.<br><br>&gt; Flexibility and Language Support: OTel is designed to be flexible and language-agnostic. It provides SDKs for various programming languages, including Java, Python, JavaScript, Go, and others.<br><br>&gt; Vendor-Neutral: OTel is vendor-neutral, allowing developers to choose from various telemetry backends and monitoring tools.<br><br>&gt; Community-Driven: It is developed and maintained by a vibrant and active community of contributors, ensuring continuous improvement and adaptation to evolving needs.<br><br>&gt; Standardization and Interoperability: By adhering to OTel standards, developersensure interoperability with multiple observability tools and platforms, streamlining the monitoring and troubleshooting process.</pre><p>In summary, OTel is focused on generating, collecting, managing, and exporting application observability data.</p><p><strong>3.0 OpenTelemetry (OTel) Operator</strong></p><p>The Kubernetes (k8s) Operator is a software framework designed to simplify the management of complex, stateful applications on Kubernetes, abstracting away the complexities of the underlying Kubernetes API. They provide a high-level abstraction that makes it easy to manage the application’s lifecycle, including things like deployment, scaling, health checks and upgrades.</p><p>In this way, the OpenTelemetry (OTel) Operator for k8s is a powerful delivery mechanism that simplifies the deployment and management of OpenTelemetry components in Kubernetes environments. It streamlines the process from instrumenting applications for distributed tracing, metrics and logs to forward them to their final storage by making the OpenTelemetry data collection process easier.</p><p>All the way from basic installs to auto pilot scheduling and scaling, the <a href="https://sdk.operatorframework.io/docs/overview/operator-capabilities/">operator SDK</a> provides best practices and definitions to achieve different capacity levels. In particular, contributions are done every day to enhance the capabilities of the OpenTelemetry Operator under this framework to ease the management of the OpenTelemetry Collector lifecycle and monitoring.</p><p>Red Hat OpenTelemetry Operator not only manages the collector piece but also the <a href="https://github.com/open-telemetry/opentelemetry-operator#opentelemetry-auto-instrumentation-injection">auto instrumentation injection</a>. This means that the operator is able to inject and configure libraries which auto-instrument the code for you after the Instrumentation resource is added with the desired configuration for the SDK . Python, Go, Java, .Net, nodeJS are supported out of box and more/other platform support on the way. The values of OTel Operator:</p><pre>[I] Automated Setup: Automates the setup and configuration of OpenTelemetry components, reducing the manual effort required to deploy and manage them. It handles the deployment of the Collector, agent, and other necessary components, ensuring a consistent and reliable configuration.<br><br>[II] Empowered Configuration: With the Operator, you can easily adjust the configuration of OpenTelemetry components without redeploying them. This dynamic configuration capability allows for real-time changes to samplingrates, trace and metric exports, and other settings, providing greater flexibility and adaptability. It also enforces a consistent configuration across different deployments, ensuring that all applications and services within the Kubernetes cluster adhere to the same OpenTelemetry settings. This consistency simplifies the management and monitoring of distributed applications.<br><br>[III] Integration with Kubernetes Resources: Integrates seamlessly with Kubernetes resources allowing you to define OpenTelemetry configurations using custom resources (CRs). This native Kubernetes integration makes it easyto manage OpenTelemetry components using familiar Kubernetes tools and concepts.<br><br>[IV] Scaling and High Availability: The Operator enables effortless scaling and high availability for OpenTelemetry components. It can automatically handlethe replication and load distribution of OpenTelemetry resources as your application&#39;s needs change.<br><br>[V] Easy Updates and Rollbacks: Upgrading or downgrading OpenTelemetry versions becomes straightforward for the Operator. It simplifies updating the components and allows for easy rollbacks in case of any issues with the new version.<br><br>[VI] Multi-Tenancy Support: The OpenTelemetry Operator supports multi-tenancy, allowing you to manage separate OpenTelemetry instances for different teams, namespaces, or projects within the Kubernetes cluster.<br><br>[VII] Centralized Management: With the Operator, you can manage OpenTelemetry across multiple clusters from a centralized control plane. This centralized approach reduces the administrative overhead of managing distributed tracing and metrics in large-scale Kubernetes environments.<br><br>[VIII] Community-Driven: The OpenTelemetry Operator benefits from a vibrant community with ongoing contributions and improvements. This means users can leverage the collective knowledge and expertise of the community to enhance their OpenTelemetry deployment.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/0*_AZpjCtGXO53WdtH" /><figcaption>Figure-1 OpenTelemetry Architecture</figcaption></figure><p>The architecture of the OTel Operator is very flexible but is based on few simple concepts:</p><pre>&gt; The receiver ingests the telemetry signal (logs, metrics, traces). The OpenTelemetry Collector offers a vendor-agnostic implementation for receiving, processing, and exporting telemetry data. In addition, it removes the need to run, operate, and maintain multiple agents/collectors to support collecting open-source telemetry data formats (e.g., Jaeger, Prometheus, etc.) and exporting them to multiple open-source or commercial back-ends.<br><br>&gt; Then, data is processed in the processor pipeline. This is an excellent place to centralize your data manipulation in a single point instead of needing to instrument data across your many microservices written in different programming languages. It can also help to sample, and extract metrics from spans or even balance.<br><br>&gt; Finally, you export the processed data to your favorite backend via the exporter component. Again, several vendors contribute here to help you achieve seamless integration. Loki, Prometheus remote-write, and Kafka exporters are just a few examples.</pre><p>Important prospects:</p><pre>&gt; Observability signals are covered in the OTLP standard. For example, if you want to send all traces, logs, and metrics to an external ingester/backend, the only exporter you&#39;d need is the OTLP one.<br><br>&gt; The OTel operator can inject and configure OpenTelemetry auto-instrumentation libraries such as Apache HTTPD, DotNet, Go, Java, NodeJS, and Python.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*crt7YjHrMOYg7fRr" /><figcaption>Figure-2 OTel Instrumentation Example</figcaption></figure><p>Being this a very active project with contributions from many companies, one can easily realize how great the challenge of supporting all of them is. From small tests to integration, end-to-end and day to day bugs. That’s why Red Hat OpenShift provides a <a href="https://github.com/os-observability/redhat-opentelemetry-collector/blob/main/manifest.yaml">list of supported components</a>: receivers, exporters, extensions, processors and connectors. This list is changing by the day, to help our customers make use of the flexibility this amazing project provides, while being certain that it performs at its best.</p><p><strong>4.0 OTel Base Configurations</strong></p><p>OpenTelemetryCollector configuration provides means to collect data via OTLP protocol and send to Jaeger (traces) and Prometheus (metrics).</p><pre>apiVersion: opentelemetry.io/v1alpha1<br>kind: OpenTelemetryCollector<br>spec:<br>  config: |<br>    receivers:<br>      otlp:<br>        protocols:<br>          grpc:<br>          http:<br>            cors:<br>              allowed_origins:<br>                - &quot;http://*&quot;<br>                - &quot;https://*&quot;<br><br>    exporters:<br>      logging:<br>      otlp:<br>        endpoint: &quot;my-jaeger-collector:4317&quot;<br>        tls:<br>          insecure: true<br>      prometheus:<br>        endpoint: &quot;0.0.0.0:9464&quot;<br>        resource_to_telemetry_conversion:<br>          enabled: true<br>        enable_open_metrics: true<br><br>    processors:<br>      batch:<br>      filter:<br>        metrics:<br>          exclude:<br>            match_type: strict<br>            metric_names:<br>              - queueSize<br><br>    service:<br>      pipelines:<br>        traces:<br>          receivers: [otlp]<br>          processors: [batch]<br>          exporters: [otlp, logging]<br>        metrics:<br>          receivers: [otlp]<br>          processors: [filter, batch]<br>          exporters: [logging, prometheus]<br>        logs:<br>          receivers: [otlp]<br>          processors: [batch]<br>          exporters: [logging]<br>  ingress:<br>    route: {}<br>  mode: deployment<br>  ports:<br>    - appProtocol: http<br>      name: otlp-prometheus<br>      port: 9464<br>      protocol: TCP<br>      targetPort: 9464<br>  replicas: 1<br>  resources: {}<br>  targetAllocator:<br>    prometheusCR: {}<br>    resources: {}<br>  upgradeStrategy: automatic<br><br>                    Figure-3 OTel Collector Configuration</pre><p>In order connect a java application to OTel Collector, instrumentation can be configured as follows:</p><pre>apiVersion: opentelemetry.io/v1alpha1<br>kind: Instrumentation<br>metadata:<br>  name: my-instrumentation<br>spec:<br>  exporter:<br>    endpoint: http://otel-collector:4317<br>  propagators:<br>    - tracecontext<br>    - baggage<br>    - b3<br>  sampler:<br>    type: parentbased_traceidratio<br>    argument: &quot;0.25&quot;<br>  java:<br>    image: &gt;-<br>      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.26.0<br>    resources:<br>      limits:<br>        cpu: 500m<br>        memory: 64Mi<br>      requests:<br>        cpu: 50m<br>        memory: 64Mi<br><br>                    Figure-4 OTel Instrumentation Configuration</pre><p>To enable injection by an annotation so that all pods within that namespace will get instrumentation, the annotation may also be applied to individual PodSpec objects, available as part of Deployment, Statefulset, and other resources as follows:</p><pre>kind: Namespace<br>apiVersion: v1<br>metadata:<br>  labels:<br>    instrumentation.opentelemetry.io/inject-java: &#39;true&#39;<br>…<br><br>                   Figure-5 Auto-Instrumentation Configuration</pre><p><strong>5.0 5G Application Observability Sample Integrations By 3GPP Specifications</strong></p><pre>/**<br>* 3GPP TS 32.404 Measurement Definition <br>*/<br>public static class Type {<br> private String ioc;<br> private String family;<br> private String typename;<br> private Method method;<br> private String unit;<br> private List&lt;Measurement&gt; measurements;<br> …<br>}<br><br>/**<br>* 3GPP TS 32.404 Collection Methods<br>*/<br>public enum Method {<br> /** Cumulative Counter*/<br> CC,<br> /** data being measured can vary up or down during the period of measurement; */<br> GAUGE,<br> /** (Discrete Event Registration),;*/<br> DER,<br> /** Status Inspection */<br> SI<br>}<br><br>    Figure-6 5G application performance measurement sample definition</pre><pre>/**<br>*  3GPP TS 28.552 Counters Definitions<br>*/<br>public enum Counter {<br> /** This measurement provides the mean and max number of PDU sessions. */<br> SessionNbr(Family.SM, Method.SI, //<br>   Measurement.of(&quot;SessionNbrMean&quot;, Aggregate.MEAN), //<br>   Measurement.of(&quot;SessionNbrMax&quot;, Aggregate.MAX)),<br><br> /** This measurement provides the number of PDU sessions requested to be created by the SMF */<br> PduSessionCreationReq(Family.SM, Method.CC),<br><br> /** This measurement provides the number of PDU sessions successfully created by the SMF.*/<br> PduSessionCreationSucc(Family.SM, Method.CC),<br><br> /** Total number of the SmContext create request.  */<br> PostSmContextsNbr(Family.PDU_SES, Method.CC),<br><br>      …<br>  <br> private final Type type;<br><br> Counter(String family, Method method) {<br>  this(family, method, (Measurement[]) null);<br> }<br><br> Counter(String typename, String family, Method method, Measurement... measurements) {<br>  this.type = new Type(&quot;SMF&quot;, family, typename == null ? name() : typename, method, measurements);<br> }<br>       …<br> }<br><br>            Figure-7 5G application (SMF) counter sample definitions</pre><pre>/**<br>*  Counter Event hold dynamic attributes for the target counter type <br>*/<br>public class CounterEvent {<br> public final Counter.Type type;<br> public final String subcounter;<br> public final long value;<br> public CounterEvent(Counter.Type type, String subcounter, long value) {<br>  this.type = type;<br>  this.subcounter = subcounter;<br>  this.value = value;<br> }<br><br> public CounterEvent of(String subcounter) {<br>  return new CounterEvent(type, subcounter, 1);<br> }<br><br> public CounterEvent of(String subcounter, long value) {<br>  return new CounterEvent(type, subcounter, value);<br> }<br>}<br><br>            Figure-8 5g application counter event sample definition</pre><pre>/**<br>*  3GPP TS 29502 SmContextCreate Rest Api JAXRS Implementation<br>*/ <br>@Component<br>@Path(&quot;/nsmf-pdusession/v1/sm-contexts&quot;)<br>public class SmContextsApi {<br> @POST<br> @Consumes({ &quot;application/json&quot; })<br> @Produces({ &quot;application/json&quot;, &quot;multipart/related&quot; })<br> @Operation(operationId = OperationId.PostSmContexts, <br>                  description = &quot;Create SM Context, response = NSmContextCreatedData.class&quot;,)<br> @ApiResponses(value = { @ApiResponse(responseCode = &quot;201&quot;, <br>                  description = &quot;successful creation of an SM context&quot;)}<br> public void createSmContext( <br>             @Context HttpHeaders headers, //<br>  @Context SecurityContext securityContext, //<br>  @Parameter String body, //<br>  @Context UriInfo uriInfo){<br> stopwatch().start(&quot;flow.onNrSessionCreate&quot;);<br> nLoggerSys.atDebug(Marks.SM).log(“...”, message);<br> counterActor.tell(PduSessionCreationReq.of(data.getSNssai()), ActorRef.noSender());<br> counterActor.tell(PduSessionCreationReq.of(data.getRequestType()), ActorRef.noSender());<br> ...<br>      }<br>}<br><br>            Figure-9 5g application data context sample definition</pre><pre>/**<br>*  Metric Collector Actor (Akka)<br>*  Process Counter Events and update metrics  <br>*/ <br>public class CounterActor extends AbstractActor {<br>/**<br>* Micrometer Meter Registry<br>*/<br>MeterRegistry meterRegistry;<br><br><br>/**<br>* Cache holds all metric data by typename and subcounter<br>* @param list<br>*/<br>Map&lt;String,Map&lt;String,AtomicLong&gt;&gt; meters;<br><br>/**<br>* Actor behavior <br>*/<br>@Override<br>public Receive createReceive() {<br> return receiveBuilder()<br>             .match(CounterEvent.class, this::process)<br>  .build();<br>}<br><br>/**<br>* Update Counter Metric Value <br>* @param list<br>*/<br>public void process(CounterEvent event) {<br> Type type = event.type;<br> var map = meters.computeIfAbsent(type.getTypename(), k -&gt; new HashMap&lt;&gt;());<br> var valueAccess = map.get(event.subcounter);<br> if (valueAccess == null) {<br>  Tags tags = Tags.of(&quot;subcounter&quot;, Utils.nvl(event.subcounter, &quot;&quot;));<br>  valueAccess = meterRegistry.gauge(type.getTypename(), tags, new AtomicLong(event.value));<br>  map.put(event.subcounter, valueAccess);<br> } else {<br>  if (type.getMethod() == Counter.Method.CC) {<br>   valueAccess.incrementAndGet();<br>  } else if (type.getMethod() == Counter.Method.GAUGE) {<br>   valueAccess.set(event.value);<br>  }<br> }<br>}<br>    …<br>}<br><br>          Figure-10 5g application counter actor sample definition</pre><pre>public static final String INTEGRATION_REQ_LOG_DEBUG_TEMPLATE = &quot;[{}_REQ]\nDIRECTION:{} TO {}\nENDPOINT:{}\nMESSAGE DETAIL: {}\n&quot;;<br><br>if(nLoggerIntegration.isDebugEnabled()) {<br> apiMsgCallback.getRequestDetails().add(Pair.of(MediaType.APPLICATION_JSON, bodyContent));<br> LoggingUtil.logDebugReqMessage(nLoggerIntegration, apiMsgCallback);<br>} else if(nLoggerIntegration.isInfoEnabled()) {<br> LoggingUtil.logInfoIncReqMessage(nLoggerIntegration, apiMsgCallback);<br>}<br><br>public static void logDebugReqMessage(NLogger logger, SmfApiMsgCallback callback) {<br> logger.atDebug(SmfMarks.INTEGRATION).log(Constants.INTEGRATION_REQ_LOG_DEBUG_TEMPLATE  <br>   , callback.getSmfService().getOperationId() <br>   , String.join(&quot;,&quot;, callback.getSmfService().getServiceConsumers())  <br>   , String.join(&quot;,&quot;, callback.getSmfService().getServiceProducers())  <br>   , callback.getEndPoint().toString()  <br>   , getDetails(nLoggerIntegration.isDebugEnabled(),callback.getRequestDetails())<br> );<br>}<br>          Figure-11 5G AMF to 5G SMF service request logging enablement</pre><pre>10:34:03.555 DEBUG [clktewiga001euc46tdfaxaeq] com.i2i_systems.n5g.smf.integration.SmfSmContextCreateTransformer - [PostSmContexts_REQ]<br>DIRECTION:AMF TO SMF<br>ENDPOINT:http://localhost:2550/nsmf-pdusession/v1/sm-contexts<br>MESSAGE DETAIL: <br>application/json : class NSmContextCreateData {<br> supi: imsi-286060000000051<br> unauthenticatedSupi: false<br> pei: imeisv-8637250421890738<br> gpsi: msisdn-905050000002<br> pduSessionId: 5<br> dnn: internet<br> selectedDnn: null<br> ...<br>application/vnd.3gpp.5gnas : BaseNasMsg [extendedProtocolDiscriminator=SESSION_MANAGEMENT_MESSAGE, pduSessionId=1, pti=5, messageType=PDU_SESSION_ESTABLISHMENT_REQUEST]<br>...<br><br>                         Figure-12 AMF/SMF Log Output</pre><p><strong>6.0 Lessons Learned</strong></p><blockquote>“Ensure that the data retention periods for your backend are sufficiently extended to facilitate the transfer to long-term storage For example; prometheus default configuration is 24h for retention unless spec.retention and spec.retenionSize are empty. (Ref:<a href="https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ByteSize">Link</a>)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*ND3_zaEFyc7sn9-2" /><figcaption>Figure-13 Prometheus Config for Retention</figcaption></figure><p><strong>7.0 Visualization/Reporting</strong></p><p>Data collected by OTel gets written to Prometheus as ephemeral backend, which can be used by data visualization (Figure-15) and/or external data pipelines consuming the same data (Figure-16) via OTel exporter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FIuaxxrS-Rm4mP4e" /><figcaption>Figure-14 Sample 5G Observability Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gXNPEb3woiBadSzB" /><figcaption>Figure-15 Sample 5G Observability Dashboard — <a href="https://github.com/ddosify/ddosify">DDosify</a></figcaption></figure><p><strong>8.0 Conclusions</strong></p><blockquote>As we conclude our exploration of OpenTelemetry (OTel) in the context of 5G networks, several key insights emerge. Firstly, OTel’s flexibility and adaptability make it an invaluable tool in the complex ecosystem of 5G technology. Through its extensive capabilities in data collection and observability, OTel enhances our understanding and management of 5G network operations.</blockquote><p>Throughout this discourse, we’ve highlighted the strategic integration of OTel with Cloud Native Network Functions (CNF) and its impact on AIOps in 5G environments. The practical applications demonstrated how OTel’s robust framework can be leveraged to streamline network operations and improve performance metrics.</p><p>However, our journey into OTel’s application in 5G is far from complete. Looking ahead, there are several avenues for further exploration and development:</p><pre>&gt; Enhanced Automation: Future studies should focus on automating more aspects of OTel’s integration with 5G, reducing manual intervention and further simplifying processes for network operators.<br><br>&gt; Interoperability and Standards: Continuing to work on standardizing OTel practices across different platforms and vendors will be crucial for ensuring seamless integration and wider adoption in the 5G realm.<br><br>&gt; Community Collaboration: Encouraging ongoing collaboration within the OTel community is vital. Sharing best practices, challenges, and solutions can accelerate innovation and application of OTel in 5G networks.</pre><blockquote>While OpenTelemetry has already proven to be a significant asset in the realm of 5G, its full potential is yet to be realized. The journey ahead is replete with opportunities for innovation and improvement, promising to revolutionize further how we understand and manage 5G networks.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bccf100e753f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/5g-open-telemetry-otel-bccf100e753f">Episode XVIII 5G Open Telemetry (OTel)</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode-XVII [BKeeper]]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xvii-bkeeper-a23882feb75?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/a23882feb75</guid>
            <category><![CDATA[ebpf]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[kernel]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Fri, 01 Sep 2023 16:03:35 GMT</pubDate>
            <atom:updated>2023-12-22T02:14:03.755Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors: [</strong>Fatih E. NAR, Dave Tucker, Arkady Kanevsky at Red Hat], [Dave Cremins, Vivek Kashyap, John Browne at Intel]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*9nAHzaqNmEsPbyWP" /><figcaption><strong>DALLe Beekeeper with Hives on the Boat</strong></figcaption></figure><p><strong>Update [December 2023]:</strong> bpfd project renamed to bpfman to have clear use and self-explanation. Red Hat has initiated a <a href="https://github.com/cncf/sandbox/issues/76">donation under CNCF Sandbox initiative</a>.</p><p><strong>1.0 Introduction</strong></p><p>In “<a href="https://medium.com/open-5g-hypercore/episode-xvi-the-eye-of-the-bee-holder-a8e81be2dfa2">Episode XVI: The Eye of the BEE-Holder</a>,” we delved into potential data sources and their augmentation for comprehensive observability solutions bolstered by AIOps capabilities.</p><p>One of the standout methods for enhancing observability, as highlighted in Episode XVI Section 3.3, is the utilization of eBPF probes and applications. Through our journey of designing eBPF applications and critically assessing others, we discerned that they could occasionally exert profound impacts on the core framework of the application platform, particularly Kubernetes (K8s) and its interlinked services. In some cases, these core K8s services teetered on the brink of dysfunction or became overshadowed by dominant eBPF loaders exerting authority across nodes.</p><blockquote>This article dissects the intricacies of deploying eBPF programs for observability.</blockquote><blockquote>Our aim? We are glad you asked 🙂; We would like to ensure optimal performance without compromising the stability and integrity of the application platform or jeopardizing the applications it hosts, while leveraging eBPF technology.</blockquote><p><strong>2.0 Background</strong></p><p>The eBPF (Extended Berkeley Packet Filter) functionality in the Linux kernel is immensely powerful, and the `CAP_BPF` capability provides access to a significant subset of these functionalities. The `CAP_BPF` offers access to great capabilities, in return for potent permissions, such as:</p><p>1. Kernel Observability: eBPF allows for the observation of kernel internals. A process with the appropriate permissions can attach eBPF programs to various kernel hooks, tracing system calls, network packets, scheduler events, and more. This can reveal sensitive information about system operations.</p><p>2. Kernel Behavior Modification: eBPF programs can modify the kernel’s behavior beyond mere observation. For example, an eBPF program can alter or block network packets. In the hands of a malicious entity, this can be misused for nefarious purposes like data tampering.</p><p>3. Access to eBPF Maps: eBPF maps are key-value stores in kernel space with which eBPF programs can interact. A process with `CAP_BPF` can read from and write to these maps, potentially changing data or behavior defined by other eBPF programs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KbIbw3aOI4wcZ9Zz" /><figcaption>Figure-1 eBPF deployment(s) in Kubernetes with chaos</figcaption></figure><p>4. Unrestricted Program Loading: Without additional controls, a process with `CAP_BPF` can load and attach eBPF programs to many kernel hooks. There isn’t an inherent mechanism to restrict which eBPF programs can be loaded or which hooks can be attached.</p><p>5. Potential for Resource Exhaustion: eBPF programs and maps consume system resources. A process that can freely load and manage eBPF entities could potentially exhaust system (hardware and/ or software) resources, leading to a Denial of Service (DoS) situation.</p><p>6. Indirect Access to Other Capabilities: Loading some eBPF program types can require other capabilities. For instance, attaching an XDP or TC eBPF program to a network interface would need an application to be granted `CAP_NET_ADMIN`.</p><p>The scope of these capabilities is much broader than just `CAP_BPF`, which goes against the principle of least privilege.</p><p>Given these powers, `CAP_BPF` should be considered equivalent to full root privileges and handled &amp; used carefully.</p><p><strong>3.0 Wrong Way [Chaos of Bees]</strong></p><p>The challenges &amp; disruptive impacts of using external (i.e., non-regulated) eBPF loaders:</p><p>[I] Privileged access requirements:</p><pre>[I.A] eBPF loaders necessitate privileged pods.<br><br>[I.B] eBPF-enabled apps mandate, at the least, <br>CAP_BPF permissions (details described in the previous section), <br>with the possibility of additional permissions based on <br>the specific program type.<br><br>[I.C] Linux capabilities are expansive, <br>making it tricky to limit a pod to the most essential privileges. <br>This vastness can lead to inadvertent or deliberate system compromises <br>(such as container escape cases).</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJQQtrZHOv3ed_LPhr0sXg.png" /><figcaption>Figure-2 DaemonSet to Daemon Acts (<a href="https://unit42.paloaltonetworks.com/kubernetes-privilege-escalation/">Link1</a>, <a href="https://security.googleblog.com/2022/05/privileged-pod-escalations-in.html">Link2</a>, <a href="https://sysdig.com/blog/analysis-of-supply-chain-attacks-through-public-docker-images/">Link3</a>)</figcaption></figure><p>[II] Handling Multiple eBPF Programs:</p><pre>[II.A] Not every eBPF hook supports numerous programs simultaneously.<br><br>[II.B] Specific software operating with eBPF may assume exclusive <br>access to an eBPF hook, potentially leading to the unintended <br>displacement of existing programs upon attachment, <br>resulting in unpredictable or silent failures.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oQXkP5Gq2tzWvr6D" /><figcaption>Figure-3 BPF Kernel Datapath Revamped Incident Reporting (<a href="https://lpc.events/event/16/contributions/1353/">Link3</a>)</figcaption></figure><p>[III] Multi-Tenancy Problem:</p><pre>[III.A] eBPF lacks namespacing, allowing privileged access to <br>all kernel-stored information. Traditionally, there were no ACLs <br>or RBAC controls to limit the loading of specific probe types.</pre><p>[IV] Debugging Deployment Challenges:</p><pre>[IV.A] Cluster administrators must be made aware of using eBPF programs <br>within a cluster.<br><br>[IV.B] Interactions between various eBPF programs can lead to <br>unexpected complications (example given II.B above).<br><br>[IV.C] SSH access or a privileged pod becomes indispensable <br>to ascertain the status of eBPF programs on each node <br>within the cluster.</pre><p>[V] Lifecycle Management Complexities:</p><pre>[V.A] While userspace libraries <br>(such as libbpf, bcc, bpftool, pyebpf, bpftrace etc.) <br>support basic eBPF program loading and unloading, <br>supplementary coding is often essential for holistic lifecycle <br>management and continuous impact management.<br><br>[V.B] Deployment within Kubernetes is more complex. <br>It demands the development of a daemon to load the eBPF bytecode, <br>subsequently deploying it via a DaemonSet. <br>This process necessitates a deep understanding of the eBPF program <br>lifecycle to ensure consistent program loading and <br>seamless handling of pod restarts, upgrades, and movement.</pre><p>[VI] Version Control Limitations:</p><pre>[VI.A] In current eBPF-enabled K8s deployments, <br>the eBPF Program is typically integrated into the userspace binary, <br>which manages and communicates with it. <br><br>This integration hinders the ability to maintain precise versioning <br>control of the bpfProgram in relation to its corresponding <br>userspace component.</pre><p>[VII] Performance considerations</p><pre>[VII.A] Since probes run in the kernel space, <br>they have a lot of powers for good and evil. <br>With so much power comes heavy responsibilities. <br>You want probes to be very fast and small. <br>Just read what you need to write quickly into a map, <br>avoiding expensive processing at all costs!<br><br>[VII.B] Poorly performing probes can impact all areas of the system. <br>For example, a user could load an uprobe to trace <br>`free` and `malloc` in libc adding a context switch <br>to the kernel (to execute the probe) at a very high frequency, <br>causing the performance of all programs that link to libc to suffer.</pre><p><strong>4.0 The Kubernetes Native Way [Right-Way, Organized Beehive]</strong></p><p><a href="https://github.com/bpfman/bpfman">bpfman</a> is an innovative software stack designed to simplify the loading, unloading, modification, and monitoring of eBPF programs, whether on an individual host or across a Kubernetes cluster. It’s developed in Rust and built upon the Aya eBPF library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*acPnasQaLQ5JiVAP" /><figcaption>Figure-4 eBPF Deployment(s) in Kubernetes the Right Way</figcaption></figure><p>Core Components:</p><p>[1] bpfman Daemon:</p><pre>bpfman daemon is asystem-wide service enabling the loading, <br>unloading, modification, and monitoring of eBPF programs, <br>accessible through a gRPC API.</pre><p>[2] eBPF CRDs:</p><pre>[2.1] Offers a collection of Custom Resource Definitions (CRDs), <br>such as XdpProgram and TcProgram, expressing the intent <br>to load specific eBPF programs.<br><br>[2.2] Features a bpfman-generated CRD (BpfProgram) that <br>depicts the runtime state of loaded programs.</pre><p>[3] bpfman-agent:</p><pre>bpfman-agent operates within a container in the bpfman daemonset, <br>this agent ensures that the desired eBPF programs are loaded <br>as required for each node.</pre><p>[4] bpfman-operator:</p><pre>Leveraging the Operator SDK, this operator oversees the installation <br>and lifecycle of both the bpfman-agent and associated CRDs within <br>a Kubernetes environment.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X0WcGgl0FYIKq1ja" /><figcaption>Figure-5 bpfman Architecture</figcaption></figure><p>Critical benefits of having a central, authoritative eBPF loader (single pane solution); <strong>bpfman</strong>:</p><p>Security:</p><pre>&gt; Maintaining stringent oversight of the bpfman daemon ensures it exclusively <br>possesses the necessary privileges to load eBPF programs, <br>thereby preserving the unprivileged status of user pods. <br>While we&#39;re committed to minimizing the spread of CAP_BPF privileged <br>within pods, we anticipate that it might never be entirely restricted. <br>Nevertheless, our aim/recommendation is to alert cluster administrators <br>about any &quot;rogue&quot; bpf programs in the k8s cluster.<br><br>&gt; Administrators gain increased control over program loading and can set <br>rules for the eBPF program networking sequence.<br><br>&gt; Facilitating bytecode ownership verification through container <br>image signing.<br><br>&gt; API access is managed through standard RBAC methods, <br>restricting which eBPF features and pods can use hooks. <br>This lets you define which programs to load and sets the stage <br>for future developments. Soon, bpfman aims to provide refined control, <br>such as inserting uprobes into &quot; myapp &quot; processes <br>within pods labeled &quot;myapp&quot; while limiting access to <br>specific functions. <br><br>&gt;As part of our commitment to ensuring the security of the <br>code bpfman loads, we are excited about our upcoming integration<br>with Sigstore/Rekor, which will vouch for the authenticity <br>of the eBPF code.</pre><p>Visibility &amp; Debuggability:</p><pre>&gt; Enhanced insight into active eBPF programs on a system, <br>optimizing debuggability for various stakeholders.<br><br>&gt; It offers a comprehensive view of all eBPF programs <br>across cluster nodes.<br><br>&gt; Can assist with collecting performance data from <br>eBPF programs to help debug system performance/instability issues.</pre><p>Multi-program Support:</p><pre>&gt; Enables multiple eBPF programs from various users <br>to coexist harmoniously.<br><br>&gt; Utilizes the libxdp multiprog protocol for allowing <br>multiple XDP programs on a single interface, with similar support <br>for TC programs.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tNO2rElWZTCQ7qSA" /><figcaption>Figure-4 Bpfman Operator</figcaption></figure><p>Productivity:</p><pre>&gt; Streamlines eBPF program deployment and management within a <br>Kubernetes setting.<br><br>&gt; Developers can focus on core tasks, leaving program lifecycle <br>concerns like loading, attaching, and pin management to bpfman.<br><br>&gt; Supports using libraries like Cilium, libbpf, and Aya <br>for eBPF development, but with loading/unloading streamlined by bpfman.<br><br>&gt; Introduces eBPF Bytecode Image Specifications for intricate <br>versioning control of userspace and kernelspace programs.</pre><p><strong>5.0 Conclusion</strong></p><p>While the potential pitfalls and challenges of eBPF deployment are undeniable, as explored in this article’s initial sections, a path exists that allows for the seamless integration of eBPF into K8s platforms. With tools like bpfman at our disposal, we can craft a solution that keeps the delicate balance within K8s application platform intact.</p><blockquote>The juxtaposition of the “wrong way” versus the “right way” elucidated that while the journey of eBPF integration is riddled with complexities, the destination — when approached with the right tools and perspective — can be one of security, visibility and enhanced productivity.</blockquote><blockquote>By implementing &amp; using bpfman, we’re not just addressing the challenges head-on; we’re moving towards an era where eBPF management becomes fluid and adaptable without breaking the integrity of application platforms and/or impacting overall performance.</blockquote><blockquote>Understanding eBPF within Kubernetes demands respect for its power, nuances, and the tools and techniques to harness its full potential. Through this article, we hope to have illuminated that path, taking one step closer to achieving harmony of performance and stability.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a23882feb75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xvii-bkeeper-a23882feb75">Episode-XVII [BKeeper]</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XVI: The Eye of the BEE-Holder]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xvi-the-eye-of-the-bee-holder-a8e81be2dfa2?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8e81be2dfa2</guid>
            <category><![CDATA[5g]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[5g-technology]]></category>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Thu, 13 Jul 2023 15:41:52 GMT</pubDate>
            <atom:updated>2023-07-14T13:49:43.184Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors: </strong>Fatih Nar (Red Hat), Abdurrahim Suslu (AIOPSONE), Eric Lajoie (Red Hat), Min Xie (Telenor), Sean Cohen (Red Hat), Dave Tucker (Red Hat), Shujaur Mufti (Red Hat)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*PXW8_wHVIi1S4zvDx1CKGQ.jpeg" /><figcaption>DALL-E: Observing the world from an eye of a bee with feeling the stress &amp; joy of a ship captain.</figcaption></figure><h4><strong>1.0 Introduction</strong></h4><p>Welcome to the second article in our AIOps series. In the first article (<a href="https://medium.com/open-5g-hypercore/episode-xv-magic-ops-cda74284fe74">link</a>), we covered the fundamentals of AIOps, exploring its definition, functionalities, and benefits that it brings to IT operations &amp; enterprise business teams. Building upon that foundation, this article delves deeper into the AIOps technology stack resulting from Red Hat, Telenor, and AIOPSONE collaboration. Here, we will provide insights into the components of the AIOps technology stack and present ready-to-use solution components backed by open-source reference projects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mosh-C4sIqnHvfh5" /><figcaption>Figure-1 AIOPSONE Solution Architecture</figcaption></figure><blockquote>To begin our exploration, we will first focus on the critical observability data sources/backends that feeds in to AIOps machinery and the related open-source technologies that can be leveraged to implement them effectively. By understanding observability data sources and technologies, you’ll realize how AIOps can be built on top of observability data.</blockquote><blockquote>Next, we will delve into the importance of standardizing observability data collection, particularly aligning with OTel and 3GPP standards. By incorporating standardized practices, we ensure data collection compatibility, interoperability, and consistency across different network functions within 5G networks and beyond.</blockquote><blockquote>Finally, we will showcase real-world operational use cases that demonstrate the capabilities of AIOps. These use cases have been developed based on field-driven requirements and highlight the practical application of AI-driven insights and automation in optimizing network performance, minimizing downtime, and enhancing overall operational and business efficiency.</blockquote><h4><strong>2.0 Fundamental Application Data</strong></h4><p>Here are some base observability data types that applications generate, gather &amp; store -&gt; that can be crunched by AIOps, along with their backend projects.</p><p><strong>2.1 Metrics</strong></p><p>A metric is a measurement of a service captured at runtime (i.e., time-bound informational elements). Logically, the moment of observing one of these measurements is known as a metric event that consists not only of the measurement itself but also the time it was captured and associated metadata. Application and request metrics are essential indicators of availability and performance. Custom metrics (such as 3GPP 5G CNF Metrics) can provide insights into how key performance indicators (KPI) impact the user experience or the business&#39;s success.</p><p>Metrics could be from; infrastructure like {CPU usage, memory usage, network IO, storage IO, etc.}, application-platform like {platform resource measures, utilization levels, performance counters, etc.} and/or custom metrics from your business application (i.e., tenant workloads such 5G CNFs). Metrics are usually stored in time-series databases, and popular time-series capable open-source technologies are;</p><ul><li><em>Prometheus is an open-source system monitoring and alerting toolkit. It offers a multi-dimensional data model, a flexible query language, and integrates with many sources to gather metrics. In a Kubernetes (K8s) environment, Prometheus can be configured to scrape metrics from individual pods, services, etc. It primarily supports a pull model for gathering metrics but can also push metrics via the Pushgateway component. It’s widely adopted in cloud-native environments and has built-in service discovery for many types of infrastructure. However, it’s primarily designed for domain-bound operation and doesn’t natively support long-term storage or global views (multi-domain) across multiple clusters, which limits large-scale deployments.</em></li><li><em>Thanos is an extension of Prometheus and was designed to address some of Prometheus’s limitations, such as long-term storage, high availability, and multi-cluster monitoring. It integrates with existing Prometheus servers and can store data in object storage systems (like S3 or GCS), allowing for cost-effective long-term storage. Thanos components are designed to be highly available and to provide a global query view across all connected Prometheus servers, which can be a significant advantage in large, distributed environments.</em></li><li><em>InfluxDB is an open-source time series database that handles high write and query loads. It’s often used for applications in IoT, analytics, and monitoring. InfluxDB supports a SQL-like query language and can work with Grafana for visualization. It’s generally easy to use and provides much functionality. However, some advanced features require the enterprise version (i.e., pay-to-use).</em></li><li><em>VictoriaMetrics is another open-source time series database that aims to provide a more cost-effective and scalable alternative to systems like Prometheus and Thanos. It can ingest data from Prometheus, Graphite, and InfluxDB, offering better performance and lower resource usage than similar systems. It can also replicate data for high availability, shard data for increased capacity, and provides long-term storage options.</em></li><li><em>Cortex is an open-source project that adds horizontal scalability to Prometheus. This is achieved by using a microservices architecture to distribute the responsibilities of Prometheus (like ingesting metrics, storing data, and querying data) across multiple nodes. This can significantly increase your monitoring setup’s scalability and add support for multi-tenancy, which means that different teams or customers can have separate, isolated views of their metrics. Like Thanos, Cortex supports long-term metrics storage in object stores like S3 or GCS.</em></li></ul><p>[Field-View]: What we see in the field is Prometheus being popular per k8s cluster domain; however, as noted above, it comes with domain-bound, scalability, redundancy/resiliency limitations where Thanos and Cortex fill the gaps -&gt; to make Prometheus more suitable for large-scale, long-term use.</p><p><strong>2.2 Logs</strong></p><p>A log is a time-stamped text record with structured (recommended) or unstructured metadata. Of all telemetry signals, logs have the most significant legacy. Most programming languages or well-known, widely used logging libraries (remember the famous Log4J? lol) have built-in capabilities. Although logs are an independent data source, they may also be attached to spans.</p><p>Logs provide insights into what happened and when it happened in your application. They’re valuable for troubleshooting issues and understanding the sequence of events leading up to a problem. Popular logging-related technologies are;</p><ul><li><em>Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus, designed to handle massive amounts of log data.</em></li><li><em>Fluentd / Fluent Bit are open-source data collectors that unify data collection and consumption. They can gather logs from many sources and deliver them to many destinations, including Elasticsearch, Cloudwatch, and Datadog.</em></li><li><em>Logstash is a server-side data processing pipeline that simultaneously ingests data from multiple sources, transforms it, and then sends it to a stash (like Elasticsearch). Logstash, like Fluentd, also offers a wide array of input, filter, and output plugins to customize its functionality. However, it’s generally more resource-intensive than Fluentd.</em></li><li><em>ELK refers to a stack composed of Elasticsearch for search, Logstash for centralized logging, and Kibana for visualization. It’s a widely used solution for log management and analytics, though it can be complex to manage at scale. It is now often referred to as the Elastic Stack since it can also include Beats, a platform for single-purpose data shippers.</em></li><li><em>EFK is similar to ELK but replaces Logstash with Fluentd. Fluentd is often lighter-weight and easier to manage than Logstash, making EFK a popular choice for Kubernetes logging.</em></li><li><em>Rsyslog is a high-performance legacy log processing system. It’s often used in traditional system logging setups and supports a wide variety of sources, transformations, and destinations. However, it needs to be more oriented toward modern, cloud-native environments and required to provide its own storage or visualization layer like ELK or EFK.</em></li></ul><p>[Field-View]: What we see in the field; is Fluentd which is popular at Infrastructure and Platform Layer software stacks; however, Loki is gaining popularity across applications DevOps community. For VNFs, we still see rsyslog being a log delivery mechanism for element management systems (EMS).</p><p><strong>2.3 Events</strong></p><p>Although not every standalone event is irrelevant from an observability perspective, events can provide robust input for RCA (Root Cause Analysis) in the AIOPS operations. Events record notable changes in your platform/services, which are carried over a messaging bus, popular message bus projects are:</p><ul><li><em>Kafka is a distributed event streaming platform that can handle real-time data feeds. It can capture all types of events in a Kubernetes environment.</em></li><li><em>RabbitMQ is a messaging broker that accepts and forwards messages. It can be used to manage event-driven applications in a Kubernetes environment.</em></li><li><em>Pulsar is a scalable, high-performance publish-subscribe messaging platform that you can use as an alternative to Kafka. Its architecture separates the serving and storage layers, making it a good choice for cloud-native applications.</em></li><li><a href="https://github.com/nats-io"><em>NATS</em></a><em> is an open-source, lightweight, and high-performance messaging system. It doesn’t support the same level of message durability as Kafka or RabbitMQ, but it is straightforward to deploy and can deliver extremely high throughput with low latency.</em></li></ul><p>[Field-View]: What we see in the field as Kafka is famous due to scalability, durability, and real-time processing performance. However, NATS is getting popular across IT administrations due to its low footprint for resource consumption and ease in lifecycle management.</p><p><strong>2.4 Traces</strong></p><p>Tracing is a powerful tool that provides insights into the path and life of a request as it travels across a distributed system. It proves invaluable when addressing latency concerns and debugging intricate interactions between services. Here are some popular tracing tech stacks:</p><ol><li><em>Jaeger is an open-source, end-to-end distributed tracing system created by Uber. It tracks requests across multiple microservices in environments such as Kubernetes.</em></li><li><em>Zipkin is considered obsolete by some due to the rise of newer solutions, Zipkin remains an open-source distributed tracing system to gather timing data needed to troubleshoot latency problems in service architectures.</em></li><li><em>Grafana Tempo is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. Tempo can ingest standard open-source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.</em></li><li><em>Elastic’s application performance monitoring (APM) system supports distributed tracing. It’s part of the Elastic Stack (ELK Stack), so it’s a seamless addition if you already use Elasticsearch for log storage.</em></li></ol><p>[Field-View]: What we see in the field as Jaeger is the popular choice due to broad language support, support for multiple storage backends, and integration well with K8s. However, distributed tracing with OTel (see Section 5.0 for more details) is catching up with a high adoption rate.</p><h4><strong>3.0 Advanced Telemetry Data</strong></h4><p>Here are some extremely valuable additional advanced observability data that can be used by AIOps, which may require different tools, stacks, and integration points to be utilized.</p><p><strong>3.1 Topology/Correlation Data (Graph Data)</strong></p><p>Topology data is a blueprint of your infrastructure/platform. It maps out the relationship and interconnections between various nodes (such as servers), services, and other resources within a network. By providing an interconnected view of all components, topology data helps you understand service dependencies and facilitates impact analysis in case a component fails or changes.</p><p>Consider a microservices-based architecture. It comprises multiple services that communicate with each other. The topology data for this environment would capture these services and the interaction paths between them. If a service fails, you can leverage topology data to identify other services that may be affected due to this failure.</p><p>Here’s a deeper dive into some potential sources for obtaining topology data:</p><p><em>1. The Kubernetes API serves as a treasure trove of data about the real-time state of a Kubernetes cluster. It tells you about the cluster’s nodes (or servers), the pods running on these nodes, and the services that tie them together. By harnessing this data, you can generate a topology map of your Platform-as-a-Service (PaaS) environment.</em></p><p><em>2. Similarly, the OpenStack API divulges comprehensive details about the current state of your Infrastructure-as-a-Service (IaaS) deployment. This comprises information about the compute nodes, active workloads, network configurations, storage volumes, and more. By collecting this data, you can draw up a topology map of your IaaS environment.</em></p><p><em>3. A service mesh emerges as a crucial consideration When assembling the topology data for a modern, microservices-based environment. This service mesh is a service layer solution to manage a high volume of network-based inter-process communication among services through APIs.</em></p><p><em>4. Network fabric data includes details about your network setup, such as segmentation data, insights about network overlays, subnet details, firewall rules, etc. Such data is critical to comprehend how different resources interact and ensure optimal network security and performance.</em></p><p>[Field-View]: There is no winner or loser in this category, as these offer a comprehensive understanding of your infrastructure/platform and how each piece interacts with the others, all of which can be used when &amp; where available for observability’s sake.</p><p><strong>3.2 External Tools</strong></p><p>Empowering AIOps machinery with data from external tools &amp; services can significantly enhance the depth of insight and precision in handling operations. Here are some additional data sources that can be leveraged:</p><ul><li><em>Container Monitoring Tools (Runtime Data): These specialized tools monitor container activities, looking for unusual patterns that may signify issues. They detect anomalies, enforce compliance rules, and protect against threats. Examples of such tools include Falco and Stackrox, an open-source project by Red Hat. They offer runtime security features and the ability to spot abnormal application behavior.</em></li><li><em>Application Performance Testing (APT): By applying performance tests and analyzing the outcomes, we can assess distributed applications’ performance, resilience, and reliability. This form of black-box testing is instrumental in detecting performance bottlenecks and potential failure points under load conditions. One of the popular open-source APT projects is </em><a href="https://github.com/ddosify/ddosify"><em>DDosify</em></a><em>.</em></li><li><em>Vulnerability Scanning Tools (Image Scanning Data): These specialized tools scan container images to identify known vulnerabilities. Examples include Stackrox, KubeArmor, Trivy, Clair, and Anchore. These tools ensure our container images are secure before deployment, helping prevent potential breaches and improve overall system security.</em></li><li><em>Network Performance Tools (Network Traffic Data): Network performance tools provide insights into the traffic between services in a Kubernetes environment. They can spot suspicious network behavior and, in some cases, prevent attacks. Tools like NetObserv Operator, Pixie, Calico, and Hubble (Cilium), which operate in Kubernetes environments, offer network policy enforcement and a certain degree of network visibility.</em></li><li><em>Identity and Access Management (IAM) Tools (Access Control Data): IAM tools maintain information about access controls in your Kubernetes cluster, detailing who has access to what resources. Examples include AWS IAM, Google Cloud IAM, and Azure Active Directory, all offering robust IAM capabilities on their respective platforms.</em></li><li><em>Compliance/Audit Tools (Compliance Data): These tools ensure that your Kubernetes configurations adhere to best practices or specific compliance frameworks. Stackrox, for instance, checks for secure deployment of Kubernetes by running checks documented in the CIS Kubernetes Benchmark. Meanwhile, Kyverno and OPA/Gatekeeper can enforce policies within your Kubernetes clusters, enhancing overall security.</em></li></ul><p><strong>3.3 Add-On Application Observability with eBPF</strong></p><p>If an application is written without observability in mind (i.e., does not generate its own consumable observability data), we can “inject” observability into it via;</p><ul><li><em>The use of service mesh that inserts a sidecar container between incoming/outgoing traffic and an application and collects metrics about the ingress and egress traffic in/out of the application. </em><a href="https://medium.com/open-5g-hypercore/episode-iii-meshville-7f0bb7ca0e3b"><em>Service mesh</em></a><em> does not offer application performance level intelligence or granular inside-the-application observability, only generic network stack-level metrics.</em></li><li><em>Using kernel-level application performance detail gathering with eBPF probes &amp; maps for; capturing granular application performance data (metrics &amp; logs), security posture analysis (sniffing), behavior modeling (low-level tracing), and advanced troubleshooting.</em></li></ul><p><strong>3.3.1 Use of eBPF for Observability</strong></p><p>Kernel-level details about a running workload can provide valuable insights into the performance and behavior of your application that were previously difficult to access. eBPF (extended Berkeley Packet Filter) is a powerful tool that enables such capabilities.</p><p>For instance, eBPF programs like Runqslower can report scheduling delays, shedding light on tasks waiting for CPU time. Funclatency, another eBPF program, can be used to instrument applications that don’t generate trace spans, revealing the average execution time of functions within a workload. Please see Figure-2 for other BPF capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XPuNsNhU-I_OhOaG" /><figcaption>Figure-2 BPF Tracing Tools (Ref: IOVISOR/BCC <a href="https://github.com/iovisor/bcc">Link</a>)</figcaption></figure><p>To simplify the management and monitoring of eBPF programs, <a href="https://github.com/bpfd-dev/bpfd">bpfd</a> has been developed. <a href="https://bpfd.dev/">bpfd</a> facilitates the loading, unloading, modifying, and monitoring of eBPF programs on individual hosts and Kubernetes clusters. It consists of the following core components:</p><ol><li><em>Daemon-Set: This system daemon supports the loading, unloading, modification, and monitoring of eBPF programs through a gRPC API.</em></li><li><em>eBPF CRDs: bpfd provides Custom Resource Definitions (CRDs) such as XdpProgram and TcProgram, which allow expressing intent to load eBPF programs. It also includes a bpfd-generated CRD (BpfProgram) that represents the runtime state of loaded programs.</em></li><li><em>bpfd-agent: This agent runs in a container within the bpfd daemon set and ensures that the requested eBPF programs for a given node are in the desired state.</em></li><li><em>bpfd-operator: An operator built with the Operator SDK manages the installation and lifecycle of bpfd-agent and the CRDs within a Kubernetes cluster.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/0*IbsX6l2qj0Ua61Wn" /><figcaption>Figure-3 eBPF Probes-Based Kubernetes Application Observability</figcaption></figure><p>The solution offers several benefits, including enhanced security and improved visibility and debuggability:</p><ul><li><em>Security: Only the tightly controlled bpfd daemon has the privilege to load eBPF programs, improving security. Access to the API can be controlled using standard RBAC methods, and administrators have control over program loading permissions.</em></li><li><em>Visibility &amp; Debuggability: The solution provides better visibility into the eBPF programs running on a system, enhancing debugging capabilities for developers, administrators, and customer support. Even if not all applications use bpfd, it still offers visibility into all eBPF programs loaded on cluster nodes.</em></li><li><em>Multi-program Support: The solution supports the coexistence of multiple eBPF programs from different users. It utilizes the libxdp multiprog protocol, allowing multiple XDP programs on a single interface. This protocol is also supported for TC programs, providing a unified multi-program experience across both TC and XDP.</em></li><li><em>Productivity: The solution simplifies the deployment and lifecycle management of eBPF programs within Kubernetes clusters. Developers can focus on program logic, while bpfd handles program lifecycle tasks such as loading, attaching, and pin management. Existing eBPF libraries can still be used for development and interaction with program maps via well-defined pinpoints managed by bpfd.</em></li><li><em>eBPF Bytecode Image Specifications: bpfd provides eBPF Bytecode Image Specifications, enabling fine-grained versioning control for userspace and kernelspace programs. It also allows for signing these container images to verify bytecode ownership. The gathered information from kernel space probes can be written into eBPF maps, which userspace processes can export, transform, and load into the desired backend/sink (please refer to section 5.0 on open telemetry for data collector and exporter mechanisms).</em></li></ul><p>By leveraging bpfd with eBPF application probes, we can gain valuable insights and streamline the management of eBPF programs within Kubernetes clusters.</p><p><strong>3.4 Hardware Observability</strong></p><p>Regarding hardware observability, two primary approaches can be employed. Each approach offers unique advantages and allows retrieving observability data from different sources. These approaches are:</p><ul><li><em>Direct Communication with Hardware Baremetal Controller: One way to achieve hardware observability is by directly communicating with the hardware’s bare-metal controller. This involves leveraging well-defined and supported communication protocols.</em></li><li><em>Retrieving Host Operating System Data: Another approach to hardware observability involves retrieving data from the host operating system through system calls. The host operating system collects and exposes various metrics and information about the underlying hardware.</em></li></ul><p><strong>3.4.1 Bare Metal Level</strong></p><p>The Redfish (when and where available properly, i.e., compliant with relevant specifications) provides a standardized, vendor-neutral interface for hardware-level observability in modern data centers and server environments with a REST API. The protocol manages distributed, converged, or software-defined resources and infrastructure.</p><p>We can leverage the Redfish API to collect real-time telemetry data from hardware components and gain insights into their health, performance, and operational status. The Redfish API offers a rich set of capabilities for hardware observability, including:</p><ol><li><em>Sensor Data Collection: The Redfish API allows the retrieval of sensor data from various hardware components, such as temperature sensors, power sensors, fan sensors, and voltage sensors. This data provides valuable information on the operating conditions of the hardware, enabling proactive monitoring, alerting, and predictive maintenance. Caveats can exist with an active collection of sensor data from a Bare Metal Controller (BMC). BMC access also needs to be restricted due to operational security needs, often resulting in a layer boundary that should be approached on a case-by-case basis.</em></li><li><em>Events: Hardware events, such as component failures, power supply changes, or system restarts, can be captured and retrieved using the Redfish API. These event data provide a historical record of critical hardware-related incidents, facilitating troubleshooting, analysis, and compliance reporting.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*31PJBrWxAerzZFnw" /><figcaption>Figure-4 Redfish Event Relay Data Flow</figcaption></figure><p>We can leverage an event relay approach (a solution to be implemented on the application platform) to subscribe applications running in your K8s cluster to events generated on the underlying bare-metal server, where the Redfish service publishes events on a node and transmits them to subscribed applications on an advanced message queue.</p><ol><li><em>Power Management: The Redfish API enables control and monitoring of power-related aspects of hardware, including power supply status, power consumption, and power capping. This allows organizations to optimize power usage, identify energy inefficiencies, and ensure stable power delivery to critical components.</em></li><li><em>Inventory and Asset Information: Through the Redfish API, organizations can retrieve detailed inventory information about hardware assets, including manufacturer details, serial numbers, firmware versions, and component configurations. This data aids in asset tracking, warranty management, and lifecycle planning.</em></li></ol><p>While the Redfish API is a widely agreed standard for hardware observability, other protocols and interfaces exist. For example, Intelligent Platform Management Interface (IPMI) is another widely used hardware management and monitoring interface.</p><p><strong>3.4.2 Operating System Level</strong></p><p>Another approach for hardware telemetry collection involves gathering data within the operating system (OS) using;</p><blockquote><em>The Linux kernel libsensors library provides an interface to access raw sensor data through the sysfs interface, which exposes hardware-related information.</em></blockquote><p>By leveraging the libsensors library, we can extract hardware telemetry data directly from the sensors present in the system. This includes temperature sensors, fan speed sensors, voltage sensors, and other relevant hardware sensors. The library provides an abstraction layer that simplifies the process of retrieving sensor data from various hardware components.</p><p>The Prometheus node exporter is a specialized component that exposes system-level metrics and data for external consumption. The node exporter includes a collector known as “hwmon,” specifically designed to collect hardware telemetry. This collector interacts with the libsensors library to gather sensor data from the system. It retrieves the raw telemetry values and converts them into Prometheus-compatible metrics, which Prometheus can scrape for further analysis, visualization, and alerting.</p><p>By combining the kernel libsensors library with the Prometheus node exporter’s hwmon collector, we can effectively collect and expose hardware telemetry data within our observability infrastructure. This approach enables us to monitor and analyze critical hardware metrics, such as temperature, fan speed, and voltage, alongside other system-level metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cJHxv2A2LJwztCNQ" /><figcaption>Figure-5 Prometheus Node Exporter HWMON (Ref: <a href="https://github.com/prometheus/node_exporter/blob/8b4dc8248891d1e08d502e74e5c49452da59e447/collector/hwmon_linux.go">Link</a>)</figcaption></figure><blockquote>Project <a href="https://github.com/sustainable-computing-io/kepler">Kepler</a> (Kubernetes-based Efficient Power Level Exporter) is a platform energy telemetry observation solution which also leverages eBPF technology (background provided in Section 3.3), to probe performance counters and kernel tracepoints that encompass hardware telemetry as well.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BwNyQ8LjYVQiggdHpjDKiA.png" /><figcaption>Figure-6 Kepler Architecture</figcaption></figure><h4><strong>4.0 Data Processing / Data Pipelines</strong></h4><p>Creating an ETL (Extract, Transform, Load) layer with sub-layers like data cleansing, data segregation, data anonymizer, data correlation, data mesh, data enrichment, and data governance is crucial to handling large amounts of data. Here’s how we can fulfill them using various open-source projects and popular products:</p><p><em>(I) Data Cleansing: This involves removing or correcting errors in the data, filling in missing values, removing duplicates, and validating the accuracy of the data.</em></p><ul><li><em>OpenRefine is a powerful tool for working with messy data, cleaning it, transforming it from one format into another, and extending it with web services or external data.</em></li></ul><p><em>(II) Data Segregation involves separating data based on specific criteria or rules.</em></p><ul><li><em>Apache NiFi is a robust data ingestion and distribution framework that allows you to segregate and route data based on various attributes.</em></li></ul><p><em>(III) Data Anonymizer ensures that sensitive data is anonymized to protect privacy.</em></p><ul><li><em>ARX Data Anonymization tool is open-source software for anonymizing sensitive personal data.</em></li></ul><p><em>(IV) Data Correlation involves discovering relationships or patterns across different fields.</em></p><ul><li><em>Elasticsearch with the Kibana visualization tool can provide robust data correlation features.</em></li></ul><blockquote><a href="https://github.com/korrel8r/korrel8r"><em>&gt;&gt; Korrel8r</em></a><em>, an advanced correlation engine, excels in navigating relationships and uncovering related data across multiple heterogeneous stores. It harnesses a comprehensive set of rules that define intricate connections between objects and signals. With a designated start object, such as an Alert within a cluster, and a specified goal, like “finding related logs,” Korrel8r dynamically scours through the available data, tracing a chain of rules to identify and retrieve the goal data associated with the start object. Korrel8r efficiently establishes connections and uncovers relevant insights through its intelligent rules-based approach.</em></blockquote><p><em>(V) Data Mesh involves creating a distributed data architecture where teams can access data from where it’s naturally produced, promoting greater agility and scalability.</em></p><ul><li><em>There needs to be a specific tool for this, as it’s more of an architectural principle. However, tools like Istio can help implement a data mesh architecture.</em></li></ul><p><em>(VI) Data Enrichment involves improving raw data by linking it with additional data sources.</em></p><ul><li><em>Apache Nifi has processors that can call out to external APIs to enrich data in the flow.</em></li></ul><p><em>(VII) Data Governance involves managing the availability, usability, integrity, and security of data in systems.</em></p><ul><li><em>Apache Atlas is a scalable and extensible set of core foundational governance services that enables enterprises to effectively and efficiently meet their compliance requirements within Hadoop.</em></li></ul><h4><strong>5.0 Open Telemetry (OTel)</strong></h4><p>Let’s pause here to close a gap that we have not touched on previously: the “standardization and instrumentation” of data collection, processing, and sharing (export), -&gt; Which OpenTelemetry (OTel) has born for!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RcGDOc2DzUfjN_YZ" /><figcaption>Figure-7 OTel Collection</figcaption></figure><p>OpenTelemetry is an open-source observability framework that provides a set of APIs, libraries, and tools for collecting, transmitting, and processing telemetry data from distributed systems. It aims to standardize the collection and instrumentation of telemetry data, making it easier to observe, monitor, and debug complex applications regardless of which programming language you are using. The full power of OpenTelemetry gets unleashed when processors sample, filter and transform the telemetry data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_2W7_6-EoD5zSWJxGc6FbA.png" /><figcaption>Figure-8 OTel Pipeline Flow</figcaption></figure><p>OTel benefits:</p><p><em>1. Standardization: It provides a vendor-neutral, community-driven standard for instrumentation and telemetry data collection across different programming languages, frameworks, and platforms. There is a very active community working behind this standardization. Distributed traces have already reached stable status, and the other signals are closer to the same level of maturity day by day.</em></p><p><em>2. Distributed Tracing: It enables distributed tracing, which involves capturing and correlating trace information across multiple services or microservices. This helps identify the path of requests as they traverse different system parts, providing end-to-end visibility and allowing for performance optimization and troubleshooting.</em></p><p><em>3. Metrics Collection: It supports the collection of metrics, which are quantitative measurements of the system’s behavior and performance. By instrumenting applications with OpenTelemetry, developers can collect metrics related to CPU usage, memory consumption, latency, error rates, and other relevant indicators. These metrics provide valuable insights into system health and performance.</em></p><p><em>4. Integration with Logging and APM: It can be integrated with logging systems and Application Performance Monitoring (APM) tools. This integration enables the correlation of logs, traces, and metrics, providing a holistic view of system behavior and allowing for deeper analysis and debugging.</em></p><p><em>5. Flexibility and Extensibility: It provides flexibility regarding the choice of collectors, exporters, and backends. It supports various telemetry data formats and can be integrated with monitoring and observability systems. It also allows custom instrumentation and extensions to meet specific monitoring requirements. OpenTelemetry also provides SDKs to instrument the code and even auto-instrument it in a long (and growing) list of programming languages.</em></p><h4><strong>6.0 3GPP’s Take On Observability</strong></h4><p>3GPP (3rd Generation Partnership Project) has defined a framework for observability in 5G networks, specifically for the Service-Based Architecture (SBA). The SBA is a critical architectural component of 5G networks that enables flexible service deployment and scalability. Observability in this context refers to the ability to gather, analyze, and gain insights from various data sources within the 5G SBA to ensure efficient network operation and performance. Compared to 4G, 5G Networks extensively use network slicing, requiring additional capabilities in the AIOps framework to consolidate KPIs at the slice level to constitute a single output for a network slice.</p><p>The observability solution for 5G SBA is based on a set of specifications defined by 3GPP. These specifications outline the protocols, interfaces, and mechanisms to enable observability within the 5G network. 3GPP has introduced Network Data Analytics Function (NWDAF), which collects the network data from other network functions and performs data analysis to help improve 5G network management automation. The NWDAF collects data from the 5G Network and provides analytics to support network automation, closed-loop operations, self-healing, experience improvement, and reporting. NWDAF helps automate and deliver network and slicing optimization, cost efficiency, and resource management to meet SLAs and quality-of-service guarantees. Utilizing intelligent modeling on refined data streams and ingested statistics, it supports a broad spectrum of analytics capabilities ranging from threshold-driven to more sophisticated algorithm-based analytics.</p><p>Critical aspects of the 3GPP TS 23.288 NWDAF-based observability framework include:</p><p><em>1. Data Collection Coordination Function (DCCF): Data Collection Coordination and Delivery coordinates the collection and distribution of data requested by NF consumers. It prevents data sources from handling multiple subscriptions for the same data and sends multiple notifications containing the same information due to uncoordinated requests from data consumers. The DCCF may collect the analytics and deliver it to the NF, or the DCCF may rely on a messaging framework to collect analytics and deliver it to the NF. A DCCF can support multiple Data Sources, Data Consumers, and Message Frameworks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GMXv_u9PBOBU_sxzW-Upfw.png" /><figcaption><em>Figure-9 3GPP Custom Data Bus Architecture for 5G</em></figcaption></figure><p><em>2. Analytics Data Repository Function (ANDRF): The 5G System architecture allows ADRF to store and retrieve the collected data and analytics. The following options are supported:</em></p><ul><li><em>ADRF exposes the Nadrf service for storing and retrieving data by other 5GC NFs (ex NWDAF), which access the data using Nadrf services.</em></li><li><em>Based on the NF request or configuration on the DCCF, the DCCF may determine the ADRF and interact directly or indirectly with the ADRF to request or store data. The interaction can be direct or indirect via a messaging framework.</em></li></ul><p><em>3. Messaging Framework Adaptor Function (MFAF): MFAF offers 3GPP-defined services that allow the 5GS to interact with the Messaging Framework. Internally, the Messaging Framework may support the pub-sub pattern, where received data are published to the Messaging Framework, and requests from 3GPP Consumers result in Messaging Framework specific subscriptions. Alternatively, the Messaging Framework may support other protocols required for integration.</em></p><h4><strong>7.0 Observability Solution Architecture Samples</strong></h4><p>In our previous discussions, we have explored different aspects of observability data characterization, including its collection, storage, processing, and sharing across various operational landscapes. We will illustrate these concepts with architectural diagrams that exemplify influential approaches to clarify understanding.</p><blockquote>[A] Reactive Approach: Incidents occur frequently, and we should focus on learning from them using the resources at hand. This approach incorporates the concept of a message bus, where observability data is reported and carried over a best-effort delivery system. Listeners can then process the data based on availability.</blockquote><p>Please note, as we covered in Section 2.3 Events, message bus frameworks are designed to pass the information of noticeable changes at the end systems, not necessarily complete state information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PkAM3MJYBDkrFRkpGgy6BA.png" /><figcaption>Figure-10 Reactive Use of Observability</figcaption></figure><p>In Figure-10, we present a sample solution for observability data collection using a message bus-driven approach. It integrates various open-source backends, following a bottom-up approach (network-fabric -&gt; server-hardware -&gt; application-platform -&gt; applications), and forwards the collected data to a central Operations Support Systems (OSS) umbrella system via a message bus. This approach aligns with the 3GPP approach depicted in Figure-9.</p><blockquote>[B] Proactive Approach: The key is to have a substantial volume of data (big data) delivered with high velocity (fresh data) and encompassing variety (data from multiple endpoints). To achieve this, we should implement high-throughput and low-latency data stream pipelines that transfer data from applications to data sinks, from data backends to data pipelines, and from data pipelines to AIOps machinery. This enables proactive operational management based on real-time (or near-real-time) data-driven insights.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nplKf8EH_N-_jD2MmcsvFQ.png" /><figcaption>Figure-11 Proactive Use of Observability</figcaption></figure><p>Figure-11 illustrates a dynamic and data-focused observability solution topology. It utilizes different technology stacks, such as RedFish, Victoria, Icinga, Loki, and Logstash, to provide comprehensive data collection and processing. The valuable and actionable data is then shared with external OSS platforms. This solution continually observes system states, offering proactive management capabilities.</p><p>[Field-View]: Both reactive and proactive approaches have been implemented by various service providers, and neither approach is inherently better than the other by definition. The choice of solution design depends on specific operational goals to be achieved.</p><p>However, from a data collection framework perspective, we suggest several viable and future-proof options that leverage the power of the community and the freedom of open source. This allows us to easily select data backends and data processing pipelines that best suit different needs and circumstances:</p><ul><li><em>Direct Collection from CNFs: CNF/VNF observability data can be collected directly from the CNFs using OTel’s OpenTelemetry Protocol (OTLP). This enables direct data ingestion from CNFs to a designated data backend (e.g., AWS S3 with per CNF buckets and isolated access credentials) via the OTel exporter.</em></li><li><em>Collection via OTel Collector: Alternatively, CNFs observability data sinked to a time-series cluster-scoped backend (e.g., Cortex with tenant isolation), which can then be collected by an OTel collector — the OTel collector aggregates, processes, and exports the data to a central observability/AIOps platform.</em></li><li><em>Utilizing eBPF: Another option is to employ eBPF to collect observability data directly from CNFs/VNFs. This requires the CNF/VNF vendor (or a platform vendor) to provide configurable eBPF probes that collect and export data in an OTel-compliant format.</em></li></ul><p>By leveraging OTel, our proposed approach enables efficient data collection, processing, and export of CNF/VNF observability data. It offers flexibility in data collection, while the OTel components handle aggregation, transformation, and integration with the central observability/AIOps platform.</p><p>These offers align with modern practices in observability, utilizing standard protocols and components to achieve efficient and scalable data collection from CNFs/VNFs.</p><h4><strong>8.0 Multi-Site Observability + AIOps Insights Machinery</strong></h4><p>Red Hat, Telenor and AIOPSONE have been collaborating to address the following;</p><ul><li><em>Full stack (Network, Compute, Storage, OS, Platform, Application) observability, pre-integrated with Thanos, InfluxDB, VictoriaMetrics, Loki backend, and built-in Kafka client to receive published events.</em></li><li><em>Multi-Site single-pane dashboard with Openshift, K8s, and OpenStack API support.</em></li><li><em>Embedded visualization and on-demand data query &amp; filtering with Grafana Dashboards and embedded native visualization capabilities.</em></li><li><em>ML/AI-driven insights generation with event-driven workflow triggers for action resolutions, pre-integrated with Ansible Tower.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*047byb5mWa5vrlBM" /><figcaption>Figure-12 AIOPSONE Single Pane Experience</figcaption></figure><p>We have worked on the following widespread operational problems with possible solution approaches enabled by AIOps:</p><p>1. Root Cause Analysis (RCA) with network observability</p><p>In our 5GCore Test Bed, we intentionally introduced a failure or bottleneck in the network fabric to simulate real-world scenarios. This results in anomalies within the Key Performance Indicators (KPIs) of the 5GCore Cloud Native Functions (CNFs) and provides various reason indications. Here, the AIOPSONE engine showcases its capabilities by performing Root Cause Analysis (RCA) and accurately identifying the source of the problem.</p><p>Key Highlights:</p><ul><li><em>Intentional network fabric failure/bottleneck simulation for realistic testing.</em></li><li><em>Anomalies observed in 5GCore CNF KPIs with exact reason indications.</em></li><li><em>AIOps engine performs accurate RCA to identify the root cause of the issue.</em></li></ul><p>Key Data Sources:</p><ul><li><em>OTel Metrics for platform network performance measurements.</em></li><li><em>Application data collected via eBPF probes.</em></li><li><em>NEP Performance KPI(s) via Rsyslogs.</em></li><li><em>OS network interface &amp; device logs and metrics via Node Exporter.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hIp9IQ9d1CzVQTK_zv0rGA.png" /><figcaption>Figure-13 AIOPSONE Prediction / Possible RCA for Network-Fabric Related Issue (MTU Misconfig)</figcaption></figure><p>Issue Resolution:</p><ul><li><em>Such insight (Figure-11) enables quick troubleshooting and problem solving (ex, use of Ansible network fabric modules for configuring problematic hops accordingly), minimizing downtime and optimizing network performance.</em></li></ul><p>2. VNF/CNF Performance Bottlenecks Analysis</p><p>In this scenario, we focused on the performance bottlenecks after upgrading to a new release of the 5G CNF. Some CNF PODs start consuming excessive CPU/Memory resources, leading to unforeseen issues reported within CNF OTel Data. AIOPSONE engine leverages this data for performing RCA analysis, identifying the POD version change (i.e., release change) as the cause of the bottleneck.</p><p>Key Data Sources:</p><ul><li><em>OTel Metrics for platform network performance measurements.</em></li><li><em>Application data collected via eBPF probes.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TiFeLAGIo2kDLM8jMZA0-Q.png" /><figcaption>Figure-14 AIOPSONE Prediction/Possible-RCA for CNF Performance/Behaviour</figcaption></figure><p>Issue Resolution:</p><ul><li><em>Such insight (Figure-14) enables the execution of an intelligent rollback to the previous POD Image Version by Ansible Tower that eliminates bottlenecks and ensures stable performance. Also, the application deployment manifest was updated by Ansible LightSpeed for using GenAI to scale a particular CNF POD by increasing the replica-set (RS) size and with optimal resource allocation.</em></li></ul><p>3. Hardware Failure Prediction: Analyzing IaaS observability data and performing predictive maintenance modeling</p><p>We analyzed received IaaS observability data in this use case to predict hardware failure patterns and create resolution paths. By utilizing advanced predictive maintenance models, our AIOPSONE engine can proactively identify potential hardware failures and provide resolution paths to mitigate risks.</p><p>Key Highlights:</p><ul><li><em>Analysis of IaaS data enables predictive hardware failure pattern recognition.</em></li><li><em>Proactive identification of potential hardware failures for timely resolution.</em></li><li><em>Predictive maintenance modeling reduces downtime and enhances network reliability.</em></li><li><em>Resolution paths were provided to address identified hardware issues efficiently.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mNNYh_xWvrgqb2SGL12nzw.png" /><figcaption>Figure-15 AIOPSONE Prediction/Possible RCA for Hardware Failure</figcaption></figure><p>Key Data Sources:</p><ul><li><em>Hardware Telemetry (mainly sensor data collection and event logging) over Redfish.</em></li><li><em>OS device metrics and logs via Node Exporter.</em></li></ul><p>Issue Resolution:</p><ul><li><em>Such insight (Figure-15) enables the execution of an external workflow order (ex, Ansible ServiceNow modules) for hardware replacement in a specific data center and server enclosure.</em></li></ul><h4><strong>9.0 Summary</strong></h4><p>This article delves into various aspects of observability and AIOps, exploring different technology stacks and implementation approaches. We emphasized the importance of data collection and highlighted vital sources such as metrics, logs, events, traces, and topology data. Open-source technologies like Prometheus, Thanos, Loki, Fluentd, Kafka, and Jaeger were discussed for data collection, processing, and enrichment.</p><p>Standardization and instrumentation in data collection led us to the OpenTelemetry framework, which provides a standardized approach for collecting, transmitting, and processing telemetry data from distributed systems. This framework enhances the observability and monitoring of complex applications.</p><p>The 3GPP’s approach to observability in 5G networks, specifically the Service-Based Architecture (SBA) and Network Data Analytics Function (NWDAF) framework, were explored. NWDAF standardizes data collection and management from user equipment, network functions, and operations and maintenance systems within the 5G network.</p><p>For application add-on observability, eBPF was highlighted as a powerful tool for injecting observability into applications. The bpfd software stack simplifies the management and monitoring of eBPF programs, providing enhanced security, visibility, multi-program support, and improved productivity.</p><p>In terms of hardware observability, two significant approaches were discussed. Direct communication with the bare-metal hardware controller using protocols like Redfish enables real-time telemetry data collection while retrieving host operating system data through system calls, and libraries like libsensors and hwmon provide insights into hardware behavior.</p><p>The collaboration between Red Hat, Telenor, and AIOPSONE in addressing operational challenges and implementing AIOps solutions were showcased, highlighting the power of AI-driven insights and automation in optimizing network performance and reducing downtime.</p><blockquote>In conclusion, the collaboration between Red Hat, Telenor, and AIOPSONE, along with open-source projects and standardization efforts like OpenTelemetry, and bpfd, empowers organizations to collect, process, and analyze data for improved observability and decision-making without vendor lock-in. Striving for ultimate open-source solutions with scalability, resilience, and efficiency remains a crucial objective.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8e81be2dfa2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xvi-the-eye-of-the-bee-holder-a8e81be2dfa2">Episode XVI: The Eye of the BEE-Holder</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XV: Magic-Ops]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xv-magic-ops-cda74284fe74?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/cda74284fe74</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Fri, 21 Apr 2023 13:41:46 GMT</pubDate>
            <atom:updated>2024-02-21T13:34:07.725Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Authors:</strong> Fatih Nar Chief Technologist at Red Hat; Vanessa Martini Senior Product Manager at Red Hat; Volker Tegtmeyer Principal Product Marketing Manager at Red Hat; Fatih Baltaci CTO at DDosify.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*KJZ4QDHzRkrxYHhjaIsc_w.png" /><figcaption>Article Thumbnail (Licensed from iStockPhoto CN:27675569)</figcaption></figure><blockquote><em>“Any sufficiently advanced technology is indistinguishable from magic.” Arthur C. Clarke</em></blockquote><p><strong>Introduction</strong></p><p>As the 5G network deployment continues, telecommunication networks’ complexity and operations are increasing rapidly. They now offer services in multi-cloud environments spanning a multilayer stack of; containers, virtual machines (VMs), and physical components. Therefore, service providers need to adopt a radical approach to network operations to achieve efficiency with timely effectiveness. One technology that can help address this challenge is artificial intelligence (AI).</p><p>Artificial Intelligence for IT Operations (AIOps) is a rising star solution that leverages big data analytics and machine learning to identify patterns and anomalies in IT infrastructure and services. This enables IT teams to automate tasks, gain valuable insights, and make informed decisions for faster and more efficient problem resolution. By leveraging advanced analytics, AIOps can analyze vast amounts of data generated by complex IT infrastructures, multi-cloud environments, applications, and services in real time. This enables AIOps to quickly detect issues, diagnose problems, provide root cause analysis (RCA) functionality, and automate remediation processes, resulting in improved speed and efficiency of IT operations. This leads to a lower risk of downtime, reduced total cost of ownership (from a site reliability engineering perspective), and a better overall user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*28pGSwGYqeHRed-H" /><figcaption>Figure-1 Google Search Trends for AIOps over Years</figcaption></figure><p>With more solutions and systems getting spread across the geographies to especially for offering edge computing with lower latency and aiming for higher user experiences, the management of these systems also becomes more complex and challenging, especially in more granular edge IT infrastructures with high traffic patterns (high population with more granular coverage areas) forcing operation models to embrace AIOps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uxYxTiAdYBLXIqig" /><figcaption>Figure-2 AIOps Market Growth (Reference: <a href="https://www.mordorintelligence.com/industry-reports/aiops-market">Link</a>)</figcaption></figure><p>In order to raise awareness and educate our audience about AIOps, demonstrate its benefits, and promote promising solutions, we will publish a three-part publishing series.</p><p>(I) The first article (this article you are reading), at a 101 level, will explain what AIOps is, what it does, and how it works. It will focus on the benefits that AIOps offers to operations and business teams.</p><p>(II) The second article, at a 201 level, will highlight market-ready solutions by providing a complete architecture with commercial products and open-source projects.</p><p>(III) Finally, at a 301 level, the third article will showcase a sandboxed, tested example solution with use-case recordings and snapshots.</p><p>Note: Depending on our resources and timing, we may/may-not merge the second and third articles into a single publication.</p><p><strong>AIOps is an approach to achieve better-informed IT operations!</strong></p><p>Although we occasionally see misguiding marketing materials positioning AIOps as a product in the field, AIOps is actually a solution built of multiple products and services from various disciplines. The critical components of the AIOps solution include extensive data collection, ETL (extract, transform, and load), and the central AI/ML core where the “<em>magic”</em> happens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*COY8GCYZVpcmF73d" /><figcaption>Figure-3 High-Level Reference AIOps Solution Diagram</figcaption></figure><p><strong>Data Collection from multiple channels</strong></p><p>This foundational base fuels (as data being the oil) complex machinery (AIOps Core) to analyze and quickly see patterns and anomalies that IT Ops teams generally wouldn’t find. Suppose you operate an extensive network, managing thousands of devices and millions of nodes globally. In that case, it is impossible to manually sift through all available and most complex data in your environments. Analyzing metrics, events, logs, traces (MELT), and other unstructured/structured supplementary data and predicting upcoming application or system failure within any reasonable time would be less likely to happen than winning the argument against your spouse.</p><p>An effective AIOps solution requires a highly flexible and comprehensive data collection mechanism. To address this challenge, data collection can be implemented through push mechanisms (ex, webhooks) and pull mechanisms (ex, SFTP).</p><blockquote>The “V3-attributes” of data; Volume, Velocity, and Variety should be considered to ensure enough, valuable, and high-quality data.</blockquote><p>Data from various channels can be, including;</p><ul><li>Network Fabric (Switch Fabric, Site, and Backbone Routers, Domain Firewalls Carrier Grade NATs, etc.),</li><li>Baremetal Layer (HP iLO, Dell iDRAC, etc.),</li><li>Operating Systems,</li><li>Virtualization Layer (KVM, ESXi, etc.),</li><li>Infrastructure as a Service (IaaS) layer (VMW, OpenStack),</li><li>Platform as a Service (PaaS) layer (Kubernetes),</li><li>Big Data and Analytics systems containing insightful data</li><li>OSS / BSS systems (Nagios, Zabbix, etc.)</li><li>Middleware Layer for Application Frameworks (such as Quarkus, Micronaut, Helidon, etc.), and</li><li>Tenant Applications (such as 5G CNFs).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQ17FcR-RFM12NbB7BPXNw.png" /><figcaption>Figure-4 Top Incident Management Challenges (Ref: <a href="https://www.opsramp.com/wp-content/uploads/2022/07/The-OpsRamp-State-of-AIOps-Report.pdf">Link</a>)</figcaption></figure><p>By leveraging V3-attributes of data from these sources, AIOps solutions can generate more accurate and actionable insights, ultimately improving the efficiency and effectiveness of IT operations. Besides of collecting existing data (oil-transport) out in the wild, generating the required valuable additional data may be needed (oil-rig), for which new-generation observability platforms would be beneficial (ex, DDosify for application performance testing -APT-) as supplementary data channel.</p><p><strong>ETL Engine makes sure data is accurate, consistent, and up-to-date</strong></p><p>Data cleansing involves identifying and correcting or removing any inaccuracies, inconsistencies, or irrelevant information from data sets to improve their quality and reliability. A segregator segregates data into categories or groups based on specific criteria to enable easy analysis and processing. Anonymizer masks or removes personal or sensitive information from data sets to comply with privacy regulations. Together, these components ensure that the AIOps solution has access to high-quality data that is organized, secure, and compliant with regulations (which may vary based on region/country).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U_2TjBKThNlwUA3g" /><figcaption>Figure-5 Data-Mesh for 5G Core Sample</figcaption></figure><p>Data correlation involves identifying and linking related data from various sources to understand better the system or application being monitored. Data mesh (example Figure-5) refers to a decentralized approach to data management, where data is organized into smaller, domain-specific units to enable efficient and scalable analysis. Enrichment involves augmenting data with additional context or metadata to provide deeper insights and enable more accurate analysis.</p><p>Governance involves defining policies, procedures, and standards for data management to ensure data quality, security, and compliance with regulatory requirements. A data governor provides oversight and control over the data used by the AIOps solution, including monitoring and auditing data usage, ensuring data privacy and security, and managing data retention and disposal. It also involves establishing clear roles and responsibilities for data management, such as defining data ownership, access controls, and accountability.</p><p><strong>AIOps Core processes and analyzes the data to extract information</strong></p><p>Observability is an essential building block of an AIOps solution as it empowers IT teams to gain immediate and actionable insights into the performance and health of their systems and applications. An observability framework collects and analyzes clean and enriched data from the ETL (Extract, Transform, Load) layer, including logs, metrics, traces, and events, to provide a comprehensive view of the system’s behavior. This feedback can be used by developers to optimize their code and improve the system’s design, leading to better overall performance and reliability.</p><p>Communication and workflow engines are two critical building blocks of an AIOps solution, facilitating the smooth functioning of IT operations. The communication engine acts as a bridge between different teams, stakeholders, and the AIOps platform, enabling the team to stay informed about the system’s status and critical alerts. With an effective communication engine, the IT team can act swiftly and address issues before they escalate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BnTEWI22-H_jUaxh" /><figcaption>Figure-6 AIOps Capability Impact Ratio (Ref: <a href="https://www.opsramp.com/wp-content/uploads/2022/07/The-OpsRamp-State-of-AIOps-Report.pdf">Link</a>)</figcaption></figure><p>On the other hand, the workflow engine automates the incident management process, from the initial diagnosis to resolution and closure, by following a well-defined workflow. By providing a centralized platform for collaboration, it streamlines communication and decision-making, leading to a more efficient and effective resolution process. The workflow engine ensures that incidents are escalated to the right team members based on their skills and availability, resulting in faster resolution times.</p><p>To integrate these communication flows, AIOps solutions often use APIs to connect with existing platforms such as Slack, Email, or CRM systems. This provides a standardized way of communicating and enables a more effective flow of information between different teams, leading to better collaboration and ultimately improving outcomes.</p><p>An AI/ML engine is the central component of an AIOps solution, providing the intelligence to automate IT operations and improve efficiency. Using machine learning algorithms and artificial intelligence techniques, it can identify patterns and anomalies in vast amounts of data, enabling IT teams to detect and diagnose issues proactively.</p><ul><li>One of the vital vertical offerings of an AIOps solution is predictive maintenance, which uses equipment sensors, historical data, and maintenance records to identify potential equipment failures before they occur. This allows organizations to perform proactive maintenance and avoid costly downtime.</li><li>Another critical offering is root cause analysis, which correlates all types of data collected by the AIOps solution to identify the problem’s root cause. This can be challenging and require significant team involvement, but it helps prevent issues from recurring in the future and mitigates the risk of equipment failure.</li><li>Service and revenue assurance is another offering that helps ensure efficient and accurate service delivery while maximizing revenue. It uses billing records, customer feedback, and service tickets to identify issues and areas for improvement.</li><li>Fraud detection and prevention is critical for organizations dealing with sensitive data and transactions. It uses transaction records, customer profiles, and social media activity to identify patterns and anomalies that may indicate fraudulent activity.</li><li>The AIOps solution can also generate insights into an organization’s operations and help make informed decisions. By analyzing customer feedback, social media activity, and financial reports, it can identify trends, patterns, and opportunities for improvement.</li></ul><p>Other essential features and services that an AI/ML engine can provide include network optimization, capacity planning, and resource allocation. It can analyze data from various sources, including network traffic, server logs, and resource usage, to identify areas for optimization and improvement. This enables organizations to maximize their networks&#39; and resources&#39; efficiency and performance while minimizing costs and downtime.</p><p><strong>AIOps &amp; OSS/BSS</strong></p><p>Operations Support Systems (OSS) and Business Support Systems (BSS) are critical components of telecommunications and IT service providers’ operations and business processes. These systems facilitate network management, service provisioning, billing, and customer care.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ffI4SK-c4iABPkNuERs1Gw.png" /><figcaption>Figure-7 Purpose of using AIOps (Ref: <a href="https://www.opsramp.com/wp-content/uploads/2022/07/The-OpsRamp-State-of-AIOps-Report.pdf">Link</a>)</figcaption></figure><p>An AIOps solution can seamlessly integrate with OSS and BSS systems using APIs, enabling effective communication and data exchange between systems. With access to OSS and BSS systems data, an AIOps solution can analyze information from various sources, such as logs, metrics, and events, providing a comprehensive view of the system’s health and performance. AIOps can report its findings, conclusions, and recommendations to OSS and BSS systems, empowering them to make better-informed decisions. AIOps can impact organizations in the following ways:</p><ul><li>Improved operational efficiency: AIOps help automate routine tasks and reduce the time and effort required for manual analysis, which improves operational efficiency. This closed-loop automation feature enables automatic actions to address problems, which helps organizations reduce costs and improve service delivery.</li><li>Enhanced customer experience: AIOps helps identify and resolve issues quickly, improving service quality and customer satisfaction. This can help organizations retain customers and increase revenue.</li><li>Predictive maintenance: AIOps can predict and prevent potential issues before they occur, which reduces downtime, minimizes the impact of issues on customers, and optimizes operations.</li><li>Effective capacity planning: AIOps can provide accurate projections about capacity usage, including peak values and usage patterns. With AI-powered calculations, organizations can make informed capacity planning decisions based on comprehensive historical usage and future usage projections.</li><li>Faster problem resolution: AIOps can identify the root cause of issues quickly and accurately, which enables operations teams to resolve problems faster, minimizing the impact on customers. This results in improved Mean Time to Repair (MTTR).</li><li>Better decision-making: AIOps provides operations and business teams with insights into system performance, which helps them make better decisions and optimize their operations.</li></ul><p>AIOps can be an empowering companion for OSS and BSS as it helps organizations improve operational efficiency, enhance customer experience, reduce costs, and increase revenue.</p><p><strong>Which AIOps Solution to Pick?</strong></p><p>As we have been trying to explain/highlight that AIOps is not a product but a very sophisticated solution, and while shopping for one, there are a few essential criteria you shall look for/seek;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mQmVgrcTD3l9yU-lklcDw.png" /><figcaption>Figure-8 Concerns for AIOps (Ref: <a href="https://www.opsramp.com/wp-content/uploads/2022/07/The-OpsRamp-State-of-AIOps-Report.pdf">Link</a>)</figcaption></figure><ol><li>Integration and interoperability: The AIOps solution should be able to integrate with your IT infrastructure, os &amp; virtualization layer, network functions, and monitoring tools. It should also provide a unified view of IT operations and correlate logs, metrics, events, and alerts across multiple tools.</li><li>AI and machine learning capabilities: The AIOps solution should have advanced AI and machine learning capabilities that can analyze vast amounts of data to cover the key aspect of the operational data in real-time with effective ML techniques to detect and resolve issues. It should also be able to provide predictive analytics to help prevent future problems.</li><li>Automation capabilities: The AIOps solution should automate routine tasks and workflows, freeing IT staff to focus on more strategic initiatives. It should also provide automated remediation workflows to resolve issues quickly by taking actions themselves or triggering external entities (ex, Ansible Tower) to provide closed-loop automation functionality.</li><li>Scalability and performance: The AIOps solution should scale to meet your organization’s needs and provide real-time insights into the performance of applications and infrastructure.</li><li>Usability and ease of use: The AIOps solution should be easy to use and offer a user-friendly interface that can be customized to meet your organization’s specific needs. It should also provide customizable dashboards and reports.</li><li>Security and compliance: The AIOps solution should meet your organization’s security and compliance requirements. It should provide secure access to data and offer audit trails and compliance reporting.</li><li>Cost and value: The AIOps solution should provide good value for acquiring and running costs. It should offer flexible pricing options and provide a clear return on investment.</li></ol><p><strong>Summary</strong></p><p>This article aimed to provide an overview of AIOps (Artificial Intelligence for IT Operations) and how it can improve IT operations management. We also discussed the critical components of AIOps, such as data collection, ETL engine, AI/ML engine, observability framework, communication and workflow engine, and data governor. Furthermore, we highlighted how an AIOps solution could integrate with existing OSS and BSS solutions inside the enterprise ecosystem.</p><p>We hereby conclude that for the right AIOps solution, you shall consider your specific needs and requirements and select the best-fitting solution that integrates well with your existing OSS/BSS ecosystem.</p><p>In this workshop series’s coming part(s), we will dive into ready-to-use reference AIOps solution(s) with sandbox testing and real-world use case(s) analysis.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cda74284fe74" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xv-magic-ops-cda74284fe74">Episode XV: Magic-Ops</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Episode XIV: Circle of Life]]></title>
            <link>https://medium.com/open-5g-hypercore/episode-xiv-circle-of-life-7a678243c2ad?source=rss----bd5258a5ea5---4</link>
            <guid isPermaLink="false">https://medium.com/p/7a678243c2ad</guid>
            <dc:creator><![CDATA[Fatih Nar]]></dc:creator>
            <pubDate>Mon, 27 Mar 2023 16:39:36 GMT</pubDate>
            <atom:updated>2023-03-28T18:06:53.135Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Author(s):</strong> Fatih Nar Chief Architect at Red Hat, Tom Conklin Senior Director at Red Hat<br><strong>Reviewer(s) &amp; Contributor(s): </strong>Azhar Sayeed Senior Director at Red Hat</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZRn9K2U_8wJHpzR7" /></figure><p><strong>Introduction</strong></p><p>When we see birds up in the air in Autumn, that reminds us the days ahead will get more challenging and hence it is time to re-think where we are and where we shall aim to be for our sakes; time goes on and at next Autumn lather, rinse, and repeat. This is also true for businesses where IT systems bring significant challenges not only for maintaining existing execution but also for scaling based on business needs and complying with standards, regulations, and trends. Hence, they stay alive and relevant in their journey. As we maintain the execution of a business process in some parts of the flow, we may notice circular patterns where some operation can/may/will move back and forth between different hostings based on a given time’s business and operational (BaO) needs.</p><p>With the rising popularity of cloud computing, there has been a hot debate about using on-premise data centers vs. cloud, and passionate camps have been established to defend each side. This article will detail each, try to gather the pros &amp; cons, and draw insights into possible cyclic BaO patterns.</p><p><strong>On-Premise vs. Cloud (Frenemy)</strong></p><p>Whether cloud computing is better than on-premise data centers (or not) depends on various factors, such as the organization’s size and needs, budget, security requirements, and the level of control, it wants to retain over the systems it consumes.</p><p>With cloud computing, businesses can;</p><ul><li>access computing resources and services on demand,</li><li>pay only for what they use (intentionally or unintentionally), and</li><li>easily scale up or down as their needs change.</li><li>pay less (even none for the most part) for infrastructure maintenance and upgrades, which can reduce the burden on in-house IT teams.</li></ul><p>While cloud computing offers many benefits, on-premise data centers can provide more granular control over infrastructure and data, which can be crucial for specific industries and organizations with high-security and regulatory requirements. The decision to choose between cloud computing and on-premise data centers ultimately depends on the specific needs and circumstances of the organization. Security, compliance, scalability, cost, and performance must be carefully considered before deciding on the right approach. With this in mind, let’s explore when to migrate workloads to or from the cloud.</p><p><strong>On-Premise -&gt; Cloud (Rush to Gold)</strong></p><p>The decision to move to the cloud from on-premise data centers can be for various reasons:</p><ul><li>Scalability: Organizations that experience fluctuating demands for computing resources may find it more cost-effective to use cloud computing instead of maintaining on-premise data centers sized for peak/burst demand. Cloud computing allows businesses to scale up or down easily as their needs change without investing in additional hardware and space.</li><li>Cost savings: With cloud computing, businesses can pay only for their resources (operational expenditure -opex-), avoiding investing in expensive hardware and infrastructure (capital expenditure -capex-).</li><li>Agility and innovation: Businesses can act faster to adopt new technologies and innovation. Cloud providers often release new features and services faster than on-premise data center vendors, giving businesses a competitive advantage.</li><li>Single Pane (All You Can Eat Menu): Cloud service providers offer a central menu/catalog and access to their products and services, making development and operations relatively easier to implement &amp; convenient with all-you-can-eat from their catalog, which dynamically changes behind the scenes.</li><li>Geographic reach: Businesses can extend their reach to new geographic locations without setting up physical infrastructure. This can be particularly useful for businesses with customers and/or employees in multiple locations.</li><li>Disaster recovery: Cloud computing can provide businesses a more robust and reliable disaster recovery solution than on-premise data centers. Cloud providers typically have redundant systems and backups in multiple geographic locations, which can help ensure business continuity in the event of a disaster based on the service/product they are offering in their catalogs.</li></ul><p><strong>Cloud -&gt; On-Premise (Back to Motherland)</strong></p><p>The move from the cloud to on-premise data centers can be based on various reasons:</p><ol><li>Cost: Depending on the usage patterns and workload characteristics, it is possible for on-premise data centers to be more cost-effective than cloud computing in some cases. For example, if a business has stable and predictable traffic that requires significant computing and storage resources over a long period, it may be more cost-effective to invest in on-premise infrastructure rather than pay for cloud resources over time.</li><li>Data sovereignty: Some organizations may be lawfully required to keep data within specific geographic locations, which may not be possible with cloud computing. The organization may deploy on-premise to comply with data sovereignty regulations in such cases.</li><li>Security and compliance: Some organizations may have strict security or compliance requirements that are difficult or impossible to meet with cloud computing. In such cases, the organization may bring the workload back on-premise, where they can better control the infrastructure and security policies.</li><li>Convenience versus Blast Radius: Despite having central access to all-you-can-eat products and services from a cloud service provider providing convenience, the cloud operational record track also showed that it increased the blast radius for failures and outages on a global scale.</li><li>Balancing Roles and Responsibilities with SLAs: The roles and responsibilities of cloud service providers and their consumers have been a key area of friction in implementing end-to-end services with SLAs. While cloud providers have established a high level of separation between these roles and responsibilities, issues can arise in the operational experience of handling these responsibilities and signing off on end-user SLAs. This can create challenges that are overwhelming for all parties involved.</li><li>Latency and performance: Applications that require low latency and high performance may not be suitable for cloud computing due to the inherent network latency and potential for contention with other workloads running on the same hardware. In such cases, on-premise data centers can provide better performance and lower latency for critical applications.</li><li>Unforeseen costs: In some cases, businesses may experience unforeseen costs associated with cloud computing, such as unexpected network transfer fees, storage costs, or licensing costs. These costs may be difficult to control or predict, leading some businesses to bring the workload back on-premise.</li></ol><p>Some statistics for reverse migration:</p><ul><li>In a survey conducted by Flexera in 2021, 20% of respondents reported moving applications from the cloud back to on-premises environments, up from 7% in 2020.</li><li>A report by Gartner in 2021 predicted that by 2024, more than 80% of organizations that have deployed cloud infrastructure as a service (IaaS) would adopt a strategy of moving workloads between cloud providers or bringing them back on-premises.</li><li>Another report by IDC in 2021 found that while only 3% of workloads have moved back to on-premises environments from the cloud, 47% of organizations that moved workloads to the cloud experienced unexpected costs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hsylw23AJd3ZrkqX" /><figcaption>Figure-1 Flexera 2023 State of the Cloud Report, Top Cloud Challenges (<a href="https://info.flexera.com/CM-REPORT-State-of-the-Cloud-2023-Thanks">Link</a>)</figcaption></figure><p><strong>Talent Factor (So You Think You Can Dance?!)</strong></p><p>Cloud migration requires a range of specific skills and strong expertise. The migration task’s difficulty will depend on various factors, such as the size and complexity of the organization, the number and types of applications and workloads being migrated, and the level of integration required with existing systems. Successful migration requires careful planning, execution, and monitoring to ensure that the process is seamless and disruption-free.</p><p>Here are some factors that influence cloud migration:</p><ul><li>Know-How: Cloud migration requires a range of skills and expertise, including cloud architecture, security, networking, DevOps, and vendor management. In addition, it may require expertise in the specific cloud platforms and technologies being used and the ability to manage the migration process from planning to execution and ongoing management.</li><li>Effective Program Management: Gathering the right team with the appropriate skills and experience is crucial to successful cloud migration. This requires a robust talent management program and well-established practices. This is where global systems integrators (GSIs) play a significant role. We have seen many cloud migration projects owned, executed, and managed by the biggest GSIs, often close cloud service provider partners with significant cloud spending commitments. A growing trend is for systems integrators to manage on-premise clouds through a “containers as a service” offering. In this model, the GSI deploys platforms and workloads on demand. It handles day-to-day lifecycle management for both, allowing for active management of IT sprawl and reduced shadow IT efforts. Partnering with a GSI can be an excellent way to improve business agility and achieve successful cloud migration.</li><li>Time Factor: Time is a crucial factor in technology for new shiny things to be developed/used and for the dynamics of personal and business life. It is proven that covid has accelerated the cloud migration, and the post-covid economic slow-down showed reverse migration is very real and happening. Over the course of time, talent variety, strength, capacity, and capability would be changing for availability, accessibility, and affordability.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cZBLEabCVL_YKnzo" /><figcaption>Figure-2 Flexera 2023 State of the Cloud Report, YoY Comparison for Top Cloud Challenges (<a href="https://info.flexera.com/CM-REPORT-State-of-the-Cloud-2023-Thanks">Link</a>)</figcaption></figure><p><strong>Sustainability &amp; Efficiency</strong></p><p>Cloud service providers operate large data centers designed to be energy efficient and rely on renewable energy sources. They tend to be located in areas with low energy costs and abundant renewable energy sources, such as hydroelectric or wind power. This results in a relatively low carbon footprint for cloud providers. However, sustainability can vary among on-premises data centers, with smaller facilities often having lower energy efficiency and relying more on non-renewable energy sources. However, larger on-premises data centers can also be designed for high energy efficiency and renewable sources.</p><p>While both cloud service providers and on-premises data centers can be efficient, cloud providers tend to offer higher levels of efficiency due to economies of scale. These providers can spread the fixed costs of data centers over more customers, resulting in lower costs for each customer. This allows cloud providers to invest more in advanced technologies, such as AI and machine learning, that improve efficiency. However, on-premises data centers can also be highly efficient, often requiring significant upfront investments in technology and infrastructure.</p><p>One way to achieve more efficient operation is through workload migration, where legacy workloads are moved to modern server hardware. Refactoring legacy monolithic applications to be cloud-native and built from microservices can further optimize power savings and reduce the total cost of ownership. Advanced orchestration and automation can also optimize server hardware utilization, with workloads distributed dynamically throughout the data center to improve cooling systems’ performance.</p><p>Integrating data from infrastructure into advanced closed-loop automation can also be a significant tool for running a sustainable enterprise, regardless of whether the data center is on-premises or in the cloud. Organizations can achieve significant gains in efficiency and sustainability by taking a hardware-oriented approach and implementing software structures and layers that optimize performance.</p><p><strong>Summary</strong></p><p>We have adopted the “circle of life” perspective to guide our approach in migrating workloads to and from the cloud. This view recognizes that deployment spans go through different stages in their lifetime, each with unique growth and needs that require optimization to meet evolving TCO versus ROI expectations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vw9SDzaIOcCE-4cW" /><figcaption>Figure-3 Yin and Yang for Cloud &amp; On-Premise</figcaption></figure><p>Throughout this article, we have emphasized the Yin and Yang of cloud versus on-premises infrastructure (Figure-3), highlighting the need for a hybrid cloud strategy to avoid the limitations of an exclusive approach. A hybrid cloud strategy provides greater flexibility, allowing workloads to be placed in the public cloud or on-premise depending on performance, cost, and security factors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1nL5kA3hLvsyHpk3Lr3UQ.png" /><figcaption>Figure-4 Flexera 2023 State of the Cloud Report, Hybrid &amp; Multi-Cloud Usage Statistics (<a href="https://info.flexera.com/CM-REPORT-State-of-the-Cloud-2023-Thanks">Link</a>)</figcaption></figure><p>If you have made it this far in reading, we don’t want to disappoint you with an ambiguous ending. However, the truth is that every business journey is unique with its own distinct characteristics that may change over time. Decisions made at a given time may need to be revisited later. Just as in the circle of life, a hybrid cloud strategy provides the freedom to adapt to changing circumstances and make necessary revisions. With a hybrid cloud approach, businesses can avoid being locked into rigid, predetermined paths and instead embrace the flexibility to evolve and grow with their changing needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a678243c2ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/open-5g-hypercore/episode-xiv-circle-of-life-7a678243c2ad">Episode XIV: Circle of Life</a> was originally published in <a href="https://medium.com/open-5g-hypercore">Open xG HyperCore</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>