<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amit Sharma on Medium]]></title>
        <description><![CDATA[Stories by Amit Sharma on Medium]]></description>
        <link>https://medium.com/@amit.sharma_14643?source=rss-d886a1fe6982------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k68pvU9c4dI1Mvcwv2dOdQ.png</url>
            <title>Stories by Amit Sharma on Medium</title>
            <link>https://medium.com/@amit.sharma_14643?source=rss-d886a1fe6982------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:14:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amit.sharma_14643/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Google Cloud Foundations -Automation Approach]]></title>
            <link>https://medium.com/engineered-publicis-sapient/google-cloud-foundations-automation-approach-a48003f33e8b?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a48003f33e8b</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 11:06:00 GMT</pubDate>
            <atom:updated>2024-11-17T17:56:52.409Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Publicis Sapient “Slingshot”</h4><p>Establish a robust and scalable GCP Cloud infrastructure that supports business growth while ensuring non-functional requirements.</p><h3><strong>Introduction</strong></h3><p>As a team, we recently completed a project on Google Cloud Platform (GCP) foundations for an enterprise organization. The goal was to establish a robust and scalable Cloud infrastructure that would support our growing business needs while ensuring security, compliance, and other non-functional requirements (NFRs).</p><p>By leveraging <strong>‘Slingshot’</strong> (Publicis Sapient’s AI assistant powered by RAG) for automation, we achieved exceptional efficiency, ensuring that our automated workflows and reusable modules were precise and closely aligned with our organization’s standards. Here are the key artifacts we generated with Slingshot while setting up the GCP foundations:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FgElp5mlIDjk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgElp5mlIDjk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FgElp5mlIDjk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae700a2c25cab7f1ef9d629dbe5cc73e/href">https://medium.com/media/ae700a2c25cab7f1ef9d629dbe5cc73e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*UIpQ9CuWvL9M_93b_xmbqw.png" /></figure><p>Automated workflows to provision the resources with minimal configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*aMxs_RZLOv-3iPjYrlByFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*ZrTYxMUTOIocmte253hugQ.png" /></figure><p><strong>Business ROI: Unlocking Productivity and Efficiency</strong></p><p>One of the primary objectives of our GCP foundations project was to provide automated and self-service Cloud-based resources for Product teams. By leveraging the power of IaaS (Infrastructure as a Service) and PaaS (Platform as a Service), we aimed to maximize productivity without compromising security and other NFRs.</p><p>The results were impressive:</p><ul><li><strong>25% reduction in provisioning time:</strong> With automated deployment scripts and Terraform configurations, our development teams can now spin-up new environments in minutes, compared to days or weeks previously.</li><li><strong>40% increase in developer productivity:</strong> By providing self-service access to Cloud resources, developers can focus on writing code rather than waiting for infrastructure to be set up. This led to a significant boost in overall productivity and velocity.</li><li><strong>90% reduction in manual errors:</strong> With automation and IAC (Infrastructure as Code) in place, the risk of human error when provisioning new environments was drastically reduced. By leveraging GCP’s automated infrastructure provisioning, we ensured environment parity across development, staging and production, reducing discrepancies and improving deployment reliability.</li><li><strong>Cost Efficiency:</strong> Through GCP’s cost management tools and automated scaling, we were able to optimize resource usage and significantly reduce overall Cloud expenditure.</li></ul><p>These metrics demonstrate the tangible business value generated from our GCP foundations project. By providing a scalable, secure and self-service Cloud infrastructure, we empowered Product teams to deliver faster, more efficiently, and with higher quality.</p><p><strong>Best Practices &amp; Gotchas</strong></p><p>There were valuable lessons learned that can be applied to similar initiatives:</p><p><strong>Best Practices:</strong></p><ul><li><strong>Adopt Terraform and IAC:</strong> Terraform is an industry-standard tool for managing Infrastructure as Code. Its benefits include:</li></ul><p>Version control of infrastructure configurations</p><p>Simplified provisioning and de-provisioning</p><p>Reduced errors through automated deployments</p><ul><li><strong>Emphasize automation:</strong> Automation is key to maximizing productivity while minimizing human error. Leverage tools like Ansible, Puppet or SaltStack to automate tasks such as Configuration management for infrastructure components.</li><li><strong>Focus on self-service access:</strong> Provide Product teams with self-service access to Cloud resources through APIs, SDKs or managed services. This will help them work independently and efficiently.</li><li><strong>Security:</strong> We ensured that each Terraform module adhered to stringent security standards, embedding best practices like encryption, IAM policies, and network configurations directly into the codebase.</li></ul><p><strong>Gotchas:</strong></p><ul><li><strong>Over-indexing on features:</strong> While it’s tempting to add every feature under the sun to your Cloud infrastructure, focus on essential capabilities that align with business goals.</li><li><strong>Requirement gathering failures:</strong> Ensure thorough requirements gathering from Product teams to avoid surprises during deployment or after go-live.</li><li><strong>Being too opinionated about our automated solutions:</strong> Sometimes, being overly restrictive in our automation can limit flexibility and hinder adaptation to evolving business needs.</li></ul><h3><strong>Conclusion</strong></h3><p>Our GCP foundations project demonstrated significant business value by providing automated and self-service Cloud-based resources for Product teams, while ensuring security and other NFRs. Key lessons learned include the importance of adopting Terraform and IAC, emphasizing automation, focusing on self-service access, and embedding security practices into the codebase. Remember to avoid over-indexing on features, ensure thorough requirements gathering, and remain flexible with automated solutions to adapt to changing business needs.</p><p>Leveraging <strong>Slingshot</strong> (our AI assistant powered by RAG) was instrumental in achieving these outcomes, enabling us to automate workflows and develop reusable modules with high precision and alignment to our organizational standards.</p><p>We hope this blog post provides valuable insights for your team’s future Cloud infrastructure projects. If you’d like to discuss these topics further or share your own experiences, feel free to comment below!</p><h3>Authors</h3><p>Mohit Gairola — Software Development Engineer 3</p><p>Amit Sharma — Senior Director, Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a48003f33e8b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/google-cloud-foundations-automation-approach-a48003f33e8b">Google Cloud Foundations -Automation Approach</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Real Value of AIOps]]></title>
            <link>https://medium.com/engineered-publicis-sapient/unlocking-the-real-value-of-aiops-b4bf35ac0cd4?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4bf35ac0cd4</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Wed, 21 Aug 2024 06:28:22 GMT</pubDate>
            <atom:updated>2024-08-21T06:28:22.518Z</atom:updated>
            <content:encoded><![CDATA[<h4>Navigating towards AI-Driven Observability with Data Foundations</h4><h4><strong>Introduction</strong></h4><blockquote><strong>The Rising Importance of AI-Driven Observability in Modern Enterprises</strong></blockquote><p>In today’s complex digital landscape, businesses face mounting pressure to maintain resilient and high-performing IT systems. AI-driven observability emerges as a critical solution, leveraging Artificial Intelligence to provide deep insights and proactive management capabilities. As organizations embrace digital transformation, AI-driven observability becomes essential for ensuring operational stability and efficiency. This article explores the crucial role of data foundations in enabling AI-driven observability, highlighting the challenges faced by enterprises and the transformative potential of automation and Generative AI.</p><blockquote><strong>The Role of Data Foundations in AIOps Adoption</strong></blockquote><p>AIOps’ success hinges on the strength of its data foundation. Without reliable, consistent and high-quality data, AIOps initiatives struggle to deliver accurate insights and automated responses. A robust data infrastructure ensures AIOps platforms receive the necessary data for precise anomaly detection and effective automation. Prioritizing a strong data foundation is not just about meeting current needs, but also about building a scalable framework for future technological advancements.</p><h4><strong>Enabling Automatic Resolution with AIOps</strong></h4><p><strong>AIOps Leverages Data Foundations for Automatic Issue Resolution, which is high stakes use case.</strong></p><p><a href="https://lion.box.com/s/hvs3wlm1teq9t7j2cx19uckxk5253p0i">Copy of aiops-v2-demo-full-edit - Made with Clipchamp (1).mp4 | Powered by Box</a></p><p>AIOps Machine Learning algorithms analyze the data, identifying patterns and anomalies in real-time and historical data. This allows AIOps to predict potential issues before they arise, enabling proactive maintenance and minimizing downtime.</p><p>Beyond prediction, AIOps automates issue resolution. Predefined rules and algorithms trigger corrective actions based on identified issues, eliminating the need for manual intervention and accelerating resolution times. As AIOps continuously learns from data, its algorithms become more refined, leading to even more accurate predictions and effective resolutions.</p><p>The integration of AIOps with robust data foundations ensures that automated resolutions are not only fast, but also accurate. This comprehensive view of the IT environment enables informed decision-making, enhancing operational stability, and ensuring the consistent delivery of high-quality IT services.</p><h3><strong>Implementing Data Foundations for AIOps</strong></h3><p>It is not easy for a large enterprise to take this journey. That’s why a curated list of items below will help you think through that process.</p><blockquote><strong>Standardization of metadata for AIOps</strong></blockquote><p>Creating organization-level standards for application and infrastructure metadata involves several key steps:</p><ol><li><strong>Assessment:</strong> Conduct a comprehensive assessment of the current metadata landscape, identifying existing sources, and evaluating data quality and consistency.</li><li><strong>Objectives and Scope:</strong> Define the objectives and scope of the standardization initiative, including specific metadata elements, target systems and desired outcomes.</li><li><strong>Governance Framework:</strong> Establish a governance framework outlining roles and responsibilities for metadata management, including data collection, validation and updating, ensuring consistency and compliance.</li><li><strong>Metadata Standards:</strong> Develop metadata standards covering structure, format and naming conventions for metadata elements, ensuring flexibility for future changes.</li></ol><blockquote><strong>Connecting and Collaborating with Different and Diverse Teams across the Organization</strong></blockquote><p>Effective standardization requires cross-team collaboration. Engage stakeholders from development, operations and business units to ensure alignment and address their needs. Establish a cross-functional team to identify common challenges, share best practices, and develop a unified approach to metadata management. Regular meetings and workshops facilitate progress, resolve issues and ensure continuous alignment.</p><blockquote><strong>Understanding Existing Standards and Use Cases</strong></blockquote><p>Understand existing standards and use cases within the organization through interviews and surveys. Gather insights into current practices, challenges and specific requirements. Document the current state of metadata across systems and applications, identifying gaps, inconsistencies and redundancies. Develop targeted strategies to address these issues and improve metadata quality and consistency.</p><blockquote><strong>Creating a List of Parameters/Metadata Values</strong></blockquote><p>Based on insights from the assessment and collaboration phases, create a list of relevant parameters or metadata values. Include key attributes related to applications and IT infrastructure, such as application name, version, dependencies, resource utilization, performance metrics and configuration details. Align these metadata values with the specific needs and use cases of different teams, ensuring comprehensive coverage.</p><p><strong>Metadata Standards Template</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rfePKJQiGAvqM5aewwpbDg.png" /></figure><h3><strong>Leveraging Automation for Data Foundation Setup</strong></h3><p>By leveraging automation, IT teams can streamline the setup of their data foundation, reducing the risk of human error and freeing up resources to focus on higher-value tasks.</p><h4><strong>Automating Data Tagging with Infrastructure as Code (IaC) using Terraform</strong></h4><p>Automating data tagging with Infrastructure as Code (IaC) using tools like Terraform significantly improves data management efficiency. By automating tagging, organizations ensure accurate categorization and easy retrieval of data assets.</p><p>One effective approach is creating a Terraform tagging module responsible for tag validation. This module accepts a list of tags as input and verifies compliance with organizational standards. It checks for missing tags, validates tag values, and generates errors if tags are non-compliant, preventing deployment.</p><p>Below, we demonstrate how to create a Terraform tagging module to automate the tagging process.</p><p>Tags can be supplied to the Terraform module using a variable, typically named tags. The validation of these tags occurs during the validation stage in Terraform. For each key-value pair, the key and value are validated using the validate block.</p><p>For example: To validate a specific tag, such as “business.region”, “technical.owner” and “application.name” we can define validation rules in the terraform module. Here’s an example:</p><pre>variable &quot;tags&quot; {<br>  description = &quot;This is key value pair of tags and there values&quot;<br>  type        = map(string)<br><br>  validation {<br>    condition = contains(&quot;North America&quot;, &quot;South America&quot;, &quot;Europe&quot;, &quot;Africa&quot;, &quot;Asia_Oceania&quot;], lookup(var.tags, &quot;business.region&quot;, &quot;&quot;)<br>    )<br>    error_message = &quot;The tags map must contain a key &#39;business.region&#39; &amp; values must be one of [North America, South America, Europe, Africa].&quot;<br>  }<br>  <br>validation {<br>  condition = (<br>    alltrue([<br>      for email in split(&quot;/&quot;, lookup(var.tags, &quot;technical.owner&quot;, &quot;&quot;))) : can(regex(&quot;^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$&quot;, trimspace(email)))<br>    ])<br>  )<br>  error_message = &quot;The tags should contain &#39;technical.owner&#39; that must be a valid email address of owner of this application. Distribution lists are also acceptable.&quot;<br>}<br><br>  validation {<br>    condition     = contains(keys(var.tags), &quot;application.name&quot;)<br>    error_message = &quot;The tags map must contain a key &#39;application.name&#39;.&quot;<br>  }<br>}</pre><p><strong>Tag categorization</strong></p><p>Within the tagging module, tags can be grouped into categories based on their applicability to different resource types. This approach facilitates the efficient management of both common and resource-specific tags. In the example below, we group the common tags in the common_tags variable and the load balancer tags in the load_balancer_map variable.</p><p>Notice that the load balancer tags are stored within a map. This design allows us to manage different sets of load balancer tags effectively. For instance, if you have Terraform code that deploys multiple load balancers, each load balancer might have its own unique set of values.</p><pre>locals {<br>  common_tags = {<br>    &quot;business.region&quot; : lookup(var.tags, &quot;region&quot;)<br>    &quot;technical.owner&quot; : lookup(var.tags, &quot;technical.owner&quot;, &quot;&quot;))<br>    &quot;application.name&quot; : lookup(var.tags, &quot;application.name&quot;, &quot;&quot;)<br>  }<br>  <br>  load_balancer_map = {<br>    for lb_key, lb_value in var.load_balancer_values : lb_key =&gt; merge(<br>      # Required Variables<br>      lookup(lb_value, &quot;lb_target_group&quot;, null) != null ? { &quot;lb.target.group&quot; : lb_value.lb_target_group } : {},<br>      lookup(lb_value, &quot;lb_listener_protocol&quot;, null) != null ? { &quot;lb.listener.protocol&quot; : lb_value.lb_listener_protocol } : {},<br>      lookup(lb_value, &quot;lb_listener_port&quot;, null) != null ? { &quot;lb.listener.port&quot; : lb_value.lb_listener_port } : {},<br>      lookup(lb_value, &quot;lb_session_stickiness&quot;, null) != null ? { &quot;lb.session.stickiness&quot; : lb_value.lb_session_stickiness } : {}<br>    )<br>}</pre><p><strong>Output</strong></p><p>Within the output file, you can combine all the tags into an output variable to make them easily accessible for other modules or resources. This practice ensures that the tags are consistently applied and can be referenced wherever needed.</p><pre>output &quot;common_tags&quot; {<br>  description = &quot;Combined tags from common tags and user-provided tags&quot;<br>value = merge(var.tags, local.common_tags)<br>}<br><br>output &quot;load_balancer_tags&quot; {<br>  value = { for k, v in local.load_balancer_map : k =&gt; merge(var.tags, v, local.common_tags) }<br>}</pre><p>Notice that the output for the load balancer is also a map, storing tag values for different load balancers. Each load balancer within the map will have its corresponding load balancer specific tags, the common tags, and any additional user-provided tags.</p><p><strong>Using the Terraform tagging module</strong></p><p>In your Terraform application infrastructure code, you can then create a module and reference your tagging module as the source, as shown below:</p><pre>module &quot;tagging &quot; {<br>    source = &quot;[add link to your terraform module source here]&quot;<br>    tags = {<br>      &quot;business.region&quot;                    = &quot;North America<br>      &quot;technical.owner&quot;                    = &quot;john@example.com&quot;<br>      &quot;application.name&quot;                     = &quot;CRM portal&quot;<br>    }<br>    <br>    load_balancer_values = {<br>        “example_lb” = {<br>            lb_target_group = “example-tg”<br>            lb_listener_protocol = “HTTP”<br>            lb_listener_port = 80<br>            lb_sticky_session = true<br>         }<br>    }<br><br>}</pre><p>Below is an example of how you can use the tagging module to create an AWS Load Balancer with tags. This example assumes you have already defined the tagging module as described in the previous sections.</p><pre>resource &quot;aws_lb&quot; &quot;example&quot; {<br>  name               = &quot;example-alb&quot;<br>  internal           = false<br>  load_balancer_type = &quot;application&quot;<br>  security_groups    = [aws_security_group.alb_sg.id]<br>  subnets            = [aws_subnet.subnet1.id, aws_subnet.subnet2.id]<br><br>  enable_deletion_protection = false<br><br>  tags = {<br>    Name = module.tagging.load_balancer_tags.example_lb<br>  }<br>}</pre><p>As shown above, the Terraform tagging module serves as a reusable component designed to standardize and automate the application of tags to Cloud resources. This can be applied to all Terraform resources that can be tagged.</p><h4><strong>Tagging governance with Open Policy Agent</strong></h4><p>As mentioned above, in modern Cloud infrastructure management, standardizing resource tags is critical for maintaining consistency, and ensuring compliance with organizational policies. This is where Open Policy Agent (OPA) comes into play. OPA is a versatile, policy-based control framework designed to unify policy enforcement across the stack. In this article, we’ll explore what OPA is, and how it can be used to validate tags in a Terraform JSON file.</p><p>OPA can be integrated with various systems such as Kubernetes, API gateways, microservices, CI/CD pipelines, and more. It allows you to define policies that enforce security, compliance, and operational best practices across your infrastructure. To standardize Cloud infrastructure tags, you can use OPA to validate Terraform plan JSON files. OPA uses a high-level declarative language called Rego to write policies. Here’s an example of Rego policy for tag validation:</p><pre>package terraform.tags<br><br># Define a rule that validates required tags<br>required_tags = [&quot;Environment&quot;, &quot;Owner&quot;, &quot;Project&quot;]<br><br># Rule to check if the resource has the required tags<br>violation[resource] {<br>  resource := input.resource_changes[_].change.after<br>  some tag in required_tags<br>  not resource.tags[tag]<br>}</pre><p>In summary, by using Open Policy Agent (OPA), you can ensure that your Cloud infrastructure adheres to standardized tagging policies, enhancing resource management and compliance. OPA’s powerful policy language, Rego, allows you to define and enforce complex policies with ease. Integrating OPA with your Terraform workflows can automate and streamline policy enforcement, helping you maintain consistent and compliant Cloud environments.</p><h3><strong>Building Automated Dashboards for Observability Tools</strong></h3><p>Automated dashboards provide real-time visibility into system performance and health, enabling proactive monitoring and management. By integrating observability tools with automated dashboards, organizations can visualize key metrics, detect anomalies, and gain actionable insights. Dashboards can also be used to gain visibility into an organization’s resource metadata. Proper and standardized tags are essential for AIOps integration, and dashboards can help monitor compliance with the organization’s observability tagging standards.</p><p>Viewing Cloud resource tags on tools like New Relic offers additional insights and robust visualization options for monitoring and managing Cloud infrastructure. Dashboards can display metrics on the number of compliant and non-compliant resources and the overall compliance percentage. Below is an example of a custom dashboard application created using the New Relic Nerdpack SDK:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7pWVa1m68vgDXY3Tu5pKw.png" /></figure><p>As seen in the dashboard image above, the tag monitoring dashboard offers a comprehensive overview of the tag compliance for Lambda resources. The compliance status reflects the percentage of mandatory tags adopted by each resource. The dashboard displays the average compliance status for all Lambda resources and provides detailed information on individual compliance status. Additionally, it highlights the specific tags missing for each entity, enabling a more detailed view of resource tags and facilitating the quick identification of missing tags for each resource.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PHhaxcnLgUk-llhdMdyqgg.png" /></figure><p>The screenshot above shows the summary page of the monitoring dashboard, providing a high-level overview of an organization’s overall tag compliance.</p><p>Overall, the tag monitoring dashboard offers several benefits. It provides a clear and concise summary of tag compliance across all resources, ensuring that essential tagging requirements are met. This tool helps maintain compliance, improves resource organization, and supports efficient operational oversight.</p><h3><strong>Conclusion</strong></h3><p>Data foundations are critical for enabling AI-driven observability, providing the necessary infrastructure to support comprehensive data management and advanced analytics. Robust data foundations ensure data integrity, accessibility and scalability, enabling enterprises to leverage AI and Machine Learning for proactive IT operations. By addressing common challenges such as data silos and fragmentation, enterprises can build a strong data foundation that supports effective AIOps adoption and enhanced observability.</p><h3>Authors:</h3><p><strong>Gaurav Mehta</strong>: Manager Technology</p><p><strong>Madina Diane</strong>: Engineer</p><p><strong>Amit Kumar Sharma</strong>: Engineering Lead</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4bf35ac0cd4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/unlocking-the-real-value-of-aiops-b4bf35ac0cd4">Unlocking the Real Value of AIOps</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Service Analytics with Generative AI]]></title>
            <link>https://medium.com/engineered-publicis-sapient/self-service-analytics-with-generative-ai-376bfe5d1121?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/376bfe5d1121</guid>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Wed, 07 Aug 2024 06:40:14 GMT</pubDate>
            <atom:updated>2024-08-07T06:40:14.741Z</atom:updated>
            <content:encoded><![CDATA[<h4>Slice &amp; dice your data using natural language conversations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/1*W6fMR2WVoEOrt-G_ZU7QAg.gif" /><figcaption>Self service analytics tool</figcaption></figure><blockquote><strong>Problem Statement : “</strong>The Data Deluge Dilemma<strong>”</strong></blockquote><blockquote>Senior leaders within organizations are increasingly seeking data-driven insights to inform strategic decisions. However, they often face a significant hurdle: accessing relevant and actionable information on operations without relying on IT resources or lengthy development cycles.</blockquote><blockquote>This challenge arises from the growing volume of data being generated by various business functions, making it difficult for leadership teams to identify key trends, perform comparative analyses, and make informed decisions in a timely manner. As a result, they often struggle with getting the insights they need, leading to missed opportunities, delayed decision-making, and potential revenue loss.</blockquote><p><strong>Solution</strong> : Generative AI-based chat assistant trained on analytics sources using RAG pipelines.</p><p>This AI-driven platform empowers senior leaders to access relevant insights and perform complex analyses with the ability to interact with the system through natural language.</p><h3>Unlocking Business Value</h3><p><strong>Enhanced Decision-Making:</strong> Senior leaders can interact with the interface using natural language queries, allowing them to explore data and gain insights without the need for technical expertise. This empowers them to make data-driven decisions quickly and effectively.​</p><p><strong>Real-Time Performance Monitoring:</strong> The interface provides real-time updates on key performance indicators (KPIs) and metrics, allowing senior leaders to stay informed about the current state of operations. This enables proactive problem-solving and timely adjustments to strategies.​</p><p><strong>Customized Analytics:</strong> With NLP capabilities, the interface can generate customized reports and analytics based on the specific needs and interests of senior leaders. This personalized approach ensures that they receive the most relevant and actionable insights.​</p><h4>How We Brought Our Generative AI Project to Life:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pz8uQ5_rzZE5OG-h" /></figure><h4>A Journey through Data, Technology and Speed</h4><p>In the ever-evolving landscape of technology, integrating Generative AI into workflows can be a game-changer. My team and I embarked on a project focused on harnessing the power of GPT (Generative Pre-trained Transformer) models to revolutionize our operations. Here’s a behind-<br>the-scenes look at how we transformed from AI novices to successfully launching our Generative AI-focused project, navigating the intricacies of data, technology and speed along the way.</p><p><strong>Starting from Scratch: Our Initial Knowledge and Challenges</strong></p><p>One of the key challenges was defining the Use Case, especially given previous vendor failures. By leveraging GPT’s expertise, we researched the client’s domain, generated multiple solution options, and aligned on scope and delivery in a short time. We differentiated our approach by previewing the specific questions GPT could answer, using sample data from similar clients. This highlighted our value and clarified the scope of work, demonstrating where our Gen-AI tool outperformed their existing BI platform.</p><h4><strong>Innovating with a Data Dictionary</strong></h4><p><strong>Understanding the Problem</strong></p><p>Directly inputting large datasets into GPT proved inefficient. We needed a method for GPT to understand and work with data without ingesting it all, leading us to the concept of a ‘data dictionary.’</p><p><strong>Creating the Data Dictionary</strong></p><p>In our context, a data dictionary serves as a system prompt for GPT. It includes names of SQL tables and the required column names representing our clients’ data structures, acting as a blueprint for data organization.</p><p><strong>Implementing the Two-Call Approach</strong></p><p>To optimize our process, we devised a two-step GPT call system:</p><ol><li><strong>Initial Query Processing</strong>: The first GPT call reads the user’s prompt and consults the data dictionary to identify the relevant tables and columns needed to fulfill the request. GPT then generates an SQL query based on this information.</li><li><strong>Data Retrieval and Analysis</strong>: We execute the generated SQL query against our database to retrieve the necessary data. This data is then passed to a second GPT call, which performs a detailed analysis and provides insightful results back to the user.</li></ol><p>This innovative approach allowed us to leverage GPT’s capabilities more effectively, making our application faster and more responsive.</p><h4>Pillars of success</h4><p><strong>Enhanced User Experience</strong></p><p>We showcased our ability to create a compelling front-end experience, including a GPT-like interactive feature, voice input, client branding, and configurable greeting prompts. By adding appropriate charts and graphs to each response, we provided parity with the existing BI tool alongside our unique differentiators.</p><p><strong>Mastering Data Pipelines</strong></p><p>One of the cornerstones of our project was setting up an efficient data pipeline. We chose <strong>data warehouse </strong>for this purpose, as it seamlessly combined data transformation and storage capabilities within a single service. This integration simplified our workflow significantly, allowing us to handle data more effectively and without the need for multiple tools.</p><p><strong>Prompt Engineering</strong></p><p>We focused on designing precise prompts for AI models to improve the quality of responses. Key principles included clarity, context provision and iterative refinement, which proved effective in various applications like content generation and data analysis.</p><p><strong>Speed: The Competitive Edge</strong></p><p>Here is where we ran into our main issue with GPT integration and data analysis. For our project, achieving responsive analysis in under 20–25 seconds was a significant challenge. Balancing model selection, deployment strategies and cost-effectiveness was key to maintaining this competitive edge.</p><h4>The Lessons We Took Away</h4><ol><li><strong>Define the source of truth as early as possible in an engagement</strong>: Our client chose to compare the results from our application with their existing BI reports as the source of truth, rather than comparing with it with the actual data we were using in the application. So the team had to spend additional time dealing with the complexities of metrics and data sources being used by the existing BI reports. But after a thorough investigation, we demonstrated that the variations were within an acceptable range and provided clear explanations for them.</li><li><strong>Experimentation is Key:</strong> Don’t be afraid to experiment with different phrasings and structures. Sometimes, a small tweak can significantly enhance the output quality.</li><li><strong>Leverage Examples:</strong> Providing examples within the prompt can guide the model towards the desired response format. For example, “Your response should be this in JSON format: {“status”: 200, “analysis_response”: “Analysis indicates a low performing store for the next few months”}.”</li><li><strong>Stay Updated:</strong> The field of AI is continuously evolving. Just this week, GPT-4o came out free for all users to test out. Understanding the differences between each model and which one works best for your application is crucial. Engaging with the community through forums, webinars, and publications can provide valuable insights.</li><li><strong>Developing a Comprehensive Data Dictionary: </strong>To manage the complexity of our data and maintain consistency, we developed a data dictionary. This tool served as a centralized repository of definitions and mappings for our data, aiding in clear communication and streamlined processes across various project stages.</li><li><strong>Automating SQL Query Creation with GPT : </strong>One of the standout innovations in our project was using GPT to automate the creation of SQL queries. This capability not only enhanced our productivity, but also minimized errors, allowing us to focus more on analyzing and interpreting the data.</li></ol><h4><strong>Conclusion</strong></h4><p>By leveraging Generative AI in this self-service analytics platform, we’ve empowered senior leaders to make data-driven decisions quickly and effectively, driving enhanced decision-making and real-time performance monitoring. This has led to significant business gains, including improved response times, reduced costs and increased productivity. Additionally, the customized analytics capabilities have provided personalized insights, enabling organizations to stay ahead of the competition.</p><p>Bringing a Generative AI project to life is a multifaceted endeavor that requires deep data management, sophisticated technological integration, and a relentless focus on speed. From grappling with the complexities of data to devising innovative solutions like the data dictionary and the two-step GPT call system, our journey was filled with valuable learnings and breakthroughs.</p><p>By partnering with Publicis Sapient, organizations can unlock the full potential of Generative AI and drive transformative business outcomes like such.</p><h3>Authors :</h3><p><strong>Ravi</strong> : AI Engineer</p><p><strong>Ensar</strong> : AI Engineer</p><p><strong>VJ</strong> : Product Manager</p><p><a href="/service/mailto:amit.sharma@publicissapient.com"><strong>Amit</strong></a> : Engineering Lead</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=376bfe5d1121" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/self-service-analytics-with-generative-ai-376bfe5d1121">Self-Service Analytics with Generative AI</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-Driven Architecture at Scale Using Kafka — Part2]]></title>
            <link>https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-using-kafka-part2-b66edde84bb2?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/b66edde84bb2</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Mon, 03 Jul 2023 08:00:12 GMT</pubDate>
            <atom:updated>2023-07-03T08:00:51.231Z</atom:updated>
            <content:encoded><![CDATA[<h3>Event-Driven Architecture at Scale Using Kafka — Part 2</h3><h4>How to decide on the right Kafka cluster option</h4><h4>Problem Statement:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*lf_AFM-2nPw8jB9CvoSeAA.png" /></figure><p>With the increasing adoption of event-driven architectures by companies, as previously highlighted in <a href="https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-2bd1ef580d71">our original Medium article</a>, there has been a growing demand for a fully-managed and highly-available Kafka service.</p><h4><strong>Solution:</strong></h4><p>AWS MSK (Cloud Native) and Confluent Cloud (SaaS) are two of the leading providers of fully-managed Kafka services. However, it is imperative to carefully evaluate their offerings and choose the right service to ensure the future-proofing of your product, cost savings, and optimization of your data pipelines. This article provides a detailed comparison of the key components of MSK, MSK Serverless, and Confluent Cloud, in order to aid architects in selecting the ideal platform for their event-driven architecture requirements.</p><h4><em>Cluster Info</em></h4><p>What are the key differences between the cluster setup?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*GbcgISOAFP3VlHCPnEuqaw.png" /></figure><h4><em>Cluster and Storage Resizing</em></h4><p>What is required to alter the capacity of a setup?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*VUEzPMHG9O3v1PmDltvdUA.png" /></figure><h4><em>Monitoring</em></h4><p>How much insight into cluster operations do we get?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*wI5Wfr7ykbvVbuu_GnUNdg.png" /></figure><h4><em>Security</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*yp9bzhHMNV9q0CnuXyldxw.png" /></figure><h4><em>Logging</em></h4><p>What logs are available to users?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*RVGI7qQEgBjJ-rGT8Clenw.png" /></figure><h4><em>Schema Registry</em></h4><p>What are the data governance options?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*289E-iKI5_ij6EJcdkoEPA.png" /></figure><h4>Cost &amp; Billing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*nayyZNTFhgruKftP-El1qw.png" /></figure><h4><em>Kafka Streams</em></h4><p>Does the streaming use case have any differences?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*I28xeYDqgFUnFo7ujo9O2Q.png" /></figure><h4><em>KSQL DB</em></h4><p>Confluent created application, similar to the Schema Registry, that allows easy use of the Kafka Streams API through a microservice with SQL commands</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*PNYygCxP7t4-rbhvsOJ78Q.png" /></figure><h4><em>Included UIs</em></h4><p>How much information can we easily see?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*E3y9C51mdPcWZYAkYk-pfw.png" /></figure><h4>Disaster Recovery</h4><p>What options are available for disaster recovery and replication?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*zh2sqsveRNLtsLvV7BKcRA.jpeg" /></figure><h4><strong>Conclusions</strong></h4><p>Both AWS MSK and Confluent Cloud are managed Kafka services that offer scalability, reliability, and security for data streaming. It’s important to note that the comparison between AWS MSK and Confluent Cloud is based on the specific use case of event-driven architectures, and that the ideal platform for your requirements may vary depending on your business needs. Depending on your use case, you may choose one over the other. For example, if you are using an AWS-only ecosystem, MSK IAM Authentication may be a desired feature. Alternatively, if you are looking for minimal involvement in Kafka Cluster setup and maintenance, then Confluent Cloud’s offerings may be preferred. Ultimately, you should evaluate both services based on the needs and goals of your project.</p><h4><strong>References</strong></h4><p>1. AWS M5 instance family for limitation assessment of MSK: <a href="https://aws.amazon.com/ec2/instance-types/m5/">https://aws.amazon.com/ec2/instance-types/m5/</a></p><p>2. Kafka version support in Confluent Cloud: <a href="https://docs.confluent.io/cloud/current/faq.html#what-version-of-ak-does-ccloud-use">https://docs.confluent.io/cloud/current/faq.html#what-version-of-ak-does-ccloud-use</a></p><p>3. Updating the broker type in MSK: <a href="https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-broker-type.html">https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-broker-type.html</a></p><p>4. Partitions and data retention in MSK Serverless: <a href="https://aws.amazon.com/blogs/big-data/create-more-partitions-and-retain-data-for-longer-in-your-msk-serverless-clusters/">https://aws.amazon.com/blogs/big-data/create-more-partitions-and-retain-data-for-longer-in-your-msk-serverless-clusters/</a></p><p>5. Cluster types in Confluent Cloud: <a href="https://docs.confluent.io/cloud/current/clusters/cluster-types.html">https://docs.confluent.io/cloud/current/clusters/cluster-types.html</a></p><p>6. Schema linking support in Confluent Cloud: <a href="https://www.confluent.io/blog/easy-data-migration-with-schema-linking/">https://www.confluent.io/blog/easy-data-migration-with-schema-linking/</a></p><p>7. Client support for various Confluent Platform features: <a href="https://docs.confluent.io/platform/current/clients/index.html#supported-cp-features-for-ak-clients">https://docs.confluent.io/platform/current/clients/index.html#supported-cp-features-for-ak-clients</a></p><p>8. AWS Glue Schema Registry: <a href="https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas">https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas</a></p><p>9. AWS MSK Pricing: <a href="https://aws.amazon.com/msk/pricing/">https://aws.amazon.com/msk/pricing/</a></p><p>10. Confluent Cloud Pricing: <a href="https://www.confluent.io/confluent-cloud/pricing/?tab=overview">https://www.confluent.io/confluent-cloud/pricing/?tab=overview</a></p><p>11. Confluent Cloud support packages: <a href="https://www.confluent.io/confluent-cloud/support/">https://www.confluent.io/confluent-cloud/support/</a></p><h4>Authors:</h4><p><a href="/service/mailto:amit.sharma@publicissapient.com"><strong>Amit Sharma</strong></a>: Head Of Engineering</p><p><strong>Gaurav Mehta</strong>: Architect</p><p><strong>Lakshumanan : </strong>Product Manager</p><p><strong>Mckenzie Murphy, Rakib Kamal &amp; Warren Bigelow: </strong>Software Developers</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b66edde84bb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-using-kafka-part2-b66edde84bb2">Event-Driven Architecture at Scale Using Kafka — Part2</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Revolutionize Your Point of Sale System with a Standardized Peripherals Framework]]></title>
            <link>https://medium.com/engineered-publicis-sapient/revolutionize-your-point-of-sale-system-with-a-standardized-peripherals-framework-a4c5041cff6b?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4c5041cff6b</guid>
            <category><![CDATA[pos]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Tue, 16 May 2023 10:01:18 GMT</pubDate>
            <atom:updated>2023-05-16T10:01:18.790Z</atom:updated>
            <content:encoded><![CDATA[<h4>Maximizing Efficiency and Cost Savings with a POS Peripherals Framework</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*bXirRnA8FJG9Ix9k2Q0wxQ.jpeg" /></figure><p>With the advancements in technology, businesses are now able to integrate multiple peripherals with their POS systems. This has led to the emergence of a standardized peripherals framework, which can help businesses streamline their operations and achieve better returns on investment (ROI). Standardizing peripheral devices used in POS systems can greatly benefit businesses by providing cost savings, compatibility, reduced maintenance, improved scalability, and above all, time-to-market for competitive features. This article discusses different standards available for integrating POS peripherals and their limitations. It also provides details on how standard specifications like OPOS and JavaPOS work and can be leveraged to build standard frameworks, and solve for the major challenges faced by retailers on device integration.</p><p><strong>Problem:</strong> In the past, each POS device vendor had its own proprietary way of communicating with its hardware devices, which made it difficult to develop applications that could work with multiple vendors’ hardware.</p><p><strong>Solution:</strong> Standardizing peripherals helps to build POS systems that are vendor-agnostic and also offers several benefits which are explained in detail below.</p><h4>Benefits:</h4><p>· <strong>Compatibility:</strong> By using a standard interface, POS systems can work with a variety of peripheral devices from different manufacturers that best fit their needs without worrying about compatibility issues.</p><p>· <strong>Cost savings: </strong>Standardization promotes healthy competition among manufacturers by no vendor locking and also there is no device integration development cost when we switch device from different vendor.</p><p>· <strong>Reduced maintenance:</strong> Using a standard interface can also simplify maintenance and support for peripheral devices. Since the interface is standardized, it can be easier to troubleshoot issues and replace devices when necessary.</p><p>· <strong>Improved scalability:</strong> Standardizing peripheral devices can also make it easier to scale up or down the restaurant point of sale system as the business grows or changes. This can save time and money in deploying and configuring peripheral devices.</p><h4>Unleashing the Power of Standardization:</h4><p><strong>Understanding Standardization:</strong></p><p>There are several standards available for integrating POS peripherals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TlKwLFOgKBzFYItrpNf3vw.png" /><figcaption>UPOS implementation Standards &amp; their Differences</figcaption></figure><p><strong>How OPOS and JavaPOS works:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*MubA_fTw2sXXlwSUUExkEA.png" /></figure><p><strong>OPOS:</strong></p><p>A POS application which needs to communicate with OPOS standard devices needs to interact with the common control object library. This can be downloaded from <a href="http://monroecs.com/oposccos_current.htm">http://monroecs.com/oposccos_current.htm</a>.</p><p>For each device, there will be an OCX file (ex: OPOSCashDrawer.ocx). This OCX file needs to be registered in windows registry. Based on the tech stack used in POS application, we can create an object of this Device class (ex: If we use .Net then we need to create a COM callable wrapper Dynamic Link Library (DLL) using Type Library Importer Tool (Tlbimp.exe) by passing the OCX file as param and use the output DLL in our project), register for events and call methods on it to communicate with the device.</p><p>The Open Method of the Device class will take the device logical name as parameter and based on the passed logical name, it will refer the windows registry and create service object. If we have to switch to a different device from another vendor, then we have to just pass the right logical name of the device to the control object, and there is no code change required.</p><p>Registering and configuring the service object is outside of POS application development. Typically, the device manufacturer will provide an installer which will take care of registering service object with its logical name, configuring device settings like port/baud rate and also install required OS device drivers.</p><p><strong>JavaPOS:</strong></p><p>JavaPOS works in the same way except the COM technology. JavaPOS controls can be downloaded from <a href="https://github.com/JavaPOSWorkingGroup/javapos-controls">https://github.com/JavaPOSWorkingGroup/javapos-controls</a>. To create object of device control class, we need to have this JavaPOS control JAR file on our class path. Also after installing the vendor provided driver, we need to add the device service mapping to Jpos.config and place the service control JAR file on class path.</p><p>The Open Method of the Device class will take the device logical name as parameter and based on the passed logical name, it will refer the Jpos.config file and create the device service object. If we have to switch to different device from another vendor, then we have to just pass the right logical name of the device to the device control and there is no code change required.</p><p><strong>The major challenges faced on device integration:</strong></p><p><strong>Non-Standard:</strong> Despite the availability of standards like UPOS (OPOS, JavaPOS), and WS-POS, many large retailers still use devices that do not support any of these standards and instead rely on custom proprietary implementations for device communication.</p><p>This can result in several challenges:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*VhEGVdX6KDOcMscnUkXx4g.png" /></figure><h4>Unlocking the Benefits of Standardized Peripherals</h4><p>To overcome the limitations, we need to make some additions to the peripheral standards, so that it can support futuristic applications and technology needs.</p><p>· The first step is to separate the device integration logic from the POS application, so that it can be changed, tested, deployed, and configured independently.</p><p>· Add an extra layer on top of OPOS/JPOS to support platform-independent communication channels such as HTTP. This will enable the integration of devices with any tech stack or platform.</p><p>· Ensure that these device services are dynamically discoverable to reflect the current environment accurately instead of relying on static configurations that will not work when the device is removed or disconnected.</p><p>· Add an API gateway, such as NGINX, between business and device services, to achieve security and uniform access from one place.</p><p>· Write service objects/device services for non-standard devices in newer programming languages, so that they will also support standard and enable unified communication. This will also eliminate the need for developers with legacy skill sets.</p><p>· Develop simulators in the form of service objects/device services, so that the POS application or business services can be automatically tested. This will also help to develop and test complex scenarios without physical devices.</p><p>· Use tools like OpenTelemetry to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help analyze your software’s performance and behaviours.</p><p>Modern POS Application Reference Architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*21arnetCnvJO3k67lFMVHw.png" /></figure><h4>ROI with a Modern POS Peripherals Framework</h4><p><strong>Isolation (Self-contained, independent app for scale for each peripheral -Microservices design)</strong></p><blockquote>Fully utilize 64 Bit OS advantages</blockquote><blockquote>Reduces testing time (Lab/Manual Tester/Travel) by unit/BDD tests.</blockquote><blockquote>Reduces Deployment time as its independent unit.</blockquote><blockquote>Allows for independent testing and shift left methodologies.</blockquote><p><strong>Support for device standard (OPOS/JPOS)</strong></p><blockquote>No vendor lock (allows one to negotiate and close a deal with less spending)</blockquote><blockquote>Faster Time to Market</blockquote><blockquote>Device Failure Recovery</blockquote><blockquote>Standardization</blockquote><p><strong>Modern Programming Language</strong></p><blockquote>Unified device communication by moving to industry standard.</blockquote><blockquote>Removes developer needs with legacy skillset, dependency &amp; cost.</blockquote><blockquote>NFRs (Performance &amp; Security)</blockquote><blockquote>Reduce downtime  more order taking customer satisfaction.</blockquote><blockquote>Power Saving  Sustainable  Environment-friendly</blockquote><blockquote>Reduce Operator Costs</blockquote><blockquote>Secure &amp; reduce attacks.</blockquote><p><strong>Cross Platform</strong></p><blockquote>Reduces license cost.</blockquote><h4>Conclusion:</h4><p>Using industry standards like UPOS can greatly benefit the development and integration of point of sale peripherals. Adding platform independent communication channel, peripherals management, device discovery, API Gateway, security on top of existing standards will add more benefits to retailers. These standards provide a unified management, communication which enables different hardware and software components to work together seamlessly, reducing complexity, development time, and costs. Additionally, standards-based solutions tend to be more stable, reliable, and maintainable over time, ensuring long-term compatibility and interoperability with other systems. While there may be some challenges and limitations to using standards, the benefits are often worth the investment.</p><h4>References:</h4><p><a href="https://www.omg.org/spec/UPOS/1.15/Beta1/PDF">https://www.omg.org/spec/UPOS/1.15/Beta1/PDF</a></p><p><a href="https://github.com/JavaPOSWorkingGroup/javapos-controls">https://github.com/JavaPOSWorkingGroup/javapos-controls</a>.</p><p><a href="http://monroecs.com/oposccos_current.htm">http://monroecs.com/oposccos_current.htm</a>.</p><h4>Authors</h4><p>Vinoth Kumar : Senior Architect</p><p>Gabriel Drilea : Senior Architect</p><p>Amit Sharma : Head of Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4c5041cff6b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/revolutionize-your-point-of-sale-system-with-a-standardized-peripherals-framework-a4c5041cff6b">Revolutionize Your Point of Sale System with a Standardized Peripherals Framework</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managing Your Margins for Business Success via Error Budgets]]></title>
            <link>https://medium.com/engineered-publicis-sapient/managing-your-margins-for-business-success-via-error-budgets-8e225d9cd97b?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e225d9cd97b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[error-budget]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[site-reliability]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Fri, 14 Apr 2023 12:08:06 GMT</pubDate>
            <atom:updated>2023-04-14T12:08:06.971Z</atom:updated>
            <content:encoded><![CDATA[<h4>Error Budgets Matter More Than Ever Before with distributed systems</h4><p>You’ve got a business product in your organization having hundreds of micro-services/jobs, and you have been running it in production for some time now. Great work!</p><p>But you have found out that there are many of these in production which keep on failing either due to frequent updates from development teams or due to some underlying code issue that is not caught in test cycles, and keeps on coming occasionally for a specific production scenario. This makes you unhappy, since now your product is not only unreliable, but you are also burning an extra chunk of effort/money to fix the issues after the fact. On top of that, you are under pressure to deliver more customer features on this product.</p><blockquote>Which makes you think that you need to balance innovation and stability. Now you can be a Google, Netflix or Uber of the world who have error budgets to manage the reliability of its platform, or can be a business like airlines or retail stores, which has suffered massive outages to tank their reputation in a minute.</blockquote><p>Think of Error budgets as they can improve the release process which will help you improve the reliability of your product, and have a better customer experience. It will also help you decide whether new features can be launched in production based on if the applications are currently running with few or no errors, or if all launches need to be frozen until the number of errors is reduced to an acceptable error budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*zEVKSdTm-JNv2MnNwopNPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*ntgzAdB1Cj1oem1KmVfFuA.png" /></figure><p>ROI of error budgets is that they help businesses avoid costly downtime and lost revenue. By setting and tracking error budgets, companies can ensure that their systems are reliable and available to customers. This can lead to increased customer satisfaction and loyalty, which can ultimately drive revenue growth. Additionally, preventing downtime and outages can help businesses avoid the costs associated with fixing issues and responding to customer complaints.</p><h4>The Key to Building Resilient Systems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*9KJGDaME3F980iPzZHCEsA.png" /></figure><p><strong>Site Reliability Engineering (SRE)</strong> and <strong>Error Budgets</strong> are concepts pioneered by Google. By definition, Site Reliability describes the stability and quality of service that an application provides to the end user. SRE improves collaboration between development and operations teams. Developers often make rapid changes to an application to release new features or fix critical bugs. On the other hand, the operations team must ensure seamless service delivery. Hence, the operations team uses SRE practices to closely monitor every update and promptly respond to any issues that arise due to changes.</p><p>Having said that, no product or application can be 100% reliable. Error or failures are imminent and Error Budgets are the basis for measuring the Reliability, Throughput, and Performance of an application. An Error Budget is downtime that the system or systems may face without violating any contractual terms in the SLA.</p><p>For example, if your Service Level Agreement states that your system will be available 99% of the time, then the Error Budget for your system’s failure will be 1% of the time. If the application downtime exceeds the Error Budget, the development team devotes all resources and attention to stabilizing the application first.</p><h4>Error Budgets 101: Understanding and Implementing</h4><p>Let’s get a little more detail-orientated.</p><p>Below are 3 key SRE metrics that are needed to calculate the Error Budget:</p><p><strong>SLI:</strong> The Service Level Indicator is a measurement the service provider uses for the SLO goal. It is a quantitative measure of some aspect of service level such as Latency, availability, error rates</p><p><strong>SLO:</strong> The Service Level Objective is a goal for a component that a service provider wants to reach. It is a target value or a range of values for a service level that is measured by an SLI</p><p><strong>SLA</strong>: Contract with customers that include consequences for meeting or missing the SLO defined in the agreement</p><p><strong>The formula for calculating SLI:</strong></p><p><strong>SLI</strong> =[Good Events or Bad events /Valid Events] *100</p><p><strong>99% SLO is equal to 1% of the Error Budget</strong></p><p><em>Here is an example:</em></p><p>Consider you have a real-time application that processes customer orders and updates the backend system in real-time. This job is critical to business as reports are being refreshed every minute to display the total number of orders which is reviewed by the Store Manager every hour. You want to measure <em>the availability</em> of this job and calculate Error Budget for it to make sure it is up and running 99% of the time. So, in this case:</p><p>SLO: SLO will be 99% i.e. this job needs to be available 99% of the time</p><p>SLI: Records processing error rate &lt;1% every 5 minutes i.e., total number of failed orders/Total number of orders*100 should be less than 1% measured every 5 minutes</p><p>Timeframe: This needs to be calculated for the last 30 days of the time frame</p><p><em>Now, based on the above values, Error Budget can be calculated for this application</em></p><p><strong><em>Calculations</em></strong></p><p><em>Number of times application to be measured in a day if the error rate is calculated every 5 minutes = 288 instances (Formula: 24hrs*60min/5)</em></p><p><em>Number of times application to be measured in 30 days = 1440 instances (Formula 288 instances *5)</em></p><p><strong><em>SLO </em></strong><em>= 99%</em></p><p><strong><em>Error Budget </em></strong><em>= 0.01 for 1 instance (Formula: 1 -SLO/100) i.e., Error Budget for an application that needs to be measured every 5 min for 30 days = </em><strong><em>14.40 </em></strong><em>(1440*0.01)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*If21FKwXKrNqa9xt3QsWxQ.png" /></figure><p>In other words, this application instance measured every 5 min is allowed to fail only ~14 times in 30 days. After 14 failures, all Error Budgets would be considered consumed and the development team should devote all resources and attention to stabilize the application. No newer feature of this application should be allowed to deploy in production.</p><h4>Measuring Success &amp; Managing Priorities with Error Budgets are getting more common now</h4><p>The release velocity is controlled by ensuring that the SLOs are met or fulfilled before planning a new release. By measuring the consumption of the Error Budgets, you can control the rate of deployment. You can either slow down or speed up as per the remaining error budget. As per the above given example, for application 1 which has an availability SLO defined as 99%, the Error Budget comes out to be as 14 per month. Which means that all new features for this application can be deployed, if it doesn’t fail more than 14 times per month in production.</p><p>The SRE team needs to understand the product requirement, and what the developers expect the SRE engagement to achieve. Both teams must strike the right balance so that developers don’t just keep working on new features or changes that let applications make highly unreliable in production and exceed the Error Budget. Whereas, when engaging with a developer team, SREs should also build a deep understanding of the product and business goals as well so that new changes don’t keep piling on in the backlog.</p><p>Teams need to regularly talk with each other about business and production priorities. The SRE and developer leadership teams should ideally work as a unit, meeting regularly and exchanging views about technical and prioritization challenges.</p><p>Developers also need to commit to dedicating a reasonable percentage of engineering time to fixing and preventing the things that are breaking reliability: resolving ongoing service issues at the design and implementation level and including SREs in new feature development early, so that they can participate in design conversations.</p><p>Error Budget can also be adjusted based on the impact on service reliability, to make sure that a new feature or a change is released as quickly as possible, but within Error Budget limits.</p><h4>Ways to monitor applications and report Error Budget</h4><p>Error Budget can be applied to any type of application, such as web, mobile, or data applications. These applications could run in real-time, batch, or have API interfaces. There are multiple ways and different tools that are available in the market which can be used to monitor your application and store your metrics data to calculate the Error Budget.</p><p>Example architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*1gsXUNS687xWvtpPcx7mPw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*8xACrP1g1iDFz_GowjeAEA.png" /></figure><h3>Conclusion</h3><p>Effective monitoring and defining an Error Budget for your application is just as important as making good technical design decisions for the application. Once defined, it helps you to:</p><p><strong>Eliminate risk</strong>: Working within Error Budgets can eliminate unnecessary risks and give your teams a threshold for reliability when pushing new features or changes to production</p><p><strong>Accountable: </strong>With SLOs in place, there will be shared ownership between the development, operational, and product teams</p><p><strong>Prioritize</strong>: With an Error Budget, you can focus and prioritize your efforts in the right direction</p><p>Error Budgets provide a tangible way for business leaders to measure and manage system reliability. By prioritizing reliability, businesses can reduce costs, improve customer satisfaction, and ultimately drive growth.</p><h4>Contributors :</h4><p><strong>Sandeep Bhanot: </strong>Senior Specialist Engineering</p><p><a href="/service/mailto:amit.sharma@publicissapient.com"><strong>Amit Sharma</strong></a>: Head of Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e225d9cd97b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/managing-your-margins-for-business-success-via-error-budgets-8e225d9cd97b">Managing Your Margins for Business Success via Error Budgets</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-Driven Architecture at Scale Using Kafka]]></title>
            <link>https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-2bd1ef580d71?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bd1ef580d71</guid>
            <category><![CDATA[schema]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Mon, 02 Jan 2023 17:17:38 GMT</pubDate>
            <atom:updated>2023-07-07T16:24:00.688Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><blockquote>You are a digital technology leader who is either starting your digital transformation (may be pandemic-forced) or neck deep into one already. Your focus is to gain a competitive advantage in your field by building highly robust, scalable, and agile applications. Using EDA helps achieve the competitive level of the non-functional attributes by needing minimal connection between systems while still allowing them to interact, making it ideal for modern, distributed systems. Modern application designs are often event-driven, requiring an advanced knowledge of the<a href="https://stackoverflow.blog/2020/03/16/how-event-driven-architecture-solves-modern-web-app-problems/"> event-driven architecture</a>.</blockquote><p>Event-driven micro-services allow for real-time micro-services communication, enabling data to be consumed in the form of events before they’re even requested. It extends core principles of micro-services, including decoupling, separation of concerns, agility, and real-time streaming of event data.</p><p><em>There are many technologies in this space - open source, cloud-native, or vendor-specific. But based on our experience, we feel </em><a href="https://kafka.apache.org/"><em>Kafka</em></a><em> is a great fit for most use cases requiring scale due to its distributed nature</em>. It is used by a vast number of companies to build high-performance data pipelines, enable real-time data analysis, and integrate data from critical applications.</p><h3>Pitfalls faced in most Large Kafka Implementations</h3><h4><strong>Error Handling/Reliability</strong></h4><p>We know this is the most important quality of modern applications, but with multiple configurations available in Kafka, it gets complicated to select the right combination of configurations. While the right circuit-breaking settings with exponential backoffs with jitter can get you a resilient application, but the emphasis is required on producer and consumer configs like acks, consumer delivery semantics, offset commit strategies, etc. to have true reliability in a distributed system. The throughput (best effort) vs reliability (never lose my message) lens is a fair game based on a use case.</p><h4><strong>Schema Evolution</strong></h4><p>Message’s schematic sanctity is a gigantic problem in distributed applications, where producer and consumer applications don’t lie in the same product boundaries. A lot of implementations fail in this space as testing or retrieving information from each other becomes irreparable. <a href="#fc81">Design first schema-led approach</a> is the best solution to this problem which still requires collaboration with cross-products teams to get it right. This way it can be ensured that a well formatted and valid message is exchanged between a producer and a consumer. Data semantics check can also be added as an add-on to achieve data quality. Doesn’t that sound amazing?</p><h4><a href="#2f13"><strong>Automation</strong></a></h4><p>Manual toil is always a killer for modern application implementations. It’s nothing different for Kafka-based implementations. If you don’t nail your IAC to get the Kafka creation, then you are already one step in the ditch. Automation with tools like CDK, terraform, etc. will get you the desired infrastructure, but think of extending that to the automated creation of topics, schemas, back-up of schemas, metric-based elasticity for brokers etc. with DLQ management and automation to make life easy.</p><h4><strong>Serverless vs long-running Compute</strong></h4><p>Another silent assassin for these failed implementations is the use of the right compute for producer and consumer applications. While serverless is very tempting, it is not always the right choice. Apply the lens of throughput, scale and long-running vs on-demand nature to decide if serverless will work for you or not. Pressure test this in your automated testing environments to make the right choices.</p><h3>So how should you proceed? Think of it as a common framework</h3><p>Templated infrastructure, producers, and consumers can be created with all the NFRs (like reliability, resiliency, security, SRE, etc.) as in built framework that can be taken by different product teams to extend in their business use-case. Centralized control with decentralized implementation can help improve the development efforts and standardization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezB7NzqVcVn6zmqlH8Yfpg.png" /></figure><p>Key aspects to take care are :</p><ul><li>Standardize the technology platform for eventing and streaming use cases like Kafka</li><li>Standardize patterns around producing/consuming events, evolving schemas (compatibility included), stream processing, etc.</li><li>Standardize SRE &amp; Auditing design for event and stream processing</li><li>Set up the data quality governance around events and their schemas within a domain and across domains.</li><li>A consistent approach to improve reliability for fault-tolerance and event reprocessing.</li></ul><h3>How to get the key aspects of Kafka implementation right</h3><h4>Schema Validation to drive design first and Data Quality</h4><p>Message Schemas could be used as a contract between Producer and Consumer, which will ensure that a producer will publish a message in a format that a consumer is expecting. This contract will enforce<br>data governance between the message that is produced and consequently consumed. Schema Registry is an application external to Kafka, that maintains the database of schemas. Confluent provides its own version of Schema Registry and AWS provides that functionality as one of the<br>features of its Glue service.</p><p>Schema Registry could become one of the points of failure. Hence, certain measures should be taken to increase the availability. It will be a good design decision to run multiple instances of Schema Registry<br>and have it deployed in an auto scale mode. In this way, Schema Registry will be available when the load increases.</p><p>Messages will need to have the schema information along with the original message. Either the entire schema or an identifier of the schema will be sent along with the message. To make this process performant, it will be beneficial to locally cache the schema so that producers and consumers do not have to make frequent calls to an external component like Schema Registry.</p><p>Any modifications or changes in schema should be done in a lower environment and should be tested before implementing them in Production environment. In this way, a well-tested and designed schema will be introduced in Prod, as it will need a bit of downtime to have a changed/new version of schema.</p><h4>Reliable Producers</h4><p>For large volumes of data transactions in an organization, it is not only important to have a performant system that can handle the heavy data traffic, but it is equally important to have a reliable system that ensures all the data that is sent is also received. Kafka is highly fault tolerant, as it has some internal mechanisms to ensure that messages are sent and received reliably. Transient issues in sending will be retried a certain number of times by Kafka in the hopes that the error will resolve.<em> However, sometimes this in-built level of error handling is not enough.</em></p><p>If the Producers are unable to send the messages to Kafka cluster, then it needs to be stored somewhere to resend them to Kafka. A Dead Letter Queue (DLQ) topic, without schema validation, could be used to hold the messages and resend the messages later. A header attribute could be used in the messages to store metadata around the context of the failure. DLQ topic will not work in case the Kafka cluster is unavailable. <em>In this scenario, DLQ will not work as there will be no connectivity with the cluster. Therefore, a different solution will need to be used that will not involve any Kafka cluster component. A separate storage mechanism like a Database (DB) could be used.</em></p><h4>Consumers that can scale</h4><p>Kafka Consumers generally take a long-polling approach when it comes to consuming messages from topics in near real-time. This is optimized by parallelizing individual consumer applications to consume from separate Topic-Partitions. Given this, it is vital to design your applications to scale as safely as possible.</p><p>There are many important factors to consider when designing scalable consumer applications (<em>like Segregated Consumer Group Naming, CommitFailedException, Static Group Membership, etc</em>). The nature of how you consume must capitalize on the parallelism unit that the framework provides, and you should avoid writing code that can interfere with scalability patterns. How your consumer handles retriable and non-retriable failures can have downstream impacts on reliability and performance, so choose an approach that makes sense from your business use case. Finally, rebalancing is an inevitable functionality in most pub-sub frameworks, so optimize your design to rebalance only when necessary.</p><h4>Automation to keep sanity</h4><ul><li>IAC using Terraform or similar tools to stand up Kafka clusters on your cloud or on prem.</li><li>Use IAC to automate topic creation or schema creations.</li></ul><pre>resource &quot;kafka_topic&quot; &quot;mytfkafkatopic&quot; {<br>  for_each           = toset(var.topic_name)<br>  name               = &quot;${var.region}-${var.market}-${var.environment}-${each.key}&quot;<br>  replication_factor = var.replication_factor<br>  partitions         = var.partitions<br>  config = {<br>    &quot;segment.ms&quot;     = var.segment<br>    &quot;cleanup.policy&quot; = var.cleanup-policy<br>    &quot;retention.ms&quot;   = var.retention-ms<br>  }</pre><ul><li>Automate broker scaling using defined metrics</li><li>Automate cluster rebalancing using cruise control</li><li>Automate testing using <a href="https://www.testcontainers.org/">test containers</a> like tools</li></ul><h4>Testing &amp; Observability</h4><p>Unit tests ensure that the code has no glaring issues, but integration and performance tests are vital in uncovering problems with an actual running system. Much of a producer’s/consumer’s functionality can only be tested alongside a running Kafka and Schema Registry setup. This is easy to achieve with local docker containers. For Jvm applications , the TestContainers library can be used to create an end-to-end integration<br>testing suite. This isolated environment is the perfect place to test all functionalities without worrying about changing an existing Kafka cluster.</p><p>To fully capitalize on those benefits, we need to closely monitor our infrastructure and catch problems before they arise. “Preventing fires is better than fighting them.”<br><em>• Monitor the Kafka Cluster<br>• Monitoring Distributed event services<br>• Monitoring Topics</em></p><h4>Administration</h4><p>Kafka does not have a default UI, but there are some third-party tools which can be used to display Kafka resources graphically.<br>Kafdrop is one of the web UIs for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages.</p><p>You can also extend this tool like we did to get additional benefits like DLQ management and automation with features like automated retry of retryable error messages, bulk fix and upload of DLQ messages for replay etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*Kf-zYyr5ldTmKWp3Krfo_A.png" /><figcaption>DLQ message update and Replay Capability</figcaption></figure><h3>Conclusion</h3><blockquote>Enterprises that use event-driven architecture can enjoy the advantages of scalable and potent real-time communication. Many strategic undertakings can benefit from this, including the internet of things (IoT), e-commerce, data integration across systems, and edge and fraud detection.</blockquote><p>There’s a lot of word around Kafka and how it can pave way for your digital transformation success. When properly executed, Kafka can be a decisive tool for sending and receiving data across your network. Improper implementations will not only deter you from processing data more efficiently, but they may also set off a technical debt cycle. So control these pitfalls and keep their solution in mind as you implement this potentially powerful tool.</p><p>Given the demand of Kafka , choosing the right cluster set up either in SaaS or cloud-native environment is very important. Please see the <a href="https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-using-kafka-part2-b66edde84bb2">part-2 of this blog</a> to review the criteria to chose the right one.</p><h3>Contributors</h3><p><a href="/service/mailto:amit.sharma@publicissapient.com"><strong>Amit Sharma</strong></a>: Head Of Engineering</p><p><strong>Kamakhya Das</strong>: Senior Architect</p><p><strong>Gopa Roy</strong>: Senior Developer</p><p><strong>Mckenzie Murphy &amp; Rakib Kamal &amp; Warren Bigelow: </strong>Software Developer</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bd1ef580d71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/event-driven-architecture-at-scale-2bd1ef580d71">Event-Driven Architecture at Scale Using Kafka</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extending Your Cloud Deployments to Edge]]></title>
            <link>https://medium.com/engineered-publicis-sapient/extending-your-cloud-deployments-to-edge-3bbd4cc4d977?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bbd4cc4d977</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Fri, 22 Jul 2022 15:10:17 GMT</pubDate>
            <atom:updated>2022-07-22T15:10:17.458Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>There may be scenarios in edge computing where the business is not worried so much about data collection and the subsequent analysis through edge devices, but rather their ability to deploy applications to their edge locations and keep them in synchronization.<br>Some companies may not have a distributed computing infrastructure yet, and are looking at options to transition to one. While other companies may already have hundreds of edge servers located in their storefronts or warehouses across the country (or even the world), seeking a way to keep the applications that are running on these servers up to date to ensure their business needs are met.</p><p>These businesses are looking for solutions to expand their ability to deploy to edge seamlessly and, when possible, integrate this extension to edge with their existing cloud-based deployments.<br>This blog caters exactly to that need of deploying and extending to the edge using edge computing on Kubernetes with an example of <a href="#a2b6">AWS Outpost and EKS below.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SlMWlQbLhX7c-IWZuDYg1Q.jpeg" /></figure><h4>Common problems of software updates at in-store locations</h4><p>Without the use of edge computing, updating and managing software on the customer premises can be a long and tedious process. Depending on the number of stores, it could take months to update all of the systems, as some of these companies rely on sending people to manually update the software on-site. The disparity between the time that the first store gets updated to the last can also lead to problems as the applications become out of sync. This applies to bug fixes as well, where small issues that need to be fixed can compound with some locations being multiple updates behind. With a distributed edge solution, businesses can manage their edge environments from a single control plane, even though the infrastructure is distributed across multiple edges. The servers sitting in their locations get the same operational consistency as the Cloud-based Kubernetes clusters. Changes made to the Cloud-based clusters will be propagated to all edge location clusters in mere minutes.</p><h4><strong>Edge Computing with Kubernetes for </strong>Centralized management of disparate infrastructure</h4><p>The features built into Kubernetes make the platform well-suited for a variety of use cases. You can use Kubernetes to manage any on-premises or cloud-based environment. It can be used in a multi-cloud context, where Kubernetes becomes the central control plane for multiple clouds; alternatively, in a hybrid context, it can manage both cloud-based and on-prem infrastructure at the same time.</p><ol><li>It can <strong><em>manage multiple types of infrastructure</em></strong> at the same time. That makes Kubernetes ideal for an edge computing scenario in which some applications (or parts of applications) reside on-premises or in the cloud, while others run at the edge. Kubernetes can manage all these resources centrally, eliminating the need to shuffle different management tools for each environment.</li><li>Because of its <strong><em>automated management features</em></strong>, Kubernetes can react instantly to changes at the edge. The fact that Kubernetes can deploy new container or application versions quickly is also an advantage for edge computing use cases. If a new location needs to be brought online or an application requires an update, Kubernetes offers the automated deployment features necessary to implement the change quickly.</li><li><strong><em>The fact that Kubernetes is a proven</em></strong> and easy-to-deploy platform makes it attractive for edge computing in the sense that edge remains a relatively new technological niche. Being able to implement edge environments using technology that organizations already know and trust, and that they can set up quickly, beats having to learn and deploy brand-new management solutions that are purpose-built for the edge.</li></ol><p><strong>Edge computing flavor with AWS Outpost?</strong></p><p>Outposts are AWS’s offering for edge servers: they afford users the ability to run AWS services on-premises through servers and racks that can be purchased from them. The infrastructure is managed by AWS, so there is less burden on the customer to perform updates. They also seamlessly connect to both the customer network and the AWS Region to cultivate a unique hybrid cloud experience. <a href="https://aws.amazon.com/outposts/">ref</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*mqyN6MbmDnZcLZCWIJ2cYA.png" /><figcaption>AWS Outpost Set up with extended VPC &amp; Subnet</figcaption></figure><p>The Outpost rack is perfect for accelerating deployment as mentioned above, as it has ample compute power and allows for the integration of container orchestration services like EKS. With a few simple actions, the customer can push updates or new software to edge locations. This acceleration of delivery is achieved by allocating a single, overarching EKS cluster and its control plane in the cloud so that the cluster and all of its nodes are reachable and updateable from anywhere with a decent connection. Outposts are linked to this cluster by creating a self-managed node group and attaching it to the cluster through the control plane. Customers will be able to send deployments to these self-managed node groups running on the on-premises Outposts from the regional cloud, or the Outpost’s node groups will be updated whenever the Outpost reaches out to the cloud (at least once a day).</p><p><strong>How can you set it up?</strong></p><p><strong>VPC</strong></p><p>To begin, we established a VPC with an assortment of private and public subnets. A majority of the subnets exist in the Availability Zones within the AWS Region. We also created a specific subnet to signify the Outpost, which could not overlap with any subnets in the Region. Per the Outpost documentation, the subnet it creates is a private subnet that extends the Availability Zone that it is in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*vYKnNr3e7Vth4Y_kJyp9Dw.png" /><figcaption>Separate Outpost Subnet</figcaption></figure><p><strong>Cluster</strong></p><p>Next, an Amazon EKS cluster was used to run the Kubernetes applications. When creating the EKS cluster, it is important to only specify subnets that run in the AWS Region which will be used for the creation of the Kubernetes control plane. While it is not necessary to specify the Outpost subnets, you should still ensure that its parent Region supports EKS.</p><p><strong>Creating Nodes &amp; Pods within Cluster</strong></p><p><em>Self-managed node groups for Edge</em></p><p>AWS Outposts currently support self-managed nodes on an EKS cluster (managed nodes and Fargate are not supported). During the creation of these self-managed node groups, you must specify the subnets that run on your AWS Outpost. With self-managed node groups, you have more control over the server, but you are responsible for managing it. This includes patching and upgrading the AMI and the nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*NPIqESJBVhD6kiMd8vL1cg.png" /><figcaption>Extension of Self managed node group on Outpost subnet</figcaption></figure><p><em>EKS Managed or Self managed node groups for Cloud</em></p><p>We also provisioned an EKS managed node group on the same cluster, but this time specifying the non-Outpost subnets in the Region.</p><p>Finally, in order to create pods in each of these node groups, we selected the groups via node labels and referenced an image stored in Amazon ECR (Elastic Container Registry). This deployed the same application to both the AWS Region and the Outpost.</p><p><strong>Think automation with Infrastructure as a Code Implementation (Terraform) same as your cloud operations</strong></p><p>We used Terraform to handle building the previously detailed infrastructure. Terraform is a leading infrastructure as code tool that allows users to quickly build huge projects and destroy them by running a few commands, while providing modules that allow for easy setup of the infrastructure. It can be used for configuration management, so we also used it to create the pods to showcase the application deployment.</p><p>Following is a Terraform snippet that creates the VPC. Note that the Outpost subnets need the CIDR Blocks that you want to assign to the Outpost as well as the Region that it is in. The use of NAT Gateways in this VPC allows the nodes to connect with each other and call out to the internet but blocks any inbound traffic from the internet, unless ingress security rules explicitly allow that traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*4ZnbEHwfrfdPay4-P34LHA.png" /><figcaption>Terraform Code</figcaption></figure><p><strong>Considerations for using an Outpost</strong></p><ol><li>The configuration of the Outpost device itself, which is established on a case-by-case basis to satisfy the needs of the business, and could include networking support, EC2 compute capacity, and EBS or S3 storage capacity.</li><li>An additional consideration is the service link, which is an encrypted set of VPN connections that are used whenever the Outpost device communicates with the AWS Region. There are two options for this service link connection, but either way it will enable communication between the Outpost and the AWS Region for both management of the Outpost as well as intra-VPC traffic between the Region and Outpost. The first option is private connectivity, which is closest to what we have implemented in our solution because it establishes the connection using an existing VPC and subnet that we have specified. You would select this option when creating the Outpost in the Outposts console and then the connection would be established after the Outpost is installed. This option minimizes public internet exposure by way of the VPC. Alternatively, the Outpost is able to create the service link VPN back to the AWS Region through public Region connectivity. With this approach, the Outpost needs connectivity to the Region’s public IP ranges, which can be achieved either through the public internet or AWS Direct Connect service.</li><li>And a final consideration is the local gateway, which serves two purposes. It provides a target in your VPC route tables for on-premises destined traffic, and performs network address translation for instances that have been assigned addresses from your customer-owned IP pool. You also use the local gateway for communication for internet-bound traffic.</li></ol><p><strong>Video Demo</strong></p><p><a href="https://lion.box.com/s/c4es0pr0rd5hy6umem4bc0fdzdzhse6a">https://lion.box.com/s/c4es0pr0rd5hy6umem4bc0fdzdzhse6a</a></p><h3>Conclusion</h3><p>With a distributed edge solution, businesses can manage their edge environments from a single control plane, even though the infrastructure is distributed across multiple edges giving the centralized control to software distribution &amp; management.</p><p>There are offering from specific vendors like HCP Nomad as well as other major cloud providers like <a href="https://azure.microsoft.com/en-us/overview/azure-stack/">Azure</a> (Azure Stacks) &amp; GCP (<a href="https://cloud.google.com/anthos">Anthos</a>) in this space which can be evaluated based on specific business needs. But the concept of deploying a containerized application to the edge will be pretty similar.</p><h3>Authors</h3><p><strong>Christian Kowalczyk</strong> : Junior Associate Software Development</p><p><strong>Shelby Theisen</strong> : Junior Associate Software Development</p><p><strong>Daniel Pimentel</strong> : Junior Associate Software Development</p><p><strong>Amit Kumar Sharma</strong> : Director Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bbd4cc4d977" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/extending-your-cloud-deployments-to-edge-3bbd4cc4d977">Extending Your Cloud Deployments to Edge</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Push-button pipeline for digital transformation API stack with an automated Canary release (using…]]></title>
            <link>https://medium.com/engineered-publicis-sapient/push-button-pipeline-for-digital-transformation-api-stack-with-an-automated-canary-release-using-7e97f0c90e5e?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e97f0c90e5e</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Tue, 06 Apr 2021 09:13:08 GMT</pubDate>
            <atom:updated>2021-04-06T17:06:06.826Z</atom:updated>
            <content:encoded><![CDATA[<h3>Push-button pipeline for digital transformation API stack with an automated Canary release (using Terraform &amp; Jenkins)</h3><h3>Introduction</h3><p>Continuous integration and continuous deployment are now mainstream among software companies. This pipeline can be easily understood as the process pathway through which we can deliver a single unit of production-ready software. For IT leadership, understanding the CI/CD pipeline is critical to keeping your organization on track for digital transformation goals. CI and CD form the backbone of the modern DevOps environment and companies are doing continuous delivery throughout its lifecycle. But <em>can your pipeline perform push-button deployments of any version of the software to any environment on demand?</em></p><p>That’s where the true challenge lies in the CI/CD process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EhyDTZK7PXazpO_nuQCAaA.jpeg" /></figure><h3>Key challenges with push-button pipeline define beleaguered DBT solutions</h3><h4>No single full-stack deployment pipeline</h4><p>A CD pipeline implementation may sound easy and smooth for deployment. However, many teams face challenges in implementing these pipelines for large digital transformation solutions. Especially when standing up full environments with all the feature functionalities like exp APIs, microservices, etc. A single microservice CI pipeline is great for development but for building integration and performance environments for on-demand for MVPs need single full-stack pipelines.</p><p><em>An interesting fact here is the accrued cost of manually setting up full environments for every release adds up to a mountain.</em></p><h4>Inadequate configuration management and non-reusable IAC (infrastructure as a code) scripts</h4><p>One of the common problems faced by organizations with huge infrastructure today is inadequate configuration managing for infrastructure, i.e not able to maintain the consistency of the services used. For example, an application has to be built and deployed in 2 environments: Test and Prod. Both environments have a different configuration as Tests need smaller capacity servers as compared to Prod. And application running in the test environment is deployed only in one geo-location. In the cloud, prod needs to be deployed in multiple regions for failure recovery.</p><h4>Lack of canary deployment, let alone automated one</h4><p>As a best practice, application code is first tested in pre-production environments. But even when continuously integrated, deploying code into production can still be manual, tedious, and error-prone. This leads to ever-larger differences between code to be deployed and code running in production, fueling a risky vicious cycle and could negatively affect site-reliability. The failure of code in production is a huge <strong>cost to the organization</strong>. And it is one of the biggest challenges to DevOps teams to prevent downtime in an application after every deployment. Because of huge interdependencies between multiple services, frequency of deployments, and with multiple underlying software, it is challenging to implement rollouts of application code in smaller versions.</p><h4>No Enforcement of Engineering Discipline</h4><p>Integrating the entire networking stack within apps/codes such as service discovery, routing, circuit breaking, load-balancing, functional testing, performance testing, and security authorization is a huge challenge with an application as an abstraction of the infrastructure layer.</p><h3>Our Framework to solve these challenges</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*XBeiJUTF2_vxsrrO_JGe9g.png" /></figure><h4>Modular &amp; Reusable Infrastructure as a Code</h4><p>Terraform <strong>as Infrastructure as a Code </strong>— A high-level configuration syntax is used for describing support. It allows a blueprint of datacenter to be versioned and treated as you would any other code. Also, infrastructure can be shared and re-used. Execution plans eliminate any surprises when Terraform manipulates infrastructure and change automation helps change and in what order can be known by the previously mentioned execution plan and resource graph, which helps in avoiding many possible human errors. By using Terraform, any N-tier DBT solution can be scaled easily by modifying a single count configuration value because the creation and provisioning of a resource are codified and automated, elastically with load becomes trivial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JYYBlNyFh8MMuKmOhx1Y_w.gif" /><figcaption>Terraform Code</figcaption></figure><h4>Externalized Configurations &amp; Consistency in deployment strategy across environments</h4><p>Configuration is externalized in AWS SSM (KV store) for seamless changes in infrastructure parameters like auto-scaling groups or instance types or ami ids.</p><p>Specification such as server capacity, geo-location, auto-scaling instance types can be managed as configurations here and can be modified as desired without any code changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n_zkUjXf3jCnUA2XaBh5KA.gif" /><figcaption>AWS SSM PS</figcaption></figure><p>Using Jenkins as the orchestrator:</p><p>• Picks the modules to execute</p><p>•Configurations are injected into Modules</p><p>•Modules apply configurations on tasks</p><p>•Tasks execute to accomplish use cases</p><p>Two different environment deployments (e.g. qa and uat) created by the same codebase and pipeline in the cloud account. Just add those values to the KV store and the pipeline will take care of deploying the tested software, maintaining consistency in deployment strategy across regions and environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A0hoiPEV-L_Q7Se04x-9XQ.png" /><figcaption>Multi Env Pipeline</figcaption></figure><h4>Cloud-Native</h4><p>Support for cloud native DBT solution. Using App Mesh, Istio and Kubernetes for the microservices which are built with enhanced resiliency in the cloud eco-system. Supporting operational analytics like monitoring or observability using the cloud specific services. All this with security in mind using IAM based controls and KV store with versioning and sensitivity control for key information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*UC6R4Cjdn6znMjYTvie1mQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*5bblbEbsooykK5TABlm-6g.png" /><figcaption>Cloud Specific architecture to deploy 2or more tier APIs</figcaption></figure><h4>Support for Canary Deployment</h4><p>This pipeline supports the canary deployment (or rollout) to introduce a new version of a service by first testing it using a small percentage of user traffic, and then if all goes well, increase, possibly gradually in increments, the percentage while simultaneously phasing out the old version. If anything goes wrong along the way, abort and rollback to the previous version.</p><p>The canary support is automated so that you can define the new service and update the KV store to define the percentage and use the pipeline to push the changes. Confirm canary option is added to allow for business teams to validate the canary results and push the changes to 100% accordingly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KbykThCtPCPCMFXJcpV3Xg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qoO8HekO9dfBxnHhkNscQ.gif" /><figcaption>Automated Canary Deployment</figcaption></figure><h4>Engineering Practices</h4><p>Service discovery, routing, circuit breaking, load-balancing are addressed with cloud-specific tools like AWS app mesh or Istio (in GCP). Being part of the deployment template, each service gets registered there and can be monitored and observed automatically once deployed. Logging follows the cloud architecture as it uses fluentd module to enable service logs to be captured by the AWS cloudwatch or cloud logging service in GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zv8ILbuOjmFByThETkNlBQ.png" /><figcaption>Observability with AWS Xray trace</figcaption></figure><p>Functional testing and performance testing are integrated as test harness and part of the pipeline. Newman (Postman command-line version) gets kicked off with the defined scripts (test cases written in Postman) to test at every level of the architecture to certify the software.<br>Security authorization follows the cloud-native principle of using IAM policies. Creation of the service accounts, roles, and privileges are created programmatically using IAC code. This ensures safety and makes the deployment less error-prone per environment.</p><h3>Framework in Action</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2PJcKir493IejhA5LlIqag.gif" /></figure><h3>Conclusion</h3><p>In a world where every company needs to think like a software company to be cutting-edge and innovative, the role of DevOps is immense. For successful DevOps and to meet the most important business objectives, CI/CD is a must. And CI/CD with the right attributes is even more critical to save labor cost, improve time-to-market, set engineering standards and quality. Our framework does just that and it alleviates IT leaders’ headaches for constant DBT solution failures and takes a step closer to achieve No-Ops.</p><h3>Contributors</h3><p><strong>Anupam Singh —</strong> Senior Associate, DevOps</p><p><strong>Amit Srivastava</strong> — Senior Software Development Engineer</p><p><strong>Amit Sharma</strong> — Director, Engineering</p><p><strong>Ravi Evani </strong>— Vice President, Engineering</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e97f0c90e5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/push-button-pipeline-for-digital-transformation-api-stack-with-an-automated-canary-release-using-7e97f0c90e5e">Push-button pipeline for digital transformation API stack with an automated Canary release (using…</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Native solution for Targeted/Context-aware Notifications in QSRs using Beacons and GPS]]></title>
            <link>https://medium.com/engineered-publicis-sapient/cloud-native-solution-for-targeted-context-aware-notifications-in-qsrs-using-beacons-and-gps-fb334cb796e2?source=rss-d886a1fe6982------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb334cb796e2</guid>
            <category><![CDATA[beacons]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Amit Sharma]]></dc:creator>
            <pubDate>Wed, 17 Mar 2021 04:51:40 GMT</pubDate>
            <atom:updated>2021-03-29T15:14:14.991Z</atom:updated>
            <content:encoded><![CDATA[<h3>Authors</h3><p><a href="/service/mailto:amit.sharma@publicissapient.com"><strong>Amit Sharma</strong> </a>— Director, Engineering</p><p><a href="/service/mailto:lourdhuraju.periyathambi@publicissapient.com"><strong>Lourdhuraju Periyathambi</strong></a> — Director, Engineering</p><h3><strong>Introduction</strong></h3><p>Location awareness is the heart of virtually all context-aware mobile apps. Although GPS technologies have enabled rough estimates of a person’s location, unfortunately, it does not provide the accuracy required for context-aware in indoor environments. This is especially true in large multi-level buildings such as restaurants, hotel and food hall environments. Therefore, it is highly likely that merging mobile marketing efforts with Beacon technology could result in higher click-through, quicker conversion, more sales and greater loyalty.</p><p>Integrating beacons with a mobile strategy will help level the playing field by responding to customers’ timely contextual content needs.</p><p>Imagine you are deciding where to have lunch. All of a sudden, your phone alerts you to an offer of two free sodas with the purchase of any entrée at a nearby restaurant. That kind of interaction might influence your decision to visit the restaurant.</p><p>This added value is critical as restaurants look to rise above their competition in a highly crowded space. Beacon technology can also be leveraged to capitalize on consumer habits.</p><p>In addition to this type of customer interaction, there are also numerous operational efficiencies that restaurants will experience when using Beacon technology.</p><p>One example is drive-thru.</p><p>For QSRs, drive-thru serves as a major way for customers to receive their orders. But what if customers order ahead and decide at the last minute that they would like to eat inside instead? Thanks to Beacon technology arranged at the parking lot entrance, the customer can be prompted upon arrival with the question of whether he or she plans to eat inside or pick up the order at the drive-thru, so the decision can be made on the spot. This allows the staff to act accordingly, change gears if needed and have the order ready the way the customer likes it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7o3rGdhOaHb1ytioeaBwA.jpeg" /></figure><h3>Challenges today</h3><h4>Data Collection</h4><p>The process of data collection and the ethics surrounding it are increasingly under fire, as evidenced by growing concerns over user privacy and the regulatory legislature aimed at protecting it. While the extensive push for data regulations creates challenges for marketers, it also provides an opportunity for businesses to build a foundation of trust with customers. By obtaining user consent and offering visibility into how location data is being used, marketers can maintain customer trust and deliver meaningful content in the path to purchase.</p><p>A lot of opt ins (customers need to install the app, always enable Bluetooth, agree to receive Push Notifications).</p><h4><strong>Accuracy/Poor User Experience on Beacon-enabled Apps</strong></h4><p>According to a recent survey conducted by Oho interactive with 35 of the top Beacon CMS and hardware providers, nearly half of the companies claimed poor user experience as one of their top concerns. These mainly fall under two categories:</p><p>1. Sending irrelevant notifications</p><p>2. Not providing for seamless experience</p><h4>Specifications Issue</h4><p>Lack of open standard, thus requiring businesses to purchase, install, and manage beacons that support each platform in addition to developing apps for both iOS and Android. Drives up the management and cost of the solution.</p><h4>Reliability/Scalability</h4><p>Data is as good as when captured and processed for the right use. IT solution exists to help with Beacons technology, but constantly struggle with the reliability and scalability of the data. On-prem solutions don’t scale with the needs and often fail to capture the data reliably to share with other systems. Poor data quality affects the efficiency and the timing of the notifications.</p><h3>The Framework to Solve the Challenges</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTFHjpm3e0KDS_Ci0UVU1A.png" /></figure><p>The following sections provide mechanics of the framework:</p><h4><strong>Cloud Native</strong></h4><p>Our solution framework brings Cloud-native to the forefront, which has become the new norm to help businesses operate.</p><p>1. Faster with greater flexibility</p><p>2. Offers elasticity, resilience, high availability and responsiveness</p><p>3. Cost-effective, as you can pay per use</p><p>4. Supports fastest Idea-to-Delivery time for Pilot Use Cases</p><h4>Streaming/Real-time Data</h4><p>Real-time is the key for the proximity-based notifications. Our framework uses serverless streaming services which capture data from beacons or GPS tracking at scale. Being serverless gives the ability to scale when needed without any management overhead.</p><h4>Relevant notifications</h4><p>As we capture the data real-time, the feedback loop to the relevant customer can go out at the same rate. With the use of right data processing, we apply personalization rules which make the notifications relevant according to their history, location, weather pattern, etc.</p><p>It’s important not to overwhelm users with notifications, as it could nudge them towards opting-out or un-installing an app. The key here is to have a deep understanding of the value offered to customers and deliver it in the best way possible. For example, allow customers to bookmark favorite dishes on the mobile app. This is so that when they walk by a restaurant any day, the restaurant can automatically push notifications with an enticing offer on the dish the customer bookmarked earlier.</p><h4><strong>Increase Reaction Time</strong></h4><p>Information is most valuable when it’s up-to-date, and in the hands of key decision-makers. The faster they’re able to respond to changes in data, the further ahead they’ll be. Proactive, smart alerts empower key decision-makers with the information most relevant to them and empower them to react immediately.</p><h4><strong>Open Source Specifications</strong></h4><p>Eddystone is an open source beacon format developed by Google and designed with transparency and robustness in mind to support Android and iOS devices (unlike iBeacons which is only compatible with iDevices). Eddystone is cross-platform and discoverable by any Bluetooth smart device.</p><p>Eddystone also supports the multi-beacon concept. This means that a single hardware beacon sends out multiple transmissions that can be used independently. The telemetry (metadata about how the beacon is operating that includes both battery and temperature) packet transmission, separate from the main identifier packet, is one example of this. Google made to combine multi-beacons into a single standard.</p><h3>Reference Architecture and Best Practices –</h3><p>Cloud-native architectures can be built with different cloud providers. AWS, GCP and Azure are leaders in this. We have captured the reference architecture for AWS and GCP below to help with the understanding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/64/1*g6R0qs-vkgzmu9H02AdIsA.png" /></figure><h3>With AWS Cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BHqUBqir8LgmX64yufjRlQ.png" /></figure><p>1. Real-time data will be captured from BLE/GPS notifications. AWS IOT Greengrass can broker the conversation with IOT sensors via the IOT rule to convert it to kinesis streams which can then be massaged with Kinesis firehose.</p><p>2. Real-time data is processed by applying windows and identifying data types to be fed to notifications system. Managed SNS service can help with this.</p><p>3. Real-time data will be used along with historical data to use AI models for personalization notifications.</p><p>4. Other AI services like text-to-speech, etc. can be used to make the messaging more engaging for customers.</p><p>5. Appsync along with AWS amoplify can help build mobile app which gives customers real-time alerts.</p><p>6. Data is also shared with Analytics engines like Amazon Redshift/Athena to do the BI. Dashboards can help with visibility to management for the success.</p><p>7. AWS CloudWatch helps with proactive monitoring with alerts, uptime checks and thresholds. This is keeping the system healthy and help automated remediations of the problems.</p><p>8. AWS IAM provides the security and access management of the data for compliance purposes.</p><p>9. Terraform and Jenkins are used to automate the deployments and management of the infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/122/1*xhCmmpKAGfxNSfb2ZxYakA.png" /></figure><h4>With Google Cloud Platform</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_zYH2DYoGc4ffk8a4VG-xA.png" /></figure><p>1. Real-time data will be captured from BLE/GPS notifications. Cloud IOT can broker the conversation with IOT sensors to convert it to streams which can be captured in Cloud pubsub and then be massaged with Cloud data flow.</p><p>2. Real-time data is processed by applying windows and identifying data types to be fed to notifications system. Cloud pub/sub is a perfect fit here.</p><p>3. Real-time data will be used along with historical data to use AI models for personalization notifications.</p><p>4. BigQuery ML/AI Platform can be used to feed off the data from data lake to build models for forecasting and accurate predictions. Out of the box recommendation algorithms can be used as well.</p><p>a. BigQuery ML supports SQL queries-based model on its storage or federated storage.</p><p>b. Auto ML can use the structured data for forecasting model without the data science knowledge.</p><p>5. Firebase can help build mobile app which gives customers real-time alerts.</p><p>6. Data is also passed on to data lake in Cloud storage or Cloud Bigtable for time series analysis and then processed further using ETL processes.</p><p>7. Data is also shared with Analytics engines like BigQuery to do the BI. Dashboards can help with visibility to management for the success.</p><p>8. Cloud Monitoring/Logging helps with proactive monitoring with alerts, uptime checks and thresholds. This is keeping the system healthy and help automated remediations of the problems.</p><p>9. Cloud IAM provides the security and access management of the data for compliance purposes.</p><p>10. Terraform and Jenkins are used to automate the deployments and management of the infrastructure.</p><h3>Conclusion</h3><p>Smartphones have made it possible for QSR brands to reach their customers more effectively and to provide them with a more engaging, more valuable brand experience — but only if they approach it the right way. Adding Context-aware technology like Beacons can be a cherry on the top.<br>Our framework helps brands that are still approaching this technology as a pure acquisition play with more meaningful or personalized messages to drive likeability rather than sending one-size-fits-all blast messages that drive a lot of customers to opt-out of messaging or uninstall apps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb334cb796e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/engineered-publicis-sapient/cloud-native-solution-for-targeted-context-aware-notifications-in-qsrs-using-beacons-and-gps-fb334cb796e2">Cloud Native solution for Targeted/Context-aware Notifications in QSRs using Beacons and GPS</a> was originally published in <a href="https://medium.com/engineered-publicis-sapient">Engineered @ Publicis Sapient</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>