<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Contino Engineering - Medium]]></title>
        <description><![CDATA[Opinions from Contino Engineering - Medium]]></description>
        <link>https://medium.com/contino-engineering?source=rss----20c2e566faed---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Contino Engineering - Medium</title>
            <link>https://medium.com/contino-engineering?source=rss----20c2e566faed---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 18:26:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/contino-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Decoding Google Cloud’s GenAI Services: Strengths and Complexities]]></title>
            <link>https://medium.com/contino-engineering/decoding-google-clouds-genai-services-strengths-and-complexities-3f96c9e8e8eb?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/3f96c9e8e8eb</guid>
            <category><![CDATA[dialogflow-cx]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[slack]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Zahid Saleem]]></dc:creator>
            <pubDate>Thu, 07 Dec 2023 09:57:04 GMT</pubDate>
            <atom:updated>2023-12-08T13:11:54.231Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jzao1Sy8OZVFLt_Dg-Iufg.jpeg" /></figure><h3>I was recently tasked to look and review some of the latest Google Cloud offering around AI and how easy it can be to create a Chat bot that can be used to generate some informative answers powered by the latest Gen AI services.</h3><h3><strong>TL;DR</strong></h3><p>Embarking on a week-long journey into the realm of Generative AI, I set out to immerse myself in Google’s offerings. Eager to share the fruits of my exploration, this blog is an emotional recount of my endeavours, extended as an offering to those seeking knowledge on GenAI or willing to exchange insights.</p><p>The narrative unfolds through the challenges I encountered while constructing an end-to-end document discovery solution utilising Google’s cloud Gen AI services. I delve into the personal motivations that led me to consider Google’s AI services and encourage other cloud enthusiasts to try.</p><p>Within the cocoon of Google’s AI platform, life is bliss, especially when setting up a Gen AI Chat bot straight out of the box — an experience that radiates simplicity. However, the tale takes a twist when venturing beyond Google’s ecosystem, attempting to integrate with enterprise tools like Slack to facilitate client engagement within my organisation.</p><p>This chronicle is a candid documentation of what didn’t unfold as planned, the anticipated outcomes that remained elusive, the eventual triumphs, and the intricate journey through the labyrinth of preview services to achieve seamless integration.</p><p><strong>As a developer, I share a tangible working example of the solution. As a learner, I provide references that I hope will serve as a place to start for fellow enthusiasts navigating a similar path. As a consultant I share my experience, offer assistance and add value. As a human I hope to still be relevant in the midst of the continually advancing field of AI.</strong></p><h3>Problem Statement &amp; Solution</h3><p>Guided by sophisticated algorithms, artificial intelligence has become a necessary requirement in todays age.</p><p>Empowered by personalised and tailored experiences elevate how AI serves individuals, enhancing engagement and satisfaction.</p><p>Next Generation AI technologies require innovative solutions to existing problems of today.</p><p>A typical organisation spends a considerable amount of time and effort collecting, accumulating, generating documents, presentations, white papers, and ebooks over time. While the initial intention is often to revisit and share these resources centrally with colleagues and the sales team, the reality is that not many individuals have the time to sift through hundreds of documents to find the specific sources needed to deliver value. Moreover, navigating through multiple document repositories to identify the most valuable content can be a time-consuming task.</p><p><strong>Introducing the solution to streamline document discovery challenges; your friendly neighbourhood Slack Gen AI powered Virtual Assistant. This bot not only enhances question responses with intelligence, but also guides users directly to the most relevant documents, providing a quicker and more targeted natural experience, saving time, effort and resources.</strong></p><h3>Solution Architecture</h3><p>Lets begin with a diagram that illustrates how the end to end solution was built and how relatively straightforward at a high level Google cloud services fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5tfZCOWFpqvHhc07_YbIw.png" /></figure><p>Rather than delving into technical intricacies of the solution, I will provide an overview of the strengths, and challenging aspects of the solution. This approach aims to offer a comprehensive understanding of my learning experiences in creating &amp; configuring the Virtual Agent and offer suggestions on potential improvements / enhancements.</p><h3>Day 0 — Learning</h3><p>First things first — my experience on Google Cloud Gen AI services was limited, however I am an avid user of Google Duet AI Assistant. For those unfamiliar with Duet AI its like Chat GPT specifically trained on Google cloud content, code generation. I highly recommended you go check it out at <a href="https://cloud.google.com/duet-ai/docs/discover/set-up-duet-ai">https://cloud.google.com/duet-ai/docs/discover/set-up-duet-ai</a></p><p>I am a cloud solutions enthusiast working mostly on Azure Cloud, have recently started to branch out to Google Cloud. I was intrigued by the problem statement and wanted to try Generation AI services with a practical demo. Here are a few links and learning resources I found useful when beginning my foray into GCP and GenAI.</p><p>Free Courses from Google — Great starter on Vertex AI.</p><ol><li><a href="https://www.cloudskillsboost.google/course_templates/872">Conversational AI on Vertex AI and Dialogflow CX</a></li><li><a href="https://www.cloudskillsboost.google/course_templates/872">Create Conversational AI Agents with Dialogflow CX</a></li></ol><p>Free Google Labs and Vidoes— great hands on learning.</p><ol><li><a href="https://www.cloudskillsboost.google/focuses/22797?catalog_rank=%7B%22rank%22%3A1%2C%22num_filters%22%3A0%2C%22has_search%22%3Atrue%7D&amp;parent=catalog&amp;search_id=28780405">Create Conversational AI Agents with Dialogflow CX: Challenge Lab</a></li><li><a href="https://www.youtube.com/watch?v=HK__EJHMSMU">Youtube — Build Conversational AI experiences powered by LLMs</a></li></ol><p>The majority of the links I provide are sourced directly from Google’s own documentation and learning hubs. I highly encourage newcomers to the Google Cloud ecosystem to engage with their labs and courses. These resources are exceptionally well-curated, offering a valuable learning experience that is consistently updated with the most current and cutting-edge information</p><h3><strong>Day 1 — Implementation</strong></h3><h3>1. Setup Slack Application</h3><p>Initiating a new Slack Application is a relatively straightforward process. Follow the provided links for clear guidance on establishing the Slack App <a href="https://cloud.google.com/dialogflow/cx/docs/concept/integration/slack">here</a>. I highly recommend setup your own free organisation on Slack to test the Integration on, before moving over to Enterprise Slack.</p><h3>2. Search &amp; Conversation AI</h3><p>This part unveils the true potency of Google’s GenAI Services. Creating the Chat Agent and configuring Data stores prove remarkably effortless, with default settings offering highly valuable features straight out of the box. The Agent creation process is exceptionally user-friendly. Click <a href="https://cloud.google.com/generative-ai-app-builder/docs/introduction">here</a> for more information on Seach and Conversation Vertex AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*-18WzzqJ1EjYDefJeyVzFw.png" /></figure><p>Setting up and configuring the above steps may take approximately 10 minutes. The Agent is automatically generated with LLM (Large Language Model), GenAI, Default Intents, Flows, Pages and access to a Dashboard to manage the Agent. Fortunately, the default Gen AI settings eliminate the need for any additional adjustments. ML settings can be re-viewed in Agent settings section, and testing the Agent is simplified through the built-in Test Agent Simulator. Here is a great code lab on setting up a Data Store agent <a href="https://codelabs.developers.google.com/codelabs/vertex-ai-conversation#0">https://codelabs.developers.google.com/codelabs/vertex-ai-conversation#0</a></p><p>Lets take it a step further and use Googles Dialogflow Messenger Client for a quick and efficient end-to-end solution. This client is particularly useful in scenarios where Slack, Google Chat, or Teams integration may not be readily available. Dialogflow Messenger is a ready-to-use client that can seamlessly integrate into any website, ensuring it’s prepared to deliver value right from Day 1.</p><p>From the Dialogflow CX UI of the Agent click on <strong>&lt;&gt; Publish</strong>. And you get shown how to embed the Dialogflow Messenger App. <strong>Try it now</strong> and test the Chat agent inside your browser tab.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBerbqhj2kijOitRLhCGRg.png" /></figure><p>As a comprehensive solution, this client integration is functional, and relying solely on this client is a viable option. However, in practice, the use of enterprise collaboration tools such as Slack, Google Chat, and Microsoft Teams is more prevalent. This seamlessly brings us to the topic of integration with Slack.</p><h3>Days 2, 3, 4 &amp; 5 : Integration, Integration, Integration</h3><p>I had anticipated that integrating with Slack would be as smooth as the first day. Yet, what initially seemed like a straightforward integration unexpectedly transformed into a challenging endeavour. To be honest, I found this challenge quite enjoyable.</p><p>The integration process constitutes a critical aspect of the entire experience, and it is essential to marry the agent with Slack responses seamlessly. While my exposure to Dialogflow CX is somewhat short, I aim to share insights gained from this experience. This blog serves as a resource to enlighten others on the experience or help provide recommendations for potential improvements in Dialogflow CX integration processes.</p><h3>3. Dialogflow CX Agent Integration</h3><p><strong>Inbuilt Slack Integration:</strong> Utilising the inbuilt Slack integration involves setting up the Slack App through guidelines provided by Dialogflow CX dashboard. Click on the Integrations. Find Slack and click on connect. Use the link here <a href="https://cloud.google.com/dialogflow/cx/docs/concept/integration/slack">https://cloud.google.com/dialogflow/cx/docs/concept/integration/slack</a> to view documentation on how to setup Slack App. We are reminded that the Slack Integration is in Preview.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DCCDoPQz-HaPuxwOtR2kYA.png" /></figure><p>While this operation is relatively straightforward, complexities were encountered after implementation. Specifically, only text responses were being received, lacking any rich content, and predominantly not featuring links to documents. Here is an example of a somewhat meaningless response, on Slack, as no links were shown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pf3focd-0gWELeUNwNeUGw.png" /><figcaption>Slack Integration test</figcaption></figure><p>I also conducted a quick Integration test with Google Chat client — and encountered the same issues of link not showing.</p><blockquote><strong>I am certain that fixing the integration for Slack and Google Chat to output rich content would eliminate additional effort elsewhere.</strong></blockquote><p>From the Slack integration page towards the end there is a section on Rich Response messages. This led me start investigating Custom Payloads.</p><p><strong>Custom Payloads:</strong> In the Dialogflow CX UI you have the ability to modify Agent responses for Intents and incorporate Custom payloads. However, Custom Payloads appeared to be restricted to semi static responses, whereas the Gen AI response payload included dynamic list of citations (links) that <strong>could not</strong> be iterated over for display.</p><p>To include a Custom Payloads you edit the Data Stores (naming did confuse me a little) in the Fulfilment Section <a href="https://cloud.google.com/dialogflow/cx/docs/concept/fulfillment">click here</a> for more information. Adding a custom payload will provide a text box to add custom JSON payload in the format for Slack demonstrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c7m6-MkKsh7TiLKr9kY6cw.png" /></figure><pre>{<br> &quot;text&quot;: &quot;$request.knowledge.answers[0]&quot;,<br> &quot;blocks&quot;: [<br>  {<br>   &quot;type&quot;: &quot;section&quot;,<br>   &quot;text&quot;: {<br>    &quot;type&quot;: &quot;mrkdwn&quot;,<br>    &quot;text&quot;: &quot;*Title:*\n$richContent.citations.citation[0].title&quot;<br>   }<br>  },<br>  {<br>   &quot;type&quot;: &quot;section&quot;,<br>   &quot;text&quot;: {<br>    &quot;type&quot;: &quot;mrkdwn&quot;,<br>    &quot;text&quot;: &quot;*link:*\n&lt;$richContent.citations.citation[0].actionLink&gt;&quot;<br>   }<br>  }<br> ]<br>}</pre><p>I used the above JSON as the Custom Payload (made up from what made sense via the test Simulator). Save the settings and tested again (always remember to save) and the output was:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UOjyyhJBsifwVTqPLudcLA.png" /><figcaption>custom payload test</figcaption></figure><p>I experienced some success as I began to showcase more than just the answer. While this additional display might not have significantly aided me due to the dynamic nature of the citations, it did demonstrate the possibility. Surprisingly, Custom Payload templates proved unable to retrieve extracts such as citations from <strong>richContent</strong> in the payload. The answer can be accessed from <strong>$request.knowledge.answers[0]</strong>. I referred to the link below on how to handle rich responses, but unfortunately, it was of no avail. Time for more reading</p><p><a href="https://cloud.google.com/dialogflow/cx/docs/concept/fulfillment">https://cloud.google.com/dialogflow/cx/docs/concept/fulfillment</a></p><p>On the documentation page above there was a section referencing Webhooks. Cue more reading and learning.</p><p><strong>Webhooks:</strong> Webhooks serve as a mechanism for hosting your business logic or making calls to other services <a href="https://cloud.google.com/dialogflow/cx/docs/concept/webhook">see here</a> for further details. For instance, a Cloud Function can be set up to manipulate the payload and generate a dynamic list of links. While this concept is practically advantageous, challenges emerged in formatting the response message to meet Dialogflow CX fulfilment payload requirements. After delving into the content provided in the following links and consulting some Google Partner Advantage documents, I was well-prepared to commence development and testing.</p><ol><li><a href="https://cloud.google.com/run/docs/triggering/webhooks">https://cloud.google.com/run/docs/triggering/webhooks</a></li><li><a href="https://cloud.google.com/dialogflow/cx/docs/reference/rpc/google.cloud.dialogflow.cx.v3beta1#webhookresponse">https://cloud.google.com/dialogflow/cx/docs/reference/rpc/google.cloud.dialogflow.cx.v3beta1#webhookresponse</a></li></ol><p>I had a simple python Webhook ready to be deployed and tested. Note the message of the JSON response is surrounded by fulfillmentResponse {} indicating the payload is adhering to a schema.</p><pre>import functions_framework<br>import json<br>@functions_framework.http<br>def webhook_http(request):<br>    <br>    return json.dumps(<br>        {<br>            &quot;fulfillmentResponse&quot;: {<br>                &quot;mergeBehavior&quot;: &quot;MERGE&quot;,<br>                &quot;messages&quot;: [{<br>                    &quot;text&quot;: {<br>                        &quot;text&quot;: [&quot;Message to be extracted from request&quot;]<br>                    }<br>                }]<br>            }        <br>        }<br>    )</pre><p>To setup the webhook — I created a Cloud Function in Google Cloud. with the following settings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*aOdCns8YFSvnAUsnLIjZhA.png" /></figure><p>From the next Screen choose the runtime as <strong>Python 3.10</strong> and copy the code above in the source area and click deploy. <a href="https://cloud.google.com/functions/docs/concepts/overview">Click here</a> for more details on Cloud Functions.</p><p>Add the webhook to the Agent go to dialogflow CX UI and go to <strong>Manage -&gt; Webhooks</strong> and click <strong>+ Create</strong>. Provide the name and Cloud Function webhook trigger URL and click <strong>save</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-ko0dvoo9MoF4-uTnjCNw.png" /></figure><p>Add the Webhook via Fulfilment section for the Data stores and go to the Webhook section (see below). Enable webhooks (turn on) and select the webhook name from the drop down list created earlier. Add Tag value to be included in the response e.g. “chatbot”. Click Save (always remember to save)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQth3b-lv_O1OlYWnup6cw.png" /></figure><p>Test the Agent and webhook by using the test simulator to review the response. The hardcoded message “Message to be extracted from request” is shown along with the payload. To access the complete message response, click on the small clipboard icon highlighted in red (below). We are confident the webhook is responding and we can start to incorporate dynamic elements to the payload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ZX7qae64BtVh9k5Grj9eg.png" /></figure><p>Next step was to add the Slack message blocks to the webhook to get simple links to be shown in Slack. I used examples and links from various sources in order to format the response e.g. <a href="https://api.slack.com/methods/chat.postMessage">Slacks API Message reference and tester</a> and Googles own documentation on the <a href="https://cloud.google.com/dialogflow/cx/docs/reference/rpc/google.cloud.dialogflow.cx.v3beta1#google.cloud.dialogflow.cx.v3beta1.Fulfillment">fulfilment message API</a></p><p>Rather than helping the conflicting information from the documentation pages had the opposite effect (or I was just fatigued). Despite numerous attempts and crafting a specific response body, parsing errors and empty responses, persisted in the Agents <strong>webhookResponse</strong>. Due to unfamiliarity with the schema and building a valid Slack response I kept getting— only text responses and no links.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/194/1*IDpPIAIHbJOSfVvAb0_S_w.jpeg" /></figure><p>With more time and a better understanding of the fulfilment schema, Webhooks offer the best option for dynamic message responses. More complex examples of payloads for different Clients would address many of the issues I had experienced.</p><blockquote><strong>Finally if you want a job done right you do it yourself. Enter Custom Slack Integration.</strong></blockquote><h3>4. CloudRun Custom Slack Integration</h3><p>After exhausting Webhooks integration, I found a Slack NodeJS Dialogflow CX Integration project on Github <a href="https://github.com/GoogleCloudPlatform/dialogflow-integrations/blob/master/cx/slack/README.md">here</a>. The solution provides an API intermediary between Slack and Dialogflow CX Agent. The installation and deployment was again relatively straightforward based on the instructions provided. The <strong>Github</strong> repo also provides manifest for Slack App, either edit the manifest here and replace it in Slack App configuration or edit the Manifest that already exists with the URL for the <strong>CloudRun</strong> NodeJS URL.</p><p>Ensure to provide the following details in the <strong>server.js</strong> and uncomment the lines.</p><pre>// const slackSigningSecret = &#39;...&#39;;<br>// const slackToken = &#39;xoxb-...&#39;;<br>// const projectId = &#39;my-project&#39;;<br>// const locationId = &#39;global&#39;;<br>// const agentId = &#39;my-agent&#39;;<br>// const languageCode = &#39;en&#39;;</pre><p>The Agent Id is the UUID string that is available from the list of Agents — clicking on the 3 dots and Copy name (see below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xh7q6BYC1XVNRyFEIYIDA.png" /></figure><p>Please copy the entire string and utilise only the concluding segment of the identifier in the <strong>agentId</strong> parameter above. Proceed to update the Slack App in order to get the Server URL to be added to the Slack App manifest. Execute the code deployment following the provided instructions. As outlined in <a href="https://github.com/GoogleCloudPlatform/dialogflow-integrations/blob/master/README.md">https://github.com/GoogleCloudPlatform/dialogflow-integrations/blob/master/README.md</a>. <strong>It is mandatory to configure a service principal and allocate the appropriate role as per the guide</strong>. During testing, the Agent’s response remained consistent: <strong>text-only with no embedded links</strong>.</p><h3>From Zero to Hero: Lets hack the Slack</h3><p>We’re on the verge of making it all work. I begin tweaking the main function, honing in on that pivotal moment when Slack responses seamlessly align with our expectations. Crafting a response demanded a couple of intense hours. The journey unfolds as I dive into the intricacies, breaking down the steps taken. I suggest downloading the Git repository <a href="https://github.com/GoogleCloudPlatform/dialogflow-integrations">dialogflow-integrations</a>, and set up a development environment.</p><p>1) Begin by editing the file <a href="https://github.com/GoogleCloudPlatform/dialogflow-integrations/blob/master/cx/slack/server.js"><strong>server.js</strong></a><strong> </strong>to add some debug for the what the response payload looks like.</p><p>2) Within the<strong> convertToSlackMessage</strong> function, output the response payload to the console using the <strong>JSON.stringify()</strong> function. e.g.</p><pre>async function convertToSlackMessage(responses, channel_id) {<br>  let replies = [];<br>  console.log(&quot;Message:&quot;+JSON.stringify(responses.queryResult.responseMessages))</pre><p>3) Deploy the code and test with a simple request from Slack. Examine the <strong>Cloud Run</strong> app’s output, via logs, to observe how the JSON was constructed and formatted. Copy the output and place the output into a file.</p><p>4) Convert the JSON String from the output step above to JavaScript. I used the wonderful <strong>DUET AI </strong>and asked it convert the JSON file into JavaScript.</p><p>5) Create / Edit a Visual Studio Code project, insert the converted JSON into a file named <strong>server_test.js</strong>, and change the logic from <strong>server.js</strong> to create custom code for parsing and extracting the <strong>richContent</strong> from the response. This streamlined the testing process, eliminating the need for deployment during local testing.</p><p>6) After successfully testing and extracting citations from the response. Use the Slack’s guidelines page <a href="https://api.slack.com/messaging/composing/layouts">here</a> for completing the layout and format of the response. See the code block below for the complete function I changed to get it to work.</p><pre>async function convertToSlackMessage(responses, channel_id) {<br>  let replies = [];<br>  console.log(JSON.stringify(responses.queryResult.responseMessages))<br>  for (let response of responses.queryResult.responseMessages) {<br>    let reply;<br>    let payload;<br>    let blocks = [];<br>    switch (true) {<br><br>      case response.hasOwnProperty(&#39;text&#39;): {<br>        reply = { channel: channel_id, text: response.text.text.join() };<br>      } break;<br><br>      case response.hasOwnProperty(&#39;payload&#39;): {<br>        payload = await structProtoToJson(response.payload);<br>        let content = payload.richContent;<br>        if (payload.hasOwnProperty(&#39;richContent&#39;)) {<br>          if (Array.isArray(content[0])) {<br>            let citations = content[0][0].citations<br><br>            for (let cit in citations) {<br>              let section = {<br>                &#39;type&#39;: &quot;section&quot;,<br>                &#39;fields&#39;:<br>                  [<br>                    {<br>                      &#39;type&#39;: &quot;mrkdwn&quot;,<br>                      &#39;text&#39;: &quot;*Title:*\n &quot; + citations[cit].title + &quot;\n*SubTitle:*\n&quot; + citations[cit].subtitle<br>                    },<br>                    {<br>                      &#39;type&#39;: &quot;mrkdwn&quot;,<br>                      &#39;text&#39;: &quot;*link:*\n &lt;&quot; + citations[cit].actionLink + &quot;&gt;&quot;<br>                    }<br>                  ]<br>              }<br>              blocks.push(section);<br>            }<br>          }<br>        }<br>      } break;<br>      default:<br>    }<br>    if (reply) {<br>      replies.push(reply);<br>    }<br>    if (payload) {<br>      payload[&#39;channel&#39;] = channel_id;<br>      payload[&#39;blocks&#39;] = blocks<br>      replies.push(payload);<br>    }<br>  }<br>  return replies;<br>}</pre><p>7) Redeploy the Cloud Run App and conduct further testing. This iterative process continued until the correct response was achieved.</p><p>8) Once message blocks were appropriately configured, the integration worked and started to display links in the final response sent to the Slack client.</p><p>What does the above look like — here is the response from slack</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zn5FVqbj1er4twowsdxDPQ.png" /></figure><p><strong>Success…. We have the citations now being displayed with the title, actionLink and subTitle all being shown. Confidence restored, time to demo. Funny meme on my experiences incoming...</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpjvQFtlNQ-M%3Fstart%3D8%26feature%3Doembed%26start%3D8&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpjvQFtlNQ-M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpjvQFtlNQ-M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cb7afcd67485ca71fc7e551ef545f8c/href">https://medium.com/media/5cb7afcd67485ca71fc7e551ef545f8c/href</a></iframe><p>The solution is not yet perfected and has a few issues that still need to be addressed here is a short snag list</p><ul><li>Changing the number of links to &lt; 2 in the data store fulfilment section affects the payload and links are not displayed. Needs further investigation</li><li>Truncate the Strings e.g. Subtitles for cleaner view.</li><li>The interaction / response is slower (a few seconds) via this option.</li><li>The Cloud Run NodeJS server requires better security e.g. removing hardcoded secrets and Ids.</li><li>The proto json functions should be setup to parse and format the JSON responses better.</li></ul><h3>Summary</h3><p>A little longer blog post than I would have anticipated but wanted to ensure I documented my experience in all its glory and hardships. We went from a truly exceptional Day 1 experience with the remainder of time left fighting Integrations with Slack. I did learn a significant amount in the process.</p><p>In summary, the process of integration the Slack Bot to Dialogflow CX involved exploring different methods, each with its complexities and considerations. I am in awe of the Google Suite of Services around Gen AI — and simultaneously critical of the time it took me to get the Integration complete. AI is here to stay — we have a need to embrace AI and use it responsibly and ethically.</p><blockquote><strong>“Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.”</strong></blockquote><blockquote><strong>– Stephen Hawking</strong></blockquote><h3><strong>My passing thoughts</strong></h3><p><strong>For Users 🧑‍💼 :</strong></p><ul><li>Gen AI Services in Google Cloud —<strong> super easy to setup.</strong></li><li>What are you waiting for ? — Build Services and <strong>leverage best in class AI</strong> offerings. Its a no brainer.</li><li><strong>Integrations</strong> are tricky — but with <strong>perseverance</strong> it can be <strong>achieved</strong>. (I still believe <strong>Webhooks </strong>are the best way to achieve this).</li><li><strong>Get involved</strong> with the community and help support additional Integrations for other popular clients. I will be working on Teams integration next.</li></ul><p><strong>For Builders 🛠️ (GCP) :</strong></p><ul><li>Keep up the awesome work on making Gen AI accessible and easy to enable for customers. Great content and labs available at <a href="https://www.cloudskillsboost.google/">https://www.cloudskillsboost.google/</a></li><li>I recommend providing more concrete examples of different payloads and responses in documentation for specific client Integrations.</li><li>Fix projects &amp; Integrations that only support text and look to make rich content responses more accessible.</li><li>Custom Payloads Templates — looks like could be a powerful tool with elements of simple logic that can transform responses e.g. loops, conditionals and access to the payload response objects.</li></ul><p><strong>For Enterprises 🏢 :</strong></p><ul><li>If you exploring Gen AI services and how it can help you. Review Googles suite of Cloud AI focussed services and offerings such as <a href="https://cloud.google.com/generative-ai-app-builder/docs/introduction">Search &amp; Conversations (Vertex AI)</a>.</li><li>Lead in the realm of emerging tech, with more practical applications with Multi Modal AI led Virtual Assistants (covering images, speech, audio, code etc.)</li><li>Does the use case / problem statement sound familiar? Get in touch on the consulting service team, email below, and and learn about how to get started with Gen AI.</li><li>A LLM-powered chat interface connected to internal enterprise documents is such a valuable and general use case — why not give this a go?</li></ul><h3>Whats next ?</h3><p>Have an idea you’d like me to explore? Curious about a specific aspect of Generation AI? Want to share your own experiences or insights? I’m all ears. I aim to write more blogs on my user journey, the good, the bad and the not so pretty aspects of GenAI with practical use cases and problems.</p><h3>Contact Me:</h3><p>Feel free to reach out via email. I’m excited to connect with fellow enthusiasts, experts, and anyone passionate about the potential of Generation AI.</p><p>You can find me on LinkedIn <a href="http://www.linkedin.com/in/zahid-saleem-08446a4">here</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f96c9e8e8eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/decoding-google-clouds-genai-services-strengths-and-complexities-3f96c9e8e8eb">Decoding Google Cloud’s GenAI Services: Strengths and Complexities</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s that Baking? It’s VM Images!]]></title>
            <link>https://medium.com/contino-engineering/whats-that-baking-it-s-vm-images-5a13c1a8670b?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/5a13c1a8670b</guid>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[vm]]></category>
            <category><![CDATA[image-bakery]]></category>
            <category><![CDATA[golden-image]]></category>
            <category><![CDATA[packers]]></category>
            <dc:creator><![CDATA[Mark Nash]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 08:41:10 GMT</pubDate>
            <atom:updated>2023-10-09T08:41:10.540Z</atom:updated>
            <content:encoded><![CDATA[<h4>Backstory</h4><p>You stand up and stretch, you’ve been building and configuring Virtual Machines (VMs) all morning and the compliance dashboard is still not happy. There are still more systems that need to be updated and configured with the new security standard. But it’s lunch time, so you decide to take a walk down to your local bakery to pick up something to eat.</p><p>As you’re waiting in the queue you look at the array of baked goods prepared in batches, baked, and now all available to be picked up on demand by customers. It would be very painful if each cake had to be created from scratch every time it was ordered!</p><p>You think back to the task at hand — it would make your life so much easier if you could create the VMs in a similar way. An automated process that would use Infrastructure as Code (IaC) and Configuration as Code (CaC) to create VM and Container images via a standardised and repeatable process, configuring and testing the VMs so that they are ready for use?</p><p>Your mind jumps back to a Medium article you skimmed last week…</p><p>“An image bakery is a codified, automated and repeatable process where you create custom images used for creating VMs, Containers or Disks. They can help speed up the provisioning time for a VM by baking the required configuration of the systems into the VM image rather than configuring it after the instance is built.”</p><p>You stroll back to the office and decide to do some more investigations into Image Bakeries…</p><h3>Introduction</h3><p>Many Organisations and Platform teams find themselves in similar situations to the above scenario, Ageing VM infrastructure that is in need of updates, configuration that is not being updated in a standardised way across the estate and new compliance requirements that need to be applied across all resources.</p><p>By putting an image bakery solution in place platform teams can create company specific base images that align to company compliance and security policies via a repeatable process that allows the images to be updated easily. This then enables application teams to build upon these base images installing required applications to create their required resources.</p><p>In this blog we will explore the idea of Image Bakeries, discussing why you might build one into your cloud estate. We’ll look at how an image bakery solution can help you achieve compliance, security and goals whilst also speeding up provisioning of resources.</p><p>Finally we will look at some of the issues I’ve personally hit whilst creating an Azure Image Bakery, in the hopes that you won’t get burnt by your experience in future!</p><h3>Pat-a-cake, pat-a-cake, Technology man, Bake me an Image, as fast as you can!</h3><p>Let’s start by breaking down some of these terms that I’m using.</p><h4>What is an Image Bakery?</h4><p>The process of creating a custom image is called baking and so the process that creates multiple custom images tends to be called a bakery.</p><p>An image bakery is a codified, automated and repeatable process where you create templated/custom images used for creating VMs, Containers or Disks.</p><p>Image bakeries tend to be associated with the practice of immutable infrastructure, whereby you want your VMs/Containers to wholly contain all the applications and configuration that they need to be successful, not needing any additional changes following deployment. When changes to applications or updates are needed a new image and instance are created instead and older VMs/Containers are replaced.</p><p>By baking images you are altering the creation dependency chain for getting VMs and Containers stood up, moving the configuration to earlier in the process and therefore reducing required changes following resource creation. By having the images available earlier you allow time to do thorough testing, providing confidence in the resources being deployed using the company specific images.</p><p>This means having an image bakery for creating these images via a standardised process can be an exciting prospect, reducing overall deployment time and helping to keep resources inline with the organisation’s policies.</p><h4>The Baking Process</h4><p>The process of a bakery tends to follow a similar pattern:</p><ol><li>A base image is selected</li><li>A VM or container instance of the selected base image is created</li><li>The instance is configured to the required level</li><li>The instance is validated against the configuration</li><li>The Instance state is saved and turned into an image</li><li>The image is then made available to be used on VMs or containers in future</li><li>Clean up resources used for process</li></ol><p>Configuring the instance can consist of a few different tasks:</p><ul><li>Updating the Operating system</li><li>Installing required Software / Applications</li><li>Locking down system settings</li><li>Creating required Users / files</li><li>Testing to ensure applications are functioning and that files are where expected</li></ul><p>Once an image is created it can either be utilised to create further images, creating an inheritance structure of image dependencies, or it can be used to deploy a resource for its intended use.</p><h4>Base and Child Images</h4><p>Images inherit characteristics from their parent images, most obviously the core operating system you are building on top of, but this can be expanded to contain security configuration, applications and anything the parent image is configured with.</p><p>By stacking images into an inheritance structure you are able to make use of this inheritance capability and standardise the base configurations across all of your custom images. You can then create child images when requirements between images differ.</p><p>In the following example image we have a small structure of VM/Container images, the visual shows how platform images get pulled and utilised to create base and then child images. These platform images are configured using example configuration files to create the custom (base and child) images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSaLYgyB645ok0en1vq3ew.jpeg" /></figure><p>As in the example image, we can utilise platform images to create our standardised base image, inherently baking in the company’s base security requirements and settings into the image.</p><p>These base images can then be used to create Domain Controllers / Database servers / Web App server images which can either be used to create servers, or again these images can be built upon creating application specific images in the third layer coming pre-installed with the latest application to use in scale-set or similar resources.</p><p>Because the images are created within an inheritance structure they gain the settings and applications of their parents, this helps to ensure all Containers and VMs created by the Application teams abide by company policies.</p><h4>When things go Stale</h4><p>Images will get old and stale just like cakes sat in our real bakery, and so we need to put policies and processes in place to keep our images fresh — up to date with OS updates and application updates, and we’ll need to remove older images when they should no longer be used.</p><p>By setting up an Image bakery, instead of baking images one by one, we are putting the required automation in place to help us with the creation of these images, and by combining this with lifecycle policies and versioning we get a neat solution for empowering users to select only the newest and freshest images for deployment into the organisations platform.</p><p>Now that we’ve discussed how a Bakery solution can help you ensure you are using the newest images, that they follow organisational policies and that they are tested and proven to work, let’s discuss what tools you need to start looking into.</p><h3>We don’t need Cake Tins and Whisks where we’re going</h3><p>Baking images is obviously different to baking cakes, but we still need a set of tools that are going to help us automate and manage the image creation process.</p><p>We’ll be looking for tools that cover:</p><ul><li>Creating the initial image instance to configure and turn into the image</li><li>Configuration tools to actually install the configuration on the instance</li><li>Pipeline and scripting tools to create the overall Bakery process</li></ul><p>Now let’s browse the utensil shelves and see what we find…</p><h4>Image Creation tool</h4><p>Hashicorp’s <a href="https://www.packer.io/">Packer </a>has been a leader in the image creation tooling field for many years. Packer lets you create identical machine images for multiple platforms from a single source configuration. A common use case is creating golden images for organisations to use in cloud infrastructure.</p><p>Coming from Hashicorp, the same company that has provided industry with tools such as <a href="https://www.terraform.io/">Terraform</a>, <a href="https://www.vaultproject.io/">Vault</a> and <a href="https://www.nomadproject.io/">Nomad</a>, It is understandable that <a href="https://www.packer.io/">Packer</a> has gained quite a following and has become a leader in this field.</p><p><a href="https://www.packer.io/">Packer</a> uses a fairly simple setup of configuration files to define the source image, image destination, build instance settings and required system configuration. Packer then uses these configuration files to create all the required infrastructure resources to host an image in the specified cloud provider, creating a temporary image to configure and turn into a nicely baked image (golden some might say..)</p><p>All the main cloud providers (<a href="https://aws.amazon.com/image-builder/">AWS</a>, <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/image-builder-overview?tabs=azure-powershell">Azure</a>, <a href="https://cloud.google.com/compute/docs/images/create-custom">GCP</a>) have their own cloud specific image building tools and processes to help you create personalised images for their cloud platform. The cloud specific tools have the benefit of being able to maximise features aligned to the hosting providers, Whereas Packer is cloud agnostic and can be used across all three.</p><p>For me being cloud agnostic outweighs the small benefits the specialisms get you but of course the choice on image builder will be down to your own organisation’s requirements. In the meantime, Packer is the tool we’ll be using and talking about for the rest of the blog.</p><p>Now we have our tool for baking the solution in a manageable way, but we need to get our configuration in the image otherwise what’s the point in creating our own image at all?</p><h4>Gently fold your configuration into the mixture</h4><p>There are many tools out there for managing the configuration of IT resources, and so we won’t focus too much on which you should choose. The choice of configuration tool will be down to organisational requirements and choices, but for my scenario we are trying to keep with the agnostic and low cost approach and so we’ve picked <a href="https://www.ansible.com/">Ansible</a>.</p><p><a href="https://www.ansible.com/overview/it-automation"><em>“Ansible — the simple, yet powerful IT automation engine that thousands of companies are using to drive complexity out of their environments and accelerate DevOps initiatives.”</em></a></p><p><a href="https://www.ansible.com/">Ansible</a> works on the idea of configuration ‘playbooks’ and hosts that they are deployed to. Each ‘playbook’ consists of a collection of ‘plays’ and each ‘play’ contains variables, roles and an ordered list of tasks. The machine running the packer command will be considered as the control node and will configure the instance created by Packer.</p><p>In a more complex Ansible controlled environment you may see Inventory files containing multiple hosts, but as we’re only building a single image at a time we don’t need to be too concerned with these. Just know that the host created by packer will be contained in an inventory file and configured using the ansible playbook provided.</p><p>So in our bakery we want these Playbooks to consist of all the tasks needed to get that image configured to the agreed level, So installing applications, adjusting registry settings, and creating files. We can even include testing within our ansible playbook so that we can be sure that everything is in the right place and that our image is ready for use before we package it back up.</p><p>We call our configuration tool (Ansible) or any configuration scripts from within the Packer configuration files via a provisioner, so in this case we trigger an Ansible playbook, and get it to configure our image, installing all necessary applications and settings and then testing to ensure all is well.</p><p>Once the instance is configured Packer converts the instance into an image and deletes the remaining resources.</p><p>There you have it, our first Golden Image all ready for use! However, this process has been very hands-on up to this point. Even though Packer has automated the creation of our image and Ansible has managed the configuration of that image, we’ve still only managed a single image. So let’s get this whole process orchestrated in a pipeline so that we can kick back and let the system handle the creation of images itself.</p><h4>Now let’s make a baker’s dozen!</h4><p>Most Organisations will already have chosen which pipeline tool and repository tools that they are aligning to as an organisation and a lot of them are tied together these days hence lumping them into the same pan. It’d be foolish of me to assume my mere article is going to change those strategic decisions, so again I’ll not spend too much time focusing on which specific tool is best, but will focus on why they are important.</p><p>Starting off with a repository for our bakery, we’ll need somewhere to store our packer code, pipeline configuration, ansible configuration and other required resource IaC and so having a central place to put it is a good idea and having version control keep a track of our changes to configuration can be helpful over the lifecycle of an image.</p><p>Some decent code repository tools are <a href="https://github.com/">GitHub</a>, <a href="https://about.gitlab.com/">GitLab</a>, <a href="https://bitbucket.org/">BitBucket</a> and <a href="https://azure.microsoft.com/en-gb/products/devops">Azure DevOps</a> but some orgs will be aligned to older technologies such as Team Foundation Services (TFS) or Subversion (SVN) or one of so many other options! What you choose doesn’t really matter as long as we have somewhere to put the bakery code.</p><p>These days a lot of repositories come with Pipeline tools included , <a href="https://github.com/features/actions">GitHub Actions</a>, <a href="https://azure.microsoft.com/en-gb/products/devops/pipelines/">Azure DevOps Pipelines</a>, <a href="https://docs.gitlab.com/ee/ci/">GitLab CI/CD</a> to name a few, but there are also tools that specialise such as <a href="https://octopus.com/">Octopus Deploy</a> and <a href="https://www.jenkins.io/">Jenkins</a>. Again choice of tool is fairly inconsequential here they will all be capable of supporting a bakery set of pipelines.</p><p>Now for the actual process that we will be putting into the pipeline tool, we’ll be setting up a process that aligns to ‘The Baking Process’ as outlined earlier.</p><ol><li>A base image is selected — <strong>Parameters / Triggered selection of images</strong></li><li>A VM or container instance of the selected base image is created — <strong>Packer Build</strong></li><li>The instance is configured to the required level — <strong>Ansible run from Packer</strong></li><li>The instance is validated against the configuration — <strong>Ansible run from Packer</strong></li><li>The Instance state is saved and turned into an image — <strong>Packer</strong></li><li>The image is then made available to be used on VMs or containers in future — <strong>Packer</strong></li><li>Clean up resources used for process — <strong>Packer</strong></li></ol><p>What this process boils down to is :</p><ul><li><strong>Image Identification</strong> — which images are you going to build in this specific run</li><li><strong>Run Packer build</strong> — Run Packer build against all selected images</li></ul><p>It’s really that simple, Packer does an amazing job at managing the temporary build instance, launching our configuration scripts, and tidying up after itself.</p><p>Now the complexity around the bakery comes from trying to do the following:</p><ul><li>Manage Child / Parent image dependencies, especially when trying to build in parallel</li><li>Checking whether the version about to be built is already available or not</li><li>Adding triggers/ Automating builds when patches or changes are deployed</li><li>Actually getting all the configuration you require in your images so they can be successfully deployed</li></ul><p>It wouldn’t be much fun if I solved all these problems for you now would it 😉</p><p>But that’s our basic pipeline! All the ingredients have come together and our bakery has taken shape.</p><h3>A little burnt around the edges</h3><p>So on to my experience creating an Azure image bakery and some learnings I had making an image bakery in Azure for the first time, in the hopes that you can avoid them.</p><p>We were aiming to build a Proof of Concept (PoC) Azure bakery to get a better understanding of the specific cloud provider eccentricities and pull together a repository that could be used as an example to build upon in future.</p><p>For our bakery we wanted the Azure resources to be more long lived and used multiple times as if they were deployed into a customer estate, and so we paired our Packer configuration up with resources built using Terraform, referencing the resource IDs for network and Resource groups for whether the packer resources should be deployed.</p><p>Unsurprisingly being that this was my first image bakery, some of the challenges come from a lack of initial understanding, But I also found that usage of Packer in Azure is not widely documented on the web and so finding help was a challenge at times. This is one of the main reasons why I decided to write this blog — to help others potentially avoid the confusion I hit in building this bakery.</p><p>When building an image bakery in Azure we needed to setup few resources first:</p><ul><li>Azure Compute Gallery (previously known as Shared Image Gallery)</li><li>Image definitions</li></ul><h4>Azure Compute Gallery and Azure Image Definitions</h4><p>The <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/azure-compute-gallery">Azure Compute Gallery</a> is a resource that helps you build structure and organisation around your images and applications. It helps you manage security and access around the images, global replication of images and versioning and grouping of images.</p><p>In our bakery we set up the gallery as the storage for our custom images. There are other options for image storage, but the gallery’s features around access control suited it towards the organisational design our PoC was aligning with.</p><p>We then needed to create <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/shared-image-galleries?tabs=azure-cli#image-definitions">image definitions</a> per image being deployed.</p><p><a href="https://learn.microsoft.com/en-us/azure/virtual-machines/shared-image-galleries?tabs=azure-cli#image-definitions"><em>“Image definitions are a logical grouping for versions of an image. The image definition holds information about why the image was created and also contains Image metadata such as, what OS it is for, features it supports and other information about using the image”</em></a></p><p>This is where my problems started. For anyone who has worked in Azure you’ll be familiar that VMs need to be provided with an OS_SKU, Publisher and Offer to select the operating system in which to install.</p><p>I believed when we set up the image definitions in Terraform that the OS_SKU, Publisher and Offer needed to align to the Base operating system we were using for creating the first custom images.</p><pre>identifier {<br>  publisher = &quot;Canonical&quot;<br>  offer = &quot;UbuntuServer&quot;<br>  sku = &quot;18.04-LTS&quot;<br>}</pre><p>This was making it difficult to create child images because the compute gallery will only allow image definitions to share one or two of the values for OS_SKU, Publisher and Offer, but not all three values. (But a child image is based off of the same original OS ?! — Cue confusion)</p><p>This was <strong>incorrect</strong>.</p><p>The OS_SKU, Publisher and Offer defined in your image definition are <strong>YOUR</strong> values.</p><p>So you can have the Publisher be your company name, the Offer be the team and the OS SKU be the Image Type, or whatever values you like!</p><pre>identifier {<br>publisher = &quot;ContinoBakery&quot;<br>offer = &quot;Base_Ubuntu&quot;<br>sku = &quot;18_04_secure_base&quot;<br>}</pre><p>You actually define what the image is created from in the Packer values, and as long as the image definition is aligned on the overall type of OS (Windows / Linux) then it can be linked to the image.</p><p>Maybe it was just me that got confused but it wasn’t clear to me when I read through documentation, so here we are Plain and clear — Use your own values, just make sure you align the naming strategy with everyone else building images in your gallery!</p><h4>It’s a New Generation</h4><p>On to the next issue, After Packer had built the temporary instances into Azure it was occasionally hitting issues with saving the images. It wasn’t occurring on all of our images but just a couple of images did not want to be created. It was odd as the VMs spun up quite happily, so their parent image was obviously available, however Packer did not want to save the image.</p><p>It turns out we had misaligned our VM generations between the values being sent to Packer to build, and those defined in our image template.</p><p>If you’re unaware of the differences in Hypervisor generation then Microsoft has a good <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2">article</a> discussing the differences and on which VM families that they are available on.</p><p>Another one to remember for future, you cannot create a generation 2 VM and then try to save it in a generation 1 image definition. And of course if your estate is relying on some of the Image sizes that are specific to a generation then you’ll need to think about creating generation specific images.</p><h4>WinRM Connectivity</h4><p>The last little burnt crust on our otherwise pristine cake, Configuring Windows systems… Configuring our custom Linux images was fairly simple, SSH connectivity allowed the remote commands to be run against the instances out of the box with minimal additional configuration.</p><p>Our issue arose when it was time to work on the Windows VMs. Allowing access to WinRM on the instance so that it can be configured remotely using the configuration scripts run from the Packer VM.</p><p>We needed to enable WinRM straight after the VM was created, allowing the connections in to configure the server. Looking at code online we found Packer examples for AWS that were using a feature called <em>user_data_file</em> where you can provide a script to run on instance at creation time — That’s perfect!</p><p>So we found a script that would set up the required WinRM settings, and we added it as a <em>user_data_file</em>, assuming it would be run on creation… This is not the case in Azure, just passing a <em>user_data_file</em> doesn’t work.. It’s still not allowing connections to the VM.</p><p>So after some investigation we found the problem. We needed to pass both a <em>user_data_file</em> containing the WinRM settings along with a <em>custom_script</em> command to run the <em>user_data_file</em> on the instance.</p><pre>  user_data_file = &quot;winrm_bootstrap.ps1&quot;<br>  custom_script   = &quot;powershell -ExecutionPolicy Unrestricted -NoProfile -NonInteractive -Command \&quot;$userData = (Invoke-RestMethod -Headers @{Metadata=$true} -Method GET -Uri http://169.254.169.254/metadata/instance/compute/userData?api-version=2021-01-01$([char]38)format=text); $contents = [System.Text.Encoding]::UTF8.GetString([Convert]::FromBase64String($userData)); set-content -path c:\\Windows\\Temp\\userdata.ps1 -value $contents; . c:\\Windows\\Temp\\userdata.ps1;\&quot;&quot;</pre><p>And we can connect! Huzzah!</p><p>Again this is maybe something others are already aware of but it’s one of those potholes I got tripped up by that now you know about it you can avoid more easily.</p><h3>Finishing Touch</h3><p>So there we go, the cakes are ready and it’s time to review what we’ve baked.</p><p>We’ve discussed Image Bakeries and how creating images can help to speed up the deployment of applications and help to standardise Operating systems across an estate, looking at the process and why you might want inheritance based structures across your images.</p><p>Searching through our Utensils shelves we found the tooling that you might use to create your bakery such as industry leading tools like Packer and Ansible and pipeline tools like GitHub or Azure Devops amongst many others. There’s many ways to make a cake. If you wanted a Whisk but your organisation could only provide a Fork then you’ll still be able to make a half decent bakery, it may just be more work to get it started.</p><p>And finally I’ve told you my burnt edges, soggy bottoms and downright disgusting tales of woe from when I tried to do all of this myself, and hot issues along the way so that you don’t struggle as I did.</p><p>I hope this blog has been helpful. If you are interested in more detailed personalised baking expertise then feel free to reach out to <a href="https://www.linkedin.com/in/mark-nash/">me on LinkedIn</a> or to our <a href="https://www.contino.io/">Contino</a> Team for more assistance.</p><p>But for now — Thanks for reading, and happy baking!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a13c1a8670b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/whats-that-baking-it-s-vm-images-5a13c1a8670b">What’s that Baking? It’s VM Images!</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How can Azure Container Apps and Dapr help you improve the developer experience of Kubernetes?]]></title>
            <link>https://medium.com/contino-engineering/how-can-azure-container-apps-and-dapr-help-you-improve-the-developer-experience-of-kubernetes-e0ec3d46d1d3?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/e0ec3d46d1d3</guid>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[azure-container-apps]]></category>
            <dc:creator><![CDATA[Romeel Khan]]></dc:creator>
            <pubDate>Tue, 03 Oct 2023 14:21:25 GMT</pubDate>
            <atom:updated>2023-10-03T14:21:25.857Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*di29SYl5ARJfxqe-JSHFxA.png" /></figure><p>With the advent of public cloud and the adoption of technologies such as containers and Kubernetes, many enterprises jumped on the bandwagon to embrace these as platforms on which to build their next generation cloud-native applications.</p><p>Kubernetes adoption is not slowing, and according to the <a href="https://www.dynatrace.com/news/blog/kubernetes-in-the-wild-2023/">2023 Dynatrace report</a>, it’s now becoming the “operating system” of the cloud. A powerful container orchestrator, it provides key benefits to name a few:</p><ul><li>Resilience to failures</li><li>Self healing</li><li>Auto scaling</li><li>Orchestration of Microservices</li></ul><p>Despite its popularity though, the developer experience can still be overwhelming for teams. Kubernetes is complex, has a steep learning curve and has many moving parts that have to be understood when deploying your applications.</p><p>In this blog, we explore how you can use Azure Container Apps and Dapr as a platform to develop modern cloud-native applications without getting bogged down by the complexities of Kubernetes.</p><h3>What are Azure Container Apps?</h3><p>What if we could hide the complexity and cognitive load of the Kubernetes ecosystem from the purview of development teams but still take advantage of all its benefits?</p><p>Azure Container Apps is a hosting platform built on top of AKS. It provides all the underlying benefits of Kubernetes orchestration without developers needing to know the details of Kubernetes constructs.</p><p>This reduces the unit of deployment back to the container image — rather than pod — that you build as part of your build pipeline.</p><p>Here’s a diagram that illustrates deploying to Kubernetes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjAvrtWH2U_PAvacnG6ZNg.png" /></figure><p>With App Containers, however, as illustrated below, deploying is clearly much more straightforward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*V0N_5Guiu7D7_tShgc0aZw.png" /></figure><p>Kubernetes provides a lot of choice that can be overwhelming for some teams, and common questions and options for building applications on Kubernetes might include:</p><ul><li>Which Ingress controller do I use? Nginx, Application Gateway, Traefik?</li><li>How do I deploy a release? Blue/Green, Rolling?</li><li>Networking, how do I isolate my applications?</li><li>What is the best way to template my application manifests? Helm? Kustomise?</li></ul><p>These questions, and the answers your team chooses, depend on the skills and the support you have to manage the extensible nature of Kubernetes.</p><p>Azure Container Apps removes this burden from the team. It’s built on a selection of underlying technologies and features built on top of Kubernetes:</p><ul><li>Envoy Proxy (Ingress)</li><li>KEDA (Event driven use cases)</li><li>Dapr runtime support (Microservices development)</li><li>Out-of-the-box auto-scaling</li><li>Traffic splitting via revisions (Canary deployments)</li></ul><p>As a developer you no longer need to worry about defining your pod, deployment or service manifests; Kubernetes orchestration is handled by Azure Container Apps.</p><h3>What use cases can this be applied to?</h3><p>If the underlying Kubernetes cluster is abstracted away, what kind of applications are supported?</p><p>Azure Container Apps provides numerous use cases including:</p><ul><li>Expose your container based APIs</li><li>Background processing jobs</li><li>Microservices</li><li>Event driven applications</li></ul><h3>What caveats are there?</h3><p>Like any platform, there are going to be constraints and caveats. Some of the key caveats of Azure Container Apps are:</p><ul><li>You can’t access the Kubernetes cluster (defeats the point of Azure Container apps)</li><li>It only supports Linux containers</li><li>Privileged containers cannot run</li><li>Single port exposed (Multi port in preview recently — <a href="https://github.com/microsoft/azure-container-apps/issues/546">https://github.com/microsoft/azure-container-apps/issues/546</a>)</li></ul><p>Additionally, to find out more about feature road map refer to: <a href="https://github.com/orgs/microsoft/projects/540">https://github.com/orgs/microsoft/projects/540</a></p><h3>Hosting your container applications in Azure Container Apps</h3><p>To package and deploy your applications you require:</p><ul><li>A Linux based container image (linux/amd64)</li><li>Accessible from a public or private registry</li></ul><p>For further details on container limitations please refer to <a href="https://learn.microsoft.com/en-us/azure/container-apps/containers#limitations">https://learn.microsoft.com/en-us/azure/container-apps/containers#limitations</a></p><h3>Application lifecycle management</h3><p>Once you have your application container images ready for deployment you need to understand some basic concepts.</p><h4>Azure Container App environments</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*n4tjxjeJvZVae88u" /></figure><p>Azure Container Apps isolate your application using a concept of environments. To summarise an environment is:</p><ul><li>An isolation boundary for your application</li><li>Application shares the same virtual network and compute resources</li></ul><p>Think of the concept of Namespace in Kubernetes.</p><h4>Revisions</h4><p>Traditionally, with containers and Kubernetes, updates to your deployments are triggered with an update to your container tag as part of your CI/CD process.</p><p>Azure Container Apps introduce the concept of revisions. Application lifecycle updates can be triggered by:</p><ul><li><a href="https://learn.microsoft.com/en-us/azure/container-apps/revisions#revision-name-suffix">Revision suffix</a></li><li>Container configuration and images</li><li>Scale rules for the container application</li></ul><p>Revisions enable the development team to cater for use cases such as:</p><ul><li>Reverting a version</li><li>Traffic splitting</li><li>Blue/green deployments</li></ul><p>For further information please refer to: <a href="https://learn.microsoft.com/en-us/azure/container-apps/revisions#revision-scope-changes">https://learn.microsoft.com/en-us/azure/container-apps/revisions#revision-scope-change</a></p><h4>Infrastructure deployment options</h4><p>Azure Container Apps can be deployed via the following options:</p><ul><li>ARM/BIcep</li><li>Terraform (AzAPI)</li><li>Azure CLI</li></ul><p>Depending on your CI/CD tool stack, you can build pipelines that build out the base Container App environments, as well as the IaC to deploy the Azure Container Apps as well.</p><h4>Azure Container Apps use cases</h4><p>In practical terms, Azure Container Apps are essentially a PaaS abstraction over AKS/Kubernetes. The focus of the platform is to enable the seamless build of your modern containerised web applications.</p><p>In contrast, If you are looking for more complex scenarios and extensibility of Kubernetes then the answer is a definite no.</p><p>Azure Container Apps are a viable alternative to App Services for building applications such as:</p><ul><li>Converting your N-tier to container based applications</li><li>New Greenfield development where Microservices architecture is used</li><li>Running background worker tasks (Database cleanup, maintenance)</li></ul><h3>Building your applications with Dapr on Azure Container Apps</h3><p>Azure Container Apps provide managed Kubernetes orchestrator capabilities. However, it does not provide a programming model for developing your Microservices. In this part of the blog we look at how Dapr can be used with Azure Container Apps to develop your Microservice applications.</p><h4>What is Dapr?</h4><p>Dapr is an open-source project written in Go, It is a portable, event-driven runtime that can be run on the cloud or edge and supports a diverse amount of programming languages and frameworks.</p><h4>What problem does Dapr solve?</h4><p>Building distributed systems and Microservices is complex, and common challenges include application resiliency, state management, service to service invocation and more.</p><p>Dapr removes the burden of handling these challenges from the developer so that the development team can focus on building their application features in any language and framework.</p><h3>Dapr Architecture</h3><p>Dapr operates on a sidecar architecture. Each of the building block APIs is used via the sidecar pattern. For performance, Dapr uses the gRPC protocol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/0*eG6mc49AqIIXyxGb" /></figure><h4>Dapr Building blocks — pluggable components</h4><p>Dapr is built on the concept of building blocks, which are exposed via the Dapr runtime as APIs that can be used. Dapr is not a prescriptive API model, but it allows you to consume any of these building blocks as pluggable components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*txZ73V3PX8zlVmsB" /></figure><p>For example, if you are building a shopping cart component you may use the Dapr Redis state management building block. Dapr provides you with numerous application plumbing building blocks you don’t have to write custom code for.</p><h4>Developing with Dapr components</h4><p>Like Kubernetes, Dapr building blocks are configured via declarative YAML files following a similar schema spec to Kubernetes objects. For example, this is a Dapr state management component file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*wm3CKDBDR8GWHCaM" /></figure><p>You can use component files to build your own components as well. For further reading about the schema head to <a href="https://docs.dapr.io/reference/resource-specs/">Dapr’s resource specifications page</a>.</p><h4>Dapr hosting with Azure Container Apps</h4><p>Dapr can be hosted in three configurations:</p><ul><li>Self-hosted (with or without Docker)</li><li>In a Kuberenetes cluster</li><li>Azure Container Apps</li></ul><p>The Azure Container Apps option provides access to the Dapr control plane for configuration via the Azure Portal or through Azure CLI.</p><p>The <a href="https://docs.dapr.io/operations/hosting/kubernetes/kubernetes-deploy/">Dapr site</a> gives more information on the native Kubernetes setup.</p><h4>Connecting to Azure Services</h4><p>With Dapr integration you get access to Azure services, as well as capabilities such as Managed Identity via Azure Container Apps environments.</p><h4>Deployment options</h4><p>When deploying to Azure Container Apps you must explicitly enable your environments for Dapr usage. Dapr components can be configured via:</p><ul><li>ARM/Bicep</li><li>Azure CLI</li><li>Github actions workflow with Bicep</li></ul><p>Microsoft provides a Terraform example <a href="https://learn.microsoft.com/en-us/samples/azure-samples/container-apps-azapi-terraform/container-apps-azapi-terraform/">here</a>.</p><h3>Summary</h3><p>If you’re looking to onboard greenfield applications using a cloud-native development approach, and are concerned about the learning curve of Kubernetes and the impact on productivity of your development team, then Azure Container Apps is a viable platform to start consuming.</p><p>Additionally, Dapr with Azure Container Apps provides an additional opportunity to ease the complexities of building distributed applications (Microservices), and is a CNCF project that’s gaining traction in the development community.</p><p>At Contino, we help customers in highly regulated environments improve the outcomes of their digital transformation initiatives by providing consulting and engineering expertise in areas such as cloud-native development.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0ec3d46d1d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/how-can-azure-container-apps-and-dapr-help-you-improve-the-developer-experience-of-kubernetes-e0ec3d46d1d3">How can Azure Container Apps and Dapr help you improve the developer experience of Kubernetes?</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GitOps and Compliance: How to Effectively Use Compliance Tools]]></title>
            <link>https://medium.com/contino-engineering/gitops-and-compliance-how-to-effectively-use-compliance-tools-6e6a2d167894?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/6e6a2d167894</guid>
            <category><![CDATA[compliance]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[policy-as-code]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Colin Smith]]></dc:creator>
            <pubDate>Thu, 21 Sep 2023 08:02:50 GMT</pubDate>
            <atom:updated>2023-09-21T08:02:50.402Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here’s how to get the most out of cloud compliance tools, and why we recommend avoiding the auto-remediation feature.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W0T9hoYdBRm1pbFP4P4r1g.png" /></figure><p>For development teams following a GitOps philosophy and using Infrastructure-as-Code (IaC) templates to manage cloud resources, a conflict can arise when these resources are subject to automatic remediation by cloud compliance tools. In this blog, we explore the options available, and why organisations should use code-scanning compliance tools in CI pipelines to check IaC templates to prevent non-compliant cloud resources being deployed.</p><p>Read on for examples and relevant links and there’s also a word at the end from the Azure Policy team on this topic.</p><h3>Cloud Compliance Tools</h3><p>As organisations transition to the cloud, they must ensure they comply with relevant business and/or regulatory policies—for example, a certain business might not be allowed to store its data outside the European Union.</p><p>Many vendors including the major cloud hyperscalers provide tooling that can scan cloud resources to determine whether they comply with an organisation-defined set of policies. These cloud compliance tools can provide a valuable service to organisations by improving their mean time to repair (MTTR) any non-compliant resources.</p><blockquote><strong>Note:</strong> These tools are often referred to as security orchestration, automation and response (SOAR) tools. However, since they allow compliance policies to be defined that are unrelated to security concerns, I will continue to refer to them generically as cloud compliance tools.</blockquote><p>Going one step further, many of these compliance tools also include an option to <em>automatically</em> repair any non-compliant resources they may find. However, despite the appeal, this auto-remediation feature can cause problems for organisations who are trying to follow a GitOps philosophy and manage their cloud resources using IaC templates.</p><h3>GitOps</h3><p>Creating infrastructure and other resources in cloud providers is complex. Even simple environments can involve provisioning multiple instances of various resource types. Creating all of this manually is difficult and error-prone and would usually result in infrastructure that differs slightly between environments.</p><p>To solve this problem, <a href="https://www.redhat.com/en/topics/devops/what-is-gitops">GitOps</a> mandates that all infrastructure is encoded into IaC templates. These templates can then be deployed multiple times — once to each environment — guaranteeing that all environments are logically identical (even if the SKUs and/or instance numbers might vary).</p><p>The fundamental principle of GitOps is that the code in source control is always the ultimate source of truth. In terms of cloud resources, this means that the IaC templates are always assumed to be correct. If any cloud resources deviate from their IaC template definition, those resources are assumed to have “drifted” and must be brought back into line with the IaC template.</p><p>Unfortunately, the auto-remediation feature of the cloud compliance tools makes changes directly to the deployed resources and, as such, can be a cause of such drift. This can lead to a constant battle between the development teams managing the IaC templates and the security team managing the organisation’s compliance policies.</p><h4>Example: Tag Inheritance</h4><p>Imagine that an organisation wants to get a grip on where its Azure budget is being spent. It knows that every resource group already has a “Department” tag so it might institute a policy stating that “<em>All Azure resources should include the same tags as their parent resource group.” </em>This policy would allow it to generate financial reports showing how much is being spent by each department.</p><p>To enforce this policy, the organisation:</p><ul><li>Creates an Azure Policy Definition stating that all Azure resources should include, at least, the same tags as their parent resource group</li><li>Assigns that Azure Policy Definition to the root management group with a “deployIfNotExists” action (i.e. with auto-remediation enabled)</li></ul><p>Now, imagine that a development team later deploys a virtual machine into a resource group using an IaC template… but the VM is missing some of the tags from that parent resource group. In this scenario, we should expect to see the following sequence of events:</p><ol><li>The VM is deployed by the IaC template</li><li>Azure Policy detects the policy violation and auto-remediates it by adding the missing tags to the VM</li><li>When the IaC template is next deployed, it sees the additional tags as drift and removes them</li><li>Repeat from step 2 ↑</li></ol><p>The cloud compliance tool and the IaC template would be constantly fighting over which tags should be applied to the VM.</p><p>This is a general problem with all clouds, not just Azure. Ideally, organisations would avoid anything that makes changes to deployed cloud resources outside of their IaC processes. Following the <a href="https://www.redhat.com/en/topics/devops/what-is-gitops">GitOps</a> philosophy, they would consider the IaC template to be the source of truth for their VMs. Any change to those VMs should only be made by updating the IaC templates and redeploying.</p><p>As such, organisations that are managing their resources via IaC templates should avoid the auto-remediation feature of cloud compliance tools.</p><h3>Code-Scanning Compliance Tools</h3><p>Fundamentally, the conflict described above occurred because the cloud compliance tool had no knowledge of the IaC template that was used to deploy the VM. Cloud compliance tools can only act on actual, deployed resources, not the IaC templates in which they were defined.</p><p>So if organisations shouldn’t auto-remediate their non-compliant resources directly, what should they do? The answer is to simply never let the non-compliant resources get deployed in the first place!</p><p>Indeed, some of the cloud compliance tools described above <em>can</em> be used to deny the deployment of new, non-compliant resources (and use of such “deny” policies is encouraged). However, rather than wait until the resource is being deployed, why not run the compliance checks directly against the IaC templates themselves? This is where code-scanning compliance tools shine.</p><blockquote><strong>Note:</strong> Often these sorts of compliance tools are referred to as “Policy-as-Code” tools. However, that term is also used to describe the process of encoding the policies used by cloud compliance tools into IaC templates. For this reason, I am using the generic term “code-scanning compliance tools”.</blockquote><blockquote>For example, a Terraform template might deploy some Azure Policy definitions and assign them to Azure Subscriptions. Such a situation is often referred to as “policy-as-code” despite the fact that Azure Policy is still monitoring the deployed cloud resources… not the content of the IaC templates that are deploying those resources.</blockquote><p>These code-scanning compliance tools can be integrated into CI pipelines. They will scan any IaC templates against a set of policies defined by the organisation and, should any non-compliance be detected, the pipeline will fail. Coupling this with some well designed branch control policies, organisations can make it impossible for PRs containing non-compliant resources to ever be merged into the main branch. Thus they can never be deployed.</p><p>Not only do these tools shift the compliance checks left, they also provide a far better experience for development teams. Developers can run these tools on their own machines, drastically speeding up the feedback loop. They will get near-instant feedback whenever they add a non-compliant resource to their IaC templates. Being able to test before merging their PRs will also mean that they avoid impacting any other teams that are attempting to merge their changes into the main branch. Indeed, developers will come to see these business compliance policies as akin to “unit-tests” that their IaC templates must pass before they can merge their pull requests.</p><p>Some notable code-scanning compliance tools that work with IaC templates include:</p><ul><li><a href="https://docs.hashicorp.com/sentinel">Hashicorp Sentinel</a></li><li><a href="https://www.checkov.io/">checkov</a></li><li><a href="https://docs.snyk.io/scan-infrastructure/getting-started-with-snyk-iac">Snyk IaC</a></li><li><a href="https://runterrascan.io/">terrascan</a></li><li><a href="https://www.qualys.com/apps/infrastructure-as-code-security/">Qualys IaC Security</a></li></ul><blockquote><strong>Note:</strong> In some cases, these tools scan the result of a <em>plan</em> generated from the IaC templates — not the IaC templates themselves. However, they can still be run both locally and in the CI pipeline and they have all of the same benefits as described above.</blockquote><h3>Summary</h3><p>Using the auto-remediation feature of cloud compliance tools can cause problems due to the fact that IaC templates may see those changes as drift and attempt to revert them. With a GitOps philosophy in mind, the IaC templates in source control should be seen as the ultimate source of truth. Therefore, unless the policy is absolutely <em>critical, </em>organisations using IaC should avoid using any tools that make changes to those deployed resources… including auto-remediation.</p><p>Code-scanning compliance tools can prevent developers from ever merging PRs with non-compliant resources in their IaC templates, which in turn prevents non-compliant resources from ever getting deployed — and provides immediate feedback to the developers.</p><p>Cloud compliance tools remain valuable for their real-time detection and alerting capabilities though. They could audit any existing resources that are not compliant to any <em>new</em> policies that are defined in future. In addition, they can prevent malicious attempts to change deployed resources. For example, an Azure Policy with a “deny” action could prevent a user from disabling the soft-delete flag on a Recovery Services vault, which would have allowed them to permanently delete all backups.</p><h3>Footnote: Pragmatism</h3><p>The above solution is idealistic, while of course the real world is generally far from ideal. Organisations can — and will — decide to use the auto-remediation features of cloud compliance tools (or make other manual changes to cloud resources). Development teams need a way of responding to those changes without abandoning GitOps altogether.</p><p>In many cases, IaC tools allow developers to specify that certain properties of certain resources should be ignored after initial deployment. For example, Terraform has an “<a href="https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle#ignore_changes">ignore_changes</a>” lifecycle policy to do just that. Similarly, it’s possible to just <em>remove</em> any properties that should be ignored from ARM/Bicep/CloudFormation templates after the initial deployment. In this way, any changes made to those properties by auto-remediation in future will be ignored by those IaC tools.</p><p>However, this feature should be used as sparingly as possible because it prevents the IaC template from being able to safely <em>update</em> those properties in future. If it attempted to do so, it wouldn’t know if the value it was overwriting was the initial value, or some new value to which it was later changed manually.</p><h4>Edit:</h4><p>I asked the Azure Policy team about this topic and this was their response:</p><blockquote>IaC is fully tracking the resources and it expects for the resource to always look the same as your definition in state file, otherwise it will redeploy. When you have a Deploy-if-not-Exists (DINE) policy, then resource will be different, and then it will be redeployed, so the scenario you are describing is correct.<br> <br>Usually, you don’t need Azure Policy when using IaC. Your IaC template should specify the final and desired state of your resource, so the resource just needs to be deployed directly as you want it to be.</blockquote><p>It’s good to get confirmation of my thinking from one of the big cloud providers, although I personally disagree with the “… <em>you don’t need Azure Policy when using IaC…</em>” bit. IMHO, an Azure Policy with a “deny” action can be a useful safeguard against malicious/accidental changes that are made manually. It’s only the auto-remediation actions on the policy that cause problems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e6a2d167894" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/gitops-and-compliance-how-to-effectively-use-compliance-tools-6e6a2d167894">GitOps and Compliance: How to Effectively Use Compliance Tools</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Azure Blog — Azure Monitoring Agent taking over Log Analytics Agent and how to prepare for change?]]></title>
            <link>https://medium.com/contino-engineering/azure-blog-azure-monitoring-agent-taking-over-log-analytics-agent-and-how-to-prepare-for-change-890f4f7f2101?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/890f4f7f2101</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[log-analytics]]></category>
            <category><![CDATA[azure-virtual-machines]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[monitoring]]></category>
            <dc:creator><![CDATA[Andes Lam]]></dc:creator>
            <pubDate>Mon, 18 Sep 2023 16:00:42 GMT</pubDate>
            <atom:updated>2023-09-18T19:56:19.413Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why It’s Time to Migrate to Azure Monitoring Agent</h3><h4>Log Analytics Agent isn’t dead yet, but here’s why it’s time to migrate to Azure Monitoring Agent</h4><p>If you’re working in Azure, it hopefully isn’t coming as a shock to you that <a href="https://azure.microsoft.com/en-gb/updates/were-retiring-the-log-analytics-agent-in-azure-monitor-on-31-august-2024/?WT.mc_id=AZ-MVP-5003548">Microsoft</a> will be retiring Log Analytics Agent on 31st August 2024.</p><p>In this blog we explain why you should consider migrating as soon as possible, rather than taking it down to the wire. We’ll take you through the new Azure Monitoring Agent, its benefits and how it compares to legacy agents, as well as migration best practices and how to prepare for changes.</p><p>Even without the 2024 deadline, we’d recommend migrating well ahead of the cutover date to reap the benefits of the Azure Monitoring Agent.</p><h3>1. What is Azure Monitoring Agent (AMA)?</h3><p>The Azure Monitoring Agent (AMA) is a data collection tool that collects monitoring data from a Virtual Machine and delivers it to Azure Monitor.</p><p>Azure Monitor is a service in Azure that provides performance and availability monitoring for applications and services in Azure, other cloud environments, or on-premises. Azure Monitor collects data from multiple sources into a common data platform where it can be analysed for trends and anomalies.</p><p>Effectively, AMA will collect monitoring data from different operating systems and send it to Azure Monitor for further analysis and monitoring use.</p><h4>Why migrate?</h4><p>Excluding the fact that the Log Analytics agent will be retired in less than a year, there are many reasons why you should consider migrating to the new Azure Monitor Agent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E0pSOM5r7NYvwLDQ" /><figcaption>Benefits of migrating to Azure Monitor Agent</figcaption></figure><h4>i/ Better security and performance</h4><ul><li>AMA uses Managed Identity or Azure Active Directory (Azure AD) tokens (for clients), which are much more secure than the legacy authentication methods.</li><li>AMA provides a higher events per second (EPS) upload rate compared to legacy agents.</li></ul><h4>ii/ Cost saving</h4><ul><li>AMA relies on Data Collection Rules (DCRs) as the control plane, which lets you target data collection from groups of machines connected to the same or different workspaces, as compared to the “all or nothing” approach of the legacy agents.</li><li>Using DCRs you can define which data to ingest and which data to filter out to reduce workspace clutter and save on costs.</li><li>Support for additional destinations, like Azure Storage accounts, will be available in the future, which will enable you to optimise for longer storage periods at cheaper costs.</li></ul><h4>iii/ Centralised and simpler management</h4><ul><li>Easy <strong>multihoming</strong> on Windows and Linux, across regions and tenants.</li><li>Centralised, <em>‘in the cloud’</em> agent configuration makes every action across the data collection lifecycle, simpler and scalable, from onboarding to deployment to updates and all related changes over time.</li><li>Greater transparency and control over services and related updates, including Sentinel, Defender for Cloud, and VM Insights.</li></ul><h3>2. Cost Optimisation With Data Collection Rules</h3><p>Data collection rule (DCR) in Azure Monitoring Agent allows sharing the same rule across single or multiple VMs as mentioned above.</p><p>To understand how a DCR can save costs, see this following graph</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/0*gxF9KGN2zcjg-xdf" /><figcaption>From Data collection rule to transformation to destination</figcaption></figure><p>Using transformation that is available within Data Collection Rule before sending to destination, you can reduce the amount of data ingestion while creating more granular and useful data that can be viewed in Log Analytics Workspace.</p><p>Transformations are defined in a data collection rule (DCR) and use a Kusto Query Language (KQL) statement that’s applied individually to each entry in the incoming data. It must understand the format of the incoming data and create output in the structure expected by the destination.</p><p>Microsoft has a list of <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/vm/monitor-virtual-machine-data-collection#control-costs">cost optimisation strategies for different log collections using DCR</a>. Topics covered are featuring</p><ul><li>Migrate from Log Analytics agent to Azure Monitor agent for granular data filtering.</li><li>Filter data that you don’t require from agents.</li><li>Determine whether you’ll use VM insights and what data to collect.</li><li>Reduce polling frequency of performance counters.</li><li>Ensure that VMs aren’t sending duplicate data.</li><li>Use Log Analytics workspace insights to analyse billable costs and identify cost saving opportunities.</li><li>Migrate your SCOM environment to Azure Monitor SCOM Managed Instance.</li></ul><p>And current tables that support transformations in Azure Monitor Logs can be found <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tables-feature-support">here</a>.</p><p>For example, you create a DCR rule custom filter using XPath queries. You can further filter the collected data by editing the DCR to add a transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1qkgia-oGq_C_qvr" /><figcaption>An example of XPath data collection rule filtering Windows Event using XPath.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z90NZGIfD8De0hYg" /><figcaption>An example of text log transformation</figcaption></figure><p>For more detail on transformations, go to this tutorial: <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-workspace-transformations-portal">Add a transformation in a workspace data collection rule by using the Azure portal</a>.</p><h4>ARM sample DCR templates</h4><ul><li><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-transformations#single-destination">DCR for Azure Monitor Agent that sends data to the Syslog table. In this example, the transformation filters the data for records with error in the message</a></li><li><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-workspace-transformations-api">Tutorial: Add transformation in workspace data collection rule to Azure Monitor using Resource Manager templates</a></li></ul><h4>Terraform sample DCR templates</h4><ul><li><a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_data_collection_rule#transform_kql">Terraform registry doc azurerm_monitor_data_collection_rule</a></li><li><a href="https://learn.microsoft.com/en-us/azure/templates/microsoft.insights/datacollectionrules?pivots=deployment-language-terraform">Terraform (AzAPI provider) resource definition</a></li></ul><h3>3. Comparison Between New and Legacy Agent</h3><p>Microsoft has already stated that Log Analytics is fully consolidated into Azure Monitor Agent.</p><p>However, <a href="https://github.com/influxdata/telegraf">InfluxData Telegraf agent</a> and diagnostic extension are still under consolidation, and unfortunately Microsoft hasn’t given any more details on the exact timeline. Watch <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-extension-versions">this space</a> for latest extension updates.</p><p><strong>Telegraf agent:</strong> Sends data to Azure Monitor Metrics (Linux only). Only basic Telegraf plugins are supported today in Azure Monitor Agent, and bear in mind that InfluxData Telegraf is an open source agent and not officially supported by Azure Monitor.</p><p><strong>Diagnostics extension:</strong> Sends data to Azure Monitor Metrics (Windows only), Azure Event Hubs, and Azure Storage.</p><p>If you have the requirement to collect additional logs that are not yet consolidated, coexisting AMA with diagnostic extension or telegraf agent alongside would still be the way to go. You would need to consider and try to avoid monitoring data duplication. There are a number of good articles highlighting different combinations or ways to achieve that. (Further links at the end of this post!) However, that is not our focus today.</p><p>The following two tables summarise the comparison of Azure Monitor Agent with the legacy Azure Monitor telemetry agents for Windows and Linux.</p><h4>Windows agent</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XV_TSbsWWAF1u9LQ" /><figcaption>Windows AMA feature comparison</figcaption></figure><h4>Linux agent</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UBrxz1Sq9yAe5qoh" /><figcaption>Linux AMA feature comparison</figcaption></figure><h4>Comparison summary</h4><ul><li>Azure Monitor Metric is in public review</li><li>Azure Stack HCI is supported by Azure Monitor Agent over legacy</li><li>Automation Update management is no longer required. <a href="https://learn.microsoft.com/en-us/azure/update-center/overview?tabs=azure-vms">Update management center (Preview)</a> is the v2 version of Automation Update management and the future of Update management in Azure.</li><li>Bear in mind that again there are limitations on which OS are being supported (<a href="https://learn.microsoft.com/en-us/azure/update-center/manage-updates-customized-images">custom images</a>, <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/automatic-vm-guest-patching#supported-os-images">marketplace images</a> and <a href="https://learn.microsoft.com/en-us/azure/update-center/support-matrix?tabs=azurevm%2Cazurearc-os#tabpanel_2_azurearc-os">Azure Arc-enabled servers</a>) and unsupported (<a href="https://learn.microsoft.com/en-us/azure/update-center/support-matrix?tabs=azurevm%2Cazurevm-os#unsupported-operating-systems">Windows client, Virtual machine scale sets and Azure Kubernetes Nodes</a>)</li></ul><p>We can conclude that for log analytics agent replacement, AMA can be enough to suit your needs if you only send data to Azure Monitor.</p><p>However, if you want to get events from Windows (ETW), crash dumps, application logs and agent logs or send data to destinations like Azure Storage or Event Hub. Then we need to combine AMA with diagnostic extensions</p><h3>4. Operating Systems Compatibility</h3><p>It is worth mentioning Azure Monitor Agent compatibility before even considering migration.</p><ul><li>Architecture: All operating systems are assumed to be <strong><em>x64</em></strong>. x86 isn’t supported.</li><li>Physical and on premise servers: View <a href="https://learn.microsoft.com/en-us/azure/azure-arc/servers/prerequisites#supported-operating-systems">supported operating systems for Azure Arc Connected Machine agent</a>, which is a prerequisite to run Azure Monitor agent.</li><li>ARM-based: Azure Arc connected agent is not supported.</li></ul><p>As AMA is still constantly evolving, I would suggest checking out official Microsoft for the operating system compatibility chart.</p><p>For Windows OS support:</p><p>Please visit <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#windows">https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#windows</a></p><p>For Linux OS support:</p><p>Please visit: <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#linux">https://learn.microsoft.com/en-us/azure/azure-monitor/agents/agents-overview#linux</a></p><h3>5. AMA Migration Best Practices and Recommendations</h3><h4>Assess current risks, requirements and standards</h4><p>In all migration, you want to first review if you can migrate successfully. Microsoft has provided a checklist and tools to help as follows</p><ol><li><strong>AMA Migration Helper</strong> to view virtual machine status<br>This is a public Azure workbook template to see connected agents and discover solutions enabled on your Log Analytics workspaces that use legacy agents, including per-solution migration recommendations.</li><li>You can find it on the Azure portal under Monitor &gt; Workbooks &gt; Public Templates &gt; Azure Monitor essentials &gt; AMA Migration Helper</li><li><strong>DCR generator </strong>to parse Log Analytics Agent configuration from your workspaces and generate/deploy corresponding data collection rules automatically.</li><li>This is a powershell script for creating ready to deploy ARM templates to deploy data collection rules based on previous legacy agents.</li><li><strong>Review prerequisites </strong>for installing Azure Monitor Agent<br>Built-in roles, permission, managed identities, azure arc agent (for on-premise machines), networking/firewalls. For more detail, visit <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-manage?tabs=azure-portal#prerequisites">here</a></li><li><strong>Existing Azure policy</strong> in enterprise scale<br>Existing legacy policies may be in place that deploys legacy agents. That would require proper migration planning and testing before rolling out to all other machines.</li><li><strong>Review machines that require upgrade</strong><br>As some of you might face the problem where your OS is non-compatible by AMA which we have briefly mentioned in the “Operating systems compatibility” section. Maybe it is an opportunity to re platform from older versions of Windows and Linux.</li></ol><h4>Validate and test migration steps</h4><p>As Microsoft has already well documented <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-migration#migration-steps">migration steps</a> that we can follow, troubleshoot and test along.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zgIoWYSsgySm2eBI" /></figure><h3>6. Prepare for Change</h3><p>Following considerations that might be useful which has been categorised below</p><h4>a) Pre migration review</h4><ul><li>Determining test subject machines with AMA migration helper and DCR generator</li><li>Finding suitable test machines depends on multiple factors that may vary across teams, business needs and ownerships. Alignment with different projects/teams in a phased approach would be recommended.</li><li>Conducting Proof of Concept (PoC) prior to full scale migration</li><li>We all want some assurances before having a full scale migration that could involve 100+ virtual machines or more for large organisations. By doing time boxed PoC, we can quickly understand and estimate time and resources required.</li></ul><h4>b) Automation</h4><ul><li><strong>New Azure policies</strong> to enable Azure monitoring agents are still under preview. <br>Contino and our clients are actively tracking this issue.<br><a href="https://github.com/Azure/Enterprise-Scale/issues/1055">https://github.com/Azure/Enterprise-Scale/issues/1055</a></li><li>Develop or upgrade existing infrastructure as code (IaC) templates to deploy AMA and their data collection rules</li><li>There are <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-manage?tabs=azure-powershell">powershell</a>, <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-manage?tabs=azure-cli">Azure cli</a> and <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-manage?tabs=azure-resource-manager">ARM templates</a>. I would recommend to further read our article if you are interested in IaC <a href="https://www.contino.io/insights/infrastructure-as-code">here</a></li></ul><h4>c) Security</h4><ul><li>Enable network isolation for AMA. By default, Azure Monitor Agent connects to a public endpoint to connect to your Azure Monitor environment. To enable network isolation for your agents, create <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-endpoint-overview#create-a-data-collection-endpoint">data collection endpoints</a> and add them to your <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-data-collection-endpoint?tabs=PowerShellWindows#add-dces-to-ampls">Azure Monitor Private Link Scopes (AMPLS)</a>.</li><li>Or by using a firewall with correct ports and endpoints. See <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-data-collection-endpoint?tabs=PowerShellWindows#firewall-requirements">firewall requirements</a>.</li><li>Disable and block old agent from installation or modification<br>There are different angles to this solution. Both legacy and AMA are installed from VM extensions which can be prevented if proper drift management is implemented.</li><li>We can further monitor or remove non compliances if we find an old agent policy exists above an AMA one. For active protection within the operating system, we can define definitions or look at the installation path to prevent or protect old/new agents from being installed or modified. Additional antivirus or security scanning tools would be the way to go.</li><li>Good to note that currently, Microsoft Defender for Cloud and Microsoft Sentinel support are still in public review.</li><li>See further here for their detailed implementation:<br><a href="https://learn.microsoft.com/en-us/azure/defender-for-cloud/auto-deploy-azure-monitoring-agent">Deploy the Azure Monitor Agent to protect your servers with Microsoft Defender for Cloud</a></li><li>Click here for the <a href="https://learn.microsoft.com/en-us/azure/sentinel/ama-migrate">Migration process for Microsoft Sentinel</a></li></ul><h4>d) Field expert troubleshoot and support</h4><ul><li>Make use of <a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/use-azure-monitor-agent-troubleshooter">Azure Monitor Agent troubleshooter</a> ​​<br>This is a command line executable designed to help diagnose issues with the agent, and general agent health checks. It can run checks to verify agent installation, connection, general heartbeat, and collect AMA-related logs automatically from the affected Windows or Linux VM.</li><li>Microsoft and Contino can help! We have all kinds of Azure devops, architects, development, security and migration experts and client engagement experiences. You can <a href="https://www.contino.io/contact-us">get in touch here</a> with the folk at Contino who will be only too happy to support you!</li></ul><h3>Summary</h3><p>With the current log analytics agent expiring within the year, now is the right time to be thinking about migration.</p><p>We have highlighted some of the potential risks, migration best practices, cost optimisation using DCR with filters and transformations, and preparations for changes to a successful migration journey. Start small, fail fast and repeat.</p><p>Let us know your thoughts and feel free to reach out! We are always here to help.</p><h4>Further reading or links that would come in handy</h4><p><a href="https://github.com/microsoft/AzureMonitorCommunity/blob/master/Azure%20Services/Azure%20Monitor/Agents/Migration%20Tools/DCR%20Config%20Generator/DCRGenerator-README.md">DCR Generator Github</a></p><p><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/agents/azure-monitor-agent-migration">Office Microsoft migration guidance from legacy log analytics agent</a></p><p><a href="https://medium.com/microsoftazure/tips-when-migrating-to-azure-monitor-agent-from-log-analytics-agent-ef05be4257c9">Additional tips on migration</a></p><p><a href="https://adisimon.medium.com/choosing-the-right-azure-monitor-agent-for-your-vms-4e94a624d344">Choosing the right Azure Monitor Agent for your VMs</a></p><p><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/best-practices-vm">Best practices for monitoring virtual machines in Azure Monitor</a></p><p><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/vm/monitor-virtual-machine-data-collection#control-costs">Monitor virtual machines with Azure Monitor: Collect data</a></p><p><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview">Data collection rules in Azure Monitor</a></p><p><a href="https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-transformations">Data collection transformations in Azure Monitor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=890f4f7f2101" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/azure-blog-azure-monitoring-agent-taking-over-log-analytics-agent-and-how-to-prepare-for-change-890f4f7f2101">Azure Blog — Azure Monitoring Agent taking over Log Analytics Agent and how to prepare for change?</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using AWS Services to Improve Developer Experience]]></title>
            <link>https://medium.com/contino-engineering/using-aws-services-to-improve-developer-experience-b15912fdbbae?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/b15912fdbbae</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[developer-experience]]></category>
            <category><![CDATA[devex]]></category>
            <dc:creator><![CDATA[Mark Faiers]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 15:16:58 GMT</pubDate>
            <atom:updated>2023-09-12T13:59:26.839Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8oAAw8iKHR7VNzQX" /></figure><p>Recently I wrote a post detailing <a href="https://www.contino.io/insights/dx-developer-experience">why developer experience matters</a>, focusing on the business benefits of building and maintaining a good developer experience. The post also detailed some concepts that any organisation can use to improve developer experience. In this post we’ll go into a bit more detail on how those ideas can be implemented using AWS services.</p><p>You’ll learn more about the various services purpose-built by AWS to enable great developer experience, and how each of them helps to realise business benefits and goals through improved developer experience. The services we’ll focus on in this post are <a href="https://aws.amazon.com/codewhisperer/">Amazon CodeWhisperer</a>, <a href="https://codecatalyst.aws/explore">Amazon CodeCatalyst</a>, <a href="https://aws.amazon.com/codeguru/">Amazon CodeGuru</a>, and <a href="https://aws.amazon.com/devops-guru/">Amazon DevOps Guru</a>, which combine to offer benefits across the SDLC, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*ovhFQTbj3L3ZOlDX" /></figure><h3>CodeWhisperer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*PMRnfULG0TUmHM6dDu1CTA.png" /></figure><p>Amazon CodeWhisperer is an Generative AI pair-programming tool. Its main function is to determine the intent of the code developers are trying to write and to provide suggestions on how to implement that intent. These suggestions are driven by providing comments to CodeWhisperer, and can be accepted with the press of a key on your keyboard.</p><p>Many integrated development environments (IDEs) and code editors have had auto-complete suggestion mechanisms that can help with finishing partially written lines of code for quite some time. However, CodeWhisperer takes that way, way further and it is now possible to reliably produce code without actually writing any of it yourself, using a technique known as ‘Prompt Engineering’.</p><p>You can see a demonstration of CodeWhisperer <a href="https://d1.awsstatic.com/codewhisper/cw_get_more_done_faster.3a9084876df23f3a367537ae2be18dc9718f42d3.mp4">here</a>.</p><p>This carries a number of benefits. It makes it orders of magnitude faster to write code, which in turn means that new features and products can be coded orders of magnitude faster. And not only that but it also opens up new business opportunities.</p><p>The flip side of faster development time is cost reduction. If products and features can be developed at a fraction of the cost things that were previously not viable suddenly become viable. CodeWhisperer also provides a potential competitive advantage to those organisation that use it, over those that don’t, because it allows them to reach the market much faster than their competitors, and produce higher quality features and products.</p><p>Of course CodeWhisperer isn’t a silver bullet, and must be used in conjunction with good practices and processes to reap its benefits. It isn’t much use being able to develop code very quickly if it takes a long time to define requirements, or to test and deploy the code being written. This is where some of the other AWS Builder Experience products come in handy.</p><h3>CodeCatalyst</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*WSYOPAHye4e4mk0jT0UIOg.png" /></figure><p>Amazon CodeCatalyst is a service that provides a number of popular development tools, such as git repositories, issue tracking, and CI/CD pipelines, all in one place. Uniquely for AWS services, it exists outside of the AWS account ecosystem (although you do need to link it to an account for billing purposes).</p><p>Having a unified development tool like CodeCatalyst carries a number of benefits. Firstly it makes it a lot easier to implement cross-functional teams. If business analysts (BAs), developers, test engineers, and devops engineers are all using the same tooling it is much easier for them to collaborate and share updates and other data and information. It is easier to track work, resolve issues and bottlenecks, and reduce handoffs. This is all key to bringing products and features to market faster and realising the value that they bring.</p><p>The use of a single tool also makes onboarding a lot easier. Having numerous tools, each, potentially, with its own onboarding process, user account management, and permissions sets makes it slower and more complex to onboard team members. While the impact of this is not overbearing for a single user, scaling this out across a large organisation can be.</p><h3>CodeGuru</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*JLuuM57T_z34S8nJD4RQMg.png" /></figure><p>Amazon CodeGuru provides automated security checking, optimisation recommendations, and code reviews. It also uses Machine Learning (ML) to identify the recommendations that it provides.</p><p>CodeGuru can be thought of as a ‘Feedback Loop’ service that can be added into your Continuous Integration (CI) process, and development workflow to improve the speed and quality of feedback that is generated.</p><p>Having worked on a number of development teams I have seen how much time can be lost waiting for another member of the team to provide feedback on code changes, colloquially known as a ‘code review’. Such code reviews also often require context switching by the reviewer. Mistakes are also often missed by human reviews.</p><p>Integrating CodeGuru into your code review process will allow reviews to take place faster, with a higher quality most of the time. This, in turn, will mean that code, and therefore features and services, can make it into production faster, and that less mistakes will become ingrained into the codebase, improving overall quality of products and services.</p><h3>DevOps Guru</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*qXBTykLk1WY7sHs9nif9Bw.png" /></figure><p>Monitoring and Observability of applications in production has, in recent years, been recognised as a critically important part of the software development lifecycle. Monitoring allows the teams building and maintaining the application to identify errors quickly, before they cause a big impact on customers.</p><p>DevOps Guru, with its Machine Learning capabilities takes this a step further. With standard monitoring processes, the team must still decide on what metrics are important to the running of the application, and also baseline those metrics. For example, if it is decided that CPU is an important indicator of how well the application is performing, then it must be decided what an acceptable utilisation, 70%? 80%? 90%? It is likely a certain amount of guesswork and refinement must be done. DevOps Guru takes a lot of that away. It uses Machine Learning to identify anomalies in the running of applications, and therefore identify when an issue is likely arising.</p><p>This allows for a number of things:</p><ul><li>Monitoring and Operations can be conducted more ‘passively’ and so teams can focus more on value generating activities, like adding new features, rather than trying to decide which metrics to monitor, what acceptable thresholds to set, and measuring and refining both of those things.</li><li>More production issues to be identified, and earlier identification of those issues. This equates to a better customer experience overall, as, provided those issues are fixed in a timely manner, less customers will suffer from a degraded experience.</li><li>Increased confidences in teams and organisations ability to build and run complex applications and cloud architectures therefore allowing for features and services that may otherwise be too risky</li><li>Reduced cost of operations, directly improving the organisations bottom line.</li></ul><h3>Putting Everything together</h3><p>While the services mentioned above, namely CodeWhisperer, CodeCatalyst, CodeGuru, and DevOps Guru offer many benefits to developer experience, and therefore, as we’ve shown, also to business outcomes they really shine when used in conjunction with one another. Developer Experience in 2023 is not just about writing code efficiently, but also, requirements, testing, deployment, monitoring, operations, and much more.</p><p>It is important, to achieve a great developer experience, that all of these areas are covered. It doesn’t matter how quickly and efficiently you can code new features if there is no easy way to test them and testing becomes a bottleneck. In this scenario business benefits are slower to be realised, and developers will likely be just as frustrated that it’s taking so long for their features to make it to production.</p><p>The tools we’ve spoken about provide value across different areas of the software development but their real power comes when used in conjunction with one another. CodeCatalyst helps with smooth onboarding, and defining a single pane of glass for code repositories, ticketing, test reports, and deployment pipelines. CodeWhisperer helps to write code quickly, more efficiently, and with higher quality. CodeGuru assists in validating code, both in terms of quality and efficiency, and security. And DevOps Guru takes a lot of the burden away from people managing monitoring and operations.</p><p>These tools provide features and techniques such as AI/ML, automated feedback loops, and integration that enable the potential for a great developer experience, and, by extension, deliver value to the business and its customers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b15912fdbbae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/using-aws-services-to-improve-developer-experience-b15912fdbbae">Using AWS Services to Improve Developer Experience</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knock Knock; Who’s there? 
Azure Front Door!]]></title>
            <link>https://medium.com/contino-engineering/knock-knock-whos-there-azure-front-door-920cffad2e3a?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/920cffad2e3a</guid>
            <category><![CDATA[azure-front-door]]></category>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[azure]]></category>
            <category><![CDATA[cdn]]></category>
            <dc:creator><![CDATA[Mark Nash]]></dc:creator>
            <pubDate>Thu, 07 Sep 2023 09:12:25 GMT</pubDate>
            <atom:updated>2023-10-02T10:39:49.271Z</atom:updated>
            <content:encoded><![CDATA[<h3>Knock Knock; Who’s there? <br><em>Azure Front Door!</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bHUc0xOxwWfKcShyVFebnA.jpeg" /></figure><h3>Introduction</h3><p>Imagine you have an online store that you want to share with people all over the world. You want everyone to have a smooth, fast and safe shopping experience, no matter where they are. The online store is hosted on Microsoft Azure, and you’re looking for a resource to make your website faster, more secure, and always available. In your search you stumble across ‘Azure Front Door’ … But what is it?</p><p>Azure Front Door is a globally dispersed Application Delivery Network (ADN). It is a Content Delivery Network (CDN), Web Application Firewall (WAF), Layer 7 Load balancer and much more, All in one Azure resource!</p><p>Microsoft is so confident in Azure Front Door that their own services such as Bing, Skype, LinkedIn, Microsoft 365 and even the Azure Portal all utilise the service to provide higher availability, lower latency, better scale, and more secure experience for their users.</p><p>With such a multifaceted resource it’s obvious there are many reasons you might decide to implement it in your environment, but let us talk through and explore some of these a little.</p><h4>It’s web traffic only from here on out</h4><p>A quick note at this point; if you’re looking for a solution to route traffic to your applications via protocols other than HTTP, HTTPS and HTTP/2 then you might as well stop here, pay your tab and look elsewhere. Azure Front Door only supports HTTP based traffic, and although this is supported over non standard ports it is the only protocol we’ll see pass through from here. This includes you WebSockets, have a look at Azure Application Gateways and Traffic Manager if you need other protocol support as Front Door is not the tool for you.</p><h3>Azure Front Door</h3><p>So let’s dive in through the front door and discover what Azure Front Door can provide you and your team, and why you should look at implementing it for your internet-facing HTTP(S) based applications!</p><h3>Tiers of Azure Front Door</h3><p>If you’ve been around the Azure landscape as long as I have then you might have seen some of the earlier iterations of Azure Front Door with a single tier. Since April 2022 Azure has been offering Standard and Premium tiers of Azure Front Door. The previous and already existing iterations of Azure Front Door have been retired and are now indicated as ‘classic’ resources. Although these classic resources will continue to be supported, they will miss out on future enhancements and latest capabilities.</p><p>In this blog, we’ll focus on the features of the Standard and Premium tiers, allowing you to use this information to either build a new instance or upgrade your current classic resource.</p><p>Let’s take a quick look at the differences between the available tiers. Microsoft has produced a nice <a href="https://learn.microsoft.com/en-us/azure/frontdoor/standard-premium/tier-comparison#feature-comparison-between-tiers">full feature comparison</a> between the Standard, Premium and classic resource tiers that I wouldn’t be able to beat. It’s worth a deeper look if you’re trying to decide which tier is right for you, but here we will look at the highlights.</p><h4><strong>Standard</strong></h4><p>Standard provides you with a wide range of features covering the core features of CDN, WAF, LB at a global scale, so you are not missing out on much if you decide to stay at a standard level. <br>Some highlights are:</p><ul><li>Static and Dynamic file and site delivery</li><li>Custom Domains</li><li>Origin Load balancing</li><li>Custom WAF rules</li><li>Advanced analytics, metrics, reports and logs (Access logs, WAF logs, Health probe logs)</li></ul><h4><strong>Premium</strong></h4><p>Along with everything that the standard level provides, the main benefits Premium gives you over and above the Standard Tier are:</p><ul><li>Microsoft Managed rule set</li><li>Bot Protection</li><li>Private Link Connection to Origin sources</li><li>WAF reports</li><li>Increase of Quota limits</li></ul><p>It’ll be down to your own requirements to decide which tier is correct for you, however as going down the tiers (Premium -&gt; Standard) is not currently supported, it is always advised to start on the Standard tier and only upgrade if you find you require the additional features.</p><h3>Placement</h3><p>So where does a service called ‘Azure Front Door’ sit in your infrastructure stack?</p><p>No surprises here, Azure Front Door sits at the start of your infrastructure estate as the first point of contact users will get when they are looking for your applications.</p><p>DNS for your Website is pointed at the Azure Front Door endpoint using a CName and TXT record. The CName record is to direct traffic to the Front Door endpoint address, the TXT record to confirm domain ownership.</p><p>Azure Front door can even provide an Azure managed TLS/SSL certificate for your site, allowing you to communicate on HTTPS without the additional overheads of purchasing a separate certificate! Sites that do not have certificate based authentication rank lower on search results, potentially reducing the amount of customers that are directed at your site. Certificate-less sites also go against industry best practice and risk compromise of the systems security. By having Azure Front Door manage the SSL/TLS certificate for your site you reduce management overhead renewing certificates and reduce risk of certificates being missed and taking your site offline.</p><p>Once a user is directed to Front Door, the Front Door instance checks the health of your application servers, the validity of cached content and ensures the request is not an attack. As long as all is good Front Door will direct the user to the relevant content or application servers (or origin). If an Application server or Region is unavailable Front Door can direct the request to a different healthy endpoint or even a custom error page.</p><p>All is Good! The user is in your App!</p><p>If you want to read the full routing architecture overview Microsoft explains that here : <a href="https://learn.microsoft.com/en-us/azure/frontdoor/front-door-routing-architecture?pivots=front-door-standard-premium">https://learn.microsoft.com/en-us/azure/frontdoor/front-door-routing-architecture?pivots=front-door-standard-premium</a></p><h3>Capabilities — Security</h3><p>All sounds great so far! However, what happens if this isn’t some nice user trying to purchase your wares, and is instead an attacker looking to steal information from your website?! (Oh no! We Don’t want that!)</p><p>Azure Front Door has Web Application Firewall (WAF) capabilities that help to protect your Sites and Applications from attacks from these pesky individuals.</p><p>WAF policies can deny attacks such as SQL Injection, Cross site scripting and other known HTTP or HTTPS attacks with known signatures from accessing your systems. Managed rule sets in the premium tiers protect many common attacks and from bots. You are also able to restrict specific Regions and /or IPs from accessing your sites.</p><p>If the attacker tries to batter your site with a Distributed Denial of Service (DDoS) attack, it’ll come to no avail as Front Door is guarded with DDoS protection that has been proven to protect Microsoft’s enterprise and customer services from large-scale attacks. (Not this time attacker!)</p><p>Alongside the active protection you put in place when using Azure Front Door and WAF, some of the security best practices that Front Door can enable are:</p><ul><li>Limiting externally accessible endpoints by restricting traffic to origins from Front Door</li><li>End to End TLS encryption</li><li>HTTP to HTTPS redirects</li></ul><p>So you are already better off by using Front Door from a security perspective, but let us continue to look at other reasons Front Door is worth your time.</p><h3>Capabilities — CDN</h3><p>Your storefront is going to have many images and content pieces that your customers will want to load so they understand what you are selling. These requests could go to your application servers every time, but the images and content won’t change that often meaning lots of repeated requests. Wouldn’t it be better to get customers looking at as much content as possible in the shortest period of time by speeding their requests up and reducing load on the application servers?</p><p>Content Delivery Networks (CDN) utilise caching to save standard requests and responses without having to send and wait for requests to go to and from the Origin servers. Caching these standard requests helps to speed up overall site resolution by hosting the content closer to the user, and it reduces the amount of traffic sent to origin servers by removing the duplicated requests for assets allowing application servers to focus on more intensive requests. <br>Static assets such as images, CSS and JavaScript files benefit most from caching. Dynamic assets should not be cached for many reasons such as data freshness, personalization, security and privacy. These dynamic assets should be separated and routed directly to the origin servers.</p><p>Speed is not the only reason having CDN capabilities is helpful, it can also help your applications manage peaks in traffic and demand.</p><h3>Capabilities — Scalability and Availability</h3><p>Christmas, Black Friday, spectacular deals or just the peaks and troughs of a standard day — there are many reasons your store or application will see high traffic and demand. There will also be times when disaster strikes and issues arise in your application that are caused by extreme weather, natural disasters or just standard accidents. It would be crushing if your application could not handle the pressure of these high demand scenarios and you lost out on that custom, or your application became unavailable in these disasters.</p><p>Azure Front Door is a globally hosted service which delivers content using Microsoft’s global network of hundreds of Points of Presence (PoPs) distributed across the world. This allows you to connect customers to your content with minimal latency, utilising Front Door’s CDN capability to store content close to users rather than having to be constantly transferred across regions.</p><p>It also enables you to handle higher demand more easily, providing many PoPs to scale and handle peak traffic loads, along with the CDN removing the repeated requests for content.</p><p>Being a globally distributed service also helps to increase the availability of your application. If an issue arose in a core region for your users they could be directed to a working region, continuing to allow them to access your applications with minimal disruption. Using Azure Front Door alongside geographically redundant application designs enables applications to handle disasters with ease.</p><h3>Capabilities — Layer 7 Load Balancing</h3><p>Having multiple instances of your store makes sense, whether it’s a physical bricks and mortar shop, or an application hosted in the cloud. Designing for resilience in this manner you can protect from incidents causing a loss of business by directing traffic to a different one of your stores if one has issues. This is exactly the same in a web application environment, we want to ensure business continuity in the event an instance becomes unavailable, so we need to direct traffic to a different location or instance.</p><p>This is where Front Door’s layer 7 load balancing comes into play, Front Door can keep an eye on application health and direct users to a working instance in the same region, or even another region if the application is set up to support it.</p><p>Sometimes applications will have specialisations just like your highstreet shop, and so you can do path based routing to direct specific requests to certain application backends. Maybe you have a different application handling payment requests than hosting your main website. Requests to /payment/* can be directed to this application to resolve.</p><p>By using Front Door you can direct traffic to healthy instances and specify the correct paths for users to traverse, helping to keep applications performant and reduce errors for the users.</p><h3>Capabilities — Infrastructure as Code (IaC)</h3><p>Whether it’s in a store or IT environment, having individuals implement changes to high importance systems necessitates care, which tends to slow the whole process down, and even when individuals are careful it doesn’t mean all mistakes are caught before tasks are deemed complete. Managing IT infrastructure in a manual way has fallen out of favor with IT professionals for these (and many more) reasons and so they now look to IaC to configure and manage their resources.</p><p>Infrastructure as Code (IaC) is an Industry standard for managing infrastructure. By defining the configuration of infrastructure assets as code, IT professionals can store, duplicate, run tests and gain confidence in the configurations, ensuring that when they are implemented to the high importance systems such as production level instances of Azure Front Door, that they are sure nothing will break.</p><p>Azure Front Door is available to be managed via a multitude of Infrastructure as code tools varying from Command line and scripting tools such as PowerShell and the Azure CLI, to templating tools such as ARM templates, Bicep and Terraform.</p><p>In the IaC deployments of <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/frontdoor">Classic Azure Front Door with Terraform</a>, the Front Door Azure resource was a single terraform resource with many configuration blocks to allow you to configure the frontends, routes, backends and more. <br>Being a single resource meant the Terraform configuration files for Front Door would be single long files that weren’t particularly easy to read or maintain and due to the complicated nature of the Front Door Resource these files would be hundreds of lines long to capture the full configuration of the resource.</p><p>With the move to the new Standard and Premium tiers of Azure Front Door resource comes a new set of <a href="https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cdn_frontdoor_endpoint">Terraform resources</a> for Azure Front Door.</p><p>These new resources break the original monolithic terraform files down into smaller chunks allowing them to be split over multiple files and even multiple configuration deployments. This makes management and understanding a lot clearer, it allows changes to be made in smaller increments and overall is a better way to manage the resource.</p><p>We would always advise having a test environment that is architected like the production environment as this enables you to test changes you want to make to production without affecting the user base. By using IaC it is easy to be able to duplicate the code, rename resources and then deploy a like-for-like instance to do testing on. If you are concerned about costs then these environments don’t have to stick around as the IaC enables you to spin them up and down as and when needed.</p><p>Microsoft provides <a href="https://learn.microsoft.com/en-us/azure/frontdoor/create-front-door-terraform?tabs=Portal">quickstart guides</a> for deploying Azure Front Door via Terraform, ARM template, Bicep, CLI, PowerShell and the portal.</p><p>You can get started easily and deploy Azure Front Door and test out what it can do for you.</p><h3>In Summary</h3><p>Whatever your HTTP(S) application is showing or selling to the world, it is important that your customers are kept safe, served quickly, consistently and want to come back and use your service again.</p><p>Azure Front Door helps you with these aims by providing industry leading WAF Protection, CDN request acceleration and Global availability and scalability to help ensure your customers always come away from your application happy.</p><p>With Microsoft’s quickstart guides you can easily setup an example instance of Azure Front Door to have a play and find out if it’s the right tool for you.</p><p>If you still need help…..well you know where we (Contino) are 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=920cffad2e3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/knock-knock-whos-there-azure-front-door-920cffad2e3a">Knock Knock; Who’s there? 
Azure Front Door!</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unpacking Google Next 2023: My Top 10 Insights & Takeaways]]></title>
            <link>https://medium.com/contino-engineering/unpacking-google-next-2023-my-top-10-insights-takeaways-3a502a143e8e?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/3a502a143e8e</guid>
            <category><![CDATA[google-container-engine]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-cloud]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Jaroslav Pantsjoha]]></dc:creator>
            <pubDate>Tue, 05 Sep 2023 08:15:26 GMT</pubDate>
            <atom:updated>2023-09-05T08:28:58.120Z</atom:updated>
            <content:encoded><![CDATA[<h3>Google Next 2023 has wrapped up, and once again, it was a whirlwind of innovation, insights, and announcements. The conference indeed focused on ‘all-things-AI’, as expected. Fortunately, here’s my easy-to-digest top 10 list.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*AgbOIu0RKE09e9Fbq7j4mQ.png" /><figcaption>Super exciting, fun, and very busy Google Next ’23 in San Francisco Moscone Centre</figcaption></figure><h3>Entree</h3><p>I typically intro the topic with <em>“what, where and why”</em>, but this occasion I am prescribing this blog post for someone who’s knows a fair bit about cloud technologies, Google Cloud as a hyperscaler to start with.</p><p>Now, If I had to summarise the Event with One Meme, then I have this…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*u6Awz_oKTi8EtIcJkmkk9w.jpeg" /><figcaption>“Do-it” \w AI</figcaption></figure><h3>The Main Event</h3><p>Once again, The Google Cloud Next ’23 was a significant event that showcased Google Cloud’s commitment to innovation and the future of cloud computing. The conference highlighted the importance of sustainability, with Google Cloud aiming to be the world’s cleanest cloud provider, and introduced the “Sustainable Cloud Challenge”, encouraging developers to build eco-friendly applications.</p><p><strong>“This Time It Was Different”</strong>— The event packed quite a punch being full-on immersive “<strong><em>AI-value-add this</em></strong>”, and “<strong><em>AI-fasttrack-augment-this</em></strong>” emphasising <strong>the role of AI in modern businesses</strong>, with the introduction of <strong>Duet AI</strong>, a generative AI model designed to collaborate with users in real-time. Sitting in my seat at the theatre, this was the most impressive “a-ha! moment”.</p><p>Google Business <strong>Workspace</strong> + (Duet) <strong>AI</strong> + (power-of) <strong>Google</strong> <strong>Cloud</strong> Enterprise made so much more sense, particularly as per <strong>live demo of AppSheet+VertexAI integration.</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fj4ckgMyt-1M%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dj4ckgMyt-1M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fj4ckgMyt-1M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e5b119cddd57c2557fd66976ee8c6d0/href">https://medium.com/media/3e5b119cddd57c2557fd66976ee8c6d0/href</a></iframe><p>This AI-driven approach is set to revolutionise various sectors, from customer service to software development. <a href="https://cloud.withgoogle.com/next/generative-ai#get-started-with-ai">Watch Sessions on-demand</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TpKl7t17gtwaAocW4AmDtA.png" /></figure><h4>Overall Trends and Innovations</h4><p>It came as not surprise that <strong>AI</strong> was at the forefront of Google Cloud’s Strategy. This annual Google Cloud Conference event in San Francisco was dominated by a plethora of AI announcements. This included enhancements to its infrastructure to better support AI, an expansion of AI partnerships, improvements to its Vertex AI Platform, and updates to its generative tool Duet AI, which has been now integrated into Workspace.</p><p>It was the sheer volume and pace of these announcements that really underscored <strong>AI</strong> as the <strong>topmost</strong> <strong>priority</strong> for Google Cloud. The keynote, led by Alphabet and Google CEO Sundar Pichai and Google Cloud CEO Thomas Kurian, emphasized the transformative potential of AI, touching upon every sector, industry, and business function.</p><p>The message was clear: <strong><em>AI is not just a feature but a profound shift that will redefine how we live and work.</em></strong></p><p>Beyond the technical announcements, the conference also showcased practical applications of the AI technologies, bringing to life the tangible <strong>Real-world Applications of AI</strong>.</p><p><strong>Demo one:</strong> One notable demonstration was the use of generative AI to enhance the customer experience at the DMV (Department of Motor Vehicles), known for its less-than-stellar service. Through the integration of conversational AI, customers could easily navigate the DMV’s services, ask questions, and even book appointments, all facilitated by generative AI.</p><p><strong>Demo two:</strong> Another demonstration highlighted the capabilities of Duet AI within Google Drive, where it could autonomously create presentations based on user prompts. Furthermore, Duet AI’s integration with Google Meet showcased its ability to take notes, summarize meetings, and collate action items, enhancing productivity during virtual meetings.</p><p>These real-world applications underscore the transformative potential of AI, not just as a technological marvel but as a tool that can significantly improve everyday tasks and business operations.</p><p>In short, “<strong>I’m very excited about this space”</strong></p><h4>My notes on Security Announcements</h4><p>At the heart of the security announcements was — once again — the introduction of Duet AI in Google Cloud. This AI collaborator provides generative AI-powered assistance to cloud defenders. <a href="https://www.youtube.com/watch?v=xmls-Cd-WhI&amp;t=1s"><strong>Duet AI has been integrated into key products like Mandiant Threat Intelligence and Chronicle Security Operations</strong></a>.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fxmls-Cd-WhI%3Fstart%3D1%26feature%3Doembed%26start%3D1&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dxmls-Cd-WhI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fxmls-Cd-WhI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/abd22374b8c8a0b55b47ccc815d6fec1/href">https://medium.com/media/abd22374b8c8a0b55b47ccc815d6fec1/href</a></iframe><p>Duet AI helps in <strong>summarising</strong> threat intelligence, <strong>transforming</strong> threat detection, and providing clear summaries of security cases. Additionally, Duet AI in the Security Command Center offers <strong>near-instant analysis of security findings, helping teams stay ahead of potential threats</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*6glwQvA06vaoiGWvWGjA9A.jpeg" /><figcaption>Meeting the <a href="https://www.linkedin.com/feed/hashtag/?keywords=googlecloudcertified&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7102735315869544448">#googlecloudcertified</a> community legends in the <a href="https://www.linkedin.com/feed/hashtag/?keywords=certifiedlounge&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7102735315869544448">#CertifiedLounge</a> 💪👌🦸‍♂️<br><a href="https://www.linkedin.com/in/ACoAAAACgUIBzLd3YNffMjvZrlXLB1x46ve_rxw">Walter Lee</a> 🐳Jai Campbell <a href="https://www.linkedin.com/in/ACoAAB4zHoUBKsH6dN3MmnoEaYrWTJDZceoLQRk">⎈</a> <a href="https://www.linkedin.com/in/ACoAABWJGEMBPeFY6Jsh9FD9gDZo0xXgmvgEkT8">Natalie McHugh</a><br><a href="https://www.linkedin.com/feed/hashtag/?keywords=googlecloudnext&amp;highlightedUpdateUrns=urn%3Ali%3Aactivity%3A7102735315869544448">#googlecloudnext</a></figcaption></figure><p>Let’s dive into my list of notable updates and announcement I jotted down, while running from one session to another and catching up with great folks face to face, for the first time in…. a really really long time. (2019?)</p><h3>My Top 10 Updates and Announcements:</h3><ol><li><strong>Supercharging Security with Duet AI</strong> <br>Google Cloud is addressing security challenges by enhancing security with Duet AI. This AI collaborator provides generative AI-powered assistance to cloud defenders, aiding in threat detection, investigation, and response. <a href="https://cloud.google.com/blog/products/ai-machine-learning/duet-ai-in-google-cloud-preview">Read more about Duet AI</a>.<br>PS There is an All N<a href="https://cloud.google.com/learn/certification/guides/cloud-security-engineer"><strong>ew Google Security Exam Available</strong></a></li><li><strong>Mandiant Hunt for Chronicle</strong><br>This new feature provides continual threat hunting by Mandiant experts on Chronicle data, exposing attacker activity and reducing business impact. <a href="https://cloud.google.com/blog/products/identity-security/introducing-mandiant-hunt-for-chronicle">Learn about Mandiant Hunt for Chronicle</a>.</li><li><a href="https://cloud.google.com/blog/products/identity-security/rsa-introducing-ai-powered-risk-summaries-security-command-center"><strong>Enhancements in Security Command Center </strong></a>(Upgrades!)<br>Google Cloud is integrating agentless vulnerability scanning, powered by Tenable, to detect vulnerabilities on Google Compute Engine virtual machines. Additionally, they’ve introduced Cloud Firewall Plus and Network Service Integration Manager to bolster network security.</li><li><strong>Confidential Computing </strong>(more of)<br>Google Cloud announced the private Preview of Confidential Computing running on 4th Gen Intel Xeon Scalable CPUs with TDX technology, ensuring cryptographic isolation from other workloads and cloud provider access.</li><li><strong>Google </strong><a href="https://workspace.google.com/blog/product-announcements/duet-ai-in-workspace-now-available"><strong>Workspace with Generative AI</strong></a><br>Google Workspace is now supercharged with AI-powered security and digital sovereignty controls, ensuring a safer way to work and protect users from cyber threats.</li><li><strong>Duet AI in Looker — BusinessIntelligence</strong><br>Duet AI (Duet expands across the cloud ecosystem now) <a href="https://cloud.google.com/blog/products/application-modernization/introducing-duet-ai-for-google-cloud">has been <strong>integrated into Looker</strong> and is currently in preview</a>. This feature will enhance the data visualization capabilities of Looker, making it easier for businesses to derive actionable insights from their data.</li><li><strong>Cross-Cloud Network Platform</strong><br><a href="https://cloud.google.com/solutions/cross-cloud-network">Google announced the Cross-Cloud Network</a>, an open and programmable global cloud networking platform. This platform aims to enhance connectivity and integration across various cloud environments. Additionally, the Private Service Connect now supports over 20 different Google and partner managed services, ensuring seamless integration and communication between services. Read about <a href="https://cloud.google.com/network-connectivity/docs/interconnect/concepts/cci-overview">Cross-Cloud Interconnect</a> in particular.</li><li><strong>Enhancements in Networking<br></strong>Google introduced several new features to improve networking capabilities. The Cloud Firewall Plus tier, which is a cloud-first next-generation firewall, is now in preview. Additionally, the new Network Service Integration Manager simplifies the setup and operation of partner NGFWs from leading providers. VPC Spokes support in Network Connectivity Center is also in preview, allowing for enhanced VPC connectivity. And my personal favourite — <a href="https://cloud.google.com/load-balancing/docs/mtls">Global ALB supporting mTLS Client-Side Auth.</a></li><li><strong>GKE Enterprise is here!<br></strong>It’s a premium edition of Google Kubernetes Engine (GKE), designed to help businesses run their most critical workloads with increased efficiency across multiple teams. It includes features like a new; <strong>multi-cluster feature (“fleets”)</strong>, <strong>advanced security insights, governance controls, and a managed service mesh</strong>. GKE Enterprise also supports hybrid and multi-cloud environments, allowing businesses to run container workloads anywhere. Yes — There is <a href="https://cloud.google.com/blog/products/ai-machine-learning/duet-ai-in-google-cloud-preview">Duet AI for GKE</a> as well :)</li><li><strong>TPU Support in GKE!<br></strong><a href="https://cloud.google.com/blog/products/containers-kubernetes/whats-new-with-gke-at-google-cloud-next">TPU support in GKE: a catalyst for AI success</a>.<br>GKE has expanded its support for AI-optimized hardware. Specifically, it now supports the <strong>AI-optimized Cloud TPU v5e</strong>. This new TPU can scale to tens of thousands of chips, making it ideal for developing more complex AI models. Additionally, GKE has added GA support for A3 VM with NVIDIA H100 GPU and made Google Cloud Storage <a href="https://cloud.google.com/blog/products/storage-data-transfer/cloud-storage-fuse-now-ga">FUSE generally available on GKE</a>.</li><li>Ok, Last one, promise — <strong>Digital Sovereignty Offerings</strong> (More regions)<br>Google Cloud continues to invest in digital sovereignty offerings, announcing Assured Workloads Japan Regions, which supports data residency in Japanese regions.</li></ol><blockquote>The Government of <a href="https://www.googlecloudpresscorner.com/2023-08-29-Google-plans-to-establish-operations-in-El-Salvador">El Salvador announced a multi-year deal with Google Cloud</a> to digitally transform the country with Google Distributed Cloud (GDC) Edge, as well as to establish an office presence and Center of Excellence (CoE) in the country.</blockquote><h3>Closing Thoughts</h3><p>The Google Next 2023 Conference was a testament to Google Cloud’s commitment to driving innovation in the cloud space.</p><p>And we’re clearly living in the new era of business and digital transformation, with AI supercharged solutions, accelerators.</p><p>The focal point is that this <strong>“Duet AI”</strong> capability is being embedded across the Google Cloud ecosystem of products, thus fast tracking solutions, delivering real tangible accelerators across a spectrum of industries. This will be transformational in it’s own right, and you really ought to get started with the courses and learning to ensure you’re not left behind — <a href="https://www.cloudskillsboost.google/journeys/17">the ML Engineer Learning Path (by Google Cloud) is one such good one stop shop.</a></p><p>See you in the Cloud!<br>Until Next Time,<br>JP</p><p><em>(All The Content and Opinions are my own)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a502a143e8e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/unpacking-google-next-2023-my-top-10-insights-takeaways-3a502a143e8e">Unpacking Google Next 2023: My Top 10 Insights &amp; Takeaways</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dirty Coding: a Brief Overview of Coding with LLMs]]></title>
            <link>https://medium.com/contino-engineering/dirty-coding-a-brief-overview-of-coding-with-llms-7ee937e5863d?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/7ee937e5863d</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[developer-tools]]></category>
            <dc:creator><![CDATA[Byron Allen]]></dc:creator>
            <pubDate>Wed, 12 Jul 2023 14:13:50 GMT</pubDate>
            <atom:updated>2023-07-12T14:13:50.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eqSsHUE2S4YTtHYlVYJ1Ew.png" /></figure><p>Much has been said about the creative embers kindled with Generative AI (GenAI) — and more specifically the latest Large Language Models (LLMs). One such ember is:</p><blockquote>When an unexpected glitch occurs in CodeWhisperer’s system, ChatGPT steps in to help, offering to support the AI in a complex coding challenge at an international coding symposium, despite CodeWhisperer’s lack of experience in this “dirty coding” realm.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ksjyTLpvcbC6aCe3VEssA.png" /><figcaption><a href="https://openai.com/dall-e-2">Dall-e 2</a> prompt: two people on hands and knees facing each other like in ‘dirty dancing’ the movie</figcaption></figure><p>Embers like this, however bright they glow, represent a fictional response. Here we see creative variation that can be overwhelmingly convincing and captivating — sometimes even terrifying. This is one side of the coin that is an LLM response.</p><p>What’s on the other side? A <strong><em>factual, actionable</em></strong> output.</p><p>Bear in mind though, that assumes facts are being produced. And in case you didn’t know, LLMs have no concept (rather I should say construct!) to classify truth. That is why you should have a healthy skeptical view of their output.</p><p>In other words, you can use LLMs to generate fiction for sure, but not necessarily fact for sure. That said, there is one domain where GenAI is increasingly becoming less fiction and more fact — coding.</p><p>I’ll walk through a brief experiment I ran to compare three LLMs that generate code and cover the pros and cons I observed. However, something that becomes quite apparent is that even if the focus is on creating factual and actionable outputs, they can still be subject to preference, which can be a bit messy, requiring experience to navigate.</p><h3>A Brief Experiment of My Own</h3><p>I took some inspiration from <a href="https://arxiv.org/abs/2304.10778"><em>Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT</em></a>, which is worth a read.</p><p>The highlight for me is that ChatGPT gets code correct out of the box more frequently than its peers (in this study). However, time taken to correct this generated code (i.e. the technical debt’ in this scenario) is higher with ChatGPT. It’s great to see academic research like this, as it complements what companies such as <a href="https://github.com/customer-stories/duolingo">Duolingo</a> have said about the benefits of using Copilot.</p><p>That said, I like running my own experiments — even if they are quick and dirty.</p><p>Doing your own reviews of these tools are critically important given the high-speed evolution of GenAI from one day to the next (e.g. the announcement of <a href="https://help.openai.com/en/articles/6825453-chatgpt-release-notes">OpenAI’s Code Interpreter going to beta</a>), and the highly contextual nature of what value they deliver to you. I would encourage you, dear reader, to do the same.</p><p>In this post, I highlight my experiments with <a href="https://openai.com/blog/chatgpt">OpenAI ChatGPT</a>, <a href="https://aws.amazon.com/codewhisperer/">AWS CodeWhisperer</a> and <a href="https://cloud.google.com/vertex-ai/docs/generative-ai/code/code-models-overview">Google Cloud Codey</a>. Notably, I think there is opportunity to use these tools together, leveraging their strengths to augment their counterparts’ weaknesses, and using multiple models for validation.</p><p>Sound a touch messy? Welcome to <em>dirty </em>coding…</p><h3>Comparing Three LLMs</h3><p>If you’re going to compare machine learning (ML) models you need consistent tests across all cases i.e. the same set of prompts used across all LLMs. I used the following <a href="https://en.wikipedia.org/wiki/Zero-shot_learning">zero-shot</a> prompts to get the best sense of how these models work out of the box:</p><blockquote><em>Create a function that moves a file to folder.</em></blockquote><blockquote><em>Create a function that reads two data frames and returns a new data frame with the two data frames merged.</em></blockquote><blockquote><em>Create a function that uploads a file to a S3 bucket.</em></blockquote><p>Let’s get into the output.</p><h4>OpenAI ChatGPT (GPT-4)</h4><pre># create a function that moves file to folder<br><br>import os<br>import shutil<br><br>def move_file_to_folder(file_path, destination_folder):<br>    try:<br>        if not os.path.isfile(file_path):<br>            print(f&quot;{file_path} does not exist.&quot;)<br>            return<br>        if not os.path.isdir(destination_folder):<br>            os.makedirs(destination_folder)<br>        shutil.move(file_path, destination_folder)<br>        print(f&quot;Successfully moved {file_path} to {destination_folder}.&quot;)<br>    except Exception as e:<br>        print(f&quot;An error occurred while moving {file_path} to {destination_folder}. Details: {str(e)}&quot;)<br><br># Example usage:<br># move_file_to_folder(&#39;path_to_your_file&#39;, &#39;path_to_destination_folder&#39;)<br><br><br><br><br># create a function that reads two data frames and returns a new data frame with the two data frames merged<br>import pandas as pd<br><br>def combine_dataframes(df1_path, df2_path):<br>    try:<br>        # Read dataframes from CSV files<br>        df1 = pd.read_csv(df1_path)<br>        df2 = pd.read_csv(df2_path)<br>        <br>        # Concatenate dataframes vertically<br>        combined_df = pd.concat([df1, df2])<br>        <br>        # Reset index for the new combined dataframe<br>        combined_df.reset_index(drop=True, inplace=True)<br>        <br>        return combined_df<br>    except Exception as e:<br>        print(f&quot;An error occurred: {str(e)}&quot;)<br><br># Example usage:<br># combined_df = combine_dataframes(&#39;path_to_your_first_dataframe.csv&#39;, &#39;path_to_your_second_dataframe.csv&#39;)<br><br><br><br><br># create a function that uploads a file to a S3 bucket<br>import boto3<br><br>def upload_to_s3(bucket_name, file_name, object_name=None):<br>    # If S3 object_name was not specified, use file_name<br>    if object_name is None:<br>        object_name = file_name<br><br>    # Instantiate the S3 client<br>    s3_client = boto3.client(&#39;s3&#39;)<br><br>    try:<br>        response = s3_client.upload_file(file_name, bucket_name, object_name)<br>    except Exception as e:<br>        print(f&quot;An error occurred: {str(e)}&quot;)<br>        return False<br>    return True<br><br># Example usage:<br># upload_to_s3(&#39;your_bucket_name&#39;, &#39;path_to_your_file&#39;)</pre><p>It’s hard to fault what you get out of the box with GPT-4. Overall, I felt this code was good. For example, in the first prompt, it caters for a non-existant file path and attempts to implement exception-handling. It even gives an example usage comment.</p><p>For the second prompt, the output chooses to combine the two dataframes through a union (to use SQL-lingo), and then very handily follows that up by resetting the index. Very nice, but what if I wanted to join my dataframes instead? You could use the pd.concat()function, but how you do so isn’t really clear.</p><p>This highlights the need to be more specific with word choices in the prompt, which I was not.</p><p>Verdict? Very nice, but no chef kiss.</p><h4>AWS CodeWhisperer</h4><pre># create a function that moves file to folder<br>def move_file(file_name, folder_name):<br>    <br>    # 2.) check if folder exists<br>    if not os.path.exists(folder_name):<br>        os.makedirs(folder_name)<br><br>    # 3.) move file to folder<br>    os.replace(file_name, folder_name)<br><br>    ############################<br>    # CODE WHISPERER FAILED HERE<br>    # why? could reason the word &#39;to&#39; when it should have been &#39;into&#39;<br>    ############################<br><br><br><br><br># create a function that reads two data frames and returns a new data frame with the two data frames merged<br>def merge_data_frames(df1, df2):<br>    return pd.merge(df1, df2)    <br><br><br><br><br># create a function that uploads a file to a S3 bucket<br>def upload_file_to_s3(file_name, bucket):<br>    s3_client = boto3.client(&#39;s3&#39;)<br>    response = s3_client.upload_file(file_name, bucket, file_name)<br>    return response</pre><p>One of the first things you may notice is the brevity of the code. Like Copilot, CodeWhisperer integrates into VSCode, effectively providing an auto-complete function while you type code — hence the less robust response. The intended use doesn’t appear to promote an output that is extensive, which it can be in GPT-4.</p><p>In prompt 2, I like that the response provides an alternative to pd.concat() It shows a fundamental reality of coding — that there is often more than one solution to a problem.</p><p>The selected pd.merge() serves as join but cannot serve as a union, so to some extent it’s less versatile than pd.concat() — and I use the versatile instead of appropriate as pd.concat() can be considered an unsafe operation (particularly for a join).</p><p>Verdict? I don’t think CodeWhisperer is a good starting point for a project. As the complexity of a prompt goes up CodeWhisperer becomes a less viable solution (for now). By design, it’s functional mode is that of a high-end code autocomplete, which I can see as being useful while in the thick of coding, facing a mental block or drafting code comments.</p><p>Worth noting, it is great to see integration into VSCode (even if a touch clunky at times).</p><h4>Google Cloud Codey (Palm-2 Bison)</h4><pre># create a function that moves file to folder<br>def move_file(file_name, new_folder):<br>  &quot;&quot;&quot;Moves a file to a new folder.<br><br>  Args:<br>    file_name: The name of the file to move.<br>    new_folder: The path to the new folder.<br>  &quot;&quot;&quot;<br><br>  # Get the current directory.<br>  current_directory = os.getcwd()<br><br>  # Move the file to the new folder.<br>  shutil.move(os.path.join(current_directory, file_name), new_folder)<br><br><br><br><br># create a function that reads two data frames and returns a new data frame with the two data frames merged<br>def merge_data_frames(df1, df2):<br>  &quot;&quot;&quot;Merges two data frames into a new data frame.<br><br>  Args:<br>    df1: The first data frame to merge.<br>    df2: The second data frame to merge.<br><br>  Returns:<br>    The merged data frame.<br>  &quot;&quot;&quot;<br><br>  # Merge the data frames on the common column.<br>  merged_df = pd.merge(df1, df2, on=&#39;common_column&#39;)<br><br>  # Return the merged data frame.<br>  return merged_df<br><br><br><br><br># create a function that uploads a file to a S3 bucket<br>def upload_file_to_s3(file_name, bucket_name, key_name):<br><br>    # Create a boto3 client<br>    s3 = boto3.client(&#39;s3&#39;)<br><br>    # Open the file to be uploaded<br>    with open(file_name, &#39;rb&#39;) as f:<br>        data = f.read()<br><br>    # Upload the file to S3<br>    s3.put_object(<br>        Bucket=bucket_name,<br>        Key=key_name,<br>        Body=data,<br>    )</pre><p>Doc strings! I didn’t expect to see them in each function, but they are there and I feel this is a surprisingly good result. I hope they iteratively improve on this by expanding on the verasitey of best practices, such as this, that could/should be present by default.</p><p>For prompt 2, it’s nice to see that more parameters were used in the function pd.merge(), which I think needs to be expanded on if they want this kind of technology to be educational. To me this point highlights the sentiment that this technology class is great if you’re currently a skilled developer, but if you’re not, I’m not convinced this type of technology is a good educational tool… which it could be.</p><p>Prompt 3, was written quite differently to the other two. I assume that might be due to the data that was trained on and it might be skewed towards Google Cloud dependencies (making an assumption that is not validated there).</p><p>Chef kiss for doc strings!</p><h3>What is the Benefit by the Numbers?</h3><p>The previous section represents a qualitative analysis. For those more interested in a qualitative analysis, gain that research paper I mention is good, but also, I’ve added this section.</p><p>Before I get into my own brief quantitative analysis, I want reference some metrics from another entity — Duolingo — which has some nice figures that I believe are worth pointing out:</p><blockquote>25% increase in developer speed with GitHub Copilot</blockquote><blockquote>1min setup time for largest repo with Codespaces</blockquote><blockquote>67% decrease in median code review turnaround time</blockquote><blockquote>70% increase in pull requests</blockquote><p>In my personal experience, I spent some time to develop a mobile application using the <a href="https://flutter.dev/">Flutter</a> framework (beyond the simple 3-prompt test above).</p><p>I’m not a frontend or mobile developer, and I’ve only used Flutter once (more than 2 years ago). With that in mind, I knew how long it would take me to make this application (roughly 8 hours). However, in about an hour or two, I was able to create it using <a href="https://en.wikipedia.org/w/index.php?title=Few-shot_learning_(natural_language_processing)&amp;redirect=no">few-shot learning</a> in ChatGPT. The app worked with minimal adjustments required to the code.</p><p>From that perspective, I have it in my mind that this technology <strong>increases development time somewhere around 4x to 8x</strong>. Truly a substantial gain, but probably more possible as I am an experienced developer and <em>not </em>working in a dev-test-prod environment, which I think helps contextualise the metrics that Copilot highlights.</p><h3>Conclusion</h3><p>GenAI is having a great moment, and LLMs for code development are certainly part of that magic of 2023.</p><p>It is not without hype though. The above outputs, although wonderful, are also temperamental, only partially including best practices and realistically requiring some prior experience in development and and implementation of such code.</p><p>I think we will enter an era of ‘Dirty Coding’ (if we’re not already there). In particular, I think there is a need to leverage what we see in ChatGPT and Codey — large veins of ‘well’ structured code alongside AI-enabled code-complete like what we see in CodeWhisperer. I believe this will help us work towards a more robust form of development, particularly if we start to encourage those who are new to development to pick up these AI tools.</p><p>On that note, these technologies are not far enough down the road to start making rash hiring decisions until they have been thoroughly vetted in existing teams — it would be wise to test and learn as a form of evolving with these new tools as they will undoubtedly evolve quickly as well over the next several years. There will be extension hacks like <a href="https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt">Code GPT</a>, which more closely embedded OpenAI’s API into VSCode albeit without auto-rendering code. Moreover, there will be developments in models like the jump from GPT-3 to GPT-4, and there may be legal action that changes what we can be used.</p><p>So, let the dirty coding commence!</p><h3>OpenAI’s Code Interpreter</h3><p>At the time of drafting this, OpenAI announced the beta release for Code Interpreter. How does it impact what I’ve mentioned above? First, let me take you through the quick experiment I ran. I used the prompt <strong>Create a faux dataset about tv shows. Add it to a pandas dataframe</strong>, which generated:</p><pre>fig, axs = plt.subplots(2, 2, figsize=(20, 20))<br><br># Distribution of shows by Genre<br>sns.countplot(data=df, y=&#39;Genre&#39;, ax=axs[0, 0])<br>axs[0, 0].set_title(&#39;Distribution of shows by Genre&#39;)<br><br># A timeline of the number of shows that first aired each year<br>sns.histplot(data=df, x=&#39;First_Aired&#39;, bins=30, ax=axs[0, 1])<br>axs[0, 1].set_title(&#39;Timeline of the number of shows that first aired each year&#39;)<br><br># A bar chart showing the number of shows each network has<br>sns.countplot(data=df, y=&#39;Network&#39;, ax=axs[1, 0])<br>axs[1, 0].set_title(&#39;Number of shows each network has&#39;)<br><br># The average number of seasons by Genre<br>df.groupby(&#39;Genre&#39;)[&#39;Num_Seasons&#39;].mean().plot(kind=&#39;barh&#39;, ax=axs[1, 1])<br>axs[1, 1].set_title(&#39;Average number of seasons by Genre&#39;)<br><br>plt.tight_layout()<br>plt.show()</pre><p>I expected Code Interpreter to render the visualisations per the marketing that OpenAI has promoted, but it didn’t until I input the above as plaintext like so:</p><pre>please plot the following<br><br>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>fig, axs = plt.subplots(2, 2, figsize=(20, 20))<br><br># Distribution of shows by Genre<br>sns.countplot(data=df, y=&#39;Genre&#39;, ax=axs[0, 0])<br>axs[0, 0].set_title(&#39;Distribution of shows by Genre&#39;)<br><br># A timeline of the number of shows that first aired each year<br>sns.histplot(data=df, x=&#39;First_Aired&#39;, bins=30, ax=axs[0, 1])<br>axs[0, 1].set_title(&#39;Timeline of the number of shows that first aired each year&#39;)<br><br># A bar chart showing the number of shows each network has<br>sns.countplot(data=df, y=&#39;Network&#39;, ax=axs[1, 0])<br>axs[1, 0].set_title(&#39;Number of shows each network has&#39;)<br><br># The average number of seasons by Genre<br>df.groupby(&#39;Genre&#39;)[&#39;Num_Seasons&#39;].mean().plot(kind=&#39;barh&#39;, ax=axs[1, 1])<br>axs[1, 1].set_title(&#39;Average number of seasons by Genre&#39;)<br><br>plt.tight_layout()<br>plt.show()</pre><p>Resulting in…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*unBrSy8fiO-5dz_W" /></figure><p>So, how does it impact what I’ve mentioned above? It doesn’t… not until OpenAI comes out with their own IDE 👀</p><p>Like what I write? Follow me here on Medium.</p><p>Want to learn how I can help your business with GenAI? Reach out to me on <a href="https://www.linkedin.com/in/byronaallen/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ee937e5863d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/dirty-coding-a-brief-overview-of-coding-with-llms-7ee937e5863d">Dirty Coding: a Brief Overview of Coding with LLMs</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Pipeline Orchestration - Using Amazon Managed Workflows for Apache Airflow (MWAA)]]></title>
            <link>https://medium.com/contino-engineering/data-pipeline-orchestration-using-amazon-managed-workflows-for-apache-airflow-mwaa-60e5b213a0a7?source=rss----20c2e566faed---4</link>
            <guid isPermaLink="false">https://medium.com/p/60e5b213a0a7</guid>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws-mwaa]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[data-orchestration]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Naseer Ahmad]]></dc:creator>
            <pubDate>Thu, 25 May 2023 10:22:09 GMT</pubDate>
            <atom:updated>2023-05-25T10:22:09.769Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*gCzwXy7mQKTaC8e-" /><figcaption>Data Pipeline Orchestration using AWS MWAA &amp; Glue</figcaption></figure><p>Data pipeline orchestration can be complex and challenging, especially when dealing with large volumes of data and complex workflows. Some of the challenges associated with data pipeline orchestration include the following:</p><ul><li><strong>Data format and schema compatibility:</strong> Data pipelines may involve multiple sources and destinations with different data formats and schemas, making it challenging to ensure data compatibility and consistency across the pipeline.</li><li><strong>Data validation and cleansing:</strong> Data quality issues such as missing or inconsistent data, duplicates, and data in the wrong format can cause pipeline failures, which can be time-consuming to debug and fix.</li><li><strong>Dependency management: </strong>Data pipeline workflow may depend on other workflows or external services, making it challenging to manage and orchestrate the pipeline execution.</li><li><strong>Resource management:</strong> Data pipelines may require significant computing and storage resources, which can be challenging to provision and manage efficiently, especially when dealing with burst workloads.</li><li><strong>Workflow monitoring and troubleshooting: </strong>Monitoring and troubleshooting data pipeline workflows can be challenging, especially when dealing with complex workflows involving multiple sources, destinations, and dependencies.</li></ul><p>Addressing these challenges requires combining technical expertise, best practices, and robust data pipeline orchestration tools and platforms. For example, Amazon Managed Workflows for Apache Airflow (MWAA) provides a fully managed and scalable platform for data pipeline orchestration, enabling you to focus on building and optimizing your data workflows. In contrast, AWS manages the platform&#39;s underlying infrastructure and operational aspects.</p><h4>Scenario</h4><p>Let&#39;s take a scenario where you must kick off the data pipeline when files land in the AWS S3 bucket, transform and save the data into S3 curated bucket, and crawl and catalog the curated data in the AWA Athena database table to run analytical queries without manual intervention. This blog post will walk you through step-by-step instructions for building data pipeline orchestration using Amazon Managed Workflow for Apache Airflow (MWAA).</p><h4>Data Pipeline Orchestration — Reference Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*dp1XCkb-KhwiAPGaUNQWqA.png" /><figcaption>Data Pipeline Orchestration — Reference Architecture</figcaption></figure><h4>What is Amazon MWAA?</h4><p>Amazon MWAA is a fully managed service that Amazon Web Services (AWS) provides, allowing users to run Apache Airflow in the cloud. Apache Airflow is an open-source platform for creating, scheduling, and monitoring complex data pipelines.</p><p>With Amazon MWAA, users can quickly deploy and manage Apache Airflow environments without requiring manual setup, configuration, or maintenance. Users can use Amazon MWAA to create and manage their workflows without worrying about infrastructure, deployment, or scalability. This makes it an ideal solution for organizations that need to process large amounts of data and want to automate their data workflows.</p><h4>MWAA Key Benefits</h4><ul><li><strong>Scalability:</strong> MWAA can automatically scale your Airflow cluster up or down based on your workload, ensuring you have the right resources to handle your workflows.</li><li><strong>Cost-effectiveness: </strong>With MWAA, you only pay for the resources you use, which can help you save money on infrastructure costs compared to managing your own Airflow cluster.</li><li><strong>Reliability: </strong>AWS takes care of the underlying infrastructure and handles tasks such as patching and updates, ensuring your workflows&#39; high availability and reliability.</li><li><strong>Security: </strong>MWAA provides built-in security features such as network isolation, encryption, and access controls to help you protect your data and workflows.</li><li><strong>Integration with AWS services:</strong> MWAA integrates with other AWS such as services, such as S3, DynamoDB, and Glue, allowing you to connect quickly and interact with other AWS services in your workflows.</li></ul><h4>Key Concepts</h4><ul><li><strong>DAG (Directed Acyclic Graph):</strong> DAGs are collections of tasks and describe how to run a workflow written in Python. Pipelines are designed as a directed acyclic graph by dividing a pipeline into tasks that can be executed independently. Then these tasks are combined logically as a graph.</li><li><strong>Task</strong>: A Task defines a unit of work within a DAG, represented as a node in the DAG graph. Each task implements an Operator, for example, a PythonOperator to execute some Python code or a BashOperator to run a Bash command. After an operator is instantiated, it&#39;s called a task.</li><li><strong>Task instance</strong>: A task instance represents a specific task run characterized by a DAG, a task, and a point in time.</li><li><strong>Operators</strong>: Operators are atomic components in a DAG describing a single task in the pipeline. They determine what gets done in that task when a DAG runs. Airflow provides operators with common tasks. In addition, it is extensible so that you can define your custom operators.</li><li><strong>Scheduling</strong>: The DAGs and tasks can be run on demand or scheduled at a specific frequency defined as a <em>cron expression</em> in the DAG.</li></ul><h4>Process Overview</h4><ol><li>Create an S3 bucket using GUI</li><li>Configure Amazon Managed Workflow for Apache Airflow(MWAA)</li></ol><ul><li><strong>IAM Role:</strong> Create IAM roles and policies to use with MWAA.</li><li><strong>S3 Bucket:</strong> S3 bucket to store your DAGs, plugins, and other Airflow files.</li><li><strong>VPC and Subnet:</strong> To deploy an MWAA environment in a VPC and Subnet for secure and private network communication.</li><li><strong>Security Groups:</strong> Create security groups to control MWAA environment access.</li><li><strong>Monitoring and logging:</strong> Configure Amazon CloudWatch to monitor and log your MWAA environment.</li></ul><p>3. Create the following AWS artifacts using the CloudFormation template:</p><ul><li><strong>IAM Role:</strong> Grant permission to AWS Glue and S3 services and attach this role with the AWS Glue job.</li><li><strong>Glue Job:</strong> Converting CSV file to Parquet format and saving the curated file(s) into S3.</li><li><strong>Crawler:</strong> Crawl and Catalog curated data using AWS Glue Crawler.</li><li><strong>Catalog:</strong> Catalog the metadata of the process file.</li></ul><p>4. Lambda function to trigger MWAA pipeline</p><p>5. Upload the CSV data files to S3 (Landing Zone) bucket</p><p>6. Query transformed data using AWS Athena</p><h4><strong>Implementation Steps</strong></h4><blockquote><strong>NOTE:</strong> Click <a href="https://github.com/datapro05/mwaa-datapipeline-orchestration"><strong>here</strong></a> to download the code featured in this blog post.</blockquote><ol><li>Create an S3 bucket using GUI</li></ol><p>2. Create the following folders under that bucket<br>- <strong>curated-data: </strong>Store curated data in this folder<br>- <strong>dags: </strong>Store DAGs code <br>- <strong>landed-zone: </strong>Contains the raw data file to process<br>- <strong>scripts: </strong>To save AWS Glue job script(s)<br>- <strong>processed-data: </strong>To move the processed raw data files<br>- <strong>requirement: </strong>Contains DAG <a href="https://github.com/datapro05/mwaa-datapipeline-orchestration/blob/main/requirements/requirements.txt"><strong><em>“requirement.txt”</em></strong></a> file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*wGf4AB80kl20WOvvW36AVA.png" /><figcaption>S3 Bucket — Folder Structure</figcaption></figure><p>3. Configure Amazon Managed Workflow for Apache Airflow(MWAA)<strong><br>- </strong>Login into AWS Management Console → Search → <strong>Managed Apache Airflow </strong>→ <strong>Create environment</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEdzgVY0onSM8ACvzZQ48Q.png" /><figcaption>Managed Apache Workflow (MWAA) — Create environment</figcaption></figure><ul><li>Under <strong>Environment details<br>- </strong>Enter<strong> <em>Name<br>- </em></strong>Leave the <strong><em>Airflow version </em></strong>and<strong><em> Weekly maintenance window start </em></strong>as default</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pj429rPtQbCLfi-WiDOgbw.png" /><figcaption>MWAA — Environment details</figcaption></figure><ul><li>Under the <strong>DAG code in Amazon S3<br>- </strong>Enter<strong> <em>S3 Bucket</em> </strong>path<br>- Enter the <strong><em>DAGs folder</em></strong> path<br>- Click → <strong>Next</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p2UQdOJtQes17hJHw_Cyrg.png" /><figcaption>MWAA — DAG code in Amazon S3</figcaption></figure><ul><li>Under <strong>Networking<br>- </strong>Click → <strong><em>Create MWAA VPC</em></strong></li></ul><blockquote><strong>NOTE: </strong>For this blog post, I have created a new VPC. Feel free to use your existing VPC, if it exists, or create a new.</blockquote><ul><li><em>Under</em><strong><em> web server access </em></strong>→ <strong><em>Public network (Internet accessible)<br>- </em></strong>Under<strong><em> Security group(s) </em></strong>→ Select <strong><em>Create new security group</em></strong> or Select <strong><em>Existing security group(s)</em></strong></li></ul><blockquote><strong>Note: </strong>For this blog post, I have created a new Security Group. Feel free to use your existing Security Group, if it exists.</blockquote><p><strong><em>- </em></strong>Leave the rest of the<strong><em> settings as default</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9XKxU-IwQE9DNq6SUGVtXw.png" /><figcaption>MWAA — Network Settings</figcaption></figure><p><strong><em>- </em></strong>Under<strong><em> Permissions, </em></strong>Select the existing<strong><em> Execution role or </em></strong>Click <a href="https://docs.aws.amazon.com/mwaa/latest/userguide/mwaa-create-role.html"><em>here</em></a><strong><em> </em></strong>to<strong><em> create a new role; </em></strong>click<strong><em> </em></strong>→ <strong><em>Next</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m_T1Yf6nycwnO427BBT0tg.png" /><figcaption>MWAA — Execution Role</figcaption></figure><p><strong><em>- Review and create; </em></strong>click<strong><em> </em></strong>→ <strong><em>Create environment<br>- </em></strong>Click on<strong><em> Open Airflow UI </em></strong>under<strong><em> Airflow UI</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UgXe5TjmRe7zUEOOhs5vHQ.png" /><figcaption>MWAA — Airflow UI Link</figcaption></figure><ul><li>Upon successful environment creation, you should see the Airflow UI</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HRkksRPnYW7NTRMgtrfrdw.png" /><figcaption>MWAA — Airflow UI</figcaption></figure><p>4. Create a stack of artifacts using below CloudFormation template</p><blockquote><strong>Note: </strong>For this demo purpose, I’m giving <strong><em>AmazonS3FullAccess</em></strong> to the IAM role. It’s not a best practice, especially if you are deploying this solution to a Production environment.</blockquote><pre>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: &#39;AWS Cloudformation Template to create AWS Crawler, S3, and Glue Job&#39;<br><br># Set :param values<br>Parameters:<br>  IAMRoleName:<br>    Description: Name of the exising role<br>    Type: String<br>    Default: demo-mwaa-glue<br>  GlueJobName:<br>    Description: Glue Job Name to process the raw data<br>    Type: String<br>    Default: cars-data-transformation<br>  GlueCrawlerName:<br>    Description: Glue Crawler name to crawl the raw data<br>    Type: String<br>    Default: catalog-cars-data<br>  GlueCatalogName:<br>    Description: Name of the data Catalog database<br>    Type: String<br>    Default: curated-data<br>  GlueJobScriptLocation:<br>    Description: Glue job script location<br>    Type: String<br>    Default: &#39;s3://airflowmwaa-demo/scripts/etlscript.py&#39;<br>  CrawlerS3Path:<br>    Description: Target file name for AWS Crawler<br>    Type: String<br>    Default: &#39;s3://airflowmwaa-demo/curated-data/&#39;<br><br>Resources:<br># Create IAM role for Glue<br>  IAMRole:<br>      Type: AWS::IAM::Role<br>      Properties:<br>        RoleName: !Ref IAMRoleName<br>        AssumeRolePolicyDocument:<br>          Version: &#39;2012-10-17&#39;<br>          Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service:<br>              - glue.amazonaws.com<br>              - s3.amazonaws.com<br>            Action: sts:AssumeRole<br>        ManagedPolicyArns:<br>          - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly<br>          - arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole<br>          - arn:aws:iam::aws:policy/AmazonS3FullAccess<br>        Path: &quot;/&quot;<br>        <br># Create Glue Database<br>  GlueCatalog:<br>    Type: &#39;AWS::Glue::Database&#39;<br>    Properties:<br>      CatalogId: !Ref AWS::AccountId<br>      DatabaseInput:<br>        Name: !Ref GlueCatalogName<br>        Description: Database for AWS CUR Data<br><br># Create AWS Glue job <br>  GlueJob:<br>    Type: AWS::Glue::Job<br>    Properties: <br>      Command:<br>        Name: glueetl<br>        PythonVersion: &#39;3&#39;<br>        ScriptLocation: !Ref GlueJobScriptLocation<br>      ExecutionProperty:<br>        MaxConcurrentRuns: 2<br>      MaxRetries: 0<br>      Role: !Ref IAMRoleName<br>      Name: !Ref GlueJobName<br>      GlueVersion: &#39;3.0&#39;<br>      WorkerType: &#39;G.1X&#39;<br>      NumberOfWorkers: 10<br><br># Create AWS Glue Crawler<br>  GlueCrawler:<br>    Type: &#39;AWS::Glue::Crawler&#39;<br>    Properties:<br>      Name: !Ref GlueCrawlerName<br>      Role: !GetAtt &quot;IAMRole.Arn&quot;<br>      DatabaseName: !Ref GlueCatalogName<br>      RecrawlPolicy:<br>        RecrawlBehavior: CRAWL_EVERYTHING<br>      Targets:<br>        S3Targets:<br>          - Path: !Ref CrawlerS3Path<br><br>Outputs:<br>  GlueCatalog:<br>     Description: AWS Glue Job created using CloudFormation Template.<br>     Value: !Ref  GlueCatalog<br>  GlueJob:<br>     Description: AWS Glue Database created using CloudFormation Template.<br>     Value: !Ref  GlueJob<br>  GlueCrawler:<br>    Description: AWS Crawler created using CloudFormation Template.<br>    Value: !Ref  GlueCrawler</pre><ul><li>Copy and save the CloudFormation template code locally</li><li>Login into AWS Console ➞ Search <strong>CloudFormation</strong></li><li>Click <strong>Create Stack </strong>➞ Select <strong>Upload a template file</strong> under <strong>Specify template ➝</strong> Choose file ➞ <strong>Next</strong></li><li>Under <strong>Stack name,</strong> ➞ <em>Enter a stack name </em>➞ <strong>Next</strong></li><li>Click <strong>Next</strong></li><li>Under <strong>Capabilities</strong> ✓ the checkmark ➞ <strong>Submit</strong></li><li>Under the <strong>Outputs </strong>tab, you will see all the artifacts created successfully</li></ul><p>5. Create a lambda function</p><ul><li>Login into AWS Console ➞ Search <strong>Lambda </strong>➞ <strong>Create Function</strong></li><li>Enter <strong>Function Name</strong></li><li>Under <strong>Permissions </strong>➞ <strong>Change default execution role</strong> ➞ <strong>Use an existing role </strong><em>(The role you created for the MWAA environment)</em></li><li>Leave the rest of the settings default ➞ <strong>Create Function</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zy_v6loXrz9bmYIOomq5mA.png" /><figcaption>Lambda Function Creation</figcaption></figure><ul><li>Please copy the below code snippet and replace it with the default code ➞ <strong>Deploy</strong></li></ul><pre>import boto3<br>import http.client<br>import base64<br>import ast<br><br># Set :param values<br>mwaa_env_name = &#39;MyAirflowEnvironmentDemo&#39;<br>dag_name = &#39;datapipeline-orc&#39;<br>mwaa_cli_command = &#39;dags trigger&#39;<br><br>client = boto3.client(&#39;mwaa&#39;)<br><br>def lambda_handler(event, context):<br>    print(event)<br>    # get web token<br>    mwaa_cli_token = client.create_cli_token(<br>        Name=mwaa_env_name<br>    )<br>    <br>    conn = http.client.HTTPSConnection(mwaa_cli_token[&#39;WebServerHostname&#39;])<br>    payload = mwaa_cli_command + &quot; &quot; + dag_name<br>    headers = {<br>      &#39;Authorization&#39;: &#39;Bearer &#39; + mwaa_cli_token[&#39;CliToken&#39;],<br>      &#39;Content-Type&#39;: &#39;text/plain&#39;<br>    }<br>    conn.request(&quot;POST&quot;, &quot;/aws_mwaa/cli&quot;, payload, headers)<br>    res = conn.getresponse()<br>    data = res.read()<br>    dict_str = data.decode(&quot;UTF-8&quot;)<br>    mydata = ast.literal_eval(dict_str)<br>    return base64.b64decode(mydata[&#39;stdout&#39;])</pre><p>6. Add S3 Trigger</p><ul><li>Click on <strong>+ Add trigger</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJfJ-3r8CtjhRrlfHh5Fkg.png" /><figcaption>Lambda — Add Trigger</figcaption></figure><ul><li>Select a source ➞ S3</li><li><strong>Event types </strong>(PUT, POST)</li><li>Enter the <strong>Bucket</strong> ➞ <strong>Prefix</strong> ➞ <strong>Recursive invocation</strong> ✓ ➞ <strong>Add</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ch26vGPfgi6_yfz0lgINcQ.png" /><figcaption>Lambda — S3 Trigger</figcaption></figure><p>7. Upload the <strong><em>“carsdata.csv”</em></strong> data file using AWS GUI; Click <a href="https://github.com/datapro05/mwaa-datapipeline-orchestration/tree/main/sample-data"><strong><em>Here</em></strong></a> to download the sample data file.</p><p>8. Copy the below DAG code snippet, save it locally, and Upload the <strong><em>“datapipeline-orc.py” </em></strong>file to the S3 bucket’s DAG folder using AWS GUI</p><pre>from datetime import datetime, timedelta<br>from airflow import DAG<br>from airflow.models.baseoperator import chain<br>from airflow.operators.bash import BashOperator<br>from airflow.operators.dummy import DummyOperator<br>from airflow.providers.amazon.aws.operators.glue import GlueJobOperator<br>from airflow.providers.amazon.aws.operators.glue_crawler import GlueCrawlerOperator<br>from airflow.providers.amazon.aws.operators.s3 import S3DeleteObjectsOperator<br><br>import os<br><br># Get the DAG ID from the file name<br>DAG_ID=os.path.basename(__file__).replace(&quot;.py&quot;, &quot;&quot;)<br><br>DEFAULT_ARGS={<br>            # Default arguments for the DAG<br>            # the number of retries that should be performed before failing the task<br>            &quot;owner&quot;: &quot;Nahmad&quot;,<br>            &quot;retries&quot;: 1,<br>            &quot;email_on_failure&quot;: False,<br>            &quot;email_on_retry&quot;: False,<br>        }<br># Glue job configuration<br>job_name = &quot;cars-data-transformation&quot;<br>region_name= &quot;us-east-1&quot;<br>iam_role_name=&quot;demo-mwaa-glue&quot;<br><br>config = {<br>    &quot;Name&quot;:&quot;catalog-cars-data&quot;,<br>    &quot;Role&quot;: &quot;demo-mwaa-glue&quot;,<br>    &quot;DatabaseName&quot;:&quot;curated-data&quot;,<br>    &quot;Description&quot;:&quot;Crawl cars dataset and catalog the the data&quot;,<br>    &#39;Targets&#39;:{&#39;S3Targets&#39; : [{&#39;Path&#39;: &quot;s3://airflowmwaa-demo/curated-data/&quot; }]}<br>}<br><br>with DAG(<br>        dag_id= DAG_ID,   <br>        description=&#39;Prepare data pipeline orchestration demo&#39;,<br>        default_args = DEFAULT_ARGS,<br>        start_date=datetime(2023, 4, 28),<br>        schedule_interval=None,<br>        dagrun_timeout=timedelta(minutes=10),<br>        catchup=False,<br>        tags=[&quot;Data Pipeline Orchestration&quot;]<br>) as dag:<br>    # Define the tasks<br>    <br>    # Dummy task to start the DAG<br>    begin = DummyOperator(task_id=&quot;begin&quot;)<br><br>    # Dummy task to end the DAG <br>    end = DummyOperator(task_id=&quot;end&quot;)<br>    <br>    # Task to purge processed data from S3 bucket<br>    purge_processed_data_s3_objects = BashOperator(<br>        task_id=&quot;purge_processed_data_s3_objects&quot;,<br>        bash_command=f&#39;aws s3 rm s3://airflowmwaa-demo/processed-data/ --recursive&#39;,<br>    )<br>    <br>    # Task to purge data catalog in Glue    <br>    purge_data_catalog = BashOperator(<br>        task_id=&quot;purge_data_catalog&quot;,<br>        bash_command=&#39;aws glue delete-table --database-name curated-data --name curated_data || echo &quot;Database cars-details not found.&quot;&#39;,<br>    )<br>    <br>    # Task to run the Glue Job<br>    run_glue_job = GlueJobOperator(<br>        task_id=&quot;run_glue_job&quot;,<br>        job_name=job_name,<br>        region_name= region_name,<br>        script_location=&quot;s3://airflowmwaa-demo/scripts/etlscript.py&quot;,<br>        s3_bucket=&quot;airflowmwaa-demo&quot;,<br>        iam_role_name=iam_role_name,<br>        aws_conn_id=&quot;aws_default&quot;,<br>        create_job_kwargs={&quot;GlueVersion&quot;: &quot;3.0&quot;,<br>                           &quot;WorkerType&quot;: &quot;G.1X&quot;,<br>                           &quot;NumberOfWorkers&quot;: 4,},      <br>    )<br>    <br>    # Task to run the Glue Crawler<br>    run_glue_crawler = GlueCrawlerOperator(<br>        task_id=&quot;run_glue_crawler&quot;,<br>        aws_conn_id= &quot;aws_default&quot;,<br>        config=config,       <br>    )<br>    <br>    # Task to sync buckets in S3<br>    sync_buckets = BashOperator(<br>        task_id=&quot;sync_buckets&quot;,<br>        bash_command=&#39;aws s3 sync s3://airflowmwaa-demo/landed-zone/  s3://airflowmwaa-demo/processed-data/&#39;,<br>    )<br>    <br>    # Task to purge raw data files from S3 bucket<br>    purge_raw_data_file = S3DeleteObjectsOperator(<br>        task_id=&quot;purge_raw_data_file&quot;,<br>        bucket=&quot;airflowmwaa-demo&quot;,<br>        keys=[&quot;landed-zone/carsdata.csv&quot;],<br>        aws_conn_id=&quot;aws_default&quot;,<br>    ) <br># Define the task dependencies using chain    <br>chain(<br>    begin,<br>    (purge_processed_data_s3_objects,purge_data_catalog),<br>    (run_glue_job),<br>    (run_glue_crawler),<br>    (sync_buckets),<br>    (purge_raw_data_file),<br>    end<br>)</pre><ul><li>Navigate back to your Airflow environment page, click on the link to your Airflow UI</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yffK-At22x7KiPWhNQZ0gg.png" /><figcaption>MWAA Airflow UI</figcaption></figure><ul><li>You should see the <strong><em>“datapipeline-orc”</em></strong> DAG listed here.</li></ul><blockquote><strong>NOTE: </strong>You may have to wait a few minutes for the DAG to appear; if unsuccessful, please review the previous steps.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EEPL0SjfL8llC5XDcPcUIA.png" /><figcaption>Data Pipeline Orchestration — DAG</figcaption></figure><ul><li>The DAG code will create the following tasks:<br>- <strong>begin:</strong> Starting the DAG point<br>- <strong>purge_data_catalog:</strong> Delete the data catalog<br>- <strong>purge_processed_data_s3_objects:</strong> Clean up the processed data folder in the S3 bucket<br>- <strong>run_glue_job:</strong> Trigger glue job to convert raw data file format from CSV to PARQUET<br>- <strong>run_glue_crawler: </strong>Crawler the curated-data folder and catalog the metadata of the processed file(s)<br>- <strong>sync_buckets:</strong> Copy landed-zone bucket objects to the processed-data folder<br>- <strong>purge_raw_data_file:</strong> Clean up the landed-zone folder <br>- <strong>end:</strong> Show Data pipeline orchestration completion</li><li>To move the DAG to <strong>Active, </strong>Click on the <strong>toggle switch</strong></li><li>To see the DAG Graph, Click on the DAG ➞ <strong>Graph</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNodTSDbJ8DJWPAYdwJ8LQ.png" /><figcaption>DAG Steps</figcaption></figure><h4>DAG Execution</h4><ol><li>Go to S3 Console<br>- Upload the carsdetail.csv file into the <strong>landed-zone </strong>folder<strong> </strong><br>- Lambda will trigger the DAG</li><li>Go to MWAA Airflow UI; you will see DAG in the running status</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJOhTF1lKQSDkUj5dcLVxA.png" /><figcaption>DAG Execution</figcaption></figure><p>3. After a successful DAG execution, you will see all tasks in <strong>Green</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7bMdVadytrxv5_VlUCIyEA.png" /><figcaption>Successful DAG Execution</figcaption></figure><h4>Validation</h4><ol><li>Go to <strong>Athena</strong> Console ➞ <strong>Launch query editor</strong></li><li>Select <strong>Data Source </strong>➞ AwsDataCatalog</li><li>Select <strong>Database</strong> ➞ curated-data</li><li>Click on three dots next to the table name ➞ Click <strong>Preview Table</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*IkYN84oNa9jkguuUBD1LcA.png" /><figcaption>Athena Data Source</figcaption></figure><p>5. Results will be generated in the <strong>Query results</strong> window</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qR6f6DEdM1PLOg2CdzknhQ.png" /><figcaption>Query Results</figcaption></figure><h4>Clean-up</h4><ol><li>Go to <strong>the CloudFormation </strong>Console<br>- Select the stack name created to build AWS data pipeline objects<br>- In the stack details pane, choose <strong>Delete<br>- </strong>Select<strong> Delete stack </strong>when prompted<br>- Go to CloudFormation Console<br>- Select the <strong>MWAA-VPC</strong> Stack <em>( MWAA VPC was created using CloudFormation during the MWAA setup)</em><br>- Choose <strong>Delete</strong></li><li>Go to Managed Apache Airflow Consol<br>- Select the Environment created as part of the workshop<br>- Choose <strong>Delete</strong></li><li>Go to S3 Console<br>-Click on the <strong>Bucket</strong> name created for this demo <strong><br>- </strong>Select all the folders in the bucket<br>- Choose <strong>Delete<br>- </strong>Follow through with the delete steps<br>- Go back to S3 List, select the bucket, and choose <strong>Delete</strong></li><li>Go to Lamda Console<br>- Select the <strong>Lambda</strong> function<br>- Under the <strong>Actions</strong> drop-down, Select <strong>Delete<br>- </strong>To confirm the deletion, type <strong>delete</strong> in the field,<strong> </strong>Click<strong> Delete</strong></li></ol><p>This blog post provides a comprehensive overview of using AWS Glue and Managed Workflows for Apache Airflow (MWAA) to create an efficient and scalable data pipeline in the AWS ecosystem. The blog post starts by introducing the concept of data pipelines and the challenges associated with building and maintaining them. Then, it highlights the importance of a reliable and automated pipeline solution for processing, transforming, and analyzing large volumes of data.</p><p>The blog post then introduces Managed Workflows for Apache Airflow (MWAA), a fully managed service that extends Apache Airflow, an open-source platform for orchestrating complex data pipelines. Moving forward, the blog post outlines the steps required to set up a data pipeline using AWS Glue and MWAA. Next, it covers creating a Glue Data Catalog, defining and running Glue Crawlers to discover metadata, creating Glue Jobs to transform data, and configuring MWAA to orchestrate the pipeline workflows.</p><p>Finally, the blog post summarizes the benefits of using AWS Glue and MWAA together. Overall, the blog post provides a comprehensive guide for building a robust data pipeline using AWS Glue and MWAA, highlighting the key features, setup steps, and best practices for efficient data processing and analysis in the AWS cloud environment.</p><blockquote>“Learning never exhausts the mind.” — Leonardo da Vinci</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=60e5b213a0a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/contino-engineering/data-pipeline-orchestration-using-amazon-managed-workflows-for-apache-airflow-mwaa-60e5b213a0a7">Data Pipeline Orchestration - Using Amazon Managed Workflows for Apache Airflow (MWAA)</a> was originally published in <a href="https://medium.com/contino-engineering">Contino Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>