<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vamsi Ramakrishnan on Medium]]></title>
        <description><![CDATA[Stories by Vamsi Ramakrishnan on Medium]]></description>
        <link>https://medium.com/@vamsiramakrishnan?source=rss-e02af0a414b8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*pAthtp13l0RC730kyGSjqA.jpeg</url>
            <title>Stories by Vamsi Ramakrishnan on Medium</title>
            <link>https://medium.com/@vamsiramakrishnan?source=rss-e02af0a414b8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:35:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vamsiramakrishnan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Say hello to Berglas — a way to keep your Secrets a Secret]]></title>
            <link>https://vamsiramakrishnan.medium.com/say-hello-to-berglas-a-way-to-keep-your-secrets-a-secret-4760a83c7bbb?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4760a83c7bbb</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[secrets]]></category>
            <category><![CDATA[gke]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 05:43:28 GMT</pubDate>
            <atom:updated>2022-12-11T09:23:35.938Z</atom:updated>
            <content:encoded><![CDATA[<h3>Say hello to Berglas — a way to keep your Secrets a Secret</h3><p>Your microservice calls other services — a database, an API endpoints or SaaS/ PaaS services and hold credentials to do so. Environment variables or Kubernetes Secrets are not really a secret and suffer from a variety of drawbacks</p><p>There are a variety of solutions for this</p><ol><li>3rd Party solutions like Vault, Mozilla SOPs, Bitnami k8s sealed secrets— which have way too many configurations to be made to get started for the simple use case of Creating Secrets, Storing secrets in an encrypted way, allow only authenticated and authorized principals to retrieve them.</li><li>Cloud KMS/HSM solutions which require integrating the SDKs into your code which is not very convenient as secrets are usually an afterthought after finishing the build prior to deployment.</li></ol><p>Enter Berglas — a painless way to manage secrets. Under the hood Berglas uses the Cloud KMS + GCS or Secrets Manager to abstract away the complexity of having to wire these things up yourself.</p><h4>The workings</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*2gvY2uhdcVPkstSvykYdbA.png" /></figure><h4>Key Steps to get it to Work</h4><pre>1. Berglas CLI installation on Cloud Shell<br>2. Use the CLI to bootstrap secrets in a bucket or secret manager <br>3. KMS keyrings are created to symmetrically to encrypt Secrets<br>4. Deploy the cloud function that acts a webhook mutation endpoint<br>5. Deploy the Webook Mutating Webhook config referencing endpoint <br>6. Create a berglas secret, grant access to a cloud service account<br>7. Create and Annotate k8s service acct to Cloud service acct<br>8. Change PodSpec to reference Berglas Secret</pre><h4>How to Get Started</h4><p><a href="https://github.com/GoogleCloudPlatform/berglas">GitHub - GoogleCloudPlatform/berglas: A tool for managing secrets on Google Cloud</a></p><h4><strong>GCP Services that works with Berglas</strong></h4><p><a href="https://github.com/GoogleCloudPlatform/berglas">GitHub - GoogleCloudPlatform/berglas: A tool for managing secrets on Google Cloud</a></p><pre>Google Kubernetes Engine<br>Cloud Run <br>Cloud Build<br>Cloud Functions<br>AppEngine Standard &amp; Flex<br>Init Scripts / Ansible scripts for Google Compute Engine</pre><h4>Other Reading</h4><ul><li><a href="https://insights.project-a.com/a-painless-way-to-manage-secrets-in-google-kubernetes-engine-3100eb35a313">A painless way to manage secrets in Google Kubernetes Engine</a></li><li><a href="https://scalesec.com/blog/a-comparison-of-secrets-managers-for-gcp/">A Comparison of Secrets Managers for GCP | ScaleSec</a></li><li><a href="https://www.kevinsimper.dk/posts/how-to-keep-secrets-secret-in-the-cloud">How to keep secrets secret in the cloud</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4760a83c7bbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started on Cloud Armor, Protect and Score Protection L]]></title>
            <link>https://vamsiramakrishnan.medium.com/getting-started-on-cloud-armor-protect-and-score-protection-l-95cbea1684ba?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/95cbea1684ba</guid>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Mon, 28 Mar 2022 03:43:39 GMT</pubDate>
            <atom:updated>2022-03-28T03:43:39.715Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started on Cloud Armor, Protect and Score Protection Levels</h3><p>If your application is hosted on Google Cloud serving internet traffic, consider enabling Cloud Armor. Which provides an extremely simple and easy to use way of protecting your applications against a variety of attack signatures.</p><p>While WAF rules and <a href="https://cloud.google.com/armor/docs/rule-tuning">tuning</a> them is beyond the scope of this post in itself If you do not know what rules you need to start with</p><p>A good Starting Point</p><pre>1. Filter Traffic from unwanted Geographies<br>2. Filter Traffic to unwanted paths in your API<br>3. Filter Traffic with unauthorized methods<br>4. OWASP Top 10 <br>5. SSL Policy to disallow TLS &lt; target version</pre><h4>Cloud Armor Basics</h4><p>For a detailed understanding of Limits , check this <a href="https://cloud.google.com/armor/quotas">page</a></p><pre>1. Targets Per Policy <br>( Which are the backends that need to be protected by this policy)</pre><pre>2. Rules Per Policy ( No Limit per policy, 200 rules per project)<br>( A collection of Allow and Deny rules )</pre><pre>3. Statements Per Rule ( 5 statements limit )<br>( What are the conditions that are evaluated for allow / deny )</pre><h4>Heuristics</h4><pre>1. Allow rules should take lower priority <br>2. Allow rules should be as specific as possible <br>3. Deny rules should be as broad as possible<br>4. Start with UAT , and enable it in Preview</pre><h4>Rule Translation</h4><p>How would all this translate as Rules in Cloud Armor. This is by no means the most comprehensive but is a good starting point. For the universal set of all rules check this <a href="https://cloud.google.com/armor/docs/rule-tuning">page</a></p><h4># Rule 1</h4><p>Deny ( 403)</p><p>Allow traffic from a specific region Only , also disallow sensitive paths or specific headers</p><pre>origin.region_code != &#39;&lt;ONLY REGION YOU WANT TRAFFIC FROM&gt;&#39; || request.path.contains(&#39;/&lt;YOUR PATH&gt;&#39;) || <br>request.headers.contains(&#39;&lt;DISALLOWED HEADERS&#39;)</pre><p><strong>#Rule 2</strong></p><p>Deny ( 403) — OWASP</p><pre>evaluatePreconfiguredExpr(&#39;xss-stable&#39;) &amp;&amp;evaluatePreconfiguredExpr(&#39;protocolattack-stable&#39;) &amp;&amp; evaluatePreconfiguredExpr(&#39;sqli-stable&#39;) &amp;&amp; evaluatePreconfiguredExpr(&#39;lfi-stable&#39;) &amp;&amp; evaluatePreconfiguredExpr(&#39;rfi-stable&#39;)</pre><p><strong># Rule 3 ( Extension of Rule 2 due to 5 Statement limit per rule)</strong></p><p>Deny (403) — OWASP</p><pre>evaluatePreconfiguredExpr(&#39;sessionfixation-stable&#39;) &amp;&amp; evaluatePreconfiguredExpr(&#39;protocolattack-stable&#39;) &amp;&amp; evaluatePreconfiguredExpr(&#39;scannerdetection-stable&#39;)</pre><h4>Inside Cloud Armor Rule Config</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYhZBXqG5uIyL4TdmMqTCA.png" /></figure><h4>How do I test the effectiveness of these rules</h4><p>Some of the popular and easy WAF testing frameworks</p><pre>1. <a href="https://github.com/wallarm/gotestwaf">https://github.com/wallarm/gotestwaf</a> Wallarm<br>2. <a href="https://github.com/signalsciences/waf-testing-framework">https://github.com/signalsciences/waf-testing-framework</a> SignalSci<br>3. <a href="https://github.com/fastly/ftw">https://github.com/fastly/ftw</a> Fastly<br>4. <a href="https://github.com/f5devcentral/f5-waf-tester">https://github.com/f5devcentral/f5-waf-tester</a> F5</pre><p>I decided to choose the one by Wallarm</p><pre>docker pull wallarm/gotestwaf<br>docker run -v ${PWD}/reports:/app/reports --network=&quot;host&quot;     wallarm/gotestwaf --url=&lt;Your WAF Endpoint&gt; </pre><p>Generates a report in this format. The Hosted backend is a Hello World Application serving on Flask so Application Secutity</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sIpWfddGWIIDa5hGtSiavA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHEDyIBisARu8h18BkhKxw.png" /></figure><h4>Next Steps :</h4><p>The report generates a folder /reports/xxx.pdf , which on examination reveals that there are a bunch of false positives which got flagged and blocked while it shouldn’t have been . Which is a good place to begin</p><h4>Tuning the False Positives</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*DtzU06EpA0oB1zc62fw8qA.png" /></figure><h4>Correlate Cloud Armor Logs to WAF Report</h4><p>Select <strong>“Logs” &gt;&gt; “Policy Logs”</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V29NiPuepeaAZQxioysGJQ.png" /></figure><p>Correlate <strong>WAF Logs</strong> to <strong>Test Execution Times</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HFKYDiDcHXsvwHwnnUH7lw.png" /></figure><h4>Tune the ModSec paranoia Levels</h4><p>Each preconfigured rule has a sensitivity level that corresponds to a ModSecurity <a href="https://coreruleset.org/faq/#paranoialevel">paranoia level</a>. A lower sensitivity level indicates a higher confidence signature, which is less likely to generate a false positive. A higher sensitivity level increases security, but also increases the risk of generating a false positive.</p><p>Repeat tests.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95cbea1684ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Code to create Import File for VertexAI Datasets in Google Cloud]]></title>
            <link>https://vamsiramakrishnan.medium.com/code-to-create-import-file-for-vertexai-datasets-in-google-cloud-a00a7ae61967?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a00a7ae61967</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[vertex-ai]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Thu, 07 Oct 2021 19:04:21 GMT</pubDate>
            <atom:updated>2021-10-07T19:14:06.018Z</atom:updated>
            <content:encoded><![CDATA[<h3>GCP Vertex AI Dataset Import File Generation</h3><figure><img alt="Vertex AI Google Cloud" src="https://cdn-images-1.medium.com/max/1024/0*SjeekKw6unvL-LS4" /><figcaption>Google Cloud Vertex AI</figcaption></figure><p>Dataset preparation for VertexAI requires creation of an Import File accompanying the dataset.</p><pre><strong>Import File contains</strong></pre><pre>1. Path of The Image <br>2. Is it Training, Test, Validation Image ?<br>3. What is the Label(s) - Classification, Where is the Bounding Box(es) for Detection etc. </pre><p>Both JSONL&amp; CSV excepting specific use cases such as Entity Extraction in Text where only JSONL is supported. The code below is for CSV File Generation along with GCS Parallelized Uploads.</p><h3>Code</h3><h4>Import Modules and Define Path to Service Account Token</h4><pre>import json<br>import os <br>import csv<br>from google.cloud import storage<br>import threading</pre><pre>os.environ[&quot;GOOGLE_APPLICATION_CREDENTIALS&quot;]=&quot;PATH_TO_SERVICE_ACCOUNT_JSON_PATH&quot;</pre><h4>Helper Functions</h4><pre>def write_csv(fName, row_data): <br>    with open(fName, &#39;a&#39;, newline=&#39;\n&#39;) as myfile:<br>        wr = csv.writer(myfile,  delimiter = &quot;,&quot;)<br>        wr.writerow(row_data)</pre><pre><strong># Skip this part if you dont want to run the upload of dataset also automatically </strong><br>def upload_to_bucket(blob_name, path_to_file, bucket):<br>    blob = bucket.blob(blob_name)<br>    blob.upload_from_filename(path_to_file)</pre><h4>Define Paths</h4><pre><strong>#Base Dataset Path from where </strong><br>dataset_path = &quot;/Users/vamramak/Downloads/dataset&quot;<br>folder_list = [&quot;training&quot;,&quot;validation&quot;]<br>label_list = [&quot;00-damage&quot;,&quot;01-whole&quot;]</pre><pre><strong># Bucket Information</strong><br>bucket_name = &quot;vamramak-automl-dataset&quot;</pre><pre><strong># Where should the CSV be created</strong><br>csv_output_path = dataset_path + &quot;/inputFile.csv&quot;</pre><pre><strong># Intialize Client</strong><br>storage_client = storage.Client()<br>bucket = storage_client.get_bucket(bucket_name)</pre><h4>Scan through the Folder Structure</h4><pre><strong># Simple Scan </strong><br>for folder in folder_list:<br>    for label in label_list:<br>        file_list_dir = dataset_path + &quot;/&quot; + folder + &quot;/&quot; + label<br>        file_list = os.listdir(file_list_dir)<br>        <br>        for file_name in file_list:<br>            <br>            path_to_file = file_list_dir + &quot;/&quot; + file_name<br>            object_name = folder + &quot;/&quot; + label + &quot;/&quot; + file_name<br>            gcs_file_path =  &quot;gs://&quot; + bucket_name + &quot;/&quot; + object_name<br>            <br>            <br>            # Skip this part and use direct function call instead of Multi Threading if you have a really large dataset<br>            # Skip the upload piece if the goal is to only generate the CSV File<br>            <br>            t = threading.Thread(target = upload_to_bucket, args=(object_name, path_to_file, bucket,)).start()<br>            row_data = [ folder , gcs_file_path , label ]<br>            write_csv( csv_output_path , row_data)</pre><h4>Finally Upload the CSV File</h4><pre>upload_to_bucket(&quot;input_file.csv&quot;, csv_output_path, bucket)</pre><h3>Gist Link</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8740815a3d57cc8deb673857efe0607/href">https://medium.com/media/e8740815a3d57cc8deb673857efe0607/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a00a7ae61967" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Weekend Reads : 3rd October, 2021]]></title>
            <link>https://vamsiramakrishnan.medium.com/weekend-reads-3rd-october-2021-74330318edca?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/74330318edca</guid>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Sun, 03 Oct 2021 17:50:13 GMT</pubDate>
            <atom:updated>2021-10-03T17:59:23.920Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>On the Internet, We’re Always Famous</strong></h3><p>What happens when the experience of celebrity becomes universal?</p><p><a href="https://www.newyorker.com/news/essay/on-the-internet-were-always-famous">https://www.newyorker.com/news/essay/on-the-internet-were-always-famous</a></p><h3>What Our Research Really Says About Teen Well-Being and Instagram</h3><p>Facebook&#39;s official position on well being research on Instagram. How it claims Wall Street Journal through its Facebook Files Mischaracterized research.</p><p><a href="https://about.fb.com/news/2021/09/research-teen-well-being-and-instagram/">https://about.fb.com/news/2021/09/research-teen-well-being-and-instagram/</a></p><h3>The other voting systems we often ignore</h3><p>A gentle introduction to some of the other kinds of voting systems - First Past the Post, Ranked Choice, Proportional Voting, Quadratic Voting <a href="https://link.medium.com/EF9lLRKy3jb">https://link.medium.com/EF9lLRKy3jb</a></p><h3>Trisolarans</h3><p>Planets that orbit three stars at once.</p><p><a href="https://www.nytimes.com/2021/09/28/science/triple-sun-planet.html">https://www.nytimes.com/2021/09/28/science/triple-sun-planet.html</a></p><h3>Fluency in reading minds</h3><p>In one classic experiment, children were told stories such as the following:</p><p>Maxi has put his chocolate in the cupboard. While Maxi is away, his mother moves the chocolate from the cupboard to the drawer. When Maxi comes back, where will he look for the chocolate?</p><p>Until the age of four, children often fail this test, saying that Maxi will look for the chocolate where it actually is (the drawer), rather than where he thinks it is (in the cupboard).</p><p>They are using their knowledge of the reality to answer the question, rather than what they know about where Maxi had put the chocolate before he left.</p><p><a href="https://aeon.co/essays/is-there-a-symmetry-between-metacognition-and-mindreading">https://aeon.co/essays/is-there-a-symmetry-between-metacognition-and-mindreading</a></p><h3>The American Aristotle</h3><p>The very first lesson that we have a right to demand that logic shall teach us is, how to make our ideas clear … To know what we think, to be masters of our own meaning, will make a solid foundation for great and weighty thought.</p><p><a href="https://aeon.co/essays/charles-sanders-peirce-was-americas-greatest-thinker">https://aeon.co/essays/charles-sanders-peirce-was-americas-greatest-thinker</a></p><h3>China’s war on fun.</h3><p>Like helicopter parents, dictators think it’s dangerous to enjoy life</p><p><a href="https://unherd.com/2021/09/the-totalitarian-urge-to-control-children/?1632676627537">https://unherd.com/2021/09/the-totalitarian-urge-to-control-children/?1632676627537</a></p><h3>A brief history of Baileys</h3><p>The real heroes of ideas are not the people who have them – they are the people who buy them</p><p><a href="https://www.irishtimes.com/culture/books/in-1973-i-invented-a-girly-drink-called-baileys-1.3240945">https://www.irishtimes.com/culture/books/in-1973-i-invented-a-girly-drink-called-baileys-1.3240945</a></p><h3>Deep Learning&#39;s Diminishing Returns</h3><p>The cost of improvement is becoming unsustainable</p><p><a href="https://spectrum.ieee.org/deep-learning-computational-cost">https://spectrum.ieee.org/deep-learning-computational-cost</a></p><h3>Misinformation Is About to Get So Much Worse.</h3><p>A conversation with the former Google CEO Eric Schmidt</p><p><a href="https://www.theatlantic.com/technology/archive/2021/09/eric-schmidt-artificial-intelligence-misinformation/620218/">https://www.theatlantic.com/technology/archive/2021/09/eric-schmidt-artificial-intelligence-misinformation/620218/</a></p><h3>China’s Power Shortage</h3><p>A bit of a dive into what is happening with China’s power situationThe core issue is the coal market.</p><p><a href="https://stratechery.com/2021/chinas-power-shortage-the-tech-angle-china-and-bitcoin/">https://stratechery.com/2021/chinas-power-shortage-the-tech-angle-china-and-bitcoin/</a></p><h3>Whatever Happened to the Population Bomb?</h3><p>The world&#39;s population has risen enormously. The signs of a slowdown are becoming visible</p><p><a href="https://spectrum.ieee.org/whatever-happened-to-the-population-bomb">https://spectrum.ieee.org/whatever-happened-to-the-population-bomb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74330318edca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Study on using Google Cloud Storage with the S3 Compatibility API]]></title>
            <link>https://vamsiramakrishnan.medium.com/a-study-on-using-google-cloud-storage-with-the-s3-compatibility-api-324d31b8dfeb?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/324d31b8dfeb</guid>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[google-cloud-storage]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Fri, 24 Sep 2021 04:57:56 GMT</pubDate>
            <atom:updated>2021-09-24T04:57:56.158Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Cloud Storage’s XML API Provides Interoperability with some of the client libraries that use S3. If you have existing applications that read and write data using the S3 API/SDK/Client Libraries within configuration changes and minimal code change. While compatibility offers ease, it is important to be aware about where things break. This post outlines some of those scenarios</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F1GbCh6AwFiPY1Zt.png" /></figure><h3>Sections</h3><p>The post has 4 major sections</p><pre>1. Where things break ( TL;DR )<br>2. Server Side Config<br>3. Client Side Config<br>4. Code Samples</pre><h3>Where things break</h3><h4>ACLs</h4><p>There are some minor differences in the way AWS’s Predefined/Canned ACLs work and the way GCPs Canned ACLs work. Before that, a small refresher in ACL Concepts</p><pre>ACLs have 2 Properties<br>1. Grantees ( Who gets access )<br>2. Scope ( How much access do they get )</pre><pre>ACLs are of 2 types<br>1. Canned ACLs ( Predefined Scopes &amp; Grantees )<br>2. Custom ACLs ( Custom Scope &amp; Grantees )</pre><pre>ACLs can be Applied at 2 Levels<br>1. Bucket <br>2. Object</pre><h4>Differences in Canned ACLs</h4><p>The bolded ones where upstream programs use these permissions will make the the Compatibility break.</p><pre>| AWS Canned ACL            | GCP Canned ACL            | Applies |<br>|---------------------------|---------------------------|---------|<br>| private                   | private                   | Both    |<br>| public-read               | public-read               | Both    |<br>| public-read-write         | public-read-write         | Both    |<br><strong>| aws-exec-read             | -                         | Both    |</strong><br>| authenticated-read        | authenticated-read        | Both    |<br>| bucket-owner-read         | bucket-owner-read         | Object  |<br>| bucket-owner-full-control | bucket-owner-full-control | Object  |<br><strong>| log-delivery-write        | -                         | Bucket  |<br>| -                         | project-private           | Both    |</strong></pre><h4>CORS</h4><pre>2 Types of CORS <br>1. Simple<br>2. Preflighted</pre><p>While both GCS and AWS S3 have the same fields in CORS the way we specify CORS Configurations are different and hence reusing the client libraries is not possible in this case.</p><pre>| AWS CORS       | GCP CORS        |<br>|----------------|-----------------|<br>| AllowedHeaders | ResponseHeaders |<br>| AllowedMethods | Methods         |<br>| AllowedOrigins | Origins         |<br>| MaxAgeSeconds  | MaxAgeSec       |</pre><p>So when setting up CORS Configuration at the bucket the following error pops up</p><pre>ClientError: An error occurred <strong>(MalformedLifecycleConfiguration)</strong> when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.</pre><h4>Object Lifecycle Policy</h4><p>While setting Object Lifecycle Policies are supported by the XML API, the request structures are different in the case of GCS , and you will recieve this common error</p><pre>ClientError: An error occurred <strong>(MalformedLifecycleConfiguration)</strong> when calling the PutBucketLifecycleConfiguration operation: The XML you provided was not well-formed or did not validate against our published schema.</pre><h4>Object Integrity Check</h4><p>If your application uses Object Integrity Checks in it’s logic during upload then you may want to read this. We encounter 2 Broad Scenarios in Object Integrity Checks</p><pre>2 Scenarios<br>1. Single Part Upload<br>2. Multi-Part Upload</pre><p>There are different types of File Integrity Checks</p><pre>Types of File Integrity Checks<br>1. CRC32C<br>2. MD5<br>3. ETags</pre><p>How do these components Relate, The client side while uploading needs to validate</p><h4>AWS Multi-Part Upload</h4><p><a href="https://stackoverflow.com/questions/37853192/amazon-s3-huge-file-integrity">Amazon S3 huge file integrity</a></p><p>On multipart uploads, the etagis computed by taking the binary encoding of each part’s md5hash, concatenating them together, doing an md5of that, hex-encoding the result, then appending the - followed by the number of parts.</p><h4>GCP Multi-Part Upload</h4><p>GCP Composite crc32cis computed by taking the individual part’s crc32cs</p><h4>A Table comparing the differences</h4><pre>|     | Single Part | Multi-Part                            |<br>|-----|-------------|---------------------------------------|<br>| GCP | MD5, CRC32C | b64 encoded CRC32C (CRC32Cs of Parts) |<br>| AWS | MD5         | Hex encoded MD5 ( MD5 of Parts )      |</pre><h3>Server Side Configuration</h3><p><strong>Step 1:</strong> Create a Custom Role (Not Mandatory), skip this step if a pre-defined role can be assigned to the Principal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*zU5lHr6CZUZ_IrTMIa7peg.png" /></figure><p><strong>Step 2:</strong> Create a Service Account/Accounts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*LMtiEPOenk-zyQTlz2BTzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*EDQDeHj64Zd1o-p-qF_puw.png" /></figure><p><strong>Step 3:</strong> Add Storage Admin Role to Service Account</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*w4839PekKUaWvOakN5ortA.png" /></figure><p>Or Assign the Custom Role that was created</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*EvobzWBqh7WP7hY4pbToGg.png" /></figure><p><strong>Step 4:</strong> Go to Cloud Storage, Copy Storage Endpoint</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*Mr8v9yff4u3BgkkxrmVWVg.png" /></figure><p><strong>Step 5: </strong>Create HMAC Keys</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*bK11KEwcdOPy9U5FfIpw8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*GzNhdPGE2sPshkV-jV7KJA.png" /></figure><p>Once you have the HMAC Keys and the Interop Endpoint Setup for that project you are all set to use the S3 Interoperability</p><h3>Client Side Configuration</h3><p>Call out to this Stack Overflow Post</p><p><a href="https://stackoverflow.com/questions/42809096/difference-in-boto3-between-resource-client-and-session">Difference in boto3 between resource, client, and session?</a></p><h4>As a Resource</h4><pre>s3_resource = boto3.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><h4>As a Session</h4><pre>session = Session()<br>s3_session = session.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=ACCESS_KEY, region_name=GCP_REGION_NAME)</pre><h4>As a Client</h4><pre>s3_client = boto3.client(&#39;s3&#39;, endpoint_url= GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><p>Or alternatively change the boto.cfg</p><pre>cat /etc/boto.cfg</pre><p>Add the right values</p><pre>[Credentials]<br>  aws_access_key_id = ACCESS_KEY<br>  aws_secret_access_key = SECRET_KEY<br>  s3_host = storage.googleapis.com</pre><h3>Code Samples</h3><p>Skipping the simple List Bucket, List Objects based examples as they are repetitive.</p><pre>1. Create Bucket<br>2. Multipart Upload<br>3. Signed URLs<br>4. Object Versioning Enable </pre><h4>Create Bucket</h4><p>Please note that the default configuration for any GCP Region with multi-region bucket configuration follows that multi-region as a default. Specifying</p><pre>import boto3</pre><pre>s3 = boto3.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><pre><br>s3.create_bucket(Bucket = BUCKET_NAME, CreateBucketConfiguration= {&#39;LocationConstraint&#39;: GCP_REGION_NAME })</pre><h4>Multi-part Upload</h4><pre>import boto3<br>from boto3.s3.transfer import TransferConfig<br></pre><pre>s3 = boto3.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><pre><br>config = TransferConfig(multipart_threshold=1024 * 25, <br>                        max_concurrency=10,<br>                        multipart_chunksize=1024 * 25,<br>                        use_threads=True)</pre><pre>file_path = os.path.dirname(__file__) + FILE_NAME</pre><pre>s3_resource.Object(BUCKET_NAME, OBJECT_NAME).upload_file(file_path,<br>                            ExtraArgs={&#39;ContentType&#39;: &#39;xxx/yyy&#39;},<br>                            Config=config)</pre><h4>Signed URLs</h4><p>It is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource.</p><pre>import boto3</pre><pre>s3 = boto3.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><pre>response = s3.meta.client.generate_presigned_url(https://medium.com/&#39;get_object&#39;,%20Params={&#39;Bucket&#39;:%20BUCKET_NAME,%20&#39;Key&#39;:%20OBJECT_NAME},%20ExpiresIn=EXPIRATION)</pre><pre>print(response.data)</pre><h4>Object Versioning</h4><p>Enable object versioning in a bucket</p><pre>import boto3</pre><pre>s3 =boto3.resource(service_name=&#39;s3&#39;, endpoint_url=GCP_URL, aws_access_key_id=ACCESS_KEY, aws_secret_access_key=SECRET_KEY, region_name=GCP_REGION_NAME)</pre><pre>versioning = s3.BucketVersioning(BUCKET_NAME)<br>versioning.enable()<br>print(versioning.status())</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=324d31b8dfeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Weekend Reads : 10th Sep, 2021]]></title>
            <link>https://vamsiramakrishnan.medium.com/weekend-reads-10th-sep-2021-af172aee0568?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/af172aee0568</guid>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Sun, 12 Sep 2021 14:30:33 GMT</pubDate>
            <atom:updated>2021-09-13T08:25:22.935Z</atom:updated>
            <content:encoded><![CDATA[<h3>Quotes that piqued my interest</h3><blockquote>Umberto Eco wrote in his encyclopedic novel of human credulity, Foucault’s Pendulum, “The conspiracy theory of society comes from abandoning God … and then asking, ‘Who is in his place?’”</blockquote><blockquote>Doubt seems to be a double-edged sword: One blade cuts through orthodoxy allowing for new truths; the other seeks to sever the head of reason from the body of society.</blockquote><blockquote>Behind the wheel of a car, aren’t we already cyborgs.</blockquote><blockquote>Nobel-laureate Pierre-Gilles de Gennes allegedly once said of nuclear fusion: “We say that we will put the sun into a box. The idea is pretty. The problem is, we don’t know how to make the box</blockquote><blockquote>You can short civilization if you want. But there’s no one to pay you if you win.</blockquote><h3>Long reads</h3><pre><strong>When in Doubt we Have to Talk About Doubt.</strong> How to tell the difference between scientific and conspiratorial skepticism.</pre><p><a href="http://m.nautil.us/issue/104/harmony/we-have-to-talk-about-doubt">We Have to Talk About Doubt - Issue 104: Harmony - Nautilus</a></p><pre><strong>Gas Cloud and transmissibility</strong> : Perhaps it&#39;s time we understood the role of the Gas Cloud and it&#39;s role in aiding airborne transmission</pre><p><a href="https://knowablemagazine.org/article/physical-world/2021/understand-airborne-transmission-disease">To understand airborne transmission of disease, follow the flow</a></p><pre><strong>Hairdressers and Algorithms : </strong>This is a fantastic story that has currency in today’s contemporary debates</pre><p><a href="https://www.exponentialview.co/algorithms-and-hairdressers/">💡 Algorithms and hairdressers</a></p><pre><strong>What is Sustainable Infrastructure </strong>The case of the Kyoto canal, or, building cities that can last a thousand years</pre><p><a href="https://wrathofgnon.substack.com/p/sustainable-infrastructure">Sustainable Infrastructure</a></p><pre><strong>One Woman’s Mission to Rewrite Nazi History on Wikipedia:</strong> Ksenia Coffman’s fellow editors have called her a vandal and a McCarthyist. She just wants them to stop glorifying fascists—and start citing better sources.</pre><p><a href="https://www.wired.com/story/one-womans-mission-to-rewrite-nazi-history-wikipedia/">One Woman&#39;s Mission to Rewrite Nazi History on Wikipedia</a></p><h3>Eye-Openers</h3><pre><strong>Number Games:</strong> Numbers don’t always mean what they seem to mean, It pays to always check what is actually being counted.</pre><p><a href="https://www.theguardian.com/theobserver/commentisfree/2021/sep/05/numbers-dont-always-mean-what-they-seem-to-mean">Numbers don&#39;t always mean what they seem to mean | David Spiegelhalter and Anthony Masters</a></p><pre><strong>Yield &amp; Fragility of Global Supply chains</strong>: If you take 99% yield through 700 steps, by the time you&#39;re done, you&#39;ll get nothing at the end which is workable. So you really need to have 99.99-plus % yield on every step.”</pre><p><a href="https://www.theverge.com/2021/8/31/22648372/willy-shih-chip-shortage-tsmc-samsung-ps5-decoder-interview">Why the global chip shortage is making it so hard to buy a PS5</a></p><pre><strong>Nuclear Fusion: </strong>The race to give nuclear fusion a role in the climate emergency. Power from fusion has proved too hard to generate at scale. Can recent breakthroughs and massive investment change that.</pre><p><a href="https://www.theguardian.com/environment/2021/aug/28/the-race-to-give-nuclear-fusion-a-role-in-the-climate-emergency">The race to give nuclear fusion a role in the climate emergency</a></p><h3>Tech News</h3><pre><strong>Apple vs Epic :</strong> Epic Leaves a Big Crack in Apple’s Walled Garden - A judge ordered significant changes to the App Store today, but the fight between Epic Games and Apple isn&#39;t over yet.</pre><p><a href="https://www.wired.com/story/epic-leaves-big-crack-in-apple-walled-garden/">Epic Leaves a Big Crack in Apple&#39;s Walled Garden</a></p><pre><strong>Censorship and Internet Shutdowns</strong> Governments are turning off the internet to silence dissenters at an ‘exponential’ rate—and threatening civil society, says the chief operating officer of Google’s Jigsaw project.</pre><p><a href="https://www.technologyreview.com/2021/09/09/1035237/internet-shutdowns-censorship-exponential-jigsaw-google/">Why you should be more concerned about internet shutdowns</a></p><pre><strong>Ex-Googlers raise $40 million to democratize natural-language AI</strong>. Their startup, called Cohere, is backed by AI luminaries Geoffrey Hinton and Fei-Fei Li.</pre><p><a href="https://www.fastcompany.com/90670635/ex-googlers-raise-40-million-to-democratize-natural-language-ai">Ex-Googlers raise $40 million to democratize natural-language AI</a></p><h3>Other interesting morsels of wisdom</h3><pre><strong>Meaning is the hottest commodity</strong>: The virtuous cycle that creates the purpose economy</pre><p><a href="https://rkauf.medium.com/the-purpose-economy-is-arriving-sooner-than-you-think-62854e9d0f6c">The purpose economy is arriving sooner than you think</a></p><pre><strong>Meet the Little-Known Genius Who Helped Make Pixar Possible</strong>. Alvy Ray Smith helped invent computer animation as we know it—then got royally shafted by Steve Jobs. Now he’s got a vision for where the pixel will take us next.</pre><p><a href="https://www.wired.com/story/pixar-animation-alvy-ray-smith-pixel/">Meet the Little-Known Genius Who Helped Make Pixar Possible</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af172aee0568" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Autoscaling Exadata Cloud at Customer / Exadata Cloud Service]]></title>
            <link>https://vamsiramakrishnan.medium.com/autoscaling-exadata-cloud-at-customer-exadata-cloud-service-f0e02039a0be?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0e02039a0be</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[oracle-cloud]]></category>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[scaleup]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 09:27:16 GMT</pubDate>
            <atom:updated>2020-07-09T09:27:16.092Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Recipe — Events, Functions, Schedule based Autoscaling</h4><p>This post is specifically about leveraging schedule-based autoscaling to scale up and scale down of Exadata Cloud at Customer and Exadata Cloud Service. For a general understanding of what are the possibilities and a primer, check the link below.</p><p><a href="https://medium.com/@vamsiramakrishnan/use-schedule-based-autoscaling-to-do-everything-except-autoscaling-a4b8e6fed288">Use schedule-based autoscaling to do everything except autoscaling</a></p><h4>Some Context</h4><p>The solution is applicable to</p><pre>Gen 2.0 Exadata Cloud at Customer<br>Exadata Cloud Service.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uGao_qjvFUvOwhHH" /></figure><h4>What makes this approach special?</h4><pre><strong>#1 Vertical Scaling<br>--------------------</strong></pre><pre>ExaCC - Exadata Cloud at Customer<br>ExaCS - Exadata Cloud Service<br>ExaCC &amp; ExaCS support Vertical Scaling without downtime</pre><p>—</p><pre><strong>#2 Hybrid Cloud<br>----------------</strong></pre><pre>Exadata Cloud at Customer is deployed behind the customer&#39;s firewall at the customer&#39;s data-center. </pre><p>—</p><h4>The Architecture</h4><p>The Gen 2.0 Architecture allows one to use a fully modern fabric to optimize Exadata Cloud at Customer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*psbLqmE1qySIegVi" /><figcaption>Gen 2.0 Exadata Cloud at Customer</figcaption></figure><h4>The Pattern</h4><p>The pattern is very similar to what I advocated earlier in this post. It builds on what a modern cloud infrastructure and platform provides.</p><h4>Fundamental Principles</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_yKZxc6OMMVGAPJo" /></figure><pre>1. Every infrastructure component is an API<br>2. Every interaction with infrastructure is a CRUD action<br>3. Every CRUD Action is logged.<br>4. Every Create, Update or Delete Action emits a lifecycle event.<br>5. Every lifecycle event provides resource contextual information.<br>6. This event can be processed to take further action</pre><pre>Schedule based Autoscaling as a Trigger<br>Events Service as a Router &amp; Feedback loop for closed loop benefits<br>Functions as an Actuator</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0wsy5dwa9TkXV33d" /></figure><h4>Tags for Passing Data &amp; Attributes for Filtering</h4><p>Tags can be used (both freeform and defined tags ) along with resource attributes to decide which ExaCC and by how much should the scaling happens. This removes the need to write multiple functions to do the same scaling action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9kRi3HYT2mv939Im" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A9Us0wLX_XKAimmf" /></figure><h3>The Why</h3><p>Assume you have an ExaCS Half Rack / ExaCC Half Rack of the total capacity of 200 Cores.</p><pre>( Mon- Friday 6 AM - 10 PM )<br>Exa Database CPU Requirement - 100 Cores</pre><pre>( Mon - Friday 10 PM - 6 AM )<br>Exa Database CPU Requirement  - 40 Cores</pre><pre>Fri - Mon ( 6 AM - 6 PM )<br>Exa Database CPU Requirement - 60 Cores</pre><pre>Fri - Mon ( 6 PM - 6 AM )<br>Exa Database CPU Requirement - 20 Cores</pre><h4>Comparison</h4><p>A yearly savings potential of 109K USD for every 100 Cores of Database footprint you have on Exadata. While the savings potential may change from use case to use case, the cost of turning this lever to save costs is quite literally “0”</p><pre>+----------+---------------+--------------+--------------+<br><strong>| Comments | Full Capacity | Demand Curve | Cost Savings |</strong><br>+----------+---------------+--------------+--------------+<br>| Month    | 24001 $       | 14865 $      | 9136$        |<br>+----------+---------------+--------------+--------------+<br>| Yearly   | 288017 $      | 178384 $     | 109,632$     |<br>+----------+---------------+--------------+--------------+</pre><h3>The Details</h3><p>Understanding how components are organized inside Exadata Cloud Service Exadata Cloud@Customer</p><pre>An Example ExaCC/ExaCS<br>1) 3 VM Clusters<br>2) VM Cluster-1 has 16 OCPUs/Cores allocated to it.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*YKa7h9-UslWAs2k7" /></figure><pre>VM Cluster - 1<br>Before Scale Up : 16 OCPUs<br></pre><pre><strong>Minimum Scale up OCPU Count = Number of VMs<br></strong>Scaling action is always for a given VM Cluster. If there are 4 VMs in a VM-Cluster , then the scaling is done in Multiples of 4</pre><pre>After Scale Up : 20 OCPUs</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aCdgl2K6u1KoZZJY" /><figcaption>Scaling Action Example</figcaption></figure><p>I hope this helps. Leave your thoughts below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0e02039a0be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use schedule-based autoscaling to do everything except autoscaling]]></title>
            <link>https://vamsiramakrishnan.medium.com/use-schedule-based-autoscaling-to-do-everything-except-autoscaling-a4b8e6fed288?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4b8e6fed288</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-native]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[oracle-cloud]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Fri, 26 Jun 2020 17:07:41 GMT</pubDate>
            <atom:updated>2020-06-27T06:27:11.928Z</atom:updated>
            <content:encoded><![CDATA[<p>For the Impatient: Skip section</p><h4>Context</h4><p>Oracle Cloud Infrastructure recently announced schedule-based autoscaling for compute instances maximizing the cost-saving proposition as it built on per-second-billing and the flexibility of Universal Credits.</p><pre>1. <a href="https://docs.cloud.oracle.com/en-us/iaas/Content/Compute/Tasks/autoscalinginstancepools.htm#ScheduleBasedAutoscaling">Scedule based autoscaling</a><br>2. <a href="https://docs.cloud.oracle.com/en-us/iaas/releasenotes/changes/20facdce-34bf-4648-a0a7-de5db73657ce/">Per-Second Billing</a><br>3. <a href="https://docs.oracle.com/en/cloud/get-started/subscriptions-cloud/csgsg/universal-credits.html#:~:text=Oracle%20Cloud%20provides%20a%20flexible,eligible%20IaaS%20and%20PaaS%20services.">Universal Credits</a></pre><h3>Long Story Short</h3><pre>Use Schedule based autoscaler on an instance pool of size = 1</pre><p>Just follow the routine to create an autoscaling policy</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*F9Y_sRQBibiE4qoDvDEhCg.png" /><figcaption>Create an Instance Config from an Instance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*If19o0DrL-__DMIIY9zJqQ.png" /><figcaption>Create an Instance Pool from Instance Config</figcaption></figure><pre>Use Schedule based autoscaling<br>Refer to <a href="http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html">Quartz Cron Expressions</a><br>Don&#39;t change the pool size<br>Let it stay at &quot;1&quot; if it is 1.</pre><p>Instead of metrics-based autoscaling, use schedule-based autoscaling. and make sure your schedule based policies don’t change the instance pool size. i.e. you trigger an event but not for autoscaling but for something else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mS4ZvdvnK1nwBr4XQeF20A.png" /><figcaption>Create Autoscaling Policy with Instance Pool</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZWf4UeCBUu7pRCkWklnew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V4djHAj8wpXT_T0r" /><figcaption>The Contraption</figcaption></figure><pre><br>1) Autoscaling generates an <strong>&quot;ScalingEvent&quot;</strong> event action<br>2) Event Action can be processed by <strong>Events Service</strong><br>3) Events Service can trigger <strong>Functions </strong><br>4) <strong>Functions </strong>become your Actuation layer</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JS16LiS9lQEF0UXhASgoQ.png" /><figcaption>Create a Function Trigger</figcaption></figure><h4>Use Attribute filters for Routing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFYgKgD8DgqJrQNqPwxmMw.png" /></figure><h3>What if I wanted a bigger Contraption</h3><p>Once the data moves to the actuation layer, Function calls get decoupled from one another using Notifications/ Streaming.</p><pre><strong>Streaming </strong>- would have data published to it by Functions, but have to be read by a group of consumers periodically, which can also be a scheduled autoscaling policy triggered function that reads from the stream every “1m”</pre><pre><strong>Notifications</strong> - We could have functions publish to a bunch of notifications that simply trigger functions on Notification publish. But limited to 60 events per minute per topic. </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MJWLP2yyVGLwSI_w" /><figcaption>The Bigger Contraption</figcaption></figure><h3>Extend the contraption</h3><p>Functions are my actuation layer, i.e. short-lived, ephemeral programs, that can leverage massive parallelism but I would like to break up a large task into small tasks using a Fan-Out Logic. Fundamental problems with this approach are having someone maintain state.</p><h4>Maintain State through the Task Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j5YjtEA_iGeamidL" /></figure><h4>Fan-Out &amp; Micro-Batch</h4><p>A single function could decide how many functions if would like to call and publish a proportionate number of Notifications. Publish to multiple topics which have the same function as the trigger action to overcome the 60 events per minute limitation too. The idea is to have state maintained in Streaming or in a database service like NoSQL database</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uZhFQkvp1oWSZOlB" /></figure><h3>Leverage the Power of Cron</h3><pre>The CRON Expression is the logic.</pre><p>https://www.freeformatter.com/cron-expression-generator-quartz.html</p><h3>More Power to you with</h3><pre>Events<br>Functions<br>Streaming<br>Notifications</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4b8e6fed288" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Better Alerts, Notifications & Incident triggering with Monitoring, Functions and Search on Oracle…]]></title>
            <link>https://vamsiramakrishnan.medium.com/better-alerts-notifications-incident-triggering-with-monitoring-functions-and-search-on-oracle-eb2ce70230da?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb2ce70230da</guid>
            <category><![CDATA[oracle]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[incident-response]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Tue, 16 Jun 2020 12:47:36 GMT</pubDate>
            <atom:updated>2020-06-16T12:51:26.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>Better Alerts, Notifications &amp; Incident triggering with Monitoring, Functions and Search on Oracle Cloud Infrastructure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*cw4x7K1r56LApu5u.jpg" /></figure><p>One of the most defining characteristics of a reliability/monitoring/production-support engineer is working on-call shifts, mitigating incidents, and responding to production issues.</p><blockquote>Running an Ops Team/ Production Support/ SRE team is a lot like running an Emergency Response Team.</blockquote><h3>MTTR ( Mean Time to Repair/Resolve/Remediate)</h3><p>Your team could be equipped with sophisticated real-time operations and incident-response tooling, it wouldn’t matter if you had an inefficient upstream incident information generation, issues will nevertheless overwhelm the system.</p><pre>Successful Emergency Responses depend on the ability of the responder(human/machine) to successfully extract maximum information from the incident reporter (human/machine) </pre><h3>What if incident reporters can report incidents better?</h3><pre>&quot;For an SRE, automation is a force multiplier, not a panacea. Of course, just multiplying force does not naturally change the accuracy of where that force is applied: doing automation thoughtlessly can create as many problems as it solves. Therefore, while we believe that software-based automation is superior to manual operation in most circumstances, better than either option is a higher-level system design requiring neither of them - an autonomous system. Or to put it another way, the value of automation comes from both what it does and its judicious application. We&#39;ll discuss both the value of automation and how our attitude has evolved over time&quot;</pre><pre><strong>Niall Murphy, John Looney, Michael Kacirek</strong></pre><blockquote>What if people were trained better to make better 911 calls.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*UebQAGhO4t5xMKjw.png" /></figure><h3>The solution</h3><p>Enrich the Alarm notifications with OCI- Functions that use OCI-Search and maybe even get resource context or even the Resource endpoint for detailed information about the resource. One could even configure the function to do special tasks that aren’t limited to pulling resource information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*nRQibIMwS7ZeekBETs74ug.png" /></figure><h3>It’s all about Context</h3><p>Let’s understand the Alarms payload,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*rroVKyvCBNPaZgFLZfHxig.png" /><figcaption>Example Alarms Payload</figcaption></figure><h4>Title</h4><p>What you configure as the alarm title, remember to write a meaningful title.</p><h4>Status</h4><ul><li>OK_TO_FIRING: Things are getting worse and need your attention</li><li>FIRING_TO_OK: Things are getting better and you can panic a little lesser</li><li>REPEAT: Whatever it is that you’re doing isn’t helping</li><li>RESET: You’re either wasting your time by checking because the resource doesn’t exist or something is very wrong</li></ul><h4>Severity</h4><p>How important is this alert for you</p><h4>Alarm Metadata</h4><p>The part that provides you with resource information on why the alarm fired.</p><pre>Alarms are the boolean evaluations of Metric Queries with a state variable to understand transitions from good to worse or vice-versa. </pre><p>They contain critical pieces of information about</p><pre>1. The Metric Query and the evaluation<br>2. What was the result of the List Metrics Activity<br>3. Number of resources where this evaluation is passing</pre><h4>About Metric Queries</h4><pre>metric[<em>interval</em>]{dimensionname=&quot;dimensionvalue&quot;}.groupingfunction.statistic</pre><pre><strong>Further Reading: </strong><a href="https://medium.com/@vamsiramakrishnan/so-lets-talk-about-metrics-in-oracle-cloud-infrastructure-de776a04a891">Metrics</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/0*8XOwhm6Lh09jiYPU" /></figure><pre>1. Multiple Resources, of same resource grouped as a namespace<br>2. Every namespace has multiple metrics, therefore a resource emits mulltiple metrics</pre><pre>n:n relationship between metrics and resources<br>1:1 relationship between metrics &amp; metric queries</pre><h3>Enrichment Options</h3><p>The Incident response mode dictates, what kind of enrichment strategy to choose, a machine-readable OCID is a better strategy to choose if the responder to the action is a machine and not a human. It could be a combination of both, so you’d probably have to mine as much information as possible.</p><pre>How does one locate this resource with ease <br>-------------------------------------------</pre><pre>1) A Resource Display name or OCID?<br>2) Which Compartment does it belong to Name or OCID? <br>3) In which region is this present ? </pre><h4><strong>Enrich with Supplementary Data</strong></h4><pre>For Compute Instances if I see the CPU Utilization Shoot Up , do I also need the Block Volume attachments to that instance and the number of TCP Connections, the upstream load-balancer is receiving ?</pre><h3>The Implementation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pFZ4LMBjPR2pIaL-" /></figure><h4>Better Incident reporting</h4><pre>Read Alarm Body -&gt; <br>Extract Alarm metadata -&gt; <br>Enrich Metadata with Resource Fetch -&gt;<br>Publish Notification -&gt;<br>Configure Notification to Publish to HTTP Endpoint/Slack/PagerDuty </pre><h4>Better Email Readability</h4><pre>Read Alarm Body -&gt; <br>Extract Alarm metadata -&gt; <br>Enrich Metadata with Resource Fetch -&gt;<br>Format Email -&gt;<br>Send Email on SMTP credentials through email delivery-&gt;</pre><p><a href="https://docs.cloud.oracle.com/en-us/iaas/Content/Email/Reference/python.htm">Integrating Python with Email Delivery</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb2ce70230da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So let’s talk about Metrics in Oracle Cloud Infrastructure]]></title>
            <link>https://vamsiramakrishnan.medium.com/so-lets-talk-about-metrics-in-oracle-cloud-infrastructure-de776a04a891?source=rss-e02af0a414b8------2</link>
            <guid isPermaLink="false">https://medium.com/p/de776a04a891</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[monitoring]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[oracle-cloud]]></category>
            <category><![CDATA[oracle]]></category>
            <dc:creator><![CDATA[Vamsi Ramakrishnan]]></dc:creator>
            <pubDate>Tue, 09 Jun 2020 19:50:57 GMT</pubDate>
            <atom:updated>2020-06-09T19:50:57.119Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aAnh34x6SNiijoMhz2da1Q.jpeg" /></figure><p>This post is to understand how metrics are organized in OCI and how would you go about exporting all metrics, from all resources, in the tenancy to a common space such as object storage. If you’ve come here to understand how to visualize OCI Metrics in an external tool like Grafana, check this page out.</p><p><a href="https://github.com/oracle/oci-grafana-plugin">oracle/oci-grafana-plugin</a></p><p>Every service in OCI emits metrics a.k.a KPIs about the service. The exhaustive list of all services and metrics in the <a href="https://docs.cloud.oracle.com/en-us/iaas/Content/Monitoring/Concepts/monitoringoverview.htm#SupportedServices">link</a></p><h3>For the Impatient</h3><p>Create a Compartment metricsCompartment copy the OCID and keep it handy. The script then uses this OCID and fetches all regions subscribed and creates a bucket for each compartmentwithin the metricscompartment and then creates compressed objects in the log.gzformat each metric within that namespace for the time period specified. Code in the gist below.</p><pre>!! Python &amp; Jupyter Understanding is a must</pre><h3>The scenario I tested it for</h3><pre>5 Regions <br>151 Compartments<br>1280 Resources<br>Avg, 5-10 Metrics per Resource<br>Mean, Median, Max, Min, Sum, Rate, Percentile(.5, 0.9, .95, .99)<br>12 Hours of data<br>1m resoultion<br>-----------------------------------------------<br>1280x10x10x12x60 = <strong>92,160,000‬ Data Points</strong> (92 Million Data Points)<br>-----------------------------------------------</pre><h3>The Entity Hierarchy</h3><pre>└── tenancy<br>     └── regions<br>            └── compartments<br>                   └── namespaces<br>                         └── metrics<br>                              └── statistics</pre><h3>Explanation of Hierarchy</h3><pre>1. <strong>Namespace </strong>is an identification of the kind of resource/service eg. Compute, Block Storage </pre><pre>2. <strong>Metrics </strong>are what the resources emit a.k.a KPIs eg. CpuUtilization</pre><pre>3. <strong>Statistics </strong>is a computation or mathematical/logical quantity <br>Eg. Min, Max, Mean, Rate, Sum over Specified Interval</pre><h3>The Parameters</h3><pre><strong>Period: </strong>    From when to Until when<br><strong>Resolution:</strong> Data points per Unit Time, also known as <strong>Interval</strong></pre><h3>The Filters</h3><pre><strong>Dimensions</strong>: A way to filter  a single resource or group multiple resources based on an identifer / name / tag etc.<br><strong>Expressions</strong>: To apply logical expressions, &gt;,&lt;,=,!= etc.</pre><h3>The Representation</h3><pre>Aggregated : (Σ) over Interval <br>Standalone : Data point at that point in time</pre><p>When you combine all of this you have the</p><blockquote><strong>MQL — Monitoring Query Language</strong></blockquote><h3>Query</h3><pre>[<br>{&quot;namespace&quot;:&quot;oci_faas&quot;,&quot;query&quot;:&quot;FunctionInvocationCount[60m].sum()&quot;,&quot;endTime&quot;:&quot;2020-06-05T12:00:00.000Z&quot;,&quot;startTime&quot;:&quot;2020-03-20T10:00:00.000Z&quot;},<br>{&quot;namespace&quot;:&quot;oci_faas&quot;,&quot;query&quot;:&quot;FunctionExecutionDuration[60m].mean()&quot;,&quot;endTime&quot;:&quot;2020-06-05T12:00:00.000Z&quot;,&quot;startTime&quot;:&quot;2020-03-20T10:00:00.000Z&quot;},<br>{&quot;namespace&quot;:&quot;oci_faas&quot;,&quot;query&quot;:&quot;FunctionResponseCount[60m]{responseType = \&quot;Throttled\&quot;}.sum()&quot;,&quot;endTime&quot;:&quot;2020-06-05T12:00:00.000Z&quot;,&quot;startTime&quot;:&quot;2020-03-20T10:00:00.000Z&quot;}<br>]</pre><p>To understand how to extend OCI Metrics into</p><h3>Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oA5FYzolpZ01deOO98gVGw.png" /></figure><h3>What If</h3><p>We wanted to store and export</p><pre>Every statistic<br>From every metric<br>Emitted by Every resource<br>In Every Namespace<br>Inside every compartment<br>of Every region subscribed to<br>in the Tenancy</pre><h3>How</h3><pre><strong>USE RECURSION</strong></pre><p>Metrics are regional resources. Metrics Service Endpoints have Regional</p><pre>https://telemetry-ingestion.ap-hyderabad-1.oraclecloud.com</pre><h4>Use Object Storage</h4><p>Object Storage is a regional resource and is massively and horizontally scalable. Object storage is fully capable of generating events on CRUD and hence is the best place for storing files.</p><pre>Compress json stream to .gz<br>Every Metric is an Object<br>Use &quot;/&quot; with Namespaces, Time etc. <br>Every Compartment is a Bucket<br>Emit Events in the Object Lifecycle so you can do ETL on Metrics</pre><h4>Bucket</h4><p>Object Storage namespaces enforce Bucket-uniqueness across compartments within a given region. Hence, within a given region’s metric compartment the bucket names should represent the compartment the namespaces belong to and hence this strategy</p><pre>oci-logs._monitoring.ocid1.compartment.oc1..aaaaaaaaqye<br><strong>BucketPrefix.Monitoring.CompartmentOCID</strong></pre><h4>Object</h4><p>The JSON Time Series file has several repetitive elements such as field and timestamp and this provides great scope for compression. However, we could use space-efficient formats such as influx formats instead of compression with gunzip as a strategy</p><pre>oci_blockstore/VolumeReadOps/02062020_0714__02142020_0814.json.gz<br><strong>Namespace/Metric/FromTime_ToTime.json.gz</strong></pre><h3>Things to remember</h3><pre>metric[<em>interval</em>]{dimensionname=&quot;dimensionvalue&quot;}.groupingfunction.statistic</pre><p>[1m] Interval on a time range of 7 dayswould provide on a metric would provide 24 x 7 x 60 = 10080 data points of a given statistic on a given metric in a given namespace in a compartment</p><h4>Limits</h4><pre>+----------+------------+------------+<br>| Interval | resolution | time range |<br>+----------+------------+------------+<br>| 1d       | 1 day      | 90 days    |<br>| 1h       | 1 hour     | 90 days    |<br>| 5m       | 5 minutes  | 30 days    |<br>| 1m       | 1 minute   | 7 days     |<br>+----------+------------+------------+</pre><h3>And when you write it as code</h3><p>For efficiency, we useconcurrent.futures to leverage asynchronously and parallelism. The code takes a good one hour to run and parallelizes runs across multiple regions if run from a single VM , what could be done is reduce the latency of data traveling back and forth by running it for a single region from a VM inside that region.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7160ac11fd07f2a6532bd8350dea13ae/href">https://medium.com/media/7160ac11fd07f2a6532bd8350dea13ae/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de776a04a891" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>