<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Peter Götz on Medium]]></title>
        <description><![CDATA[Stories by Peter Götz on Medium]]></description>
        <link>https://medium.com/@peter.gtz?source=rss-c6dc59138c47------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dZJUZo7Vu4pZJdK-KxAYlQ.jpeg</url>
            <title>Stories by Peter Götz on Medium</title>
            <link>https://medium.com/@peter.gtz?source=rss-c6dc59138c47------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:38:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@peter.gtz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MyGPT — or how the first ChatGPT powered Alexa Skill was born]]></title>
            <link>https://medium.com/@peter.gtz/mygpt-or-how-the-first-chatgpt-powered-alexa-skill-was-born-72e28187c860?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/72e28187c860</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[alexa]]></category>
            <category><![CDATA[chatgpt-chatbot]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[alexa-skills]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Sat, 06 Jul 2024 21:08:40 GMT</pubDate>
            <atom:updated>2024-07-06T21:23:33.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kyNKlOTxDLAPcS4o_1sKQ.png" /></figure><h3>MyGPT — or how the first ChatGPT powered Alexa Skill was born</h3><p><em>This is blog post #1 of a mini series of posts about my </em><a href="https://www.amazon.ca/dp/B0CHSGVFBM/"><em>Alexa Skill that uses ChatGPT</em></a><em> to make Alexa smart.</em></p><p>At the beginning of 2023, the world was still hyped by the advent of ChatGPT. But for some reason, no one had seriously made an effort to combine it with Alexa. That was strange, because ChatGPT was a purely language based assistant, and Alexa is a purely voice controlled assistant, just not as smart as ChatGPT. And so I set out to fill that gap and connect the two in an Alexa Skill.</p><p>The first prototype was rather quick to do. I already had a Wikipedia Skill (<a href="https://github.com/petergtz/alexa-wikipedia">written in Go</a>) publicly available (<a href="https://www.amazon.com/Peter-G%C3%B6tz-Wikipedia/dp/B07BKRT478">en</a>, <a href="https://www.amazon.de/Peter-G%C3%B6tz-Wikipedia/dp/B07BKRT478">de</a>, <a href="https://www.amazon.es/Peter-G%C3%B6tz-Wikipedia/dp/B07BKRT478">es</a>). All I had to do, was fork it, and replace the <a href="https://www.mediawiki.org/wiki/API:Main_page">API calls to Wikimedia</a> with ones to <a href="https://platform.openai.com/docs/api-reference">OpenAI</a>. It was literally done in a day.</p><p>And wow, what a result. Alexa suddenly felt like the Star Trek Computer. You could ask her arbitrary things, have actual conversations! Yes, just like all the jaw dropping things people experience when using ChatGPT the first time. But now you could… just talk!</p><p>And so, MyGPT was born.</p><p><em>If you want to </em><strong><em>try it</em></strong><em> out, before you continue to read, </em><strong><em>activate</em></strong><em> it for your country here:</em><a href="https://www.amazon.de/dp/B0CHSGVFBM/"><em> </em>🇩�</a>� |<a href="https://www.amazon.com/dp/B0CHSGVFBM/"> 🇺�</a>� |<a href="https://www.amazon.ca/dp/B0CHSGVFBM/"> 🇨�</a>� |<a href="https://www.amazon.in/dp/B0CHSGVFBM/"> 🇮�</a>� |<a href="https://www.amazon.co.uk/dp/B0CHSGVFBM/"> 🇬�</a>� |<a href="https://www.amazon.com.au/dp/B0CHSGVFBM/"> 🇦�</a>� | <a href="https://www.amazon.fr/dp/B0CHSGVFBM/">🇫🇷</a> <em>. These link directly to the successor skill that can be invoked using the name “ChatGPT”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*lnMl2adsrU9sDvY8EQK9Ng.png" /><figcaption>The MyGPT logo (later also used for its sibling skill “Chatbot powered by ChatGPT”)</figcaption></figure><p>Coming back to the implementation, the truth is, it wasn’t that easy. The prototype did work, yes, but it had many issues. They mostly had to do with how to deal with slow responses from OpenAI and general network connection issues. To get to 100% success rate on Alexa responses, I had to address a lot of subtle issues. The rest of this post explains these.</p><p>Some background on what an Alexa Skill is and how it works: Alexa Skills are the equivalent to apps on smartphones. Users need to activate them explicitly, and they get invoked by the user saying something like “Alexa, open My Encyclopedia”.</p><p>The way an Alexa Skill works, is, that when a user asks: “Alexa, ask MyGPT how tall the Eiffel tower is”, the device sends the audio to the Alexa service, the Alexa service transcribes it (speech to text), determines that it’s a request to a specific skill, sends it off to the skill’s endpoint, in MyGPT’s case an <a href="https://aws.amazon.com/lambda">AWS Lambda function</a>, and passes a <a href="https://en.wikipedia.org/wiki/JSON">JSON</a> payload to it, that includes the utterance from the user. Now the Alexa service will wait for a maximum of 8s for a response from that endpoint. I.e., a JSON response payload that includes the written speech that Alexa should read to the user. If the response doesn’t come within 8s, Alexa will tell the user that there was an error with the skill and end the session.</p><p>It’s important to note here, that there’s no streaming option available with Alexa. It’s a classic synchronous request/response model. Mostly at least. But we’ll get to that in a second.</p><p>OpenAI, however, does offer a <a href="https://platform.openai.com/docs/api-reference/streaming">streaming API</a>. So we can make an HTTP request against OpenAI and collect tokens until 7.5s have passed (a little bit of time buffer, just in case), cut off the connection here, and send back to the user whatever we got so far. That’s not a good user experience though yet. Instead, we need to cut off at the end of a sentence. In Go, there is a convenient library to identify the end of a sentence: <a href="https://pkg.go.dev/github.com/neurosnap/sentences">neurosnap/sentences</a>. With that, it’s possible for Alexa to give the response proper intonation, because it’s made up of complete sentences. We also know when the answer is incomplete, because we cut off the response from OpenAI. When it is incomplete, we need to ask the user if they want to hear more.</p><p>That solves the problem of sessions ending with an error due to timeout. However, asking the user is if they want to hear more, is certainly a controversial design choice about this Skill. Indeed, people complain about it often, wondering why the Skill doesn’t just keep on talking. Unfortunately, I believe there’s no perfect solution with Alexa’s 8s limit and the limited throughput OpenAI provides. So I settled with that approach.</p><p>The experience was still not ideal for another reason: For some answers, you’d have to wait for almost 8s, even though OpenAI provided a partial response much earlier. Waiting for 8s can seem like an eternity when talking to an assistant. In reality, this wasn’t acceptable. Some googling revealed, there is a way to “stream” responses to Alexa. It’s not really streaming, but instead you can send “<a href="https://developer.amazon.com/en-US/docs/alexa/custom-skills/send-the-user-a-progressive-response.html">Progressive Responses</a>” using Alexa’s API. I.e., before providing your final answer, you can already send partial responses to the user. So I changed the implementation to not just iterating through the received tokens from OpenAI and collecting them, but also to send off every 1s a partial response including all complete sentences received so far.</p><p>That worked very well! With one final small issue: For some reason, Alexa reads progressive responses with a quieter voice than the final response. It really sounded awkward, because she would seem to start shouting at you for the last piece of response she gave you. Of course, the fix was easy: use the <a href="https://developer.amazon.com/en-US/docs/alexa/custom-skills/speech-synthesis-markup-language-ssml-reference.html#prosody">volume attribute</a> in the response to turn up the progressive response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5l3HbOot2sPOAi6TFWYsQ.png" /><figcaption>Showing the median and p90 response time in milliseconds (not including progressive responses; these are even faster). As we can see, the median is mostly below 2.5s. It also shows that the slowest 10% requests can easily go to 7s and above.</figcaption></figure><p>At this point, I felt confident that the skill was good enough to be published. So I did submit it and now it’s available in six different countries in English and German (it’s sibling, Chatbot powered by ChatGPT, also supports French and is available in 7 countries).</p><p>The experience was still not perfect. Every once in while, OpenAI would be so slow, that not even a single sentence was available after 1s, 2s or 3s had passed. After a bit of experimenting, I inserted an “um… um…” in that case. However, people got annoyed by this, because it happened more often than I thought. Eventually, I changed it to a subtle jingle.</p><p>Finally, there’s error conditions where the HTTP connection just hangs. It took me quite a while to understand what’s going on. Fortunately, there’s a intuitive configuration parameter in Go’s HTTP client called timeout. Setting it to 7s made sure, the client would back out in time and the skill is able to tell the user, something is wrong and asking them if the Skill should try again.</p><p>HTTP errors are another issue. The simplest I found to deal with all of them, is to use Hashicorp’s excellent <a href="https://pkg.go.dev/github.com/hashicorp/go-retryablehttp">go-retryablehttp</a> library and set retry wait times to very low values and only retry twice. Only this setting gives us fast enough reaction to try again, or back out completely.</p><pre>retryableHttpClient := retryablehttp.NewClient()<br><br>retryableHttpClient.HTTPClient.Timeout = 7000 * time.Millisecond<br>retryableHttpClient.RetryMax = 2<br>retryableHttpClient.RetryWaitMin = 10 * time.Millisecond<br>retryableHttpClient.RetryWaitMax = 50 * time.Millisecond</pre><p>Finally, to make sure it really never ever hangs for whatever reason, we wrap everything into a select statement with a timer:</p><pre>chatCompletionTimer := time.NewTimer(7500 * time.Millisecond)<br>fullSentencesSinceLastProgressiveResponseChan := make(chan string, 1)<br>entireSpeechChan := make(chan string, 1)<br><br>go func() {<br>    // get the actual responses from ChatGPT ...<br>    // write to fullSentencesSinceLastProgressiveResponseChan,<br>    //          entireSpeechChan<br>}()<br><br>var fullSentencesSinceLastProgressiveResponse, entireSpeech string<br>select {<br>case &lt;-chatCompletionTimer.C:<br>    fullSentencesSinceLastProgressiveResponse = l.Get(r.TakingLongerToGetResponse)<br>    entireSpeech = fullSentencesSinceLastProgressiveResponse<br>case fullSentencesSinceLastProgressiveResponse = &lt;-fullSentencesSinceLastProgressiveResponseChan:<br>    entireSpeech = &lt;-entireSpeechChan<br>    chatCompletionTimer.Stop()<br>}</pre><p>With these changes, I got 100% success rate on responses. No timeouts, no errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3albt21z51ifHBMZiddQg.png" /></figure><p>And that’s it for this post. In the next posts, I will talk about how I introduced a feedback features, user frustration, a new skill name, different voices and more.</p><p><em>Again, if you want to </em><strong><em>try it</em></strong><em> out, </em><strong><em>activate</em></strong><em> it for your country here:</em></p><p><a href="https://www.amazon.ca/dp/B0CHSGVFBM/">🇩�</a>� |<a href="https://www.amazon.com/dp/B0CHSGVFBM/"> 🇺�</a>� |<a href="https://www.amazon.ca/dp/B0CHSGVFBM/"> 🇨�</a>� |<a href="https://www.amazon.in/dp/B0CHSGVFBM/"> 🇮�</a>� | <a href="https://www.amazon.co.uk/dp/B0CHSGVFBM/">🇬🇧</a> |<a href="https://www.amazon.com.au/dp/B0CHSGVFBM/"> 🇦�</a>� |<a href="https://www.amazon.fr/dp/B0CHSGVFBM/"> 🇫�</a>� .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72e28187c860" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mocking your Way into Go Generics]]></title>
            <link>https://medium.com/@peter.gtz/mocking-your-way-into-go-generics-fdf8c2a41a18?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdf8c2a41a18</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[mocking]]></category>
            <category><![CDATA[generics]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Thu, 11 May 2023 16:13:16 GMT</pubDate>
            <atom:updated>2023-05-11T16:13:16.306Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Pegomock supports Go Generics in its new release</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nSSRmzESyfE1OJhxLJsdBg.png" /><figcaption>Pegomock is a powerful, yet simple mocking framework for the Go programming language</figcaption></figure><p>I started <a href="https://github.com/petergtz/pegomock">Pegomock</a> in 2015, because I wasn’t happy with the existing frameworks, such as <a href="https://github.com/golang/mock">GoMock</a>, <a href="https://github.com/stretchr/testify">Testify</a>, or <a href="https://github.com/maxbrunsfeld/counterfeiter">Counterfeiter</a>. I missed the ease of use I knew from Java’s <a href="https://javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/Mockito.html">Mockito</a>. To jumpstart the development of Pegomock, I forked GoMock and bent its <a href="https://en.wikipedia.org/wiki/Domain-specific_language">DSL</a> to my needs.</p><p>Since then, Go has evolved quite a bit. Most notably, it has introduced <a href="https://go.dev/doc/tutorial/generics">generics</a>. However, until recently, Pegomock did not support generics, just like Counterfeiter or GoMock don’t support it. The only exception was Testify. All this changed though with Pegomock’s new <a href="https://github.com/petergtz/pegomock/releases/tag/v3.0.1">versions 3</a> and <a href="https://github.com/petergtz/pegomock/releases/tag/v4.0.0">version 4</a>, which I’ve released earlier this week. This means, you can now mock interfaces that have type parameters. Let’s look at an example:</p><pre>type Repository[Item any] interface {<br>    func GetBy(id int) Item<br>    func Store(id int, item Item)<br>}</pre><p>Repository is a popular pattern, used to abstract away from the concrete implementation of some kind of persistence, such as a database, flatfiles, or even in-memory. To generate a mock, in a terminal we can now run:</p><pre>pegomock generate Repository</pre><p>This will give us a new struct type MockRepository[T any] and a function NewMockRepository[T any], which we can use in our tests. E.g.:</p><pre>package repo_test<br><br>import (<br>    &quot;testing&quot;<br>    &quot;github.com/petergtz/pegomock/v4&quot;<br>)<br><br>type User struct {<br>    Username string<br>}<br><br>func TestGetFromRepo(t *testing.T) {<br>    repo := NewMockRepository[User](pegomock.WithT(t))<br>    When(repo.GetBy(AnyInt()).ThenReturn(User{Username: &quot;Tom&quot;})<br><br>    // use mock here<br>}</pre><p>Furthermore, matcher handling has also become much easier. Let’s assume we want to verify that something was stored in the repository:</p><pre>func TestStoreInRepo(t *testing.T) {<br>    // given:<br>    repo := NewMockRepository[User](pegomock.WithT(t))<br><br>    // when:<br>    ifUserIsEligibleStoreIn(repo, User{Username: &quot;Tom&quot;})<br><br>    // then:<br>    repo.VerifyWasCalledOnce().Store(AnyInt(), Eq(user))<br>}</pre><p>The Eq matcher now takes a generic argument. Before generics, we had to <em>generate</em> matchers for custom types, and would generate lots of functions like EqUser().</p><p>So next time you need a mocking framework in Go, try Pegomock. Install it via:</p><pre>go install github.com/petergtz/pegomock/v4/pegomock@latest<br>go get github.com/petergtz/pegomock/v4@latest</pre><p><a href="https://github.com/petergtz/pegomock">GitHub - petergtz/pegomock: Pegomock is a powerful, yet simple mocking framework for the Go programming language</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdf8c2a41a18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backing Up Google Photos to Amazon S3, Part 2]]></title>
            <link>https://medium.com/better-programming/backing-up-google-photos-to-amazon-s3-part-2-eaddc9087d1f?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaddc9087d1f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-cdk]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Mon, 13 Feb 2023 16:31:18 GMT</pubDate>
            <atom:updated>2023-02-13T16:31:18.305Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using the AWS Cloud Development Kit (CDK) for Infrastructure as Code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2uOIV8jtp9_TB94fS4biqQ.png" /></figure><p>I wanted to learn Rust, and also try out AWS Cloud Development Kit (CDK) — and secure myself from the unlikely case of being locked out of my Google account and losing 500GB worth of photos and videos.</p><p>In Part II of this two-part blog series, I’m describing how I embedded the binary written in Rust from <a href="https://betterprogramming.pub/backing-up-google-photos-to-amazon-s3-221e42a32a95">Part I</a> into an automated system for backups using CDK and GitHub Actions.</p><p>Again, if you’d like to skip the story here and just set up the same solution I’m describing here, feel free to head over directly to <a href="https://github.com/petergtz/g2s3">https://github.com/petergtz/g2s3</a> and follow the instructions. It’s not a zero-conf setup and it’ll require some technical knowledge to set up, but it’s doable.</p><h3>Idea</h3><p>As explained in Part I, we use Google Takeout to regularly export all data from Google Photos into Google Drive. I.e., every two months, the data will be dropped into a Takeout folder in Google Drive, ready to be picked up and copied over to S3.</p><p>Obviously, we will need an S3 folder to store the data and some regular job to copy the data; we will need a Docker image with the executable inside that performs the copy, etc. But let’s do this one by one. Let’s first get an overview of the entire system, and then let’s dive into each section and see how we solve it using CDK. Here’s an overview of the architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Zi-mg2qz7ImdjVQj03iqg.png" /><figcaption>The micro-service architecture in AWS Cloud</figcaption></figure><p>We use an <a href="https://aws.amazon.com/eventbridge/">Amazon EventBridge</a> Rule to trigger an <a href="https://aws.amazon.com/batch/">AWS Batch</a> job every two months (1). The Batch job will first fetch some secrets needed to access our Google Drive account (2). Then it will schedule a Task in <a href="https://aws.amazon.com/ecs/">Amazon ECS</a> (3), which runs back-up-drive-folder, which in turn will talk to Google Drive through an Internet Gateway (4) and copy the backup data into an S3 Bucket (5).</p><p>Once done, AWS Batch will publish a completion event on EventBridge (6) which will call an <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> function (7), which will then send a success or failure message to an <a href="https://aws.amazon.com/sns/">Amazon SNS</a> topic (8), which will send an email to us with the status (9).</p><h3>Configuration</h3><p>To simplify the setup, I’ve defined two config structs as follows:</p><pre>interface Config {<br>    backup_definitions: BackupDefinition[];<br>    email?: string,<br>    image?: string,<br>}<br><br>interface BackupDefinition {<br>    google_drive_folder: string;<br>    google_secrets: SecretProperty[],<br>    s3_url: string,<br>    should_create_bucket: boolean,<br>    storage_class?: string,<br>    schedule?: CronOptions;<br>}</pre><p>With this, we’re quite flexible in setting things up, and it even allows us to define multiple backups. My setup looks roughly like this:</p><pre>{<br>  &quot;email&quot;: &quot;my.email@example.com&quot;,<br>  &quot;backup_definitions&quot;: [<br>    {<br>      &quot;s3_url&quot;: &quot;s3://&lt;MY_PERSONAL_BUCKET&gt;/{date}/Takeout&quot;,<br>      &quot;should_create_bucket&quot;: false,<br>      &quot;storage_class&quot;: &quot;DEEP_ARCHIVE&quot;,<br>      &quot;google_drive_folder&quot;: &quot;Takeout&quot;,<br>      &quot;google_secrets&quot;: [<br>        ...snip...<br>      ],<br>      &quot;schedule&quot;: {<br>        &quot;minute&quot;: &quot;0&quot;,<br>        &quot;hour&quot;: &quot;0&quot;,<br>        &quot;day&quot;: &quot;1&quot;,<br>        &quot;month&quot;: &quot;*/4&quot;,<br>        &quot;year&quot;: &quot;*&quot;<br>      }<br>    }<br>  ]<br>}</pre><h3>The AWS Batch Job</h3><p>Let’s start with the heart of our CDK setup: the batch job. There are quite some things to configure here. With an instance backupDef of BackupDefinition and config of Config, let’s define our job:</p><pre>import * as batch from &#39;aws-cdk-lib/aws-batch&#39;;<br><br>// ...<br><br>const jobDefinition = new batch.CfnJobDefinition(<br>    scope,<br>    `google-${backupDef.google_drive_folder}-backup-to-s3-job-def`,<br>    {<br>        type: &quot;container&quot;,<br>        jobDefinitionName: `google-${backupDef.google_drive_folder}-backup-to-s3`,<br>        containerProperties: {<br>            command: command,<br>            image: config.image,<br>            jobRoleArn: batchJobRole.roleArn,<br>            executionRoleArn: batchJobExecutionRole.roleArn,<br>            networkConfiguration: {assignPublicIp: &quot;ENABLED&quot;,},<br>            resourceRequirements: [<br>                {type: &quot;VCPU&quot;, value: &quot;1&quot;},<br>                {type: &quot;MEMORY&quot;, value: &quot;2048&quot;}<br>            ],<br>            secrets: backupDef.google_secrets,<br>        },<br>        platformCapabilities: [&quot;FARGATE&quot;],<br>    });</pre><p>As can be seen by the Cfn-prefix, we use a so-called <a href="https://docs.aws.amazon.com/cdk/v2/guide/constructs.html#constructs_lib">L1 construct</a> to define the job, as an L2 construct does not exist for AWS Batch. Some of the properties we provide here are intuitive to use, some are not. Let’s look at some of those:</p><ul><li>command: this is the command we actually want to execute within our container. No magic here:</li></ul><pre>let command = [&quot;/back-up-drive-folder&quot;, backupDef.google_drive_folder, backupDef.s3_url];<br>if (backupDef.storage_class) {<br>    command.push(&quot;--s3-storage-class&quot;, backupDef.storage_class)<br>}</pre><ul><li>image: If no image is provided as part of configuration, this defaults to: config.image = &quot;ghcr.io/petergtz/g2s3/g2s3:latest&quot;</li><li>jobRoleArn and executionRoleArn: These are frankly the most confusing ones to me and the documentation really doesn’t make it easier. I found by trial and error how to configure them:</li></ul><pre>import * as iam from &quot;aws-cdk-lib/aws-iam&quot;;<br><br>// ...<br><br>let batchJobRole = new iam.Role(scope, &quot;google-takeout-backup-job-role&quot;, {<br>    assumedBy: new iam.ServicePrincipal(&#39;ecs-tasks.amazonaws.com&#39;),<br>})<br>let batchJobExecutionRole = new iam.Role(scope, &quot;google-takeout-backup-execution-job-role&quot;, {<br>    assumedBy: new iam.ServicePrincipal(&#39;ecs-tasks.amazonaws.com&#39;),<br>    managedPolicies: [<br>        iam.ManagedPolicy.fromAwsManagedPolicyName(&quot;SecretsManagerReadWrite&quot;),<br>        iam.ManagedPolicy.fromAwsManagedPolicyName(&quot;service-role/AWSBatchServiceRole&quot;),<br>    ],<br>});</pre><ul><li>networkConfiguration: We must enable public IP here for thecontainer to be able to talk to Google Drive.</li><li>resourceRequirements: We don’t have any specifically challenging requirements. So one CPU and 2GB of memory is sufficient.</li><li>secrets: These are the secrets necessary to access Google Drive. I have copied the secrets into AWS Secrets Manager manually through the console.</li><li>platformCapabilities: We’re hosting everything on <a href="http://ws.amazon.com/fargate">Fargate</a>, as this is the simplest setup and we have no specific hardware requirements.</li></ul><h3>Triggering the Batch Job</h3><p>If the job is supposed to run on a schedule defined by backupDef.schedule , we can define a rule to do so:</p><pre>import aws_events as events from &#39;aws-cdk-lib&#39;;<br><br>// ...<br><br>new events.Rule(scope, `run-google-${backupDef.google_drive_folder}-backup-to-s3-event-rule`, {<br>    enabled: true,<br>    ruleName: `run-google-${backupDef.google_drive_folder}-backup-to-s3`,<br>    schedule: Schedule.cron(backupDef.schedule),<br>    targets: [new targets.BatchJob(jobQueue.attrJobQueueArn, jobQueue, jobDefinition.ref, jobDefinition,<br>        {jobName: `google-${backupDef.google_drive_folder}-backup-to-s3`}<br>    )],<br>});</pre><p>To make this work, we need an AWS Batch job queue, here referred to asjobQueue , in which the job can run.</p><h3>Defining an AWS Batch Job Queue</h3><p>For our use case, there are no special requirements for the job queue, because we don’t have lots of jobs with competing priorities or similar. Still, the construction of it is, again, a bit more complex:</p><pre>const jobQueue = new batch.CfnJobQueue(scope, &quot;default-job-queue&quot;, {<br>        jobQueueName: &quot;default-job-queue&quot;,<br>        computeEnvironmentOrder: [{<br>            computeEnvironment: new batch.CfnComputeEnvironment(scope, &quot;default-compute-env&quot;, {<br>                computeEnvironmentName: &quot;default-compute-environment&quot;,<br>                type: &quot;MANAGED&quot;,<br>                computeResources: {<br>                    type: &quot;FARGATE&quot;,<br>                    maxvCpus: 256,<br>                    subnets: vpc.publicSubnets.map(s =&gt; s.subnetId),<br>                    securityGroupIds: [vpc.vpcDefaultSecurityGroup],<br>                },<br>                serviceRole: `arn:aws:iam::${Stack.of(scope).account}:role/aws-service-role/batch.amazonaws.com/AWSServiceRoleForBatch`,<br>            }).attrComputeEnvironmentArn,<br>            order: 0,<br>        }],<br>        priority: 0<br>    });</pre><p>We’re using L1 constructs again. Taking a closer look, the complex part is not the queue itself, but the compute environment for it. The compute environment requires us to specify a type which we set to &quot;MANAGED&quot;, because, we don’t have special requirements. And we need to define the compute resources.</p><p>We must define where these compute resources live, and therefore define a dedicated VPC for it. Since our job needs to talk to the outside world, we need a public subnet with an Internet gateway. So let’s create all that:</p><pre>import * as ec2 from &quot;aws-cdk-lib/aws-ec2&quot;;<br><br>// ...<br><br>let vpc = new ec2.Vpc(scope, &quot;vpc&quot;, {<br>    cidr: &quot;10.0.0.0/16&quot;,<br>    natGateways: 0,<br>    subnetConfiguration: [<br>        {<br>            name: &#39;public&#39;,<br>            cidrMask: 24,<br>            subnetType: ec2.SubnetType.PUBLIC<br>        }<br>    ],<br>    vpcName: &quot;google-backup-vpc&quot;,<br>})</pre><h3>Reporting Status via EventBridge, Lambda, and SNS</h3><p>Let’s first define the rule in EventBridge that “listens” to completed AWS Batch jobs:</p><pre>import * as targets from &quot;aws-cdk-lib/aws-events-targets&quot;;<br><br>// ...<br><br>new events.Rule(scope, jobDefinition.jobDefinitionName + &quot;-completed&quot;, {<br>    eventPattern: {<br>        source: [&quot;aws.batch&quot;],<br>        detailType: [&quot;Batch Job State Change&quot;],<br>        detail: {<br>            &quot;status&quot;: [&quot;FAILED&quot;, &quot;SUCCEEDED&quot;],<br>            &quot;jobDefinition&quot;: [jobDefinition.ref],<br>        }<br>    },<br>    enabled: true,<br>    ruleName: jobDefinition.jobDefinitionName + &quot;-completed&quot;,<br>    targets: [new targets.LambdaFunction(lambdaToForwardBatchCompletion)],<br>});</pre><p>The properties should be pretty self-explanatory. We could have used an SNS topic directly as a target. However, as it turned out, the default subject and message body, are not very descriptive. And so we put a Lambda function in between to fix this, as decribed <a href="https://aws.amazon.com/premiumsupport/knowledge-center/change-sns-email-for-eventbridge/">here</a>. Let’s see what lambdaToForwardBatchCompletion looks like:</p><pre>import * as lambda from &#39;aws-cdk-lib/aws-lambda&#39;;<br><br>// ...<br><br>const lambdaToForwardBatchCompletion = new lambda.Function(<br>    scope,<br>    &#39;lambda-forward-batch-completion-to-sns&#39;,<br>    {<br>        functionName: &quot;ForwardBatchCompletionStatusToSnS&quot;,<br>        runtime: lambda.Runtime.PYTHON_3_9,<br>        handler: &#39;index.handler&#39;,<br>        code: lambda.Code.fromInline(`<br>import boto3, json<br>def handler(event, context): boto3.client(&quot;sns&quot;).publish(<br>    TargetArn=&quot;${snsTopic.topicArn}&quot;, <br>    Message=json.dumps(event, sort_keys=True, indent=4), <br>    Subject=f&quot;AWS Batch job {event[&#39;detail&#39;][&#39;jobName&#39;]} {event[&#39;detail&#39;][&#39;status&#39;]}&quot;)`),<br>    });</pre><p>We provide the code for this function inline in Python. All this function does, is that it sends an SNS notification via the boto3 client.</p><p>Finally, we need to define the SNS topic and optionally an email subscription:</p><pre>import aws_sns as sns from &#39;aws-cdk-lib&#39;;<br><br>// ...<br><br>const snsTopic = new sns.Topic(scope, &quot;GoogleBackupToS3&quot;, {<br>    topicName: &quot;GoogleBackupToS3&quot;,<br>})<br>if (config.email) {<br>    new sns.Subscription(scope, &quot;GoogleBackupToS3-SnsSubscription&quot;, {<br>        protocol: SubscriptionProtocol.EMAIL,<br>        endpoint: config.email,<br>        topic: snsTopic,<br>    })<br>}</pre><p>And that’s it. We can now deploy this via cdk deploy.</p><h3>Generating the Container Image via GitHub Actions</h3><p>Let’s make building the container image reproducible too! For that, we can define a GitHub Actions workflow. The specific part that actually builds the image and pushes it to GitHub Container Registry, looks like this:</p><pre>- name: Build and publish Docker Images to GitHub Container registry<br>  uses: VaultVulp/gp-docker-action@1.5.0<br>  with:<br>    github-token: ${{ secrets.GITHUB_TOKEN }}<br>    image-name: g2s3<br>    image-tag: latest<br>    build-context: ./rust<br>    dockerfile: ./rust/Dockerfile</pre><p>There are other Actions available too, but I found this one to be quite straightforward.</p><p>That’s it. We’re done. The entire system is now persisted in code. Again, if you want to install it for yourself, head over to my <a href="https://github.com/petergtz/g2s3">GitHub Repository</a> and follow the instructions.</p><p><a href="https://github.com/petergtz/g2s3">GitHub - petergtz/g2s3: G2S3 is a solution to regularly back up your Google data by copying it over to S3.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaddc9087d1f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/backing-up-google-photos-to-amazon-s3-part-2-eaddc9087d1f">Backing Up Google Photos to Amazon S3, Part 2</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backing Up Google Photos to Amazon S3, Part 1]]></title>
            <link>https://medium.com/better-programming/backing-up-google-photos-to-amazon-s3-221e42a32a95?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/221e42a32a95</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[google-photos]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[backup]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Sun, 11 Dec 2022 22:32:32 GMT</pubDate>
            <atom:updated>2025-09-19T13:51:41.504Z</atom:updated>
            <content:encoded><![CDATA[<h4>Harness the power of Rust</h4><figure><img alt="An arrow hinting at moving the Google Photos logo into an Amazon S3 bucket." src="https://cdn-images-1.medium.com/max/1024/1*53L-0VCRsuXB7qZmZUO0gw.png" /></figure><p>I wanted to learn Rust, and also try out AWS Cloud Development Kit (CDK) — and secure myself from the unlikely case of being locked out of my Google account and losing 500GB worth of photos and videos.</p><p>So in this blog post, I’m describing my journey and the system I’ve developed for it — a system that would regularly back up all my photos and videos into Amazon S3. I’m sure there are off-the-shelf solutions for this, but that would be much less fun. This is Part I of a two-part blog series (<a href="https://betterprogramming.pub/backing-up-google-photos-to-amazon-s3-part-2-eaddc9087d1f">Part II</a>). So let’s get into it.</p><p><em>But wait</em>, if you’d like to skip the story here and just set up the same solution I’m describing here, feel free to head over directly to <a href="https://github.com/petergtz/g2s3">https://github.com/petergtz/g2s3</a> and follow the instructions. It’s not a zero-conf setup and it’ll require some technical knowledge to set up, but it’s doable.</p><h3>Idea</h3><p>Now, let’s get into it. The idea was simple: use Google’s Photos Library API to regularly scan the entire Google Photos library and copy photo and video data to <a href="https://aws.amazon.com/s3/storage-classes/glacier">Amazon’s S3 Glacier Deep Archive</a>.</p><p>At a price point of $0.00099 per GB that would cost $0.50 per month for my entire Photos library. Seemed reasonable.</p><h3>Access to the Google Photos Library API</h3><p>First, I needed programmatic access to my Google Photos library. So I headed over to <a href="https://console.cloud.google.com/">https://console.cloud.google.com/</a>, created a new project “Photos Backup” and enabled the <a href="https://console.cloud.google.com/apis/library/photoslibrary.googleapis.com">Photos Library API</a> for this project.</p><p>Google only allows OAuth 2.0 access for that API, no Google Service Account access, so I also needed to <a href="https://console.cloud.google.com/apis/credentials/consent">create an OAuth consent screen</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1qbD5cXGg7HSQYBwnwXxPg.png" /><figcaption>Creating an OAuth consent screen in Google Cloud</figcaption></figure><p>The important part here is to specify the correct authorization scope as …/auth/photoslibrary.readonly, as explained in the <a href="https://developers.google.com/photos/library/guides/authorization">documentation</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-x7H0Lhpe9WFmR7S8lh7Q.png" /><figcaption>Authorization scope for Google Photos</figcaption></figure><p>Since we only want to <em>download</em> data from the library, readonly is fine. Still, since this makes it possible for an app to access a user’s private photos, this falls into the category of <em>sensitive scopes</em>, and if I wanted to make this app available to everyone, I’d have to go through Google’s proper verification process. Luckily, this is only for private use, or “testing purposes”.</p><p>Next, we can download our Client ID and Client Secret by clicking on the download icon on the right:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4Np7G60FUGUfhGNuoloMg.png" /><figcaption>OAuth 2.0 Client ID and Client Secret can be found under the Credentials section</figcaption></figure><p>To make an actual request against the Photos Library API, we need an access token. We can get this by following the <a href="https://developers.google.com/identity/protocols/oauth2/native-app">OAuth 2.0 authorization process</a>. Getting this right is a grind. Google’s error messages, when making a mistake, are not helpful. Eventually, I did get it right though. To cut this short, the result is is in <a href="https://github.com/petergtz/google-backup-to-s3/blob/main/rust/src/bin/retrieve-google-tokens.rs">this Rust binary</a>.</p><p>Now that we have the access token, we can finally start.</p><h3>Downloading Photos Using Photos Library API in Rust</h3><p>So I thought “let’s create a new Rust project and get it done”. But as anyone knows, who has started learning Rust, it’s not that easy. And the first few hours I fought more with the borrow checker than got anything done to download image data from Google Photos.</p><p>Eventually, though, I hacked together a first prototype to download a photo from Google Photos. I used the <a href="https://crates.io/crates/google-photoslibrary1">google-photoslibrary1</a> crate and the code looked roughly like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13a414bd4d4edbc4f7f384f912ad1107/href">https://medium.com/media/13a414bd4d4edbc4f7f384f912ad1107/href</a></iframe><p>Super happy about the result, I opened the downloaded photo in an image viewer locally. Everything was there, except, the photo missed GPS location Exif data. Not an absolute tragedy in an unlikely disaster case, but it was a nagging annoyance, and so it didn’t quite satisfy me. I had to find a better solution that would give me my photo and video data as is.</p><h3>Google Takeout to the Rescue</h3><p>Turns out that <a href="https://takeout.google.com/">Google Takeout</a>, the service Google provides for users to export their personal data, keeps the original data. But Google Takeout doesn’t provide an API. What it does provide though, is the option to perform a takeout regularly every 2 months for one year. It also offers to drop the data into Google Drive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RRjMOZP_JDj4RDz5gHMxTg.png" /></figure><p>So I could simply set up a job that runs every 2 months and copies the latest takeout from Google Drive into S3. It’s not a 100% hands-free solution, because I now have to remember to trigger the takeout once a year. But given that this is <em>only</em> once a year and it takes about 5 minutes, it’s a compromise I’m willing to make.</p><h3>Downloading Photos Using Google Drive API in Rust</h3><p>Back to square one: Let’s enable the Google Drive API in our Photos Backup project in Google Cloud. Let’s edit the OAuth consent screen and let’s use a different authorization scope, namely …/auth/drive.readonly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqU-6H0Zj3EZN6zRWnG3PA.png" /><figcaption>Authorization scope for Google Drive</figcaption></figure><p>Now, let’s put together the code to download data from Google Drive. This time, I used the <a href="https://crates.io/crates/google-drive3">google-drive3</a> crate and the parts for creating its “drive hub” and for downloading actual data look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44768b58cff8a765273d12b4fac183eb/href">https://medium.com/media/44768b58cff8a765273d12b4fac183eb/href</a></iframe><p>The hyper::Response&lt;Body&gt; object can be used directly to upload it to S3, using the S3 client from the <a href="https://crates.io/crates/aws-sdk-s3">aws-sdk-s3</a> crate:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49c3da25a71618f861b5f0fef19f0f8f/href">https://medium.com/media/49c3da25a71618f861b5f0fef19f0f8f/href</a></iframe><p>What we can also see here, is that we can use the Google Drive API File struct’s md5_checksum property (in case it’s available) to specify it in the S3 upload request. That way, S3 can automatically verify that the file uploaded to S3 is identical to the one downloaded from Google Drive.</p><h3>Performance</h3><p>Of course, things have to be reasonably fast. While this matters a little less when everything happens in the clouds anyway, it’s still kind of nice when the copying job finishes quickly. Besides, all these libraries are already implemented as async libraries. So the jump to taking advantage of it shouldn’t be too difficult. Right? Again, it turned out not as straightforward to do this in Rust. But the book <a href="https://rust-lang.github.io/async-book/">Asynchronous Programming in Rust</a> is a lifesaver.</p><p><a href="https://rust-lang.github.io/async-book/">Asynchronous Programming in Rust</a></p><p>I realized, without making things too complex, I could simply first get the list of all Google File API objects in the Takeout folder (in my case that’s about 500 1 GB files), create a futures stream out of it, and iterate through all files by copying four files concurrently over to S3.</p><p>An unbounded channel is used to keep track of asynchronous copy results. The call to rx.close() will automatically ensure that all files have been copied before continuing.</p><p>It looks like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d72db6444e9ce47c7f348232e61eed7f/href">https://medium.com/media/d72db6444e9ce47c7f348232e61eed7f/href</a></iframe><p>The critical part happens in line 7 where we use for_each_concurrent(4, ...) which causes the 4 simultaneous copies. Line 11 then performs the actual asynchronous copy.</p><p>Handling errors in this implementation is still a bit unpolished, and I will have to go over this in future refactoring. However, what’s important is that the job will fail, if at least one file copy failed, and all errors will be logged.</p><h3>Making it Nice</h3><p>To make this binary a bit easier to use, I decided to use a command line argument parser library. In this case, <a href="https://crates.io/crates/clap">clap</a> looked exactly like what I needed. With it, I could simply define a struct that defines my command line arguments:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a1decdc5a5701dbb96bb49f4cf7b2ca/href">https://medium.com/media/3a1decdc5a5701dbb96bb49f4cf7b2ca/href</a></iframe><p>Which produces:</p><pre>$ back-up-drive-folder --help</pre><pre><strong>Usage: back-up-drive-folder</strong> [OPTIONS] &lt;SOURCE&gt; &lt;DESTINATION&gt;</pre><pre><strong>Arguments</strong>:<br>  &lt;SOURCE&gt;       The Google Drive folder to back up<br>  &lt;DESTINATION&gt;  Where to copy the files. This must be in the format s3://bucket-name/some/folder Can also use {date} which will get substituted by the current date</pre><pre><strong>Options</strong>:<br>  <strong>-s</strong>, <strong>--s3-storage-class</strong> &lt;S3_STORAGE_CLASS&gt;<br>          Storage class to use when storing objects in S3. Possible values: DEEP_ARCHIVE, GLACIER, GLACIER_IR, INTELLIGENT_TIERING, ONEZONE_IA, OUTPOSTS, REDUCED_REDUNDANCY, STANDARD, STANDARD_IA [default: STANDARD]<br>  <strong>-h</strong>, <strong>--help</strong><br>          Print help information<br>  <strong>-V</strong>, <strong>--version</strong><br>          Print version information</pre><p>With that, I had a working binary that could be used to copy the contents of a Google Drive folder into Amazon S3 via:</p><pre>$ back-up-drive-folder --s3-storage-class DEEP_ARCHIVE \<br>                       Takeout \<br>                       s3://my-backup-bucket/some/folder</pre><p>This wraps up Part I of this two-part blog series. In <a href="https://betterprogramming.pub/backing-up-google-photos-to-amazon-s3-part-2-eaddc9087d1f">Part II</a>, I will write about the automation I put in place, using CDK, Docker, and GitHub Actions, to deploy this system.</p><p>If you want to see the full source code of this project, head over to <a href="https://github.com/petergtz/g2s3">https://github.com/petergtz/g2s3</a> and feel free to contribute or leave feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=221e42a32a95" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/backing-up-google-photos-to-amazon-s3-221e42a32a95">Backing Up Google Photos to Amazon S3, Part 1</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why We Rewrote the Cloud Foundry Bits-Service in Go]]></title>
            <link>https://medium.com/@peter.gtz/why-we-rewrote-the-cloud-foundry-bits-service-in-go-afa7f2b8b8ff?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/afa7f2b8b8ff</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[cloud-foundry]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[bits-service]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Wed, 13 Jun 2018 14:43:51 GMT</pubDate>
            <atom:updated>2018-07-04T20:43:41.065Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MiPcEAv3GmhK8xXI9Bxow.png" /></figure><p>Rewriting a service in a another language is never a small endeavor and it always comes with a risk. Yet, we decided was worth it. Here’s our story about dropping the <a href="https://www.ruby-lang.org/">Ruby</a> implementation of the <a href="https://www.cloudfoundry.org/">Cloud Foundry</a> <a href="https://github.com/cloudfoundry-incubator/bits-service">Bits-Service</a> and <a href="https://github.com/petergtz/bitsgo">rewriting it in Go</a>.</p><h3>Issues with the Ruby implementation</h3><p>It started when we realized there was a lot of duplicate code in the Bits-Service. Packages, droplets and buildpacks are treated almost identically, yet the Bits-Service had a separate code path for all of them. We agreed this needed to be refactored, because it had led several times to hard-to-track-down bugs due to subtle differences or forgotten changes in some parts.</p><p>We used Sinatra as framework for the Ruby Bits-Service. Unfortunately, we couldn’t find a way to use proper dependency injection with that framework. Therefore, the implementation required some really dumb config value checks on every request which seemed very bad performance-wise. We decided we’d need to migrate to another framework that would allow us to inject config values.</p><p>Next, our log files were a mess. It was nearly impossible to understand what was going on from the log file in case of an error. The error was usually logged three times from different locations in the code, but important information was in fact missing.</p><p>The Ruby service required <a href="http://nginx.org/en/">nginx</a> as a frontend. Originally, this was intended to be optional, but over time, it became a hard dependency. Unfortunately, nginx was part of the <a href="http://bosh.io/">BOSH</a> release of the Bits-Service, not the Bits-Service itself. And so to run the Bits-Service, we had to do the BOSH dance, i.e.</p><pre>bosh create-release<br>bosh upload-release<br>bosh deploy manifest.yml</pre><p>Waiting for that to finish can suck out motivation from anyone.</p><p>Sure, we could have changed that and provided scripts that started nginx + Ruby Bits-Service locally. We could have also fixed our logging and necessarily the error handling. We could have migrated to another web service framework. And we could have refactored the whole code base to remove code duplication. After these changes, there wouldn’t have been much left from the original code. We could have made all these changes, and we’d have invested a lot of time.</p><p>Yet, we’d still be stuck with nginx, which we learned to despise due to its unintuitive and complicated configuration, due to the intransparent interaction when serving files through nginx, and finally, because there is no easy way to send response times directly to statsd. We’d still be stuck with a language whose libraries often use language features that are complex and hard to understand.</p><h3>Re-Implementation</h3><p>We decided instead, we’d re-implement it. — Okay, so that’s not entirely true. We had it already ready. Mostly. Except for a few small features. The small side-story here is that I wanted to investigate web service frameworks for Go quite a while ago, and thought the Bits-Service would be a really good real-world example to do that (Gorilla mux + Negroni it was, by the way).</p><p>So we had that almost-done re-implementation of the Bits-Service and we decided to use that.</p><p>From the outside, it might seem like we were dropping a battle-tested Ruby implementation for an immature Go implementation. However, that’s not true. While the Ruby implementation had been around for about two years, it had only been used in test environments, not production environments. Those same test environments are now used to test the Go version. We kept all system-level tests and CATs.</p><h3>Pay-off</h3><p>There are some great benefits that came with it:</p><ul><li>Speed! On GET requests the Go implementation is <em>1000 times faster</em> when running concurrent load against the service. We didn’t make any specific performance optimizations.</li><li>Half the memory consumption (300–400MB instead of 1GB).</li><li>We got what <a href="http://twitter.com/nparsons08">@nparsons08</a> calls <a href="https://codeburst.io/why-we-switched-from-python-to-go-60c8fd2cb9a9">Reason 6 — The Ability to Build a Team</a>.</li><li>Issues explained in <a href="https://movio.co/en/blog/migrate-Scala-to-Go/">Making The Move From Scala To Go, And Why We’re Not Going Back</a> are existent in Ruby code bases too. And we had them. Understanding where certain methods come from magically is far from being straight forward.</li><li>We dropped the leaky fog-abstraction existing in the Ruby version and exposed well-defined properties.</li></ul><h3>Conclusion</h3><p>When we decided to switch from Ruby to Go, it wasn’t simply because we think Go is cool. Instead, we suffered from a lot of basic code issues, and fixing them would have cost us almost as much as rewriting in Go, without getting the benefits we got from Go. Looking at the greater Cloud Foundry community, I think long-term, this decision will have even more positive impact, because the Community is by now mostly Go-based.</p><p><em>Excited about all things </em><a href="https://www.cloudfoundry.org/"><em>Cloud Foundry</em></a><em> and want to know more about the </em><a href="https://github.com/cloudfoundry-incubator/bits-service"><em>Bits-Service</em></a><em>? The Bits-Service is being developed in the IBM-Lab Boeblingen, Germany and we usually hang out on </em><a href="https://cloudfoundry.slack.com/messages/C0BNGJY0G"><em>cloudfoundry.slack.com in #bits-service</em></a><em>. Come and talk to us!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afa7f2b8b8ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Write Better Unit Tests in Go Using Mocks]]></title>
            <link>https://itnext.io/how-to-write-better-unit-tests-in-go-using-mocks-4dd05e867b17?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd05e867b17</guid>
            <category><![CDATA[mocking]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[cloud-foundry]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Mon, 23 Apr 2018 21:48:42 GMT</pubDate>
            <atom:updated>2018-06-25T12:30:39.772Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, as part of my work on the Go re-implementation of the <a href="https://www.cloudfoundry.org/">Cloud Foundry</a> <a href="https://github.com/petergtz/bitsgo">Bits-Service</a>, I’ve implemented a feature that requires a set of actions to be executed in the right order. Such a set of actions is also known as <em>protocol</em>.</p><p>The protocol in my case looked like this:</p><ol><li>A file upload request is made against Bits-Service</li><li>Bits-Service tells Cloud Controller that it’s PROCESSING_UPLOAD for a file.</li><li>Bits-Service uploads the file to a Blobstore (such as S3).</li><li>Bits-Service tells Cloud Controller that the file upload either (a) FAILED or (b) succeeded and the file is READY to be used.</li></ol><p>Since any of those calls can return an error in which case the set of actions changes, I wanted to unit test the interaction of the Bits-Service with its two collaborators to get this exactly right. Both collaborators, the Cloud Controller and the Blobstore, were already defined as interfaces, Updater and Blobstore respectively. Still, creating mocks for protocols is not trivial, because the order in which the mocks are called is important.</p><p>Using ad-hoc, manually written mocks is possible; they could share a common counter to track the calling order etc. But getting their internal behavior right is not trivial, and requires its own set of unit tests. That’s a lot of additional work just to write a simple unit test.</p><p>Luckily, there’s <a href="https://github.com/petergtz/pegomock"><strong>Pegomock</strong></a>, an excellent mocking framework for Go. All I had to do, was:</p><pre>$ pegomock generate Blobstore <br>$ pegomock generate Updater</pre><p>Then the unit test code looked trivial:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f98e69186702b4eda4b1c4e1738026e/href">https://medium.com/media/0f98e69186702b4eda4b1c4e1738026e/href</a></iframe><p>It’s really that simple.</p><p>While we’re at it, of course Pegomock isn’t only good for this specific use case. In fact, it’s excellent for <em>all</em> kinds of mocking use cases. Want to stub some phone book-like interface? Here’s how:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4434f4830e1fcc3b544d8e221264b71d/href">https://medium.com/media/4434f4830e1fcc3b544d8e221264b71d/href</a></iframe><p>You can use matchers to always return the same value:</p><pre><strong>When</strong>(phoneBook.<strong>GetPhoneNumber</strong>(<strong>AnyString</strong>())).<strong>ThenReturn</strong>(&quot;123-456&quot;)</pre><p>Or, in case you need something more sophisticated, you can even implement your own stub function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7b46c223ffa9b4cdeeaf6f1d7570f30/href">https://medium.com/media/e7b46c223ffa9b4cdeeaf6f1d7570f30/href</a></iframe><p>Need to panic? Here it is:</p><pre><strong>When</strong>(phoneBook.<strong>GetPhoneNumber</strong>(&quot;Unknown&quot;)).<strong>ThenPanic</strong>(&quot;Invalid name&quot;)</pre><p>However, the real beauty in Pegomock surfaces when you need to combine stubbing and verification, because you use the same kind of matchers for both, and so it feels very natural. You don’t even notice when you switch between them, except that your tests look amazingly clean:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c2f87418fcbc2b4b9ff6e075a0b1ffa/href">https://medium.com/media/0c2f87418fcbc2b4b9ff6e075a0b1ffa/href</a></iframe><p>Next time you need a mocking framework for Go, why not try Pegomock? <a href="https://github.com/petergtz/pegomock#getting-pegomock">Just </a><a href="https://github.com/petergtz/pegomock#getting-pegomock">go get it</a>.</p><p><em>By the way, the Go re-implementation of the Cloud Foundry Bits-Service is being developed in the IBM-Lab Boeblingen, Germany and we usually hang out on </em><a href="https://cloudfoundry.slack.com/messages/C0BNGJY0G"><em>cloudfoundry.slack.com in #bits-service</em></a><em>. Come and talk to us!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd05e867b17" width="1" height="1" alt=""><hr><p><a href="https://itnext.io/how-to-write-better-unit-tests-in-go-using-mocks-4dd05e867b17">How to Write Better Unit Tests in Go Using Mocks</a> was originally published in <a href="https://itnext.io/">ITNEXT</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lifting the Veil of Mystery around the Cloud Foundry Bits-Service]]></title>
            <link>https://medium.com/@peter.gtz/lifting-the-veil-of-mystery-around-the-cloud-foundry-bits-service-5e386845d33f?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e386845d33f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ibm]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[cloud-foundry]]></category>
            <category><![CDATA[bosh]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Thu, 07 Dec 2017 14:37:11 GMT</pubDate>
            <atom:updated>2017-12-07T14:37:11.346Z</atom:updated>
            <content:encoded><![CDATA[<p>We have been in stealth mode for quite a while, trying to hide as much as possible of our development and activity around the Bits-Service. Of course, some information leaked here and there, where we couldn’t avoid it. But overall, we managed to keep it secret quite well.</p><p>That is of course non-sense. The <a href="https://github.com/cloudfoundry-incubator/bits-service">Bits-Service</a> has been developed in the open like any other true Open Source project, and is being developed as a <a href="https://github.com/cloudfoundry-incubator">Cloud Foundry Incubator</a> project. What is true, though, is the fact, that there hasn’t been much information out there about the Bits-Service and what problem it solves, except for some <a href="https://www.youtube.com/watch?v=91P-Sg16bIQ">Youtube videos</a> from past <a href="https://www.cloudfoundry.org/events/">Cloud Foundry Summits</a>.</p><p>This is about to change soon, however, as we are getting closer to becoming a standard component of <a href="https://www.cloudfoundry.org/">Cloud Foundry</a>. That of course begs the questions “What is, in fact, the Bits-Service?” and “Can you start using it in your Cloud Foundry deployment today?”</p><p>The answer to the latter question is “Yes, you can use it!” And the rest of this post explains how and why you should use it.</p><h3>What is the Bits-Service and Why Use It?</h3><p>To explain that, let’s first get a bit of context. Over time, the <a href="https://docs.cloudfoundry.org/concepts/architecture/cloud-controller.html">Cloud Controller</a> has acquired quite a few responsibilities. Some might say “it got a bit bloated”. In an effort to separate out some of the responsibilities currently being held by the Cloud Controller, we decided to extract the handling of bits into its own service, the Bits-Service. What are “bits” in this context? Anything dealing with files: your source code, aka package, droplets, buildpacks, etc.</p><p>So, on the one hand the Bits-Service really is a design hygiene effort, which allows us to have an <em>independent</em> team working on Cloud Foundry bits-handling, avoiding to expand the CAPI team (which develops the Cloud Controller) to a bigger team, meaning we just followed good microservice architecture best practices.</p><p>On the other hand, extracting the bits-handling out of the Cloud Controller opened up new and interesting opportunities to make Cloud Foundry more performant. Why is that?</p><p>The Cloud Controller currently serves (among other things) two purposes:</p><ol><li>It responds to short-lived requests such as cf apps, which is simply querying the database and returning a response immediately.</li><li>It responds to long-running requests such as uploading package bits, or downloading droplet bits.</li></ol><p>The latter requests are expensive for the Cloud Controller, because they block valuable threads which could be used to serve short-lived requests instead. Of course we could simply scale up the number of Cloud Controllers and solve any bottlenecks. But that would be a waste of resources, as Cloud Controllers require “fat” machines, while bits-handling can be handled with much more light-weight machines.</p><p>Now with bits-handling extracted, we can have the cf CLI start talking to the Bits-Service directly. That frees the Cloud Controller from that responsibility and let’s us see real performance gains.</p><p>For small Cloud Foundry deployments, such as test environments, a lot of the above doesn’t matter that much. However, in a <a href="https://www.ibm.com/cloud/cloud-foundry">Cloud Foundry deployment of the scale of the IBM Cloud</a>, where we have a massive number of Cloud Controllers and a heavy load on bits-handling, this “refactoring” actually makes a difference.</p><p>Since we are in the middle of this migration, the performance improvements are not there yet. But by enabling the Bits-Service in your Cloud Foundry deployment, you’re making space for the above described performance improvements that we expect to introduce iteratively over the coming months.</p><p>With that, how can you start using it?</p><h3>Enabling the Bits-Service in Cloud Foundry</h3><p>The guide below is geared towards developers who are already familiar with the deployment of a vanilla Cloud Foundry using <a href="http://bosh.io/">BOSH</a>.</p><p>The new and recommended way to deploy Cloud Foundry is through <a href="https://github.com/cloudfoundry/cf-deployment">cf-deployment</a>. <a href="https://github.com/cloudfoundry/cf-release">cf-release</a> is <a href="https://lists.cloudfoundry.org/g/cf-dev/topic/6404202">soon going to be deprecated</a>, so I won’t explain the approach there.</p><p>Thanks to cf-deployment’s modular structure, there is actually not much to do, except for including a couple of experimental <a href="https://bosh.io/docs/cli-ops-files.html">BOSH Ops files</a> and setting necessary <a href="https://bosh.io/docs/cli-int.html">Ops variables</a>, assuming there is already a valid <a href="https://bosh.io/docs/cloud-config.html">cloud-config</a> on the BOSH director as explained incf-deployment’s documentation.</p><p><a href="https://flintstone.ci.cf-app.com/">Our Concourse pipeline</a> e.g. deploys Cloud Foundry with Bits-Service into several environments, covering <a href="https://www.ibm.com/cloud/info/softlayer-is-now-ibm-cloud">IBM Cloud</a>, <a href="https://aws.amazon.com/">AWS</a> and <a href="https://bosh.io/docs/bosh-lite">Bosh-Lite</a>.</p><p>The BOSH manifest generation command for our IBM Cloud deployment looks more or less like:</p><pre>bosh interpolate cf-deployment/cf-deployment.yml \<br>    --vars-store deployment-vars-cf.yml \<br>    -v system_domain=cf-deployment.dynamic-dns.net \<br>    -o cf-deployment/operations/experimental/bits-service.yml \<br>    -o cf-deployment/operations/experimental/enable-bits-service-https.yml \<br>    -o cf-deployment/operations/experimental/bits-service-s3.yml \<br>    -o cf-deployment/operations/scale-to-one-az.yml \<br>    -v &#39;aws_region=&quot;eu-west-1&quot;&#39; \<br>    -v &#39;blobstore_access_key_id=&quot;AKIAIOSFODNN7EXAMPLE&quot;&#39; \<br>    -v &#39;blobstore_secret_access_key=&quot;wJalrXUtnF...EXAMPLEKEY&quot;&#39; \<br>    -v &#39;resource_directory_key=&quot;ci-bits-service-blobs&quot;&#39; \<br>    -v &#39;buildpack_directory_key=&quot;ci-bits-service-blobs&quot;&#39; \<br>    -v &#39;droplet_directory_key=&quot;ci-bits-service-blobs&quot;&#39; \<br>    -v &#39;app_package_directory_key=&quot;ci-bits-service-blobs&quot;&#39;</pre><p>Obviously, you need to adjust the parameters system_domain, blobstore_access_key_id,blobstore_secret_access_key, and *_directory_keys to match your specific environment.</p><p>Now bosh deploy this and that’s it! Seems simple? It is!</p><p>You’ll see that there is now a new job called bits-service in the deployment. This is what now takes care of the bits-handling.</p><p>Note that you can do this in an existing Cloud Foundry deployment, given that you use the same storage configuration for the Bits-Service as you have used for the Cloud Controller. The migration is seamless and you can always go back. We have taken great care to make sure you can enable the Bits-Service in an <a href="https://hackernoon.com/interruption-free-service-migrations-how-you-can-learn-from-the-cashiers-at-your-local-supermarket-36c1fe72397b">interruption-free manner</a> in your existing Cloud Foundry deployment.</p><h3>Conclusion</h3><p>The Bits-Service is there and it’s real. It’s soon going to be a standard part of Cloud Foundry and hence, we encourage people to try it out. In this post I have explained why we made the Bits-Service and how anyone can start using it today.</p><p><em>Excited about all things </em><a href="https://www.cloudfoundry.org/"><em>Cloud Foundry</em></a><em> and want to know more about the </em><a href="https://github.com/cloudfoundry-incubator/bits-service"><em>Bits-Service</em></a><em>? The Bits-Service is being developed in the IBM-Lab Boeblingen, Germany and we usually hang out on </em><a href="https://cloudfoundry.slack.com/messages/C0BNGJY0G"><em>cloudfoundry.slack.com in #bits-service</em></a><em>. Come and talk to us!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e386845d33f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Thinking About New Ways of Error Handling in Go 2]]></title>
            <link>https://medium.com/@peter.gtz/thinking-about-new-ways-of-error-handling-in-go-2-e56d116952f1?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/e56d116952f1</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[error-handling]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Mon, 04 Sep 2017 21:09:07 GMT</pubDate>
            <atom:updated>2018-11-13T10:19:35.440Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I was doing some refactoring on existing Go code and eventually ended up with this code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb31ffe152c1e99ebb732ebae7a02c54/href">https://medium.com/media/cb31ffe152c1e99ebb732ebae7a02c54/href</a></iframe><p>Presented in this isolated way, it’s probably easy to spot that there is a bug in the error handling. Seeing this in a big junk of code, however, is a different story.</p><p>It’s obvious how this happened: a simple copy-and-paste mistake. Not a big problem in this little example. This is a CLI utility and if the server flag is not set, it would simply die with a panic saying something like “nil pointer dereference” and give me a stack trace which helps to solve the problem.</p><p>But in the production code of a critical service the effects of such a mistake can be pretty harmful.</p><p>One could argue that following best practices by writing a test could have helped. True. But while I’m a firm believer in writing tests, reality is, it’s not always possible or not always practical to write a test. Furthermore, Go is a statically typed language and it’s fair to put a certain amount of static correctness checking into the compiler.</p><p>I have been thinking about error handling <a href="https://medium.com/@peter.gtz/whats-wrong-with-go-s-error-handling-d2b58c058e0e">more</a> <a href="https://hackernoon.com/correct-error-handling-is-hard-307ea72759c7">than</a> <a href="https://medium.com/@peter.gtz/correct-error-handling-is-hard-part-2-bba10ce73c46">once</a> in the past. What I’d like to do in this post, is to propose a new approach to error handling. It’s neither exception based nor return code based. It reuses some of the concepts, but essentially tries to match what I will describe as “requirements for a sane error handling experience”.</p><h3>Requirements for a Sane Error Handling Experience</h3><p>The error handling mechanism should:</p><ul><li><strong>Force the developer to handle errors: </strong>Traditional error code mechanism as in C do not force the developer to handle the error. Developers can easily forget to handle an error. It’s hence error prone (no pun intended). Go goes a step further by requiring to use an assigned error variable. However, it’s still too easy to mess up, as the following example shows, by not handling the first e:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99b255b5d13491b9204e18797e69d4bb/href">https://medium.com/media/99b255b5d13491b9204e18797e69d4bb/href</a></iframe><ul><li><strong>Have local scopes for errors to avoid handling mistakes: </strong>Both examples above already describe this. We simply want to have the error only available while handling it, not afterwards.</li><li><strong>Have an expressive syntax: </strong>making clear where an error path starts, and making that part as concise as possible helps reading the code and will improve developer productivity.</li><li><strong>Have a simple syntax to “fail fast”:</strong> Start-up code or test code do not need to handle errors in a way that gives the caller the chance to recover. They are simply interested in bailing out quickly. Instead of the typical if e != nil { panic(e) }, it would be nice if this doesn’t need three lines.</li></ul><h3>Proposal</h3><p>By introducing three new keywords we can satisfy all the requirements listed above. They are:</p><ul><li>fails_with Used in the function signature to denote the error type of this function and simply formalizes the existing convention of making the last return value the error.</li><li>fail Used to return from a function by failing. It takes the error value as argument.</li><li>on_error Used in the calling code to react to an error.</li></ul><p><strong>Let’s have a look at this proposal in action through a minimal example:</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e604eef17733746af1ac9dc2c6006c0/href">https://medium.com/media/0e604eef17733746af1ac9dc2c6006c0/href</a></iframe><p>By specifying the error through fails_with in the signature, the compiler can enforce an on_error block-statement on the caller side. e has the type of the argument in fails_with and gets assigned the error value. That value is only available in the on_error block. So accidental reuse of the error variable is not possible. The syntax is short and expressive, letting us handle the error where it happens.</p><p>By allowing to omit the curly braces, we could implement the fail fast version simply by:</p><pre>result := SomeFunction(arg) on_error e panic(e)</pre><p>Not requiring the curly braces could make the syntax short and concise in many other scenarios too:</p><pre>result := SomeFunction(arg) on_error e fail e</pre><h3>FAQ</h3><p><strong>Introducing new keywords makes the language more complex. Isn’t there a better solution?</strong></p><p>Introducing new keywords might not be the typical Go strategy. The language has been marketed specifically to be simple because there are so few constructs or keywords. However, keywords themselves do not make a language complex. What makes a language complex, is having many concepts. The introduced keywords here really just formalize what is best practice and an existing concept in Go already. The benefit comes from being less error prone.</p><p><strong>Isn’t this just adding some syntactic sugar?</strong></p><p>It seems like on first sight. But letting the compiler help you avoid forgetting to handle an error is more than just syntactic sugar. Also, scoping the error variable is a huge improvement over the current situation.</p><p><strong>Isn’t this an exception mechanism in disguise?</strong></p><p>The idea behind exceptions is to automatically propagate failure conditions and handle multiple exceptions at common places to avoid “littering” the code with error handling. Exceptions are mostly concerned about <em>what type</em> of error happened. I believe, and that probably goes in line with what the Go designers had in mind, that it is more important to know <em>where</em> an error happened, and under which <em>context</em>. That allows precise error handling, and subsequently more useful error handling. Anyone who starts doing precise error handling with exceptions ends up with code convoluted with try ... catch.</p><h3>Feedback</h3><p>I’d love to get feedback on this proposal. It’s obviously a very early draft based on some thoughts of mine. There’s a lot that could be improved, like e.g. should we change the keywords? Could we avoid some of them? What other things am I completely missing? Is there a way to simplify the constructs I introduced? Any thoughts in the comments section are welcome!</p><h3>Update Nov. 13, 2018</h3><p><em>Here’s an example what the popular </em>CopyFile<em> function would look like that is used in the current </em><a href="https://go.googlesource.com/proposal/+/master/design/go2draft.md"><em>Go 2 draft design documents</em></a><em> for </em><a href="https://go.googlesource.com/proposal/+/master/design/go2draft-error-handling-overview.md"><em>Error Handling</em></a><em>:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a949b8555fa7f71b28a94cc972b77fd/href">https://medium.com/media/7a949b8555fa7f71b28a94cc972b77fd/href</a></iframe><p><em>As can be seen, the flow is still linear (i.e. no jumping back and forth between some pre-set handler functions or similar). We still use the </em>defer<em> construct to handle automatic closing of the source file. No </em><strong><em>automatic/implicit</em></strong><em> error propagation, but a concise way to propagate errors </em><strong><em>explicitly</em></strong><em>.</em></p><p><em>And as a nice bonus, “command” functions, such as </em>Copy<em> and </em>Close<em> are not hidden in </em>if<em> constructs, but are easily identifiable as such.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e56d116952f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interruption-free service migrations: what you can learn from the cashiers at your local…]]></title>
            <link>https://medium.com/hackernoon/interruption-free-service-migrations-how-you-can-learn-from-the-cashiers-at-your-local-supermarket-36c1fe72397b?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/36c1fe72397b</guid>
            <category><![CDATA[bosh]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[operational-excellence]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[cloud-services]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Thu, 27 Jul 2017 22:14:45 GMT</pubDate>
            <atom:updated>2019-05-16T12:00:45.959Z</atom:updated>
            <content:encoded><![CDATA[<h3>Interruption-free service migrations: what you can learn from the cashiers at your local supermarket</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GYRCR06qVazwN5WDH9ilcA.jpeg" /></figure><p><a href="https://hackernoon.com/tagged/amazon">Amazon</a> can do it, <a href="https://hackernoon.com/tagged/google">Googl</a>e can do it, a lot of others can do it too: interruption-free service migrations.</p><p>So why are so many enterprise businesses, banks, and others not able to do it? Why do I still get a notification by my bank that there will be service interruptions between Saturday 4 p.m. and Sunday 10 a.m. Why is there still that senior manager asking the question upon a change request: &quot;will that change be disruptive?&quot; 20 years ago, that might have been a reasonable question, but when was the last time you went to <a href="http://google.com/">google.com</a> and got a “service down due to maintenance”?</p><p>I do not know why we are still talking about this. However, I do know how to do interruption-free service migrations. It’s very simple:</p><h4><strong>Just watch and learn from the cashiers at your local supermarket</strong></h4><p>Let’s say they have 4 counters. #1, 2, and 3 are open and have a cashier behind them. Now the cashier behind counter #2 is done for the day and needs a replacement.</p><p><strong>Here’s what the cashier won’t do:</strong> stop everything, take the cash drawer out, wait for her replacement, who has to put in his cash drawer, all while the whole queue of customers on that counter is waiting.</p><p><strong>In fact, this is what will happen:</strong> A fourth cashier will open counter #4 and start serving customers. Next, the cashier at counter #2 will put up a <em>closed</em> sign to avoid getting more customers in the queue. She will still serve the remaining customers in her queue. Then she will close it.</p><p>Notice how this is a 4-step process:</p><ol><li>Open a new service</li><li>Route traffic to the new service</li><li>Serve all remaining traffic in the old service</li><li>Close the old service</li></ol><p>Also, notice how this leaves all possibilities open: The cashier at counter #2 could suddenly decide to re-open, or not close at all, while the cashier at counter #4 still serves. Or the cashier at counter #4 could decide that it wasn’t a good idea, put a <em>closed</em> sign and leave after the remaining queue was served. It’s all seamless.</p><h3>Production Systems</h3><p>So why do we still have these nail-biting upgrade processes in production systems? These all-or-nothing-high-risk-no-rollback upgrade processes, potentially with service interruption, when those cashiers provide us with the perfect blueprint of how to do it?</p><p>Okay — I do realize that production systems are a bit more complex than what I described above. But as always, the way to deal with complexity, is to break it down into small, digestible chunks. This is the hard part. Once that is done, any service upgrade can be done gradually with low-risk and rollback at any point in time, just like the cashiers could do it above.</p><p>Let me try to use some examples in the following to make my point clear.</p><h3>Examples</h3><h4>Changing Certificates</h4><p>Almost any non-trivial system needs to deal with certificates. The problem is always, how do you make sure your clients <em>and </em>your servers get updated simultaneously?</p><p>The answer is, they don’t. Instead, the side verifying the validity of a certificate should allow a new one <em>and</em> an old one for the transition period.</p><p>Here are the steps:</p><ol><li>add new certificate to the the list of accepted certificates on the client (assuming this is the verifying side)</li><li>On server, replace old certificate with new one</li><li>Wait for all requests using the old certificate to finish on client</li><li>remove old certificate from client</li></ol><h4>Update DB Schema</h4><p>Let’s say you have 1000 machines accessing an SQL database to feed a web app. A new version of the web app uses a renamed column in the database. So you need to rename that column in the database and simultaneously upgrade all your 1000 machines to the new web app version. How can you do that?</p><p>You can’t. Instead, your new version must be able to understand a schema which contains both the new column name and the old column name. It must always try to use the new one first, and falling back to the old one when necessary. When writing, it should always write using the new and old column name.</p><p>Here’s how you deploy:</p><ol><li>Add the new column to the database. (Open counter #4)</li><li>Deploy your new web app version to your machines in staggered manner, potentially with a canary (Reroute traffic to counter #4, put up a <em>closed sign</em>)</li><li>Have a background job that copies all old column values to the new one. (Serve all remaining customers at counter #2)</li><li>Remove the fallback mechanism in the web app and deploy again (again staggered). (Close counter #2)</li><li>Optionally: Remove the old column. (Tear down counter #2 :-))</li></ol><h4>A New Storage Service Backend</h4><p>A very similar one as the previous. Let’s say you used S3 so far, but want to migrate to another S3 compatible storage service from another vendor. Your service may consist of thousands of machines.</p><p>Again, to do this, you introduce a code change: writes now go to the new <em>and</em> old storage service. Reads go to the new, but fall back to the old when a resource is not found. Deletes also go to both.</p><p>The migration plan goes like this:</p><ol><li>Set up the new data store. (Open counter #4)</li><li>Deploy your updated system to your machines (Reroute traffic to counter #4, put up a <em>closed sign</em>)</li><li>Have a background that copies all your data from the old storage service to the new one (Serve all remaining customers at counter #2)</li><li>Remove the code change, configure <em>only</em> the new data store. (Close counter #2)</li></ol><p>Note that you might need to increase the number of machines for your service temporarily, because of the more expensive data storage operations.</p><h3>Implications</h3><p>Looking at the examples, specifically the last one, one thing has become clear by now: to do migrations like these, you need to own the code. If you’re operating a system that you cannot change, you won’t be able to make changes that help interruption-free migrations.</p><p>It’s for that reason probably, that newer forms of organizations, e.g. like Amazon’s two-pizza teams that follow a “You build it, you run it” culture are better suited for such interruption-free service migrations than the more classic ones where there is a strict separation between development and operations.</p><p>It’s also obvious that these kinds of migration plans require teams to be able to do frequent deployments with small changes. At any point in time during such migration you must be able to stop or rollback. That’s difficult when your update contains other unrelated changes and when your deployment process is manual and slow.</p><p>It also helps to have a mechanism to deploy to so-called canaries: single machines that get the update first. Only if they prove working correctly will the update be rolled out on to more machines.</p><p>If you wonder what can help you do this kind of deployments, use <a href="http://bosh.io/">BOSH</a>. It probably cannot compete in scale with Amazon’s internal deployment system, <a href="http://www.allthingsdistributed.com/2014/11/apollo-amazon-deployment-engine.html">Apollo</a>. But it’s one of the most powerful you can currently get, especially being free and open source.</p><h3>Conclusion</h3><p>Cashiers at your local supermarket can give you the blueprint for doing interruption-free service migrations. Using this blueprint, you can break down your (potentially huge) migration into small changes that provide seamless, interruption-free migrations. Embracing an organizational structure that merges developers and operators helps to implement such small changes that aid the actual migration. Finally, use a powerful deployment system like <a href="http://bosh.io/">BOSH</a> to deploy the changes in an automated way.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36c1fe72397b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/interruption-free-service-migrations-how-you-can-learn-from-the-cashiers-at-your-local-supermarket-36c1fe72397b">Interruption-free service migrations: what you can learn from the cashiers at your local…</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Correct Error Handling is Hard, Part 2]]></title>
            <link>https://medium.com/@peter.gtz/correct-error-handling-is-hard-part-2-bba10ce73c46?source=rss-c6dc59138c47------2</link>
            <guid isPermaLink="false">https://medium.com/p/bba10ce73c46</guid>
            <category><![CDATA[error-handling]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[operational-excellence]]></category>
            <category><![CDATA[bugs]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Peter Götz]]></dc:creator>
            <pubDate>Sun, 02 Jul 2017 21:26:54 GMT</pubDate>
            <atom:updated>2017-07-31T20:56:03.962Z</atom:updated>
            <content:encoded><![CDATA[<p>Following my last post “<a href="https://medium.com/@petergtz/correct-error-handling-is-hard-307ea72759c7">Correct Error Handling is Hard</a>” I had an <a href="https://twitter.com/zekjur/status/876837493378879489">interesting discussion on Twitter</a> and got interesting comments in the comments section, which I want to use as an opportunity for another post.</p><p>In that previous post I introduced a helper type called SafeCloser, which helps to correctly close io.Closer objects in Go exactly once.</p><p>In the follow-up discussions it was pointed out to me that os.File can handle multiple closes gracefully and that in fact none of the Go code on GitHub panics when calling Close twice.</p><p>For os.File this is true. Right now. In Go version 1.8. We don’t know if that is true for future versions.</p><p>And while a lot of code is hosted on Github, what about all those proprietary code repositories hosted within companies? Can you also assume they all handle multiple closes gracefully?</p><p>The fact that it is neither guaranteed through io.Closer’s documentation nor in os.File’s, and very likely not through any other I/O object’s documentation, means that you cannot count on it. Yes, it’s unlikely, but in a future version of Go, os.File’s Close() method might panic when called twice.</p><p>And just to illustrate that changes like this in new Go versions are in fact not that unlikely, take e.g. net/http’s NewRequest function. This function had a <a href="https://github.com/golang/go/commit/aa1e063efd7376e268ee592ebe078c6d05b0bdf8#diff-b97af51863ce82bf2a13003b52034aa9">breaking change in Go version 1.8</a>, which was not documented. What happened?</p><p>In versions prior to 1.8, when you passed in a bytes.Buffer as body to NewRequest, it would only read that buffer when the request was actually executed. With version 1.8, the buffer was read in the NewRequest call already. It’s a minor change, but it’s enough if your logic depends on the fact, that the buffer is read lazily.</p><p>Similar things could happen in os.File Close() or other Close() methods.</p><p>“Not a problem”, you could argue, “tests will catch it.” Maybe. Maybe not. The panic could come from a subtle bug which only triggers once in a million. So it easily passes all your tests. Unfortunately, according to Murphy’s Law, “Anything that can go wrong will go wrong.”</p><p>Once your production system starts serving 100 requests per second, this problem will arise roughly every 3 hours. Add to this a couple of other issues in your production system, and you will have a lot of work in front of you digging yourself out of that hole. You better have goodies like a log archiver and visualizer and a fast deployment cycle, otherwise this can become a real nightmare.</p><p>For io.Closer itself, I wish the documentation would be updated from</p><blockquote>The behavior of Close after the first call is undefined. Specific implementations may document their own behavior.</blockquote><p>to be more specific. I.e. either say</p><blockquote>Callers must not call Close more than once. Otherwise the behavior is undefined, implementations may choose to panic.</blockquote><p>or</p><blockquote>Implementations of Close must be idempotent, i.e. callers can call it as often as they want. After the first call it must be a no-op.</blockquote><p>After all, how does an interface help, when the contract is not the same across different implementations? Unfortunately, making either change would break code, so it’s difficult to introduce.</p><p>In my experience it’s better to just avoid any of this trouble and use a construct like SafeCloser or any other that accomplishes the same.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bba10ce73c46" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>