<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daz Wilkin on Medium]]></title>
        <description><![CDATA[Stories by Daz Wilkin on Medium]]></description>
        <link>https://medium.com/@DazWilkin?source=rss-5e4c7deea497------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*7eoQ4m8uoWX5NxvY.</url>
            <title>Stories by Daz Wilkin on Medium</title>
            <link>https://medium.com/@DazWilkin?source=rss-5e4c7deea497------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 19:26:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@DazWilkin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Google Trillian for Noobs (1d)]]></title>
            <link>https://medium.com/@DazWilkin/google-trillian-for-noobs-1d-5be929a69319?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/5be929a69319</guid>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[google-trillian]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Mon, 30 Sep 2019 20:45:08 GMT</pubDate>
            <atom:updated>2019-09-30T20:45:08.901Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Way back when I worked at Google, I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-9b81547e9c4a">wrote</a> a series of posts on Google’s <a href="http://github.com/google/trillian">Trillian</a>. At that time, I contemplated writing a gRPC client for a Trillian personality in Rust.</p><p>I’ve succeeded (barely) and, for closure, am referencing the post here:</p><p><a href="https://pretired.dazwilkin.com/posts/190930/">PyPi Transparency Client (Rust)</a></p><blockquote><strong>NB</strong> Feedback welcome but — please be gentle — I’m a Rustacean Noob ;-)</blockquote><blockquote><strong>NB</strong> The client is for a different Trillian personality than I used in these posts but, it’s sufficiently trivial that it should be straightforward to migrate it.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5be929a69319" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[(p)retired]]></title>
            <link>https://medium.com/google-cloud/p-retired-621563bf4c6d?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/621563bf4c6d</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Wed, 21 Aug 2019 02:36:37 GMT</pubDate>
            <atom:updated>2019-09-08T03:56:10.733Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong>Update</strong>: <a href="https://pretired.dazwilkin.com/">https://pretired.dazwilkin.com</a></blockquote><p>I (p)retired from Google last week.</p><p>The plan is to continue to do more of the same mostly using Golang (though I’m learning Rust), mostly using Kubernetes (though Docker Compose is better for local development) and mostly using Google Cloud Platform (though I’ve a soft spot for DigitalOcean and will continue to use that too).</p><p>I’m writing a personality for Google <a href="http://github.com/google/trillian">Trillian</a>, plan to write at least one more, and have several ideas for this compelling platform.</p><p>I plan to stop using Medium. Partly to draw a notional line under my musings as a Googler. More because I just don’t like Medium’s write-but-don’t-read constraint. I started with Medium because there’s a Google Cloud Platform publication but this is no longer a limitation for me (or GCP).</p><p>Thanks for reading my stuff and best wishes!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621563bf4c6d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/p-retired-621563bf4c6d">(p)retired</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cloud Build, Go Modules & gRPC]]></title>
            <link>https://medium.com/@DazWilkin/cloud-build-go-modules-grpc-12c09b4204d5?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/12c09b4204d5</guid>
            <category><![CDATA[go-modules]]></category>
            <category><![CDATA[google-cloud-build]]></category>
            <category><![CDATA[grpc]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 05:10:36 GMT</pubDate>
            <atom:updated>2019-07-11T17:25:50.065Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Last one! :-)</p><p>We’ll now combine this week’s adventures with Go Modules, immutable package repos and <a href="https://cloud.google.com/cloud-build">Cloud Build</a> into a more realistic Golang project, hosted on GitHub, that (a) regenerates the <a href="http://grpc.io/">gRPC</a> bindings; (b) builds multiple Golang binaries; and c) generates container images in <a href="https://cloud.google.com/container-registry">Google Container Registry</a> (GCR) for the binaries, on each git push.</p><h4>Protobuf Compiler image</h4><p>The Protobuf compiler is called protoc. We’ll use Cloud Build’s community-provided protoc image (<a href="https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/protoc">link</a>).</p><p>Unfortunately, unlike Google’s cloud-builders, these community contributions aren’t automatically built by Cloud Build (the irony!) and aren’t accessible from a container image registry.</p><p>So, you need to Cloud Build protoc for yourself :-(</p><h4>Cloud Build</h4><p>You do you, but ensure Cloud Build is enabled in your project. I recommend starting out by adding cloudbuild.yaml to your Golang project’s local repo clone. This permits more rapid testing.</p><p>Let’s start by defining some global options for our Cloud Build. We’ll set GO111MODULE and GOPROXY once here and defines a volume called go-modules that will hold our Golang package repo:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/803d1f188f775541ae1ceda70ed70c1e/href">https://medium.com/media/803d1f188f775541ae1ceda70ed70c1e/href</a></iframe><p>Assuming you have a protos subdirectory within your repo containing your *.protos, you’ll need to complement the above protoc image with — at a minimum — the Golang protoc compiler plugin. Following the convention protoc-gen-[[LANGUAGE]], this is called <a href="https://github.com/golang/protobuf/tree/master/protoc-gen-go">protoc-gen-go</a>. And, while protoc is C++, protoc-gen-go is written in Golang.</p><p>A wrinkle here is that protoc looks for plugins in PATH. Because we’re pipelining container images like monsters, this is something we don’t have. Fortunately, there’s a solution, when invoked,protoc permits explicit references to plugins to be enumerated by their paths.</p><p>But, first we need to go get the plugin. Here’s we’ll begin accruing packages in go-modules:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12b4f5adb6e0bd7360a3a80f58c8d19e/href">https://medium.com/media/12b4f5adb6e0bd7360a3a80f58c8d19e/href</a></iframe><blockquote><strong>NB</strong> Because we defined options above, we needn’t define env and volumes for this step. They’re automatically applied for us. Theprotoc-gen-go binary will be available in /go/bin/protoc-gen-go at the completion of this step.</blockquote><p>Now we may run protoc and reference protoc-gen-go:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3c52004bb3b4f93c874848440e99506/href">https://medium.com/media/f3c52004bb3b4f93c874848440e99506/href</a></iframe><blockquote><strong>NB</strong> If you are using other plugins, simply enumerate the list of binaries here, each preceeded with --plugin=protoc-gen-[[NAME]]=path/to/protoc-gen-[[NAME]].</blockquote><blockquote><strong>NB</strong> I was unable to use *.protos as I would usually do. This appears to be a limitation of the container image. You’ll see a solution to this in the next step that is not possible here. You may enumerate each protobuf file that you wish to compile.</blockquote><p>A curious side-effect of using Modules and holding sources outside of GOPATH is that protoc generates Golang files assuming the use of GOPATH. I’ve yet to discover an elegant solution to this (see: <a href="https://stackoverflow.com/q/56978791/609290">question</a>).</p><p>In the protoc step, you’ll see that /go/src is used as ouput. The only solution I have is to then move the generated Golang files back to my source. I’m placing them alongside *.proto in protos. So:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b52d424a42db2d9725cf348f10b88cc9/href">https://medium.com/media/b52d424a42db2d9725cf348f10b88cc9/href</a></iframe><p>Add any other pre-build steps that you may need. For my project, although I need a slightly more involved protoc step, this is all I need and I can now build my project binaries. I have 2 but you may create a step for each binary:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a3df167f4e863072530d6ebb5b751ac/href">https://medium.com/media/4a3df167f4e863072530d6ebb5b751ac/href</a></iframe><p>Then, I have a simple Dockerfile that adds the binary to gcr.io/distroless/base and add an ENTRYPOINT:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed20782645fc553e11185b56bc9f83cc/href">https://medium.com/media/ed20782645fc553e11185b56bc9f83cc/href</a></iframe><blockquote><strong>NB</strong> This tags the image with ${COMMIT_SHA}. When running locally, this value will be unavailable but you may provide it using gcloud builds submit ... --substitutions=COMMIT_SHA=$(git rev-parse HEAD)</blockquote><p>Wrap it all up by enumerating the images to push to GCR:</p><pre>images:<br>- gcr.io/${PROJECT_ID}/server</pre><p>You may test the build:</p><pre>TAG=$(git rev-parse HEAD)<br>gcloud builds submit . \<br>--config=cloudbuild.yaml \<br>--substitutions=COMMIT_SHA=${TAG} \<br>--project=${PROJECT}</pre><p>Once you have everything working to your satisfaction, you may configure the Cloud Build job to be triggered by GitHub pushes. Visit:</p><p><a href="https://pantheon.corp.google.com/cloud-build/triggers?project=image-transparency">https://console.cloud.google.com/cloud-build/triggers?project=</a>[[YOUR-PROJECT]]</p><p>And, if you’ve not done so before “Connect Repository”. Otherwise follow “Create Trigger” and follow the self-explanatory instructions. Once completed, you should see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e6XWF43GvqWibHVll_hzFw.png" /></figure><p>And, every time you push to your GitHub repo — including the push that includes the addition of cloudbuild.yaml — a build will be triggered.</p><h4>Conclusion</h4><p>Hopefully, it’s helpful to see an end-to-end Cloud Build scenario like this documented. This example</p><ul><li>Uses Cloud Build options to share env and volumes across steps</li><li>Shares packages across distinct Golang steps to save duplicate gets</li><li>Uses the Cloud Build community contribution protoc</li><li>Add protoc-gen-go and configures protoc to use it</li><li>Builds potentially multiple binaries and generates images for them</li><li>Runs a series of build verification steps…. kidding… I’m done.</li></ul><p>That’s all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12c09b4204d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Modules & Immutability #2]]></title>
            <link>https://medium.com/google-cloud/golang-modules-immutability-2-96cb76a6630e?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/96cb76a6630e</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[go-modules]]></category>
            <category><![CDATA[multi-stage-builds]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 04:02:48 GMT</pubDate>
            <atom:updated>2019-07-11T04:02:48.463Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Yesterday, I <a href="https://medium.com/google-cloud/golang-modules-immutability-9b2fddda1dd8">explored</a> some ways to take advantage of an immutable Golang package store using Docker and Cloud Build. It seems as though there may be a way to take advantage of this immutability using <em>deconstructed</em> multi-stage builds.</p><h4>Multi-Stage Builds</h4><p>This is my boilerplate for Golang multi-stage builds and Google’s distroless</p><pre>FROM golang:1.12 as build<br>WORKDIR /app<br>COPY . .<br>RUN GO111MODULE=on GOPROXY=https://proxy.golang.org go build app</pre><pre>FROM gcr.io/distroless/base<br>COPY --from=build /go/bin/app /<br>ENTRYPOINT [&quot;/app&quot;]</pre><p>Each time this process is run, the golang image’s /go/pkg will be populated by packages relevant to the build.</p><blockquote><strong>Hypothesis</strong>: Interim containers are anonymous but must be persisted to disk. It should be possible to discover these interim containers. But…</blockquote><blockquote><strong>Hypothesis</strong>: by deconstructing a multi-stage build, it should be possible to enforce persisting the interim containers and we could use such an interim to aggregate distinct package pulls from multiple builds into a single source.</blockquote><h4>Deconstructed Multi-stage Builds</h4><p>I’m envisage something like this seed step done once with your first go.mod:</p><pre>FROM <strong>golang:1.12</strong> as <strong>origin<br></strong>WORKDIR /app<br>COPY go.mod .<br>RUN GO111MODULE=on GOPROXY=https://proxy.golang.org <strong>go</strong> <strong>mod</strong> <strong>download</strong></pre><p>and build the image:</p><pre>docker build --tag=golang:modules --file=Dockerfile.orig .</pre><p>Now we have our baseline golang:modules.</p><p>And — subsequently —<strong> for each Go build</strong>— add this build’s packages to golang:modulesrather than golang:</p><pre>FROM <strong>golang:modules</strong> as <strong>modules<br></strong>WORKDIR /app<br>COPY go.mod .<br>RUN GO111MODULE=on GOPROXY=https://proxy.golang.org <strong>go</strong> <strong>mod download</strong></pre><blockquote><strong>NB</strong> I discovered that a directory containing only go.mod failed with e.g. go get ./... However, there is a go mod download command that does exactly what we need.</blockquote><p>and (re)build the golang:modules (packages) image:</p><pre>docker build --tag=<strong>golang:modules</strong> --file=Dockerfile.mods .</pre><blockquote><strong>NB</strong> Yes, golang:modules is potentially going to become problematic with so many layers being added. Docker has an experimental docker build … -squash ... command that may help.</blockquote><p>And then, to build this project:</p><pre>FROM <strong>golang:modules</strong> as <strong>modules</strong></pre><pre>FROM golang:1.12 as <strong>build</strong><br>COPY --from=<strong>modules</strong> /go/pkg /go/pkg<br>WORKDIR /app<br>COPY . .<br>RUN GO111MODULE=on GOPROXY=https://proxy.golang.org <strong>go build app</strong></pre><pre>FROM gcr.io/distroless/base<br>COPY --from=<strong>build</strong> /foo /<br>ENTRYPOINT [&quot;/foo&quot;]</pre><p>and build the image:</p><pre>docker build --tag=thisproject --file=Dockerfile.proj .</pre><h4>Conclusion</h4><p>Using this — admittedly more complicated approach — we (should) be able to accrue distinct packages in golang:modules so that, once pulled, a package is retained and never pulled a second time even across distinct builds.</p><p>The deconstructed multi-stage build Dockerfile may be reconstructed by binding the two build statements together.</p><p>That’s all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96cb76a6630e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/golang-modules-immutability-2-96cb76a6630e">Golang Modules &amp; Immutability #2</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Modules & Immutability]]></title>
            <link>https://medium.com/google-cloud/golang-modules-immutability-9b2fddda1dd8?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b2fddda1dd8</guid>
            <category><![CDATA[go-modules]]></category>
            <category><![CDATA[google-cloud-build]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[cloud-build]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 00:21:20 GMT</pubDate>
            <atom:updated>2019-07-11T04:07:53.961Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Yesterday, I wrote a <a href="https://medium.com/google-cloud/golang-before-after-modules-273b5a5df838">summary</a> of my recent switch to Go Modules. In the conclusion, I wrote that I’m moving to a single ${GOPATH} across my projects. One of the advantages of Modules is that, a package version should be immutable. This implies that, once you’ve pulled the package once, you should never (have to) pull the package again.</p><p>But, of course, that works if you only use one machine. But, what, for example, happens when you use Docker? Is there a way to extend this to Google Cloud Build?</p><h4>Docker Build</h4><p>Let’s add a distroless Docker Build to the example. Create this Dockerfile in ${WORKDIR}:</p><pre>FROM golang:1.12 as build</pre><pre>RUN printf &quot;[timer] start\t%s\n&quot; $(date +%s%N)</pre><pre>COPY ./foo /foo<br>WORKDIR /foo<br>RUN GO111MODULE=on GOPROXY=<a href="https://proxy.golang.org/">https://proxy.golang.org</a> go build foo</pre><pre>RUN printf &quot;[timer] end\t%s\n&quot; $(date +%s%N)</pre><pre>FROM gcr.io/distroless/base</pre><pre>COPY --from=build /foo /<br>CMD [&quot;/foo&quot;]</pre><p>Here’s the results of running this 10 times ensuring Docker doesn’t cache layers between builds with --no-cache:</p><pre>for t in {1..10}<br>do<br>  docker build \<br>  <strong>--no-cache</strong> \<br>  --tag=foo:latest . \<br>  | grep &quot;^\[timer\]&quot;<br>done</pre><pre>[timer] start 1562692830352570907<br>[timer] end   1562692833701779047<br>[timer] start 1562692837743692624<br>[timer] end   1562692841088710635<br>[timer] start 1562692845149766319<br>[timer] end   1562692848465842721<br>[timer] start 1562692852464160691<br>[timer] end   1562692855829662065<br>[timer] start 1562692859765082861<br>[timer] end   1562692863149075221<br>[timer] start 1562692867096511753<br>[timer] end   1562692870563710328<br>[timer] start 1562692874552237719<br>[timer] end   1562692877931096022<br>[timer] start 1562692881991442336<br>[timer] end   1562692885298613761<br>[timer] start 1562692889252781564<br>[timer] end   1562692892541429803<br>[timer] start 1562692896460210629<br>[timer] end   1562692899837684581</pre><p>Which according to Sheets has a mean of 3357913011 and a standard deviation of 50407471 nanoseconds.</p><h4>Docker COPY</h4><p>This isn’t a great solution but it’s not possible to mount e.g. a Docker Volume (which would be more elegant) during Docker builds. Amend (or create another Dockerfile):</p><pre>FROM golang:1.12 as build</pre><pre>RUN printf &quot;[timer] start\t%s\n&quot; $(date +%s%N)</pre><pre><strong>COPY ./go/pkg /go<br></strong>COPY ./foo /foo<strong><br></strong>WORKDIR /foo<br>RUN GO111MODULE=on GOPROXY=<a href="https://proxy.golang.org/">https://proxy.golang.org</a> go build foo</pre><pre>RUN printf &quot;[timer] end\t%s\n&quot; $(date +%s%N)</pre><pre>FROM gcr.io/distroless/base</pre><pre>COPY --from=build /foo /<br>CMD [&quot;/foo&quot;]</pre><p>This time, we’ll run the test by mounting the golang-module-mirror into the container as /go. On the first run which we’ll discard, our mirror will be created, subsequent runs should utilize it:</p><pre>[timer] start 1562694281225041674<br>[timer] end   1562694285811156939<br>[timer] start 1562694289319807596<br>[timer] end   1562694293776883985<br>[timer] start 1562694297206823489<br>[timer] end   1562694301530111138<br>[timer] start 1562694304936211074<br>[timer] end   1562694309252913601<br>[timer] start 1562694312650657096<br>[timer] end   1562694316880599577<br>[timer] start 1562694320360184122<br>[timer] end   1562694324700460041<br>[timer] start 1562694328110722373<br>[timer] end   1562694332475305957<br>[timer] start 1562694335900959477<br>[timer] end   1562694340500560831<br>[timer] start 1562694343962632837<br>[timer] end   1562694348394060653<br>[timer] start 1562694351868504357<br>[timer] end   1562694356285662159</pre><p>Presumably because I’ve such a miniscule local cache (glog only), the time to COPY dramatically outweighs the benefit here. Mean of 4406618035 and a standard deviation of 117921035.6.</p><h4>Google Cloud Build</h4><p>I’ve written several times about Google Cloud Build. It’s an understated and compelling service that provides a mechanism to run a series of container images, pipelining the results from one to the next. Most (!) of the time, the service is used to build container images but it can be used to build many more assets.</p><p>I realized that, by default, using a library/Golang image rather than gcr.io/cloud-builders/go there’s a need to persist e.g. /go between build steps in order that 2 distinct golang steps can share a ${GOPATH}.</p><p>The solution to this is to use Cloud Build volumes (see below). In this example, the volume named go-modules is created and mounted on /go for the first step that needs it and the volume persists until no more steps reference it. For every step that references it subsequently, the cache of e.g. go get packages is retained.</p><blockquote><strong>2019–07–10 Update</strong>: I learned that Cloud Build permits an options section (<a href="https://cloud.google.com/cloud-build/docs/build-config#options">link</a>) that permits env and volumes sections to be defined once and be applied to each step rather than repeated in every step as I have below.</blockquote><pre>steps:</pre><pre>- name: golang:1.12<br>  env:<br>  - GOPATH=/go<br>  - GO111MODULE=on<br>  - GOPROXY=<a href="https://proxy.golang.org/">https://proxy.golang.org</a><br>  <strong>dir</strong>: foo<br>  args:<br>  - go<br>  - build<br>  - -o<br>  - /go/bin/<strong>test1</strong><br>  - foo<br>  <strong>volumes</strong>:<br>  - name: go-modules<br>    path: /go</pre><pre>- name: golang:1.12<br>  env:<br>  - GOPATH=/go<br>  - GO111MODULE=on<br>  - GOPROXY=<a href="https://proxy.golang.org/">https://proxy.golang.org</a><br>  <strong>dir</strong>: foo<br>  args:<br>  - go<br>  - build<br>  - -o<br>  - /go/bin/<strong>test2</strong><br>  - foo<br>  <strong>volumes</strong>:<br>  - name: go-modules<br>    path: /go</pre><pre>- name: golang:1.12<br>  env:<br>  - GOPATH=/go<br>  - GO111MODULE=on<br>  - GOPROXY=<a href="https://proxy.golang.org/">https://proxy.golang.org</a><br>  <strong>dir</strong>: foo<br>  args:<br>  - go<br>  - build<br>  - -o<br>  - /go/bin/<strong>test3</strong><br>  - foo<br>  <strong>volumes</strong>:<br>  - name: go-modules<br>    path: /go</pre><pre>- name: busybox<br>  args:<br>  - ls<br>  - -l<br>  - /go/bin<br>  <strong>volumes</strong>:<br>  - name: go-modules<br>    path: /go</pre><blockquote><strong>NB</strong> I think the GOPATH=/go is redundant here as this is the default|working directory for the Golang images.</blockquote><blockquote><strong>NB</strong> Because Cloud Build uses /workspace as a working directory, our sources will be located in /workspace/foo but our packages etc. are persisted in /go.</blockquote><blockquote><strong>NB</strong> Because our sources are in a subdirectory of /workspace, I’m using Cloud Build’s dir modifier to make /workspace/foo the working directory for the builds.</blockquote><blockquote><strong>NB</strong> I’m using GOPROXY to utilize the Golang team’s Go Modules Mirror.</blockquote><p>So, in this admittedly trivial example, each build (of the same thing) creates a differently named binary (testX) and puts this in the shared /go/bin directory.</p><p>The purpose of this though is to show how the packages are cached in /go/pkg once pulled for the first build:</p><pre>gcloud builds submit \<br>--config=./cloudbuild.yaml \<br>--project=[[YOUR-PROJECT]]</pre><pre>...<br>starting build &quot;22137c6c-df17-418b-8177-c231dd34b719&quot;</pre><pre>FETCHSOURCE<br>...<br>BUILD<br>Starting Step #0<br>Step #0: Pulling image: golang:1.12<br>Step #0: 1.12: Pulling from library/golang<br>Step #0: Digest: sha256:3fee5835...<br>Step #0: Status: Downloaded newer image for golang:1.12<br><strong>Step #0: go: finding github.com/golang/glog v0.0.0</strong><br>Finished Step #0<br>Starting Step #1<br>Step #1: Already have image: golang:1.12<br>Finished Step #1<br>Starting Step #2<br>Step #2: Already have image: golang:1.12<br>Finished Step #2<br>Starting Step #3<br>Step #3: Pulling image: busybox<br>Step #3: Using default tag: latest<br>Step #3: latest: Pulling from library/busybox<br>Step #3: Digest: sha256:bf510723...<br>Step #3: Status: Downloaded newer image for busybox:latest<br>Step #3: total 5892<br>Step #3: <strong>test1</strong><br>Step #3: <strong>test2</strong><br>Step #3: <strong>test3</strong><br>Finished Step #3<br>PUSH<br>DONE</pre><p>However, if we remove|comment out the references to volumes and remove the busybox step since no files have been created ingo/bin and rerun the build:</p><pre>starting build &quot;3837eb7d-dae6-476e-9e72-67ee2f57de86&quot;</pre><pre>FETCHSOURCE<br>...<br>BUILD<br>Starting Step #0<br>Step #0: Pulling image: golang:1.12<br>Step #0: 1.12: Pulling from library/golang<br>Step #0: Digest: sha256:3fee5835...<br>Step #0: Status: Downloaded newer image for golang:1.12<br><strong>Step #0: go: finding github.com/golang/glog v0.0.0</strong><br>Finished Step #0<br>Starting Step #1<br>Step #1: Already have image: golang:1.12<br><strong>Step #1: go: finding github.com/golang/glog v0.0.0</strong><br>Finished Step #1<br>Starting Step #2<br>Step #2: Already have image: golang:1.12<br><strong>Step #2: go: finding github.com/golang/glog v0.0.0</strong><br>Finished Step #2<br>PUSH<br>DONE</pre><p>You’ll see this time that the glog package is pulled each time the build is run. This is because we’re not sharing go/pkg across the build steps.</p><p>I suspect that this mechanism would be a time-saver if, for example, you have an existing multi-stage Dockerfile or you have a repo comprising several binaries and you want to emit different container images for each binary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b2fddda1dd8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/golang-modules-immutability-9b2fddda1dd8">Golang Modules &amp; Immutability</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Golang Before||After Modules]]></title>
            <link>https://medium.com/google-cloud/golang-before-after-modules-273b5a5df838?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/273b5a5df838</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[gopath]]></category>
            <category><![CDATA[go-modules]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Mon, 08 Jul 2019 20:46:36 GMT</pubDate>
            <atom:updated>2019-07-09T16:34:12.475Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Although I am familiar (and a fan of) the old ${GOPATH} way, to remain current and because of the many benefits, I’ve begun to use Go Modules.</p><p>Like others, I found the switch to be confusing. So…</p><h4>Before Modules</h4><pre>WORKDIR=[[PATH-TO-YOUR-WORKING-DIRECTORY]]</pre><pre>mkdir -p ${WORKDIR}/go<br>export GOPATH=${WORKDIR}/go<br>export PATH=${GOPATH}/bin:${PATH}</pre><pre>mkdir -p {${WORKDIR}/go/src/foo, ${WORKDIR}/go/src/foo/bar}</pre><p>Then create ${WORKDIR}/go/src/foo/bar/library.go:</p><pre>package bar</pre><pre>func Something() (string) {<br>    return &quot;Hello Freddie&quot;<br>}</pre><p>Then create ${WORKDIR}/go/src/foo/main.go:</p><pre>package main</pre><pre>import (<br>    &quot;fmt&quot;<br>    &quot;foo/bar&quot;<br>)</pre><pre>func main() {<br>    fmt.Printf(&quot;%s&quot;, bar.Something())<br>}</pre><p>You’ll have a structure like this:</p><pre>.<br>└── go<br>    └── src<br>        └── foo<br>            ├── bar<br>            │   └── library.go<br>            └── main.go</pre><p>Then you may run this in any of the following ways:</p><pre>GO111MODULE=<strong>off</strong> go run ${WORKDIR}/go/src/foo/main.go<br>Hello Freddie!</pre><pre>cd ${WORKDIR}/go/src/foo<br>GO111MODULE=<strong>off</strong> go run main.go<br>Hello Freddie!</pre><pre>GO111MODULE=<strong>off</strong> go run foo<br>Hello Freddie!</pre><blockquote><strong>NB</strong> With the above, you may omit GO111MODULE=off because this is implicit when sources are within ${GOPATH}. I’ve included it to be explicit with intent.</blockquote><h4>After Modules</h4><p>Assuming you’ve done the above!</p><p>You don’t need to do this step but this is the new best practice. We’re moving our sources outside of ${GOPATH}. ${GOPATH} is still used to store our versioned packages.</p><blockquote><strong>NB</strong> In this trivial example, we&#39;re not using external packages so ${GOPATH} remains empty. See the end for an example.</blockquote><pre>mv ${WORKDIR}/go/src/foo ${WORKDIR}<br>rm ${WORKDIR}/go/src</pre><p>You should have a structure like this:</p><pre>.<br>├── <strong>foo</strong><br>│   ├── bar<br>│   │   └── library.go<br>│   └── main.go<br>└── go</pre><blockquote><strong>NB</strong> Our sources rooted on <strong>foo</strong> are now outside of ${GOPATH}</blockquote><pre>cd ${WORKDIR}/foo</pre><pre>GO111MODULE=<strong>on</strong> go mod init foo<br>more go.mod<br>module foo</pre><pre>go 1.12</pre><pre>GO111MODULE=<strong>on</strong> go run foo<br>Hello Freddie!</pre><pre>GO111MODULE=<strong>on</strong> go run main.go<br>Hello Freddie!</pre><h4>For Completeness</h4><p>To show the difference with external packages. Not using Modules, go get pulls the latest version of the package into ${GOPATH}/src:</p><pre>GO111MODULE=<strong>off</strong> go get github.com/golang/glog</pre><pre>.<br>├── foo<br>│   ├── bar<br>│   │   └── library.go<br>│   ├── go.mod<br>│   ├── go.sum<br>│   └── main.go<br>└── <strong>go</strong><br>    └── <strong>src</strong><br>        └── github.com<br>            └── golang<br>                └── glog<br>                    ├── glog_file.go<br>                    ├── glog.go<br>                    ├── glog_test.go<br>                    ├── LICENSE<br>                    └── README</pre><p>Versus using Modules, pulls a specific version (or versions) of the package into ${GOPATH}/pkg:</p><pre>GO111MODULE=<strong>on</strong> go get github.com/golang/glog</pre><pre>.<br>├── foo<br>│   ├── bar<br>│   │   └── library.go<br>│   ├── go.mod<br>│   ├── go.sum<br>│   └── main.go<br>└── go<br>    └── <strong>pkg</strong><br>        ├── linux_amd64<br>        │   └── github.com<br>        │       └── golang<br>        │           └── glog.a<br>        └── mod<br>            ├── cache<br>            │   ├── download<br>            │   │   └── github.com<br>            │   │       └── golang<br>            │   │           └── glog<br>            └── github.com<br>                └── golang<br>                    └── glog@v0.0.0-20160126235308-23def4e6c14b<br>                        ├── glog_file.go<br>                        ├── glog.go<br>                        ├── glog_test.go<br>                        ├── LICENSE<br>                        └── README</pre><h4>Go Modules Mirror</h4><p>See <a href="https://proxy.golang.org/">https://proxy.golang.org/</a></p><p>You include PROXY=https://proxy.golang.org in all the above commands to take advantage of the Golang team’s (Google Trillian based) Mirror:</p><pre>GO111MODULE=on \<br>GOPROXY=https://proxy.golang.org \<br>go get github.com/golang/glog</pre><h4>Trick</h4><p>Thanks to this <a href="https://www.reddit.com/r/golang/comments/9blgn0/using_go_modules_with_vendorprovided_protobuf/">link</a> for this useful trick finding packages that will be installed somewhere in go/pkg/mod when using Modules:</p><pre>GO111MODULE=on \<br>go list -f &#39;{{ .Dir }}&#39; github.com/golang/glog</pre><pre>go: finding github.com/golang/glog latest<br>/path/to/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b</pre><h4>Visual Studio Code</h4><p>A few folks asked for my configuration settings for Visual Studio Code w/ Go Modules. Here’s what I have:</p><pre>{<br>    &quot;go.autocompleteUnimportedPackages&quot;: true,<br>    &quot;go.useLanguageServer&quot;: true,<br>    &quot;[go]&quot;: {<br>        &quot;editor.snippetSuggestions&quot;: &quot;none&quot;,<br>        &quot;editor.formatOnSave&quot;: true,<br>        &quot;editor.codeActionsOnSave&quot;: {<br>            &quot;source.organizeImports&quot;: true<br>        }<br>    },<br>    &quot;gopls&quot;: {<br>        &quot;usePlaceholders&quot;: true // add parameter placeholders when completing a function<br>    },<br>    &quot;files.eol&quot;: &quot;\n&quot;, // formatting only supports LF line endings    <br>    &quot;go.toolsEnvVars&quot;: {<br>        &quot;GO111MODULE&quot;: &quot;on&quot;,<br>        &quot;GOPROXY&quot;: &quot;<a href="https://proxy.golang.org/">https://proxy.golang.org</a>&quot;<br>    },<br>}</pre><p>A combination of help from this <a href="https://github.com/Microsoft/vscode-go/wiki/Go-modules-support-in-Visual-Studio-Code">page</a> and this <a href="https://github.com/golang/go/issues/32903#issuecomment-507817147">reply</a> from a Google Engineer. I try to remain current but the Golang team’s gopls had passed me by entirely :(</p><p><a href="https://github.com/golang/go/wiki/gopls">https://github.com/golang/go/wiki/gopls</a></p><h4>Conclusion</h4><p>Modules are the <a href="https://github.com/golang/go/wiki/Modules">future</a> of package management in Golang. Having now begun the plunge, I mostly see benefits. Even with my simple Golang repos, there’s utility in being able to define specific versions of packages that are imported.</p><p>Initially I didn’t understand (and maybe still don’t understand) the uility in not including my sources under ${GOPATH} but, one advantage is that it becomes more evident which sources are my sources and which are others packages. When everything’s smooshed under ${GOPATH} this was confusing.</p><p>In part, the use of semantic versioning (aka SemVer) also means that, once you’ve go get useful-package@v1.0.0 you should (!) never need to re-get it. I’ve previously always kept my ${GOPATH} directories distinct by project to keep code isolated. But, by pulling my sources out of ${GOPATH} and because I only need one copy of useful-package@v1.0.0, and (!) because I know this package is immutable, I will now begin using a single, central ${GOPATH} for all my projects.</p><p>That’s all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=273b5a5df838" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/golang-before-after-modules-273b5a5df838">Golang Before||After Modules</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Trillian for Noobs (1c)]]></title>
            <link>https://medium.com/google-cloud/google-trillian-for-noobs-1c-dddb66ddd09f?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/dddb66ddd09f</guid>
            <category><![CDATA[google-trillian]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[zipkin]]></category>
            <category><![CDATA[opencensus]]></category>
            <category><![CDATA[jaeger]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Fri, 05 Jul 2019 20:56:20 GMT</pubDate>
            <atom:updated>2019-07-05T23:29:16.260Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>The Missing Manuals series</h4><p>Last week I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-9b81547e9c4a">documented</a> what I hope is the simplest possible Trillian personality. Yesterday, I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-1a-c87a78e5e585">documented</a> adding an inclusion proof. Earlier today, I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-1b-16097474ee3d">documented</a> building a gRPC-based client and server for the personality. Here is a small addition that adds metrics (stats) and traces.</p><h4>OpenCensus Exporter</h4><p>With the addition of a straightforward configuration for an OpenCensus Exporter using the OpenCensus Agent, we have the ability to configure the Agent to convert incoming stats|traces into a wide selection of 3rd-party services.</p><p>Here’s the Basic Personality server configuration:</p><pre>oc, err := ocagent.NewExporter(<br> ocagent.WithAddress(*ocagEndpoint),<br> ocagent.WithInsecure(),<br> ocagent.WithReconnectionPeriod(10*time.Second),<br> ocagent.WithServiceName(serviceName),<br>)<br>if err != nil {<br> log.Fatal(err)<br>}<br>defer oc.Stop()</pre><p>And here’s the Agent’s configuration to receive incoming stats|traces on :55678 and export the stats to Prometheus (on :9100) and the route the traces to thezipkin service on its port :9411:</p><pre>receivers:<br>  opencensus:<br>    address: &quot;:55678&quot;</pre><pre>exporters:<br>  jaeger:<br>    collector_endpoint: &quot;<strong>jaeger</strong>:14268/api/traces&quot;<br>  prometheus:<br>    address: &quot;:9100&quot;<br>  zipkin:<br>    endpoint: &quot;<strong>zipkin</strong>:9411/api/v2/spans&quot;</pre><pre>zpages:<br>    port: 9999</pre><blockquote><strong>NB</strong> A potential source of confusion. These DNS names are provided by Docker Compose to the services running on the network it creates. These names are accessible to other services within the solution. In this case, the OpenCensus Agent (itself known as opencensus-agent) is able to refer to these other services by their service names (jaefer ,zipkin) but only within the solution. Externally, we need to port map these services onto an available port on localhost.</blockquote><h4>OpenCensus Agent</h4><p>The OpenCensus Agent export as a Prometheus Exporter on the host on :9100 so you can query <a href="http://localhost:9100/metrics">http://localhost:9100/metrics</a> and hopefully see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGcHHOHiV7EjBNh0TotDsA.png" /></figure><p>We can confirm the OpenCensus Agent by checking its logs:</p><pre>docker-compose \<br>--file=deployment/docker-compose.yml \<br>logs <strong>opencensus-agent</strong></pre><pre>Attaching to deployment_opencensus-agent_1<br>opencensus-agent_1          | {&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1562357412.0980296,&quot;caller&quot;:&quot;config/config.go:490&quot;,&quot;msg&quot;:&quot;Trace Exporter enabled&quot;,&quot;exporter&quot;:&quot;zipkin&quot;}<br>opencensus-agent_1          | {&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1562357412.0983222,&quot;caller&quot;:&quot;config/config.go:497&quot;,&quot;msg&quot;:&quot;Metrics Exporter enabled&quot;,&quot;exporter&quot;:&quot;prometheus&quot;}<br>opencensus-agent_1          | 2019/07/05 20:10:13 Running OpenCensus Trace and Metrics receivers as a gRPC service at &quot;:55678&quot;<br>opencensus-agent_1          | 2019/07/05 20:10:13 Running zPages on port 9999</pre><p>The OpenCensus Agent is configured (:9999) to show zPages so we can interrogate that endpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*araLUG_hqdc7ubtQnfw19A.png" /><figcaption>/debug/tracez</figcaption></figure><blockquote><strong>NB</strong> I would have expected to see upstream gRPC traces reflected by the Agent but they’re not. So this is one problem.</blockquote><h4>Prometheus</h4><p>The OpenCensus Agent is now a scrape target (via the above Exporter) for the Prometheus server. So the Basic Personality client and server gRPC metrics are exported to the OpenCensus Agent, which exports them as a Prometheus Exporter and they’re then captured by this server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1IkugrEa0OIh2k_vD184gw.png" /></figure><h4>Zipkin</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvNxpDTqWImnu7FEsFbdgw.png" /></figure><p>For some reason — to be explored — Zipkin either isn’t receiving or isn’t able to process the traces routed through the OpenCensus Agent that are generated by the automatic ocgrpc handler.</p><p>You can confirm the Zipkin service logs too though these are verbose and not included here:</p><pre>docker-compose \<br>--file=deployment/docker-compose.yml \<br>logs zipkin</pre><h4>Jaeger</h4><p>When Zipkin wasn’t showing traces, I added <a href="https://www.jaegertracing.io/">Jaeger</a> (another trace monitoring tool) to see whether it was Zipkin, the upstream OpenCensus Agent, or something else. Jaeger is reported its own traces but no others. So, there’s something either in my configuration or the OpenCensus Agent that’s not working:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkIa7eOq3bn_y1WERGrD5g.png" /></figure><h4>Conclusion</h4><p>An ‘amuse bouche’ for integration stats and traces into the Basic Personality. It would be more awesome if it worked perfectly ;-)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dddb66ddd09f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-trillian-for-noobs-1c-dddb66ddd09f">Google Trillian for Noobs (1c)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Trillian for Noobs (1b)]]></title>
            <link>https://medium.com/google-cloud/google-trillian-for-noobs-1b-16097474ee3d?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/16097474ee3d</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[merkle-tree]]></category>
            <category><![CDATA[google-trillian]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[zpages]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Fri, 05 Jul 2019 19:11:58 GMT</pubDate>
            <atom:updated>2019-07-05T19:24:36.273Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>The Missing Manual series</h4><p>Last week I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-9b81547e9c4a">documented</a> what I hope is the simplest possible Trillian personality. Yesterday, I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-1a-c87a78e5e585">documented</a> adding an inclusion proof. Today we’ll split the main.go into a client and a server and reconnect them using gRPC. This is a gRPC rather than Trillian work but, it helps to evolve the personality.</p><h4>Setup</h4><p>You’ll need the Database and Trillian Servers described in my previous post.</p><h4>A gRPC-based Personality Server</h4><p>This time, either clone gRPC:</p><pre>git clone \<br>--single-branch \<br>--branch=gRPC \<br><a href="/service/mailto:git@github.com">git@github.com</a>:DazWilkin/simple-trillian-log-1.git</pre><p>Or you may just run the Docker Compose file:</p><pre>docker-compose --file=./deployment/docker-compose.yml up</pre><blockquote><strong>NB</strong> In either case you will need to have created the Database.</blockquote><p>Docker Compose will create 7 services:</p><pre>Name                       State  Ports<br>--------------------------------------------------------------------<br>adminer                    Up     0.0.0.0:7777-&gt;8080/tcp<br>basic-personality-client   Up     0.0.0.0:9997-&gt;9999/tcp<br>basic-personality-server   Up     50051/tcp<br>                                  0.0.0.0:8080-&gt;8080/tcp<br>                                  0.0.0.0:9998-&gt;9999/tcp<br>db                         Up     3306/tcp<br>prometheus                 Up     0.0.0.0:9090-&gt;9090/tcp<br>trillian-log-server        Up     0.0.0.0:54051-&gt;50051/tcp, 8080/tcp<br>trillian-log-signer        Up     50051/tcp, 8080/tcp</pre><p>I’ll not describe adminer and db much further. Suffice to say that the db service exposes port 3306 within (!) the network created by Docker Compose to the Trillian Log Server and Log Signer.</p><p>The basic-personality-client exposes a zPages endpoint internally on basic-personality-client:9999 that will be exposed on your host on :9997. You can browse this endpoint using either localhost:9997/debug/rpcz or localhost:9997/debug/tracez.</p><p>The basic-personality-server exposes :50051 internally (only) and this is used by basic-personality-client to access it using gRPC. This service also exposes :8080 (http) and a zPages endpoint internally on basic-personality-server:9999 that will be exposed on your host on :9998. You can browse this endpoint using either localhost:9998/debug/rpcz or localhost:9998/debug/tracez.</p><p>The prometheus services exposes its endpoint on :9090 (see below).</p><p>The trillian-log-server exposes its gRPC endpoint on the host on :54051. This is to provide a host-accessible endpoint to use the createtree binary to create new logs.</p><h4>Prometheus</h4><p>Currently only the Trillian services are instrumented. You can access the Prometheus server on:</p><pre><a href="http://localhost:9090/targets">http://localhost:9090</a><br><a href="http://localhost:9090/">http://localhost:9090</a>/graph<br>http://localhost:9090/status<br>http://localhost:9090/targets<br>http://localhost:9090/-/healthy</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKXujz8aWGn7GjuuMuKnsw.png" /></figure><blockquote><strong>NB</strong> Because Prometheus is part of the Docker Compose network, the service is able to reference the Trillian services using their network names (trillian-log-server and trillian-log-signer). You will need to create port bindings to expose these services’ :8080 ports on the host if you wish to access their /metrics endpoints directly.</blockquote><h4>zPages</h4><p>The trivial addition of OpenCensus’ zPages package, provides details of gRPC calls including traces. Google’s convention is to pluralize (pluralise?) these with a ‘-z’. So, you can browse:</p><pre><a href="http://localhost:9998/rpcz">http://localhost:9998/debug/rpcz</a><br>http://localhost:9998/debug/tracez</pre><p>And — for the basic-personality-server (exposed on :9998) you should see:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-YsAns1--8iY-0swX7R-Uw.png" /><figcaption>/debug/rpcz</figcaption></figure><p>And:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*1MUtcPLxyLVf03t8AENvog.png" /><figcaption>/debug/tracez</figcaption></figure><blockquote><strong>NB</strong> For tracez the first column is in-flight|running traces so you have to refresh at the right time. The latency histogram provides more opportunity for drilldown.</blockquote><h4>Channelz</h4><p>The Basic Personality gRPC server exposes a <a href="https://grpc.io/blog/a_short_introduction_to_channelz/">channelz</a> endpoints over its gRPC endpoint (:50051). If you wish to explore the functionality of channelz, please see <a href="https://github.com/grpc/grpc-experiments/tree/master/gdebug">gdebug</a>. The simplest solution is to revise the Docker Compose file to expose this port to the host so that you may run gdebug on the host and so that gdebug can access the Basic Personality service.</p><h4>Protobuf</h4><p>The essence of the changes in this branch of the repo are a result of the addition of protobuf definitions for the messages our original client (main.go) was shipping to and from the Trillian Log Server. We’d refactored this code so it appeared to be interacting with a server when, in fact this was purely a naming|package conceit. You’ll recall we have 3 functions with similar signatures. Of the form:</p><pre>server.<strong>put</strong>(&amp;Request{<br>  thing: *thing,<br>  extra: *extra,<br> }</pre><p>After creating protobuf definitions of these functions and messages. Then new client code looks similar now using a gRPC client proxy:</p><pre>client.PutThing(ctx, &amp;pb.ThingRequest{<br>   Thing: &amp;pb.Thing{...},<br>   Extra: &amp;pb.Extra{...},<br>})</pre><p>and the gRPC server proxy is just:</p><pre>func (s *server) PutThing(ctx context.Context, r *pb.ThingRequest) (*pb.ThingResponse, error) { return s.<strong>put</strong>(r) }</pre><p>I’m not wanting to bore you with a gRPC tutorial but, for completeness, pb.ThingRequest and pb.ThingResponse types are described in the protobuf definition along with our renamed — but otherwise similar — functions PutThing, GetThing, and WaitThing.</p><pre>syntax = &quot;proto3&quot;;</pre><pre>package google.basicpersonality.v1;</pre><pre>option go_package = &quot;github.com/DazWilkin/basic-personality/protos&quot;;<br>option java_package = &quot;com.google.dazwilkin.basicpersonality&quot;;</pre><pre>service BasicPersonality {<br>  rpc PutThing(ThingRequest) returns (ThingResponse) {};<br>  rpc GetThing(ThingRequest) returns (ThingResponse) {};<br>  rpc WaitThing(ThingRequest) returns (ThingResponse) {};</pre><pre>message <strong>ThingRequest</strong> {<br>  Thing thing = 1;<br>  Extra extra = 2;<br>}<br>message <strong>ThingResponse</strong> {<br>  string status = 1;<br>}<br>message Thing {<br>  string name = 1;<br>}<br>message Extra {<br>  string name = 1;<br>}</pre><p>All that we need do is provide the above definition to the protobuf compiler (protoc) , configure it to generate Golang bindings for us and, we’re almost done.</p><p>We then need to decouple the original main.go into a client and a server that each use the machine-generated gRPC bindings for BasicPersonality and let the gRPC runtime take care of passing messages between them.</p><h4>Docker Compose</h4><p>As mentioned previously, the Docker Compose file is extended to run the basic-personality-client and basic-personality-server alongside the Database, a Trillian Log Server and Log Signer, and Prometheus. We configure each with the gRPC port (:50051) to bind them together.</p><p>The client is configured to run forever so albeit with 15-second pauses. When you up the Docker Compose file, be careful not to leave it running as you’ll eventually blow up the database.</p><h4>Conclusion</h4><p>We now have a very basic Trillian personality called “Basic Personality” that is implemented as Golang gRPC server that itself uses gRPC to communicate with the Trillian services. For simplicity, we generated a Golang gRPC client that integrates with our Basic Personality server.</p><p>Because Basic Personality has a protobuf service definition and exposes a gRPC endpoint using this definition, you could take the protobuf and generate a client in any gRPC language to talk with the service.</p><p>In the next steps, I think I’m going to (a) instrument these services using OpenCensus; (b) generate a non-Golang gRPC client. I Googled Rust gRPC and it seems non-trivial for a Rust beginner. If anyone reading this would like to show how this could be done, I’d be interested.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16097474ee3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-trillian-for-noobs-1b-16097474ee3d">Google Trillian for Noobs (1b)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Trillian for Noobs (1a)]]></title>
            <link>https://medium.com/google-cloud/google-trillian-for-noobs-1a-c87a78e5e585?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/c87a78e5e585</guid>
            <category><![CDATA[go-modules]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[google-trillian]]></category>
            <category><![CDATA[merkle-tree]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Wed, 03 Jul 2019 20:26:55 GMT</pubDate>
            <atom:updated>2019-07-04T20:47:05.843Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>The Missing Manuals series</h4><p>Last week I <a href="https://medium.com/google-cloud/google-trillian-for-noobs-9b81547e9c4a">documented</a> what I hope is the simplest possible Trillian personality. This is an interim post as I realized I’d missed some important functionality in my sample, an inclusion proof: effectively incontrovertible evidence that some specified data is part of the transparent log.</p><h4>Setup</h4><p>You’ll need the Database and Trillian Servers described in my previous post.</p><h4>A Basic++ Personality</h4><p>This time, please clone inclusion-proof:</p><pre>git clone \<br>--single-branch \<br>--branch=inclusion-proof \<br><a href="/service/mailto:git@github.com">git@github.com</a>:DazWilkin/simple-trillian-log-1.git</pre><p>Then:</p><pre>GO111MODULES=on \<br><strong>GOPROXY=</strong><a href="https://proxy.golang.org/"><strong>https://proxy.golang.org</strong></a> \<br>go run github.com/DazWilkin/simple-trillian-log-1 \<br>--tlog_endpoint=:8090 \<br>--tlog_id=${LOGID}</pre><blockquote><strong>NB</strong> Very interestingly, the Go team is evaluating a module mirror for Go Modules. The mirror not only intends to provide a fast proxy for downloads but, it leverages the hashes of go.sum to provide verifiability of packages. Very, very interestingly, this solution is a Trillian personality ;-)</blockquote><p>You should see output similar to:</p><pre>[main] Entered<br>[main] Establishing connection w/ Trillian Log Server [:8090]<br>[main] Creating new Trillian Log Client<br>[main] Creating Server using LogID [5201727448936460865]<br>[server] Creating<br>[main] Creating a &#39;Thing&#39; and something &#39;Extra&#39;<br>[thing:new] Creating: [2019-07-03T13:01:34-07:00] Thing<br>[extra:new] Creating: Extra<br>[main] Submitting it for inclusion in the Trillian Log<br>[main] Awaiting Inclusion (Proof) in the Trillian Log<br>[main] Retrieving it from the Trillian Log<br>[main:get] Entered<br>[server:get] Entered<br>[thing:marshal] Marshaling: [2019-07-03T13:01:34-07:00] Thing<br>[server:get] hash: 0800eb65...<br>[main:wait] Entered<br>[server:wait] Entered<br>[main:put] Entered<br>[server:put] Entered<br>[thing:marshal] Marshaling: [2019-07-03T13:01:34-07:00] Thing<br>[extra:marshal] Marshaling: Extra<br>[server:wait] Root hash: 68f7634b...<br>[main:get] Status:ok<br>[main:get] Done<br>[thing:marshal] Marshaling: [2019-07-03T13:01:34-07:00] Thing<br><strong>[main:wait]</strong> rpc error: code = NotFound desc = No leaf found for hash: 0800eb65... in tree size 39<br>[main:wait] Status:<br>[main:wait] Sleeping<br>[main:put] Status:ok<br>[main:put] Done<br>[server:wait] Entered<br>[server:wait] Root hash: 6c7a5673...<br>[thing:marshal] Marshaling: [2019-07-03T13:01:34-07:00] Thing<br>[main] proof[0],hash[0] == 9dc9b7d5...<br>[main] proof[0],hash[1] == 384214f3...<br>[main] proof[0],hash[2] == 36e35370...<br>[main] proof[0],hash[3] == 880846d5...<br><strong>[main:wait]</strong> Status:ok<br>[main:wait] Done<br>[main] Done</pre><p>This code attempts to mirror usage of a log. There are 3 goroutines in main.go. One of these — put — attempts to asynchronously add a Thing to the log. Another — get — attempts to asynchronously get the Thing from the log. The third — wait — attempts to perform an inclusion proof on the Thing added to the log. get and wait cannot proceed until put has succeeded. More importantly, wait cannot succeed until it’s able to calculate the inclusion proof. For this reason, in the above log output, although get succeeds on its first attempt, wait fails, sleeps for one second, and succeeds on the retry.</p><h4>Trillian Client</h4><p>When I asked for details on implementing the inclusion proof, the Trillian team pointed me to the Trillian Client (trillian/client) package. I did not realize this existed :-) The code in this sample uses the machine-generated (from Trillian’s protobufs) gRPC client. However, the Trillian Client is an abstraction above this that also includes functions for inclusion proofs. In your code, I recommend you use trillian/client rather than the raw gRPC client.</p><h4>Conclusion</h4><p>We now have a client (main.go) personality that interacts with a Trillian Log to add, retrieve and prove existence of Things. In the next story, we’ll turn this client into a server, expose it using gRPC and generate a (test) client to use against it. In the final (?) post, I plan (!) to try my hand at Rust and generate a Rust client for this personality. That’s not really Trillian but — hopefully — it goes some way to helping you understand how you may use this solution.</p><p>That’s all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c87a78e5e585" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-trillian-for-noobs-1a-c87a78e5e585">Google Trillian for Noobs (1a)</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prometheus Exporter for Particle]]></title>
            <link>https://medium.com/@DazWilkin/prometheus-exporter-for-particle-6203bf70abf5?source=rss-5e4c7deea497------2</link>
            <guid isPermaLink="false">https://medium.com/p/6203bf70abf5</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[prometheus]]></category>
            <category><![CDATA[particle]]></category>
            <dc:creator><![CDATA[Daz Wilkin]]></dc:creator>
            <pubDate>Mon, 01 Jul 2019 01:30:00 GMT</pubDate>
            <atom:updated>2019-07-01T04:31:04.584Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h4>Weekend Hacking</h4><p><a href="http://prometheus.io/">Prometheus</a> is one of the technologies that I find elegant. Reading on its extensive list of <a href="https://prometheus.io/docs/instrumenting/exporters/">integrations</a> on Friday afternoon while walking my dog, I was inspired to write an Exporter for <a href="http://particle.io/">Particle</a>.</p><p>The following documents some weekend hacking. I have a working solution but more works needs to be done defining Particle metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_lT9lhq15ljzlscjd9c6Q.png" /><figcaption>Prometheus Exporter for Particle</figcaption></figure><p>The <a href="https://github.com/DazWilkin/particle-exporter/commit/508e488a5ae8291cc605f67f4265a6479eb2c17b">current</a> solution decouples Particle — as a metric source — from Prometheus-as a metric sink — and there is a functional set of Interfaces but the code will benefit from more work, better definition of metrics, and a move from the Expose interface to the Export interface.</p><h4>Particle</h4><p><a href="http://particle.io/">Particle</a> provides a comprehensive REST-based (Particle Cloud) <a href="https://docs.particle.io/reference/device-cloud/api">API</a> that’s commonly accessed through Particle’s <a href="https://console.particle.io/">Console</a>. Authentication is achieved using access tokens. You’ll need an access token to curl the API and the Prometheus Exporter will need an access token to access the API on your behalf. E.g.:</p><pre>TOKEN=[[YOUR-PARTICLE-TOKEN]]<br>curl \<br>--silent \<br>--request GET \<br>--header &quot;Authorization: Bearer ${TOKEN}&quot; \<br><a href="https://api.particle.io/v1/devices">https://api.particle.io/v1/devices</a> \<br>| jq .</pre><p>The API return JSON representations of Particle resources. My only quibble with the API is that different resource representations are returned for the same resource depending on the API method. For example, the above /v1/devices returns a different (summary) representation of each device than the per-device call /v1/devices/{{DEVICEID}}.</p><p>Fortunately the Golang JSON parser is tolerant of superset values and will only marshal the contents of the struct that’s defined.</p><p>The Exporter implements a simple Particle client that can create correct URL requests and handle responses. The client makes requests against the Particle Cloud asynchronously and enqueues the metrics that result.</p><h4>Prometheus</h4><p>For systems that can’t be instrumented with a Prometheus-compliant metrics library, Exporters provide a scrape endpoint using a trivial text-oriented “exposition” format that’s straightforward to implement. Prometheus provides <a href="https://prometheus.io/docs/instrumenting/writing_exporters/">best-practices for Exporters</a>; I’m going to read the page now :-)</p><p>Here’s an example of a Particle metric rendered in Prometheus’ format:</p><pre># HELP device system memory total.<br># TYPE device_system_memory_total gauge<br>device_system_memory_total{core_id=&quot;123456789abcdef12345678&quot;} 170576</pre><p>A gauge (and a counter) type comprise three lines:</p><ol><li>HELP</li><li>TYPE</li><li>the metric name, labels and its value</li></ol><p>That’s it! In the current implementation, the Interface requires that Particle metrics implement a function that returns string (in this format). The improved (but not implemented) Interface requires that Particle metrics implements one of several metrics types (Gauge, Counter, Histogram) and the implementation of these requires an Export method that renders the above Prometheus format.</p><p>The Exporter implements a simple Exporter server. This dequeues metrics created by the client and renders them, using the above format as the response to requests (by default) on its /metrics endpoint.</p><pre># HELP device system memory used.<br># TYPE device_system_memory_used gauge<br>device_system_memory_used{core_id=&quot;123456789abcdef12345678&quot;} 33072<br># HELP device system memory total.<br># TYPE device_system_memory_total gauge<br>device_system_memory_total{core_id=&quot;123456789abcdef12345678&quot;} 83200<br># HELP device uptime.<br>#TYPE device_uptime gauge<br>device_uptime{core_id=&quot;123456789abcdef12345678&quot;} 721<br># HELP Exporter Milliseconds to produce Metrics.<br># TYPE exporter_produce_time gauge<br>exporter_produce_time{type=&quot;devices&quot;} 1329<br># HELP Exporter Milliseconds to consume Metrics.<br># TYPE exporter_consume_time gauge<br>exporter_consume_time 1330</pre><p>Then, it’s straightforward to configure a Prometheus server to scrape this endpoint periodically to amass time-series data that — most people — will onsend to Grafana for charting.</p><h4>Deploy</h4><p>The following you should get you a running deployment:</p><pre>export TOKEN=[[YOUR-PARTICLE-TOKEN]]<br>WORKDIR=[[YOUR-WORKING-DIRECTORY]]</pre><pre>cd ${WORKDIR}</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2561b25912d39497c227813370046014/href">https://medium.com/media/2561b25912d39497c227813370046014/href</a></iframe><p>Put both files in ${WORKDIR}, the ${TOKEN} variable must be exported to be accessible by Docker Compose, and then:</p><pre>docker-compose up</pre><p>You can check that both services are running:</p><pre>docker-compose ps<br><br>Name                  Command                 Ports<br>--------------------------------------------------------------------<br>particle-exporter_1   /particle-exporter...   0.0.0.0:9999-&gt;9999/tcp<br>prometheus_1          /bin/prometheus...      0.0.0.0:9090-&gt;9090/tcp</pre><blockquote><strong>NB</strong> I will now fix the absence of the healthcheck on the Exporter :-)</blockquote><p>If both services come up, you can browse them using:</p><pre>google-chrome <a href="http://localhost:9090/">http://localhost:9090</a>         # Prometheus<br>google-chrome <a href="http://localhost:9999/metrics">http://localhost:9999/metrics</a> # Exporter</pre><p>The Prometheus Target should be similar to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OxWWXqcCOSbHI_9jVtbJCQ.png" /></figure><p>And you may then browse any of the metrics listed by :9999/metrics including exporter_produce_time and exporter_consume_time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_lT9lhq15ljzlscjd9c6Q.png" /><figcaption>Prometheus Exporter for Particle</figcaption></figure><p>There’s a bug in these currently as they both report the same value. This is highly unlikely.</p><h4>Develop</h4><p>If you wish to review|contribute to the code:</p><pre>git clone <a href="/service/mailto:git@github.com">git@github.com</a>:DazWilkin/particle-exporter.git<br># or go get github.com/DazWilkin/particle-exporter</pre><p>The code uses Golang’s new Modules, so:</p><pre>GO111MODULES=on \<br>go run github.com/DazWilkin/particle-exporter \<br>--token=${TOKEN}</pre><h4>Conclusion</h4><p>This is <strong>not</strong> production ready code! Please do not use it in production.</p><p>It does hopefully show how straightforward it is to gather metrics from arbitrary services using Prometheus. If — like me — you’re a satisfied customer of Particle but would like to surface Particle device measurements using Prometheus, hopefully this helps.</p><p>That’s all!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6203bf70abf5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>