<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by John Doak on Medium]]></title>
        <description><![CDATA[Stories by John Doak on Medium]]></description>
        <link>https://medium.com/@johnsiilver?source=rss-ebed66dcde0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cqOftU-Ot5NFtEtM9M8KAQ.png</url>
            <title>Stories by John Doak on Medium</title>
            <link>https://medium.com/@johnsiilver?source=rss-ebed66dcde0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 10:14:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@johnsiilver/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Stringer 2: The Revenge]]></title>
            <link>https://medium.com/@johnsiilver/stringer-2-the-revenge-9363fb238b8d?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9363fb238b8d</guid>
            <category><![CDATA[golang-development]]></category>
            <category><![CDATA[design-pattern-in-golang]]></category>
            <category><![CDATA[golang-tutorial]]></category>
            <category><![CDATA[golang-tools]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Tue, 10 Mar 2026 16:32:13 GMT</pubDate>
            <atom:updated>2026-03-10T16:32:13.607Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*POoe5OLRq4HFd9kXZM99nQ.png" /></figure><p><strong>One of the most useful tools that the Go team has provided</strong> <strong>is the </strong><a href="https://pkg.go.dev/golang.org/x/tools/cmd/stringer"><strong>stringer</strong></a><strong> tool</strong>.</p><p>It provides generation of a String() string method on integer enumerators that output the constant name as a string. The is paired with the the go generate tool to allow automatic generation.</p><pre>//go:generate go tool stringer -type Fruit -linecomment<br><br>type Fruit uint8<br><br>const (<br>  Unknown Fruit = 0<br>  Apple Fruit = 1 // apple<br>  Orange Fruit = 2 // orange<br>)</pre><p>Running go generatecauses a new file to be created in the same package, but as a separate file:</p><pre>// Code generated by &quot;stringer -type Fruit -linecomment&quot;; DO NOT EDIT.<br><br>package inventory<br><br>import &quot;strconv&quot;<br><br>func _() {<br>        // An &quot;invalid array index&quot; compiler error signifies that the constant values have changed.<br>        // Re-run the stringer command to generate them again.<br>        var x [1]struct{}<br>        _ = x[Unknown-0]<br>        _ = x[Apple-1]<br>        _ = x[Orange-2]<br>}<br><br>const _Fruit_name = &quot;Unknownappleorange&quot;<br><br>var _Fruit_index = [...]uint8{0, 7, 12, 18}<br><br>func (i Fruit) String() string {<br>        if i &gt;= Fruit(len(_Fruit_index)-1) {<br>                return &quot;Fruit(&quot; + strconv.FormatInt(int64(i), 10) + &quot;)&quot;<br>        }<br>        return _Fruit_name[_Fruit_index[i]:_Fruit_index[i+1]]<br>}</pre><p>If you have not used go generate or stringer before, here is the official <a href="https://go.dev/blog/generate">blog</a> that explains how these work. To make this scale across your team, use the Go tool <a href="https://www.bytesizego.com/blog/go-124-tool-directive">directive</a>.</p><p>This eliminates code that used string enumerators to simply make human readable output messages or the hassle of maintaing these mappings by hand.</p><h3>Why not use string enumerators?</h3><pre>type Fruit string<br><br>const (<br>  Unknown Fruit = &quot;unknown&quot;<br>  Apple Fruit = &quot;apple&quot;<br>  Orange Fruit = &quot;orange&quot;<br>)</pre><p><strong>Speed and size</strong>, plain and simple.</p><p>Most enumerators can be a uint8, meaning a single byte is required to compare values. A string enumerator is rarely less that 4 bytes .</p><p>When working with non-trivial amounts of data, this can prove to be significant. When you want to get into deep optimizations, its hard to bin pack or do other tricks with string data.</p><p>A string is also allocated, where as an integer type is generally not.</p><p>A string also limits comparisons to equality. With numerics you can group items together, such as success codes being between 10–20 and failure codes that are &gt;20. Instead of checking failure values in a loop, you can do a simple number check.</p><p>string is a necessary data type, but <a href="https://wiki.c2.com/?StringlyTyped">stringly</a> typing data is almost always a mistake when we want to optimize. In the era of large data sets, AI training and cloud compute, the amount of money that gets wasted due to simply poor data choices is staggering.</p><p>If you are OpenAI with an unlimited budget, maybe it doesn’t matter. If you are the scrappy startup, this kind of thing might cause your AWS bill to burn through your funds. Its not just CPU time, memory costs, machine costs…. but network cost can be mulitples of storage costs. Smaller and faster can be better when you are processing lots of data.</p><h3>Things I wish Stringer did</h3><p>stringer is great. But after using it for years, I’ve found it has a few things missing that I almost always have to maintain by hand.</p><p>1st</p><p><strong>Value validation</strong>. I want something that tells me if the value is valid. Often I will receive a value that was encoded over the wire. In those cases after the conversion to my type, I want to know that the value is valid. A value may not be valid for many reasons, like because a client or server supports newer values that the other does not, having not been upgraded yet.</p><p>2nd</p><p><strong>Recognizing invalid values</strong>. Certain values that are valid values in one sense (I defined the value) are not actually valid. When I am designing enumerators, the 0 value which is also Go’s zero value for integer types, stands for Unknown. This allows me always know that code did not set a field correctly and I have a bug. So in almost every implementation it is an invalid value if I receive 0. So I want to be able to recognize those values as invalid when I do a check vs. it being considered valid because I defined it.</p><p>3rd</p><p><strong>Reverse </strong><strong>string conversion</strong>. The Go protocol buffer compiler supports an enum type. When it generates the Go values, it does so in a similar way to stringer. But it creates reverse maps that allow converting a string into the enum type.</p><p>I want a more powerful version of that which allows a conversion either with an exact match or a case insensitive match. There are many times that I must deal with bad JSON data formats and I need to have a reverse lookup to get the enumerator from stringly format into strongly typed. Some data is hand typed by a person and case insensitive allows me to be more forgiving.</p><p>4th</p><p><strong>JSON Marshal/Unmarshal strings. </strong>Because these values might be stored as a stringly type in JSON, it would be nice to generate the Marshal/Unmarshal methods that do the conversions.</p><p>In the past I’ve defined and manually maintained this type of code. It was past time to just fork the stringer code and add these for myself.</p><h3>Extended Stringer tool</h3><p>You can install this for a repo with:</p><p>go get -tool github.com/johnsiilver/stringer</p><p>Using this tool vs the standard stringer simply becomes:</p><p>//go:generate go tool stringer -type=EnumType -linecomment -valid -invalid=0 -reverse</p><p>It supports all the standard stringer flags, with a few additions.</p><p>— valid will generate a Valid() method on the type that will validate its current value. Useful if you take in a value that is valid for the underlying type, but not defined.</p><p>— invalid takes a comma deliminated list of values or <strong>&lt;</strong>{{value}}, <strong>&gt;</strong>{{value}}, <strong>≤</strong>{{value}}, <strong>≥</strong>{{value}} that causes those values to be marked invalid when considered by Valid() . So this: -invalid=0,&gt;=100 would mark a value of 0 or ≥ 100 as invalid values. This is only useful if those values are defined for some reason.</p><p>— reverse generates a Reverse{{type}}() function that takes a string and converts it back to the type and lets you know if the type is valid. If --valid or --invalid are used, it will also check those instead of just the string representation.</p><p>— marshal generates MarshalJSON/UmarshalJSON methods. This allows conversion from JSON that is storing this value as a string into our enumeration type. Without this, the value would need to be a numeric representation. This uses case sensitive matching. This will also write invalid values out to JSON, as that is sometimes desired when passing data around that you might not need to understand the meaning of but need to pass on. You can of course do a Valid() check before doing the marshal call if you want to disallow that.</p><p>— marshalinsensitive is the same as — marshal except that we do case insensitive matches.</p><p>-replaceallows you to replace words or characters in the string representation, like a — with a _ before Reverse{{type}}() does its thing. This affects how unmarshalling works as well, as it uses Reverse{{type}}().</p><p>These options cover the majority of my uses cases. If I need to go beyond this I can always revert to implementing the functionality by hand for a specific need.</p><p>If you need this kind of functionality, consider giving this a whirl.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9363fb238b8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto-Cleanup Weak Pointer Caches]]></title>
            <link>https://medium.com/@johnsiilver/auto-cleanup-weak-pointer-caches-1d65d7e243e7?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d65d7e243e7</guid>
            <category><![CDATA[golang-tutorial]]></category>
            <category><![CDATA[caching]]></category>
            <category><![CDATA[design-pattern-in-golang]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[golang-development]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Wed, 05 Nov 2025 20:42:44 GMT</pubDate>
            <atom:updated>2025-11-05T20:42:44.624Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZQiCNn8fieSJnIq0_rhEMg.png" /></figure><h3><strong>Introduction</strong></h3><p>Would you like a cache that automatically cleans up data when it is not in use?</p><p>Go recently added weak pointers to the standard library. If you have never used a weak pointer, these are pointers whose value can be collected by the garbage collector if only the weak pointer is pointing at it. A normal pointer keeps the value alive as long as the pointer is alive.</p><p>This means that the value of a weak pointer at any time can become nil. It becomes nil whenever the garbage collector runs and sees the weak pointer being the only value pointer.</p><p>This can be perfect for certain caching situations. Normal caches require you to deal with deleting values when its been in the cache too long, when there isn’t enough space or when a TTL expires. But this requires some type of scan or queue system that deals with cleanup. You can end up with values that aren’t in use but are still hanging around taking up memory.</p><p>Weak pointers take care of this for you, cleaning up the value when it is no longer needed. This is useful when keeping the lowest memory footprint is your primary concern.</p><p>Here is a a very simple weak pointer cache using a standard map and sync.RWMutex:</p><pre>// Cache is a weakPointer cache. It provides a cache of objects that have <br>// weak pointers to them. This cache is thread-safe.<br>type Cache[K comparable, V any] struct {<br>  mu sync.RWMutex<br>  m map[K]weak.Pointer[V]<br>}<br><br>type opts struct{}<br><br>// Option is an option for New(). No options exist, this is future proofing.<br>type Option func(opts) (opts, error)<br><br>// New creates a new Cache with the given options.<br>func New[K comparable, V any](ctx context.Context, options ...Option) (*Cache[K, V], error) {<br> o := opts{}<br> for _, option := range options {<br>  var err error<br>  o, err = option(o)<br>  if err != nil {<br>   return nil, err<br>  }<br> }<br><br> return &amp;Cache[K, V]{m: map[K]weak.Pointer[V]{}}, nil<br>}<br><br>// Len returns the number of entries in the cache.<br>func (c *Cache[K, V]) Len() int {<br>  c.mu.RLock()<br>  defer c.mu.RUnlock()<br>  return len(c.m)<br>}<br><br>// Get retrieves a value at Key.<br>func (c *Cache[K, V]) Get(k K) (value *V, ok bool) {<br>  if c == nil || c.m == nil {<br>   return nil, false<br>  }<br>  c.mu.RLock()<br>  defer c.mu.RUnlock()<br><br>  wp, ok := c.m[k]<br>  if !ok {<br>   return nil, false<br>  }<br><br>  v := wp.Value()<br>  // We check if the value actually exists, as a weak pointer can become nil<br>  // at any point, which indicates it is deleted.<br>  if v == nil {<br>   return nil, false<br>  }<br><br> return v, true<br>}<br><br>// Set stores a key with value. If the key already exists, this will <br>// overwrite it. In that case we return the previous value and true. <br>// If v is nil, we delete the key and return the previous value.<br>func (c *Cache[K, V]) Set(k K, v *V) (Prev *V, ok bool) {<br>  if c == nil || c.m == nil {<br>    return<br>  }<br> <br>  if v == nil {<br>    return c.Del(k)<br>  }<br><br>  c.mu.Lock()<br>  defer c.mu.Unlock()<br><br>  wp := weak.Make(v)<br><br>  oldWP, ok := c.m[k]<br>  if !ok {<br>    c.m[k] = wp<br>    return nil, false<br>  }<br><br>  oldV := oldWP.Value()<br>  if oldV == nil {<br>    c.m[k] = wp<br>    return nil, false<br>  }<br><br>  c.m[k] = wp<br>  return oldV, ok<br>}<br><br>// Del removes a value from the cache.<br>func (c *Cache[K, V]) Del(k K) (prev *V, ok bool) {<br>  if c == nil || c.m == nil {<br>    return<br>  }<br><br>  c.mu.Lock()<br>  defer c.mu.Unlock()  <br>  <br>  wp, ok := c.m[k]<br>  delete(c.m, k)<br><br>  if !ok {<br>    return nil, false<br>  }<br>  v := wp.Value()<br>  if v == nil {<br>    return nil, false<br>  }<br>  return v, ok<br>}</pre><p>Here is this code in the <a href="https://go.dev/play/p/xaCUlZNkp87">playground</a>.</p><h3><strong>The Problem with this Simple Weak Pointer Cache</strong></h3><p>There are a few issues with just using a weak pointer to clear out memory.</p><ul><li>Maps, even with sync.RWMutex are not super efficient for jobs with a lot of concurrent read/write.</li><li>Maps and sync.Map do not shrink if you delete keys. The values are deleted but the map size itself does not get smaller.</li><li>This version doesn’t delete keys automatically. We would need to add some scanning and a delete loop that would need to hold a write lock. This would be expensive, similar to the garbage collection cycle where nothing can be done while this is happening. If the cache is large, this can be costly.</li><li>Might be nice if it supported a minimum TTL to avoid adding a value and it being collected just because the garbage collector ran right after.</li></ul><h3><strong>Sharded Maps for Concurrency and Deletes</strong></h3><p>Sharded maps have multiple map structures underneath and spreads values over the shards. These shards each have their own sync.RWMutex which allows better concurrent reads and writes if under heavy writes.</p><p>This version will use a sharded map:</p><pre>package cleancache<br><br>import (<br> &quot;context&quot;<br> &quot;runtime&quot;<br> &quot;weak&quot;<br><br> &quot;github.com/gostdlib/base/concurrency/sync&quot;<br>)<br><br>type Cache[K comparable, V any] struct {<br> shardedMap *sync.ShardedMap[K, weak.Pointer[V]]<br>}<br><br>type opts struct{}<br><br>type Option func(opts) (opts, error)<br><br>func New[K comparable, V any](ctx context.Context, options ...Option) (*Cache[K, V], error) {<br> o := opts{}<br> for _, option := range options {<br>  var err error<br>  o, err = option(o)<br>  if err != nil {<br>   return nil, err<br>  }<br> }<br><br> c := &amp;Cache[K, V]{shardedMap: &amp;sync.ShardedMap[K, weak.Pointer[V]]{}}<br><br> return c, nil<br>}<br><br>func (c *Cache[K, V]) Len() int {<br> return c.shardedMap.Len()<br>}<br><br>func (c *Cache[K, V]) Get(k K) (value *V, ok bool) {<br> if c == nil || c.shardedMap == nil {<br>  return nil, false<br> }<br><br> wp, ok := c.shardedMap.Get(k)<br> if !ok {<br>  return nil, false<br> }<br><br> v := wp.Value()<br> if v == nil {<br>  return nil, false<br> }<br><br> return v, true<br>}<br><br>func (c *Cache[K, V]) Set(k K, v *V) (Prev *V, ok bool) {<br> if c == nil || c.shardedMap == nil {<br>  return<br> }<br> if v == nil {<br>  return c.Del(k)<br> }<br><br> wp := weak.Make(v)<br><br> oldWP, ok := c.shardedMap.Set(k, wp)<br> if !ok {<br>  return nil, false<br> }<br> oldV := oldWP.Value()<br> if oldV == nil {<br>  return nil, false<br> }<br><br> return oldV, ok<br>}<br><br>func (c *Cache[K, V]) Del(k K) (prev *V, ok bool) {<br> if c == nil || c.shardedMap == nil {<br>  return<br> }<br><br> wp, ok := c.shardedMap.Del(k)<br> if !ok {<br>  return nil, false<br> }<br> v := wp.Value()<br> if v == nil {<br>  return nil, false<br> }<br> return v, ok<br>}</pre><p>This sharded map implementation is based on <a href="https://github.com/tidwall">Josh Baker’s</a> <a href="https://github.com/tidwall/shardmap">sharded map.</a> The key difference is it uses generics (generics didn’t exist when Josh’s wrote his). It uses its own map implementation which can shrink with deletes.</p><p>But we still need a way to find deleted values and remove them from the map. Let’s use another recent Go feature: runtime.AddCleanup().</p><h3><strong>runtime.AddCleanup() for the Win</strong></h3><p>runtime.AddCleanup() is a method that allows you to cause a function to run when a value goes out of scope.</p><p>Let’s modify our Set() call to use runtime.AddCleanup() to automatically cleanup our cache maps:</p><pre>// Set stores a key with value. If the key already exists, this will overwrite it. In that case<br>// we return the previous value and true. If v is nil, we delete the key and return the previous value.<br>func (c *Cache[K, V]) Set(k K, v *V) (Prev *V, ok bool) {<br> if c == nil || c.shardedMap == nil {<br>  return<br> }<br> if v == nil {<br>  return c.Del(k)<br> }<br><br> wp := weak.Make(v)<br><br> runtime.AddCleanup[V, K]( // &lt;- clean the value when it becomes nil<br>  v,<br>  func(k K) {<br>   c.Del(k)<br>  },<br>  k,<br> )<br><br> oldWP, ok := c.shardedMap.Set(k, wp)<br> if !ok {<br>  return nil, false<br> }<br> oldV := oldWP.Value()<br> if oldV == nil {<br>  return nil, false<br> }<br><br> return oldV, ok<br>}</pre><p>Now, whenever our value V is going to be cleaned up by the garbage collector, a function that calls Del() runs.</p><p>Now, it is <em>possible </em>the value will be re-Set() before this function is called and we will delete the new key/value from the cache . This will not cause any issues with code holding the value and is likely a rare case, but it is sub-optimal.</p><p>I provide a version with a customized sharded map specifically for weak pointer values that eliminates this issue. See the links below.</p><p>There is some runtime costs involved with automatic cleanup and you are amoratizing this over each Get() and Set() call.</p><h3><strong>An Alternate Timer Based Approach</strong></h3><p>Instead of using runtime.AddCleanup(), we can simply loop over each sharded map for a delete at intervals. This give some higher latency on calls for maps that are locked, but sharded maps help with this by spreading the cost over lots of maps instead of a single map.</p><p>To make this efficient I used the custom weak pointer sharded map and added a CleanShards method:</p><pre>// CleanShards removes all entries with nil values from the map.<br>func (m *Map[K, V]) CleanShards() {<br> m.initDo()<br> for shard := range m.maps {<br>  m.mus[shard].Lock()<br>  for k, v := range m.maps[shard].All() {<br>   if v.Value() == nil {<br>    if _, deleted := m.maps[shard].Delete(k); deleted {<br>     m.count.Add(-1)<br>    }<br>   }<br>  }<br>  m.mus[shard].Unlock()<br> }<br>}</pre><p>This keeps only a single shard locked at any time, minimizing the overhead. But it is not as efficient in memory utilzation since it requires an interval to occur.</p><p>Again, links to the code are below.</p><h3>Adding a minimum TTL</h3><p>Maybe you want a minimum TTL for a cache entry. This can be accomplished by having a map type that holds a strong pointer for the value until the TTL interval expires. Then the value will expire only when all other strong references expire.</p><p>This gives the best of both worlds by allowing the values to stay for some minimum time and then cleanup when nothing is using it, vs just a TTL which doesn’t handle if the values are still in use. However it adds a slight cost by needing to lock multiple maps during Set() and Del() calls.</p><p>Links below.</p><h3><strong>Using SingleFlight</strong></h3><p>Sometimes you might have several Goroutines trying to get the same value. We can use the <a href="https://pkg.go.dev/golang.org/x/sync/singleflight">singleflight</a> package to avoid multiple lookups.</p><p>Now for in-memory caches, vs caches distributed across machines or backed by disk, this will have zero benefits. The extra locks are going to be more costly. I’ve added it here to introduce the concept of singleflight to those who haven’t seen it before. For things like database lookups on cache misses or calling to a distributed cache service, this can be a lifesaver.</p><p><strong>Note</strong>: This uses a custom singleflight package that is generic instead of the official one that uses any. Again, the original was written before generics.</p><p>Links below.</p><h3><strong>Are these worth it over LRU or other Go cache packages?</strong></h3><p>This is going to depend on your use case. Its hard to know which cache will work out better without testing on a product’s actual work load. Micro-benchmarks tend to be misleading except in apples to apple benchmarks for caches.</p><p>Set() calls become expensive. Its hard to tell exactly how much, because benchmarks require runtime.KeepAlive() calls for some things. But probably 2–3x. Get() calls are faster in some cases, slower in others. Warming up memory and the garbage collector before benchmarks has a positive effect on our cache, negative on others.</p><p>If memory is the greatest concern, it is likely that this will outperform other caches. Runtime costs are going to make per call latency more expensive. An LRU is a glorfied queue, so by comparison, it is much simplier in terms of CPU overhead but worse in the memory management department.</p><p>Also, if you are looking for a great distributed cache (with free and paid hosted versions), checkout <a href="https://pogocache.com/">pogocache</a> by Josh Baker (whose code the sharded map is based on).</p><p>Full disclosure: I know Josh through a few message exchanges and have no business relationship. But I do greatly admire his code.</p><h3><strong>Sample packages</strong></h3><p><a href="https://github.com/johnsiilver/cleancache">Github Link</a></p><ul><li><strong>cleancache/</strong>: AddCleanup + Weak Pointers + Custom Sharded Map + Supports minimum TTL + SingleFlight capability</li><li><strong>cleancache/original/</strong>: AddCleanup and Weak Pointers + Sharded Map</li><li><strong>cleancache/timer/</strong>: WeakPointers + Custom Sharded Map + Cleanup loop</li></ul><p><strong>Note:</strong> These set of packages are provided as examples for this article and not as a well groomed, highly tested (in a production environment) set of packages with instrumentation and other advanced features. Feel free to copy them, modify them, etc for your own uses.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d65d7e243e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fsets, a useful pattern for testing]]></title>
            <link>https://medium.com/@johnsiilver/fsets-a-useful-pattern-for-testing-50d2f46281e1?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/50d2f46281e1</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang-development]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Mon, 30 Jun 2025 23:13:10 GMT</pubDate>
            <atom:updated>2025-07-02T12:13:51.616Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the things that is always a struggle is testing call chains.</p><p>I’ve written about this before in my statemachine article (<a href="https://medium.com/@johnsiilver/go-state-machine-patterns-3b667f345b5e">go-state-machine-patterns</a>)which details how statemachines can reduce your testing difficulty. But statemachines aren’t a hammer you should hit every nail with. It is good when you have a lot of states and those states are non-deterministic when called. It adds a lot of overhead when there are few states and the functions to be called are always the same in the same order (deterministic).</p><p>So what do I do when I have a call chain A() → B() → C() and when testing A() I don’t want to fake out things for B() and C() because they are complicated?</p><p>There are several options, but I’ve come to like a pattern I call the Fsets pattern.</p><p>It acts like a statemachine, except it calls functions that are in a deterministic order. Like the statemachine, it passes an object, in this case called a StateObjectalong the call path.</p><p><em>(The </em><a href="https://pkg.go.dev/github.com/johnsiilver/fsets"><em>godoc</em></a><em> is here for reference.)</em></p><p>Here is a simple example:</p><pre>fset := fsets.Fset[MyDataType]{}<br>fset.Adds(<br>  // C stands for Call<br>  // F stands for function<br>  // B stands for backoff<br>  // O stands for option<br>  // Yeah, they are super short in the veinf of Timer.C, simply because if<br>  // you write these a lot, it becomes obvious. If not, its annoying to look<br>  // up what it means. Tradeoffs.<br>  fsets.C[MyDataType]{F: retriableFunc, B: myBackoff},<br>  fsets.C[MyDataType]{F: nonRetriableFunc},<br>)<br><br>so := fsets.StateObject[MyDataType]{Data: myData}<br>so = fset.Run(ctx, so)<br>if so.Err() != nil {<br>  log.Error(&quot;Error running function set&quot;, &quot;error&quot;, err)<br>}</pre><p>Once you’ve created an Fset you can use it multiple times. fsets support exponential backoff if you provide the backoff object with the call you want to make.</p><p>Want the functions to be methods with access to an object’s state, no problem:</p><pre> fset := fsets.Fset[MyDataType]{}<br> fset.Adds(<br>  fsets.C[MyDataType]{F: myStruct.myMethod1, B: myBackoff},<br>  fsets.C[MyDataType]{F: myStruct.myMethod2},<br> )<br><br> so := fsets.StateObject[MyDataType]{Data: myData}<br> so = fset.Run(ctx, so)</pre><p>You can assign an fset in a New() constructor to an object attribute and remove the cost of creating these on the fly. Just be sure to use attributes on the object and not set them unless you have some syncronization. Use the StateObject to mutate data between calls for a particular call.</p><h3>Nothing in this world is free</h3><p>Like a statemachine, this isn’t free, it adds about 30–40ns per call as the runtime has to execute the function call. This is usually a wash the moment some I/O operation happens, but in tight loops where your counting cache lines or something, this can be painful, especially if you have a long chain.</p><p>This also eliminates inlining of function calls by the compiler, so this isn’t a silver bullet. However, not everyone is writing Redis.</p><p>Testing becomes easier because A() does not call B(). B() is simply executed after A(). <strong><em>That difference means everything.</em></strong></p><p>Now you can test A() without faking anything needed by B(). If B() needs some data that A() collects, you simply set it in your data object that is contained within the StateObject.</p><p>In addition this package allows you to easily implement parallel operations that are bounded by some parallelism constraint (as long as your functions are thread-safe, you could always call .Run() on your object infinite times, but usually it is better to be bounded by some limit).</p><p>Here is a concurrent version using pooled goroutines equal to GOMAXPROCS x Number of Functions:</p><pre> type RPCServer struct {<br>  fset fsets.Fset[RPCRequest]<br>  in  fsets.PromiseQueue[RPCRequest]<br> }<br><br> func New(ctx context.Context) *RPCServer {<br>  // Create a new Fset for the RPC server.<br>  fset := fsets.Fset[RPCRequest]{}<br><br>  r := &amp;RPCServer{}<br>  // Add the functions to the Fset. These can be methods or functions.<br>  fset.Adds(<br>   fsets.C[RPCRequest]{F: r.handleAuth, B: myBackoff},<br>   fsets.C[RPCRequest]{F: r.handleData},<br>   fsets.C[RPCRequest]{F: r.handleResponse},<br>  )<br><br>  in, _ := fset.Concurrent(ctx, -1)<br><br>  r.fset = fset<br>  r.in = in<br>  return r<br> }<br><br> func (s *RPCServer) Serve(ctx context.Context, req RPCRequest) (RPCResponse, error) {<br>  // Create a new promise for the request.<br>  p := s.fset.Promise(req)<br><br>  if err := s.in.Send(ctx, p); err != nil {<br>   return RPCResponse{}, err // Context cancelled or other error.<br>  }<br><br>  // Wait for the response.<br>  resp, err := p.Get(ctx)<br>  if err != nil { // Only happens if Context is cancelled.<br>   return RPCResponse{}, err<br>  }<br><br>  return resp.V, resp.Err()<br> }</pre><p>It also supports pipelines:</p><pre>fset := fsets.Fset[Data]{}<br><br>// Add the functions to the Fset. These can be methods or functions.<br>fset.Adds(<br>  fsets.C[Data]{F: handleAuth, B: myBackoff},<br>  fsets.C[Data]{F: handleData},<br>  fsets.C[Data]{F: handleResponse},<br>)<br><br>// Create a channel to get the results on.<br>out := make(chan promises.Promise[StateObject[T], StateObject[T]], 1)<br>in, _ := fset.Concurrent(ctx, -1, WithPipeline[Data](out))<br><br>// Send data to the input channel.<br>go func() {<br>  defer close(in) // Close the input channel when done.<br>  for _, data := range inputData{<br>    p := fset.Promise(data)<br>    in &lt;- p<br>  }<br>}()<br><br>// Get data from the output channel.<br>for promise := range out {<br>  // Process the promise results.<br>  resp, err := p.Get(ctx)<br>  if err != nil { // Only happens if Context is cancelled.<br>    log.Error(&quot;Error getting promise result&quot;, &quot;error&quot;, err)<br>    continue<br>  }<br>  if resp.Err() != nil {<br>    log.Error(&quot;Error processing data&quot;, &quot;error&quot;, resp.Err())<br>    continue <br>  }<br><br>  log.Info(&quot;Processed data: &quot;, resp.V)<br>}</pre><h3>This next section is me going down a rabbit hole, you’ve been warned!</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*kblhMKHPFrXVX4K7.jpg" /></figure><h3>What if you want the nanoseconds back and still want to use fsets.</h3><p>You could just use the pattern without the Fset object to achieve this:</p><pre>type callData struct {<br>  args Args<br>  aResult aResult<br>  finalResult results<br>}<br><br>type MyStruct struct{}<br><br>func (m *MyStruct) Call(ctx context.Context, args Args) (results, error) {<br>  so := fsets.StateObject[callData]{Data: callData{args: args}}<br>  <br>  so, err := m.a(fsets.StateObject[callData]{Ctx: ctx, Data: callData{args: args}})<br>  if err != nil || so.GetStop() {<br>    return so.Data.finalResults, err<br>  }<br>  so, err = m.b(fsets.StateObject[callData]{Ctx: ctx, Data: callData{args: args}})<br>  if err != nil || so.GetStop() {<br>    return so.Data.finalResults, err<br>  }<br>  return so.finalResults, nil<br>}<br><br>func (m *MyStruct) a(so fsets.StateObject[callData]) (fsets.StateObject[callData], error) {<br>  // do stuff<br>  return so, nil<br>}<br><br>func (m *MyStruct) b(so fsets.StateObject[callData]) (fsets.StateObject[callData], error) {<br>  // do stuff<br>  return so, nil<br>}</pre><p>Now you can test a() and b() in isolation. Someone might point out that if you want to test Call() that you’d have to test everything, but I wouldn’t test Call(). If your a stickler for 100% coverage, then this pattern isn’t going to be good for you. I’m an 80% guy.</p><p>With this structure its easy to use the Fsets package for convenience with small call chains and then add a single method that just decomposes that call loop to gain back any lost time if performance becomes an issue. With this there is no support for the automatic backoff mechanism. Writing a backoff helper would go back to runtime overhead, so you would need to bake backoff directly into the calls that require that.</p><h3>I’m too lazy for my shirt, too lazy for my shirt, so lazy it hurts</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FP5mtclwloEQ%3Fstart%3D24%26feature%3Doembed%26start%3D24&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DP5mtclwloEQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FP5mtclwloEQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/387599a2d88bff6de30ebc1228406f9c/href">https://medium.com/media/387599a2d88bff6de30ebc1228406f9c/href</a></iframe><p>I’m afraid I’m too damn lazy for that. So I spent some time “vibe coding” (yeah, I know exactly what is going through your mind, so lets move on) to build a basic code generator that can compile my fset and then I can use a .Compile() method before I do Run(). This causes us to use a “compiled” (aka, a version that does a more standard call chain) version instead of the looped code. This also handles using exponential backoff if we set that option.</p><p>The generated code removes some of that overhead, though inlining is still a problem. I could probably just inline some of the code myself to help the compiler when that option exists, but I think I’ve taken this as far as I’m going to. I’d call this an experiment and its a little awkward to use, but does work.</p><p>If you have this structure:</p><pre>type RPCRequest struct {<br>  User string<br>  Data string<br>}<br><br>type RPCResponse struct {<br>  Data string<br>}<br><br>type RPCServer struct {<br>  serveFset *fsets.Fset[serveSO]<br>}<br><br>//go:generate fsetcodegen -var=serveFset -func=New<br><br>func New(ctx context.Context) *RPCServer {<br>  // Create a new Fset for the RPC server.<br>  serveFset := fsets.Fset[serveSO]{}<br>  <br>  r := &amp;RPCServer{}<br>  <br>  // Add the functions to the Fset. These can be methods or functions.<br>  serveFset.Adds(<br>    fsets.C[serveSO]{F: r.handleAuth},<br>    fsets.C[serveSO]{F: r.handleData},<br>  )<br>  <br>  r.serveFset = &amp;serveFset<br>  return r<br>}<br><br>type serveSO struct {<br>  req  RPCRequest<br>  resp RPCResponse<br>}<br><br>func (s *RPCServer) Serve(ctx context.Context, req RPCRequest) (RPCResponse, error) {<br>  so := s.serveFset.Run(ctx, fsets.StateObject[serveSO]{Data: serveSO{req: req}})<br>  return so.Data.resp, so.Err()<br>}<br><br>func (s *RPCServer) handleAuth(so fsets.StateObject[serveSO]) (fsets.StateObject[serveSO], error) {<br>  // Handle authentication logic here.<br>  // For example, check if the user is authenticated.<br>  if so.Data.req.User == &quot;&quot; {<br>    return so, errors.New(&quot;user not authenticated&quot;)<br>  }<br>  if so.Data.req.User != &quot;admin&quot; {<br>    return so, errors.New(&quot;unauthorized user&quot;)<br>  }<br>  return so, nil<br>}<br><br>func (s *RPCServer) handleData(so fsets.StateObject[serveSO]) (fsets.StateObject[serveSO], error) {<br>  // Handle data processing logic here.<br>  // For example, process the data and return a response.<br>  if so.Data.req.Data == &quot;&quot; {<br>    return so, errors.New(&quot;no data provided&quot;)<br>  }<br>  <br>  so.Data.resp.Data = &quot;Processed: &quot; + so.Data.req.Data<br>  return so, nil<br>}</pre><p><em>(Notice the </em><em>//go:generate line in the above code)</em></p><p><em>(Example can be found </em><a href="https://github.com/johnsiilver/fsets/tree/main/testing/fakeRPC"><em>here</em></a><em>)</em></p><p>We can now call go generate . on the package and it will generate a file with our compiled version.</p><p>Now we can add a single line to use the compiled version:</p><pre>fset.Adds(...) // ... stands for the functions we were adding<br>serveFset.Compiled(newServeFsetCompiled(r))</pre><p>When serveFset.Run() is called, it will use the compiled version. Your testing and everything else stay the same.</p><p>The compiler is experimental and quirky because:</p><ul><li>I vibe coded it because I hate writing AST stuff</li><li>No, I mean it, I hate writing AST stuff, its sooooo painfull………</li><li>The AST code has lots of corner cases and I am certainly not covering them all. There are notes on how you must structure an fset in the README</li><li>If you have multiple fsets, they all need to have unique names. I imagine that otherwise it will erase subsequent ones</li></ul><h3>Ok, I’m a liar</h3><p>So, I went to sleep after writing this article and woke up still not happy that I was loosing time. Depending on what I ran the benchmark on, I could lose up to the 30–40ns or 10–12ns when compiled.</p><p>Now honestly, I think what I’m doing is silly, because as soon as I touch I/O or do anything complicated its going to cost me more time that this. Everyone likes to quote “over optimization is the root of all evil”, which is always taken out of context, but here it probably applies.</p><p><a href="https://www.reddit.com/r/programming/comments/ppsn13/premature_optimization_is_the_root_of_all_evil/">Donald Knuth on what he meant</a> when he said that.</p><p>So I wrote an inliner that that will inline the code. Now if the compiler is experimental, the --inline flag is super experimental. I certainly haven’t flushed out any corner cases and have only used this for very basic implementations.</p><p>With that said it puts the standard call chain and an fset within less than 2ns of each other:</p><pre>BenchmarkInline-10              29917340                40.04 ns/op           16 B/op          1 allocs/op<br>BenchmarkCallChain-10           31291388                38.16 ns/op           16 B/op          1 allocs/op</pre><p>Now that is more like it (seriously, I should not celebrate doing something like this)!</p><p>If you use this, definitely inspect the generated code.</p><h3>Final Thoughts</h3><p>This package works, though I’d probably stay away from the compiled stuff. This is me getting too creative.</p><p>What is important about it is simply that you can simplify your testing using the methodology (regardless if you use the package) for complex call chains with complicated testing.</p><p>This package is not in its final form, so if you use it, expect possible changes. Feel free to fork it and slim it up for your use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50d2f46281e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go: Optional Arguments in the age of Generics]]></title>
            <link>https://medium.com/@johnsiilver/go-optional-arguments-in-the-age-of-generics-f8eba23103ad?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8eba23103ad</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 17:30:09 GMT</pubDate>
            <atom:updated>2024-10-21T17:30:09.116Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*9gePdzPuPf8PtYEIrozd4A.png" /></figure><p>Go’s standard method of providing optional arguments to functions has been a combination of a function type and variadic arguments.</p><p>You can see this pattern in many packages, sometimes with the With prefix, though this is not universally agreed on.</p><p>Here is an example of that form:</p><pre>type DB struct {<br>  endpoint string<br>}<br><br>type Option func(db *DB)<br><br>func WithEndpoint(endpoint string) Option {<br>  return func(db *DB) {<br>    db.endpoint = endpoint<br>  }<br>}<br><br>func New(options ...Option) *DB {<br>  db := &amp;DB{<br>    endpoint: &quot;endpoint.myservice.com&quot;,<br>  }<br>  for _, o := range options {<br>    o(db)<br>  }<br>  return db<br>}</pre><p>This simple pattern allows for having an easily readable constructor call like so:</p><pre>db := New(WithEndpoint(&quot;alternate.myservice.com&quot;))</pre><p>This has a clean look and is easy to decipher, usually without looking up struct field descriptions.</p><p>Unfortunately, the introduction of generics has made this less readable.</p><h3>Generic Ugliness</h3><p>Let’s look at what generics have done to this optional pattern.</p><pre>type MyGeneric[T any] struct {<br>  log *slog.Logger<br>}<br><br>type Option[T any] func(g *MyGeneric[T])<br><br>func WithLogger[T any](l *slog.Logger) Option[T] {<br>  return func(g *MyGeneric[T]) {<br>    g.log = l<br>  }<br>}<br><br>func New[T any](options ...Option[T]) *Generic[T] {<br>  g := &amp;MyGeneric{<br>    log: slog.Default(),<br>  }<br><br>  for _, o := range options {<br>    o(g)<br>  }<br>  return g<br>}</pre><p>What the use looks like:</p><pre>g := New[int](WithLogger[int](customLogger))</pre><p>All of the extra generics are now getting in the way of our clean optional syntax.</p><p>We can see this new pattern in libraries that share both generic types and optional arguments.</p><p>This is a <a href="https://sourcegraph.com/search?q=context:global+lang:Go+Option%5BT&amp;patternType=keyword&amp;sm=0">list</a> of packages via SourceGraph that use this new pattern (3.3K found before the search limit is hit).</p><p><em>Btw SourceGraph reminds me of Google’s internal code search system and is pretty awesome (I have no relationship with SourceGraph).</em></p><p>But this new pattern stinks. You at first forget that you need to pass the type parameter, then you end up with a much uglier version.</p><h3>Upgrading the ugly duckling</h3><p>However, we can get back to having our old options with just a little indirection.</p><pre>type MyGeneric[T any] struct {<br>  options options<br>}<br><br>type options struct {<br>  log *slog.Logger<br>}<br><br>func (o options) defaults() options {<br>  o.log = slog.Default()<br>  return o<br>}<br><br>type Option func(o options) options<br><br>func WithLogger(l *slog.Logger) Option {<br>  return func(o options) options {<br>    o.log = l<br>    return o<br>  }<br>}<br><br>func New[T any](options ...Option) *Generic[T] {<br>  opts := options{}<br>  g := &amp;MyGeneric{<br>    options: opts.defaults()<br>  }<br><br>  for _, o := range options {<br>    g.options = o(opts)<br>  }<br>  return g<br>}</pre><p>Now our options look like they did before:</p><pre>g := New[int](WithLogger(customLogger))</pre><p>One thing you may notice here is that options are always done via reassignment and not via pointers. Because we don’t have to modify a pointer type, we can get away with options that are stack allocated and not burden the garbage collector.</p><p><strong>What if you need an option with type parameters?</strong></p><p>Now, this is slightly harder. You can do it, but it does require a runtime check instead of a compile time check.</p><p>In the following example, we want to be able to pass a function with an option that is called when deleting a key/value pair. This function requires the key and value types that are defined in type parameters.</p><p>Aka:</p><pre><br>type Cache[K any, V any] struct {<br> ...<br> opts    opts<br> delFunc func(key K, value V, storage *Storage)<br>}</pre><p>First, let’s define our opts type:</p><pre>type opts struct {<br> a string<br><br> delFunc any<br>}</pre><p>Inside our opts type, we have defined a delFunc that is set to any. This is where we are going to store the function we get when a person sets the options.</p><p>Let’s setup the option type and our various option calls:</p><pre>type Option func(o opts) (opts, error)<br><br>func WithA(a string) Option {<br> return func(o opts) (opts, error) {<br>  if a == &quot;&quot; {<br>   return o, fmt.Errorf(&quot;WithA must not be an empty string&quot;)<br>  }<br>  o.a = a<br>  return o, nil<br> }<br>}<br><br>func WithDelFunc(f any) Option {<br> return func(o opts) (opts, error) {<br>  if f == nil {<br>   return o, fmt.Errorf(&quot;WithDelFunc must not have f == nil&quot;)<br>  }<br>  o.delFunc = f<br>  return o, nil<br> }<br>}</pre><p>Now, we are only doing a compile check on the f parameter here, because we don’t have access to the type parameters of the generics. Don’t worry, we will check it, but it will be a runtime check and not a compile check.</p><p>Let’s create our New constructor:</p><pre><br>func New[K any, V any](storage *Storage[K, V], options ...Option) (*Cache[K, V], error) {<br>  // These simply hold the zero values of our K and V parameters.<br>  var (<br>    zeroK K<br>    zeroV V<br>  )<br><br>  b := &amp;Cache[K, V]{<br>    ...<br>    opts: opts{<br>     a: &quot;default&quot;,<br>    },<br>    delFunc: defaultDel[K, V], // For this example, see full code link below.<br>  }<br><br>  var err error<br>  for _, o := range options {<br>    b.opts, err = o(b.opts)<br>    if err != nil {<br>     return nil, err<br>    }<br>  }<br>  if b.opts.delFunc != nil {<br>    // Here is our runtime check.<br>    f, ok := b.opts.delFunc.(func(K, V))<br>    if !ok {<br>     return nil, fmt.Errorf(&quot;WithDelFunc was given a func %T, want func[K %T, V %T](K, V)&quot;, b.opts.delFunc, zeroK, zeroV)<br>    }<br>    // Assign to our type specific variable.<br>    b.delFunc = f<br>  }<br>  return b, nil<br>}</pre><p>Here we do our standard options work, with a twist. We do a type check on the opts.delFunc to make sure it is the right type. If it isn’t, we return an error. If it is the right type, we assign it to our typed variable stored in Cache.delFunc .</p><p>The only negative to this is really the runtime error possibility. This is a negligible cost that is similar to checks done on other parameters passed to a function call.</p><p>This means that you will want to include a proper test for the option you are passing to make sure it will pass:</p><pre>f, ok := b.opts.delFunc.(func(K, *Storage[K, V]))<br>if !ok {<br>  return nil, fmt.Errorf(&quot;WithDelFunc was given a func %T, want func[K %T, V %T](K, V)&quot;, b.opts.delFunc, zeroK, zeroV)<br>}</pre><p>Here is a longer <a href="https://go.dev/play/p/Jw7x35KIXC6">working version</a> with everything defined.</p><h3>Conclusion</h3><p>While doing options in the manner of Option[T any] can be done, the above provides a nicer implementation for your users.</p><p>It avoids the cost of WithOption[string, int](blah), WithOtherOption[string, int](someOtherOption), ... and replaces it with the old pattern WithOption(blah), WithOption(someOtherOption) .</p><p>The only cost is a possible runtime check in certain cases, which isn’t much different than checks against passed arguments.</p><p>If you like this article, you might want to pair it with: <a href="https://medium.com/dev-genius/go-call-option-that-can-be-used-with-multiple-methods-6c81734f3dbe">Call Options That Can Be Used with Multiple Methods</a> (not for the feint of heart).</p><h3>Shameless Plug</h3><p>If you like articles like this and want to know how to leverage Go for DevOps, checkout my book:</p><p><a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894?source=post_page-----3b667f345b5e--------------------------------">Go for DevOps: Learn how to use the Go language to automate servers, the cloud, Kubernetes, GitHub, Packer, and Terraform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8eba23103ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go: Faking Method Calls Within a Struct]]></title>
            <link>https://medium.com/@johnsiilver/go-faking-method-calls-within-a-struct-2bf3e5af9e29?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bf3e5af9e29</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[golang-tutorial]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Fri, 26 Jul 2024 17:42:30 GMT</pubDate>
            <atom:updated>2024-07-26T17:42:30.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A gopher wearing a disguise." src="https://cdn-images-1.medium.com/max/1024/1*PXdlrQfM2GLYogljJcVVwQ.jpeg" /></figure><p>This is a follow up on an <a href="https://medium.com/@johnsiilver/go-state-machine-patterns-3b667f345b5e">article</a> I wrote on using functional State Machines to ease testing requirements.</p><p>In that article, we discussed how to use states that return the next function to break up complicated call chains. This allows us to ease testing requirements by only testing what is required for each state instead of a chain of function calls.</p><p>But what happens if you have a helper method on a struct that is called by other methods? You don’t want to test its logic over again if the logic is complicated.</p><p>You could move the logic into another struct and then create an interface on your current struct to hold it. Then you could pass either a production version or a test version. However, we often overuse interface and <strong>overlook that Go functions are first-class objects.</strong></p><p>interface also tends to be sub-optimal if you need to change internal state on the parent struct when calling your fake.</p><p>And while the state machine we talked about in the previous article helps with these call chain problems, it helps when you are making sequential calls for a long list of methods. State functions are more rigid than standard function calls and the state machine pattern is not a catch all solution.</p><p>Below I will present a method, for your amusement, that can help with these types of problems.</p><h3><strong>Methodology</strong></h3><p>In cases like these, you can attach a field with the same function signature as the helper function.</p><p>I’ve played around with several different methods to achieve testing the a method and not every sub-method. This is what I’ve settled on:</p><pre>type MyStruct struct {<br>  // You can inject test code here<br>  fakeRun func(ctx context.Context, arg string) error<br>}<br><br>func (*m *MyStruct) PublicFuncThatCallsRunSomething(ctx contex.Context, arg string) error{<br>  ... // Stuff you want to do before calling the helper.<br>  result := m.runSomething(ctx, arg)<br>  ... // The rest of the code.<br>}<br><br>func (m *MyStruct) runSomething(ctx context.Context, arg string) error {<br>  if m.fakeRun != nil { <br>    return m.fakeRun(ctx, arg) // test code gets run here<br>  }<br>  ... // regular code<br>}</pre><p>This makes it easy to see what is going on. If we set fakeRun to a value, when runSomething() is executed, it uses our fake. Now we can easily do fakes in our tests and avoid retesting runSomething() (a helper function we can fully test on its own) when testing PublicFuncThatCallsRunSomething(). Now we can just concentrate on dealing with how to handle the result in our code.</p><h3><strong>What if I need to mutate internal state?</strong></h3><p>Sometimes what you are faking needs to mutate internal state and the example of above doesn’t allow that.</p><p>In that example, we copied the exact pattern of the function we were going to call. But we don’t have to do this. Remember the difference between a function that operates on an object and a method is only how we pass our type to the call.</p><p>If we needed to mutate MyStruct above, we could have made our fake call look like:</p><pre>type MyStruct struct {<br>  ...<br><br>  // Notice that we now pass MyStruct here.<br>  fakeRun func(m *MyStruct, ctx context.Context, arg string) error<br>}<br><br>func (*m *MyStruct) PublicFuncThatCallsRunSomething(ctx contex.Context, arg string) error{<br>  ... // Stuff you want to do before calling the helper.<br>  result := m.runSomething(ctx, arg)<br>  ... // The rest of the code.<br>}<br><br>func (m *MyStruct) runSomething(ctx context.Context, arg string) error {<br>  if m.fakeRun != nil { <br>    return m.fakeRun(m, ctx, arg) // test code gets run here<br>  }<br>  ... // regular code<br>}</pre><p>Now you can easily mutate state when running the test.</p><h3>What if I need to check the call arguments or that a call happened?</h3><p>Sometimes due to a decision in the function you are testing, you need to validate that a call did or did not happen under certain conditions.</p><p>Additionally, sometimes the arguments to the call are constructed in your function and you want to verify they are correct for the call.</p><p>You could mutate some fields that are in the structfor testing, but I find that it leaves more mess than I want.</p><p>There are several ways we can handle this.</p><h4><strong>If you only are faking in a single test</strong></h4><p>This is the easiest scenario, because you can manipulate variables defined in the test:</p><pre>func TestPublicFuncThatCallsSomethingRunner(t *testing.T) {<br> called := false // We will set this if the fake is called.<br><br> fake := func(context.Context, arg string) error {<br>  called = true<br>  return nil<br> }<br><br> ms := &amp;MyStruct{<br>  fakeRunner: fake,<br> }<br><br> ms.PublicFuncThatCallsRunSomething(context.Background(), &quot;arg&quot;)<br><br> if !called {<br>  t.Fatalf(&quot;TestPublicFuncThatCallsSomethingRunner: did not call somethingRunner&quot;)<br> }<br>}</pre><p>See it in the playground: <a href="https://go.dev/play/p/3HMp93BKwZK">https://go.dev/play/p/JiSrIUjP1ES</a></p><h4>Faking in multiple tests</h4><p>Instead of creating a function and recording variables, you can create an object with the fake function being a method. This allows you to record whatever your test needs for later inspection.</p><p>This is similar to how you normally implement a fake with an interface except you only pass the method instead of an entire object.</p><pre>type faker struct { // This records results<br> called bool<br>}<br><br>func (f *faker) fakeRun(m *MyStruct, ctx context.Context, arg string) error {<br> f.called = true<br> m.argMutate = arg<br> return nil<br>}<br><br>func TestPublicFuncThatCallsRun(t *testing.T) {<br> f := &amp;faker{}<br><br> ms := &amp;MyStruct{<br>  fakeRun: f.fakeRun,<br> }<br><br> arg := &quot;arg&quot;<br><br> ms.PublicFuncThatCallsRun(context.Background(), arg)<br><br> if !f.called {<br>  t.Fatalf(&quot;TestPublicFuncThatCallsRun: did not call run&quot;)<br> }<br> if ms.argMutate != arg {<br>  t.Fatalf(&quot;TestPublicFuncThatCallsRun: argMutate: got %q, want %q&quot;, ms.argMutate, arg)<br> }<br>}</pre><p>See it in the playground: <a href="https://go.dev/play/p/dUDNXz3ZtIs">https://go.dev/play/p/dUDNXz3ZtIs</a></p><p>This method also lets you set various parameters on how run() should respond by checking the struct fields. For example, you can have an attribute such as shouldErr that if set will cause the method to emit an error.</p><h4>Shameless Plug</h4><p>If you like articles like this and want to know how to leverage Go for DevOps, checkout my book:</p><p><a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894?source=post_page-----3b667f345b5e--------------------------------">Go for DevOps: Learn how to use the Go language to automate servers, the cloud, Kubernetes, GitHub, Packer, and Terraform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bf3e5af9e29" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go State Machine Patterns]]></title>
            <link>https://medium.com/@johnsiilver/go-state-machine-patterns-3b667f345b5e?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b667f345b5e</guid>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 15:02:28 GMT</pubDate>
            <atom:updated>2024-03-12T22:13:11.740Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ugpP0DHejfNCnjeB" /></figure><h3>Introduction</h3><p>In many of the jobs I’ve had over the years, there has been a need to do sequential operations that depended on some operational state.</p><p>Examples of this include:</p><ul><li>Parsing configuration languages, programming languages, etc…</li><li>Executing operations on systems, routers, clusters, …</li><li>ETL (Extract Transform Load)</li></ul><p>A long time ago, I came across Rob Pike’s talk on<a href="https://www.youtube.com/watch?v=HxaD_trXwRE"> lexical scanning</a> in Go. This is a complex talk that took me a few views to really understand. But one of the most fundamental pieces of knowledge he introduces during the talk is his version of a Go state machine.</p><p>This takes advantage of Go’s ability to make a type out of a function and assign a function to a variable.</p><p>This state machine he introduces in that talk is powerful and breaks the logic of having functions do an <strong>if/else</strong> and call the next needed function. It replaces it with a version where each state returns the next function that should be called.</p><p>This allows us to break the call chain into more easily testable parts.</p><h3>A Call Chain</h3><p>Here is an example of accomplishing something with a simple call chain:</p><pre>func Caller(args Args) {<br>  callA(args)<br>  callB(args)<br>}</pre><p>or</p><pre>func Caller(args Args) {<br>  callA(args)<br>}<br><br>func callA(args Args) {<br>  callB(args)<br>}<br><br>func callB(args Args) {<br>  return<br>}</pre><p>Both of these represent a call chain where <strong>Caller()</strong> calls <strong>callA()</strong> which eventually leads to <strong>callB()</strong>. You can see how we can end up with a series of calls that are executed.</p><p>There is nothing wrong with this design of course, but when these callers are making remote calls to other systems, those remote calls must be mocked/faked out in order to provide hermetic testing.</p><p>You also may want to have conditional call chains, where depending on some argument or state, you use <strong>if/else </strong>to call different functions under certain conditions.</p><p>This means that to hermetically test <strong>Caller()</strong> you may need to handle fakes all the way through the entire call chain. If there are 50 stages, you may need to handle all the mock/fakes for every stage below the function you want to test.</p><p>This is where Pike’s state machine design can really shine.</p><h3>The state machine pattern</h3><p>First, we define a state:</p><pre>type State[T any] func(ctx context.Context, args T) (T, State[T], error)</pre><p>This represents a function/method that receives a set of arguments as any type <strong>T </strong>and returns the arguments for the next state and the next <strong>State</strong> to run or an <strong>error</strong>.</p><p>If the <strong>State</strong> returned is <strong>nil</strong>, then the state machine will stop running. If the <strong>error</strong> is set, the state machine will also stop. Because you are returning what the next <strong>State</strong> to run is, you have different next <strong>State(</strong>s) depending on various conditions.</p><p>One of the differences of this version from Pike’s state machine is the inclusion of generics here and the returned <strong>T</strong>. This allows us to create a purely functional state machine (if desired) that can return a stack allocated type that can be passed to the next <strong>State</strong>. Pike did not have access to generics back when he originally wrote this design.</p><p>Now to make this work we need a state runner:</p><pre>func Run[T any](ctx context.Context, args T, start State[T]) (T, error) {<br>  var err error<br>  current := start<br>  for {<br>    if ctx.Err() != nil {<br>      return args, ctx.Err()<br>    }<br>    args, current, err = current(ctx, args)<br>    if err != nil {<br>      return args, err<br>    }<br>    if current == nil {<br>      return args, nil<br>    }<br>  }<br>}</pre><p>Now we have a functional state runner in a few lines of code.</p><p>Let’s look at an example where we write the state machine for service turndown operations in with a cluster:</p><pre>package remove<br><br>...<br><br>// storageClient provides the methods on a storage service<br>// that must be provided to use Remove().<br>type storageClient interface {<br>  RemoveBackups(ctx context.Context, service string, mustKeep int) error<br>  RemoveContainer(ctx context.Context, service string) error<br>}<br><br>// serviceClient provides methods to do operations for services <br>// within a cluster.<br>type servicesClient interface {<br>  Drain(ctx context.Context, service string) error<br>  Remove(ctx context.Context, service string) error<br>  List(ctx context.Context) ([]string, error)<br>  HasStorage(ctx context.Context, service string) (bool, error)<br>}</pre><p>Here we define a couple of private interfaces that we will need clients to implement in order to remove our service from cluster.</p><p>We define private interfaces to prevent others from using our definitions, but we will expose them via a public variable. This keeps us loosely bound to the clients with just the methods we require.</p><pre>// Args are arguments to Service().<br>type Args struct {<br>  // Name is the name of the service.<br>  Name string<br>  <br>  // Storage is a client that can remove storage backups and storage<br>  // containers for a service.<br>  Storage storageClient<br>  // Services is a client that allows the draining and removal of<br>  // a service from the cluster.<br>  Services servicesClient<br>}<br><br>func (a Args) validate(ctx context.Context) error {<br>  if a.Name == &quot;&quot; {<br>    return fmt.Errorf(&quot;Name cannot be an empty string&quot;)<br>  }<br><br>  if a.Storage == nil {<br>    return fmt.Errorf(&quot;Storage cannot be nil&quot;)<br>  }<br>  if a.Services == nil {<br>    return fmt.Errorf(&quot;Services cannot be nil&quot;)<br>  }<br>  return nil<br>}</pre><p>This sets up the arguments we are going to pass through our states. We could include private fields that would be set in one <strong>State</strong> and passed into another <strong>State</strong>.</p><p><strong>Notice</strong> that <strong>Args</strong> <em>is not a pointer.</em></p><p>Because we modify and pass the <strong>Args</strong> to each <strong>State</strong>, we do not need to burden the garbage collector. For some operation like this, it is trivial savings. In a heavy hit ETL pipeline, this could be the difference in long pauses that you don’t need.</p><p>We include a <strong>validate()</strong> method to let us test our arguments meet the base minimum to be used.</p><pre>// Service removes a service from a cluster and associated storage.<br>// The last 3 storage backups are retained for whatever the storage retainment<br>// period is.<br>func Service(ctx context.Context, args Args) error {<br>  if err := args.validate(); err != nil {<br>    return err<br>  }<br>  <br>  start := drainService<br>  _, err := Run[Args](ctx, args, start)<br>  if err != nil {<br>    return fmt.Errorf(&quot;problem removing service %q: %w&quot;, args.Name, err)<br>  }<br>  return nil<br>}</pre><p><strong>Service() </strong>is what the package users will call. They simply pass in the <strong>Args</strong> and receive an error if something goes wrong. They don’t see the state machine pattern and don’t need to understand it in order to perform the operation.</p><p>We simply validate the <strong>Args</strong> look correct, set the starting state of our state machine to a function called <strong>drainService</strong> and call the <strong>Run()</strong> function we defined above.</p><pre>func drainService(ctx context.Context, args Args) (Args, State[Args], error) {<br>  l, err := args.Services.List(ctx)<br>  if err != nil {<br>    return args, nil, err<br>  }<br><br>  found := false<br>  for _, entry := range l {<br>    if entry == args.Name {<br>      found = true<br>      break<br>    }<br>  }<br>  if !found {<br>    return args, nil, fmt.Errorf(&quot;the service was not found&quot;)<br>  }<br><br>  if err := args.Services.Drain(ctx, args.Name); err != nil {<br>    return args, nil, fmt.Errorf(&quot;problem draining the service: %w&quot;, err)<br>  }<br><br>  return args, removeService, nil<br>}</pre><p>Our first state is called <strong>drainService()</strong>. It implements the <strong>State</strong> type we defined above.</p><p>It uses the <strong>Services</strong> client defined in <strong>Args</strong> to list all the services in a cluster. If it does not find the service, it returns an error and the state machine ends.</p><p>If it find the service, it performs a service drain on the service. Once that completes we move on to the next state, <strong>removeService().</strong></p><pre>func removeService(ctx context.Context, args Args) (Args, State[Args], error) {<br>  if err := args.Services.Remove(ctx, args.Name); err != nil {<br>    return args, nil, fmt.Errorf(&quot;could not remove the service: %w&quot;, err)<br>  }<br><br>  hasStorage, err := args.Services.HasStorage(ctx, args.Name)<br>  if err != nil {<br>    return args, nil, fmt.Errorf(&quot;HasStorage() failed: %w&quot;, err)<br>  }<br>  if hasStorage{<br>    return args, removeBackups, nil<br>  }<br><br>  return args, nil, nil<br>}</pre><p><strong>removeService()</strong> uses our <strong>Services</strong> client to remove the service from running in the cluster.</p><p>Once that complete, is determines if we have storage by calling the <strong>HasStorage()</strong> method. If we have storage, we move on to the <strong>removeBackups()</strong> <strong>State</strong>, otherwise we return <strong>args, nil, nil</strong> which causes the state machine to exit with no errors.</p><p>This is an example of how you can branch in your state machine based on either information in <strong>Args</strong> or remote calls that your code makes.</p><p>The other <strong>State </strong>calls are up to you to determine. Let’s look at how this design is better suited for testing this type of action.</p><h3>Testing Advantages</h3><p>The first thing this pattern can encourage is small blocks of testable code. It makes things easily divisible so that when a block gets too big, you simply create a new <strong>State</strong> that isolates the block.</p><p>But the bigger advantage is removing the need for a large end to end test. Because each stage in an operational flow needs to call the next stage, you end up in one of the following scenarios:</p><ul><li>The top-level caller calls all the sub functions in some order</li><li>Each caller calls the next function</li><li>Some mixture of the two</li></ul><p>Both lead to some type of end to end test that shouldn’t be needed.</p><p>If we coded the top level caller method, it might look like this:</p><pre>func Service(ctx context.Context, args Args) error {<br>  ...<br>  if err := drainService(ctx, args); err != nil {<br>    return err<br>  }<br><br>  if err := removeService(ctx, args); err != nil {<br>    return err<br>  }<br><br>  hasStorage, err := args.Services.HasStorage(ctx, args.Name)<br>  if err != nil {<br>    return err<br>  }<br><br>  if hasStorage{<br>    if err := removeBackups(ctx, args); err != nil {<br>      return err<br>    }<br>    if err := removeStorage(ctx, args); err != nil {<br>      return err<br>    }<br>  }<br>  return nil<br>}  </pre><p>As you can see, you can write individual tests for all your sub-functions, but to test <strong>Service()</strong>, now you have to fake either all the clients for each call or other unsavory methods. That starts to look like end to end tests, and for this type of code it is usually a bad idea.</p><p>If we move to a functional call chain, the situation doesn’t get much better:</p><pre>func Service(ctx context.Context, args Args) error {<br>  ...<br>  return drainService(ctx, args)<br>}<br><br>func drainService(ctx context.Context, args Args) (Args, error) {<br>  ...<br>  return removeService(ctx, args)<br>}<br><br>func removeService(ctx context.Context, args Args) (Args, error) {<br>  ...<br>  hasStorage, err := args.Services.HasStorage(ctx, args.Name)<br>  if err != nil {<br>    return args, fmt.Errorf(&quot;HasStorage() failed: %w&quot;, err)<br>  }<br>  <br>  if hasStorage{<br>    return removeBackups(ctx, args)<br>  }<br><br>  return nil<br>}<br>...</pre><p>Now when we test, the tests get more difficult to implement as we get closer to the top of the call chain. At <strong>Service() </strong>we have to test <strong>drainService(), removeService() </strong>and every call below it.</p><p>There are several ways to do this, but none are fun.</p><p>With the state machine version, we can simply test that each individual stage does what we want and returns the next stage we want.</p><p>The top level caller doesn’t even need to be tested. It calls a <strong>validate() </strong>method we can write a test for and calls a <strong>Run()</strong> function we should have tests for.</p><p>Let’s write a table-driven test for <strong>drainService(). </strong>I will include the <strong>drainService() </strong>code to minimize having to scroll.</p><pre>func drainService(ctx context.Context, args Args) (Args, State[Args], error) {<br>  l, err := args.Services.List(ctx)<br>  if err != nil {<br>    return args, nil, err<br>  }<br><br>  found := false<br>  for _, entry := range l {<br>    if entry == args.Name {<br>      found = true<br>      break<br>    }<br>  }<br>  if !found {<br>    return args, nil, fmt.Errorf(&quot;the service was not found&quot;)<br>  }<br><br>  if err := args.Services.Drain(ctx, args.Name); err != nil {<br>    return args, nil, fmt.Errorf(&quot;problem draining the service: %w&quot;, err)<br>  }<br><br>  return args, removeService, nil<br>}<br><br>func TestDrainSerivce(t *testing.T) {<br>  t.Parallel()<br><br>  tests := []struct {<br>    name      string<br>    args      Args<br>    wantErr   bool<br>    wantState State[Args]<br>  }{<br>    {<br>      name: &quot;Error: Services.List() returns an error&quot;,<br>      args: Args{<br>        Services: &amp;fakeServices{<br>          list: fmt.Errorf(&quot;error&quot;),<br>        },<br>      },<br>      wantErr: true,<br>    },<br>    {<br>      name: &quot;Error: Services.List() didn&#39;t contain our service name&quot;,<br>      args: Args{<br>        Name: &quot;myService&quot;,<br>        Services: &amp;fakeServices{<br>          list: []string{&quot;nope&quot;, &quot;this&quot;, &quot;isn&#39;t&quot;, &quot;it&quot;},<br>        },<br>      },<br>      wantErr: true,<br>    },<br>    {<br>      name: &quot;Error: Services.Drain() returned an error&quot;,<br>      args: Args{<br>        Name: &quot;myService&quot;,<br>        Services: &amp;fakeServices{<br>          list:  []string{&quot;yes&quot;, &quot;mySerivce&quot;, &quot;is&quot;, &quot;here&quot;},<br>          drain: fmt.Errorf(&quot;error&quot;),<br>        },<br>      },<br>      wantErr: true,<br>    },<br>    {<br>      name: &quot;Success&quot;,<br>      args: Args{<br>        Name: &quot;myService&quot;,<br>        Services: &amp;fakeServices{<br>          list:  []string{&quot;yes&quot;, &quot;myService&quot;, &quot;is&quot;, &quot;here&quot;},<br>          drain: nil,<br>        },<br>      },<br>      wantState: removeService,<br>    },<br>  }<br><br>  for _, test := range tests {<br>    _, nextState, err := drainService(context.Background(), test.args)<br>    switch {<br>    case err == nil &amp;&amp; test.wantErr:<br>      t.Errorf(&quot;TestDrainService(%s): got err == nil, want err != nil&quot;, test.name)<br>      continue<br>    case err != nil &amp;&amp; !test.wantErr:<br>      t.Errorf(&quot;TestDrainService(%s): got err == %s, want err == nil&quot;, test.name, err)<br>      continue<br>    case err != nil:<br>      continue<br>    }<br>  <br>    gotState := methodName(nextState)<br>    wantState := methodName(test.wantState)<br>    if gotState != wantState {<br>      t.Errorf(&quot;TestDrainService(%s): got next state %s, want %s&quot;, test.name, gotState, wantState)<br>    }<br>  }<br>}</pre><p>You can play with this on the Go playground <a href="https://go.dev/play/p/HcgYkQOjeIz">here</a>.</p><p>As you can see we avoid any need to test an entire call chain while still ensuring that the next function in the chain will be called when we want it to.</p><p>The tests are easily divisible and are easy for a maintainer to follow.</p><h3>Other Possibilities</h3><p>There are variations on this pattern where the <strong>State</strong> is determined from a field set in <strong>Args</strong> and you can track <strong>State </strong>execution to prevent loops.</p><p>In the first case, the state machine package might look like:</p><pre>type State[T any] func(ctx context.Context, args T) (T, State[T], error)<br><br>type Args[T] struct {<br>  Data T<br><br>  Next State<br>}<br><br><br>func Run[T any](ctx context.Context, args Args[T], start State[T]) (T, error) {<br>  var err error<br>  current := start<br>  for {<br>    if ctx.Err() != nil {<br>      return args, ctx.Err()<br>    }<br>    args, current, err = current(ctx, args)<br>    if err != nil {<br>      return args, err<br>    }<br>    current = args.Next // Set our next stage<br>    args.Next = nil // Clear this so to prevent infinite loops<br><br>    if current == nil {<br>      return args, nil<br>    }<br>  }<br>}</pre><p>And you can easily do things like integrate distributing tracing or logging into this design.</p><p>If you are looking to push a lot of data and take advantage of concurrency AND parallelism, you might give the <a href="https://github.com/gostdlib/concurrency/tree/main/pipelines/stagedpipe">stagedpipe package</a> a try. It is based on this structure with a ton of advanced features built in. There are videos and a README to introduce you to using the package.</p><p>Hopefully this article has given you a good understanding of the Go state machine design pattern. You now have powerful new tool in your toolbelt.</p><h3>Shameless Plug</h3><p>If you like articles like this and want to know how to leverage Go for DevOps, checkout my book:</p><p><a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894?source=post_page-----6a8edad4159d--------------------------------">Go for DevOps: Learn how to use the Go language to automate servers, the cloud, Kubernetes, GitHub, Packer, and Terraform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b667f345b5e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go: And Leave Your Java Factories Behind]]></title>
            <link>https://medium.com/@johnsiilver/go-and-leave-your-java-factories-behind-179067818e9f?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/179067818e9f</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Sat, 09 Mar 2024 16:29:07 GMT</pubDate>
            <atom:updated>2024-06-03T17:11:29.913Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5bL1-8FclyaDhI9x" /></figure><p>Recently I saw an article come up about using the Go Factory pattern. The article was encouraging people to use a pattern that may have a place in languages such as Java and C#, but have no place within Go.</p><p>Unfortunately, I have run into various Go code that uses a factory pattern. The code is normally brittle because of the misuse of the Go interface types.</p><p>I believe this pattern shows up because people tend to bring the patterns that they used in other languages to new languages. When I began writing Go code, it looked more like Python code, the language I had recently been using.</p><p>I don’t write in Java and I wrote my last C# code 20+ years ago. So I am not familiar with the nature of interfaces in those languages. But my understanding from hearsay is that there is tight coupling between those interfaces and the static types that implement them.</p><p>Go has loose coupling with interfaces. You do not need to declare what types implement what, they implement them simply by having the same method signatures as the interface. They can have methods that the interface does not, but they have to have the minimum set to implement the interface.</p><h3>An Example of A Go Factory Pattern</h3><p>Let’s look at the factory code example that I saw in the article I referenced above. I am not going to link the original article, as this article isn’t about shaming anyone. It is about helping to encourage better Go programming that is easier to understand, use, and extend.</p><pre>type Database interface {<br>    Query(query string) string<br>}<br><br>type MySQL struct {}<br><br>func (m MySQL) Query(query string) string {<br>    // implementation for MySQL<br>    return &quot;MySQL result&quot;<br>}<br><br>type PostgreSQL struct {}<br><br>func (p PostgreSQL) Query(query string) string {<br>    // implementation for PostgreSQL<br>    return &quot;PostgreSQL result&quot;<br>}<br><br>type DatabaseFactory func() Database<br><br>func NewMySQL() Database {<br>    return MySQL{}<br>}<br><br>func NewPostgreSQL() Database {<br>    return PostgreSQL{}<br>}<br><br>type Service struct {<br>    dbFactory DatabaseFactory<br>}<br><br>func (s *Service) GetData(query string) string {<br>    db := s.dbFactory()<br>    return db.Query(query)<br>}<br><br>func main() {<br>    service := Service{dbFactory: NewMySQL}<br>    fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: MySQL result<br><br>    service.dbFactory = NewPostgreSQL<br>    fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: PostgreSQL result<br>}</pre><p>You can run the code <a href="https://go.dev/play/p/4aPtql-M1Iz">here</a>.</p><p>This code provides two constructors: <strong>NewMySql()</strong> and <strong>NewPostgreSQL()</strong> that return an interface type called <strong>Database.</strong></p><p>There is another type, <strong>DatabaseFactory</strong>, that is a function that returns a <strong>Database</strong>.</p><p>The <strong>Service</strong> type has a <strong>DatabaseFactory </strong>that creates a database that is used by the method <strong>GetData()</strong>.</p><p>To start with, the <strong>DatabaseFactory</strong> is a pretty useless here.</p><p>Simply ditching the <strong>DatabaseFactory</strong> for <strong>Database </strong>will yield you the same results:</p><pre>…<br>type Service struct {<br> db Database // &lt;- Changed From DatabaseFactory to Database type<br>}<br>…<br>func main() {<br> service := Service{db: MySQL{}}<br> fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: MySQL result<br>service.db = Postgres{}<br> fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: PostgreSQL result<br>}<br></pre><p>You can see the runnable example <a href="https://go.dev/play/p/PKVxP600JWw">here</a>.</p><h3>Even with that change, the code is brittle</h3><p>With the removal of <strong>DatabaseFactory</strong>, we still have brittle code. Let’s add a method to the <strong>Database</strong> interface that is not needed by <strong>Service </strong>and<strong> </strong>see what happens:</p><pre>type Database interface {<br> Query(query string) string<br> Create(query string) error // &lt;- New method<br>}</pre><p>Run it <a href="https://go.dev/play/p/QAA8nNx2Wsz">here</a>.</p><pre>./prog.go:25:9: cannot use MySQL{} (value of type MySQL) as Database value in return statement: MySQL does not implement Database (missing method Create)<br>./prog.go:29:9: cannot use PostgreSQL{} (value of type PostgreSQL) as Database value in return statement: PostgreSQL does not implement Database (missing method Create)<br>./prog.go:41:25: cannot use MySQL{} (value of type MySQL) as Database value in struct literal: MySQL does not implement Database (missing method Create)<br>./prog.go:44:15: cannot use PostgreSQL{} (value of type PostgreSQL) as Database value in assignment: PostgreSQL does not implement Database (missing method Create)</pre><p>Now, both <strong>PostgreSQL</strong> and <strong>MySQL</strong> no longer work. But <strong>Service</strong> never required <strong>Create</strong> and we have broken it for no reason.</p><p>In a real life scenario, we probably have these implementations or wrappers in other packages and not in the same package. This makes this pattern even more brittle because we don’t immediately see that we broke something.</p><p>If we wrote test versions of <strong>Database</strong> so that our tests are hermetic, those would also break.</p><p>This is because Go interfaces are for <em>CONSUMERS</em> and not <em>CREATORS</em>.</p><h3>Let’s stop returning interfaces</h3><p>Instead let’s return concrete types and consume interfaces.</p><pre>type Database interface {<br> Query(query string) string<br>}<br><br>type MySQL struct{}<br><br>func (m MySQL) Query(query string) string {<br> // implementation for MySQL<br> return &quot;MySQL result&quot;<br>}<br><br>func (m MySQL) Create(query string) error { // &lt;- added Create here, didn&#39;t break anything<br> return nil<br>}<br><br>type PostgreSQL struct{}<br><br>func (p PostgreSQL) Query(query string) string {<br> // implementation for PostgreSQL<br> return &quot;PostgreSQL result&quot;<br>}<br><br>func NewMySQL() MySQL { // &lt;- Return the concrete type<br> return MySQL{}<br>}<br><br>func NewPostgreSQL() PostgreSQL { // &lt;- Return the concrete type<br> return PostgreSQL{}<br>}<br><br>type Service struct {<br> db Database // We don&#39;t need Create, so we don&#39;t change the interface<br>}<br><br>func (s *Service) GetData(query string) string {<br> return s.db.Query(query)<br>}<br><br>func main() {<br> service := Service{db: NewMySQL()}<br> fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: MySQL result<br><br> service.db = NewPostgreSQL()<br> fmt.Println(service.GetData(&quot;SELECT * FROM table&quot;)) // Outputs: PostgreSQL result<br>}</pre><p>Now, when we need to add something like <strong>Create()</strong> to one of our database implementations for use in another package, we don’t need to expose it in <strong>Database </strong>interface. <strong>Database</strong> should be local to our package to define what our types need to consume.</p><p>In fact, if we changed <strong>Database</strong> to <strong>database </strong>(private type), outside packages could still implement <strong>database</strong> and this package could still consume them, but without as much danger that changes to <strong>database</strong> would affect other packages.</p><p>The long of this is, <strong>database</strong> should define what we will consume, not a return type. By doing so, we can add functionality to our implementors without affecting this package or other packages designed like it.</p><h3>A few interface pointers</h3><p>Interfaces are great in Go, but can also give headaches if not used thoughtfully.</p><p>Here are some general rules to follow:</p><ul><li>Consume interfaces, do not create them with constructors</li><li>Keep your consumed interfaces defined locally to prevent breaking changes</li><li>Do not expose public interfaces unless they will <strong>NEVER change</strong><br>– The <strong>io</strong> library is an example of interfaces that do not change<br>– Rarely is this a good idea and rarely do they not change</li><li>If you need to expose interfaces that could change, allow only internal types to implement them<br>– This can be accomplished by including a <strong>private()</strong> method on the interface<br>– A more sophisticated method for this: <a href="https://medium.com/@johnsiilver/writing-an-interface-that-only-sub-packages-can-implement-fe36e7511449">https://medium.com/@johnsiilver/writing-an-interface-that-only-sub-packages-can-implement-fe36e7511449</a></li></ul><h3>Note: A factory-like structure that is safe to use</h3><p>There is a type of Go struct that I’ve seen used and I use myself occasionally that has some characteristics of a factory pattern but without all the dangerous problems.</p><p>Normally, this is when I’m wrapping some type of autogenerated REST client to create a Go client for a service. As many of you have seen, Cloud clients can be quite large and ugly.</p><p>This use case allows for Go doc pages that are concise and easy to follow for users.</p><p>The pattern looks like the following pseudo code:</p><pre>import (<br>  &quot;../subClientA&quot;<br>  &quot;../subClientB&quot;<br>)<br><br>type Client struct {<br>  conn *restClient<br>  ...<br>}<br><br>func New(conn *restClient) (*Client, error) {<br>  ...<br>}<br><br>func (c *Client) SubClientA() *subClientA.Client {<br>  return subClientA.New(c.conn)<br>}<br><br>func (c *Client) SubClientB() *subClientB.Client {<br>  return subClientB.New(c.conn)<br>}</pre><p>I have used this pattern when I want to divide a large client with hundreds of methods into digestible chunks. The various sub-clients are concrete types that simply bundle like methods together. This allows sharing of a single connection among the client and sub-clients and allows you to create the sub-clients individually if you want by going directly to the sub-packages.</p><p>This pattern lacks the brittle nature of the factory pattern, has a purpose that makes sense in the context of Go and is easily testable.</p><h3><strong>Note: Never say never</strong></h3><p>Last but not least, while Factories are *mostly* an antipattern in Go, that doesn’t mean there isn’t ever a use case.</p><p>It just means that it shouldn’t be the first tool to grab in your toolbox, but a specialized tool that your rarely need because Go has better options in most situations, where other languages are built on this pattern.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=179067818e9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writing an interface that only sub-packages can implement]]></title>
            <link>https://medium.com/@johnsiilver/writing-an-interface-that-only-sub-packages-can-implement-fe36e7511449?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe36e7511449</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Thu, 11 May 2023 03:13:38 GMT</pubDate>
            <atom:updated>2023-05-13T16:15:56.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*54JDYmQoXMppqMuhnAAFOQ.jpeg" /></figure><p>Sometimes we need to provide an interface that we only want our packages to implement.</p><p>The most common way of doing this is having a package that has a private method, so only types in that package can implement it.</p><pre>type MyInterface interface {<br>  SayHello()<br>  internalOnly()<br>}<br><br>type Implementation struct {}<br>func (i Implementation) SayHello() {<br>  fmt.Println(&quot;Hello&quot;)<br>}<br>func( i Implementation) internalOnly() {}</pre><p>The code above provides MyInterface which has method internalOnly which can only be implemented by a type in that package. This is great and I often use this method to prevent others from implementing an interface that I expose.</p><p>Allowing others to implement an interface you define is dicey. I only do it when there is both a need and I KNOW that the interface won’t change. Otherwise adding a method at a later date will break all of my user’s code.</p><p>But this method doesn’t extend to handle the case where you want to organize your code with sub-packages. In that case, you might want to have a main package that defines an interface and sub-packages that implement that interface. But if you use the method above, the sub-packages cannot implement the interface. If you get rid of the private method, users can implement the interface and your interface becomes unchangeable without breaking user code (or doing a major version increment, which is breaking to the users in a different way). You can make a superset of the interface, but this clutters your API.</p><p>And while you can put all the types that implement an interface in the same package, I don’t find that as clean as using sub-packages.</p><h3>A solution to the problem</h3><p>Recently I came up with a solution that uses the internal/ package directory (I’ve never seen this method in use before, but it doesn’t mean someone hasn’t done it). For those that don’t know, internal/ can hold packages that can only be accessed by packages within a certain folder hierarchy. You can read about it <a href="https://go.dev/doc/go1.4#internalpackages">here</a>.</p><p>To achieve my goal, I create an internal package internal/local :</p><pre>package local<br><br>type InternalInter interface {<br>  internalOnly()<br>}<br><br>type InternalImpl struct {}<br>func(i InternalImpl) internalOnly(){}</pre><p>Now this looks similar to what we do when we have a single package. InternalInter can only be implemented by a type in the local package. In this case we create the implementation InternalImpl .</p><p>Now, let’s say our package structure looks like the following:</p><pre>hello/<br>  hello.go<br>  internal/<br>    local/<br>      local.go<br>  sayit/<br>    sayit.go</pre><p>In the above structure, hello.go will hold our public interface. sayit.go will hold an implementation of the interface in hello.go .</p><p>hello.go will look like:</p><pre>package hello<br><br>import &quot;.../internal/local&quot;<br><br>type SayHelloer interface {<br>  SayHello()<br>  local.InternalInter<br>}</pre><p><em>Note: “…” stands for the package’s path.</em></p><p>So now the interface we have defined can only be implemented if they implement local.InternalInter we defined in our local package. No user packages can import local.</p><p>But our sub-packages can take advantage of this in our by doing the following:</p><pre>package sayit<br><br>import &quot;.../internal/local&quot;<br><br>type SayIt struct {<br>  local.InternalImpl // embeds the local.InternalImpl<br>}<br><br>func (s SayIt) SayHello() {<br>  fmt.Println(&quot;Hello&quot;)<br>}</pre><p>Now you have a sub-package that implements the interface in our package hello , but the public interface cannot be implemented by external packages.</p><p>Hopefully this method will help those of you trying to organize your code in a similar manner.</p><p>Good luck and happy coding!</p><h3>Shameless plug</h3><p>If you like this article and you are interested in using Go for DevOps work, check out my book:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QF1HcVcmXjXMrd9MRiGVsQ.png" /><figcaption>Buy it on Amazon or any online book seller (I do not recommend the Kindle edition)</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe36e7511449" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Testing gRPC methods]]></title>
            <link>https://medium.com/@johnsiilver/testing-grpc-methods-6a8edad4159d?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a8edad4159d</guid>
            <category><![CDATA[go]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[grpc]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Fri, 28 Apr 2023 22:22:03 GMT</pubDate>
            <atom:updated>2023-04-28T22:22:03.377Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r95SUaQDYI86yz0qQvzd6A.jpeg" /></figure><p>I was recently asked by a colleague on the best way to test a gRPC method for streaming. They were trying to find a simple way to test their code without spinning up a gRPC server.</p><p>gRPC testing is similar to testing the net/http packages HandleFunc and Handler types. You can use net/http/httptest , but most of the time there is no need. We don’t want to test gRPC or an HTTP server itself, we simply want to test our method’s logic.</p><p>The simple answer to this question is to de-couple gRPC’s work from the actual work.</p><p>Let’s pretend we need to receive a request and then stream back a response. This might look something like this in proto3:</p><pre>service Authority {<br> // ProcessExporter instructs the Processor to process some data from an exporter and<br> // stream the results back.<br> rpc Servers(ServersReq) returns (stream ServerMsg) {}<br>}<br><br>message ServersReq {<br>  string name_filter_re = 1;<br>  repeated string datacenter_filter = 2;<br>}<br><br>message ServerMsg {<br>  string name = 1;<br>}</pre><p>The above code represents a simple authoritative data store for server information. You can query with a ServerReq that optionally takes a name filter (an RE2 regular expression) and a filter for data centers the servers are in.</p><p>The Servers() call returns a stream of server information held in the ServerMsg type.</p><p>In gRPC for Go, this would require you to implement this:</p><pre>type AuthorityServer interface {<br> Servers(*ServersReq, Authority_ServersServer) error<br> mustEmbedUnimplementedServersServer()<br>}</pre><p>This is autogenerated by the protobuf compiler. If you are using Google’s proto compiler, I suggest switching to the <a href="http://buf.build/">http://buf.build</a> compiler, which is much easier to use (if you are in a single repo for all protocol buffer definitions. If not you do need their pay product, BSR).</p><p>For simplicity, this example will have all our server data in a file we access instead of the more common use case of a database.</p><h3><strong>The Naive way</strong></h3><p>The naive way to implement this is to do it in the gRPC implementation directly:</p><pre>// Authority implements the gRPC AuthorityServer interface.<br>type Authority struct {<br>  dataFilePath string<br><br>  pb.UnimplementedAuthorityServer // required by gRPC<br>}<br><br>type serversRec struct {<br> Name       string<br> Datacenter string<br>}<br><br>// Servers implements AuthorityServer.Servers .<br>func (a *Authority) Servers(req *pb.ServersReq, stream pb.Authority_ServersServer) error {<br>  f, err := os.Open(a.dataFilePath)<br>  if err != nil {<br>    return err<br>  }<br>  defer f.Close()<br><br>  // Compile our name filter if it was given.<br>  var nameFilter *regexp.Regexp<br>  if req.NameFilterRe != &quot;&quot; {<br>    var err error<br>    nameFilter, err = regexp.Compile(req.NameFilterRe)<br>    if err != nil {<br>      return status.Errorf(codes.InvalidArgument, &quot;name_filter_re did not compile: %s&quot;, err)<br>    }<br>  }<br><br>  // dcs if not nil has keys that are the datacenters we want to filter for.<br>  var dcs map[string]bool<br>  if len(req.DatacenterFilter) &gt; 0 {<br>    dcs = map[string]bool{}<br>    for _, dc := range req.DatacenterFilter {<br>      dcs[dc] = true<br>    }<br>  }<br><br>  // Stream our entries if they match the various filters in the request.<br>  dec := json.NewDecoder(f)<br>  for {<br>    if stream.Context().Err() != nil {<br>        return status.Errorf(codes.DeadlineExceeded, stream.Context().Err().Error())<br>    }<br><br>    var r serversRec<br>    if err := dec.Decode(&amp;r); err == io.EOF {<br>      break<br>    } else if err != nil {<br>      return err<br>    }<br><br>    // If the name filter was defined, reject any entries that don&#39;t match.<br>    if nameFilter != nil &amp;&amp; !nameFilter.MatchString(r.Name) {<br>        continue<br>    }<br><br>    // If the DC filter was provided, reject any entries that don&#39;t match.<br>    if dcs != nil &amp;&amp; !dcs[r.Datacenter] {<br>      continue<br>    }<br><br>    // If we get here, stream the ServerMsg to the caller.<br>    if err := stream.Send(&amp;pb.ServerMsg{Name: r.Name}); err != nil {<br>      return err<br>    }<br>  }<br>  return nil<br>}</pre><p>This methodology is pretty straight forward. Stream all entries from a data file (encoded with JSON entries on disk) and return any entries that match our filters.</p><p>Testing this requires either faking pb.Authority_ServersServer or running the gRPC service and using the generated gRPC client. And while those are valid ways to test, it is more bootstrapping than needed and it tests gRPC, which isn’t needed (gRPC already has an entire test suite).</p><h3><strong>A Simpler Way</strong></h3><p>Since we don’t want to test gRPC and want to make this easier for ourselves to test, we simply need to divide this into two methods:</p><pre>func (a *Authority) Servers(req *pb.ServersReq, stream pb.Authority_ServersServer) error {<br>  ...<br>}<br>func (a *Authority) servers(req *pb.ServersReq) chan StreamMsg[*pb.ServerMsg] {<br>  ...<br>}</pre><p>Servers() is what is going to handle gRPC streaming. It won’t need tests.</p><p>servers() is going to hold the bulk of our code and can be easily tested.</p><p>StreamMsg is a new generic type that can hold any data and signal a stream error.</p><p>Let’s look at StreamMsg :</p><pre>type StreamMsg[T any] struct {<br>  Data T<br>  Err error<br>}</pre><p>This is a generic holder of streaming information that can be used for any channel. Most channels need a way to signal an error and this is an excellent way to do that using generics.</p><p>Here is our simplified Servers() implementation:</p><pre>func (a *Authority) Servers(req *pb.ServersReq, stream pb.Authority_ServersServer) error {<br>  for msg := range a.servers(stream.Context(), req) {<br>    if msg.Err != nil {<br>      return msg.Err<br>    }<br>    if err := stream.Send(&amp;pb.ServerMsg{Name: msg.Data.Name}); err != nil {<br>      return err<br>    }<br>  }<br>  return nil<br>}</pre><p>Now Servers() simply handles error detection and sending on the gRPC stream. This pattern is simple enough that it doesn’t require a test.</p><p>Let’s have a look at servers():</p><pre>func (a *Authority) servers(ctx context.Context, req *pb.ServersReq) &lt;-chan StreamMsg[*pb.ServerMsg] {<br>  ch := make(chan StreamMsg[*pb.ServerMsg], 1)<br><br>  go func() {<br>    defer close(ch)<br><br>    f, err := os.Open(a.dataFilePath)<br>    if err != nil {<br>      ch &lt;- StreamMsg[*pb.ServerMsg]{Err: err}<br>      return<br>    }<br>    defer f.Close()<br><br>    // Compile our name filter if it was given.<br>    var nameFilter *regexp.Regexp<br>    if req.NameFilterRe != &quot;&quot; {<br>      var err error<br>      nameFilter, err = regexp.Compile(req.NameFilterRe)<br>      if err != nil {<br>        ch &lt;- StreamMsg[*pb.ServerMsg]{<br>          Err: status.Errorf(codes.InvalidArgument, &quot;name_filter_re did not compile: %s&quot;, err),<br>        }<br>        return<br>      }<br>    }<br><br>    // dcs if not nil has keys that are the datacenters we want to filter for.<br>    var dcs map[string]bool<br>    if len(req.DatacenterFilter) &gt; 0 {<br>      dcs = map[string]bool{}<br>      for _, dc := range req.DatacenterFilter {<br>        dcs[dc] = true<br>      }<br>    }<br>  <br>    // Stream our entries if they match the various filters in the request.<br>    dec := json.NewDecoder(f)<br>    for {<br>      if ctx.Err() != nil {<br>        ch &lt;- StreamMsg[*pb.ServerMsg]{Err: ctx.Err()}<br>        return<br>      }<br><br>      var r serversRec<br>      if err := dec.Decode(&amp;r); err == io.EOF {<br>        return<br>      } else if err != nil {<br>        ch &lt;- StreamMsg[*pb.ServerMsg]{Err: err}<br>        return<br>      }<br>  <br>      // If the name filter was defined, reject any entries that don&#39;t match.<br>      if nameFilter != nil &amp;&amp; !nameFilter.MatchString(r.Name) {<br>          continue<br>      }<br>  <br>      // If the DC filter was provided, reject any entries that don&#39;t match.<br>      if dcs != nil &amp;&amp; !dcs[r.Datacenter] {<br>        continue<br>      }<br>  <br>      select {<br>      case &lt;-ctx.Done():<br>        ch &lt;- StreamMsg[*pb.ServerMsg]{Err: ctx.Err()}<br>        return<br>      case ch &lt;- StreamMsg[*pb.ServerMsg]{Data: &amp;pb.ServerMsg{Name: r.Name}}:<br>      }<br>    }<br>  }()<br>  return ch<br>}</pre><p>This model allows testing without using gRPC directly. Calling this method allows reading from the returned channel until you encounter an error or the channel is closed. If an error is found, the stream will be closed.</p><p>Simply switching out the dataFilePath allows using test data and calling servers() allows you to avoid spinning up a service and connecting to it with a client.</p><p>This methodology of work division can be used to test code that handles http.HandleFunc or http.Handler types as well, avoiding the necessity to spin up an HTTP server or use the httptest package.</p><p>If you’d like to see this code in action, you can see it here: <a href="https://github.com/johnsiilver/medium/tree/main/grpc_testing">code path</a>.</p><p><strong>Note: </strong>This package differs from our example in that I have divided the code into smaller methods. It adds a little more text to allow for better maintainability.</p><p>Until next time!</p><h3><strong>Shameless plug</strong></h3><p>If you like articles like this and want to know how to leverage Go for DevOps, checkout my book:</p><p><a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894">Go for DevOps: Learn how to use the Go language to automate servers, the cloud, Kubernetes, GitHub, Packer, and Terraform</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a8edad4159d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Go: Support Universal Binaries Using Shell Script]]></title>
            <link>https://blog.devgenius.io/go-support-universal-binaries-using-shell-script-ec9478ac716?source=rss-ebed66dcde0------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec9478ac716</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[John Doak]]></dc:creator>
            <pubDate>Tue, 16 Aug 2022 21:55:01 GMT</pubDate>
            <atom:updated>2022-08-24T06:19:25.861Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z7yatnawvL9hUhfG" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>Have you ever wanted to have a Go program that ran on multiple platforms from a single binary? I’d always kinda hoped that Go would have “fat” binaries like the old Mac’s used to have when we converted from PowerPC to Intel that let you run on either platform.</p><p>So what can we do today if we want to distribute our Go program so that it supports a multitude of Platform/OS combinations from a single executable?</p><p>Trusty old shell scripts.</p><h3>Advertisement</h3><p>If you are interested in using Go for DevOps work, checkout my book on Amazon: <a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894/ref=cm_cr_arp_d_pdt_img_top?ie=UTF8">Go For DevOps</a> .</p><p>The proceeds for the authors (not the publisher) is going to <a href="https://www.doctorswithoutborders.org/">Doctors Without Borders</a>, a great organization.</p><h3>Let’s get this out of the way</h3><p>I actually detest shell scripts for most use cases. They are error prone and testing is a pain. My main use case for shell scripts is running a <strong>for</strong> loop around a flaky tests to try and get it to trip again.</p><p>But like anything, there are exceptions. <strong>sh</strong> is on every Unix platform and therefore its universal nature makes it perfect for this use.</p><h3>What caused me to write this?</h3><p>I recently ran into someone trying to use shell to push data to a service that had SDKs in multiple languages. But that person didn’t want to have different binaries for all the systems they were going to support and didn’t want to have to install other runtimes (Python, .Net, …). This meant doing some reverse engineering on that platform’s REST streams and doing some complicated gymnastics in shell while invoking <em>cURL.</em></p><p>I thought that there must be a better way and cooked this up as a demo. The idea comes from this methodology Google and Facebook use to package up Python programs for distribution.</p><h3>The Idea</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl4EpfbRWMHRGhUjRe%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl4EpfbRWMHRGhUjRe%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl4EpfbRWMHRGhUjRe%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/01af70a19ffa7788a7a2ccdcd798152c/href">https://medium.com/media/01af70a19ffa7788a7a2ccdcd798152c/href</a></iframe><p>We are going to wrap multiple Go binaries in a shell script and the shell script will detect the OS/Platform and decode the binary we want to the correct location and invoke it. We can pass the arguments to the shell script directly to the binary.</p><p>Now this will make the shell script large if you have enough binaries or large programs. An alternate version could have the script download the correct version from GitHub or another location to save space. But for this demo we will just bake the binaries in.</p><p>All the code for this demo can be found here: <a href="https://github.com/johnsiilver/shellembed">https://github.com/johnsiilver/shellembed</a></p><h3>The Wrapping Script</h3><p>To make this all work, we need a script that can wrap our Go binaries and then unwrap them when invoked.</p><pre>#!/bin/bash <br># define the tasks that need to be done with the extracted content<br>process_tar() {<br>    cd $WORK_DIR&quot;/bin&quot;<br>    # do something with the extracted content<br>    eval &quot;./hello_${os}_${arc} $@&quot;<br>}</pre><p>This starts by defining a function we are going to call later called <em>process_tar()</em>. That function is going to enter a work directory we will create later and run the <em>eval</em> command on a binary that will be in the work directory, passing it the arguments the script was passed (the <em>$@</em>).</p><pre># line number where payload starts<br>PAYLOAD_LINE=$(awk &#39;/^__PAYLOAD_BEGINS__/ { print NR + 1; exit 0; }&#39; $0)</pre><p>Next we use <em>awk</em> to scan the script file we are in and record the line number that has <em>__PAYLOAD_BEGINS__. </em>Everything after this will be our Go binaries that have put in a compressed <em>tar</em> format at the end of the shell script.</p><pre># Determine the os and arch.<br>if [[ &quot;$OSTYPE&quot; == &quot;linux-gnu&quot;* ]]; then<br>  os=&quot;linux&quot;<br>elif [[ &quot;$OSTYPE&quot; == &quot;darwin&quot;* ]]; then<br>  os=&quot;darwin&quot;<br>else <br>  echo &quot;unsupported OS: &quot; + `uname -a` <br>  exit 1<br>fi</pre><pre>if [[ `uname -m` == &quot;arm64&quot; ]]; then  <br>  arc=&quot;arm64&quot;<br>elif [[ `uname -m` == &quot;x86_64&quot; ]]; then <br>  arc=&quot;amd64&quot;<br>else <br>  echo &quot;unsupported arch: &quot; + `uname -m` <br>  exit 1<br>fi</pre><p>We want to support Linux and Darwin both running on x86_64 and arm64 processors. We use <em>$OSTYPE</em> and <em>uname</em> to determine our OS and platform. If any other combination is present, we give an error.</p><pre># make our directory if it doesn&#39;t exist.<br>mkdir -p /tmp/helloshell<br>WORK_DIR=/tmp/helloshell</pre><p>We then make some work directories where we will dump our extracted tar file to get at our Go executables.</p><pre># extract the embedded tar file<br>tail -n +${PAYLOAD_LINE} $0 | tar -zpx -C $WORK_DIR &gt;/dev/null 2&gt;&amp;1</pre><pre># perform actions with the extracted content<br>process_tar $@</pre><pre>exit 0<br>__PAYLOAD_BEGINS__</pre><p>We then make some work directories where we will dump our extracted tar file to get at our Go executables.</p><p>We then <em>untar</em> our Go executable and execute our <em>process_tar()</em> function from above.</p><p>Now we have a script that can untar some Go binaries and run them if we support the platform. But how do we get the Go binaries into the script?</p><h3>Build For Platform Script</h3><p>Here is a small script that builds our binaries for each OS/Platform combination we wanted and then embeds them in a file called <strong>run.sh. </strong>This file is a copy of <strong>hello.sh</strong> with the Go binaries embedded as compressed <em>tar</em> files.</p><p>This builder script will be called <strong>build_for_platform.sh</strong></p><pre>#!/bin/bash</pre><pre>export GOOS=linux <br>export GOARCH=amd64 <br>go build -o bin/hello_linux_amd64</pre><pre>export GOOS=linux <br>export GOARCH=arm64 <br>go build -o bin/hello_linux_arm64</pre><pre>export GOOS=darwin<br>export GOARCH=amd64 <br>go build -o bin/hello_darwin_amd64</pre><pre>export GOOS=darwin<br>export GOARCH=arm64 <br>go build -o bin/hello_darwin_arm64</pre><p>This first section simply builds our Go binaries in a directory called <em>bin/</em> indexing them by OS and platform.</p><pre>tar -czvf hello.tar.gz bin/</pre><p>This next section creates our <em>tarball</em> of <em>bin/</em></p><pre>cp hello.sh run.sh<br>cat hello.tar.gz &gt;&gt; run.sh</pre><p>Now we create our copy of <strong>hello.sh</strong> as <strong>run.sh</strong> and append our tarball to the end of <strong>run.sh</strong> .</p><pre>rm hello.tar.gz<br>rm -rf bin/</pre><p>Finally, we do some cleanup.</p><h3>Drumroll, if you will</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FItItdif4XcSiSNw9Zh%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FItItdif4XcSiSNw9Zh%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FItItdif4XcSiSNw9Zh%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/f709c6171f37331b8631ef8de4092e64/href">https://medium.com/media/f709c6171f37331b8631ef8de4092e64/href</a></iframe><p>You can now simply run:</p><pre>./build_for_platform.sh</pre><p>Which will create your <strong>run.sh</strong> file.</p><p>Now let’s execute our <strong>run.sh</strong> and pass it an argument (which is a name it will print out):</p><pre>./run.sh John</pre><p>This will output something like this (output depends on the platform):</p><pre>hello John from: darwin/arm64</pre><h3>Conclusion</h3><p>As you can see from above, it is possible to use shell scripts to have a universal binary for Go across all supported platforms. If you wish to keep the shell script small, you could have it simply download the binary you need and call with the same arguments.</p><p>But this is a good way when you don’t want to depend on the internet.</p><p>If you’d like to see more of my writing, you should check out my book on Amazon: <a href="https://www.amazon.com/Go-DevOps-language-Kubernetes-Terraform/dp/1801818894/ref=cm_cr_arp_d_pdt_img_top?ie=UTF8">Go For DevOps</a></p><p>You can also find more writings on various topics here on medium or my blog at <a href="http://golangsre.com/">golangsre.com</a>. And if you want to learn Go from some interactive video lessons, see my free site <a href="https:// golangbasics.com">golangbasics.com</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QF1HcVcmXjXMrd9MRiGVsQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec9478ac716" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/go-support-universal-binaries-using-shell-script-ec9478ac716">Go: Support Universal Binaries Using Shell Script</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>