<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nathan Smith on Medium]]></title>
        <description><![CDATA[Stories by Nathan Smith on Medium]]></description>
        <link>https://medium.com/@nate510?source=rss-12e694b81dd8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*v9xonwKsQhPOrnF6.jpeg</url>
            <title>Stories by Nathan Smith on Medium</title>
            <link>https://medium.com/@nate510?source=rss-12e694b81dd8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:15:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nate510/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Structuring Go gRPC microservices]]></title>
            <link>https://medium.com/@nate510/structuring-go-grpc-microservices-dd176fdf28d0?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd176fdf28d0</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[grpc]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Sun, 19 Jul 2020 22:56:12 GMT</pubDate>
            <atom:updated>2020-07-20T15:53:45.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*P0z3ortvUH4gZtIyj4al_A.png" /></figure><p>Protocol Buffers and gRPC are popular technologies for defining microservices that communicate efficiently over the network. Many companies build gRPC microservices in Go. Some have even published the frameworks they have developed, such as the <a href="https://github.com/nytimes/gizmo">New York Times’ Gizmo</a>, which uses <a href="https://github.com/go-kit/kit">go-kit</a> under the hood.</p><p>There are many great open source libraries out there, but as with many things in Go, rolling your own implementation is easy, fun, and surprisingly simple. In this tutorial, I will outline how we structure gRPC microservices at my company, <a href="https://weareher.com/">HER</a>. All the code discussed below is <a href="https://github.com/neocortical/mysvc">available on Github</a>.</p><h3>Motiviation</h3><p>There are surprisingly few examples of how to structure gRPC services in Go. I found one <a href="https://medium.com/@amsokol.com/tutorial-how-to-develop-go-grpc-microservice-with-http-rest-endpoint-middleware-kubernetes-daebb36a97e9">tutorial by Aleksandr Sokolovskii</a> that is very good and quite thorough. However, I didn’t find all of the design decisions to be practical for my purposes, so I developed my own template for laying things out. Hopefully between the two guides, you’ll find something that works well for you!</p><h3>Prerequisites</h3><p>It is assumed you are familiar with Go and can create a new project, compile, etc. I will be including a lot of code examples, but I won’t be going over things like compiling, using go mod, etc.</p><h3>Defining Our Service</h3><p>I’m going to create a service for loading users for my awesome (hypothetical) app. I’m not going to think about any gRPC stuff yet. I will write the main service interface in a file called service.go in the root directory. It will consist of two methods, one to load a single user by their ID and another that accepts multiple user IDs and returns a map of Users keyed by their IDs. The single user method returns an error if no user is found while the multiple user method will just return an empty map in that case.</p><pre>package mysvc</pre><pre>// ErrNotFound signifies that a single requested object was not found.<br> var ErrNotFound = errors.New(&quot;not found&quot;)</pre><pre>// User is a user business object.<br>type User struct {<br>   ID int64<br>   Name string<br>}</pre><pre>// Service defines the interface exposed by this package.<br>type Service interface {<br>   GetUser(id int64) (User, error)<br>   GetUsers(ids []int64) (map[int64]User, error)<br>}</pre><p>I want to write the service definition first because I want to think about the business logic first and foremost. I place it in the root directory because I want the package structure to have an intuitive hierarchy (you’ll see what I mean in a second). I also want to have a simple import path and naming when I’m referring to business objects such as user.Service and user.User (this stutters a bit, but it’s clear).</p><h3>Implementing the Service</h3><p>The next thing I am going to do is implement the core of the service. This means writing all the code that will accomplish what I want my microservice to do. I will create a sub-package and file called core/service.go with the following code:</p><pre>package core</pre><pre>import (<br>  &quot;errors&quot;</pre><pre>  &quot;github.com/neocortical/mysvc&quot;<br>)</pre><pre>type service struct {<br>  // a database dependency would go here but instead we&#39;re going to have a static map<br>  m map[int64]mysvc.User<br>}</pre><pre>// NewService instantiates a new Service.<br>func NewService( /* a database connection would be injected here */ ) mysvc.Service {<br>  return &amp;service{<br>    m: map[int64]mysvc.User{<br>      1: {ID: 1, Name: &quot;Alice&quot;},<br>      2: {ID: 2, Name: &quot;Bob&quot;},<br>      3: {ID: 3, Name: &quot;Carol&quot;},<br>    },<br>  }<br>}</pre><pre>func (s *service) GetUser(id int64) (result mysvc.User, err error) {<br>  // instead of querying a database, we just query our static map<br>  if result, ok := s.m[id]; ok {<br>    return result, nil<br>  }</pre><pre>  return result, mysvc.ErrNotFound<br>}</pre><pre>func (s *service) GetUsers(ids []int64) (result map[int64]mysvc.User, err error) {<br>  // always a good idea to return non-nil maps to avoid nil pointer dereferences<br>  result = map[int64]mysvc.User{}</pre><pre>  for _, id := range ids {<br>    if u, ok := s.m[id]; ok {<br>      result[id] = u<br>    }<br>  }</pre><pre>  return<br>}</pre><p>At this point I have a working implementation of the code I care about. If I decide that I don’t want to create a separate microservice after all, I can just write userService := mysvc.NewService() in my app to get the functionality I need (again, in a real-world example you would be required to configure the service’s dependencies as well).</p><h3>Adding gRPC</h3><p>Now it’s time to create the gRPC service definition. This design pattern treats gRPC as a transport layer only, keeping it entirely separate from our business objects and logic. It may feel counterintuitive to create two different interfaces for the same thing, but this allows us to reason about our core service and correct gRPC behavior in isolation, producing a cleaner, easier to maintain end product.</p><p>I’m going to create a sub-package called grpc and add a protobuf file called service.proto:</p><pre>syntax = &quot;proto3&quot;;</pre><pre>package grpc;</pre><pre>service UserService {<br>  rpc GetUsers (GetUsersRequest) returns (GetUsersResponse) {}<br>}</pre><pre>message User {<br>  int64 id = 1;<br>  string name = 2;<br>}</pre><pre>message GetUsersRequest {<br>  repeated int64 ids = 1; <br>}<br>  <br>message GetUsersResponse {<br>  repeated User users = 1; <br>}</pre><p>Hey wait, I’ve only added half of the user service interface and it doesn’t even match, what gives? This is a matter of choice, but for this example I wanted to demonstrate that you can use the glue code between the gRPC layer and your core service layer to map and simplify the gRPC code you need to write.</p><p>Next, I will generate the Go gRPC code using protoc. Getting protoc installed is beyond the scope of this tutorial, but once it’s installed you just need to run go install google.golang.org/protobuf/cmd/protoc-gen-go to add the Go plugin. Check out this <a href="https://developers.google.com/protocol-buffers/docs/gotutorial">Protocol Buffers guide on Google’s developer website</a> for more info.</p><p>To generate the Go code, I just need to cd into the grpc directory and run:</p><pre>protoc --go_out=plugins=grpc:. *.proto</pre><p>I always forget the exact command, so I highly recommend creating a small run script or README to document it!</p><p>Now I have a Go file in the same directory called service.pb.go that contains all the autogenerated protobufs code I will need to wire up my gRPC service. Let’s do that now.</p><h3>gRPC Service: The Controller</h3><p>The first thing I’m going to write is the controller that will implement the gRPC service method I created. I’m going to create a new sub-package and file called grpc/service/controller.go with the following code:</p><pre>package main</pre><pre>import (<br>  &quot;context&quot;</pre><pre>  &quot;github.com/neocortical/mysvc&quot;<br>  mysvcgrpc &quot;github.com/neocortical/mysvc/grpc&quot;<br>)</pre><pre>// userServiceController implements the gRPC UserServiceServer interface.<br>type userServiceController struct {<br>  userService mysvc.Service<br>}</pre><pre>// NewUserServiceController instantiates a new UserServiceServer.<br>func NewUserServiceController(userService mysvc.Service) mysvcgrpc.UserServiceServer {<br>  return &amp;userServiceController{<br>    userService: userService,<br>  }<br>}</pre><pre>// GetUsers calls the core service&#39;s GetUsers method and maps the result to a grpc service response.<br>func (ctlr *userServiceController) GetUsers(ctx context.Context, req *mysvcgrpc.GetUsersRequest) (resp *mysvcgrpc.GetUsersResponse, err error) {<br>  resultMap, err := ctlr.userService.GetUsers(req.GetIds())<br>  if err != nil {<br>    return<br>  }</pre><pre>  resp := &amp;mysvcgrpc.GetUsersResponse{}<br>  for _, u := range resultMap {<br>    resp.Users = append(resp.Users, marshalUser(&amp;u))<br>  }</pre><pre>  return<br>}</pre><pre>// marshalUser marshals a business object User into a gRPC layer User.<br>func marshalUser(u *mysvc.User) *mysvcgrpc.User {<br>  return &amp;mysvcgrpc.User{Id: u.ID, Name: u.Name}<br>}</pre><p>This code implements the service side of our gRPC contract. It is simply a gRPC-aware wrapper around our core user service whose job it is to marshal the core service’s responses for transport via gRPC. This may seem a little cumbersome now, but we’ll see why I do things this way in a minute.</p><h3>gRPC Service: The Server</h3><p>But first, let’s start our service. We need a gRPC server. I will add a simple main function in grpc/server/main.go. It’s job is to wire up the core service, inject it into the controller, and start a gRPC server.</p><pre>package main</pre><pre>import (<br>  &quot;net&quot;<br>  &quot;os&quot;</pre><pre>  mysvccore &quot;github.com/neocortical/mysvc/core&quot;<br>  mysvcgrpc &quot;github.com/neocortical/mysvc/grpc&quot;<br>  &quot;google.golang.org/grpc&quot;<br>  &quot;google.golang.org/grpc/reflection&quot;<br>)</pre><pre>func main() {</pre><pre>  // configure our core service<br>  userService := mysvccore.NewService()</pre><pre>  // configure our gRPC service controller<br>  userServiceController := NewUserServiceController(userService)</pre><pre>  // start a gRPC server<br>  server := grpc.NewServer()<br>  mysvcgrpc.RegisterUserServiceServer(server, userServiceController)<br>  reflection.Register(server)</pre><pre>  con, err := net.Listen(&quot;tcp&quot;, os.Getenv(&quot;GRPC_ADDR&quot;))<br>  if err != nil {<br>    panic(err)<br>  }</pre><pre>  err = server.Serve(con)<br>  if err != nil {<br>    panic(err)<br>  }<br>}</pre><p>Now, we can build and run our gRPC service with the following commands:</p><pre>cd grpc/server<br>go build -o ./mysvc .<br>GRPC_ADDR=&quot;:9000&quot; ./mysvc</pre><p>Voilà — we have a gRPC user service up and running! But how do we talk to it?</p><h3>gRPC Service: The Client</h3><p>The last thing we need to build is the gRPC client interface. This is also the core service interface that we will use to call our microservice. Its job is to mediate between calls to the service and the gRPC transport layer underneath. This is where we do all the cool stuff like mapping our two service methods down to one gRPC call and handling the details of gRPC communication. I’ll add a sub-package and file called grpc/client/client.go:</p><pre>package client</pre><pre>import (<br>  &quot;context&quot;<br>  &quot;time&quot;</pre><pre>  &quot;github.com/neocortical/mysvc&quot;<br>  mysvcgrpc &quot;github.com/neocortical/mysvc/grpc&quot;<br>  &quot;google.golang.org/grpc&quot;<br>)</pre><pre>var defaultRequestTimeout = time.Second * 10</pre><pre>type grpcService struct {<br>  grpcClient mysvcgrpc.UserServiceClient<br>}</pre><pre>// NewGRPCService creates a new gRPC user service connection using the specified connection string.<br>func NewGRPCService(connString string) (mysvc.Service, error) {<br>  conn, err := grpc.Dial(connString, grpc.WithInsecure())<br>  if err != nil {<br>    return nil, err<br>  }</pre><pre>  return &amp;grpcService{grpcClient: mysvcgrpc.NewUserServiceClient(conn)}, nil<br>}</pre><pre>func (s *grpcService) GetUsers(ids []int64) (result map[int64]mysvc.User, err error) {<br>  result = map[int64]mysvc.User{}</pre><pre>  req := &amp;mysvcgrpc.GetUsersRequest{<br>    Ids: ids,<br>  }</pre><pre>  ctx, cancelFunc := context.WithTimeout(context.Background(), defaultRequestTimeout)<br>  defer cancelFunc()<br>  resp, err := s.grpcClient.GetUsers(ctx, req)<br>  if err != nil {<br>    return<br>  }</pre><pre>  for _, grpcUser := range resp.GetUsers() {<br>    u := unmarshalUser(grpcUser)<br>    result[u.ID] = u<br>  }<br>  return<br>}</pre><pre>func (s *grpcService) GetUser(id int64) (result mysvc.User, err error) {<br>  req := &amp;mysvcgrpc.GetUsersRequest{<br>    Ids: []int64{id},<br>  }</pre><pre>  ctx, cancelFunc := context.WithTimeout(context.Background(), defaultRequestTimeout)<br>  defer cancelFunc()<br>  resp, err := s.grpcClient.GetUsers(ctx, req)<br>  if err != nil {<br>    return<br>  }</pre><pre>  for _, grpcUser := range resp.GetUsers() {<br>    // sanity check: only the requested user should be present in results<br>    if grpcUser.GetId() == id {<br>      return unmarshalUser(grpcUser), nil<br>    }<br>  }</pre><pre>  return result, mysvc.ErrNotFound<br>}</pre><pre>func unmarshalUser(grpcUser *mysvcgrpc.User) (result mysvc.User) {<br>  result.ID = grpcUser.Id<br>  result.Name = grpcUser.Name<br>  return<br>}</pre><p>There are a couple things worth noting here. I’m mapping the two service methods onto a single gRPC method while preserving the different behavior (GetUser() returns an error if no user is found whereas GetUsers() returns an empty map and no error). Also, I’m able to work around the lack of maps in protobufs in a way that is transparent to the user of the service.</p><p>Most importantly, all gRPC-specific code is encapsulated here so that downstream developers don’t have to think about it. The package chooses a reasonable timeout deadline for the context and manages calling its cancel function opaquely. I could also expose the ability to set the timeout and/or other gRPC-related configuration here. The goal is that the only place your code needs to know it is using the gRPC version of the service and not the core service is where it is configured.</p><h3>Conclusion</h3><p>I hope you found this tutorial useful. All the code is <a href="https://github.com/neocortical/mysvc">available on Github</a>. The README goes into a little more detail and has instructions for running the service as well as a command line tool to test out the gRPC client.</p><p>If you have any questions, comments, or suggestions please feel free to comment below or <a href="https://twitter.com/NateAtHome">reach out on Twitter</a>. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd176fdf28d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How President Trump Can Heal a Divided America]]></title>
            <link>https://medium.com/@nate510/in-his-victory-speech-on-tuesday-night-donald-trump-offered-an-olive-branch-to-the-people-that-did-538aa8a0135a?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/538aa8a0135a</guid>
            <category><![CDATA[donald-trump]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[2016-election]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Thu, 10 Nov 2016 07:27:02 GMT</pubDate>
            <atom:updated>2016-11-10T07:35:18.503Z</atom:updated>
            <content:encoded><![CDATA[<p>In his victory speech on Tuesday night, Donald Trump offered an olive branch to the people that did not support him:</p><blockquote>“Now it’s time for America to bind the wounds of division; have to get together. To all Republicans and Democrats and independents across this nation, I say it is time for us to come together as one united people.”</blockquote><p>For the majority of voters who voted against Trump, and for all those he has maligned over the course of his campaign, those were kind words. To some, they even signaled hope that President Trump would be a different man than Trump the campaigner — that the vitriolic and often patently offensive outbursts that punctuated his march to the presidency are not reflective of the man that will lead our nation over the next four years.</p><p>But if Donald Trump was being sincere about reuniting our fractured electorate, then he must do more than pay lip service to the millions of Americans that he offended during his campaign.</p><p>Actions speak louder than words, and the next four years will illuminate Trump’s commitment to “be president for all Americans.” But the singularly divisive nature of Trump’s campaign has left many deeply incredulous at the idea that he will be a fair and just commander in chief.</p><p>Let me spell out what I believe Trump must say — publicly, before his inauguration — to even begin to heal the vast divide this presidential election has created.</p><p>First and foremost, Donald Trump must publicly and unequivocally disavow the endorsement given him by the Ku Klux Klan. It it indisputable that during his campaign he enjoyed without rebuke support from those in our country that view African Americans and other races as inferior. Racism persists in this nation on many levels, but to tacitly accept the support of outwardly racist groups such as the KKK is impossible to square with his Tuesday message of harmony.</p><p>Trump must state explicitly that any and all hate groups are not welcome in his base. He must reject his supporters that feel otherwise. There is simply no unity possible until this is done.</p><p>Second, Donald Trump must state for the record that he views sexual harassment to be abhorrent in all its forms. During his campaign, Trump expressed reluctant regret for comments he made regarding women, but he has never stated clearly his support for women’s rights nor his opposition to those who would invade those rights.</p><p>Women make up over half of our country. If Donald Trump wants to represent all of us, he must reassure women that he respects them and will fight for them.</p><p>Third, Donald Trump must clarify his support for legal immigrants to the US. Over the course of his campaign, he deeply offended many Americans with his rhetoric regarding Muslims, Mexican Americans, and other groups. Our country was founded by immigrants, expanded by immigrants, and has a long history of offering a better way of life to those found wanting in their native countries.</p><p>Whatever his policy decisions regarding new immigration, Donald Trump must explicitly reiterate his support for people of all colors and faiths (more than 10% of us) who are in this country legally.</p><p>Fourth, Donald Trump must state without reservation that he is not an anti-Semite. Though never overtly stated by him, his campaign touched on anti-Semitic themes that cannot plausibly be explained short of pandering to a certain constituency.</p><p>The specter of anti-Jewish sentiment has haunted the world for far too long. Given the events of the past year, it is sadly the job of the president-elect to reiterate that those sentiments have no place in American democracy.</p><p>Lastly, Donald Trump must walk back his claims that he will have Hillary Clinton prosecuted once he takes office. The widespread chants of “Lock her up!” at his rallies might be begrudgingly forgiven as parroted propaganda by diehard supporters, but we a nation of laws and democratic norms. The president of the United States cannot set the precedent of vindictively persecuting his political opponents; that is the stuff of petty dictators and banana republics.</p><p>Trump must state that he will respect the findings of the FBI and the attorney general with regard to the investigation of Hillary Clinton’s email server while secretary of state. He must pledge to not propose or support any investigation into Hillary Clinton’s affairs beyond what is deemed warranted by the FBI and the Justice Department.</p><p>Donald Trump’s campaign for president was underscored by a divisiveness not seen since the Great Depression. Some are calling it the most divisive campaign in US history. To walk away the victor in such a campaign, it takes more than vague preamble to one’s victory speech to rebuild a nation. It takes strength and dedication.</p><p>The above are the necessary assertions I believe Donald Trump must make prior to taking office in order to build any kind of unified support for his presidency. To simply rest on his election-night statement is not sufficient by any stretch of the imagination. Trump was correct in his Tuesday night speech that a presidential campaign is “nasty and it’s tough.” To be the leader of the most powerful democracy on the planet is far tougher and it remains to be seen if Donald Trump can rise to the challenge.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=538aa8a0135a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[UC Davis, meet the Streisand Effect]]></title>
            <link>https://medium.com/@nate510/uc-davis-meet-the-streisand-effect-85c3eeaaa2da?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/85c3eeaaa2da</guid>
            <category><![CDATA[free-speech]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Fri, 15 Apr 2016 20:11:26 GMT</pubDate>
            <atom:updated>2016-04-15T20:11:26.261Z</atom:updated>
            <content:encoded><![CDATA[<p>In November 2011, the UC Davis police responded violently to an Occupy protest on campus. You may remember reading about how UC policeman Lt. John Pike infamously hosed down peaceful, seated protesters with pepper spray.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*ZQFZUwmL0DXDfXAqE6l4jA.jpeg" /><figcaption>UC Davis Lt. John Pike uses pepper spray on nonviolent demonstrators. UCD Chancellor Linda P.B. Katehi apologized for the incident but recently oversaw efforts to use school money to cover up the event.</figcaption></figure><p>UCD Chancellor Linda P.B. Katehi apologized for the incident a few days later, saying that the campus police acted against her orders. Two days ago, however, the Sacramento Bee reported that UC Davis has spent at least $175,000 on PR efforts to <em>improve</em> search results related to the incident. Basically, they used taxpayer dollars to embark on a whitewashing campaign.</p><p>Davis is my hometown, and UC Davis is my alma mater. So I thought I’d use a little be of my own time to help improve search results by ensuring that ill-advised whitewashing campaigns like this backfire.</p><p>Go Aggies!</p><p>You can read more about the <a href="https://en.wikipedia.org/wiki/UC_Davis_pepper-spray_incident">UC Davis pepper-spray incident on Wikipedia</a>.</p><p>You can read about the <a href="http://www.sacbee.com/news/local/education/article71659992.html">attempted coverup at SacBee.com</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85c3eeaaa2da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic JSON umarshalling in Go]]></title>
            <link>https://medium.com/@nate510/dynamic-json-umarshalling-in-go-88095561d6a0?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/88095561d6a0</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Tue, 02 Feb 2016 19:18:16 GMT</pubDate>
            <atom:updated>2016-02-02T19:18:16.578Z</atom:updated>
            <content:encoded><![CDATA[<p>When dealing with third-party JSON data (e.g. from an API), it is sometimes the case that field values can have multiple types depending on the context. A common case is when a field can hold either the ID of a sub-object or embed the entire object in an expanded view.</p><p>Here is an example of the Spacely Sprockets API:</p><pre>// Simple API response<br>{<br>  “desc”: “Sub-object is an ID”, <br>  “sprocket”: 42<br>}</pre><pre>// Expanded API response<br>{<br>  “desc”: “Sub-object is an object”, <br>  “sprocket”: {<br>    “id”: 42,<br>    “size”: “large”,<br>    “gears”: 15 <br>  }<br>}</pre><p>This is bad API design. Please don’t do it. It forces client developers to jump through hoops to unmarshal data. This is especially problematic for statically typed languages like Scala and Go, because your data object’s field must be somehow typed to handle more than one type of data.</p><p>For those wondering what a better schema would be, I would point to the Facebook Graph API. They use partial objects everywhere, consistently containing ID, Name, and CreateTime fields. This object can be expanded via query shaping without requiring a different object on the client side to unmarshal. Another alternative is to have two fields (e.g. sprocket and sprocket_id) that store an object and its ID respectively, but this is redundant, can be error-prone, and requires flow-control logic on the client side to figure out which field was passed.</p><p>In this post, I’m going to demonstrate a reasonably elegant way of dealing with dynamic JSON field types in Go. The code used in the examples is <a href="https://github.com/neocortical/flexjson">available in my Github repo</a>.</p><h4>Unmarshalling</h4><p>In our example, we need to handle two cases for the “sprocket” field:</p><ol><li>sprocket is an integer ID</li><li>sprocket is a Sprocket object</li></ol><p>Since we can’t have a variable of both types, we’re going to build a Sprocket object and a SprocketWrapper object that embeds a Sprocket (I’ll explain why in a second):</p><pre>// Example is the top-level API response for this demo<br>type Example struct {<br>  Desc string `json:”desc”`<br>  Sprocket SprocketWrapper `json:”sprocket”`<br>}</pre><pre>// Sprocket is a sprocket object in our business logic<br>type Sprocket struct {<br>  ID int `json:”id”`<br>  Size string `json:”size,omitempty”`<br>  Gears int `json:”gears,omitempty”`<br>}</pre><pre>// SprocketWrapper wraps a Sprocket object with <br>// additional functionality<br>type SprocketWrapper struct {<br>  Sprocket<br>  Partial bool `json:”-”`<br>}</pre><p>Notice that the top-level API object specifies “sprocket” as type SprocketWrapper. When when our code attempts to unmarshal a JSON response…</p><pre>var example flexjson.Example<br>err := json.Unmarshal([]byte(exampleJSON), &amp;example)</pre><p>… Go’s json package will attempt to call UnmarshalJSON() on our wrapper object. In order to correctly unmarshal either an ID or an object, we can now override the default behavior:</p><pre>func (w *SprocketWrapper) UnmarshalJSON(data []byte) error {<br>  if id, err := strconv.Atoi(string(data)); err == nil {<br>    w.ID = id<br>    w.Partial = true<br>    return nil<br>  }<br>  return json.Unmarshal(data, &amp;w.Sprocket)<br>}</pre><p>If an ID is passed, as in the first example response above, we detect it by calling Atoi() on the raw data and store it in the Sprocket object’s ID field. We also mark the object as being a partially populated object using a flag. This allows our API client code to differentiate between a partial object and a sprocket with no Size or Gear information. If an object is passed, the attempt to convert the raw data to an int fails, and we fall through to the packages default unmarshal method on the underlying Sprocket object.</p><p>Notice that having the wrapper allows us to preserve the default unmarshalling behavior for Sprocket. This makes things much cleaner.</p><h4>Marshalling</h4><p>If you need to marshal your business objects back into JSON for any reason, there are two options for dealing with a partial Sprocket:</p><ol><li>Marshal into an object with only the ID field populated and mark that object as partial in the JSON</li><li>Mimic the API and output an integer ID</li></ol><p>While the first method cleans up the technical debt of the poorly constructed API, the second method may be necessary if you need to PUT/POST back to the same API. It’s also more fun to implement, so let’s do that.</p><p>To achieve our goal, let’s write a custom MarshalJSON() method for our wrapper that is aware of partial data:</p><pre>func (w SprocketWrapper) MarshalJSON() ([]byte, error) {<br>  if w.ID == 0 {<br>    return []byte(“null”), nil<br>  }<br>  if w.Partial {<br>    return []byte(strconv.Itoa(w.ID)), nil<br>  }<br>  return json.Marshal(w.Sprocket)<br>}</pre><p>This method can output three different things:</p><ol><li>If the object is empty (i.e. it’s ID field is zero), output JSON null</li><li>If the object was constructed as a partial, only output the integer ID</li><li>If the object is populated, fall through to the default marshal function and output the object’s JSON representation.</li></ol><p>Again, notice that we still have Sprocket’s default marshal behavior, thanks to our wrapper object. Additionally, the Sprocket object itself isn’t cluttered up with any fields or methods that are not part of its business logic. This decouples API-level hackery from your core business logic code. Since Go’s MarshalJSON() hook allows any type of data to be output, it’s relatively straightforward to switch between integer and object values when constructing your JSON.</p><h4>Conclusion</h4><p>The JSON encoding format allows for schema-less objects, which sometimes makes life difficult for developers. This technique uses Go’s well-designed JSON package to allow for custom object marshalling/unmarshalling with just a few lines of code, and without polluting your primary business objects with JSON format-specific fields or code. Hopefully someone finds this useful, and hopefully someone else finds this before they design their API to have dynamically typed fields. :)</p><p>Check out <a href="https://github.com/neocortical/flexjson">the Github repo</a> for a working example of unmarshalling and marshalling the different types of data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88095561d6a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 2286 Bug]]></title>
            <link>https://medium.com/@nate510/the-2286-bug-65697bb1b908?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/65697bb1b908</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Sun, 24 Jan 2016 20:23:04 GMT</pubDate>
            <atom:updated>2016-01-24T20:23:04.720Z</atom:updated>
            <content:encoded><![CDATA[<p>Unix timestamps represent time as seconds elapsed since January 1, 1970. The current-ish Unix timestamp as of this writing is 1453665679. On November 20, 2286, another digit will be added to this number as it reaches 10000000000. This could cause issues for some programs, specifically those programs that sort lexicographically on a string representation of this number. How many programs do this that will ever need to represent dates later than Nov 20, 2286? Who knows; it’s probably not very common.</p><p>A better-known and possibly more common bug will arise in 2038, when the Unix timestamp hits 2147483647, the maximum value that can be represented by a signed, 32-bit integer. This number also happens to be the 8th <a href="https://en.wikipedia.org/wiki/Mersenne_prime">Mersenne prime</a>, according to Wikipedia. Then there’s the maximum value that can be stored in an unsigned, 32-bit integer, 4294967295, which will be hit by Unix time on February 7, 2106.</p><p>Hopefully by the time these bugs strike, most computing platforms will be using 64-bit integers, which can safely represent Unix timestamps until long after the sun cooks Earth to a crisp.</p><p>But the 2286 bug is interesting because it doesn’t have anything to with overflow, which can be generally fixed. Lexicographic ordering of integers as been a source of bugs for a long time because data is messy and often formatted incorrectly. In the early 23rd century, I imagine Fortran programmers at banks will be gnashing their teeth over it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65697bb1b908" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t use Go’s default HTTP client (in production)]]></title>
            <link>https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/4804cb19f779</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Fri, 22 Jan 2016 22:19:11 GMT</pubDate>
            <atom:updated>2017-01-09T08:23:08.430Z</atom:updated>
            <content:encoded><![CDATA[<p>Writing Go programs that talk to services over HTTP is easy and fun. I’ve written numerous API client packages and I find it an enjoyable task. However, I have run into a pitfall that is easy to fall into and can crash your program very quickly: the default HTTP client.</p><p><strong>TL;DR</strong>: Go’s http package doesn’t specify request timeouts by default, allowing services to hijack your goroutines. <strong>Always specify a custom http.Client</strong> when connecting to outside services.</p><h4>The Problem by Example</h4><p>Let’s say you want to want to talk to spacely-sprockets.com via their nice JSON REST API and view a list of available sprockets. In Go, you might do something like:</p><pre>// error checking omitted for brevity<br>var sprockets SprocketsResponse<br>response, _ := http.Get(&quot;spacely-sprockets.com/api/sprockets&quot;)<br>buf, _ := ioutil.ReadAll(response.Body)<br>json.Unmarshal(buf, &amp;sprockets)</pre><p>You write your code (with proper error handling, please), compile, and run. Everything works great. Now, you take your API package and plug it into a web application. One page of your app shows a list of Spacely Sprockets inventory to users by making a call to the API.</p><p>Everything is going great until one day your app stops responding. You look in the logs, but there’s nothing to indicate a problem. You check your monitoring tools, but CPU, memory, and I/O all look reasonable leading up to the outage. You spin up a sandbox and it seems to work fine. What gives?</p><p>Frustrated, you check Twitter and notice a tweet from the Spacely Sprockets dev team saying that they experienced a brief outage, but that everything is now back to normal. You check their API status page and see that the outage began a few minutes before yours. That seems like an unlikely coincidence, but you can’t quite figure out how it’s related, since your API code handles errors gracefully. You’re still no closer to figuring out the issue.</p><h4>The Go HTTP package</h4><p>Go’s HTTP package uses a struct called Client to manage the internals of communicating over HTTP(S). Clients are concurrency-safe objects that contain configuration, manage TCP state, handle cookies, etc. When you use http.Get(url), you are using the http.DefaultClient, a package variable that defines the default configuration for a client. The declaration for this is:</p><pre>var DefaultClient = &amp;<a href="https://golang.org/pkg/net/http/#Client">Client</a>{}</pre><p>Among other things, http.Client configures a timeout that short-circuits long-running connections. The default for this value is 0, which is interpreted as “no timeout”. This may be a sensible default for the package, but it is a nasty pitfall and the cause of our application falling over in the above example. As it turns out, Spacely Sprockets’ API outage caused connection attempts to hang (this doesn’t always happen, but it does in our example). They will continue to hang for as long as the malfunctioning server decides to wait. Because API calls were being made to serve user requests, this caused the goroutines serving user requests to hang as well. Once enough users hit the sprockets page, the app fell over, most likely due to resource limits being reached.</p><p>Here is a simple Go program that demonstrates the issue:</p><pre>package main</pre><pre>import (<br>  “fmt”<br>  “net/http”<br>  “net/http/httptest”<br>  “time”<br>)</pre><pre>func main() {<br>  svr := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {<br>    time.Sleep(time.Hour)<br>  }))<br>  defer svr.Close()</pre><pre>  fmt.Println(“making request”)<br>  http.Get(svr.URL)<br>  fmt.Println(“finished request”)<br>}</pre><p>When run, this program will make a request to a server that will sleep for an hour. Consequently, the program will wait for one hour and then exit.</p><h4>The Solution</h4><p>The solution to this problem is to always define an http.Client with a sensible timeout for your use case. Here is an example:</p><pre>var netClient = &amp;http.Client{<br>  Timeout: time.Second * 10,<br>}</pre><pre>response, _ := netClient.Get(url)</pre><p>This sets a 10 second timeout on requests made to the endpoint. If the API server exceeds the timeout, Get() will return with the error:</p><pre>&amp;httpError{<br>  err:     err.Error() + &quot; (Client.Timeout exceeded while awaiting headers)&quot;,<br>  timeout: true,<br>}</pre><p>If you need finer-grained control over the request lifecycle, you can additionally specify a custom net.Transport and net.Dialer. A Transport is a struct used by clients to manage the underlying TCP connection and it’s Dialer is a struct that manages the establishment of the connection. Go’s net package has a default Transport and Dialer as well. Here’s an example of using custom ones:</p><pre>var netTransport = &amp;http.Transport{<br>  Dial: (&amp;net.Dialer{<br>    Timeout: 5 * time.Second,<br>  }).Dial,<br>  TLSHandshakeTimeout: 5 * time.Second,<br>}</pre><pre>var netClient = &amp;http.Client{<br>  Timeout: time.Second * 10,<br>  Transport: netTransport,<br>}</pre><pre>response, _ := netClient.Get(url)</pre><p>This code will cap the TCP connect and TLS handshake timeouts, as well as establishing an end-to-end request timeout. There are other configuration options such as keep-alive timeouts you can play with if needed.</p><h4>Conclusion</h4><p>Go’s net and http packages are a well-thought out, convenient base for communicating over HTTP(S). However, the lack of a default timeout for requests is an easy pitfall to fall into, because the package provides convenience methods like http.Get(url). Some languages (e.g. Java) have the same issue, others (e.g. Ruby has a default 60 second read timeout) do not. Not setting a request timeout when contacting a remote service puts your application at the mercy of that service. A malfunctioning or malicious service can hang on to your connection forever, potentially starving your application.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4804cb19f779" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Open Letter to the FCC on Net Neutrality]]></title>
            <link>https://medium.com/@nate510/an-open-letter-to-the-fcc-on-net-neutrality-cd0e26cf8edd?source=rss-12e694b81dd8------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd0e26cf8edd</guid>
            <dc:creator><![CDATA[Nathan Smith]]></dc:creator>
            <pubDate>Sun, 27 Apr 2014 23:40:27 GMT</pubDate>
            <atom:updated>2014-04-27T23:40:27.930Z</atom:updated>
            <content:encoded><![CDATA[<h4>Proposed rule changes by the FCC threaten to allow service providers to charge for preferential deliverability. These changes will hurt both consumers and small businesses and should not be enacted.</h4><p>To whom it may concern,</p><p>I am writing today to oppose the FCC’s proposed changes for the governance of Internet service providers regarding the delivery of information across their networks. Reports indicate that the FCC intends to allow these providers to give preferential treatment to certain kinds of traffic, dependent upon source. This change has been heavily lobbied for by providers, as it will allow them to both charge for preferential access as well as to offer such access to their own content-creating properties for free. However, these changes will be bad for consumers and for the health of the Internet, and the should not be adopted.</p><p>On June 28, 2006, Senator Ted Stevens of Alaska uttered a now-infamous statement regarding how the infrastructure of the Internet works, describing the global electronic network as a “series of tubes” that could be “filled” by the large amount of information that flows through them. Those comments became the butt of many jokes—some fair, some less so. To the technically minded, Ted Stevens clearly did not have a strong understanding of how the Internet functioned. To others, his comments simply sounded silly.</p><p>The context of Sen. Stevens’ speech was a debate over legislation that would mandate the equal treatment of Internet traffic—that is, data such as email, web pages, streaming audio and video, etc.—by Internet service providers such as Comcast, Verizon, AT&amp;T, and others. All of the aforementioned companies lobbied Washington heavily to prevent the legislation from passing, and Sen. Stevens was arguing in their favor when he made his unfortunate remarks. The primary arguments put forth by providers center on the ideas that preferential treatment of certain kinds of data is necessary to promote innovation and will ultimately benefit consumers. Both of these arguments are false and the evidence used to support them is misleading.</p><p>Ted Stevens was right: Internet service providers are in the business of providing a service infrastructure, similar in many ways to the pipes of a water system or the wires of an electrical grid. Similar to these other services, consumer demand for Internet service is growing. Dissimilar to water and electrical services however, that demand is currently growing at an enormous rate. This puts pressure on providers to deploy ever faster software and hardware to keep up with demand. These improvements are being made by Internet service providers worldwide, and must also occur in the US for us to remain competitive in this key arena.</p><p>Allowing providers to throttle certain types of Internet traffic will disincentivise them from deploying upgraded network technologies such as fiberoptic and 4G networks. Instead of reinvesting their profits in their infrastructure, companies such as Comcast and AT&amp;T will be able to exploit a brand new market—charging for preferential content delivery—allowing them to extract additional profits from outdated infrastructure. In some cases, such as Comcast, it will also promote the anti-competitive practice of preferring their own content over that of their competitors. In a climate where much of the developed world is already enjoying better Internet speeds than the US, this will move the US even further to the rear. This will hurt consumers, while helping service providers increase their already high net profit margins.</p><p>Consumers will also be hurt directly by the preferential treatment given to content providers that pay service providers for deliverability guarantees. Netflix—a streaming video provider that is often cited as an example in these matters—will most likely elect to pay service providers for deliverability guarantees in an environment where such guarantees could be sold. This hurts consumers because Netflix will have a smaller percentage of its operating budget available for content creation, and infrastructure. It also hurts consumers because competitors to Netflix large and small will face an unnecessary barrier to competition in degraded deliverability over the network. This is a very real and inevitable effect of allowing preferential delivery. Netflix itself was once a nascent player in online content delivery, as were other popular content streaming services such as YouTube, Vimeo, Spotify, Pandora, etc. Their current popularity owes much to the level playing field that existed when they began delivering content. In short, the absence of equal deliverability guarantees will make it difficult for the next generation of great content content providers to find their audiences, directly hurting the consumer.</p><p>FCC rules that promote the principle of equal deliverability of content over the network serve to ensure a level playing field for all companies that wish to do business on the Internet. The degradation of these rules will create artificial barriers to many companies, hurting innovation, reducing quality of service to consumers, and entrenching the US’s current inferiority in consumer network speeds. I urge the FCC to rescind the proposed rule changes and continue on its previously steady course of protecting the unique and wonderful resource that is the open Internet.</p><p>Thank you for you time, and for protecting the tubes.</p><p>Sincerely,<br>Nathan Smith</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd0e26cf8edd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>