<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Betalo Engineering - Medium]]></title>
        <description><![CDATA[Building the future of Nordic banking - Medium]]></description>
        <link>https://medium.com/betalo-engineering?source=rss----b7f86917576b---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Betalo Engineering - Medium</title>
            <link>https://medium.com/betalo-engineering?source=rss----b7f86917576b---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:19:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/betalo-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Troubles automating iOS app releases (using Fastlane and Travis-CI)]]></title>
            <link>https://medium.com/betalo-engineering/troubles-automating-ios-app-releases-using-fastlane-and-travis-ci-957f8884064a?source=rss----b7f86917576b---4</link>
            <guid isPermaLink="false">https://medium.com/p/957f8884064a</guid>
            <category><![CDATA[fastlane]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[travis-ci]]></category>
            <dc:creator><![CDATA[Jason Fellin]]></dc:creator>
            <pubDate>Thu, 12 Apr 2018 09:36:09 GMT</pubDate>
            <atom:updated>2018-04-12T09:36:09.402Z</atom:updated>
            <content:encoded><![CDATA[<p><em>This post assumes you’re familiar using </em><a href="https://travis-ci.com/"><em>Travis</em></a><em> with </em><a href="https://fastlane.tools/"><em>Fastlane</em></a><em>. There are plenty of </em><a href="https://medium.com/practo-engineering/build-distribution-automation-with-fastlane-and-travis-ci-ios-f959dff2799f"><em>how</em></a><em>-</em><a href="http://artsy.github.io/blog/2017/07/31/fastlane-travis-weekly-deploys/"><em>tos</em></a><em> out there to get up to speed.</em></p><p>As the lead iOS developer at <a href="https://www.betalo.com/en">Betalo</a>, I wanted to find a way to offload the beta and production releases of our iOS app to free up our local machines for development. I had recently integrated Travis with our Github repo to run our tests on a every pull request. In the process, I came across the Travis API which allows you to <a href="https://docs.travis-ci.com/user/triggering-builds/">trigger</a> the build process independently from your default .travis.yml file. Cool, let Travis do the work.</p><p>I built a simple ReactJS web app to handle calling Travis. Along the way I hit a few bumps in the road that required work arounds, which I wanted to share with my fellow travelers.</p><h3>Automate build number bumps before pushing to <a href="https://fabric.io/kits/android/crashlytics">Crashlytics</a></h3><p>We use <a href="https://datasift.github.io/gitflow/IntroducingGitFlow.html">git flow</a> for our development and release process. One aspect of this process is requiring an approved pull request to merge into our develop, release and master branches. ANY change to the repo is essentially blocked. This makes automating build number bumps (<em>Crashlytics demands a unique build number for each release</em>) a royal pain in the pinky toe.</p><p>Like good little iOS developers, we were already using Fastlane to streamline our releases and take the pain out of <a href="https://docs.fastlane.tools/actions/match/">code signing</a>. You can <a href="https://docs.fastlane.tools/actions/increment_build_number/#increment_build_number">increment</a> the build number easily enough but in our case we needed to create a commit and push these changes back to Github. But how can we get these changes pushed to a protected branch? We came up with the following solution:</p><blockquote>Create a Github bot user, with admin rights, and clone the repo using a personal access token of said bot user.</blockquote><p>Here are the steps:</p><h4>1. Exclude admins from the approved pull request requirement</h4><p>If your branch is protected on Github, the settings screen for said branch will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mE-lCBpJ84adOtleCyxHAQ.png" /></figure><p>Leave the <strong><em>Include administrators</em> </strong>box unchecked. This allows users with admin status to push directly to the protected branch without an approved pull request.</p><h4>2. Create a <a href="https://stackoverflow.com/questions/29177623/what-is-a-bot-account-on-github?utm_medium=organic&amp;utm_source=google_rich_qa&amp;utm_campaign=google_rich_qa">bot user</a> with admin status on your repo</h4><p>When you have the user created, add it to the repository Settings &gt; Collaborators &amp; teams. Ours looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*4hDmBXc0hRKf1gszlnGvag.png" /></figure><p>We used a bot user that was already created by our backend team for the personal access token so it’s not attached to an actual person (you never know the longevity of your employees).</p><h4>2. Create a personal access token for the bot user</h4><p>To create the token on Github, log in is as the bot user and go to:</p><p><em>Settings &gt; Developer Settings &gt; Personal access tokens</em></p><p>Create one and copy it.</p><h4>3. Add the token as an <a href="https://en.wikipedia.org/wiki/Environment_variable">environment variable</a> in Travis</h4><p>Go to your repo’s Settings screen in Travis and find the Environment Variables section. Ours looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPo9VUU3ZySx6sOf9N29hA.png" /><figcaption>Environment variables section on Travis</figcaption></figure><p>The GITHUB_TOKEN is the bot user’s personal access token. Paste yours and name it whatever you want.</p><p>(Pro-tip: leave “Display value” off and the value is encrypted)</p><h4>4. Re-clone the repo in Travis with the above token</h4><p>As mentioned above, in order to have permission to push directly to a protected branch, we have to clone the repo using the personal access token of an admin user. That command looks something like this:</p><pre>// GITHUB_TOKEN is the ENV variable from above<br>git clone https://${GITHUB_TOKEN}@github.com/your-company/your-repo.git</pre><p>By default, Travis will clone whatever branch you tell it to when calling the <a href="https://developer.travis-ci.com/resource/requests#Requests">requests</a> endpoint. This happens before any of the <a href="https://docs.travis-ci.com/user/customizing-the-build/#The-Build-Lifecycle">build lifecycle</a> steps that you can hook into in your YAML file. There’s no way to override this clone script that I found. So just re-clone it!</p><p>I decided to hook into the before_install lifecycle step with this command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57fad4dac1cc5d2c45282f37d9c548ac/href">https://medium.com/media/57fad4dac1cc5d2c45282f37d9c548ac/href</a></iframe><p>(You could lower the depth if you want since we only need the most recent commit.)</p><p>If you’re confused by the syntax above, remember we’re using the Travis API to request the build. This means sending a JSON object with the custom build step(s). To issue more than one command per step, it’s a simple string array. You can review how Travis merges the commands in the JSON object with your .travis.yml file <a href="https://docs.travis-ci.com/user/triggering-builds/#Customizing-the-build-configuration">here</a>.</p><h4>5. Create a shell script ‘git push’ action in Fastlane</h4><p>Since we’re using Fastlane to do the actual <a href="https://github.com/SiarheiFedartsou/fastlane-plugin-versioning">build bump</a>, we chose to use a <a href="https://docs.fastlane.tools/actions/sh/">shell script</a> action to handle the git commit and push. You could just as easily have one lane handle the build number bump, issue the git commands, then call another lane to do the build and release. It feels cleaner to me to have Fastlane deal with all of it.</p><p>Since your bot user cloned the repo, the git push should succeed! 👏👏👏</p><p>…and now code signing fails, wah wah waaaaah.</p><h3>Make Travis play nice with <a href="https://docs.fastlane.tools/actions/match/">Match</a></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*cyFz-9jYbtEHMRVPbs9yFw.png" /></figure><p>After I had the above finished, I didn’t understand why Match kept failing. I followed Fastlane’s guide to <a href="https://docs.fastlane.tools/best-practices/continuous-integration/#travis-integration">integrate with Travis</a>. Everything checked out. What the heck?</p><p>A key component of using Match is storing the certs and profiles in a private repository. Key word, <em>private</em>. Of course! Travis needs permission to clone the repo when it’s private. So I changed our Match command to look something like this:</p><pre>match(type: &quot;development&quot;,<br>      app_identifier: bundle,<br>      force_for_new_devices: true,<br>      git_url: &quot;https://#{ENV[&#39;GITHUB_TOKEN&#39;]}<a href="http://twitter.com/github">@github</a>.com/your-company/ios-certificates.git&quot;</pre><p>There’s that GITHUB_TOKEN again. The same one we created for our bot user above. This’ll give Travis permission to clone it. Don’t forget to add the bot user as a collaborator on the ios-certificates repo. Read permission will be sufficient this time.</p><p>You’ll also need to add the MATCH_PASSWORD environment variable to Travis like you did in step 3 above. Everything else should just work.</p><h3>Is there anywhere to go from here?</h3><p>I know your life may feel complete after this read. Mine sure did when I was able to trigger an app release on my subway ride home.</p><p>Anyone else using Travis to release your iOS app? What have been your pain points? Let us know in the comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=957f8884064a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/betalo-engineering/troubles-automating-ios-app-releases-using-fastlane-and-travis-ci-957f8884064a">Troubles automating iOS app releases (using Fastlane and Travis-CI)</a> was originally published in <a href="https://medium.com/betalo-engineering">Betalo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Integrating with Third-Parties via a Go Forwarding HTTPS Proxy]]></title>
            <link>https://medium.com/betalo-engineering/integrating-with-third-parties-via-a-go-forwarding-https-proxy-6f0e5575d93e?source=rss----b7f86917576b---4</link>
            <guid isPermaLink="false">https://medium.com/p/6f0e5575d93e</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[https]]></category>
            <category><![CDATA[proxy-server]]></category>
            <dc:creator><![CDATA[Uwe Dauernheim]]></dc:creator>
            <pubDate>Mon, 19 Mar 2018 08:00:05 GMT</pubDate>
            <atom:updated>2018-03-19T08:00:05.081Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article we provide guidance for integrating with third-parties via a HTTPS proxy in production as well as establishing confidence during testing.</p><p>Head over to <a href="https://github.com/betalo-sweden/forwardingproxy">betalo-sweden/forwardingproxy</a> to see the final result.</p><h3>Use case</h3><p>If you provide a banking service to customers, it is likely that you have to integrate with third-parties such as other banks and payment providers. To guarantee a high level of security the participating parties often require the communication channel to be encrypted, destination and origin restricted. For web services, encryption is addressed by using HTTPS with transport layer security (TLS) or a virtual private network (VPN) connection. Destination restrictions are enforced by firewalls and load balancers with port and packet filtering. Origin restrictions are enforced by static IP addresses such as <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html">Elastic IPs</a> (EIP) provided by Amazon’s web services (AWS) or <a href="https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address">static external IP addresses</a> provided by Google Cloud Platform (GCP). If your services are already running within the realms of these cloud providers you can just connect your integrating service to these static IPs. However, if your services are hosted on a platform as a service (PaaS) provider such as <a href="https://heroku.com/">Heroku</a> you have to consider additional integration work to satisfy origin restriction. This is were a HTTPS proxy comes in to provide encapsulation.</p><h3>What is a HTTP/S proxy</h3><p>A proxy is a server or application that acts as an intermediary for requests from clients to destinations. An HTTP/S proxy is a proxy that supports HTTP request forwarding and HTTP tunneling. It commonly operates on Open Systems Interconnection (OSI) layer-7 which distinguishes it from a Network Address Translation (NAT) application which commonly operates on OSI layer-3. In the case of HTTP tunneling there are many alternative names for this kind of proxy such as <em>gateway</em> or <em>transparent proxy</em>. To proxy HTTP/S requests, one broadly can choose between two types: <em>Forwarding Proxy</em> and <em>Reverse Proxy</em>.</p><h3>Forwarding Proxy</h3><p>A forwarding proxy can come in two variants:</p><p>One variant, let’s call it <em>replication</em>, in which the proxy terminates an incoming client request, creates its own outgoing request to the destination, and then responds to the client with the destination’s reply. This works for HTTP but not for HTTPS as it would come with the important side-effect of being able to inspect (and modify) the client’s request, which violates TLS integrity.</p><pre>+--------+             +-------+             +-------------+<br>| Client |             | Proxy |             | Destination |<br>+--------+             +-------+             +-------------+<br>    |   1. HTTP Request    |                       |<br>    | -------------------&gt; |   2. HTTP Request     |<br>    |                      | --------------------&gt; |<br>    |                      |   3. HTTP Response    |<br>    |   4. HTTP Response   | &lt;-------------------- |<br>    | &lt;------------------- |                       x<br>    x                      :</pre><p>The other variant, let’s call it <em>tunneling</em>, in which the proxy uses the <a href="https://en.wikipedia.org/wiki/HTTP_tunnel">HTTP Tunneling</a> feature via the HTTP <a href="https://www.ietf.org/rfc/rfc2817.txt">CONNECT</a> method. By this, a proxy accepts an initial CONNECT request entailing the entire URL as HOST value, rather than just the host address. The proxy then opens up a TCP connection to the destination and transparently forwards the raw communication from the client to the destination. This allows the proxy to inspect and evaluate only the initial CONNECTrequest from the client e.g. for authorization, however, any further communication is not intercepted thus TLS communication can&#39;t be read by the proxy. This allows the initial CONNECT method to be HTTP or HTTPS to the proxy and then HTTPS to the destination via the proxy.</p><p>An example HTTP conversation:</p><pre>&gt; CONNECT example.host.com:443 HTTP/1.1<br>&gt; Host: example.host.com:443<br>&gt; Proxy-Authorization: Basic base64-encoded-proxy-credentials<br>&gt; Proxy-Connection: Keep-Alive<br>&lt; HTTP/1.1 200 OK</pre><pre>&gt; GET /foo/bar?baz#qux HTTP/1.1<br>&gt; Host: example.host.com<br>&gt; Authorization: Basic base64-encoded-destination-credentials<br>&lt; HTTP/1.1 200 OK<br>&lt; Connection: close</pre><pre>+--------+              +-------+             +-------------+<br>| Client |              | Proxy |             | Destination |<br>+--------+              +-------+             +-------------+<br>    |   1. HTTP/S CONNECT    |                       |<br>    | ---------------------&gt; |   2. TLS handshake    |<br>    |                        | --------------------&gt; |<br>    |                        |   3. Established      |<br>    |   4. HTTP/S 200 OK     | &lt;-------------------- |<br>    | &lt;--------------------- |                       |<br>    |   5. HTTPS Request     |                       |<br>    | ---------------------&gt; |   6. TCP stream       |<br>    |                        | --------------------&gt; |<br>    |                        |   7. TCP stream       |<br>    |   8. HTTPS Response    | &lt;-------------------- |<br>    | &lt;--------------------- |                       x<br>    x                        :</pre><p>In either variant, the client has direct influence over the choice of destination as two different URLs are involved: the proxy’s URL and the destination’s URL.</p><h3>Reverse Proxy</h3><p>A reverse proxy accepts incoming requests from clients and routes them to a specific destination based on the request. Discriminators for destination selection could be the requested host, path, query parameters, any header even the body payload. A reverse proxy, in any case, intercepts and terminates the HTTP and HTTPS connections and creates new requests to the destinations. Thus when using a reverse proxy, the client has not direct influence over the choice of destination.</p><h3>Rehash</h3><p>Now that we know the different HTTP/S proxy types and variants we can evaluate our requirements and work on an appropriate proxy implementation:</p><p>We want to be in full control of the destination choice, thus choose a forwarding proxy, and we want to guarantee high level of privacy and integrity, thus choose HTTP tunneling.</p><h3>Implementation</h3><p>Let’s build our own naïve forwarding tunneling proxy. The initial implementation is heavily inspired by a <a href="https://medium.com/@mlowicki/http-s-proxy-in-golang-in-less-than-100-lines-of-code-6a51c2f2c38c">blog post</a> by Michał Łowicki but goes beyond his prototype by addressing his outlined caveats such as handling hop-by-hop headers and setting up timeouts. You can follow along the final source code results over at <a href="https://github.com/betalo-sweden/forwardingproxy">betalo-sweden/forwardingproxy</a>.</p><p>We primarily need two components: A HTTP/S server and request handler. The HTTP/S server and request handler will act as intermediary for incoming client requests and the request handler will act as outgoing request initiator for destination requests deciding which method to use depending on the requested forwarding variant, tunneling or replication. The end result will be packages as command-line (CLI) tool ready to be started from a terminal or as daemon for supervisors such as <a href="https://www.freedesktop.org/wiki/Software/systemd/">Systemd</a>.</p><p>As HTTP/S server we use the Go’s standard <a href="https://golang.org/pkg/net/http/#Server">HTTP server</a>, which comes out of the box with HTTPS support as long as we provide a server certificate and private key for TLS handshakes:</p><pre>srv := &amp;http.Server{<br>    Addr:    &quot;:8080&quot;,<br>    Handler: proxy,<br>}</pre><p>Then, depending on using a HTTP or HTTPS server, we start the server:</p><pre>if certPath != &quot;&quot; &amp;&amp; keyPath != &quot;&quot; {<br>    srv.ListenAndServeTLS(certPath, keyPath)<br>} else {<br>    srv.ListenAndServe()<br>}</pre><p>To generate a certificate and private key pair we can use OpenSSL:</p><pre>$ openssl req \<br>    -newkey rsa:2048 \<br>    -nodes \<br>    -new \<br>    -x509 \<br>    -sha256 \<br>    -days 3650 \<br>    -keyout key.pem \<br>    -out cert.pem</pre><p>So far, nothing unusual or proxy related. Let’s have a look at the server’s handler. proxy is a struct that implements the http.Handler interface method and additionally holds a httputil.ReverseProxy; more on the reverse proxy later.</p><pre>type Proxy struct {<br>    HTTPProxy *httputil.ReverseProxy<br>}</pre><p>The ServeHTTP method mainly delegates the request handling responsibility: for destination URLs with http scheme it continues with the <em>replication</em> variant and for destination URLs with https scheme it continues with the <em>tunneling</em> variant:</p><pre>func (p *Proxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {<br>    if r.URL.Scheme == &quot;http&quot; {<br>        p.handleHTTP(w, r)<br>    } else {<br>        p.handleTunneling(w, r)<br>    }<br>}</pre><h3>Handle replication variant</h3><p>p.handleHTTP takes care of the <em>replication</em> by delegating the request handling responsibility to a reverse proxy:</p><pre>func (p *Proxy) handleHTTP(w http.ResponseWriter, r *http.Request) {<br>    p.HTTPProxy.ServeHTTP(w, r)<br>}</pre><p>p.HTTPProxy is a <a href="https://golang.org/pkg/net/http/httputil/#ReverseProxy">httputil.ReverseProxy</a> but acts as forwarding proxy. The names sound contradicting, so let&#39;s take a short de-tour and resolves the confusion.</p><h4>httputil.ReverseProxy interlude</h4><p>Go’s httputil.ReverseProxy is commonly used, as the name suggests, as reverse proxy. Its strength is its strict compliance to proxying HTTP specifications (RFC) such as <a href="http://www.w3.org/Protocols/rfc2616/rfc2616.html">RFC2616</a> for Hop-by-hop headers. The documentation states:</p><blockquote><em>ReverseProxy is an HTTP Handler that takes an incoming request and sends it to another server, proxying the response back to the client.</em></blockquote><p>Reading just this gives no indication to wether it actually acts as reverse or forwarding proxy. The httputil.ReverseProxy is in essence a http.Handler with a director, allowing to manipulate incoming requests before proxying them. The only constructor function <a href="https://golang.org/pkg/net/http/httputil/#NewSingleHostReverseProxy">httputil.NewSingleHostReverseProxy</a> constructs by default a reverse proxy by taking a target URL and then routing all requests to this target, rewriting the client&#39;s request URL. Its director looks like the following:</p><pre>director := func(req *http.Request) {<br>    req.URL.Scheme = target.Scheme<br>    req.URL.Host = target.Host<br>    req.URL.Path = singleJoiningSlash(target.Path, req.URL.Path)<br>    if targetQuery == &quot;&quot; || req.URL.RawQuery == &quot;&quot; {<br>        req.URL.RawQuery = targetQuery + req.URL.RawQuery<br>    } else {<br>        req.URL.RawQuery = targetQuery + &quot;&amp;&quot; + req.URL.RawQuery<br>    }<br>    if _, ok := req.Header[&quot;User-Agent&quot;]; !ok {<br>        // explicitly disable User-Agent so it&#39;s not set to default value<br>        req.Header.Set(&quot;User-Agent&quot;, &quot;&quot;)<br>    }<br>}</pre><h4>Back to p.HTTPProxy</h4><p>Back to our p.HTTPProxy. By constructing our own httputil.ReverseProxy and using a custom director, we can create a forwarding proxy instead of reverse proxy. All the director has to do is to not rewrite the request URL:</p><pre>func NewForwardingHTTPProxy() *httputil.ReverseProxy {<br>    director := func(req *http.Request) {<br>        if _, ok := req.Header[&quot;User-Agent&quot;]; !ok {<br>           // explicitly disable User-Agent so it&#39;s not set to default value<br>            req.Header.Set(&quot;User-Agent&quot;, &quot;&quot;)<br>        }<br>    }<br>    return &amp;httputil.ReverseProxy{Director: director}<br>}</pre><p>This is all that is required to establish a simple forwarding proxy for replicating HTTP requests.</p><h3>Handle tunneling variant</h3><p>The tunneling variant is a bit more work than the <em>replication</em> variant but this straight-forward. We have to implement a process of five steps:</p><ol><li>Expect and evaluate the CONNECT method as first request.</li><li>Create a TCP connection to the client’s requested destination.</li><li>Acknowledge the CONNECT request.</li><li>Hijack the connection and await a second request.</li><li>Bidirectionally copy incoming TCP streams between client and destination.</li></ol><p>Step (5) is the main behavior that gives <em>HTTP Tunnel</em> its name: It’s an unmodified TCP stream through a proxy tunnel.</p><p>Let’s go through each step and see how to implement these in Go:</p><p>Step (1) verifies we got the correct HTTP method CONNECT as first request:</p><pre>func (p *Proxy) handleTunneling(w http.ResponseWriter, r *http.Request) {<br>    if r.Method != http.MethodConnect {<br>        http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed)<br>        return<br>    }</pre><pre>    :</pre><p>Step (2) requires to dial a TCP connection to the destination:</p><pre>    :</pre><pre>    destConn, err := net.DialTimeout(&quot;tcp&quot;, r.Host, p.DestDialTimeout)<br>    if err != nil {<br>        http.Error(w, err.Error(), http.StatusServiceUnavailable)<br>        return<br>    }</pre><pre>    :</pre><p>This gives us a new connection, the connection to the destination. Note that the host of the client request comprises of the full address: host and port.</p><p>Step (3) simply writes back a success HTTP header:</p><pre>    :</pre><pre>    w.WriteHeader(http.StatusOK)</pre><pre>    :</pre><p>Step (4) requires to take over (<em>hijack</em>) the established client connection by moving ownership of the connection from the server to the handler. This has the consequence that the hijacker is fully responsible for error handling and not leaking any resources, thus draining and closing the connection:</p><pre>    :</pre><pre>    hijacker, ok := w.(http.Hijacker)<br>    if !ok {<br>        http.Error(w, &quot;Hijacking not supported&quot;, http.StatusInternalServerError)<br>        return<br>    }<br>    clientConn, _, err := hijacker.Hijack()<br>    if err != nil {<br>        http.Error(w, err.Error(), http.StatusServiceUnavailable)<br>        return<br>    }</pre><pre>    :</pre><p>We now have two connections, the client connection and the destination connection.</p><p>Step (5) streams data bidirectionally between client and destination. Go provides a simple way to do buffered byte copying, and using Go routines we stream bidirectionally by spawning two stream copiers:</p><pre>    :</pre><pre>    go transfer(destConn, clientConn)<br>    go transfer(clientConn, destConn)<br>}</pre><pre>func transfer(dest io.WriteCloser, src io.ReadCloser) {<br>    defer func() { _ = dest.Close() }()<br>    defer func() { _ = src.Close() }()<br>    _, _ = io.Copy(dest, src)<br>}</pre><p>Note that we don’t handle the potential closing and copy errors. The client and the server will implicitly close their connection whenever they are done sending data, causing the copying to exit with a close error. However, these are impractical to match for as no sentinel errors nor interface methods exist for them yet.</p><h3>Putting it all together</h3><p>Combining all snippets from before gives us a HTTP/S server that can <em>replicate</em> HTTP requests and <em>tunnel</em> HTTPS requests. We left out some details for simplification such as proxy authorization, timeout handling, and error handling.</p><p>Timeout handling can be complex as there are many timeout knobs to tweak: timeouts for the initial client connection (read, read-headers, write, idle), timeouts for the new established destination connection (dial, read, write) as well as the hijacked client connection (read, write).</p><p>Proxy authorization is standardized by <a href="http://www.w3.org/Protocols/rfc2616/rfc2616.html">RFC2616</a> and can be enforced via the Proxy-Authentication header (Proxy-Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l). Note that clients that support the proxy specification will only add these headers during the CONNECT method when tunneling and omit them during the actual request. Curl, for example, requires since <a href="https://daniel.haxx.se/blog/2014/04/04/curl-and-proxy-headers/">v7.37.0</a> proxied headers to be specifically declared separately from normal headers, in order to not leak proxy credentials to the destination.</p><p>The final source code at <a href="https://github.com/betalo-sweden/forwardingproxy">betalo-sweden/forwardingproxy</a> takes care of these topics.</p><p>Packaging everything as a small command-line tool and we can test it using Curl:</p><pre>forwardingproxy$ forwardingproxy -cert cert.pem -key key.pem -addr :443 -user Aladdin -pass OpenSesame</pre><pre>client$ curl -v -L --proxy-insecure --proxy https://Aladdin:OpenSesame@localhost:443 https://example.com/<br>* Rebuilt URL to: https://example.com/<br>*   Trying 127.0.0.1...<br>* TCP_NODELAY set<br>* Connected to localhost (127.0.0.1) port 443 (#0)<br>* Establish HTTP proxy tunnel to example.com:443<br>* Proxy auth using Basic with user &#39;Aladdin&#39;<br>&gt; CONNECT example.com:443 HTTP/1.1<br>&gt; Host: example.com:443<br>&gt; Proxy-Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l<br>&gt; User-Agent: curl/7.54.0<br>&gt; Proxy-Connection: Keep-Alive<br>&gt;<br>&lt; HTTP/1.0 200 OK<br>&lt;<br>* Proxy replied OK to CONNECT request<br>:<br>&gt; GET / HTTP/2<br>&gt; Host: example.com<br>&gt; User-Agent: curl/7.54.0<br>&gt; Accept: */*<br>&gt;<br>* Connection state changed (MAX_CONCURRENT_STREAMS updated)!<br>&lt; HTTP/2 200<br>&lt; cache-control: max-age=604800<br>&lt; content-type: text/html<br>&lt; date: Thu, 15 Mar 2018 17:50:35 GMT<br>&lt; etag: &quot;1541025663+gzip+ident&quot;<br>&lt; expires: Thu, 22 Mar 2018 17:50:35 GMT<br>&lt; last-modified: Fri, 09 Aug 2013 23:54:35 GMT<br>&lt; server: ECS (dca/53FA)<br>&lt; vary: Accept-Encoding<br>&lt; x-cache: HIT<br>&lt; content-length: 1270<br>&lt;<br>:</pre><p>Success! We can even use HTTP/2.</p><h3>Testing</h3><p>If you are using Docker and Docker Compose for integration testing of your services, having a small and easy to install forwarding proxy can be used to simulate the production infrastructure’s interaction.</p><p>Packaging the Go service as a Docker image is straight forward:</p><pre>FROM golang:1.10</pre><pre>RUN mkdir -p /go/src/github.com/betalo-sweden/forwardingproxy<br>WORKDIR /go/src/github.com/betalo-sweden/forwardingproxy</pre><pre>COPY ./vendor vendor<br>COPY *.go ./</pre><pre>RUN go install</pre><pre>EXPOSE 80 443</pre><pre>ENTRYPOINT [&quot;forwardingproxy&quot;]</pre><p>The proxying service and the proxy service can then be represented as Docker Composer services:</p><pre># docker-compose.forwardingproxy.yml</pre><pre>version: &#39;2.3&#39;</pre><pre>services:<br>  forwardingproxy:<br>    image: betalo/forwardingproxy<br>    init: true<br>    networks:<br>      - backend</pre><p>And an imaginary backend service:</p><pre># docker-compose.server.yml</pre><pre>version: &#39;2.3&#39;</pre><pre>networks:<br>  backend:</pre><pre>services:<br>  backend:<br>    image: dummy-service<br>    depends_on:<br>      - forwardingproxy<br>    init: true<br>    networks:<br>      - backend<br>    environment:<br>      PROXY_URL: http://forwardingproxy:8080</pre><p>This service uses a standard Go HTTP client with our proxy service as proxy configured:</p><pre>c := &amp;http.Client{<br>    Transport: &amp;Transport{<br>        Proxy: http.ProxyURL(os.Getenv(&quot;PROXY_URL&quot;)),<br>        :<br>    },<br>    :<br>}</pre><pre>:<br>c.Get(&quot;http://...&quot;)<br>:</pre><p>Note that Go can also directly read the proxy URL from environment variables, specifically HTTP_PROXY , HTTPS_PROXY, and NO_PROXY(or the lowercase versions thereof), which is the default behaviour for the default global Transport RoundTripper. HTTPS_PROXY takes precedence over HTTP_PROXY.</p><p>That’s all and we are ready to make requests between containerized services (or even outbound requests) for which the receiving service can verify that the client (backend service) did indeed proxy requests via our forwarding proxy:</p><pre># http.Handler middleware</pre><pre>func(w http.ResponseWriter, r *http.Request) {<br>    if !strings.HasPrefix(r.RemoteAddr, &quot;forwardingproxy&quot;) {<br>        log.Fatalln(&quot;Error: Request did not got proxied&quot;)<br>    }<br>})</pre><h3>Alternatives</h3><p>One can consider alternatives in network protocols, off-the-shelf products and <em>Make vs Buy</em> decisions.</p><p>Alternatives in network protocols to a proxy would be a NAT. This would require more network or hardware configuration even if it’s abstracted by IaaS providers.</p><p>Alternative off-the-shelf products would be for example <a href="http://squid-cache.org/">Squid</a>, <a href="https://www.nginx.com/">NGINX</a>, <a href="http://haproxy.org/">HAProxy</a>, <a href="http://varnish-cache.org/">Varnish</a>, <a href="https://tinyproxy.github.io/">TinyProxy</a>, and many more. NGINX is primarily meant to be used as a reverse proxy and can be difficult to set up when require to act as a forwarding proxy. HAProxy is similar to NGINX meant to be used as reverse proxy as well as HTTP cache. Varnish is primarily meant to be used as HTTP cache. Squid and TinyProxy are closest to be working as forwarding proxies, however it can be difficult to set them up on a new Amazon Linux 2 AMI EC2 instance, and TinyProxy is not maintained anymore since several years.</p><p>Alternatives for a <em>Make vs Buy</em> decision would be:</p><p>Heroku’s <a href="https://www.heroku.com/private-spaces">Private Spaces</a>, which provide <a href="https://devcenter.heroku.com/articles/private-spaces#stable-outbound-ip-addresses">stable outbound IPs</a> as a feature:</p><blockquote><strong><em>Stable outbound IPs</em></strong><em><br>Securely connect apps to third party cloud services and corporate networks.</em></blockquote><p>Choosing Heroku’s Private Spaces comes with many more benefits but would also weight in with a price tag of either $1000 or $3000 add-on credits/month for advanced features for regulatory compliance, respectively.</p><p>Another <em>Make vs Buy</em> alternative would be hosted Heroku Add-ons that would offer stable IP support such as <a href="https://elements.heroku.com/addons/fixie">Fixie</a>, <a href="https://elements.heroku.com/addons/guru301">Guru301</a>, <a href="https://elements.heroku.com/addons/proximo">Proximo</a>, <a href="https://elements.heroku.com/addons/quotaguardstatic">QuotaGuard Static</a>.</p><p><em>Fixie</em> and <em>Guru301</em> only support Heroku’s <em>United States</em> region, not Europe. <em>QuotaGuard Static</em> and <em>Proximo</em> both support Heroku’s <em>Europe</em> region and would offer 20,000 respectively 50,000 requests per month for $19 respectively $25 per month.</p><h3>Conclusion</h3><p>It’s fairly easy to implement a naïve forwarding proxy in Go, as many network server’s are easy to implement in Go. This is commonly Go’s strength. Using your own forwarding proxy allows you full understanding and customization and is simple and cheap to use during development and integration testing to raise your confidence level.</p><p>We hope you gained a good understanding about http forwarding proxies. If you feel curious, give our forwarding proxy a try; we are happy to receive feedback.</p><p>Source code: <a href="https://github.com/betalo-sweden/forwardingproxy">betalo-sweden/forwardingproxy</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f0e5575d93e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/betalo-engineering/integrating-with-third-parties-via-a-go-forwarding-https-proxy-6f0e5575d93e">Integrating with Third-Parties via a Go Forwarding HTTPS Proxy</a> was originally published in <a href="https://medium.com/betalo-engineering">Betalo Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>