<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[The Quiq Blog - Medium]]></title>
        <description><![CDATA[Technology, Life, Random Musings. From the team at Quiq. - Medium]]></description>
        <link>https://medium.com/quiq-blog?source=rss----ff232b4e5904---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>The Quiq Blog - Medium</title>
            <link>https://medium.com/quiq-blog?source=rss----ff232b4e5904---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:17:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/quiq-blog" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to restrict and track Linux user outbound connections]]></title>
            <link>https://medium.com/quiq-blog/how-to-restrict-and-track-linux-user-outbound-connections-8fb98125c3c5?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/8fb98125c3c5</guid>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[iptables]]></category>
            <category><![CDATA[http-proxy]]></category>
            <category><![CDATA[openresty]]></category>
            <dc:creator><![CDATA[Lev Petrushchak]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 23:19:49 GMT</pubDate>
            <atom:updated>2022-08-05T23:19:49.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f8ENGJ0t9XSiutZP34DM6A.jpeg" /><figcaption>Online media photo created by creativeart — <a href="https://www.freepik.com/photos/online-media">www.freepik.com</a></figcaption></figure><p>Restricting outbound (egress) connections is an essential part of network security. Many articles explain why this is so important and how to do it overall. In simple words — you can have a super secure application and environment, but you can’t avoid using third-party software which may be compromised. In most cases, such software is not harmful until it downloads some helper tools, which can download backdoor, scrape and send sensitive information, etc. Without the possibility to upload/download stuff, malicious software in most cases can be disarmed.</p><p>It’s also the case where a vulnerability existing in a piece of software which can allow access into your environment by a hacker exploiting that vulnerability, if then they can’t easily contact the internet it makes it much harder to make lateral movements.</p><p>Simply blocking connections to outside is not a solution. There are lots of services that are dependent on external connectivity to do their job. For example — your OS needs downloading updates, so at least this requires DNS and HTTP/HTTPS connections.</p><p>For now, let’s imagine we have a Linux server with an unprivileged user.</p><blockquote>Unprivileged — means user with /usr/sbin/nologin shell and other limitations. It’s used to run the Docker containers in a more secure environment.</blockquote><p>For this user, for example, outbound connectivity will be blocked by the firewall, but not everything. Let say we want to make an exception for DNS, HTTP, and HTTPS traffic. While DNS requests can be easily restricted and tracked with a firewall, HTTP(S) is more complex and also encrypted.</p><p>So for this, we will use a proxy, built with <a href="https://openresty.org/"><strong><em>Openresty</em></strong></a> with the HTTP(S) proxy CONNECT module <a href="https://github.com/chobits/ngx_http_proxy_connect_module"><strong>ngx_http_proxy_connect</strong></a></p><p>Openresty is an enhanced version of the Nginx web server and ngx_http_proxy_connect module provides support for the CONNECT method request. That’s how it is described in <strong>RFC7231</strong>:</p><blockquote>The CONNECT method requests that the recipient establish a tunnel to the destination origin server identified by the request-target and, if successful, thereafter restrict its behavior to blind forwarding of packets, in both directions, until the tunnel is closed.</blockquote><p>This method must be supported by both — the proxy and the client. And the client is a tricky part. Not all clients support this method.</p><p>For example, <strong>wget </strong>— a popular tool to download content from the web, doesn’t support it, especially the <em>Busybox</em> version used in Alpine Linux. Fortunately, this is rather an exception and most modern tools work just fine. In this case, <strong>wget</strong> can be replaced by <strong>curl.</strong></p><p>But let’s get back to our Linux unprivileged user, we need somehow to enforce all his outbound connections to go only through the proxy. This can be achieved with the most common Linux firewall —<strong> iptables</strong>, and this is the first part of our setup.</p><p>We will create a new chain:</p><pre>$ iptables --new-chain SANDBOX</pre><p>route all traffic from that user to the sandbox chain</p><pre>$ iptables -A OUTPUT -m owner --uid-owner john.doe -j SANDBOX</pre><p>now in sandbox chain we care only about new connections, everything else can go</p><pre>$ iptables -A SANDBOX -m state ! --state NEW -j RETURN</pre><p>allow local subnet<em> (put your subnet mask here)</em></p><pre>$ iptables -A SANDBOX -p tcp -d 172.16.0.0/12 -j RETURN</pre><pre>$ iptables -A SANDBOX -p udp -d 172.16.0.0/12 -j RETURN</pre><p>allow DNS from Cloudflare</p><pre>$ iptables -A SANDBOX -p udp -d 1.1.1.1/32 --dport 53 -j RETURN</pre><p>and log everything else, that came to this point</p><pre>$ iptables -A SANDBOX -m state --state NEW -j LOG --log-prefix &quot;Filtered outbound connection: &quot;</pre><p>then drop it</p><pre>$ iptables -A SANDBOX -j REJECT</pre><p>now we can try to run a sample request and see what happens</p><pre>$ sudo -H -u john.doe curl -v ifconfig.me<br>* Trying 34.160.111.145:80…<br>* TCP_NODELAY set<br>* connect to 34.160.111.145 port 80 failed: Connection refused<br>* Failed to connect to ifconfig.me port 80: Connection refused<br>* Closing connection 0<br>curl: (7) Failed to connect to ifconfig.me port 80: Connection refused</pre><p>so if there’s an outbound connection attempt to something not allowed in the firewall and not using the HTTP(S) proxy, in /var/log/syslog you will see a message like this:</p><pre>kernel: Filtered outbound connection: IN= OUT=ens5 SRC=172.16.0.5 DST=34.160.111.145 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=16088 DF PROTO=TCP SPT=42218 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0</pre><p>With this information, it’s easy to catch what else should be whitelisted or routed through the HTTP(S) proxy and also (what is even more important) — track if something is trying to connect outside which is not supposed to do so.</p><p>The second part is the proxy setup. We have our ready-to-use Ansible role <a href="https://github.com/Quiq/egress-proxy">https://github.com/Quiq/egress-proxy</a></p><p>Once Ansible setup finished on the server, let’s login there and see what we can do with it</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/624b23a9c49df0f7320e6ee5f531610d/href">https://medium.com/media/624b23a9c49df0f7320e6ee5f531610d/href</a></iframe><p>We set environment variables http_proxy and https_proxy to 127.0.0.1:3128 which is our egress-proxy container, and <strong>curl</strong> seeing these variables will use them to access the proxy and connect to the destination host with CONNECT method</p><p>At the end docker logs -f egress-proxy command will show us two connections opened through the proxy.</p><p>In our Ansible role we have whitelist.yml with the list of domains allowed to connect and ifcomfig.me is there. Others would get403 — Forbidden</p><p>Now if we run our docker containers under this user, we can be sure, that our system has one more solid layer of security, all outbound connections are restricted and all connection attempts will be logged in the system log.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fb98125c3c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/how-to-restrict-and-track-linux-user-outbound-connections-8fb98125c3c5">How to restrict and track Linux user outbound connections</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resolving Greenbone data loading issues]]></title>
            <link>https://medium.com/quiq-blog/resolving-greenbone-data-loading-issues-8f873de66d12?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/8f873de66d12</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[openvas]]></category>
            <category><![CDATA[greenbone]]></category>
            <category><![CDATA[scan-configs]]></category>
            <category><![CDATA[nvt]]></category>
            <dc:creator><![CDATA[Phill Mell-Davies]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 23:16:02 GMT</pubDate>
            <atom:updated>2022-08-05T23:16:01.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3EuwBx4Z8j2pwGyFUkquuw.png" /></figure><p>At Quiq we use Greenbone Vulnerability Management system (GVM, also known as OpenVAS) to periodically scan our networks for vulnerable network services. It’s a powerful piece of software and we love it; it’s one of many that helps us ensure our platform is as secure as possible. However it’s also a complicated tool, and as with all tools with high complexity sometimes issues arise.</p><p>We run GVM in docker containers on several different platforms: Linux machines as well as Macs, x64 and ARM. Occasionally, for reasons we don’t fully understand, some of the data that GVM requires in order to run fails to load. This article describes one such situation along with the steps we’ve found to resolve it. We wanted to write it up because even though we’ve found a number of articles with suggestions on how to resolve similar issues, those suggestions didn’t quite take care of it for us.</p><p><strong>TL; DR: </strong><br>NVTs and scan configs didn’t show up in the GVM console. Running `<em>sudo -u gvm gvmd — rebuild &amp;&amp; sudo -u gvm gvmd — rebuild-gvmd-data=all</em>` ultimately fixed it for us.</p><p><strong>The Setup:</strong></p><ul><li>Running a version of the Secure Compliance GVM image (<a href="https://hub.docker.com/r/securecompliance/gvm/">https://hub.docker.com/r/securecompliance/gvm/</a>)</li><li>GVM data stored in three separate docker volumes, mapped to: /var/lib/gvm, /opt/database, /var/lib/openvas/plugins</li></ul><p><strong>The Symptom:</strong></p><ul><li>Start up GVM container and wait several hours, as is suggested in a number of different entries on StackOverflow and community.greenbone.net</li><li>Log into the GVM admin panel and check the data feeds page, and see that all the feeds are marked as current. In some cases a feed may be marked “too old”, this is okay as well:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QT7o3eBs6AkZPrH8q07aYQ.png" /><figcaption>Data feeds are current</figcaption></figure><ul><li>However, discover that there are no NVT’s (network vulnerability tests) or default scan configs in the system:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6MpFChjhe5E-ajZipb0Sg.png" /><figcaption>No NVTs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bw6cA1Wg4EGiEkYuBIwVKg.png" /><figcaption>No scan configs</figcaption></figure><p>Tailing the container log didn’t give us any insights into what was going wrong, everything in it looked normal. However, there are also logs of interest under <em>/var/log/gvm</em>.</p><p><strong>Our Solution:</strong></p><ul><li>Looking at the logs we could see all the data being “loaded” (fetched from the internet and stored in some form), but they still didn’t show in the console. After a fair bit of research and experimentation though, we came across the following commands that forced the NVTs and scan configs to be rebuilt, and this was the solution: `<em>sudo -u gvm gvmd — rebuild &amp;&amp; sudo -u gvm gvmd — rebuild-gvmd-data=all</em>`</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKIlpj5fYYfg4s7Qhh4TGw.png" /></figure><ul><li>These commands take a while (approximately 30 minutes for us) to complete, but once they finished, everything was there!</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LklrLupvEvw2JUBESrZD1g.png" /><figcaption>NVTs loaded</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6Sdpwsw9hkplm9Hl0BUKfw.png" /><figcaption>Scan configs loaded</figcaption></figure><p>Footnote: These articles helped us get to the bottom of this issue and ultimately resolve it, so if this one doesn’t help resolve your situation, maybe these ones will:</p><ul><li><a href="https://github.com/greenbone/gvmd/blob/main/INSTALL.md">gvmd/INSTALL.md at main · greenbone/gvmd</a></li><li><a href="https://community.greenbone.net/t/update-in-progress/9866/8">Update in progress</a></li><li><a href="https://community.greenbone.net/t/missing-scan-configs-gvm-20-08-1/9604/3">Missing scan configs (GVM 20.08.1)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f873de66d12" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/resolving-greenbone-data-loading-issues-8f873de66d12">Resolving Greenbone data loading issues</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Consul balancer for Openresty/Nginx]]></title>
            <link>https://medium.com/quiq-blog/consul-balancer-for-openresty-nginx-32d4a86e65b3?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/32d4a86e65b3</guid>
            <category><![CDATA[openresty]]></category>
            <category><![CDATA[consul]]></category>
            <dc:creator><![CDATA[Roman Vynar]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 01:35:42 GMT</pubDate>
            <atom:updated>2022-08-05T01:35:41.858Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we would like to opensource our Consul balancer LUA code which can be used with Openresty, a free nginx alternative with LUA module, for service discovery and load balancing on the discovered addresses.</p><p>Check this out: <a href="https://github.com/Quiq/lua-consul-balancer">https://github.com/Quiq/lua-consul-balancer</a></p><p>We are using this implementation at Quiq for 4 years or so and it is very fast and reliable with even 50 services in total and multiple addresses per each one. It doesn’t produce much load on Openresty if any at all.</p><p>This is also a good replacement of “Internal Load Balancer” resource from AWS or any other cloud provider especially if you use Hashicorp Consul and rely on it for service discovery. There is no reason to do a discovery on the network level of ports and instances if you can straight point your web server <em>proxy_pass</em> directive to one of the addresses being discovered.</p><p>Thanks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32d4a86e65b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/consul-balancer-for-openresty-nginx-32d4a86e65b3">Consul balancer for Openresty/Nginx</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Next-level Authentication with Webauthn]]></title>
            <link>https://medium.com/quiq-blog/next-level-authentication-with-webauthn-4bbfa2f225a3?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/4bbfa2f225a3</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[webauthn]]></category>
            <category><![CDATA[yubikey]]></category>
            <category><![CDATA[quiq]]></category>
            <category><![CDATA[web]]></category>
            <dc:creator><![CDATA[Phill Mell-Davies]]></dc:creator>
            <pubDate>Wed, 26 Jan 2022 18:40:40 GMT</pubDate>
            <atom:updated>2022-01-26T18:40:40.065Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEgSNzJzyzaTZXjCiBYJ2A.jpeg" /></figure><p>In 2021, as part of our never-ending quest to improve security at Quiq, we started exploring options for adding additional layers of authentication for some of our most critical services. We already utilize federated authentication (single sign-on) for our internal and customer facing services, through which we require multi-factor authentication, but the question arose, could we do more?</p><p>This search led us to explore Webauthn. What is it exactly? Webauthn is a <a href="https://webauthn.guide/#intro">standard</a> that allows websites to easily support passwordless, public key-based authentication. What does this mean in practice? It means that you can use a wide range of modern mechanisms such as Yubikey, Apple Touch ID, Windows Hello, and more for logging into your critical services.</p><p>Adding Webauthn into our defense-in-depth strategy is big win! There’s just one problem. While all major web browsers have supported Webauthn for some time, adoption on the server side has been <a href="https://sec.okta.com/articles/2020/04/webauthn-great-and-it-sucks">very slow</a>. We rely very heavily on self-hosted, open source tools and the way we apply federated authentication is using <a href="https://github.com/oauth2-proxy/oauth2-proxy">OAuth2 Proxy</a>. This allows us to add a little configuration to NGinx and voilà! Any requests to the service require users to log in to our single sign-on provider.</p><p>Unfortunately there wasn’t a proxy that allowed us to easily do the same thing with Webauthn. There were a few projects that aimed to do this but they all had drawbacks: too immature, not easy to automate with our DevOps tools, or requiring dependencies that we didn’t want to have to manage. So to solve this problem, we created our own.</p><p>As of today we are open sourcing Quiq’s Webauthn proxy: <a href="https://github.com/Quiq/webauthn_proxy">https://github.com/Quiq/webauthn_proxy</a></p><p>The proxy is compact and easy to deploy. Written in Golang, it is simple to configure, in fact there are only two parameters you are required to change in its configuration file to get started. It stores configuration and user credentials in a simple YAML file so it’s a breeze to manage with tools like Ansible or Puppet.</p><p>The README file in the github repo illustrates how to use the proxy with NGinx. It also demonstrates a more advanced configuration with OpenResty that chains the OAuth2 proxy and Webauthn proxy, requiring both methods to succeed in order to log in. We are certain that it will work with other reverse proxies and webservers but thus far it has only been tested with NGinx and OpenResty.</p><p>There are a couple of caveats to be aware of. First of all, we’re strictly using Webauthn proxy internally; as of today we don’t support Webauthn for our customers to log into Quiq’s products. Many of our customers use their own single sign-on solutions with Quiq anyway, so it will be up to them to support it. Secondly, passwordless authentication should always be used as part of a multi-factor approach, not by itself; otherwise a thief who steals your authentication device could potentially log in as you.</p><p>We really hope that this project will be used by as many teams as possible to help secure their own infrastructure. We also look forward to contributions so if you find a way to make it more useful for you, please open an issue or pull request. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bbfa2f225a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/next-level-authentication-with-webauthn-4bbfa2f225a3">Next-level Authentication with Webauthn</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adventures in Deploying a Deep Learning Model in the Browser]]></title>
            <link>https://medium.com/quiq-blog/adventures-in-deploying-a-deep-learning-model-in-the-browser-ef0e5bfc192c?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/ef0e5bfc192c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflowjs]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Kyle McIntyre]]></dc:creator>
            <pubDate>Thu, 13 Jan 2022 01:10:47 GMT</pubDate>
            <atom:updated>2022-01-13T19:12:31.577Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nP_ZV6diSmbSy9UXrQKOFQ.png" /></figure><p>Last year we embarked on a project to develop a predictive typing feature for conversational business messaging. This article doesn’t focus on the feature or model itself but rather our experiences deploying it in the browser. We discuss our reasons for attempting a browser-based deployment, selection of a browser runtime for deep learning models and some lessons learned about how much of a ML feature should live in the model vs. in the application.</p><h3>Background</h3><p>A predictive typing model (one that helps you complete your words and thoughts via tab-complete) is inherently high-scale. At Quiq, we often think of our scaling factors in rough terms like — does it scale in the number of logged in users, conversations or messages? In this case, the scaling factor is at the sub-message level. We call it typing scale. In addition to being very high scale the model must also exhibit low latency. If the latency is too high, we won’t be able to get suggestions in front of our users’ eyes before they proceed to complete their own words and thoughts.</p><p>Our web-based software is often used within contact centers, some of which are operated overseas. We’re excited to see where edge computing is headed especially as it relates to ML, but we take it as a given that some of our users will have significant network latency to our servers. This, coupled with concerns about our cloud computing bill led us to consider the approach of deploying the model in the browser.</p><p>We had plenty of concerns about pursuing a browser-based approach as well:</p><ul><li>Will there be a myriad of browser compatibility issues/unsupported browsers?</li><li>Will we tank the performance of the app hosting the model?</li><li>Will the latency saved by avoiding the network be lost due to lower compute power on client machines?</li><li>How much will we have to weaken the model accuracy to make it viable client-side?</li><li>How much code will have to be ported to Javascript and dual-maintained?</li></ul><p>We were confident, however, that a browser-based approach would be operationally cheaper. Moreover, we weren’t technologically prepared to host the model at scale server-side even if we were willing to pay the costs and assume the sunk network latency cost. So — with significant work in either direction — we decided to give the browser-based approach a shot.</p><p>Our starting point was a Python/<a href="https://www.tensorflow.org/">TensorFlow 2</a> training pipeline that was successfully producing TF SavedModels that did a pretty good job at predicting text.</p><h3>Finding a Deep Learning Browser Runtime</h3><p>Our first attempt to serve the model in the browser centered around <a href="https://onnx.ai/">ONNX</a> (Open Neural Network Exchange). We liked the idea of being able to deploy models trained in various deep learning frameworks onto ONNX as a common runtime based on an open standard. But we encountered a significant problem: At least as of late 2021, the ONNX browser runtime project seems incomplete and inactive. There are a bunch of critical ops that it doesn’t support. So while we were able to get the TF SavedModel → ONNX converter working just fine, the ONNX browser runtime couldn’t run it. ONNX seems like a great project, but the browser runtime simply isn’t viable.</p><p>Next we turned to what seemed like the only other real possibility: <a href="https://www.tensorflow.org/js">TensorFlowJS</a> (TFJS). We were initially under the impression that TFJS was tightly coupled with TensorFlow but we have since learned that TFJS is a standalone deep learning framework. Even if you train in another framework like TensorFlow or PyTorch, you can generally convert your saved models to TFJS’s model format — similar to what we were hoping to achieve with ONNX. So far so good!</p><p>TFJS supports several ‘backends’ — runtimes capable of executing your TFJS model, some of which are supported in web browsers. Here are the runtimes that work in the browser: WebGL, WASM (SIMD, SIMD+threads) &amp; CPU. WebGL is currently the recommended runtime and was the top performer in our model benchmarks throughout our development lifecycle. WASM was decent especially w/ SIMD &amp; threading support &amp; CPU was glacially slow as expected. WebGL also had sufficient <a href="https://caniuse.com/webgl">browser support </a>for our purposes.</p><p>We started with a basic model that was trained using the keras Layers API. We were able to convert the model to TFJS format and run it with minimal effort! This was a great start to our project. However, our full model took more effort to convert.</p><h3>Being Compatible</h3><p>TFJS doesn’t support everything that can easily wind up in your TF SavedModel. In particular, we realized that TFJS didn’t have support for any of the text stuff we were doing in our TF program. Here’s some stuff we had to reimplement outside the model:</p><ul><li>Routines from tensorflow_text for normalization &amp; tokenization</li><li>Preprocessing layers from keras.experimental.preprocessing for things like vocabulary encodings</li></ul><p>The realization that we’d be handling all text preprocessing ourselves outside the framework made us more cautious about doing fancier text tokenization schemes since we’d likely have to port it to Javascript. Our mantra became ‘code it up in Python, and if it significantly helps prediction accuracy we’ll investigate porting it’.</p><p>It should be noted that TFJS supports custom Ops/Kernels that might facilitate keeping your preprocessing in the model. But we needed to do a port either way and decided to skip learning how to do custom ops. We later realized this was a stroke of luck because of op scheduling overhead, which we’ll discuss later in the post.</p><h3>The Thrill of Victory, the Agony of Defeat</h3><p>It was very exciting when we first saw our full model run in the browser. It WORKED! And the numbers were dead nuts accurate with the Python version. Unfortunately, it was running at the approximate speed of a snail. Things were taking between 10-20x longer than we had expected based on estimates from our simple Keras/layers model. Both were executing on the recommended WebGL backend for TFJS and had roughly similar network complexity.</p><p>At this point we learned a bit more about how the WebGL backend works: it actually translates your TFJS model operations into GLSL shader programs and executes them on your graphics device. This allows it to leverage any available hardware acceleration and is what enables it to perform heavy operations faster.</p><p>The translation of a TFJS model to various shader programs is fairly slow: it took between 1–2 seconds on our model/machines. Consequently TFJS maintains a cache of shader programs based on the ‘method signatures’ of your model &amp; ops. The cause of our big performance problem turned out to be our use of inputs and outputs with dimensions of dynamic length. These were continually breaking through the method signature-based cache of shader programs.</p><p>We converted our dynamic-length inputs to a fixed length of sufficiently large size and then padded them to work around this problem. At this point our performance was roughly where we had predicted. However, the more we played with it in the app on various hardware, the more it seemed like it still wasn’t fast enough. We also realized we needed more control over the model as it executed.</p><h3>Deploy a Network, Not a Program</h3><p>At this point, all of the text preprocessing was done outside the model, but the model took care of everything else. What do we mean by everything else? We modeled the predictive text problem as a sequential word prediction task. The basic steps are as follows:</p><ol><li>Tokenize the text typed so far</li><li>Run each token sequentially through an autoregressive model to update its hidden state</li><li>Now you’re in the ‘present’ i.e. where the typing cursor is at</li><li>Begin predicting tokens in the future — greedy &amp; beam search variants</li></ol><p>Steps 2 and 4 have loops and other control flow in them. Where should the loops live? TFJS supports control flow so we had a choice to put them in there, or outside the model in Javascript. Once again, we tried putting it all in the model to save porting the logic and hopefully run faster. This approach works but is unfortunately slow.</p><p>As hinted at in the last section, TFJS models are based on a graph representation that cannot only express deep learning networks but also general programs/execution. A graph is a graph of Ops — operations to execute. It is our understanding that most (if not all) of the TFJS backends, and especially WebGL, incur a latency cost (e.g. milliseconds) to schedule and execute an op, regardless of how heavy or lightweight the op is.</p><p>This makes a lot of sense in the context of the WebGL backend: it executes your models’ ops asynchronously as graphics shader programs on your graphics device. This requires OS-level scheduling and coordination. So even though WebGL can execute heavy math stuff pretty fast by leveraging any hardware acceleration you might have, it can be slow if your model is composed of lots of tiny ops. It’s for this reason that we took the loops and other control flow described in steps 2 &amp; 4 out of the model and implemented them in Javascript. Those basic loops and conditions were really slowing us down! That’s one reason we think TFJS should be used to ‘deploy a network, not a program’.</p><p>Aside from performance, it’s also advantageous to move control flow outside of the model so that you have greater control over things at runtime. You may want to introspect on time remaining or application state so that you can potentially interrupt and return early. This is especially true in sequential problems. It’s possible to perform such introspection inside of your TFJS model if you pass enough data in, but it will likely be harder to implement and slower to execute.</p><p>By the time we released the feature, the only stuff left in the TFJS model was a <em>single time-step/iteration of the purely numeric deep learning network</em>. No text preprocessing or postprocessing. No search, no loops. The sequential nature of the problem was tackled in Javascript.</p><p>Although the ‘final’ version didn’t look much like we had originally envisioned, we were happy with the result. In addition to getting our performance to an acceptable level, invoking the model in a less monolithic way provided us with much greater control and ability to optimize. For example, we could now easily add caching and loop-breaking timeouts. By using asynchronous Javascript we were able to ‘throttle’ model execution more easily so as to not impact rendering in the main app. The single iteration of the numeric deep learning network was the building block we needed TFJS to tackle for us — everything else worked better outside TFJS.</p><h3>You’re in the Browser — Now What?</h3><p>You’ve got a compatible model that’s a network and not a full-fledged program. You’ve got the surrounding Javascript to make it all work. What’s next? Well there are other concerns associated with hosting the model in the browser that you need to tackle if you’re going to deliver a production-ready feature.</p><p>With any interactive client application it’s important to tune your code with an eye towards performance. After all, even the best predictive model isn’t going to be useful if running it ruins the user experience. One of the first things we knew we had to address was delivering the model assets to the browser promptly to make sure we could get the model running in time to be useful. We decided to try and cache the assets in the browser in order to save the network round-trip time as well as keep bandwidth free for the other network calls our application requires. Although the model assets are too large to save in browser local storage, TFJS has a pre-built adapter for the IndexedDB API. The larger storage limits on this API allows us to save and load the model artifacts on the client side. With a little extra work to store our additional client and vocabulary data alongside the model, we now only incur download costs a single time per model when the user first turns on the feature (assuming the browser doesn’t delete it to reclaim storage space). We also wanted to make sure our users wouldn’t load the hefty TFJS libraries if they didn’t have this feature enabled — a problem solved thanks to <a href="https://webpack.js.org/">Webpack’s</a> dynamic import code-splitting feature.</p><p>Once the model is loaded and running in the browser we want to give it enough system resources to deliver timely yet accurate predictions. However this is balanced by the need to prevent blocking the main javascript thread so that the UI can continue to update and respond to user interactions. We investigated running the model in a Web Worker to stay off the main thread completely, but so far it doesn’t seem that the WebGL backend is fully supported in Web Workers. Instead we opted to run on the main thread, using async programming to ensure our prediction loop ceded control back to the browser at regular intervals. This strategy provided us with a good balance of model performance and UI responsiveness, and really made it feel like a powerful, modern feature that would both be useful and feel good to use.</p><p>We also took care to prevent running duplicate instances of our models to keep memory usage in check. This was especially important as our application can be iFramed within other sites, potentially running several separate instances on a single page. We set up a message passing service using a Shared Worker to communicate across the iFrame boundaries and allow each instance of our app to access the feature while only one of them was in charge of the model. TFJS also requires some manual garbage collection, which we tied into the model lifecycle to prevent any leakage.</p><h3>Results &amp; Conclusion</h3><p>We deployed the feature to a subset of our users last fall. So far the feature has executed robustly and we haven’t been plagued by browser compatibility issues. We instrumented the feature’s execution time and the observed times out in the wild are similar to those we saw during our late stage testing and are acceptable. We haven’t received any complaints or observed any metrics that would point to general application or machine lag, which we attribute to ceding control back to the browser loop regularly and often.</p><p>Perhaps if we were to do it all over again, we would just do the project in Javascript/TFJS from the beginning to reduce our concerns about duplicate code maintenance. However, to date we don’t have any experience using TFJS for model training or authoring non-trivial forward passes etc.</p><p>In conclusion, we were able to successfully deploy a non-trivial, sequential deep learning model for predictive text to the browser using TFJS. The final result didn’t look like what we expected, but we’re happy with it. TFJS is a great piece of open source that opens up a world of ML possibilities. We hope that our experience and conclusions are valid and help other folks make great use of it.</p><h3>Acknowledgements</h3><p>The following individuals contributed heavily to this project and blog post:</p><ul><li><a href="https://www.linkedin.com/in/avishek-biswas">Avishek Biswas</a></li><li><a href="https://www.linkedin.com/in/talondaniels">Talon Daniels</a></li><li><a href="https://www.linkedin.com/in/joseph-debruycker-056520121">Joe DeBruckyer</a></li><li><a href="https://www.linkedin.com/in/kylejmcintyre">Kyle McIntyre</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef0e5bfc192c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/adventures-in-deploying-a-deep-learning-model-in-the-browser-ef0e5bfc192c">Adventures in Deploying a Deep Learning Model in the Browser</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Plotly Timelines to Visualize Thread Activity]]></title>
            <link>https://medium.com/quiq-blog/using-plotly-timelines-to-visualize-thread-activity-c135bb17880?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/c135bb17880</guid>
            <category><![CDATA[timeline]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[Kyle McIntyre]]></dc:creator>
            <pubDate>Tue, 14 Dec 2021 14:32:51 GMT</pubDate>
            <atom:updated>2021-12-14T14:32:51.399Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever wanted to visualize concurrent activity in a multi-threaded application? With <a href="https://plotly.com/python-api-reference/generated/plotly.express.timeline.html">Plotly’s timeline</a> chart type and some reliable logging, it’s easy to create such a visualization.</p><p>We recently had a service instance in production saturate its available CPU. This caused requests to lag substantially and triggered upstream retry. Unfortunately the problem didn’t resolve itself and we had to manually intervene. We of course wanted to learn from this event and needed to understand exactly what happened.</p><p>Service monitoring and diagnostics are essential to detecting and diagnosing these types of problems. We also love using sampling-based profilers like <a href="https://github.com/jvm-profiling-tools/honest-profiler">Honest Profiler</a> to understand CPU usage out in production. However, sometimes you need a more holistic view of what’s going on as opposed to focusing on metrics or flame graphs. A thread timeline can help bridge that gap.</p><h3>Example Thread Timeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KevkJsTH2Zfysavoa4Mv-w.png" /><figcaption>Timeline of thread activity using Plotly Express</figcaption></figure><p>In the timeline above, the Y axis corresponds to different threads in our application. We use DropWizard/Jetty in our Scala services, hence thread names like dw-5702. The rectangular events correspond to individual inbound web requests. Our application is multi-tenant and performs logical isolation at the application layer, so we’re using colors here to denote the tenant associated with each request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MaqRQiaIvTjB9K5KJQqYKw.png" /><figcaption>Interactive tooltips for further exploration</figcaption></figure><p>The events can have additional metadata attached to them that is available within interactive tooltips. This makes the timeline very powerful for data exploration. We’re using plotly within Jupyter notebooks for the greatest ease and interactivity.</p><h3>The Code Snippet</h3><p>Generating a thread timeline is really easy if you already have some standard logging in place that includes the thread in its output. For example, if you have a standard HTTP request log, you can process those logs to construct events for the timeline. Such request logs typically have a timestamp and a duration, allowing you to properly size and position your events. But of course nothing about this is specific to web servers. As long as you can construe your thread activity as requests/events with a start (or end) time and duration, you’re good to go.</p><p>Assuming you’ve constructed a list called events that looks something like this:</p><pre>[<br>  {<br>    &#39;thread&#39;: &#39;dw-5697&#39;, <br>    &#39;type&#39;: &#39;http&#39;, <br>    &#39;tenant&#39;: &#39;redacted&#39;, <br>    &#39;start&#39;: Timestamp(&#39;2021-12-13 18:51:00.038999808&#39;), <br>    &#39;end&#39;: Timestamp(&#39;2021-12-13 18:51:00.046999808&#39;), <br>    &#39;note&#39;: &#39;&quot;GET /external/conversations&#39;, <br>    &#39;duration&#39;: 8,  <br>    &#39;correlation_id&#39;: &#39;181c810c-1ff6-ec9a-1395-5cc89a333be7&#39;<br>  },<br>  ...<br>]</pre><p>You can easily generate the timeline as follows:</p><pre>import plotly.express as px</pre><pre>category_order = sorted(list(set([e[&#39;thread&#39;] for e in events])))</pre><pre>fig = px.timeline(<br>    events, x_start=&quot;start&quot;, x_end=&quot;end&quot;, y=&quot;thread&quot;, <br>    hover_data=[&#39;note&#39;, &#39;duration&#39;, &#39;tenant&#39;, &#39;correlation_id&#39;], <br>    color=&#39;tenant&#39;, height=400, width=1000, <br>    category_orders={&#39;thread&#39;: category_order}<br>)</pre><pre>fig.show()</pre><h3>Conclusion</h3><p>Plotly makes it really easy to render timelines, and threads can be viewed as actors in a timeline. The resulting plot is interactive and useful for data exploration, but probably isn’t something you’ll publish without further work. This method becomes less useful if you have many hundreds of threads but hopefully gives you ideas of other ways you can use timelines to understand your applications at runtime.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c135bb17880" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/using-plotly-timelines-to-visualize-thread-activity-c135bb17880">Using Plotly Timelines to Visualize Thread Activity</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying TensorFlow Models in Microservices]]></title>
            <link>https://medium.com/quiq-blog/deploying-tensorflow-models-in-microservices-78d422d10c2c?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/78d422d10c2c</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[nomad]]></category>
            <category><![CDATA[model-serving]]></category>
            <category><![CDATA[deployment]]></category>
            <dc:creator><![CDATA[Kyle McIntyre]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 22:37:58 GMT</pubDate>
            <atom:updated>2021-12-08T22:37:58.690Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Hfu0HqgEhlmGTEUge7iWg.jpeg" /></figure><p>If you’re like us, you’ve got an application that’s built on a mature microservice architecture. Your microservices are built on docker images and primarily expose REST APIs. You’ve also got one or more TensorFlow models that you want to deploy as part of your application. You’re hoping that you can just wrap your model inside of a microservice. Why? Because your architecture has already tackled the following problems:</p><ul><li>Zero downtime upgrades &amp; version management</li><li>Redundancy, horizontal scalability, auto-scaling</li><li>Independent release cycles</li><li>Unified monitoring &amp; logging</li><li>Multi-tenancy concerns</li><li>Compliance &amp; regulatory concerns</li></ul><p>You view the world as a collection of microservices and you don’t really want to add something special to your environment. Is it possible to wrap your TF model in a microservice, and is it wise?</p><h3>Background</h3><p>First off, some background on our architecture:</p><ul><li>Pure HTTP/REST microservices built on docker images. We currently have 60 or so</li><li><a href="https://www.nomadproject.io/">HashiCorp Nomad</a> for container orchestration</li><li><a href="https://www.consul.io/">HashiCorp Consul</a> for service discovery</li><li>Docker containers based on Alpine or Ubuntu Linux</li></ul><p>It’s notable that we don’t use Kubernetes for container orchestration. If we did we might have tried using <a href="https://www.kubeflow.org/">Kubeflow</a> to deploy our models. However, we prefer Nomad for its more focused and lightweight operational nature.</p><p>When building a new microservice in our environment, there are some standard HTTP resources you need to implement:</p><ul><li>A healthcheck</li><li>A resource for forcible shutdown</li><li>A resource for monitoring (e.g. prometheus metrics)</li><li>Resources related to change tracking</li><li>A Swagger page</li></ul><p>In addition to the standard HTTP resources, there are implicit requirements for your service</p><ul><li>Registering yourself with Consul on startup</li><li>Log in a common format</li><li>Negotiate secrets in a standard way</li><li>Various security measures such as checking JWTs and implementing an IP allow-list for all inbound requests</li></ul><p>We have implementations of this core ‘contract’ in Scala, Go &amp; Python. Then individual microservices extend those implementations with actual application logic. The Python skeleton is based on Flask + GUnicorn.</p><p>Given this landscape, it’s <em>really</em> tempting to try running our TensorFlow model inside of a Flask-based microservice because we’ve already blazed that trail.</p><h3>A Naïve Approach</h3><p>For our first attempt, we tried the most obvious approach: we created a microservice container with the following contents:</p><ul><li>Our Flask microservice skeleton</li><li>Tensorflow &amp; associated dependencies</li><li>The model, statically built into the image</li><li>A REST API to invoke the model and perform the necessary preprocessing and postprocessing steps</li></ul><p>In this setup, the model was loaded statically during app initialization. The results of this approach were ….*drum roll*…. abysmal . The service would typically become unresponsive after a single invocation. What was the problem??</p><p>Unbeknownst to us, the Python tensorflow module isn’t set up very well for concurrency at the user level. Sure, it can parallelize computation with the best of them, but it’s designed for doing one task at a time. We don’t claim to be experts in this area of TensorFlow (or any of it, really) — but it’s related to the TF concept of a Session. Receiving multiple web requests and attempting to have them all run inferencing on a statically initialized model resulted in deterministic deadlock — even if they didn’t run concurrently. The architectural problem seemed to be that TensorFlow isn’t fork-safe, but we needed to use a forking web server.</p><p>We were able to work around the deadlock issue by initializing the model once per request (although it still seemed a bit shaky). But of course, this is totally unworkable from a scalability perspective. These are deep learning models after all! Hundreds of megabytes is totally common. We couldn’t afford the load latency or the crazy amounts of RAM it would require to service concurrent requests.</p><p>At this point we realized there must be a reason people went to the effort of building the separate <a href="https://www.tensorflow.org/tfx/guide/serving">TensorFlow Serving</a> project 😂</p><h3>TensorFlow Serving</h3><p>There are a lot of articles about TensorFlow Serving, but too many of them just repeat the same blurb from its homepage. Here’s our take on the project:</p><ul><li>It is <em>the</em> way to deploy TF SavedModel formats server-side. The main Python tensorflow project is unsuitable for this purpose.</li><li>The most important architectural aspect of TF serving is that it’s designed for request concurrency. A model only gets loaded once but can be leveraged multiple times, simultaneously</li><li>Almost as important: TF serving provides an abstraction around request batching, which is essential if you want to save on inferencing costs by leveraging a GPU/TPU</li><li>It’s kind of opinionated and it solves other problems you may not care about i.e. model versioning and upgrades</li></ul><p>After doing some research and experimentation with TensorFlow Serving, we were convinced it was the path forward. But, it seemed heavy in the sense that it’s meant to run as a gRPC or HTTP server. We were hoping for something more akin to a library. But, short of dissecting the TensorFlow Serving build and/or trying to write python wrappers around the C++ API, we were stuck with running TF Serving as well, a server. We considered three ways of doing this:</p><ol><li>Running TensorFlow Serving as it was meant to be run(?) — a specialized, networked entity within our environment that would be explicitly managed, almost like a database. This would require significant Ops support</li><li>Nomad has a concept of Jobs that encapsulate ‘something you need to do’ via container orchestration. Jobs can actually have multiple tasks, which can use different containers but kind of run next to each other. We considered running TF Serving in one task, and our microservice application in another.</li><li>Embedding TF Serving within the same container as our microservice application and running it as a child process</li></ol><p>Option #1 just didn’t feel right to us. Our microservice-tainted minds wanted to avoid a centralized database style approach &amp; associated operational maintenance costs.</p><p>Option #2 seemed promising, but the more we considered this approach the more we realized that a big part of the value proposition of a microservice is its <em>atomicity. </em>Creating two bound entities complicates issues like logging and monitoring.</p><h3>Wrapping TensorFlow Serving in a Microservice</h3><p>In the end we were able to get Option #3 working without much trouble. We were able to avoid building TF Serving from source because our containers were binary-compatible with the official TF Serving docker containers, enabling us to use docker’s <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage build pattern</a> to pluck the TF serving artifact out of their image and stick it in ours:</p><pre>ARG TF_SERVING_VERSION=latest<br>ARG TF_SERVING_BUILD_IMAGE=tensorflow/serving:${TF_SERVING_VERSION}-devel<br>FROM ${TF_SERVING_BUILD_IMAGE} as build_image</pre><pre>FROM docker.quiq.sh/quiqml-tf:2.4.4<br>COPY --from=build_image /usr/local/bin/tensorflow_model_server /usr/bin/tensorflow_model_server</pre><p>We then start up TF Serving during our web server initialization:</p><pre>tfx_child = subprocess.Popen([<br>            &quot;/usr/bin/tensorflow_model_server&quot;,<br>            &quot;--model_base_path=/models/ro_inferencer&quot;,<br>            &quot;--model_name=ro_inferencer&quot;,<br>            &quot;--rest_api_port=0&quot;,<br>            &quot;--port=0&quot;,<br>            &quot;--grpc_socket_path=/tmp/grpcsocket&quot;<br>     ])</pre><p>You might be wondering why we set rest_api_port and port to zero. We run our containers in host network mode. If we didn’t set these ports to zero, which disables them, TF Serving would actually bind ports on the host which is messy and doesn’t look very secure. Thankfully TF Serving supports a socket only communication mode (although this commits you to doing gRPC requests instead of REST).</p><p>For our healthcheck, we simply wrapped the TF Serving <a href="https://www.tensorflow.org/tfx/serving/api_rest#model_status_api">model status API</a>.</p><h3>What about Model Versioning?</h3><p>TF Serving supports multiple model versions and transitioning between them. However, we don’t use this feature at all. The reason is because we handle all versioning concerns at our microservice layer. We like this approach better because our models often have associated pre and post-processing steps that are performed ‘outside’ the model, but should still be versioned with it since they must match the model’s training conditions.</p><h3>Results</h3><p>This approach has worked very well for our needs. The resulting microservice is indeed atomic and is indistinguishable from our other services. We can say confidently that the service is stable and has consistent memory use (no leaks). Wrapping another server within your microservice perhaps feels a bit kludgy, but for us it’s incredibly pragmatic and valuable.</p><p>It’s important to note we still haven’t leveraged GPUs for model inferencing, but we have no reason to believe that it won’t work. We’ll just need to run ML services in a separate auto scaling group with different hardware specs. Doing that would allow us to potentially reduce hardware costs through batch inferencing — but plain old CPU is sufficient for our current use cases, models &amp; load.</p><h3>Acknowledgement</h3><p>This article, and the work underpinning it, was done in conjunction with my friend and coworker Talon Daniels.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78d422d10c2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/deploying-tensorflow-models-in-microservices-78d422d10c2c">Deploying TensorFlow Models in Microservices</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenVPN with Two-Factor authentication]]></title>
            <link>https://medium.com/quiq-blog/openvpn-with-two-factor-authentication-4dce9199c3af?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/4dce9199c3af</guid>
            <category><![CDATA[vpn]]></category>
            <category><![CDATA[2fa]]></category>
            <category><![CDATA[openvpn]]></category>
            <category><![CDATA[mfa]]></category>
            <dc:creator><![CDATA[Roman Vynar]]></dc:creator>
            <pubDate>Thu, 22 Apr 2021 02:40:59 GMT</pubDate>
            <atom:updated>2021-04-22T02:40:59.528Z</atom:updated>
            <content:encoded><![CDATA[<p>This article explains how to setup 2FA with OpenVPN server and how it’s supposed to work with VPN clients.</p><p>OpenVPN is one of the most popular open-source VPN server software. Nowadays when security plays a critical role we try to protect various authorization services by implementing multi-factor authentication. This introduces “something you have” in addition to “something you know” to verify your identity. In case of VPN, it is notably important.</p><p>Assuming you have already got OpenVPN server&lt;&gt;clients working and you just need to add 2FA capability. Let’s say we want to ask for OTP (one-time password) at least once a week, when started using a different client, when coming from a different IP or whenever user fully disconnects. Laptop sleep/awake is not considered as full disconnect unless your VPN client does this explicitly.</p><p><strong>Client config</strong></p><p>The client should do password auth with static challenge using the following options:</p><pre>auth-user-pass<br>static-challenge &quot;Enter your OTP&quot; 0</pre><p>Then the client software such as Tunnelblick, Viscosity will prompt for an OTP code after the regular username/password:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nesc8-2yVuiMo0Hd0SsFNA.png" /></figure><p><strong>Server config</strong></p><p>Server config should include the following options:</p><pre># User auth script.<br>auth-user-pass-verify /etc/openvpn/openvpn_otp_auth.py via-file<br>script-security 2<br>tmp-dir /dev/shm<br># Allow to reconnect automatically w/o triggering a static-challenge prompt for OTP.<br># Generate auth-token for 12h on every control channel/connection renegotiation.<br>auth-gen-token 43200<br># Perform data channel renegotiation every 1h (default), so the server can validate auth-token.<br>reneg-sec 3600</pre><p>Note, it’s just an excerpt related to 2FA and you need other options to employ a secure setup, TLS authentication etc. The script <em>openvpn_otp_auth.py</em> mentioned is the key item and we have an example for you below. However, at first let’s see how things work together.</p><p><strong>How it works</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CIQXeKCxrbtsj4MV3HCjWg.png" /></figure><p>There are 2 channels established when user session is created: control channel (CC) and data channel(DC).</p><p><em>openvpn_otp_auth.py </em>is run only on CC renegotiation which is triggered on specific events listed on the picture. Only from the script we can do all kinds of session validation, password and OTP code checks.</p><p>DC renegotiation doesn’t involve the script run and only checks auth-token expiration and when expired it triggers CC renegotiation when our script is run.</p><p><strong>Python script</strong></p><p>Here is a sample of <em>openvpn_otp_auth.py </em>written in Python: <a href="https://github.com/roman-vynar/random-scripts/blob/main/openvpn_otp_auth_sample.py">https://github.com/roman-vynar/random-scripts/blob/main/openvpn_otp_auth_sample.py</a></p><p>You can get a clue what’s done there and use it on your own. The script does a password check, OTP code check, VPN client version check and IP validation.</p><p>Such setup should work great with Tunnelblick (however you need to disable “Disconnect when computer goes to sleep” in Settings &gt; Advanced) and Viscosity VPN clients.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dce9199c3af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/openvpn-with-two-factor-authentication-4dce9199c3af">OpenVPN with Two-Factor authentication</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Advanced health check for Openresty (Nginx) backend]]></title>
            <link>https://medium.com/quiq-blog/advanced-health-check-for-openresty-nginx-backend-ddf45a5a2a70?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/ddf45a5a2a70</guid>
            <category><![CDATA[nginx]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[lua]]></category>
            <category><![CDATA[health-check]]></category>
            <category><![CDATA[openresty]]></category>
            <dc:creator><![CDATA[Lev Petrushchak]]></dc:creator>
            <pubDate>Thu, 17 Dec 2020 00:11:04 GMT</pubDate>
            <atom:updated>2020-12-17T00:11:04.091Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VACGK8q1PEztDUDXo1rzfQ.jpeg" /></figure><p>At Quiq we are using <a href="https://github.com/openresty/openresty">Openresty</a> as a replacement for the Nginx web server. This is an open-source Web Platform which extends Nginx with Lua scripting language.</p><p>I have faced an interesting case. Let’s say there are two upstream (backend) servers. One is main and used to serve traffic and another one is used by a separate “<em>server</em>” block for other purposes. Both backends are listening on different ports on <em>localhost </em>together with Openresty server<em>. </em>In order to send traffic to the main upstream, I need to know the health status of both upstreams. In case when either first or second is down, I need Openresty to return 503.</p><p>There is an advanced library for doing health checks <a href="https://github.com/openresty/lua-resty-upstream-healthcheck">https://github.com/openresty/lua-resty-upstream-healthcheck</a> and while I figured out how to use this advanced library, I came to a more simple and elegant solution — use direct LUA in access_by_lua_block</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b457c52b5013f3af24358d726bdf775/href">https://medium.com/media/0b457c52b5013f3af24358d726bdf775/href</a></iframe><p>So here I used core <strong>resty.http </strong>library and simple GET method to get the status of the server. The main backend must return <strong>HTTP 200</strong> and “other” backend <strong>HTTP 204</strong>. As simple as that.</p><p>The final <em>nginx.conf</em> would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4678f85de269fe762d69e9f72fdc4ee/href">https://medium.com/media/a4678f85de269fe762d69e9f72fdc4ee/href</a></iframe><p>Worth to mention, Nginx itself has two built-in types of health checks for upstreams. Active and passive <a href="https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/">https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/</a> Passive health check is not quite what I needed and active health check is available only in Nginx Plus version</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddf45a5a2a70" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/advanced-health-check-for-openresty-nginx-backend-ddf45a5a2a70">Advanced health check for Openresty (Nginx) backend</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Redis as a highly available database]]></title>
            <link>https://medium.com/quiq-blog/redis-as-a-highly-available-database-22322263a58e?source=rss----ff232b4e5904---4</link>
            <guid isPermaLink="false">https://medium.com/p/22322263a58e</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[haproxy]]></category>
            <category><![CDATA[high-availability]]></category>
            <dc:creator><![CDATA[Lev Petrushchak]]></dc:creator>
            <pubDate>Fri, 24 Apr 2020 19:19:22 GMT</pubDate>
            <atom:updated>2020-04-24T19:19:22.243Z</atom:updated>
            <content:encoded><![CDATA[<p>Once upon a time, we decided to store some data. We have chosen Redis — in-memory data structure store, which may be used as a database, cache and message broker and we are using pretty much all of it.</p><p>We started with one popular hosted Redis solution which in our case was nice and simple. But as the product grew, we were getting more and more strict requirements, like 100% uptime, control over maintenance windows, splitting DBs with no downtime, and so on.</p><p>Moreover, if your company wants to pass a security audit (like SOC 2) you have to keep all data encrypted. This, of course, adds more overhead to your application and latency to requests.</p><p>As a result, we decided to move out of a remotely hosted service in order to keep Redis and application services together in the same place.</p><p>Since we are using AWS as our main cloud provider, it was obvious to try Amazon ElastiCache Redis. And here we go again. In terms of Amazon, it’s just a cache and they don’t seem to care about data loss. When you need to reboot the whole cluster, all data will be flushed. In case of failure, acceptable failover time is near 1 minute which was not ok for us. One more fun thing about ElastiCache is that you can’t change the DB password, you just have to recreate your database.</p><p>It was more than enough now to understand that we needed something custom. Luckily, Redis is open source and it has Sentinel as HA so we decided to try it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*YVCJZgyYuj6RUwkL" /></figure><p>After lots of hard research and testing, we finally built it. <em>High available Redis cluster </em><a href="https://github.com/Quiq/ha-redis"><em>https://github.com/Quiq/ha-redis</em></a> which can lose master and failover will take up to 7 seconds. What is most important is that we have confidence that no accidental writes will happen to old master if it was simply restarted. With Haproxy we can have SSL encrypted transport from application to Redis host (which is still not yet supported by Redis itself). The application is connected to Sentinels and can follow them with all failovers. With additional Python scripts, we can make manual failover in less than 2 seconds. We covered a lot of other edge cases to make a reliable, highly available Redis cluster.</p><p>Please try out our solution and send us any feedback. This code has been used in production since last June 2019 with no downtime or data loss. This included upgrading Redis and kernels. Having an always-on database is one of the biggest challenges to startups so hoping other companies will find this helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22322263a58e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/quiq-blog/redis-as-a-highly-available-database-22322263a58e">Redis as a highly available database</a> was originally published in <a href="https://medium.com/quiq-blog">The Quiq Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>