<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Diogok on Medium]]></title>
        <description><![CDATA[Stories by Diogok on Medium]]></description>
        <link>https://medium.com/@diogok?source=rss-55a9db13f0b3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T996314k4m-wGX0-pemdzQ.jpeg</url>
            <title>Stories by Diogok on Medium</title>
            <link>https://medium.com/@diogok?source=rss-55a9db13f0b3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 01:14:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@diogok/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Guaranteed message delivery with RabbitMQ]]></title>
            <link>https://blog.devgenius.io/guaranteed-message-delivery-with-rabbitmq-5211cff5f1e3?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5211cff5f1e3</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[messaging]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Sun, 28 Jun 2020 09:49:53 GMT</pubDate>
            <atom:updated>2020-07-01T07:42:44.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J-qRZYLVksm_iCT7" /><figcaption>Photo by <a href="https://unsplash.com/@mparzuchowski?utm_source=medium&amp;utm_medium=referral">Michał Parzuchowski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>RabbitMQ is a mature and stable messaging server that uses the AMQP protocol to interchange messages between producers and consumers.</p><p>It can serve several uses-cases, including: transient channels, fan-out messages, federation and others.</p><p>Here I will explain some (maybe all…) configurations and parameters for my most common use-case: Guaranteed message delivery for worker job queues.</p><p>This means that if a message is accepted by RabbitMQ, it will be consumed and only removed from the queue once acknowledge.</p><p>This will cover exchange, queue, consumer and producer setup. And will not cover RabbitMQ server reliability and clustering. I also do not care for ordering as I use multiple consumers most of the time.</p><h3>Quick RabbitMQ recap</h3><p>To use rabbitmq you open a connection to the server. On this connection you can open one or more channels, those channels are what you use to publish and consume messages.</p><p>Recommendation is that you publish in one channel and consume from another. Also in most implementations channels are not thread-safe, so you need one channel per thread or make sure to synchronize its access.</p><p>To “put a message on rabbitmq” you publish it to an exchange. An exchange receives a message and route it to a queue.</p><p>To get messages from rabbitmq you consume from a queue. To remove messages you ack or reject them. You can also nack and requeue, to keep the message on the queue.</p><p>Declare is the act of creating an exchange and a queue. Bind is what connect an exchange to a queue.</p><h3>Not losing your exchanges and queues</h3><p>First we need to make sure that our exchanges and queues survive restarts and disconnects, for this we make our exchanges and queue “durable”.</p><p>For exchange this means that `durable=true`, `auto-delete=false` and `internal=false` need to bet set as arguments when you declare it.</p><p>For queue this means that `durable=true`, `auto-delete=false` and `exclusive=false` need to be set as arguments when you declare it. But there are more arguments for the queue in a few paragraphs below.</p><p>This means that the exchange and queue will continue to exist even if no more client exist and in case of server restart. The declared exchange and queue are durable.</p><p>You will also need to bind the exchange to a queue, to make the messages accessible.</p><h3>Not losing rejected messages</h3><p>Now, when consuming messages fails they are rejected. If you want to catch those failed messages, you set up a dead-letter-queue.</p><p>A dead-letter-queue , accompanied by a dead-letter-exchange, is a way for you to capture rejected messages from the queue.</p><p>You set up them up as regular exchange, normally suffixed with “.dlx” and queue ending in “.dlq” and bind them. Also remember to make both durable.</p><p>When declaring you queue you setup `x-dead-letter-exchange=exchange-name.dlx` and `x-dead-letter-routing-key=` as arguments.</p><p><strong>Summary of exchange and queues declaration</strong></p><p>You declare, in order:</p><ul><li>A durable exchange</li><li>A durable exchange for failed messages (dlx)</li><li>A durable queue for failed messages (dlq)</li><li>Bind dlx to dlq</li><li>A durable queue, with dead-letter arguments</li><li>Bind the first exchange to the last queue</li></ul><p>Of course this is in case of a single exchange to a single queue, if you want to route to many queues you need to repeat the dlq and queue steps.</p><h3>Consuming messages safely</h3><p>There are actually more than on way to consume messages in AMPQ, but the common and recommended one is using the consume method.</p><p>You should set `auto-ack` to false, and be sure to only ack messages when they are good to be removed.</p><p>If you fail to process a message you should reject it so they go to the DLQ.</p><p>If you fail to ack or reject the messages, it will go back to the queue eventually.</p><p>This will ensure messages only leave the queue when you are done with them, and that they go to the dlq in case of failure.</p><p>Once you consume and ack a message in the DLQ it will be gone. You can also shove the messages from the DLQ back into the original queue to reprocess.</p><h3>Publishing safely</h3><p>Having a durable queue does not mean a durable message. Messages will still be kept in memory and be lost on restart or in case of non-routable messages.</p><p>To make sure a message will be routed to a queue you set it as `mandatory=true`. This means publish will only return success if the message end up in a queue, and you receive error if the message is not routable (like no binding on the exchange, or a bad routing key rule).</p><p>To make sure the message goes to disk you set `persistent=true`. This will make sure rabbitmq write it to disk on the queue.</p><p>And for last. you should wait for confirmation that the message was written.</p><p>There are other methods such as transactions, but the faster and recommended way is to use waiting for confirmation.</p><p>You call the confirm or wait_for_confirms on an open channel to wait for rabbitmq to confirm that the messages sent on this channels were written to where they should.</p><p>You can wait on each message or publish a batch and wait once. Waiting is per channel.</p><p>It can also be useful to use `lazy` messages to control memory usage to make rabbitmq write to disk as soon as possible. Unless you use quorum queue, but that is for another post.</p><h3>Conclusion</h3><p>These are all the measure that I know to make sure your rabbitmq messages are safe. Apart from cluster, but that is another story.</p><p>Important to note that each persistence, each guarantee we add will impact performance. Direct transient messages are probably the faster you can go. That said, this is still a very fast and efficient messaging server.</p><p>If you keep a close eye on memory consumption, disk space and number of open connections to your server, RabbitMQ is very likely to not fail you, even without all above measures.</p><p>So, did I miss something? Got any other tips?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5211cff5f1e3" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/guaranteed-message-delivery-with-rabbitmq-5211cff5f1e3">Guaranteed message delivery with RabbitMQ</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Technologies I learned and that I do not use]]></title>
            <link>https://medium.com/@diogok/technologies-i-learned-and-that-i-do-not-use-e7e8af99e67d?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7e8af99e67d</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Wed, 01 Apr 2020 12:04:23 GMT</pubDate>
            <atom:updated>2020-04-07T06:02:30.432Z</atom:updated>
            <content:encoded><![CDATA[<p>I was reflecting the other day on the tools I know and, specially, the ones I do not know. That led me to think “why do I not know these tools by now?”.</p><p>One of the reasons is that I learnt other things, and is what lead to this text.</p><p>That might sound obvious, but I actually have to choose what to invest my time, and make trade offs.</p><p>Here I share some technologies I learned and invested and that did not really pay off, that I abandoned or that just made me take a long run.</p><p>These are not the most recent or older, most bad or good, these are just the ones I remember better and find most interesting to share.</p><h3>JavaFX</h3><p>Now, this was a bet. For those for do not know, JavaFX was a new UI toolkit for java. For desktop, that is.</p><p>Back when JavaFX came out it was already time for “rich web apps”, so people had already moved on from desktop apps, so it this was a risky shot at least.</p><p>What I found interesting about it was that it had a functional programing style (or, at least, function as first class elements), two-way binding of properties and the UI and heavy focus on components and composability. You could even style components with what resembled CSS.</p><p>That said, it was still java on desktop, it looked slow at that time, and people were already on the web. Apart from that, I had no desktop app to build.</p><h3>Docker before v1.0</h3><p>There was actually a turning point in docker where it became stable, sometime after v1.0.</p><p>I began investing in docker around v0.5 or something. At that time a lot of functions were missing, and docker-compose was either not existent or very limited.</p><p>Due to being so new and lacking functionality, I have written and adopted several wrappers around docker for that year to provided much needed resources.</p><p>It was interesting that at each release of docker or docker-compose I could remove one extra tool or remove on more script from my custom setup in favor of the official docker way.</p><p>Still this one payed off in the end, as docker is the industry standard, but being an early adopter was a long and hard run.</p><h3>CouchDB</h3><p>My favorite DB. CouchDB offered a schemaless, HTTP and Map-reduce enabled database that was very resilient and easy to use. It even included replication over http and a feed endpoint. It was really awesome.</p><p>I actually got to build a major system in my career on top of it, paired with Elasticsearch. But after that, it did no gain enough traction.</p><p>MongoDB basically killed it, becoming the go to nosql at that time. And the fact that CouchDB was slow, view building specially, did not help.</p><p>CouchDB is still improving, now includes search, clustering is easier and other goodies.</p><p>Now a days I mostly use PostgreSQL and Elasticsearch, paired with something from the cloud my system might be using (dynamodb, firestore…).</p><h3>JS Games and Augmented Reality</h3><p>Now this was just for fun, but I always go back and forth in learning something related to games in the browser. I find it really fascinating and I really liked to work with the Canvas on HTML5.</p><p>One of the experiments that I spent most time with was building AR experiences for mobile browser. That was hard, limited experience and very much fun.</p><p>This was obviously just a hobby, and it did take time away from learning tools that I would actually use on a daily basis. But it was worth it nevertheless.</p><h3>Lua</h3><p>My favorite language that I never used.</p><p>Lua is an easy, simple and small language that you can embed anywhere and that have implementation on every platform. Used a lot in games, in nginx by resty (and Kong, based on that) and who knows where else.</p><p>I do know it well enough to safely use it in production. Even right now I am testing Fennel, a lisp in lua. But I have never used it for anything serious. Maybe a kong plugin soon?</p><h3>Conclusion</h3><p>I do not regret learning any of those, they all added something interesting, I had fun playing with as hobby and some were really useful for some time.</p><p>Now, time to learn something new that might or might not stick.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7e8af99e67d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient Clojure multistage docker images, with java and native-image]]></title>
            <link>https://medium.com/@diogok/efficient-clojure-multistage-docker-images-with-java-and-native-image-c7c80b93c8?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7c80b93c8</guid>
            <category><![CDATA[clojure]]></category>
            <category><![CDATA[graalvm]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Wed, 15 Jan 2020 17:33:10 GMT</pubDate>
            <atom:updated>2020-01-15T17:33:49.420Z</atom:updated>
            <content:encoded><![CDATA[<p>Here I explore a few optimization when building docker images for your clojure apps.</p><h3>Image versions</h3><p>One easy way to make it faster for you local development and for CI/CD is to just use smaller images, and to reuse images.</p><p>Using common public images make it more likely that you will use the same image over and over again, also pinning to the most specific version help assure the base image have not changed between builds. Choosing alpine or slim images can reduce the image size.</p><p>For the base image I use clojure:openjdk-13-tools-deps-slim-buster and openjdk:13-slim-buster. I prefer buster images over alpine due to compatibility with most native libs, and rumor has it that due to libc versions it can be faster.</p><h3>Build cache</h3><p>The next step is to leverage the docker image build cache, so the order of the steps you use for building the image matter.</p><p>You generally want to set non-changing configurations like ENV, WORKING_DIR and EXPOSE first.</p><p>To decouple installing the deps from actually building the artifact, the next thing you add is your deps files and install it.</p><p>The code is what changes most, so it goes last, right before actually building the uberjar.</p><h3>Garbage collector and heap size</h3><p>Not optimization of images but a tip.</p><p>Current versions of openjdk support running in containers, so it is best to use relative memory limits and container support with -XX:+UseContainerSupport and -XX:MaxRAMPercentage=85 just we don`t have to mess with Xmx or Xms at runtime anymore.</p><p>Remember that the JVM uses a little more memory than the heap, so give it some extra space.</p><h3>Multistage</h3><p>To further reduce the image size and remove clutter from base image, we can start with a JVM only image and copy the generated jar over. This will remove clojure specific tools, sources-code, intermediate artifacts and others.</p><h3>Result</h3><p>After applying these tips, here is the resulting Dockerfile:</p><pre>FROM clojure:openjdk-13-tools-deps-slim-buster as builder<br><br>WORKDIR /usr/src/app<br><br># install cambada builder, as it will not change<br>RUN clj -Sdeps &#39;{:deps {luchiniatwork/cambada {:mvn/version &quot;1.0.0&quot;}}}&#39; -e :ok <br><br># install main deps, sometimes change<br>COPY deps.edn /usr/src/app/deps.edn<br>RUN clj -e :ok<br><br># add files and build, change often<br>COPY resources/ /usr/src/app/resources<br>COPY src/ /usr/src/app/src<br>RUN clj -A:uberjar<br><br># use clean base image<br>FROM openjdk:13-slim-buster</pre><pre># set static config<br>ENV PORT 8080<br>EXPOSE 8080</pre><pre># set the command, with proper container support<br>CMD [&quot;java&quot;,&quot;-XX:+UseContainerSupport&quot;,&quot;-XX:MaxRAMPercentage=85&quot;,&quot;-XX:+UnlockExperimentalVMOptions&quot;,&quot;-XX:+UseZGC&quot;,&quot;-jar&quot;,&quot;/usr/src/app/app.jar&quot;]</pre><pre># copy the ever changing artifact<br>COPY --from=builder /usr/src/app/target/app-1.0.0-SNAPSHOT-standalone.jar /usr/src/app/app.jar</pre><h3>Native image</h3><p>Now, for bonus, we can also setup a native image using graalvm tools. This will reduce image size by a lot, as it does not depend on the JVM, and potentially reduce memory usage.</p><p>Note that native-image is only compatible with linux x86-64, and is new tech, so a lot o frameworks can break it. It also does not give better performance (latency, throughput, gc times…) comparing to JVM version.</p><p>Some flags may change depending on your tools of choice.</p><p>The native image will build upon the previous tips, but has to be based of java 11 instead of latest.</p><p>Here is the dockerfile:</p><pre>FROM clojure:openjdk-11-tools-deps as builder<br><br>WORKDIR /usr/src/app<br><br>RUN clj -Sdeps &#39;{:deps {luchiniatwork/cambada {:mvn/version &quot;1.0.0&quot;}}}&#39; -e :ok <br><br>COPY deps.edn /usr/src/app/deps.edn<br>RUN clj -e :ok<br><br>COPY resources/ /usr/src/app/resources<br>COPY src/ /usr/src/app/src<br><br>RUN clj -A:uberjar</pre><pre># the native-image tool<br>FROM oracle/graalvm-ce:19.3.1-java11 as native<br><br>RUN gu install native-image<br><br>WORKDIR /usr/src/app<br><br>COPY --from=builder /usr/src/app/target/app-1.0.0-SNAPSHOT-standalone.jar /usr/src/app/app.jar</pre><pre># lots of specific flags<br>RUN native-image --allow-incomplete-classpath --static --enable-http --no-fallback  --no-server --initialize-at-build-time --report-unsupported-elements-at-runtime  --initialize-at-run-time=io.netty.channel.epoll.EpollEventArray,io.netty.channel.unix.Errors,io.netty.channel.unix.IovArray,io.netty.channel.unix.Socket,io.netty.channel.epoll.Native,io.netty.channel.epoll.EpollEventLoop,io.netty.util.internal.logging.Log4JLogger -jar app.jar app<br><br># use clean image<br>FROM scratch<br><br>COPY --from=native /usr/src/app/app /app<br><br>CMD [&quot;/app&quot;]</pre><p>Note that there is a lot of netty specific config there, as I use aleph for HTTP.</p><h3>Opensource example</h3><p>All this experiments and other framework choices are available at my github <a href="https://github.com/diogok/klj-api">klj-api </a>project.</p><p>Hope it helped.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7c80b93c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An incomplete comparison of geospatial file formats]]></title>
            <link>https://medium.com/@diogok/an-incomplete-comparison-of-geospatial-file-formats-bd6c870793e1?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd6c870793e1</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[geojson]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[topojson]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 11:46:01 GMT</pubDate>
            <atom:updated>2018-10-17T11:46:01.964Z</atom:updated>
            <content:encoded><![CDATA[<p>A few years ago I was working with geospatial analysis and came to study and compare a few alternative file formats for storing and, especially, transferring such data.</p><p>I worked mainly with vector data (polygons and points) and served that for the web. The motivation of this work was to test the gains and cost of TopoJSON, and how it fared in size (important for transfer time) and encoding time (important for server resource usage).</p><p>A more complete description of advantages of the available formats can be found at <a href="http://switchfromshapefile.org/">Shapefile must die!</a> , a good resource even if a bit strong on words (FYI: As a developer, I really dislike working with shapefile).</p><p>Even if I never got to finish it, I guess I can share the results anyway.</p><h3>The contender file formats</h3><p>First, let`s make some groups. Here we will compare three things:</p><ul><li>Data structure</li><li>Format encoding</li><li>Compression</li></ul><p>For <strong>data structure,</strong> I am are comparing how data is organized inside the file:</p><ul><li>Shape</li><li>GeoJSON structure</li><li>TopoJSON structure</li></ul><p>For <strong>format encoding</strong>, I am comparing how this structure is serialized:</p><ul><li>Binary (shapefile)</li><li>JSON</li><li>MessagePack</li></ul><p>For <strong>compression</strong>, I am comparing compression algorithms:</p><ul><li>None</li><li>Deflate/GZIP</li><li>XZ/LZMA</li></ul><p>The choices of what to test were made based on tooling availability on programing language(Java/Clojure) and ease of use from the web(Javascript).</p><p>A few notes of formats not tested:</p><p><strong>CSV </strong>was not tested cause it is not as standard as it looks with several combinations of separators, enclosing and record separator. And also would have to compare WKT and WKB and other geometry encodings. But it would probably fare well as it can be streamed and well compressed.</p><p><strong>Spatialite </strong>is a bit more complex to handle, as you will not only need an SQLite library but with extensions for Spatialite. Also would have to define table structure and such.</p><p>Given more time I would include more tests on both.</p><p>Shapefile was only tested as a baseline.</p><h3>Overall results</h3><p>The code and results can be found at <a href="https://github.com/diogok/geof">my github</a>.</p><pre>| Structure | Format  | Compress | Size  | Time |<br>|-----------|---------|----------|-------|------|<br>| Shapefile | -       | -        | 5MB   | -    |<br>| Shapefile | -       | zip      | 3.2MB | -    |<br>| Geo       | JSON    | -        | 9MB   | 10s  |<br>| Geo       | JSON    | gz       | 2.5MB | 11s  |<br>| Geo       | JSON    | xz       | 1.4MB | 21s  |<br>| Geo       | MsgPack | -        | 5.2MB | 9s   |<br>| Geo       | MsgPack | gz       | 3.5MB | 11s  |<br>| Geo       | MsgPack | xz       | 1.7MB | 15s  |<br>| Topo      | JSON    | -        | 524KB | 22s  |<br>| Topo      | JSON    | gz       | 84KB  | 20s  |<br>| Topo      | JSON    | xz       | 64KB  | 22s  |<br>| Topo      | MsgPack | -        | 256KB | 21s  |<br>| Topo      | MsgPack | gz       | 76KB  | 20s  |<br>| Topo      | MsgPack | xz       | 60KB  | 22s  |</pre><p>Shapefiles are the baseline, they do not compress very well.</p><p>As expected raw GeoJSON files are huge, but as text files, they compress very well, and are reasonably fast to encode.</p><p>TopoJSON files are minimal in size but take a long time to encode. Also not included in this test is the fact that topology encoding takes a lot of memory, as it has to hold the whole collection to iterate over it.</p><p>MessagePack as a format offers reasonable space efficiency being a binary format and encode faster. They add more complexity to the web and lose most gains after compression. They are faster to read/write on the server but slower on the browser.</p><p>Deflate/GZ offer expected compression results. They are standard on the web which makes it an easy choice, you server already have them, and so does the browser.</p><p>LZMA/XZ is a bit harder to use on the browser, but it is able to deliver even more.</p><h3>A few conclusions</h3><p>This test is incomplete and you should run your own on your set of data to get more practical results to your reality.</p><p>But here is my take on it:</p><ul><li>Shapefiles sucks cause they are a lot of files and with several limitations</li><li>If nothing else, at least enable DEFLATE on your geojson serving</li><li>TopoJSON is complex to deal with and expensive to encode</li><li>TopoJSON offers insane compaction, specially on polygons with a lot of shared lines</li><li>MsgPack offers nice compaction over text, but most of that is lost over compression</li><li>LZMA/XZ adds a little complexity but it gave good gains on bigger files</li></ul><p>So, if you can afford to encode only once with resources to spare: TopoJSON with XZ gives the most value. If you have to encode/decode on the fly: GeoJSON with XZ. If you can spare disk, offer both: TopoJSON and GeoJSON with XZ and GZ.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd6c870793e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Terraform workspaces and locals for environment separation]]></title>
            <link>https://medium.com/@diogok/terraform-workspaces-and-locals-for-environment-separation-a5b88dd516f5?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5b88dd516f5</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Mon, 11 Sep 2017 12:01:01 GMT</pubDate>
            <atom:updated>2019-11-13T21:33:27.649Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://terraform.io/">Terraform</a> is this amazing tool to provision and manage changes on your cloud infrastructure while following the great practice of keeping your infrastructure-as-code.</p><p>One common need on infrastructure management is to build multiple environments, such as testing and production, with mostly the same setup but keeping a few variables different, like networking and sizing.</p><p>The first tool help us with that is <a href="https://www.terraform.io/docs/state/workspaces.html">terraform workspaces</a>. Previously called environments, it allows you to create different and independent states on the same configuration. And as it’s compatible with remote backend this workspaces are shared with your team.</p><p>As an example, let’s work with the following simple infrastructure:</p><pre>provider &quot;aws&quot; {<br>  region= &quot;us-east-1&quot;<br>}</pre><pre>resource &quot;aws_instance&quot; &quot;my_service&quot; {<br>  ami=&quot;ami-7b4d7900&quot;<br>  instance_type=&quot;t2.micro&quot;<br>}</pre><p>Now we have defined a single aws ec2 instance andterraform apply will have your testing server up.</p><p>But that is only one environment, in this simple example one might think that it would be ok to simple replicate and call one service “testing_my_service” and the other “prod_my_server”, but this approach will quickly lead to confusion as your setup grows in complexity and more resources are added.</p><p>What you can do instead is use workspaces to separate them.</p><pre>terraform workspace new production</pre><p>With this, you are now in production workspace. This one will have the same configuration, since we are in the same folder and module of terraform, but nothing created. Thus, if you terraform apply it will create another server with the same configuration but not changing the previous workspace.</p><p>To go back to testing you can terraform workspace select default , since we are using default as the testing environment to make sure we are not working on production by mistake.</p><p>But, obviously, there are differences between testing and production, and the first approach would be to use variables and an if switch on the resources. Instead a better approach would be to use the recently introduced <a href="https://www.terraform.io/docs/configuration/locals.html">terraform locals</a> to keep resources lean of logic:</p><pre>provider &quot;aws&quot; {<br>  region= &quot;us-east-1&quot;<br>}</pre><pre>locals {<br>  env=&quot;${terraform.workspace}&quot;</pre><pre>  counts = {<br>    &quot;default&quot;=1<br>    &quot;production&quot;=3<br>  }</pre><pre>  instances = {<br>    &quot;default&quot;=&quot;t2.micro&quot;<br>    &quot;production&quot;=&quot;t4.large&quot;<br>  }</pre><pre>  instance_type=&quot;${lookup(local.instances,local.env)}&quot;<br>  count=&quot;${lookup(local.counts,local.env)}&quot;<br>}</pre><pre>resource &quot;aws_instance&quot; &quot;my_service&quot; {<br>  ami=&quot;<a href="https://console.aws.amazon.com/ec2/home?region=us-east-1#launchAmi=ami-7b4d7900">ami-7b4d7900</a>&quot;<br>  instance_type=&quot;${local.instance_type}&quot;<br>  count=&quot;${local.count}&quot;<br>}</pre><p>The main difference from variables is that locals can have logic in them instead of in the resources, while variables allow only values and pushed the logic into the resources.</p><p>One thing to keep in mind is that terraform is rapidly evolving and is worth to keep an eye on it’s changes to make sure you making the most of it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5b88dd516f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Golang, “static” binaries, cross-compiling and plugins]]></title>
            <link>https://medium.com/@diogok/on-golang-static-binaries-cross-compiling-and-plugins-1aed33499671?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/1aed33499671</guid>
            <category><![CDATA[plugins]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Wed, 10 May 2017 22:29:48 GMT</pubDate>
            <atom:updated>2017-05-10T22:29:58.487Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a simple sum up of a few experiences I had with golang with regarding to compiling binaries, “static” binaries, cross compiling and using plugins.</p><h3>The basics</h3><p>We all start on the same basic, to compile a simple go program you do:</p><pre>$ go build -o mybin *.go</pre><p>That will generate a binary “mybin” at your folder. A few things for this to work:</p><ul><li>It’s a main package</li><li>It have a func main()</li></ul><h3>Cross compiling</h3><p>That is awesome, but now I actually want this to work on my raspberry pi or my Scaleway arm servers. How do we go?</p><pre>$ GOOS=linux GOARCH=arm GOARM=6 go build -o mybin-arm *.go</pre><p>That will build an linux + arm (armel) compatible binary (think raspberry pi 3 and zero), depending on your devices you might change to GOARCH=arm7 and skip the GOARM flag (some more high end devices).</p><p>In case you wonder, if you are actually runing an arm and want to compile to x64:</p><pre>$ GOOS=linux GOARCH=amd64 go build -o mybin-arm *.go</pre><p>Now, you might be missing some dependencies if you need to use CGO : arm-linux-gnueabi. To fix see next part.</p><h3>Debian cross-compile dependencies</h3><p>Boring but easy, actually :</p><pre># aptitude install cross-gcc-dev -t jessie-backports<br># dpkg --add-architecture armhf<br># dpkg --add-architecture armel<br># aptitude update<br># aptitude install crossbuild-essential-armel crossbuild-essential-armhf</pre><p>Them compile with</p><pre>$ CC=arm-linux-gnueabi-cc GOOS=linux GOARCH=arm GOARM=6 go build -o mybin-arm *.go</pre><h3>More binary</h3><p>Now, for when things like libc version differences, net libraries, missing certificates and such gives you trouble, you go for more static binaries.</p><pre>$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo -ldflags &#39;-w&#39; -o mybin *.go</pre><p>The command is growing, right?</p><p>Now we disabled CGO , which you may not need depending on the libs and things you do, forced a rebuild with -a and tagged netgo to make sure we use built-in net package and not the system’s one.</p><p>Disabling CGOalso removes the need for the cross-compile dependencies.</p><p>This ldflags -w just disables debug letting the file be smaller.</p><p>But if you actually need CGO integration, you might consider this:</p><pre>$ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo -ldflags &#39;-w -extldflags &quot;-static&quot;&#39; -o mybin *.go</pre><p>This will make the linked C part also static into the binary, reducing compatibility risk.</p><p>And if you are running in docker (I am) you can use a very small image, such as the following Dockerfile:</p><pre>FROM scratch<br>COPY mybin /mybin<br>CMD [&quot;/mybin&quot;]</pre><p>This means your docker image is actually just your binary, no bloat included. For now.</p><h3>Library vs Binary</h3><p>This is sort of a thing a part, but sometimes you might want to build a library but that also have a default binary.</p><p>Instead of making different projects, you can have you library with the desired package name, and inside this library have folder and file called “cmd/myproj/main.go”.</p><pre>package main<br>import &quot;myproj&quot;<br>func main() { <br>}</pre><p>Now you can use the library in other projects by simply importing but you can make a binary out of it:</p><pre>$ go build -o mybin cmd/myproj/main.go</pre><p>Bonus point since you do not need to compile *.go.</p><h3>Plugins</h3><p>Now this is the fun part that made this text happen. I was very happy with my binaries with netgo and no cgo included, but them I decided to try and use plugins.</p><p>Don’t worry, it’s not horrible, but it has a few gotchas.</p><ul><li>You will need CGO . The plugin system uses the linker;</li><li>The plugin itself needs to be a main package, but can’t have a main func;</li><li>It can import your project as a library;</li><li>Can not use static compiled binaries;</li><li>Cross-compiles work, but..</li><li>It only works on Linux, and…</li><li>Not working on ARM.</li></ul><p>So, <a href="https://github.com/golang/go/issues/18397">golang plugin not working on ARM</a> kind let me down.</p><p>Also if it uses plugin it can hardly be static. But that means you can’t use CGOENABLED=0 and can’t use -extldflags &quot;-static&quot; , which also makes me need FROM debian:jessie as my base Dockerfile.</p><p>So, you can build the plugin as follows:</p><pre>$ CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -buildmode=plugin -a -tags netgo -o plugin.so plugin.go</pre><p>Attention to buildmode=plugin and plugin.so as the plugin extension.</p><h3>Bottom line</h3><p>The story probably does not end here, while I was satisfied with the results, not working on ARM broke me a little. So I will have to hold on using plugins for some cases. Still, a nice resource to have.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1aed33499671" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Providing extinction risk assessments for biodiversity with Elasticsearch]]></title>
            <link>https://medium.com/@diogok/providing-extinction-risk-assessments-for-biodiversity-with-elasticsearch-b4a8bded9539?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4a8bded9539</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[biodivesity]]></category>
            <category><![CDATA[conservation]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Fri, 24 Feb 2017 14:11:22 GMT</pubDate>
            <atom:updated>2017-02-24T14:12:29.160Z</atom:updated>
            <content:encoded><![CDATA[<p>I am very proud to announce that the project by <a href="http://eduardo.dalc.in/">Eduardo Dalcin</a> and me, developed inside Brasil’s <a href="http://cncflora.jbrj.gov.br/">National Centre for Flora Conservation</a>, is featured in <a href="http://elastic.co/">Elastic.co</a> user stories blog.</p><p>In <a href="https://www.elastic.co/blog/providing-extinction-risk-assessments-for-biodiversity-with-elasticsearch">Providing extinction risk assessments for biodiversity with Elasticsearch</a> we give an overview of the processes of assessing brasillian flora with the help of the Elastic stack, using Elasticsearch for indexing and Kibana for exploratory data analysis.</p><p>The mentioned tool is the <a href="https://br.biodiversity.cloud/">Rapid Risk Assessment Application (RRAPP)</a>.</p><p>It was a great surprise already that we were invited to run for the <a href="https://www.elastic.co/causeawards">Elastic Cause Awards</a>, <em>which recognizes organizations using the Elastic Stack to do great good in the world.</em> We got a feel steps forward in the awards but were not able to make it till the end, unfortunally.</p><p>Even so, it was inspiring to see Elastic noticing it’s comunity projects and giving relevance and some spotlight to us.</p><p>So, enjoy the reading and more articles on Conservation and Biodiveristy informatics are due.</p><p><a href="https://www.elastic.co/blog/providing-extinction-risk-assessments-for-biodiversity-with-elasticsearch"><strong>Providing extinction risk assessments for biodiversity with Elasticsearch</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4a8bded9539" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Power from simplicity of chat bots]]></title>
            <link>https://medium.com/@diogok/power-from-simplicity-of-chat-bots-7348a75af1dc?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/7348a75af1dc</guid>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[ux]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Sat, 10 Sep 2016 17:25:41 GMT</pubDate>
            <atom:updated>2016-09-10T17:25:41.270Z</atom:updated>
            <content:encoded><![CDATA[<p>Or: Non-conversational interactive chat interfaces.</p><p>Since <a href="https://blog.bufferapp.com/7-examples-of-how-creative-constraints-can-lead-to-amazing-work">constraints</a> provokes <a href="https://medium.com/stanford-d-school/want-some-creativity-crank-up-the-constraints-5728a988a635#.o1czrsve9">creativity</a> and <a href="https://techcrunch.com/2015/11/11/no-ui-is-the-new-ui/">no-UI</a> is the <a href="https://medium.com/@jasonyogeshshah/why-no-ui-is-the-best-ui-4f066c2f95a2#.gwg9wf70c">best UI</a> (simplicity is king), it really is looking good to be on the chat focused apps side of things.</p><p>And from a backend developer, who can’t really come up with nice designs, it really makes sense: the UI is already there, is dead simple and there are already users. All you got to do is write an API, and in that we excel.</p><p>Talking about simplicity, you don’t even need to go the conversational AI path. I find conversational bots (those that trhow AI and pretend to be a person) actually frustating. It adds a lot of complexity and make interaction longer and results sometimes are unexpected.</p><p>Simple command based chat bots feels faster and direct. Let your bot be the robot it really is, no make-up and no deceiving.</p><p>Also, make interaction one step at a time and confirm me it’s all right at each one. Be responsive: I send you one comand, you reply ok and ask one field of data, them the next and so on. And list me options when available.</p><p>So I built some <a href="https://kbots.net/">simple bots for Telegram</a>, and am really enjoying it.</p><p>It’s all about making something useful and usable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7348a75af1dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I made something]]></title>
            <link>https://medium.com/@diogok/i-made-something-4ae50d97620e?source=rss-55a9db13f0b3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ae50d97620e</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[makers]]></category>
            <dc:creator><![CDATA[Diogok]]></dc:creator>
            <pubDate>Mon, 18 Jan 2016 23:42:26 GMT</pubDate>
            <atom:updated>2016-01-19T14:18:21.554Z</atom:updated>
            <content:encoded><![CDATA[<p>At this new year break I made a blog for my girlfriend. A simple, straightforward and customized wordpress blog, and I felt pretty good about it.</p><p>It took me about two days to air it, and I felt good because I put something for the public and it was without any hassle, and it has been a while since I was able to put something out there that was both simple and useful.</p><p>Lately all my programming world resolved around “complex” stuff: Polyglot programming, multiple databases, containers, geospatial analytisys, micro-services... and, above all that, little of that was made public, and the part that were was not really making just for all the work underneath it. Thou it’s all opensource, at least.</p><p>And, in two easy days, side-by-side with my “client”, I put up a pretty product that made her happy and made me fell like I did more than a good good, a complete job.</p><p>That did light a sparkle, so I filled up my trello with stuff to do (this writing included), <a href="http://biodiv.diogok.net/">published</a> and <a href="http://github.com/diogok">opened source code</a> of what I had in the backlog, and will take this forward as much a possible.</p><p>So, this text is about that, delivering simple stuff that matter, and feeling good about it.</p><p>At last, let’s take the challenge with <a href="https://medium.com/u/a02dd1574d08">Justin Jackson</a> and <a href="http://megamaker.co/challenge/">make stuff</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ae50d97620e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>