<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yusuf Syaifudin on Medium]]></title>
        <description><![CDATA[Stories by Yusuf Syaifudin on Medium]]></description>
        <link>https://medium.com/@yusufs?source=rss-b9cc63eced88------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*GniJ-cjqBH0WZN8sZX_69A.jpeg</url>
            <title>Stories by Yusuf Syaifudin on Medium</title>
            <link>https://medium.com/@yusufs?source=rss-b9cc63eced88------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 11:33:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yusufs/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Streaming Data Made Easy with Benthos]]></title>
            <link>https://yusufs.medium.com/streaming-data-made-easy-with-benthos-78f14ba90d6d?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/78f14ba90d6d</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[benthos]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Fri, 06 May 2022 08:14:30 GMT</pubDate>
            <atom:updated>2022-05-06T08:14:30.618Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*3bMG7mJwFQyE93dkHq8h2w.png" /><figcaption>Benthos logo</figcaption></figure><p>I sometimes heard about “How if we unified our pub/sub system, so developer is agnostic to Kafka?”, and this is my second time I heard of that. I sometimes wonder about why are we avoid interacting with Kafka so much while the concept is rather simple: publish (from your producers) and subscribe (on your consumers). But, when talking about creating consumer, the task seems repeatable and boring: we create consumer service, add some logic and etc. Then at some point we must add extra care because sometime we need to process the same data in different service (fan-out), or messages cannot be processed and we need to add Dead Letter Queue, or sometimes we need to do retry mechanism — and we’re doing this for every service that needs consume Kafka.</p><p>Maybe we can solve this by creating template, but another approach maybe we can using ETL tooling such as Benthos. In this article I will show you what and how we work with Benthos.</p><h3>What is Benthos?</h3><blockquote>Benthos is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless <a href="https://www.benthos.dev/docs/components/processors/about">processing steps</a>. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit.</blockquote><p><a href="https://v4.benthos.dev/docs/about">What is Benthos for? | Benthos</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLOcUbYS_uQblPAfKB86kA.png" /><figcaption>Source: <a href="https://v4.benthos.dev/docs/about">https://v4.benthos.dev/docs/about</a> taken at May 6th, 2022</figcaption></figure><h3>How we can use Benthos to stream our data?</h3><p>The idea of “getting rid of Kafka” from developer’s view is usually convert the asynchronous data (in Kafka) processed into REST API interface system rather than creating standard template when building Kafka consumer’s services. In simple words, we actually create one middleware system that subscribe all message from list of Kafka topics, and then routing them into the destination service by calling REST API.</p><p>Because the nature of REST API is synchronous and we remove the ability to stop reading from Kafka if process has failure (stop send acknowledge to Kafka), we need to add some retry mechanism (back-off retry or publishing the message again so it can be processed later) and Dead Letter Queue if message still failed.</p><p>However, not all use-case can handle “delayed message”, some message like journal system or transaction system need to keep the order of incoming message. But, if the incoming message still failed after consecutive retry, maybe we can drop the data and send it to DLQ to manually reviewed by human.</p><p>So, here the use-case and requirements that I want to show you in this article:</p><ol><li>I will create the oversimplified wallet system where user can top-up and spend the money in specified walletID. The system will error if we spend money when we don’t have sufficient balance.</li><li>Incoming data to the REST API must in the sequential order. The order is must exactly the same like the offset in Kafka because our source of truth is from Kafka topic.</li><li>Apart from sending message to the Wallet Service, we need to send to another journal system to record each incoming message. This journaling system is good for analytic, auditing, or accounting. To minimize error, we save the journal data into file output.</li><li>If incoming data is failed to be processed or system is down, we need to make a request retry.</li><li>If consecutive retry still failed, we need to save it in the specific file as final DLQ.</li></ol><p>By above requirement we know that in point 1 we need to implement distributed lock (will not explained because it is out of topic). Also, in point 2 and 3 it means that we need to do fan-out message delivery, meaning that sending data to Wallet Service via REST API and sending to journal system via writing to file can be done with concurrent processing as long as the order send to both system is the same. In point 4 and 5, we need to implement back-off retry and DLQ if data is failed to process.</p><p>To keep this article short, I will not share detailed process step by step as usual. Instead, I will share how to run and check if retry mechanism and DLQ is supported by Benthos.</p><p><a href="https://github.com/yusufsyaifudin/benthos-sample/tree/v1.0.0-alpha">GitHub - yusufsyaifudin/benthos-sample at v1.0.0-alpha</a></p><p>First, run Kafka, redis and Wallet Service (written in Golang) using docker compose and commandgo run main.go wallet-server -p 8081 --redis-addr localhost:6379 --debug . You can use pre-built binaries downloaded from here <a href="https://github.com/yusufsyaifudin/benthos-sample/releases/tag/v1.0.0-alpha">https://github.com/yusufsyaifudin/benthos-sample/releases/tag/v1.0.0-alpha</a></p><p>Seed data into Kafka using:</p><pre>$ cd golang<br>$ sudo chmod -R 757 wallet_seed_kafka.sh<br>$ ./wallet_seed_kafka.sh wallet_kafka.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MhTaSgCCQFEcRhQkwxiwEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x0XkroAaoUi2zuRRZfA6WA.png" /><figcaption>Seed data into Kafka topic: walletservice</figcaption></figure><p>In the script wallet_seed_kafka.sh I create publish message to specific partition 0. This because if we use multiple partition in Kafka, we will get out-of-order message in the consumer which is not good for transaction system that need strong ordering.</p><p>Imagine we push data:</p><pre>DATA 1 =&gt; GOES TO PARTITION 0<br>DATA 2 =&gt; GOES TO PARTITION 1<br>DATA 3 =&gt; GOES TO PARTITION 2<br>DATA 4 =&gt; GOES TO PARTITION 3<br>DATA 5 =&gt; GOES TO PARTITION 0<br>DATA 6 =&gt; GOES TO PARTITION 1<br>DATA 7 =&gt; GOES TO PARTITION 2<br>DATA 8 =&gt; GOES TO PARTITION 3</pre><p>or represented like this:</p><pre>Partition 0 =&gt; [1, 5]<br>Partition 1 =&gt; [2, 6]<br>Partition 2 =&gt; [3, 7]<br>Partition 3 =&gt; [4, 8]</pre><p>When consumer using the consumer-group name subscribe this message, the message arrived may be out of order. The consumer may receive data with this order: [1, 3, 2, 4, 8, 7, 6, 5] because Kafka only maintain ordering for the same partition. To avoid this behavior, the data will publish specific to partition 0 in the Kafka (Kafka topic start from 0 index).</p><p>Then we can start the Benthos streams:</p><pre>benthos -c simple/config/benthos.yaml -r &quot;simple/resources/*.yaml&quot; streams simple/streams/*.yaml</pre><p>When we run this command it will do an action similar like the requirements I defined above:</p><ol><li>Consume message from Kafka and then trying to add Kafka metadata using processor proc_wallet_service_add_meta.</li><li>If processor proc_wallet_service_add_meta is failed, for example due to non-valid JSON, then try to catch the error using errored check in the switch-case and processor out_file_wallet_service_proc_error.</li><li>If processor proc_wallet_service_add_meta success then:</li><li>Send all the message to the file (for Journaling log) using out_file_wallet_service and to the Wallet Service <a href="http://localhost:8081/">http://localhost:8081</a> using resource out_http_wallet_service. Writing to Journal Log file (out_file_wallet_service) should not return any error (in production we may send it into standard output and then pipe to another service such as Elasticsearch or database to minimize error). Sending to HTTP is prone to error, so we need to create `fallback` mechanism.</li><li>If sending to HTTP service is failed after 3 consecutive retry, then send to DLQ out_file_wallet_service_error.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHS8B3x_CffoAfKeQETFzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRWa3-l1E-RrzZMF8TxPkQ.png" /><figcaption>The processed message must be in the order.</figcaption></figure><p>You see that the final amount on wallet bis 49049.</p><ul><li>The wallet dataset contain 1001 line of data, every 10 first data is top-up with amount: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. Then after that it will “spend” with amount of 11.</li><li>This mean that we have 91 repeated pattern (1001/11=91).</li><li>Also mean that the top-up amount is: (10 + 20 + 30 + 40 + 50 + 60 + 70 + 80 + 90 + 100) * 91 = 50050.</li><li>And spending amount is: (11 * 91) = 1001.<br>The final value is`(50050–1001) = 49049. This 49049 amount must be the final value we get from the API.</li></ul><p>To double-check, see from cURL:</p><pre>curl -X POST ‘localhost:8081/amount’ -H ‘Content-Type: application/json’ — data-raw ‘{“walletID”:”b”}’ | jq .</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OU948vSsgiFBS3pUlGYryQ.png" /><figcaption>Response after checking wallet b</figcaption></figure><h4>Testing the errored data: must go to DLQ</h4><p>Now we only have 1 file with name service_wallet_out.txt because we don’t have any error both in Wallet Service or in the message processor.</p><p>We can validate that all data/message consumed by Benthos from Kafka are passed to the Wallet Service. To verify we have the right order, use jq and bash script:</p><pre>$ sudo chmod -R 757 golang/check_sorted.sh<br>$ ./golang/check_sorted.sh service_wallet_out.txt</pre><p>It must not return anything, if error it will show message something like this:</p><pre>kafkaOffset: prev offset 999 larger than current offset 100</pre><p><strong>Produce non-valid data</strong></p><p>Now we try to produce data that is not valid JSON string. Following data is missing &quot;after {in the key walletID:</p><pre>echo ‘{walletID”:”b”, “operation”: “topup”, “amount”: “10”}’ | kcat -P -t walletservice -b localhost:9092 -p 0</pre><p>We now will have new file service_wallet_proc_error.txt with content like this:</p><pre>{“content”:”{walletID\”:\”b\”, \”operation\”: \”topup\”, \”amount\”: \”10\”}”,”error”:”failed assignment (line 1): unable to reference message as structured (with ‘this’): parse as json: invalid character ‘w’ looking for beginning of object key string”,”meta”:{“kafka_lag”:”0&quot;,”kafka_offset”:”1001&quot;,”kafka_partition”:”0&quot;,”kafka_timestamp_unix”:”1651819063&quot;,”kafka_topic”:”walletservice”},”reqID”:”28mQ1MGrZ7AWzGwil2BC7VJKIam”,”time”:”2022–05–06T13:37:44.298612+07:00&quot;}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AcRhtNxXKQXXtiRsG6Gzyw.png" /><figcaption>Benthos log when processor is failed</figcaption></figure><p>The file service_wallet_proc_error.txt act as DLQ when processor is error. In this case, the processor is error when adding Kafka metadata because it is not valid JSON string:</p><pre>failed assignment (line 1): unable to reference message as structured (with ‘this’): parse as json: invalid character ‘w’ looking for beginning of object key string</pre><p>The data will never be passed to Wallet Service because it stops when processor error.</p><p><strong>Produce valid JSON data but cannot processed by Wallet Service</strong></p><p>Now, we try to produce data that cannot be processed by Wallet Service.</p><pre>echo ‘{“walletID”:”b”, “operation”: “topup”, “amount”: “10”}’ | kcat -P -t walletservice -b localhost:9092 -p 0</pre><p>The payload may look valid, but actually is not. It because the Wallet Service expect amount to be number, but we send as string.</p><p>Now, see the Wallet Service’s logs. It shows that first call is error, then Benthos will retry every 3 seconds up-to 3 times. We can validate this by looking at the reqID value is still the same for all 4 requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgVwgw1ufLI6MTm7p0rfxQ.png" /><figcaption>When error, it will retry up-to maximum retry value we defined in Benthos configuration file. When all retry done but still failed, it goes to DLQ.</figcaption></figure><pre>2022/05/06 13:47:09 {“path”:”/”,”request”:{“header”:{“Accept-Encoding”:”gzip”,”Content-Length”:”172&quot;,”Content-Type”:”application/json”,”User-Agent”:”Go-http-client/1.1&quot;},”data_object”:{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}},”response”:{“header”:{“Content-Type”:”application/json”},”data_object”:{“error”:”failed read request body: json: cannot unmarshal string into Go struct field ReqData.amount of type int64&quot;}}}<br>2022/05/06 13:47:12 {“path”:”/”,”request”:{“header”:{“Accept-Encoding”:”gzip”,”Content-Length”:”172&quot;,”Content-Type”:”application/json”,”User-Agent”:”Go-http-client/1.1&quot;},”data_object”:{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}},”response”:{“header”:{“Content-Type”:”application/json”},”data_object”:{“error”:”failed read request body: json: cannot unmarshal string into Go struct field ReqData.amount of type int64&quot;}}}<br>2022/05/06 13:47:15 {“path”:”/”,”request”:{“header”:{“Accept-Encoding”:”gzip”,”Content-Length”:”172&quot;,”Content-Type”:”application/json”,”User-Agent”:”Go-http-client/1.1&quot;},”data_object”:{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}},”response”:{“header”:{“Content-Type”:”application/json”},”data_object”:{“error”:”failed read request body: json: cannot unmarshal string into Go struct field ReqData.amount of type int64&quot;}}}<br>2022/05/06 13:47:18 {“path”:”/”,”request”:{“header”:{“Accept-Encoding”:”gzip”,”Content-Length”:”172&quot;,”Content-Type”:”application/json”,”User-Agent”:”Go-http-client/1.1&quot;},”data_object”:{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}},”response”:{“header”:{“Content-Type”:”application/json”},”data_object”:{“error”:”failed read request body: json: cannot unmarshal string into Go struct field ReqData.amount of type int64&quot;}}}</pre><p>See the time value which incremented by 3 seconds, meaning that the next request will be made after 3 seconds if previous call is failed.</p><p>After 4 failed requests (1 for first request and 3 for retry), Benthos will save the failed request into file service_wallet_error.txt. The content of file service_wallet_error.txt will look like this:</p><pre>{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}</pre><p>Benthos will also save to file service_wallet_out.txt regardless all retries already done or not, this is used for journal log. The content of file service_wallet_out.txt will be appended with new data:</p><pre>{“amount”:”10&quot;,”kafkaOffset”:”1002&quot;,”kafkaPartition”:”0&quot;,”operation”:”topup”,”reqID”:”28mRAGUy6nWVpSrw41rFHy8vsNj”,”time”:”2022–05–06T13:47:09.758318+07:00&quot;,”walletID”:”b”}</pre><p>The kafkaOffset is 1002 because the offset 1001 is used by previous data when we try to publish non-valid JSON string.</p><h3>Lesson learned</h3><p>During trying Benthos, some thoughts come in my mind (please also share yours if you have one):</p><p>If your use-case is simple i.e: just consume data and then process it (like creating worker service), you can use Benthos. But, please beware if you have multiple topic and complex routing, for example you have 1 Kafka cluster for all topic and you need to routing each topic to different services. If each routing must have it’s own retry configuration, DLQ and so on, this will make your configuration complex and may resulting cognitive overhead to read and debug YAML file.</p><p>In my opinion, I think it is better to deploy specific Benthos configuration for specific use-case. If you think to define all resources in one place and then define each streams per node, you will get wrong routing.</p><p>When we define input in the resource (in my case is Kafka input),<br>and then use it multiple times (in the stream configuration) for each service, then the message routing is not work as expected. Sometimes message from topic A is still routed to the service B even though we already define usingswitch-check.</p><p>I mean, when we define this input resources:</p><pre><em># define input resources<br></em>input_resources:<br>  - label: in_kafka_service<br>    kafka:<br>      addresses:<br>        - localhost:9092<br>      topics:<br>        - calcservice<br>        - walletservice<br>      consumer_group: &quot;benthos_consumer12&quot;<br>      checkpoint_limit: 1</pre><p>Then create two file:</p><p>First, named service_calc.yaml :</p><pre><em># doing pipeline<br></em>input:<br>  resource: in_kafka_service<br><br>output:<br>  switch:<br>    cases:<br>      - check: meta(&quot;kafka_topic&quot;) == &quot;calcservice&quot;<br>        output:<br>          broker:<br>            pattern: fan_out<br>            outputs:<br>              - resource: out_file_calc_service<br>              - resource: out_http_calc_service<em><br>                </em>processors:<br>                  - resource: proc_calc_service_json<br><br>      - check: errored()<br>        output:<br>          resource: out_file_calc_service_error</pre><p>Second, named service_wallet.yaml :</p><pre><em># doing pipeline<br></em>input:<br>  resource: in_kafka_service<br><br>output:<br>  switch:<br>    cases:<br>      - check: meta(&quot;kafka_topic&quot;) == &quot;walletservice&quot;<br>        output:<br>          broker:<br>            pattern: fan_out<br>            outputs:<br>              - resource: out_file_wallet_service<br>              - fallback:<br>                  - resource: out_http_wallet_service<br>                  - resource: out_file_wallet_service_error <em>#DLQ HTTP<br><br>      </em>- check: errored()<br>        output:<br>          resource: out_file_wallet_service_error <em>#DLQ proc error<br>  </em>processors:<br>    - resource: proc_wallet_service_add_meta</pre><p>If we don’t know about Kafka consumer group, we may expect that each message will route to the Calc and Wallet service properly because we already define switch-check. But, it is not! Since we use consumer group, Benthos will consume all topic under the same consumer group. Then Benthos will try to process data to stream Calc even though the data come from topic walletservice. The data will stop there, it will not try to fan-out to all stream. This is the behavior that I got when I use Benthos version 4.0.0 (the same version that I use to write this article).</p><pre>benthos -v<br>Version: 4.0.0<br>Date: 2022-04-21T08:35:06Z</pre><p>Also, service_calc.yaml and service_wallet.yaml is not error nor really valid. When I tried to run this, the check: errored() never be called, but when I try to create simple streams I learn that check error must in the first check (or I need to add continue as documented here <a href="https://v4.benthos.dev/docs/components/outputs/switch/#casescontinue">https://v4.benthos.dev/docs/components/outputs/switch/#casescontinue</a> ?). Regardless what should I do, to use Benthos we must learn about the Benthos configuration. Again, if we only use small subset of Benthos feature, then it is okay. But if we’re going to write complex routing system just for only one type input (for example Kafka) and one output (i.e HTTP) and save all failed processed data into shared DLQ (with data partition using tag), I think it is better to reconsider.</p><p>Yogyakarta<br>Friday, May 6th, 2022 03:00PM</p><p>Ied Al-Fitr 1443 Hijr Mubarak!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78f14ba90d6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Create Geo-aware System: How to Better Detect Whether User Is Inside the City [Bonus: Implement…]]></title>
            <link>https://yusufs.medium.com/create-geo-aware-system-how-to-better-detect-whether-user-is-inside-the-city-bonus-implement-3107292cbc6f?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/3107292cbc6f</guid>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[tile38]]></category>
            <category><![CDATA[geocoding]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[geolocation]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Tue, 20 Jul 2021 14:03:23 GMT</pubDate>
            <atom:updated>2021-07-20T14:03:23.884Z</atom:updated>
            <content:encoded><![CDATA[<h3>Create Geo-aware System: How to Better Detect Whether User Is Inside the City [Bonus: Implement Using Tile38]</h3><p>To engage user in a particular area, we need to attract them with local promotion or information. This can be done by capturing user location via GPS (of course by asking their permission first). Then queries to our database whether an user is need to know an information or eligible for a promotion based on their location data. The data sometimes only valid for a certain cities (which may have small area for each city), but how to ensure that user is really inside the city? How about if user 1KM away from the city, is it still valid?</p><p>This article will show you how to convert user latitude and longitude data into location name.</p><h4>Using Paid Service</h4><p>Well, the easier but costly solution is using 3rd party service. You can use:</p><ul><li>Google Maps API <a href="https://developers.google.com/maps/documentation/geocoding/overview#ReverseGeocoding">https://developers.google.com/maps/documentation/geocoding/overview#ReverseGeocoding</a></li><li>Bing Maps <a href="https://www.bing.com/api/maps/sdk/mapcontrol/isdk/searchbypoint">https://www.bing.com/api/maps/sdk/mapcontrol/isdk/searchbypoint</a></li><li>…</li><li>or any similar services.</li></ul><p>This is good and scale very well, but for cost-aware start-ups we may need free basic solution before proving the business model gains revenue. For free solution, we need to write our own system. In the last section of this article I will share how tile38 — an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence, can be used to solve this.</p><p>Before using Tile38, I want to share that we should never use single city’s latitude and longitude as the main data.</p><h4>Using City Point</h4><p>We starting with simple basic idea: list all of possible city latitude and longitude. When we get the user’s location, we match the user’s geo location with all of the possibility. But, this simply can’t be done because there will be small amount of the user (or it maybe zero) that have the exact match. So, then we added radius.</p><p>Supposed that we have only one city with name Sleman. When I search in Google maps with keyword “Sleman” it shows <a href="https://www.google.com/maps/place/Sleman+Regency,+Special+Region+of+Yogyakarta/@-7.689355,110.2411879,11z/data=!3m1!4b1!4m5!3m4!1s0x2e7a5ee1c5671249:0x3027a76e352bc20!8m2!3d-7.7325213!4d110.402376">this URL</a>. Its center point is -7.689355, 110.2411879.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfjeS8D81c8H4HnwbRrMlg.png" /><figcaption><a href="https://www.google.com/maps/place/Sleman+Regency,+Special+Region+of+Yogyakarta/@-7.689355,110.2411879,11z/data=!3m1!4b1!4m5!3m4!1s0x2e7a5ee1c5671249:0x3027a76e352bc20!8m2!3d-7.7325213!4d110.402376">https://www.google.com/maps/place/Sleman+Regency,+Special+Region+of+Yogyakarta/@-7.689355,110.2411879,11z/data=!3m1!4b1!4m5!3m4!1s0x2e7a5ee1c5671249:0x3027a76e352bc20!8m2!3d-7.7325213!4d110.402376</a></figcaption></figure><p>For example, we have a user located in -7.7677384, 110.3771634 (it is my campus location 😝). It actually in Sleman regency. But, as we see below, the user location is too far from city’s point. Even after we add radius 1KM it still couldn’t reach city’s center.</p><p>In order to make this easier to understand, I create the page in here <a href="https://yusufsyaifudin.github.io/wilayah-indonesia/radius.html">https://yusufsyaifudin.github.io/wilayah-indonesia/radius.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*krCwGrGylcS7VKP-32pEyw.png" /></figure><p>How if we move city’s point? Yes, we can, but it leads to another problem because we know:</p><ul><li>City’s border is not round nor rectangle, it better represented using polygon.</li><li>If we add too large radius, it may resulting wrong values as it may near different city than actual user’s city location.</li></ul><h4>Using Polygon as Geo fencing</h4><p>As we’ve seen in Google Maps screenshot above, Sleman city have a border somewhat like a mountain (pyramid). So, every users inside the border should be detected as inside the Sleman city.</p><p>To draw where is Sleman border, we can use <a href="https://geojson.org/">GeoJson</a>.</p><p><a href="https://geojson.io/">geojson.io</a></p><p>But, it may leave a question, how do I know the border of the city? Thanks to Openstreet Map for it’s free to use and under open license of data.</p><p><em>If you already have or know about GeoJSON data, you may skip this part. In this part I will tell you how to get GeoJSON data of the city’s border.</em></p><p>For example, if you want to get city’s border of Sleman, Indonesia. You can open <a href="https://www.openstreetmap.org/">https://www.openstreetmap.org</a> and search “Sleman Indonesia”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjN40zJFxQw_3Zj8mt-58Q.png" /><figcaption>Search result of “Sleman Indonesia” on openstreetmap.org at 20 July 2021 16:26 GMT+7</figcaption></figure><p>Then select “County Boundary <a href="https://www.openstreetmap.org/relation/5615254">Sleman Regency, Special Region of Yogyakarta, Indonesia</a>”, there you find the Sleman city’s boundary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIUCCqFsGMcW0zwreiW-xg.png" /><figcaption>Sleman’s boundary on openstreetmap.org</figcaption></figure><p>In the top left corner you find that Sleman Regency have OpenStreetMap ID (OSMID) of 5615254. To get detail of this, we can use Overpass API:</p><p><a href="https://wiki.openstreetmap.org/wiki/Overpass_API">Overpass API</a></p><p>You can select one of Overpass API Open in browser this URL and you will prompted to download the file output. It may take a while before download begin.</p><pre><a href="http://overpass-api.de/api/interpreter?data=(relation(5615254);&gt;;);out;">http://overpass-api.de/api/interpreter?data=(relation(5615254);&gt;;);out;</a></pre><p>If you took too long to get the response, it may because the server is busy. You can change using different server with the same payload:</p><pre><a href="https://overpass.kumi.systems/api/interpreter?data=(relation(5615254);&gt;;);out;">https://overpass.kumi.systems/api/interpreter?data=(relation(5615254);&gt;;);out;</a></pre><p>or you can use cURL:</p><pre>curl &#39;https://overpass.kumi.systems/api/interpreter?data=(relation(5615254);&gt;;);out;&#39; &gt; 5615254.xml</pre><p>Please note that the format data=(relation(osmid);&gt;;);out; is <strong>MUST</strong> be written like that. <a href="https://wiki.openstreetmap.org/wiki/Overpass_API/Overpass_QL#Recurse_down_.28.3E.29">It is a Overpass Query Language</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQyYAFO5YWuXvPS5sWZEUA.png" /><figcaption>OSM XML data from Overpass API</figcaption></figure><p>Then you can convert it using <a href="https://tyrasd.github.io/osmtogeojson/">https://tyrasd.github.io/osmtogeojson/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CoHEtqIBQ5v42Tir-7bvhQ.png" /><figcaption>Voila, you get the GeoJSON data from OSM XML!</figcaption></figure><p>This process may exhausting if you do manual labor. You need to create a system to automate this process. But, for Indonesia, I have done it when I do my bachelor thesis. I forget where I put my script to crawl the data.</p><p><a href="https://github.com/yusufsyaifudin/wilayah-indonesia">yusufsyaifudin/wilayah-indonesia</a></p><p>Back to the main topic, we already have one city’s border. Then we still assume that our user is on -7.7677384, 110.3771634. To illustrate this, I create <a href="https://yusufsyaifudin.github.io/wilayah-indonesia/geojson.html">https://yusufsyaifudin.github.io/wilayah-indonesia/geojson.html</a></p><p>And we got this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-u2wZoYPZVgushbcS6bDw.png" /><figcaption>User is inside the Sleman city.</figcaption></figure><p>Zoom in the area!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YKiuJz_DPitY3-ysX57d5w.png" /><figcaption>The user location (-7.7677384, 110.3771634) is inside the area (Sleman, Indonesia).</figcaption></figure><p>But, it still leave a problem. How if the polygom we’ve got from the internet or the one we manually create from <a href="https://geojson.io/">https://geojson.io</a> have sub-meters wrong data? Since the actual city border (IRL) vs in the map is different, we need to add some error threshold in our system.</p><h4>Using Polygon as Geo Fencing and Add User Radius</h4><p>Still using Sleman city as the polygon data, now we have an user <a href="https://www.google.com/maps/place/Affandi+Museum/@-7.7828038,110.3963476,21z/data=!4m5!3m4!1s0x2e7a59c49f681dbd:0x3e9d55bf26695d4a!8m2!3d-7.782713!4d110.396397">at Museum Affandi</a>. The latitude and longitude is -7.7828038, 110.3963476. Now using the same geojson data, we got this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R8ELy4NSfN7LKBkwfwsxVQ.png" /><figcaption>Museum Affandi is not inside the polygon, but IRL it located in Sleman regency.</figcaption></figure><p>Now we add radius 500 meters from the user position as the error threshold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*brvnxxJ1DC0bfhljGczhQQ.png" /><figcaption>After adding the radius 500 meter, the circle is intersect with Sleman regency polygon.</figcaption></figure><p>Now, after adding the radius of 500m, the user location’s circle is intersected with the polygon. Here we can assume that now system can perfectly detect user location. But, wait, this is not the end. How if we add neighbor city’s polygon? Try add Yogyakarta, we got this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOrzk0oqU2ZEYhFuMSLadQ.png" /><figcaption>Circle is intersected with both Yogyakarta and Sleman.</figcaption></figure><p>In this situation, the system may return Yogyakarta as the first rank result than Sleman. This should be acceptable because we don’t have true border data of each city. Also, compared to the first method which using city’s lat long only, this system should have less false positive and false negative.</p><p>Another example is when user is located in <a href="https://www.google.com/maps/place/Masjid+Baitul+Arqom/@-7.7928974,110.3983692,21z/data=!4m5!3m4!1s0x2e7a59df13b0a329:0xe4f56f32e6ad0589!8m2!3d-7.7928819!4d110.3984034">Masjid Baitul Arqom</a> (-7.7928974, 110.3983692) to do prayer. This mosque is located in Bantul, so none of the current polygon in the map (Sleman and Yogyakarta) is match. But, since we use radius of 500 meter, we will see this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cVC1jFQ_9e3fz51Utz2ekQ.png" /><figcaption>User may be detected in Yogyakarta because of circle radius 500 meter is have more intersected area near Yogyakarta than Sleman.</figcaption></figure><p>User may be detected in Yogyakarta because of circle radius 500 meter is have more intersected area near Yogyakarta than Sleman.</p><h4>Implement Using Tile38</h4><p>We have done with the theory, now we can implement it using Tile38. I will not write thorough steps of the installation since it already documented very well in the website.</p><p><a href="https://tile38.com/">Tile38 · Ultra Fast Geospatial Database &amp; Geofencing Server</a></p><p>Supposed that you have successfully install and run the tile38-server.</p><p>You can add polygon of Sleman city using HTTP:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;SET cities sleman OBJECT geojson from <a href="https://github.com/yusufsyaifudin/wilayah-indonesia/blob/master/data/geojson/regency/3404.geojson">https://github.com/yusufsyaifudin/wilayah-indonesia/blob/master/data/geojson/regency/3404.geojson</a>&#39;</pre><p>Now try to get user in location -7.7677384, 110.3771634 with radius 0:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7677384 110.3771634 0&#39;</pre><p>It will return sleman :</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 1,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;47.899µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SD7CHVUxuX-TE9shHZBZaQ.png" /><figcaption>With user located in -7.7677384, 110.3771634 it detects Sleman.</figcaption></figure><p>Now, test if user located in Museum Affandi (-7.7828038, 110.3963476):</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7828038 110.3963476 0&#39;</pre><p>It return empty response:</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [],<br>    &quot;count&quot;: 0,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;36.427µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqRR_qjcgIiZ_DF5uRxaNQ.png" /><figcaption>With user located in -7.7828038, 110.3963476 it not detect any city.</figcaption></figure><p>Now, add radius of 500 meter:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7828038 110.3963476 500 &#39;</pre><p>It detect Sleman:</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 1,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;142.258µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQTbIhFOKVxpSdwucvDERg.png" /><figcaption>With user located in -7.7828038, 110.3963476 and radius 500 meter it detects Sleman.</figcaption></figure><p>Now, add new city Yogyakarta:</p><pre>SET cities yogyakarta OBJECT geojson from <a href="https://github.com/yusufsyaifudin/wilayah-indonesia/blob/master/data/geojson/regency/3471.geojson">https://github.com/yusufsyaifudin/wilayah-indonesia/blob/master/data/geojson/regency/3471.geojson</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ry2MxAiK_75wJ1WfctiesA.png" /><figcaption>Success adding Yogyakarta</figcaption></figure><p>And test the same user located in Museum Affandi (-7.7828038, 110.3963476) with radius 500 meter:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7828038 110.3963476 500 &#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWnp2RgISQaynLXfU6vTKw.png" /><figcaption>Museum Affandi (-7.7828038, 110.3963476) with radius 500 meter still return Sleman as first result even though is has more intersected area in Yogyakarta.</figcaption></figure><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        },<br>        {<br>            &quot;id&quot;: &quot;yogyakarta&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.80279421806335,<br>                &quot;lon&quot;: 110.3762512207<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 2,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;155.937µs&quot;<br>}</pre><p>The result wasn’t expected, because Yogyakarta must on the first rank than Sleman. So I tried to lowering radius to 300, and it result the same:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jy8bjsxKxpFtEACssM4pCw.png" /><figcaption>Museum Affandi (-7.7828038, 110.3963476) with radius 300 meter still return Sleman as first result even though is has more intersected area in Yogyakarta.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRjpjR5-ExlUfRK1Z8Mt2g.png" /><figcaption>Museum Affandi (-7.7828038, 110.3963476) with radius 300 meter using Tile38.</figcaption></figure><p>When I tried to set radius to 0, it works as expected where Yogyakarta is the only polygon that match:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S8oKmWuuwhWb7NrfXA4UeQ.png" /><figcaption>Museum Affandi (-7.7828038, 110.3963476) with radius 0 meter works as expected: return Yogyakarta only.</figcaption></figure><p>After deleting all polygon data and then insert Yogyakarta first then Sleman, using same queries I got Yogyakarta. Maybe this is an issue of Tile38 where insertion order is matter.</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7828038 110.3963476 500 &#39;</pre><p>Response:</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;yogyakarta&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.80279421806335,<br>                &quot;lon&quot;: 110.3762512207<br>            }<br>        },<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 2,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;151.785µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G0dlCghpaoWUA7ESxESBpw.png" /><figcaption>After changing insertion order (Yogyakarta then Sleman), Museum Affandi (-7.7828038, 110.3963476) with radius 500 meter return Yogyakarta as first rank.</figcaption></figure><p>Now, back with inserting Sleman then Yogyakarta and user location is at <a href="https://www.google.com/maps/place/Masjid+Baitul+Arqom/@-7.7928974,110.3983692,21z/data=!4m5!3m4!1s0x2e7a59df13b0a329:0xe4f56f32e6ad0589!8m2!3d-7.7928819!4d110.3984034">Masjid Baitul Arqom</a> (-7.7928974, 110.3983692) which actually located at Bantul:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7928974 110.3983692 500 &#39;</pre><p>Again, it return Sleman although in above we see that it has more area intersected in Yogyakarta rather than Sleman.</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        },<br>        {<br>            &quot;id&quot;: &quot;yogyakarta&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.80279421806335,<br>                &quot;lon&quot;: 110.3762512207<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 2,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;129.766µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nH3WFa6bXHUS6ZVkqzYsEA.png" /><figcaption>Masjid Baitul Arqom (-7.7928974, 110.3983692) with radius 500 meter return Sleman as first result.</figcaption></figure><p>I tried to lowering radius to 230 meter and got that Yogyakarta is the only match:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5bOdvl5cRbt3wFaiJZnKyg.png" /><figcaption>Masjid Baitul Arqom (-7.7928974, 110.3983692) with radius 230 meter return only Yogyakarta.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L0gayVghEbyFd4pdpMNHBA.png" /><figcaption>Masjid Baitul Arqom (-7.7928974, 110.3983692) with radius 230 meter only intersected with Yogyakarta.</figcaption></figure><p>Same as above, I tried to change the insertion order. Now, I insert Yogyakarta then Sleman and do the same exact query:</p><pre>curl -X POST &#39;localhost:9851&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;INTERSECTS cities POINTS CIRCLE -7.7928974 110.3983692 500  &#39;</pre><p>And got the result Yogyakarta first:</p><pre>{<br>    &quot;ok&quot;: true,<br>    &quot;points&quot;: [<br>        {<br>            &quot;id&quot;: &quot;yogyakarta&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.80279421806335,<br>                &quot;lon&quot;: 110.3762512207<br>            }<br>        },<br>        {<br>            &quot;id&quot;: &quot;sleman&quot;,<br>            &quot;point&quot;: {<br>                &quot;lat&quot;: -7.6903140544890505,<br>                &quot;lon&quot;: 110.38651657104501<br>            }<br>        }<br>    ],<br>    &quot;count&quot;: 2,<br>    &quot;cursor&quot;: 0,<br>    &quot;elapsed&quot;: &quot;121.298µs&quot;<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cJx3eDS1j10TicSJVx48Tg.png" /><figcaption>When changing insertion order, Masjid Baitul Arqom (-7.7928974, 110.3983692) with radius 500 meter return Yogyakarta as first result.</figcaption></figure><p>For now we can consider this as a minor bug. This because when we query using radius of 0 meter we can still got the result if it really inside the polygon and got no result if outside the polygon. For the workaround, we can add radius by 10 meter gradually to see the nearest city, but it may cost the operation. For asynchronous system it may still acceptable, but for synchronous call it may cause severe system performance.</p><h4>Conclusion</h4><p>Reverse geocoding (translating latitude and longitude data into location name) is not easy to solve. But knowing limitation of each possible solution will minimize the problem or at least we know it is just the right system behaviour when system is wrong detect location info.</p><p>Yogyakarta, July 20th, 2021 20.48 GMT+7 <br><strong><em>Happy Ied Al Adha 1442 Hijri!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3107292cbc6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Example of Implementing OAuth 2.0 Using ory/hydra]]></title>
            <link>https://yusufs.medium.com/practical-example-of-implementing-oauth-2-0-using-ory-hydra-fbaa2765d94f?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbaa2765d94f</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[oauth2]]></category>
            <category><![CDATA[authorization]]></category>
            <category><![CDATA[oauth]]></category>
            <category><![CDATA[authentication]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 11:00:12 GMT</pubDate>
            <atom:updated>2021-02-08T11:00:12.877Z</atom:updated>
            <content:encoded><![CDATA[<p>When we building a server application that needs to protect some API resource, usually we only direct implemented simple authentication method. We send user’s credential directly to the server and then server respond with a session or returning a random string (access token) that we save in our database: stating that this {access_token} is owned by this {user_id}. Most often we produce JSON Web Token (JWT) that we don’t save in database, leading a session management issue because we cannot revoke specific JWT before it expired.</p><p>Above approach may fits in common use case, but it has pros and cons.</p><p><strong>Pros:</strong></p><ul><li>The authentication and authorization process can be done in one handler/API, makes it easy to understand and implement.</li></ul><p><strong>Cons:</strong></p><ul><li>There is no separation of concern between authorization and authentication process.</li><li>Your user service may be blocked by tremendous data of access tokens.</li><li>Hard to maintain multi-application with same user’s credential. For example, service Gmail may consist of n services behind it and Youtube m services, but if we only build Auth service for Gmail only, it sometimes hard to Youtube to use the same login credential.</li></ul><p>The latter cons may reminds you about “Sign In With Google” button that you can easily find when accessing web application that needs user account. That tiny button and flow behind it is powered by OAuth 2.0 authorization protocol. By implementing it into your app, you will be able to decouple authentication and authorization process. You can also create an Single Sign On (SSO) if you need the same user credential to be used across your service.</p><h3>OAuth 2.0</h3><p>Before we go, it will be useful to know what is OAuth itself. OAuth is not an authentication protocol, therefore it does not need to know user’s context. Although we can implement <a href="https://openid.net/connect/">Open ID Connect</a> as an authentication layer on top OAuth 2.0, it is a different topic that I will not explain here.</p><blockquote>OAuth 2.0 is the industry-standard protocol for authorization. OAuth 2.0 focuses on client developer simplicity while providing specific authorization flows for web applications, desktop applications, mobile phones, and living room devices. This specification and its extensions are being developed within the <a href="https://www.ietf.org/mailman/listinfo/oauth">IETF OAuth Working Group</a>.<br> — <a href="https://oauth.net/2/">https://oauth.net/2/</a></blockquote><h4>Authentication VS Authorization</h4><p>Citing from <a href="http://homakov.blogspot.com/2013/03/oauth1-oauth2-oauth.html">Egor Homakov</a>, he wrote:</p><p><strong>authorize</strong> = permit 3rd party Client to access your Resources (/me, /statuses/new)<br> <strong>authenticate</strong> = prove that “this guy” is “that guy” which has account on the website.</p><p>In long words, we can read the analogy explanations from Okta:</p><blockquote>Consider a person walking up to a locked door to provide care to a pet while the family is away on vacation. That person needs:</blockquote><blockquote><strong>Authentication</strong>, in the form of a key. The lock on the door only grants access to someone with the correct key in much the same way that a system only grants access to users who have the correct credentials.</blockquote><blockquote><strong>Authorization, </strong>in the form of permissions. Once inside, the person has the authorization to access the kitchen and open the cupboard that holds the pet food. The person may not have permission to go into the bedroom for a quick nap.</blockquote><p><a href="https://www.okta.com/identity-101/authentication-vs-authorization/">Authentication vs. Authorization | Okta</a></p><p>From above explanation, there is word “permission” in the authorization process. In OAuth 2.0 itself, “permission”(s) are defined by key “scope” in the request payload when requesting grant access to <strong>Authorization server</strong>. Once granted, user will be given an access token to access a <strong>Protected Resource</strong> to proof that he or she is permitted to access it. For example, whether current user is able to access protected endpoint only for <a href="https://yourdomain.com/me">https://yourdomain.com/me</a> or <a href="https://yourdomain.com/statuses/new">https://yourdomain.com/statuses/new</a> or both endpoint.</p><p>Wait, it will be long explanation if I explain one by one. For now, I will list down it here:</p><ul><li><strong>Third Party Application (Client):</strong> or OAuth 2.0 Client is the <em>external application </em>(can be mobile app or web app or any application) that will access <strong>Protected Resource </strong>if <strong>Resource Owner (User)</strong> gives the access. To know that, the Client will ask <strong>Authorization Server</strong>.</li><li><strong>Resource Owner (User): </strong>is a end-user who have username and password (or other credentials) to access <strong>Protected Resource</strong>.</li><li><strong>Authorization Server:</strong> is a server to proof that this user is authorized or permitted to access the Protected Resource. In the last section we will use ory/hydra as Authorization Server.</li><li><strong>Resource Server (Authentication Server): </strong>or others called it as <strong>an Identity Provider</strong>, is a server to proof that this user has the right credentials to do the operation in the Protected Resource. It commonly a username and password.</li><li><strong>Protected Resource: </strong>or sometimes called as <strong>Resource Provider</strong>, is a server(s) where the data can only be accessed if user is authenticated <strong>AND</strong> has permission (authorized) to access that data.</li></ul><p>For more information about the term, you can see:</p><ul><li><a href="https://www.ory.sh/hydra/docs/v1.8/concepts/oauth2/">https://www.ory.sh/hydra/docs/v1.8/concepts/oauth2/</a></li><li><a href="https://aaronparecki.com/oauth-2-simplified/">https://aaronparecki.com/oauth-2-simplified/</a></li></ul><p>You might think, why do we need this “<em>permission</em>”? Isn’t it clear, that if user already proof that it is him (authenticated using username and password), means that he can access all data inside protected resource (of course with WHERE user_id = :current_user_id)?. If you build a monolithic service, it is correct. But, if you still remember about what SSO that I briefly mentioned above, you may change your mind.</p><p>Imagine that you are now building a user-based services where your user credential is not only accessed by application that you make on your own (highly privileged application), but another application created by 3rd party may access user’s info. Hence, this 3rd party application must be given approval by the user.</p><p>For example, when you implement “Sign In With Google” in your app, you are not accessing email and password of the Google’s user. Instead, you only ask Google authorization service<em>, “Hey Google, I have an application and wanna make an access and request on your user’s behalf.”</em>, and then Google said, <em>“Okay, please redirect my user to our </em><strong><em>authentication page</em></strong><em> and ask them </em><strong><em>what permissions</em></strong><em> they give to your application.” </em>Then Google will validates whether email and password is correct. If correct, then it will ask their user what permission that our application can access. Once given, Google will give an access token to our application for further operation.</p><p>Those processes is happen because the application that will access Google Account data it not created by Google (highly privileged application from Google), but it is an application or service that you created. So, in order to protect their user’s Google Account credentials, Google need to make sure that their user only send their data to their system (in Google authentication page) and ask their user whether they permitted your application to access your data on your behalf (in Authorization page).</p><p>Now, lets the position is swapped: you are now creating something like Google Account service and you give the possibilities for 3rd party app uses your user account on their system.</p><p>This is what OAuth do. It gives you “an industry standard” about how authorization process will be made. By implementing this, you will save your development time because you don’t need to think about authorization flow because it already use by major company. In addition, with it vast adopter, your client (the ones who will use your user’s info) doesn’t need to learn your new proprietary logic because once they know OAuth, they will easily integrates in your system. Also, you don’t need to create your own library or SDK just for obtaining access token, because there is plenty OAuth Client library out there.</p><h4>Grant Type: A Method To Authorize Your Client</h4><p>OAuth 2.0 offers some grant type to authorize the 3rd party apps/clients to access to your data. The authorization process for mobile application or website will differs from smart-TV application or back-end’s service authorization process. This because each application runs in different capability limitation, and since OAuth 2.0 usually need web browser to redirect and run to authentication page process, some grant type is arranged to comply with device limitations.</p><p>In simple words: grant type is a set of flow about how the client (3rd party app) request the access token to access the protected resources.</p><p><strong>OAuth 2.0 Grant Type: Authorization Code</strong></p><p><strong><em>TLDR;</em></strong><em> If you still don’t understand about my explanation below, may be this great article will easier to understand by you:</em></p><p><a href="https://aaronparecki.com/oauth-2-simplified/">OAuth 2 Simplified</a></p><p>One of grant type that is commonly used is Authorization Code. The flow diagram is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hd4Y-gYigO_Eoe0W.png" /><figcaption>Figure 1. Authorization Code Flow. Image source: <a href="https://darutk.medium.com/diagrams-and-movies-of-all-the-oauth-2-0-flows-194f3c3ade85">https://darutk.medium.com/diagrams-and-movies-of-all-the-oauth-2-0-flows-194f3c3ade85</a> accessed at 6 February 2021 18.39 GMT+7</figcaption></figure><p>You may confused about term “<strong><em>Service ABC Resource Server</em></strong>” here, because if we refer to above terminology, the <strong>Resource Server</strong> is the API server used to access the user’s information, but in this image it seems like a <strong>Protected Resource</strong>. Yes, if we refer to above glossary, the “<strong><em>Service ABC Resource Server</em></strong>” itself actually is a <strong>Protected Resource</strong>.</p><p>In this diagram, the <strong>Resource Server</strong> and the <strong>Authorization Server</strong> (based on above glossary) is in the same server, called “<strong><em>Authorization Server</em></strong>”. So, the term <strong>Resource Server</strong> and <strong>Authorization Server</strong> can be used interchangeably if we see the diagram. But, for the clarity, I still call them as two different entity: <strong>Resource Server</strong> referring to a server who know about user’s identity, and <strong>Authorization Server</strong> referring to server to issues an access token and to check whether issued token is valid and has specific grant or not.</p><p><strong>Step 1:</strong> <strong>Resource Owner (User)</strong> will click a button “Sign With {Your App Name}” in the <strong>Third Party Application (Client)</strong>.</p><p><strong>Step 2:</strong> <strong>Step 1</strong> button will redirect page to <strong>Authorization Server</strong> to make authorization request. In OAuth 2.0, this request will be made using these query parameters:</p><ul><li>client_id: A Client ID that the <strong>Third Party Application (Client) </strong>owner obtained from the OAuth 2.0 service. This includes client_secret.</li><li>redirect_uri: a Front-End or website page or URI where the access token (and sometimes refresh token) will given back. This URI must be registered first when creating an OAuth 2.0 Client. Usually it will be redirected to the <strong>Third Party Application (Client)</strong> page.</li><li>response_type: for Authorization Code grant, use “<strong>code</strong>”.</li><li>scope: list of permission that client (or the 3rd party) asks to access protected resource on user’s behalf.</li><li>state: a random code to be validated later when access_token is given back to the application. This is used to prevent <a href="https://www.owasp.org/index.php/Cross-Site_Request_Forgery_%28CSRF%29">CSRF attacks</a>.</li></ul><p>The final URI will be like this: <a href="http://your-authorization-server.com/oauth2/auth?client_id=myclient&amp;redirect_uri=http%3A%2F%2Flocalhost%3A1234%2Fcallbacks&amp;response_type=code&amp;scope=users.write+users.read+users.edit+users.delete+offline&amp;state=2nJR8aiorREJYUuePL29z0JucIo4guU7rx1qZLT4mTc%3D">http://your-authorization-server.com/oauth2/auth?client_id=myclient&amp;redirect_uri=http%3A%2F%2Flocalhost%3A1234%2Fcallbacks&amp;response_type=code&amp;scope=users.write+users.read+users.edit+users.delete+offline&amp;state=2nJR8aiorREJYUuePL29z0JucIo4guU7rx1qZLT4mTc%3D</a></p><p><strong>Step 3, 4:</strong> Login page will be displayed to the user.</p><p><strong>Step 5:</strong> The user then input their credentials that will be validated by <strong>Resource Server (Authentication Server)</strong>. In this image, the process of Authentication may be done in the same server of <strong>Authorization Server</strong>. Actually, the processes in this step are:</p><ul><li><strong>Resource Server</strong> will validates the username and password.</li><li>Once <strong>Resource Server</strong> say “okay”, then <strong>Resource Serve</strong>r may request to <strong>Authorization Server</strong> to ask <strong>Resource Owner (User)’s</strong> consent about what permissions he/she give to <strong>Third Party Application (Client) </strong>to access. This process done by sending “scope” parameters to the <strong>Authorization Server.</strong></li><li>When authentication and permission process is done,<strong> Resource Server</strong> then ask <strong>Authorization Server</strong> to issue short-live <strong>authorization code</strong>. This code will be used later by the <strong>Third Party Application (Client) </strong>to be exchanged with Access Token to access <strong>Protected Resource</strong>.</li></ul><p><strong>Step 6:</strong> A short-lived authorization code is given back to the <strong>Third Party Application (Client) </strong>by <strong>Authorization Server.</strong></p><p><strong>Step 7: Third Party Application (Client)</strong> then exchange the authorization code with the Access Token via <strong>Authorization Server</strong> by presenting the short-lived code itself and a client_secret.</p><p><strong>Step 8: Authorization Server</strong> issues an Access Token.</p><p><strong>Steps A-D</strong> shows a process in the Back-end called <em>Service ABC </em>when a <strong>Third Party Application (Client) </strong>requesting a protected data.</p><p><strong>Step A: Third Party Application (Client) </strong>request a protected data, for example an user profile to <strong><em>Service ABC</em></strong> with an access token obtained from Step 1–8.</p><p><strong>Step B:</strong> <strong><em>Service ABC </em></strong>then validate the access to <strong>Authorization Server.</strong></p><p><strong>Step C: Authorization Server</strong> then return the response, stating that “this access token is owned by this subject (can be user id or any unique identifier) and has these several permissions (via granted scope in the authorization process).”</p><p><strong>Step D:</strong> <strong><em>Service ABC </em></strong>receives the access token’s subject and granted scope info. If accessing user profile is granted by the user (in the authorization process), then <strong><em>Service ABC </em></strong>must return the requested data, that is user info.</p><h3>Implementing OAuth 2.0 using ory/hydra</h3><p>The better way to learn is by practicing. Here I will show you how to implement Authorization Code grant flow in OAuth 2.0 using ory/hydra.</p><blockquote>ORY Hydra is a hardened, <strong>OpenID Certified OAuth 2.0 Server and OpenID Connect Provider</strong> optimized for low-latency, high throughput, and low resource consumption. ORY Hydra <em>is not</em> an identity provider (user sign up, user login, password reset flow), but connects to your existing identity provider through a <a href="https://www.ory.sh/docs/next/hydra/oauth2#authenticating-users-and-requesting-consent">login and consent app</a>. Implementing the login and consent app in a different language is easy, and exemplary consent apps (<a href="https://github.com/ory/hydra-consent-app-go">Go</a>, <a href="https://github.com/ory/hydra-consent-app-express">Node</a>) and <a href="https://www.ory.sh/docs/next/hydra/sdk/">SDKs</a> are provided.<br> — Hydra readme, <a href="https://github.com/ory/hydra/tree/v1.9.2">https://github.com/ory/hydra/tree/v1.9.2</a></blockquote><p>As you can read, Hydra is not an identity provider. This means that does not manage your users credential and profile. <a href="https://www.ory.sh/hydra/docs/v1.8/#flexible-user-management">This allows you to implement user management and login your way, in your technology stack, with authentication mechanisms required by your use case (token-based 2FA, SMS 2FA, etc).</a></p><p>Let’s we begin to spin up Hydra server and our <strong>Resource Server (Identity Provider)</strong>. Although <a href="https://www.ory.sh/hydra/docs/v1.8/5min-tutorial">the 5-minutes tutorial</a> already explains how to run hydra using <strong>Identity Provider</strong> implemented in NodeJS in the <a href="https://github.com/ory/hydra-login-consent-node/tree/v1.9.2">ory/hydra-login-consent-node repository</a>, but it will be easier to understand if we implement by ourself.</p><p>Before we go, we need to understand to integrates our <strong>Identity Provider</strong> to Hydra by looking at following sequence diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*saomZY_cfShClpHNeIgUkA.png" /><figcaption>Figure 2. OAuth 2.0 Flow using Ory/Hydra. Image source: <a href="https://www.ory.sh/hydra/docs/concepts/login">https://www.ory.sh/hydra/docs/concepts/login</a> accessed 7 February 2021 21.13 GMT+7</figcaption></figure><p>By looking at Figure 2, we know that we need to implements 4 (four) HTTP handlers (and I will also explains the logic of respective functions):</p><ol><li>Login Page — a Web UI that show a form for user to input their credentials. This is needed in process <em>“Redirects end user with login challenge”</em> and to <em>“Fetches login info”</em>. In this page, we need to capture a login_challenge from query parameter that sent by Hydra when “<em>OAuth 2.0 Client initiates OAuth2 Authorize Code or Implicit Flow.” <br></em>We then need to call Hydra to check whether current user already logged in (<a href="https://github.com/ory/hydra-client-go/blob/v1.9.2/models/login_request.go#L48-L52">checked via cookie or remember flag</a>). If already logged in (response key skip is true) then we do not need to show this login page, instead we use previous login info data to accept the login request and <em>“Redirects end user to redirect url with login verifier”</em>. Otherwise, we need to show a login form.</li><li>Login Handler — an API to handle user’s info login. This handler will retrieve username and password of the User, then it will validate whether the inputted credentials is true. When valid, we need to accepts the login request to Hydra using login_challenge code sent by Hydra in process “<em>Redirects end user with login challenge”</em>. After accepting the request, Hydra will generate a consent_challenge and <em>“Redirects end user with consent challenge”</em> to Consent Page.</li><li>Consent Page — a Web UI shows a list of permission that Resource Owner (User) need to accepts. For example, if the app need to access User’s profile info, it probably show a checkbox to access user info where the User can unchecked it (disagree) or checked it (agree). In this page, we will receive a consent_challenge that we need to send back to Hydra with list of scopes (permissions) that user agreed.<br>We need to ask Hydra whether current request has requested the same scopes from the same user previously. <a href="https://github.com/ory/hydra-client-go/blob/v1.9.2/models/consent_request.go#L61-L64">If yes then </a><a href="https://github.com/ory/hydra-client-go/blob/v1.9.2/models/consent_request.go#L61-L64">skip parameter is true and you must not ask the user to grant the requested scopes.</a></li><li>Consent Handler — an API handler to receive action of Consent Page (process 3). This handler will retrieve a grant_scope that User agreed in Consent Page UI, then ask Hydra to accepts Consent Request. The final response of this proccess is a redirect URI where Hydra will <em>“Redirects to redirect url with consent verifier”. </em>Hydra then verifies grant and transmits authorization code/token back to the redirect_uri in <strong>Third Party Application (Client)</strong>. The Client then needs to exchange this Authorization code with an access token.</li></ol><p>You may still cannot imagine what Login Page and Consent Page looks like. In “Sign In With Google” the Login Page will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3M2QoLNQlobqwwlsjmwm6Q.png" /><figcaption>Figure 3. Sample of Login Page in “Sign In With Google” button. Image source: <a href="https://developers.google.com/identity/sign-in/web/sign-in">https://developers.google.com/identity/sign-in/web/sign-in</a> accessed at 8 February 2021 09:04 GMT+7</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/0*qInRignvtqh-ApDJ.jpg" /><figcaption>Figure 4. Consent Page in “Sign In With Google”. Image source: <a href="https://stackoverflow.com/questions/60954001/how-can-i-configure-google-oauth-consent-screen-to-not-show-checkboxes">https://stackoverflow.com/questions/60954001/how-can-i-configure-google-oauth-consent-screen-to-not-show-checkboxes</a> accessed at 8 February 2021 09:08 GMT+7</figcaption></figure><p>To implement those functions, I created a new Identity Provider based on Golang project and put those logic in authcdirectory. For each HTTP handlers above, I created these endpoint:</p><ol><li>GET /authentication/login — <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/login_get.go">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/login_get.go</a></li><li>POST /authentication/login — <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/login_post.go">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/login_post.go</a></li><li>GET /authentication/consent — <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/consent_get.go">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/consent_get.go</a></li><li>POST /authentication/consent — <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/consent_post.go">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/consent_post.go</a></li></ol><p>All of those endpoint is located in <a href="http://localhost:8000/">http://localhost:8000</a> so we need to configure Hydra:</p><pre>URLS_LOGIN=http://localhost:8000/authentication/login<br>URLS_CONSENT=http://localhost:8000/authentication/consent</pre><p>To know more about OAuth 2.0 Authorization Code Grant flow, we also need the <strong>Third Party Application (Client) </strong>to initiates OAuth 2.0 Authorization Code request. In <a href="https://www.ory.sh/hydra/docs/v1.8/5min-tutorial">the 5-minutes tutorial</a> it use hydra token user command using below docker compose command:</p><pre>docker-compose -f quickstart.yml exec hydra \<br>hydra token user \<br>--client-id auth-code-client \<br>--client-secret secret \<br>--endpoint http://127.0.0.1:4444/ \<br>--port 5555 \<br>--scope openid,offline</pre><p>It actually run this code <a href="https://github.com/ory/hydra/blob/v1.9.2/cmd/token_user.go">https://github.com/ory/hydra/blob/v1.9.2/cmd/token_user.go</a> in port 5555.</p><p>Again, to make it more understandable, we need to create our own Front-End system that act as OAuth 2.0 <strong>Third Party Application (Client). </strong>We will create it using Golang with official library golang.org/x/oauth2 and run it in port 1234. You can see this code inside frontenddirectory. This application only consist of 2 endpoints:</p><ul><li>GET <a href="http://localhost:1234/">http://localhost:1234/</a> — return a Homepage contains a Sign In button URL to the Hydra Public Authorization URL (http://localhost:4444/oauth2/auth). In this page, we only generate OAuth 2.0 Authorization Request by sending state unique code to prevent CRSF attack. <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/frontend/main.go#L86-L113">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/frontend/main.go#L86-L113</a></li><li>GET <a href="http://localhost:1234/callbacks">http://localhost:1234/callbacks</a> — a page to retrieve an authorization code in Authorization Code grant flow and statethat we generate in Homepage. We need to ensure that the stateis issued by our application, then in this handler we need to exchange this authorization code with access token. <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/frontend/main.go#L115-L148">https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/frontend/main.go#L115-L148</a><br>After retrieve the access token, it’s up to you to save the access token (and refresh token) in session cookie or local storage, depending in what platform your application will run. If you are building a website client, then you can save it in session cookies. If you are building a mobile apps, you can save it in secure storage. <a href="https://auth0.com/docs/tokens/token-storage">https://auth0.com/docs/tokens/token-storage</a></li></ul><p>Now, we can run Hydra using this docker compose:</p><pre>version: &#39;3.7&#39;<br><br>services:<br>  hydra-migrate:<br>    image: oryd/hydra:v1.9.0<br>    restart: on-failure<br>    networks:<br>      - ory-hydra-network<br>    command:<br>      migrate sql -e --yes<br>    environment:<br>      - DSN=postgres://hydra:secret@postgresd:5432/hydra?sslmode=disable&amp;max_conns=20&amp;max_idle_conns=4<br>    depends_on:<br>      - postgresd<br><br>  hydra:<br>    image: oryd/hydra:v1.9.0<br>    restart: on-failure<br>    networks:<br>      - ory-hydra-network<br>    ports:<br>      - &quot;4444:4444&quot; <em># Public port<br>      </em>- &quot;4445:4445&quot; <em># Admin port<br>      </em>- &quot;5555:5555&quot; <em># Port for hydra token user, testing purpose only<br>    </em>command:<br>      serve all --dangerous-force-http<br>    environment:<br>      <em># https://www.ory.sh/hydra/docs/reference/configuration<br>      # https://github.com/ory/hydra/blob/aeecfe1c8f/test/e2e/docker-compose.yml<br>      </em>- SECRETS_SYSTEM=this-is-the-primary-secret<br>      - URLS_LOGIN=http://localhost:8000/authentication/login <em># Sets the login endpoint of the User Login &amp; Consent flow.<br>      </em>- URLS_CONSENT=http://localhost:8000/authentication/consent <em># Sets the consent endpoint of the User Login &amp; Consent flow.<br><br>      # set to Hydra public domain<br>      </em>- URLS_SELF_PUBLIC=http://localhost:4444 <em># to public endpoint<br>      </em>- URLS_SELF_ISSUER=http://localhost:4444 <em># to public endpoint<br>      </em>- DSN=postgres://hydra:secret@postgresd:5432/hydra?sslmode=disable&amp;max_conns=20&amp;max_idle_conns=4<br>      - SERVE_PUBLIC_PORT=4444<br>      - SERVE_PUBLIC_HOST=0.0.0.0<br>      - SERVE_PUBLIC_CORS_ENABLED=true<br>      - SERVE_ADMIN_PORT=4445<br>      - LOG_LEVEL=debug<br>      - TRACING_PROVIDER=jaeger<br>      - TRACING_PROVIDERS_JAEGER_SAMPLING_SERVER_URL=http://jaeger:5778/sampling<br>      - TRACING_PROVIDERS_JAEGER_LOCAL_AGENT_ADDRESS=jaeger:6831<br>      - TRACING_PROVIDERS_JAEGER_SAMPLING_TYPE=const<br>      - TRACING_PROVIDERS_JAEGER_SAMPLING_VALUE=1<br>    depends_on:<br>      - postgresd<br>      - jaeger<br><br>  postgresd:<br>    image: postgres:13<br>    restart: on-failure<br>    networks:<br>      - ory-hydra-network<br>    ports:<br>      - &quot;5433:5432&quot;<br>    environment:<br>      - POSTGRES_USER=hydra<br>      - POSTGRES_PASSWORD=secret<br>      - POSTGRES_DB=hydra<br>    volumes:<br>      - ./logs:/var/lib/postgresql/data<br><br>  adminer:<br>    image: adminer<br>    restart: always<br>    networks:<br>      - ory-hydra-network<br>    ports:<br>      - 9000:8080<br><br>  jaeger:<br>    image: jaegertracing/all-in-one:1.7.0<br>    restart: on-failure<br>    networks:<br>      - ory-hydra-network<br>    ports:<br>      - 5775:5775/udp<br>      - 6831:6831/udp<br>      - 6832:6832/udp<br>      - 16686:16686 <em># Web App GUI to see traces<br><br></em>networks:<br>  ory-hydra-network:<br>    name: ory-hydra-net</pre><p>Then you can run using docker-compose up in terminal. It will spin up a two HTTP server, one in port 4444 for Hydra Public Endpoint. This is an OAuth 2.0 server and should be publicly available by the <strong>Third Party Application (Client)</strong>. Then port 4445 is for Hydra Admin and must not be available to public. You can deploy it behind a VPN where only your <strong>Identity Provider</strong> can access it. This Hydra Admin is used for accepting login request and consent request as well as revoking it.</p><p>We already set in URLS_LOGIN and URLS_CONSENT to <a href="http://localhost:8000/">http://localhost:8000</a> as our <strong>Identity Provider </strong>server, now we can try to run the pre-built binaries to serve Login and Page consent by downloading the binaries from <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/releases/tag/v0.1.0">https://github.com/yusufsyaifudin/oauth2-example-hydra/releases/tag/v0.1.0</a> or by git clone and run directly on your machine:</p><p>Clone the repository:</p><pre>$ git clone git@github.com:yusufsyaifudin/oauth2-example-hydra.git<br>$ cd oauth-example-hydra<br>$ git checkout v0.1.0</pre><p>Then download dependencies:</p><pre>$ go mod download</pre><p>Now run the <strong>Identity Provider:</strong></p><pre>$ go run cmd/authc/main.go</pre><p>It will run a HTTP server in port 8000.</p><p>Before we can use the front-end application, we need to create an OAuth 2.0 Client App by requesting to Hydra Admin URL in different terminal tab:</p><pre>curl -X POST &#39;http://localhost:4445/clients&#39; \<br>-H &#39;Content-Type: application/json&#39; \<br>--data-raw &#39;{<br>    &quot;client_id&quot;: &quot;myclient&quot;,<br>    &quot;client_name&quot;: &quot;MyApp&quot;,<br>    &quot;client_secret&quot;: &quot;mysecret&quot;,<br>    &quot;grant_types&quot;: [&quot;authorization_code&quot;, &quot;refresh_token&quot;],<br>    &quot;redirect_uris&quot;: [&quot;http://localhost:1234/callbacks&quot;],<br>    &quot;response_types&quot;: [&quot;code&quot;, &quot;id_token&quot;],<br>    &quot;scope&quot;: &quot;offline users.write users.read users.edit users.delete&quot;,<br>    &quot;token_endpoint_auth_method&quot;: &quot;client_secret_post&quot;<br>}&#39;</pre><p>In above command, we create an OAuth 2.0 Client with client_id myclient and client_secret mysecret. This client is only for OAuth 2.0 grant type authorization_code and refresh_token. We also define a custom scope for this client, that is users.write, users.read, users.edit, and users.delete. The offline and openid scope is for Open ID Connect, see here <a href="https://www.ory.sh/hydra/docs/v1.8/debug#oauth-20-refresh-token-is-missing">https://www.ory.sh/hydra/docs/v1.8/debug#oauth-20-refresh-token-is-missing</a></p><p>Here, we also set the Front-End URL (or Mobile Apps URI) to <a href="http://localhost:1234/callbacks">http://localhost:1234/callbacks</a> as a whitelist to retrieve authorization code.</p><p>Then, open a new terminal tab and run:</p><pre>REDIRECT_URL=http://localhost:1234/callbacks CLIENT_ID=myclient CLIENT_SECRET=mysecret go run cmd/frontend/main.go</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8oljhUyUKgWkn9zjdj0gg.png" /><figcaption>Figure 5. Left tab is for docker compose, top-right tab is for Identity Provider run in port 8000, bottom-right is for Front-End run in port 1234.</figcaption></figure><p>Now, open in private session web browser to <a href="http://localhost:1234/">http://localhost:1234,</a> it should show following page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qdzplsf--TYn2dljSVQDkQ.png" /><figcaption>Figure 6. Page show “Click here to _Sign In with YourApplicationName_”</figcaption></figure><p>If you cannot understand what the page says, it is in Indonesian and freely translated in English as “Click here to _Sign In with YourApplicationName_”. Then if we click that button, it will redirect to Login Page from Identity Provider authc that we build in above:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zepSyjZsnKBJTee4C7iV9Q.png" /><figcaption>Figure 7. A Login Page in Identity Provider.</figcaption></figure><p>Look at the address bar, you see that Hydra send login_challenge payload in query parameter. Use username user@example.com and password password to authenicate the user with id 1. Then it will show the Consent Page with consent_challenge in query parameter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hlxIlwcmBSsEVOOICgu1YA.png" /><figcaption>Figure 8. A Consent Page in Identity Provider</figcaption></figure><p>In this example, we request all possible scope that myclient can access. So, it will show like Figure 8. But, in reality we can make offline and openid scope to always included in the request (when accepting the Consent Challenge).</p><p>Please also keep in mind that the scope it self is just a set of definition that your Identity Provider should be handle. Hydra as an Authorization Server only save it and does not do anything about it. For example, when we say users.write it does not tell Hydra to give user permission to write data. Instead, your Identity Provider (Resource Server) should define and also handle the name of each scopes. That’s why when you are implementing several OAuth 2.0 provider like Google or Github, they have their own scope definition.</p><p><a href="https://docs.github.com/en/developers/apps/scopes-for-oauth-apps">Scopes for OAuth Apps</a></p><p>Back to figure 8, when we click button “Authorize” it will then <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/authc/handler/consent_post.go#L49-L52">accepts the consent request in the Identity Provider (endpoint POST /authenticate/consent)</a> and also redirect back to the front-end. Front-end then will retrieve the authorization code in the query parameter and <a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/blob/v0.1.0/cmd/frontend/main.go#L139-L140">exchange it with the real access token</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Z9Sm1_ecE0bSsCIfmejEQ.png" /><figcaption>Figure 9. A Front-End callback retrieve a authorization code and state to be validated.</figcaption></figure><p>The process after retrieving token is up to you. You can save it in session or secure place in your application. Please note that this example is just for education purpose only, for deploying it into production you must be aware of several parameter. For example, in Mobile Apps application you must implements PKCE to add security layer in the authorization code exchange process.</p><p><a href="https://github.com/yusufsyaifudin/oauth2-example-hydra/tree/v0.1.0">yusufsyaifudin/oauth2-example-hydra</a></p><h3>References which also good to read</h3><ul><li><a href="https://developer.okta.com/blog/2018/04/10/oauth-authorization-code-grant-type">https://developer.okta.com/blog/2018/04/10/oauth-authorization-code-grant-type</a> accessed at 7 February 2021 20:43 GMT+7</li><li><a href="https://aaronparecki.com/oauth-2-simplified/">https://aaronparecki.com/oauth-2-simplified/</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://www.ory.sh/hydra">https://www.ory.sh/hydra</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://stackoverflow.com/questions/7561631/oauth-2-0-benefits-and-use-cases-why">https://stackoverflow.com/questions/7561631/oauth-2-0-benefits-and-use-cases-why</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://stackoverflow.com/questions/4113934/how-is-oauth-2-different-from-oauth-1">https://stackoverflow.com/questions/4113934/how-is-oauth-2-different-from-oauth-1</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://oauth.net/articles/authentication/">https://oauth.net/articles/authentication/</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://darutk.medium.com/diagrams-and-movies-of-all-the-oauth-2-0-flows-194f3c3ade85">https://darutk.medium.com/diagrams-and-movies-of-all-the-oauth-2-0-flows-194f3c3ade85</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://security.stackexchange.com/questions/214980/does-pkce-replace-state-in-the-authorization-code-oauth-flow">https://security.stackexchange.com/questions/214980/does-pkce-replace-state-in-the-authorization-code-oauth-flow</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://www.scottbrady91.com/OAuth/Why-the-Resource-Owner-Password-Credentials-Grant-Type-is-not-Authentication-nor-Suitable-for-Modern-Applications">https://www.scottbrady91.com/OAuth/Why-the-Resource-Owner-Password-Credentials-Grant-Type-is-not-Authentication-nor-Suitable-for-Modern-Applications</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://www.identityserver.com/articles/an-introduction-to-the-oauth-device-flow/">https://www.identityserver.com/articles/an-introduction-to-the-oauth-device-flow/</a> accessed at 8 February 2021 17:46 GMT+7</li><li><a href="https://stackoverflow.com/questions/17679523/am-i-right-in-thinking-oauth-1-0-has-been-deprecated-in-favour-of-oauth-2-0">https://stackoverflow.com/questions/17679523/am-i-right-in-thinking-oauth-1-0-has-been-deprecated-in-favour-of-oauth-2-0</a> accessed at 8 February 2021 17:46 GMT+7</li></ul><p>Yogyakarta, 8 February 2021. Work From Home.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbaa2765d94f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn How to Use and Deploy Jaeger Components in Production]]></title>
            <link>https://medium.com/swlh/learn-how-to-use-and-deploy-jaeger-components-in-production-fddb9947b2b2?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/fddb9947b2b2</guid>
            <category><![CDATA[opentracing]]></category>
            <category><![CDATA[opentelemetry]]></category>
            <category><![CDATA[jaeger]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Thu, 19 Nov 2020 14:59:49 GMT</pubDate>
            <atom:updated>2020-11-24T09:59:10.821Z</atom:updated>
            <content:encoded><![CDATA[<p>Past week I got questioned again about Jaeger — an end to end distributed tracing. That question come from my workmates that they’re in doubt whether the Jaeger can be single point of failure. Though this post will not answer that case explicitly, but it will give a new perspective about how Jaeger should be properly deployed in production (both for me and for you — the readers).</p><p>First time when I get my hand dirty with the so called things as “distributed tracing”, I always use Jaeger all-in-one docker image which just works. At that time I don’t think that it would be hard for DevOps or SRE to give the resilient environment of Jaeger for the developer so that developer can focus on their code and help them to trace the request’s flow.</p><p>So, for the end of this preamble, I just want to clarify that I write this post just to documented about what I learn instead giving some advice about dos and don’ts.</p><h3>Development and Playground Mode</h3><p>In Development, Localhost or even Playground mode, the all-in-one Jaeger installation is the right choice for the developer to play with Jaeger. Using below docker compose YAML file, we can easily run all Jaeger’s component and play with it. This is what I have done when I learn about Jaeger back in last year of 2018.</p><p><a href="https://medium.com/@yusufs/understand-your-system-behavior-by-implement-distributed-tracing-using-jaeger-9041df1104ec">Understand Your System Behavior By Implement Distributed Tracing Using Jaeger</a></p><pre>version: &#39;3&#39;<br><br>services:<br>  jaeger:<br>    image: jaegertracing/all-in-one:1.7<br>    container_name: jaeger<br>    restart: on-failure<br>    ports:<br>      - 5775:5775/udp<br>      - 6831:6831/udp<br>      - 6832:6832/udp<br>      - 5778:5778<br>      - 16686:16686<br>      - 14268:14268<br>      - 9411:9411<br>    environment:<br>      - COLLECTOR_ZIPKIN_HTTP_PORT=9411</pre><p>At first, I imagine that the installation of Jaeger may look like this even in the production mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*rzRLQquUyyFoRAi74liN7w.png" /><figcaption>Wrong Jaeger Deployment Installation</figcaption></figure><p>This because I didn’t read the documentation thoroughly, not before the question about “how to deploy Jaeger in production” came. So, after that, I revisit Jaeger website and learn about the Architecture and Deployment strategy.</p><h3>Jaeger Architecture</h3><p><a href="https://www.jaegertracing.io/docs/1.21/architecture/">Architecture</a></p><p>Although this looks like useless point and all the documentation about this already written in official website, I think it is not that worthless to mention it again.</p><p>As I stated above, we can deploy Jaeger using all-in-one strategy. This means that any Jaeger components can be installed in one high spec machine. But, this is come into problem when we talk about “single point of failure”, and isn’t it kind of weird when we want to trace the large distributed service but send the metrics into only one “monolith” service?</p><blockquote><strong><em>Isn’t it kind of weird when we want to trace the large distributed service but send the metrics into only one “monolith” service?</em></strong></blockquote><p>So, I take a look into official documentation, Jaeger can provide two way to deploy as a scalable distributed system:</p><ol><li>Directly write metrics into persistent storage</li><li>Buffer the trace metrics into Kafka before writing into persistent storage</li></ol><h4>Directly write metrics into persistent storage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*UKRDDvjdKuOCoV34bke5wA.png" /><figcaption><em>Illustration of direct-to-storage architecture. Image from </em><a href="https://www.jaegertracing.io/docs/1.21/architecture/">https://www.jaegertracing.io/docs/1.21/architecture/</a></figcaption></figure><p>In this deployment strategy, the Jaeger collector directly writes the trace metrics into Database. Then Jaeger UI access DB to visualize the system behavior.</p><p>This scenario works well with small to medium traffic, because Jaeger supports two types of persistent storage that renowned works well under high workloads. Also, both type of storage can be deployed in cluster mode so this will eliminate our concerns about single point of failure in storage. For your additional information, in the official documentation, “For large scale production deployment the Jaeger team <a href="https://www.jaegertracing.io/docs/1.21/faq/#what-is-the-recommended-storage-backend">recommends Elasticsearch backend over Cassandra</a>.”</p><h4>Buffer the trace metrics into Kafka before writing into persistent storage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*KbycqRzXUrPiqMxQO2ZRUQ.png" /><figcaption><em>Illustration of architecture with Kafka as intermediate buffer. </em>Image from <a href="https://www.jaegertracing.io/docs/1.21/architecture/">https://www.jaegertracing.io/docs/1.21/architecture/</a></figcaption></figure><p>This is the deployment scenario that I may choose for production because of several reasons.</p><ol><li>First, as far as I know, Kafka can scale and work very well even under big data produced into it. You can learn more about Kafka in <a href="https://medium.com/@yusufs/getting-started-with-kafka-in-golang-14ccab5fa26">my another blog post about Kafka</a> or take a quick look about this <a href="https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines">benchmark</a>.</li><li>Second, we can decoupled the persistent storage or DB (that will takes time to persist data into disk) from the collector (which may receive hundreds metric per second to be saved).</li><li>Third, as the complement for the first and second reason, we can deploy many Jaeger collector and Jaeger ingester, so that metrics can be seen via Jaeger UI instantly.</li></ol><p>Ok, this third reason may leads to the new question: How we can vertically scale Jaeger collector and ingester? Would it means metrics data spreadly send into multi collector?</p><p><a href="https://www.jaegertracing.io/docs/1.21/deployment/">Deployment</a></p><p>To answer that question, again we must refer to the official Jaeger documentation. Thanks to it’s enough clarity. And yes, metrics data can send into multi collector. Below is the resume how each component should be installed:</p><p><strong>Jaeger Agent</strong></p><blockquote><a href="https://www.jaegertracing.io/docs/1.21/deployment/#agent">Jaeger client libraries expect <strong>jaeger-agent</strong> process to run locally on each host.</a></blockquote><p>This is clear statement that Agent should be installed in host machine or if you prefer you can directly send your trace metrics directly into Jaeger collector. This also suggested by the creator of Jaeger, Yuri Shkuro in this responses:</p><blockquote>Jaeger agent should always run on the same host as the application, as a sidecar or a host agent. Alternatively, Jaeger clients can be configured to send spans directly to the collector, which then can run anywhere.<br>- <strong>Yuri Shkuro</strong></blockquote><p><a href="https://medium.com/@YuriShkuro/jaeger-agent-should-always-run-on-the-same-host-as-the-application-as-a-sidecar-or-a-host-agent-99ef5dd3eb97">Jaeger agent should always run on the same host as the application, as a sidecar or a host agent.</a></p><p>If you use Kubernetes to deploy your services, you can also refer into this great article:</p><p><a href="https://medium.com/jaegertracing/deployment-strategies-for-the-jaeger-agent-1d6f91796d09">Deployment strategies for the Jaeger Agent</a></p><p>Jaeger agent acts as intermediate buffer between your application to the Jaeger collector. Having this close to your application will benefit to your performance since your application will send data using UDP protocol (commonly port 6831) and then buffer it to Jaeger collector using gRPC (commonly port 14250 in Jaeger collector). Because UDP is stateless protocol, it makes sense that installing in same host will reducing the risk of lost of data when sending metrics.</p><p><strong>Jaeger Collector</strong></p><blockquote><a href="https://www.jaegertracing.io/docs/1.21/deployment/#collectors">The collectors are stateless and thus many instances of <strong>jaeger-collector</strong> can be run in parallel.</a></blockquote><p>Actually we can run only once Collector for many Agent, but for scalability we can also deploy it as many as we want</p><p><strong>Jaeger Ingester</strong></p><blockquote><a href="https://www.jaegertracing.io/docs/1.21/deployment/#ingester"><strong>jaeger-ingester</strong> is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).</a></blockquote><p>Ingester works as a worker to consume Kafka message. We can also deploy many Jaeger ingester to make it more as responsive as possible to write tracing data into our data storage (Elasticsearch) when traces message published into Kafka.</p><p>The only limitation about how many you should deploy Jaeger ingester is how many you setup your Kafka topic partition. This is because when there is more consumer (ingester) than the topic partition, some of the ingester will idle because it will not receives the message. To learn about this, you should learn about the Kafka consumer group id which <a href="https://medium.com/@yusufs/getting-started-with-kafka-in-golang-14ccab5fa26">I explained in Kafka tutorial post.</a></p><p><strong>Jaeger Query UI</strong></p><blockquote><a href="https://www.jaegertracing.io/docs/1.21/deployment/#query-service--ui"><strong>jaeger-query</strong> serves the API endpoints and a React/Javascript UI. The service is stateless and is typically run behind a load balancer, such as</a> <a href="https://www.nginx.com/"><strong>NGINX</strong></a>.</blockquote><p>Jaeger Query UI is a dashboard to visualize about the tracing. It connects directly into Elasticsearch (or Cassandra) to query our tracing data. In all-in-one Docker image we may not see any persistent storage and that’s true! In all-in-one binary, we can use In Memory or Badger storage which is not recommended for heavy tracing metrics.</p><h3>Example Using Docker Compose</h3><p>In the end of the learning, as usual, I always included the working example because common software engineer tend to try the code instead only reading and to give the reader a chance to get their hand dirty too. :)</p><p>This example just give you the idea and proof that multi installation Jaeger Agent, Collector and Ingester is just works fine and answering the first question that the Jaeger components can be scaled.</p><p>In this example I want to create two services, which having architecture design like this (including Jaeger components):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*CmPGYEawk5eT3EfKQ5vkXg.png" /><figcaption>Example services architecture</figcaption></figure><p>We will create two services one named Dora The Explorer (<a href="https://medium.com/@yusufs/understand-your-system-behavior-by-implement-distributed-tracing-using-jaeger-9041df1104ec">you may familiar with this from my previous blog</a>) and another one is Umbrella service.</p><p>For service Dora we send tracing data into Jaeger Agent port 1111 and for Umbrella’s to 1112. Actually, you must install Jaeger Agent in the same host, so using localhost:6831 for each application will never be conflicted.</p><p>However, since this example we use Docker Compose, we mimicking that application in host 1 is using Jaeger Agent port 1111 and application in host 2 using port 1112.</p><p>Each application send data into their respective Jaeger Agent in the same host through UDP. Each Jaeger Agent then (both in port 1111 and 1112) then send with static load balancing to 2 Jaeger Collector using gRPC protocol in port 14250. Collector then publish message into same Kafka topic.</p><p>This is the part that you must give an attention: you must create more than one partition regarding how many you would deploy Jaeger ingester. For example, if you want to deploy 4 Jaeger Ingester, you need at least 4 partition of Kafka topic to make it work in maximum efforts.</p><p><strong>Let’s try it!</strong></p><p>First, let’s run below docker compose file using this command:</p><pre>MY_IP=$(ifconfig | sed -En &#39;s/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p&#39;) docker-compose -f docker-compose-copy.yml up</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f88aa0905bf8ebec7c15b2031bbbe6ed/href">https://medium.com/media/f88aa0905bf8ebec7c15b2031bbbe6ed/href</a></iframe><p>Clone this repo and checkout into branch with-external-service :</p><pre>$ git clone <a href="https://github.com/yusufsyaifudin/go-opentracing-example.git">https://github.com/yusufsyaifudin/go-opentracing-example.git</a><br>$ cd go-opentracing-example<br>$ git checkout with-external-service</pre><p>Run main.go and external_service/main.go in separate terminal. Then do cURL into it:</p><pre>curl -X GET &#39;localhost:1323/dora-the-explorer?is_rainy_day=true&#39;</pre><p>You will get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wu33u5rm3HotmuprnGlEpA.png" /><figcaption>Running two API service and cURL into it.</figcaption></figure><p>Open browser then go to <a href="http://localhost:16686/search">http://localhost:16686/search</a>, search for newcoming tracing, you will get view something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGEyAd05hCTM44QahN9CCg.png" /><figcaption>Jaeger UI shows one trace data already come in 4 minutes ago.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*50E9kMP9MJoCWFtNtIcDtg.png" /><figcaption>The detail of the request life-span.</figcaption></figure><p>Now, try to run load testing using k6 to proof that even under load test the Jaeger can still receives the tracing data without no error:</p><pre>$ k6 run --vus 10 --duration 30s load-testing.js</pre><p><a href="https://k6.io/">Load testing for engineering teams | Grafana k6</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r2S_0_v7NLDD5pj14C4InQ.png" /><figcaption>k6 Load Testing to prove that Jaeger Agent won’t blocking out the request and make the single point of failures.</figcaption></figure><p>After that, you can back into Jaeger UI (<a href="http://localhost:16686/">http://localhost:16686/</a>) and then check if many tracing span show in Jaeger UI. Please note that it may take a while for Jaeger UI to show the list of tracing. But, in my experience the Jaeger consistently show all span using this deployment strategy (deploying each Jaeger component with different docker image). Back then when I use all-in-one Docker Image, I sometimes find that some spans is missing and I don’t find it again using this new scenario of deployment as far as I observe.</p><h3>What Next?</h3><p>Again as I stated in the preamble that this post only documented my experience during learning deploying Jaeger components as individual service. We then can conclude that Jaeger can works under high workloads with its ability to deploy in vertical scale.</p><p><a href="https://medium.com/jaegertracing/ticketmaster-traces-100-million-transactions-per-day-with-jaeger-38ec6cf599f0">Ticketmaster Traces 100 Million Transactions per Day with Jaeger</a></p><p>Some of us may already using Kubernetes in your architecture. For this case, you can learn from Jaeger Medium blog posts:</p><ul><li><a href="https://medium.com/jaegertracing/deployment-strategies-for-the-jaeger-agent-1d6f91796d09">Deployment strategies for the Jaeger Agent</a></li><li><a href="https://medium.com/jaegertracing/a-guide-to-deploying-jaeger-on-kubernetes-in-production-69afb9a7c8e5">A Guide to Deploying Jaeger on Kubernetes in Production</a></li></ul><p><strong><em>Written by Yusuf on Wed 18 — Thu 19, November 2020 during the Covid-19 Pandemic.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fddb9947b2b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/learn-how-to-use-and-deploy-jaeger-components-in-production-fddb9947b2b2">Learn How to Use and Deploy Jaeger Components in Production</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating Distributed KV Database by Implementing Raft Consensus Using Golang]]></title>
            <link>https://yusufs.medium.com/creating-distributed-kv-database-by-implementing-raft-consensus-using-golang-d0884eef2e28?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0884eef2e28</guid>
            <category><![CDATA[key-value-store]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[raft]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[distributed-database]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Sat, 30 May 2020 02:55:25 GMT</pubDate>
            <atom:updated>2020-05-30T02:55:25.856Z</atom:updated>
            <content:encoded><![CDATA[<p>Well, you may heard a lot about “distributed system” or “distributed database” itself. But, you may wonder how to create it or the internal behind it. In this post, I will try to explain how to create one of it using Raft consensus. This post more discussed about how to code and test the Raft cluster rather than discussing the internal Raft.</p><h3>What is Raft Consensus?</h3><p>When it comes to distributed system, we need a consensus to agree on some values. Since servers are located in different place it is hard to maintain the agreement. Imagine that you and your workmates want to make a discussion to solve a problem. When you all are in same meeting room, you can just wave your hand when you have an idea. But, when you WFH, there is some barrier that prevent you to speak, such as slow internet connection, your friends are speaking at the same time, etc.</p><p>To prevent this, you may create some rules, e.g: just one person who talk at one time, other person must mute their mic, etc. These set of rules is a consensus in order to achieve agreement.</p><p>In Raft Consensus, in order to broadcast the message to all node it must be sent by the Leader. The Leader will be elected by voting mechanism. You can search in Google how Raft consensus elect the leader. Article on <a href="https://www.consul.io/docs/internals/consensus.html">Consul</a> and <a href="https://www.vaultproject.io/docs/internals/integrated-storage#raft-protocol-overview">Vault</a> already tell some of common terms in Raft.</p><p><a href="https://raft.github.io/">Raft Consensus Algorithm</a></p><h3>Let’s Implement Raft</h3><p>I will use library from Hashicorp to create Raft cluster, and save it persistently in each node using <a href="https://github.com/dgraph-io/badger">BadgerDB</a>. First, you may already heard about <a href="https://www.sciencedirect.com/topics/engineering/finite-state-machine">Finite State Machine (FSM)</a> in Raft. It is the place where you can process the data in the local node.</p><p>First, you may notice (and must aware) that all operation in Raft Consensus is must be through the Leader. So, we need to ensure that all request must be sent to Leader node. Leader node then will send the command to all the follower, and wait to majority the server process the command. How many numbers of “majority” is symbolized by N number called “quorum”. Each follower then will do these:</p><ol><li>After receiving the command operation, data will be save in Write Ahead Log mode as a log entries.</li><li>After successfully writing log entries, the data will be sent to FSM where you must process your data in a deterministic way.</li><li>After you successfully process the data in FSM, it will return the data. Then the leader will be notified that this node already success. Once the Leader thinks that there is enough follower telling that they already successfully process the data, then the Leader will tell the client that data already “distributed” to N quorum.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HmKrS15OxCeBsZV3" /><figcaption>Image source: <a href="https://www.sofastack.tech/en/projects/sofa-jraft/consistency-raft-jraft/">https://www.sofastack.tech/en/projects/sofa-jraft/consistency-raft-jraft/</a></figcaption></figure><p>So, here is the flow diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/0*zuLBXKd2XHd5L3sv" /><figcaption>Image source: <a href="https://www.sofastack.tech/en/projects/sofa-jraft/consistency-raft-jraft/">https://www.sofastack.tech/en/projects/sofa-jraft/consistency-raft-jraft/</a></figcaption></figure><ul><li><a href="https://github.com/hashicorp/raft">GitHub - hashicorp/raft: Golang implementation of the Raft consensus protocol</a></li><li><a href="https://github.com/dgraph-io/badger">dgraph-io/badger</a></li></ul><h3>Coding!</h3><p>Okay, we already know the higher level of data flow in <a href="https://github.com/hashicorp/raft">hashicorp/raft</a> library and what the process behind it before client received “ok” by the leader server. Now, let me to tell you the implementation in code based on three step above one by one.</p><p>1. Create the server! In this example, we will create two server: HTTP server and RPC server. HTTP server is for us to interact with the each of node server, while RPC server is the built-in server from the <a href="https://github.com/hashicorp/raft">hashicorp/raft </a>library to communicate and exchange command between Leader to Follower(s).</p><p>Here is the snippet creating Raft:</p><pre>raftServer, err := raft.NewRaft(raftConf, fsmStore, cacheStore, store, snapshotStore, transport)<br>if err != nil {<br> log.Fatal(err)<br> return<br>}</pre><p>And here is the snippet creating HTTP server using Echo Framework:</p><pre>e := echo.New()<br>e.HideBanner = true<br>e.HidePort = true<br>e.Pre(middleware.RemoveTrailingSlash())<br>e.GET(&quot;/debug/pprof/*&quot;, echo.WrapHandler(http.DefaultServeMux))<br><br>// Raft server<br>raftHandler := raft_handler.New(r)<br>e.POST(&quot;/raft/join&quot;, raftHandler.JoinRaftHandler)<br>e.POST(&quot;/raft/remove&quot;, raftHandler.RemoveRaftHandler)<br>e.GET(&quot;/raft/stats&quot;, raftHandler.StatsRaftHandler)<br><br>// Store server<br>storeHandler := store_handler.New(r, badgerDB)<br>e.POST(&quot;/store&quot;, storeHandler.Store)<br>e.GET(&quot;/store/:key&quot;, storeHandler.Get)<br>e.DELETE(&quot;/store/:key&quot;, storeHandler.Delete)</pre><p>Let me briefly explain it, in order to create a cluster, Raft need to know each node. That’s why we create 3 endpoint using prefix /raft which consists of handler to add (join) and remove the node (Voter in Raft), and one to see each node statistics (or status).</p><p>To store, fetch and delete data, we also create 3 other endpoint: POST /store, GET /store/:key and DELETE /store/:key. I will show one of this endpoint later in step 4.</p><p>2. In step 2 above, I mentioned about log entries. It is place where <a href="https://github.com/hashicorp/raft">hashicorp/raft</a> save all log related to any cluster change: adding nodes, adding services, new key-value pairs, etc. It is a sequential entries, as already stated in Consul docs:</p><blockquote>Log — The primary unit of work in a Raft system is a log entry. The problem of consistency can be decomposed into a <em>replicated log</em>. A log is an ordered sequence of entries. Entries includes any cluster change: adding nodes, adding services, new key-value pairs, etc. We consider the log consistent if all members agree on the entries and their order.<br><a href="https://www.consul.io/docs/internals/consensus.html#raft-protocol-overview">https://www.consul.io/docs/internals/consensus.html#raft-protocol-overview</a></blockquote><p>3. Finite State Machine. In <a href="https://github.com/hashicorp/raft">hashicorp/raft</a>, you must implement raft.FSM interface to create Finite State Machine. It consists of 3 functions: <br>* Apply will be invoked when Raft already committed log entries in step 2. <br>* Snapshot is used to support log compaction. This can be used to save a point-in-time snapshot of the FSM.<br>* Restore is used to restore an FSM from a snapshot.</p><p>You may already noticed in step 1 when we creating new Raft instance, we pass the variable fsmStore. This actually just from calling this function fsm.NewBadger(badgerDB) with below code of implementation using BadgerDB:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10321f6c559d6d67937a2ffce9ee6a8e/href">https://medium.com/media/10321f6c559d6d67937a2ffce9ee6a8e/href</a></iframe><p>In Apply method you will see that it only handle raft.LogCommand which has type CommandPayload. This is because in this library, it pass the []byte slices where in this program we passing the JSON string using CommandPayload structure (I will explain it in step 4).</p><p>You also see that it handle each command SET, GET, and DELETE using function b.set, b.get and b.delete. It actually just a wrapper function to save, fetch and delete data in BadgerDB. You can see the implementation in this link: <a href="https://github.com/yusufsyaifudin/raft-sample/blob/48254d0c71/fsm/badger.go#L18-L93">https://github.com/yusufsyaifudin/raft-sample/blob/48254d0c71/fsm/badger.go#L18-L93</a></p><p>4. So, let’s take the POST /store function for example. As mentioned before, all raft operation should be done in Leader node, but in this program for GET /store/:key we will not use Leader node, instead we will use local BadgerDB to get the data. It may lead an inconsistent result, but this is intended for something that we call “eventual-consistency”.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d700c968e2e7e48579c01deec76eff5/href">https://medium.com/media/9d700c968e2e7e48579c01deec76eff5/href</a></iframe><p>In line 37–41, we checked that the operation must be done in Leader node, otherwise the handler will return error response. When the request is on Leader node, it will then build the payload using CommandPayload with operation SET. This will be encoded to JSON as []byte slices and passed to raft.Apply in line 56. If you still remember, in FSM code, we convert the []byte to CommandPayload, this is where those data are come from.</p><p>The raft.Apply then will broadcast the message to all raft nodes, and when is felt that enough Follower node saying “success” doing the Applyin their respective machine, the leader then return no error. Otherwise, the error will be returned in applyFuture.Error().</p><p>In previous FSM code, we always returned pointer of ApplyResponse, if we want get that value, we can also fetch it from applyFuture.Response and try convert it as *fsm.ApplyResponse.</p><p>This post maybe will long enough to explain all code line by line, but I think above explanation already cover the key point where you can start coding your own Raft cluster. For full code, you can refer to this Github repository (while writing this post I use the commit ID <a href="https://github.com/yusufsyaifudin/raft-sample/commit/45285b23763409f03ca638700423a6690311a617">45285b2</a> so you can do git checkout 45285b2 by yourself):</p><p><a href="https://github.com/yusufsyaifudin/raft-sample">yusufsyaifudin/raft-sample</a></p><h3>Testing</h3><p>Testing in one critical part in software development, it ensures that our system already works as expected. Let’s do our manual testing for the software we just built. We will do these steps:</p><ul><li>Run the program.</li><li>Join all nodes to create cluster.</li><li>Testing some store, get from two other Follower node and delete data.</li><li>Testing remove one Follower node and store some data. Rejoin recently removed node to the cluster, try fetch the data from it.</li><li>Shutdown one Follower node, store a data, Start again the terminated node, fetch the data from it.</li><li>Start new node (say, node4) and add the cluster then fetch data that already posted before from this node.</li></ul><p>Done</p><h4><strong>Run the program</strong></h4><p>We will trying creating a raft cluster with 3 nodes: 1 Leader, 2 Follower. If you already learn in somewhere else (or again in <a href="https://www.consul.io/docs/internals/consensus.html#raft-protocol-overview">Consul documentation</a>), the 3 servers means that it need at least 2 nodes to say “okay” when Leader broadcast the raft.Apply before the Leader return “okay” too. Two nodes can be consists of the Leader itself and at least 1 other Follower in the cluster.</p><p>Why we create the 3 node is because we also want to check the raft consistency when node one node have a problem. When using 3 nodes, raft has failure tolerance 1.</p><p>To run the 3 nodes, run this command in different terminal tab:</p><pre>$ SERVER_PORT=2221 RAFT_NODE_ID=node1 RAFT_PORT=1111 RAFT_VOL_DIR=node_1_data go run ysf/raftsample/cmd/api</pre><pre>$ SERVER_PORT=2222 RAFT_NODE_ID=node2 RAFT_PORT=1112 RAFT_VOL_DIR=node_2_data go run ysf/raftsample/cmd/api</pre><pre>$ SERVER_PORT=2223 RAFT_NODE_ID=node3 RAFT_PORT=1113 RAFT_VOL_DIR=node_3_data go run ysf/raftsample/cmd/api</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZdHdQyMIw7YrAjLIzNwqTw.png" /><figcaption>Run the program in three different tab</figcaption></figure><p>Now, check each server, all three of them must be the Leader now:</p><ul><li><a href="http://localhost:2221/raft/stats">http://localhost:2221/raft/stats</a></li><li><a href="http://localhost:2222/raft/stats">http://localhost:2222/raft/stats</a></li><li><a href="http://localhost:2223/raft/stats">http://localhost:2223/raft/stats</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06hD9Lb2im2rMg6s22Bxgg.png" /><figcaption><a href="http://localhost:2221/raft/stats">http://localhost:2221/raft/stats</a> is the Leader</figcaption></figure><h4><strong>Join all nodes to create cluster</strong></h4><p>Now, we manually select that <a href="http://localhost:2221/raft/stats">http://localhost:2221/raft/stats</a> is the Leader, and we want to join the other node as the Follower. In raft, we already mark each node with distinct node id:</p><ul><li>Node ID: node1<br>Raft RPC Port: 1111<br>HTTP Port: 2221</li><li>Node ID: node2<br>Raft RPC Port: 1112<br>HTTP Port: 2222</li><li>Node ID: node3<br>Raft RPC Port: 1113<br>HTTP Port: 2223</li></ul><p>To join node2 as Follower to node1 as the Leader, we can cURL by posting the node id and the Raft RPC address of the node2 to node1 using POST /raft/join.</p><pre>curl -X POST &#39;<a href="http://localhost:2221/raft/join&#39;">http://localhost:2221/raft/join&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>--data-raw &#39;{<br> &quot;node_id&quot;:&quot;node2&quot;,<br> &quot;raft_address&quot;:&quot;<a href="http://localhost:1112/">localhost:1112</a>&quot;<br>}&#39;</pre><p>So, we must join the node3 to node1 which currently elected as the Leader.</p><pre>curl -X POST &#39;<a href="http://localhost:2221/raft/join&#39;">http://localhost:2221/raft/join&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;{<br> &quot;node_id&quot;:&quot;node3&quot;,<br> &quot;raft_address&quot;:&quot;<a href="http://localhost:1113/">localhost:1113</a>&quot;<br>}&#39;</pre><blockquote><strong>Please note that in </strong><strong>raft_address in JSON body, we only use </strong><strong>localhost without </strong><strong>http://. This is because the Raft address should be RPC call, not the HTTP.</strong></blockquote><p>Now, check again in <a href="http://localhost:2221/raft/stats">http://localhost:2221/raft/stats</a> you will see that now the latest_configuration is contained all three servers. At this point, raft cluster already set. We can try to store, fetch, and delete data now.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7IXnZNDHkTXeIt2u4YrB1A.png" /><figcaption>Raft Cluster created</figcaption></figure><h4>Testing some store, get from two other Follower node and delete data</h4><p>We already know that the Leader node is on node1 with HTTP address <a href="http://localhost:2221/">http://localhost:2221</a>. To store new data, we can using this cURL:</p><pre>curl -X POST &#39;<a href="http://localhost:2221/store&#39;">http://localhost:2221/store&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>--data-raw &#39;{<br> &quot;key&quot;: &quot;foo&quot;,<br> &quot;value&quot;: &quot;bar&quot;<br>}&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*PN0fEzQ2_ghIbupWeDRL8Q.png" /><figcaption>Success persisting data</figcaption></figure><p>Then, try to fetch the data from 2 other Follower nodes:</p><pre>curl -X GET &#39;<a href="http://localhost:2222/store/foo&#39;">http://localhost:2222/store/foo&#39;</a><br>curl -X GET &#39;<a href="http://localhost:2223/store/foo&#39;">http://localhost:2223/store/foo&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T-wNNM304hGIEdZwYq10Aw.png" /><figcaption>Success fetching data from node2 and node3</figcaption></figure><p>It shows that 2 others node (node2 and node3) already received and persisted the data that we’ve sent in the Leader node (node1).</p><p>Now, try to remove the data (in this example using key foo), then retry above fetch cURL command. It must shows that the value in response is null.</p><pre>curl -X DELETE &#39;<a href="http://localhost:2221/store/foo&#39;">http://localhost:2221/store/foo&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*GSrloL4R0bUNnctFK9FvRw.png" /><figcaption>Delete data</figcaption></figure><p>When get the data, it shows that the key foo is not found in the storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*LEa1Y4TuMx_5wx5uEksPvg.png" /><figcaption>Error fetching on non-exist or deleted data</figcaption></figure><h4>Testing remove one Follower node and store some data. Rejoin recently removed node to the cluster, try fetch the data from it</h4><p>Now, try to remove node3 from the cluster. Again, the operation of join or removing node must be done by the Leader, so request it to node1.</p><pre>curl -X POST &#39;<a href="http://localhost:2221/raft/remove&#39;">http://localhost:2221/raft/remove&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;{<br> &quot;node_id&quot;:&quot;node3&quot;,<br> &quot;raft_address&quot;:&quot;localhost:1113&quot;<br>}&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wThYsVZU4j38gPPa2AUqjw.png" /><figcaption>Removing node3 from the cluster</figcaption></figure><p>Check the status in <a href="http://localhost:2221/raft/stats">http://localhost:2221/raft/stats</a> now it shows that the cluster only consist of node1 and node2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fNCwtTbgTfaxFl_V3iRBIA.png" /><figcaption>node3 already removed from latest_configuration</figcaption></figure><p>Now, post the data to the leader (node1) and try fetching it from node2 using key removednode :</p><pre>curl -X POST &#39;<a href="http://localhost:2221/store&#39;">http://localhost:2221/store&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>--data-raw &#39;{<br>        &quot;key&quot;: &quot;removednode&quot;,<br>        &quot;value&quot;: &quot;my value&quot;<br>}&#39;</pre><pre>curl -X GET &#39;<a href="http://localhost:2222/store/removednode&#39;">http://localhost:2222/store/removednode&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W9XmUSAUr1OZDsXqyViKUw.png" /><figcaption>Post on node1 and fetch on node2 successfully synced the data</figcaption></figure><p>Now, join the node3 again to node1. And oops! leadership lost while commiting log !</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*Oh2fGRPfm5UN1CniDZMJuQ.png" /><figcaption>Fail when trying rejoining the removed node (node3)</figcaption></figure><p>I don’t know what happen, but when I try it the second time it show success. Please comment below if you know what happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NpWcGekjTw-AunMZT-3IdQ.png" /><figcaption>Rejoining node3 to the cluster success</figcaption></figure><p>Now, try fetching data from node3. Yeay! It already synced with the Leader node!</p><pre>curl -X GET &#39;<a href="http://localhost:2223/store/removednode&#39;">http://localhost:2223/store/removednode&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58UZwloTxBuA7ImW-uTCmw.png" /><figcaption>Fetching data from node3 after rejoining the cluster shows the synced data.</figcaption></figure><h4>Shutdown one Follower node, store a data, Start again the terminated node, fetch the data from it.</h4><p>Same as the previous test case, but now instead of explicitly removing the node3 from the cluster, we try to shutting it down.</p><p>Let’s shutting the node3 down!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08HZdhUkNhAi_0gh6k_Reg.png" /><figcaption>node3 is shutting down</figcaption></figure><p>Once you terminatted the node3, node1 as the leader will notified and show the error in console log which look like this:</p><pre>2020-05-29T17:17:55.816+0700 [DEBUG] raft: failed to contact: server-id=node3 time=1m37.425816508s</pre><p>Then, store to node1 and fetch from node2.</p><pre>curl -X POST &#39;<a href="http://localhost:2221/store&#39;">http://localhost:2221/store&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>--data-raw &#39;{<br>        &quot;key&quot;: &quot;problemnode&quot;,<br>        &quot;value&quot;: &quot;hello world&quot;<br>}&#39;<br>curl -X GET &#39;<a href="http://localhost:2222/store/problemnode&#39;">http://localhost:2222/store/problemnode&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ctm8ed7MCISjvCjdKKDrVw.png" /><figcaption>Success persisting data even with one failure node.</figcaption></figure><p>Now, start again the node3 with same configuration (volume data, raft rpc port, http port), so it will use last state before it terminated.</p><pre>$ SERVER_PORT=2223 RAFT_NODE_ID=node3 RAFT_PORT=1113 RAFT_VOL_DIR=node_3_data go run ysf/raftsample/cmd/api</pre><p>After started again, the node1 will notified that node3 already up again.</p><pre>2020-05-29T17:20:58.043+0700 [ERROR] raft: peer has newer term, stopping replication: peer=&quot;{Voter node3 localhost:1113}&quot;<br>2020-05-29T17:20:58.043+0700 [INFO]  raft: entering follower state: follower=&quot;Node at :1111 [Follower]&quot; leader=<br>2020-05-29T17:20:58.043+0700 [INFO]  raft: aborting pipeline replication: peer=&quot;{Voter node2 localhost:1112}&quot;<br>2020-05-29T17:20:58.174+0700 [DEBUG] raft: lost leadership because received a requestVote with a newer term<br>2020-05-29T17:20:58.216+0700 [WARN]  raft: rejecting vote request since our last index is greater: candidate=:1113 last-index=12 last-candidate-index=11<br>2020-05-29T17:20:59.659+0700 [WARN]  raft: heartbeat timeout reached, starting election: last-leader=<br>2020-05-29T17:20:59.659+0700 [INFO]  raft: entering candidate state: node=&quot;Node at :1111 [Candidate]&quot; term=265<br>2020-05-29T17:20:59.720+0700 [DEBUG] raft: votes: needed=2<br>2020-05-29T17:20:59.720+0700 [DEBUG] raft: vote granted: from=node1 term=265 tally=1<br>2020-05-29T17:20:59.832+0700 [DEBUG] raft: vote granted: from=node2 term=265 tally=2<br>2020-05-29T17:20:59.833+0700 [INFO]  raft: election won: tally=2<br>2020-05-29T17:20:59.833+0700 [INFO]  raft: entering leader state: leader=&quot;Node at :1111 [Leader]&quot;<br>2020-05-29T17:20:59.833+0700 [INFO]  raft: added peer, starting replication: peer=node2<br>2020-05-29T17:20:59.833+0700 [INFO]  raft: added peer, starting replication: peer=node3<br>2020-05-29T17:20:59.833+0700 [INFO]  raft: pipelining replication: peer=&quot;{Voter node2 localhost:1112}&quot;<br>2020-05-29T17:20:59.843+0700 [WARN]  raft: appendEntries rejected, sending older logs: peer=&quot;{Voter node3 localhost:1113}&quot; next=12<br>2020-05-29T17:20:59.881+0700 [INFO]  raft: pipelining replication: peer=&quot;{Voter node3 localhost:1113}&quot;</pre><p>Then, try to fetch the data from node3:</p><pre>curl -X GET &#39;<a href="http://localhost:2223/store/problemnode&#39;">http://localhost:2223/store/problemnode&#39;</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q8WcqKd6EFAY4DPa4TcvOA.png" /><figcaption>node3 also have the latest data after it up again</figcaption></figure><p>As expected, because we already try using remove-rejoin node where the data still synced up while node lost it’s contact, in this scenario we also see that even after it’s failure the Follower node can keep up sync with the leader node.</p><h4>Start new node (say, node4) and add the cluster then fetch data that already posted before from this node.</h4><p>Start the new node,</p><pre>SERVER_PORT=2224 RAFT_NODE_ID=node4 RAFT_PORT=1114 RAFT_VOL_DIR=node_4_data go run ysf/raftsample/cmd/api</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hzSErh-ACkNIVzarIdVLQg.png" /><figcaption>Starting new node4</figcaption></figure><p>Then join it to the cluster and fetch the data from it:</p><pre>curl -X POST &#39;<a href="http://localhost:2221/raft/join&#39;">http://localhost:2221/raft/join&#39;</a> \<br>-H &#39;Content-Type: application/json&#39; \<br>-H &#39;Content-Type: text/plain&#39; \<br>--data-raw &#39;{<br> &quot;node_id&quot;:&quot;node4&quot;,<br> &quot;raft_address&quot;:&quot;localhost:1114&quot;<br>}&#39;</pre><pre>curl -X GET &#39;<a href="http://localhost:2224/store/problemnode&#39;">http://localhost:2224/store/problemnode</a></pre><p>You will see that our previously stored data using key problemnode with value hello world can be fetched from this node4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wKdMxMoEeWse_lzbaUk1cQ.png" /><figcaption>Join node4 to node1 and fetch data with key `problemnode` that already stored before from node4.</figcaption></figure><p><strong><em>Done is done!</em></strong></p><h3>Conclusion</h3><p>Creating Distributed System now easier than before because there are many libary providing Raft Consensus such as <a href="https://github.com/hashicorp/raft">hashicorp/raft</a> or <a href="https://github.com/lni/dragonboat">dragonboat</a>. Although Raft is not the only one consensus that we can use, but it is the <a href="https://raft.github.io/">alternative consensus that easier to understand compared to Paxos.</a> As you already follow this post, you will feel that implementing Raft (using hashicorp/raft) is as easy as you only to implement raft.FSM using your own logic. Raft consistently replicate the data to the newly joined node, so creating the distributed system on top of this consensus generally safe.</p><p><em>28–30 May 2020<br>Yogyakarta, Indonesia</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0884eef2e28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand Your System Behavior By Implement Distributed Tracing Using Jaeger]]></title>
            <link>https://yusufs.medium.com/understand-your-system-behavior-by-implement-distributed-tracing-using-jaeger-9041df1104ec?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/9041df1104ec</guid>
            <category><![CDATA[distributed-tracing]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[tracing]]></category>
            <category><![CDATA[golang-tutorial]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Wed, 12 Dec 2018 07:56:35 GMT</pubDate>
            <atom:updated>2018-12-12T07:56:35.833Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been more than 3 years in programming and building many applications from simple website to an API that needs to scale. Whether it hobby project or work related job, since choosing title “back-end developer” as the first career, I’m aware that building a good back-end is not easy as what I’ve imagine. One challenges that comes into my work is how could I well-knowing about the system that I face every day?</p><h4>Believe me, you will never know your Software well!</h4><p>Have you think that you are deeply known of your software that you’ve created like you think that it’s the part of yours? Until it came into moment when your software scream at you at the night, waking you up and you didn’t know what it exactly doing. You even don’t know what it want.</p><p>To give you some clear picture, what I mean here is, you may think that all the lines of code that you write is well. But, you forget that the software itself is not just what you write, you may use reusable code created by other developer, or one of your function calls another service made by other department. When you test it in development or staging mode, you will find it works well. Yet, when it comes to production, the things may gone wrong and you lost your mind what caused that problem. Whether it your function slow down the request, slow database query, or the other service that you call is slow?</p><p>If you only call one service, you will easy to notice by checking to that service. How about if you call multiple service? Or, when it comes to monolith service, how do you know that the bottleneck is your database, not your logic or unreachable Redis cache that you forget to handle? This is why you need to trace every single process in your software. In terms of back-end service which handle the HTTP API request, you <strong>**MUST** </strong>trace it start from client request received in the server until it writes the response back to client.</p><h4>Why do you need distributed tracing and what is it?</h4><p>In terms of micro-service architecture, one service responsible to one <a href="https://microservices.io/patterns/decomposition/decompose-by-business-capability.html">business capability</a> or <a href="https://microservices.io/patterns/microservices.html">other strategies that you can read here</a>. This leads to the need of tracing multi-separated service which much harder than tracing monolith architecture. In monolith architecture the whole process life-cycle is on one project, but in micro-service architecture to handle one request it needs to call n other services.</p><p>You may thinks that it is enough to create separate log service that records all request log for each services. But, when your API hits by hundreds request per second, how do you know that one request log in service A corresponds to log in service B and so on?</p><blockquote>… distributed tracing helps you, as a developer, to know deeply about your software behavior.</blockquote><p>By this far, I should expect that you got the idea why distributed tracing is really needed. I copied the statement from <a href="https://searchitoperations.techtarget.com/definition/distributed-tracing">here</a>: “ Distributed tracing helps pinpoint where failures occur and what causes suboptimal performance.” From there, it clear stated that distributed tracing helps you, as a developer, to know deeply about your software behavior.</p><h4>How tracing works?</h4><p>To easy to follow I will made some analogous example. Supposed you are Dora the Explorer and you given a task to get to sand castle from your current position, tree house. To get there you must face some obstacle that takes time. Here’s your map to get to sand castle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*eoQMngpSCKFeez3P.jpg" /><figcaption>Dora the Explorer map template. Source: <a href="https://commixus.com/dora-the-explorer-map-template-58475/">https://commixus.com/dora-the-explorer-map-template-58475/</a> (accessed at 2 Dec 2018.)</figcaption></figure><p>From above picture you can see that to reach sand castle from tree house, you must pass through the forest, crossing over the lake, enter the pyramid and voila you arrived to sand castle!</p><p>But, for every obstacle you met, it takes time to reach the final destination. You want to know how much time taken between the tree house and the sand castle. Therefore, you may record the current time before you depart and once after you arrived.</p><p>Now, I want you, as Dora, to report to me which obstacle that takes time you the most: how much time it takes, what causing you takes the long time and every detail that you may find it good to report. So, you may record the start time and end time of each obstacle, note some additional information when you find it useful to report, etc. Here’s the visual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1GE1wqOO7uxYYkMnX4aVQ.png" /><figcaption>Breakdown process to get from tree house to sand castle</figcaption></figure><p>If you already got the idea, there you go! Now, you should already knowing about the tracing works. Each obstacle is like a function, then the way between them is a function calls, that may be ignored to consider since it must be fast.</p><h4>Let’s coding!</h4><p>Now, to be more techie, we can bring that Dora journey into code and trace what she do in every obstacle. First, we need to create 4 function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*5Fr82tuO369FZCsC8G6sqg.png" /></figure><ol><li>startTheJourneyFromTreeHouse where the Dora prepare some food for her travel’s stock</li><li>passTheForest where Dora walk with Boot, her monkey, through the forest.</li><li>crossTheLake where the Dora cross the lake. We add some condition here, if the day is rainy, then the process will be slower because Dora doesn’t want to be drowned by her boat since the wind blows fast. This condition also to show you that the tracing can help you to pinpoint where the slowing process occurred.</li><li>enterThePyramid where Dora enter to the pyramid and surpass the dark room inside the pyramid.</li></ol><p>As you can see, there is always context parameter in first argument of each function. And from that context, we create a span using opentracing.StartSpanFromContext. Each span represents function process and therefore must be initiated in the first of function process and finish using deferred function. Context carry the information of tracing, so when you call, for example function passTheForest , the context will brings the information from previous function call which is startTheJourneyFromTreeHouse .</p><p>The function call will done <a href="https://github.com/yusufsyaifudin/go-opentracing-example/blob/91d3abba58/main.go#L141-L144">like this</a>:</p><pre>serverSpan := eCtx.Get(&quot;serverSpan&quot;).(opentracing.Span)<br>ctx := opentracing.ContextWithSpan(<br>   context.Background(),<br>   serverSpan,<br>)<br><strong>defer </strong>ctx.Done()</pre><pre>...</pre><pre>out, _ = startTheJourneyFromTreeHouse(ctx, in)<br>out, _ = passTheForest(ctx, out)<br>out, _ = crossTheLake(ctx, out, isRainyDay)<br>out, _ = enterThePyramid(ctx, out)</pre><p>Variable serverSpan is a value that already set in traceMiddleware . It contains the parent span that wrap all function call span:</p><pre>spanCtx, _ := opentracing.GlobalTracer().Extract(<br>   opentracing.<em>HTTPHeaders</em>,<br>   opentracing.HTTPHeadersCarrier(req.Header),<br>)</pre><pre>serverSpan := opentracing.StartSpan(<br>   c.Request().URL.Path,<br>   ext.RPCServerOption(spanCtx),<br>)<br><br>c.Set(&quot;serverSpan&quot;, serverSpan)</pre><p>So, each time when you call function, it always carry the information from previous function call. And then you will get the stack trace (which visualized using span) that look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RnVjN6ayQO94PMloyIoCxw.png" /><figcaption>Tracing visualization of Dora the explorer API</figcaption></figure><p>Since in every span we record the info such as a message, we can see that by expanding them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N3Pw4PP5PCj0gpjGfgFgJw.png" /><figcaption>Expanding all the process information.</figcaption></figure><p>Now, you notice that the longest process is the function crossTheLake with parameter is_rainy_day true. To ensure that this is not an anomaly, you can try to look for another trace in same end point. Finally, using this clue, you can try to optimize that function without guessing like a seer, because you have the data here!</p><h4>Final words</h4><p>You already know that knowing the exact problem in back-end can be tricky and painful when the traffic is high. But, the tracing system can help you to minimize your headache to trace which process that causing bottleneck of the system.</p><p>You may want to implement it soon once you see this article, but you may wondering how to ensure that your current code meet the requirement to implement this tracing. Tracing works by pipe-lining information using context and send it’s information using UDP to Jaeger. So, if your current Go code doesn’t have context, you must add it before you can trace your application.</p><p>Note:</p><ol><li>The working example of above snippet can be found here: <a href="https://github.com/yusufsyaifudin/go-opentracing-example">https://github.com/yusufsyaifudin/go-opentracing-example</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9041df1104ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with Kafka in Golang]]></title>
            <link>https://yusufs.medium.com/getting-started-with-kafka-in-golang-14ccab5fa26?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/14ccab5fa26</guid>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[golang-tutorial]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Wed, 15 Aug 2018 08:55:43 GMT</pubDate>
            <atom:updated>2018-08-15T08:55:43.239Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*6SQiJ4tinE0p4sjoBexxuA.png" /><figcaption>Go and Apache Kafka official logo</figcaption></figure><p>This is my experience in these past two weeks while I get my hand dirty in Apache Kafka. Even I was introduced with Kafka by my CTO several months ago, but I still have some problems about how to produce and consume a data to Kafka and just have a spare time to take a look at it again. The harder part that I’ve encountered was when I try to setup Kafka cluster running using docker. While there are some open-source docker images out there, but they just give an example to running Kafka in single node.</p><p>In this post I will tell you briefly about what is Kafka, how to setup it using Docker, and create simple program which produce and consume message from Kafka.</p><p>The behind story of this post is about “how to make an API faster to save data, hence make the lower latency and more clients can be served.” Also, this post is intended to be used as an internal engineering show at <a href="https://www.qiscus.com/">Qiscus</a> (yes, we at Qiscus periodically sets the internal sharing to what we’ve learned so far). So, let’s start with what is Apache Kafka itself.</p><h3>What Is Apache Kafka?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*iUWxneAQ_kozzLPkFsrakw.png" /><figcaption>Kafka Architecture. Source: <a href="http://kth.diva-portal.org/smash/get/diva2:813137/FULLTEXT01.pdf">http://kth.diva-portal.org/smash/get/diva2:813137/FULLTEXT01.pdf</a></figcaption></figure><p>Apache Kafka is an open-source stream processing software platform which started out at Linkedin. It’s written in Scala and Java. Kafka is based on commit log, which means Kafka stores a log of records and it will keep a track of what’s happening. This <a href="https://stackoverflow.com/a/45141245/5489910">commit log is similar with common RDBMS uses</a>.</p><p>To make it easier to imagine, it’s more like a queue where you always append a new data into the tail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*fsjNrvgiqvxmjhuYK6-wLA.png" /><figcaption>How Kafka stores a log, image source: <a href="https://www.confluent.io/blog/okay-store-data-apache-kafka/">https://www.confluent.io/blog/okay-store-data-apache-kafka/</a></figcaption></figure><p>While it seems simple, but in Kafka it can maintain the records into several partitions with same <em>topic</em>. Based on Kafka documentation’s, a topic is a category or feed name to which records are published. So, rather than just write into one queue like the image above, Kafka can writes into several queue with same topic name.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*GoRlq7O8qMNui6tvnq30cg.png" /><figcaption>For each topic, the Kafka cluster maintains a partitioned log that looks like this. Source: <a href="https://kafka.apache.org/documentation/#design">https://kafka.apache.org/documentation/#design</a></figcaption></figure><p>Imagine a topic is a name of a hotel. And imagine that the partition topic is like that hotel have 3 lift elevators with same direction, let’s say: up direction. In a busy day, when a guest of a hotel want to go to upstairs, they will choose one of three elevators which have a few passengers on it. With same analogical logic, when a data arrives, we can tell Kafka to write into specific partition.</p><p>It’s all about how Kafka handle an incoming data. Now, let’s see how we can read or consume the data from Kafka.</p><p>In Kafka you can consume data from specific partition of a topic, or you can consume it from all partition. Interestingly, you can subscribe the data using several clients/workers and make each of it retrieve different data from different partition using consumer group. So, what is a consumer group?</p><p>Consumer group is like a label of a group of consumer. For each consumer under the same label, you will consume different messages. Hmm okay, in simpler way, it’s like when you are at school, your teacher wants you to make a group of 3 persons. Then, your teacher will give each group a label of name, for example group 1 given a name “Tiger”, while the other one “Apple”, and so on. From now on, you and your other 2 friends are recognized as one entity rather than 3 entity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*J-0xbraSo0fbyrrPCXedlg.png" /><figcaption>Kafka consumer group. Source: <a href="https://kafka.apache.org/documentation/#design">https://kafka.apache.org/documentation/#design</a></figcaption></figure><p>In Kafka, as the consumer already grouped into one label, it can consume different message, so the job of each person in the same group will be not redundant. Let’s say, I joined into group “Hore” with my two friends. There, I and my friends, will not snatch the same things to be processed. For example, if our group given assignment to sweep the classroom, we may split our job where I sweep the left side, one of my friend is in the middle, and another one is in the right side of the class. Just like the pupils in Japan while they wiped the classroom’s floor together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*bk6rAukUe3Rxi8wL7qjcKQ.jpeg" /><figcaption>Students wipe the floor. Source: <a href="http://lucky-japan.blogspot.com/2014/10/japanese-students-clean-classrooms-on.html">http://lucky-japan.blogspot.com/2014/10/japanese-students-clean-classrooms-on.html</a></figcaption></figure><p>You can read more about the design behind of Kafka here:</p><p><a href="https://kafka.apache.org/documentation/#design">Apache Kafka</a></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FElilYxUOjOQ%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DElilYxUOjOQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FElilYxUOjOQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/87145226d62427e0ca5e51b070e36875/href">https://medium.com/media/87145226d62427e0ca5e51b070e36875/href</a></iframe><p>Now, you have the idea about how the Kafka’s topic work. Apart from it’s internal design structure, now I will tell you about how to use Kafka’s publish (produce) and subscribe (consume) feature to create a worker.</p><p>To easy to follow, we can use case study where we need a system to retrieve a message through REST API then consume it and process it in workers. It can be sending email, push notification, save to database, or other time-taking process.</p><h4>Setup Kafka With Docker</h4><p>I will use Docker and Docker Compose to setup Kafka cluster in my local environment. Here’s my docker version that I use while writing this post:</p><pre>Client:<br> Version:           18.06.0-ce<br> API version:       1.38<br> Go version:        go1.10.3<br> Git commit:        0ffa825<br> Built:             Wed Jul 18 19:11:02 2018<br> OS/Arch:           linux/amd64<br> Experimental:      false</pre><pre>Server:<br> Engine:<br>  Version:          18.06.0-ce<br>  API version:      1.38 (minimum version 1.12)<br>  Go version:       go1.10.3<br>  Git commit:       0ffa825<br>  Built:            Wed Jul 18 19:09:05 2018<br>  OS/Arch:          linux/amd64<br>  Experimental:     false</pre><p>and my docker-compose version</p><pre>docker-compose version 1.21.2, build a133471<br>docker-py version: 3.3.0<br>CPython version: 3.6.5<br>OpenSSL version: OpenSSL 1.0.1t  3 May 2016</pre><p>You may have newer version than me, but I think it’s okay since it can run docker compose yml file with version 2.</p><p>In this tutorial, I use docker image from Confluent: <a href="https://github.com/confluentinc/cp-docker-images">https://github.com/confluentinc/cp-docker-images</a>.</p><p>Long story short, previously, I use <a href="https://github.com/wurstmeister">wurstmeister</a>/<a href="https://github.com/wurstmeister/zookeeper-docker">zookeeper-docker</a> and <a href="https://github.com/wurstmeister/kafka-docker">wurstmeister/kafka-docker</a>, but the Zookeeper is not support clustering mode, so, I change it to <a href="https://github.com/31z4/zookeeper-docker">31z4/zookeeper-docker</a>. But, after trying it, I have a problem to publish message to some partition. Specifically, when I create a topic called “foo” with 4 partitions and 2 replication factors, I only got success publishing message into partition 1 and 3, while publishing message into partition 0 and 2 keeps return error like this:</p><blockquote>Tried to send a message to a replica that is not the leader for some partition. Your metadata is out of date.</blockquote><p>Maybe it’s not the fault of the docker image, maybe it just my wrong configuration so that the Kafka doesn’t run as expected. Here’s my docker-compose.yml using <a href="https://github.com/wurstmeister/kafka-docker">wurstmeister/kafka-docker</a> and <a href="https://github.com/31z4/zookeeper-docker">31z4/zookeeper-docker</a> that have a problem (maybe you can check it and let me know if I have configured it wrong):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/335e2e82f45eabd6a4751f17c6ac4095/href">https://medium.com/media/335e2e82f45eabd6a4751f17c6ac4095/href</a></iframe><p>Luckily, I’ve found<a href="https://gist.github.com/mhowlett/14f70af1a5b44fba80c9d1857a44bb98"> gist</a> by <a href="https://gist.github.com/mhowlett">Matt Howlett</a> to run Kafka and Zookeeper in cluster mode using Confluent Platform. Here’s is the docker compose file that I use to test Kafka in this post:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20afc76b3e09509dac52ad6310b74f9e/href">https://medium.com/media/20afc76b3e09509dac52ad6310b74f9e/href</a></iframe><p>Now, using that file, you can run it with MY_IP=your-ip docker-compose up. You can find your IP using ifconfig -a in your terminal.</p><p>After all service is up, you can create the topic, for example “foo” with this command:</p><pre>docker run --net=host --rm confluentinc/cp-kafka:5.0.0 kafka-topics --create --topic foo --partitions 4 --replication-factor 2 --if-not-exists --zookeeper localhost:32181</pre><p>It will create the topic “foo” with 4 partitions and replication factors = 2. If success, the above command will return something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UuMkWKNdwMkCi9jJqtuU4Q.png" /><figcaption>The left side is the docker logs while running Kafka and Zookeeper. The right side is the command to create new topic in Kafka.</figcaption></figure><h4>Test using kafkacat</h4><p>Now, you successfully running the Kafka and Zookeeper in cluster mode. You might wonder if it really functional before you want to write any code. For that matter, you can use command line tool named “kafkacat” which you can install using your beloved package manager like brew or apt.</p><p><a href="https://github.com/edenhill/kafkacat">edenhill/kafkacat</a></p><p>First, open new terminal and type:</p><pre>kafkacat -C -b localhost:19092,localhost:29092,localhost:39092 -t foo -p 0</pre><p>It will listen to topic “foo” in partition 0 (Kafka start the partition index from 0).</p><p>Then, from the other terminal you can publish a message specific into partition 0 using this command:</p><pre>echo &#39;publish to partition 0&#39; | kafkacat -P -b localhost:19092,localhost:29092,localhost:39092 -t foo -p 0</pre><p>If success, the first command will retrieve “publish to partition 0” message which sent by second command. You can do it respectively for partition 1, 2, and 3. You must ensure that all partition can receive the message as well as the consumer can subscribe it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CkrrhMMMzNALQigSm1XfgA.png" /><figcaption>Testing all partition can read and write the message using kafkacat.</figcaption></figure><p>As you can see in the above figure that when I sent a message into specific partition, it must be retrieved only by the destination partition.</p><h4>Writing Golang-based REST API</h4><p>There you have it! A working Kafka cluster and the tool to ensure that it was running well. Now, you can start write the code to push a message into Kafka.</p><p>To connect into Kafka, I use <a href="https://github.com/segmentio/kafka-go">segmentio/kafka-go</a> since <a href="https://github.com/Shopify/sarama">Shopify/sarama</a> is not support consumer group. Also, even when I use <a href="https://github.com/bsm/sarama-cluster">bsm/sarama-cluster</a> it stills have a problem when subscribing into Kafka version 2 as posted in this issue <a href="https://github.com/Shopify/sarama/issues/1144#issuecomment-412066477">https://github.com/Shopify/sarama/issues/1144#issuecomment-412066477</a>. So, finally I will use the <a href="https://github.com/segmentio/kafka-go">segmentio/kafka-go</a> library both for consumer and producer when writing this post after two weeks giving a try to both <a href="https://github.com/Shopify/sarama">Shopify/sarama</a> and <a href="https://github.com/bsm/sarama-cluster">bsm/sarama-cluster</a>. (Yes, you read it right, I just move to <a href="https://github.com/segmentio/kafka-go">segmentio/kafka-go</a> today when I write this post 😅 ).</p><p><a href="https://github.com/segmentio/kafka-go">GitHub - segmentio/kafka-go: Kafka library in Go</a></p><p>To serve and handle the request from HTTP, I will use <a href="https://github.com/gin-gonic/gin/">gin-gonic/gin</a> as a router. With this, you can create an end-point as easy as this snippet:</p><pre>gin.SetMode(gin.ReleaseMode)                                                router := gin.New()<br>router.POST(&quot;/api/v1/data&quot;, postDataToKafka)</pre><pre>return router.Run(listenAddr)</pre><p>The main part of publishing message into Kafka is you must create the connection into Kafka then you can produce a message to Kafka using established connection. You can find this code in this link <a href="https://github.com/yusufsyaifudin/go-kafka-example/blob/accadecfca65c956bf03a6199ff1f3944bc6d7dc/dep/kafka/config.go">https://github.com/yusufsyaifudin/go-kafka-example/blob/accadecfca65c956bf03a6199ff1f3944bc6d7dc/dep/kafka/config.go</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b0c7b07a788fab530e3ade225e9a623/href">https://medium.com/media/6b0c7b07a788fab530e3ade225e9a623/href</a></iframe><p>Once the connection established and been set in writer variable, it can be use to push the message into Kafka with this code:</p><pre>package kafka</pre><pre>import (<br>  &quot;context&quot;<br>  &quot;time&quot;<br>  &quot;github.com/segmentio/kafka-go&quot;<br>)</pre><pre>func Push(parent context.Context, key, value []byte) (err error) {<br>  message := kafka.Message{<br>    Key:   key,<br>    Value: value,<br>    Time:  time.Now(),<br>  }</pre><pre>  return writer.WriteMessages(parent, message)<br>}</pre><p>Then you call it using:</p><pre>err := kafka.Push(context.Background(), nil, []byte(&quot;your message content&quot;))</pre><p>Since this post will contains full of code if I explain it line by line, you can take a look the full code here:</p><p><a href="https://github.com/yusufsyaifudin/go-kafka-example/tree/accadecfca65c956bf03a6199ff1f3944bc6d7dc">yusufsyaifudin/go-kafka-example</a></p><p>However, I assume you have got the idea of publishing message into Kafka. It’s just like common API handling which accept the request, validate it and then save the data into database, but now it is in Kafka. Later, rather than finish the data life-cycle in Kafka, we need to create a worker to queue it to write into database following these reasons:</p><ul><li>it is hard to query right from Kafka (even now you can do it using KSQL)</li><li>if you save the data in Kafka forever, once you create a new consumer-group, it will consume all data from beginning, which is too large (even it is okay to store data in Apache Kafka for forever)</li></ul><ul><li><a href="https://www.confluent.io/product/ksql/">KSQL: Streaming SQL for Apache Kafka | Confluent</a></li><li><a href="https://www.confluent.io/blog/okay-store-data-apache-kafka/">It&#39;s Okay To Store Data In Kafka</a></li></ul><h4>Writing the Consumer as a Worker</h4><p>Writing the consumer is much simpler. The main idea is just retrieve data from Kafka then process it. For each consumer group, we can do different works.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/204461995098508ac8a1f0424db980b3/href">https://medium.com/media/204461995098508ac8a1f0424db980b3/href</a></iframe><p>If we look the code above, we can easily understand that the idea is straightforward:</p><ul><li>We read configuration such as Kafka brokers URL, topic that this worker should listen to, consumer group ID and client ID from environment variable or program argument.</li><li>Then we make the connection to Kafka to subscribe particular topic in line 42–52. In line 52, you may notice that there is reader.Close() in deferred mode. It will tells the application to close the connection to Kafka if the program exit.</li><li>In line 55–73 we wrap the ReadMessage function in forever loop. This will make program listening to incoming message.</li><li>In line 63, we check the compression codec of incoming message. Since it publisher we use Snappy compression, we must decode it using Snappy too.</li><li>Line 72 is the final stage where we can really read the message content. Starting from this point, you can now process the data. For example, marshaling it into Go struct, write to persistent database, sent data to Elasticsearch to indexing or anything.</li></ul><h4>So, Why Using Kafka?</h4><p>Maybe we still have a question, why do we using Kafka instead do any processing task right in the controller. Can we just write into database, cache and index service right after we receive the data? Or another question is, why don’t we use Redis pub/sub feature? Isn’t it fast enough since it processed in RAM?</p><p>I once think the same about it, and it depends on your use case. But, let’s consider this:</p><ul><li>If you do any processing in the controller, it will cause the large latency to your client since for each request your client must waiting all the process is really done. You might think it can be tricked by using go routine or any spawning new thread mechanism. But, if this process is takes longer time, you will end-up having a system that always busy to handle your background thread process. Maybe it’s either kill your CPU or RAM. As you know, writing into database is already takes time, adding with time to writing to Elasticsearch, Redis, etc.</li><li>If you’re considering Redis pub/sub, maybe it’s okay if you only need to get notification and tolerate the message lost. It because Redis pub/sub mechanism is not queued, and even less persisted. <a href="https://stackoverflow.com/a/18081263/5489910">They are only buffered in the socket buffers, and immediately sent to the subscribers in the same event loop iteration as the publication. If a subscriber fails to read a message, this message is lost for the subscriber.</a> In Kafka, the message will retain for several days or months, depends on your retention log configuration. So, when a consumer (worker) is fail processing a particular data, it can be read from the beginning time as long as the data still on retention window time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nzuqFUIgFdKHjbS7PwUR1w.png" /><figcaption>Kafka’s official website stated: “Kafka will perform the same whether you have 50 KB or 50 TB of persistent data on the server.” Source: <a href="https://kafka.apache.org/intro">https://kafka.apache.org/intro</a></figcaption></figure><ul><li>Kafka’s performance is guaranteed to be the same under the small data sets or when it grow into terabyte of data. It because Kafka avoid trees (BTress or LSM Tree) on it’s internal structure. It uses queue mechanism, where data is appended to the system. To know more deeply why Kafka is so fast, here’s some link that you may be interested:</li></ul><ul><li><a href="https://medium.freecodecamp.org/what-makes-apache-kafka-so-fast-a8d4f94ab145">Here’s what makes Apache Kafka so fast</a></li><li><a href="http://searene.me/2017/07/09/Why-is-Kafka-so-fast/">Why is Kafka so fast</a></li><li><a href="https://engineering.linkedin.com/kafka/running-kafka-scale">Running Kafka At Scale</a></li><li><a href="https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines">Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)</a></li></ul><p>Yogyakarta, 11 August 2018. Written at the UGM library and continued at Jogja City Library.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14ccab5fa26" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My First Try On TypeScript Programming]]></title>
            <link>https://yusufs.medium.com/my-first-try-on-typescript-programming-a5e0a4aa1d05?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5e0a4aa1d05</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[es5]]></category>
            <category><![CDATA[es6]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Thu, 26 Jan 2017 14:38:13 GMT</pubDate>
            <atom:updated>2021-01-27T04:16:07.729Z</atom:updated>
            <content:encoded><![CDATA[<p>I have tried coding in JavaScript using different standard, start from ES5 to ES6. My impression was gained when I tried some simple code using ES6. Why? Because in ES6 I can declare class, even I know that when it translated to ES5 it just a variable which return a function.</p><p>For example:</p><pre>class MyClass {<br>  <br>  greeting(name) {<br>    return &quot;Hello &quot; + name;<br>  }<br>  <br>}</pre><p>Which later will be translated to ES5 as below:</p><pre>&quot;use strict&quot;;</pre><pre>function _classCallCheck(instance, Constructor) { if (!(instance instanceof Constructor)) { throw new TypeError(&quot;Cannot call a class as a function&quot;); } }</pre><pre>var MyClass = function () {<br>  function MyClass() {<br>    _classCallCheck(this, MyClass);<br>  }</pre><pre>  MyClass.prototype.greeting = function greeting(name) {<br>    return &quot;Hello &quot; + name;<br>  };</pre><pre>  return MyClass;<br>}();</pre><h4>How it can be translated?</h4><p>For simple and non-dependent code (single file only), you can use <a href="https://babeljs.io/repl/">BabelJS REPL</a>. But, for real application, I suggest you to use Grunt or Gulp to make converter task from ES6 standard to ES5 using BabelJS.</p><p><a href="https://babeljs.io/repl/">Babel · The compiler for next generation JavaScript</a></p><p>Here is the gulpfile.js example to convert from ES6 to ES5 (ES2015):</p><pre>const gulp = require(&#39;gulp&#39;);<br>const babel = require(&#39;gulp-babel&#39;);</pre><pre>gulp.task(&#39;convert&#39;, () =&gt;<br>  gulp.src(&#39;src/**/*.js&#39;)<br>  .pipe(babel({<br>    presets: [&#39;es2015&#39;]<br>  }))<br>  .pipe(gulp.dest(&#39;dist&#39;))<br>);</pre><pre>gulp.task(&#39;watch&#39;, [&#39;convert&#39;], () =&gt; {<br>  gulp.watch(&#39;src/**/*.js&#39;, [&#39;convert&#39;]);<br>});</pre><pre>gulp.task(&#39;default&#39;, [&#39;watch&#39;])</pre><p>Using above code, you can now write your JS code using ES6 standard. I will not show you the rest of other code, you can learn by your self and see ES6 features from this website:</p><p><a href="http://es6-features.org/">ECMAScript 6: New Features: Overview and Comparison</a></p><h4>How About TypeScript?</h4><p><a href="https://www.typescriptlang.org/">JavaScript With Syntax For Types.</a></p><p>I didn’t dive into <a href="https://www.typescriptlang.org/">TypeScript</a> deeply yet, at first try I don’t see any differences between JS using ES6 standard, except TypeScript support static typing binding. Furthermore, I found TypeScript have controlling access to member class. This prevent you to access wrong variable value before it has assigned.</p><p>For instance, you have following code in ES6 standard (and later compiled using BabelJS):</p><pre>// Multiply.js<br>class Multiply {<br>  <br>  constructor(a, b) {<br>    this.a = a;<br>    this.b = b;<br>  }<br>  <br>  result() {<br>    return this.a * this.b;<br>  }<br>  <br>}</pre><pre>let multiply = new Multiply(2, 3);<br>console.log(multiply.a);<br>console.log(multiply.b);<br>console.log(multiply.result());</pre><p>and below code written using TypeScript:</p><pre>// Multiply.ts<br>class Multiply {</pre><pre>  private a: number;<br>  private b: number;<br>  <br>  constructor(a: number, b: number) {<br>    this.a = a;<br>    this.b = b;<br>  }<br>  <br>  result() {<br>    return this.a * this.b;<br>  }<br>  <br>}</pre><pre>let multiply = new Multiply(2, 3);<br>console.log(multiply.a);<br>console.log(multiply.b);<br>console.log(multiply.result());</pre><p>You may noticed, that Multiply.js is just standard JS code and Multiply.ts is JS code + type checking + access level attribute. For the first one code, you can convert it into ES5 using BabelJS preset `es2015-loose`. For the second one, you can compile it using TypeScript with following configuration (tsconfig.json):</p><pre>{<br>    &quot;compilerOptions&quot;: {<br>        &quot;target&quot;: &quot;es5&quot;,<br>        &quot;module&quot;: &quot;commonjs&quot;,<br>        &quot;outDir&quot;: &quot;dist&quot;,<br>        &quot;moduleResolution&quot;: &quot;node&quot;,<br>        &quot;emitDecoratorMetadata&quot;: true,<br>        &quot;experimentalDecorators&quot;: true,<br>        &quot;sourceMap&quot;: true,<br>        &quot;alwaysStrict&quot;: true<br>    },<br>    &quot;exclude&quot;: [<br>        &quot;node_modules&quot;<br>    ]<br>}</pre><p>If you have expect that both code results same code after compiled into ES5, you’re right! <a href="https://www.diffchecker.com/IxCLJK7t">Here is the diffs</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ECw0k3-Uc2sZDxj21rw2qA.png" /><figcaption>Multiply.js and Multiply.ts after compiled into Multiply.js using ES5 preset. Left code is from Multiply.js and the right one is from Multipy.ts</figcaption></figure><p>There are no many differences between above two files, just class type-checking in BabelJS compile result.</p><h4><strong>Is that TypeScript static typing language?</strong></h4><p>The answer for this question is depends. If you see from the language (JavaScript) itself, it’s not a static typing languange inborn. But, “<a href="https://en.wikipedia.org/wiki/Source-to-source_compiler">transpiler</a>” like BabelJS or JavaScript superset like TypeScript, is just checking the type of variable and just return error notice, but don’t stop it from translates into JavaScript code.</p><p>However, if you see from TypeScript side, you may consider that TypeScript is a type-checking (okay we can call it “partial type-checking”), since it just check the type of your assigned variable parameter and doesn’t stop you to convert it as JavaScript ES5 code — it just complains you. This also occurred when you try to access private or protected member of a class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mOoeZWz1EMBBSSulm7Qyg.png" /><figcaption>TypeScript just give you a notice, but don’t stop your “transpiling” process.</figcaption></figure><h4>Final Words</h4><p>I think TypeScript have its own future. It redefine JavaScript syntax into clearer and nicer syntax — oh I almost forgot about mentioned it as neat language compared to JavaScript. TypeScript will allow JS developer to code in logical syntax, no morevar ClassName = function () {} each times they want to declare a class (although eventually it will be converted into it).</p><p><a href="https://github.com/Microsoft/TypeScript/wiki/Coding-guidelines">Microsoft/TypeScript</a></p><p>Finally, as long as JS developer follows <a href="https://github.com/Microsoft/TypeScript/wiki/Coding-guidelines">TypeScript coding guidelines</a>, I think it will make they keep away from painful feels when coding in JavaScript, such when they need to type checking or make an object class.</p><p>You can see the code I used for this post through this link <a href="https://github.com/yusufsyaifudin/babel-vs-typescript">https://github.com/yusufsyaifudin/babel-vs-typescript</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5e0a4aa1d05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Procrastination is not habit, it’s habituation]]></title>
            <link>https://yusufs.medium.com/procrastination-is-not-habit-its-habituation-6dc8afcfadea?source=rss-b9cc63eced88------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dc8afcfadea</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[self-esteem]]></category>
            <category><![CDATA[procrastination]]></category>
            <dc:creator><![CDATA[Yusuf Syaifudin]]></dc:creator>
            <pubDate>Fri, 06 Jan 2017 10:48:41 GMT</pubDate>
            <atom:updated>2021-01-27T04:15:31.675Z</atom:updated>
            <content:encoded><![CDATA[<p>This is my first post in medium since I registered one year ago. I write this because I’m triggered by my friend’s post at my Facebook news feed. He wrote some tutorial about Data Science, and another one wrote about Deep Learning. But that’s not only my reason. Another reason is I must learn to write English in correct grammar (so please correct my grammar if I’m write some mistake) and I must push myself to do it. (And if you are still asking another “why” question, then this post doesn’t answer all of your “kepo” behavior).</p><h4>Preamble</h4><p>The Doraemon’s opening song translation in Bahasa Indonesia said that “Aku ingin begini, aku ingin begitu. Ingin ini, ingin itu, banyak sekali. (I wanna be like this, I wanna be like that. Wanna this, wanna that, so much thing that I want).” Yeah you know, sometimes beginner always hopes to see the result as fast as they starts learning.</p><p>Doing something in a rush will not make you get result faster, but you’ll end up <a href="https://medium.com/the-mission/multitasking-is-killing-your-brain-79104e62e930#.l9t94rdl3">multitasking which not effective at all</a>, because <a href="https://aeon.co/essays/your-brain-does-not-process-information-and-it-is-not-a-computer">your brain is not a computer</a>. You may want to learn many things at once in short time, but believe me, you can’t.</p><blockquote><a href="https://www.youtube.com/watch?v=vGJTaP6anOU">Wise men say: “Only fool rush in.”</a></blockquote><p>You <strong>must focus to do something at one time.</strong> Don’t try to learn X while you are still focus at Y. But, will it murder your creativity and curiosity? Yeah, maybe.</p><h4>Focus is not about forgetting</h4><p>Focus doesn’t mean you forget the other things. It mean you must allocate time to fully focus into it. For example, when you want to study about matrix multiplication in mathematics, you must focus about how to multiply it. In the same time, you don’t need to think about <a href="http://stackoverflow.com/questions/11627279/running-java-helloworld">how to running your hello world program in Java</a>, because it’s not correlated. But, once you want to learn about coding in Java, you should switch your focus for a while about your problem in math.</p><p>It’s not easy, right? Yap, I agree. Even I still hard to follow this advice. So, what can we do? I think this behavior is related to how you manage your plan. A bad plan will return a bad result. Then, the second question which is the biggest problem of all human is: how are we fight to procrastination?</p><h4>Procrastination is not habit, it’s habituation</h4><p>I made this subtitle because I think all people don’t want to procrastinate their to-dos. But, since they are lazy or &lt;put all thousand reason here&gt;, they think that they will do it in the near future. Then when future comes, they just do a little bit part of hundred task’s steps. To make it worse, they said that “at least I have done that and made progress,” without looking back at their plan and re-check whether they called “progress” is far enough to be called “progress” or not. Later, it can be called as “justification”.</p><blockquote>How you can say it’s a “progress” if you are just done with counting one lamb, where you have to count 1000 lambs on the field?</blockquote><p>This recursive “justification” leads into some biased view, accusing that “procrastination is bad habit”. I say BIG NO for it. Procrastination isn’t habit, it’s habituation. So, instead of saying that “procrastination is bad habit” you must say that “procrastination is bad habituation”. Why? Because, source of procrastination is your justification that leads to habituation, then, even worse, you said it is a “habit”. Hey, you made it. So, please admit that it’s your made and fault.</p><blockquote>Procrastination isn’t habit, it’s habituation.</blockquote><h4>How we fight procrastination?</h4><p>I don’t know the exact answer for this question. Even I, still trying to fight it. But, recently I read <a href="http://www.bbc.com/news/business-38496775">BBC’s article</a> which mentioned some tips to fight it. In brief, it said that we must control our negative thought over our task. It means, you must avoid some judgement about our task, i.e: “Ouch, I think this task will make me bored.” or “It’s better to stroll around the city park to get good mood before doing this task (then you never touch your task again after coming home with reason that you are tired).”</p><p>Then, as I said before, you must plan your task. Push yourself to do it. Say to yourself, like “if I’m doing it right I will get … &lt;insert some achievement that you will get&gt;”. Try harder to push yourself to do it. Then, this good habituation will become habit. And, yap, you can say good habituation results to “habit”. To be clear, for example: when you can be morning person already because of the result of your workout, you can say “Wake up early in the morning is my habit.”</p><p>But, again, don’t try to call the result of bad habituation as “habit”. Just don’t!</p><h4>Changes log:</h4><p>2017–01–08<br>* Thanks to <a href="https://medium.com/@amirahff">Farah</a> for correcting my grammar.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dc8afcfadea" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>