<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Victor Gil on Medium]]></title>
        <description><![CDATA[Stories by Victor Gil on Medium]]></description>
        <link>https://medium.com/@victorgil_91367?source=rss-d464b56be56------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9YF8d43as01a-c-q.</url>
            <title>Stories by Victor Gil on Medium</title>
            <link>https://medium.com/@victorgil_91367?source=rss-d464b56be56------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:50:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@victorgil_91367/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kafka + Cadence + WebSockets + Angular: managing event-driven microservices state with Uber Cadence]]></title>
            <link>https://medium.com/@victorgil_91367/kafka-cadence-websockets-angular-managing-event-driven-microservices-state-with-uber-cadence-3363aae00be7?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/3363aae00be7</guid>
            <category><![CDATA[cadence]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Tue, 17 Mar 2020 17:42:40 GMT</pubDate>
            <atom:updated>2020-03-17T19:15:59.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*fd1lgLekrA-kBZQ_ExIw-A.png" /></figure><p>In the the previous post of the <strong><em>Event-driven microservices with Kafka</em></strong> series (see <a href="https://dev.to/victorgil/kafka-websockets-angular-event-driven-microservices-all-the-way-to-the-frontend-12aa">here</a>), I showed how to extend the asynchronous event-driven communication all the way from Kafka to the Web frontend passing through the Java backend.</p><p>In that proof of concept application which processes money transfers, one Kafka topic is used to receive incoming transfer messages and a second topic is used to store the account balances state (no need of a database) and also to broadcast changes (i.e., to push notifications from the backend to the frontend).</p><p>The first topic (called “ <em>transfers</em>”) and the second topic (called “ <em>account-balances</em>”) are connected through Kafka Streams.</p><h3>Uber Cadence</h3><p>In this post we are bringing <a href="https://cadenceworkflow.io/"><strong>Uber Cadence</strong></a> into the mix to manage the state of the application (i.e., to keep the balance of the accounts updated), thus, <strong>Cadence replaces Kafka Streams</strong>.</p><p>Cadence is an orchestration/workflow engine which unlike most of the other workflow engines out there (e.g., Zeebe, Camunda and many <a href="https://github.com/mengjin001/awesome-workflow-engines">others</a>), does not rely on <a href="http://www.bpmn.org/">BPMN</a> (nor on any other type of XML) to define a workflow.</p><p>It uses regular programming languages such as Go and Java instead, which is a huge advantage as eloquently explained by Cadence’s lead developer in <a href="https://medium.com/@mfateev/i-absolutely-agree-that-zeebe-is-the-only-cloud-native-and-scalable-solution-in-bpmn-space-9bfa549d635c">this comment</a>.</p><p>It is out of the scope of this post to provide a detailed description or an introduction to Cadence, you are encouraged to perform your own (even if superficial) research and also watch <a href="https://youtu.be/LKfZAFx56kI">the companion video for this post</a> but I will mention its main advantages:</p><ul><li>It makes your application easily scalable</li><li>It hugely simplifies the error handling</li><li>It comes with an admin Web UI and CLI</li></ul><p>In short, it takes care of most of the heavy lifting inherent to a distributed application.</p><h3>Enhanced PoC application</h3><p>Let’s go trough the data flow to get an overview of the application functionality.</p><p>A new service called <a href="https://github.com/VictorGil/cadence_transfers_recording_service">Cadence transfers recording service</a> listens for incoming transfer messages on a Kafka topic.</p><p>If the message represents the act of opening a new account, the service creates a new <a href="https://cadenceworkflow.io/docs/03_concepts/01_workflows">workflow</a> by using the corresponding Cadence Java client API <a href="https://github.com/VictorGil/cadence_transfers_recording_service/blob/master/src/main/java/net/devaction/cadence/transfersrecordingservice/processor/TransferProcessor.java#L89">method</a>.</p><p>The Cadence server will assign the execution of the new workflow to one of the previously started Cadence workers. One important detail to note is that the id of the workflow matches the account id, i.e., there is one workflow per account.</p><p>If the incoming Kafka message represents a money transfer which changes the balance of an existing account, the Cadence transfers recording service calls the <a href="https://github.com/VictorGil/cadence_transfers_recording_service/blob/master/src/main/java/net/devaction/cadence/transfersrecordingservice/processor/TransferProcessor.java#L70">required method</a> (which must be annotated with <a href="https://github.com/VictorGil/account_balance_workflow_api/blob/fc5766f7d93237b9d6973391a2c8548fa7352b72/src/main/java/net/devaction/cadence/accountbalanceworkflow/AccountBalanceWorkflow.java#L24">@SignalMethod</a>) on the workflow whose id matches the account id.</p><p>That triggers the remote execution of the code on one of the Cadence workers, which after updating the value of the account balance as required, will send a Kafka message to the “ <em>account-balances</em>” topic to announce the event.</p><p>Next, the “ <em>transfers websockets</em>” service will pull that message from the “ <em>account-balances</em>” Kafka topic and broadcast it to the interested Angular Web clients through websockets.</p><p>In addition to this asynchronous/push communication, an Angular client may query the “ <em>transfers websockets</em>” service synchronously to get the current balance or the history of an account (i.e., the list of transfers for that account).</p><p>The “ <em>transfers websockets</em>” service, in order to fullfill those requests, will query the Cadence server by calling the corresponding method which must be annotated with <a href="https://github.com/VictorGil/account_balance_workflow_api/blob/fc5766f7d93237b9d6973391a2c8548fa7352b72/src/main/java/net/devaction/cadence/accountbalanceworkflow/AccountBalanceWorkflow.java#L29">@QueryMethod</a>.</p><h3>Diagram</h3><p>This is a high-level diagram of the PoC application, I am sorry if it looks a little messy, graphic design is not my strongest skill but it should help you to get an overview of the components and their interactions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*fd1lgLekrA-kBZQ_ExIw-A.png" /></figure><h3>See it in action</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLKfZAFx56kI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLKfZAFx56kI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FLKfZAFx56kI%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c1870acaf8e2db368f38bf4e74281c6/href">https://medium.com/media/0c1870acaf8e2db368f38bf4e74281c6/href</a></iframe><h3>Source code repositories</h3><p><a href="https://github.com/VictorGil/cadence_transfers_recording_service">https://github.com/VictorGil/cadence_transfers_recording_service</a><br><a href="https://github.com/VictorGil/account_balance_workflow_api">https://github.com/VictorGil/account_balance_workflow_api</a><br><a href="https://github.com/VictorGil/transfers_websockets_service">https://github.com/VictorGil/transfers_websockets_service</a> <br><a href="https://github.com/VictorGil/transfers_frontend">https://github.com/VictorGil/transfers_frontend</a> <br><a href="https://github.com/VictorGil/transfers_recording_service">https://github.com/VictorGil/transfers_recording_service</a> <br><a href="https://github.com/VictorGil/transfers_api">https://github.com/VictorGil/transfers_api</a> <br><a href="https://github.com/VictorGil/kafka_util">https://github.com/VictorGil/kafka_util</a></p><p><em>Originally published at </em><a href="https://www.devaction.net/2020/03/cadence.html"><em>https://devaction.net</em></a><em> on March 17, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3363aae00be7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kafka + WebSockets + Angular: event-driven microservices all the way to the frontend]]></title>
            <link>https://medium.com/@victorgil_91367/https-medium-com-victorgil-91367-kafka-websockets-angular-event-driven-microservices-9cb8e66525f2?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/9cb8e66525f2</guid>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[websocket]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Sat, 09 Nov 2019 12:38:00 GMT</pubDate>
            <atom:updated>2019-11-19T07:17:30.606Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*rNJ1uGTDvVjluvKPOtyhUg.gif" /><figcaption>Realtime data updates in action</figcaption></figure><p>In the the initial post of the <strong><em>Event-driven microservices with Kafka</em></strong> series (see <a href="https://medium.com/@victorgil_91367/using-apache-kafka-to-implement-event-driven-microservices-810a26f1418f"><strong>here</strong></a>), I talked about the advantages of using event-driven communication and Kafka to implement stateful microservices instead of the <em>standard</em> stateless RESTful ones.<br>I also presented the architecture and the source code of a related proof of concept application.</p><p>In this post, I would like to show how to extend the asynchronous event-driven communication all the way from Kafka to the Web frontend passing through the Java backend.<br>Hence, in the first post of this series, we got rid of HTTP as the communication protocol among microservices in the backend, and now we are also replacing it (with WebSockets) as the communication protocol between the frontend and the backend.</p><h3>Ok, but why would you do that?</h3><p>Because it provides a better experience to the end user!.<br> Using WebSockets you can build <strong><em>legit</em></strong> real-time user interfaces, the updates are pushed immediately from the server to the client, no need for the user to click any button nor to refresh the browser.</p><h3>Wait, but we can do the same using good old HTTP, can’t we?</h3><p>Well… HTTP was not designed to provide real-time communication, but since people really wanted it, a number of workaround techniques and technologies have been create on top of it, such as:</p><ul><li>Long polling</li><li>Short polling</li><li>Server-Sent Events</li></ul><p>Those are little more than hacks, IMHO. <br> WebSockets, on the other hand, provide <strong>full-duplex bidirectional</strong> communication, which means that information can flow from the client to the server and also in the opposite direction simultaneously, it cannot get more <em>real-time</em> than that (that’s what I meant when I said ‘ <em>legit</em> real-time applications’ before).</p><figure><img alt="Image depicting HTTP vs WebSockets" src="https://cdn-images-1.medium.com/max/1024/1*P7Os0XpDjZHF4Wy8tf3lKA.png" /><figcaption><em>HTTP vs WebSockets (source: </em><a href="https://blogs.windows.com/windowsdeveloper/2016/03/14/when-to-use-a-http-call-instead-of-a-websocket-or-http-2-0)"><em>https://blogs.windows.com/windowsdeveloper/2016/03/14/when-to-use-a-http-call-instead-of-a-websocket-or-http-2-0)</em></a></figcaption></figure><h3>The example</h3><p>I added a new service to the same PoC application which I used in the previous post and also an Angular web frontend.</p><h4>Ok, but what does this new service do?</h4><p>It allows the frontend user to receive information about the current balance of any account of her/his choice as well as live updates when the balance changes.</p><p>Also, information about any transactions related to that currently selected account (i.e., any amount of money credited or debited) past or live, is displayed to the users.</p><h3>Architecture</h3><p>The new <a href="https://github.com/VictorGil/transfers_websockets_service"><strong><em>Transfers WebSockets</em></strong></a> service main components are:</p><ul><li>Two Kafka <a href="https://github.com/VictorGil/transfers_websockets_service/blob/1.0.0/src/main/java/net/devaction/kafka/consumer/TopicConsumerImpl.java"><strong>consumers</strong></a> (one for each topic) to retrieve messages from the Kafka cluster</li><li>Two <em>Kafka Streams</em> <a href="https://github.com/VictorGil/transfers_websockets_service/blob/1.0.0/src/main/java/net/devaction/kafka/transferswebsocketsservice/localstores/LocalStoresManagerImpl.java"><strong>local stores</strong></a><strong> </strong>to retrieve the latest data associated with a given key (id)</li><li>A <a href="https://github.com/VictorGil/transfers_websockets_service/blob/1.0.0/src/main/java/net/devaction/kafka/transferswebsocketsservice/transferscustomstore/TransfersStoreImpl.java"><strong>custom local store</strong></a><strong> </strong>implemented using a simple Map to store the list of transactions for a given account.</li><li>Two <em>dispatchers</em> which keep track of what WebSockets sessions (i.e., frontend clients) are registered to receive live <a href="https://github.com/VictorGil/transfers_websockets_service/blob/1.0.0/src/main/java/net/devaction/kafka/transferswebsocketsservice/processor/dispatcher/BalanceUpdatesDispatcherImpl.java"><strong>balance updates</strong></a> or live <a href="https://github.com/VictorGil/transfers_websockets_service/blob/1.0.0/src/main/java/net/devaction/kafka/transferswebsocketsservice/processor/dispatcher/TransferDataDispatcherImpl.java"><strong>transfers data</strong></a> related to which account and then dispatch the updates accordingly.</li></ul><p>And these are the main components of the <a href="https://github.com/VictorGil/transfers_frontend"><strong>Angular frontend</strong></a>:</p><ul><li>A <a href="https://github.com/VictorGil/transfers_frontend/blob/1.0.0/src/app/websockets.service.ts"><strong>WebSockets service</strong></a> which handles the connection lifecycle and dispatches the received messages to the corresponding <strong>RxJS Subject</strong>. Please note that different types of JSON messages (requests, responses and updates) flow through the only WebSockets connection.</li><li>A <a href="https://github.com/VictorGil/transfers_frontend/blob/1.0.0/src/app/account-balance-form/account-balance-form.component.ts"><strong>form</strong></a> which passes the user input (i.e., the id of the chosen account) to the WebSockets service so it can be sent to the remote server.</li><li>An Angular to display the account balance which gets asynchronously notified of balance changes (updates) through an <strong>RxJS Observer</strong> which is subscribed to the corresponding <strong>RxJS Subject</strong> in the previously mentioned WebSockets service, and then re-renders the value accordingly.</li><li>Another Angular <a href="https://github.com/VictorGil/transfers_frontend/blob/1.0.0/src/app/display-transfers/display-transfers.component.ts"><strong>component</strong></a> which similarly as the previous one, gets asynchronously notified of any new transaction related to the selected account and adds it to the list of displayed transaction data.</li></ul><p>This is a diagram depicting the PoC application components and several frontend instances:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/0*E8mP-iwmC16gyr5-.png" /></figure><h3>See it in action</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/0*pq2T2QpPDb5KJoBw.gif" /></figure><p>Check out this <a href="https://youtu.be/6b07L7floKE"><strong>Youtube video</strong></a><strong> </strong>for a demo, where a <a href="https://github.com/VictorGil/transfers_recording_service/blob/1.1.0/src/test/java/net/devaction/kafka/transfersrecordingservice/randomtransfersproducer/RandomTransferProducer.java">test process</a> which continuously generates Kafka messages representing random money transfers is used to feed the application.<br> And on the Web UI the user can see how the transfers pop up as they happen and the balance of the related account changes accordingly, all in real time.</p><h3>Final Thoughts</h3><p>HTTP communication protocol has had a huge success, it made possible the WWW as we know it today.<br> It has been so successful that it has spread to the backend (RESTful services and open APIs) where it has been the king for more than a decade already.<br> However, technology evolves, paradigms come and go and for some use cases other tools and designs may be a better choice.<br> I do not know how software applications will look like in the future, all I can do is guess.<br><em>Event-driven microservices</em> look very powerful and implementing them with Kafka and adding WebSockets to the mix seems promising.<br> In fact, some companies are able to see beyond REST APIs and are starting to offer WebSockets APIs to their clients for faster streaming access to data (<a href="https://blog.kraken.com/post/2892/kraken-launches-websockets-private-api/"><strong>example</strong></a>).<br> And there are other alternative ways to implement event-driven microservices, for example, recently I learnt about <a href="https://github.com/uber/cadence"><strong>Uber Cadence</strong></a> but I’ll save that for a future post 😃</p><p><em>Originally published at </em><a href="https://www.devaction.net/2019/11/kafka-websockets-angular.html"><em>http://www.devaction.net</em></a><em> on November 9, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9cb8e66525f2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Apache Kafka to implement event-driven microservices]]></title>
            <link>https://medium.com/coinmonks/using-apache-kafka-to-implement-event-driven-microservices-810a26f1418f?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/810a26f1418f</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[stream]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Sun, 18 Aug 2019 15:48:00 GMT</pubDate>
            <atom:updated>2020-09-20T20:10:24.903Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*BfZR9mOT1p9fSr_DyWfyhg.png" /><figcaption>Diagram of the example/PoC Kafka Streams application</figcaption></figure><p>When talking about microservices architecture, most people think of a network of stateless services which communicate through HTTP (one may call it RESTful or not, depending on how much of a nitpicker one is). <br>But there is another way, which may be more suitable depending on the use case at hand.</p><p>I am talking about event-driven microservices, where in addition to the classic request-response pattern, services publish messages which represent events (facts) and subscribe to topics (or queues depending on the terminology used) to receive events/messages.</p><p>To fully understand and embrace this new software design paradigm is not straight-forward but it is totally worth it (at least looking into it).<br>There are several interconnected concepts which need to be explored in order to discover the advantages of event-driven design and the evolutionary path which led to it, for example:</p><ul><li>Log (including log-structured storage engine and write-ahead log)</li><li>Materialized View</li><li>Event Sourcing</li><li>Command Query Responsibility Segregation (CQRS)</li><li>Stream processing</li><li>“Inside-out” databases (a.k.a. “un-bundled” databases)</li></ul><p>I’d like to point you to the following books to get familiar with those topics:</p><ul><li><a href="https://amzn.to/31S2Trb"><strong>Designing Data-Intensive Applications</strong></a><strong> </strong>by by Martin Kleppmann — this is a very comprehensive book, it starts covering single-node application concepts, then distributed systems and finally batch and stream processing.</li><li><a href="https://www.confluent.io/designing-event-driven-systems"><strong>Designing Event-Driven Systems</strong></a> by Ben Stopford — it focuses on Apache Kafka as the backbone of event-driven systems.</li><li><a href="https://amzn.to/2KDRqWc"><strong>Kafka: The Definitive Guide</strong></a><strong> </strong>by Neha Narkhede, Gwen Shapira, et al. — a more practical book than the other two.</li></ul><p>I read those three books and then I started building a simple PoC since learning new design ideas is great but it is not completed until you also put them in practice.<br>Also, I was not happy with the examples of event-driven applications/services available online, I found them too simplistic and not properly explained, so I decided to create one.</p><h3>The proof of concept</h3><p>The source code is split in two GitHub repositories (as per <a href="https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html">the Clean Architecture</a>):</p><ul><li><a href="https://github.com/VictorGil/transfers_api">transfers_api</a> → contains Java entities and Avro data definition files</li><li><a href="https://github.com/VictorGil/transfers_recording_service">transfers_recording_service</a> → contains the business logic and the Kafka-related code</li></ul><p>The proof of concept service keeps track of the balance available in bank accounts (like a ledger).<br>It listens for <a href="https://github.com/VictorGil/transfers_api/blob/1e7e4519dce6a6d1bf7628738e3358490c706d6f/src/main/resources/avro/transfer.avsc">Transfer</a> messages on a Kafka topic and when one is received, it updates the balance of the related account by publishing a new <a href="https://github.com/VictorGil/transfers_api/blob/1e7e4519dce6a6d1bf7628738e3358490c706d6f/src/main/resources/avro/accountBalance.avsc">AccountBalance</a> message on another Kafka topic.<br> Please note that each entity type is represented by two different classes:</p><ul><li>one is generated by Apache Avro and it’s used for serialization and deserialization (so they can be sent and received from Kafka) → see <a href="https://github.com/VictorGil/transfers_api/tree/1e7e4519dce6a6d1bf7628738e3358490c706d6f/src/main/resources/avro">avro</a> directory.</li><li>the other one is a POJO which may contain some convenience constructors and does not depend on Avro → see <a href="https://github.com/VictorGil/transfers_api/blob/1e7e4519dce6a6d1bf7628738e3358490c706d6f/src/main/java/net/devaction/entity">net.devaction.entity</a> package.</li></ul><p><a href="https://github.com/VictorGil/transfers_api/tree/1e7e4519dce6a6d1bf7628738e3358490c706d6f/src/main/java/net/devaction/kafka/avro/util">net.devaction.kafka.avro.util</a> package holds converters to move back and forth from one data representation to the other. <br>In the beginning, Apache Kafka may seem overwhelming, even though it resembles a classic messaging broker such as ActiveMQ or RabbitMQ, it is much more than that and it works very differently internally.<br>Also, there are several Kafka client APIs, which adds more confusion to the learner.<br> We are going to focus on the following three:</p><ul><li>The <a href="https://kafka.apache.org/documentation/#producerapi">Producer API</a> and the <a href="https://kafka.apache.org/documentation/#consumerapi">Consumer API</a></li><li>The <a href="https://kafka.apache.org/documentation/#streamsapi">Streams API</a></li></ul><p>The Producer and the Consumer APIs are lower level and the Streams API is built on top of them. <br> Both sets of APIs have advantages and disadvantages.<br>The Producer/Consumer API provides finer control to the application developer at the cost of higher complexity.<br>On the other hand, the Streams API is not as flexible but it allows the implementation of some standard operations more easily and it requires much less code.</p><p>The “transfers recording” example/PoC service can be started in one of the following two modes:</p><ul><li>The (explicit) <strong>polling</strong> mode which uses the Producer API, the Consumer API and also the Streams API (to create a queryable materialized view backed by a local data store, more on that later) → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/main/TransfersRecordingServicePolling.java">TransfersRecordingServicePolling.java</a></li><li>The “<strong>join streams</strong>” mode which only uses the Streams API → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/joinstreams/TransfersRecordingServiceJoinStreams.java">TransfersRecordingServiceJoinStreams.java</a></li></ul><p>The two modes provide exactly the same functionality which is quite convenient for comparison purposes.</p><h3>The (explicit) polling mode</h3><p>It has four main components:</p><ul><li>A <a href="https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html">consumer</a> which listens on the “<em>transfers</em>” topic → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/transferconsumer/TransferConsumer.java#L103">TransferConsumer.java</a></li><li>A <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html">ReadOnlyKeyValueStore</a> (which is part of the Streams API) to materialized the “<em>account-balances</em>” topic data into a queryable view, so we can use the accountId value to retrieve the latest/current balance of a specific account → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/accountbalanceproducer/AccountBalanceRetrieverImpl.java#L93">AccountBalanceRetrieverImpl.java</a>.<br>Please note that the <em>accountId</em> value is extracted from the “<em>transfer</em>” data message received by the consumer.</li><li>The business logic which creates a new/updated <em>AccountBalanceEntity</em> object from the received <em>TransferEntity</em> object and the current <em>AccountBalanceEntity</em> present in Kafka → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/core/NewAccountBalanceProvider.java#L15">NewAccountBalanceProvider.java</a></li><li>A <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html">producer</a> which publishes the updated balance by sending a message to the “<em>account-balances</em>” topic → and the local data store will get updated accordingly.</li></ul><h3>The “join streams” mode</h3><p>As we said before, this second operating mode only uses the Streams API/DSL and taking advantage of it, we can code at a higher level of abstraction:</p><ul><li>Instead of a <a href="https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html">consumer</a>, we use a <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/KStream.html">KStream</a> → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/271f1e7716b0298e18b90578eed6d978e0cd1d5f/src/main/java/net/devaction/kafka/transfersrecordingservice/joinstreams/TransfersRecordingServiceJoinStreams.java#L62">TransfersRecordingServiceJoinStreams.java</a></li><li>Instead of a <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/state/ReadOnlyKeyValueStore.html">ReadOnlyKeyValueStore</a>, we use a <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/KTable.html">KTable</a> → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/271f1e7716b0298e18b90578eed6d978e0cd1d5f/src/main/java/net/devaction/kafka/transfersrecordingservice/joinstreams/TransfersRecordingServiceJoinStreams.java#L69">TransfersRecordingServiceJoinStreams</a>.java again.</li><li>The business logic remains the same but now we call it from the <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/streams/kstream/ValueJoiner.html">ValueJoiner</a> → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/eecd02b7f893afcb7d96a32907a968713b7534ca/src/main/java/net/devaction/kafka/transfersrecordingservice/joinstreams/TransferAndAccountBalanceJoiner.java#L40">TransferAndAccountBalanceJoiner.java</a></li><li>Instead of a <a href="https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html">producer</a>, we redirect the output of the streams join to the desired topic (i.e., “<em>account-balances</em>”) → see <a href="https://github.com/VictorGil/transfers_recording_service/blob/271f1e7716b0298e18b90578eed6d978e0cd1d5f/src/main/java/net/devaction/kafka/transfersrecordingservice/joinstreams/TransfersRecordingServiceJoinStreams.java#L75">TransfersRecordingServiceJoinStreams.java</a> once more → and the KTable will get updated with each message, closing the loop.</li></ul><p>We can see that the code is much more compact than in the previous mode.<br>We do not need to explicitly map the <em>KStream</em> key to the <em>KTable</em> key, that’s exactly what the <em>join</em> does (see line 11 in the code snippet below). Hence, we need to choose the Kafka keys accordingly. In this case, both message keys represent the <em>account id</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9182d0977cfbd54f6f176ec9d8d33271/href">https://medium.com/media/9182d0977cfbd54f6f176ec9d8d33271/href</a></iframe><h3>Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*6ayv8RCXS1eDfvfe.png" /><figcaption>Diagram of the example/PoC Kafka Streams application</figcaption></figure><h3>Running the code</h3><p>To build and run the PoC application, in addition to Maven and Java, we also need a Kafka broker.<br>I decided to install the <a href="https://www.confluent.io/product/confluent-platform">Confluent Platform</a> which includes a Kafka broker (or a cluster depending on the configuration chosen) with some example topics and pre-configured integration with ElasticSearch and Kibana. But more importantly, it also includes an admin Web UI called “<em>Control Center</em>” which comes in very handy.</p><p>I hit a few bumps when running the Confluent Platform for the first time on my Fedora 30 computer. Namely, I had to manually install a couple of software packages (i.e., “<em>jot</em>” and “ <em>jq</em>”). <br>And I had to separately install the <a href="https://docs.confluent.io/current/cli/index.html"><em>Confluent CLI</em></a>. <br>I also had to perform some several changes to some properties files and bash scripts to be able to run the Confluent Platform using a non-root user, <a href="https://gist.github.com/VictorGil/5866f90dc626126897a76dbee53deeec">here</a> are the changes, please modify the configuration values as per your environment.</p><p>Watch the following <a href="https://youtu.be/w-Vy6_0buYo">YouTube video</a> to get all the details including starting the Confluent Platform and running the example Streams application:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fw-Vy6_0buYo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dw-Vy6_0buYo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fw-Vy6_0buYo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee539e4d6db7566fdd5efab7773bc31c/href">https://medium.com/media/ee539e4d6db7566fdd5efab7773bc31c/href</a></iframe><p><em>Originally published at </em><a href="https://www.devaction.net/2019/08/event-driven-microservices.html"><em>www.devaction.net</em></a><em> on August 23rd, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=810a26f1418f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/using-apache-kafka-to-implement-event-driven-microservices-810a26f1418f">Using Apache Kafka to implement event-driven microservices</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vert.x microservices: an (opinionated) application]]></title>
            <link>https://medium.com/@victorgil_91367/vert-x-microservices-an-opinionated-application-56e44c0b45b4?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e44c0b45b4</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[vertx]]></category>
            <category><![CDATA[android]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 22:38:05 GMT</pubDate>
            <atom:updated>2019-01-24T08:30:43.861Z</atom:updated>
            <content:encoded><![CDATA[<p>First of all, sorry for the <a href="https://en.wikipedia.org/wiki/Tautology_(language)"><strong>tautology</strong></a> in the title, a library can be either opinionated or un-opinionated (such as <a href="https://vertx.io/"><strong>Vert.x</strong></a> is), but an application can only be opinionated. However, I decided to include “opinionated” to help getting my point across even though is redundant.</p><h3>The Motivation</h3><p>I am a big fan of Vert.x and the official documentation is quite good, yet it is not straight-forward to understand how it works and how to use it. <br>There are a lot of blogs and articles describing Vert.x terminology and its concurrency model.<br>There are also tons of “Hello World” Vert.x applications on Github, and the rest seem to be just variations of the already typical “<em>Web chat application using Vert.x</em>”.<br>On top of that, many of them are outdated (using AngularJS instead of Angular2+, for example).</p><p>The only two exceptions which I found are:<br><a href="https://github.com/cescoffier/vertx-microservices-workshop/"><strong>vertx-microservices-workshop</strong></a>: a demo application by the Vert.x development team.<br><a href="https://github.com/benorama/ngrx-realtime-app"><strong>ngrx-realtime-app</strong></a>: proof of concept application focused on state management which just updates a shared counter.</p><p>Those are good example applications and I am thankful to the authors for sharing with the community. <br>However, one problem I have with them is that the source code of all the microservices is put together in the same repository (I guess for the sake of simplicity). <br>I am not trying to be a purist, but for me, the main requisites of a microservice is that it must be independently deployed, upgraded and run, which basically means that each microservice should have its own version string and its own code repository. <br>Also, each microservice should have a well defined (contracted) API. And regarding APIs, JSON is good… for JavaScript, but for Java (and other statically-typed languages for that matter) there is something better (I warned you this was going to be an opinionated post ☺): <a href="https://github.com/protocolbuffers/"><strong>Google Protocol Buffers</strong></a>, but I am getting ahead of myself…</p><p>And one more thing, the devil is in the details, and these applications avoid dealing with “boring” stuff such as basic security (HTTPS configuration), which leaves information gaps for these important topics.</p><p>So in short, I was lacking a comprehensive and non-trivial (yet simple) microservices application based on Vert.x, and since I believe the best way of learning is to roll up one’s sleeves and code, I wrote one in my spare time.</p><p>And then, after I had already written almost all of this post (and completed my example microservices application), I found this: <a href="https://github.com/sczyh30/vertx-blueprint-microservice"><strong>vertx-blueprint-microservice</strong></a>, which is almost what I was looking for (despite of being monorepo). But it seems to be going through a <a href="https://github.com/sczyh30/vertx-blueprint-microservice/issues/17"><strong>major refactoring</strong></a> which has not been completed after two years and it actually looks outdated (the frontend is still based on AngularJS).</p><h3>The Code</h3><p>I have just “released” (umbrella) version 2.0.0.0 of my example application (this previous <a href="https://medium.com/@victorgil_91367/a-java-dev-journey-to-full-stack-first-chapter-521cc8946edb"><strong>post</strong></a> presented the first version), which is composed of <strong>six</strong> different Github repositories:</p><p><a href="https://github.com/VictorGil/mylocation-backend"><strong>mylocation-backend</strong></a> version 1.1.0.0 → main microservice and API gateway.<br><a href="https://github.com/VictorGil/mylocation-last_known_location"><strong>mylocation-last_known_location</strong></a> version 1.0.0.0 → another microservice, including its protobuf API.<br> <a href="https://github.com/VictorGil/mylocation-last_known_location-persistence"><strong>mylocation-last_known_location-persistence</strong></a> version 1.0.0.0 → yet another microservice, also including its own protobuf API.</p><p><a href="https://github.com/VictorGil/vertx-utility-extensions"><strong>vertx-utility-extensions</strong></a> version 1.0.0.0 → library used by the previous three microservices to avoid boilerplate code when starting and initializing Vert.x.</p><p><a href="https://github.com/VictorGil/mylocation-android"><strong>mylocation-android</strong></a> version 1.0 → Android application which sends live location data messages to <a href="https://github.com/VictorGil/mylocation-backend">mylocation-backend</a>.</p><p><a href="https://github.com/VictorGil/mylocation-frontend"><strong>mylocation-fronten</strong>d</a> version 1.1.0 → Angular 7 application which asynchronously displays either live location data (if possible) or the latest previously saved location.</p><p>If you think that having to manage and push commits to 6 different source code repositories involves quite a lot of hassle, you are totally right, but I do believe the advantages overcome the disadvantages (more on that in a future post). Again, this is <em>my</em> opinion, and please note that it is not based on irrational fads but on years of experience.<br>And of course, I am always happy to change my mind whenever facts, circumstances or enlightening discussions show me a better way☺.</p><p>I mentioned <a href="https://github.com/protocolbuffers/"><strong>Google Protocol Buffers</strong></a> before because all the communication between the Java microservices is done through protobuf binary encoding, not only for <a href="https://auth0.com/blog/beating-json-performance-with-protobuf/"><strong>better performance</strong></a> but also because of the automatic generation of the required API classes.<br>And for the sake of modularisation, each protobuf API is built separately into its own Maven (jar) artifact.</p><p>The communication between the backend and the frontend is done through JSON though, since I did not want to deal with protobuf in JavaScript/TypeScript. <br>For similar reasons the communication between the Android client app and the backend also uses JSON.</p><p>Please note that currently, despite using clustered Vert.x instances, all the microservices are meant to run on the same host (for the sake of simplicity), I plan to change that in the future.</p><p>And one comment about the coding style, as soon as the implementation of a <a href="https://vertx.io/docs/apidocs/io/vertx/core/Handler.html"><strong>Handler</strong></a> grew bigger than a few lines, I decided to create a new class in a separated file instead of an anonymous class or a lambada expression.<br>I prefer that because I like methods (and classes) to be small, as a direct consequence of applying the Single Responsibility Principle.</p><h3>The Functionality</h3><p>Ok, so what does the application do?<br>When a new Web client connects, the <a href="https://vertx.io/docs/vertx-web/java/#_sockjs_event_bus_bridge"><strong>event bus bridge</strong></a> is set up, then the latest saved location (previously sent by my mobile phone to the backend) is requested, retrieved (through the event bus bridge) and displayed. Simultaneously (everything is asynchronous), it starts listening for live location updates again on the Vert.x event bus bridge, and as soon as the live updates start coming (through <strong><em>server push</em></strong>), they replace the saved location on the display.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/0*Tcg8onKvWqz_NMkn.png" /></figure><p>The app is up and running on a small Amazon Lightsail VM (1GB RAM), so you may see it in action yourself: <a href="http://mylocation.devaction.net/"><strong>http://mylocation.devaction.net</strong></a></p><h3>Final Thoughts</h3><p>I could write much more on related topics such as: <br>— How I combined Vert.x and Spring in the backend (please don’t freak out, I just use Spring purely for dependency injection, I dislike Spring gigantic ecosystem as a whole, specially Spring Boot — again, just my opinion). <br>— The Android service which fetches the location provided by the GPS. <br>— The Vert.x service which saves each received location in a local file (in binary format) since it is not efficient to use a database just for that.<br> — The related Vert.x service which provides that saved location.<br> — Client-side certificate (in Android) in addition to server-side.<br> — The application configuration (resolving and loading properties values, password encryption). <br>— RxJS usage in the frontend.<br> — The Vert.x event bus SockJS bridge.<br> — How “WINCH” Linux signal is used to gracefully shutdown Vert.x<br> —The social implications of sharing my live location with potentially everybody (spoiler alert: I introduced a “masking” security mechanism in the Android code to keep the area where I live under the radar).</p><p>But again, I will leave that for future posts, in any case, everything is in the code, so feel free to peek at it on Github or ask me☺.</p><p><em>Originally published at </em><a href="https://www.devaction.net/2019/01/vertx-microservices.html"><em>www.devaction.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e44c0b45b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Java dev journey to full-stack: first chapter]]></title>
            <link>https://medium.com/@victorgil_91367/a-java-dev-journey-to-full-stack-first-chapter-521cc8946edb?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/521cc8946edb</guid>
            <category><![CDATA[rxjs]]></category>
            <category><![CDATA[vertx]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[angular]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 22:16:47 GMT</pubDate>
            <atom:updated>2019-01-19T17:27:27.247Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>The Motivation</strong> <br>I am an experienced Java developer and unsurprisingly, I like Java. I know it is not perfect but it works just fine for me (I enjoy type-safety and I do not consider verbosity a disadvantage, quite the opposite). I also know that some people dislike Java, which is also fine.</p><p>But recently I decided to step out of my confort zone as developer, my goal isn’t to be one of the “cool kids” neither trying to monetize a new skill in the job market. <br>I have a quite practical motivation: I want to be able to build more (different) stuff. That’s exactly the same reason why I learnt Android development by myself a couple of years ago.</p><p>Web applications are ubiquitous, even more than native mobile apps, and thanks to cloud computing, one can easily and inexpensively release their idea/app to the World Wide Web.</p><p>I already did some Web development in the past, in the <em>bad</em> old days of JSP and JSF, but the process was slow and painful.</p><p>Nowadays the Web landscape has totally changed though, JavaScript is a mature language, it provides <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes"><strong>classes</strong></a> (well, not really, but bear with me) and a whole ecosystem which includes dependency injection, dependency resolution, a backend server (Node.js) and advanced IDEs.</p><p><strong>The JavaScript Framework <br></strong>So the first step was to decide what to use for frontend development, among the numerous options, there are two main ones (by popularity): React or Angular.</p><p>After some research it seemed that Angular was <a href="https://jaxenter.com/java-developers-prefer-angular-140390.html"><strong>more suitable</strong></a> for devs coming from the Java world, mainly because of two reasons:</p><ul><li>Typescript is more strongly typed than “naked” JavaScript, which makes it more similar to Java.</li><li>Angular is a framework (not a library like React) which means that it provides more guidance (it’s opinionated) and includes a comprehensive set of helper tools (e.g., a CLI).</li></ul><p>Hence, I chose Angular and VSCode as my Integrated Development Environment.</p><p><strong>The Proof Of Concept<br></strong>After having decided <strong><em>with what </em></strong>I’d code, one more important question remained open: <strong><em>what</em></strong> to code.</p><p>All I knew is that I wanted to make a modern (albeit simple) single page Web site. And by modern I essentially mean asynchronous live data loading onto the UI without user interaction (i.e., server data push, no need to click the “refresh” button).</p><p>But what would be the source of the data? That’s when I decided to make things more interesting by throwing Android into the mix.</p><p>I decided that the application would push location updates from my phone to the backend running on AWS and then to the Web browser. All the data transfers would be in JSON format and secured by HTTPS (self-signed certificate).</p><p>And since I was already in ‘adventurous mode’, I decided to use Vert.x (a Java framework to build reactive applications) in the backend because it had been <a href="https://www.devaction.net/2018/05/verx-01.html"><strong>on my radar</strong></a> for a while already and it also provides good integration with JavaScript through its <a href="https://vertx.io/docs/vertx-web/java/#_sockjs_event_bus_bridge"><strong>SockJS event bus bridge</strong></a>.</p><p>You can see how (version 1 of) the web application works in the following video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FeYnAtV6A0wA%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DeYnAtV6A0wA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FeYnAtV6A0wA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27c49080165c86bda5f69f66b34d8edb/href">https://medium.com/media/27c49080165c86bda5f69f66b34d8edb/href</a></iframe><p><strong>The Code<br></strong>After several weeks of after-work coding and dealing with three different Git repositories, three different IDEs (AndroidStudio, Eclipse and VSCode) and two programming languages (Java and Typescript), I am ready to share the code with anyone who would care to have a look at it:</p><p><a href="https://github.com/VictorGil/mylocation-frontend"><strong>mylocation-frontend</strong></a><br><a href="https://github.com/VictorGil/mylocation-backend"><strong>mylocation-backend</strong></a><br> <a href="https://github.com/VictorGil/mylocation-android"><strong>mylocation-android</strong></a></p><p>And the first version (<strong><em>UPDATE</em></strong>: <a href="https://medium.com/@victorgil_91367/vert-x-microservices-an-opinionated-application-56e44c0b45b4"><strong><em>second version</em></strong></a><strong><em> </em></strong><em>is already available</em>) of the application is live at <a href="http://mylocation.devaction.net/"><strong>http://mylocation.devaction.net</strong></a></p><p>Please note that you will get “<em>your connection is not private</em>” warning (or some other similar one depending on your browser of choice) when accessing it because of the self-signed certificate, but as long as you check that the SHA-256 fingerprint matches the following hexadecimal digits, you are safe: DA C1 F8 06 D2 4E 17 C9 C4 F4 D7 47 40 AF CD F1 22 00 36 3E 97 CB 41 C2 63 DD 28 1F A0 51 12 29</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*DQPWlFhB6kYA62hA.png" /></figure><p>Also, I do not think the backend will be able to handle much load since it runs on the smallest type of Amazon Lightsail instances (just 512 MB memory and one core processor →<strong><em>UPDATE</em></strong>: VM upgraded to 1GB RAM) so if it is not able to keep up with the demand at least it would mean that this post has become somewhat popular :-)</p><p>I could write much more on related topics such as: <br>— How I combined Vert.x and Spring in the backend.<br> — The Android service which fetches the location provided by the GPS. <br>— Client-side certificate (in Android).<br> — The application configuration (loading, password encryption). <br>— RxJS. <br>— The Vert.x event bus SockJS bridge. <br>— The social implications of sharing my live location with potentially everybody (spoiler alert: I introduced a “masking” security mechanism in the Android code to keep the area where I live under the radar).</p><p>But I will leave that for future chapters of this journey since this learning exercise is far from over yet, stay tuned!<br><strong><em>UPDATE</em></strong>: <a href="https://medium.com/@victorgil_91367/vert-x-microservices-an-opinionated-application-56e44c0b45b4"><strong>next episode</strong></a>.</p><p><em>Originally published at </em><a href="https://www.devaction.net/2018/11/full-stack-01.html"><strong><em>www.devaction.net</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=521cc8946edb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avenger Bot: A Twitter Bot To Avenge Customer Abuse]]></title>
            <link>https://medium.com/coinmonks/avenger-bot-a6270ff7a957?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6270ff7a957</guid>
            <category><![CDATA[bots]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Sun, 14 Oct 2018 14:15:41 GMT</pubDate>
            <atom:updated>2020-08-31T14:47:58.379Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*DGAoIDBaRCnaeYPRcNVo-g.png" /></figure><h4>Building a multi-threaded bot which replies to selected tweets</h4><p><strong>TL;DR: </strong>Complete Java code of a multi-threaded Twitter bot which spreads the word about customer abuse and how to (not) get banned by Twitter.</p><p>The Twitter API was one of the first available from a major Internet organization and it is probably the most used API worldwide, even though in the past few years Twitter has been making it tougher for developers to use it, you may check <a href="https://www.theverge.com/2018/5/16/17362138/twitter-api-third-party-apps-changes-explained"><strong>this</strong></a> as an example. Not everybody is on Twitter but almost every business is, especially big telcos. Most of them even provide customer support through Twitter, and despite being usually terrible (the support), at least it is an open channel and you can share your frustration with other frustrated fellow customers.</p><h3>Motivation</h3><p>When one of the major telecommunications operators in Spain started aggressively chasing my mom and my whole family because someone used her name and national identity number to fraudulently buy services from them which were never paid (it seems they <em>intentionally?</em> have barely any security controls in place), I decided to take revenge on Twitter.</p><p>I’ve got very mad, <strong>Jazztel</strong> (the telco) was not interested in finding the truth whatsoever, they were accusing my mom of not paying her bills and threatening her to go to court, this <a href="https://youtu.be/9cHGkn_4Sk4"><strong>telephonic conversation</strong></a> (in Spanish) proves it.</p><p>I’ve got inspiration from <a href="https://medium.freecodecamp.org/how-i-used-my-programming-skills-to-buy-a-stroller-2778cb85e8b2"><strong>this post</strong></a> which I read a few months ago.</p><p>But my “<em>avenger bot</em>” is more sophisticated than that, it uses the <a href="https://developer.twitter.com/en/docs/tutorials/consuming-streaming-data.html"><strong>Twitter streaming API</strong></a> to listen for tweets addressed to Jazztel and then replies to them with an angry complain (I am aware that it probably won’t make any difference but at least it was a way to release my anger while learning something since it was technically challenging and I also found it interesting).</p><h3>Implementation</h3><p>So the first thing I needed was a Twitter account under which my bot would run and then also a “Twitter app” in order to retrieve the four security keys required for authentication and authorization against the Twitter API.</p><p>Luckily I had all that from working on a <a href="https://medium.com/p/bfd402a770d3"><strong>previous pet project</strong></a> of mine, because since few months ago, Twitter forces new users of their API to register for a developer account and wait for the registration to (hopefully) be approved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*r6vxd8KbpuhDIrri.png" /></figure><p>Then I started working on my bot, and after few evenings I had it ready, the whole source code (Java 8) is available on <a href="https://github.com/VictorGil/avenger-bot"><strong>GitHub</strong></a>.</p><p>These are the<strong> main features:</strong></p><p>- The four security keys:<br><em>Consumer Key (API Key)<br>Consumer Secret (API Secret) </em><br> <em>Access Token</em><br> <em>Access Token Secret</em><br> are encrypted in the <a href="https://github.com/VictorGil/avenger-bot/blob/master/src/main/resources/config/config.json"><strong>JSON configuration file</strong></a> and decrypted at runtime using a password stored in an environment variable.</p><p>- It uses very simple dependency injection through Spring Framework, I chose XML configuration over annotations because as Sam Atkinson said in his <a href="http://samatkinson.com/why-i-hate-spring/"><strong>famous post</strong></a> more than four and a half years ago: <em>I don’t like magic.<br></em>I do not like to “<em>code</em>” in XML either, that is why I said “<strong><em>very</em></strong><em> </em><strong><em>simple</em></strong><em> dependency injection</em>” before, I strongly believe that Spring (or any other DI framework for that matter) should be used <strong>just to wire beans together</strong>, nothing more, no smart/custom type conversions, no SpEL, etc…</p><p>- It implements the producer/consumer pattern using a <a href="https://github.com/VictorGil/avenger-bot/blob/master/src/main/java/net/devaction/avengerbot/tweetscapturedproducer/TweetsQueueImpl.java"><strong>blocking queue</strong></a>.</p><p>- The application properly shuts down (disconnecting gracefully from the stream of tweets) on Linux after receiving the WINCH signal by implementing the <a href="https://github.com/VictorGil/avenger-bot/blob/master/src/main/java/net/devaction/avengerbot/main/Manager.java#L17"><strong>SignalHandler</strong></a> interface (the complete motivation and explanation of this shutdown mechanism deserves its own post which I may write in the near future).</p><p>- The bot uses <a href="http://twitter4j.org/"><strong>Twitter4j</strong></a> library so I do not need to deal directly with REST/HTTP.</p><p>- There are no unit tests but at least some “<em>tester</em>” classes are included which helped me to validate the code as I was writing it.</p><h3>Execution</h3><p>After I completed the coding, I was ready to run it and see the bot in action.</p><p>The first time I started it, everything seemed to be working fine, every time a new tweet containing the keywords defined in the <a href="https://github.com/VictorGil/avenger-bot/blob/master/src/main/resources/config/config.json#L7"><strong>config.json</strong></a> was published, the bot would reply after waiting a semi-random amount of time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/0*BFJTWcIBCtHtqEU_.png" /></figure><p>But after an hour or so, I got the following ugly error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*ZeHQ7RkvhfNWtvsf.png" /></figure><p>Which meant that my application had been banned from publishing tweets, even though it could still read them. I could see it also in the Twitter Web UI</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/0*7r4wF6vvB7izGt_Z.png" /></figure><p>But why? I had been cautious not to reach the <a href="https://iag.me/socialmedia/guides/do-you-know-the-twitter-limits/"><strong>Twitter API limits</strong></a>, my bot always waited for more than 2 and a half minutes between two consecutive tweet submissions, much longer than the 36 seconds which seems to be the limit.</p><p>After some research I found the answer in the Twitter spamming rules:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*IONa2qOABwWLTf4T.png" /></figure><p>My bot randomly selects the content of the tweets from a pool of twenty texts, so after several tweets were published containing the same text, my application was banned.</p><h3>Future work</h3><p>Something that all software developers hate are loose requirements but they are everyday fare and whole methodologies such as Scrum have been created to tackle this issue.</p><p>In this case, several unanswered questions come to my mind:<br><em>What exactly makes two tweets “substantially similar”?<br>If two tweets contain the same words but the order is different does Twitter consider them duplicates? <br>What if only 80% of the words are the same? <br>And what if only 60%? <br>Is it ok to repeat the same tweet just once a day? <br>Or maybe once a week?</em></p><p>The Twitter documentation intentionally does not answer these questions and the information available from other sources is <a href="https://code.tutsplus.com/tutorials/using-the-twitter-api-to-tweet-repetitive-content--cms-28096"><strong>quite ambiguous</strong></a>.<br>So my only option is experimentation and testing.</p><p>Does this mean that all the hassle was in vain? Absolutely not! <br>I had an idea, a challenge, and I went through with it. <br>Something bothered me and I did something about it (and I also enjoyed it). So even though the result is not ideal (yet) because of some unforeseen limitation, the whole experience was totally worth it. <br>I have got to experiment with something new to me and I also get the chance to share the whole episode with whoever might be interested in it.</p><p>Currently, I am waiting for my Twitter developer request to be accepted so that I can create a new app. Once I am able to publish tweets using the API again, I may even do some research and throw in some AI in order to generate distinct enough tweets to be published without Twitter banning me…</p><p>So if I find a workaround, I may also launch my Avenger Bot against another company which disrespects their customer/passengers: <a href="https://medium.com/p/826c1bb4add0"><strong>SmartWings</strong></a>.<br>Stay tuned!</p><p><em>Originally published at </em><a href="https://www.devaction.net/2018/10/avenger-bot.html"><strong><em>www.devaction.net</em></strong></a><em>.</em></p><blockquote><a href="https://coincodecap.com/?utm_source=coinmonks">Get Best Software Deals Directly In Your Inbox</a></blockquote><figure><a href="https://coincodecap.com/?utm_source=coinmonks"><img alt="" src="https://cdn-images-1.medium.com/max/584/0*OJ-qb5G6i863msBB.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6270ff7a957" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/avenger-bot-a6270ff7a957">Avenger Bot: A Twitter Bot To Avenge Customer Abuse</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My first day playing around with Vert.x]]></title>
            <link>https://medium.com/@victorgil_91367/my-first-day-play-around-with-vert-x-8cdf00b55271?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cdf00b55271</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[reactive-programming]]></category>
            <category><![CDATA[vertx]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Thu, 31 May 2018 05:43:24 GMT</pubDate>
            <atom:updated>2018-05-31T06:26:58.762Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been reading quite a lot about <a href="https://vertx.io/"><strong>Vert.x</strong></a> recently and finally the day before yesterday I sat down and spent some time exploring it by writing some code, today I would like to present the outcome.</p><h4>A little bit of history</h4><p>Like all <strong>reactive</strong> frameworks/tool-kits/libraries, Vert.x is inspired by Node.js. Back when it was released, Node.js shocked many Java developers who looked down on Javascript, on top of owning the Web client side, JS wanted to conquer also the server side! And thanks to its non-blocking nature, Node.js performed better than typical Java application servers such as Tomcat or Glassfish. Then the Java world reacted (pun intended) and today Vert.x is one of the most matured reactive technologies for the JVM.</p><h4>First Vert.x application</h4><p>When trying to build my first super-simple <em>reactive</em> (not really) application using Vert.x, I started following <a href="http://escoffier.me/vertx-hol/"><strong>this tutorial</strong></a><strong> </strong>from the lead developers of Vert.x.</p><p>That’s great stuff, but I found it a little overwhelming as the first hands-on experience. I understand that the authors are eager to highlight most of the nice features of the technology but I decided to write my own example focusing just in the very basics.</p><p>The example application uses Java 8 and Maven and no custom Vert.x configuration (just the default). The <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/MainClass.java"><strong>main class</strong></a> creates one <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/MainVerticle.java"><strong>main verticle</strong></a> and then that verticle creates two more verticles:<br><strong>PublisherVerticle</strong> and <strong>ConsumerVerticle</strong>.<br>The <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/PublisherVerticle.java"><strong>PublisherVerticle</strong></a> simply publishes (broadcasts) five messages using JSON as serialization and the <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/ConsumerVerticle.java"><strong>ConsumerVerticle</strong></a> receives the messages and just logs their contents.<br>The messages are just <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/SomeMoney.java"><strong>POJOs</strong></a>.</p><p>At this point, you may wonder what a verticle is. Vert.x uses its own terminology, I recommend <a href="https://www.cubrid.org/blog/inside-vertx-comparison-with-nodejs/"><strong>this post</strong></a> in order to get familiar with it. In simple words, a verticle is a Java class whose life-cycle is managed by a Vert.x instance and it can deploy and start other verticles, it can also communicate with other verticles using the Event Bus.</p><p>This example is useful for people who are new to vert.x since it showcases the following:</p><ul><li>how to deploy and start verticles</li><li>how to serialize/marshal a POJO to be encapsulated inside a vert.x</li><li>message using JSON</li><li>how to publish a message to the vert.x event bus</li><li>how to receive a message from the vert.x event bus</li><li>how to construct a POJO from the received message</li><li>the asynchronous behaviour of vert.x</li></ul><p>And here you have <a href="https://github.com/VictorGil/simple-vertx-example-01"><strong>the code</strong></a>.</p><p>Looking at the <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/pom.xml#L20"><strong>pom.xml</strong></a> you’ll see that just adding the vertx-core dependency is enough, as usually, Maven will download the required transient dependencies for us:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/0*UShxTf2yKuZiT9iP.png" /><figcaption>Dependencies gathered by Maven</figcaption></figure><p>As you can see, Vert.x uses Netty for the network communication (I like that it does not force you to use HTTP only) and Jackson for JSON serialization.</p><p>To run the example application you just need to execute its <a href="https://github.com/VictorGil/simple-vertx-example-01/blob/master/src/main/java/net/devaction/vertx/simplevertxexample01/MainClass.java"><strong>main class</strong></a> class, and looking at the output helps to understand the “<strong>everything asynchronous by default</strong>” philosophy of Vert.x.</p><p>Even the call to start Vert.x is asynchronous, that is why the main method is the first one to finish:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ue1iSDTitu9l0z_e.png" /><figcaption>Output when running the main class</figcaption></figure><p>We can also confirm that Vert.x manages the pool of threads for us so we do not need to worry about it.</p><p>Since this is a simplistic example, all verticles run on the same JVM/process, but one of the features of Vert.x is its convenience to implement microservices, thus, I could easily refactor the application in order to place each verticle in a different (Maven) module (and run it on its own runtime environment or in different machines) but that’s subject for a future post.</p><p>Vert.x is a broad topic and I would like to highlight what this post does not cover and I may explore in the future:</p><ul><li>using a different serialization other than JSON (Google protobuffers for example)</li><li>communication with external services (using HTTP or simply TCP or Websockets) either directly or through the Event Bus</li><li>using Vert.x as the back-end for a Web app (front end could be Angular or ReactJS)</li><li>using Vert.x as the back-end for a native mobile app (probably Android since I already have some experience)</li><li>Vert.x cache capabilities (Shared Map)</li><li>using different instances of Vert.x in the same application</li><li>the polyglot capabilities of Vert.x (i.e., using it with other languages supported by the JVM such as Kotlin)</li></ul><p>Thanks for reading!</p><p><em>Originally published at </em><a href="http://www.devaction.net/2018/05/verx-01.html"><strong><em>www.devaction.net</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cdf00b55271" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proof of Social Network Identity: a proof of concept]]></title>
            <link>https://medium.com/coinmonks/proof-of-social-network-identity-a-proof-of-concept-bfd402a770d3?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfd402a770d3</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Tue, 08 May 2018 19:21:12 GMT</pubDate>
            <atom:updated>2018-05-09T13:12:08.167Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qh4cGVb-TDBC9ykEe3Cskw.png" /></figure><p><em>TL;DR: start mining a new cryptocurrency based on Ethereum using a regular (so far Linux only) computer and a Twitter account in just 15 minutes: </em><a href="https://youtu.be/5VxZ8qhordo"><em>Youtube video</em></a></p><p>One of the main criticisms to blockchain technology is based on the huge consumption of electricity required by the biggest networks (e.g., Bitcoin and Ethereum).<br> That huge electric power consumption comes from using <strong>Proof of Work</strong> (PoW) algorithms to prevent “<em>double spending</em>” which is closely related to the “<em>51% attack</em>”.</p><p>Some other types of algorithm have been developed as an alternative to PoW, the more popular one being <strong>Proof of Stake</strong> (PoS), which basically implies that individuals with higher stake (i.e., more crypto money) have more power (i.e., are more trusted) than the rest.<br> Hence, PoS, creates an entry barrier for those who want to become members of the blockchain network, since one needs to own a stake in order to join the network.</p><p>The key to prevent “<em>double spending</em> / <em>51% attacks”</em> is to use <em>something</em> in the block validation mechanism which is complicated or expensive enough for some malicious actor to replicate or to clone or to create (whatever you call it) in order to gain control of the network.</p><p>In PoW that <em>something</em> is computer power (which requires electricity power)<br>In PoS that <em>something</em> is a cryptocurrency.</p><p>But maybe there is something else which one could use to create a secure-enough blockchain network, even if it meant compromising on some other feature of the current blockchain networks.</p><p>Let me elaborate a little, I read somewhere that the main features of the blockchain are:<br>- Integrity (via hashing)<br>- Authenticity (via signing)<br>- Confidentiality (via encryption)</p><p>Those three points above are an oversimplification, but it will work for the purpose of this post.</p><p>I would split down the last point in the following two points:<br>- Confidentiality of the one who <strong>submits</strong> a transaction<br>- Confidentiality of the one who <strong>validates</strong> a transaction (a.k.a. miner)</p><p>Arguably, the first type of confidentiality is much more important than the second one.</p><p>In fact, it is currently quite easy to identify who are the validators (a.k.a. miners in PoW) in Bitcoin for example, simply because mining requires such a huge investment in computers and electricity that miners stand out and are easy to find, and also there are publicly-advertised mining pools whose solely purpose is to share the mining costs.<br> So in general, people do not really care about the validators’ identity.</p><p>The identity of who submits a transaction, or more broadly, the identity of anyone who owns a blockchain address is a big issue though, and it should not be possible to obtain that information from the blockchain network.</p><p><strong>Now this is my proposal: what if by giving away the current not-so-effective protection of the validators identities we could implement a new type of algorithm which would not require huge computing power and neither the ownership of any stake in order to participate in a blockchain network?</strong></p><p>It occurred to me that we could use social networks (Twitter, Facebook, LinkedIn, StackOverflow, Github, etc…) for this.</p><p>More specifically, we could use the <em>verified</em> social networks accounts together with the power of the <strong>Open Web APIs</strong> which most of them provide in order to prevent double spending and 51% attacks.</p><p>Think about it, the social networks have made a lot of progress in order to prevent abuse and misuse. Of course you can still create fake accounts in social networks, but in order to be verified you would need more than just an email account, you will also need a working mobile telephone number.</p><p>And this new type of algorithm (let’s call it <strong>Proof of Social Network Identity</strong>, since just Proof of Identity has been used in the past to refer to other stuff) can implement tougher requirements in order to increase the security, for example:</p><blockquote>“In order to be a validator, you need to own a Twitter account which is more than a year old, and it must have 10 or more followers”</blockquote><p>Another example:</p><blockquote>“In order to be a validator, you need to own a Twitter account which is more than a year old, and it has 10 or more direct followers and 50 or more secondary followers (followers of your followers) and you must also own a Stackoverflow account with more than 10 points of reputation”</blockquote><p>Social network accounts which meet those requirements are not easy to obtain nor fake. And if the blockchain network is big enough, one malicious actor would need to be able to create thousands of them to be able to succeed or alternatively hack into and take control of Twitter, Facebook, etc… So the possibilities are endless. What requirements to choose? Well, it is a compromise between ease of access to the network versus security of the network… One could just start with low requirements and then while using the network (through experimentation and trial and error) those requirements may be increased if deemed necessary.</p><p>Since I am a hands-on person, I decided to build a proof of concept based on this new <strong>Proof of Social Network Identity</strong> (<strong>PoSNI</strong>) and I called it <a href="https://github.com/VictorGil/social-ledger"><strong>Social Ledger</strong></a>.</p><p>I did not build it from scratch though, one of the main beauties of open source code is that anyone can reuse an existing project, and that is exactly what I have done.</p><p>Initially I thought of forking the <a href="https://github.com/ethereum/go-ethereum"><strong>Go Ethereum</strong></a> client repository, but since I have no experience with Go programming language, then I decided to use <a href="https://github.com/ether-camp/ethereum-harmony"><strong>Harmony</strong></a> (which is in turn based on <a href="https://github.com/ethereum/ethereumj"><strong>EthereumJ</strong></a>) which is an implementation of the Ethereum client in Java.</p><p>Ok, so how does it work? To start the validator (or miner, even though we do not really mine in the sense that no expensive calculations are required), you need to have a Twitter account with an app associated to it.</p><p>If you do not know how to create an app in Twitter, you may follow the instructions in the following Youtube video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKPHC2ygBak4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKPHC2ygBak4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKPHC2ygBak4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/16577ebf62008f1087f503a01b8f02ec/href">https://medium.com/media/16577ebf62008f1087f503a01b8f02ec/href</a></iframe><p>Whenever a new block containing new transactions is created, the Twitter username of the miner node which is creating that block is included in one of the block fields (the <em>extradata</em> field), and also, the hashcode of that block is tweeted by that same Twitter account using the Twitter public Web API. Then the block is broadcasted to the network. Whenever a validator receives a block for the network, on top of all the regular Ethereum-standard checks, it will: <strong>1.-</strong> Get the Twitter username of the node which created/mined that block from the extradata block field value. <br><strong>2.-</strong> Look for a tweet using the Twitter public Web API containing the hashcode of the block whose author is that same Twitter account.<br>Such tweets look like these ones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/0*tLMn4Ixjj24AWzAE.png" /></figure><p>Also, the same miner (i.e., the same Twitter account) is not allowed to mine/create two blocks in a row. And if there are two competing blocks, the one whose Twitter account has been longer without creating a block, will have preference.</p><p>This is quite a lot of information to digest, so I would recommend you to just <a href="https://www.dropbox.com/s/sc1jgfph22a0a0g/social_ledger_1.0.0.0.zip?dl=0"><strong>download</strong></a> the Social Ledger distribution zip file, install it, configure it and run it (so far only on Linux computers, I have not tested it on Mac and to make it work on Windows it would require to write the equivalent Windows start script) as per the instructions in <a href="http://www.devaction.net/2018/05/socialledger-02.html"><strong>this post</strong></a> or the following Youtube video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F5VxZ8qhordo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D5VxZ8qhordo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F5VxZ8qhordo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b6ffd1a8eb85c9ecfe0d885ad244310/href">https://medium.com/media/3b6ffd1a8eb85c9ecfe0d885ad244310/href</a></iframe><p>I have an Amazon Lightsail VM running 24x7 so your local node should be able to join the network.</p><p>The testing which I have performed so far is minimal (just three nodes) so I am not sure what will happen once more nodes join the network but if I see there is enough interest, I may work on make the current implementation more robust and escalable or I may create a new implementation of the <strong>Proof of Social Network Identity</strong> (PoSNI) algorithm using a more flexible blockchain platform such as Hyperledger or Corda (or something else).</p><p><em>Originally published at </em><a href="http://www.devaction.net/2018/05/socialledger-01.html"><em>www.devaction.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfd402a770d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/proof-of-social-network-identity-a-proof-of-concept-bfd402a770d3">Proof of Social Network Identity: a proof of concept</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apology not accepted]]></title>
            <link>https://medium.com/@victorgil_91367/apology-not-accepted-826c1bb4add0?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/826c1bb4add0</guid>
            <category><![CDATA[travel]]></category>
            <category><![CDATA[customer-experience]]></category>
            <category><![CDATA[airlines]]></category>
            <category><![CDATA[customer-engagement]]></category>
            <category><![CDATA[european-union]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Sun, 01 Apr 2018 21:03:44 GMT</pubDate>
            <atom:updated>2020-09-26T20:02:17.688Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TL;DR: </strong>After being disrespected multiple times as customer by<a href="https://www.smartwings.com/"> Smartwings</a> airlines over the years, now I can finally have my little revenge by claiming 400 euros for a delayed flight. The airline sent me an insincere and pathetic apologetic email message. Apology not accepted.</p><p><strong>UPDATE from September 2020</strong>: Almost two and a half years after the flight, I got my compensation thanks to <a href="https://wingstoclaim.com/">Wingstoclaim.com</a>, see the details at the bottom.<br> <strong><br></strong>I have been a regular customer of Smartwings airlines for quite a long time already.<br>It was not by choice. The reason I fly with them is because it is the only airline which flies directly from Prague (where I live) to Valencia (my hometown).<br>Although I appreciate that they include 15 kg of check-in luggage at no extra charge, the ticket prices are quite high compared to other low cost airlines (Wizzair and Ryanair, for example).<br> <br>Also, they do not seem to care much about their customers. Several times they cancelled a flight just few days before the departure date due to “operational reasons”. It would seem to me that the real reason was that they did not sale as many tickets for that flight as they expected, so it would be more profitable to just cancel the flight and screw those of us who already bought tickets.<br> <br>Another example: in the past they used to advertise direct flights between Prague and Malaga when in reality, it was a Prague → Valencia → Malaga flight.<br>Let me try to explain it because it was quite surreal.<br>Passengers flying to both Valencia and Malaga would get on board the same airplane in Prague.<br>Then the plane would land at Valencia airport, where only the passengers who purchased a PRG-VLC ticket would get out of the plane. Immediately after that, <strong>new</strong> passengers flying VLC → PRG would get into the plane.<br>Then the plane would fly from Valencia to Malaga! In Malaga, the passengers who joined the flight in VLC would remain seated, while the passengers who bought a ticket from PRG to Malaga would finally get out of the plane and again <strong>new</strong> passengers heading to Prague would get on board the plane…<br> <br>Also, the schedules of their flights are unreliable. Delays are frequent.<br>One of such delays happened last Thursday March 29th, 2018.<br>The flight <strong>QS1054</strong> from PRG to VLC was packed because of the upcoming Easter weekend.<br>The departure time was scheduled at 18:35 and the arrival time was 21:20.<br>But after almost 4 hours of waiting, the plane took off at 22:16 and we landed at 01:00 AM (3 hours and 40 minutes delayed).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*id98mOuP_TmIQiby.jpg" /></figure><p>According to the EU regulation, each passenger of that flight is entitled to a 400 euros compensation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/0*-Umn0TSP0mo0OaIP.jpg" /></figure><p>During the flight, the captain apologized several times and literally mentioned “extraordinary circumstances” (out of the airlines control) as the cause of the delay.<br>He specifically said that the bad weather <strong>in Egypt</strong> caused a chain of delays which resulted in a late departure of our flight. What a lame excuse!!! <br> <br>This is what the EU regulation says about the extraordinary circumstances:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*gje1E1BjXMw8JanE.jpg" /></figure><p>I am pretty sure that the “adverse weather conditions” should occur either at the departure airport or at the destination or somewhere in between to be counted as an “extraordinary circumstance”, and as far as I know, Egypt is nowhere in the route from Prague to Valencia.<br> <br>But it gets better (or more ridiculous), the next day, every single passenger of the flight got the following apology by email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/0*R3aso9gdKqwdzlBh.png" /></figure><p>This seems so fake to me, the only reason they sent that was because they hope that maybe someone would not claim their compensation after reading it.<br>Money is the only apology I want, hence, apology not accepted.</p><p><strong>UPDATE from September 2020<br></strong>Almost six months after the flight (in September 2018), I finally submitted a claim through <a href="https://wingstoclaim.com/">WingsToClaim.com</a> which is not a very popular site, my first choice was Airhelp.com but they literally told me that they were too busy at that time…</p><p>Anyway, I am satisfied with WingsToClaim service, coincidentally, the are based in Valencia, my hometown. The communication with them was fluent and mainly by telephone, I never had to wait in order to talk to a customer service employee and they informed me about the status of our claim. Please note that I talked to them in Spanish, but they should be able to speak English as well.</p><p>When WingsToClaim contacted SmartWings, they refused to pay, so they took them to court and finally, SmartWings had to pay 400 euros for each of the 5 passengers, i.e., 2000 euros in total.</p><p>The net amount which I received on my bank account this month (September 2020) was 1153 euros though, because WingsToClaim keeps a 25% fee plus 21% VAT and on top of that, since they had to take Smartwings to court, they charge a 40 euros fee per each passenger (plus 21% VAT), so the number match my calculation:<br>2000 - 0.25 * 2000 * 1.21 - 5 * 40 * 1.21 = 1153</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*9x1wxlmqzN61ZI2qCzm-Lg.png" /><figcaption>After almost 2 and half years, our money arrived, sweet revenge!</figcaption></figure><p>I hope more and more frustrated passengers go through the process of claiming their compensation so that SmartWings and other airlines with similar behavior learn the lesson and stop disrespecting their costumers.</p><p><em>Originally published at </em><a href="http://www.devaction.net/2018/04/apology-not-accepted.html"><em>www.devaction.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=826c1bb4add0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shared Ledger Simulator]]></title>
            <link>https://medium.com/@victorgil_91367/shared-ledger-simulator-f060416cd390?source=rss-d464b56be56------2</link>
            <guid isPermaLink="false">https://medium.com/p/f060416cd390</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[Victor Gil]]></dc:creator>
            <pubDate>Mon, 11 Dec 2017 21:45:50 GMT</pubDate>
            <atom:updated>2017-12-11T23:13:32.198Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been interested in the shared/distributed ledger technology (a.k.a. block chain, a.k.a. the <em>magic</em> behind cryptocurrencies) for more than a year already but recently I had finally put real time and effort into it.<br> <br> And since I believe that the best way to understand something is to get your hands dirty, that is what I have done, after I got a grasp of the core principles (or that is what I thought back then), I decided to code my own shared ledger simulator.<br> <br> Initially, I also considered to look into the main existing code bases (e.g., the source code of the main/official Bitcoin client/miner or Ethereum’s) since they are open source, but seeing code like <a href="https://sourceforge.net/p/bitcoin/code/133/tree/trunk/main.cpp#l1630"><strong>this</strong></a><strong> </strong>put me off… That file is 3229 lines big!!! Plus it is written in C++.<br> Do not get me wrong, I truly believe Satoshi Nakamoto (i.e., whoever hides behind that name) is a genius and also a great software developer, but he/she/they for sure did not have readability as their main priority.<br> <br> I also noticed that some other people had the same idea before but I did not really like how they were implemented, for example: <br>This <a href="https://hackernoon.com/learn-blockchains-by-building-one-117428612f46">one</a> → as pointed out in one of the comments, it does not include the hash of the previous block in the current block… trying to make things simple is not an excuse for this, if you do not have this core feature, then what you have is not a shared ledger implementation at all, period.<br> Or this other <a href="https://medium.freecodecamp.org/how-does-blockchain-really-work-i-built-an-app-to-show-you-6b70cd4caf7d">one</a> is not exactly what I had in mind, I wanted my running code to demonstrate how the algorithm works by generating multiple blocks and transactions in an automated way (more on this later). <br> <br> So I chose Java to code my shared ledger simulator from scratch mainly because it is my favorite programming language and also because it has better support for cryptography than many other languages out there. The complete source is available on <a href="https://github.com/VictorGil/shared-ledger-simulator">GitHub</a> under the MIT license.<br> <strong><br></strong> <strong>Assumptions and compromises made</strong><br> I wanted my implementation to be compliant with the core shared ledger principles while keeping things simple (or at least not too complicated), so these are the main characteristics/assumptions/compromises I made (in no particular order):<br> <br> <strong>1</strong>.- Bitcoin (and Ethereum too) use <a href="https://en.bitcoin.it/wiki/Elliptic_Curve_Digital_Signature_Algorithm">Elliptic Curve Digital Signature Algorithm</a> but I used good old RSA instead. Conceptually there is not much difference, since both rely in the same common concepts such as private key, public key and signature<br> <br><strong>2</strong>.- The addresses are represented as arrays of bytes and each address is unequivocally associated to an RSA key pair. For simplicity, I took the address as the last 20 bytes (or 40 hexadecimal digits/characters) of the public key. <br> <br> <strong>3</strong>.- There are only two type of nodes: full nodes (i.e., miners) and thin clients (which submit transactions). Each node runs on a different thread, analogously to real life where each node runs on a different computer which may be geographically very distant from other nodes.<br> <br> <strong>4</strong>.- To simulate the network, I used two simple abstract data types, <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/master/src/main/java/net/devaction/sharedledgersimulator/network/BlocksInTime.java">BlocksInTime.java</a> and <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/master/src/main/java/net/devaction/sharedledgersimulator/network/TransactionsInTime.java">TransactionsInTime.java</a>. Every thread/node has access to both ADTs:<br> — Full Node (Miners) have “read and add” access to BlocksInTime and “read only” access to TransactionsInTime<br> — Thin clients have “read only” access to BlocksInTime (to check when is safe to consider a submitted transaction as confirmed, e.g., after five blocks, but please note that this functionality has not been implemented yet) and “add” access to TransactionsInTime<br> <br> <strong>5</strong>.- Every transaction must be signed (by a thin client) using the private key associated with the source address of the transaction → this is one of the core principles of shared ledger<br> <br> <strong>6</strong>.- Thin clients do not explicitly set a transaction fee, the difference between total inputs and total outputs of the transaction is considered to be the transaction fee → exactly how Bitcoin works<br> <br> <strong>7</strong>.- The higher the fee is, the higher chance for the transaction to be included in the next block by the <em>lucky</em> miner, i.e., transactions with higher fee have higher priority.<br> <br> <strong>8</strong>.- Signing a transaction is implemented with the help of the standard Java serialization, once we get the array of bytes which represent an unsigned transaction, the bytes are signed using the private key of the thin client and the resulting signature is added (“<em>stamped</em>”) to the signed transaction object, you may check <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/eb5d7e911d185441317763de1fc25728bb93e82d/src/main/java/net/devaction/sharedledgersimulator/rsa/RsaKeyPair.java#L78">this method</a>.<br> <br><strong>9</strong>.- The number of maximum transactions than can fit in a block is set through a property in the <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/master/src/main/resources/application.properties">properties file</a> (Bitcoin limits the size of the block in bytes but I just limited the number of transactions for the sake of simplicity), so if the transaction fee is too low, it may take longer (or even forever) to be included in a mined block but please note that submitted transactions never really expire, more info <a href="https://en.bitcoin.it/wiki/Transaction_expiration">here</a>.<br> <br><strong>10</strong>.- Every block (except the genesis block) must contain the hash of its parent block → this is another core principle of shared ledger<br> <br> <strong>11</strong>.- Every block contains the address of the miner, and includes a transaction with no source address which sends the miner’s reward amount to that address, this is how new coins are being generated, thus the origin of every single coin/amount in every transaction can be traced to a <em>mining reward</em> transaction.<br> <br> <strong>12</strong>.- In Bitcoin, the number of coins is limited (to 21 million) and at some point in the future (the year 2140) no more bitcoins will be created from mining, this limitation was not implemented in the simulator for the sake of simplicity and because I did not considerate it essential. <br> <br><strong>13</strong>.- In the current implementation, a very basic (but reliable) Proof of Work mechanism is used when generating new blocks, I may add other mechanisms (e.g., Proof of Stake or something else which I already have in mind) in the future. <br> <br><strong>14</strong>.- The most difficult part to come up with was the algorithm for a full node (miner) to decide which block is the latest valid one from the longest chain (yes, there are multiple chains of blocks, I actually think that the term block chain is somehow misleading, <strong>block tree</strong> would be a better one). I had to put into practice a little of Graph theory (more precisely Depth First Search/Traversal algorithm) to calculate the distance (a.k.a. depth in a tree node) of a given block from the Genesis block. <br> This may actually deserve its own post.<br> <br><strong>15</strong>.- There is no single data structure to permanently store the balances (i.e., what amount is available on what address/account). Balances are calculated on the fly by checking every single transaction present in a block chain → this is another core principle of share ledger which my implementation honors.<br> <br> <strong>16</strong>- I added several “sleep statements” across the source code in order to slow down the execution for didactic purposes.<br> <br> <strong>Running the code</strong> <br>So let’s see the code in action.<br>In the this <a href="https://youtu.be/FxrYhWDksEo">video</a> you can see the execution of the <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/master/src/test/java/net/devaction/sharedledgersimulator/client/FullNodeAndThinClientTester.java">FullNodeAndThinClientTester.java</a> which showcases two full nodes (miners) competing to generate the next block and also two thin clients submitting transactions to the network which then are included into the blocks.<br>You may execute the code yourself in your computer and even modify the timings of the main events (e.g., when to start a miner to join the network and start mining) or change the value of the <a href="https://github.com/VictorGil/shared-ledger-simulator/blob/master/src/main/resources/application.properties">properties</a> to see how that impacts the outcome of the execution (i.e., the blocks generated).</p><p>Also, please let me know in case I missed anything or any of my assumptions or compromises made are not accurate or properly explained.</p><p><em>Originally published at </em><a href="http://www.devaction.net/2017/12/shared-ledger-simulator.html"><em>www.devaction.net</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f060416cd390" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>