<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Piers Wermbter on Medium]]></title>
        <description><![CDATA[Stories by Piers Wermbter on Medium]]></description>
        <link>https://medium.com/@onepiers?source=rss-79bb666fc3b2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sJbjHEN2VyZ_yEc7DTWiNA.jpeg</url>
            <title>Stories by Piers Wermbter on Medium</title>
            <link>https://medium.com/@onepiers?source=rss-79bb666fc3b2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:27:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@onepiers/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Event sourcing using gRPC streams as Kafka replacement]]></title>
            <link>https://medium.com/coyo-tech/event-sourcing-using-grpc-streams-as-kafka-replacement-d5655bf84ef9?source=rss-79bb666fc3b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5655bf84ef9</guid>
            <category><![CDATA[translation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[language]]></category>
            <category><![CDATA[events]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Piers Wermbter]]></dc:creator>
            <pubDate>Thu, 19 Nov 2020 14:25:41 GMT</pubDate>
            <atom:updated>2020-11-19T14:25:41.962Z</atom:updated>
            <content:encoded><![CDATA[<p>Back in May, we started our <a href="https://medium.com/coyo-tech/moving-to-microservices-i18n-coyo-part-1-995a9d87020b">long-awaited journey towards microservices</a> that are self-contained and not dependent on synchronous calls to other services. To operate independently, the services will need a local representation of the domain data from other services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TBRPnhv9lhNEVadcrtbIyg.png" /><figcaption>Overriding message keys</figcaption></figure><p>One example is the i18n service from the first part of this series. It needs to know about all tenants and their URL to provide customer overridden message keys (as you can see above) mixed in with our default values to the frontend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*iphUGSQH4W70Hp8PP1i8lg.png" /></figure><p><strong>The concept</strong></p><p>One concept to deal with this is <a href="https://microservices.io/patterns/data/event-sourcing.html">EventSourcing</a>. A producing service publishes state change events. These events reflect the made changes to the underlying domain data and can be subscribed to by consuming services that need a local representation of the data to be able to act on their own. Most implementations of the EventSourcing concept heavily rely on a central component. <strong><em>Apache Kafka</em></strong> is a popular choice for this.</p><p>While this is a very desirable target architecture for us, it is currently not implementable due to the operational model of our product here at COYO. Next to our SaaS offerings we also offer an on-premises solution that is installable on the customer&#39;s own metal or virtual machines. Our customers in that model range from ten to over ten-thousand users. Still, we would need a reliable Kafka component that persists the events without gaps or outages otherwise, services are not in sync. That being said, adding Kafka to the operational model would be a nightmare for most of our customers as it would raise the operational costs by a lot.</p><p>With Kafka ruled out we made our minds up about a decentralized solution, not relying on that one central component, and which better fits our restrictions. At the same time, it removes the need for one of our product teams to own and maintain the additional central component. One decentralized approach is to work with Atom feeds where every service publishes a feed of changes that others can subscribe to and replicate the data. Those feeds are backed by a persistence layer in the service itself. Each service needs to reimplement the refresh and event persistence logic itself and in an ideal world provide the feed in an aligned architecture-wide format. While we liked the overall approach writing Atom feeds felt kind of <em>old-school</em>. So we had a look at <strong><em>gRPC</em></strong>.</p><p><strong>The gRPC “EventSourcing” contract</strong></p><blockquote>gRPC is a modern open source high performance <a href="https://en.wikipedia.org/wiki/Remote_procedure_call">RPC</a> framework that can run in any environment. It can efficiently connect services in and across data centers with pluggable support for load balancing, tracing, health checking and authentication.¹</blockquote><p>gRPC is based on ProtoBuf and is build on contracts that each server or client can implement and rely on. It is based on HTTP/2 and therefore also supports server-side streaming. A perfect solution to build a subscription endpoint that delivers change events to all interested services. Our gRPC service definition looks like this:</p><pre>[...]</pre><pre>message EventSubscribeRequest {<br>    repeated EventTypeSubscribeRequest types = 1;<br>    bool includePastEvents = 2;<br>}</pre><pre>message EventTypeSubscribeRequest {<br>    string typeName = 1;<br>    int64 startingId = 2;<br>}</pre><pre>message EventResponse {<br>    int64 id = 1;<br>    string eventType = 2;<br>    string entityType = 3;<br>    string entityId = 4;<br>    google.protobuf.Timestamp time = 5;<br>    string tenantId = 6;<br>}</pre><pre>[...]</pre><pre>service EventService {<br>    rpc subscribeEvents(EventSubscribeRequest) returns (stream EventResponse) {};<br>    rpc getEvent(StateRequest) returns (StateResponse) {};<br>}</pre><p>A producing service will store change events to a certain entity type in its database with an incremental id. These events are then sent out to all-consuming services in the order they are raised. If such a connection is broken up, the consumer can reconnect with its last received event id and continue consumption from there on.</p><p>After receiving an event, it is up to the consuming service if it is interested in the current state of the domain data or not. We at COYO do not follow the principle of event sourcing thoroughly though, as we just emit the event without containing any change information, to allow for speedy event distribution. In case of interest, the consumer can get the current state of the entity through the second endpoint and will receive a JSON representation of the domain data which can be locally stored.</p><pre>message StateRequest {<br>    int64 id = 1;<br>}<br><br>message ErrorResponse {<br>    string type = 1;<br>    string message = 2;<br>}<br><br>message StateResponse {<br>    google.protobuf.Struct content = 1;<br>    string tenantId = 2;<br>    ErrorResponse error = 3;<br>}</pre><p>Out of these contracts, we generate libraries that can be used in NodeJS and Java to write client and server implementations of the eventing interfaces.</p><p><strong>Pros and cons</strong></p><p>We were able to build a version of the EventSourcing concept that does not rely on Kafka or any other central component, which is a huge benefit and necessity as mentioned earlier. Also having an “API contract” through the proto definitions made life easier. But the approach also has its downsides.</p><ul><li>Consumers need to strictly follow the order of the events and can only consume one event at a time when they fetch the states manually in order. <em>(Something we needed to accept as we wanted to keep the events as slim as possible due to the still existing monolithic main application.)</em></li><li>There is no historic state of the entities, just a stream of status changes in form of event names, and one can only fetch the current state of the entity.</li><li>We experienced some technical difficulties scaling the implementation to millions of events.</li><li>Every consumer and server need to reimplement the logic of consuming and providing events. <em>(We are currently working on a spring library and a client to cope with this.)</em></li></ul><p>Especially the second to the last point of the “contra” list is worth sharing in terms of pitfalls that can happen while putting gRPC streaming under the load of six million events both on the client (NodeJS) and server (Java) side. So stay tuned for the next part of this series.</p><p>[1] gRPC Authors. (2020). <em>gRPC </em><a href="https://grpc.io/">https://grpc.io/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5655bf84ef9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coyo-tech/event-sourcing-using-grpc-streams-as-kafka-replacement-d5655bf84ef9">Event sourcing using gRPC streams as Kafka replacement</a> was originally published in <a href="https://medium.com/coyo-tech">Haiilo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hackathon: Prototyping a linear recommendation engine]]></title>
            <link>https://medium.com/coyo-tech/hackathon-prototyping-a-linear-recommendation-engine-346b99f5c9fa?source=rss-79bb666fc3b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/346b99f5c9fa</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Piers Wermbter]]></dc:creator>
            <pubDate>Wed, 17 Jun 2020 08:01:24 GMT</pubDate>
            <atom:updated>2020-06-17T08:01:24.529Z</atom:updated>
            <content:encoded><![CDATA[<p>COYO, being a mix of a social network and a classic intranet, is very depended on the content users create to attract and retain an active base of users. Over time that content will grow to an amount that finding relevant and useful information, especially for new users, can become cumbersome in a jungle of “relevant” information. A search might help, but let’s be realistic. It does not, if you do not even know what to search for. Recommendations to the rescue!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lip42_4U9aCfBFcTlR8r-A.png" /></figure><h4>The idea</h4><p>COYO’s static content is mostly provided by editors through blog articles in so called pages. These pages can be subscribed by company members or might be mandatory to follow.</p><p>The created blog articles can then be promoted through a news-feed to existing subscribers. But newly joining members of the company have missed those promotions and won’t read the article.</p><p>Let’s look at how you as new employee may behave in your first moments on the platform:</p><p>You might be looking at the overview of pages and will find some interesting content next to the must-follow pages, but in the amount of over a hundred pages you will not immediately find the page that might be most fitting to you.</p><p>After you have subscribed an amount of pages, you will receive new articles in your news feed the next days and will start to read them, even though you are still new to the company and might not understand every part of the article as the broader scope is missing.</p><p>In this months Hackathon we wanted to tackle exactly those two issues in a prototypic way:</p><blockquote>Suggest pages to the user that he might like.</blockquote><blockquote>Suggest similar articles to the user that will be of interest to read to help him understand more about the topic.</blockquote><h4>The implementation</h4><p>To be as fast as possible in the context of a three day Hackathon of which half a day had already passed, we decided to prototype a new microservice. For our new “recommendation engine” we first chose Micronaut.io as a JVM based micro-framework, then bootstrapped the project by integrating it into our deployments and finally added JWT auth to the required endpoints. The rest of first day was spend to research ways how to actually build a recommendation engine in a Java context and brainstorming which data we would like to take for the recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNb24y9Qg-GXkcOfxpBtoA.jpeg" /></figure><p>Based on that brainstorming we integrated Apache Mahout’s linear in-memory taste algorithm into our project and started to fetch data from the required data sources via COYO’s gRPC service communication. Learn more in this <a href="https://medium.com/coyo-tech/moving-to-microservices-i18n-coyo-part-1-995a9d87020b">series</a> shortly. For our first use case we implemented a user based recommendation model in Mahout based on all page subscriptions of all users and for the second use-case an item based recommender for all blog articles and the user liking those.</p><p>It does not sound like a lot of implementation work what we did and it definitely is not, but for a group of people that never had touch points with recommendations we spend most of our time playing around and building the front-end for the newly created recommendation endpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9lMTAFqcCcLv07-nPFc6Vw.png" /><figcaption>How pages are recommended</figcaption></figure><h4>The result</h4><p>COYO is now, in a very prototypic way, able to recommend you pages you like by comparing your follows with those of others and picking the ones you might like. Looking at the graphic, the algorithm will first try to find users that match your existing structure of followed pages (2,3) the most. In this simple example that would be users three and four. You will then receive those pages as a suggestion that those two follow on top of two and three being 4 and 5. While this is a very simple example, you might imagine that in a bigger dataset with a hundreds of pages and even more users you will find much more suggestions that are then weighted by how many others users follow them on top.</p><p>The blog article recommender works with the same two dimensional way, by building a table of which users liked which blog articles and then suggests you those articles that are most equal in terms of likes. Think of Amazons “Users who bought this also bought…” for COYO it would be “Users who liked the article also liked this one …”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*chJTDyl9k9AxJDRkHtem-A.png" /></figure><p>In the end we are very happy with the result we were able to produce in the three days with our limited knowledge: Providing a working example that is capable of not only recommending you pages but also blog articles, heaving the experience for new users to the next level.</p><p>Check out what others did in this times COYO Hackathon in our article series shortly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=346b99f5c9fa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coyo-tech/hackathon-prototyping-a-linear-recommendation-engine-346b99f5c9fa">Hackathon: Prototyping a linear recommendation engine</a> was originally published in <a href="https://medium.com/coyo-tech">Haiilo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Moving to microservices: i18n @ COYO Part 1]]></title>
            <link>https://medium.com/coyo-tech/moving-to-microservices-i18n-coyo-part-1-995a9d87020b?source=rss-79bb666fc3b2------2</link>
            <guid isPermaLink="false">https://medium.com/p/995a9d87020b</guid>
            <category><![CDATA[internationalization]]></category>
            <category><![CDATA[translation]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[language]]></category>
            <category><![CDATA[engineering]]></category>
            <dc:creator><![CDATA[Piers Wermbter]]></dc:creator>
            <pubDate>Tue, 26 May 2020 10:53:00 GMT</pubDate>
            <atom:updated>2020-05-26T10:53:00.991Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fgrntHYYJ2ANmw_grLAbLw.png" /></figure><p>COYO, being a software that was rapidly developed over the course of multiple years, is currently not only facing the challenge of a growing code base but also of a growing team, well beyond 25 developers, working on the same project. At the same time, the international expansion of the company itself asks for more languages to be supported in the product, next to German and English. Adding these to an already heavy frontend did not seem like a feasible option. Therefore the decision was made to introduce a central internalisation service that provides message key bundles for all services.</p><p>Follow our story in this two part series, on extracting the first service out of the existing product. In the first part we are going to focus on extracting the bundles from the other services into a central “key store”.</p><p>As mentioned before, the main goal was to provide a central store for all message key translations that delivers all interface translations in a single call for a dependent service, in a format that can be presented to the user without the need of any calculation - think of necessary fallbacks to the system language or English.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66ae00a707742efdd729e956cd75bd52/href">https://medium.com/media/66ae00a707742efdd729e956cd75bd52/href</a></iframe><p>Coming from a monolithic application engineered with Spring Boot 2 and a frontend application that is gradually moving from AngularJS to Angular, the general skillset of our development team can be pinned to JavaScript/TypeScript and Java. The possibilities with these languages are nearly endless but can be narrowed down to the following popular frameworks, in the early state of the microservice transition that we are in:</p><ul><li>Spring Boot</li><li>NodeJS with Express</li><li>NestJS</li></ul><p>To have a very lightweight application with typing support the decision was made for NestJS as it provides typing (via TypeScript) with a Spring Boot like application architecture on top of Express + NodeJs and provides a fast learning curve for all developers of a team being frontend or backend.</p><p>After deciding on the technology and setting up the project including CI, health checks, logging, management endpoints, the goal was to move the existing message key translations from being bundled into the frontend application next to the actual components to one central file that is delivered by the new service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XLho2oF9tPar4AtZxd6x2g.png" /><figcaption>Figure 1: Project structure (outline)</figcaption></figure><p>While the implementation is rather simple as we just needed to remove all the files and merge them into a central one and then statically serve it from a NestJS controller (Figure 1). The benefit of this is mainly visible on the second look, as it not only removes 1.5 MB of message keys from the frontend bundle but also made us introduce a central workflow tool (Zanata) to work with translation agencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vuhk108drR47ynAtMweEkg.png" /><figcaption>Figure 2: Auto push of english keys</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJ5K6h9CSN9fCrumL0GrPg.png" /><figcaption>Figure 3: Pulling of none english translations</figcaption></figure><p>A central tool also allows developers to only work on the English translations in the project and each change is automatically pushed (Figure 2). All other translations are pulled from Zanata (Figure 3). A major benefit especially to international colleagues that do not speak German (the second supported language at that point) is the possibility to add more languages easily in the future without the need of code changes.</p><p>At that point the first version of the service went live and already served the bundles for the frontend and in a second step for the backend components (e.g. push and email notifications). Still we haven’t reached our goal yet as there are still multiple calls and calculations needed in the frontend to present the user language, as the product allows every customer to overwrite each single translation for all services and these are still residing in the monolith.</p><p>That fact needs a lot of technical ground work due to the needed data synchronisation between services. To see how we coped with this challenge, stay tuned for part two.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=995a9d87020b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coyo-tech/moving-to-microservices-i18n-coyo-part-1-995a9d87020b">Moving to microservices: i18n @ COYO Part 1</a> was originally published in <a href="https://medium.com/coyo-tech">Haiilo</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>