<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Smur89 on Medium]]></title>
        <description><![CDATA[Stories by Smur89 on Medium]]></description>
        <link>https://medium.com/@smur89?source=rss-a47d2b47f232------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*69JHa6PIDHoQYff4</url>
            <title>Stories by Smur89 on Medium</title>
            <link>https://medium.com/@smur89?source=rss-a47d2b47f232------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 19:42:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@smur89/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CRUD to Event-Driven Retention with CDC]]></title>
            <link>https://medium.com/swissborg-engineering/crud-to-event-driven-retention-with-cdc-23d0a0f452a7?source=rss-a47d2b47f232------2</link>
            <guid isPermaLink="false">https://medium.com/p/23d0a0f452a7</guid>
            <category><![CDATA[swissborg]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[fintech]]></category>
            <dc:creator><![CDATA[Smur89]]></dc:creator>
            <pubDate>Tue, 16 Dec 2025 15:12:10 GMT</pubDate>
            <atom:updated>2025-12-16T15:12:10.425Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uI6wfSb5BkKmN6cfcVp29A.png" /></figure><p>At SwissBorg, as a financially regulated entity, we are required to retain records for a minimum of five years once a business relationship concludes. Recently, the first of our community members crossed this five-year threshold and reached the end of their data retention period, presenting the challenge of managing these records effectively.</p><p>Coordinating the handling of sensitive data across our distributed services highlighted the need for a dedicated solution to manage the data retention lifecycle of our users. Consequently, we built a service that now sits at the heart of our compliance efforts. It leverages a simple CRUD interface, underpinned by PostgreSQL tables, triggers, and constraints, while integrating with Kafka and leveraging Change Data Capture (CDC) via Debezium.</p><p>In this article, we will explore how we built the core of our data-retention functionality on a simple CRUD foundation. By adding a few simple building blocks — Change Data Capture (CDC), database triggers, and cron jobs — we transformed a simple table into an event-driven orchestration hub that propagates retention status changes to downstream services. This approach ensures regulatory compliance while keeping the system simple, reliable, and scalable, turning everyday operations into powerful automation:</p><ul><li><strong>Simple CRUD operations</strong> become distributed events via CDC.</li><li><strong>Basic database triggers</strong> provide zero-code auditing and enforce constraints.</li><li><strong>Standard cron jobs</strong> orchestrate complex multi-service workflows.</li><li><strong>Plain SQL updates</strong> trigger enterprise-wide data lifecycle management.</li></ul><h3>Regulatory Context and Requirements</h3><p>Companies offering services to individuals in the EU must comply with the General Data Protection Regulation (GDPR) for managing personal data. This includes obligations around data retention (Article 5.1.e) and the right to erasure (Article 17), which we, as an entity serving European users, are required to comply with.</p><p>However, as a regulated Virtual Asset Service Provider, we face additional obligations that complement GDPR requirements. While GDPR mandates that personal data be kept no longer than necessary and grants users the right to erasure, financial regulations (e.g. in France and Estonia) require that certain data be retained for the duration of the business relationship and for a minimum of five years following its termination.</p><p>The data involved is highly sensitive, and the expiration of a retention period is a volatile event: incorrect handling could result in unrecoverable data loss. We must therefore act deliberately, employing safeguards to protect data while fully meeting our regulatory responsibilities.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*cc56qG0XoDdD5N0yI3gnSA.png" /><figcaption>Data Retention Service Architecture</figcaption></figure><p>At its core the data-retention-service is a simple CRUD service managing a single table; retention_info which stores each user’s data retention details. The table captures the following key fields:</p><ul><li><em>status</em>: current state of the retention period (PendingExpiry, Expired)</li><li><em>expires_at</em>: timestamp indicating when the retention period ends</li><li><em>retention_policy</em>: either Default (5 years) or Extended</li><li><em>end_of_business_relationship</em>: date when the user’s relationship concluded</li><li><em>created_at</em>, <em>updated_at</em>: timestamps for record tracking</li><li><em>updated_by</em>: the author of the last change</li><li><em>comment</em>: optional justification for manual updates</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*MrMcic319YJjKEiffpFBFA.png" /><figcaption>retention_info table structure</figcaption></figure><p>All functionality of the service is built around this table, which can be updated either via gRPC requests or in response to Kafka events.</p><p>By leveraging Change Data Capture (CDC), we extend this simple CRUD model into an event-driven system. Each database change triggers a corresponding domain event — for example, notifying when a user’s retention period transitions to Expired — allowing downstream services to react appropriately while maintaining architectural simplicity.</p><h3>Account Closure to Data Expiry</h3><p>When an account is closed, the upstream system emits an AccountClosed event, which the data-retention-service consumes. Upon receiving this event, the service sets a default retention period (typically five years) and emits a retention-info-updated event with a status of pending-expiry.</p><p>Once a retention period lapses, a daily cron job marks the data as expired, which in turn triggers an expired event for each user. Downstream systems then consume these events and take appropriate actions, such as redacting, nullifying, or deleting data, based on their respective data handling policies.</p><h3>Change Data Capture (CDC)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*kQ80-7xOODhbG1_KPJgPJw.png" /><figcaption>CDC Flow</figcaption></figure><p>By leveraging <strong>Change Data Capture (CDC)</strong> on our PostgreSQL table, every operation automatically becomes a domain event broadcast downstream. Simple database operations can now drive complex, coordinated workflows across our entire backend.</p><p><strong>Debezium </strong>enables this by reading database changes from PostgreSQL’s <strong>Write-Ahead Log (WAL)</strong>, transforming each relevant change in the retention_info table into a domain event, defined in <strong>Protocol Buffers</strong>, and emitting it to the data-retention-service event bus. This <strong>Change Data Capture (CDC) </strong>approach enables near real-time, auditable event propagation without the overhead of a separate outbox table or dedicated kafka publishing, ensuring that no retention updates are missed.</p><p>By relying on <strong>Kafka</strong> and CDC instead of the traditional outbox pattern, we gain several key advantages:</p><ul><li><strong>Reduced overhead</strong> — no additional outbox table, insert logic, flag marking, or cleanup process.</li><li><strong>Latency</strong> — events are emitted as soon as they are committed to the WAL, enabling near-instant downstream reactions.</li><li><strong>Increased Fault tolerance</strong> — Debezium provides exactly-once semantics and offset management, ensuring safe recovery from failures without event duplication.</li><li><strong>Monitoring and alerting</strong> — our existing Kafka observability stack (metrics, lag monitoring, and alerting) applies directly to CDC pipelines, reducing operational complexity.</li><li><strong>Auditability</strong> — since the WAL is the source of truth, emitted events reflect exactly what occurred in the database, ensuring regulatory traceability.</li></ul><h3>Scheduled Expiration with Cron Jobs</h3><p>To ensure retention periods are enforced consistently, the data-retention-service runs a daily cron job that simply collects all rows with status = pending-expiry where the expires_at date has passed. The cron job updates the record’s status to expired directly on the database. This is possible since data integrity is enforced at the database level (as we’ll see later)</p><pre>UPDATE retention_info<br>SET updated_at = NOW(),<br>updated_by = &#39;system&#39;,<br>status = &#39;expired&#39;<br>WHERE status = &#39;pending-expiry&#39;<br>AND expires_at &lt;= NOW()<br>AND end_of_business_relationship &lt;= current_date - interval &#39;5 years&#39;;</pre><p>The cron job is implemented as a Kubernetes CronJob that uses a PostgreSQL client image to perform the update directly within the database in a single transaction. The job runs with access to the services’ database credentials through a provided secrets volume.</p><p>The job is configured with concurrency policy <strong>Forbid</strong> to prevent overlapping runs.</p><h3>Auditing Changes with Database Triggers</h3><p>In addition to emitting domain events via CDC, the <strong>data-retention-service</strong> maintains a complete audit trail of all retention changes using a <strong>database trigger</strong>. Every insert or update to the retention_info table automatically writes a corresponding record to retention_info_history, capturing the state of the record at that point in time.</p><p>This mechanism ensures that the service remains <strong>fully auditable and transparent</strong>, providing compliance teams and internal reviewers with a record of all changes. It also allows for <strong>event replayability</strong>, as the history table mirrors the evolution of each record over time. By separating operational data from historical logs, the main table stays concise and optimised for day-to-day operations, while the history table preserves a complete trace for governance and review.</p><pre>-- Function to store the history of data_retention changes<br>CREATE OR REPLACE FUNCTION record_data_retention_history() RETURNS trigger<br>    LANGUAGE &#39;plpgsql&#39; AS<br>$$<br>BEGIN<br>  INSERT INTO data_retention_history(user_id, status, expires_at, retention_policy, end_of_business_relationship, created_at, updated_at, updated_by, comment)<br>  VALUES (NEW.user_id,<br>            NEW.status,<br>            NEW.expires_at,<br>            NEW.retention_policy,<br>            NEW.end_of_business_relationship,<br>            NEW.created_at,<br>            NEW.updated_at,<br>            NEW.updated_by,<br>            NEW.comment);<br>    RETURN NEW;<br>END;<br>$$;</pre><pre>CREATE TRIGGER data_retention_post_update_trigger<br>    AFTER INSERT OR UPDATE<br>    ON data_retention<br>    FOR EACH ROW<br>EXECUTE PROCEDURE record_data_retention_history();</pre><h3>8. Ensuring Data Integrity and Compliance</h3><p>To prevent accidental or unlawful data loss, the data-retention-service enforces <strong>strict constraints at the database level</strong> via a PostgreSQL trigger on the retention_info table. The trigger raises exceptions when updates or inserts violate the rules, ensuring sensitive user data cannot be deleted or altered improperly.</p><p>Key constraints include:</p><ol><li>expires_at ≥ now() + 1 month<br>Enforces a minimum grace period before data can expire, preventing premature deletion.</li><li>status = ‘pending-expiry’ on creation<br>Ensures records cannot be inserted as already expired.</li><li>Prevent expiration if expires_at has not passed<br>Prevents records from being marked as expired before the actual expiry date</li></ol><p><strong>Why use database-level enforcement</strong></p><ul><li>Enforces <strong>immutability of critical rules</strong> independently of the application layer.</li><li>Protects against <strong>accidental or malicious direct database updates</strong>.</li><li>Ensures compliance with GDPR and financial regulations by making premature deletions impossible without explicit trigger removal.</li></ul><p>By placing these safeguards in the database, the data-retention-service provides <strong>robust, auditable enforcement of retention rules</strong>.</p><h3>9. Downstream event consumption</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*bGjtin8sUc78ih4bRR8oQw.png" /><figcaption>Event Consumption Flow Diagram</figcaption></figure><p>When a user’s retention period expires, the data-retention-service emits a <strong>DataRetentionInfoUpdated</strong> event with status = expired. Downstream services storing <strong>Personally Identifiable Information (PII) </strong>must consume this event and delete or anonymise all associated user data in a GDPR-compliant manner. Services should handle events idempotently, as Kafka may deliver the same event multiple times due to retries or replay scenarios. Services may also verify the user’s status via the data-retention gRPC API before taking irreversible actions. While the event is fire-and-forget, teams are expected to retry on failure and can emit downstream confirmation events to improve observability and traceability, ensuring consistent, auditable, policy-driven retention across services.</p><h3>10. Documentation</h3><p>To ensure transparency and facilitate compliance reviews, every service that handles sensitive data undergoes regular auditing for <strong>Personally Identifiable Information (PII)</strong>. Services identified as containing PII are annotated in our <strong>Backstage (</strong>internally named Merlin<strong>) </strong>with metadata such as the <strong>last audit date</strong>, making it easy for compliance teams, auditors, and internal reviewers to track and verify the state of sensitive data.</p><p>Each audited service also exposes a <strong>GDPR.md file</strong>, hosted in our <strong>Backstage</strong>, which documents:</p><ul><li>All GDPR-relevant data fields.<br>How each data field is handled when a retention period expires or when a deletion request is processed.</li><li>The precise location of each data point across databases, projections, and journals.</li></ul><p>This standardised documentation ensures operational transparency, reduces compliance risk, and provides a reliable reference during audits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xf49sBkN7xVOU5DzMDQsMA.png" /></figure><p>By maintaining this level of documentation, we provide a clear, auditable map of data handling practices to both internal and regulatory stakeholders, ensuring that data-retention policies are transparent, enforceable, and verifiable.</p><h3>11. Conclusion</h3><p>The data-retention-service demonstrates that you don’t need complex frameworks or sophisticated architectures to build powerful, event-driven systems. By composing simple software architecture building blocks together, we can orchestrate complex workflows across an entire platform while ensuring auditability and compliance. The sophistication emerges from composition, not complexity.</p><p>Key lessons from this implementation include:</p><ul><li><strong>Database constraints:</strong> Enforcing rules at the database level prevents accidental or unlawful changes, providing a safety net independent of application logic.</li><li><strong>CDC powers event-driven workflows:</strong> transforms local operations into distributed workflows</li><li><strong>Transparent auditing and documentation are critical:</strong> Maintaining clear records of PII handling ensures regulatory alignment and builds trust.</li><li><strong>Simplicity accelerates delivery:</strong> Simple building blocks enables rapid, reliable delivery</li></ul><p>This approach is now being applied to more complex services, where CDC-based events drive complex lifecycle triggers; allowing services to respond to their <strong>own change events</strong>. The data-retention-service demonstrates that a modest CRUD-based system, when combined with CDC, can form the backbone of reliable, event-driven, and compliance-critical workflows.</p><p>For teams facing similar regulatory or operational challenges, this example illustrates how CDC can transform a simple service into a <strong>powerful compliance tool</strong>, ensuring that sensitive data is handled consistently, auditable end-to-end, and in alignment with legal requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23d0a0f452a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swissborg-engineering/crud-to-event-driven-retention-with-cdc-23d0a0f452a7">CRUD to Event-Driven Retention with CDC</a> was originally published in <a href="https://medium.com/swissborg-engineering">SwissBorg Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On the road to Scala 3]]></title>
            <link>https://medium.com/swissborg-engineering/on-the-road-to-scala-3-99e5cc9d0e4?source=rss-a47d2b47f232------2</link>
            <guid isPermaLink="false">https://medium.com/p/99e5cc9d0e4</guid>
            <category><![CDATA[scalafmt]]></category>
            <category><![CDATA[scala-3]]></category>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Smur89]]></dc:creator>
            <pubDate>Fri, 26 May 2023 11:14:58 GMT</pubDate>
            <atom:updated>2023-05-26T11:14:58.606Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/304/0*dQrWZQqoHHtqF3cB.png" /></figure><p>Like many Scala developers, at SwissBorg we’re excited about migrating to Scala 3. Although we’re not in a position to fully migrate to Scala 3, mostly due to 3rd party libraries not yet being available for Scala 3 and partly because we’re not prepared for the inevitable “braces or no braces?” debate.</p><p>We started to look at how we might make this transition easier for ourselves when the time comes to migrate; reducing the uncertainty around runtime errors and performance, as most of our code will be already adapted.</p><p>There’s nothing groundbreaking to be discussed here, and the official <a href="https://docs.scala-lang.org/scala3/guides/migration/compatibility-intro.html">Scala 3 Migration Guide</a> is a great resource for this. However, without rewriting the migration guide, we wanted to share our approach and experiences nonetheless.</p><h3>Scala 2.13</h3><p>A prerequisite is to have already updated to Scala 2.13.x; the official standard library for Scala 3. From here it is rather easy to port the code to align more with Scala 3, assuming the following additional <a href="https://docs.scala-lang.org/scala3/guides/migration/tutorial-prerequisites.html">prerequisites</a> are met:</p><ul><li>It must not depend on a macro library that has not yet been ported to Scala 3.</li><li>It must not use a compiler plugin that has no equivalent in Scala 3.</li><li>It must not depend on scala-reflect.</li></ul><p>Here’s our first taste of the compatibility improvements we get with Scala 3; we don’t need to migrate everything all at once. It is both backwards and forwards compatible and 2.13 &lt;-&gt; 3 cross-compilation actually works quite well on JVM with the help of the -Ytasty-reader option. This means we can already start to mix Scala 3 dependencies with our existing 2.13.x dependencies.</p><p>The exception here is our macro-based libraries, which need to be rewritten. For 3rd party macro-based libraries, we can temporarily put the code generated by them into a separate module and depend on that. It’s possible to create a Scala 3 sandwich of sorts</p><figure><img alt="module A in Scala 3, module B in Scala 2.13 — depends on A, module C in Scala 3 — depends on B" src="https://cdn-images-1.medium.com/max/151/1*gUtajXqbQAbqBRtFxB4Fnw.jpeg" /><figcaption>Scala 3 sandwich</figcaption></figure><p>However, there are already efforts to provide alternatives to some popular macro based libraries. For example, one we use is scala-newtype, which could be easily replaced by Monix’s Newtypes which uses no macros, and no magic based on implicits.</p><h3>Scala 3 dialect for Scala 2</h3><p>The -XSource:3 compiler option is provided in 2.13.x to encourage early migration. It enables some Scala 3 syntax and behaviour such as:</p><ul><li>Most deprecated syntax generates an error.</li><li>Infix operators can start a line in the middle of a multiline expression.</li><li>Implicit search and overload resolution follow Scala 3 handling of contravariance when checking specificity.</li><li>Ability to use &amp; instead of within type ascription</li><li>Ability to use import foo.* and import foo.Bar as Bazsyntax</li><li>Fix for smart constructors in case classes<br>case class Foo private (bar: Bar)<em>still generated apply and copy but now it correctly hides them</em></li></ul><h3>Kind projector underscore placeholders</h3><p>Scala 3.3 dictates the use of the _ symbol for all anonymous type parameters and aligns with Java syntax on the wildcard symbol, which is now ?. This means in Scala 3 syntax, the following expressions would be rewritten</p><ul><li>EitherT[F, String, *] as EitherT[F, String, _]</li><li>ApplicativeError[*[_], Throwable] as ApplicativeError[_[_], Throwable]</li><li>List[_] as List[?]</li></ul><p>In order to make this change with a dependency on <a href="https://github.com/typelevel/kind-projector">kind-projector</a> we need to enable the new syntax explicitly using the compiler flag -P:kind-projector:underscore-placeholders .</p><h3><strong>Rewriting</strong></h3><p>There are a few options when it comes to handling the syntax updates, including the <a href="https://github.com/scalacenter/scala3-migrate">scala3-migrate</a> plugin. However, as we already use <a href="https://github.com/scalameta/scalafmt">ScalaFmt</a> we could instead rely on that to do the heavy lifting for us.</p><p>ScalaFmt provides a dedicated Scala213Source3 dialect for this which allows it to format some of the new syntax backported from Scala 3. Combined with the additional rewrite rules provided for Scala 3, we can have most of the rewrites taken care of by simply running sbt scalafmtAll Test/scalafmtAll</p><p>Here’s how your configuration file might look if you follow a similar approach</p><pre>version = 3.7.3<br><br>runner.dialect = Scala213Source3<br><br>rewrite.scala3.convertToNewSyntax = true<br>rewrite.scala3.removeOptionalBraces = false<br>runner.dialectOverride.allowUnderscoreAsTypePlaceholder = true</pre><h3>Gotcha’s</h3><p>So, where did we trip up along the way?</p><h4>Implicit Resolution</h4><p>Due to an <a href="https://github.com/scala/bug/issues/12437">issue</a> with ScalaJS, the default implicit resolution when using the -XSource:3 has <a href="https://github.com/scala/scala/pull/10012">changed</a> and needs to be explicitly enabled using the -Yscala3-implicit-resolution. This will keep the previous behaviour of implicit resolutions from scala 2.13</p><h4>Macros</h4><p>Macro based libraries need to be rewritten for Scala 3. In many services we make use of such libraries, which prevents us from moving forward with a full upgrade to Scala 3. For example, we use <a href="https://github.com/scalalandio/chimney">Chimney</a> in many services for our data transformations, which relies on Scala 2 macros. In the case of Chimney, this is something which is <a href="https://github.com/scalalandio/chimney/milestone/3">actively being worked on</a>.</p><h4>Manual rewrites</h4><p>Sometimes, we had to manually update some syntax in order to make our code bases compile.</p><p>With the changes for Kind Projector, ScalaFmt by default does not rewrite it correctly. Worse, when we manually updated to the correct style, ScalaFmt would know better and change it back! Eventually we found this <a href="https://github.com/scalameta/scalameta/pull/2811/files">Pull Request</a> which solves the issue by introducing an override in the configuration. After adding</p><pre>runner.dialectOverride.allowUnderscoreAsTypePlaceholder = true </pre><p>to our config, ScalaFmt relented.</p><p>A minor annoyance if you decide to instead opt to go with utilising the -Compile / scalafixOnCompile := true method of rewriting is that it seems to not remove unused wildcard imports.</p><h4>ADTs</h4><p>In the case of ADTs, we noticed that leaf nodes of the ADT are inferred to be the root type. This caused some confusing compilation issues for us.</p><pre>sealed trait Foo<br><br>object Foo {<br>  case class Bar() extends Foo<br>}<br><br>val foo = Foo.Bar() // inferred type = Foo</pre><h4>ScalaPb generated code</h4><p>If you have a module which combines in-house code with generated code, as we had, it is recommended to split the two into separate modules. In this way, the module containing your own code can depend on the module with the generated code.</p><p>For us, this was the case with ScalaPb, and we took the opportunity to clean up and split our module as described above. This module for the generated code does not apply the -XSource:3compiler option and is consumed as a dependency from the other modules.</p><h4>Tooling</h4><p>When we initially performed the migration, we noticed that IDEs had some trouble correctly displaying the code highlights, even though the build was successful. Since then, support has improved a lot and this seems to be less of an issue. At the time, we took the approach of leveraging the fileOverride option and setting rewrite.scala3.convertToNewSyntax to false if it was particularly bothersome.</p><h3>Next Steps</h3><p>We are already working on migrating our core internal libraries to Scala 3, with some of our CI helper scripts utilising Scala 3 already. Many of these libraries are not dependent on incompatible libraries like <a href="https://github.com/estatico/scala-newtype">newtype</a> and <a href="https://scalalandio.github.io/chimney/">Chimney</a>, which will allow us to already begin our migration. This is something we’re actively working on, and we’ll begin our journey in Scala 3 with Scala 2 code style before discussing which syntax changes we want to embrace.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99e5cc9d0e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swissborg-engineering/on-the-road-to-scala-3-99e5cc9d0e4">On the road to Scala 3</a> was originally published in <a href="https://medium.com/swissborg-engineering">SwissBorg Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SwissBorg Notifications]]></title>
            <link>https://medium.com/swissborg-engineering/swissborg-notifications-83d38dbd377f?source=rss-a47d2b47f232------2</link>
            <guid isPermaLink="false">https://medium.com/p/83d38dbd377f</guid>
            <category><![CDATA[notifications]]></category>
            <category><![CDATA[cyrpto]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[swissborg]]></category>
            <dc:creator><![CDATA[Smur89]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 11:23:58 GMT</pubDate>
            <atom:updated>2022-11-10T15:05:11.161Z</atom:updated>
            <content:encoded><![CDATA[<p>How we manage them.</p><p>Notifications are an additional tool we can leverage to communicate with our community outside of social media channels. Moreover, notifications are an important security, educational and marketing tool at our disposal.</p><p>Certain notifications can be used to communicate important information to users even when they aren’t actively using the App. With this feature users don’t need to keep the app open or be on constant alert if waiting for an action, like a deposit to arrive. Instead, they’ll receive a push notification on their device as soon as it arrives.</p><p>From a security standpoint, notifications are crucial. When someone’s account info changes, like their email or phone number, we want to alert them to make sure they know about it in case the change was made by a malicious actor. Notifications also play a role in our verification process whereby a new user receives an SMS challenge code to verify their phone number. When a big news event comes along, like the UST depeg, we also push educational content and warnings to users in an effort to assist them in protecting their assets.</p><p>Finally, we also use notifications in marketing campaigns and for user outreach. These include product announcements, new coin listings, competitions and voting events, to name a few. When choosing what notifications to send, we are conscious of ‘alert fatigue’ and follow a HIVR (Helpful, Interesting, Valuable and Relevant) formula when deciding what content should be sent out.</p><p>We use two main tools to communicate with our users in this way; <a href="https://www.hubspot.com/">HubSpot</a> and an internal solution centred around our <strong>notification-service</strong>. This service was built to provide a central point to manage our notifications ecosystem; being responsible for templating, translations, InApp notifications and managing our 3rd party integrations.</p><p>In this post we dive into our internal solution and show how notifications are created, managed and sent.</p><h3>Notifications Terminology</h3><p>Before we get to the crux of the matter, let’s clarify some terminology and cover how we classify different notification types. We use the term ‘notification’ in a general sense, though a notification can be broken down further into its constituent parts. Notifications terminology is often confused, so for the sake of this article, we will define them as follows:</p><h4>InApp</h4><p>InApp notifications encompass any messages shown to a user inside the app. They are commonly used to help with onboarding, guidance, driving user discovery, and increasing feature adoption.</p><p>For this notification type, the app itself determines how they are styled and displayed. Users of the app will recognise them as the expanded text block displayed on the Portfolio screen and in the Notifications History view. This is our default InApp notification style, and until recently, the only InApp display style we supported; more on this later.</p><figure><img alt="Example of an InApp Notification" src="https://cdn-images-1.medium.com/max/1024/0*A4n7u_nzTHnku_0X" /></figure><h4>Push</h4><p>A push notification is more of a trigger sent out to mobile devices to elicit an action from the app, even if the app is not in the foreground or is completely closed.</p><p>App developers can choose how the app reacts to these triggers. For example, they may preload some data while users are not using the app in order to improve the app’s performance when the user next opens it, or to show a message within the app. At SwissBorg, push notifications are most commonly used to trigger system notifications.</p><h4>System</h4><p>System notifications are controlled by the OS of the device that receives the notification, with the notification behaviour and styling dictated therein. In general, we may only decide on the type of system notification we want, for example, the App Badge, Lock Screen, or Notification Bar. Depending on the system notification, we can provide predetermined parameters for that system notification style, such as tap-action, additional action intents, icon, title and text.</p><p>In SwissBorg’s case, we employ the use of system notifications when sensitive account changes are detected, and when there are update announcements, feature launches, and transaction notifications, among others.</p><figure><img alt="Example of a system notification triggered by a push notification" src="https://cdn-images-1.medium.com/max/1024/0*Oyv_ZxK1o3J6EQGO" /></figure><h4>SMS</h4><p>SMS notifications are generally reserved for communicating with users who cannot be reached via InApp notifications, such as someone who has not yet been onboarded. For example, if a <a href="https://swissborg.com/smart-send">Smart Send</a> transaction is sent to one of these users, we send an SMS notification to their phone number, notifying them of the transaction. In this case, the transaction is tied to their phone number, waiting to be received after onboarding.</p><figure><img alt="Example of a SMS Notification" src="https://cdn-images-1.medium.com/max/1024/0*muUHVvZMYKA4DGEQ" /></figure><h4>Email</h4><p>Email notifications are where we have the most flexibility with regards to notification layout and styling since the messages here can be longer and contain richer content. As such, these notifications are mostly used as Marketing Messages, but are also often used to reinforce the messages in other notification types.</p><p>For emails, we also have the ability to send these using HubSpot rather than directly going through our internal service. Since this is a core product of HubSpot, they offer a more feature rich experience for email marketing messages.</p><figure><img alt="Example of an Email Notification" src="https://cdn-images-1.medium.com/max/395/0*Vjo-MXNqKFw3sPci" /></figure><h3>Notification Topology / Management</h3><h3>Templating</h3><p>Now that we know which notification types are in the tool kit, how do we set the content for them? We start by creating a template with its associated Template Content. This separation between template and its content is done to track versioning history for the content as it evolves. Later we’ll see that each sent notification is linked back to its originating template and template content version. Each time the template content is updated, the template is also updated to point to the latest version.</p><blockquote>“Organisation is the root of all satisfaction, and satisfaction is the root of all happiness.”</blockquote><blockquote>– <a href="https://medium.com/@michael_SB">Our Technical Writer</a></blockquote><p>As the name suggests, the template content allows for templated variables to be used in the content. That is to say, the content is not static. If you’ve ever received an email from a company that began with “Hello {{ firstName }}” then you’ve encountered a templated notification before. It allows us to use templated content, while still keeping the message somewhat personal and specific.</p><p>Each Notification Type allows for templating of different fields, as shown by the ADT definition below:</p><pre>object TemplateData {</pre><pre>  final case class Email(subject: String, body: String, plainText: String) extends TemplateData</pre><pre>  final case class Push(popupTitle: String, popupBody: String, deepLink: Option[String]) extends TemplateData</pre><pre>  final case class Sms(value: String) extends TemplateData</pre><pre>  final case class InApp(<br>    title: String,<br>    body: String,<br>    buttonLabel: Option[String],<br>    buttonLink: Option[String], // Later rendered using handlebars, validated and converted to Option[URI]<br>    iconLink: Option[String], // Later rendered using handlebars, validated and converted to Option[URI]<br>    displayProperties: List[TemplateDisplayProperties]<br>  ) extends TemplateData</pre><pre>}</pre><p>To support this feature, we rely on a popular templating engine called <a href="https://handlebarsjs.com/guide/#what-is-handlebars">HandleBars</a>. We use this library to convert the templated content provided through our backoffice, into the final substituted version that our users will finally receive.</p><p>Which exact variables are available for substitution depends on the context of the notification, but some examples of variables we can use here include User Profile information, Product information, and Transaction information.</p><h4>Template Collections</h4><figure><img alt="Database UML model" src="https://cdn-images-1.medium.com/max/1024/0*p8JDi6dBda8iebQj" /></figure><p>The final piece of our notifications topology is our Template Collections. These collections represent one or more notification types and their templated content to send. This allows us to group multiple notification types into one logical group for sending. This is used quite heavily, for example, when we send a system notification to the user, it is often accompanied by an InApp notification relaying the same message, perhaps with additional details.</p><p>We enforce a constraint of 1 notification type per collection to ensure each collection represents one piece of information to convey. We do not allow two distinct Email notifications to be part of the same collection, as an example. This is enforced at the database level by the composite primary key of the template content — <strong>PK(collection_id, type, version)</strong>.</p><p>Our marketing agents create and edit collections and template content via a BackOffice interface, which issues commands to our notification-service’s gRPC endpoints.</p><h4><strong>Notification categories</strong></h4><p>We differentiate notification collections into two groups. The distinction between the two groupings can be boiled down to the following points.</p><p><strong>Transactional Notifications (TNs)</strong></p><ul><li>A notification triggered as a result of some internal event.</li><li>For example this might occur when a user’s account is verified, funds arrive after a deposit or a user’s transaction has failed.</li><li>They serve to notify users of important events concerning their account when the application is closed, either via a direct notification (push, email, sms) or the next time they open the app (InApp).</li></ul><p><strong>Marketing Messages (MMs)</strong></p><ul><li>A notification triggered as a result of some external event.</li><li>A Marketing Message can be triggered for a variety of reasons from outside the SwissBorg ecosystem. When and Why these get triggered is defined by our Marketing Agents.</li><li>The can be used during a user journey to notify them about new features, compliance issues, or more generally for direct communication with specific user cohorts.</li></ul><h3>Translations</h3><p>If you’ve read our previous blog post about <a href="https://medium.com/swissborg-engineering/internationalisation-i18n-at-swissborg-7b44152d0f58">Internationalisation at SwissBorg</a>, you’ll remember we try to centralise our translations in <a href="https://lokalise.com/">Lokalise</a>. As our approach to notifications predates multi-language support in the app, and we have the additional complexity of heavily templated strings, this generalisation is not held inside the <strong>notification-service</strong>. Currently, we have somewhat of a hybrid approach where TNs, which have well defined static content are resolved with Lokalise, but Marketing Messages (MMs) are handled differently.</p><p>In the case of MMs, translations are managed by the notification service itself, storing the template content for each language individually. The data we store for Template Content contains a map of Languages to TemplateData.</p><pre>final case class TemplateContent(version: Int, kind: NotificationType, langs: Map[Lang, Option[TemplateData]])</pre><p>I won’t go any further into detail here as this is an exception to the rule, and we have already discussed translations at some length in another post.</p><h3><strong>Architecture</strong></h3><figure><img alt="Architecture Diagram" src="https://cdn-images-1.medium.com/max/947/0*2zd2WHg6n4nOv0O3" /></figure><p>Now that we’ve seen how notifications are managed, we can take a look at how they get from this point to our user’s devices.</p><h4>Queueing</h4><p>As we handle both TNs and MMs in the same service, we can expect there to be an uneven distribution of load. TNs have a relatively predictable and steady flow, however with a new marketing campaign or new feature launch we see periodic spikes of MMs to process, as shown in the graph.</p><figure><img alt="Graph of Marketing Messages to process" src="https://cdn-images-1.medium.com/max/1008/0*u6Lt-wV2c4ZTWVGe" /></figure><p>As a result, processing notifications is inherently suited to a queue design by its nature. We are utilising streaming quite heavily, using Kafka via <a href="https://doc.akka.io/docs/alpakka/current/index.html">Akka’s alpakka client library</a> in our implementation. This brings with it the guarantee of ‘At least once delivery’ ensuring that notifications are processed by the service</p><p>However, since we are not guaranteed ‘At most once’ or ‘Exactly once’ delivery, we also take some measures to deduplicate incoming events. We do this by storing the originating <strong><em>correlation_id</em></strong> of the event with the resulting notification. When a duplicate event is processed, we’ll see we have already emitted a notification for this <strong><em>correlation_id</em></strong> and skip it.</p><p>This protects us from duplicate notifications being issued, however, it does not guarantee ‘exactly once delivery’ as it’s possible the service itself fails between sending a notification and updating the internal status.</p><p>Kafka is responsible for managing the incoming events for the service, while an internal postgres table manages the commands currently being processed. This approach was chosen over a solely kafka based solution because of the advantages of its low-tech implementation, which is quite resilient and easy to debug. The implementation allows us to manage notification priority (more on that later) and retry logic in a central place. It is quite performant, and can handle the load easily as illustrated by the following graph.</p><figure><img alt="Queue size of pending notifications" src="https://cdn-images-1.medium.com/max/997/0*3h_4dGILQWCz17zs" /></figure><p>During normal operation, with a full queue of notifications to send, we poll our postgres <strong><em>notifications_queue</em></strong> postgres table for ‘reservable’ notifications to process, throttling only when downstream back pressures. This polling is implemented using <a href="https://doc.akka.io/docs/akka/current/stream/index.html">Akka Streams</a> and an SQL query issued through <a href="https://tpolecat.github.io/doobie/">doobie</a>. We’re taking this approach over Postgres’ <a href="https://www.postgresql.org/docs/current/sql-listen.html">LISTEN</a> functionality because LISTEN also involves polling on the PQnotifies function, and support in JVM libraries is relatively poor.</p><pre>Source<br>  .repeat(())<br>  .via(RestartFlow.withBackoff(restartSettings)(() =&gt;<br>    Flow[Unit].mapAsync(1) { _ =&gt; dispatcher.unsafeToFuture {       <br>      repository.reserveNotifications(<br>        notificationType,<br>        config.redeliverInterval,<br>        throttleConfig.batch()<br>      )<br>    }})<br>  )</pre><p>If the <strong><em>notifications_queue</em></strong> is empty, we initiate a backoff period using <a href="https://doc.akka.io/api/akka/2.6/akka/stream/scaladsl/Flow.html#throttle(elements:Int,per:scala.concurrent.duration.FiniteDuration):FlowOps.this.Repr[Out]">Akka Streams custom cost calculation</a> feature, whereby we stop polling the database for some time in order to reduce the number of calls to the database.</p><pre>Stream.throttle(<br>  1,<br>  config.emptyQueueBackoff,<br>  list =&gt; if (list.isEmpty) 1 else 0<br>)</pre><p>Since adding this optimisation with a 500 millisecond backoff, we’ve seen a significant reduction in queries hitting the database to our current rate of around 1080 requests per minute.</p><h4><strong>Priority</strong></h4><p>Not all notifications are created equal, and given the constraints around rate limits, sometimes we have to pick favourites. As mentioned previously, this is handled during the polling query to our database <strong><em>notifications_queue</em></strong> table.</p><p>When creating notifications, we allocate a numeric priority to each of them. This <em>priority</em> value is used in the ORDER BY clause of the query in order to return queued notifications in ascending priority. In effect, we send notifications about transactions and account changes before we send notifications about new features.</p><p>In the future, if we see a degradation of performance, we may need to get more granular with our priority levels. An example of this; perhaps account update notifications should take priority over transaction updates, for security reasons.</p><p>Below, you can see the polling query behind the <strong><em>repository.reserveNotifications(…) </em></strong>call from the previous snippet to illustrate this.</p><pre>UPDATE notification_queue<br> SET status = &#39;reserved&#39;, updated_at = CURRENT_TIMESTAMP<br>   WHERE notification_id IN (<br>     SELECT notification_id FROM notification_queue<br>       WHERE (<br>         status = $available OR<br>         updated_at + ($delayMillis || &#39;milliseconds&#39;)::interval &lt; CURRENT_TIMESTAMP) AND<br>         process_after &lt; CURRENT_TIMESTAMP AND<br>         notification_type = $notificationType<br>       ORDER BY priority, process_after<br>       FOR UPDATE SKIP LOCKED<br>       LIMIT $limit<br>)</pre><pre>RETURNING *;</pre><h4>Rate limits</h4><p>As we rely heavily on 3rd party providers for delivering notifications, rate limiting is a concern here. The number of notifications sent grows linearly with the size of our user base, and also with each new notification added. We needed to address the issue of having a shared rate quota in a horizontally scalable setup.</p><p>Our solution to this is a small utility class which utilises <a href="https://doc.akka.io/docs/akka-management/current/discovery/kubernetes.html">Akka Kubernetes Discovery API</a> to determine the current number of service pods running, and splits the allocated quota between them; <strong><em>quota / numPods</em></strong> per service. This information is refreshed every minute. The implementation assumes all pods know the total rate limit, which may not hold, as an example, during configuration changes to a lower overall rate.</p><p>Using this information, we throttle the processing inside each pod to its allotment of the total quota.</p><h4>Retries</h4><p>Using our <strong><em>notification_queue</em></strong> table also simplifies our retry logic for notifications. Had we gone down the route of additional retry topics, the number of kafka topics needed by the service would have increased dramatically.</p><p>When we fail to send a notification, and consider the error to be a retryable error, we reschedule the notification by updating the status from ‘reserved’ (as set in our polling query) back to ‘available’, updating other fields required by the retry logic such as a <strong><em>process_after</em></strong> timestamp and the retries counter for the notification. The polling query will pick up this notification again when it is ready to be reprocessed.</p><pre>UPDATE notification_queue <br>  SET status = &#39;available&#39;,<br>      updated_at = CURRENT_TIMESTAMP,<br>      process_after = CURRENT_TIMESTAMP + ($backoffMillis || &#39;milliseconds&#39;)::interval,<br>      retries = retries + 1<br>WHERE notification_id = $notificationId</pre><p>Some error cases are determined to be non-retryable. In these cases the message is logged and removed from the postgres queue. InApp notifications are also not retried, as our retry solution relies on writing to postgres, and failing to send an InApp notification would be a result of failing to write to postgres, as we’ll see later.</p><h4>Time To Live</h4><p>At some point it does not make sense to try to deliver a failing notification anymore, even if we have not reached the maximum number of retries allowed.</p><p>As a trivial example, imagine we send a user a TAN code challenge to verify their phone number, which they need to enter within 1 hour. If the notification has not been sent in 1 hour, does it make sense to keep retrying? The user may eventually receive the notification, but the code will be invalid.</p><p>For this reason, each notification command has a Time To Live (TTL) value, beyond which point the notification is considered outdated, and will not be sent. We configure this based on the originating trigger of the notification, examples being, 1 hour for challenge sms, 1 day for deposit notifications, 1 day for marketing messages, 7 days for account changes. This TTL is considered both in our first attempt to send the notification and also when executing our retry logic.</p><h3>Sending and Overview</h3><p>Finally, we get to the point where we send the notification. Let’s briefly recap. Notifications are triggered in three ways; via a gRPC endpoint or through kafka, either by reacting on a known transactional event or by a webhook call from an external service. In general, most notifications are triggered via a webhook call from our external analytics tool. Triggering via gRPC is usually reserved for testing or edge cases but follows the same flow once inside the service.</p><p>From these input triggers, we first transform them into a notification-service domain model object called a NotificationCommand, one of which is a SendNotification command. For all input triggers, the trigger provides at least the ID of the template collection and the user. From these IDs we retrieve and ‘explode’ the template collection; splitting the collection into separate SendNotification commands for each configured notification type and resolving the required user information from their ID — phone number, email, language, push tokens, etc. These commands are then put on another internal kafka topic for further processing.</p><p>Reading from this queue, the CommandTransformer sends the notification to the user, linking the notification to a specific version of the template content for reference.</p><p>The flow inside the CommandTransformer goes as follows:</p><ol><li>Checks if a notification was already issued with the same correlationID.</li><li>Checks if the command is still valid based on the TTL.</li><li>Creates the notification in a Pending state.</li><li>Enqueues the event to be sent.</li><li>Emits an event to notify of a status change, moving the notification to ‘Pending’.</li></ol><p>‘Sending’ the event usually means inserting it into the <strong><em>notifications_queue</em></strong> postgres table. However, in the case of InApp notifications — as these are provided to the apps by a REST API endpoint and get pulled from the server — ‘Sending’ for InApps means rendering the template (resolving templated values in the notification) and inserting directly to a <strong><em>rendered_notifications</em></strong> table, which is queried by the endpoints.</p><p>In contrast to our InApp notifications, all other notifications are delivered by a 3rd Party integration. These delivery methods do not have the limitation of requiring a user to open the app. Multiple vendors are typically used per type, and the vendor is chosen non-sequentially from a list of possible vendors in order to spread the load.</p><p>For testing purposes, we avoid incurring costs associated with our 3rd party providers by redirecting some notifications either to the void, or to a cheaper alternative such as a Console log or Slack.</p><h3><strong>Building out FullScreen</strong></h3><p>As we touched on earlier, InApp notifications materialise as an expanded block of text displayed on the Portfolio screen and are viewable in the Notifications History. These are our default InApp notification style, and until recently, the only InApp display style we supported.</p><p>We recently extended our InApp notifications to support different display stylings within the app. The feature was built out specifically to support Full Screen notifications, but with the possibility of enabling other display options in the future.</p><p>For this, we decided to refactor our InApp Notification model to allow for the configuring of different display options. Our InApp notifications now contain a list of display properties that they can exhibit in the app, this allows us to support new display styles in the future simply by defining a new DisplayProperties type for this array.</p><p>When creating the template content, we validate that this list of display properties contain only distinct values. In this way, it would not be possible to create an InApp notification that has a duplicate configuration for a display style.</p><p>The mobile apps then in turn interpret these properties on their side.</p><p><strong>Before</strong></p><pre>final case class InAppNotification(<br>  notificationId: Option[NotificationId],<br>  title: String,<br>  body: String,<br>  buttonLabel: Option[String],<br>  buttonLink: Option[URI],<br>  iconLink: Option[URI],<br>  createdAt: Option[Instant],<br>  readAt: Option[Instant]<br>)</pre><p><strong>After</strong></p><pre>final case class InAppNotification(<br>  notificationId: Option[NotificationId],<br>  createdAt: Option[Instant],<br>  readAt: Option[Instant],<br>  displayProperties: List[DisplayProperties]<br>)</pre><pre>sealed trait DisplayProperties</pre><pre>final case class Simple(<br>  title: String,<br>  body: String,<br>  buttonLabel: Option[String],<br>  buttonLink: Option[URI],<br>  iconLink: Option[URI]<br>) extends DisplayProperties</pre><pre>final case class FullScreen(<br>  title: String,<br>  body: String,<br>  buttonLabel: Option[String],<br>  buttonLink: Option[URI],<br>  asset: Asset<br>) extends DisplayProperties</pre><p>In our two currently supported display properties, Simple InApp notifications are interpreted by the apps as the existing behaviour, where a notification appears on the Portfolio screen and in the Notifications History. A FullScreen InApp notification will be displayed covering the whole screen when the user opens the app. A notification configured with both Simple and FullScreen display properties will exhibit the behaviours of both types.</p><p>Et voila, Full Screen Notification for the launch of Thematics in the app!</p><figure><img alt="Example Full Screen InApp Notification" src="https://cdn-images-1.medium.com/max/720/0*-kUv3waSDRRvFQms" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83d38dbd377f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swissborg-engineering/swissborg-notifications-83d38dbd377f">SwissBorg Notifications</a> was originally published in <a href="https://medium.com/swissborg-engineering">SwissBorg Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Internationalisation (i18n) at SwissBorg]]></title>
            <link>https://medium.com/swissborg-engineering/internationalisation-i18n-at-swissborg-7b44152d0f58?source=rss-a47d2b47f232------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b44152d0f58</guid>
            <category><![CDATA[lokalise]]></category>
            <category><![CDATA[internationalization]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[localization]]></category>
            <category><![CDATA[i18n]]></category>
            <dc:creator><![CDATA[Smur89]]></dc:creator>
            <pubDate>Fri, 06 May 2022 11:43:04 GMT</pubDate>
            <atom:updated>2022-05-06T11:43:04.144Z</atom:updated>
            <content:encoded><![CDATA[<h3>The beginning of our i18n journey at SwissBorg</h3><blockquote>“If I’m selling to you, I speak your language. If I’m buying, dann müssen Sie Deutsch sprechen!“</blockquote><blockquote>— Willy Brandt</blockquote><p>At SwissBorg we are in the unique position of having our home base in a country with four official national languages in a space of 350km wide by 220km tall — about 8% the size of France — in the geographic heart of Europe. This affords us the perspective and appreciation for the usefulness of a well-internationalised app.</p><figure><img alt="A Map of Europe colour coded by number of official languages showing switzerland with the most amount of official languages in eurioe" src="https://cdn-images-1.medium.com/max/1024/0*1aViZObCfcD2-YWE" /></figure><p><em>Source: </em><a href="https://www.instagram.com/languages.5/"><em>https://www.instagram.com/languages.5/</em></a></p><p>We have all used online services to translate directly from one language to another, only to have the result either not make sense or not quite make it — missing the <a href="https://news.mit.edu/2017/analyzing-language-color-0918">language of colour</a>, culture, emoji interpretation, and more.</p><p>Thankfully, we can leave this part in the capable hands of our Translation Team and show you the technical challenges translation brings.</p><h3>Technical Challenges</h3><h4>Pluralisation</h4><p>Pluralisation (p11n) involves applying a mapping from plural rules to plural forms. In languages like English, this is relatively trivial, since there are only two forms, singular and plural. This becomes more complicated in languages with multiple forms, as is the case in some Slavic languages with three or more plural forms, Irish with five and Arabic having six.</p><p>To illustrate this, consider the following example taken and expanded upon from <a href="https://www.forasnagaeilge.ie/wp-content/uploads/2017/06/ce_mhead_uair-aghaidh-agus-c%C3%BAl.pdf">this</a> Irish language school book</p><figure><img alt="Table comparing the counting of Hours in English and Irish" src="https://cdn-images-1.medium.com/max/1024/1*9e3oiLHvfAi9yR7JOM-fSg.png" /></figure><p>In practice, this means our display string depends on two inputs; our translation key and the quantity of the item we are presenting. For English, this might look something like the following:</p><pre>quantity match {<br>  case 1 =&gt; my_display_string.singular // (“hour”)<br>  case _ =&gt; my_display_string.plural // (“hours”)<br>}</pre><h4>Number, date and currency formatting</h4><p>Countries and cultures denote numbers and dates in different ways, with an obvious example being the date formats on either side of the Atlantic; <em>mm/dd/yyyy</em> v.s. <em>dd/mm/yyyy</em>.</p><p>Perhaps less obvious and closer to home is number formatting.Consider the following examples of the same monetary value formatted according to each locale.</p><figure><img alt="Table comparing the formatting of currency values between France, Germany and Ireland" src="https://cdn-images-1.medium.com/max/1024/1*9smQYhFquwpso9xrHsG4CQ.png" /></figure><p>Although it might seem like a minor adjustment, ignoring these differences can easily lead to confusion among users accustomed to seeing monetary values in their local format. The capacity to format numbers also applies to other kinds of numbers, like phone numbers.</p><p>Currently at SwissBorg, our solution for this is not perfect; we have a custom defined formatting for numeric values that does not align with any one locale for now.</p><p>You can explore these differing locales using the <a href="https://icu4c-demos.unicode.org/icu-bin/locexp?d_=en">ICU Locale Explorer</a>.</p><h4>Centralising Display Strings</h4><p>Within the SwissBorg ecosystem, our display strings are spread out across many different areas. From the mobile applications, website and help centre, through to emails, notifications and the App Store descriptions, they get around.</p><p>As much as possible, we aim to centralise the source of these strings and provide a simple process for our translators to work with. To this end, we utilise a Translation Management System (TMS) called Lokalise. Using this approach we decrease complexity and improve efficiency, enabling our translators to directly leverage the features provided by Lokalise in their workflows. Features used include Computer-Aided Translation (CAT), QA tooling, branching, project statistics and context (comments, screenshots, etc.).</p><h3>Backend Integration</h3><figure><img alt="Diagram of the Backend integration with Lokalise" src="https://cdn-images-1.medium.com/max/767/1*4Z44bzE2WNOLdZ_bbjPuJw.png" /></figure><p>We leverage the <a href="https://lokalise.com/product/integrations/cloud-file-storage/amazon-s3">Lokalise AWS S3 integration</a> to push translation files into our backend systems. Once the files are inside our system, we replicate the data to all environments using <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html">AWS S3 Replication</a>. At this point, the localisation files are ready to be consumed by our services.</p><p>Using this approach, we decouple ourselves from any third party dependency, in this case Lokalise, and move our integration dependencies closer to home inside our Virtual Private Cloud (VPC). In doing so, the risk of downtime, high latency, efficiency and rate limits on the Lokalise API are mitigated.</p><p>Our S3 buckets are configured to invoke a lambda function for <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html">S3 Event Notifications</a> on receiving a PUT. This lambda is a simple function responsible for writing the notification to Kafka which we later use to trigger a refresh of each service’s translation files.</p><p>When discussing speed, we have seen that from start to finish, the pipeline takes 20 to 30 seconds; from triggering a build in Lokalise to being visible in the apps. Of course, this may vary per environment and is largely dependent on the <a href="https://aws.amazon.com/s3/sla-rtc/">AWS S3 Replication</a> SLA of 15 minutes. Considering this can be handled without any developer involvement, we’re quite happy with the relatively small lag.</p><p>Our agent library retrieves and caches the most recent files from S3 on startup. It relies on notification events from Kafka to trigger a cache update while the service is running. For this, we use Kafka’s ’auto.offset.reset’ <a href="https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_auto.offset.reset">configuration</a> to read from the latest offset, skipping any previous update notification events.</p><h4>Cache Size</h4><p>Due to the relatively small footprint of the translations required in our backend services, we can keep all translations cached in memory within our services. Currently, the size on disk of all translations is 49KB.</p><p>Now, in the case that memory becomes a concern, we can first look at reducing the number of keys that each service keeps in memory. Currently each service holds all keys from our backend project in Lokalise, even the keys it has no use for. We can easily achieve this by defining a unique bucket prefix per service to export from lokalise. Then it is a matter of a configuration change to read from the new file location in each service.</p><p>The internal cache implementation uses a <a href="https://typelevel.org/cats-effect/docs/std/deferred">Deferred</a> and a <a href="https://typelevel.org/cats-effect/docs/std/ref">Ref</a>; Deferred in order to wait for the cache to be initialised and Ref to provide safe concurrent access and modification. For more specific details, our approach is quite similar to that outlined by SoftwareMill in their <a href="https://blog.softwaremill.com/cats-concurrency-basics-with-ref-and-deferred-399e0335630">blog post</a>.</p><h4>Monitoring</h4><blockquote>“Software testing is a sport like hunting, it’s bughunting.”</blockquote><blockquote>– Amit Kalantri</blockquote><p>To mitigate possible debugging issues we introduced the concept of versioning our translations cache. This will help us to know which file is being used internally in our services. The ‘version’ is derived from the most recently modified timestamp of the last modified file in the bucket.</p><p>We expose this version as a <a href="https://prometheus.io/docs/concepts/metric_types/">Prometheus Info Metric</a>, which is a Gauge Metric with one possible value. This metric type is designed to store labels whose values may change over time, as is the case with versions.</p><figure><img alt="Table of Prometheus Info Metrics in Grafana showing current translation version" src="https://cdn-images-1.medium.com/max/756/0*AOCfObihhp2XJ-0K" /></figure><h4>Translation Agent</h4><p>The majority of our translation requirements are handled by our mobile apps. The scope of the requirements on our backend translations is quite light. Due to the fact that the majority of backend translations revolve around our notifications, the use of something like <a href="https://unicode-org.github.io/icu/userguide/icu4j/">ICU4J</a> — which implements the full ICU specification — was unnecessary and would only serve to provide a convoluted API to work with. In the future, if translations become more complicated, we may revisit the use of this type of library.</p><p>Our main requirements on the backend were support for the following:</p><ul><li>plain text translation</li><li>interpolation</li><li>plurals</li></ul><p>Our solution is to provide an internal agent library, which is published to our artifactory for any service which may need to resolve display keys to utilise. Internally, this library is responsible for managing the cache of localisation files from S3 and exposing these through a simple API for use in our services.</p><p>The library exposes a single method in its API…</p><pre>def get(key: String, lang: Lang): F[I18nValue]</pre><p>…which internally utilises a Java properties file, and this i18nValue type in turn exposes…</p><pre>def withArgs[F[_]: Raise[*[_], Throwable]](args: Any*)(quantity: Int): F[String]</pre><pre>def singular: String</pre><pre>def zero: String</pre><pre>def plural(quantity: Int): String</pre><p>…to allow us to handle different plural categories. Currently we have a static plural rules mapping for all languages, which is taken from the <a href="https://cldr.unicode.org/index/cldr-spec/plural-rules#h.m2v80lrdfvr9">CLDR guidelines</a>. We can later provide a mapping per language as new languages are introduced.</p><h3>App Integration</h3><p>Although we focus on the less-discussed backend integration here, it would be prudent to briefly mention how our mobile apps interact with Lokalise, as it is slightly different from our backend integration.</p><p>To provision translation keys in Lokalise in our build pipeline we utilise the <a href="https://lokalise.com/product/integrations/code-repositories/gitlab">Gitlab Integration</a> provided by Lokalise. We have found that this fits well with our preexisting workflow between design and development, whereby the base translation, in English, is provided by our designers through <a href="https://www.figma.com/">Figma</a>. Our developers then take the display text from Figma and add it directly to the English String dictionary in the codebase.</p><p>As part of the build pipeline on our master/main branch, the String dictionary is pushed to Lokalise, creating or updating the keys/values for the base language. We never delete keys from Lokalise with this integration. This operation is possible because we maintain two separate <a href="https://docs.lokalise.com/en/articles/1400460-projects">Lokalise projects</a> for iOS and Android. In this way, each ecosystem can define and rely on its own set of keys. A drawback is that it results in duplicated work for our translators.</p><p>Once the base keys are in Lokalise, we can move forward with the process of translating the strings to each supported language, after which, a new bundle is built from Lokalise. On each start of the app we query Lokalise for the latest values which allows for changes to display strings without issuing a new App release.</p><p>The current process is under review with plans to move the ‘source of truth’ for the base language out of the codebase and into Lokalise. We are also considering integrating the creation of new keys directly from Figma using the <a href="https://www.figma.com/community/plugin/818840482503404814">Lokalise plugin</a> for that.</p><h3>Unicode Consortium</h3><p>The Unicode Consortium provides a wide variety of data, libraries and information regarding software internationalisation through their International Components for Unicode (ICU) project. A full description of what is provided is available on their <a href="https://unicode-org.github.io/icu/">website</a>.</p><h4>Unicode Common Locale Data Repository (CLDR)</h4><p>The <a href="https://cldr.unicode.org/">Unicode Common Locale Data Repository (CLDR)</a> is maintained by the Unicode Consortium and provides the underlying data and building blocks for ICU. For a given locale, the CLDR can provide you the locale’s script, its preferred calendar, number system, date formats, pluralisation rules, etc.</p><h4>Plurals</h4><p>English has a simple distinction between singular and plural forms — including zero forms. For more complex languages, defining the plurals is a more involved and subtle task; the <a href="https://cldr.unicode.org/index/cldr-spec/plural-rules">CLDR</a> defines 6 distinct categories for plural forms:</p><ul><li>zero</li><li>one (singular)</li><li>two (dual)</li><li>few (paucal)</li><li>many (also used for fractions if they have a separate class)</li><li>other (required — general plural form — also used if the language only has a singular form)</li></ul><p>A more in-depth view of the ruleset per language can be found <a href="https://unicode-org.github.io/cldr-staging/charts/latest/supplemental/language_plural_rules.html">here</a>, again as defined by the CLDR.</p><h3>Encoding</h3><p>The method by which characters get encoded, passed around and displayed is important in the localisation discussion. Broken character encodings can materialise when encodings are not recognised by the software displaying the information.</p><p>As an example, let’s say we provide a localisation for Arabic languages, but use the Latin-1 (ISO 8859–1) character set for encoding. If an Arabic localisation is requested, the user will see Mojibake. ‘Mojibake’ describes instances where text is improperly decoded, resulting in nonsense or random symbols, e.g. squares and other characters instead of letters. It can also be encountered with certain characters in western languages.</p><figure><img alt="Example of how Mojibake looks" src="https://cdn-images-1.medium.com/max/725/0*2_m-cn5MDS55rbIz" /></figure><p><em>Source: </em><a href="https://commons.wikimedia.org/wiki/File:Mojibake_iso8859-1_em_utf-8.png"><em>https://commons.wikimedia.org/wiki/File:Mojibake_iso8859-1_em_utf-8.png</em></a></p><p>This issue can be resolved by including the character encoding as part of the message sent between subsystems. UTF-8 has become the de facto and convenient way to represent non-Latin characters. As we use UTF-8 across our application, it should be capable of representing pretty much any character we might come across. Using it for all languages means we do not need to track and convert various encodings depending on the locale being served, with the trade-off of an increased encoded size compared to an encoding supporting a smaller set of characters.</p><p>Since Java 9 even properties files, which we use in this integration, are loaded in UTF-8 encoding, replacing the Latin-1 encoding used previously. Recently we updated our services from Java 11 to Java 17, you can learn more about that in our earlier <a href="https://medium.com/swissborg-engineering/migrating-to-java-17-2ed137ee2eec">blog post</a>.</p><h3>Conclusion</h3><p>The process of localising an IT product is no trivial task, and in this post we have only presented thoughts on decoupling our backend services from display strings.</p><p>There is much more to consider including the following:</p><ul><li>RTL (right-to-left) scripts such as Arabic, Hebrew, Persian, Pashto, Urdu and Sindhi.<br><em>Accounting for design, UX and space constraints.</em></li><li>Localising notifications per device.<br><em>What if a user has the application on multiple devices, each with a different locale setting?</em></li><li>Character limits creating a display issue.<br><em>A constant occurrence usually resolved by collaboration between translators, designers and developers.</em></li><li>Non Gregorian calendars.</li></ul><p>The process of internationalising our app is an ongoing project at SwissBorg.The foundations have been laid and we are continuing to improve. There are areas we would like to enhance, open questions and issues to resolve. We aim to also iron out the processes between departments and reduce unnecessary overhead on our translators and developers alike.</p><p>After all, part of making wealth management fun, fair and community-centric is making it easily accessible and welcoming to everyone in their native language.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b44152d0f58" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swissborg-engineering/internationalisation-i18n-at-swissborg-7b44152d0f58">Internationalisation (i18n) at SwissBorg</a> was originally published in <a href="https://medium.com/swissborg-engineering">SwissBorg Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>