<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Helicap Engineering - Medium]]></title>
        <description><![CDATA[Enabling Alternative Financing Access (https://heli-cap.com/) - Medium]]></description>
        <link>https://medium.com/helicap-engineering?source=rss----82cc3c1b69ad---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Helicap Engineering - Medium</title>
            <link>https://medium.com/helicap-engineering?source=rss----82cc3c1b69ad---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:50:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/helicap-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How we built an auto scaling CICD solution for 57 cents/month]]></title>
            <link>https://medium.com/helicap-engineering/how-we-built-an-auto-scaling-cicd-solution-for-57-cents-month-f6c3adeb171a?source=rss----82cc3c1b69ad---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6c3adeb171a</guid>
            <category><![CDATA[gitlab]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Lim Yi Sheng]]></dc:creator>
            <pubDate>Fri, 13 Mar 2020 07:23:55 GMT</pubDate>
            <atom:updated>2020-03-13T07:23:55.041Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will be covering how our company, Helicap, uses AWS and Gitlab to create a CICD solution to build and deploy our applications, for a grand total of 57 cents/month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*7X0u8uluCuQHlXhIIUwuIg.png" /></figure><p>For those who do not know Helicap, we are a FinTech investment firm specialising in the alternative lending space in the Asia Pacific region. We help people improve their livelihoods by driving financial inclusion and enabling our partner platforms to transform the way consumers and SMEs access credit today.</p><h3>What is CICD?</h3><p>To better understand the importance of such a pipeline, we need to delve into the concept of what CICD is.</p><p>In software engineering, CICD generally refers to the combined practices of continuous integration and continuous deployment. Continuous integration is the practice of merging all developers’ working copies to a shared mainline several times a day.</p><p>What does that mean? You can imagine it as two people building a car. One will build the engine, another person takes the chassis. By being able to constantly merge, integrate, and fix issues that come up, the two people can ensure that the design of the engine and the chassis is able to synergise perfectly constantly. This is as compared to each of them building their own parts in a silo-ed environment, only to find out after a few months, that the engine and chassis built are incompatible with each other.</p><p>Continuous deployment is the practice of constantly releasing software. This is extremely important for demoing to stakeholders and business users. To reuse the car analogy, every so often, the engineering team will assemble the latest prototype of the car for the driver to drive. With a tighter feedback loop between the driver and engineers, the team will find it easier to build a better car.</p><p>By building a robust and scalable CICD workflow, the business and software teams at Helicap are able to synergise perfectly to deliver quality products for our clients.</p><h3>Helicap’s CICD Architecture</h3><p>For our architecture, we have decided to go with Amazon Web Services and Gitlab. This is what our CICD flow looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*6aymosZM6TuRgWAoVueeyQ.png" /><figcaption>Helicap’s CICD Architecture</figcaption></figure><p>Typically, when a developer needs to build and deploy their source code, they need build machines to do that. Depending on the source code that needs to be built and deployed, these machines may consume quite a fair bit of resource. However, there is a problem. These machines will not be building and/or deploying source code all the time. In fact, most of the time, these machines will remain idle, until the developers decide they want to integrate and deploy. Even so, we still have to pay for these machines even if they are idle.</p><p>And to make it worse, as the software engineering team scales up, we will have multiple developers wanting to build and deploy their source code at the same time. So the team either has to deal with a) new jobs being queued until previous build jobs are done, or b) having to pay for more machines to deal with any demand spikes.</p><p>The ideal solution is obvious at this point. Why don’t we purchase those machines only when we need them? We can purchase however many machines we need during demand spikes, and however large it needs to be to deal with the job.</p><p>So we reduce our need to a single auto-scaling runner. Our runner is a small 1 CPU core, 1GB RAM machine that listens out for any jobs that require CICD. Upon receiving such jobs, it is responsible for renting larger machines from AWS, that will be responsible for building and deploying the source code. Once the build jobs are done, the machines will be returned back to AWS.</p><p>By doing so, we reduce our overhead from multiple, large build machines, to a single small runner, yet still retain the functionality of being to scale to a) build large jobs, b) build multiple jobs at once.</p><h3>Step-by-step Guide</h3><p>To set this up, we will be primarily following <a href="https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/">this fantastic article</a> written by the Gitlab team.</p><p>This is written for anyone who has no DevOps background, and would like to just follow a series of steps to set such a CICD pipeline up.</p><p><strong>Prerequisites</strong></p><ul><li>Amazon Web Services account</li><li>Gitlab Account and Project</li></ul><p><strong>Setting up EC2 infrastructure</strong></p><ol><li>Log into your AWS account, go to <em>EC2</em> and select ‘<em>Launch Instance</em>’</li><li>Select which operating system you are more comfortable with. For the purposes of this guide, we will be choosing an <em>Ubuntu Server 18.04 LTS</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7uL5WuX4aP3Yphm4UFGC_Q.png" /><figcaption>Runner’s Operating System</figcaption></figure><p>3. Next, select the instance type you would want. As this runner scales automatically, you would not want a very large instance. For practical purposes, we will choose a <em>t2.micro</em> instance because it is free to use.</p><p>4. Next, follow through on all the steps to create an instance. When you come to this page, we will suggest creating a new key pair for your runner. In this case, we are using a new key pair called ‘<em>gitlab</em>’. Make sure that you store your .pem file in a safe and secure place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBhjAexXm59AFqZoeTq-rQ.png" /><figcaption>Create Key-pair</figcaption></figure><p>And that’s it! Your machine is now ready to go.</p><p><strong>Installing Gitlab Runner</strong></p><ol><li>Get the public IP of your EC2 instance, and SSH into your new EC2 instance. The command for our case is:</li></ol><p>ssh -i &quot;gitlab.pem&quot; ubuntu@&lt;ec2-public-ip&gt;</p><p>2. The first thing we want to do is install the Gitlab runner. So we need to add the official repository:</p><pre>curl -L <a href="https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh">https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh</a> | sudo bash</pre><p>3. Following which, we can install the runner with:</p><pre>sudo apt-get install gitlab-runner</pre><p><strong>Installing Docker CE</strong></p><p>1. Next, we need to install Docker CE into our machine. The way our runner automatically scales is via Docker Machine, where it will spawn new Docker machines on demand.</p><p>So first, we run a quick update.</p><pre>sudo apt-get update</pre><p>2. Then we need to allow apt to use a repository over HTTPS:</p><pre>sudo apt-get install \<br>    apt-transport-https \<br>    ca-certificates \<br>    curl \<br>    gnupg-agent \<br>    software-properties-common</pre><p>3. Then we add Docker’s GPG key. GPG is a tool that apt uses to sign files and check their signatures. In short, this allows you to know that the files you are going to download comes from a secure source.</p><pre>curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -</pre><p>4. Next, we will add Docker’s repository for us to retrieve the installation files. As our AWS instance isusing an <em>x86</em> or <em>amd64</em> architecture, we can use the following command:</p><pre>sudo add-apt-repository \<br>   &quot;deb [arch=amd64] https://download.docker.com/linux/ubuntu \<br>   $(lsb_release -cs) \<br>   stable&quot;</pre><p>5. We run a quick update.</p><p>sudo apt-get update</p><p>6. And we install the latest version of Docker CE with the following command:</p><pre>sudo apt-get install docker-ce docker-ce-cli containerd.io</pre><p>Optionally, we can test that this is working with:</p><pre>sudo docker run hello-world</pre><p>This command downloads a test image and runs it in a container. When the container runs, it prints an informational message and exits.</p><p><strong>Installing Docker Machine</strong></p><ol><li>We will install Docker Machine that allows our runner to scale up by spawning new machines on demand. The command is as follows:</li></ol><pre>base=<a href="https://github.com/docker/machine/releases/download/v0.16.0">https://github.com/docker/machine/releases/download/v0.16.0</a> &amp;&amp; curl -L $base/docker-machine-$(uname -s)-$(uname -m) &gt;/tmp/docker-machine &amp;&amp; sudo mv /tmp/docker-machine /usr/local/bin/docker-machine &amp;&amp; chmod +x /usr/local/bin/docker-machine</pre><p><strong>Getting Gitlab Authorisation Token</strong></p><p>Before we can proceed further, we need to obtain the registration token for our Gitlab. This will then connect our Gitlab project to our machine.</p><ol><li>Open your Gitlab project, navigate to Settings =&gt; CI/CD =&gt; Runners and expand the tab.</li></ol><p>You should see something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*tikLD6cXYc3e0sS0I2UW9Q.png" /><figcaption>Gitlab Runner Registration Token</figcaption></figure><p><strong>Setting up AWS IAM</strong></p><p>We will now be setting up permissions for our runner to be able to programmatically access our AWS console to rent larger machines.</p><ol><li>Go to “<em>IAM</em>”, then go to the “<em>User</em>” tab and select “<em>Add a new User</em>”</li><li>Give it the following permissions: <em>AmazonEC2FullAccess</em>, and <em>AmazonS3FullAccess. </em>The former allows the runner to rent new EC2 machines to build, and the latter allows the runner to use S3 as a distributed cache where selected directories and/or files are saved and shared between subsequent jobs.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4E85xFpmSaGmeEItXSXPA.png" /><figcaption>Gitlab AWS User Permissions</figcaption></figure><p>3. Add the user. Take note of the <em>Access Key ID</em> and <em>Secret Access Key</em>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MoCelZjX9GegUyF2QjR3Ug.png" /><figcaption>Gitlab AWS User Access</figcaption></figure><p><strong>Setting up distributed cache</strong></p><ol><li>Go to “<em>S3</em>”</li><li>Create a new bucket. Take note of the name of the bucket, as well as the region that this bucket is created under.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmiLl_EckJayma8ZqNf62Q.png" /><figcaption>S3 Cache</figcaption></figure><p><strong>Setting up VPC</strong></p><ol><li>Go to “<em>VPC</em>”. There should already be a VPC created along with your EC2 machine. Go to “<em>Subnet</em>” and you should see this.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pA3yT514OGd26P8ovj6BNA.png" /><figcaption>AWS Subnet</figcaption></figure><p>Take note of the following:</p><ul><li>Subnet ID</li><li>VPC ID</li><li>Region (in this case it is found under the Availability Zone column, ap-southeast-1)</li><li>Zone (in this case, it is the alphabet behind, either a, b, or c)</li></ul><p><strong>Configuring the Auto-scaling Runner</strong></p><p>Phew! We have finally reached the final step! We now need to take everything from we have assembled from before and configure our runner.</p><ol><li>Register the runner with the following command:</li></ol><pre>sudo gitlab-runner register</pre><p>When asked for the token, use the registration token you got from Gitlab earlier. When asked for the executor type, enter docker+machine. For image you can just use alpine:latest</p><p>2. After all this is done. You can edit the configuration file with:</p><pre>sudo vi /etc/gitlab-runner/config.toml</pre><p>3. Edit the file as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c14306608cc3f13a99b908dbd2396a3/href">https://medium.com/media/8c14306608cc3f13a99b908dbd2396a3/href</a></iframe><p>There are some points to note for this configuration file.</p><p>Under the [[runners]], we have to add in environment = [&quot;DOCKER_TLS_CERTDIR=&quot;]. We also have to modify[runners.docker] to add tls_verify = false. <a href="https://gitlab.com/gitlab-org/gitlab-runner/issues/4501">This is due to some breaking changes in Docker 19.03.</a></p><p>[runners.cache] is where we configure our distributed cache where we will make use of the S3 that we configured earlier.</p><p>[runners.machine] and MachineOptions is where we configured the runner to rent machines. The runner will rent machines under the VPC we created earlier. For amazonec2-instance-type , we decided to go with t3a.small as that is the size that we are most comfortable with. Please upgrade or downgrade where you deem fit. amazonec2-spot-price is set to empty. In this case, the default recommended price by Amazon is used. This is set up in such a way so that we are able to rent our machines immediately, without having to wait for our bid to be matched.</p><p>4. Restart our runner with:</p><pre>sudo service gitlab-runner restart</pre><p>For you to debug and see service logs, you can use the following command:</p><pre>journalctl -f</pre><p>5. Navigate back to your Gitlab Runner’s page, and you should be able to see your runner with a green light next to it. You are now ready to use the runner!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*hYb-G0ZYFkgtTiAhoTQ-Cw.png" /><figcaption>Successful Runner Integration</figcaption></figure><h3>Cost Benefit Analysis</h3><p>For anyone who has been using Gitlab, you will notice that Gitlab does provide some default runners for users to use. However, as of the time of writing, you are only allowed 2000 free minutes. As a startup eventually starts to scale up, the CICD pipeline will become a critical bottleneck.</p><p>Hence, our engineering team here at Helicap has decided to experiment with implementing this auto-scaling runner sometime last year.</p><p>To our surprise, our final overall cost was extremely small. The image below is taken from our AWS billing from last month.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*laJClmU5zf9fM4R2COJEIg.png" /><figcaption>Auto-scaling Runner February 2020 Costs</figcaption></figure><p>Given that we do not have to pay for the <em>t2.micro</em> machine, our costs only comprises of the uptime of the <em>t3a.small</em> machines that we spin up for our builds.</p><p>As of the month of February 2020, we have approximately 4760 minutes of build time, and our final overall cost is USD 0.57.</p><p>To have an idea of how much cost savings you can get, we tried to compare it with the subscription plans of other CI solution providers, namely Gitlab, Travis CI and CircleCI. We selected the plans based on the number of concurrent jobs the plan allows, as well as the number of build minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*6iHcykEY5oKZkNYhgwgVaw.png" /><figcaption>CI Solution Providers Benchmark as of March 2020</figcaption></figure><p>Please take note that the other CI solution providers also have a bunch of other features included in their subscription plans beyond what our autoscaling runner can provide, so this benchmarking is not an exact one-to-one value comparison. But if you are a startup simply looking for a CI solution that can offer you concurrent builds with a large number of build minutes, you will probably be looking at these plans as a baseline, even without all the bells and whistles that these plans will bring to the table.</p><h3>Conclusion</h3><p>Bootstrapping a startup is an extremely difficult task to accomplish. However, despite limited resources, there are a few things that are absolutely essential for a modern technology startup to harness and multiply the value of their team. A robust CICD pipeline is one such thing.</p><p>Thank you for reading this article, and we hope that this helps to provide some insights into implementing a scalable CICD solution at a minimal cost.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6c3adeb171a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/helicap-engineering/how-we-built-an-auto-scaling-cicd-solution-for-57-cents-month-f6c3adeb171a">How we built an auto scaling CICD solution for 57 cents/month</a> was originally published in <a href="https://medium.com/helicap-engineering">Helicap Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Audit trails with MongoDB’s change stream]]></title>
            <link>https://medium.com/helicap-engineering/audit-trails-with-mongodbs-change-stream-a3b5739c33a7?source=rss----82cc3c1b69ad---4</link>
            <guid isPermaLink="false">https://medium.com/p/a3b5739c33a7</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[golang]]></category>
            <dc:creator><![CDATA[Lim Yi Sheng]]></dc:creator>
            <pubDate>Mon, 10 Feb 2020 08:45:56 GMT</pubDate>
            <atom:updated>2020-02-10T14:09:31.444Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we will be covering how our company, Helicap, uses MongoDB’s change streams to track any real time data changes, as well as implement audit trails.</p><p>For those who do not know Helicap, we are a FinTech investment firm specialising in the alternative lending space in the Asia Pacific region. We help people improve their livelihoods by driving financial inclusion and enabling our partner platforms to transform the way consumers and SMEs access credit today.</p><p>Being in the financial space, trust and accountability is paramount, and one of main and earliest concerns, was to set up an audit trail. Our database of choice (<a href="https://medium.com/helicap-engineering/how-we-chose-a-tech-stack-for-our-startup-103538e79249">as elaborated on an earlier post</a>) was MongoDB due to the unstructured nature of the data we work with. One of the reasons we chose was because of Change Streams, which was introduced in version 3.6. To quote directly from the <a href="https://docs.mongodb.com/manual/changeStreams/">documentation</a>:</p><blockquote>Change streams allow applications to access real-time data changes without the complexity and risk of tailing the <a href="https://docs.mongodb.com/manual/reference/glossary/#term-oplog">oplog</a>. Applications can use change streams to subscribe to all data changes on a single collection, a database, or an entire deployment, and immediately react to them. Because change streams use the aggregation framework, applications can also filter for specific changes or transform the notifications at will.</blockquote><p>That sounds perfect for our requirements of creating an audit trail. Any changes made to our database could be captured and logged elsewhere.</p><h3>Architecture</h3><p>So our architecture will look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*0KzM9tSzVkDOV87Sh7su0g.png" /><figcaption>Change Stream Watcher architecture</figcaption></figure><p>We will have applications that will perform CREATE, READ, UPDATE, and DELETE operations from the application database. We will then use our Change Stream watcher to listen to the Change Stream of our application database, and persists any Write operations to a separate audit database.</p><p>The reason why we chose a separate audit database is due to separation of concerns, and possible performance impact if both the audit trail and the actual operational data are kept on the same database.</p><h3>Setup</h3><p>We will be writing a small application using Go to connect to the Change Streams, and propagating the changes to the audit database. These is the list of software we will be using:</p><ul><li>MongoDB version 4.2</li><li>Go version 1.13</li><li><a href="https://github.com/mongodb/mongo-go-driver">Official MongoDB Go driver</a> version 1.3.0</li></ul><h3>Code</h3><p><a href="https://gist.github.com/yish91/5f3dbe33f229a3f36f13ebd6491c706e">The gist can be found here</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4ec6bbe3b79d0d5905bd15332adaa38/href">https://medium.com/media/d4ec6bbe3b79d0d5905bd15332adaa38/href</a></iframe><p>So let us step through the code:</p><pre>func ChangeStreamWatcher(client *mongo.Client, db *mongo.Database) {</pre><p>This program has a function called ChangeStreamWatcher. It takes in a Mongo client. This client is one which we will connect to create our audit database.</p><p>It also takes in a database. This is the application database which we will be tracking and persisting an audit trail for.</p><pre>auditDatabase := &quot;audit&quot; <br>token := &quot;token&quot;</pre><p><em>Audit</em> is the name of database that we are saving our audit trails. <em>Token</em> is actually a collection within the audit database, which we save our resume token for Change Stream.</p><p>Change Stream tracks all events as <a href="https://docs.mongodb.com/manual/reference/change-events/#change-stream-output">change stream event documents</a>. As our watcher processes each event document, we will then save the id of the last document processed into the <em>token</em> table. We will touch on how we use the <em>token</em> collection very shortly.</p><pre>var pipeline mongo.Pipeline <br>ctx := context.Background() <br>opts := options.FindOne() <br>opts.SetSort(bson.D{{&quot;$natural&quot;, -1}})</pre><p>This code is just basic initialisation of the MongoDB pipeline for our change stream. The options is just to sort the token documents, so that the latest document is first.</p><pre>tokenDoc := &amp;bsoncore.Document{} <br>err := client.Database(auditDatabase).Collection(token).FindOne(context.TODO(), bson.D{{}}, opts).Decode(&amp;tokenDoc) <br>var cur *mongo.ChangeStream</pre><p>This is where we query the <em>token</em> collection, and check if there are any existing tokens that we have already saved.</p><pre>if err == nil {  <br>  resumeToken := strings.Trim(tokenDoc.Lookup(&quot;_data&quot;).String(), &quot;\&quot;&quot;)  <br>  cso := options.ChangeStream()  <br>  cso.SetResumeAfter(bson.D{{&quot;_data&quot;, resumeToken}})  <br>  cur, err = db.Watch(ctx, pipeline, cso)  <br>  if err != nil {   <br>    log.Println(err)  <br>  } <br>}</pre><p>This is where we set the <em>resumeAfter </em>for the Change Stream. By extracting the token of the latest event document that we have processed up till, we can then resume processing whatever event documents that are unprocessed.</p><p>E.g. There are 10 event documents in Change Stream, the first document starts with an id of 1, and the last document ending with 10. So the <em>token</em> collection will store the id of 10.</p><p>Assuming that my Change Stream watcher crashes, but somehow, more changes were made to my database, which results in 2 more documents being added with an id of 11 and 12 respectively.</p><p>As the Change Stream watcher is restarted, we will then go to the <em>token</em> collection, extract the token which we have processed up to (which is 10), and set our Change Stream to resume processing documents after the id of 10. Our watcher will then pick up documents 11 and 12.</p><p>This <em>token</em> collection helps to ensure the resiliency of the system. Should our watcher crash, we will not miss out on any event changes once we restart our watcher.</p><pre>else {  <br>  cur, err = db.Watch(ctx, pipeline)  <br>  if err != nil {   <br>    log.Println(err)  <br>  } <br>}</pre><p>This code snippet just means, if there are no documents, simply just watch the Change Stream for any changes.</p><pre>defer cur.Close(ctx)</pre><p>This line just ensures that we will close our connection when we shut the program down.</p><pre>for cur.Next(ctx) {  <br>  elem := &amp;bsoncore.Document{}  <br>  if err := cur.Decode(elem); err != nil {   <br>    log.Println(err)  <br>  }<br>  rawCollectionName := elem.Lookup(&quot;ns&quot;).Document().Lookup(&quot;coll&quot;).String()  <br>  collectionName := strings.Trim(rawCollectionName, &quot;\&quot;&quot;)</pre><p>This code snippet dictates how the program will handle every change stream event document that passes through our pipeline. It will basically decode the byte stream into a bson document for processing. As per the Mongo document on <a href="https://docs.mongodb.com/manual/reference/change-events/#change-stream-output">change stream event document</a>, we will extract out the name of the collection which the change originates from.</p><pre>session, _ := client.StartSession()  <br>_ = session.StartTransaction()  <br>err = mongo.WithSession(ctx, session, func(sc mongo.SessionContext) error {</pre><p>We will start a Mongo session so that we can make a transaction which involves two changes 1) the change stream event document, 2) the token to persist in the <em>token</em> collection.</p><p>Here is where you may ask, if the id of the change stream event document is used as the resume token, is this not a duplication of efforts?</p><p>That is true. However, we are saving the change stream event document in each of their respective collection. So if it comes from the <em>user</em> collection, it will subsequently be saved in the corresponding <em>user</em> collection in the <em>audit</em> database.</p><p>So for us to search the latest document across all the collections will be a rather expensive task, as compared to just saving the token in a separate collection of its own.</p><pre>_, _ = client.Database(auditDatabase).Collection(collectionName).InsertOne(ctx, elem)   <br>  if err != nil {    <br>    _ = session.AbortTransaction(sc)    <br>    return err   <br>  }   <br>  _, err = client.Database(auditDatabase).Collection(token).InsertOne(ctx, cur.ResumeToken())   <br>  if err != nil {    <br>    _ = session.AbortTransaction(sc)    <br>    return err   <br>  }   <br>  _ = session.CommitTransaction(sc)   <br>  return nil  <br>})<br>session.EndSession(ctx)</pre><p>So the code snippet above saves the change stream event document in its corresponding collection, as well as save the token. As this is in a transaction, both of these changes will have to be persisted for the transaction to be successful. This is to prevent any desynchronisation between our <em>token</em> collection, and the changes that we are persisting.</p><p>After the transaction is committed successfully, the session is then terminated, and the program will listen to another change due to <em>for cur.Next(ctx).</em></p><h3>Recap</h3><p>With this, we have created a lightweight watcher that listens to any changes made to an application database, and persist said changes to a separate audit database. These changes will be persisted in a collection structure reflective of that of the application database. So if the application database has 3 collections: <em>x, y, z</em>, our audit database will also have the same collections, on top of a <em>token</em> collection.</p><p>This organization and structure will help us greatly when we want to do rollbacks, or even a poor man’s version of event sourcing.</p><h3>Conclusion</h3><p>Our team here at Helicap love using MongoDB given the unstructured nature of our data. With Change Streams being introduced to our workflow, we were able to set up an audit trail in a matter of minutes.</p><p>On top of creating an extremely useful tool, the audit trail also serves as proof of concept, that Change Streams can be used to really extend the capabilities of any application using MongoDB.</p><p>We will be elaborating a lot more on how we are improving our application with such technologies. So do watch this space!</p><p>Thank you for reading this article, and we hope that this has helped anyone looking to use Change Streams in their workflow as well.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b5739c33a7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/helicap-engineering/audit-trails-with-mongodbs-change-stream-a3b5739c33a7">Audit trails with MongoDB’s change stream</a> was originally published in <a href="https://medium.com/helicap-engineering">Helicap Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How we chose a tech stack for our startup]]></title>
            <link>https://medium.com/helicap-engineering/how-we-chose-a-tech-stack-for-our-startup-103538e79249?source=rss----82cc3c1b69ad---4</link>
            <guid isPermaLink="false">https://medium.com/p/103538e79249</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Lim Yi Sheng]]></dc:creator>
            <pubDate>Tue, 07 Jan 2020 09:02:15 GMT</pubDate>
            <atom:updated>2020-02-09T17:13:16.398Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome to the first post from Helicap’s engineering team, where we will try to bring you through our tech stack, and explain how we ended up with the choices that we made.</p><p>For those who do not know Helicap, we are a FinTech investment firm specialising in the alternative lending space in the Asia Pacific region. We help people improve their livelihoods by driving financial inclusion and enabling our partner platforms to transform the way consumers and SMEs access credit today.</p><p>To build a startup, the first step is always the hardest; and this goes double for any tech enthusiast in a startup. One of the reasons why software engineers gravitate towards startups is the freedom that it brings. However, this often brings with it the problem of paralysis of choice. Our engineering team too spent quite a considerable amount of time evaluating pros and cons of each technology choice. We were Goldilocks locked in a house with infinite bowls of porridge with an infinite range of temperature.</p><p>As such, we decided that an overarching philosophy was needed first in order to lay common ground, and we will subsequently select each aspect of our tech stack based on our common philosophy.</p><h3>Philosophy</h3><p>At Helicap, we are big fans of <a href="https://en.wikipedia.org/wiki/John_Gall_(author)#Gall&#39;s_law">Gall’s Law</a>:</p><blockquote>A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.</blockquote><p>In short, Keep It Simple, Stupid.</p><p>We are also big fans of YAGNI. You Are Not Gonna Need It. Or in the words of Ron Jeffries:</p><blockquote>Always implement things when you actually need them, never when you just foresee that you need them.</blockquote><p>But that does not mean that we should build things without consideration for the future; it just means that we don’t over-engineer.</p><p>With those two guiding philosophies, we created our own rubrics to judge a technology.</p><ol><li>Simplicity. Is the framework/technology easy to grasp? Does it contain any complexities that we do not need?</li><li>Need Fulfilment. Does the framework/technology fulfil our current needs?</li><li>Extensibility. Even if we are not going to implement certain features, how confident are we that said framework/technology may be able to scale/extend towards the future?</li></ol><h3>Frontend</h3><p>For our frontend choice, after taking into consideration maturity, and community support, the top two contenders came out to be a fight between Angular and React, both of which are always ranked in the top web frameworks to use in <a href="https://insights.stackoverflow.com/survey/2019#technology">StackOverflow’s Developer Survey</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xo-LEsm5__OwzRgw-8xOkw.png" /></figure><p>However, we need to see how they stack up against our rubrics.</p><h4>Simplicity</h4><p>Angular is a full-fledged framework for UI development, whilst React, is a more lightweight library. While we are absolutely confident in Angular’s capabilities, we felt that Angular was more complex to understand, with many opinionated intricacies built into its framework, whilst React was far easier to pick up once a project structure was defined.</p><p><strong>Winner: React</strong></p><h4>Need Fulfilment</h4><p>Both of these technologies really served to fulfil our needs, which was to build a reliable and robust frontend. We think that no one will really go wrong with either one of them.</p><p><strong>Winner: Draw</strong></p><h4>Extensibility</h4><p>Given the maturity of both these technologies, we believe that both of them will be able to fulfil any future needs that we may plausibly have.</p><p><strong>Winner: Draw</strong></p><h4>Our Choice : React</h4><h3>Backend</h3><p>For our backend choice, we had two options that were tabled. The first was Java using Spring Boot. This was a framework that our team felt the most comfortable with, having had some prior experience building applications with it before. Our second choice was one that rose out of the results of some experimentation; and that was Go.</p><p>After a few months of development with each, we felt confident in our ability to judge each technology against our rubrics.</p><h4>Simplicity</h4><p>One obvious difference we noticed is the declarative nature of Spring Boot’s annotations, versus Go’s largely imperative way of coding. For Go, there is little reliance on dependency injection, and everything is handcrafted. Whilst in Spring Boot, most of the dependency injections are handled by annotations and configurations. This is convenient, but at times, these can also invert control and add more complexity when optimising certain aspects of Spring Boot. While, the initial process of having to handcraft Go’s infrastructure had rather annoying pains to overcome; but once we had something of a pattern going on in our code, we could really focus on writing business logic.</p><p><strong>Winner: Go</strong></p><h4>Need Fulfilment</h4><p>Both of them fulfilled our needs quite nicely. Spring Boot has a superb ecosystem, while Go’s libraries really catered to all our requirements.</p><p><strong>Winner: Draw</strong></p><h4>Extensibility</h4><p>Given the maturity of Java, and Spring Boot, we are confident that it can take any number of kitchen sinks that we care to throw at it.</p><p>Go, on the other hand, is still an up and coming language. However, the huge community support, as well as the ability to call C libraries actually gave us a huge sense of security with regards to the extensibility of this language.</p><p><strong>Winner: Draw</strong></p><h4>Our Choice: Go</h4><h3>Database</h3><p>NoSQL vs SQL. Those are murky waters I dread to wade into.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*irNDXJdGj3-rJLlcedx2OQ.png" /></figure><p>But wade into these waters, I must.</p><p>So for our database choices, our team was split rather evenly between Postgres and MongoDB.</p><h4>Simplicity</h4><p>Considering that both of these databases have simple one-click managed services (Aurora PostgresSQL and Mongo Atlas for example), most of the complexities are taken off our hands.</p><p><strong>Winner: Draw</strong></p><h4>Need Fulfilment</h4><p>For our startup, we needed to process documents of varying data types and structures, of which, some of them have a tendency to change sporadically without any notice. In short, we have a lot of unstructured data to process on a regular basis, which NoSQL seems to be the better option for.</p><p><strong>Winner: MongoDB</strong></p><h4>Extensibility</h4><p>Given the maturity and community support of these two technologies, we believe that both of them will be able to fulfil any future needs that we may plausibly have.</p><p>However, one really interesting feature we would like to highlight is the change streams introduced with Mongo 4.x. This opens up possibilities for us to use change streams for audit trails, event sourcing, and in memory caching. We will definitely elaborate on this further in future articles.</p><p><strong>Winner: Draw</strong></p><h4>Our Choice: MongoDB</h4><h3>Conclusion</h3><p>So after a couple of rounds of evaluation and prototyping, we managed to successfully decide on a tech stack that we feel is well positioned to serve our needs. What really saved our bacon, was deciding on a common team philosophy that all of us could really get behind. And once that was established, selecting our choices became a lot simpler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*-TmkAmwM7hKb8qy_ZQBaUA.png" /></figure><p>However, all of us are also cognisant of the fact that all of this is not set in stone. As they say, change is the only constant in a software’s lifecycle! What we want to ensure is that we build our software using established software design patterns and principles. By maintaining the modularity, extensibility, scalability, and resilience of our platform, we are then best equipped to handle any change and future needs of the platform gracefully.</p><p>We will be elaborating a lot more on these principles, as well as various challenges we faced along-the-way and how we overcame them in future articles. So do watch this space!</p><p>Thank you for reading this article, and we hope that this has helped in providing more clarity in choosing a tech stack for your startup too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=103538e79249" width="1" height="1" alt=""><hr><p><a href="https://medium.com/helicap-engineering/how-we-chose-a-tech-stack-for-our-startup-103538e79249">How we chose a tech stack for our startup</a> was originally published in <a href="https://medium.com/helicap-engineering">Helicap Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>