<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adinata Thayib on Medium]]></title>
        <description><![CDATA[Stories by Adinata Thayib on Medium]]></description>
        <link>https://medium.com/@dieend?source=rss-8e8bc291fb9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*1b7JU_-vb7e0mmagbY4M2Q.jpeg</url>
            <title>Stories by Adinata Thayib on Medium</title>
            <link>https://medium.com/@dieend?source=rss-8e8bc291fb9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:10:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dieend/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Running Unit Test on Top of Serverless Service]]></title>
            <link>https://medium.com/pocket-gems/running-unit-test-on-top-of-serverless-service-d8c223077512?source=rss-8e8bc291fb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8c223077512</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[parallel-computing]]></category>
            <category><![CDATA[testing]]></category>
            <dc:creator><![CDATA[Adinata Thayib]]></dc:creator>
            <pubDate>Fri, 03 Apr 2020 15:01:00 GMT</pubDate>
            <atom:updated>2020-05-04T20:44:35.477Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B0-eS8y3XB5f_fZjl22vog.png" /></figure><p>Pocket Gems hit its 10th year in 2019, and our game server unit tests have numbered 2,756 to date. Three years ago, it took 30 mins to run a full unit tests suite with a 2016 i7 MacBook Pro. It currently takes 23 mins to run the full suite with a 2018 MacBook Pro (2.9 GHz Intel Core i9, 32GB RAM, 1TB SSD) with 4 process parallelizations (user CPU time is &gt; 90 minutes).</p><p>This is a pain point on our deployment, and we have horror stories because, in the past, engineers skipped running unit tests before deployment. To accommodate our growing unit tests, we created a tool that enables running the full unit test suite in less than 3 minutes.</p><h3>Stateless Unit Tests</h3><p>Each unit test can be run independently with each other. In theory, you can run each single unit test in parallel at the same time. It will then limit the number of CPU cores or your one slowest test.</p><p>We can put this into practice, where we can spawn multiple server nodes that run the unit tests. It sounds easy, but in reality, many things can happen. What would you do in the case of node failure? How would you monitor the node’s health? Do you need an auto-scaling policy? How much will it cost?</p><p>Nowadays, technology has gone in the direction of e serverless computing. Examples of such services are AWS Lambda and Google Cloud Function. The good thing about these services is that they charge you only when you are using it. You also don’t need to monitor the node health, because usually, the system is short-lived, which means it only runs when you need it.</p><p>In general, we can categorize serverless architecture usage in two use cases:</p><ul><li>Webserver with API gateway</li><li>Short-lived event-driven function</li></ul><p>We had an idea to utilize its parallelization to run unit tests. However, unit tests are not a webserver, and it’s not precisely short-lived.</p><h3>Implementation</h3><p>We ended up being creative, by creating a simple function that takes two parameters, shard number, and total shards and runs a different set of unit tests based on that given value. We created a unit test plugin that shard unit tests into a different set of unit tests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2PvlQ0HfkcxXYLN8" /></figure><p>It allows us to run only a subset of unit tests with the given parameter. To allow parallel execution of two different sources code, each engineer owns a single function (or more) in Lambda. Every time we run a full unit tests suite, it’ll upload the current code base into the function, and we can execute hundreds of shards in parallel.</p><p><strong>1. Upload Codebase</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lXF6WUYLtoVZd-oK" /></figure><p><strong>2. Send Request</strong><br>request = [TestRequest(idx, n_shard).start() for idx in xrange(n_shard)]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/0*7sT6WWMJXQEOqdg8" /></figure><p><strong>3. Combine Output</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/0*GvaLazjwne4xKcbW" /></figure><h3>The Challenge</h3><p>When implementing this architecture, we encountered multiple issues:</p><ol><li><strong>Source Code Packaging</strong><br>Our codebase is not pure Python and our first try failed because of the incompatibility of the binary dependencies. We had to re-compile the binaries using the platforms where the Lambda function is running and store it as a dependencies zip file.</li><li><strong>Maximum package size in Lambda limit</strong><br>We were uploading our full virtualenv folder, and that caused the package size to bloat. We filtered out non-essential files before zipping them together with dependencies zip.</li><li><strong>Multi-User</strong><br>If multiple users are trying to update the function while someone else is running the unit tests, it can cause inconsistencies. We handled this by having a unique function per user.</li></ol><p>We are considering open-sourcing this tool, but our requirements and situations are pretty unique in allowing this to happen.</p><ul><li>Pure Python integration testing</li><li>Small 3rd-party dependencies (rare updates of the dependencies)</li><li>Deployment happened multiple times a day and running the whole unit tests suite takes forever</li></ul><p>If you are also in a similar situation, comment below and let us know to consider open sourcing this tool.</p><h3>Conclusion</h3><p>This tool allows us to save a lot of engineering time. To this day, the tool is already being used by over <strong>70 different engineers</strong>, with roughly 30 executions a day. Assuming the situation at the beginning constant, that each run saves ~25 minutes of running time, we had saved 12.5 hours/day of engineering time (roughly $600/day, &gt; $10k/month), while costing us only ~$100/month. The result is, profit!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*478CFkRJLaLoJy3v" /></figure><p>Personally, what I like the most about this tool is my laptop resources are not being used at all. I don’t have to listen to the CPU fan spinning when running the whole unit test suite. I also don’t have the frustration of waiting for the unit tests to finish running before being able to do something else.</p><p>We, as a company, are continuously iterating and improving our products. As part of that, we are also improving our quality of life, including building productivity tools that remove our pain points like this.</p><p>If you are interested in working in a highly productive team, <a href="https://grnh.se/35c5f2151us">we are hiring</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8c223077512" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pocket-gems/running-unit-test-on-top-of-serverless-service-d8c223077512">Running Unit Test on Top of Serverless Service</a> was originally published in <a href="https://medium.com/pocket-gems">Pocket Gems Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pocket Gems Infrastructure: Building a Highly Concurrent Transactional Virtual World]]></title>
            <link>https://medium.com/pocket-gems/pocket-gems-infrastructure-building-a-highly-concurrent-transactional-virtual-world-7af4c9551327?source=rss-8e8bc291fb9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7af4c9551327</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[google-app-engine]]></category>
            <category><![CDATA[google-datastore]]></category>
            <category><![CDATA[google-pubsub]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Adinata Thayib]]></dc:creator>
            <pubDate>Thu, 05 Sep 2019 16:37:33 GMT</pubDate>
            <atom:updated>2020-07-20T18:24:24.888Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JP36mZTRGv5863zGG7ax9w.png" /></figure><p>At the end of 2017, War Dragons released <a href="https://wardragons.com/blog/120/atlas-upcoming-launch-announcement">Atlas Expansion</a>. It’s a massive virtual world created for highly engaged players where they battle against each other in a persistent world. You can raid your enemy’s castle and take what was theirs as yours. The expansion is highly social, and you need to collaborate with 49 other teammates to decide who will be the main attacker, and who will be the defender.</p><h3>Technical Challenge</h3><p>To model the virtual world, we store each castle as a single model, and each player has a “physical” state in the castle. Any player in a team can attack any other player from another team as long as they are physically nearby. These attacks can occur sporadically, but during a war, it occurs in a high volume.</p><p><strong>In summary, the problem is simplified into the following statements</strong>:</p><ul><li>We have a persistent world</li><li>Imagine each castle as a single row/entity in the database</li><li>In theory, every player not in your team can attack the castle</li><li>Each attack from the player can affect the same row/entity</li></ul><p>Pocket Gems servers are hosted on<a href="https://cloud.google.com/appengine/docs/standard/runtimes"> Google App Engine Standard</a> using Python 2.7 Runtime. It allows us to build a highly scalable system and focus on delivering the best product to our players. We use Google Datastore, which is a non-relational database built on top of Google BigTable, which has limitations for<a href="https://cloud.google.com/datastore/docs/concepts/limits"> write rate to an entity is 1/s</a>.</p><h3>Gameplay &amp; Technology Progression</h3><p>In the alpha version, traveling from one area to another area takes time (it can take hours to move from one area to another area). We assumed that the travel times would hide the contention from the player’s perspective. Whenever there is a contention, we retry the “actions” ( — termed WorkItems) until it eventually succeeds. For that, we used the task queue to store the WorkItems being executed by the player and keep the WorkItems there until it is completed.</p><p>The approach worked okay on a smaller scale, but it’s become much more noticeable when we add more players. The contention would cause a holdback in the queue, snowballing the effect to the whole handlers.</p><p>We realized we couldn’t keep ignoring the problem, especially when we need to change the gameplay to change the traveling time to almost none (from hours to less than 5 seconds of animation).</p><p>These precious 5 seconds give us the chance to finish the database transaction.</p><h3>Reducing Scope</h3><p>From the problem statement, we can roughly translate it to <em>number_of_guilds * number_of_player transactional</em> update per second or almost 60K writes/second to a single entity in a database. The high frequency means that there are high possibilities for database contention.</p><p>60K attacks at the same time are bound to have a slim chance to happen. To understand the actual scope of the problems, we would need to look at the real-world situation.</p><p>Guild wars happen between 2 different alliances, which usually occur in a cluster of areas. To optimally win the battle, you would need to spread the team attack. A single attack will finish between 3–4 minutes and ends with 5 seconds of animation. In these situations, WorkItems will spread out between 3–4 entities, of 600–1000 updates/minutes.</p><p>Based on this fact, we can reduce the scope to<strong> allow 50 updates in a single entity to complete transactionally in 5 seconds</strong>. One approach is to batch those 50 WorkItems into a single transaction.</p><p><strong>The problem changes into:</strong></p><ul><li>Retrieving all update actions for a single entity</li><li>Batching the WorkItems in a scalable way</li><li>Delivering the transaction results to all affected client</li></ul><h3>Single-Queue Worker</h3><p>For every update action, we put the WorkItems into a queue (Kafka/Google PubSub/Kinesis). In a second window, we will group the WorkItems based on the entity we want to update. Then we do the process in batch, make it into a single update, and apply it in a transaction.</p><p>The easiest way to ensure it doesn’t have contention is by having a single worker that gets all the WorkItems from the queue and then update the entity in batch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OyYTshCk7BWHS26J" /></figure><p>We thought that just having a single queue was not scalable because we can’t fetch WorkItems for only a specific area. The WorkItems also compute-intensive where we need to check the existing state and do battle logics to calculate the battle reward. Having a single worker might cause a delay in processing because of a CPU bottleneck.</p><h3>Multi-Queue Worker</h3><p>Multi-Queue Worker is an extended version of the Single-Queue Worker approach. Rather than having a single queue for all WorkItems and entities, we shared it, so that a subset of WorkItems will be handled by a single queue. An individual worker will process each queue. Google AppEngine Standard doesn’t allow you to spawn a new process, so we used thread as the worker.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TK8XfWn6isKoLMsJ" /></figure><p>The thread runs on top of Google AppEngine manual scaling, and it pulled the data from Google Pub/Sub. We used manual scaling as the workers because Google Datastore can only be accessed using the AppEngine.</p><p>Each thread will pull multiple WorkItems from a single queue, execute numerous WorkItems, and persist the update into datastore at the end of the transaction.</p><p><strong>Using a single-queue worker that batches actions for each entity alleviates the contention issue, and allows us to better the player experience in the high load area.</strong></p><p>Unfortunately, it created a new problem. From our monitoring, we found that some action in areas had a delay in being processed. The workers were still active, and the WorkItems were in the queue. This issue happened sporadically to one or two areas, and returned to normal variably after 15–45 minutes, or if we restarted the workers.</p><p>After researching into the root cause, we hypothesized the root cause is<a href="https://mail.python.org/pipermail/python-dev/2009-October/093321.html"> python GIL</a>.</p><blockquote><em>“… This, with the opcode counting scheme, is the reason why some people have been complaining about latency problems when an I/O thread competes with a computational thread (the I/O thread wouldn’t be scheduled right away when e.g. a packet arrives; or rather, it would be scheduled by the OS, but unscheduled immediately when trying to acquire the GIL, and it would be scheduled again only much later). …”</em></blockquote><p>We had IO threads (pulling from Google Pub/Sub subscription) that competed with CPU thread (which executed the actions) and caused the sporadic delay.</p><h3>Multi-Queue with External Worker</h3><p>Using process instead of thread can fix the GIL issue. Unfortunately, we can’t use the multiprocessing module because of the limitation of Google AppEngine Standard.</p><p>To fix the issue, we migrated a subset of the system. The execution of actions still needs to be in the Google AppEngine, as it requires interaction with existing game logic. Router/Forwarder and Batcher can be extracted outside of the system to be a standalone microservice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pU04FNzC6Ygtqz92" /></figure><p>The Batcher will send a list of actions to be executed in a single transaction to the Batch Handler webserver.</p><p>This approach allows us to separate the IO thread and CPU thread, with a small added latency. Its latency is low enough that it is no more an IO-bound application, but CPU-bound.</p><p>On an additional note, the battle doesn’t happen all the time in all the areas. We used auto-scaling to optimize the resource. When a dormant area becomes active, we start the Batcher for that area. We used containerized worker as the Batcher. We also used Kubernetes to manage the container and the autoscale policy.</p><p>When the area is back to dormant, the container will stop, and the autoscale policy is updated.</p><h3>Conclusion</h3><p>Making the Forwarder and the Batcher a standalone microservice enables us to use the same system on another part of the game.</p><p>We were iterating the system for six months, implementing what works, and tackling what didn’t work one by one. In doing so, we were able to learn a substantial amount throughout the development process.</p><ul><li>Using a single-queue worker and batched multiple actions in a single transaction can address the contention issue and allow us to run 6 actions/s.</li><li>The way Python GIL in Python 2 work can cause IO thread blocked by CPU thread</li></ul><p>With this architecture, we were able to decrease the battle processing 99th percentile latency of 1 hour to 20s, and the average latency down to 2s.</p><p>Join our team! Pocket Gems is <a href="https://pocketgems.com/jobs/">hiring</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7af4c9551327" width="1" height="1" alt=""><hr><p><a href="https://medium.com/pocket-gems/pocket-gems-infrastructure-building-a-highly-concurrent-transactional-virtual-world-7af4c9551327">Pocket Gems Infrastructure: Building a Highly Concurrent Transactional Virtual World</a> was originally published in <a href="https://medium.com/pocket-gems">Pocket Gems Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>