<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ecky Putrady on Medium]]></title>
        <description><![CDATA[Stories by Ecky Putrady on Medium]]></description>
        <link>https://medium.com/@eckyputrady?source=rss-e97af06479d5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fDRZoSGijiVDXmF_CbapoA.png</url>
            <title>Stories by Ecky Putrady on Medium</title>
            <link>https://medium.com/@eckyputrady?source=rss-e97af06479d5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:06:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@eckyputrady/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Prefer Integration Tests Over Unit Tests]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/prefer-integration-tests-over-unit-tests-2079856f9cfc?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2600/0*WtZx63bJW9PxTVM-" width="7500"></a></p><p class="medium-feed-snippet">The pros of not writing unit tests</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/prefer-integration-tests-over-unit-tests-2079856f9cfc?source=rss-e97af06479d5------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/prefer-integration-tests-over-unit-tests-2079856f9cfc?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/2079856f9cfc</guid>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-testing]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Wed, 07 Jun 2023 16:22:21 GMT</pubDate>
            <atom:updated>2023-06-07T16:22:21.834Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Postgres Performance Tips: Reducing Query Overhead]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devgenius.io/postgres-performance-tips-reducing-query-overhead-958cacc614ab?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2600/0*6iwKr5-sCJwggDyK" width="6240"></a></p><p class="medium-feed-snippet">There is more to Postgres performance than setting up indexes.</p><p class="medium-feed-link"><a href="https://blog.devgenius.io/postgres-performance-tips-reducing-query-overhead-958cacc614ab?source=rss-e97af06479d5------2">Continue reading on Dev Genius »</a></p></div>]]></description>
            <link>https://blog.devgenius.io/postgres-performance-tips-reducing-query-overhead-958cacc614ab?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/958cacc614ab</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Sat, 20 May 2023 12:11:46 GMT</pubDate>
            <atom:updated>2023-05-20T12:11:46.383Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How To Use MongoDB with Rust]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devgenius.io/how-to-use-mongodb-with-rust-8bd2fa3f6074?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2600/0*nlLo5SKVKLx61kcL" width="4979"></a></p><p class="medium-feed-snippet">The concise guide for getting things done.</p><p class="medium-feed-link"><a href="https://blog.devgenius.io/how-to-use-mongodb-with-rust-8bd2fa3f6074?source=rss-e97af06479d5------2">Continue reading on Dev Genius »</a></p></div>]]></description>
            <link>https://blog.devgenius.io/how-to-use-mongodb-with-rust-8bd2fa3f6074?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bd2fa3f6074</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[mongodb]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Thu, 18 May 2023 16:45:52 GMT</pubDate>
            <atom:updated>2023-05-18T16:45:52.572Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How To Use Postgres as a Message Queue]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/how-to-use-postgres-as-a-message-queue-5620ee8faa20?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2600/0*rYAPJiHXKIW8nEYD" width="5456"></a></p><p class="medium-feed-snippet">It&#x2019;s not the best tool for the job, but it might be the best for your situation</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/how-to-use-postgres-as-a-message-queue-5620ee8faa20?source=rss-e97af06479d5------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/how-to-use-postgres-as-a-message-queue-5620ee8faa20?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/5620ee8faa20</guid>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[performance-testing]]></category>
            <category><![CDATA[message-queue]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Mon, 15 May 2023 15:53:56 GMT</pubDate>
            <atom:updated>2023-05-15T15:53:56.846Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[PostgreSQL Index Best Practices]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.devgenius.io/postgresql-index-best-practices-f6315e36a1ac?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2048/1*cvwPJVWWDMLjrKpg1rnjZQ.png" width="2048"></a></p><p class="medium-feed-snippet">Foundation and best practices to set up the right indexes for your PostgreSQL database.</p><p class="medium-feed-link"><a href="https://blog.devgenius.io/postgresql-index-best-practices-f6315e36a1ac?source=rss-e97af06479d5------2">Continue reading on Dev Genius »</a></p></div>]]></description>
            <link>https://blog.devgenius.io/postgresql-index-best-practices-f6315e36a1ac?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6315e36a1ac</guid>
            <category><![CDATA[index]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Sat, 25 Mar 2023 15:46:56 GMT</pubDate>
            <atom:updated>2023-03-25T15:46:56.901Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Interact With PostgreSQL From Rust Using SQLx]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/how-to-interact-with-postgresql-from-rust-using-sqlx-cfa2a7c758e7?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/2600/0*bjksNggvZJroFY1Z" width="5384"></a></p><p class="medium-feed-snippet">The important things you need to know to get the job done.</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/how-to-interact-with-postgresql-from-rust-using-sqlx-cfa2a7c758e7?source=rss-e97af06479d5------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/how-to-interact-with-postgresql-from-rust-using-sqlx-cfa2a7c758e7?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfa2a7c758e7</guid>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[rust]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Mon, 23 Jan 2023 18:10:24 GMT</pubDate>
            <atom:updated>2023-01-23T18:10:24.949Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Database Replication: Benefits and Trade-Offs]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/better-programming/database-replication-benefits-and-trade-offs-d929ca4494c5?source=rss-e97af06479d5------2"><img src="https://cdn-images-1.medium.com/max/1280/1*vA522OcViu95t-TcC0mxHA.jpeg" width="1280"></a></p><p class="medium-feed-snippet">Design better systems by understanding the trade-offs and benefits of database replication</p><p class="medium-feed-link"><a href="https://medium.com/better-programming/database-replication-benefits-and-trade-offs-d929ca4494c5?source=rss-e97af06479d5------2">Continue reading on Better Programming »</a></p></div>]]></description>
            <link>https://medium.com/better-programming/database-replication-benefits-and-trade-offs-d929ca4494c5?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d929ca4494c5</guid>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Wed, 09 Mar 2022 16:02:58 GMT</pubDate>
            <atom:updated>2022-03-09T16:02:58.502Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Software Testing Spectrum]]></title>
            <link>https://blog.devgenius.io/the-software-testing-spectrum-d5268b3513cc?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5268b3513cc</guid>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[testing]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Sun, 11 Jul 2021 08:44:59 GMT</pubDate>
            <atom:updated>2021-07-11T08:44:59.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>A better guideline for choosing your software testing strategy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZaCbi3iy9TQJDG7T" /><figcaption>Photo by <a href="https://unsplash.com/@jsshotz?utm_source=medium&amp;utm_medium=referral">Jorge Salvador</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>One of the software engineering practices that I’m against is <strong>Software Test Pyramid</strong>. The recommendation is along the lines of “do have more isolated tests and fewer integration tests”:</p><blockquote>Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end.<br> — <a href="https://martinfowler.com/articles/practical-test-pyramid.html">The Practical Test Pyramid</a> (2018)</blockquote><blockquote>Its (The test pyramid) essential point is that you should have many more low-level <a href="https://martinfowler.com/bliki/UnitTest.html">UnitTests</a> than high level <a href="https://martinfowler.com/bliki/BroadStackTest.html">BroadStackTests</a> running through a GUI.<br> — <a href="https://martinfowler.com/bliki/TestPyramid.html">TestPyramid</a> (2012)</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*ackCJ5NnWTUtai_h.png" /><figcaption>The Software Engineering Test Pyramid from <a href="https://leaddev.com/agile-other-ways-working/demystifying-software-engineering-test-pyramid">leaddev.com</a></figcaption></figure><p>Such a recommendation shouldn’t be applied blindly in every situation. Like everything else, the answer should be “it depends”. In this article, I’d like to articulate the “it depends” part.</p><h3>The spectrum</h3><p>Depending on which test pyramid diagram that you see, you will find multiple types of test: Unit, integration, UI, end-to-end, manual, acceptance, etc. They are quite confusing and sometimes overlapping (e.g., Are UI and end-to-end tests the same?).</p><p><strong>What I find more helpful to think about tests is to lay them out in a spectrum of how many components are involved</strong>, where a component is defined as the smallest testable unit (i.e., a function).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*vNvCtjcYvsVkgWvqs9356Q.png" /><figcaption>The spectrum</figcaption></figure><p>On the far left, there’s only one component involved in the test. We’re only testing one single function in isolation. This corresponds to the bottom part of the test pyramid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*8JuCaxlTTYaImkn70pmv5g.png" /><figcaption>Software Testing Spectrum — The far left</figcaption></figure><p>On the far right, all components in the system under test are involved. Here, we are testing the system end-to-end, even with having the 3rd party system set up for the test. This corresponds to the top part of the test pyramid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*PvFwTNWSFaWh8Kxyi5GVRw.png" /><figcaption>Software Testing Spectrum — The far right</figcaption></figure><p>In the middle, we test a subset of the system that involves more than one component. An example test in this category would be an API-driven test. This corresponds to the middle part of the test pyramid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*wY9l3tgQWUw-Ffhy1uiAww.png" /><figcaption>Software Testing Spectrum — The middle</figcaption></figure><h3>The dimensions</h3><p>Along the same axis, we can add more dimensions that describe the characteristics tendency of the tests:</p><ul><li><strong>Confidence</strong>: How the passing of the tests ensure that the feature “works”. The more components involved, the more confident we are on the test, and vice-versa. (<a href="https://gfycat.com/thoseunsungbeetle">Obligatory unit vs integration test joke</a>)</li><li><strong>Test Churn</strong>: How likely the existing tests are modified when existing codes are modified. The more components involved, the less likely you need to modify the existing tests. It’s because you will tend to test higher-level flows. The change in implementation details will not matter. In contrast, if you write test for each function, you may need to modify existing tests for every refactor.</li><li><strong>Run Cost</strong>: The cost it takes to run the test. The cost here could be system resources, time, or even money. The cost tends to be higher when more components are involved due to longer test startup time and more CPU/RAM/Disk needs to be allocated.</li><li><strong>Case Variations Setup</strong>: How hard it is to set up tests that cover all the various cases. The fewer components involved, the easier it is to set up various cases (e.g., using mocks). On the other hand, the more component involved, the harder it is to set up various cases. For example, if the test case is “should return X when component A throws an error”, it might be difficult to reproduce an environment where component A will throw an error.</li><li><strong>Pinpointing Issue</strong>: How hard it is to pinpoint bugs that cause your test to fail. The more components involved, the harder it is to find which components (could be more than one!) are buggy. If you only test one component, you can easily pinpoint where the bug is.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*241Id5soxp7yR5Mde0NNyQ.png" /><figcaption>The software testing spectrum &amp; dimensions</figcaption></figure><p>The Software Test Pyramid recommends that you should have more tests that fall on the left side of the spectrum compared to the right. However, In Software Testing Spectrum, where your test falls on the spectrum is much less important.</p><p>The important thing is to find a testing strategy that <strong>maximizes the test confidence while minimizes the test churn, the run cost, the difficulty of setting up various cases, and the difficulty of pinpointing issues</strong>.</p><h3>Example</h3><p>Let’s say that we have an e-commerce application. Architecturally, the client is an SPA that talks to the backend through APIs. Among its many features, let’s pick two and decide what testing strategy best suits them. The two features are <strong>user registration</strong> and <strong>cart pricing</strong>.</p><h4>User registration</h4><p>User registration feature is usually simple. User registers with email and password. If the email is already taken, we reject the registration. We also do input validation such as making sure that the password is longer than 8 characters. Otherwise, we register the user in the system.</p><p>Let’s say that we write class-level tests, i.e. tests that fall on the left side of the spectrum. Let’s then try to assign values for each dimension. Note that there’s no exact science on assigning such values. It comes down to your estimate. In my case, here are my estimates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*KBRELz3NIe07OIbeuUtXbw.png" /><figcaption>User registration testing strategy: Exclusively class-based tests</figcaption></figure><p>We have a mid-level of confidence because while we test every class in isolation, we don’t test whether the “plumbing” between all those classes is set up correctly. The test churn is high since class-level tests are easily broken by refactoring. The other dimensions, however, are at their most ideal value.</p><p>Can we do better? What would the dimensions look like if we instead do API-driven tests?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*zc304EZaEkuAtIBFzVDh1g.png" /><figcaption>User registration testing strategy: Exclusively API-driven tests</figcaption></figure><p>Both confidence and test churn moves significantly to their ideal place, while the other dimensions are only marginally worse. Run cost, for example, will now take, let’s say, ~5 seconds instead of ~1.5 seconds. Pinpointing issues and setting up various test cases are not difficult either considering the simplicity of the feature.</p><p>In this scenario, I would skip the class-level tests and just adopt the API-driven tests.</p><h4>Cart pricing</h4><p>In contrast to user registration, cart pricing is much more complex. Many business-critical cases need to be tested. For example, promotions calculation, coupons, item availability, etc. Implementation-wise, a single cart-pricing function could be composed of multiple sufficiently complex functions.</p><p>If we pick the same testing approach as the user registration feature, the dimensions would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*JgvdLuDpbbPm-DDigbB62g.png" /><figcaption>Cart pricing testing strategy: Exclusively API-driven tests</figcaption></figure><p>While the confidence and test churn dimensions are good, the test cases set up and issue pinpointing dimensions are at their worst. For example, if the test fails because the final pricing is off by $0.2, how easily would you know where the bug is?</p><p>A better approach is to combine the lower-level and the higher-level tests. The lower-level tests exercise all the varying test cases as it’s easier to set them up at this level. Additionally, this will greatly reduce the difficulty of pinpointing issues. In contrast, the higher-level tests only exercise few test cases. This boosts confidence that the feature is working correctly, at least for the few test cases that we set up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*gFqkfsVlYR27JTl1kQY0oQ.png" /><figcaption>Cart pricing testing strategy: Class-based and API-driven tests combined</figcaption></figure><p>In the end, <strong>this looks exactly like what Software Test Pyramid recommends</strong>: More isolated tests and fewer integration tests.</p><p>However, what I want to emphasize is that such a recommendation is not always the most applicable in every situation. It makes sense for this feature but makes less sense for the user registration feature.</p><h3>Conclusion</h3><p>This article introduces <strong>Software Testing Spectrum</strong> as an alternative guideline to <strong>Software Test Pyramid</strong> for choosing the right software testing strategy. While Software Test Pyramid recommends having more isolated tests and fewer integration tests, Software Testing Spectrum recommends choosing a test strategy on case-by-case basis by considering its impact on various dimensions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5268b3513cc" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/the-software-testing-spectrum-d5268b3513cc">The Software Testing Spectrum</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Data-Backed Ways To Significantly Speed Up Your MySQL Bulk Inserts]]></title>
            <link>https://medium.com/data-science/3-data-backed-ways-to-significantly-speed-up-your-mysql-bulk-inserts-28e20b42d51e?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/28e20b42d51e</guid>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Sun, 30 May 2021 22:29:25 GMT</pubDate>
            <atom:updated>2021-05-30T22:29:25.607Z</atom:updated>
            <content:encoded><![CDATA[<h4>See what the data says about the best ways to optimize MySQL bulk inserts.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*isaaBs2eRW3sRdPp" /><figcaption>Photo by <a href="https://unsplash.com/@claybanks?utm_source=medium&amp;utm_medium=referral">Clay Banks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In Data Science projects, a common last step in the data pipeline is to persist the result into a database (e.g. MySQL). The result of this data pipeline is usually big, so optimizing the writes into the database is important to achieve the acceptable pipeline latency.</p><p>I’ve benchmarked and analyzed many MySQL bulk insert setups to get insights on what really matters to significantly speed up such workload. In this article, I’ll share them so you can save time on researching and immediately start to implement the right setup. Interested? Read on.</p><h3>1. Defer Building The Index</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*KO_bjZLep4Y1egZKcESYJg.png" /></figure><p>Index is extremely bad for inserts. In the graph above, index=EARLY creates the index before any of the inserts are done, while index=LATE creates the index after all of the inserts are done. We can see that:</p><ol><li>As the input size gets bigger, inserting into tables with index <strong>doesn’t scale</strong> as well as tables without index.</li><li><strong>The gap between the two is 1 hour</strong> when the input size is 50 million rows.</li></ol><p>The graph also suggests that the table with index has non-linear latency growth. Does writing to tables with index gets slower over time?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*jK3XV1XdGT5R_tmBZI5erQ.png" /></figure><p>Oh yes, it does. As the input size is bigger, the write rate is getting slower. Contrast this to the table without index where the write rate is consistent.</p><h3>2. Inserts Concurrently</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*PTaIkucdmqBPA6QrffRWUw.png" /></figure><p>You can <strong>cut up to 50% latency</strong> from your bulk inserts by inserting concurrently to tables without index.</p><p>From the figure above, we also see another downside of index: <strong>concurrent inserts into tables with index have no advantage or even worsen the latency</strong>.</p><p><strong>Concurrent inserts are surprisingly better compared to LOAD DATA</strong>. This sounds unbelievable given the endorsement LOAD DATA got in the MySQL reference, but it is what it is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*joC1DiPF619uSWkKTRaQQA.png" /></figure><p>Well, to be fair, LOAD DATA is still a lot faster compared to single-threaded inserts. However, it’s just not as fast as 4 concurrent inserts.</p><h3>3. Use Larger Batch Size With The Right Frequency of Commits</h3><p>Batching is the use of extended inserts (insert into table values (1, 2), (3, 4)). The impact? Up to <strong>83% latency reduction</strong>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*VuKNXzQfPBM_Ct1ykcI55A.png" /></figure><p>Now, let’s talk about <strong>COMMIT</strong> operation. When you ask the database to COMMIT, the database will perform a log flush to disk, which “formally” persists the data in the database.</p><p>Ideally, we only commit once: At the end of the workload. However, it isn’t always practical. Consider this: If you have so many data that your bulk inserts take 6 hours to run, you can’t always guarantee that the database connection is stable on that long 6 hours. Once it disconnects, you lose the whole progress and need to restart from the beginning. So, a more practical way is to COMMIT every few rows.</p><p>Since log flush operation is slow, then fewer commits should speed up the overall workload, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*yh-7FoFgHOR8UMD98qjJZQ.png" /></figure><p>Surprisingly, No. More frequent commits are slightly faster than less frequent commits. However, the trend doesn’t hold if we tune down the batch size from 30,000 rows to 300 rows as we can see in the next figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*pWONX24uI9IJ-LmNU5q2NA.png" /></figure><p>The takeaway is that there’s an ideal number of rows (or, to be precise, bytes) before a COMMIT should be performed.</p><h3>4. Bonus: Use Auto-Incrementing Primary Key</h3><p>One tip that I’ve received is that the auto-increment primary key is faster than no primary key at all. But really, how much faster it is?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*_AKfAA967kcDjiI-PXE0cQ.png" /></figure><p>Answer: Not much at all, but still is the fastest.</p><p>The slowest setup is when the primary key is supplied along with the data in an unordered manner. Even then, it’s not significantly worse.</p><p>It kinda makes sense though, supplying the primary key in the data means you send more bytes to the database. The small extra bytes per row stack up once you have millions of rows.</p><h3>Conclusion</h3><p>In this article, we’ve learned 3 ways that give you the biggest ROI for reducing latency on bulk inserts workload. They are:</p><ol><li>Defer building the index</li><li>Inserts concurrently</li><li>Use larger batch size with the right frequency of commits</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28e20b42d51e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/3-data-backed-ways-to-significantly-speed-up-your-mysql-bulk-inserts-28e20b42d51e">3 Data-Backed Ways To Significantly Speed Up Your MySQL Bulk Inserts</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rendering Charts in Laravel Applications]]></title>
            <link>https://medium.com/better-programming/rendering-charts-in-laravel-applications-b6cd1ef982f6?source=rss-e97af06479d5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6cd1ef982f6</guid>
            <category><![CDATA[php]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[laravel]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ecky Putrady]]></dc:creator>
            <pubDate>Tue, 02 Mar 2021 15:35:15 GMT</pubDate>
            <atom:updated>2021-03-02T15:35:15.145Z</atom:updated>
            <content:encoded><![CDATA[<h4>The easiest and quickest way to render charts in Laravel apps</h4><figure><img alt="Man viewing chart on tablet" src="https://cdn-images-1.medium.com/max/1024/1*xTJtSQEGM4gkapqbeMRhAA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@adeolueletu?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Adeolu Eletu</a> on <a href="https://unsplash.com/s/photos/analytics?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>.</figcaption></figure><p>People say that a picture is worth a thousand words. That couldn’t be truer for charts, which allow you to condense data into a meaningful message that one can easily digest.</p><p>In this article, we will discuss how to render charts in a <a href="https://laravel.com/">Laravel</a> application. The approach that we will take requires zero library installation, which translates to a quick and easy way to render the charts.</p><h3>What We Will Build</h3><figure><img alt="Example report page with charts" src="https://cdn-images-1.medium.com/max/812/1*N837sXAyRWOB5qRLtAvGdw.png" /><figcaption>Example report page with charts</figcaption></figure><p>Let’s pretend that we are building a report page for an online store. This report might contain:</p><ol><li>Monthly orders — The number of orders placed by customers per month</li><li>Monthly revenue — The amount of revenue generated per month</li><li>Share of products — A list of the top-selling products within a period</li></ol><p>The report is accessed as a web page in a Laravel application.</p><h3>Technology Selection</h3><p>There are few approaches to rendering charts in Laravel. For example, using a JavaScript frontend library such as <a href="https://www.chartjs.org/">Chart.js</a>, <a href="https://www.highcharts.com/">Highcharts</a>, or <a href="https://vega.github.io/vega-lite/">Vega-Lite</a>. However, this has the drawback of including the library in question in the frontend. For clients whose JavaScript is deactivated (e.g. emails), this is a no-go.</p><p>Another approach would be to render the chart in the server and send it to the client as an image. There isn’t much competition in this space. One option that I can find is <a href="https://github.com/szymach/c-pchart">c-pchart</a>, which unfortunately feels too verbose for my taste. The downside of rendering the chart in the server is the increased processing power required for the server. Rendering charts is not as cheap as a typical server workload (getting data from a DB, transforming, etc.).</p><p>Another approach is to use <a href="https://getchart.me/?m=mdm.laravel.0">GetChart.me</a> for rendering the charts. In short, GetChart.me is a service that allows us to get a chart image from a chart definition encoded in a URL. This means embedding an image in an HTML page is as easy as &lt;img src=&quot;{url}&quot;/&gt;. The downside of this approach is having a dependency on an external service, which means the availability of chart images is subject to the service&#39;s availability.</p><p>In this article, we will go with GetChart.me for the following reasons:</p><ol><li>It’s easy and quick to get started — Just build a URL and embed it as an image in the HTML document.</li><li>No JavaScript — The charts can cover more clients (e.g. emails).</li><li>Uses open source chart definition (Vega-Lite) — If the service is down, we can fall back to the “client-side library” approach by adding Vega-Lite to the HTML page.</li></ol><h3>Tutorial</h3><p>I assume you already have a Laravel app set up. If not, you can set one up by following the <a href="https://laravel.com/docs/8.x#your-first-laravel-project">official guide</a>.</p><h4>Setting up a Blade template</h4><p>First, let’s create a new <a href="https://laravel.com/docs/8.x/blade">Blade template</a> named report.blade.php in the resource/views folder. As the name suggests, this template will display the report page. The content of the Blade template is as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/17c4210a5bd6fbac10e5956b2c73df15/href">https://medium.com/media/17c4210a5bd6fbac10e5956b2c73df15/href</a></iframe><p>There’s nothing fancy here. As you can see, the charts are basically images pointing to some URLs. The URLs will be passed in from the controller via three template parameters: monthlyOrdersImageUrl, monthlyRevenuaImageUrl, and shareOfProductsImageUrl.</p><h4>Creating the controller and wiring up the Blade template</h4><p>Create a new file named ReportController.php the in app/Http/Controllers folder. Put the following code inside the file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0515304b02019dbd1229651d74e68385/href">https://medium.com/media/0515304b02019dbd1229651d74e68385/href</a></iframe><p>From the source code, we wire up the Blade template that we created earlier using view(&#39;report&#39;, ...). We also pass in the three template variables, although they are all empty strings for now.</p><h4>GetChart.me URL helper function</h4><p>Before making the actual chart, let’s create a helper function for generating the GetChart.me URL.</p><figure><img alt="Diagram showing how GetChart.me works" src="https://cdn-images-1.medium.com/max/635/1*FYNSDHXHzokk3QlOFst7NA.png" /><figcaption>Diagram showing how GetChart.me works</figcaption></figure><p>The way GetChart.me works is by sending a URL-encoded Vega-Lite chart definition as a URL parameter. In PHP, we implement it as a simple one-liner like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8400bd505293eeede62fb086f417c415/href">https://medium.com/media/8400bd505293eeede62fb086f417c415/href</a></iframe><p>This function receives a Vega-Lite chart definition (in PHP maps) and outputs a valid GetChart.me URL.</p><h4>Creating the chart</h4><p>Creating a chart involves two things:</p><ul><li>Defining how the chart will look.</li><li>Defining the data for the chart.</li></ul><p>The data for the chart can come from anywhere. A database is a typical data source. In this article, we will just hardcode the data for conciseness.</p><p>Defining the chart visuals is basically the same thing as writing a Vega-Lite chart definition. There are numerous charts in <a href="https://vega.github.io/vega-lite/examples/">Vega-Lite examples</a>. The <a href="https://vega.github.io/vega-lite/docs/">library’s documentation</a> is pretty solid too. In our case, we are using a bar chart, a line chart, and a horizontal bar chart.</p><p>The code for creating the charts is then as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f747b3a31bbf328d384df1b8372a9c2c/href">https://medium.com/media/f747b3a31bbf328d384df1b8372a9c2c/href</a></iframe><p>Refresh the page. You should see that the report page is now looking very useful with charts.</p><p>If you’ve successfully reached this point, then we are done. Pretty easy, right?</p><h3>Conclusion</h3><p>In this article, we created a report page containing charts in Laravel. There are various ways to render charts in a Laravel application: with a client-side chart library, server-side chart rendering, or a third-party chart-rendering service (GetChart.me). We chose to use GetChart.me due to its low complexity and portability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6cd1ef982f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/rendering-charts-in-laravel-applications-b6cd1ef982f6">Rendering Charts in Laravel Applications</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>