<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kim Gault on Medium]]></title>
        <description><![CDATA[Stories by Kim Gault on Medium]]></description>
        <link>https://medium.com/@kimgault?source=rss-f9d76302c088------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XitnrS36QWqx-qB_5uEm-g.jpeg</url>
            <title>Stories by Kim Gault on Medium</title>
            <link>https://medium.com/@kimgault?source=rss-f9d76302c088------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:18:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kimgault/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Build a High-Throughput System With DynamoDB]]></title>
            <link>https://medium.com/swlh/how-to-build-a-high-throughput-system-with-dynamodb-d8067e12ea00?source=rss-f9d76302c088------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8067e12ea00</guid>
            <category><![CDATA[nosql-database]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[high-throughput]]></category>
            <category><![CDATA[dynamodb]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Kim Gault]]></dc:creator>
            <pubDate>Wed, 04 Nov 2020 07:19:53 GMT</pubDate>
            <atom:updated>2021-11-23T21:19:47.880Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to Utilize DynamoDB Correctly When You’ve Exclusively Worked with Relational Databases</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*AQRFKxBUeS0zT7Jl9VqaQQ.jpeg" /><figcaption>A visual interpretation of a database in the 1995 film, Hackers. How far we’ve come!</figcaption></figure><p>Relational databases have long dominated the field of software engineering, but the adoption of NoSQL databases has been rapidly catching up. NoSQL (also known as “Not Only SQL”) databases emerged as a solution to the growing limitations of relational databases and store data differently than relational databases. In 2012, Amazon unveiled its managed database service, DynamoDB, joining popular databases MongoDB and Apache Cassandra in the category of NoSQL databases. This article will focus specifically on DynamoDB.</p><h3>NoSQL Database vs. Relational Database</h3><p>A common mistake developers make when when comparing the two databases is coming to the conclusion that one type of database is better than the other. They have different strengths and weaknesses that it doesn’t make sense to make that conclusion without factoring in the specific circumstances of how the database will be used.</p><p>Relational databases have fixed schemas. NoSQL databases do not have fixed schemas. Generally, NoSQL databases offer flexibility when storing data that comes at the cost of less flexible queries. Relational databases may not have the same flexibility when storing data, but it offers flexibility with its data queries. The data queries does come at a higher cost than NoSQL databases. Relational database queries tend to be slower and can’t handle high-throughput as well as NoSQL queries. If your data is low volume and have a consistent pattern, a relational database can be the better choice.</p><p>Another difference between the two databases is how each type of database scales. In most growth situations, relational databases tend to rely on scaling vertically by adding more resources to the server while NoSQL databases are meant to be scaled horizontally by adding more instances of the database. Scaling vertically is usually more expensive and can cause more disruption in production.</p><p>Both of NoSQL’s advantages in relation to performance and horizontal scalability comes at a cost to reliability. If you are looking for a database with ACID (Atomicity, Consistency, Isolation, Durability) Compliancy, a relational database is the better choice for a database.</p><p>Essentially, when prioritizing scalability, performance or schema freedom, NoSQL is the stronger contender when choosing between relational and NoSQL.</p><h3>Why Choose DynamoDB</h3><p>Of the many NoSQL databases to choose from, there are a few factors that make DynamoDB preferable to the rest:</p><h4>Fully Managed by Amazon</h4><p>DynamoDB differs from some of the other NoSQL databases by being fully-managed by Amazon. This means servers are provisioned by Amazon. Availability and tolerance are built into DynamoDB by Amazon. It also has the unique benefit of having one of the only pay-per-use models for NoSQL databases. On the other hand, it can also be a black box. This may not be a good option depending on how much low-level control a company would like for its database.</p><h4>Plays Well in Serverless Ecosystem</h4><p>Another driving force for choosing DynamoDB is the adoption of AWS Lambda. When considering databases in the serverless ecosystem, there are only two choices: DynamoDB and Aurora. Amazon released Aurora Serverless shortly after DynamoDB as AWS’s relational serverless database. An AWS serverless database, such as DynamoDB and Aurora, and AWS Lambda work better together than with a database outside of the serverless ecosystem.</p><p>The serverless ecosystem is about hyper-ephemeral computing. As instances are created and destroyed with each call, it can be difficult and costly to have a consistent TCP connection with a traditional database server.</p><p>Credential management is very straight-forward when all products are Amazon severless products. DynamoDB uses AWS IAM for authentication and access will be granted to AWS Lambdas by an IAM policy. If it were a MySQL database on its own instance, each lambda would have to retrieve the database credentials to connect every time. Additionally, when the database is not available to the public internet, the lambda will have to be behind the same VPC as the database. This results in prolonged cold starts.</p><h4>Better Scalability and Reliability</h4><p>Although DynamoDB is the perfect accompanying tool for AWS Lambda, DynamoDB should be considered regardless if the application is in a serverless ecosystem or not. First and foremost, NoSQL databases were created to out scale a relational database.</p><p>When considering databases for reliability and scalability, DynamoDB and Cassandra are favored contenders. DynamoDB and Cassandra both support a multi-primary model, while NoSQL databases like MongoDB support a single primary model. Databases with multi-primary models scale better and are more resilient than their peers. Write access is limited only to primary servers. Since MongoDB follows the single primary node model, write access is limited. Deploying more shards will increase write capability, but it will always be smaller in comparison than to a multi-primary model.</p><p>DynamoDB and Cassandra also handle failure and replication better. When a primary table goes down, one of the replicas is selected as a primary. This process results in downtime for write access. With a multi-primary table, this isn’t an issue.</p><h3>How to Setup DynamoDB for Success</h3><p>Many companies choose DynamoDB for its effectiveness as a high-throughput database, but then fail to set it properly up for success. DynamoDB can query data more efficiently than a relational database, but only in the way it was intended to query. Outside of those intended ways, DynamoDB is burdensome in both cost and time.</p><h4>Breaking Away from the Relational Mindset</h4><p>When designing a relational table, normalization is one of the most important aspects of designing a database. The table is completely designed and populated before you even think about access patterns. Access patterns aren’t a consideration until building the query. DynamoDB is the opposite. In DynamoDB, the design of the database is based on access patterns. The hardest part of designing a DynamoDB database is forcing developers to break away from the relational and more intuitive mindset when designing the database.</p><p>Generally speaking, most applications will be able to design a database where there is the same activity across all partition keys in the table and its secondary indexes, and most applications should be able to store all data per system using a one table design. Accomplishing this design pattern will require creative and non-traditional planning prior to building the database.</p><p>A great method is to keep attribute names agnostic so that partition and sort keys can be reused for multiple types of records. Utilizing secondary indexes is another way to leverage multiple access patterns in one table. Another important key difference is data redundancy. Relational database developers have been taught to avoid redundancy in relational table design. Redundancy is allowed and encouraged for DynamoDB as it can help fulfill the required access patterns needed by the business.</p><p>A good (albeit not quite official) indicator for success when designing for a DynamoDB table is when the table becomes no longer as readable as a relational table. For instance, to maintain a sort key that encompasses the access pattern of multiple type of records, a common pattern used for sort key attributes is “{Type}#{attribute-specific sort data}”. Records can now be retrieved that match a type and either part or all of the specific sort data. It’s a flexible yet unintuitive way to create database design and will leave the DynamoDB table in a very unreadable state, especially if the reader is used to the almost Excel-like readability of a relational table. Key attributes will also seem foreign, consisting of hashes and other data purely used for sorting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/0*woyhcm1bPQGBdXaN" /><figcaption>An example of DynamoDB records from Amazon.</figcaption></figure><p>Most business requirements can be accomplished by querying or doing a get by primary key if enough consideration is placed into access patterns when designing the database records. If the application is doing a scan in DynamoDB for a task other than migrations, something is wrong.</p><p>For those looking to adopt DynamoDB, Amazon offers a great article for best practices for many-to-many relationships <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-adjacency-graphs.html">here</a>. To read more about best practices for designing and using partition keys effectively, <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-partition-key-design.html">click here</a>.</p><p>The philosophies behind a relational table design and a DynamoDB design are completely different. Data structures should be tailored to the specific requirements of use cases with DynamoDB, whereas with a relational database, query optimization has nothing to do with schema design. Almost all of the work for DynamoDB is in planning the data model up front.</p><h4>Build Out the Application Layer</h4><p>There are multiple possible drawbacks to designing a table based on access patterns rather than relationships.</p><ol><li>Bad data could be written because there is no schema enforced on the table level.</li><li>A one table design leads to unreadability.</li><li>Adding an access pattern or a data migration after the database is written might be costly and time-consuming.</li></ol><p>These drawbacks will be reduced in the application code. Validation and readability is in the table layer for a relational table. For the DynamoDB table, the validation is in the application layer. It’s very important to code efficiently and with scalability in mind when coding the data models and database layer so that everything can be optimized.</p><p>For example, classes can be created for each entity type. Each class type will know how to transform the entity object into a DynamoDB record, as well as validate itself. It can also map the response it gets when retrieving a DynamoDB record into a readable object for the application. A lot is happening in the application code before it gets to DynamoDB.</p><p>Configuring unplanned access patterns or data migration is hairy regardless of the type of database, but it can be more difficult with DynamoDB. It’s why the planning phase of DynamoDB is so important. Considering all access patterns and keeping the partition and sort key agnostic so those keys can be reusable plays a big part in the success of DynamoDB.</p><p>When an application’s database and data model layers are built using guidelines that promote modular-based design, such as <a href="https://www.digitalocean.com/community/conceptual_articles/s-o-l-i-d-the-first-five-principles-of-object-oriented-design">SOLID principles</a> or <a href="https://stackoverflow.com/questions/1527393/what-is-orthogonality">orthogonal design</a>, the application will be able to leverage concurrency to make processes, such as adding another index or a migration (sped up by batch transactions), run quickly and efficiently.</p><h4>Switching Modes for Extremely High-Throughput Cases</h4><p>A little trick picked up when working with a greenfield system that is expected to have high-throughput immediately is to first provision the read/write and then switch to On Demand.</p><p>Although DynamoDB is great when it comes to scalability when in On Demand mode, there is a known delay when the through-put doubles twice within thirty minutes. Knowing this might be an issue when the database went into production initially, the database was provisioned to have read and write capacity for one million and auto scaling was set up. After the initial database went live and the spike in users was less volatile, capacity was switched to On Demand. When a table is updated from provisioned to On Demand mode, read and write throughput expected for the application to perform doesn’t need to be specified. Most databases would be fine with provisioned reads/writes and setting up auto scaling. Since this system was expected to have unpredictable burst speeds, some as high as twenty-five million per minute, it was safer to switch to On Demand for these sudden bursts of high-throughput.</p><h4>Amazon DynamoDB Accelerator Cluster</h4><p>Amazon DynamoDB Accelerator (DAX) is an in-memory cache for DynamoDB that is fully managed by Amazon. Amazon manages the servers, the scaling, and cache invalidation. Having a DAX cluster reduces the latency of a few milliseconds down to microseconds. It’s another nice tool in Amazon’s managed wheelhouse.</p><h3>Conclusion</h3><p>NoSQL is a great choice for a database when performance, high-throughput and velocity are priorities for an application. Although DynamoDB was built as a solution for companies that have out scaled their relational databases, it can have poor performance when not designed and utilized correctly. Designing and coding for DynamoDB databases requires breaking away from the traditional relational database mindset. When the DynamoDB paradigm is adopted, DynamoDB can be one of the best choices for a database solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8067e12ea00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-to-build-a-high-throughput-system-with-dynamodb-d8067e12ea00">How to Build a High-Throughput System With DynamoDB</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using GitHub Actions and CircleCI]]></title>
            <link>https://medium.com/instrument-stories/using-github-actions-and-circleci-b0a30c0a2608?source=rss-f9d76302c088------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0a30c0a2608</guid>
            <category><![CDATA[github]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[circleci]]></category>
            <dc:creator><![CDATA[Kim Gault]]></dc:creator>
            <pubDate>Tue, 14 Apr 2020 22:09:07 GMT</pubDate>
            <atom:updated>2020-11-11T00:07:00.172Z</atom:updated>
            <content:encoded><![CDATA[<h3>GitHub Actions vs CircleCI</h3><p><strong>Comparing GitHub Actions and CircleCI for Continuous Integration and Continuous Delivery</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nesT6v9_bsU-i_Da.png" /></figure><p>As backend engineers at Instrument, one of our duties is to streamline the development process for our teams. Our team needs to deliver websites and software at a high velocity without sacrificing performance or security. One of the ways we accomplish this goal is to setup a continuous integration and continuous delivery pipeline.</p><p>We normally use CircleCI, a named leader in the cloud-native continuous integration. At the end of 2019, GitHub Actions announced that it was releasing its beta to the public. Switching to GitHub Actions and GitHub secrets would give us the luxury of having our repository, pipeline and sensitive information under one platform. This streamlines our standard pipeline commands. We switched to GitHub Actions on our next project and documented the differences between the tools, CircleCI and GitHub Actions.</p><h3>Continuous Integration and Continuous Delivery</h3><p>Continuous integration and continuous delivery are best practices for software development. Continuous integration is the software development practice of merging code changes into the project early and often. Instead of building out features in isolation and then integrating them at the end of a development cycle, code integrates with the shared repository multiple times throughout the day.</p><p>Continuous Integration (CI) grew out of a need to have developers working on features in parallel and on separate branches without fear of conflict. CI pipelines include automatic tools and processes (i.e. unit tests, linting tools) that automatically verify the code changes before merging it into the repository. Developers are less likely to be blocked by another developer’s work. CI results in changes that are easier to test and to revert. New work is introduced quickly in smaller bites. This also creates easier to review pull requests. These practices lead to a reduction in the time it takes to review PRs, find bugs and QA changes, while also maintaining the integrity and stability of the project.</p><p>Continuous Delivery (CD) ensures code changes are tested and ready for production deployment as soon as they are merged into the project. While the code does not have to be deployed to production, it should have been tested on a staging or a production environment. Our CI pipeline also includes automatic tools and processes (i.e. integration tests, end-to-end tests) that automatically test that the code is production-ready before releasing it to the repository.</p><p>When CD and CI are done in tangent, developers can rapidly push quality features and bug fixes. The automation of this requires a robust pipeline.</p><h3>GitHub Actions in Action</h3><p>We decided to use GitHub Actions on our next greenfield project, which included Craft CMS hosted on Heroku and a front-end component system.</p><p>Building a component system with multiple developers working on database schema changes in parallel is an extremely sticky situation, and building a website with multiple developers and writers working on it simultaneously would be even more difficult. This would be a great case study for a CI/CD tool like GitHub Actions.</p><p>We utilized both GitHub branch rules and GitHub Actions to create a pipeline that would fit our workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m43axcW17YIlZPwl.png" /></figure><h3>Unreliability of GitHub Actions in Beta</h3><p>GitHub Actions during its initial beta release was fairly unstable. There were many changes throughout the first few months of the initial release. Thankfully, GitHub switched from <a href="https://github.com/hashicorp/hcl">HCL</a> over to YAML before we began our project, but we still experienced other breaking changes such as updates to the check-out process, pushes and workflow syntaxes. GitHub Actions was also down for frequent half-days in its infancy, although it seems to have stabilized toward the end of our project timeline. The GitHub Actions dashboard, which is crucial to development as it is the only way to get our jobs’ logs or statuses, proved to be unreliable throughout the beginning months.</p><p>As shown in our pipeline diagram above, this is why we took GitHub Actions out of automating anything for production and relied on a manual process that our cloud platform, Heroku, provided instead.</p><h3>GitHub Actions vs CircleCI</h3><p>Github actions did not provide all the niceties that CircleCI had:</p><ul><li>Re-running a GitHub Action (update: you can now re-run a <em>failed</em> GitHub Action, but not a successful one like you can in CircleCI.)</li><li>Manual approval/trigger</li><li>SSH into the GitHub Action environment</li><li>Caching dependencies to speed up workflows (update: now allowed as of 11/04/2019)</li></ul><p>However, GitHub has been steadily releasing features since the beta. To get an idea of their velocity, below are some features that rolled out in the two months between our project launch and this article:</p><ul><li>Proxy support for self-hosted runners</li><li>GitHub CLI beta released</li><li>GitHub Actions API released (runner is now open sourced)</li><li>Caching dependencies to speed up workflows</li><li>Ability to use GitHub Packages with Actions to simplify package management</li></ul><p>GitHub Actions prides itself on being “community-powered”. Anyone can publish an Action and any repository can use a published Action. Many have been using GitHub’s millions of open source libraries or their own repositories to create new Actions. There are now over 1000 community actions available.</p><p>Features like waiting on a manual trigger to run the next job (readily available by a single command in CircleCI) can be implemented by including another user’s published GitHub Action or creating your own using GitHub’s REST API to check for PR approval on every PR review update. It’s not quite as ready out-of-the box as CircleCI, but what it lacks in ease, it makes up in customization and control. There are Actions for popular vendors such as <a href="https://github.com/netlify/actions">Netlify</a> and <a href="https://github.com/actions/heroku">Heroku</a>, and it’s easy to add your own using any client CLI. These actions are continuing to grow as GitHub Actions gains more traction in the market.</p><p>Both CircleCI and GitHub Actions jobs are written in YAML, but the way to approach both are starkly different. CircleCI expects one config file in each repository, while GitHub Actions works best with multiple files. Our CircleCI config files span hundreds of lines, but CircleCI supports anchors and aliases to <a href="https://www.oreilly.com/library/view/97-things-every/9780596809515/ch30.html">DRY</a> up the config file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42ae0288e038a15114249d448408d583/href">https://medium.com/media/42ae0288e038a15114249d448408d583/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HoRZNOFwkcHtrVB7.png" /><figcaption>An example of the short GitHub Actions file in a folder of many GitHub Actions files.</figcaption></figure><p>The dashboard on GitHub Actions is sparse compared to CircleCI. CircleCI provides in-depth project insights and works well with integrations like Sumo Logic to better track analytical data on all of the jobs. GitHub Actions only showed the build for a week before deleting the build log. GitHub Actions is still rolling out new features. Since the project’s launch, it has added the artifacts, annotations and workflow file to the dashboard (not shown in the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IENlDzvMCghsvclD.png" /><figcaption>All of Github Actions Dashboard (image from Github)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Kdbus556ycyd-BDj.png" /><figcaption>CircleCI Dashboard</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*97nYIiPzhVdKEwk_.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qSCYh5McRA55_5DF.png" /><figcaption>A few of the different components of CircleCI Dashboard (images from CircleCI)</figcaption></figure><h3>Conclusion</h3><p>The purpose of a CI/CD pipeline is to have a high development velocity while maintaining the same stability that our clients have come to expect from us. GitHub Actions succeeds in this purpose. It increases our productivity, code quality and delivery. The amount of custom code needed to get a workflow in place for GitHub Actions is surprisingly more than needed for CircleCI, but conversely, adding more scripts for GitHub Actions is more intuitive than adding the extra YAML lines needed for CircleCI.</p><p>Although CircleCI currently has more features built in, provides better analytics and is more reliable than GitHub Actions, GitHub Actions’ ability to house our code repository and pipeline under the same platform and its ability to have multiple pipeline config files are noteworthy conveniences. GitHub Actions is a strong contender when considering a CI/CD tool.</p><p><em>Header image by Lizelle Galaz, UX Strategist at Instrument.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0a30c0a2608" width="1" height="1" alt=""><hr><p><a href="https://medium.com/instrument-stories/using-github-actions-and-circleci-b0a30c0a2608">Using GitHub Actions and CircleCI</a> was originally published in <a href="https://medium.com/instrument-stories">Instrument Stories</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>