<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Festicket Tech - Medium]]></title>
        <description><![CDATA[Building the world’s biggest live events platform. - Medium]]></description>
        <link>https://medium.com/festicket-tech?source=rss----8a8113fea244---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Festicket Tech - Medium</title>
            <link>https://medium.com/festicket-tech?source=rss----8a8113fea244---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 19:54:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/festicket-tech" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deployments at Festicket]]></title>
            <link>https://medium.com/festicket-tech/deployments-at-festicket-1de861acf2f0?source=rss----8a8113fea244---4</link>
            <guid isPermaLink="false">https://medium.com/p/1de861acf2f0</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Bruno Alla]]></dc:creator>
            <pubDate>Wed, 06 May 2020 16:23:41 GMT</pubDate>
            <atom:updated>2020-05-06T16:23:40.944Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_a82XUCMLWb3I-3taCperQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@serjosoza?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">sergio souza</a> on <a href="https://unsplash.com/s/photos/shipping-container?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Deploying code to production for a website can be done in a lot of different ways, it could be as simple as dropping your HTML files on the production server via FTP, running rsync from the developer machine or doing a git push to Heroku. However, such methods are prone to human error and don’t scale well as your team grows. As business revenue depends more and more on software, it’s important to keep the quality high. Best practices like continuous integration (CI), code reviews and QA have gained a lot of momentum over the years, but introducing them can add friction and slow down delivery. In this post, I want to share our deployment process and explain the automation we’ve put in place to reduce process friction while increasing quality and confidence.</p><h3>A bit of context</h3><p>The Festicket website was started around 8 years ago, at the time it was a classic Django monolith, rendering our templates with a sprinkle of javascript on top. Around 3 years ago, we embarked on the journey to move our front-end to a single page web application using React, and we shifted our Django app to be mainly composed of API endpoints. We do a lot of server-side rendering in React for SEO and performance reasons, which means our front-end app is not just serving static files, we need a Node.js server. Along with these main pieces of infrastructure running the application, we have Nginx routing requests to React or Django, several Celery workers handling tasks depending on their priority and few other tools.</p><p>Running all these components predictably in different environments can become challenging. How do you run them locally? How do you provision a website for staging? All our application components are Dockerized, which simplifies setting up local development, and we heavily use IaC (Infrastructure as Code) &amp; automation to build a staging site in no time.</p><h3>Local Setup</h3><p>Our local stack is using Docker Compose to spin up all the components. Each Django &amp; React app is hosted in its own Git repository, and we have a “development” repository that has some base Docker images that the apps extend, the compose files and the 2 main apps repositories as <a href="https://git-scm.com/book/en/v2/Git-Tools-Submodules">git submodules</a>. The services in the Docker Compose stack are:</p><ul><li><a href="https://hub.docker.com/r/fsouza/docker-ssl-proxy/">SSL Proxy</a></li><li>Nginx</li><li>Django backend</li><li>React Node.js server</li><li>Postgres database</li><li>Celery worker</li><li>Redis</li></ul><p>Here is a diagram giving an overview of our configuration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lcXqG6bSXJhEKh7gjmpdaA.png" /><figcaption>Each service will wait for its dependents to have started first (Nginx depends on Django &amp; React).</figcaption></figure><p>We are on macOS, so we also have a Brewfile in the development repo, which enables us to install all the OS dependencies using <a href="https://github.com/Homebrew/homebrew-bundle">Homebrew bundle</a>. In a few steps, a newcomer can spin up their entire development environment with SSL enabled locally.</p><h3>Development workflow</h3><p>We work over sprints of 2 weeks, and use Jira with a Kanban board to get our tasks done. Tasks on the board go through the following states: “to do”, “in progress”, “1st code review”, “2nd code review”, “QA” and finally “releasable”. Working with Jira can be cumbersome, so let’s see how we’ve automated things.</p><h4>Branching</h4><p>We use a simple naming convention for all our git branches: they should start with the Jira card number, for example FES-1234. This enables us to automate most of the workflow using our Slack bot “Doge”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CrQDD-4gerwDAhx-uinL4w.jpeg" /><figcaption>The Doge</figcaption></figure><p>When a branch is pushed to GitHub, our bot tries to extract a card number from the branch name, and tries to find it in Jira. If a card is found, it does the following:</p><ul><li>Open a pull request against master with a link to Jira</li><li>Find the implementer based on the recent commits</li><li>Assign the pull request to the implementer</li><li>Add a link to the pull request to a custom Jira field</li><li>Add implementer to Jira</li><li>Assign the card to the implementer</li><li>Moves the card in progress on the agile board</li></ul><p>Pushing to the branch also triggers a build on our CI, we’ll get back to that in a minute.</p><h4>Code reviews</h4><p>Once the developer is done with their change, they can move to code review. We have 2 rounds of code reviews, and ideally we want to have a staging site ready before moving to code review, so that potential visual changes can be checked if needed.</p><p>It sounds like a lot of things to do before even going to code review, but a lot of things are automated. The developer doesn’t have to interact with Jira, the command line or AWS. It’s all done from a Slack command taking the card number as argument. This instructs the bot to do the following:</p><ul><li>Check that CI is passed</li><li>Spin up a new staging site with all the required components</li><li>Add a QA link to the staging site in Jira</li><li>Move the card to code review</li></ul><p>One of the nice things is that the developer doesn’t need to wait for the build to finish to send this command, the bot may wait for it instead. If the build fails, it notifies the developer. Also, spinning up a new staging site takes only a few minutes.</p><p>If the pull request is approved by someone else in GitHub, the Jira card is moved automatically to the next step and, if changes are requested, the card goes back to the implementer. Either way, they are also notified in Slack. The 2nd code review is very similar, which then brings the card in QA. The bot saves who made the 2 code reviews in custom fields in the Jira card.</p><h4>QA</h4><p>A staging site is closer to the production setup than the local one, the main differences are that PostgreSQL is hosted in <a href="https://aws.amazon.com/rds/">AWS RDS</a> and Redis is hosted in <a href="https://aws.amazon.com/elasticache/">AWS ElastiCache</a> instead of being in Docker. The rest is running in Docker containers on <a href="https://aws.amazon.com/ecs/">AWS ECS</a> (the SSL proxy is not needed). All these components are described in a <a href="https://aws.amazon.com/cloudformation/">Cloudformation</a> template for a site. The deployment process is able to apply any required database migrations automatically.</p><p>The staging machines for the cards in QA are kept up to date. As things get merged into master, the bot keeps the feature branches up to date by merging master and redeploying to staging. In case of merge conflict, the developer is notified by Slack, and the card is assigned to them in Jira.</p><p>Once the QA process is passed, the tester sends a command in Slack which again automates the workflow:</p><ul><li>Tear down the staging machine</li><li>Move the Jira card to the ‘releasable’ status</li><li>Notifies the implementer in Slack</li><li>Updates the approval status in Github</li></ul><p>At this point, the card is ready to be released.</p><h4>Continuous Integration</h4><p>Our CI is responsible for a few things:</p><ul><li>Build the docker images</li><li>Push the images to <a href="https://aws.amazon.com/ecr/">AWS ECR</a></li><li>Run our entire test suite</li></ul><p>We can start the deployment as soon as the Docker images are published in ECR. The bot watches for updates in ECR and makes the appropriate requests to AWS to update the template with the new versions of each app.</p><h4>Production Deployments</h4><p>When all approval &amp; testing is done, it’s time to release, which is -most of the time- just a matter of merging the pull request. Again here, the rest is automated, let’s go over the steps:</p><ul><li>GitHub triggers a build for the master branch on CI</li><li>Doge creates a new git tag derived from the date (e.g. v20200425.3)</li><li>Doge creates a release in Github &amp; Jira</li><li>Doge marks the Jira card as being part of that release</li><li>Doge sends a notification to a public channel about the new release</li></ul><p>Once the build completes, our CI tags the Docker images with the release name and pushes them to ECR on our live AWS account. When the push is completed, the deployment starts, orchestrated by Doge:</p><ul><li>Notifies in a public channel that a deployment is starting</li><li>Runs any database migrations if needed</li><li>Updates the Cloudformation stack for our production site</li><li>Notifies in a public channel when the update is finished</li><li>Closes the Jira card as done</li></ul><p>This whole process between merge and deployment finished takes a bit under one hour, but the deployment itself takes around 30 mins. We usually deploy around 5 to 10 releases a day, but we sometimes go up to ~15 when we have lots to push. Usually there aren’t that many things to release.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXtOkYLu4UfN4cnCsZW81w.png" /><figcaption>Example of a day with 8 releases.</figcaption></figure><p>We do one deployment for each card in Jira, if something goes wrong with a release, we can cancel the Cloudformation update or rollback to the previous version.</p><h3>Wrapping up</h3><p>As you can see, our development workflow &amp; deployment process relies a lot on Slack. The information is all aggregated in Slack, giving an overview across all the tools we use (Jira, GitHub, CI, AWS) in a single place. Our process is structured, yet simple from the developer perspective. We’ve abstracted the complex tasks, grouped them into high level actions, and connected the tools together to remove duplicated manual updates.</p><p>New developers are quickly up to speed and less technical people feel empowered. They just need to learn a couple of Slack commands to provision or tear down a staging site in a couple of minutes.</p><p>This process reduces the room for human error to almost zero, enabling us to ship changes to production fast but with confidence and without friction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1de861acf2f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/festicket-tech/deployments-at-festicket-1de861acf2f0">Deployments at Festicket</a> was originally published in <a href="https://medium.com/festicket-tech">Festicket Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A 15% performance boost by removing unused database indexes]]></title>
            <link>https://medium.com/festicket-tech/a-15-performance-boost-by-removing-unused-database-indexes-ec32e03d5fa9?source=rss----8a8113fea244---4</link>
            <guid isPermaLink="false">https://medium.com/p/ec32e03d5fa9</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Lloyd Wilson]]></dc:creator>
            <pubDate>Wed, 15 Jan 2020 09:24:13 GMT</pubDate>
            <atom:updated>2020-01-15T09:24:13.898Z</atom:updated>
            <content:encoded><![CDATA[<h3>Removing unused database indexes lead to a 15% performance boost</h3><p>A recent analysis of indexes within our production database allowed us to identify and remove those with zero scans, providing a performance boost in write based tasks of up to 15%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5jik_tsbeBvLNz1sEt0ew.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Liverani</a> on <a href="https://unsplash.com/s/photos/data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Database Indexes are typically used to speed up read queries, however using them also comes at a cost;</p><ul><li>Indexes use up space, this is often expensive within a database.</li><li>Indexes slow down data modification. Inserts, Updates and Deletes are all slower when indexes are being used, as the indexes must be updated for the whole table.</li></ul><p>Having unused indexes brings all of the cost with none of the benefit. The scope of our initial work was to improve baseline performance metrics, this formed the basis of the investigation and therefore our priority was to speed up the table. The focus of this article is a simple table that currently has tens of thousands of rows and is frequently updated via a synchronisation job. The gid and url fields both had existing indexes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*gycDADq3-L3seLq795nUag.png" /><figcaption>Our example database table</figcaption></figure><h3>Finding unused indexes</h3><p>The following script was run on the PostgreSQL database to select the index size, tuple read and the number of scans, this is outlined in more details on the <a href="https://wiki.postgresql.org/wiki/Index_Maintenance#Index_size.2Fusage_statistics">PostgreSQL wiki</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65caa12a3bd94f702c2def1894215956/href">https://medium.com/media/65caa12a3bd94f702c2def1894215956/href</a></iframe><h3>Analyzing the results</h3><p>The SQL script brought back results for every index in the database, exporting this into a more user friendly interface (hello Google Sheets) and filtered the columns based on number of scans (looking for values of 0 implying they had never been used), in conjunction with total rows &amp; index size. If an index has a low number of scans but is high in rows or size, it may be worth investigating if it is worth keeping.</p><p>Two indexes on our example table had a high amount of rows and no scans, perfect candidates for removal!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0228416994be833a7e49795fd0b710c/href">https://medium.com/media/c0228416994be833a7e49795fd0b710c/href</a></iframe><h3>Removing the indexes</h3><p>Now that we know which indexes we want to remove, we can go ahead and do it. Making changes to a high traffic database at scale requires us to use a zero downtime approach, ensuring locks don’t interrupt normal operation. Braintree wrote an <a href="https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680">excellent article</a> on this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cbec02c972183fb218d9bcbec2f95625/href">https://medium.com/media/cbec02c972183fb218d9bcbec2f95625/href</a></iframe><p>The SQL script swaps the standard form of DROP INDEX. This operation will aquire and ACCESS EXCLUSIVE lock against the whole table while the index is being removed. In a large table this could take considerable time. Instead we use DROP INDEX CONCURRENTLY, this acquires a SHARE UPDATE EXCLUSIVE lock instead, allowing for read/writes to continue while the operation runs. <a href="https://github.com/tbicr/django-pg-zero-downtime-migrations/">Zero Downtime Migrations </a>is another good resource for further reading on this.</p><h3>Testing the impact</h3><p>To test the impact of removing the indexes, I wrote a simple insert script, to insert 10,000 rows into the table with fake data. This was done both before and after the index removal. Previously the query ran in <strong>0.249</strong> seconds. Without the indexes, it took just<strong> 0.215.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b3ec0bb711962b61c1f6b907c0422496/href">https://medium.com/media/b3ec0bb711962b61c1f6b907c0422496/href</a></iframe><p>This results were around a 15% boost in time taken to write the data. In line with our performance target and time based budgets, this was a great gain that cost nothing in read performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec32e03d5fa9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/festicket-tech/a-15-performance-boost-by-removing-unused-database-indexes-ec32e03d5fa9">A 15% performance boost by removing unused database indexes</a> was originally published in <a href="https://medium.com/festicket-tech">Festicket Tech</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>