<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[SquadStack Engineering - Medium]]></title>
        <description><![CDATA[Stories from the people who build SquadStack - Medium]]></description>
        <link>https://medium.com/squad-engineering?source=rss----256a175005d---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>SquadStack Engineering - Medium</title>
            <link>https://medium.com/squad-engineering?source=rss----256a175005d---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:45:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/squad-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Synergy Part-1: Planning and Evaluation]]></title>
            <link>https://medium.com/squad-engineering/synergy-part-1-planning-and-evaluation-5c9874e159ab?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/5c9874e159ab</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airbyte]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[clickhouse]]></category>
            <dc:creator><![CDATA[Ujjwal Gupta]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 10:26:27 GMT</pubDate>
            <atom:updated>2023-09-12T10:26:27.552Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FlnhZ6YtaKtZ0P5YrynhRw.jpeg" /></figure><p>Welcome back to our blog series on our in-house data warehouse solution! In the previous post, we gave <a href="https://medium.com/squad-engineering/synergy-part-0-introduction-to-our-in-house-data-warehouse-solution-a31e4c2ab39">an introduction to our in-house data warehouse solution</a>. This blog post will delve into the planning and evaluation phase of setting up our in-house pipeline. Let’s get started!</p><h3>Planning the Overall Process</h3><p>To set up an in-house data warehousing solution, a well-thought-out planning process is essential. Here is a simplified overview of the steps we took in our planning phase:</p><ol><li><strong>Identifying Requirements</strong>: We determined the specific data needs and goals of our organization, considering the types of data to collect, data update frequency, desired performance and scalability, and budget constraints.</li><li><strong>Defining Data Sources</strong>: We carefully documented the various data sources within our organization, including databases, applications, files, and external APIs that would provide the data for processing and storage in our data warehouse.</li><li><strong>Choosing the Right Data Warehouse Solution</strong>: After identifying the data sources, we evaluated and selected the appropriate database for storing our analytical data. This database would serve as the foundation for efficiently storing and organizing our data.</li><li><strong>Planning Data Extraction and Transformation</strong>: We meticulously planned the process of extracting data from the sources, transforming it into a suitable format, and loading it into the data warehouse. This involved selecting the appropriate ETL/ELT processes and techniques to ensure smooth and effective data integration.</li></ol><h3>Data Warehouse Evaluation</h3><p>A data warehouse serves as a centralized and integrated repository for structured, historical, and voluminous data collected from diverse sources within an organization. During our evaluation of data warehouse solutions, we carefully assessed them based on the following criteria:</p><ol><li><strong>High Performance: </strong>The data warehouse should demonstrate high performance, enabling quick loading and retrieval of analytical data. It should efficiently handle large volumes of data, ensuring fast query execution and response times.</li><li><strong>Cost Effectiveness: </strong>The data warehouse solution should be cost-effective, and capable of efficiently managing and storing the organization’s data without incurring excessive expenses. It should offer scalability options that align with the organization’s data growth and workload requirements.</li><li><strong>Data Security and Privacy:</strong> Data security and privacy are critical considerations. The data warehouse should adhere to data compliance regulations and industry standards. It should provide robust security measures such as role-based access control (RBAC), encryption of data at rest and in transit, and PII masking and scrubbing features.</li><li><strong>SQL Compatibility: </strong>Enables easy and efficient data analysis for data analysts and business users, leveraging their existing SQL skills and compatibility with SQL-based tools and applications.</li><li><strong>Managed Service:</strong> The chosen data warehouse solution should provide a highly available and robust managed service. It should offer features such as automated backups, disaster recovery options, and proactive monitoring to ensure uninterrupted availability and reliability.</li></ol><figure><img alt="Data Warehouse Decision Matrix" src="https://cdn-images-1.medium.com/max/1024/1*YR6y74EXbhNVRVehZX8Wpg.png" /><figcaption>Data Warehouse Decision Matrix</figcaption></figure><blockquote>After a thorough evaluation, our final choice for a data warehouse solution was <strong>Snowflake</strong>. Snowflake stood out due to its elasticity, separation of storage and compute, robust data security measures, cost efficiency, and SQL compatibility, making it the ideal fit for our organization’s needs.</blockquote><h3><strong>ETL/ELT Evaluation</strong></h3><p>ELT (Extract, Load, Transform) and ETL (Extract, Transform, Load) are data integration processes commonly used in data warehousing and analytics. The key difference lies in the sequence of the transformation step.</p><p>During the evaluation, we specifically considered the data sources we needed to extract information. The primary data source in our case was <strong>Postgres</strong>, which served as the foundational data repository for our organization. In addition to Postgres, we identified other important data sources such as Trello, HubSpot, and Google Sheets.</p><p>Here are the evaluation factors we considered for assessing the ETL/ELT tools:</p><ol><li><strong>Low Replication Lag</strong>: A low replication lag ensures that the data warehouse&#39;s data is up-to-date and reflects the most recent changes.</li><li><strong>Data Security and Privacy</strong>: We looked for an ETL tool that offers robust security measures to protect sensitive information during data extraction, transformation, and loading. This includes encryption mechanisms, access controls, and compliance with data protection regulations.</li><li><strong>Multi-Source Connector Support</strong>: Given that we had multiple data sources such as Postgres, Trello, HubSpot, and Google Sheets, we needed an ETL tool that could seamlessly connect and extract data from these diverse sources.</li><li><strong>Cost Effectiveness</strong>: Cost-effectiveness was a key factor in selecting an ETL tool. We considered the licensing costs, maintenance expenses, and scalability options to ensure that the chosen tool aligns with our budget constraints while providing the necessary functionalities.</li><li><strong>Support for Snowflake</strong>: Since we opted for Snowflake as our data warehouse solution, it was essential to choose an ETL tool that could seamlessly integrate with Snowflake.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LxpR4RfseT7OMegH_jZ0rA.png" /><figcaption>Data Pipeline Decision Matrix</figcaption></figure><blockquote>After evaluating numerous tools, we discovered that none of them met our specific requirements. Some exceeded our budget, others didn’t meet our SLA standards, and some lacked essential security features. This left us in a dilemma, unsure of how to proceed. <strong>After extensive discussion, we concluded that the best course of action was to build our own cost-effective and SLA-compliant in-house data pipeline.</strong></blockquote><p>In addition, we came across <a href="https://airbyte.com/"><strong><em>Airbyte</em></strong></a>, a highly effective open-source tool designed to handle data loading from secondary data sources such as HubSpot, Trello, and more. Recognizing its value, we made the decision to self-host Airbyte, allowing us to have greater control and flexibility in managing our data integration processes.</p><h3>Summary</h3><ol><li>The blog post discusses the planning and evaluation phase of setting up an in-house data warehouse solution.</li><li>Key steps in the planning process include identifying requirements, defining data sources, and choosing the appropriate data warehouse solution.</li><li>Evaluation criteria for data warehouse solutions include high performance, cost-effectiveness, data security and privacy, SQL compatibility, and managed service.</li><li>Snowflake is chosen as the data warehouse solution due to its elasticity, separation of storage and compute, robust security measures, cost efficiency, and SQL compatibility.</li><li>The evaluation criteria for ETL/ELT tools include low replication lag, data security and privacy, multi-source connector support, cost-effectiveness, and support for Snowflake as a data warehouse destination.</li><li>Despite evaluating various tools, none met the organization’s specific requirements, leading to the decision to build an in-house data pipeline. Airbyte, an open-source tool, is selected for data loading from secondary sources like HubSpot and Trello.</li></ol><p>In our next blog post, we will provide a detailed technical discussion on the development of our in-house data pipeline. Stay tuned for more and show your support by giving us a few claps if you enjoyed this post. 🤗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c9874e159ab" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/synergy-part-1-planning-and-evaluation-5c9874e159ab">Synergy Part-1: Planning and Evaluation</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Synergy Part-0: Introduction to our in-house data warehouse solution]]></title>
            <link>https://medium.com/squad-engineering/synergy-part-0-introduction-to-our-in-house-data-warehouse-solution-a31e4c2ab39?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/a31e4c2ab39</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <dc:creator><![CDATA[Ujjwal Gupta]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 12:00:09 GMT</pubDate>
            <atom:updated>2023-08-11T12:00:09.698Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*cVBJO7SAxKxD4TSv.png" /></figure><p>This blog post series highlights how we built our in-house data warehousing solution. We’ll also share our challenges while looking for the optimal solution.</p><h4>Why Synergy?</h4><p>“Synergy” refers to the combined effect or interaction of multiple layers, components, systems, or technologies that produce a result greater than the sum of their individual effects. So, we decided to call it “Synergy” because it perfectly captures the strong collaborative spirit of our team.</p><p>Here are some topics that we’ll talk about in this blog post:</p><ol><li>About Data Warehouse Solution</li><li>An Overview of Synergy</li><li>What problems we were facing before Synergy?</li><li>Why do we choose to build a data warehouse solution?</li><li>SLAs that Synergy will help us achieve</li></ol><h3>About Data Warehousing Solution</h3><p>A data warehouse solution is a centralized system that stores and integrates large volumes of data from various sources. It consists of data sources, processes for extracting, transforming, and loading data, the data warehouse, data modeling, business intelligence tools for analysis and reporting, metadata management, and security measures.</p><p>This solution enables organizations to efficiently store, analyze, and gain insights from data to support decision-making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N2xJFcv3AHgX9UFy5TEYOw.png" /><figcaption>Synergy: High-Level Diagram</figcaption></figure><h3>An Overview of Synergy</h3><ul><li>Synergy is a data warehousing solution that allows us to draw analytics to gain quicker real-time insights.</li><li>Business users get instant access to crucial data from various sources on a single platform, they can rapidly make informed decisions on key initiatives. They won’t waste precious time retrieving data from multiple sources.</li><li>The outcome we want to draw from synergy is to boost communication and speed up decision-making.</li><li>From Synergy, large amounts of historical data are kept in a data warehouse so you can study various periods and trends to foretell the future.</li></ul><h3>What problems were we facing before Synergy?</h3><p>As we are a fast-growing startup so the biggest challenge was to handle fast generative data. We listed some of the problems below:</p><h4>1. Slow Insights</h4><p>Slow insights lead to unhurried decision-making, so we must move to faster insights to improve decision-making. It was not possible to gain insights from a massive amount of data using a transactional database.</p><h4>2. Handling PII and Data Security</h4><p>Handling PII is one of the most challenging problems we were trying to solve recently. Before Synergy, it was difficult to restrict internal users to access PII.</p><h4>3. Customer-Facing Analytics</h4><p>Customer-facing analytics was one of the major issues that we faced recently, so we must address this issue to improve the customer experience.</p><h3>Why do we choose to build an in-house data warehouse solution?</h3><p>Building an in-house solution was the best option for our use case due to the following reasons:</p><ol><li><strong>Cost Effectiveness </strong>— Most of the tools in the market were too expensive and didn’t align with our budget.</li><li><strong>Immature Tools</strong> — Some tools didn’t deliver the desired output and had inadequate support.</li><li><strong>PII Handling — </strong>This could be due to a variety of reasons such as a lack of customization options, inadequate support, or limitations in the tools’ functionalities. This is why we decided to build an in-house solution to address the issue of PII Data Scrubbing and ensure that our company’s data privacy policies are met.</li></ol><h3>SLAs that we planned to achieve from Synergy</h3><p>To overcome the challenges we faced while evaluating tools, we decided to build our data warehousing solution. To ensure the success of the project, we defined the following Service Level Agreements (SLAs):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JkoHklRJfOzASH5MDwqYg.png" /></figure><ol><li><strong>Service Availability:</strong> Ensuring that the service is available and functioning properly during the designated working hours. This is important to ensure that the users can access the data and perform their required operations without any disruptions. We started with an availability SLA of 97.5%. We now have &gt; 99.5% availability, and we plan to hit <strong>&gt; 99.8%</strong> in the future.</li><li><strong>Replication Lag:</strong> This SLA defines the acceptable time delay between the source and destination systems. A replication lag of no more than 90 minutes is expected, meaning that the data should be updated and available within 90 minutes of the original change for end users.</li><li><strong>Cost Effective:</strong> The solution should be cost-effective and not exceed the budget of ~$2500/month.</li><li><strong>Hold Historical Data:</strong> The solution must be capable of storing 2 years of raw data and 1 year of transformed data. This is important for keeping a record of historical data for analysis and reporting purposes.</li><li><strong>Fast Analytics:</strong> The solution must be able to execute queries on data older than 6 months within 5 seconds. This is critical for ensuring that the users can quickly analyze the data and make informed decisions.</li></ol><p>Stay tuned for our next blog post, where we’ll reveal the planning and evaluation process behind building Synergy. If you enjoyed this post, don’t forget to show some support by giving it a few claps. 🤗</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a31e4c2ab39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/synergy-part-0-introduction-to-our-in-house-data-warehouse-solution-a31e4c2ab39">Synergy Part-0: Introduction to our in-house data warehouse solution</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monorepo for React Native: When to Use, When Not to Use]]></title>
            <link>https://medium.com/squad-engineering/monorepo-for-react-native-when-to-use-when-not-to-use-32dfd6dea635?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/32dfd6dea635</guid>
            <category><![CDATA[react-native]]></category>
            <category><![CDATA[monorepo]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[cross-platform-mobile-app]]></category>
            <dc:creator><![CDATA[Ramanpreet Singh]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 09:33:28 GMT</pubDate>
            <atom:updated>2022-11-09T08:02:17.088Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>By <a href="https://www.linkedin.com/in/ramanpreet-singh-a47005145/">Ramanpreet Singh</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B550hm26kUF8tZWAmTevgA.jpeg" /></figure><h3><strong>Introduction</strong></h3><p>It’s been a year since we started to rewrite our Android app in React Native, and during this period, we made many choices and decisions with respect to development, and choosing monorepo architecture was one of them. In this article, we will take a closer look at when we should use a monorepo and when not for React Native and share a glimpse of our experience.</p><h3><strong>What is a monorepo</strong></h3><p>A monorepo (aka monolithic repository) is an architectural concept in which we use one repository for storing multiple projects instead of managing multiple repositories. While these projects may be related, they are often logically independent. For example, a single repository for web and mobile apps.</p><h3><strong>When to use monorepo</strong></h3><ul><li><strong>Code sharing &amp; More collaboration: </strong>When you have a use case in which code sharing between different projects<strong> </strong>is required, monorepo can be helpful. As all code exists in a single repository, there is an increased likelihood of code reuse, less code duplication, and more teams working on shared infrastructure.</li><li><strong>Ease of abstractions: </strong>Monorepos simplifies code abstraction. We can easily build layers of abstractions on top of existing code and third-party libraries and keep them in separate modules for usage across all the projects by simply importing them wherever needed and using API exposed from them. In this way, it also makes the code isolated from business logic and provides separation of concerns.</li><li><strong>Easier to publish packages for independent code:</strong> If you want to publish a part of your project, you can do so with monorepo as your code can live in isolation, decoupled from other logic as an independent project when compared to the case in which you have this code in a single repository. In this case, pulling out pieces that are supposed to be published from a single code repository would be more complicated.</li><li><strong>Easier dependency management: </strong>Monorepo helps prevent dependency conflicts, as all modules are hoisted and used from a single place. This also prevents the need to re-install dependencies in each project whenever you want to update them. This can also help with packages from node_modules being bundled twice in case tree-shaking is not configured correctly.</li><li><strong>Atomic changes and easier refactors:</strong> Large-scale refactoring becomes easier with monorepo as a developer can update several packages or projects in a single commit as compared to when working with multiple repositories.</li></ul><h3><strong>When not to use monorepo</strong></h3><ul><li>When you have a simple use case for the app without any need to share code between multiple projects. It might also be overkill if you feel the logical separation of modules can be achieved with simple folder structuring instead of maintaining separate projects.</li><li>While creating POCs, MVPs, or small demo apps where the focus is on speed. Building with a monorepo approach requires maintenance and setup efforts from time to time, which can take time away from working on product features.</li><li>When different teams are working on different independent projects, it is better to have separate repositories than monorepo. Because with monorepo architecture, the size of the codebase will increase significantly making it difficult for developers to search through or navigate within the entire codebase once the project scales to a reasonable level.</li><li>Working with monorepo can involve a significant learning curve and can involve diving into your dependencies from time to time to make them work with your project. This bit is especially true for react native, as many native dependencies include hard-coded paths inside their Gradle build files. This causes issues with auto-linking and metro not being able to find certain modules while building.</li></ul><h3><strong>Why we chose monorepo</strong></h3><ul><li>While starting with the project, we had a clear goal — To share as much code as possible between the web and the mobile platform. And monorepo architecture helped us support segregating the codebase in a way, where we could reuse pieces apart from the UI layer between these two platforms.</li><li>Monorepo helped us manage dependencies better by hoisting them to the root level and sharing them across various packages.</li><li>Monorepo allowed us to write code without coupling business logic and services with our UI layer.</li><li>Structuring business logic into separate projects can also make orchestrating platform-specific features easier instead of adding Platform.OS checks everywhere.</li></ul><h3><strong>How we use monorepo</strong></h3><p>While working with a monorepo architecture, you need a certain set of tools to manage how your packages interact with each other. We use <a href="https://classic.yarnpkg.com/blog/2017/08/02/introducing-workspaces/">Yarn Workspaces</a> for this, as it works well with React Native. Other examples include <a href="https://turbo.build/">turborepo</a>, <a href="https://nx.dev/">Nx</a>, <a href="https://lerna.js.org/">Lerna</a>, <a href="https://pnpm.js.org/en/">pnpm</a> which even go beyond repository management with features like build caching, changelog generation, better build performance, etc.</p><pre>&lt;project_root&gt;<br>├── node_modules<br>├── package.json<br>└── packages<br>    ├── app<br>    ├── components<br>    ├── core    <br>    ├── mobile<br>    └── styling</pre><p>The image above showcases what a monorepo project could look like, where each package depicts a separate project with its own package.json and other configuration files. We have <strong>app</strong>, <strong>components,</strong> <strong>core,</strong> <strong>mobile, </strong>and <strong>styling</strong>.</p><p><strong>Core Package</strong></p><p>This package contains the core services of our app, like classes and wrappers written for third-party libraries, for example, the network layer responsible for API calls, analytics, RUM services, logging, etc. No business logic is present in this package to make services reusable and easily pluggable into any apps we add into the monorepo.</p><p><strong>App Package</strong></p><p>This is where all of our app UI (screens, navigators, context, app hooks) and business logic live. This is kept separate from the react native project with the intention to re-use it in the web app.</p><p><strong>Mobile Package</strong></p><p>This is the package containing the react native project <em>(along with native android and ios files). </em>It has configuration files for the metro bundler, environment files (<em>env), </em>build scripts, and project initialization files <em>(</em><em>index.js and </em><em>App.js). </em>Its package.json file contains scripts for running the project along with all the dependencies that have native files. No business logic is written here, and it imports all of it from the app package declared above.</p><p><strong>Components Package</strong></p><p>All reusable UI components reside here. Separating out UI components this way also makes it possible to develop them in isolation from the actual screens enforcing reusability. Examples of components that reside here are — Card, Bottom Sheet, Carousel, Input, Button, List, Modal, Tooltip, etc.</p><p><strong>Styling Package</strong></p><p>All styling and design system-related configurations are stored in this package and used in the entire project from here only. It includes different colors, font families, text and custom variants, spacing values, and typography values as per our design system. It also contains a wrapper for <a href="https://github.com/Shopify/restyle">shopify’s restyle</a>, which we use as our third-party styling library.</p><h3>Common problems we faced</h3><ul><li>While building the app, some dependencies had hard-coded paths to node_modules, assuming the structure was created by a react native CLI project. Fixing these issues required creating patches using something like <a href="https://www.npmjs.com/package/patch-package">patch-package</a></li><li>Some configuration is required to make the metro bundler compatible with Yarn workspaces. And why that? Because of metro bundler’s limitation to follow symlinks (<a href="https://github.com/facebook/metro/issues/1">Github issue</a>) due to which metro is not able to detect dependencies that are hoisted since they are installed in mobile/node_modules as symlinks from &lt;root&gt;/node_modules. Also, metro won’t even be able to resolve other workspaces like app, components, etc., since they’re outside of the mobile directory.<br>So to fix this problem, we have many <a href="https://facebook.github.io/metro/docs/configuration/">metro configuration options</a> available. And we can update the metro configuration file with the help of <a href="https://github.com/mmazzarolo/react-native-monorepo-tools">react-native-monorepo-tools</a> and make metro compatible with Yarn workspaces.<br>You can refer to <a href="https://engineering.brigad.co/react-native-monorepos-code-sharing-f6c08172b417">this blog</a> to help set it up.</li></ul><h3>Conclusion</h3><p>So, as we have described monorepo has many advantages, but at the same time, they are not the right fit for every team. Understand your team’s strengths &amp; weaknesses and your project’s use case to determine if a monorepo is the right choice.</p><p>Thank you for reading this article. If you enjoyed this article, please give a few claps so that it reaches more people who could benefit from it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32dfd6dea635" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/monorepo-for-react-native-when-to-use-when-not-to-use-32dfd6dea635">Monorepo for React Native: When to Use, When Not to Use</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migration to Python3: The SquadStack Experience]]></title>
            <link>https://medium.com/squad-engineering/migration-to-python3-the-squadstack-experience-4af4ed7715e7?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/4af4ed7715e7</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Yoshita Bajaj]]></dc:creator>
            <pubDate>Fri, 30 Sep 2022 08:19:22 GMT</pubDate>
            <atom:updated>2022-09-30T11:43:00.600Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Written by <a href="https://www.linkedin.com/in/kunal-mahajan-b10437121/">Kunal Mahajan</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vJBHoALwcwt_UJ6Rrhq6fw.jpeg" /></figure><h3>Introduction</h3><p>Python is the primary coding language at SquadStack. Our monolith project is 8 years old with more than 500k lines of python code. A few months back we migrated our codebase from Python2.7 to Python3<em>. </em>In this article, we’ll go through various steps taken and the challenges faced in migrating the codebase from Python 2 to Python 3.</p><h3>Why do you need to migrate your codebase from Python2.7 to Python 3?</h3><p>This will be the first thing that will definitely come to your mind when you plan to work on this project. Here are some of the reasons:-</p><ol><li>Python2.7 is no longer supported by the python contributors. This means if you face any issue w.r.t to Python2.7 no fixes would be done. No security patches would be applied.</li><li>New libraries, tools, modules, and frameworks are getting written in Python 3 only.</li><li>An outdated Python version can show up as a red flag during a security audit.</li><li>Improved threading and multiprocessing library — including shared data between processes in Python 3. (Improvement in GIL).</li><li>Python 3 is usually faster than Python 2.7.</li></ol><p>There are many more differences between Python3 and Python2.7. If you want to refer to the whole list, refer to the official docs <a href="https://docs.python.org/3.10/whatsnew/3.0.html">here</a>.</p><h3>How did we get started with the migration process at SquadStack?</h3><ul><li>We <strong>removed all the extra/redundant code from our codebase</strong>. We did not waste time upgrading that part of the codebase which was never going to get executed. <em>We learned from this that we could have been more regular with such clean-ups</em>.😅</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/0*yXvYIbDLZox9HoKE" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yc3oASDXDEENzq5t" /></figure><p>Some cleanup screenshots</p><ul><li>After the above step, we made sure that we had a <strong>good test coverage of our codebase</strong>. Without good test coverage, it would be a humongous task to upgrade our codebase to Python3 in a bug-free way.</li><li>We started running two CI pipelines for both Python 2.7 and Python 3. By running two CI pipelines we ensured that the code is working in both environments.</li><li>After the above steps, the real migration started. Before migrating to Python3 we wanted to make our code compatible with both Python2.7 and 3. <a href="https://python-future.org/compatible_idioms.html"><em>Here</em></a><em> is a </em><strong><em>cheat sheet</em></strong><em> that you can refer to make your code Python2/3 compatible. </em>We used some tools to ensure that our compatibility is not breaking like:<br>- <a href="https://docs.python.org/3.8/library/2to3.html"><strong><em>2to3</em></strong></a><strong>:</strong><em> Automated Python 2 to 3 code translation.<br>- </em><a href="https://python-future.org/futurize.html"><strong><em>futurize</em></strong></a><strong><em>:</em></strong><em> It allows you to use a single, clean Python 3 -compatible codebase to support both Python 2 and Python 3 with minimal overhead.<br>- </em><a href="https://six.readthedocs.io/"><strong><em>six</em></strong></a><strong><em>:</em></strong><em> It allows you to support codebases that work on both Python 2 and 3 without modifications.</em></li><li>To ensure that no one from the team is pushing incompatible code we added some custom <em>Github actions</em>. These actions ran on every commit. For example, if any <em>__future__</em> library got missing at the start of the file then the GitHub action would throw an error.<br>We also integrated <a href="https://python-future.org/futurize.html"><em>futurize</em></a> in the Github actions. This helped us in automating Python2/3 compatibility with every new commit.</li><li>Once we made our code Python2/3 compatible we finally started the last step of migration which was to migrate completely to Python3.</li></ul><p>All the steps were straightforward except the last step. Let’s discuss some of those challenges in the next section.</p><h3>What challenges did we face in the migration process?</h3><p>One of the biggest issues we faced was <strong><em>pickling and un-pickling objects in python</em></strong>. Let’s discuss this issue in detail.</p><p>In SquadStack, we follow the practice of<a href="https://django-cacheback.readthedocs.io/en/latest/api.html#queryset-jobs"> caching querysets</a>. So, before caching the queryset we pickle the queryset first and then store it in Redis.<br><strong>What problems occurred because of pickling the querysets?<br></strong>Unpickling a queryset in Python 3 which got pickled in Python2.7 resulted in a lot of <em>UnicodeDecodeError</em>. This happened because Python 3 uses a different protocol version to pickle objects as compared to Python 2.7.<br><strong>How did we fix this?<br></strong>We took the following approach:-<br>- Invalidated cache in case UnicodeDecodeError occurs.<br>- Pickled the queryset using a protocol version that is compatible with Python2.7 and Python3. Doing this definitely meant that the <strong>load on our database would increase</strong>. So, to ensure that this does not hamper any of our applications we <strong>increased</strong> the size of our database.<br>- Now, once the queryset got pickled using a compatible protocol version(the <em>2.0 version is compatible with both Python 2.7 and 3</em>) this problem was mitigated. :)</p><p>I know increasing the load on your database is not something with which you can go with every time. <em>Please suggest alternate approaches in the comments below. :)</em></p><pre>CACHES = {<br>    “default”: {<br>        “BACKEND”: “redis_cache.RedisCache”,<br>        “LOCATION”: “dummy”,<br>        “TIMEOUT”: 500,<br>        “BINARY”: True,<br>        “OPTIONS”: {<br>            “PICKLE_VERSION”: 2, <strong># pickle version set to 2 to make</strong><br>                                 <strong># it compatible with Python 2 and 3</strong></pre><pre>         },</pre><pre>    } </pre><pre>}</pre><p><strong>Text vs binary issues</strong>. In Python2.7 there was no distinction between text and binary strings. But now in Python3, there is a <strong>clear difference</strong> between text and binary. This was the most common issue we faced. This is easy to fix but most time-consuming. You can look into the cheat sheet attached above.</p><p>There were some issues <strong>related to Django</strong> like:-</p><ol><li><strong>Creation of new migration files</strong>: On running makemigration on Python3 we saw the creation of some new migration files.</li></ol><p><em>1. Running </em><em>makemigration on python3 resulted in a file without any </em><strong><em>b prefix </em></strong><em>for string values. String values can be anything like </em><em>verbose_name, </em><em>help_text, </em><em>choices in any choice field. This happens because there is a clear difference between text(Unicode) and bytes in Python 3. Whereas in Python 2.7 there is no clear difference between them. This resulted in a file with a </em><strong><em>b prefix</em></strong><em> for non-unicode values in Python 2.</em></p><p><strong><em>Compatible Solution:</em></strong><em> The easiest way to get the same migration file in both python 2 and python 3 environments is by adding from</em><em> __future__ import unicode_literal to all the models file. For existing migration files either we run makemigration and that should only happen once, or we can remove b prefix from existing migration files.</em></p><p>2. <strong>Object representation on Admin/Shell: </strong>__unicode__()method is not available in Python3. But we were using this in a lot of places to show a Unicode representation of an object.</p><p><strong><em>Compatible Solution:</em></strong><em> To support Python 2.7 and Python 3, we added </em><em>__str__() method and made sure it returns unicode under Python 2.7 by using </em><a href="https://consideratecode.com/2017/08/25/what-does-python_2_unicode_compatible-do/"><em>@python_2_unicode_compatible</em></a><em> decorator.</em></p><p>Once we migrated the whole codebase to Python3, we now had to roll out the changes to production. Let’s deep dive into that as well.</p><h3>How did we deploy this migration to production?</h3><p>Since we have a monolith architecture, <strong>deployment</strong> became a big challenge for us. We have a huge codebase. A single issue could have affected all the verticals be it our supply team/demand team/internal team. So we did not do a 100% rollout at once. Instead, we did roll out in phases. Currently, our system has 4 main components:-</p><ol><li>app — This component handles the android application and the customer dashboard.</li><li>api — This component handles all the API requests.</li><li>celery — This component handles all the asynchronous processes.</li><li>admin — This component handles all internal workflows/operations.</li></ol><p>We have dedicated servers for these components. So we rolled it out in the following manner:-</p><ol><li>We created<strong><em> new Python3 instances</em></strong> for every component.</li><li><strong><em>Rolled out less traffic for a single component</em></strong> so that our business doesn’t get affected . We did a weighted routing with only 5–10% traffic on the Python3 servers.</li><li><strong><em>Rolled out full traffic for a short period for a single component </em></strong>and fixed the bugs. Bugs were related to string formatting, comparison related, text vs binary, etc.</li><li><strong><em>Rolled out traffic for other components.</em></strong> (We started with sending traffic to the less busy servers first).</li><li>Once the confidence got built <strong><em>we rolled out 100% traffic</em></strong> for all our Python3 instances.</li><li>Once everything got stable in the Python 3 environment, we <strong><em>stopped all the Python2.7 instances</em></strong>.</li></ol><h3>Learnings made from this whole migration exercise</h3><ol><li><strong><em>Planning is the most important thing</em></strong>. If you plan well, you’ll always end up finishing any project/exercise/task efficiently.</li><li>Writing good and efficient test cases is a <strong><em>big-big must </em></strong>if you want to deploy your code with minimal errors.</li><li>Take out some bandwidth for tech debts. Stretching some tech debts for a very long time can lead to nightmares.</li></ol><h3>Conclusion</h3><p>Finally, we transitioned to<strong><em> Python3</em></strong> smoothly without having any downtime along with continuous development. Next up, we have started our Django migration process. <em>So, stay tuned for our next migration blog on Django </em>😉</p><h3>References</h3><ol><li><a href="https://python-future.org/compatible_idioms.html#">https://python-future.org/compatible_idioms.html#</a></li><li><a href="https://www.stxnext.com/blog/why-migrate-from-python-2-to-python-3/">https://www.stxnext.com/blog/why-migrate-from-python-2-to-python-3/</a></li><li><a href="https://medium.datadriveninvestor.com/why-companies-are-moving-from-python-2-to-python-3-86d948e529c0">https://medium.datadriveninvestor.com/why-companies-are-moving-from-python-2-to-python-3-86d948e529c0</a></li><li>How the other companies migrated like <a href="https://thenewstack.io/instagram-makes-smooth-move-python-3/">Instagram</a>, <a href="https://lwn.net/Articles/758159/">Facebook</a>, etc.</li></ol><p><strong>Special Notes</strong></p><blockquote>We are Hiring! 😀It’s an exciting time to join the SquadStack engineering team. Please have a look at the SquadStack careers page. We have a lot of exciting problems to solve in many domains like Scalability, DevOps, Backend, System design, etc.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af4ed7715e7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/migration-to-python3-the-squadstack-experience-4af4ed7715e7">Migration to Python3: The SquadStack Experience</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cross-region RMQ message migration using Shovel Plugin]]></title>
            <link>https://medium.com/squad-engineering/cross-region-rmq-message-migration-using-shovel-plugin-371b784d22d3?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/371b784d22d3</guid>
            <category><![CDATA[amazon-mq]]></category>
            <category><![CDATA[cross-region]]></category>
            <category><![CDATA[rabbitmq]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[shovel]]></category>
            <dc:creator><![CDATA[Ujjwal Gupta]]></dc:creator>
            <pubDate>Tue, 24 May 2022 06:04:15 GMT</pubDate>
            <atom:updated>2022-12-30T18:20:42.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Cross-region RMQ message migration using Shovel Plugin" src="https://cdn-images-1.medium.com/max/1024/0*o0_AKzqOwI_n0D5o" /></figure><blockquote>This blog details Squadstack’s experience with moving RMQ messages from one AWS region to another.</blockquote><h3>Brief</h3><p>Due to data compliance reasons, we at <a href="https://www.squadstack.com/">Squadstack</a> migrated our existing infrastructure from one AWS region to another. RMQ migration was one of the major aspects of the infrastructure move that we had to deal with.</p><p><strong>The approach we followed was a traditional one:</strong></p><ol><li>Created a new private RMQ cluster in a new region.</li><li>Point all-new message requests to the new RMQ cluster.</li><li>Stop request ingestion to the old RMQ cluster.</li><li>Old RMQ was only responsible to deliver the old messages (Unacked + Ready).</li><li>Stop the old RMQ cluster once the remaining message count becomes <strong>0</strong>.</li></ol><blockquote><em>Looks Simple, Right? But it’s not simple as it seems.</em></blockquote><h4><strong>Problems that we faced by using the above traditional approach</strong></h4><ol><li><strong>Infinite Queued tasks</strong> — Some tasks were re-queuing to the same queue again &amp; again. This becomes a deadlock for us.</li><li><strong>High Latency </strong>— As part of infra migration we already migrated our RDS to the Mumbai region. So, while executing tasks from Singapore’s RMQ cluster the query is getting a timeout.</li><li><strong>Extra Cost Overhead</strong> — Running multiple RMQ clusters one in the Mumbai region (for primary) and another in the Singapore region. We also placed dedicated celery servers for both of the clusters. This becomes overhead to our Infra budget.</li></ol><p><em>So, How do we overcome these problems 🤔? And the answer is…</em></p><h3>Shovel Plugin as a Message Transferer</h3><p>Shovel plugin is a core RMQ plugin that <strong>unidirectionally moves messages from a source to a destination</strong>. It is necessary to reliably and continually move messages from a source (typically a queue) in one cluster to a destination (an exchange, topic, etc) in another cluster.</p><p>The source and destination can be in the same cluster (typically in different vhosts) or distinct ones. Shovels support AMQP 0.9.1 and AMQP 1.0 sources and destinations. The source and destination do not have to use the same protocol, so it is possible to move messages from an AMQP 1.0 broker to RMQ or vice versa.</p><p>You can read more about the shovel plugin and its benefits <a href="https://www.rabbitmq.com/shovel.html#benefits"><strong>here</strong></a><strong>.</strong></p><h3>Steps to enable shovel plugin for RMQ cluster</h3><p>To enable the shovel plugin, we need to follow certain steps which as mentioned below:</p><h4>Make your destination RMQ cluster publicly accessible.</h4><ol><li>Create a server in a public subnet.</li><li>Install Nginx using sudo apt-get install nginx</li><li>Check whether <strong>nginx</strong> is installed successfullynginx -v</li><li>Check for brokers’ private IPs.</li><li>Configure TCP proxy using Nginx upstream at /etc/nginx/nginx.conf</li></ol><pre>stream {<br>    # List of upstream AMQP connections<br>    upstream stream_amqp {<br>        least_conn;<br>        server 10.0.0.24:5672;  # Broker IP and AMQP default port<br>        server 10.0.0.108:5672; # Broker IP and AMQP default port<br>        server 10.0.0.124:5672; # Broker IP and AMQP default port<br>    }</pre><pre>    # AMQP definition<br>    server {<br>        listen 12345; # the port to listen on this server<br>        proxy_pass stream_amqp;  # forward traffic to upstream group<br>        proxy_timeout 3s;<br>        proxy_connect_timeout 3s;<br>    }<br>}</pre><pre><strong>File: nginx.conf</strong></pre><p>6. Reload Nginx: nginx -s reload</p><p>7. Allow inbound rules to RMQ Security group for port 5672 and Public Server IP as a Source.</p><p>8. Make sure the connection between publisher and consumer is successful.</p><ul><li>install rabbitpy: pip install rabbitpy</li><li>Create a python script for testing the connection. Use public server IP while making connections to publisher and consumer.</li></ul><pre>import rabbitpy</pre><pre><strong>def publish(channel)</strong>:<br>    body = &quot;Testing Connection...&quot;<br>    message = rabbitpy.Message(channel, body, {&#39;content_type&#39;:              &#39;text/plain&#39;})<br>    response = message.publish(&#39;rabbitpy-tests&#39;, &#39;ha-routing-key&#39;, mandatory=True)</pre><pre>    if response:<br>        print(&#39;Message delivered successfully&#39;)<br>    else:<br>        print(&#39;Message delivery failed&#39;)</pre><pre><strong><em><br># Created a new admin user &quot;test&quot; with password &quot;test&quot;<br># The below is the IP of the proxy<br># url_pattern:            &quot;amqp://username:password@public_host_addr:AMQP_port+vhost&quot;</em></strong></pre><pre>if __name__ == &#39;__main__&#39;:<br>    url = &quot;amqp://test:test@10.0.100.198:12345/%2f&quot;  <strong><em># RMQ Conn Str</em></strong></pre><pre>    try:<br>       with rabbitpy.Connection(url) as connection:<br>            with connection.channel() as channel:<br>                <strong><em># enabled publisher confirms</em></strong><br>                channel.enable_publisher_confirms()<br>                publish(channel)<br>    except rabbitpy.exceptions.MessageReturnedException as ex:<br>        print &quot;Failed to publish %s&quot; % ex<br>    except KeyboardInterrupt:<br>        print(&#39;exiting...&#39;)</pre><pre><strong><br>File: publisher_test.py</strong></pre><h4><strong>Output</strong></h4><pre>$ python publisher_test.py<br><strong>Message delivered successfully</strong></pre><h3>Make Shovel Connection using RMQ Management Portal</h3><ol><li>Go to your Source RMQ management dashboard and logged in as an admin user.</li><li>Select the Admin tab and go into Shovel Management settings.</li><li>Add a new Shovel with config:</li></ol><pre>NAME: Test Shovel Plugin<br>SOURCE: AMQP 0.9.1<br>URI: amqp://user:password@Source_address:5672/%2f<br>QUEUE: test_shovel_plugin_sour  <strong><em># Source Queue name that you want to migrate</em></strong></pre><pre>DESTINATION: AMQP 0.9.1<br>URI: amqp://admin_user:admin_password@destination_address:12345/%2f<br>QUEUE: test_shovel_plugin_dest <strong><em># Destination Queue name where you want to migrate</em></strong></pre><pre><br><strong>Shovel Config</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gtUw7uzMaiRLj4VXDKs52A.png" /><figcaption>Shovel Config</figcaption></figure><p>4. Check <strong>Shovel Status</strong> and make sure that the connection is successful.</p><p>5. The last step is to remove all the workers from the source queue and make sure that the consumer count is only <strong>1</strong>.</p><p>6. Tadda message migration completed 👏🏻</p><h3>Extra Points to look for</h3><ol><li>Make sure no other consumer is connected to the source queue while migrating. Otherwise, it won’t migrate messages to the destination queue.</li><li>Make sure you revoke public access post message migration.</li><li>Pass patterns while providing the source queue name to the shovel configuration. <strong>Example</strong>: test_*</li><li>Make sure you append <strong>%2f</strong> at the end of the URI to the shovel configuration. The vhost is a single segment. Thus, any “/” characters that appear in the vhost name must be percent-encoded. <strong>Example</strong>: amqp://test:test@dest_add:5672/%2f</li></ol><h3>References</h3><ol><li><a href="https://www.rabbitmq.com/shovel.html">https://www.rabbitmq.com/shovel.html</a></li><li><a href="http://codereform.com/blog/post/configuring-an-nginx-tcp-proxy-for-my-rabbitmq-cluster-under-10-minutes/">http://codereform.com/blog/post/configuring-an-nginx-tcp-proxy-for-my-rabbitmq-cluster-under-10-minutes/</a></li><li><a href="https://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-December/024730.html">https://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2012-December/024730.html</a></li></ol><blockquote>Thanks for reading this article. If you enjoyed this article, please give a few claps so that it reaches more people who could benefit from it!</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=371b784d22d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/cross-region-rmq-message-migration-using-shovel-plugin-371b784d22d3">Cross-region RMQ message migration using Shovel Plugin</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mistakes that we usually do while handling on-call duty & How we can overcome this?]]></title>
            <link>https://medium.com/squad-engineering/mistakes-that-we-usually-do-while-handling-on-call-duty-how-we-can-overcome-this-6a578bdf2ae5?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/6a578bdf2ae5</guid>
            <category><![CDATA[sre]]></category>
            <category><![CDATA[mistakes-to-avoid]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[ad-hoc]]></category>
            <category><![CDATA[on-call]]></category>
            <dc:creator><![CDATA[Ujjwal Gupta]]></dc:creator>
            <pubDate>Fri, 20 May 2022 03:34:50 GMT</pubDate>
            <atom:updated>2022-05-20T03:34:50.244Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Mistakes that I made on my first on-call &amp; how I improved</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fuHdvLtVIfl6ftVONhoKcg.jpeg" /></figure><blockquote><em>This article is solely on my personal experience of what happened when I was first assigned to an on-call duty and the mistakes I made in carrying out that responsibility.</em></blockquote><h3>About On-Call Responsibility</h3><p>On-Call is a crucial duty that ensures that the system always should be up and running. On-Call is a practice that we usually follow in <a href="http://www.squadstack.com/"><strong>Squadstack</strong></a>. As part of on-call duties, we assign a dedicated engineer as an on-call engineer (rotational). The key goal for the engineer is to ensure uptime of critical systems at desired quality levels. They take up the following responsibilities:</p><ol><li>Resolve end-user escalations &amp; handle requests as they happen.</li><li>Act on infra alerts &amp; pages and make sure infra is up and running.</li><li>Suggesting &amp; setting up a better alerting and monitoring system (if time permits). Thereby enabling future on-call engineers to take proactive measures rather than reactive measures.</li><li>(IF you’re lucky &amp; time permits) Reducing Technical debt.</li></ol><blockquote>Even more important than daily work is the improvement of daily work. — Mike Orzen</blockquote><h3><strong>Mistakes that I made on my first on-call</strong></h3><h3>1. Making Assumptions 🤔</h3><p>It’s so easy to make an assumption. All you need is incomplete information coupled with urgency about a situation.<br><em>Our Assumptions are not the Truth, it’s just a tool that our minds are using to conclude any given scenario.</em></p><p><strong>Why do we make assumptions while we handle on-call?</strong></p><ol><li>We might not have the full context of the system.</li><li>We want to fix anything ASAP, which causes us to make assumptions.</li></ol><h4><strong>How we can overcome this?</strong></h4><ol><li><strong><em>Avoid making assumptions:</em> </strong>might sound cliche but it is what it is; as more you make assumptions, the easier it is to continue making them.</li><li><strong><em>Never make assumptions until you know the right context</em> - </strong>try to figure out the “why” behind things &amp; get to the root cause of the problem.</li><li><strong><em>Conclude anything when you have proof and are 100% sure</em>.</strong></li></ol><figure><img alt="Ignoring errors and assuming that the system will auto-heal while handling on-call" src="https://cdn-images-1.medium.com/max/735/1*7vZFaVEo0INZsMb-AHweGw.jpeg" /><figcaption>Ignoring errors and assuming that the system will self-heal</figcaption></figure><h3><strong><em>2. Let’s Hack-fix the system 👨‍🚒</em></strong></h3><p>I wouldn’t call this a mistake, but it is far more than that if it becomes a habit. <em>So, what prompts an on-call engineer to hack-fix a system 🤔?</em></p><ol><li><strong><em>Adhoc with an urgent tag</em></strong> causes panic among engineers, thus the first thought that springs to mind in that situation is to hack-fix the system. They forget that it’s only a temporary patch, not a permanent one; and they’ll repeat it again and again.</li><li><strong><em>Too many escalations</em> </strong>cause frustration which leads to the idea of just hack-fixing the system.</li></ol><h4><strong>How to overcome this?</strong></h4><p>Panic is the root of all errors, so engineers must keep their cool and concentrate on the following steps.</p><ol><li>First, <strong><em>identify the problem</em></strong> and assign a priority based on whether it is urgent and how much urgent.</li><li><strong><em>Acknowledge the business owners</em></strong> and offer them a rough time frame.</li><li>Based on the severity of the issue, decide whether to go with a temporary or permanent solution.</li></ol><h3><strong><em>3. Too much Dependent on Your Peer 🎭</em></strong></h3><p>When we begin our first on-call duty, we may not have a complete understanding of what we must handle, as well as our responsibilities and expectations. So, it’s fine to enlist the help of your peers or seniors in these situations, but we often overlook the fact that we are the primary owners of on-call, not our peers, and we continue to ping them for assistance.</p><p><strong>Why does this happen?</strong></p><ol><li>We as a developer always takes precautionary measures whenever we handle any new responsibility. We even thought that our actions might break the whole system.</li><li>Laziness is also a crucial factor here, as we don’t want to spend time figuring out the root cause of the problem.</li></ol><h4><strong>How we can overcome this?</strong></h4><ol><li>Keep in mind that your peers are there to assist you, not as your playbook.</li><li>Take assistance when it is genuinely vital or breaking or if you are having a pool of escalation.</li></ol><h3><strong><em>4. Improper Prioritization of Tasks 😕</em></strong></h3><p>When it comes to addressing on-call responsibilities, task prioritizing is critical, and we often overlook it. <br>This occurs because we believe that every problem must be solved as soon as possible, or else be prepared for consequences. But, as we discussed above, we must first examine the problem and its implications before prioritizing it.</p><h4>What methods can we use to examine and prioritize problems?</h4><ol><li>Prioritization is a difficult process, and it demands experience; thus, we should take peer help in assessing the problem. But, before seeking help, we must first assess the problem on our own.</li><li>Prioritization is entirely determined by the affected audience i.e the users which are affected by the problem.</li></ol><h3><strong><em>5. The System will Self-heal 👽</em></strong></h3><p>The Myth that most developers assume in their day-to-day work. When they see a green light in a red blood ocean. They feel “Wooh… everything going good we don’t need to worry 😎”</p><h4><strong>How to overcome this?</strong></h4><blockquote>Just recall my very first point — <strong><em>never make assumptions</em></strong>.</blockquote><p>Make sure to check the reason behind the failure and whether it will self-heal or not.</p><h3><strong><em>6. Ignoring Errors</em></strong></h3><p>You’ll encounter a slew of errors while managing on-call responsibilities, some of which should be addressed and others that should be ignored. Differentiating between the two would be extremely difficult for an on-call engineer. So, instead of ignoring them all, we straightforwardly ask our peers for help without first analyzing the problem. “This looks like something unknown, let’s leave this or let’s ping peer developers; they might have a greater context on that”.</p><h4><strong>How to overcome this?</strong></h4><blockquote><em>This will pass with time; the more you understand the system, the more comfortable you will be with it.</em></blockquote><h3>Other meta points I realized while handling On-call Responsibilities</h3><ol><li>Things might be hard for you in the beginning but you should keep your calm.</li><li>Your peers are always there for help if you need it.</li><li>Don’t exhaust yourself too much. Take breaks whenever you got time.</li></ol><p>Thank You for reading this article. If you enjoyed this article, please give a few claps so that it reaches more people who could benefit from it!</p><blockquote><em>PS:- We’re hiring across tech &amp; non-tech roles, check out </em><a href="https://www.squadstack.com/careers/"><em>https://www.squadstack.com/careers/</em></a><em> for all open positions &amp; for sneak peak into our culture;</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a578bdf2ae5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/mistakes-that-we-usually-do-while-handling-on-call-duty-how-we-can-overcome-this-6a578bdf2ae5">Mistakes that we usually do while handling on-call duty &amp; How we can overcome this?</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Remote Onboarding Experience At SquadStack]]></title>
            <link>https://medium.com/squad-engineering/my-remote-onboarding-experience-at-squadstack-45e42fa0806?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/45e42fa0806</guid>
            <category><![CDATA[culture]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[onboarding]]></category>
            <dc:creator><![CDATA[Yasin Munshi]]></dc:creator>
            <pubDate>Tue, 19 Oct 2021 08:44:37 GMT</pubDate>
            <atom:updated>2021-10-19T08:44:37.597Z</atom:updated>
            <content:encoded><![CDATA[<h4>Onboarding that makes you feel at home.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b8OmfzDF7UHkVOosVmXqaw.jpeg" /><figcaption>Image by <a href="https://www.shutterstock.com/g/BROvector">BRO.vector</a> from <a href="https://www.shutterstock.com/home">shutterstock.com</a></figcaption></figure><p>If you’re reading this, you’ve probably heard of <a href="https://www.squadstack.com/"><strong>SquadStack</strong></a> (and hopefully wish to co-build awesome tech alongside us).</p><p>In case you haven’t, we’re a bunch of highly passionate, committed developers solving complex problems and learning new things every day.</p><p>We’re also growing like crazy, and looking for smart, like-minded engineers who can get shit done.</p><blockquote>Let’s get to the point?</blockquote><p>As we all know, first impressions are significant. Besides the application and interview process, the onboarding experience is one of those first impressions of an organisation that can have a lasting impact on you and your work there. In fact, it’s a powerful experience that can dictate the mindset a new employee would have for the first few months.</p><p>I’m Yasin, and I recently joined SquadStack as a Product Engineer — 2. This blog aims to paint a picture of what it’s like getting onboarded as a new member in the SquadStack Tech Team. To be honest, it’s nothing short of exhilarating! The most surprising part is that even though the entire process was remote, it didn’t feel that way at all.</p><p>Here are some things that stood out the most about my remote onboarding:</p><h4>1. Pre-Onboarding Material</h4><p>One week after I accepted the offer, I got paired with a buddy (more on this later), who provided me with reading material on how the organisation functions, the product and its use-cases, engineering principles that the team operates on, etc. This helped give me a realistic idea of what my day would look like when I finally joined.</p><h4>2. Buddy Pairing</h4><p>This was the most memorable part. Anyone joining the engineering team gets assigned a buddy who will be their “go-to” person for basically anything and everything. My buddy helped me quickly gain context of the organisation, the team, tools &amp; what they’re used for, and communication/workflow procedures.</p><h4>3. Product Onboarding Sessions</h4><p>These were in-depth onboarding sessions that helped give me a thorough understanding of the product roadmap, use cases, user personas, competitors — basically everything product!</p><h4>4. Introduction Meetings</h4><p>To get to know my team members better, I had one-on-one introduction meetings with each of them. The importance of these sessions cannot be overemphasised. Especially in a remote work setting in which going up to people and making connections on your own is just not an option.</p><h4>5. Engineering Wiki</h4><p>A super comprehensive central database of all things engineering! From engineering operating principles and best practices to processes, infrastructure, tooling, and how-to guides, this has everything an engineer in the team would ever need to know. Truly mind-blowing documentation! Here’s a sneak peek:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tkgUU_NlDqgkhxymVyK_AA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FHb78ry1oF8h76ik6_BwYQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pwQ4x0yhfxvV_MWsBG6svg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GlvBkXedOZ6sbHrFLdZB6w.jpeg" /></figure><p>Needless to say, this was one of the smoothest onboarding experiences I have ever had. And it indeed would not have been possible without the people-first culture that SquadStack has cultivated.</p><blockquote>One of SquadStack’s guiding philosophies is to always put people above products and profits, and it shows. It’s the people here that make every day at work a treat.</blockquote><p>Want to co-build with us? Check out engineering open positions <a href="https://www.squadstack.com/careers/"><strong>here</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45e42fa0806" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/my-remote-onboarding-experience-at-squadstack-45e42fa0806">My Remote Onboarding Experience At SquadStack</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[#RealTalk — Womxn in Tech | Q/A with Shalini Aggarwal (Tech Lead, SquadStack)]]></title>
            <link>https://medium.com/squad-engineering/realtalk-women-in-tech-q-a-with-shalini-aggarwal-tech-lead-squadstack-360c2a2a10d6?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/360c2a2a10d6</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[diversity-and-inclusion]]></category>
            <category><![CDATA[women-in-tech]]></category>
            <category><![CDATA[culture]]></category>
            <dc:creator><![CDATA[Simran Kaur]]></dc:creator>
            <pubDate>Tue, 19 Oct 2021 08:27:33 GMT</pubDate>
            <atom:updated>2021-10-19T08:27:33.519Z</atom:updated>
            <content:encoded><![CDATA[<h3>#RealTalk — Womxn in Tech | Q/A with Shalini Aggarwal (Tech Lead, SquadStack)</h3><p>Celebrating the stories of underrepresented genders in tech — one talk at a time.</p><blockquote>SquadStack’s #RealTalk Q/A series seeks to encourage gender equality in the workplace, by amplifying the voices of women and non-binary people making a difference in the Tech World.</blockquote><p>In the first edition, the spotlight is on our very own <a href="http://linkedin.com/in/aggarwalshalini"><strong>Shalini Aggarwal</strong></a> (Tech Lead, <a href="https://www.squadstack.com/"><strong>SquadStack</strong></a>). So naturally, this one is quite special. Go, Shalini!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gliuphxWejjq3Cle3RgGrg.jpeg" /></figure><h3>About Shalini</h3><p>With over five years of experience working in tech, Shalini is skilled in database design, complex programming, and cloud infrastructure.<br> <br>From being the first woman developer in our team to leading a team of women developers, she’s had quite the journey!</p><p>In this interview, we’ll explore several topics ranging from the skillsets needed to get into tech, the challenges women face in the industry, and finally, how to build a diverse and inclusive workspace.</p><h3>Let’s get talking!</h3><h4>1. What inspired you to get into tech?</h4><p>My journey with tech began quite early in life. I had a cousin who was way into tech, constantly introducing me to different games and software. He would show me how they make our lives easier by giving insights into how they are built and function.</p><p>Schooling played a role too. In class 3, we had computer classes where they taught logo turtle (the little turtle you could move around). That was my first interaction with any programming language, and it just left me in awe. I was fascinated! We could just make the system do something on our command and generate actual results!? Now that I look back, that was the moment I realised that this is something I’d want to explore in the future.</p><p>And I explored. I got into animation and saw another side of tech, started opting for things that would eventually get me into this industry, and coding in class 11 was when it became clear that this is where I belong.</p><h4>From then till now, my journey has taught me …</h4><blockquote>“Determination and love for your dream can help you overcome every obstacle that comes your way.”</blockquote><h4>2. What skill sets and qualities do you need?</h4><p>I believe I&#39;m constantly in the process of learning and growing into a better version of myself. Although technical skills are essential and there are a few I need to up-skill myself with, soft skills are equally important - especially when you&#39;re working with a team. I&#39;ve always been a people person, due to which I believe I&#39;m good at observing behavioural patterns. This ability has been beneficial in understanding team dynamics and leadership better.</p><h4>3. What is the best thing about working in tech?</h4><p>Building stuff that has an actual impact on the people around us. You can even end up building something that completely changes how a large population of the world works, which leaves me in awe.</p><h4>4. What are the biggest challenges you have faced as a woman in tech?</h4><p>It starts right from school. My batch hardly had any women in STEM. Sadly, there is a general belief that women are not smart enough for this industry, and it shows. One time, I submitted a game I developed as a project &amp; the evaluator dismissed it as something I couldn&#39;t have built. In college, I got selected for an internship along with the assertion that I&#39;m the first woman they&#39;ve offered this to and that it took &#39;convincing&#39; people to give me a chance.</p><p>From getting paid less than the men doing the same work (sometimes, with lesser experience) to being treated a little less than the guys on the team, I&#39;ve faced some form of sexism in this industry for most of my life. A common belief is that women are just not good at this stuff and need to work harder because it doesn&#39;t come as naturally to a woman as to a man. And I&#39;ve been told that, too.</p><h4>5. What are the things that are essential to creating an inclusive workplace?</h4><p>Treat everyone equally, give opportunities to people based on their skills and not their gender. When I was actively looking for new opportunities, I remember SquadStack&#39;s hiring process standing out so much more than the rest. I honestly didn&#39;t know I was the first woman to have joined the Tech Team, which was a little unexpected because I couldn&#39;t even sense a hint of sexism during the whole process! It&#39;s been quite a journey, and I can proudly say we&#39;re building a diverse and inclusive workplace where everyone feels like they belong. Here are a few things we do to ensure inclusivity at SquadStack:</p><ul><li>Use gender-inclusive terms instead of exclusionary terms like “bro” or “guys”, when referring to a group.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SYRgodg97GqO0U23ACgqOQ.png" /><figcaption><strong>We have a custom Slackbot message that pops up when someone uses “guys” or “bro” in the workplace.</strong></figcaption></figure><ul><li>Normalise women in tech instead of looking at them like some extraordinary creatures who&#39;ve somehow beaten all odds to be there.</li><li>Normalise women&#39;s needs instead of letting them stay taboo.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*dNPCjpS1X0-rRaJoLc6y_g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*KZ6d4e08TCVQXSdZVj0wYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*JJZMVmJ3Rr7dH-x5uVDzSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*z6L1seunDtDji7SbMDNCXw.png" /><figcaption><strong>No, we don’t have specified “period leaves.” Instead, we’ve created an environment where people can talk about it openly like any other normal thing, because that’s what it is — normal.</strong></figcaption></figure><ul><li>Accept that gender doesn&#39;t play a role in how good one can be in their work, and it shouldn&#39;t affect performance evaluation. It’s a good practice to keep re-visiting roles to ensure there’s no selection bias for or against hiring women or men.</li></ul><p>At SquadStack, we’re proud of the healthy 1.5:1 gender ratio of men to women we’ve been able to create. And there’s no secret formula for it. We’ve been able to do this by simply hiring the best person for the job, giving equal importance to all genders.</p><h4>6. Do you have any advice for future women in tech?</h4><p>Don&#39;t let go of something you love or want to do just because it&#39;s a little hard for you, rather fight for it and show people that the tech industry is no bar for women. Help normalise the existence of women in tech so that we don&#39;t even need to take such initiatives in the future. In the end, we are the ones who&#39;d have to take the baton and cross the finish line by actually running the race.</p><p>We’re looking for smart, like-minded engineers to co-build SquadStack with us. Interested? Check out engineering open positions <a href="https://www.squadstack.com/careers/"><strong>here</strong></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=360c2a2a10d6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/realtalk-women-in-tech-q-a-with-shalini-aggarwal-tech-lead-squadstack-360c2a2a10d6">#RealTalk — Womxn in Tech | Q/A with Shalini Aggarwal (Tech Lead, SquadStack)</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Celery in production: Three more years of fixing bugs]]></title>
            <link>https://medium.com/squad-engineering/celery-in-production-three-more-years-of-fixing-bugs-2ee462cef39f?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/2ee462cef39f</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[celery]]></category>
            <dc:creator><![CDATA[Ayush Shanker]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 08:17:44 GMT</pubDate>
            <atom:updated>2021-08-10T08:17:44.599Z</atom:updated>
            <content:encoded><![CDATA[<p>This post covers the various issues we faced with celery in production over the last year or so. An ex-team member had iterated on a bunch of celery issues around 3 years before this iteration and had blogged about his experience here:</p><p><a href="https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d">Two years with Celery in Production: Bug Fix Edition</a></p><p>While that iteration fixed a lot of problems at that time, some problems had only gone hidden &amp; then reappeared a few months later. So it was given that a more definitive iteration would be required to fix the problems this time around.</p><h3>Our instances would always be on high RAM usage, and manually scaling them to 2x or 4x the size would not fix the problem</h3><p>This is so counter-intuitive, right? If I have an instance with 90% memory usage, doubling its resources should result in approx 45% memory usage. But the expected didn’t happen. We use AWS instances for celery workloads, so the team would scale up the instance in hopes of containing the memory requirement. The instance would still show 90–100% RAM usage so someone tried to change the instance type from c5 to t3 (c5 is for CPU bound tasks, and t3 is burstable CPU instance type for non-CPU bound workloads). This didn’t work either.</p><p><strong>The Solution: </strong>Whenever your celery processes (or any other, for that matter) takes a lot of RAM, your first response should be to gather a list of processes with resident &amp; shared memory usage. Without this data, you’re very likely to be shooting an arrow in the dark. So anyway I went ahead on the culprit server, which had 16 gigs of memory. A quick view of htop &amp; then sorted by memory usage showed that the process with max memory usage was using ~320MBs of memory. I already knew that our application takes ~280MBs when I run python manage.py shell, so it’s not a case of a few rogue celery processes taking too much RAM. Next, I ran</p><pre>$ ps aux | grep &#39;celery worker&#39; | wc -l<br>  52</pre><p>If you do the math, 280MB * 52 ~= 14.5GB. So once all the celery workers are launched, the instance is already around 90% full. But why does this not get fixed when I double the instance size? That’s because if you don’t specify --concurrency for each worker explicitly, it defaults to using the number of cores available as the default value for concurrency. This option controls how many children will be forked for each worker defined. So when you double the instance, you usually also double the CPUs in the instance, thus increasing the number of celery processes in the process. A similar thing happens when you switch from c5 to t3 instance, you get more (but much weaker) CPU cores.</p><p>The final solution was to explicitly specify the concurrency in the worker launch command:</p><pre>$ celery worker -A project (...) --concurrency=4</pre><h3>Some workers were using 4–5x times the RAM taken by a freshly launched child worker process</h3><p>Just by reading the description of the problem, you can tell that there’s a strong hint of memory leak here. I had seen in htop that the culprit workers were all at least a few hours old. This problem was new, not something we faced in last iteration. So why did this happen now? Well in the previous iteration (See “<strong>Worker servers always had an unexplainably high CPU usage” </strong>section), we had removed the <strong>— max-memory-per-child</strong> flag from options, which now meant that the processes were not dying after regular intervals. So we made a mistake; while that setting was not working for us at that time, we should have changed it to a higher value than outright removing it.</p><p>So while this was easily fixable by re-introducing the --max-memory-per-child flag, at that time I had a script lying around that’d kill a process matching certain properties, like if it goes beyond a threshold in memory usage. The kill mechanism itself is first it tries to soft-kill (letting the running task complete), and then initiates hard kill if the process is still running after a timeout. This method is still inferior to using --max-memory-per-child, but probably better for us right now.</p><h3>Instances would frequently use more RAM than they should be using</h3><p>This is the same problem as section “<strong>Worker servers always had an unexplainably high RAM usage</strong>” in <a href="https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d"><strong>https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d</strong></a></p><p>We thought it had been fixed until it re-appeared weeks later 😣.</p><p>So this time we already knew what the problem was: child worker orphans that don’t die out once their master worker has been killed in the deployment process. Now we know that all orphans get “adopted” by PID 1 once their parent die. So all I had to do was to write a small script that’d kill any celery worker process that had PID 1 as its immediate parent. We run our celery workers as multiple workers in each instance, all under supervisord which in turn is controlled by systemd.</p><p>Now, this is not the best way to solve this. Ideally, the service manager (supervisord) should be aware of all the forked children &amp; kill them when the parent is dead. A quick search tells that Systemd already does that, and is also available on the instance to be used. But we now generate the celery worker definitions from a YAML file into supervisord configs, and replacing supervisord with systemd would mean that I’ll also have to change that YAML to config generator accordingly. That would’ve taken more time than what I had then, so I decided to leave it this way. And it has been working fine till now with no issues so I won’t complain much either.</p><h3>A few celery workers would hang after a few hours of usage, and would just stop consuming any more tasks</h3><p>Let’s get you a little bit of background on our crons setup: Around the same time as the old blogpost, we switched from basic crond on a single large instance to celery beat with workers across multiple machines. We had migrated to <a href="https://healthchecks.io/"><strong>healthchecks.io</strong></a> for alerting in case a scheduled cron misses its run.<br>Now, after a few hours of usage, a few celery workers would stop consuming any more tasks. So I went in the instance &amp; straced the child worker:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3857851e7c699db94a2a111e2ffdca36/href">https://medium.com/media/3857851e7c699db94a2a111e2ffdca36/href</a></iframe><p>This looks familiar 🧐</p><p>(see section “<strong>Workers stayed idle, not consuming any tasks”</strong> in <a href="https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d#13b4"><strong>https://medium.com/squad-engineering/two-years-with-celery-in-production-bug-fix-edition-22238669601d</strong></a>)<br>So this has reappeared as well 😿</p><p>strace gives me the system call that it’s stuck on, but that’s not good enough. I want to see what application code, what library code, what underlying C function is responsible here. Basically, I need the backtrace of the process. Let’s begin!</p><ol><li>install python debug build of python: sudo apt install python-dbg</li><li>install gdb: sudo apt install gdb</li><li>take a core dump of the process: sudo gcore $PID</li><li>launch the gdb attached to core dump: gdb python core.$PID</li><li>inside the gdb, use py-bt to get python level backtrace &amp; bt for C level backtrace</li></ol><p>After I collected backtraces from all the idle workers, I found that all of them had a similar backtrace, of top few lines are</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6ce80866fb49c231828848ced272581f/href">https://medium.com/media/6ce80866fb49c231828848ced272581f/href</a></iframe><p>and the corresponding C backtrace is</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/638130ea547ddcbf986f6dc50674f0fc/href">https://medium.com/media/638130ea547ddcbf986f6dc50674f0fc/href</a></iframe><p>It looks like urllib3 is waiting to acquire GIL before it could decompress the HTTP body chunk, possibly stuck in a deadlock. Not much I can do here, this looks like a bug — which won’t be fixed now that Python 2 has reached EOL. We eventually created a workaround for this that works this way:</p><ol><li>once every few seconds, when a celery worker picks a new task, it sends a heartbeat to a local redis instance</li><li>A cron checks redis &amp; kills whichever worker hasn’t sent a heartbeat in last few minutes, and sends a SIGTERM followed by a SIGKILL.</li></ol><p>There’s a bit more detail to this workaround, but that’s a story for another time.</p><p>Also: it’s easier to just use <a href="https://github.com/benfred/py-spy"><strong>py-spy</strong></a> for this.</p><p><a href="https://github.com/benfred/py-spy">GitHub - benfred/py-spy: Sampling profiler for Python programs</a></p><h3>IO vs CPU bound tasks</h3><p>We had some instances which were having high CPU usage, while there were others with very low CPU usage. This is not a problem in itself, but I wanted to have a better visibility of CPU bound vs IO bound load in each queue to be able to allocate infra accordingly.</p><p>Celery provides some signals on which you can attach your listeners. <a href="https://docs.celeryproject.org/en/stable/userguide/signals.html"><strong>https://docs.celeryproject.org/en/stable/userguide/signals.html</strong></a>. We are particularly interested in 2 of these signals:</p><ol><li><a href="https://docs.celeryproject.org/en/stable/userguide/signals.html#task-prerun"><strong>task_prerun</strong></a></li><li><a href="https://docs.celeryproject.org/en/stable/userguide/signals.html#task-postrun"><strong>task_postrun</strong></a></li></ol><p>If I log the current clock time and CPU time at each of these signals, I can get the time taken by each task. I can then use the CPU time difference divided by clock time difference to get an approximate idea of CPU vs IO bound workload for each task. Let’s create the listeners:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fdcf6ed237857ac90865532cfd313fb/href">https://medium.com/media/8fdcf6ed237857ac90865532cfd313fb/href</a></iframe><p>After the logs are collected, I’d run a script that would transform the above lines into a table like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*OBRURpNbHeFJAw7ozP-RNA.png" /></figure><p>A plot of cpu_time_difference divided by clock_time_difference makes it clear which tasks are IO bound &amp; which are CPU bound. The IO bound tasks would peak around lower X-axis values. Here’s the plot from our system (I have defaced the actual task names with hex codes), where you can see that the IO bound tasks peak towards the left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/0*swXXXvhIoyMzbfFB.png" /><figcaption>The Y axis is indicative of number of tasks</figcaption></figure><h3>Use Gevent as worker pool</h3><p>This is not a problem, but rather an optimization that I wanted to try out. We started using gevent as the pool option instead of the default prefork in some of our IO bound workers. This didn’t go well for because some libraries as well as application code were not thread-safe, causing some deadlocks &amp; other annoyances. So this was reverted within minutes in all but 1 worker whose only tasks were shooting webhooks, and that too was reverted days later because it was too unsafe to let it be on production.</p><h3>Key Takeaways</h3><ul><li>Celery has been one of the more unstable pieces in our infra over the years.</li><li>Supervisord should’ve killed celery’s forked workers once the parent dies. Either it is not reliable or we haven’t been able to make this happen in at least a few attempts now. I’m inclined to believe the former is true, but neither outcomes will look good for supervisord.</li><li>Verify if the bug is truly solved, or just dormant. Sometimes it may mean not closing the bug tracker issue before your changes have been running for few weeks in production.</li><li>Don’t debug just by google/github issues searches. Brendan Gregg has written some great tips around performance debugging, some of which are relevant when debugging errors as well: <a href="http://www.brendangregg.com/methodology.html"><strong>http://www.brendangregg.com/methodology.html</strong></a></li><li>There are too many thread-unsafe libraries in python ecosystem. It’s also very easy to create your own thread unsafe code. As a result, gevent (or eventlet) will remain a pipedream except in very limited scenarios.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ee462cef39f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/celery-in-production-three-more-years-of-fixing-bugs-2ee462cef39f">Celery in production: Three more years of fixing bugs</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Optimizing the runtime of CircleCI builds on a Django application]]></title>
            <link>https://medium.com/squad-engineering/optimizing-the-runtime-of-circleci-builds-on-a-django-application-f1702575a2a5?source=rss----256a175005d---4</link>
            <guid isPermaLink="false">https://medium.com/p/f1702575a2a5</guid>
            <category><![CDATA[circleci]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[continuous-integration]]></category>
            <category><![CDATA[django]]></category>
            <dc:creator><![CDATA[Kunal Mahajan]]></dc:creator>
            <pubDate>Sun, 28 Feb 2021 16:00:42 GMT</pubDate>
            <atom:updated>2021-02-28T16:00:42.193Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlXJED0uomS1YSjQd_rXXA.jpeg" /></figure><h3>Introduction</h3><p>If you are pretty knackered by the amount of time your automated builds take to finish, then this blog is for you. We at <a href="https://www.squadstack.com/">Squadstack</a> have been using Continuous Integration for a long time now but it is only recently that we started using our build pipeline more efficiently. We have added some simple steps to our pipeline which has now helped us in reducing the total time to <strong>just a fraction of the time</strong> which our pipeline used to take earlier.</p><p>We majorly work on <strong>python and Django</strong> and we have used some hacks related to Django and some shell scripting which helped us in improving the runtime of our test cases. Before going straight towards the optimization let’s start from scratch and understand what is Continuous Integration and how we have integrated it into our system.</p><h4>What is Continuous Integration?</h4><p>Continuous Integration(CI) is a development practice that most companies follow. It is guided by some of the key principles like revision control, build automation and automation testing. We at SquadStack follow this practice quite religiously. As an early-age startup, we are growing day by day, so it is important that all our code changes are error-less and don’t affect our existing codebase. Continuous Integration helps us integrating code changes into a shared code repository. This practice encourages committing small changes more often than committing large changes infrequently. Each commit triggers a build during which tests are run that helps us to identify if anything was broken by the changes we committed.</p><h4>What benefits did we get after integrating Continuous Integration in our system?</h4><ul><li>We were able to <strong>detect errors more quickly</strong>. Whenever we commit some changes in our shared repository, that commit is followed by some automation scripts which we wrote to ensure if everything is working as expected or not. If that automation script fails then the developer gets to know that something is broken and then the developer can take appropriate actions to fix the error.</li><li>The result of every automation script is <strong>visible to every developer in our team</strong>. Everyone can see if the script is working fine or throwing some errors. This ensures accountability within the team when there is an issue in the respective commit that should be fixed before deploying to production. The result of every commit is visible to us on slack.</li><li>It generated a <strong>fast feedback loop</strong>. Things that as a software developer can hurt us most are the lack of feedback on the quality and impact of the changes we made. As a software developer, it is easy to write code that we think will work as expected but more often than not it is quite the opposite. We as software developers want to quickly commit code and move on to another task without running any sanity checks. But this can hit us quite hard after sometime when we try to figure out which change and from when (and by whom) broke something. Continuous integration removes such situations to happen because after each commit we will get to know if that commit is working as expected or not.</li><li>Writing <strong>robust unit/integration test cases</strong> made our CI pipeline more robust. Writing effective/optimized test cases is the core principle to achieve a robust CI pipeline. Good quality test cases include covering edge cases, covering all business logics, number of queries being executed in every block of code, mocking third-party APIs response, etc., etc.<em> In SquadStack, we always try to write effective test cases and it is quite a feat that till now we never had to hire a software engineer in testing</em>.</li></ul><h4>How did we achieve Continuous Integration in SquadStack?</h4><p>Here are some basics which we used to get started with CI</p><ol><li>Implemented version control(Git, Bitbucket, SVN, etc). In our case it is Git.</li><li>Writing quality test cases for critical components of our product.</li><li>Got a suitable<strong> continuous integration and delivery service</strong> that will enable us to run those precious tests on every push to the repository. Our continuous integration service is <a href="https://circleci.com/product/#features"><strong>CircleCI</strong></a>.</li></ol><h4>How CircleCI can be implemented to achieve Continuous Integration?</h4><ul><li>There is a YAML file that CircleCI reads to prepare an environment. In that environment, we can install some OS-level requirements, project-level requirements, database, Redis, etc. We are using docker to prepare this environment.</li><li>After that, we write steps to run our custom migrations and test cases. Below is a snippet of the config.yml file which we have made to set up our CircleCI environment.</li></ul><pre># Use Circle CI Version 2.0<br>version: 2</pre><pre>jobs:<br> build:<br> working_directory: ~/circleci-django</pre><pre>docker:<br> — image: circleci/python:2.7.15<br> environment:<br> PIPENV_VENV_IN_PROJECT: true<br> DATABASE_URL: postgresql://postgres@localhost/databse</pre><pre>- image: circleci/postgres:11.5<br> environment:<br> POSTGRES_USER: postgres<br> POSTGRES_DB: database</pre><pre>- image: redis</pre><pre>steps:<br> — checkout</pre><pre>- restore_cache:<br> key: cache-key</pre><pre>- run:<br> name: Install Dependencies</pre><pre>command: |<br> sudo pip uninstall pipenv -y<br> sudo pip install pipenv==&lt;version_number&gt;<br> pipenv install</pre><pre>name: Wait for Redis, and Postgres<br> # Here we download dockerize and wait for each service to respond.<br> # 5432 is the port for DB (Postgres)<br> # 6379 is the port for redis<br> command: |<br> sudo wget <a href="https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz">https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz</a> &amp;&amp; sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz &amp;&amp; rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz</pre><pre> sudo dockerize -wait tcp://localhost:5432 -timeout 10m</pre><pre> sudo dockerize -wait tcp://localhost:6379 -timeout 10m</pre><pre>- run:<br> name: Run Migrations<br> command: pipenv run python manage.py migrate</pre><pre>- run:<br> name: Run Tests<br> command: pipenv run python manage.py test</pre><ul><li>On every commit, CircleCI reads this file, follows all the steps here, and runs our migrations and test cases.</li><li>In the CircleCI dashboard, we have the option to tell CircleCI which file should it read after each commit.</li><li>CircleCI also gives us an option to link it with our GitHub repo so that it can track all the commits occurring in that repository.</li><li>After running the test cases the status of that build is sent to us on a respective channel on slack.</li><li>CircleCI also sends the coverage report which would tell how much code is covered, where coverage is lacking, and how to improve it.</li></ul><h4>How did we optimize build run time using CircleCI?</h4><p>As discussed above, we highly focus on writing good quality test cases since we don’t have any dedicated testing team. As of now we have &gt; 4200 tests in one of our repositories and running these test cases along with migrations, os dependencies, pip dependencies would easily take around <strong>45 minutes or more</strong>. But after adding some logic we brought down that time to <strong>12–15 minutes</strong> i.e. reduced the time to approximately 1/3rd time. Here are the few simple things which we did to optimize the time taken:-</p><ol><li>We started running test cases <strong>in parallel</strong>. Running test cases in parallel allows running test cases in different threads. Each thread will have a different test database. So, suppose there are 4 parallel threads, there would be 4 different databases made on different threads.</li><li>One major issue which we were facing when running test cases in parallel was the cache backend would still be shared across all parallel threads. To fix that we used <strong>LocMemcache</strong>. LocMemCache is an in-memory cache (not persistent) in Django and so multiple caches are made for multiple parallel test processes. <strong><em>But why do we need multiple caches for multiple test processes? Why can’t we just use one cache across every parallel thread?</em></strong> <em>The answer to this problem is quite simple. Since we are creating different test databases for every thread, there can be a possibility two completely different things have the same id in different databases and when we try to cache that id then that could break things.</em> So, it is necessary to have different cache across different threads.</li><li>There was a <a href="https://gist.github.com/kunal5/73c687bab0ed64a2a6f1ce4c25002e73"><strong>custom TransactionTestCaseWithoutFlush</strong></a><strong> </strong>that was added since the flush operation takes a lot of time.</li><li>CircleCI gives us an option if we want to store something in the cache for every build. So, for every build, we <strong>cached all the OS/pip dependencies</strong>. The cache would invalidate if we add a new dependency and then the cache would be updated again. This saved us a lot of time as well. <a href="https://circleci.com/docs/2.0/caching/">Caching Dependencies in CircleCI</a></li><li>We started running migrations only when needed. If in a specific commit, any dev has created new migrations then only migrations should run on CircleCI. For achieving this <em>firstly we calculated the md5 hash of all the migration files separately and stored them inside per_migration_hash.txt. Secondly, we sorted the per_migration_hash.txt file so that the order remains consistent because the first step isn’t ordered. Thirdly, we calculated the md5 hash sum by combining the hash value of all the files which we got from the first step. And finally, stored the combined hash in the cache</em>. Now, whenever there will be a new migration file, the md5 hash sum stored in the cache would be different due to a new migration file which will also have some hash value, and combining the hash value of that file with the rest of the files will result in a different combined md5 hash sum. This step<strong> avoided unnecessary running of migration in each build</strong>. Below is some of the code which we wrote to achieve this step.</li></ol><pre>- run:<br> name: Get Migrations Hash</pre><pre>#First Step:- Calculating the md5 hash of all the migration files separately and storing inside per_migration_hash.txt.<br> #Second Step:- Sorting per_migration_hash.txt file so that the order remains consistent because the above step isn’t an ordered step.<br> #Third Step:- Calculating the md5 hash sum by combining the hash value of all the files which we got from the first step<br> #Final Step:- Printing the final hash<br> command: |<br> find apps/ -path “*migrations*” -name “*.py” -exec md5sum {} \; &gt; per_migration_hash.txt<br> sort per_migration_hash.txt &gt; sorted_per_migration_hash.txt<br> md5sum sorted_per_migration_hash.txt | cut -d’ ‘ -f1 &gt; combined_migrations_hash.txt<br> echo “Hash generated for migration files is $(&lt;combined_migrations_hash.txt)”</pre><pre>- restore_cache:<br> # Checking if the hash sum which we got in the above step is present in the cache or not. Caching key format is branch_name-checksum of combined_migrations_hash.txt .<br> # Note: These migrations caching will be branch dependent.<br> name: Restore Migrations Cache<br> key: cache-{{ .Branch }}-{{ checksum “combined_migrations_hash.txt” }}</pre><pre>- run:<br> name: Run Migrations</pre><pre># Inside save_cache step every time when the job is running we are storing the hash which we got from the<br> # Get Migrations Hash step inside ~/cached_hashed_migration/hash.txt in cache<br> # If hash.txt is present then migrations will not run printing “No changes in migrations detected”<br> # Else if hash.txt is not present, it will mean that there was a cache miss and there are some sort of changes in migrations file.<br> # So, now the migrations will run if the branch is one of develop or master<br> command: |<br> FILE=~/cached_hashed_migration/hash.txt<br> if test -f “$FILE”; then<br> echo “No changes in migrations detected”<br> else<br> echo “Change in migrations detected. Run Migrations”<br> if [ $RUN_MIGRATION_ON_ALL_BRANCHES == true ] || [ “${CIRCLE_BRANCH}” == “master” ] || [ “${CIRCLE_BRANCH}” == “develop” ]; then<br> pipenv run python manage.py migrate — settings=test_settings<br> echo “✅ ran migrate — thanks for waiting”<br> else<br> echo “Migrations to run only for develop or master branch 😎”<br> fi</pre><pre>fi<br> mkdir -p ~/cached_hashed_migration/<br> touch ~/cached_hashed_migration/hash.txt</pre><pre>- save_cache:<br> name: Save Migrations Cache<br> # Storing combined_migrations_hash.txt which we got in the Get Migrations Hash step inside cache every time when the job runs<br> key: cache-{{ .Branch }}-{{ checksum “combined_migrations_hash.txt” }}<br> paths:<br> — ~/cached_hashed_migration/hash.txt</pre><h4>What are the things which we are trying to improve in our current test case/CircleCI setup?</h4><ul><li>In general, we are using Redis as our cache backend but we use LocMemcache as our cache backend to run test cases in parallel as discussed above. Because we use LocMemcache in our test cases we are <strong>not able to run Redis-specific commands</strong> in our test cases. We have to mock those Redis commands to make our test cases work. This is one of the drawbacks as of now. We are still finding a way to use Redis instead of LocMemcache.</li><li><strong>Continuous Delivery</strong> is still something which we need to set up. Our current deployment process is still manual. Using Continuous Delivery we can automate this process.</li></ul><h4>Conclusion</h4><p>We as a company always believe in some of our <a href="https://www.squadstack.com/culture/">core values</a>. One such value is <strong><em>Do The Right Thing</em></strong> and to make the right product, we have to use the right tools which help us to be on track and support our development process in a meaningful way. We always follow some principles to run a successful CI pipeline:-</p><ol><li>Write good quality tests.</li><li>Fix or delete the tests we don’t longer need.</li><li>Keep builds green :-)</li></ol><p><em>Also, if anyone is interested then we are </em><a href="https://www.squadstack.com/careers/"><em>hiring</em></a><em> as well ;-). Come join us and together let’s put a dent in the world.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1702575a2a5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/squad-engineering/optimizing-the-runtime-of-circleci-builds-on-a-django-application-f1702575a2a5">Optimizing the runtime of CircleCI builds on a Django application</a> was originally published in <a href="https://medium.com/squad-engineering">SquadStack Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>