<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Sendoso Engineering - Medium]]></title>
        <description><![CDATA[Welcome to the Sendoso engineering blog! Find out what makes our engineering team tick. - Medium]]></description>
        <link>https://medium.com/sendoso-engineering?source=rss----93874bc3ef83---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Sendoso Engineering - Medium</title>
            <link>https://medium.com/sendoso-engineering?source=rss----93874bc3ef83---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:26:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/sendoso-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Life at Sendoso — ft. PK Office]]></title>
            <link>https://medium.com/sendoso-engineering/life-at-sendoso-ft-pk-office-c0c2498189e5?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/c0c2498189e5</guid>
            <category><![CDATA[sendoso]]></category>
            <category><![CDATA[engineering-culture]]></category>
            <category><![CDATA[employee-experience]]></category>
            <category><![CDATA[employee-engagement]]></category>
            <category><![CDATA[lifeblog]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 10:50:27 GMT</pubDate>
            <atom:updated>2022-06-08T09:34:47.636Z</atom:updated>
            <content:encoded><![CDATA[<h3>Life at Sendoso — ft. PK Office</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Cqca4leM_Doe89odlAGVw.png" /></figure><p>Teamwork, like friendship, is a basic human need that alters our moods and brings us together. We have an inspiring and unique employee culture that has helped us get there. At Sendoso, culture isn’t just about ping pong tables, sports fests, annual tours, or free lunches (although Sendoso has them all). Culture is the set of unspoken rules, guidelines, and values that govern a group of people. Independence and trust are the names of the game at Sendoso.</p><p>Teams are given complete autonomy and are expected to make their own decisions and be responsible for their deliverables from conception to deployment. More than managing people, our goal is to inspire them. We have faith in our teams to do what they believe is best for Sendoso.</p><p><em>We don’t want to keep our culture the way it is; we want to make it better.</em></p><h4><strong>Meet the Dream Team</strong></h4><p>For Sendoso, a dream team is one in which each member excels at what they do and works well together. Our concept of a great workplace does not include great perks, though we do have plenty. It is about investing in a dream team of talented individuals eager to pursue ambitious shared goals.</p><p>When the appropriate individuals with practices and tools aim to achieve a common goal, whether it’s launching a new product, starting a political movement, or serving a delicious meal, that’s what we call <strong><em>teamwork</em></strong>. We hire the best and value respect, excellence, integrity, inclusion, and collaboration.</p><h4><strong>Our Gems</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WqPzXB8JFpirIrQil21uEA.png" /></figure><p>Meet<strong> M. Raza Aslam</strong> — <em>Tech Lead Manager &amp; Senior Software Engineer II</em>. <strong><em>Workaholic</em></strong> is how he describes himself. Whether he’s having a productive or gloomy day, you’ll always find him with his laptop. One can spot Raza working tirelessly in the office or at home. M. Raza has been part of the Sendoso family from the beginning. <strong><em>Ruby on Rails, Vue.Js, CircleCI, GraphQL, Google Chrome Extension, </em></strong>and <strong><em>Third-party integrations</em></strong> are some of the main expertise of Raza. And yes! He is proficient in team management.<br>He is a knight in shining armor and the one who is never afraid to bell the cat. <strong>M. Raza </strong>is excited to share his experience so far in the company.</p><p>Let’s hear him out.</p><blockquote>A friendly environment, flexible working hours, and learning and growth opportunities are some of the main reasons for loving Sendoso. There are many learning and growth opportunities in Sendoso due to the wonderful and talented global team working on the latest technologies, security-related stuff, and many complex features.</blockquote><blockquote>I’ve worked with Fawad Naeem, Waqas Ali, and many other engineering and project managers. They are all very supportive, and they trust me. Fawad Naeem instilled in me the ability to believe in myself. That’s why I believe trust is the most important thing in each relationship. Their trust gives me more confidence and motivates me to perform outstandingly. Sendoso has a lot to offer, from stock options to international trips and the most supportive HR. That is what stands out about working at Sendoso from other places.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nwx8tgmODYO0d-PSfy4mnw.png" /></figure><p>His name has helped him break the ice on many occasions, and we love its uniqueness. Meet <strong>M. Ushay </strong>(You — Shay). Ushay values work-life balance, which he manages by reading books, watching true-crime shows, and trying new cafes. He has all the qualities that engineers admire. He is a good storyteller with exceptional writing skills. He is always on the rails, searching for rubies (as he is an RoR enthusiast). This easy-going, passionate person gets along with everybody and looks forward to traveling the world.</p><p>His expertise is in <strong><em>Ruby on Rails, Vue.Js, and GraphQL.</em></strong> His drive to take on a challenge head-on makes him a true gem and a great team player.</p><p>Here’s what<strong> Ushay</strong> says about working here at Sendoso:</p><blockquote>To name a few, Sendoso has a lot of amazing things to offer: The organization’s culture, The skills and knowledge that an individual acquires as a member of the organization, the Potential for advancement, Effective leadership, And, of course, the people. When it comes to Sendoso’s culture, I feel it extends well beyond the office atmosphere; for me, it’s about employee engagement, transparency, a clear vision and values, professional growth opportunities, and recognizing each other’s achievements.</blockquote><blockquote>Sendoso consistently offers a high-trust workplace environment for everyone, regardless of who they are or what they perform for the company. Furthermore, at Sendoso, effective leadership strengthens and promotes teamwork, cultivates a sense of the greater good, and provides direction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m6J4RT0Vq_ERYP3igN74dw.png" /></figure><p>This is <strong>Maham Masood</strong> — <em>Software Engineer II</em>. An extremely intelligent and social girl. She has been a part of everything during her two years at Sendoso. Name hackathon, sports fests, potluck, or annual tours; Maham is always ready to give it a shot.</p><p>She is a perfect company to have. She has expertise in <strong><em>Ruby on Rails, VueJS-Apollo, and GraphQL. </em></strong>Moreover, she is always passionate about managing events.</p><p>Let’s hear her out how life is at the Sendoso PK office:</p><blockquote>Sendoso has given me many learning opportunities and paved a path for me to grow in my career. For me, the culture at Sendoso stands out because of how the team always comes together to build and innovate and how the leadership is responsive to employee needs. The team here is not only great at what they do, but they also set commendable examples of collaboration, ownership, and recognition.</blockquote><blockquote>At Sendoso, I continue to learn something new every day and feel that the opportunities I have been given by my seniors, including Fawad Naeem, Nawab Iqbal, and Waqas Ali, have been a reflection of their trust in me to strive for the best. That is what motivates me to exceed expectations and keep growing everyday.</blockquote><h4><strong>The QA Clan</strong></h4><p>Have you ever met someone that was sunshine in the human form? Here’s our QA Team to brighten up your day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q3nwYjCfVMe0ie5ohVOMAA.png" /></figure><p><strong>M. Abubakr </strong>— <em>Senior Quality and Automation Engineer</em> — A big cheese in the QA team. Knowing yourself is the beginning of all wisdom, and Abubakr is the statue of self-reflection. He is never afraid to go back to the drawing board. Abubakr is a huge football fan. Reading and traveling are two of his favorite pastimes. He has been in the testing game for eight years, excelling at automation strategy and execution.</p><p>His areas of expertise include <strong><em>JavaScript, Circle CI, Jenkins, webDriver.io, Puppeteer, Cypress, Protractor, and Jmeter.</em></strong></p><p>He is a senior asset of the Sendoso fam who is optimistic about his future and aspires to be a people person.</p><p>Let’s take a look at what he’s saying:</p><blockquote>Having worked at multiple different IT firms before joining Sendoso four years ago, I can confidently say it is the best place to be at. The number of opportunities to grow both professionally and personally is enormous. Every new employee in the company is warmly welcomed aboard and quickly becomes part of the culture. At Sendoso, diversity of opinions matters.</blockquote><blockquote>All employees, regardless of their designation, are respected equally. Growth is not based on your years of experience but on your skill and courage to lead. Ownership is valued a lot, and our performances are critically analyzed. We get honest reviews from our leadership, which helps us improve ourselves and learn from others.</blockquote><blockquote>We are a well-connected family. There is always room for improvement, but this group thrives for excellence.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MXlOeAXXDv34qIwvYRK8qQ.png" /></figure><p>Meet <strong>M. Ather Naseem — </strong><em>Quality &amp; Automation Engineer II-L2</em>. He is much more pure and simple than he appears to be. Although you won’t catch him talking in a crowded room, that doesn’t mean he doesn’t enjoy chit-chatting with others. If you leave him with his friends, he will not stop talking. He will listen to you whether or not he can fix your problem. Despite being the least talkative one on the team, he can knock one’s socks off with his automation skills.</p><p>He has a lot of things to share:</p><blockquote>I have been working as a Software Test Engineer for four years. I started my career with this company because I knew Sendoso would help me to enhance my skills and gain more knowledge in the industry. And so it happened. I learned a lot from here related to my career and improved my skills. I started my career as a manual test engineer. But now, I’ve learned a lot about automation. I have learned how to operate like one team and understand and learn the release process.</blockquote><blockquote>Furthermore, I got a chance to improve my soft skills as well. The company has also given me a promotion to the next level. People are very supportive and help each other at work.</blockquote><blockquote>PK office provides me with a good and friendly environment and helps me succeed in my life and career. Sendoso is where my efforts and hard work are recognized, and opportunities and challenges are thrown at me to prove myself.</blockquote><p>Communication is everywhere, but everyone is a good listener in the QA Team. People are told not to try to please their managers. Instead, try to help Sendoso grow. It’s fine if you disagree with your manager. It’s never a good idea to keep something hidden. <em>“I know you disagree, but I’m going to do X because I believe it is a fruitful solution.”</em> Everyone can talk to their boss openly.</p><h4><strong>Design Team</strong></h4><p>Not only does the engineering side have the OGs, but we have an incredible <strong>design team</strong> as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2bAbej1x4qZr3dINUV1KA.png" /></figure><p>This is <strong>Saleh Riaz Qureshi — </strong><em>Senior Product Designer </em>at Sendoso. Interacting with design ever since he was 12 years old and having 7+ years of experience in B2B and B2C SaaS, Saleh has helped products reach their maximum potential. Currently, he is reducing the UX debt for the financial platform of Sendoso and would love to be an advocate of innovation at Sendoso. He consistently tries to give back to the design ecosystem and community to set others up for success. His favorite functional areas are <em>interaction design</em> and <em>research</em>. He has been working to humanize the design so the next billion users can quickly adapt to technology. <strong>Saleh</strong> is always as keen as mustard to try new things.</p><p>From Saleh’s perspective, here’s how life is at the Sendoso PK office:</p><blockquote>I have been working on B2B and B2C SaaS products for more than seven years. Working in the capacity of a <strong>Senior Product Designer</strong> and primarily owning the design for Financial Platform. What’s great about working here at Sendoso is the people themselves. Sendoso is Sendoso because of the people. The high maturity level of empathy is really the key since we all contribute to the core values of <strong>One Team</strong> and <strong>Relationships Matter.</strong> Design at Sendoso is a team of multi-disciplinary designers covering all verticals that enable every individual collaborator to learn from one another and grow together. The leaders at Sendoso have a growth and collaborative mindset, and the top-down level of trust ensures that everyone is working in a comfortable environment rather than in silos.</blockquote><blockquote>Sendoso allows me to experiment and think out of the box while developing crucial solutions for innovation. This keeps me curious and allows me to take the initiative and gets the work done. This constant growth mindset has provoked me to explore the different realms of design, so I spent my time learning and giving back to the community. I mentor students and bridge the gap between technology and people. This is why I like to call myself a tech therapist for the conversation between humans and machines. I am also working on personal products that contribute to my ambitious goal of humanizing design and making unconventional design solutions that break the grid but are still easier to learn for the users.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJsXQgyFHMRxDp9HilaI7g.png" /></figure><p>Meet <strong>Adil Mahmood </strong>— <em>Senior Product Designer. </em>Adil has made his name because of his incredible work. He has been in the design field for six years and has been rocking ever since. He is well aware of when to fish or cut bait. If you need help, call on Adil, he can come at the drop of a hat. He is a cartoon-freak, name <strong>Dexter</strong> and he will be above the grounds.</p><p>If you are his friend, you will get the privilege of many cool things, as Adil likes to paint the town red after every sprint. He is a decent package of awesomeness. He has been a part of the Sendoso family for almost three years and is excited to share his experience.</p><p>Let’s hear him out:</p><blockquote>Sendoso is a place where you get the opportunity to grow as the product evolves. The company shows that they care about its employees. The team members are very kind and supportive. Great culture, People here are fun to work with, office with an enjoyable &amp; calming environment. Sendoso is a very diverse company; not only are the people working from different countries, but we also have all the skin colors, and Sendoso provides equality to them very well. Even in Pakistan, we have people working from different provinces and cities. Gender parity is also a plus point.</blockquote><blockquote>I have learned about so many new practices, processes, tools, and terms that I wasn’t aware of. And since the product is growing from a startup to a unicorn, we were able to see different phases of how a large company is formed. Therefore, learning is a solid reason to be working here.</blockquote><p>There’s one unique thing about our design team worth mentioning; when one employee brought lunch from home while the others went out to eat, the design team came together and brought lunch back to the office so that the whole team could eat together. Because <em>inclusivity</em> is essential for the PK team, little things can significantly impact company culture.</p><p>At the PK office, employees have mutual respect and appreciation. They have faith that their co-workers and managers are looking out for their best interests and are transparent with good and bad news.</p><p>Honest and open communication is one of the main pillars of the PK Team. Everyone here treats people with respect, independent of their status and disagreement. We encourage employees to share their ideas. Everyone listens well instead of reacting fast so they can better understand.</p><p>PK team culture encourages employees to make tough decisions without agonizing and take smart risks. The cherry on top is that everyone maintains calm poise in stressful situations, no one adds fuel to the fire, and no one witch hunts.</p><p><strong><em>Wanna know more about our dream team? Stay tuned for the next blog post!!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0c2498189e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/life-at-sendoso-ft-pk-office-c0c2498189e5">Life at Sendoso — ft. PK Office</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Effective Incident Escalation Process of Sendoso]]></title>
            <link>https://medium.com/sendoso-engineering/an-effective-incident-escalation-process-of-sendoso-18aa67bf0a4f?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/18aa67bf0a4f</guid>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[escalation-management]]></category>
            <category><![CDATA[incident-management]]></category>
            <category><![CDATA[sendoso]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Tue, 19 Apr 2022 08:36:07 GMT</pubDate>
            <atom:updated>2022-04-19T08:46:42.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2L-ykqMTGTRCIYZ59JgBaw.png" /></figure><p>Sendoso, the most widely used Sending Platform, helps companies stand out by giving them new ways to engage with customers throughout the buyer’s journey. Our customers use the product for a number of reasons, including closing a deal, celebrating an employee’s birthday, or reaching a new customer. Our goal is to continuously improve how we operate so that we can always deliver a reliable and resilient product for our customers.</p><p>In today’s evolving tech world, service providers are expected to be available 24/7. When something goes wrong, team members must respond quickly and restore service, whether an outage or a broken feature. This process is known as <strong>incident management</strong>, and it is a constant and complex challenge for both large and small businesses.</p><p>Escalation entails more than simply assigning a problem to an internal team or a third-party provider. It’s all about efficiently re-igniting a well-oiled machine that coordinates cross-team and cross-functional efforts. Sounds complex? In an easier way, It brings structure, order, and additional resources to situations that could otherwise cause significant business damage.</p><p>Let’s take a closer look at how the world’s leading sending platform handles incident escalation.</p><h3>Incident Management at Sendoso</h3><p>Sendoso has a streamlined incident escalation process that assures a coordinated and well-orchestrated effort, resulting in quickly addressing critical issues and ensuring unresolved issues do not linger, as we believe that traditional methods of responding to critical incidents are no longer adequate in today’s increasingly fast-paced business environment.</p><p><strong>Practical Escalation Policy</strong></p><p>We’ve developed policies to effectively respond to major incidents. At a high level, the process consists of distinct stages. During each stage, the team works toward specific objectives to efficiently leverage available resources.</p><p>The following stages are:</p><ul><li><strong>Preparation: </strong>Resourcing, tooling, and training allow for an efficient response that happens before the incident.</li><li><strong>Detection &amp; Analysis:</strong> Detection of the issue and decision whether or not it is an incident so that the team can respond and communicate to impacted stakeholders. The goal is to determine the thresholding of what’s normal and what is not normal.</li><li><strong>Containment: </strong>Short-term recovery and prevention of further damage or reduction of the impact.</li><li><strong>Eradication:</strong> Cleanup may be necessary depending on the incident.</li><li><strong>Recovery:</strong> Restoration of systems to operational status and achieving a better state than before, prior to the incident.</li><li><strong>Postmortem: </strong>Document what happened, why it happened, and what we can do to keep it from happening again.</li></ul><p><strong>Production Review</strong></p><p>A periodical meeting is held at the end of an on-call shift, usually, once a week, also known as the on-call review or production review. The meeting aims to increase visibility and allow everyone to be aware of the main production issues and how they affect the bigger picture. It also clarifies how each team’s services and code fit and affect the larger service architecture and customer experience, even if indirectly. As a result, each team will get a better picture of how their work may affect other teams or how they might be affected by other teams. It helps prioritize which issues should be resolved first and helps align teams to work together towards the same goal. Looking at what can be improved and creating action items systematically results in quality improvements across the board.</p><p><strong>Tracking Incidents</strong></p><p>Every incident is documented as a Jira ticket, with a follow-up issue to track postmortem completion. In most cases, a support engineer creates an incident issue in response to a customer ticket or when a developer recognizes a monitoring alert as an incident. If someone is concerned about something, we encourage them to create a ticket rather than wait for it to escalate.</p><p><strong>Staffing and Prioritizing</strong></p><p>We have team members involved in the incident response process. Before, every ticket would be investigated by Support Engineering. Now, we have an App engineer on-call who is always on the ball and each team has an on-call schedule for their team.</p><p>App and Sub-Team On Call members prioritize triaging issues that come in through our monitoring systems. If there is a high volume of high-priority tickets, they can work with folks on-call or ask for additional resources from their team.</p><p>Support engineering can escalate issues to App and Sub-Team on-call members. Having clear owners of incoming requests means that we are better equipped to meet the Service Level Agreement (SLA).</p><p><strong>Preventing Issues from Being Introduced</strong></p><p>To keep issues from being introduced, we have assigned code owners which are team members who know a certain area of the code really well and are automatically assigned as reviewers.</p><p>Some teams have even proactively created alerts for issues that we want to avoid customers experiencing — like slow loading times or gift card outages. These proactive changes give us confidence that we are improving support for our customers by addressing the issues that they face and foreseeing issues that may occur.</p><p><strong>Trace Incident Metrics</strong></p><p>We use Mean Time To Respond and Mean Time To Recover (MTTR) as our metrics within the incident response, surfacing team, and summary reporting that help dive into operational issues. We are able to do this by building a data warehouse that centralizes the data from our development and monitoring tools.</p><p><strong>Organized Documentation</strong></p><p>What happened, why it happened, and what can be done to prevent it from happening again are all documented. The need for future ticket escalations is reduced by documenting key information and sharing knowledge across the team.</p><p>Furthermore, having documented processes and information makes onboarding new resources much easier. Instead of escalating tickets to more senior agents while they learn, they’ll have the information they need to resolve more tickets right away.</p><p><strong>Incident Postmortem</strong></p><p>After an incident, an incident postmortem is conducted to determine the root cause and assign actions to address it before a repeat incident occurs.</p><p>​​<strong>Implemented Tools for Proactive Monitoring</strong></p><p>Some teams don’t have tickets to go to the Support and Engineering teams. Instead, these teams create alerts in our monitoring systems for issues we foresee. There are team members on call every week to address Ad Hoc issues. Additionally, the code owners prevent us from making mistakes in the code.</p><h4><strong>Our Values</strong></h4><p>We empower our teams with general guidance in the form of values because we know that an incident management process can’t cover every possible scenario.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hMUX_ZWkfy1FQIipbZGdJQ.png" /></figure><p>We are clear that incidents are an inevitable part of providing services. Rather than playing blame games, we improve services by holding teams accountable. Also, make sure we never face the same incident twice.</p><h4><strong>Conclusion</strong></h4><p>Maintaining the big wheel of the incident escalation process was not as smooth as it seemed while reading this. Like any other company, for Sendoso it was no less than a plum job. While understanding this journey, there were many challenges to face, from unclear priorities to changing the relationship between engineering and customer support. But Sendoso has ingenious engineering teams that are always looking for creative and practical solutions to convoluted problems and are equally committed to seeing Sendoso flourish.</p><p>Sendoso has a mature and dependable method for prioritizing incidents, resolving them more quickly, and providing better service to users. The advanced escalation process has clarified ownership throughout the process, making it so that when teams face an issue, they are confident to respond effectively and recover quickly. We commit to realistic delivery timelines resulting in customer satisfaction and retention. We are confident that minor problems will not become large problems, and large problems won’t turn into catastrophes.</p><p><strong>Acknowledgment<br></strong>This blog has been made possible by the contributions of our engineers. Thank you <a href="https://www.linkedin.com/in/ana-medrano-fernandez/">Ana Medrano Fernandez</a>, <a href="https://www.linkedin.com/in/pauldavidwilson/">Paul Wilson</a>, and <a href="https://www.linkedin.com/in/taha2040qureshi/">Muhammad Taha </a>for agreeing to be interviewed and sharing the knowledge. A huge shoutout to <a href="https://www.linkedin.com/in/br3tt/">Brett Suwyn </a>and <a href="https://www.linkedin.com/in/luis-davim-a349307/">Luis Davim</a> for their responses to our queries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18aa67bf0a4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/an-effective-incident-escalation-process-of-sendoso-18aa67bf0a4f">An Effective Incident Escalation Process of Sendoso</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Team Page Redesign — Redefining Corporate Hierarchy]]></title>
            <link>https://medium.com/sendoso-engineering/team-page-redesign-redefining-corporate-hierarchy-d218778e543f?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/d218778e543f</guid>
            <category><![CDATA[vuex]]></category>
            <category><![CDATA[graphql]]></category>
            <category><![CDATA[redesign]]></category>
            <category><![CDATA[vuejs]]></category>
            <category><![CDATA[apollo]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Mon, 28 Feb 2022 16:17:24 GMT</pubDate>
            <atom:updated>2022-03-01T05:53:13.243Z</atom:updated>
            <content:encoded><![CDATA[<h3>Team Page Redesign — Redefining Corporate Hierarchy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QsSlgi0ot3D-mteZ29uvJw.png" /></figure><p>Sendoso, the leading Sending Platform, helps companies stand out by giving them new ways to engage with customers throughout the buyer’s journey.</p><p>The <em>Teams</em> Page in the Sendoso platform is the go-to place for customers who want to manage their teams, users within the teams, and the level of access for said teams and users.</p><p>The<em> Team Page Redesign</em> project has revamped the Teams Page in its entirety. It now features an improved customer experience, with easier navigation, reduced load times, and better performance.</p><h4>Importance of the Team Page in the Sendoso Platform</h4><p>The Team Page plays a significant role in the platform, providing the following functionality:</p><ul><li>Managing teams and departments</li><li>Managing users and their access to sections of the application</li><li>User roles and permissions</li><li>Invitation management</li><li>Team-based gift management</li><li>Managing team settings</li></ul><p>As important as this page is, its past form had a complex User Interface (UI) and frequent performance issues.</p><p>To address these issues, we envisioned <em>Team Page Redesign</em>.</p><h3>Existing Problems</h3><p>One of the primary reasons for the redesign was to improve the platform’s limited ability to effectively manage a user’s access to different features.</p><p>Previously, the Sendoso platform had a single corporate hierarchy. At root level was a team, composed of multiple team groups. Each team group had members, and each of those members had a specific role assigned.</p><p>There were two specific roles to choose from:</p><ol><li><strong>Manager:</strong> A user with this role had <em>read/write access</em> to all financial and inventory management activities, and <em>read</em> access to team activities.</li><li><strong>Regular: </strong>A user with this role had<em> read access</em> to their own activities and the ability to send gifts.</li></ol><p>However, as the platform expanded and began to serve larger enterprises, the system outgrew its original user management design, resulting in several issues:</p><ol><li><strong><em>Security Risks</em>:</strong><br>Manager users had access to perform various Create/Read/Update/Delete (CRUD) operations on the platform, leading to potential security risks.</li><li><strong><em>Binary roles with little checks</em>:</strong><br>The system gave Manager users complete access to move funds across departments and users while giving minimal access to non-manager users.</li><li><strong><em>Multiple customer instances</em>:<br></strong>Since there was a lack of enhanced hierarchical levels, multiple instances were needed for a single customer to manage numerous organizations.</li><li><strong><em>Inability to Scale</em>:</strong><br>The model did not allow Sendoso to onboard hundreds of users under a single parent organization. Onboarding was challenging if users belonged to various teams under different departments and needed to be consolidated under a single parent corporation.</li><li><strong><em>No distinction between Manager and Owner:</em><br></strong>There was a lack of separation between Manager users and Owners since a Manager user could perform all the actions attributed to an Owner.</li></ol><p>To resolve these problems and accommodate growing customer needs, Sendoso needed to expand its hierarchy, and add new roles and permissions.</p><h3>Design and Goals</h3><p>With the new Teams Page, we built the foundation for redefined hierarchical levels and scalable user management, which achieves the following goals:</p><ol><li><strong><em>Platform Security</em>: </strong><br>Makes the system more secure by limiting user access based on their role</li><li><strong><em>Financial Security</em>:</strong><br>Increases checks and balances within the system to reduce chances of fraud</li><li><strong><em>Scalability</em>:</strong><br>Builds the foundation for scalable user management that is compatible with the needs of enterprise-level customers</li><li><strong><em>Customized User Experience</em>:</strong><br>Allows users to perform all desired actions on the platform without any friction</li></ol><p>We also devised a new authorization system with the following features:</p><p><strong>Custom Roles and Permissions</strong></p><p>In addition to <em>Manager</em> and <em>Regular</em> roles, there are two new default roles, which support a greater range of permissions. The four default roles are now:</p><ul><li><strong>Team Admin</strong> — previously the <em>Manager</em> role</li><li><strong>Department Admin</strong></li><li><strong>Manager</strong></li><li><strong>Sender</strong> — previously the <em>Regular</em> role</li></ul><p>In case of a scenario where the user wants an access level that does not map to any of the above roles, they can now create a custom role, which is a <br> combination of<em> </em>a list of <strong><em>Permissions</em></strong> and <strong><em>Scope</em>.</strong></p><ul><li><strong><em>Permission</em></strong> describes a specific action a user can perform</li><li><strong><em>Scope</em></strong> describes at what level the user can perform the action</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k9BiSlsQ6ZyBRkVXJ3r_Ng.png" /></figure><p><strong>Expanded levels of Hierarchy</strong></p><p>There are new hierarchical levels whereby a parent organization can exist with child organizations as subsidiaries. This allows a customer to manage multiple instances of Sendoso for different business units belonging to the same parent organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*3vUd3d_X8AFkvzzL1TOB6w.png" /></figure><p>Each child organization can have multiple departments. Each department can have different teams, and in these teams, team members can have specific permission sets assigned to them.</p><p>Finally, there are Admin users with the highest read/write access level at every level.</p><p><strong>User Hierarchy: Old vs New</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*znqobXwm2i38P2CSGycj3w.png" /><figcaption><strong>Old Hierarchy</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*dnMjW60bhX2QfmqAlUjvDg.png" /><figcaption><strong>New Hierarchy</strong></figcaption></figure><h3>Challenges and Strategies</h3><p>With the project goals set in mind, our first implementation challenge was to decide on the right technologies to transform our vision into reality.</p><p>Instead of choosing the default path and opting for technologies already in use in the platform, e.g. Grape, REST APIs, we decided to take a step back and investigate whether any modern tools would provide a better solution. We focused our investigation on GraphQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Aw_ZtuNXgH8ZN90N6zBspw.png" /></figure><p>Even though GraphQL was the talk of the town at the time, we needed to decide if it was the right long-term choice for our organization. Thus we began a process of extensive research to evaluate GraphQL and identify the best practices for its implementation.</p><p><strong>Some of the reasons why we chose GraphQL were:</strong></p><ol><li>Clients can request the minimum data needed, solving the age-old problems of <em>under-fetching</em> and <em>over-fetching</em>.</li><li><em>GraphQL-Federation</em> allows teams to grow and deploy new APIs with little resistance. This played a significant role in our decision.</li><li>The GraphiQL UI is a beneficial tool that enhances developer productivity and makes it easier to present API logic to project managers.</li><li>At the time, we also needed to start building APIs for Sendoso’s multiple UIs. GraphQL was the right choice to maximize responsiveness and reduce latency.</li></ol><p>Once GraphQL was finalized as the language to build our new APIs, our next step was to figure out the best framework to integrate those APIs with our preferred frontend framework, VueJS. From the two popular choices, <em>VueX</em> and <em>Apollo</em>, we chose Apollo because:</p><ol><li>Apollo offers declarative data querying, making it responsible for the complete request cycle without the user writing any middleware.</li><li>One of the framework’s most vital selling points was the freedom to choose a fetch policy for different endpoints to retrieve data from the server or the cache as needed.</li><li>Apollo’s extensive features and libraries seemed promising for making development convenient and enhancing performance.</li></ol><p>Retrospectively, the reason behind this feature’s success is the strength of the planning, research, and collaboration that took place before we began any implementation.</p><h3>Result and Achievements</h3><p><em>Team Page Redesign </em>was a feature that set out to improve the customer experience on the Teams Page and pave the way for using <strong><em>GraphQL-Apollo</em></strong> in the company’s future projects. That is precisely what we accomplished.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jk_PaWEmeSfv4hooNYiNBg.png" /><figcaption><strong>Then</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqK1WfMqs1abVRQLQ2uaLA.png" /><figcaption><strong>Now</strong></figcaption></figure><p>For the Teams Page, we improved performance and response time by as much as 20x by achieving the following:</p><ul><li>N+1 queries were removed from the page</li><li>Performance was improved at the database level by using techniques such as enhanced indexing and data modeling, enabling cache, etcetera.</li><li>Different sections were converted to a single go-to section for Team management</li><li>Some sections were moved from the Teams page to more relevant locations, e.g., the ‘<em>Manage Account Balance Settings</em>’ option, which previously had a view on the Teams page, was moved to the <em>Account Balance</em> tab where it’s better suited</li><li>Pagination was added in several sections where it was lacking</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUrQWqVXhdvIigI82WC6yw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cr8OD97wi4k0IjwdPg_wbw.png" /></figure><p>In addition, Sendoso customers welcomed the <em>Custom Roles and Permissions</em> feature with great enthusiasm and use it frequently to manage user permissions on an enterprise level.</p><p>As for new technologies, our experiment bore great results. Today, GraphQL-Apollo is being used company-wide to implement new features and is doing wonders to enhance performance and code quality. That, in itself, speaks volumes about the success of <em>Team Page Redesign</em>!</p><h3>Acknowledgment</h3><p><a href="https://www.linkedin.com/in/maham-masood-49753217a/">Maham Masood</a> is the brains behind this blog post. After demonstrating her skills on exciting features like Team Page Redesign, Iterable Integration, and International Address Validation, Maham is currently redesigning and enhancing Sendoso’s Notifications Infrastructure.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d218778e543f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/team-page-redesign-redefining-corporate-hierarchy-d218778e543f">Team Page Redesign — Redefining Corporate Hierarchy</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Innovation Comes Together In Sendoso Hackathon]]></title>
            <link>https://medium.com/sendoso-engineering/inside-look-innovation-comes-together-in-sendoso-hackathon-c1aa484d69fc?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/c1aa484d69fc</guid>
            <category><![CDATA[sendoso]]></category>
            <category><![CDATA[hackathons]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[diversity]]></category>
            <category><![CDATA[collaboration]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Thu, 20 Jan 2022 12:15:36 GMT</pubDate>
            <atom:updated>2022-01-28T07:14:58.998Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IpZ15ZgHf720l1xzJetuLw.png" /></figure><p>Sendoso engineers are always looking for creative and practical solutions to complex problems. Here at Sendoso, our innovative team of engineers and employees from various groups teamed up to help solve these challenges in our annual Sendoso Hackathon.</p><p>Whether it’s providing a new experience for our customers or improving internal processes with new tools, innovative solutions help the company evolve.</p><p>At the beginning of Sendoso and with only a handful of employees, every project was considered a hack. Since then, our team has grown, and we’ve come a long way in bringing solutions to the forefront. However, we’ve never forgotten the great ideas that came from our humble beginnings.</p><h4>What is a Hackathon?</h4><p>If you aren’t familiar with a hackathon, here’s a quick explanation:<strong><em> Hackathon</em></strong> is a marathon event in which participants design ‘hacks,’ which result in practical and creative solutions to specific problems. It’s becoming increasingly common in technology, programming, and IT development to have hackathon events.</p><p>The Hackathon concept has many definitions and interpretations. Sendoso’s VP of Engineering <a href="https://www.linkedin.com/in/fawadnaeem/"><strong><em>Fawad Naeem</em></strong></a> has a simple quote to sum up the hackathon event:</p><blockquote>“Steven Johnson, an American writer, says ‘If you look at history, innovation does not come just from giving people incentives; it comes from creating environments where their ideas can connect.’ This is exactly what Sendoso Hackathon is all about.”</blockquote><h4>Innovation at Sendoso</h4><p>To continue to innovate, Sendoso organizes a yearly hackathon event for employees. The goal is to allow employees to work on ideas outside their regular day-to-day projects. Recognizing that good ideas don’t just come from those who push code, the engineering team opened up projects to the entire company.</p><p>The first Hackathon created a plethora of applied ingenuity, not just software products or features; anything from product and pitches to a new market experiment was considered a hack. The following are some notable examples from the previous Hackathon:</p><ul><li>Gmail Integration</li><li>Scheduled Sends</li><li>Test Automation</li></ul><h4>The Hackathon Experience</h4><p>Employees are placed into small groups, showcasing a wide range of skills from various teams. This helps the groups unite, and it brings vibrant ideas into practice. They are able to work as a team and bring innovation to the forefront.</p><p>The rules are simple, a panel of six judges from Sendoso’s leadership team, including CEO <a href="https://www.linkedin.com/in/rudeegraap/"><strong>Kris Rudeegraap</strong></a><strong> </strong>and CTO <a href="https://www.linkedin.com/in/qaseemshaikh/"><strong>Qaseem Shaikh</strong></a>, judge each team under the following criteria:</p><ul><li>Group participation: At least five participants per group</li><li>Innovative idea: Commit to an idea, and show off your skills</li><li>Time crunch: 72 hours — and time starts now!!</li></ul><p>This year, we had 85 participants and 32 teams across 3 continents. Out of all these ideas and groups, a tough decision was made to shortlist nine teams to compete in the Grand Finale Stage. The teams were judged on the following five categories:</p><ol><li>Best Customer Experience</li><li>Best Employee Experience</li><li>Best Engineer Experience</li><li>Most Likely to Change the Product Experience</li><li>People’s Choice</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QeUlvmaVRTh2tyy-dCmfqA.png" /></figure><p>For 72 hours, the teams went through a myriad of emotions. Laughter turned to weeping. Arguments flared. But amid all these emotions, the teams united to create features that we wanted to see in our products. Some of the happy hacks on this three-day event were:</p><ul><li>Sending NFTs</li><li>Fix Flaky / Slowest / Most failed Specs</li><li>Improve Speed of Local Development</li><li>Google Calendar Plugin</li><li>Zoom Integration</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x6EAVjQOADs1TxltH_4Q_w.png" /><figcaption>Eventually, It made us smile so BIG!!</figcaption></figure><h4>Sendoso’s Hackathon Winners Are…</h4><p>The final groups were narrowed down to nine. The innovative ideas were presented at the Grand Finale Stage. The winners are:</p><ul><li><strong>Best Customer Experience: <em>Osama Inayat and Ayaz Ahmad Tarar</em></strong> for <em>Pulling Together an easy to use Zoom Integration</em></li><li><strong>Best Employee Experience: <em>Junaid and Syed Fahad Ali</em></strong> for<em> Building a CSM Dashboard</em></li><li><strong>Best Engineer Experience: <em>Muhammad Raza, Mubeen Tahir, GS Shahid, Fahad Idrees, and Hamza Irfan</em></strong> for their work <em>Fixing Slow and Flaky Specs</em></li><li><strong>Most likely to change the Product Roadmap: <em>Joe Soboleski, Grace Donegan, Martyn Duffy, Isidro, and Maude Collin</em>s</strong> for their work <em>Gamifying The Sending Experience</em></li><li><strong>People’s Choice: <em>Joel, Marcel, Navid, Fawad, and Grant Shively</em></strong> for their work <em>Improving our Local Development Speed</em></li></ul><h4>It’s More Than a Hackathon</h4><p>We discovered that hackathons provide networking opportunities, resulting in increased collaboration, a better understanding of the product, and solving future problems. These events encourage developers at Sendoso to try out unique ways to approach a problem and engage all the other aspects of a business. They could pitch in with their creativity and share ideas with peers, making them an ideal testbed for new tech concepts.</p><p>Hackathons at Sendoso show how to design innovation by associating cross-functional teams together. For Sendoso, the Hackathon is more than just a competition; it is an opportunity to nurture team bonding and establish new relationships with people we wouldn’t usually have the chance to work with but who are equally committed to seeing Sendoso thrive. At the same time, team morale and collaboration improved. And diversity is no exception.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1aa484d69fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/inside-look-innovation-comes-together-in-sendoso-hackathon-c1aa484d69fc">Innovation Comes Together In Sendoso Hackathon</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Sendoso Handles Addresses Globally]]></title>
            <link>https://medium.com/sendoso-engineering/how-sendoso-handles-addresses-globally-dc2631a7bdc5?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/dc2631a7bdc5</guid>
            <category><![CDATA[sending]]></category>
            <category><![CDATA[information]]></category>
            <category><![CDATA[metadata]]></category>
            <category><![CDATA[sendoso]]></category>
            <category><![CDATA[address]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Mon, 20 Dec 2021 09:38:49 GMT</pubDate>
            <atom:updated>2021-12-20T10:08:11.804Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BxPmLkbqp-lO5OnTNCw0yw.png" /></figure><p><a href="https://sendoso.com/"><strong><em>Sendoso</em></strong></a> is the leading Sending Platform that delivers direct mail, personalized gifts, eGifts, and other Physical Impressions™ at scale.</p><p>For every <em>send</em> to deliver the impact, every order needs to be delivered to the recipient. And for that, an accurate recipient address is required.</p><p><strong>Address</strong></p><p>Address is an essential information to deliver gifts, that’s obvious. What’s not obvious is that collecting address information accurately is not easy. There are nitty-gritty details that you need to highlight when delivering internationally. Addresses can be a combination of house number, street/road, city, zip/postal code and country.</p><p>A standardized address form generally consists of the following fields:</p><ul><li>Full name</li><li>Address</li><li>Address 1</li><li>Address 2</li><li>City</li><li>Zip Code</li><li>State/Province</li><li>Country</li></ul><p>Occasionally, an intersection or nearby landmark name may be required.</p><p><strong>Challenges for verifying addresses around the globe</strong></p><p>Many parts of <em>address </em>are not standard across the world, For example:</p><p>Some fields are not applicable in certain countries:</p><ul><li>Few countries like <strong>(Anguilla, Aruba, Bermuda</strong>, <strong>Israel,</strong> <strong>Falklands)</strong> don’t have a sub-national level state or region.</li><li><strong>Gibraltar, Cayman Islands, Macau, Singapore</strong> do not require <em>city </em>to identify an individual’s address.</li></ul><p>Fields are labeled differently across the boundaries:</p><ul><li><strong>Region </strong>is called “State” in the United States, “Provinces” in Canada, and “District” in the Cayman Islands.</li><li><strong>Zip</strong> is named as “ZIP code” in the United States, “Postal code” in Pakistan, or “Postcode” in Jersey</li><li><strong>City</strong> is called “Emirate” in the UAE and “Locality” in Barbados.</li></ul><p>The fields have different formats or validity requirements:</p><ul><li>Different countries use different <strong>Zip/Postal Code formats</strong>. Canada has “A1A 1A1,” and the United States has “99999” as zip format</li></ul><p>Multiple names, abbreviations and codes are used for the same country. e.g. PRC = People’s Republic of China = China (ISO codes: CN, CHN)</p><ul><li>Similarly, people routinely enter country = England when they actually mean the United Kingdom, whose full name is The United Kingdom of Great Britain and Northern Ireland, and ISO codes are GB and GBR.</li><li>A similar issue is differences in translations, where data entered through some integrations used different vowels while referring to the same region; e.g. Québec vs Quebec.</li></ul><p><strong>Solution</strong></p><p>Provide users with localized forms everywhere. To achieve this, we needed metadata on address formats for all countries. We built this database through external sources. This data is needed in various experiences within Sendoso. To make it centrally available, we wrapped it in a microservice such that any application can call the service to retrieve address metadata and render the right address form.</p><p>Here are examples of country-specific forms that can be rendered by reading schema from the Address Service.</p><p>United States</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LsFOqhKR78mLsGWPnDy2Sw.png" /></figure><p>United Kingdom</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MqTwbC-AQyQp7taJQsR5Og.png" /></figure><p><strong>Architecture</strong></p><p>The team decided to create a new microservice to avoid adding more complexity to the existing application. This also helped us quickly iterate without a dependency on the core application that has its own release cycle. The new Address Service will eventually become the single source of truth for providing country-specific address forms and validating address data at Sendoso.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jc3tQAhiyn_P4W_ckIdlSA.png" /></figure><p>To make sure we have accurate addresses and tracking, a periodic sync and version control were added. This makes sure any changes in the external API (e.g., a bugfix or creation of a new country, etc.) will be reflected in our system.</p><p>As we built this service, the team realized that a lot of times data, especially country data was incomplete, obsolete or had discrepancies. For example, one API was returning an obsolete zip code format for Israel or marked <em>city</em> as optional for Pakistan — while <em>city</em> is mandatory for correct package routing in Pakistan.</p><p>To work around some of these issues, we added a feature to allow Sendoso admins to customize their fields by adding new ones and changing the settings (show/hide and required/optional) of existing fields. In addition, this feature assisted us in adding otherwise non-address fields to the address form, which are required in some countries e.g., India, Brazil and Russia require the recipient’s <em>tax ID</em> to receive a foreign package.</p><p><strong>Admin User Interface</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fSgH9LPyQL1jnKJwI5vJ8w.png" /></figure><p><strong>Storing Address Metadata</strong></p><p>For each country in our database, we parsed the external API response and saved all of the data as a JSON schema. To store the country’s address fields (i.e. the address metadata), we chose the JSON-based approach instead of the standard one field per database column approach. This approach allowed us to dynamically add more fields without having to run database schema updates. An example of one country’s configuration field is shown below.</p><p>The initially bootstrapped data (from the third-party API) is stored in the <strong><em>`base_fields`</em></strong> key, while country-specific admin’s overrides for labels or fields visibility etc. are stored as <strong><em>`admin_overrides`</em></strong>. Any additional country-specific fields added by the admin are stored in <strong><em>`admin_defined_fields`</em></strong>.</p><p>Some fields, for which we needed to query the table, were created as database columns for efficient queries (e.g. country name, country code, country’s validity for delivering from Sendoso, etc.)</p><p><strong>How Address Service Stores Data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-84PsIqCSFOPQoTSgs2ng.png" /></figure><p>Similarly, we chose the JSON format to cater dynamically defined fields for the address data. Any fields which are considered useful for querying or reporting (country, city, state, zip codes etc.) were also added as columns in the address data table.</p><p><strong>Sample Address Row</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MxQaaY1sIftjQM52bQExQ.png" /></figure><p><strong>Accepting multiple names for the same region and country</strong></p><p>Occasionally a country has more than one acceptable or commonly used aliases. When a user enters a country directly on our platform, we provide a list of options to select from.</p><p>However, we don’t have the option to select the country from the dropdown in case of integration scenarios where the input is added using a different application and sent to the address service for validation. As ISO codes aren’t always used, we match the provided input with a list of countries’ names, codes and aliases i.e. UK, GB, GBR, the United Kingdom, Great Britain, will all map to the United Kingdom.</p><p>While using auto-suggestion from a different third-party API, we encountered scenarios where the regions had varying names during the translation process when compared to the list of region names in our service’s metadata.</p><p>For autosuggest/autofill, the third-party API also provided a country or region code. Since ISO codes are standard, and far less prone to any abbreviation or transliteration variation, we used the code to compare against options of regions in our metadata.</p><p><strong>Personal Identifiable Information Protection</strong></p><p>Sending requires the inclusion of contact information as mentioned above. Name, Street, Block, ZIP/Postal Code, Email Address, and Phone Number are all common details. When it comes to delivering something or getting in touch with someone, these minute details are essential.</p><p>Address-Service only stores the address metadata that is needed to render the country-specific form and validate the address.</p><p>Adheres to the privacy regulations, Sendoso ensures that the recipient’s information needed for sending is only kept for a short time. The required information is deleted from the system once the package has been delivered as nothing is more important for Sendoso than the client’s privacy.</p><p><strong>Achievements</strong></p><ul><li>Improved customer (and recipient) experience for international sends.</li><li>Reduced volume of paused, delayed and returned orders due to improved address accuracy.</li><li>Reduced shipping costs due to the reduction in address correction fees from carriers.</li></ul><p><strong>Next Action Items</strong></p><p>We’ve handled various name variations for <em>Country Name</em>, but we haven’t maintained a similarly flexible mapping for <em>Region Names</em>. It’s something we’ll think about in the future.</p><p>Furthermore, smoothing the way by facilitating Customer Support and Application Admins to edit this synonym list through our admin portal, so they can provide such <em>Region Name</em> translations without involving the engineering team.</p><p>By providing the complete list of schema, we can reduce the number of API calls. We’ll render the form on the first call. When the user selects a different country from the drop-down, we’ll use the cached schema instead of making a network call to the Address Service.</p><p>We are currently working on extending this service by introducing address validation by calling the <em>Easypost</em> service to validate addresses. We will add other vendors to improve address validation even further.</p><p><strong>Acknowledgment</strong></p><p>This blog is driven by <a href="http://www.linkedin.com/in/naveed-fiaz">Muhammad Naveed Fiaz</a> Software Engineer II at Sendoso under the leadership of <a href="https://www.linkedin.com/in/iqbalasad/">Nawab Iqbal</a> Senior Engineer and Technical Leader at Sendoso along with the mutual efforts of <a href="https://www.linkedin.com/in/ehab-babar-8139b7197/">Ehab Babar</a> Quality and Automation Engineer, <a href="https://www.linkedin.com/in/maham-masood-49753217a/">Maham Masood</a> Software Engineer II, and <a href="https://www.linkedin.com/in/rabiya-adnan-0a763817a/">Rabiya Adnan</a> Software Engineer II. The amazing Notifications Team of Sendoso is now focusing on enhancing the emails deliverability.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc2631a7bdc5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/how-sendoso-handles-addresses-globally-dc2631a7bdc5">How Sendoso Handles Addresses Globally</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQA: The Secret Sauce to Sendoso’s Success]]></title>
            <link>https://medium.com/sendoso-engineering/sqa-the-secret-sauce-to-sendosos-success-7f93b8c79c80?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/7f93b8c79c80</guid>
            <category><![CDATA[automation-testing]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[sendoso]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-quality-testing]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Mon, 15 Nov 2021 10:28:58 GMT</pubDate>
            <atom:updated>2022-04-22T10:12:40.659Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XU9rUGXIKHEKQytuoz6jng.png" /></figure><p>Software Quality Assurance is a set of methods and activities to ensure that the developed software meets all requirements. It is a continuous process of verification and validation of all functionalities, which ensures the delivery of a high-quality product to the end-user. Each phase of the SQA life cycle determines the platform’s performance and usability. Also, it certifies all the technical and business requirements with an ability to scale.</p><h3><strong>Test Automation and its importance in Sendoso</strong></h3><p>In Software Quality Assurance, automation testing is essential for the Software Development Life-Cycle (SDLC). Automation testing executes test scenarios at various application levels using different tools to help ensure a quick and reliable testing solution.</p><p>An extremely dynamic platform like Sendoso requires an effective testing strategy that allows it to quickly scale and innovate itself to keep up with the challenging demands of consumers without any issues. To achieve this, Automation Testing becomes a key element in any test strategy.</p><h3>Early Challenges</h3><p>In the early days at Sendoso, Quality Assurance (QA) processes were heavily reliant on manual testing because of a small number of SQA engineers and a limited consumer base. Sendoso has a very dynamic business model with multiple releases every day and SQA Engineers must execute bulk test scenarios on a daily basis to make sure that the platform is production-ready.</p><p>Some of the challenges at the start were as follows:</p><ul><li>Manual execution of repetitive test scenarios was time-consuming for daily releases</li><li>Dependency on full-stack engineers to define testing scope due to limited involvement of QA in project planning</li><li>Delays in release deployment caused by inaccurate estimations of manual testing efforts</li><li>Consistent human errors while testing manually</li><li>Inconclusive and ambiguous reporting of bugs</li><li>Lack of documentation</li></ul><h3>Our Main Strategy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*psMefGyg3uFYelL3bI1pcA.png" /></figure><p>While manual testing is here to stay, we understand the importance of designing and following a structured strategy for test automation. Creating test strategies means that teams are better informed on how to go about the testing.</p><p><strong>Single Source of Truth</strong></p><p><a href="https://www.gurock.com/testrail/docs/user-guide/getting-started/walkthrough">TestRail</a>, a test case management tool, was introduced for documentation, providing engineers with a single source of truth for all domain-relevant documentation. Anyone within the company can review product-related test cases using TestRail instead of navigating through multiple platforms. TestRail also assisted with the onboarding of new engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KArNj1UwbnxgqL2l20DL8A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uyaEbsnWON9ciUmIq0-8kw.png" /></figure><p><strong>Identifying the Right Tool</strong></p><p>Initially, each team started with a separate automation tool but quickly identified that using different tools implied different capabilities. To overcome this, SQA engineers researched a variety of tools including Selenium, Cypress, WebDriver.io, Puppeteer and also explored numerous frameworks like Jasmine and Cucumber. After carefully analyzing the requirements and limitations of Sendoso’s platform, the SQA engineers identified <a href="https://developers.google.com/web/tools/puppeteer">Puppeteer</a> as the Automation tool that best manages all expectations.</p><p>Sendoso Platform has many integrations which require API callbacks and multi-domain redirections. Puppeteer is a Node library that provides a high-level API to control Chromium or Chrome over the Devtools protocol. It uses a hybrid framework, i.e., Behavior-Driven Development (BDD) and Data-Driven automation framework with Cucumber and JEST with Javascript as the scripting language. Puppeteer is a fast, consistent, and reliable automation tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6fV1IQ-vGe_ohF-nMUViQ.png" /></figure><p><strong>Deciding What To Automate</strong></p><p>Through organized documentation that all the relevant engineers review, we were able to identify and plan efficiently which tests needed to be part of the automated smoke suite. This especially made it easier to manage the daily release testing process. Moreover, it became clear what:</p><ul><li>can be automated</li><li>cannot be automated</li><li>should not be automated</li></ul><p>This helped us in automating the major features that needed to be tested recurringly to ensure long-term sustainability. Major features can be defined as all those functionalities that allow the user to successfully navigate through most of the platform. This way any feature enhancement or new functionality request affecting the rest of the platform can be integrated in a risk-free manner.</p><p>We understand the limitations that automation has and try to bridge that gap through carefully planned manual testing. Functionalities that are not part of the major flow of the platform or suffer no impact from any enhancement throughout the application should not be part of the smoke test suite. Similarly, email verification that requires third-party navigation and hence, cannot be recurrently tested.</p><p><strong>Setting up the Right Environment</strong></p><p>When performing automation testing, a stable environment is needed that is not rapidly changing and has the same capability as the production environment. This is useful when identifying any performance-related issues on our platform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TQEIJMI4NcW04o451CoK9w.png" /></figure><p><strong>Scripting and PR Reviews</strong></p><p>To enhance and scale our Automation Test Suite, each Github PR is reviewed by SQA engineers as well as full-stack engineers. We attempt to follow the standard review practices with valuable feedback. This allows us to refactor and incorporate any changes in functionality to improve the reliability of the automation test suite consistently. The testing scope also becomes clear for every engineer by having SQA Engineers involved in the Pull Request process. SQAs help Software Engineers learn about how the larger product works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*05b5BHN45Y0ljD4nIDxD8A.png" /></figure><p><strong>Execution on CircleCI (Continuous Integration)</strong></p><p>Once the PRs are reviewed, approved, and merged, we can use Continuous Integration (CI) to ensure that the automation suite is executed with the daily release deployment. We cannot execute an exhaustive test script on each deployment as it takes a lot of time to test each functionality in detail; instead we add part of the script that is called smoke suite with the release process. Repetitive manual effort is no longer required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DVy0xov14SY-9ma0yItouw.jpeg" /></figure><p><strong>Bug Reporting</strong></p><p>Bug reporting has improved massively as well. Through our automated test execution on CI, the issues are quickly identified and reported as bugs with all the relevant information added automatically. We follow a standardized bug reporting template and assign it to the relevant engineer on JIRA. The engineer reviews the bug and can track all the required information with a complete and precise automated test report.<strong> </strong>If the engineer understands the issue and fixes it, will mark it as <strong>‘ready to test.’</strong> Once the bug is fixed, the SQA engineer verifies the fix before deploying it to any shared environment. After this, the smoke suite will execute again and ensure that the issue is resolved and has not affected any other part of the platform. Finally, the engineer marks the bug as<strong> ‘resolved or fixed.’</strong></p><p><strong>Hackathon</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M26VrETXlUFYQYAzxwkISQ.png" /></figure><p>Engineers at Sendoso always look to conquer challenging problems with creative and practical approaches. For this purpose, Sendoso organized a Hackathon event that provided every engineer with a forum to express their creativity that ignites innovation. It provided the engineers with a rigorous but extremely valuable experience and a steep learning curve to enhance their skills and get extra valuable exposure.</p><p>QA engineers participated in Hackathon as well, with the aim of fast-tracking automation testing into the project life-cycle. They researched different tools and tried various challenging scenarios to better understand the testing capacity and reliability of the tool. This played a vital role in redefining our testing strategy.</p><h3>Contributions of Automation Testing at Sendoso</h3><p>The impact of automation, as anticipated, was positively acknowledged at multiple levels in the company’s QA processes. Some of the key contributions of Automation at Sendoso were:</p><ol><li>A drop in recurring issues</li><li>Increased productivity translated from platform stability</li><li>Achieved faster and reliable testing</li><li>Boosted ROI on profitability, enhanced customer experience and satisfaction</li><li>Huge drop in the manual testing effort</li><li>Rapid test execution and project delivery</li><li>Improved confidence in the platform’s capabilities</li><li>Received swift feedback on failures, in line with the “fail fast, fail early” principle</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*B30_FON8i81U9MVXCplt5w.jpeg" /><figcaption>The chart shows the number of issues based on the <strong>Created</strong> field</figcaption></figure><p>The test plans are developed alongside regular roadmap feature development then executed automatically by continuous software integration tools. At Sendoso, automation testing promotes lean QA team size and enables them to focus on more sensitive features.</p><h3>Way Forward</h3><p>There could be more benefits of automation testing as Sendoso is progressing immensely around it. Being mindful that human error is bound to occur and can be eliminated only by investing in test automation, Sendoso is quickly on its way to maximize coverage for better reliability on test automation.</p><p>Sendoso actively carries out continuous research to identify better strategies and test tools. This pursuit also reinforced the fact that clear and accurate reporting is key. Engineers firmly believe in comprehensive reporting as it assists in drastically improving processes. We believe in our ability to create a robust and efficient test plan to help detect bugs early on in the software development life cycle.</p><h3>Acknowledgments</h3><p>This blog is driven by <a href="https://www.linkedin.com/in/muhammed-abubakr-21650887/">Muhammed Abubakr</a>, Senior Test Automation Engineer at Sendoso. He has been in the testing game for seven years, excelling at automation strategy and execution.</p><p>Shoutout to <a href="https://www.linkedin.com/in/afshan-shakoor-8209bb162/">Afshan Shakoor</a>, for her prodigious support related to designing, implementing, and improving the company’s quality processes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7f93b8c79c80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/sqa-the-secret-sauce-to-sendosos-success-7f93b8c79c80">SQA: The Secret Sauce to Sendoso’s Success</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Sendoso is using Kafka for Event-Driven Architecture]]></title>
            <link>https://medium.com/sendoso-engineering/how-sendoso-is-using-kafka-for-event-driven-architecture-d7ae2a519c39?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/d7ae2a519c39</guid>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[sendoso]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[event-driven-architecture]]></category>
            <dc:creator><![CDATA[Amna Malik]]></dc:creator>
            <pubDate>Mon, 13 Sep 2021 14:06:07 GMT</pubDate>
            <atom:updated>2021-11-29T10:32:19.044Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HtjN7YbOnheo5dYG5Wj6gA.png" /></figure><h3><strong>Event-Driven Architecture using Kafka</strong></h3><p>Sendoso started as a single repository and a monolith application, just like any web application. A single repository and application have benefits. Any developer can clone the repo, build, deploy, and dive into any aspect of the application without having to navigate another repository. After all these years, however, our repository has grown significantly. Build and deploy steps take a good part of the hour. The cognitive load has increased too. No developer has the capacity or time to learn the details of every scenario, nor should they be expected to do so. Daily builds and deployments to production require sign-offs from multiple teams. A single bug or unexplained behavior anywhere in the application can derail the release process, and impact every team’s progress.</p><p>The key, of course, is splitting the monolith into manageable subdomains. The goal is to have subdomains with bounded context that’s easily be understood by its domain expert. The same process can be repeated recursively as a subdomain gets increases in scope over time.</p><p>There’s a flip side to this. While engineers and test engineers benefit from service-oriented architecture, SOA means operational overhead, infrastructure, and observability costs. Without tooling and automation, SRE/Ops team will bear the brunt of deploying, monitoring, and maintaining multiple applications.</p><h3><strong>Microservices and Sendoso</strong></h3><p>Microservices are a set of services that act together to make a whole application operate. Microservices are self-reliant and need to coordinate as well, this makes development and deployment quick. So, if a developer needs to update a specific service, they will only update that service, and the rest of the services will remain the same. It makes the process faster.<br>For creating microservices, we have decided to start with the low-hanging fruit. Any new functionality which is not tightly dependent on the existing application is being moved to its service. Using a shared database in a microservice architecture is an anti-pattern. Service interfaces/APIs should be used for information exchange between different services and the core. One note of caution — introducing too many APIs will also imply coupling or incorrectly defined service boundaries. That means that code is separated but data isn’t decoupled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CqdLknE-2ZGyKStTMSliKQ.png" /></figure><h3><strong>Hello World Service and Cost Reduction</strong></h3><p>To make new service creation easy, we are templatizing services <em>a “hello world”</em> service that can be cloned, built, and deployed to an environment. The service template comes with monitoring and alerting built-in.</p><h3><strong>Confide on Kafka for Data Processing</strong></h3><p>Message-passing in a distributed system is key for scale. We decided to use Kafka as a message bus. Kafka is a distributed platform based on the publish and subscribe (pub-sub) model. This model contains two main entities, producer and consumer. As the names indicate, Producer produces the message and consumer consumes the message. Producer publishes messages to the data stream. Consumer, which is subscribed to the data stream, processes the messages. There is a data stream between producer and consumer. The data stream acts as a broker between the producer and consumer for data exchange. The exchange of data or messages between the producer and consumer happens through the broker as there is no connection between the producer and consumer.</p><p>Kafka has a solid Java Virtual Machine (JVM) based stream processing framework. It enables a computer to run Java programs as well as programs written in other languages that are also compiled to Java bytecode, which would have been an obvious first choice for a greenfield project.</p><p>A number of Sendoso applications are ruby-based. We decided to use Karafka gem for consumers and waterdrop/ruby-kafka gems for producers. This works well, with just one pod, we were able to move 1GB between producer and consumer in roughly a minute, using t3.small AWS MSK brokers.</p><h3><strong>Source of Truth</strong></h3><p>Event sourcing is an effective architectural pattern to record changes to the application state. Event sequence is important — we need changes as they were originally applied. incoming events are first persisted into Kafka and then processed by services independently. Kafka, hence, becomes our source of truth (SOT), a data source that gives a complete picture of the data object as a whole. This, however, meant dramatic changes in our core application. Our source of truth is still the core database, but we generate events in Kafka when data gets persisted. To ensure transactional behavior, we employed <a href="https://microservices.io/patterns/data/transactional-outbox.html">Transactional Outbox pattern</a>. Essentially, we <strong>a)</strong> created a new events table in the database, <strong>b)</strong> wrote event data in the same transaction when we update our SOT table. Kafka Connect is subsequently used to read this table and insert records in relevant Kafka topics. This ensures that we never have an inconsistent situation where data was inserted in the database but the event is not added to Kafka topic or vice versa. We evaluated a few connectors for sourcing data from Mysql (<a href="https://github.com/confluentinc/kafka-connect-jdbc/blob/master/src/main/java/io/confluent/connect/jdbc/JdbcSourceConnector.java">JdbcSourceConnector</a>, and Debezium). Our scenario was supported out of the box in JdbcSourceConnector, making it possible to have one event table in Mysql where different rows could be routed to a relevant topic based on the topic field.</p><h3><strong>TransactionalOutbox Table Schema:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/1*d95yq3RY8fRHakB4Ojd2Tg.png" /></figure><p>The connector picks a row and creates the event (Key, Payload) in the respective Kafka Topic. The timestamp field helps JdbcSourceConnector to track its progress.<br>The connector is configured as follows in Kafka Connect, which is deployed in Kubernetes:<br><a href="https://docs.confluent.io/kafka-connect-jdbc/current/source-connector/source_config_options.html">Notice</a> how key, topic, payload, and timestamp fields are used.</p><h3><strong>At Least Once and In Order Guarantees</strong></h3><p>Kafka Connect provides robust defaults out of the box. E.g. to ensure delivery of every event, it sets acks=all so that the producer doesn&#39;t move forward until the data is committed to at least ISR (in-sync-replicas). Kafka-connect does it by default, but we needed to do it wherever we are directly producing to <a href="https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#producerconfigs_acks">Kafka topics.</a><br>It is a good idea to set in-sync-replicas to at least 2 or preferably 3; so that Kafka will not lose data in the face of a broker crash.<br>There are few other configurations which are important to ensure in-order guarantee. E.g., setting max.in.flight.requests.per.connection=1 and following timeout values to a reasonably high value. request.timeout.ms, max.block.ms, delivery.timeout.ms.<br>In terms of multiple partitions in the topic, the object key should be used for partitioning. This ensures that all updates to the object land on the same partition and are consumed by the same consumer, hence applied in the original order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KCOrUok8HmQ3sGs7AC-Mg.png" /></figure><h3><strong>Retry and Dead Letter Queues</strong></h3><p>We also used Retry and dead letter queues, as explained <a href="https://eng.uber.com/reliable-reprocessing/">here</a>. This ensures progress if some poison or invalid event has entered the stream. However, in this case, the messages may be processed out of order, so (asynchronous) retry queues will not be suitable for scenarios where in-order delivery is important.</p><h3><strong>Kafka Observability and Operations</strong></h3><p><a href="https://github.com/obsidiandynamics/kafdrop"><em>Kafdrop</em></a> is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers. It also has topic creation and deletion functionality. Besides that, it allows us to observe consumer lag, and messages from any partition and offset.<br>While this is not a replacement for monitoring and observability, it has been beneficial to visualize what is happening on the Kafka brokers. We have configured separate Kafdrop pods for each MSK cluster.<br>For monitoring and alerting Kafka metrics, we are relying on <a href="https://github.com/newrelic/nri-prometheus">newrelic-prometheus integration</a>, to forward MSK metrics to New Relic. This has also been deployed as a Kubernetes pod.</p><h3><strong>Next In Line</strong></h3><p>This is an important step in the right direction. It also brings many new challenges. One common issue with breaking into microservices is that the contributors lose sight of the overall context. The drawbacks can be mitigated by correctly drawing the responsibility/service boundaries and adjusting the ownership as the domain evolves. Another popular option is to use orchestration services i.e. a service that talks to many other microservices. The latter microservices may not need the full picture, but the orchestration service provides the cognitive context. Similarly, as we add more services, we plan on investing in load balancing, security, circuit-breakers, and request throttling between services. A service mesh will be an important consideration at that point.</p><h3><strong>Acknowledgments</strong></h3><p>Kafka is complemented by many open-source tools and technologies. For this project, we benefited from Kafka, kafdrop, Karafka, Kafka-connect, and Kafka MySQL connector, to name a few.</p><p><a href="https://medium.com/u/a07a45803b5b">Nawab Iqbal</a> is a Senior Engineer and Technical Leader at Sendoso. He has experience in multiple technologies and paradigms. These days he is participating in scaling the architecture and processes to support the company’s growth. Without the experience and support from Nawab Iqbal, this blog would not exist.</p><p>Shout out to Maciej Mensfeld for his responses to our queries related to throughput issues in Karafka.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7ae2a519c39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/how-sendoso-is-using-kafka-for-event-driven-architecture-d7ae2a519c39">How Sendoso is using Kafka for Event-Driven Architecture</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka®. (An Overview)]]></title>
            <link>https://medium.com/sendoso-engineering/apache-kafka-an-overview-3db2c7b5a8d0?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/3db2c7b5a8d0</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[rails]]></category>
            <dc:creator><![CDATA[Zain Butt]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 18:45:10 GMT</pubDate>
            <atom:updated>2022-01-25T10:27:23.141Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Kafka was originally developed at LinkedIn and was later open-sourced and is currently being managed by a company called <a href="https://www.confluent.io/">Confluent</a> that is a team of people that originally developed it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_3dsvpnqEseyQlgyXOz1Rg.png" /></figure><p>Apache Kafka® is a high throughput, distributed, fault-tolerant, and scalable messaging system. Too many fancy words in one sentence. Don’t worry, you will get the hang of it.</p><p>To understand Kafka we have to look at how we traditionally develop applications. It&#39;s one monolith and some kind of datastore. To elaborate more on this is that we have a source and a target system and we exchange data between them and everything is fine until we have other applications to exchange data with that makes it hard to track the flow of data and is a maintenance nightmare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*x303fS0CMvxio4uRUmZjig.png" /></figure><p>Apache Kafka helps to eliminate this kind of uncertainty and complex architecture where we have many integrations and too many points of failure. Apache Kafka serves as a broker in between your applications and streamlines the data flow immensely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*3EDLeIAL-mRYp-OYkTA8aw.png" /></figure><p>Kafka decouples all the data streams and systems. Now, all your data coming from source systems would go to Apache Kafka and can be distributed to any target system without them knowing which app sent it. You see how this uniforms the data flow and decouples all the systems involved.</p><h4>Architecture</h4><p>Apache Kafka consists of a broker (a single Kafka server) along with some producers and consumers. A Kafka cluster usually consists of three brokers for replication purposes. A broker is responsible for receiving messages from producers and serve data to consumers. Every message coming in is replicated to topic partitions of all brokers and an acknowledgment is sent back to confirm that it has been persisted and can be consumed safely (configurable). If one or more brokers fail for any reason the data can still be served from other brokers in the cluster.</p><h4>Topic</h4><p>Every message sent to Kafka is stored in a topic that helps categorize the data. e.g User data might be sent to the users topic. A topic is further divided into partitions that are usually scattered across the cluster for redundancy. Partitions have a leader that serves reads and writes, and in-sync replicas are passive partitions that just copy data. In case of failure of a leader partition, one of the replicas becomes the leader.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*GoRlq7O8qMNui6tvnq30cg.png" /><figcaption>Partitions of a topic</figcaption></figure><h4>Message</h4><p>The message is the single unit of data persisted on disk in byte format. Apache Kafka is a distributed event log where all the messages are immutable and are appended at the end of the log and can be retained for as much time as we want by configuring something called ‘<em>Retention Policy</em>’. The messages have an id (incremental) called offset and are immutable as well.</p><h4>Producer</h4><p>Producers are applications/services that send data to Kafka (leader broker). A topic has to be provided when sending messages. There is no guarantee to which partition a message would be sent to (round-robin by default). A consistent message key is used to ensure delivery by hashing it to a specific partition.</p><p>Messages can be sent in a batch by using compression, increasing throughput, and decreasing network request but with a trade-off with latency. You can experiment with different combinations according to your preferences/ needs.</p><h4>Consumer</h4><p>Consumers are applications that fetch data from Kafka by subscribing to a specific topic. The way consumers get data from Kafka is by pulling as opposed to a push-based mechanism in other messaging systems like <a href="https://www.rabbitmq.com/">RabbitMQ</a>. This way the applications do not choke and a buffer is created and can consume data at its own pace. Data is read in order from one partition but is not guaranteed between partitions. Once a consumer consumes a message it commits back to the broker, this is called a <strong>consumer</strong> <strong>offset</strong>.</p><p>In general, whenever a consumer is created it has to be associated with a group represented by a unique id and all the consumers in a group read from different partitions of a topic in parallel. This helps us to scale by increasing the data consumption rate and makes the system capable of reading a high volume of data. By being in a group each consumer reads from a different partition and if one the consumer goes down we still have other consumers in that group fetching data from Kafka. If we have more consumers than partitions then the extra consumers would stay idle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ngtGxYoCPfyvYpbO-Aoag.png" /></figure><h4><strong>Use Cases</strong></h4><p>Some of the places where we can leverage Kafka.</p><ul><li><a href="http://kafka.apache.org/uses#uses_messaging">Messaging</a></li><li><a href="http://kafka.apache.org/uses#uses_website">Website Activity Tracking</a></li><li><a href="http://kafka.apache.org/uses#uses_metrics">Metrics</a></li><li><a href="http://kafka.apache.org/uses#uses_logs">Log Aggregation</a></li><li><a href="http://kafka.apache.org/uses#uses_streamprocessing">Stream Processing</a></li><li><a href="http://kafka.apache.org/uses#uses_eventsourcing">Event Sourcing</a></li><li><a href="http://kafka.apache.org/uses#uses_commitlog">Commit Log</a></li><li><a href="https://www.youtube.com/watch?v=I32hmY4diFY">ETL</a></li></ul><p>Well, this brings us to the end of the overview of Kafka. Every topic discussed has so much more but it is helpful to know what high-level pieces are there and gives us a starting point or a blueprint of things.</p><p>Docs: <a href="http://kafka.apache.org/documentation/">http://kafka.apache.org/documentation/</a></p><p>Youtube: <a href="https://www.youtube.com/channel/UCmZz-Gj3caLLzEWBtbYUXaA">https://www.youtube.com/channel/UCmZz-Gj3caLLzEWBtbYUXaA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3db2c7b5a8d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/apache-kafka-an-overview-3db2c7b5a8d0">Apache Kafka®. (An Overview)</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ActiveRecord::NotNullViolation when testing with RSpec]]></title>
            <link>https://medium.com/sendoso-engineering/activerecord-notnullviolation-when-testing-with-rspec-7402e0cf0784?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/7402e0cf0784</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[rspec]]></category>
            <category><![CDATA[tdd]]></category>
            <dc:creator><![CDATA[Zain Butt]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 18:45:09 GMT</pubDate>
            <atom:updated>2022-01-26T10:57:37.209Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6JOK7EvsaEMpGgzM8RPSsw.png" /></figure><p>When testing Rails Model and you encounter a <em>ActiveRecord::NotNullViolation, </em>this means you need to provide some default value for an attribute. Normally Mysql wont bug you to provide it, but in newer versions it does require it.</p><p>In my case it was due to created_at field without a default value.</p><pre><em>ActiveRecord::NotNullViolation:</em></pre><pre><em>Mysql2::Error: Field &#39;created_at&#39; doesn&#39;t have a default value</em></pre><p>This happens when Mysql turns on its strict mode by default and requires you to provide a default value for the attribute. If you don’t want to provide a value then you can just simply turn off strict mode by adding this to database.yml file:</p><pre>test:<br>  host: ...<br>  username: ...<br>  strict: false</pre><p>Now run your specs and they will pass without throwing this error.</p><p>Happy Coding!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7402e0cf0784" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/activerecord-notnullviolation-when-testing-with-rspec-7402e0cf0784">ActiveRecord::NotNullViolation when testing with RSpec</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How not to use a background job in a transaction]]></title>
            <link>https://medium.com/sendoso-engineering/how-not-to-use-a-background-job-in-a-transaction-1c797187f01e?source=rss----93874bc3ef83---4</link>
            <guid isPermaLink="false">https://medium.com/p/1c797187f01e</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[sidekiq]]></category>
            <dc:creator><![CDATA[Zain Butt]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 18:45:06 GMT</pubDate>
            <atom:updated>2022-01-25T12:20:52.453Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern web applications, transactions and background jobs play an important role to ensure consistency and performance. But sometimes if not used wisely and carefully can end up creating bugs that are difficult to identify in a big application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WU9VerjIVTthD5bmODdDJw.png" /></figure><p>While identifying such an issue in an application where we were sending a webhook to an external third party app to cancel an order and update item quantities. But somehow the wrong status was being sent for order along with the wrong quantities.</p><p>We identified the discrepancy between the expected and actual data and after crossing off other reasons from our list we realized it was due to the fact that the background jobs were invoked in the transaction.</p><pre>Model.transaction <em>do<br>  ... # some code<br>  cancel_order_and_send_webhook # invokes background job<br>  ... # some more code<br>end</em></pre><h4>Reason</h4><p>Now when the method was executed it enqueued a couple of background jobs to send webhooks. So, by the time the jobs ran the transaction wasn’t committed to the database and it used old values for the order and item quantities and hence sending the wrong values in the payload.</p><h4>Solution</h4><p>Moving background jobs out of the transaction did the thing and correct values were picked up.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c797187f01e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sendoso-engineering/how-not-to-use-a-background-job-in-a-transaction-1c797187f01e">How not to use a background job in a transaction</a> was originally published in <a href="https://medium.com/sendoso-engineering">Sendoso Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>