<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravi Ganta on Medium]]></title>
        <description><![CDATA[Stories by Ravi Ganta on Medium]]></description>
        <link>https://medium.com/@grkumar82?source=rss-41fccc9c296d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6UMNfzwATmrOchpA8u6nHA.jpeg</url>
            <title>Stories by Ravi Ganta on Medium</title>
            <link>https://medium.com/@grkumar82?source=rss-41fccc9c296d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:25:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@grkumar82/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Career Pathways of Data Engineers]]></title>
            <link>https://blog.dataengineerthings.org/career-pathways-of-data-engineers-2bc4465483d0?source=rss-41fccc9c296d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bc4465483d0</guid>
            <category><![CDATA[engineering-mangement]]></category>
            <category><![CDATA[career-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[individual-contributor]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Ravi Ganta]]></dc:creator>
            <pubDate>Wed, 06 Sep 2023 18:29:22 GMT</pubDate>
            <atom:updated>2023-09-06T18:29:22.841Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Exploring career trajectories available to Data Engineers, (drawing from my 8+ years of experience as a DE).</strong></h4><p>The career progression for Data Engineers is not a linear journey, unlike what might be expected. A straightforward trajectory, similar to that of Software Engineers ascending from entry-level individual contributors to executive leadership roles, does not necessarily apply to Data Engineers. This discourse delves into the intricate pathways that Data Engineers can undertake as they advance in their careers.</p><h3>Introduction</h3><p>To begin, let’s examine the trajectory of a typical Software Engineer’s career progression to contextualize the divergence in Data Engineers’ paths. While it’s common to witness Software Engineers transitioning to roles like Chief Technology Officers (CTOs) in the peak stages of their careers, the analogous path for Data Engineers remains less clear. The inquiry here centers around whether Data Engineers ascend to comparable roles such as CTOs or even become CTOs themselves. My personal experience suggests that only a limited number of Data Engineers attain executive leadership positions. While exceptions exist, this observation largely holds true.</p><p>This prompts the following questions:</p><h4>Why don’t Data Engineers attain executive leadership roles? How can Data Engineers advance their careers towards executive levels?</h4><p>My response to these Q’s is divided into two segments:</p><ul><li>The journey within the Individual Contributor (IC) career path</li><li>The path involving both IC responsibilities and an Engineering Manager role.</li></ul><p>These two tracks diverge as Data Engineers accumulate more expertise in their field.</p><h3><strong>IC Career Pathway</strong></h3><p>This pathway is where most Data Engineers start their career, working in a Data Engineering Org alongside other Data Engineers and reporting to a DE or Analytics Manager or hired as a the first Data Engineer within an Engineering Org. Either way, this first step sets-up the various way Data Engineers can navigate their career as they grow within an organization and accumulate more years of experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2tAv3de7PeBLab8_FCR-WA.png" /><figcaption>Pathways for IC Data Engineer</figcaption></figure><h4><strong>Path A — Stay in DE lane</strong></h4><p>For Data Engineers who intend to remain in technical roles without transitioning to people management, this trajectory is the default. After a few years, they can advance to the Senior Data Engineer level and possibly ascend to Staff or Principal roles afterward. However, it’s important to note that a significant proportion of Data Engineers remain at the Senior level, with fewer progressing to Staff or Principal positions.</p><p>A respected colleague of mine, a former Director, shared that the specific scope of work required for Staff or Principal Data Engineer positions is relatively uncommon among pure Data Engineers. This doesn’t imply that such opportunities don’t exist, but they are infrequent. Should a Data Engineer broaden their scope to encompass areas like Platform Engineering or backend engineering, they can display impactful contributions that merit promotions to these higher tiers. This demands adeptness in cross-functional collaboration and orchestrating cooperation among diverse teams. If a Data Engineer operates within a Software Engineering team (outside of Data Engineering org), attaining the Staff Engineer position can be challenging. To overcome this, Data Engineers need to proactively engage in meaningful discussions with their managers, and even higher-ups, to showcase their contributions and seek projects with substantial technical breadth that propel them toward the Staff Engineer role. Failing to do so may result in career stagnation and diminished motivation.</p><p>Notably, the title of Principal Data Engineer is rare, with only a few companies adopting this level for individual contributors. One exception is Amazon which lacks a Staff Engineer level, therefore the immediate promotion for a Senior Data Engineer is to Principal Data Engineer.</p><h4><strong>Path B</strong> — Expand to Full Stack Engineering in Data Space</h4><p>Before delving into this path, it’s essential to define the Data landscape where a Data Engineer operates. At one end are conventional Data Engineers who construct and manage pipelines in collaboration with stakeholders. They often use tools set up by other teams and aren’t responsible for tool maintenance (e.g., an Airflow service maintained by the Data Platform team). At the opposite end are teams that furnish tools employed by Data Engineers, Data Scientists, and ML Engineers.</p><p>Navigating this path equips Data Engineers with a holistic perspective of the Data domain. They serve as both consumers of tools provided by the Data Platform team and as creators and maintainers of such tools. These versatile engineers, termed (in my opinion) as ‘Full Stack Data Engineers’, are projected to be in demand due to their ability to set up end-to-end Data Architecture. This path confers greater career progression options for ICs. Exposure to the full spectrum of the Data domain empowers them to select projects that facilitate advancement to roles like Staff/Senior Staff or even Principal levels.</p><h3><strong>Engineering Manager Pathway</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9K5qmOgsiW7QzBSa8j65Q.png" /><figcaption>Pathways for Engineering Manager</figcaption></figure><h4><strong>Path C</strong> — Grow in DE/Analytics Org</h4><p>This pathway merges IC responsibilities with people management for Data Engineers. After reaching the senior level, they can opt to transition to People Manager roles, often supervising teams of Data Engineers. If within a Data Engineering organization, the progression is typically well-defined, extending from Manager to Director or VP. However, if a Data Engineer reports to a Software Engineering Manager, the pathway becomes less certain. As outlined in Path A, proactive dialogues are imperative to chart a clear plan, with the Data Engineer playing a pivotal role in shaping this trajectory.</p><h4>Path D — Head of Data Platform (including DE)</h4><p>Similar to Path B, this route is less frequented yet carries higher potential for eventual organizational leadership or even participation in the executive team. Engineers traversing this path boast comprehensive experience managing both Data Engineer teams and teams responsible for developing and maintaining tools utilized by Data professionals. In some instances, this structure encompasses Machine Learning teams also. This amalgamated approach can yield a Chief Data Officer or equivalent role, overseeing data governance, utilization, provisioning, quality assurance, and updates that bolster corporate decision-making.</p><p>This discussion aims to offer insights into the diverse career pathways accessible to Data Engineers, facilitating a long-term perspective. The pathways delineated here are based on my personal experiences and may not be an exhaustive list. If there are overlooked pathways, I encourage you to comment or connect with me on <a href="https://www.linkedin.com/in/ganta/">LinkedIn</a> with your detailed feedback.</p><h3>Conclusion</h3><p>In conclusion, the career progression of Data Engineers is notably distinct from the conventional paths often associated with Software Engineers. This discussion has illuminated two primary avenues: the Individual Contributor (IC) path and the Engineering Manager path, each with its unique challenges and strategies for advancement. To thrive in the dynamic field of Data Engineering, professionals must embrace adaptability, broaden their skill sets, and proactively shape their career trajectories. While the road to senior leadership roles may not be as straightforward, the potential for growth and impact within the data domain remains substantial, making this career field an exciting and evolving one for those who are well-prepared and forward-thinking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bc4465483d0" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/career-pathways-of-data-engineers-2bc4465483d0">Career Pathways of Data Engineers</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Preparation for Data Engineer Interviews in 2023]]></title>
            <link>https://blog.dataengineerthings.org/preparation-for-data-engineer-interviews-in-2023-e5d4e99c24d8?source=rss-41fccc9c296d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5d4e99c24d8</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ravi Ganta]]></dc:creator>
            <pubDate>Fri, 28 Jul 2023 00:42:01 GMT</pubDate>
            <atom:updated>2023-07-28T01:38:24.278Z</atom:updated>
            <content:encoded><![CDATA[<p>As mentioned in my previous <a href="https://blog.det.life/interviewing-for-data-engineering-in-2023-6cc5e6086216">article</a>, the data engineering landscape has evolved significantly in the past 2–3 hours. In this article, I will explore effective ways to prepare for your upcoming interview. Keep in mind that while this list covers major topics you should be aware of, it is not exhaustive. Additionally, remember that being overly focused on interviews can have its pitfalls, which I’ll address in a later article.</p><blockquote>Before we proceed, a couple of notes: (a) I have no sponsorship ties with any of the books/material mentioned below. (b) My interviewing experience primarily revolves around US-based companies, so the approach outlined here may or may not apply to other regions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZEnrQcoAofXEvc1jGMYTA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@wocintechchat">Christina</a> on <a href="https://unsplash.com/photos/eF7HN40WbAQ">Unsplash</a></figcaption></figure><p><strong>Data Modeling</strong>: Many interviewees expect to have some knowledge of data modeling, but often, they lack sufficient preparation to ace this round. We might become complacent due to the familiarity of the concept, but based on my experience, expert interviewers can quickly identify gaps in knowledge. To overcome this, thorough preparation is essential. Consider organizing mock interviews with friends and time yourselves. There are numerous resources available, including the quintessential book like <a href="https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/books/data-warehouse-dw-toolkit/">Kimball’s “The Data Warehouse Toolkit”</a> which has stood the test of time. <a href="https://www.amazon.com/Agile-Data-Warehouse-Design-Collaborative/dp/0956817203">“Agile Data Warehouse Design”</a> by Lawrence Corr with Jim Stagnitto is another book to consider. Also, stay updated with the latest approaches in data modeling, such as “one big table,” and understand its relevance in today’s big data space.</p><p><strong>System Design:</strong> This round is pivotal and can leave a lasting impression on the interviewing team. While some experienced engineers might assume they can wing it because they understand basic Data Engineering topics (e.g., ETL/ELT, Storage, Cloud Data Warehouses, etc.) and have some knowledge of Big Data architectures like Hadoop, this might not be sufficient. Interviewers can take you down unexpected paths, such as focusing on streaming-based approaches or data capture at the application code level. Prepare for different scenarios and potential gaps in your understanding. Recommending two books for a basic understanding: <a href="https://www.oreilly.com/library/view/fundamentals-of-data/9781098108298/">“Fundamentals of Data Engineering” </a>by Joe Reis and Matt Housley, and <a href="https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/">“Designing Data-Intensive Applications”</a> by Martin Kleppmann.</p><p><strong>Coding:</strong> Before reaching the System Design and Data Modeling rounds, you must pass the coding round. While there are plenty of resources available on how to prepare for coding interviews, I have some specific suggestions:</p><ul><li><strong>Practice for velocity and accuracy:</strong> Set a timer during your practice sessions, ensuring your code compiles without errors. Compile issues can be frustrating during a live interview, even if the result is incorrect initially.</li><li><strong>Code for real-life scenarios:</strong> Some interviewers are moving away from traditional LeetCode-type questions and focusing on coding real-life situations, such as querying APIs, ingesting and processing data, and producing results based on specific criteria. Don’t limit your practice to just algorithmic problems; work on real-life scenarios too.</li><li>Regularly conduct mock interviews to test your skills and gauge your progress.</li></ul><p><strong>ML/Data science</strong>: This knowledge can be a valuable addition for Data Engineers, although it’s not always mandatory. Some companies may expect Data Engineers to have a basic understanding of this realm. To get started, consider exploring short courses that provide an overview of key fundamental concepts in ML/Data Science. While I don’t have specific resources to suggest at this moment, there are numerous online platforms that offer such courses, and you can find one that suits your learning style and level of expertise. Look for courses that cover topics such as:</p><ul><li>Machine Learning algorithms and techniques</li><li>Data preprocessing and feature engineering</li><li>Model evaluation and validation</li><li>Supervised and unsupervised learning</li><li>Natural Language Processing (NLP) and Computer Vision (CV) if relevant to your interests</li></ul><p>Additionally, understanding how ML models are deployed and integrated into production systems can be beneficial. Remember, the goal is not to become an expert data scientist but rather to have a foundational understanding of ML/Data Science concepts that can complement your data engineering skills. This knowledge can be advantageous when working with data science teams, collaborating on projects, or contributing to the overall data-driven culture of your organization.</p><p>I hope these suggestions aid in your preparation. If you have specific questions, feel free to connect with me via <a href="https://www.linkedin.com/in/ganta/">LinkedIn</a>.</p><p>Best of luck with your preparation!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5d4e99c24d8" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/preparation-for-data-engineer-interviews-in-2023-e5d4e99c24d8">Preparation for Data Engineer Interviews in 2023</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interviewing for Data Engineering in 2023]]></title>
            <link>https://blog.dataengineerthings.org/interviewing-for-data-engineering-in-2023-6cc5e6086216?source=rss-41fccc9c296d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cc5e6086216</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ravi Ganta]]></dc:creator>
            <pubDate>Thu, 20 Jul 2023 23:05:34 GMT</pubDate>
            <atom:updated>2023-07-20T23:05:34.774Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you getting ready for Data Engineering interviews in 2023? If so, brace yourself for some unexpected challenges! This article presents my personal observations from interviewing for Data Engineering roles in 2023. (Note: I interviewed for US-based roles so these experiences reflect only the US market).</p><p>I found myself in the job market due to a recent downsizing at my company. Anticipating major changes, I began preparing for interviews a month before the official announcement. Since technical screening typically comes first, I dedicated myself to practicing LeetCode-style questions. While this proved helpful, it didn’t fully equip me for what awaited me during interviews. Here are my key observations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q-imKWYEqci5oMKAdYFTFA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@miinrad">Mina Rad</a> on <a href="https://unsplash.com/photos/a-man-and-a-woman-sitting-at-a-table--P7kHROe-6A">Unsplash</a></figcaption></figure><p><strong>1. Evolving requirements:</strong> In the past, possessing knowledge of SQL, Python, data modeling, and data engineering-related system design, coupled with experience working with cloud providers like AWS or Azure, would have sufficed. However, expectations have changed in 2023. Don’t be caught off guard if you encounter a round solely focused on Spark architecture and troubleshooting common Spark issues like OOM problems. They might even expect proficiency in coding with PySpark or Scala.</p><p><strong>2. Striving for perfection: </strong>With a saturated pool of candidates, some interviewers now demand flawless algorithms with optimal time and space complexities (e.g., O(N) or better time complexity with O(1) space complexity) within the 45-minute window. Failing to meet these expectations, even if your code solves all the provided test cases and isn’t a brute force solution, can result in rejection. Fortunately, more experienced interviewers see the flaws in this approach. I believe this trend is specific to 2023, as recent layoffs have flooded the job market, making interviewers more selective.</p><p><strong>3. Inconsistent interview processes:</strong> Data engineering interviews can vary significantly, even among companies within the same sector (e.g., gig economy or social media). Some companies focus on data modeling and machine learning questions, while others prioritize software development concepts during onsite interviews. This inconsistency poses challenges for candidates, requiring them to adapt and be prepared for different areas of emphasis depending on the company. For instance, I interviewed with a company that dedicated an onsite round to Machine Learning and Data Science. While it’s valuable knowledge, I believe it’s not a core skill for a Data Engineer, who already needs to excel in a broader spectrum of areas.</p><p><strong>4. SDE overlap:</strong> In certain companies, data engineering falls under the software development engineer (SDE) job family. Although not new, this means that interviews may include a System Design round assessing knowledge not only of data architecture but also overall system design. This might surprise Data Engineers who expect to focus solely on Data Architecture but find the scope to be much broader.</p><p><strong>5. Uncommon data structures for Data Engineers: </strong>Data Engineers may not regularly use data structures like Linked Lists or Tress in their day-to-day work, leading them to skip practicing them during interview preparation. However, some companies might include these structures in their coding rounds. For instance, a startup I interviewed with asked me to code a hashmap-like data structure without using any native Python libraries. While I managed to create a working solution, the interviewer pointed out that my approach, while functional, was not optimized and suggested considering Linked Lists instead.</p><p>As you embark on your interview journey, keep these lessons in mind and adjust your approach accordingly to better prepare. In my next article, I will share the approach that helped me secure multiple offers.</p><p>Best of luck with your interviews!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cc5e6086216" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/interviewing-for-data-engineering-in-2023-6cc5e6086216">Interviewing for Data Engineering in 2023</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>