<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sreeja Edulapalle on Medium]]></title>
        <description><![CDATA[Stories by Sreeja Edulapalle on Medium]]></description>
        <link>https://medium.com/@SreejaEdulapalle?source=rss-a3e523dc0dee------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*dhUFT3skAnYjRDGj</url>
            <title>Stories by Sreeja Edulapalle on Medium</title>
            <link>https://medium.com/@SreejaEdulapalle?source=rss-a3e523dc0dee------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:49:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@SreejaEdulapalle/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Raw Data to Reliable Systems: What I Learned About Modern Data Engineering]]></title>
            <link>https://medium.com/@SreejaEdulapalle/from-raw-data-to-reliable-systems-what-i-learned-about-modern-data-engineering-cc6604ad6d7f?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc6604ad6d7f</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 13:15:44 GMT</pubDate>
            <atom:updated>2026-09-25T13:15:44.863Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently completed <strong>Course 1 of the DeepLearning.AI Data Engineering Professional Certificate</strong>, and one thing became clear very quickly: data engineering is much more than moving data from one database to another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HN5fywhblQtu-QRvRa7wtw.png" /></figure><p>It is about designing systems that can reliably collect, store, transform, govern, and serve data while balancing scalability, security, cost, performance, and business requirements.</p><p>This course helped me connect many concepts I had previously encountered individually — data pipelines, AWS services, batch and streaming systems, orchestration, DataOps, architecture, security, and requirements gathering — into one complete picture.</p><p>This post summarizes the most important ideas I took away from the course.</p><h3>What Exactly Is Data Engineering?</h3><p>At a high level, <strong>data engineering is the practice of designing, building, and operating systems that make data available and useful</strong>.</p><p>A data engineer sits between the systems that generate data and the people or applications that consume it.</p><p>A simplified flow looks like this:</p><p><strong>Source Systems → Ingestion → Storage → Transformation → Serving → Consumers</strong></p><p>Those consumers might be:</p><ul><li>Data analysts building dashboards</li><li>Data scientists training machine learning models</li><li>Business teams making operational decisions</li><li>Applications consuming data through APIs</li><li>Operational systems receiving data through reverse ETL</li></ul><p>This is essentially the <strong>data engineering lifecycle</strong>.</p><p>But the pipeline itself is only part of the job. Running underneath the entire lifecycle are what I learned as the <strong>data undercurrents</strong>:</p><p><strong>Security → Data Management → DataOps → Data Architecture → Orchestration → Software Engineering</strong></p><p>These aren’t isolated stages. They influence every part of the system.</p><h3>Understanding the Data Lifecycle</h3><p>Before data can be analyzed, it has to come from somewhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6x3L0Ytk-QPXjB5wmlmwYA.png" /></figure><h3>1. Source Systems</h3><p>Data is generated by operational systems such as:</p><ul><li>Application databases</li><li>Transaction systems</li><li>APIs</li><li>IoT devices</li><li>Logs</li><li>SaaS platforms</li><li>Message queues</li><li>Event streams</li></ul><p>On AWS, examples include relational databases through <strong>Amazon RDS</strong>, NoSQL databases through <strong>Amazon DynamoDB</strong>, and streaming sources through services such as <strong>Amazon Kinesis Data Streams</strong>, <strong>Amazon SQS</strong>, and <strong>Amazon MSK</strong>.</p><p>The important lesson here is that the data engineer usually doesn’t control the upstream source.</p><p>If an application changes its schema, the downstream pipeline may break.</p><p>That is why understanding <strong>upstream and downstream dependencies</strong> is critical.</p><h3>2. Data Ingestion</h3><p>Once data exists, it needs to enter the data platform.</p><p>There are two major ingestion patterns.</p><h3>Batch ingestion</h3><p>Data is collected over a period of time and processed together.</p><p>For example:</p><p><strong>Operational Database → Daily Extract → S3 → Transformation → Data Warehouse</strong></p><p>Batch pipelines are often appropriate when the business doesn’t require second-by-second updates.</p><h3>Streaming ingestion</h3><p>Data is processed continuously as events arrive.</p><p>For example:</p><p><strong>Application → Event Stream → Stream Processor → Analytics System</strong></p><p>This becomes useful for scenarios such as fraud detection, monitoring, recommendations, and real-time operational analytics.</p><p>The interesting question isn’t simply:</p><blockquote><em>“Should this pipeline be streaming?”</em></blockquote><p>A better question is:</p><blockquote><em>“How quickly does the business actually need this data?”</em></blockquote><p>If the business needs information once every 24 hours, building a complex real-time streaming architecture could create unnecessary cost and operational complexity.</p><h3>3. Storage: Warehouse, Lake, or Lakehouse?</h3><p>Once data has been ingested, it needs somewhere to live.</p><p>Three major architectural patterns stood out.</p><h3>Data Warehouse</h3><p>A data warehouse stores structured, curated data optimized for analytics.</p><p>Amazon Redshift is an example.</p><p>Warehouses are especially useful when organizations need predictable analytical queries, reporting, and structured business datasets.</p><h3>Data Lake</h3><p>A data lake stores large amounts of raw and processed data, often using object storage.</p><p>On AWS, <strong>Amazon S3</strong> commonly serves as the storage layer.</p><p>Unlike traditional warehouses, lakes can hold structured, semi-structured, and unstructured data.</p><h3>Data Lakehouse</h3><p>The lakehouse approach attempts to combine the flexibility and lower-cost storage characteristics of a data lake with capabilities traditionally associated with data warehouses.</p><p>The important lesson wasn’t that one architecture is universally better.</p><p>Architecture depends on the problem.</p><h3>4. Querying, Modeling, and Transformation</h3><p>Raw data usually isn’t immediately useful.</p><p>It may contain:</p><ul><li>Duplicates</li><li>Missing values</li><li>Incorrect types</li><li>Different naming conventions</li><li>Conflicting business definitions</li><li>Unnecessary fields</li></ul><p>Transformation converts raw data into reliable datasets that consumers can understand.</p><p>Data modeling then determines how those datasets are structured.</p><p>One model covered in the course was the <strong>star schema</strong>, where a central fact table connects to dimension tables.</p><p>For example:</p><p><strong>Sales Fact</strong></p><p>↳ Customer Dimension<br>↳ Product Dimension<br>↳ Date Dimension<br>↳ Store Dimension</p><p>The goal isn’t simply to organize tables nicely. Good modeling makes analytical queries easier to understand and often improves performance.</p><h3>5. Serving Data</h3><p>Eventually, data has to create value.</p><p>The serving layer makes processed data available to consumers.</p><p>That can include traditional <strong>business intelligence</strong>, operational analytics, embedded analytics, machine learning systems, or reverse ETL workflows.</p><p>AWS provides several services in this area.</p><p><strong>Amazon Athena</strong> allows SQL queries directly against data stored in S3.</p><p><strong>Amazon Redshift</strong> provides a managed analytical data warehouse.</p><p><strong>Amazon QuickSight</strong> provides visualization and BI capabilities.</p><p>Reverse ETL flips the traditional direction of analytics pipelines by taking processed data from analytical systems and pushing it back into operational tools.</p><p>So instead of:</p><p><strong>Application → Warehouse</strong></p><p>we might also have:</p><p><strong>Warehouse → CRM / Marketing Platform / Operational Application</strong></p><h3>The Undercurrents of Data Engineering</h3><p>One of my biggest takeaways was that building pipelines is only one part of data engineering.</p><p>Several disciplines run underneath everything.</p><h3>Security</h3><p>Security needs to exist throughout the lifecycle rather than being added at the end.</p><p>One fundamental idea is the <strong>principle of least privilege</strong>:</p><blockquote><em>Give users and systems only the permissions they actually need.</em></blockquote><p>AWS Identity and Access Management (IAM) helps implement these controls.</p><p>Another important concept is avoiding routine use of the AWS root user. Administrative access should be carefully controlled and everyday workloads should use appropriately scoped identities and permissions.</p><p>Security decisions also depend on <strong>data sensitivity</strong>.</p><p>Public information, internal operational data, customer information, financial data, and highly sensitive personal information should not necessarily receive identical controls.</p><h3>From Hardened Perimeters to Zero Trust</h3><p>Traditional systems often relied heavily on a hardened perimeter.</p><p>The idea was roughly:</p><p><strong>Outside = untrusted<br>Inside = trusted</strong></p><p>Modern distributed systems make this assumption increasingly difficult.</p><p>Zero Trust takes a different approach:</p><p><strong>Never automatically trust access simply because it originates inside the network.</strong></p><p>Identity, authorization, device/system context, and access policies become increasingly important.</p><h3>Data Quality, Management, and Governance</h3><p>A pipeline running successfully doesn’t necessarily mean the data is correct.</p><p>Imagine a pipeline processes 50 million rows without errors but assigns revenue to the wrong customers.</p><p>Technically, the job succeeded.</p><p>From the business perspective, it failed.</p><p>That distinction helped me understand why <strong>data quality</strong> is such an important engineering responsibility.</p><p>Data management and governance address questions such as:</p><ul><li>Who owns this dataset?</li><li>What does each field mean?</li><li>Where did the data originate?</li><li>Who can access it?</li><li>How long should it be retained?</li><li>Is the dataset trustworthy?</li></ul><p>On AWS, services such as <strong>AWS Glue Data Catalog</strong>, <strong>AWS Glue Crawlers</strong>, and <strong>AWS Lake Formation</strong> can participate in metadata management, discovery, cataloging, and governance workflows.</p><h3>DataOps</h3><p>DataOps applies many software engineering and DevOps ideas to data systems.</p><p>Instead of treating pipelines as scripts that someone manually executes, data systems should be observable, testable, repeatable, and deployable.</p><p>Important areas include:</p><p><strong>Automation, monitoring, testing, deployment, observability, reliability, and collaboration.</strong></p><p>AWS services such as <strong>Amazon CloudWatch</strong>, CloudWatch Logs, and <strong>Amazon SNS</strong> can support monitoring, logging, and notifications.</p><p>This changes the mindset from:</p><blockquote><em>“Did my pipeline run?”</em></blockquote><p>to:</p><blockquote><em>“Can I operate this pipeline reliably in production?”</em></blockquote><h3>Orchestration and DAGs</h3><p>A real pipeline rarely consists of one task.</p><p>Imagine:</p><p><strong>Extract Orders → Validate → Transform → Load Fact Table → Refresh Dashboard</strong></p><p>Some tasks cannot start until previous tasks succeed.</p><p>This is where orchestration becomes important.</p><p>A <strong>Directed Acyclic Graph (DAG)</strong> represents dependencies between tasks.</p><p>Tools such as:</p><ul><li>Apache Airflow</li><li>Dagster</li><li>Prefect</li><li>Mage</li></ul><p>can coordinate these workflows.</p><p>Orchestration handles much more than scheduling. It can manage dependencies, retries, failures, execution order, and monitoring.</p><h3>AWS as a Data Engineering Platform</h3><p>The course connected the data engineering lifecycle to AWS services.</p><p>A simplified AWS pipeline might look like:</p><p><strong>RDS / DynamoDB / Applications</strong></p><p>↓</p><p><strong>DMS / Glue / Kinesis / SQS / MSK</strong></p><p>↓</p><p><strong>Amazon S3</strong></p><p>↓</p><p><strong>AWS Glue / EMR / Stream Processing</strong></p><p>↓</p><p><strong>Amazon Redshift / Athena</strong></p><p>↓</p><p><strong>QuickSight / ML / Applications</strong></p><p>Around this pipeline are IAM, CloudWatch, orchestration tools, metadata systems, CI/CD tooling, and governance services.</p><p>Seeing AWS this way was more useful than memorizing dozens of individual services.</p><p>Each service solves a particular problem within the lifecycle.</p><h3>EC2, Networking, and Infrastructure Fundamentals</h3><p>Understanding data platforms also requires understanding the infrastructure underneath them.</p><p><strong>Amazon EC2 (Elastic Compute Cloud)</strong> provides virtual compute instances.</p><p>Different instance families are optimized for different workloads, such as general-purpose, compute-intensive, memory-intensive, or storage-intensive workloads.</p><p>Behind virtualization is the <strong>hypervisor</strong>, which allows physical computing resources to support isolated virtual machines.</p><p>Networking introduces another important AWS concept: the <strong>Virtual Private Cloud (VPC)</strong>.</p><p>A VPC provides a logically isolated network where AWS resources can operate.</p><p>Inside a VPC, networks can be divided into <strong>subnets</strong>.</p><p>To understand those networks, concepts such as IP addresses and <strong>CIDR (Classless Inter-Domain Routing)</strong> become important because they define address ranges.</p><p>Suddenly networking concepts that can initially feel disconnected from data engineering become relevant when designing real cloud systems.</p><h3>Servers, Containers, and Serverless</h3><p>Another important architectural decision is how much infrastructure we actually want to manage.</p><p>There is a spectrum:</p><p><strong>Servers → Containers → Serverless</strong></p><p>With servers such as EC2, engineers receive significant infrastructure control but also more operational responsibility.</p><p>Containers package applications and dependencies consistently while allowing more portability.</p><p>Serverless services abstract much of the underlying infrastructure.</p><p>Examples include:</p><ul><li>AWS Lambda</li><li>Amazon Athena</li><li>AWS Glue</li></ul><p>Serverless can reduce infrastructure management and scale dynamically, but it also means giving up some control.</p><p>That introduces one of the recurring themes of architecture:</p><p><strong>Control vs. Convenience</strong></p><p>More control often means more responsibility.</p><p>More convenience often means accepting more constraints from the managed platform.</p><h3>EMR vs. AWS Glue: Another Example of the Trade-Off</h3><p>Suppose we need large-scale distributed data processing.</p><p><strong>Amazon EMR</strong> provides considerable flexibility and control over big-data frameworks and cluster configurations.</p><p><strong>AWS Glue</strong> provides a more managed experience for ETL workloads.</p><p>Neither is automatically the correct choice.</p><p>The decision depends on questions such as:</p><ul><li>How much infrastructure control do we need?</li><li>What frameworks are required?</li><li>How much operational overhead can the team handle?</li><li>How predictable is the workload?</li><li>What are the cost requirements?</li></ul><p>Architecture is ultimately a series of trade-offs.</p><h3>Streaming: Kinesis vs. MSK</h3><p>The same principle appears in streaming systems.</p><p><strong>Amazon Kinesis Data Streams</strong> provides a managed AWS-native streaming service.</p><p><strong>Amazon MSK</strong> provides managed Apache Kafka.</p><p>Kafka provides a powerful ecosystem and significant flexibility, while Kinesis can provide a more integrated AWS-managed experience.</p><p>Again, the decision can be viewed partly through:</p><p><strong>Control ←→ Convenience</strong></p><p>If an organization already has extensive Kafka expertise or needs Kafka compatibility, MSK may fit naturally.</p><p>If a team wants tighter AWS integration and less Kafka-specific operational responsibility, Kinesis may be attractive.</p><p>The correct answer depends on requirements.</p><h3>Streaming Architecture Is More Than Choosing Kafka</h3><p>Streaming architecture introduces a different mental model.</p><p>Instead of periodically processing collections of records, systems continuously process a <strong>stream of events</strong>.</p><p>Frameworks such as <strong>Apache Flink</strong> can process these event streams.</p><p>Several architectural approaches have emerged.</p><h3>Lambda Architecture</h3><p>Lambda architecture combines separate batch and real-time processing paths.</p><h3>Kappa Architecture</h3><p>Kappa architecture emphasizes stream processing and avoids maintaining separate batch and streaming processing logic where possible.</p><h3>Unified Batch and Streaming</h3><p>Modern processing frameworks increasingly try to provide common abstractions for both bounded and unbounded data.</p><p>The underlying goal is reducing duplicated logic while still supporting different latency requirements.</p><h3>Reliability: Plan for Failure</h3><p>One architecture lesson I found particularly important was simple:</p><p><strong>Systems fail.</strong></p><p>Servers fail.</p><p>Networks fail.</p><p>Dependencies fail.</p><p>Deployments fail.</p><p>Human mistakes happen.</p><p>Good architecture assumes this rather than pretending it won’t happen.</p><p>Two useful concepts are:</p><h3>Recovery Time Objective (RTO)</h3><p>How long can the system remain unavailable?</p><h3>Recovery Point Objective (RPO)</h3><p>How much data can the organization afford to lose?</p><p>If the RPO is one hour, losing 45 minutes of data may be acceptable.</p><p>If the RPO is effectively zero, the architecture becomes significantly more demanding.</p><p>Reliability requirements therefore directly affect architectural complexity and cost.</p><h3>Scalability and FinOps</h3><p>A system that works for 10,000 records might fail at one billion.</p><p>Architecture therefore has to consider scalability from the beginning.</p><p>But infinite scalability at any cost isn’t good engineering either.</p><p>That leads to <strong>FinOps</strong> — bringing financial accountability into cloud engineering decisions.</p><p>Engineers need to think about questions such as:</p><ul><li>How much does this pipeline cost per run?</li><li>Are resources sitting idle?</li><li>Are we over-provisioning?</li><li>Could serverless reduce idle infrastructure?</li><li>Would another storage tier be cheaper?</li><li>Is the performance improvement worth the additional cost?</li></ul><p>This connects to <strong>Total Cost of Ownership (TCO)</strong>.</p><p>But cost isn’t only infrastructure spending.</p><p>There is also <strong>Total Opportunity Cost of Ownership (TOCO)</strong>.</p><p>A cheaper system that requires months of engineering effort might actually cost the business more than purchasing or adopting an existing managed solution.</p><h3>Build vs. Buy</h3><p>Engineers naturally like building things.</p><p>But building isn’t always the right business decision.</p><p>Before creating a custom solution, we should ask:</p><p><strong>Does an existing service already solve this problem well enough?</strong></p><p>A custom solution can provide flexibility and differentiation, but it also introduces maintenance, testing, security, infrastructure, staffing, and operational costs.</p><p>Sometimes the best engineering decision is choosing not to build something.</p><h3>On-Premises, Cloud, or Hybrid?</h3><p>Data platforms can operate in several environments.</p><p><strong>On-premises</strong> infrastructure gives organizations significant physical and operational control.</p><p><strong>Cloud</strong> infrastructure provides elasticity, managed services, and rapid provisioning.</p><p><strong>Hybrid</strong> environments combine both.</p><p>Again, architecture depends on requirements such as regulation, latency, existing infrastructure, security, cost, and organizational strategy.</p><h3>Monolithic vs. Modular Systems</h3><p>Architecture also determines how tightly system components depend on one another.</p><p>A monolithic system combines significant functionality into a tightly integrated application.</p><p>A modular architecture separates responsibilities into components with clearer interfaces.</p><p>Modularity can make systems easier to evolve independently, but distributed architectures introduce their own complexity.</p><p>APIs become particularly important because they establish contracts between systems.</p><p>This relates closely to <strong>Conway’s Law</strong>: system architectures often reflect the communication structures of the organizations that build them.</p><p>Architecture is therefore not purely technical.</p><p>Organizations shape systems.</p><h3>One-Way and Two-Way Door Decisions</h3><p>Not every architecture decision deserves the same amount of analysis.</p><p>A <strong>two-way door decision</strong> is relatively easy to reverse.</p><p>Try something. Measure it. Change it if necessary.</p><p>A <strong>one-way door decision</strong> is expensive or difficult to reverse.</p><p>Those decisions deserve considerably more analysis.</p><p>This idea helps prevent teams from spending weeks debating decisions that could simply be tested safely.</p><h3>Always Architecting</h3><p>Another useful idea was that architecture isn’t something completed once at the beginning of a project.</p><p>Requirements change.</p><p>Traffic changes.</p><p>Technology changes.</p><p>Teams change.</p><p>Costs change.</p><p>Regulations change.</p><p>Good systems therefore evolve.</p><p>Architecture is an ongoing engineering activity.</p><h3>The AWS Well-Architected Framework</h3><p>AWS provides the <strong>Well-Architected Framework</strong> for evaluating cloud architectures.</p><p>For data workloads, the <strong>Data Analytics Lens</strong> applies architectural thinking specifically to analytics systems.</p><p>Rather than focusing only on whether a system technically works, architecture should consider areas such as security, reliability, performance, operational excellence, cost, and sustainability.</p><p>This creates a much better question than:</p><blockquote><em>“Does the pipeline work?”</em></blockquote><p>The better question becomes:</p><blockquote><em>“Will this pipeline continue working reliably, securely, and economically as requirements and scale change?”</em></blockquote><h3>Building a Three-Tier Architecture</h3><p>The course also included a practical three-tier architecture.</p><p>The architecture separates a system into:</p><p><strong>Presentation Tier → Logic Tier → Data Tier</strong></p><p>The presentation tier handles the user-facing interface.</p><p>The logic tier handles application processing and business rules.</p><p>The data tier handles persistence.</p><p>Infrastructure components such as an <strong>Application Load Balancer</strong> can distribute requests across multiple EC2 instances.</p><p>An <strong>Auto Scaling Group</strong> can increase or decrease compute capacity depending on demand.</p><p>DNS maps human-readable domain names to network resources, while ports determine which network services receive traffic.</p><p>Tools such as <strong>Apache Benchmark</strong> can generate requests to test system behavior under load.</p><p>CloudWatch then provides metrics that allow us to observe what happens as traffic changes.</p><p>Seeing autoscaling happen while load changes makes cloud scalability much more concrete than simply reading the definition.</p><h3>Requirements Come Before Architecture</h3><p>Perhaps the most important part of architecture happens before selecting any technology.</p><p>It starts with <strong>requirements gathering</strong>.</p><p>Before deciding between Redshift, Athena, Glue, EMR, Kinesis, or MSK, we need to understand the business problem.</p><p>Requirements generally fall into two groups.</p><h3>Functional Requirements</h3><p>What must the system do?</p><p>For an analytical dashboard:</p><blockquote><em>Users must be able to view daily revenue by region.</em></blockquote><p>For a recommendation system:</p><blockquote><em>The application must generate personalized product recommendations.</em></blockquote><h3>Non-Functional Requirements</h3><p>How well must the system operate?</p><p>Examples include:</p><ul><li>Latency</li><li>Availability</li><li>Scalability</li><li>Security</li><li>Compliance</li><li>Reliability</li><li>Cost</li></ul><p>Two systems can provide exactly the same functionality while requiring completely different architectures because their non-functional requirements differ.</p><h3>The Iron Triangle: Time, Cost, and Scope</h3><p>Engineering projects operate under constraints.</p><p>Three major constraints are:</p><p><strong>Time — Cost — Scope</strong></p><p>Changing one usually affects the others.</p><p>Want more functionality in the same amount of time?</p><p>You may need more resources.</p><p>Want to reduce cost?</p><p>You may need to reduce scope or extend the timeline.</p><p>Want everything faster and cheaper?</p><p>Something else usually has to change.</p><p>Understanding these trade-offs makes requirements gathering much more realistic.</p><p>Instead of promising everything, engineers can discuss which constraints actually matter most to the business.</p><h3>Choosing a Batch Architecture</h3><p>One useful way to think about batch processing is to move from requirements to service selection.</p><p>Suppose we have:</p><p><strong>Source Database → Batch Processing → Data Warehouse → BI</strong></p><p>Potential AWS services could include:</p><p><strong>DMS / Glue → S3 → Glue or EMR → Redshift → QuickSight</strong></p><p>But service selection depends on scale, transformations, expertise, control requirements, and cost.</p><p>For lightweight managed ETL, Glue may be sufficient.</p><p>For large distributed workloads requiring more framework or cluster-level flexibility, EMR may make more sense.</p><p>For SQL analytics directly over files in S3, Athena may eliminate the need to operate a traditional analytical database for certain workloads.</p><p>Architecture starts with requirements — not with a favorite AWS service.</p><h3>Choosing a Streaming Architecture</h3><p>The same approach applies to streaming.</p><p>Imagine millions of application events being generated continuously.</p><p>A pipeline could look like:</p><p><strong>Applications → Kinesis / MSK → Stream Processing → S3 / Redshift → Analytics</strong></p><p>Or data may be delivered using <strong>Amazon Data Firehose</strong> when the requirement is primarily managed delivery into supported destinations.</p><p>The decision between Kinesis and MSK should not simply be:</p><blockquote><em>“Which technology is better?”</em></blockquote><p>A better set of questions is:</p><p><strong>Do we need Kafka compatibility?<br>How much operational control do we need?<br>What throughput and latency are required?<br>What expertise does the team already have?<br>How much infrastructure responsibility do we want?</strong></p><p>This brings us back to the same recurring architectural trade-off:</p><p><strong>Control vs. Convenience.</strong></p><h3>The Biggest Shift in My Understanding</h3><p>Before going through these concepts together, it was easy to think of data engineering primarily as:</p><p><strong>SQL + Python + ETL + Cloud</strong></p><p>Those skills matter, but they are only part of the job.</p><p>A more complete view is:</p><p><strong>Data Engineering = Data + Software Engineering + Cloud Infrastructure + Architecture + Operations + Business Requirements</strong></p><p>A strong data engineer doesn’t simply ask:</p><blockquote><em>“How do I move this data?”</em></blockquote><p>They ask:</p><blockquote><em>Where is the data coming from?</em></blockquote><blockquote><em>How fresh does it need to be?</em></blockquote><blockquote><em>Who will consume it?</em></blockquote><blockquote><em>How sensitive is it?</em></blockquote><blockquote><em>What happens when the pipeline fails?</em></blockquote><blockquote><em>How will it scale?</em></blockquote><blockquote><em>How will we monitor it?</em></blockquote><blockquote><em>How much will it cost?</em></blockquote><blockquote><em>What is the simplest architecture that satisfies the actual business requirement?</em></blockquote><p>That last question may be the most important.</p><p>Modern data engineering isn’t about using the largest number of technologies.</p><p>It’s about making deliberate engineering decisions.</p><h3>Final Takeaway</h3><p>Course 1 helped me see the data platform as one interconnected system rather than a collection of individual tools.</p><p>From source systems and ingestion to storage, transformation, orchestration, governance, serving, and monitoring, every component exists because of a requirement.</p><p>AWS provides a large toolbox — EC2, S3, RDS, DynamoDB, DMS, Glue, Kinesis, MSK, Redshift, Athena, QuickSight, IAM, CloudWatch, Lambda, and many others — but knowing the service names isn’t the real skill.</p><p>The real skill is understanding <strong>when and why to use them</strong>.</p><p>And architecture isn’t about finding a universally “best” technology.</p><p>It’s about understanding requirements, recognizing trade-offs, designing for failure, controlling cost, securing data, and building the simplest reliable system that solves the problem.</p><p>That is the biggest lesson I’m taking forward as I continue through the <strong>DeepLearning.AI Data Engineering Professional Certificate</strong>.</p><p>Stay tuned for my next blog, where I’ll continue sharing what I’m learning as I dive deeper into data engineering! 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc6604ad6d7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Earned Microsoft PL-900 and PL-300 — And Why It Changed How I Think as a Data Analyst]]></title>
            <link>https://medium.com/@SreejaEdulapalle/how-i-earned-microsoft-pl-900-and-pl-300-and-why-it-changed-how-i-think-as-a-data-analyst-6f612745e4a5?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f612745e4a5</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Wed, 10 Dec 2025 03:46:23 GMT</pubDate>
            <atom:updated>2025-12-10T03:46:23.694Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I Earned Microsoft PL-900 and PL-300 — And Why It Changed How I Think as a Data Analyst</h3><p>I’ve spent the last four years working with data across systems, tools, and business workflows. But I realized something critical: experience alone doesn’t guarantee growth — structured learning does. That insight pushed me to pursue two certifications that reshaped my approach to analytics:</p><ul><li><strong>Microsoft Power Platform Fundamentals (PL-900)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*TjbMLsfcvNc8U4xrEPwB_w.png" /></figure><ul><li><strong>Microsoft Power BI Data Analyst Associate (PL-300)</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*RUH_s_7pGTUado_EBciSaw.png" /></figure><p>This post is not about “exam tips.”<br> It’s about how I built real capability — not just credentials.</p><h3>Why I Chose Microsoft Power Platform &amp; Power BI</h3><p>I didn’t pick these certifications randomly.</p><p>I chose them because modern data analysts are no longer just report builders — we’re solution builders.</p><p>I wanted to:</p><ul><li>Automate workflows (not just analyze data)</li><li>Understand enterprise data models</li><li>Build end-to-end solutions using Power BI + Power Platform</li><li>Create insight pipelines, not static dashboards</li></ul><p>PL-900 gave me the ecosystem view.<br> PL-300 forced me into execution.</p><h3>How I Prepared for PL-900 (Power Platform Fundamentals)</h3><p>PL-900 is not about memorization — it’s about <strong>conceptual clarity</strong>.</p><p>I focused on:</p><ul><li>Understanding <em>why</em> each tool exists (Power Apps, Power Automate, Power BI, Dataverse)</li><li>Learning how enterprises design low-code architectures</li><li>Connecting Power Platform to real business scenarios</li></ul><h3>My approach:</h3><ul><li>Studied Microsoft Learn modules deeply</li><li>Practiced by mapping enterprise workflows mentally to Power Automate</li><li>Learned how business rules turn into applications, not just charts</li></ul><p>PL-900 taught me this:<br> A good analyst doesn’t just understand data — they understand systems.</p><h3>How I Prepared for PL-300 (Real Power BI Mastery)</h3><p>PL-300 is where theory ends and engineering begins.</p><p>I treated this certification like a real job assignment.</p><h3>What I trained seriously:</h3><ul><li>Data modeling and relationships</li><li>DAX logic and evaluation context</li><li>Performance optimization</li><li>Row-level security</li><li>Business-driven reporting design</li><li>Data transformation using Power Query</li></ul><h3>What I did not rely on:</h3><p>❌ rote memorization<br> ❌ shortcuts or dumps<br> ❌ surface-level learning</p><p>Instead, I:</p><ul><li>Rebuilt models from scratch</li><li>Practiced DAX beyond formulas — focused on logic</li><li>Visualized problems before opening Power BI</li><li>Treated errors as tutorials, not frustration</li></ul><p>PL-300 changed how I work:<br> I stopped “building reports.”<br> I started designing <strong>analytical systems</strong>.</p><h3>What These Certifications Say About Me</h3><p>I didn’t chase certificates.</p><p>I built <strong>foundations</strong>.</p><p>These certifications represent:<br> ✔ My commitment to disciplined learning<br> ✔ My ability to master tools deeply<br> ✔ My seriousness about analytics as a career — not just a job<br> ✔ My growth from user → designer → problem-solver</p><p>I don’t just use tools.<br> I understand how tools fit inside business reality.</p><h3>What I’m Looking For</h3><p>I’m actively seeking opportunities as a:</p><ul><li><strong>Data Analyst</strong></li><li><strong>Business Intelligence Analyst</strong></li><li><strong>Power BI Analyst</strong></li><li><strong>Analytics Engineer (Entry-level / Junior)</strong></li></ul><p>Where I add value:</p><ul><li>Power BI (Data modeling, visualization, DAX, security)</li><li>SQL</li><li>Business analytics</li><li>Process automation (Power Platform)</li><li>Clean data design</li><li>Decision-centric reporting</li></ul><h3>Final Thought</h3><p>Certifications don’t make you great.</p><p>But discipline does.</p><p>Structure does.</p><p>Ownership does.</p><p>And I bring all three.</p><p>📌 <strong>If you’re a recruiter or hiring manager looking for a driven Data Analyst who understands both analytics and business systems — I’m open to conversations and opportunities.</strong></p><h3>Footer (my contact)</h3><p>📧 Email: sreeja2039@gmail.com<br> 🔗 LinkedIn: <a href="https://www.linkedin.com/in/sreeja-edulapalle/">https://www.linkedin.com/in/sreeja-edulapalle/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f612745e4a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Analyzed Bike-Share Data to Understand Member vs Casual Riders (Using R)]]></title>
            <link>https://medium.com/@SreejaEdulapalle/how-i-analyzed-bike-share-data-to-understand-member-vs-casual-riders-using-r-eded6f4958ce?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/eded6f4958ce</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Wed, 03 Dec 2025 15:53:57 GMT</pubDate>
            <atom:updated>2025-12-03T15:53:57.633Z</atom:updated>
            <content:encoded><![CDATA[<p>When a company relies on subscriptions, the hardest question is not <em>how many users you have</em> — it’s <strong>who is worth converting</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uPCi8hphtjv8-pLBeLV-7A.png" /></figure><p>In this case study, I analyzed real bike-share trip data from Cyclistic (Chicago) to understand how <em>annual members</em> and <em>casual riders</em> behave differently — and how those differences can guide subscription growth strategy.</p><p>This post walks through what I did, what I found, and how it translates into real business decisions.</p><h3>The Business Problem</h3><p>Cyclistic offers:</p><ul><li>Annual memberships</li><li>Single-ride and daily passes</li></ul><p>While casual riders bring volume, <strong>members are where long-term revenue lives</strong>.</p><p>The business objective was simple:</p><blockquote><em>How do member behavior and casual behavior differ — and which casual riders should be targeted for conversion?</em></blockquote><h3>The Dataset</h3><p>I worked with two years of trip data:</p><ul><li>2019 (older schema, different column names)</li><li>2020 (newer schema)</li></ul><p>Each trip included:</p><ul><li>Start/end timestamps</li><li>Start and end stations</li><li>User type (member or casual)</li><li>Ride duration</li></ul><p>Before analyzing anything, I had to do what real analytics work always starts with:</p><p>✅ standardize schemas<br> ✅ clean timestamps<br> ✅ remove invalid records<br> ✅ engineer usable time features</p><h3>How I Prepared the Data in R</h3><p>This wasn’t just an analysis — it was an engineering task.</p><ul><li>Unified two different schemas into one clean structure</li><li>Converted timestamps into proper datetime formats</li></ul><p>Removed:</p><ul><li>Negative rides</li><li>Multi-day rides</li><li>Missing rider types</li></ul><p>Created analytical features:</p><ul><li>weekday</li><li>hour</li><li>month</li><li>ride_length_min</li><li>day_type (weekday vs weekend)</li></ul><p>This produced a single, clean dataset ready for analysis.</p><blockquote><em>Cleaning is not the “boring part” of data work — it’s where real analysts earn their pay.</em></blockquote><h3>What I Analyzed</h3><p>I focused on four dimensions that affect subscription decisions:</p><ol><li><strong>Ride frequency vs duration</strong></li><li><strong>Weekday vs weekend usage</strong></li><li><strong>Hourly behavior</strong></li><li><strong>Geographic concentration</strong></li></ol><h3>Key Insight #1</h3><h3>Members ride often. Casual riders ride longer.</h3><p>Members created most rides overall — but casual riders spent more time per ride.</p><p>That tells you something important:</p><ul><li>Members optimize for utility (commuting, routine travel)</li><li>Casual riders optimize for experience (leisure, sightseeing)</li></ul><p>Casual riders consistently have longer average ride durations across the entire week. Members ride shorter distances but do so more frequently, suggesting practical transportation use compared to leisure behavior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5VZBeD3jJSzZvcIx-Qt8uA.png" /></figure><h3>Key Insight #2</h3><h3>Members behave like commuters. Casuals behave like tourists.</h3><ul><li>Member trips spike on weekdays</li><li>Casual trips spike on weekends</li><li>Members ride most during morning and evening hours</li><li>Casual users ride mostly midday and evenings</li></ul><p>Two completely different use-cases.</p><p>Members follow a clear workday pattern with peaks in the morning and evening commute hours. Casual riders peak later in the day and show less rigid time structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j5L9NwODyQ0cnK1wT3E5Ig.png" /></figure><h3>Key Insight #3</h3><h3>Membership is a “money-saving product,” not a “fun product”</h3><p>This data shows:</p><ul><li>Casual users are likely paying more per minute than members.</li><li>Frequent casual riders could easily save money by switching — but they aren’t being nudged clearly.</li></ul><p>That’s not a pricing problem.</p><p>It’s a <strong>positioning problem</strong>.</p><h3>Key Insight #4</h3><h3>A small number of stations generate a large share of casual rides</h3><p>High-traffic leisure locations produce:</p><ul><li>More casual riders</li><li>Longer rides</li><li>Higher likelihood of conversion</li></ul><p>These locations are where marketing should be focused — not randomly across the city.</p><h3>Key Insight #5</h3><h3>Weekend behavior is your biggest conversion opportunity</h3><p>Casual rider activity spikes sharply on weekends, while member usage stays relatively stable throughout the work week. This contrast shows that casual riders are using the service primarily for leisure, not transportation.</p><p>Weekends are when casual users:</p><ul><li>Ride more often</li><li>Ride longer</li><li>Interact most with the product</li></ul><p>That makes weekends the <strong>highest-leverage moment</strong> to push membership offers. These riders already see value in the service — the barrier is not usage, it’s positioning.</p><p>Instead of promoting memberships evenly across the week, Cyclistic should concentrate messaging, in-app offers, and station-level promotions on <strong>Saturday and Sunday behavior</strong>, when casual users are most engaged and most likely to convert.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABIfAYTSlsKlFCcvYyBnpQ.png" /></figure><h3>Business Recommendations</h3><p>Based on behavior, not guesses:</p><h3>1. Convert weekend riders aggressively</h3><p>If someone rides every weekend, they’re already a member in denial.<br> Target them with post-ride membership offers.</p><h3>2. Market memberships as a cost-saving tool</h3><p>Don’t sell it as a “club.”<br> Sell it as:</p><blockquote><em>“You’re already paying too much — membership fixes that.”</em></blockquote><h3>3. Trigger discounts after ride milestones</h3><p>If a casual rider completes:</p><ul><li>3 rides</li><li>or crosses a certain spend threshold<br> → Send an automated membership discount.</li></ul><h3>4. Advertise where casual riders actually ride</h3><p>Not across the whole city.<br> Only at stations with:</p><ul><li>Long rides</li><li>High foot traffic</li><li>Weekend spikes</li></ul><h3>The Bigger Lesson</h3><p>This project reinforced something that’s easy to forget:</p><blockquote><em>Data isn’t valuable because it exists.<br> Data is valuable when it changes decisions.</em></blockquote><p>The goal was never charts.<br> The goal was <strong>revenue logic</strong>.</p><h3>Final Takeaway</h3><p>If Cyclistic wants more members, it doesn’t need:</p><ul><li>New features</li><li>New bikes</li><li>New pricing</li></ul><p>It needs better segmentation.</p><p>The data already tells us:</p><ul><li>Who rides often</li><li>Who rides long</li><li>Who rides differently<br> …and that’s all you need to start converting intelligently.</li></ul><h3>About This Project</h3><p>This analysis was built entirely in R using:</p><ul><li>tidyverse</li><li>lubridate</li><li>ggplot2</li><li>rmarkdown</li></ul><p>If you’re interested in the full technical pipeline (data cleaning, feature engineering, and code), feel free to connect with me on LinkedIn <a href="https://www.linkedin.com/in/sreeja-edulapalle/">https://www.linkedin.com/in/sreeja-edulapalle/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eded6f4958ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ What I Learned in My Final Module of the Google Data Analytics Certificate: How AI Changes the…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/what-i-learned-in-my-final-module-of-the-google-data-analytics-certificate-how-ai-changes-the-c8cb0c6434e2?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8cb0c6434e2</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Sat, 29 Nov 2025 07:08:48 GMT</pubDate>
            <atom:updated>2025-11-29T07:08:48.369Z</atom:updated>
            <content:encoded><![CDATA[<h3>🧠 What I Learned in My Final Module of the Google Data Analytics Certificate: How AI Changes the Data Analyst Career</h3><p>By Sreeja Edulapalle</p><p>“AI does not replace data analysts. It supercharges them.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DIEN8jZtny0DavzDmfIGlg.png" /></figure><p>This final module felt like stepping back and looking at the whole journey. SQL, R, spreadsheets, Tableau, case studies. All the skills I practiced now fit together. But the part that stood out the most was how AI is shaping the work of a modern analyst.</p><p>This blog is a wrap up of the program, what I learned about using AI in analysis, the skills I am adding to my resume, and what comes next for me as I build my portfolio and step into the job market.</p><h3>🔁 Catch up on my previous blog:</h3><p><a href="https://medium.com/@SreejaEdulapalle/%EF%B8%8F-what-i-learned-in-module-7-of-the-google-data-analytics-certificate-r-programming-for-data-ba7097a3dd45">🖥️ What I Learned in Module 7 of the Google Data Analytics Certificate: R Programming for Data…</a></p><h3>🤖 AI and Data Analysis: A Powerful Partnership</h3><p>AI is now part of a data analyst’s day to day work. Not as a shortcut, but as a support system that speeds up thinking, eliminates busywork, and sharpens decision making.</p><p>Here are the ways AI makes a real difference.</p><h3>🧹 Cleaning and Preparing Data</h3><p>Cleaning data used to take the most time. Now, AI tools like ChatGPT, Copilot or Gemini help you:</p><ul><li>Write cleaning code in R</li><li>Detect missing values, outliers and inconsistent names</li><li>Suggest mutate(), filter(), drop_na() steps</li><li>Explain why something in the data looks off</li></ul><p><em>You still make the decisions, but AI gives you the best possible starting point.</em></p><h3>📊 Transforming and Organizing Data</h3><p>When you are building pipelines in R with tidyverse, AI can suggest:</p><ul><li>Logic for summarise() and group_by()</li><li>Optimized mutate() formulas</li><li>Better ways to use pipes</li><li>More efficient transformations for large datasets</li></ul><p><em>It is like having a second pair of eyes that never gets tired.</em></p><h3>❓ Asking More Focused Analytical Questions</h3><p>Strong questions lead to strong insights. AI helps you:</p><ul><li>Refine hypotheses</li><li>Frame KPIs clearly</li><li>Turn vague questions into measurable ones</li><li>Think about multiple angles before you start coding</li></ul><p><em>This is a huge confidence builder when you are new to analytics.</em></p><h3>🎨 Improving Visualizations</h3><p>When you create charts in ggplot2, AI can suggest:</p><ul><li>The most effective chart type</li><li>Better labels, themes, and aesthetics</li><li>Layer ideas, like facet grids or smoothing lines</li><li>Ways to present the same insight more clearly</li></ul><p><em>Good visuals make your analysis speak for itself. AI helps you get there faster.</em></p><h3>🧠 Debugging and Improving R Code</h3><p>R can be tricky. AI can:</p><ul><li>Interpret error messages</li><li>Suggest faster or cleaner versions of your code</li><li>Explain concepts like tibbles, lists or joins</li><li>Review entire scripts and point out inefficiencies</li></ul><p><em>It turns debugging into learning instead of frustration.</em></p><h3>💡 The T C R E I Prompt Framework</h3><p>A good prompt makes AI more accurate and more helpful. This framework is simple and effective.</p><p><strong>T. Task — </strong>Say exactly what you want.</p><p><strong>C. Context — </strong>Give background and goals.</p><p><strong>R. References — </strong>Add examples or preferred formats.</p><p><strong>E. Evaluate — </strong>Check if the answer is right.</p><p><strong>I. Iterate — </strong>Ask again with improvements.</p><p><strong><em>The better your prompts, the better your results. As an analyst, that skill matters.</em></strong></p><h3>📦 Skills I Am Adding to My Resume</h3><p>This program gave me a strong toolkit. Here are the skills I now feel confident listing:</p><ul><li>Write SQL queries</li><li>Sort, filter and transform data in SQL</li><li>Create dashboards and visuals in Tableau</li><li>Build visuals in spreadsheets</li><li>Make structured presentations from analysis</li><li>Define user data needs</li><li>Use design thinking to improve dashboards</li><li>Build pivot tables</li><li>Clean data in spreadsheets</li><li>Install and use tidyverse</li><li>Run scripts in RStudio</li><li>Build visualizations in ggplot2</li><li>Create reports in R Markdown</li><li>Maintain change logs</li><li>Create data cleaning reports</li></ul><p>These skills build the foundation of real world analytics work.</p><h3>📁 Building My Portfolio</h3><p>A portfolio is the bridge between “I learned it” and “I can do it”. I am setting up mine with these principles:</p><ul><li><strong>Personal —</strong> so people know who I am</li><li><strong>Simple —</strong> so my work is easy to explore</li><li><strong>Relevant —</strong> to the job descriptions I apply to</li><li><strong>Presentable —</strong> with polished samples</li><li><strong>Original</strong> — with clear citations</li></ul><p>Great places to host projects:</p><ul><li>GitHub</li><li>Kaggle</li><li>Tableau Public</li><li>Medium</li><li>WordPress</li><li>Google Sites</li></ul><p>My next goal: build a portfolio that reflects all eight modules.</p><h3>🎓 Interview Stages I Am Preparing For</h3><p>The program also covered what to expect in the job process.</p><p><strong>Stage 1. Introduction</strong><br> Resume and portfolio review.</p><p><strong>Stage 2. Skill Test</strong><br> Usually a case study or technical exercise.</p><p><strong>Stage 3. Compatibility Interview</strong><br> Team fit and communication style.</p><p><strong>Stage 4. Decision Making</strong><br> Where everything comes together.</p><p>Google’s Warmup Tool is great practice for this.</p><h3>🧩 Final Reflections</h3><p>Finishing this certificate feels like connecting the dots.</p><p>R taught me reproducibility.<br> SQL taught me precision.<br> Tableau taught me communication.<br> Case studies taught me thinking.<br> AI taught me speed and clarity.</p><p>Together they form a complete toolkit for a modern data analyst.</p><p>This course did not just prepare me for one job. It trained me to think in a structured, analytical and scalable way. And that is exactly what today’s AI powered world needs.</p><p>Coming up next: my portfolio projects.<br> This is where everything I learned will transform into real examples of my work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8cb0c6434e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[️ What I Learned in Module 7 of the Google Data Analytics Certificate: R Programming for Data…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/%EF%B8%8F-what-i-learned-in-module-7-of-the-google-data-analytics-certificate-r-programming-for-data-ba7097a3dd45?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba7097a3dd45</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Wed, 19 Nov 2025 12:07:05 GMT</pubDate>
            <atom:updated>2025-11-19T12:07:05.813Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>🖥️ What I Learned in Module 7 of the Google Data Analytics Certificate: R Programming for Data Analysts</strong></p><p><em>By Sreeja Edulapalle</em></p><blockquote><em>“R is where raw data turns into reproducible insight.”</em></blockquote><p>If Module 6 was all about visualizing insights, Module 7 was about building the engine behind them.</p><p>This is where the Google Data Analytics Certificate really shifts from tools to <strong>code</strong>. And the star of the show is <strong>R</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gVhE2fCgC9f7EqmANsIiTA.png" /></figure><p>In this blog, I am sharing what I learned about R, RStudio, tidyverse, pipes, data structures, cleaning, visualization and reporting. In other words, how analysts use code not just to analyze data, but to tell a story that can be reproduced again and again.</p><h3>🔁 Catch up on my previous blog:</h3><p><a href="https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-6-of-the-google-data-analytics-certificate-the-power-of-data-941350216885">📊 What I Learned in Module 6 of the Google Data Analytics Certificate: The Power of Data…</a></p><h3>🧮 What Is R and Why Does It Matter?</h3><p>R is a <strong>programming language and environment</strong> made for:</p><ul><li>Statistical computing</li><li>Data analysis</li><li>Data visualization</li></ul><p>What I like about R as an analyst:</p><ul><li>It is <strong>data centric</strong></li><li>It has a <strong>huge ecosystem</strong> of packages</li><li>It is built for <strong>reproducible analysis</strong></li></ul><p>Instead of clicking around in a spreadsheet, you write scripts that you can rerun, share, tweak and track. That single shift changes how you think about analysis.</p><h3>📊 R vs Excel vs Python</h3><p>The course also framed R in context with other tools:</p><ul><li><strong>Excel</strong><br> Great for quick, manual work, but repetitive and not easily reproducible.</li><li><strong>R</strong><br> Fantastic for <strong>structured data</strong>, statistics and visualization. Ideal when you want transparency and repeatability.</li><li><strong>Python</strong><br> Very flexible across data types and excellent for <strong>machine learning</strong> and production systems.</li></ul><p>For many data analyst workflows, R hits the sweet spot between <strong>power</strong> and <strong>clarity</strong>.</p><h3>🖥️ RStudio — Your Home Base for R</h3><p>R by itself runs in a console. RStudio turns it into a smooth working environment.</p><p>RStudio is an <strong>IDE</strong> for R, and it is organized into panes:</p><ul><li><strong>Source</strong> — where you write scripts</li><li><strong>Console</strong> — where code actually runs</li><li><strong>Environment</strong> — where you see your data and variables</li><li><strong>Plots / Help / Packages / Files</strong> — where you visualize, explore and manage</li></ul><p>It is built specifically for R, which makes your workflow more focused. You can write code, run it line by line, see outputs, and keep everything organized in one place.</p><h3>📦 Packages, Tidyverse and Dates</h3><p>One of the biggest mindset shifts in R is understanding <strong>packages</strong>.</p><p>A package is like a toolbox: functions, datasets and documentation bundled together.</p><p>Typical workflow:</p><pre>install.packages(&quot;tidyverse&quot;)  # once<br>library(tidyverse)             # every session</pre><p>Some of the core packages I met in this module:</p><p><strong>tidyverse</strong><br> A collection of packages that work together:</p><ul><li>dplyr for data manipulation</li><li>ggplot2 for visualization</li><li>tidyr for reshaping data</li><li>readr for importing CSV files</li><li>tibble, stringr, forcats and more</li></ul><p><strong>lubridate</strong><br> This one is all about <strong>dates and times</strong>, which are usually messy.</p><pre>library(lubridate)<br>ymd(&quot;2025-11-09&quot;)  # returns a proper Date object</pre><p>Being able to handle dates correctly is huge for time based analysis like sales trends, web traffic or operational metrics.</p><h3>🧠 R Programming Fundamentals</h3><p>Module 7 also covered the basics of the R language itself:</p><p><strong>Variables and data types</strong></p><pre>x &lt;- 10          # numeric<br>y &lt;- &quot;Hello&quot;     # character<br>z &lt;- TRUE        # logical</pre><p><strong>Comments</strong></p><pre># This is a comment that explains code</pre><p><strong>Functions</strong></p><pre>sum(1, 2)<br>mean(c(10, 20, 30))</pre><p>You start to see that R is not just “statistics software”. It is a real programming language with structure and logic.</p><h3>🧱 Data Structures: Vectors, Lists, Data Frames and More</h3><p>R works with a few core data structures:</p><p><strong>Vectors</strong><br> The most basic building block.</p><pre> nums &lt;- c(1, 2, 3)<br> typeof(nums)<br> length(nums)</pre><p><strong>Lists</strong><br> Can store mixed types.</p><pre>mylist &lt;- list(name = &quot;John&quot;, age = 30, scores = c(90, 85, 88)) <br>mylist$name</pre><p><strong>Data frames</strong><br> Like a spreadsheet in code. Columns can be different types.</p><pre>df &lt;- data.frame(name = c(&quot;A&quot;, &quot;B&quot;), score = c(90, 80))</pre><p><strong>Tibbles</strong><br> A modern, friendlier version of data frames from the tidyverse. Better printing, no automatic conversion to factors.</p><pre>library(tibble) tb &lt;- tibble(name = c(&quot;Alice&quot;, &quot;Bob&quot;), age = c(25, 30))</pre><p>Understanding these structures is key, because almost everything in R sits on top of them.</p><h3>📥 Getting Data In: readr, readxl and Built in Datasets</h3><p>To do any analysis, you need data in your environment.</p><p>Some tools we used:</p><ul><li><strong>readr</strong> for flat files</li></ul><pre>library(readr)<br>df &lt;- read_csv(&quot;data.csv&quot;)</pre><ul><li><strong>readxl</strong> for Excel files</li></ul><pre>library(readxl)<br>df &lt;- read_excel(&quot;data.xlsx&quot;)</pre><ul><li><strong>data()</strong> to load built in datasets</li></ul><pre>data(mtcars) <br>head(mtcars)</pre><p>readr functions return <strong>tibbles</strong> by default, which integrate nicely with the rest of tidyverse.</p><h3>🔁 Pipes: Writing Readable Data Workflows</h3><p>One of my favorite parts of R so far is the <strong>pipe operator</strong> %&gt;% from dplyr.</p><p>Instead of nesting functions inside functions, you can write your steps as a sequence:</p><pre>library(dplyr)</pre><pre>mtcars %&gt;%<br>  filter(mpg &gt; 20) %&gt;%<br>  select(mpg, cyl)</pre><p>This feels like reading a sentence:<br> “Take mtcars, then filter rows where mpg &gt; 20, then select mpg and cyl.”</p><p>Pipes make complex operations more readable and easier to debug.</p><h3>🧹 Cleaning and Transforming Data</h3><p>Module 7 also brought in some very practical cleaning and transformation tools.</p><p>A few highlights:</p><p><strong>janitor</strong> for cleaning column names</p><pre>library(janitor) <br>df &lt;- clean_names(df)   # &quot;First Name&quot; becomes &quot;first_name&quot;</pre><p><strong>skimr</strong> for quick summaries</p><pre>library(skimr) <br>skim_without_charts(df)</pre><p><strong>dplyr</strong> for core manipulation</p><ul><li>select() to pick columns</li><li>filter() to keep certain rows</li><li>mutate() to create new columns</li><li>group_by() + summarise() for grouped summaries</li><li>arrange() to sort</li><li>drop_na() to remove missing values</li></ul><p>Example cleaning pipeline:</p><pre>df_clean &lt;- df %&gt;%<br>  clean_names() %&gt;%<br>  drop_na() %&gt;%<br>  filter(price &gt; 0) %&gt;%<br>  group_by(region) %&gt;%<br>  summarise(avg_sales = mean(sales, na.rm = TRUE))</pre><p>You start with messy data and end with something structured, tidy and ready for visualization or modeling.</p><h3>🔄 Reshaping Data with tidyr</h3><p>Real world data is not always in the shape you need. That is where tidyr comes in.</p><p>Some key functions:</p><ul><li>unite() to combine columns</li><li>separate() to split a column</li><li>pivot_longer() to go from wide to long</li><li>pivot_wider() to go from long to wide</li></ul><p>Example:</p><pre>data_long &lt;- data %&gt;%<br>  pivot_longer(cols = starts_with(&quot;year&quot;),<br>               names_to = &quot;year&quot;,<br>               values_to = &quot;sales&quot;)</pre><p>Reshaping is a big part of making data analysis ready.</p><h3>🎨 Visualizing with ggplot2</h3><p>We also went deeper into visualization, this time using <strong>code</strong> with ggplot2.</p><p>Core concepts:</p><ul><li>ggplot() starts a plot</li><li>aes() maps data to visual properties like x, y, color, size</li><li>geom_ functions define the type of chart</li><li>facet_ functions split plots into panels</li><li>labs() adds titles and labels</li><li>themes control the overall look</li></ul><p>Example:</p><pre>library(ggplot2)</pre><pre>ggplot(data = mpg, aes(x = displ, y = hwy, color = class)) +<br>  geom_point() +<br>  facet_wrap(~ class) +<br>  labs(<br>    title = &quot;Displacement vs Highway MPG&quot;,<br>    x = &quot;Engine Displacement&quot;,<br>    y = &quot;Highway MPG&quot;<br>  ) +<br>  theme_minimal()</pre><p>ggplot2 follows a “grammar of graphics” idea. You build a plot layer by layer, which gives you a lot of flexibility and control.</p><p>We also learned how to:</p><ul><li>Add <strong>smoothing lines</strong> with geom_smooth()</li><li>Use <strong>facets</strong> for comparisons</li><li>Add <strong>annotations</strong> like titles, subtitles, captions and text labels</li><li><strong>Save plots</strong> with ggsave() for reports and presentations</li></ul><h3>📄 R Markdown, Notebooks and Shiny</h3><p>The last part of Module 7 was all about <strong>documentation and reporting</strong>.</p><h3>R Markdown</h3><p>R Markdown lets you mix:</p><ul><li>Explanatory text</li><li>Code</li><li>Outputs like tables and charts</li></ul><p>in a single .Rmd file that can be “knit” into HTML, PDF or Word.</p><p>Basic structure:</p><pre>---<br>title: &quot;My Analysis Report&quot;<br>author: &quot;Sreeja Edulapalle&quot;<br>date: &quot;`r Sys.Date()`&quot;<br>output: html_document<br>---</pre><p>Code chunks sit inside the document:</p><pre>```{r, echo = TRUE}<br>summary(cars)<br>```</pre><p>This is incredibly useful for <strong>reproducible reporting</strong>. You change the data, re knit and your report updates.</p><h3>Notebooks and Dashboards</h3><p>The course also mentioned tools like:</p><ul><li>Jupyter and Kaggle notebooks</li><li>R Markdown notebooks</li><li>Dashboards with flexdashboard</li><li><strong>Shiny</strong> apps for interactive web applications built with R</li></ul><p>Even if you do not build dashboards right away, it is good to know that R can power full interactive experiences, not just static reports.</p><h3>📝 Final Thoughts</h3><p>Module 7 was a big step up. It was not just “learning a language”. It was learning a new way to think about analysis:</p><ul><li>From one time work to <strong>reproducible workflows</strong></li><li>From clicking to <strong>coding</strong></li><li>From static outputs to <strong>flexible, scripted reporting</strong></li></ul><p>R + RStudio + tidyverse give data analysts a powerful toolkit to:</p><ul><li>Clean and reshape data</li><li>Explore and visualize patterns</li><li>Document and share work in a transparent way</li></ul><p>In short, this module showed me that R is not just a skill on a resume. It is a mindset about how to work with data.</p><p><strong>Coming up next: Blog #8</strong><br> Where I bring even more pieces together and reflect on how all these modules connect in real world analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba7097a3dd45" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ What I Learned in Module 6 of the Google Data Analytics Certificate: The Power of Data…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-6-of-the-google-data-analytics-certificate-the-power-of-data-941350216885?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/941350216885</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Sun, 09 Nov 2025 05:52:29 GMT</pubDate>
            <atom:updated>2025-11-09T05:52:29.903Z</atom:updated>
            <content:encoded><![CDATA[<h3>📊 What I Learned in Module 6 of the Google Data Analytics Certificate: The Power of Data Visualization</h3><p><strong>By Sreeja Edulapalle</strong></p><blockquote>“A good chart doesn’t just look good — it drives clarity and action.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XA83YSuG0Rs-EJjLfYkMDQ.png" /></figure><p>If organizing data is about laying the foundation, visualization is about building the front door — the part everyone sees and interacts with.</p><p><strong>Module 6</strong> made one thing crystal clear:<br> ✅ <em>Well-designed visuals don’t just show data — they tell a story, guide decisions, and create impact.</em></p><h3>🔁 Catch up on my previous blog:</h3><p><a href="https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-5-of-the-google-data-analytics-certificate-organizing-data-for-7bea43dd20b6">📂 What I Learned in Module 5 of the Google Data Analytics Certificate: Organizing Data for…</a></p><h3>🎯 Why Data Visualization Matters</h3><p>📌 Data is powerful — but only if people can understand it.</p><p>Great visuals help to:</p><ul><li>Simplify complex datasets</li><li>Reveal patterns, trends, and outliers</li><li>Communicate quickly across teams</li><li>Drive smarter, faster decisions</li></ul><h3>🧩 The McCandless Method</h3><p>One of the biggest takeaways was <strong>David McCandless’s framework</strong>:</p><blockquote><strong><em>Information → Story → Goal → Visual Form = Impact</em></strong></blockquote><p>This changed how I think about charts. It’s not about <em>what looks good</em> — it’s about <em>what moves the viewer to act.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7k6nbw_SKO35UfynlRUaw.png" /></figure><h3>📐 A 3-Question Visual Checklist</h3><p>Before making a chart, I ask:</p><ol><li>What’s the <strong>business question</strong>?</li><li>What does the <strong>data actually say</strong>?</li><li>What should the <strong>visual highlight</strong>?</li></ol><p>If I can’t answer all three, the chart isn’t ready.</p><h3>🖌️ Pre-Attentive Attributes: Design That Works Instantly</h3><p>Our brain reacts <em>automatically</em> to certain cues. These make or break visual clarity:</p><p><strong>Position : </strong>Most accurate (bar, line charts)</p><p><strong>Size : </strong>Show magnitude</p><p><strong>Color : </strong>Group or alert</p><p><strong>Shape : </strong>Differentiate categories</p><h3>📊 Chart Types — and When to Use Them</h3><p><strong>Bar — </strong>Comparing categories</p><p><strong>Line — </strong>Showing trends over time</p><p><strong>Pie — </strong>Part-to-whole (rarely ideal)</p><p><strong>Heat Map — </strong>Intensity or frequency</p><p><strong>Histogram — </strong>Distribution</p><p><strong>Scatter Plot — </strong>Correlations</p><p><strong>Gantt — </strong>Timelines</p><p><strong>Bubble — </strong>Comparing categories + values</p><p><strong>Box Plot — </strong>Spread and outliers</p><h3>🎨 Good Design = Better Storytelling</h3><p>Design isn’t just aesthetics — it’s <em>function</em>.</p><p>Follow these principles:</p><ul><li>🧼 <strong>Clarity</strong>: One idea per chart</li><li>🧲 <strong>Contrast</strong>: Use color/size to highlight</li><li>🎯 <strong>Consistency</strong>: Fonts, scales, units</li><li>🧍‍♀️ <strong>White space</strong>: Let visuals breathe</li></ul><blockquote><strong><em>The 5-Second Rule</em></strong><em>: If someone can’t understand your slide in under 5 seconds, redesign it.</em></blockquote><h3>🧠 Design Thinking in Visualization</h3><p>Use this 5-step framework to build human-centered visuals:</p><ol><li><strong>Empathize</strong> — Know your audience’s needs</li><li><strong>Define</strong> — Frame the core data problem</li><li><strong>Ideate</strong> — Explore visual solutions</li><li><strong>Prototype</strong> — Draft charts quickly</li><li><strong>Test</strong> — Get feedback and revise</li></ol><h3>♿ Accessibility Matters</h3><p>To keep visuals inclusive:</p><ul><li>Label everything (axes, titles, annotations)</li><li>Don’t rely on color alone</li><li>Use patterns/textures for colorblind users</li><li>Add alt text in reports and dashboards</li></ul><h3>🛠️ Tools of the Trade: Tableau Public</h3><p>Tableau was a standout in this module. It’s intuitive and powerful.</p><ul><li>Drag-and-drop charts</li><li>Live filters and interactivity</li><li>Gallery of pre-built dashboards</li><li>Easy publishing and sharing</li></ul><p><strong>Common Tableau Charts:</strong></p><ul><li>Heat Maps</li><li>Highlight Tables</li><li>Gantt Charts</li><li>Bullet Graphs</li><li>Filled Maps</li><li>Box Plots</li></ul><h3>📖 Data Storytelling 101</h3><p>You need more than clean data — you need a <strong>clear narrative</strong>.</p><blockquote><em>Combine </em><strong><em>data + visuals + storytelling = insight</em></strong></blockquote><p><strong>Story structure to follow:</strong></p><ol><li>Engage (Who’s the audience?)</li><li>Visualize (What supports your message?)</li><li>Narrate (What’s the key takeaway?)</li></ol><p>Use <em>spotlighting</em> (annotations, color, callouts) to guide the eye.</p><h3>🗣️ Presenting Charts with Impact</h3><p>Use the <strong>McCandless Walkthrough Method</strong>:</p><ol><li>Name the chart</li><li>Call out major insights</li><li>Explain the <em>why</em></li><li>State the impact</li><li>Recommend action</li></ol><p><strong>Pro Tip:</strong> Practice aloud. Focus on <em>why it matters</em>, not just <em>what it shows</em>.</p><h3>📝 Final Thoughts</h3><p>Module 6 reminded me that a great visualization isn’t decoration — it’s translation.</p><p>It’s how we bridge data and decision. It’s where analytics turns into action.</p><p>In a world full of dashboards, the visuals that <em>speak clearly</em> are the ones that <em>get remembered</em>.</p><p>Coming up next: Blog #7</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=941350216885" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ What I Learned in Module 5 of the Google Data Analytics Certificate: Organizing Data for…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-5-of-the-google-data-analytics-certificate-organizing-data-for-7bea43dd20b6?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bea43dd20b6</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Sat, 01 Nov 2025 17:49:52 GMT</pubDate>
            <atom:updated>2025-11-01T17:49:52.807Z</atom:updated>
            <content:encoded><![CDATA[<h3>📂 What I Learned in Module 5 of the Google Data Analytics Certificate: Organizing Data for Analysis</h3><p><strong>By Sreeja Edulapalle</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hS0k6PXaOM74UFdbB77L7g.png" /></figure><p>If data cleaning is about fixing the mess, organizing is about preventing it in the first place. Module 5 made one thing clear: <strong>organization is power</strong>. The more structured your data is at the start, the smoother your analysis will be. No guesswork, no chaos — just clean pathways to insight.</p><p>Here’s my breakdown of what stood out:</p><h3>Check out my previous blog here</h3><p><a href="https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-4-of-the-google-data-analytics-certificate-cleaning-validating-and-09095e466b08">🔍 What I Learned in Module 4 of the Google Data Analytics Certificate: Cleaning, Validating, and…</a></p><h3>🔄 The 4 Phases of Data Organization</h3><p>This module split data organization into four sharp phases. Each one builds on the last — skip one, and the whole thing can fall apart.</p><h3>1️⃣ Organize the Data</h3><p>Start with a raw data scan:</p><ul><li>Missing fields?</li><li>Duplicates?</li><li>Weird formats?</li></ul><p>Then:</p><ul><li>Set clear <strong>file names</strong> and <strong>folder structures</strong>.</li><li>Build a <strong>data dictionary</strong> to describe every column (it’s a life-saver later).</li></ul><h3>2️⃣ Format and Adjust</h3><p>Make everything uniform:</p><ul><li>Standardize dates, currency, and number formats.</li><li>Use spreadsheet tools like:</li><li>TRIM(), UPPER(), PROPER()</li><li>Conditional formatting to flag issues fast</li></ul><p>This is where raw turns into readable.</p><h3>3️⃣ Get Input from Others</h3><p>Data isn’t just yours. Collaborate:</p><ul><li>Check in with stakeholders and teammates.</li><li>Validate assumptions and fill in gaps.</li><li>Let others catch what you missed.</li></ul><h3>4️⃣ Transform the Data</h3><p>Now comes the action:</p><ul><li>Use formulas, pivot tables, and SQL queries.</li><li>Clean, combine, and filter until your data fits the question.</li></ul><p>Think of this phase as the bridge between raw data and insight-ready data.</p><h3>🧰 Tools That Make It All Work</h3><p>This module gave a full toolkit for getting data in shape:</p><p><strong>Spreadsheets</strong>:</p><ul><li>SORT, FILTER, VLOOKUP, LEN, CONCAT, VALUE, ROUND</li><li>Pivot Tables for dynamic summaries</li><li>Data Validation (dropdowns, checkboxes, constraints)</li></ul><p><strong>SQL</strong>:</p><ul><li>ORDER BY, WHERE, JOIN, GROUP BY</li><li>Aggregates: SUM(), AVG(), COUNT()</li><li>CAST() and SAFE_CAST() for type conversion</li><li>Subqueries, CASE, HAVING, temporary tables</li></ul><p><strong>Collaboration Tools</strong>:</p><ul><li>Google Sheets sharing, comments, and real-time edits</li><li>Connected Sheets (hello, BigQuery integration!)</li></ul><h3>🔎 Sorting, Filtering, and Validation — Why It Matters</h3><p>Sorting isn’t just cosmetic — it clarifies priorities.<br> Filtering helps you zoom in fast.<br> Validation keeps your data from going rogue.</p><p>Every formula or filter is about control — so your analysis doesn’t go off track.</p><h3>🧠 Data Calculations, Pivot Tables &amp; Lookup Magic</h3><p>This module also leaned into analysis prep with:</p><ul><li>Lookup functions: VLOOKUP(), INDEX(), MATCH()</li><li>Conditional logic: IF(), IFS(), AND(), OR()</li><li>Calculations like profit margin, weighted averages (SUMPRODUCT)</li><li>Aggregations via Pivot Tables or SQL queries</li></ul><h3>💡 Big Picture: Why Organizing Data Matters</h3><ul><li>It saves time</li><li>Prevents errors</li><li>Makes teamwork easier</li><li>Lays the groundwork for strong, ethical, bias-aware analysis</li></ul><p>The real insight? <strong>The way you organize your data defines the quality of the decisions you’ll make from it.</strong> Good data analysts don’t wing it — they systemize everything.</p><h3>📝 Final Thoughts</h3><p>Module 5 reminded me: cleaning is essential, but organizing is empowering. It turns messy rows into structured stories. Before you analyze, <strong>build your data house on solid ground.</strong></p><p>Coming up next: Blog #6</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bea43dd20b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ What I Learned in Module 4 of the Google Data Analytics Certificate: Cleaning, Validating, and…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-4-of-the-google-data-analytics-certificate-cleaning-validating-and-09095e466b08?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/09095e466b08</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Thu, 16 Oct 2025 16:33:54 GMT</pubDate>
            <atom:updated>2025-10-16T16:33:54.748Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔍 What I Learned in Module 4 of the Google Data Analytics Certificate: Cleaning, Validating, and Preparing Data</h3><p>By Sreeja Edulapalle</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZyXWo6SxczwURDfXFxMjg.png" /></figure><p>If Module 3 was about picking the right data, <strong>Module 4 dives into what happens next</strong> — cleaning, verifying, and prepping that data for real-world analysis. This is where the messy part of data work happens. And it matters — because even the best analysis means nothing if the data behind it is flawed.</p><p>Here’s my breakdown of what stood out in this module:</p><h3>Check out my previous blog here</h3><p><a href="https://medium.com/@SreejaEdulapalle/blog-3-what-i-learned-in-module-3-of-the-google-data-analytics-certificate-working-with-data-9e886b9f4794">🧩 Blog 3: What I Learned in Module 3 of the Google Data Analytics Certificate: Working with Data</a></p><h3>🧱 Data Integrity: Trust Starts Here</h3><p>Data integrity ensures that your data is <strong>accurate, consistent, and reliable</strong> over time. It’s enforced with:</p><ul><li>✅ Validation rules</li><li>✅ Constraints (like NOT NULL, UNIQUE)</li><li>✅ Access control</li><li>✅ Audit logs</li></ul><p>It’s not just about quality — it’s about trust.</p><h3>🔁 Key Concepts in Data Handling</h3><p><strong>Data Replication — </strong>Making copies across systems for backup or availability</p><p><strong>Data Transfer — </strong>Moving data through APIs, pipelines, etc.</p><p><strong>Data Manipulation — </strong>Changing data for use (filtering, updating, transforming)</p><p><strong>Data Constraints — </strong>Rules that enforce structure and integrity</p><h3>🎯 Align Data to Business Goals</h3><p>If the data doesn’t align with your business questions, the insights will be off. Clean, valid, and relevant data helps you avoid misleading conclusions and bad decisions.</p><h3>🧼 Data Cleaning: Problems &amp; Fixes</h3><p><strong>No data — </strong>Check source access and collection method</p><p><strong>Too little data — </strong>Extend time range or increase sample size</p><p><strong>Wrong data — </strong>Cross-check with source, apply correction logic</p><p><strong>Errors — </strong>Use functions like ISNUMBER(), LEN(), regex</p><h3>🔢 Sample Size &amp; Statistical Power</h3><p>A <strong>larger sample</strong> increases confidence, reduces error, and improves significance.</p><p>Your sample must be:</p><ul><li>Representative</li><li>Random (when possible)</li><li>Big enough to matter</li></ul><p><strong>Statistical power</strong> = the chance of detecting a real effect</p><blockquote><em>80% is the standard benchmark<br> It depends on:</em></blockquote><ul><li>Sample size</li><li>Effect size</li><li>Significance level</li><li>Variability</li></ul><h3>📐 Central Limit Theorem</h3><p>Even if your raw data is messy or skewed, the <strong>CLT</strong> says your sample means will form a <strong>normal distribution</strong> if the sample size is big enough (n ≥ 30). That’s a game-changer for analysis.</p><h3>🔍 Dirty Data: Types &amp; Fixes</h3><p>Dirty Data Type — What It Looks Like — Fix</p><p><strong>Duplicate — </strong>Same record twice — REMOVE DUPLICATES, fuzzy matching</p><p><strong>Outdated — </strong>Old info — Refresh sources</p><p><strong>Incomplete — </strong>Missing fields — Impute or flag</p><p><strong>Incorrect — </strong>Misspellings, wrong entries — Validation checks</p><p><strong>Inconsistent — </strong>“CA”, “California”, “Calif.” — Standardize using lookup tables</p><p>Best practices include:</p><ul><li>Validation at entry</li><li>Cleaning logs</li><li>ETL steps in your pipeline</li><li>Regular audits</li><li>Staff training</li></ul><h3>⚙️ Data Cleaning Tools</h3><p><strong>Excel/Sheets — </strong>Quick manual cleanups</p><p><strong>OpenRefine — </strong>Clustering, transformations</p><p><strong>Python (Pandas) — </strong>Scalable, programmatic cleaning</p><p><strong>R (tidyr/dplyr) — </strong>Statistical cleaning/wrangling</p><p><strong>SQL — </strong>Filtering, deduplication, updates</p><p><strong>Power Query — </strong>ETL within Excel/Power BI</p><p><strong>BigQuery/Snowflake — </strong>Cloud-scale cleaning</p><h3>🔗 Merging &amp; Mapping Data</h3><ul><li><strong>Merging</strong> = Combining datasets with joins (INNER, LEFT, RIGHT, FULL)</li><li><strong>Mapping</strong> = Aligning fields between sources (e.g., cust_id → customer_id)</li></ul><p>Don’t just merge — <strong>map carefully</strong> to avoid garbage in, garbage out.</p><h3>🧠 Build Your Cleaning Strategy</h3><ol><li>Know the <strong>business context</strong></li><li>Assess <strong>data quality</strong></li><li>Plan the <strong>cleaning approach</strong></li><li>Choose the <strong>right tools</strong></li><li>Apply <strong>fixes systematically</strong></li><li>Validate the cleaned data</li><li><strong>Document everything</strong></li></ol><h3>🛠️ SQL for Data Cleaning</h3><p>Remove duplicates — ROW_NUMBER(), DISTINCT</p><p>Handle nulls — IS NULL, COALESCE()</p><p>Fix formatting — UPPER(), TRIM(), REPLACE()</p><p>Conditional changes — CASE WHEN</p><p>Split strings — SUBSTRING(), SPLIT()</p><p>Type conversions — CAST(), SAFE_CAST()</p><p>SQL is <strong>scalable</strong>, <strong>automated</strong>, and <strong>auditable</strong> — a must for large data tasks.</p><h3>☁️ BigQuery Highlights</h3><p>Google’s BigQuery is a <strong>serverless warehouse</strong> that makes massive queries fast. It handles nested data, integrates with Sheets and Looker, and supports functions like:</p><ul><li>SAFE_CAST() — safe type conversions</li><li>ARRAY &amp; STRUCT — nested/semi-structured support</li><li>CASE WHEN, COALESCE(), SUBSTRING() — core cleaning functions</li></ul><h3>✅ Final Cleaning Checklist</h3><p>Before calling your data “ready,” check:</p><ul><li>❓ Nulls filled or flagged</li><li>🧹 Duplicates removed</li><li>🔡 Strings cleaned &amp; trimmed</li><li>📆 Dates formatted</li><li>🔠 Case standardized</li><li>✅ Validation applied</li><li>🔧 Data types corrected</li><li>📓 Log created &amp; documented</li><li>👀 Verified with business goals</li></ul><h3>🌍 Diversity in Data = Better Outcomes</h3><p>Diverse data teams bring broader perspectives. That matters for:</p><ul><li>Identifying bias in data</li><li>Interpreting results more fairly</li><li>Designing inclusive solutions</li><li>Ensuring ethical, respectful handling of sensitive information</li></ul><p>It’s not just good practice — it’s better data science.</p><h3>🌟 Final Thoughts</h3><p>This module hammered home one thing: <strong>data cleaning is non-negotiable</strong>. It’s not glamorous, but it’s foundational. Without clean, validated, well-documented data, your analysis is a guess at best.</p><p><strong>Module 4 taught me that great data analysts don’t just work with data — they <em>shape</em> it into something trustworthy.</strong></p><p>Stay tuned for Blog #5!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09095e466b08" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Blog 3: What I Learned in Module 3 of the Google Data Analytics Certificate: Working with Data]]></title>
            <link>https://medium.com/@SreejaEdulapalle/blog-3-what-i-learned-in-module-3-of-the-google-data-analytics-certificate-working-with-data-9e886b9f4794?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e886b9f4794</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 15:45:27 GMT</pubDate>
            <atom:updated>2025-10-16T15:45:43.810Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Sreeja Edulapalle</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PzExBkP_mcnIV9do45tdWA.png" /></figure><p>If Module 2 was all about structured thinking, Module 3 dives into the <strong>heart of data itself</strong> — how to select, structure, transform, and safeguard the information that powers analysis. I learned that before we even start analyzing, it’s crucial to ask: <em>Do I have the right data, and is it reliable?</em></p><p>Here’s my breakdown of what I learned in this module.</p><h4>Check out my previous blog here</h4><p><a href="https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-2-of-the-google-data-analytics-certificate-structured-thinking-in-3cec629e98b0">https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-2-of-the-google-data-analytics-certificate-structured-thinking-in-3cec629e98b0</a></p><p><strong>🎯 Selecting the Right Data</strong></p><p>Good analysis begins with asking the right questions:</p><ul><li>What decision am I supporting?</li><li>What question am I trying to answer?</li><li>Who is the audience or stakeholder?</li></ul><p>➡️ The right data is <strong>relevant, accurate, and timely</strong>.</p><p><strong>📂 Types of Data by Source</strong></p><p><strong>Primary Data — </strong>Collected directly for a purpose</p><p>Surveys, interviews, experiments</p><p><strong>Secondary Data — </strong>Pre-collected by others</p><p>Reports, public datasets, web scraping</p><p><strong>Internal Data — </strong>From within the organization</p><p>CRM logs, sales reports, financials</p><p><strong>External Data — </strong>From outside sources</p><p>Market trends, census, APIs</p><p><strong>🗂️ Types of Data by Structure</strong></p><p><strong>Structured — </strong>Organized into rows/columns</p><p>Excel, SQL tables</p><p><strong>Unstructured — </strong>No fixed format</p><p>Videos, emails, social media</p><p><strong>📏 Types of Data by Measurement Level</strong></p><p><strong>Nominal — </strong>Categories, no order</p><p>Gender, location</p><p><strong>Ordinal — </strong>Ordered categories</p><p>Survey ratings</p><p><strong>Discrete — </strong>Countable, whole numbers</p><p>Purchases, logins</p><p><strong>Continuous — </strong>Any value in a range</p><p>Revenue, time spent</p><p><strong>Qualitative — </strong>Describes attributes</p><p>Product type, sentiment</p><p><strong>Quantitative — </strong>Numerical measurements</p><p>Sales, profit</p><p>📌 Why it matters: How data is structured and measured impacts <strong>storage efficiency, query performance, scalability, and the quality of insights</strong>.</p><p><strong>🏗️ Levels of Data Modeling</strong></p><p>Data modeling connects business needs with technical systems:</p><ul><li><strong>Conceptual Model</strong> → High-level, “what data do we need?” (business focus)</li><li><strong>Logical Model</strong> → Attributes, keys, constraints (analyst/architect focus)</li><li><strong>Physical Model</strong> → Tables, indexes, partitions (engineer focus)</li></ul><p>Techniques include <strong>ERDs, UML diagrams, and Data Dictionaries</strong>.</p><p><strong>🔄 Wide vs. Long Data</strong></p><p><strong>Wide — </strong>One row per subject, variables spread across columns</p><p>Monthly sales in separate columns</p><p><strong>Long — </strong>Each observation has its own row</p><p>Month column + sales column</p><p>➡️ Wide is easier to read, but long is easier to analyze in tools like R, Python, or Power BI.</p><p><strong>🔧 Data Transformation</strong></p><p>Transformation = <strong>cleaning, modifying, or restructuring data</strong> to make it usable.</p><p>Goals include:</p><ul><li>Organization 🗂️</li><li>Compatibility 🔗</li><li>Migration ↔️</li><li>Merging 📊</li><li>Enhancement ✨</li><li>Comparison 📈</li></ul><p><strong>⚖️ Data Bias &amp; Source Quality</strong></p><p>Not all data is neutral.</p><ul><li><strong>Biased data</strong> skews insights; <strong>unbiased data</strong> represents reality fairly.</li><li>Common bias types: <strong>sampling, observer, interpretation, confirmation</strong>.</li></ul><p>✅ Use the <strong>ROCCC framework</strong> to test data sources: <strong>Reliable, Original, Comprehensive, Current, Cited</strong>.</p><p><strong>🔐 Data Ethics &amp; Privacy</strong></p><p>Ethical handling of data is non-negotiable.</p><ul><li><strong>Principles</strong>: Ownership, Transparency, Consent, Currency, Privacy, Openness.</li><li><strong>PII (Personally Identifiable Information)</strong> like names, SSNs, or biometrics must be protected.</li><li><strong>Anonymization</strong> (irreversible) vs. <strong>De-identification</strong> (reversible).</li><li>Follow laws like <strong>GDPR, HIPAA, CCPA</strong>.</li></ul><p><strong>🌍 Open &amp; Third-Party Data</strong></p><ul><li><strong>Open Data</strong> → Accessible, machine-readable, openly licensed (e.g., government, research institutions).</li><li><strong>Third-Party Data</strong> → From vendors or platforms (e.g., Google Ads). Must be reviewed for <strong>accuracy, legality, and ethics</strong>.</li></ul><p><strong>🗄️ Databases &amp; Metadata</strong></p><ul><li><strong>Databases</strong> store and manage structured information (Relational vs. NoSQL).</li><li>Core relational concepts: <strong>Primary Keys, Foreign Keys, Composite Keys</strong>.</li><li><strong>Metadata = Data about data</strong> → helps with discovery, governance, and lineage.</li><li>Stored in <strong>metadata repositories/data catalogs</strong> (e.g., Collibra, Alation).</li></ul><p><strong>📊 Working with Google Sheets &amp; SQL</strong></p><ul><li>Import functions like IMPORTDATA, IMPORTRANGE, IMPORTHTML, IMPORTXML, GOOGLEFINANCE.</li><li><strong>Sorting &amp; Filtering</strong> for quick exploration.</li><li><strong>Databases vs. Spreadsheets</strong>: scale and structure differences.</li><li><strong>BigQuery</strong> for massive, cloud-scale analytics.</li><li>SQL basics: SELECT, WHERE, GROUP BY, JOIN.</li></ul><p><strong>🗃️ File Organization &amp; Security</strong></p><ul><li>Use <strong>clear naming conventions</strong>, ISO dates, and version control.</li><li>Balance <strong>analytics with security</strong>:</li><li><strong>Encryption</strong> (reversible) → protects data in storage or transit.</li><li><strong>Tokenization</strong> (irreversible) → protects sensitive fields for analytics.</li></ul><p><strong>🌟 Final Thoughts</strong></p><p>This module made me realize: <em>choosing, structuring, and protecting data is just as important as analyzing it.</em> Without clean, reliable, ethical data, analysis won’t create business value.</p><p>Module 3 gave me a strong foundation in <strong>data quality, structure, and ethics</strong> — tools I’ll carry into every future project.</p><p>Stay tuned for Blog #4!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e886b9f4794" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ What I Learned in Module 2 of the Google Data Analytics Certificate: Structured Thinking in…]]></title>
            <link>https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-2-of-the-google-data-analytics-certificate-structured-thinking-in-3cec629e98b0?source=rss-a3e523dc0dee------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cec629e98b0</guid>
            <dc:creator><![CDATA[Sreeja Edulapalle]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 16:45:42 GMT</pubDate>
            <atom:updated>2025-10-16T15:43:04.248Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>🧩 What I Learned in Module 2 of the Google Data Analytics Certificate: Structured Thinking in Action</strong></p><p><strong><em>By Sreeja Edulapalle</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*87rjgh57N5-msd5eWMxgug.png" /></figure><p><strong>When I first heard the term “structured thinking,” it sounded like just another buzzword. But in Module 2 of the Google Data Analytics Professional Certificate, I realized it’s one of the most important skills a data analyst can have.</strong></p><p><strong>Structured thinking isn’t about making things more complicated — it’s about breaking big, messy problems into smaller, solvable pieces so that the answers are clear, logical, and useful.</strong></p><p><strong>Here’s my breakdown of what I learned about structured thinking and how it applies to data analytics.</strong></p><h4>Check out my previous blog here</h4><p><a href="https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-1-of-the-google-data-analytics-certificate-foundations-of-data-d7c734b42931">https://medium.com/@SreejaEdulapalle/what-i-learned-in-module-1-of-the-google-data-analytics-certificate-foundations-of-data-d7c734b42931</a></p><p><strong>🧠 What Is Structured Thinking?</strong></p><p><strong>Structured thinking is a logical, organized approach to problem-solving.</strong></p><p><strong>It helps analysts:</strong></p><ul><li><strong>Define problems clearly.</strong></li><li><strong>Break them into smaller, testable parts.</strong></li><li><strong>Use data to create insights that lead to action.</strong></li></ul><p><strong>Instead of jumping straight into spreadsheets, structured thinking ensures the analysis stays focused on business goals and outcomes.</strong></p><p><strong>🔑 Key Elements of Structured Thinking</strong></p><ol><li><strong>Define the Problem Clearly</strong></li></ol><ul><li><strong>Ask “What exactly are we solving for?”</strong></li><li><strong>Use frameworks like <em>What, Why, How</em>.</strong></li></ul><p><strong>2. Break It Into Smaller Parts</strong></p><ul><li><strong>Decompose problems into chunks.</strong></li><li><strong>This avoids vague, unmanageable analysis.</strong></li></ul><p><strong>3. Use Frameworks</strong></p><ul><li><strong><em>MECE</em> (Mutually Exclusive, Collectively Exhaustive)</strong></li><li><strong><em>5 Whys</em> for root cause analysis</strong></li><li><strong>SWOT, Funnel Analysis, etc.</strong></li></ul><p><strong>4. Prioritize &amp; Sequence</strong></p><ul><li><strong>Not every question matters equally. Start with the ones that give the most impact.</strong></li></ul><p><strong>5. Think From the End Goal</strong></p><ul><li><strong>Keep the bigger picture in sight. Insights must align with business needs.</strong></li></ul><p><strong>6. Use Visual Aids</strong></p><ul><li><strong>Mind maps, charts, and flow diagrams help communicate thinking clearly.</strong></li></ul><p><strong>📊 The Six Types of Data Problems Analysts Face</strong></p><p><strong>Structured thinking also means recognizing the <em>kind of problem you’re solving</em>. Module 2 outlined six recurring types:</strong></p><ol><li><strong>Making Predictions — Forecasting sales, churn, or demand.</strong></li><li><strong>Characterizing Things — Profiling customers or products.</strong></li><li><strong>Spotting Unusual Events — Fraud detection or anomaly spotting.</strong></li><li><strong>Identifying Themes — Sentiment analysis or survey insights.</strong></li><li><strong>Discovering Connections — Correlations, market basket analysis.</strong></li><li><strong>Finding Patterns — Clustering, trends, recurring behaviors.</strong></li></ol><p><strong>Knowing the problem type helps you choose the right methods and tools.</strong></p><p><strong>🎯 Asking SMART Questions</strong></p><p><strong>A big part of structured thinking is asking better questions. I learned the SMART framework to keep questions focused and effective:</strong></p><ul><li><strong>S — Specific: Narrow and clear.</strong></li><li><strong>M — Measurable: Can be answered with data.</strong></li><li><strong>A — Action-Oriented: Leads to decisions.</strong></li><li><strong>R — Relevant: Tied to business goals.</strong></li><li><strong>T — Time-Bound: Anchored in a timeframe.</strong></li></ul><p><strong>Example: Instead of asking <em>“How’s our performance?”</em> ask <em>“How did our Q2 email campaign affect new user signups?”</em></strong></p><p><strong>🧩 Beyond Numbers: Context, Communication &amp; Stakeholders</strong></p><p><strong>Structured thinking isn’t just technical — it’s about people too.</strong></p><ul><li><strong>Work with Stakeholders: Understand their goals, translate data into business value.</strong></li><li><strong>Ask the 6Ws: What, Who, When, How, Where, Why.</strong></li><li><strong>Communicate Clearly: Always consider your audience, what they know, and what they need.</strong></li><li><strong>Balance Speed vs. Accuracy: Early insights can be shared, but clarity about limitations is key.</strong></li></ul><p><strong>🛠️ Tools That Bring Structure to Life</strong></p><ul><li><strong>Pivot Tables: Summarize, group, and explore data quickly.</strong></li><li><strong>Dashboards: Strategic, operational, and analytical dashboards organize insights for different needs.</strong></li><li><strong>Frameworks for Big Data: Volume, Variety, Velocity, Veracity (the 4 V’s).</strong></li><li><strong>Spreadsheet Techniques: Absolute referencing, data ranges, and combining functions reduce errors.</strong></li></ul><p><strong>🚧 Dealing With Real-World Data</strong></p><p><strong>Structured thinking also prepares you for messy realities:</strong></p><ul><li><strong>Imperfect Data: Missing, misaligned, or dirty datasets.</strong></li><li><strong>Error Handling: Knowing how to spot and fix common spreadsheet errors (#DIV/0!, #REF!, etc.).</strong></li><li><strong>Data Storytelling: Turning insights into a clear narrative: Problem → Analysis → Insight → Recommendation.</strong></li></ul><p><strong>🌟 Final Thoughts</strong></p><p><strong>If Module 1 was about <em>understanding the data ecosystem,</em> Module 2 is about <em>thinking smarter with data.</em></strong></p><p><strong>Structured thinking is the skill that turns raw data into real business value. It’s what helps analysts avoid rabbit holes, ask better questions, and present insights in a way that actually drives action.</strong></p><p><strong>Stay tuned for Blog #3!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cec629e98b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>