<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zoe Mousmoula on Medium]]></title>
        <description><![CDATA[Stories by Zoe Mousmoula on Medium]]></description>
        <link>https://medium.com/@zoemousmoula?source=rss-8ee0572a9c70------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*h5CekaBwdeAnccQqqUmPVQ.png</url>
            <title>Stories by Zoe Mousmoula on Medium</title>
            <link>https://medium.com/@zoemousmoula?source=rss-8ee0572a9c70------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:29:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zoemousmoula/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ How to Perform Customer Sentiment Analysis in Minutes (Without Being an ML Engineer)]]></title>
            <link>https://medium.com/@zoemousmoula/how-to-perform-customer-sentiment-analysis-in-minutes-without-being-an-ml-engineer-2221ea0d05fb?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/2221ea0d05fb</guid>
            <category><![CDATA[sagemaker]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 13:56:00 GMT</pubDate>
            <atom:updated>2025-07-08T13:56:00.923Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*omdAhBxfeyLA9Du3.gif" /><figcaption>AWS Comprehend</figcaption></figure><blockquote><em>Turn your raw customer feedback into actionable insights with just a few lines of code, all using Amazon SageMaker and AWS Comprehend.</em></blockquote><h3>How Raw Feedback Becomes Business Strategy</h3><p>Imagine you’re launching a new product line. Wouldn’t it be helpful to know instantly what customers love — or hate — about it? With this pipeline, you can monitor sentiment from day one and quickly adjust your marketing, customer support, or product design strategies.</p><p>In today’s AI-driven world, you no longer need to be a machine learning engineer to extract value from customer data. Tools like <strong>Amazon SageMaker</strong> and <strong>AWS Comprehend</strong> allow data analysts, product teams, and even non-technical stakeholders to perform advanced natural language processing (NLP) tasks in just a few minutes.</p><p>That doesn’t mean we replace ML engineers, quite the opposite. They’re solving increasingly complex modeling, optimization, and deployment problems. But when it comes to lightweight use cases like <strong>sentiment analysis</strong>, AWS gives us plug-and-play solutions to get to insights fast.</p><p>In this walkthrough, I’ll show how I performed sentiment analysis on real customer reviews using:</p><ul><li><strong>Amazon SageMaker</strong> (Jupyter notebooks environment)</li><li><strong>Amazon Comprehend</strong> (for pre-trained sentiment classification)</li><li><strong>Matplotlib / Seaborn</strong> (for data visualization)</li></ul><p>And I’ll do all of this without building or training a single machine learning model.</p><h3>Dataset Overview</h3><p>To explore real-world sentiment, I used the<a href="https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews"> <strong>Women’s Clothing E-commerce Reviews dataset</strong></a><strong> </strong>from Kaggle, which is also available directly from AWS.</p><p>It contains <strong>23,486 reviews</strong>, each with structured and unstructured features:</p><ul><li><strong>Clothing ID</strong>: Categorical ID representing the specific product reviewed</li><li><strong>Age</strong>: Age of the reviewer</li><li><strong>Title</strong>: Short summary or title of the review</li><li><strong>Review Text</strong>: Full body of the customer’s written review</li><li><strong>Rating</strong>: Product rating given by the customer, from 1 (worst) to 5 (best)</li><li><strong>Recommended IND</strong>: Binary indicator of whether the reviewer recommends the product (1 = yes, 0 = no)</li><li><strong>Positive Feedback Count</strong>: Number of other users who marked the review as helpful</li><li><strong>Division Name</strong>: High-level product category (e.g., General, Petite)</li><li><strong>Department Name</strong>: Product department name (e.g., Tops, Dresses)</li><li><strong>Class Name</strong>: More granular product class name (e.g., Blouses, Knit Tops)</li></ul><h4>1. Download and Prepare the Dataset on AWS SageMaker</h4><p>This dataset is publicly hosted on AWS and can be copied directly into your SageMaker notebook instance using the following command:</p><pre>!aws s3 cp s3://dlai-practical-data-science/data/raw/womens_clothing_ecommerce_reviews.csv ./womens_clothing_ecommerce_reviews.csv</pre><p>We then read the data using Pandas:</p><pre>import pandas as pd<br><br>df = pd.read_csv(‘./womens_clothing_ecommerce_reviews.csv’, index_col=0)<br>df = df[[‘Review Text’, ‘Rating’]].dropna()<br>df = df.rename(columns={‘Review Text’: ‘review_body’, ‘Rating’: ‘star_rating’})<br>df</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*Z_QFbwEE3L5hRYkOxGtm6g.png" /></figure><h4>2. Define Ground Truth from Star Ratings</h4><p>Before using AWS Comprehend, I first defined a “ground truth” label based on star ratings. This is essential so that at the end of the process, we can objectively evaluate the <strong>accuracy</strong> of the sentiment analysis model:</p><ul><li>Ratings 4–5 → POSITIVE</li><li>Rating 3 → NEUTRAL</li><li>Ratings 1–2 → NEGATIVE</li></ul><pre>def to_sentiment(rating):<br>if rating in [1, 2]: return ‘NEGATIVE’<br>elif rating == 3: return ‘NEUTRAL’<br>else: return ‘POSITIVE’<br><br>df[‘ground_truth_sentiment’] = df[‘star_rating’].apply(to_sentiment)<br>df</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*6PLaz4tcV63VKufFnVecyA.png" /></figure><h4>3. Analyze Text Sentiment Using AWS Comprehend</h4><p>We use the <strong>pre-trained sentiment model</strong> from AWS Comprehend to analyze review text.</p><pre>import boto3<br><br>comprehend = boto3.client(&#39;comprehend&#39;, region_name=&#39;eu-west-1&#39;)<br><br>def detect_sentiment(text):<br>    try:<br>        if not isinstance(text, str) or text.strip() == &#39;&#39;:<br>            return &#39;ERROR&#39;<br>        if len(text.encode(&#39;utf-8&#39;)) &gt; 4999:<br>            return &#39;TOO_LONG&#39;<br>        response = comprehend.detect_sentiment(Text=text, LanguageCode=&#39;en&#39;)<br>        return response[&#39;Sentiment&#39;]<br>    except Exception:<br>        return &#39;ERROR&#39;<br><br># For test/demo purposes (due to Comprehend quotas)<br>df_sample = df.sample(n=1000, random_state=42).copy()<br>df_sample[&#39;comprehend_sentiment&#39;] = df_sample[&#39;review_body&#39;].apply(detect_sentiment)<br>df_sample</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*5EEVs7qbLvGRhw-NgQrBjw.png" /></figure><h4>4. Evaluate Comprehend’s Accuracy</h4><p>We compare Comprehend predictions with the ground_truth_sentiment:</p><pre>from sklearn.metrics import classification_report, confusion_matrix<br><br># Remove &#39;MIXED&#39; or &#39;ERROR&#39; results<br>df_valid = df_sample[~df_sample[&#39;comprehend_sentiment&#39;].isin([&#39;MIXED&#39;, &#39;ERROR&#39;])].copy()<br><br># Align formats<br>y_true = df_valid[&#39;ground_truth_sentiment&#39;].astype(str).values<br>y_pred = df_valid[&#39;comprehend_sentiment&#39;].astype(str).values<br><br># Print results<br>print(classification_report(y_true, y_pred))<br>print(&quot;Confusion Matrix:\n&quot;, confusion_matrix(y_true, y_pred))</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*KGlKhWYnUIoZGqkNDasG0A.png" /></figure><h3>🔍 Results</h3><p>In a business setting, these insights could power:</p><ul><li><strong>CX dashboards</strong> that track sentiment by product or season</li><li><strong>Automated routing of complaints</strong> to customer success teams</li><li><strong>A/B testing validation</strong> by comparing sentiment across variants</li><li><strong>Comprehend performs very well on POSITIVE sentiment</strong> (precision 0.99)</li><li>Accuracy dips for NEGATIVE/NEUTRAL (fewer examples)</li><li>Overall weighted F1 score: <strong>0.92</strong></li></ul><h4>Visualize Customer Sentiment</h4><p>Finally, let’s visualize sentiment distribution across all reviews:</p><pre>import matplotlib.pyplot as plt<br>import seaborn as sns<br><br>sentiment_counts = df[&#39;ground_truth_sentiment&#39;].value_counts()<br><br>plt.figure(figsize=(6, 4))<br>sns.barplot(x=sentiment_counts.index, y=sentiment_counts.values, palette=&#39;pastel&#39;)<br>plt.title(&#39;Customer Sentiment Distribution&#39;)<br>plt.xlabel(&#39;Sentiment&#39;)<br>plt.ylabel(&#39;Number of Reviews&#39;)<br>plt.tight_layout()<br>plt.show()</pre><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*dZqTAzW39FdwZgKbhlv9dA.png" /></figure><p>Most reviews are <strong>positive</strong>, but a clear share of negative and neutral opinions exist, signaling to product and marketing teams.</p><h3>What This Means for Data Analysts</h3><p>Organizations can use this workflow to:</p><ul><li>Reduce manual review overhead by 80%</li><li>Flag negative feedback within minutes</li><li>Improve product ratings by rapidly iterating on insights from real users</li></ul><p>What you’ve seen here is <strong>real NLP on real data</strong>, executed in minutes using the AWS ecosystem. You don’t need to train or fine-tune models. AWS has done the hard work for you.</p><p>If you’re a:</p><ul><li><strong>Data analyst</strong> looking to surface insights</li><li><strong>Product manager</strong> monitoring customer feedback</li><li><strong>Developer</strong> building review dashboards</li></ul><p>…this workflow is for you.</p><h3>🛠 Tools Recap</h3><ul><li><strong>Amazon SageMaker Notebooks</strong></li><li><strong>AWS Comprehend</strong> (pre-trained NLP)</li><li><strong>pandas / seaborn</strong> (data wrangling + viz)</li></ul><p>With a few Python commands, your business can unlock the voice of the customer and actually listen.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2221ea0d05fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Scalable Data Platform]]></title>
            <link>https://medium.com/@zoemousmoula/building-a-scalable-data-platform-7faa7672b283?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/7faa7672b283</guid>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Sat, 21 Jun 2025 20:52:41 GMT</pubDate>
            <atom:updated>2025-06-21T20:52:41.380Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A Blueprint for Federated Analytics and Machine Learning</em></p><h3>⚠️ Why Most Data Platforms Break</h3><p>Before we talk solutions, let’s highlight what goes wrong:</p><ul><li>❌ <strong>Central bottlenecks</strong>: 1 team, 10+ departments</li><li>❌ <strong>Brittle ETLs</strong>: schema change = broken pipeline</li><li>❌ <strong>Unclear ownership</strong>: nobody knows who owns what</li><li>❌ <strong>Trust issues</strong>: outdated dashboards</li><li>❌ <strong>ML headaches</strong>: CSV exports for training sets</li></ul><blockquote>The fix? A federated, cloud-native architecture inspired by <strong>Data Mesh principles</strong>.</blockquote><h3>🧭 Core Principles of the Architecture</h3><p><strong>✅ Federated Data Ownership</strong><br>Each domain team owns their data. The platform enforces governance and access.</p><p><strong>✅ Modular Pipelines</strong><br>Built with dbt and Airflow, versioned in Git, deployed with CI/CD.</p><p><strong>✅ Observability from Day One</strong><br>Lineage, freshness, SLAs tracked via DataHub, Great Expectations, and Grafana.</p><p><strong>✅ Auto-Scaling Compute</strong><br>Powerful, elastic performance using Snowflake Virtual Warehouses.</p><p><strong>✅ ML-Native by Design</strong><br>Supports feature stores, model tracking, and real-time inference pipelines.</p><p><strong>✅ Built-In Governance</strong><br>Role-based access, data catalogs, and metadata sync.</p><h3>🧱Data Storage</h3><p>This architecture follows the <strong>Medallion Architecture</strong>:<br>(Bronze → Silver → Gold)</p><h4>Bronze &amp; Silver in S3 (Parquet)</h4><ul><li><strong>Bronze</strong>: Raw ingested data (Kafka, Fivetran)</li><li><strong>Silver</strong>: Cleaned &amp; enriched domain data</li></ul><p>✅ Optimized for batch/stream<br>✅ Schema evolution support<br>✅ Cost-effective + audit-friendly</p><p>Partitioned by date, tenant, and region for performance and CDC.</p><h4>Gold Layer in Snowflake</h4><ul><li>Curated data marts by domain: mart_finance, mart_sales</li><li>Dimensional modeling with dbt (e.g., fact_sales, dim_customer)</li><li>High concurrency, BI-ready, ML-friendly</li></ul><p>✅ Auto-scaling, secure, fast<br>✅ Native integrations (Tableau, SageMaker, etc.)</p><blockquote><em>❗ Why not keep everything in S3?<br> Because </em><strong><em>analysts and ML workflows need fast, governed access</em></strong><em>.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LxhTGbYItWEbmLPI" /></figure><h3>🏗️ End-to-End Platform Architecture</h3><figure><img alt="Architecture Diagram" src="https://cdn-images-1.medium.com/max/600/1*Rg2_bNxehH3i2Qm31roanw.png" /></figure><h3>🔄 From Raw to Refined: What Happens Between the Layers</h3><p>Each layer adds structure and value as data moves through the platform.</p><h4>Bronze → Silver (S3 → S3)</h4><ul><li><strong>Input:</strong> Raw data lands in Parquet format (Kafka, Fivetran, etc.)</li><li><strong>Process:</strong> Type casting, deduplication, null handling, enrichment</li><li><strong>Tools:</strong> <strong>dbt</strong> + <strong>Airflow</strong>, with optional <strong>Lambda</strong> functions</li><li><strong>Output:</strong> Cleaned, partitioned Parquet files</li><li><strong>Validated:</strong> dbt tests, versioned in Git, CI/CD enforced</li></ul><blockquote><em>This creates a trusted, consistent base for analytics.</em></blockquote><h4>Silver → Gold (S3 → Snowflake)</h4><ul><li><strong>Input:</strong> Cleansed domain tables from Silver</li><li><strong>Process:</strong> Dimensional modeling (facts/dims), aggregations, KPI logic</li><li><strong>Tools:</strong> dbt + Snowflake materializations</li><li><strong>Output:</strong> Analytics-ready marts, per domain</li><li><strong>Use Cases:</strong> Dashboards, ML features, APIs</li></ul><blockquote><em>Gold is optimized for performance, security, and business trust.</em></blockquote><h3>🔎 Data Serving &amp; Use Cases</h3><p>The <strong>Gold layer in Snowflake</strong> powers:</p><ul><li>Dashboards: <strong>Tableau, Power BI, Metabase</strong></li><li>Ad-hoc analytics: <strong>SQL notebooks, AI tools</strong></li><li>ML training sets: <strong>Feast, S3 exports, SageMaker</strong></li><li>Expose data via APIs</li><li>Anomaly monitoring &amp; KPI alerts</li></ul><h3>📚 Metadata, Governance, and Quality Built-In</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*MpT1a5_eSkJmrSG2ZLi3WA.png" /></figure><h3>🧠 From Analytics to ML (Optional, Ready Anytime)</h3><p>Supports full ML lifecycle:</p><ul><li>Real-time inference pipelines</li><li>Export training sets</li><li>Feature stores via Feast (Snowflake or Redis)</li><li>Model tracking with MLflow / W&amp;B</li></ul><h3>🧰 DevOps &amp; Infra as Code (IaC)</h3><p>Provision everything with <strong>Terraform + GitHub Actions</strong>:</p><ul><li>Snowflake roles/schemas</li><li>Airflow DAGs</li><li>S3 buckets + policies</li><li>Kafka configs</li><li>Monitoring stack</li><li>CI/CD validation for every PR</li></ul><h3>💸 Cost Considerations</h3><p><strong>S3 (Bronze &amp; Silver layers)</strong>:<br>Cheap, durable storage, pay only for data stored. No compute cost.</p><p><strong>Snowflake (Gold layer)</strong>:<br>Pay-per-second compute, auto-scales up/down. Use small warehouses for light workloads, and isolate large ones for heavy queries.</p><p><strong>dbt + Airflow</strong>:<br>Run on-demand. No cost when idle. Can use open-source or managed (dbt Cloud/Astronomer), depending on team size.</p><p><strong>Kafka / Airbyte</strong>:<br>Kafka is infra-heavy; use only where needed. Airbyte scales with usage.</p><p><strong>Observability Tools (e.g., DataHub, Grafana)</strong>:<br>Mostly open-source. Managed versions add cost but reduce ops overhead.</p><p><strong>ML Tools (Sagemaker, Feast, MLflow)</strong>:<br>Use only when needed. Feast with Redis/Snowflake scales linearly with features/requests.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7faa7672b283" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ Customer LTV Forecasting Engine — Modular, AI-Ready, Serverless]]></title>
            <link>https://medium.com/@zoemousmoula/customer-ltv-forecasting-engine-modular-ai-ready-serverless-2f4ee8156c11?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f4ee8156c11</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[customer-lifetime-value]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Thu, 12 Jun 2025 12:47:59 GMT</pubDate>
            <atom:updated>2025-06-12T12:52:48.230Z</atom:updated>
            <content:encoded><![CDATA[<h3>📦 Customer LTV Forecasting Engine — Modular, AI-Ready, Serverless</h3><blockquote>“What’s this customer really worth?”<em><br> That single question can change how your business spends, grows, and retains.</em></blockquote><p>In this post, I’ll walk you through how I built a <strong>Customer Lifetime Value (LTV) Forecasting Engine</strong>. This fully serverless and AI-ready pipeline helps us predict <strong>future revenue</strong> from each user.</p><h3>What is Customer LTV?</h3><p>Customer Lifetime Value (LTV) is one of the most important metrics in any business. It gives you a sense of how much <strong>revenue</strong> a customer is expected to bring over their entire relationship with your company.</p><h3>Why Predict Customer LTV?</h3><p>Customer LTV is more than just a marketing metric. When done right, it powers:</p><ul><li><strong>Smarter acquisition</strong> — Spend more to acquire high-LTV users.</li><li><strong>Better personalization</strong> — Tailor product experiences based on user value.</li><li><strong>Accurate forecasting</strong> — Align product, sales, and finance around data.</li><li><strong>Resource prioritization</strong> — Invest where the long-term payoff is highest.</li></ul><h3>🧩 Architecture Overview</h3><p>Here’s what I built:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*Js-ds4ypMu9hG7ELeRB1Iw.png" /><figcaption>Serverless Architecture Diagram</figcaption></figure><p>This pipeline allows us to go from <strong>raw customer data → predicted value</strong> in a fully automated and retrainable loop.</p><p>It lets us predict user value at scale with <strong>minimal ops</strong> and <strong>maximum flexibility</strong>.</p><h3>1. Data Modeling with dbt</h3><p>I started by organizing the raw customer data into well-defined models using dbt, <strong>connecting to S3 via Athena</strong>:</p><ul><li><strong>Events</strong>: signup, purchases, logins, session activity</li><li><strong>Attributes</strong>: device type, referrer, user cohort</li><li><strong>Aggregates</strong>: avg order value, revenue per user, session count, avg time on site</li></ul><p>Data was transformed and stored back in <strong>S3</strong>, ready for downstream feature generation and modeling. Using <strong>dbt with Athena</strong> allowed me to reduce storage and compute costs by eliminating the need for Redshift.</p><h3>2. Feature Engineering</h3><p>I used a combination of <strong>SQL</strong> and <strong>Python</strong> scripts to convert modeled data into <strong>ML-friendly features</strong>:</p><ul><li>RFM scores (recency, frequency, monetary value)</li><li>Time since last purchase or session</li><li>Device usage patterns</li><li>Clickstream sequence stats (avg session depth, funnel drop-off)</li></ul><p>These features were stored as Parquet files in S3 and used for both training and scoring.</p><h3>3. ML Model: Forecasting LTV</h3><p>Model training was done using <strong>Amazon SageMaker</strong>, which allowed us to scale compute resources as needed, track experiments, and easily deploy models into production.</p><p>I trained a <strong>regression model</strong> to forecast LTV over 30, 90, and 180-day periods.</p><ul><li><strong>Model type:</strong> XGBoost (fast, explainable, handles missing values well)</li><li><strong>Target variable:</strong> total revenue per user over the time horizon</li><li><strong>Training cadence:</strong> monthly, with performance monitoring</li></ul><h4>Model Performance</h4><p>After training the model in SageMaker, I exported it to a model registry and deployed it to a<strong> real-time endpoint</strong>. I also evaluated its predictive power:</p><ul><li>RMSE and MAE on holdout sets</li><li>Decile analysis to measure ROI between the top and bottom predicted segments</li><li>Feature importance to understand drivers (e.g., early activity was a strong predictor)</li></ul><p>My goal wasn’t perfection, but <strong>operational accuracy</strong>. Even a 10% gain in targeting high-LTV customers leads to significant ROI.</p><h3>4. Orchestration with Prefect</h3><p>All jobs were scheduled and monitored using Prefect:</p><ul><li>Hourly/daily <strong>feature extraction jobs</strong></li><li>Monthly <strong>model retraining jobs</strong></li><li>Real-time and batch <strong>scoring pipelines</strong>, storing predictions in <strong>S3 or DynamoDB</strong></li></ul><p>Prefect offered flexibility, good observability, and reduced the ops overhead of managing infrastructure-heavy alternatives like Airflow.</p><h3>5. Real-Time Inference API</h3><p>To serve predictions in real-time, I deployed an <strong>API Gateway + AWS Lambda</strong> endpoint that interacts with a <strong>SageMaker-hosted model</strong>:</p><ul><li><strong>API Gateway</strong>: Exposes a secure HTTP endpoint to external systems.</li><li><strong>Lambda</strong>: Accepts lightweight JSON input (e.g., user_id), retrieves or computes required features, and sends them to the SageMaker endpoint.</li><li><strong>SageMaker Endpoint</strong>: Loads the pre-trained model, performs inference, and returns the predicted LTV.</li><li><strong>Response</strong>: Lambda formats the prediction and returns it via API Gateway. Results can also be saved to <strong>DynamoDB</strong> or <strong>S3</strong> for downstream use.</li></ul><p>This setup decouples serving logic from model infrastructure and allows us to scale each component independently while keeping latency <strong>under 300ms</strong>.</p><h3>🔧 Monitoring &amp; Logging</h3><p>With the pipeline running fully serverless, observability is critical. Here’s how I ensure reliability across the system:</p><p><strong>Prefect</strong>:</p><ul><li>Monitors task execution and retries failed jobs automatically.</li><li>Provides a UI for visualizing DAGs and real-time status.</li><li>Sends alerts via email on job failures.</li></ul><p><strong>SageMaker</strong>:</p><ul><li>Tracks model metrics like training loss, inference latency, and invocation counts through <strong>CloudWatch</strong>.</li><li>Logs are centralized for both training jobs and deployed endpoints.</li><li>Endpoint health is monitored for availability and cold-start behavior.</li></ul><p><strong>AWS Lambda + API Gateway</strong>:</p><ul><li><strong>API Gateway</strong> provides request-level logs, throttling alerts, and latency metrics.</li><li><strong>Lambda logs</strong> (including errors and input payloads) are streamed to <strong>CloudWatch Logs</strong>, with alarms set on failure rates or timeouts.</li></ul><p><strong>Batch Jobs &amp; Audits</strong>:</p><ul><li>All batch predictions are versioned and logged in <strong>DynamoDB</strong>, making it easy to trace model output history by timestamp and cohort.</li></ul><h3>📈 Example Use Cases</h3><ul><li><strong>Paid acquisition:</strong> LTV scores used to optimize ad bids</li><li><strong>CRM campaigns:</strong> Segmenting users by value</li><li><strong>Product experiments:</strong> Serving different flows based on predicted LTV</li><li><strong>Finance:</strong> Accurate revenue forecasts by cohort</li></ul><h3>🧪 Key Learnings</h3><p>Some things learned the hard way:</p><ol><li><strong>Start simple</strong>. Your first model doesn’t need to be deep learning. A good regression model beats no model.</li><li><strong>Feature freshness matters</strong>. LTV changes over time; make sure your features are updated often.</li><li><strong>Business context is everything</strong>. Know how your model will be used, accuracy isn’t enough.</li></ol><h3>🔁 TL;DR</h3><p>I built a <strong>Customer LTV Forecasting Engine</strong> that is:</p><ul><li><strong>Modular</strong> (dbt, Prefect, XGBoost)</li><li><strong>AI-Ready</strong> (retrainable, versioned models)</li><li><strong>Serverless</strong> (AWS Lambda, API Gateway, SageMaker, DynamoDB, S3)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f4ee8156c11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Never Trust a Dashboard Again (Until You Do This)]]></title>
            <link>https://medium.com/@zoemousmoula/never-trust-a-dashboard-again-until-you-do-this-0e213360348a?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e213360348a</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Wed, 28 May 2025 11:05:56 GMT</pubDate>
            <atom:updated>2025-05-28T11:12:57.300Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Modern Data Contract Stack with dbt, Snowflake &amp; Airflow</h3><p>🚨 <em>“Your dashboard is lying again.”</em></p><p>Upstream schema changes, silent nulls, or missing rows often ripple through pipelines unnoticed, until the <strong>CFO</strong> asks:</p><blockquote>“Why does churn look like zero this month?!! 😱”<br>If you’ve been on a data team, you’ve probably been there.</blockquote><p><strong>But what if your data could self-validate, alert on breakage, and block deployment if KPIs looked weird?</strong></p><p>In this post, I’ll walk you through building a <strong>modern, automated data contract system</strong> using <strong>dbt</strong>, <strong>Snowflake</strong>, and <strong>Airflow</strong>, with optional CI/CD, alerting, and visualization.</p><h3>🧭 Solution Overview</h3><p>This project sets up a complete data contract workflow using:</p><ul><li>✅ <strong>dbt</strong> to define models and validate schema, uniqueness, nulls, and accepted values</li><li>❄️ <strong>Snowflake</strong> as your data warehouse</li><li>🌌 <strong>Airflow</strong> to run tests and send alerts</li><li>🔁 <strong>CI/CD</strong> (optional) to block contract violations in pull requests</li><li>📈 <strong>Dashboards</strong> (optional) to monitor contract health</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*LOC1JDAR6KT8IQL_LnRM3A.png" /><figcaption>Diagram showing dbt applying contracts to Snowflake Medallion Layers, orchestrated by Airflow with email alerts on failure.</figcaption></figure><h3>🧾 What Are Data Contracts?</h3><p>Time for some theory…<br>Think of a <strong>data contract</strong> as a formal agreement between upstream data <strong>producers</strong> (engineering) and downstream data <strong>consumers</strong> (analytics):</p><ul><li>Which <strong>columns</strong> should exist</li><li>Which <strong>data types</strong> are expected</li><li>What <strong>values</strong> are allowed (or disallowed)</li><li>Who gets <strong>notified</strong> when the contract breaks</li></ul><p>It’s like <strong>unit testing</strong> for your data warehouse.</p><blockquote>🧪 These data contract checks are applied <strong>across all layers of your Medallion Architecture</strong>:</blockquote><blockquote>In the <strong>Bronze layer</strong> (raw ingestion), contracts <strong>monitor schema drift or missing fields</strong> from upstream systems — such as renamed or dropped columns from APIs, databases, or product teams.</blockquote><blockquote>In the <strong>Silver layer</strong> (cleaned data), contracts <strong>enforce data typing, null constraints, and value domains</strong> to ensure model consistency.</blockquote><blockquote>And in the <strong>Gold layer</strong> (business logic or KPI tables), contracts become even stricter — validating <strong>business-critical logic</strong>, <strong>referential integrity</strong>, and <strong>analytic expectations</strong> before data powers dashboards or reports.</blockquote><h3>🏗️ Step-by-Step Implementation</h3><h4>1. Set Up Your Snowflake Playground</h4><p>First, log in to your Snowflake account and create a database, schema, and sample table. You can do this using the <strong>worksheet</strong> interface.</p><pre>/* Step 1: Set role */<br>USE ROLE SYSADMIN;<br><br>/* Step 2: Create a new database for the project */<br>CREATE OR REPLACE DATABASE DBT_DATA_CONTRACTS;<br><br>/* Step 3: Create warehouse (skip if already exists) */<br>CREATE OR REPLACE WAREHOUSE DBT_WH<br>  WITH WAREHOUSE_SIZE = &#39;XSMALL&#39;<br>  AUTO_SUSPEND = 60<br>  AUTO_RESUME = TRUE;<br><br>/* Step 4: Use the new DB and warehouse */<br>USE DATABASE DBT_DATA_CONTRACTS;<br>USE WAREHOUSE DBT_WH;<br><br>/* Step 5: Create Medallion Architecture Schemas */<br>CREATE OR REPLACE SCHEMA bronze;<br>CREATE OR REPLACE SCHEMA silver;<br>CREATE OR REPLACE SCHEMA gold;<br><br>/* Step 6: Create a sample table to test contracts */<br>USE SCHEMA bronze;<br><br>CREATE OR REPLACE TABLE customers_raw (<br>    customer_id STRING,<br>    signup_date DATE,<br>    country STRING<br>);<br><br>/* Step 7: Insert test data with some intentional issues */<br>INSERT INTO customers_raw VALUES<br>    (&#39;u_001&#39;, &#39;2023-01-01&#39;, &#39;US&#39;),         -- valid record<br>    (NULL,     &#39;2023-01-05&#39;, &#39;FR&#39;),        -- NOT valid record<br>    (&#39;u_003&#39;,  &#39;2023-02-10&#39;, NULL),        -- NOT valid record<br>    (&#39;u_004&#39;,  NULL,          &#39;DE&#39;);       -- NOT valid record<br><br>select * from customers_raw;</pre><p>Expected output — created table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePbnDMt5nGGbnS6NHfISJQ.png" /></figure><h4>2. Initialize Your dbt Project</h4><p>From your terminal:</p><pre>pipx install dbt-snowflake --include-deps<br>mkdir ddata_contracts &amp;&amp; cd ddata_contracts<br><br># Initialize dbt project<br>dbt init .</pre><p>Configure profiles.ymlwith:</p><pre>user: your_user<br>password: your_password<br>role: SYSADMIN<br>warehouse: DBT_WH<br>database: DBT_DATA_CONTRACTS<br>schema: SILVER<br>threads: 1</pre><h4>3. Build Your First Data Contract</h4><p>Folder: models/silver<br>File 1: customers_cleaned.sql</p><pre>SELECT<br>  customer_id,<br>  signup_date,<br>  country<br>FROM {{ source(&#39;bronze&#39;, &#39;customers_raw&#39;) }}</pre><p>File 2: schema.yml</p><pre>version: 2<br><br>sources:<br>  - name: bronze<br>    schema: bronze<br>    database: DBT_DATA_CONTRACTS<br>    tables:<br>      - name: customers_raw<br><br>models:<br>  - name: customers_cleaned<br>    description: Cleaned customer data<br>    columns:<br>      - name: customer_id<br>        description: Primary key<br>        tests:<br>          - not_null<br>          - unique<br>      - name: signup_date<br>        tests:<br>          - not_null<br>      - name: country<br>        tests:<br>          - not_null<br>          - accepted_values:<br>              values: [&quot;US&quot;, &quot;FR&quot;, &quot;DE&quot;]</pre><p>Run:</p><pre>dbt debug &amp;&amp; dbt run &amp;&amp; dbt test</pre><h3>🌌 Automate with Airflow</h3><h4>4. ⚙️ Install and Configure</h4><p>Create a virtual environment:</p><pre>python -m venv airflow_env<br>source airflow_env/bin/activate<br><br>pip install dbt-snowflake==1.9.4<br>pip install apache-airflow==2.8.1 \<br>    apache-airflow-providers-smtp \<br>    apache-airflow-providers-snowflake \<br>    dbt-core</pre><pre>airflow db init<br><br># Create admin user<br>airflow users create \<br>  --username admin \<br>  --firstname YourName\<br>  --lastname YourSurname \<br>  --role Admin \<br>  --email your@email.com \<br>  --password airflowpass<br><br># Launch UI<br>airflow webserver --port 8080</pre><p>Open: <a href="http://localhost:8080/">http://localhost:8080</a></p><h4>5. 📜 Create Airflow DAG</h4><p>Place the following in ~/airflow/dags/data_contract_check_dag.py</p><pre>from airflow import DAG<br>from airflow.operators.bash import BashOperator<br>from airflow.operators.email import EmailOperator<br>from airflow.utils.trigger_rule import TriggerRule<br>import pendulum<br><br>default_args = {<br>    &#39;owner&#39;: &#39;FistName LastName&#39;,<br>    &#39;email&#39;: [&#39;email&#39;],<br>    &#39;email_on_failure&#39;: False,<br>    &#39;email_on_retry&#39;: False,<br>    &#39;retries&#39;: 0,<br>}<br><br>with DAG(<br>    dag_id=&#39;data_contract_check_dag&#39;,<br>    default_args=default_args,<br>    start_date=pendulum.today(&#39;UTC&#39;).add(days=-1),<br>    schedule=&#39;@daily&#39;,<br>    catchup=False,<br>    tags=[&#39;contracts&#39;, &#39;dbt&#39;, &#39;quality&#39;]<br>) as dag:<br><br>    run_dbt_tests = BashOperator(<br>        task_id=&#39;run_dbt_contract_tests&#39;,<br>        bash_command=&#39;cd /path/to/ddata_contracts &amp;&amp; ../../airflow_env/bin/dbt test --select path:models/silver&#39;,<br>    )<br><br>    send_email_on_failure = EmailOperator(<br>        task_id=&#39;send_contract_failure_alert&#39;,<br>        to=&#39;email&#39;,<br>        subject=&#39;🚨 dbt Data Contract Test Failed&#39;,<br>        html_content=&quot;&quot;&quot;<br>        &lt;h3&gt;Data contract violation detected&lt;/h3&gt;<br>        &lt;p&gt;At least one dbt test has failed on your silver layer. Please check the logs in Airflow.&lt;/p&gt;<br>        &quot;&quot;&quot;,<br>        trigger_rule=TriggerRule.ONE_FAILED<br>    )<br><br>    run_dbt_tests &gt;&gt; send_email_on_failure</pre><h4>6. 📧 Configure Email Alerts</h4><p>Update airflow.cfg or .env:</p><pre>[smtp]<br>smtp_host = smtp.gmail.com<br>smtp_starttls = True<br>smtp_user = your@gmail.com<br>smtp_password = your_app_password<br>smtp_port = 587<br>smtp_mail_from = your@gmail.com</pre><h3>📊 Advanced Features</h3><h4>7. 🗂️ Store Violations in Metadata Table</h4><pre>CREATE TABLE IF NOT EXISTS data_contract_violations (<br>    run_time TIMESTAMP,<br>    model_name STRING,<br>    column_name STRING,<br>    test_name STRING,<br>    failure_reason STRING<br>);</pre><p>Parse target/run_results.json and insert records via a Python post-hook in Airflow.</p><h4>8. 📈 Visualize Contract Status</h4><p>Use Metabase, Streamlit, or Looker to show:</p><ul><li>Last run time</li><li>Test pass/fail rate</li><li>Failing models</li><li>DAG/dbt log links</li></ul><h4>9. 🔁 Enforce Contracts in CI/CD</h4><p>Use GitHub Actions:</p><pre>jobs:<br>  test:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - run: dbt test</pre><p>❌ Automatically <strong>fail PRs</strong> if any tests fail.</p><h3>✅ TL;DR: What You Have Now</h3><ul><li>Data contracts for every layer of your warehouse</li><li>Daily tests automated with Airflow</li><li>Email alerting on failures</li><li>Optional dashboards and CI pipelines</li></ul><h3>💰 Cost Estimate</h3><ul><li><strong>Snowflake</strong>: Compute costs based on dbt runs (~low for daily tests)</li><li><strong>Airflow</strong>: Free locally, or use MWAA/Astronomer</li><li><strong>dbt</strong>: Open-source (or dbt Cloud)</li></ul><h3>👇 What’s Next?</h3><p>I’m working on:</p><ul><li>A GitHub template with this setup</li><li>A dbt package to manage and track contracts</li><li>Prebuilt DAGs and Email alerts for fast adoption</li></ul><p>💬 DM or comment if you’d like <strong>early access</strong> or help implementing this in your stack!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e213360348a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ User Churn Prediction Microservice — Real-Time, Serverless]]></title>
            <link>https://medium.com/@zoemousmoula/user-churn-prediction-microservice-real-time-serverless-1f541775a616?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f541775a616</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <category><![CDATA[crm]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Wed, 14 May 2025 18:45:23 GMT</pubDate>
            <atom:updated>2025-05-14T18:45:23.403Z</atom:updated>
            <content:encoded><![CDATA[<h3>📉 User Churn Prediction Microservice — Real-Time, Serverless</h3><h3>🔍 The Problem: Churn Is Silent, But Costly</h3><p>Every SaaS product fights churn. Users stop logging in. They ghost support. They cancel without warning. And by the time you realize they’re gone, it’s too late.</p><p>What founders and growth teams need is <strong>a way to detect churn before it happens</strong>, based on behavior patterns that predict when someone is about to slip away.</p><h3>⚙️ So I Built This: A Real-Time Churn Prediction Microservice</h3><p>A fully serverless microservice that:</p><ul><li>Collects product usage events</li><li>Scores users in real-time for churn risk using a machine learning model</li><li>Sends alerts (Slack/email) for high-risk users</li><li>Stores predictions in S3 for analytics or CRM sync</li></ul><h3>🧾 Suppose We Have Source Data Available from the Product</h3><p>Assume that <strong>user activity events are collected and stored in an Amazon S3 bucket</strong> in a structured format. Each record might look like this:</p><pre>{<br>  &quot;user_id&quot;: &quot;u_12345&quot;,<br>  &quot;event_type&quot;: &quot;used_feature&quot;,<br>  &quot;timestamp&quot;: &quot;2025-05-14T14:22:01Z&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;feature&quot;: &quot;export_csv&quot;,<br>    &quot;plan&quot;: &quot;starter&quot;,<br>  }<br>}</pre><p>Here’s what each field represents:</p><ul><li><strong>user_id</strong>: A unique identifier for each user.</li><li><strong>event_type</strong>: Describes the kind of user activity. Examples:<br>- used_feature: The user used a product feature.<br>- ticket_created: The user initiated a support ticket.<br>- login: The user logged into their account.</li><li><strong>metadata</strong>: Contains additional details:<br>- feature: Specifies which feature the user interacted with, such as export_csv, dashboard, or analytics.<br>- plan: Indicates the user’s subscription (e.g., starter, pro, or enterprise).</li></ul><p>Over time, this forms a behavioral history for each user, <strong>partitioned</strong> daily like:</p><pre>s3://product/events/year=2025/month=04/day=11/</pre><p>This event stream forms the <strong>input dataset</strong>.</p><p>*** I don’t partition by user_id because it would create millions of small partitions in S3, which is inefficient for metadata operations. Instead, I filter by user in memory after retrieving the last 7 days of data.</p><h3>🧠 Churn Model</h3><p>The churn prediction model is a <strong>supervised machine learning model</strong> trained to classify users as <em>likely to churn</em>, based on behavioral signals.</p><h3>🔖 Training Dataset</h3><p>For training, we maintain a separate <strong>churn label dataset</strong> in S3 that looks like this:</p><pre>{<br>  &quot;user_id&quot;: &quot;u_12345&quot;,<br>  &quot;churned&quot;: 1,<br>  &quot;churn_date&quot;: &quot;2025-03-12&quot;<br>}</pre><p>This is <strong>not part of the live event stream</strong>. It’s derived from internal business rules, such as:</p><ul><li>Explicit cancellation</li><li>Subscription status = inactive</li></ul><p>During model training, we <strong>join this churn label</strong> with each user’s historical activity data from the event stream, forming a full training set with features + labels.</p><h3>🧪 Training Process</h3><ul><li><strong>Frequency</strong>: Retrain the model <strong>weekly or monthly </strong>in <strong>SageMaker</strong>, depending on product’s data volume.</li><li><strong>Input</strong>: Activity data from S3 + above churn labels</li><li><strong>Algorithm</strong>: Typically, a tree-based model like <strong>XGBoost</strong> or <strong>LightGBM</strong></li><li><strong>Output</strong>: A serialized .joblib model file is saved back to S3:</li></ul><pre>s3://models/churn/model-v6.joblib</pre><p>This process gives us:</p><ul><li>🔄 Versioned training cycles</li><li>📁 Clear model lineage with rollbacks if needed</li><li>🔌 Seamless integration with Lambda (load from S3)</li></ul><h3>🔄 Workflow</h3><p>Every time a <strong>new event is written to S3</strong>, Lambda is triggered and runs the following:</p><ol><li>Reads the <strong>last 7 days of activity</strong> for the user from S3</li><li>Aggregates the user’s behavior to extract features</li><li>Loads the <strong>latest churn model</strong> from S3</li><li>Runs the model and computes a <strong>churn probability</strong></li><li>If the score is above a threshold (e.g., 0.75):</li></ol><ul><li>Sends a <strong>Slack or email alert</strong> via SNS</li><li>Writes the prediction result back to S3</li><li>Optionally tags the user in your CRM using AppFlow</li></ul><h3>🔧 Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zSBYgjUCfsRiL84tZo0SOw.png" /><figcaption>Pipeline in AWS Cloud</figcaption></figure><h3>📊 Feature Engineering — What the Model Sees</h3><p>Let’s break down the features the model uses and how they’re derived from the event stream.</p><h3>1. Feature Usage Frequency</h3><p><strong>How to extract it:</strong></p><ul><li>Count the number of used_feature events per user in the last 7 days</li></ul><p><strong>Why it matters:</strong></p><ul><li>Drop in engagement is a strong churn signal</li><li>Consistent use of key features implies stickiness</li></ul><p><strong>Example:</strong></p><pre>“feature_usage_count_7d”: 3,<br>&quot;feature_diversity_7d&quot;: 2</pre><h3>2. Support Ticket Volume</h3><p><strong>How to extract it:</strong></p><ul><li>Count ticket_created events</li></ul><p><strong>Why it matters:</strong></p><ul><li>High or sudden increases in support interactions can indicate user dissatisfaction or difficulties.</li></ul><p><strong>Example:</strong></p><pre>“support_tickets_7d”: 2</pre><h3>3. Plan Type</h3><p><strong>How to extract it:</strong></p><ul><li>Parse the plan value from the event metadata</li></ul><p><strong>Why it matters:</strong></p><ul><li>Churn behavior varies by plan. For example, free-tier users may churn silently, while enterprise users churn after long downgrades</li></ul><p><strong>Example:</strong></p><pre>“plan_starter”: 1,<br>&quot;plan_pro&quot;: 0,<br>&quot;plan_enterprise&quot;: 0</pre><h3>🔧 Error Handling &amp; Monitoring</h3><ul><li>Lambda logs execution to <strong>CloudWatch Logs</strong></li><li>Alerts are routed through <strong>SNS</strong> to Slack/email</li><li>Model errors (load or inference) are caught and logged in <strong>CloudWatch Logs</strong></li><li><strong>Dead Letter Queue (DLQ)</strong> handles failed Lambda events by capturing them for debugging and reprocessing.</li></ul><h3>💰 Cost Estimation</h3><p>Assuming ~5,000 user events/day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*ww8hdnuib06Qm_nGvTV-8Q.png" /></figure><p><strong>→ Total: ~$20–35/month</strong></p><h3>✅ TL;DR — Why This Matters</h3><p>This microservice turns raw user activity into real-time churn predictions:</p><ul><li>🧠 Machine learning model hosted in S3</li><li>⚡️ Scored per-user in milliseconds via Lambda</li><li>🔔 Alerts your team when users show signs of churn</li><li>🔁 Easy retraining with SageMaker</li><li>🧾 Fully auditable via S3 prediction logs (Every churn prediction is logged in Amazon S3, making it easy to analyze performance and improve future predictions)</li></ul><h3>👇 What’s Next?</h3><p>I’m preparing a CloudFormation template that lets you deploy this churn prediction pipeline in minutes.</p><p>Want access? DM me or drop your email below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f541775a616" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automated Lead Capture & Enrichment Pipeline]]></title>
            <link>https://medium.com/@zoemousmoula/automated-lead-capture-enrichment-pipeline-c195d227f3bb?source=rss-8ee0572a9c70------2</link>
            <guid isPermaLink="false">https://medium.com/p/c195d227f3bb</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[lead-generation]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[crm-automation]]></category>
            <dc:creator><![CDATA[Zoe Mousmoula]]></dc:creator>
            <pubDate>Sat, 19 Apr 2025 16:13:56 GMT</pubDate>
            <atom:updated>2025-04-19T16:19:17.502Z</atom:updated>
            <content:encoded><![CDATA[<h3>🔍 The Lead Chaos Dilemma</h3><p>Startups thrive on leads — from landing pages, ad campaigns, contact forms, or email lists. But here’s the kicker:</p><ul><li>Leads are often <strong>incomplete</strong>, missing job titles, company names, or even valid emails.</li><li>Sales teams waste time chasing <strong>invalid or duplicate leads</strong>.</li><li>Founders want <strong>fast insights</strong> to act before a hot lead goes cold.</li></ul><p>Manual filtering? Slow. Human errors. Not scalable.</p><p>So I built a <strong>fully automated, serverless pipeline</strong> to capture, clean, and store leads in real-time.</p><h3>🧠 The Solution: A Serverless Lead Pipeline</h3><p>I’ve built a <strong>serverless AWS pipeline</strong> that:</p><ul><li>Accepts raw lead data</li><li>Enriches lead info via external APIs (like Clearbit or Hunter.io)</li><li>Stores clean, validated leads as CSV in Amazon S3</li><li>Automatically syncs those files to a CRM system (like Salesforce) via AppFlow</li></ul><p>This approach gives startup founders:</p><ul><li>Clean and enriched lead data</li><li>Real-time or batched CRM sync</li><li>No need to manage or query databases</li><li>Fully auditable S3 storage (great for analytics and compliance)</li></ul><h3>🛠️ Services Used</h3><figure><img alt="Services used in AWS" src="https://cdn-images-1.medium.com/max/926/1*x_NJYJayMmEjbzmhwUF09g.png" /></figure><h3>🔄 Pipeline Overview</h3><h4>Workflow:</h4><ol><li><strong>Frontend/Webhook</strong> → sends a POST request with raw lead data</li><li><strong>API Gateway</strong> triggers <strong>Lambda</strong></li><li><strong>Lambda</strong>:</li></ol><ul><li>Validates and deduplicates</li><li>Calls the Clearbit/Hunter API for enrichment</li><li>Stores the clean leads to <strong>Amazon S3</strong> in partitions (/leads/year=yyyy/month=mm/day=dd/lead-UUID.json)</li><li>Triggers <strong>SNS</strong> to notify sales/ops of new leads</li></ul><p>4.<strong> S3 Event Notification</strong> detects a new file upload</p><p>5. <strong>AppFlow</strong> syncs the new lead to the CRM automatically</p><h3>🔧 <strong>Diagram</strong></h3><figure><img alt="Solution Diagram in AWS" src="https://cdn-images-1.medium.com/max/769/1*4A7TBcavCEy8Cz-vyUzeMg.png" /></figure><h3>🚨 Error Handling &amp; Alerts</h3><ul><li>All Lambda functions have try-catch logic</li><li>Failures log to <strong>CloudWatch Logs</strong> and trigger <strong>CloudWatch Alarms</strong></li><li>Use <strong>SQS Dead Letter Queues (DLQs)</strong> for failed Lambda executions</li><li><strong>CloudWatch Alarms</strong> for AppFlow sync failures (optional via EventBridge monitoring)</li><li>SNS can notify a Slack webhook or email when something fails or when a lead matches a certain priority</li></ul><h3>💸 Cost Estimation</h3><p>For a startup processing ~500 leads/day:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Cgy36U_yXd2F8i2V_8cbjw.png" /></figure><p>➡️ <strong>Estimated Total: $15–30/month</strong> for a complete lead pipeline.</p><h3>✅ TL;DR — Value to Founders</h3><p>This pipeline turns messy form data into enriched, qualified, CRM-ready leads.</p><p>🔹 No more spreadsheets<br>🔹 No more manual sorting<br>🔹 Alerts when good leads arrive<br>🔹 All built serverlessly — low-cost, fast, and scalable</p><h3>👇 What’s Next?</h3><p>I’m building CloudFormation templates and full deployment instructions to make this pipeline easy to deploy. If you’re a startup founder or builder wanting help to automate your lead flow, hit me up!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c195d227f3bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>