<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vighnesh Gaikwad on Medium]]></title>
        <description><![CDATA[Stories by Vighnesh Gaikwad on Medium]]></description>
        <link>https://medium.com/@vighnesh3116?source=rss-3fb5eba7f924------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UCg4czgpx0YnkNMS</url>
            <title>Stories by Vighnesh Gaikwad on Medium</title>
            <link>https://medium.com/@vighnesh3116?source=rss-3fb5eba7f924------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:09:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vighnesh3116/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond the Binge: How Big Data Powers OTT Content Analytics]]></title>
            <link>https://medium.com/@vighnesh3116/beyond-the-binge-how-big-data-powers-ott-content-analytics-c07c659fec23?source=rss-3fb5eba7f924------2</link>
            <guid isPermaLink="false">https://medium.com/p/c07c659fec23</guid>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ott-analytics]]></category>
            <category><![CDATA[ott-platform]]></category>
            <dc:creator><![CDATA[Vighnesh Gaikwad]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 17:58:37 GMT</pubDate>
            <atom:updated>2026-10-11T17:58:37.171Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Vighnesh Gaikwad and Yash Lokare</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ERxhYYy5S1MiYjDPPl9xSA.png" /></figure><h3>Introduction: The Billion-Event Question</h3><p>It’s 11 PM and you open your favorite streaming app. Within two seconds, a row of titles appears, and somehow one of them is exactly what you wanted. That moment is the visible tip of a massive analytics iceberg.</p><p>Over-the-top (OTT) platforms like Netflix, Disney+ Hotstar, Prime Video, and JioCinema serve millions of concurrent viewers. Every play, pause, seek, rewind, buffer stall, search query, and abandoned episode generates an event. At scale, a single platform can produce <strong>billions of events per day</strong>.</p><p>This creates a real business problem:</p><ul><li><strong>Content investment is risky.</strong> A single original series can cost tens of millions of dollars. Which genres, actors, and release cadences actually retain subscribers?</li><li><strong>Churn is expensive.</strong> Acquiring a subscriber costs far more than retaining one. Can we detect disengagement <em>before</em> a user cancels?</li><li><strong>Quality of Experience (QoE) is fragile.</strong> A few seconds of buffering can trigger abandonment. Can we detect streaming degradation in near real time?</li><li><strong>Discovery is overwhelming.</strong> With thousands of titles, users give up scrolling after about a minute. Recommendations must be personalized and fast.</li></ul><p>A traditional relational database cannot absorb this volume, velocity, and variety of data. Click streams arrive as semi-structured JSON, catalog metadata is relational, and device telemetry is high-frequency time-series data. This is the classic Big Data problem: the <strong>3Vs</strong> of volume, velocity, and variety.</p><p>In this post, we walk through the architecture of an OTT content analytics platform that we built as a Big Data project, covering how the data flows, which tools we chose, and why.</p><h3>What Are We Measuring?</h3><p>Before designing a pipeline, you have to know the questions it must answer. We focused on four analytics domains:</p><p>DomainExample Metrics<strong>Engagement</strong>Watch time, completion rate, session length, DAU/MAU<strong>Content performance</strong>Top titles by genre/region, drop-off points, binge rate<strong>Quality of Experience</strong>Startup time, rebuffering ratio, bitrate switches<strong>Retention &amp; churn</strong>Cohort retention, inactivity windows, churn-risk score</p><p>Each domain has different latency requirements. QoE alerts must fire within seconds, while content-performance reports can tolerate hours. That distinction drives the whole architecture.</p><h3>Data Pipeline &amp; Architecture</h3><p>We adopted a <strong>Lambda-style architecture</strong>, with a speed layer for real-time insights and a batch layer for deep historical analysis. Both feed a unified serving layer.</p><pre>┌──────────────┐   ┌─────────┐   ┌──────────────────────┐<br>│ Player/App   │──▶│  Kafka  │──▶│ Spark Structured     │──▶ Real-time dashboards<br>│ Events (SDK) │   │ Topics  │   │ Streaming (Speed)    │    &amp; alerts<br>└──────────────┘   └────┬────┘   └──────────────────────┘<br>                        │<br>                        ▼<br>                  ┌───────────┐   ┌──────────────┐   ┌────────────────┐<br>                  │ HDFS Raw  │──▶│ Spark Batch  │──▶│ Hive / Parquet │──▶ BI &amp; ML<br>                  │ (Bronze)  │   │ ETL (Silver) │   │ (Gold)         │<br>                  └───────────┘   └──────────────┘   └────────────────┘</pre><h4>1. Ingestion</h4><p>Client SDKs on mobile, web, and smart TVs emit JSON events such as play_start, heartbeat, buffer_start, seek, and play_end. These are published to <strong>Apache Kafka</strong>, which acts as a durable, replayable buffer between producers and consumers.</p><p>Key design decisions:</p><ul><li><strong>Topic partitioning by </strong><strong>user_id.</strong> This guarantees ordering of a single user’s events within a partition, which matters for sessionization.</li><li><strong>Separate topics per event class</strong> (playback_events, qoe_metrics, search_events) so each can scale and retain data independently.</li><li><strong>Schema enforcement</strong> using a schema registry (Avro) to prevent malformed events from poisoning downstream jobs.</li></ul><p>A sample event looks like this:</p><p>json</p><pre>{<br>  &quot;event_id&quot;: &quot;e7a1c2d4-...&quot;,<br>  &quot;user_id&quot;: &quot;u_483920&quot;,<br>  &quot;content_id&quot;: &quot;c_10293&quot;,<br>  &quot;event_type&quot;: &quot;heartbeat&quot;,<br>  &quot;position_sec&quot;: 1240,<br>  &quot;bitrate_kbps&quot;: 3200,<br>  &quot;device&quot;: &quot;android_tv&quot;,<br>  &quot;region&quot;: &quot;IN-MH&quot;,<br>  &quot;event_ts&quot;: &quot;2026-10-11T17:42:10Z&quot;<br>}</pre><h4>2. Stream Processing (Speed Layer)</h4><p><strong>Spark Structured Streaming</strong> consumes from Kafka and performs windowed aggregations. For QoE, we compute the rebuffering ratio per region and device in 1-minute tumbling windows, with watermarking to handle late-arriving events from flaky mobile networks.</p><p>python</p><pre># Placeholder: see /streaming/qoe_stream.py in the repo<br>qoe = (events<br>    .withWatermark(&quot;event_ts&quot;, &quot;2 minutes&quot;)<br>    .groupBy(window(&quot;event_ts&quot;, &quot;1 minute&quot;), &quot;region&quot;, &quot;device&quot;)<br>    .agg(<br>        sum(&quot;buffer_ms&quot;).alias(&quot;total_buffer_ms&quot;),<br>        sum(&quot;play_ms&quot;).alias(&quot;total_play_ms&quot;)<br>    )<br>    .withColumn(&quot;rebuffer_ratio&quot;,<br>        col(&quot;total_buffer_ms&quot;) / col(&quot;total_play_ms&quot;)))</pre><p>When the ratio crosses a threshold, an alert is pushed to the operations dashboard, so engineers see a CDN problem in a particular city before social media does.</p><h4>3. Batch Processing and the Medallion Layout</h4><p>In parallel, raw events land in <strong>HDFS</strong> as immutable files. We organize storage in three tiers:</p><ul><li><strong>Bronze (raw):</strong> Events exactly as received, partitioned by date. This is the source of truth and enables reprocessing.</li><li><strong>Silver (cleaned):</strong> Deduplicated, validated, and enriched with catalog metadata (genre, language, cast) via broadcast joins.</li><li><strong>Gold (aggregated):</strong> Business-ready tables such as daily_title_engagement, user_cohort_retention, and session_summary.</li></ul><p>The most important batch step is <strong>sessionization</strong>. Raw heartbeats must be stitched into viewing sessions. We use a 30-minute inactivity gap, implemented with Spark window functions:</p><p>python</p><pre># Placeholder: see /batch/sessionize.py<br>w = Window.partitionBy(&quot;user_id&quot;).orderBy(&quot;event_ts&quot;)<br>sessions = (events<br>    .withColumn(&quot;prev_ts&quot;, lag(&quot;event_ts&quot;).over(w))<br>    .withColumn(&quot;new_session&quot;,<br>        (col(&quot;event_ts&quot;).cast(&quot;long&quot;) - col(&quot;prev_ts&quot;).cast(&quot;long&quot;) &gt; 1800).cast(&quot;int&quot;))<br>    .withColumn(&quot;session_id&quot;, sum(&quot;new_session&quot;).over(w)))</pre><h4>4. Storage and Serving</h4><p>We store Silver and Gold data as <strong>Apache Parquet</strong>, a columnar format that compresses well and lets analytical queries read only the columns they need. For a typical engagement query that touches 3 of 40 columns, this cuts I/O dramatically. <strong>Apache Hive</strong> provides a metastore and SQL interface over these files, so analysts can query petabyte-scale data using familiar SQL.</p><h4>5. Machine Learning on Top</h4><p>With clean Gold data, we built two models using <strong>Spark MLlib</strong>:</p><ul><li><strong>Recommendation:</strong> Collaborative filtering with Alternating Least Squares (ALS) on implicit feedback (watch time rather than explicit ratings).</li><li><strong>Churn prediction:</strong> A gradient-boosted tree classifier trained on features such as 7-day watch-time trend, days since last session, and support-ticket count.</li></ul><p>Using implicit feedback matters. Most users never rate anything, but how long they watched is a strong signal of preference.</p><h3>Technologies Used</h3><p>LayerTechnologyWhy We Chose ItIngestion<strong>Apache Kafka</strong>High-throughput, fault-tolerant, replayable logStream processing<strong>Spark Structured Streaming</strong>Unified API for batch and streaming; exactly-once semantics with checkpointingBatch processing<strong>Apache Spark (PySpark)</strong>In-memory processing, rich DataFrame APIStorage<strong>HDFS + Parquet</strong>Scalable distributed storage, columnar efficiencyWarehouse/SQL<strong>Apache Hive</strong>SQL access to the data lakeOrchestration<strong>Apache Airflow</strong>DAG-based scheduling and dependency managementML<strong>Spark MLlib</strong>Distributed training close to the dataVisualization<strong>Grafana / Apache Superset</strong>Real-time and BI dashboardsContainerization<strong>Docker Compose</strong>Reproducible local cluster setup</p><p>The repository follows this structure:</p><pre>ott-content-analytics/<br>├── ingestion/        # Kafka producers &amp; event simulator<br>├── streaming/        # Structured Streaming jobs<br>├── batch/            # ETL and sessionization<br>├── ml/               # ALS &amp; churn models<br>├── dags/             # Airflow DAGs<br>├── dashboards/       # Grafana/Superset configs<br>└── docker-compose.yml</pre><p>Since real OTT data is proprietary, we wrote an <strong>event simulator</strong> that generates realistic viewing behavior, including time-of-day peaks, binge sessions, and occasional network degradation. This let us test the pipeline end to end without privacy concerns.</p><h3>Lessons Learned</h3><ol><li><strong>Late data is the norm, not the exception.</strong> Mobile users go through tunnels. Watermarks and reprocessing from Bronze saved us repeatedly.</li><li><strong>The small-files problem is real.</strong> Streaming jobs that write every few seconds create thousands of tiny files and cripple HDFS NameNode performance. Compaction jobs are essential.</li><li><strong>Data skew hurts.</strong> A blockbuster premiere creates one hot content_id. Salting keys during aggregation kept executors balanced.</li><li><strong>Decide your metric definitions early.</strong> “Active user” and “completed view” mean different things to different teams. Document them once and enforce them in the Gold layer.</li></ol><h3>Conclusion</h3><p>OTT content analytics is where Big Data stops being abstract. Every technology in the stack maps to a tangible outcome: Kafka keeps no event from being lost, Spark turns raw clicks into insight, Parquet and Hive make petabytes queryable, and ML closes the loop by feeding recommendations back into the product.</p><p>The same architecture generalizes to e-commerce, gaming, and ad-tech, anywhere users leave a trail of behavioral events. If you’re learning Big Data, building a streaming-analytics project for a domain you actually use is one of the best ways to understand why these tools exist.</p><p>We’d love your feedback. Clone the repo, spin up the Docker cluster, and tell us what you’d improve.</p><h3>Author Contribution Statement</h3><ul><li><strong>Vighnesh Gaikwad:</strong> Led the research on OTT analytics use cases and metric design. Designed the Kafka ingestion layer and event simulator, implemented the Spark Structured Streaming QoE jobs, and set up the Docker Compose environment.</li><li><strong>Yash Lokare:</strong> Led the batch-processing layer, including HDFS storage layout, Parquet/Hive modeling, sessionization logic, and the Airflow DAGs. Built the ALS recommendation and churn-prediction models in Spark MLlib.</li><li><strong>Joint contributions:</strong> Both authors co-designed the overall architecture, reviewed each other’s code, built the dashboards, wrote and edited this blog post, and documented the GitHub repository.</li></ul><p>Self-Declaration: It took me 3 hour(s) and 30 minute(s) to solve the assignment, ensuring accuracy, completeness, and adherence to professional and ethical standards.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c07c659fec23" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>