<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Manoj Ch on Medium]]></title>
        <description><![CDATA[Stories by Manoj Ch on Medium]]></description>
        <link>https://medium.com/@manojch_10?source=rss-1b21f69d6381------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*fJcMctf7XNbomzYX</url>
            <title>Stories by Manoj Ch on Medium</title>
            <link>https://medium.com/@manojch_10?source=rss-1b21f69d6381------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:56:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@manojch_10/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[BigQuery Under the Hood: A Deep Dive into Its Architecture and Performance]]></title>
            <link>https://medium.com/@manojch_10/bigquery-under-the-hood-a-deep-dive-into-its-architecture-and-performance-7ab20ad370b6?source=rss-1b21f69d6381------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ab20ad370b6</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[bigdata-analytics]]></category>
            <dc:creator><![CDATA[Manoj Ch]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 04:18:51 GMT</pubDate>
            <atom:updated>2025-02-28T04:18:51.531Z</atom:updated>
            <content:encoded><![CDATA[<p>BigQuery, Google Cloud’s fully managed, serverless data warehouse, has become a cornerstone of modern data analytics. Its ability to handle petabyte-scale datasets with blazing-fast SQL queries has made it a favorite among data engineers, analysts, and scientists. But what makes BigQuery so powerful? In this article, we’ll dive deep into BigQuery’s architecture, internals, and best practices to help you unlock its full potential.</p><h3>1. Introduction</h3><p>BigQuery is a cloud-native data warehouse designed for speed, scalability, and ease of use. Unlike traditional data warehouses, BigQuery is serverless, meaning you don’t need to manage infrastructure. It automatically scales to handle your data and query demands, making it ideal for everything from ad-hoc analytics to machine learning.</p><p>But to truly harness BigQuery’s power, it’s essential to understand how it works under the hood. In this article, we’ll explore BigQuery’s architecture, storage and compute layers, query execution, and optimization techniques.</p><h3>2. BigQuery Overview</h3><h3>What is BigQuery?</h3><p>BigQuery is a fully managed, serverless data warehouse that enables super-fast SQL queries using the processing power of Google’s infrastructure. It’s designed for:</p><ul><li><strong>Scalability</strong>: Handle petabytes of data effortlessly.</li><li><strong>Speed</strong>: Execute complex queries in seconds.</li><li><strong>Ease of Use</strong>: No infrastructure management; just focus on your data.</li></ul><h3>Key Features</h3><ul><li><strong>Serverless</strong>: No servers to manage; automatic scaling.</li><li><strong>Columnar Storage</strong>: Optimized for analytical queries.</li><li><strong>SQL Support</strong>: Standard SQL for querying data.</li><li><strong>Integration</strong>: Works seamlessly with other Google Cloud services like AI Platform, Dataflow, and Looker.</li></ul><h3>3. BigQuery Architecture</h3><p>BigQuery’s serverless architecture decouples storage and compute and allows them to scale independently on demand. This structure offers both immense flexibility and cost controls for customers because they don’t need to keep their expensive compute resources up and running all the time. This is very different from traditional node-based cloud data warehouse solutions or on-premise massively parallel processing (MPP) systems. This approach also allows customers of any size to bring their data into the data warehouse and start analyzing their data using Standard SQL without worrying about database operations and system engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*cd7LLWkvcldQXT00.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*kWIcMDcA4Y4xk7Ul.jpg" /></figure><p>BigQuery’s architecture is built on two key principles:</p><ol><li><strong>Separation of Storage and Compute</strong>.</li><li><strong>Distributed Processing</strong>.</li></ol><p>Let’s break it down:</p><h3>3.1 Storage Layer</h3><p>BigQuery uses <strong>Google Colossus</strong>, a global distributed file system, to store data. Colossus ensures durability, availability, and scalability. Data is stored in a <strong>columnar format</strong> called <strong>Capacitor</strong>, which is optimized for compression and fast scanning.</p><ul><li><strong>Columnar Storage</strong>: Unlike row-based storage, columnar storage stores data by column. This is ideal for analytical queries that often scan specific columns rather than entire rows.</li><li><strong>Capacitor File Format</strong>: BigQuery’s proprietary format that provides high compression rates and efficient encoding.</li></ul><h3>3.2 Compute Layer</h3><p>BigQuery uses <strong>Dremel</strong>, a massively parallel query execution engine, to process queries. Dremel breaks queries into smaller tasks and distributes them across thousands of machines.</p><ul><li><strong>Tree Architecture</strong>: Dremel uses a tree-like structure for query execution. The root server receives the query, breaks it into smaller tasks, and distributes them to leaf nodes. The leaf nodes process the data and return results to the root server.</li><li><strong>Slots</strong>: Slots are units of computational capacity. BigQuery dynamically allocates slots to execute queries in parallel.</li></ul><p><strong>Compute and storage talk to each other through the petabit </strong><a href="https://cloudplatform.googleblog.com/2015/06/A-Look-Inside-Googles-Data-Center-Networks.html"><strong>Jupiter</strong></a><strong> network.</strong></p><ul><li>In between storage and compute is ‘shuffle’, which takes advantage of Google’s Jupiter network to move data extremely rapidly from one place to another.</li></ul><p><strong>BigQuery is orchestrated via </strong><a href="https://research.google.com/pubs/pub43438.html"><strong>Borg</strong></a><strong>, Google’s precursor to </strong><a href="https://kubernetes.io/"><strong>Kubernetes</strong></a><strong>.</strong></p><ul><li>The mixers and slots are all run by Borg, which allocates hardware resources.</li></ul><h3>3.3 Metadata Management</h3><p>BigQuery maintains metadata about datasets, tables, and schemas in a centralized catalog. This metadata is used for query planning, optimization, and access control.</p><h3>4. Query Execution in BigQuery</h3><h3>4.1 Dremel Execution Engine</h3><p>When you submit a SQL query, BigQuery’s Dremel engine takes the following steps:</p><ol><li><strong>Query Parsing</strong>: The query is parsed and validated.</li><li><strong>Query Planning</strong>: The query planner generates an execution plan.</li><li><strong>Query Optimization</strong>: The optimizer rewrites the query for efficiency (e.g., predicate pushdown, column pruning).</li><li><strong>Execution</strong>: The query is executed in parallel across thousands of machines.</li></ol><h3>4.2 Slots and Parallelism</h3><ul><li><strong>Slots</strong>: Each slot is a virtual CPU that processes a portion of the query. BigQuery automatically allocates slots based on query complexity and available resources.</li><li><strong>Shuffle</strong>: Intermediate results are shuffled between slots for operations like JOINs and GROUP BY.</li></ul><h3>4.3 Query Caching</h3><p>BigQuery caches query results for 24 hours. If the same query is executed again, BigQuery returns the cached result, saving time and resources.</p><h3>5. Storage Internals</h3><h3>5.1 Colossus</h3><p>Colossus is Google’s distributed file system that underpins BigQuery. It provides:</p><ul><li><strong>Durability</strong>: Data is replicated across multiple zones.</li><li><strong>Scalability</strong>: Handles petabytes of data seamlessly.</li><li><strong>Performance</strong>: Low-latency access to data.</li></ul><h3>5.2 Capacitor File Format</h3><p>Capacitor is BigQuery’s columnar storage format. Key features include:</p><ul><li><strong>Compression</strong>: Reduces storage costs and improves query performance.</li><li><strong>Encoding</strong>: Efficient encoding schemes for faster data scanning.</li><li><strong>Indexing</strong>: Built-in indexing for faster lookups.</li></ul><h3>5.3 Partitioning and Clustering</h3><ul><li><strong>Partitioning</strong>: Divide large tables into smaller chunks based on a column (e.g., date). This reduces the amount of data scanned during queries.</li><li><strong>Clustering</strong>: Sort data within partitions based on one or more columns. This further improves query performance.</li></ul><h3>6. Performance Optimization</h3><h3>6.1 Best Practices for Query Performance</h3><ul><li>**Avoid SELECT ***: Select only the columns you need.</li><li><strong>Use Partitioning and Clustering</strong>: Reduce the amount of data scanned.</li><li><strong>Optimize JOINs</strong>: Use smaller tables on the left side of JOINs.</li><li><strong>Leverage Materialized Views</strong>: Precompute results for faster queries.</li></ul><h3>6.2 Slot Utilization</h3><ul><li><strong>On-Demand Pricing</strong>: Slots are dynamically allocated.</li><li><strong>Flat-Rate Pricing</strong>: Reserve slots for predictable workloads.</li></ul><h3>6.3 Query Caching</h3><p>Enable query caching to avoid reprocessing the same data.</p><h3>7 Real-world usecase</h3><ul><li><strong>Data warehousing &amp; business intelligence: </strong>BigQuery serves as a centralized data warehouse where data from various sources can be stored and analyzed. It allows businesses to make informed decisions by analyzing data across different dimensions. For instance, a company can use BigQuery to analyze sales data across regions, helping them understand market trends and customer preferences.</li><li><strong>Big data analytics &amp; reporting:</strong> BigQuery excels in handling massive datasets, making it ideal for big data analytics and reporting. It uses columnar storage to quickly read and aggregate data, enhancing the speed of SQL queries. A common use case is analyzing customer behavior patterns to design targeted marketing campaigns.</li><li><strong>Real-time analytics &amp; decision support:</strong> BigQuery’s capability for processing streaming data enables real-time analytics. This feature allows businesses to gain real-time insights and make immediate decisions. For example, a company can analyze website traffic data in real-time to optimize user experience and engagement.</li><li><strong>Machine learning &amp; AI development:</strong> BigQuery integrates seamlessly with machine learning tools, aiding in data preparation for AI models. It can load data from various data sources, process it, and prepare it for machine learning algorithms. A typical use case is preparing customer data for churn prediction models, helping businesses retain their customers by predicting and addressing churn factors.</li></ul><p>Reference:</p><p><a href="https://www.comp.nus.edu.sg/~vldb2010/">https://www.comp.nus.edu.sg/~vldb2010/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ab20ad370b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Leading the Future of AI: DeepSeek-R1]]></title>
            <link>https://medium.com/@manojch_10/pioneering-the-next-era-of-ai-deepseek-r1-9fea3d0a0ae4?source=rss-1b21f69d6381------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fea3d0a0ae4</guid>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[deepseek-r1]]></category>
            <dc:creator><![CDATA[Manoj Ch]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 03:34:44 GMT</pubDate>
            <atom:updated>2025-01-28T03:55:42.938Z</atom:updated>
            <content:encoded><![CDATA[<p>DeepSeek, a Chinese AI startup founded in 2023 by entrepreneur Liang Wenfeng, has rapidly emerged as a formidable player in the artificial intelligence landscape. On January 20, 2025, the company unveiled its groundbreaking AI model, DeepSeek-R1, which has garnered significant attention for its innovative approach and impressive capabilities.</p><h3><strong>Revolutionizing AI Training with Reinforcement Learning</strong></h3><p>Reinforcement learning (RL) is redefining the landscape of AI, and DeepSeek-R1 exemplifies this transformation. It’s not just an upgrade; it’s a leap forward, showcasing how large language models (LLMs) can solve complex problems and develop reasoning capabilities without the need for traditional supervised fine-tuning. Here’s an exploration of what makes this innovation so significant.</p><h3>Skipping Fine-Tuning, Yet Thriving</h3><p>DeepSeek-R1-Zero demonstrates that reinforcement learning alone can unlock advanced reasoning capabilities.</p><ul><li><strong>What’s groundbreaking?</strong> It achieves self-checking, reflective thinking, and the ability to form detailed, multi-step reasoning patterns — all without external fine-tuning.</li><li><strong>Why it’s impactful:</strong> The model adapts over time, tackling increasingly challenging tasks independently.</li></ul><h3>The Influence of Structured Data</h3><p>Imagine starting a task with a clear blueprint. That’s what structured, cold-start data provided:</p><ul><li>It enabled the model to produce well-organized and higher-quality outputs.</li><li>It accelerated training and enhanced overall performance.</li></ul><h3>Striking a Balance Between Usability and Safety</h3><p>DeepSeek-R1 prioritizes user needs while maintaining ethical standards:</p><ul><li><strong>User-focused outputs:</strong> Summaries and results are crafted to be clear and relevant.</li><li><strong>Safe interactions:</strong> Measures are in place to reduce risks of biased or harmful content.</li></ul><h3>Bridging Languages Seamlessly</h3><p>This model isn’t bound by language barriers. It handles reasoning across languages, often blending English and Chinese. However, ensuring consistent performance across multiple languages remains a challenge.</p><h3>Small Yet Powerful Models</h3><p>Efficiency is another hallmark of DeepSeek-R1. Its distilled versions prove that smaller models can punch well above their weight:</p><ul><li><strong>Example:</strong> The Qwen-32B Distilled Model achieved 72.6% accuracy on AIME 2024, outperforming many larger models.</li><li>Compact models like these make advanced reasoning tools more accessible and practical.</li></ul><h3>Performance That Speaks Volumes</h3><p>DeepSeek-R1 shines across various benchmarks:</p><h3>Problem-Solving and Logic</h3><ul><li>Achieved top scores in AIME 2024 and MATH-500.</li><li>Excelled in programming tasks with an impressive 96.3% in competitive coding.</li></ul><h3>Knowledge and Content Creation</h3><ul><li>Scored over 90% on tests for diverse knowledge areas.</li><li>Delivered exceptional results in tasks requiring summarization and open-ended answers.</li></ul><h3>How DeepSeek-R1 got to the “aha moment</h3><p>A particularly intriguing phenomenon observed during the training of DeepSeek-R1-Zero is the occurrence of an “aha moment”. This moment, as illustrated in Table 3, occurs in an intermediate version of the model. During this phase, DeepSeek-R1-Zero learns to allocate more thinking time to a problem by reevaluating its initial approach. This behavior is not only a testament to the model’s growing reasoning abilities but also a captivating example of how reinforcement learning can lead to unexpected and sophisticated outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jn7OwyB2lhBlFokXSYPUaA.png" /></figure><h3>Challenges and Future Possibilities</h3><h3>Where It Struggles</h3><ul><li>Fine-tuning language consistency, especially for multilingual scenarios, is still a hurdle.</li><li>Performance in coding benchmarks has room for growth due to limited RL-specific training data.</li></ul><h3>What’s Next</h3><ul><li>Broaden support for more languages to resolve consistency issues.</li><li>Enhance capabilities for tasks like dialogue generation and real-world coding applications.</li><li>Experiment with novel training approaches like rejection sampling to further boost performance.</li></ul><h3>Why This Is a Game-Changer</h3><p>DeepSeek-R1 isn’t just another step in AI evolution; it’s a shift in how we think about machine learning. By leveraging RL, it’s paving the way for smarter, more adaptable systems that can reason, learn, and grow. The future of AI has never looked brighter.</p><p><strong>Open-Source Commitment and Community Engagement</strong></p><p>DeepSeek has embraced an open-source philosophy by releasing DeepSeek-R1 under the MIT license. This move allows developers and researchers worldwide to access, modify, and build upon the model, fostering innovation and collaboration within the AI community. The open-source release challenges the proprietary nature of many existing AI models and encourages a more inclusive approach to AI development.</p><p><strong>Looking Ahead: The Future of AI Reasoning</strong></p><p>DeepSeek’s innovative use of reinforcement learning in training large language models represents a significant shift in AI research and development. By demonstrating that advanced reasoning capabilities can emerge through RL without supervised fine-tuning, DeepSeek-R1 paves the way for more efficient, accessible, and collaborative AI advancements. As the AI community continues to explore and build upon these developments, the potential for creating more intelligent and adaptable systems becomes increasingly attainable.</p><p>Ref:</p><p><a href="https://arxiv.org/pdf/2501.12948">https://arxiv.org/pdf/2501.12948</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fea3d0a0ae4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unlocking the Power of Stream Processing with Apache Beam]]></title>
            <link>https://medium.com/@manojch_10/unlocking-the-power-of-stream-processing-with-apache-beam-0100c7387ffa?source=rss-1b21f69d6381------2</link>
            <guid isPermaLink="false">https://medium.com/p/0100c7387ffa</guid>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[dataflow]]></category>
            <dc:creator><![CDATA[Manoj Ch]]></dc:creator>
            <pubDate>Mon, 27 Jan 2025 15:05:55 GMT</pubDate>
            <atom:updated>2025-01-28T03:42:05.511Z</atom:updated>
            <content:encoded><![CDATA[<p>Stream processing is becoming increasingly vital in a world where data flows in real-time from millions of sources. Whether you’re analyzing clickstreams, monitoring IoT devices, or processing financial transactions, stream processing enables timely insights and decisions. In this article, we’ll explore how Apache Beam simplifies stream processing and why it’s a powerful tool for developers and data engineers alike.</p><h3>What is Apache Beam?</h3><p>Apache Beam is an open-source, unified programming model for defining both batch and stream processing pipelines. It provides portability across various distributed processing backends like Apache Flink, Apache Spark, and Google Cloud Dataflow, allowing you to write your pipeline once and execute it on the platform of your choice.</p><h3>Core Concepts of Apache Beam:</h3><ul><li><strong>Pipeline:</strong> The entire workflow of your data processing job.</li><li><strong>PCollection:</strong> Represents a distributed dataset, either bounded (batch) or unbounded (stream).</li><li><strong>Transforms:</strong> Operations performed on PCollections, like mapping, filtering, or aggregations.</li><li><strong>Windowing:</strong> Divides unbounded PCollections into logical windows based on time.</li><li><strong>Triggers:</strong> Determines when to emit results for each window.</li></ul><h3>Why Use Apache Beam for Stream Processing?</h3><ol><li><strong>Unified Batch and Streaming Model:</strong> Apache Beam’s unified model allows developers to focus on logic without worrying about underlying execution nuances.</li><li><strong>Portability:</strong> Develop pipelines once and run them on multiple backends without rewriting code.</li><li><strong>Rich Set of Features:</strong> Beam provides advanced features like windowing, event-time processing, and customizable triggers, making it a powerful framework for real-time processing.</li><li><strong>SDK Support:</strong> Beam offers SDKs for Java, Python, and Go, catering to diverse developer ecosystems.</li></ol><h3>Key Components of a Stream Processing Pipeline</h3><h3>1. Setting Up the Environment</h3><p>First, choose your SDK and install Apache Beam. For Python, you can install it with:</p><p>pip install apache-beam</p><h3>2. Define Your Pipeline</h3><p>The pipeline typically consists of these steps:</p><ul><li><strong>Data Source:</strong> Ingest data from sources like Pub/Sub, Kafka, or a file.</li><li><strong>Transformations:</strong> Apply operations to process and enrich data.</li><li><strong>Windowing and Triggers:</strong> Organize data into time-based windows for aggregation.</li><li><strong>Output:</strong> Send the processed data to sinks like BigQuery, Elasticsearch, or storage systems</li></ul><h3>3. Example Code: WordCount in Streaming Mode</h3><p>Here’s a simple example to process a stream of text and count word occurrences:</p><p>import apache_beam as beam<br> import re</p><p># Function to split a line of text into words using regex<br> def split_into_words(line):<br> return re.split(r’\W+’, line)</p><p># Example pipeline<br> with beam.Pipeline() as pipeline:<br> (<br> pipeline<br> | ‘Read input data’ &gt;&gt; beam.Create([<br> “Hello, World! This is an example.”,<br> “Apache Beam makes data processing easy.”,<br> “Regex can help in text splitting.”<br> ])<br> | ‘Split into words’ &gt;&gt; beam.FlatMap(split_into_words)<br> | ‘Filter empty words’ &gt;&gt; beam.Filter(lambda word: word.strip()) # Remove empty strings<br> | ‘Write output’ &gt;&gt; beam.Map(print) # Print results<br> )</p><h3>Execution</h3><p>Run the pipeline on your preferred runner, such as Google Cloud Dataflow or Apache Flink, by specifying the necessary configurations.</p><h3>Advanced Features</h3><ul><li><strong>Event-Time Processing:</strong> Ensures that late-arriving data is still processed correctly.</li><li><strong>Side Inputs and Outputs:</strong> Share additional data with transforms or write to multiple destinations.</li><li><strong>State and Timers:</strong> Maintain stateful information across streaming events.</li></ul><h3>Real-World Use Cases</h3><ol><li><strong>Log Analytics:</strong> Process logs in real-time to detect anomalies.</li><li><strong>IoT Analytics:</strong> Monitor sensor data streams for predictive maintenance.</li><li><strong>Fraud Detection:</strong> Identify suspicious transactions as they occur.</li></ol><h3>Conclusion</h3><p>Apache Beam empowers developers with a robust framework for building scalable and portable stream processing pipelines. Its flexibility and portability make it an excellent choice for tackling real-time data challenges. Whether you’re working on a small prototype or a large-scale production pipeline, Apache Beam has you covered.</p><p>Ready to dive into stream processing? Start building with Apache Beam today and unlock the power of real-time data!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0100c7387ffa" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>