<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shamim Bhuiyan on Medium]]></title>
        <description><![CDATA[Stories by Shamim Bhuiyan on Medium]]></description>
        <link>https://medium.com/@shamim_ru?source=rss-c3b65fc9a1f8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*gDbbhT13ylqASHVy.jpg</url>
            <title>Stories by Shamim Bhuiyan on Medium</title>
            <link>https://medium.com/@shamim_ru?source=rss-c3b65fc9a1f8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:55:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shamim_ru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Microsoft Clarity Agent: A Thinking Partner Before You Start Building]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/microsoft-clarity-agent-a-thinking-partner-before-you-start-building-955117b8584d?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/1040/1*ahT611vAhBJViQIkElkPEw.png" width="1040"></a></p><p class="medium-feed-snippet">Note: this is not Microsoft&#x2019;s web analytics product, which shares the same name. It&#x2019;s a separate open-source project.</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/microsoft-clarity-agent-a-thinking-partner-before-you-start-building-955117b8584d?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/microsoft-clarity-agent-a-thinking-partner-before-you-start-building-955117b8584d?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/955117b8584d</guid>
            <category><![CDATA[sdlc]]></category>
            <category><![CDATA[pdlc]]></category>
            <category><![CDATA[microsoft-clarity-agent]]></category>
            <category><![CDATA[sdd]]></category>
            <category><![CDATA[ai-thinking-partner]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 09:52:16 GMT</pubDate>
            <atom:updated>2026-11-11T09:53:08.191Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[I Made My Local Coding Agent Know When to Give Up]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/i-made-my-local-coding-agent-know-when-to-give-up-7a4126eaa089?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/1020/1*hms3s0uCE7J9oNz8HCbnRA.png" width="1020"></a></p><p class="medium-feed-snippet">A debugging journal: porting a model-escalator plugin to OpenCode 2.x, making a local Qwen model hand off to DeepSeek only after it&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/i-made-my-local-coding-agent-know-when-to-give-up-7a4126eaa089?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/i-made-my-local-coding-agent-know-when-to-give-up-7a4126eaa089?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a4126eaa089</guid>
            <category><![CDATA[conditional-model-handoff]]></category>
            <category><![CDATA[model-escalation-policy]]></category>
            <category><![CDATA[cost-optimized-ai-agent]]></category>
            <category><![CDATA[hybrid-ai-coding-agent]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 17:53:27 GMT</pubDate>
            <atom:updated>2026-11-04T17:54:38.815Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Automating architecture decisions in OpenSpec + Superpowers frameworks: Pre-check, Classify…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/automating-architecture-decisions-in-openspec-superpowers-frameworks-pre-check-classify-6ec5d2d579fb?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/1440/1*kNQWn19B1QGdTQLL5lry0A.png" width="1440"></a></p><p class="medium-feed-snippet">Disclaimer: Everything I describe in this post is an experiment &#x2014; a working prototype that I built in my own time to explore a specific&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/automating-architecture-decisions-in-openspec-superpowers-frameworks-pre-check-classify-6ec5d2d579fb?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/automating-architecture-decisions-in-openspec-superpowers-frameworks-pre-check-classify-6ec5d2d579fb?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ec5d2d579fb</guid>
            <category><![CDATA[superpowers]]></category>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[openspec]]></category>
            <category><![CDATA[sdd]]></category>
            <category><![CDATA[nimble]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 13:27:12 GMT</pubDate>
            <atom:updated>2026-11-09T17:56:39.515Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From product intent to production: where the Enterprise Architect fits in AI-Native SDD pipeline]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/from-product-intent-to-production-where-the-enterprise-architect-fits-in-ai-native-sdd-pipeline-5884b1bb2175?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/790/1*Jz68L02yhks7t36FmYVn1g.png" width="790"></a></p><p class="medium-feed-snippet">Over the last few weeks, I have been working on several pieces of the same engineering problem:</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/from-product-intent-to-production-where-the-enterprise-architect-fits-in-ai-native-sdd-pipeline-5884b1bb2175?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/from-product-intent-to-production-where-the-enterprise-architect-fits-in-ai-native-sdd-pipeline-5884b1bb2175?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/5884b1bb2175</guid>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[google-okf]]></category>
            <category><![CDATA[pdlc]]></category>
            <category><![CDATA[sdd]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 14:43:06 GMT</pubDate>
            <atom:updated>2026-10-10T09:29:08.156Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From a Local Knowledge Brain to Architecture artifact]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/from-a-local-knowledge-brain-to-architecture-artifact-73eedcf32795?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/600/0*EQeSrT7JyV5yWJEc.png" width="600"></a></p><p class="medium-feed-snippet">In my last two posts I built a local knowledge environment for my work as an Enterprise Architect.</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/from-a-local-knowledge-brain-to-architecture-artifact-73eedcf32795?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/from-a-local-knowledge-brain-to-architecture-artifact-73eedcf32795?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/73eedcf32795</guid>
            <category><![CDATA[enterprise-architecture]]></category>
            <category><![CDATA[archify]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[google-okf]]></category>
            <category><![CDATA[gbrain]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 13:44:29 GMT</pubDate>
            <atom:updated>2026-09-13T08:38:10.578Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From an AI Knowledge base to a Local Knowledge Brain]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/from-an-ai-knowledge-base-to-a-local-knowledge-brain-bca06736bbec?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/600/0*j5khFl40ar_4oZLr.png" width="600"></a></p><p class="medium-feed-snippet">In my last post I walked through how I built an AI Knowledge base compiler. The setup was: take a huge amounts of enterprise documents &#x2014;&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/from-an-ai-knowledge-base-to-a-local-knowledge-brain-bca06736bbec?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/from-an-ai-knowledge-base-to-a-local-knowledge-brain-bca06736bbec?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/bca06736bbec</guid>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[gbrain]]></category>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[google-okf]]></category>
            <category><![CDATA[qwencode]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Sun, 30 Aug 2026 10:32:34 GMT</pubDate>
            <atom:updated>2026-09-13T08:39:06.700Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building an AI Knowledge Compiler with Google’s Open Knowledge Format and Qwen Code CLI]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@shamim_ru/building-an-ai-knowledge-compiler-with-googles-open-knowledge-format-and-qwen-code-cli-83f0d5f3bd78?source=rss-c3b65fc9a1f8------2"><img src="https://cdn-images-1.medium.com/max/1521/0*KAxJrJnW16S0-9iZ.png" width="1521"></a></p><p class="medium-feed-snippet">When Google introduced the Open Knowledge Format (OKF), most discussions focused on the specification itself.</p><p class="medium-feed-link"><a href="https://medium.com/@shamim_ru/building-an-ai-knowledge-compiler-with-googles-open-knowledge-format-and-qwen-code-cli-83f0d5f3bd78?source=rss-c3b65fc9a1f8------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@shamim_ru/building-an-ai-knowledge-compiler-with-googles-open-knowledge-format-and-qwen-code-cli-83f0d5f3bd78?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/83f0d5f3bd78</guid>
            <category><![CDATA[knowledge-compiler]]></category>
            <category><![CDATA[local-llm]]></category>
            <category><![CDATA[qwencode]]></category>
            <category><![CDATA[google-okf]]></category>
            <category><![CDATA[knowledge-repository]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 17:17:00 GMT</pubDate>
            <atom:updated>2026-09-13T08:40:00.005Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[AI-driven observability platforms with Toto 2.0]]></title>
            <link>https://medium.com/@shamim_ru/ai-driven-observability-platforms-with-toto-2-0-d087f8ca8041?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d087f8ca8041</guid>
            <category><![CDATA[observability]]></category>
            <category><![CDATA[aiops-platform]]></category>
            <category><![CDATA[toto-2]]></category>
            <category><![CDATA[aiops]]></category>
            <category><![CDATA[predictive-monitoring]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Sat, 23 May 2026 13:54:51 GMT</pubDate>
            <atom:updated>2026-05-23T14:03:16.849Z</atom:updated>
            <content:encoded><![CDATA[<p>Datadog recently released <a href="https://github.com/datadog/toto#installation"><strong>Toto 2.0</strong></a><strong> </strong>(Time Series Optimized Transformer for Observability) as a foundation model for time-series forecasting, specifically optimized for observability data such as:</p><ul><li>infrastructure metrics,</li><li>application telemetry,</li><li>monitoring signals,</li><li>cloud system performance,</li><li>logs/metrics/traces patterns.</li></ul><p>In a nutshell, it’s like a “GPT-style foundation model” for machine telemetry and operational metrics over time rather than text. It can be used to predict abnormal application or infrastructure behavior before incidents occur.</p><p>Why is it so important?</p><p>Modern distributed systems generate enormous amounts of time-series data, including:</p><ul><li>CPU usage</li><li>Memory consumption</li><li>Request latency</li><li>API error rates</li><li>Kubernetes pod metrics</li><li>Network traffic</li><li>Database throughput</li><li>Cloud infrastructure telemetry</li></ul><p>Traditional forecasting systems often:</p><ul><li>Require task-specific training</li><li>Struggle with noisy telemetry</li><li>Fail on sparse or irregular signals</li><li>Do not generalize well across services</li></ul><blockquote>One of the main pain points for teams is fine-tuning or training ML models on task-specific datasets. Fine-tuning or training a model from scratch is both time-consuming and expensive. Not every company or team can afford the required resources and infrastructure.</blockquote><blockquote>Unfortunately, I have seen several teams attempt to use general-purpose LLMs to predict abnormal behavior in systems and infrastructure.</blockquote><p>Toto was designed to act as a general-purpose forecasting engine for the observability signals mentioned earlier. Because of this, I decided to take a closer look at the foundation model and develop a simple application to test its features.</p><p>According to sources from Datadog, the Toto 2.0 model was trained on more than 1 trillion time-series data points, with approximately 75% of the training data coming from real observability telemetry collected by the company. As a result, Toto 2.0 can be considered a large pre-trained transformer model for multivariate temporal data.</p><h3>Architecture overview.</h3><p>From an architectural perspective, Toto is based on a decoder-only Transformer architecture. It also incorporates specialized attention mechanisms designed for multivariate telemetry data. By borrowing concepts from both Large Language Models (LLMs) and Vision Transformers, Toto can efficiently process massive telemetry streams and perform scalable forecasting.</p><p>It is important to understand that Toto 2.0 is not a silver bullet. The model is not designed for tasks such as root-cause analysis or causal reasoning. Instead, it is intended to complement existing observability solutions. In a production environment, Toto would typically work alongside:</p><ul><li>tracing systems,</li><li>topology graphs,</li><li>alert correlation engines,</li><li>incident management systems.</li></ul><p>A modern observability platform could integrate Toto across multiple architectural layers, as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uToxexaTXq-oCRBM.png" /></figure><h3>Use cases.</h3><p>Let’s go through a few use cases where Toto could be a good fit. As a quick disclaimer, I do not have hands-on experience working in a modern operations team, so these use cases are based on my own perspective and understanding.</p><p>1. Predictive anomaly detection.</p><p>This is the most obvious use case. On the traditional monitoring:</p><pre>CPU &gt; 90% → alert</pre><p>In Toto-style monitoring, it should be something like these:</p><pre>CPU pattern predicts saturation in 15 minutes</pre><p>So, instead of traditional threshold-based alerts:</p><ul><li>the model learns normal temporal behavior,</li><li>forecasts future states.</li></ul><p>2. Incident precursor detection</p><p>This is where the Toto foundation model becomes especially powerful. In many cases, system outages are preceded by early warning signals or anomalous behavior patterns:</p><ul><li>queue buildup,</li><li>retry growth,</li><li>cache degradation,</li><li>network jitter,</li><li>thread pool saturation.</li></ul><p>Toto can learn multivariate temporal correlations such as</p><pre>Pattern A + Pattern B + Pattern C<br>→ often leads to incident type X</pre><p>This enables:</p><ul><li>earlier detection,</li><li>predictive incident management.</li></ul><p>3. Capacity forecasting</p><p>Most often, observability platforms constantly store telemetries:</p><ul><li>resource utilization,</li><li>request rates,</li><li>storage growth,</li><li>DB throughput.</li></ul><p>Toto can use these telemetries to forecast:</p><ul><li>cluster saturation,</li><li>storage exhaustion,</li><li>bandwidth limits,</li><li>database scaling needs.</li></ul><p>For example,</p><pre>Kafka retention storage will exhaust in 9 days.</pre><p>However, there are many additional use cases where Toto could provide significant value, such as intelligent alert correlation, root-cause assistance, and pattern similarity search.</p><p>For now, this overview should be enough. If you would like to explore the topic further, please refer to the official product documentation.</p><h3>Hands on lab.</h3><p>Toto 2.0 introduces several new features, including Zero-Shot Forecasting, High-Dimensional Support, and Probabilistic Predictions, among others. In this short tutorial, I will focus on Probabilistic Predictions, as this is one of the more practical and commonly applicable use cases for operational teams.</p><p>For a simple demonstration, I chose the following example:</p><pre>Predict future database load using application telemetry logs and metrics.</pre><p>The example simulates:</p><ul><li>PostgreSQL query throughput</li><li>active DB connections</li><li>API request spikes</li><li>slow queries</li><li>cache misses</li></ul><p>Then Toto 2.0 forecasts future DB load. This is useful for:</p><ul><li>proactive autoscaling,</li><li>saturation prediction,</li><li>SLA protection.</li></ul><p>The steps of the demo are shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2oZ1wfcMesZZfh4K.png" /></figure><h3>Let’s get started.</h3><p><strong>Step 1.</strong> Install Toto 2.0</p><p>I am using Google Colab to develop and run the application. You can either download the .ipynb file from the <a href="https://colab.research.google.com/drive/1iRM0A-5F3QpoCETZ7rFSqnbupB8uip11?usp=sharing">repository</a> and run it directly, or create a new Colab project and copy the code snippets from this page.</p><pre>Change the runtime type to T4 GPU</pre><p>Install Toto 2.0</p><pre>!pip install -q &quot;toto-2 @ git+https://github.com/DataDog/toto.git#subdirectory=toto2&quot;<br>!pip install -q pandas matplotlib numpy</pre><p><strong>Step 2.</strong> Imports</p><pre>import numpy as np<br>import pandas as pd<br>import torch<br>import matplotlib.pyplot as plt</pre><pre>from toto2 import Toto2Model</pre><p><strong>Step 3.</strong> Check Cudo</p><pre>evice = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;</pre><pre>print(&quot;Using device:&quot;, device)</pre><p>If CUDA is enabled, you should see a similar output: &quot;Using device: cuda&quot;</p><p><strong>Step 4. </strong>Generate realistic observability telemetry</p><p>This simulates:</p><ul><li>business traffic cycles,</li><li>DB bottlenecks,</li><li>noisy production metrics.</li></ul><pre>np.random.seed(42)</pre><pre># Must be divisible by 32<br>n_steps = 1536</pre><pre>t = np.arange(n_steps)</pre><pre># Base database load<br>db_qps = (<br>    300<br>    + 80 * np.sin(2 * np.pi * t / 144)<br>    + 40 * np.sin(2 * np.pi * t / 48)<br>)</pre><pre># Add upward growth trend<br>db_qps += t * 0.25</pre><pre># Add random noise<br>db_qps += np.random.normal(0, 20, n_steps)</pre><pre># Add BIG overload spikes<br>for i in range(25):</pre><pre>    spike_start = np.random.randint(0, n_steps - 30)</pre><pre>    spike_size = np.random.randint(250, 500)</pre><pre>    db_qps[spike_start:spike_start+30] += spike_size</pre><pre>df = pd.DataFrame({<br>    &quot;db_qps&quot;: db_qps<br>})</pre><pre>print(df.head())</pre><p><strong>Step 5. </strong>Visualize observability metrics</p><pre>plt.figure(figsize=(15,6))</pre><pre>plt.plot(df[&quot;db_qps&quot;])</pre><pre>plt.title(&quot;Synthetic PostgreSQL Database Load&quot;)</pre><pre>plt.xlabel(&quot;Time&quot;)</pre><pre>plt.ylabel(&quot;Queries Per Second&quot;)</pre><pre>plt.show()</pre><p>You should see a similar output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*x8GEhrWXaQX2vAfi.png" /></figure><p><strong>Step 6.</strong> Prepare Toto input tensor</p><pre>data = df[&quot;db_qps&quot;].values.astype(np.float32)</pre><pre>mean = data.mean()<br>std = data.std()</pre><pre>data_norm = (data - mean) / std</pre><pre>print(&quot;Mean:&quot;, mean)<br>print(&quot;Std:&quot;, std)</pre><pre>tensor_data = torch.tensor(data_norm)</pre><pre># Add variable dimension<br>tensor_data = tensor_data.unsqueeze(0)</pre><pre># Add batch dimension<br>tensor_data = tensor_data.unsqueeze(0)</pre><pre>tensor_data = tensor_data.float()</pre><pre>tensor_data = tensor_data.to(device)</pre><pre>print(&quot;Tensor shape:&quot;, tensor_data.shape)</pre><p><strong>Step 7.</strong> Load Toto 2.0 model</p><pre>model = Toto2Model.from_pretrained(<br>    &quot;Datadog/Toto-2.0-22m&quot;<br>)</pre><pre>model = model.to(device)</pre><pre>model.eval()</pre><p>Toto 2.0 supports models from:</p><ul><li>4M</li><li>22M</li><li>313M</li><li>1B</li><li>2.5B parameters.</li></ul><p>I am using the 22M parameter model, which fits within the 16 GB memory available in Google Colab.</p><p>Step 8. Forecast future database load</p><pre>eries_ids = torch.tensor([[0]]).to(device)</pre><pre>target_mask = torch.ones_like(<br>    tensor_data,<br>    dtype=torch.bool<br>).to(device)</pre><pre>inputs = {<br>    &quot;target&quot;: tensor_data,<br>    &quot;series_ids&quot;: series_ids,<br>    &quot;target_mask&quot;: target_mask<br>}</pre><pre>prediction_length = 120</pre><pre>with torch.no_grad():</pre><pre>    forecast = model.forecast(<br>        inputs,<br>        horizon=prediction_length<br>    )</pre><pre>forecast_np = forecast.cpu().numpy()</pre><pre>print(&quot;Forecast shape:&quot;, forecast_np.shape)</pre><pre>THRESHOLD = 500</pre><pre>future_risk = predicted_db_qps &gt; THRESHOLD</pre><pre>if future_risk.any():</pre><pre>    risk_points = np.where(future_risk)[0]</pre><pre>    print(&quot;⚠️ Predicted database overload detected&quot;)</pre><pre>    print(<br>        f&quot;First overload predicted in &quot;<br>        f&quot;{risk_points[0]} future timesteps&quot;<br>    )</pre><pre>else:</pre><pre>    print(&quot;✅ No overload predicted&quot;)</pre><p>Here, you should see the following message:</p><pre>&quot;Predicted database overload detected. First overload predicted in 0 future timesteps.&quot;</pre><p>This indicates that Toto 2.0 predicts the database is already entering an overload state at the very beginning of the forecasting horizon.</p><p>Step 9. Visualize again</p><pre>historical_db_qps = df[&quot;db_qps&quot;].values</pre><pre>plt.figure(figsize=(16,6))</pre><pre>plt.plot(<br>    range(len(historical_db_qps)),<br>    historical_db_qps,<br>    label=&quot;Historical DB Load&quot;<br>)</pre><pre>plt.plot(<br>    range(<br>        len(historical_db_qps),<br>        len(historical_db_qps) + prediction_length<br>    ),<br>    predicted_db_qps,<br>    label=&quot;Forecasted DB Load&quot;<br>)</pre><pre>plt.axvline(<br>    x=len(historical_db_qps),<br>    linestyle=&quot;--&quot;<br>)</pre><pre># Overload threshold<br>plt.axhline(<br>    y=THRESHOLD,<br>    color=&#39;red&#39;,<br>    linestyle=&#39;--&#39;,<br>    label=&#39;Overload Threshold&#39;<br>)</pre><pre>plt.legend()</pre><pre>plt.title(&quot;Toto2 Forecast — PostgreSQL Database Load&quot;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xmIef76d5uy2IxWh.png" /></figure><p>Actually, what happened internally? We configured</p><pre>THRESHOLD = 500</pre><p>Then Toto2 predicted something like Predicted DB QPS =</p><p>495</p><p>510</p><p>517</p><p>Since: 517&gt;500, the system immediately triggered overload detection.</p><p>Conceptually, Toto2 is saying “Based on recent telemetry patterns, the database is expected to already be overloaded as soon as the forecast starts.”</p><p>The demo is not perfect but demonstrate how predictive AIOps systems work.</p><p>Instead of:</p><pre>CPU overloaded NOW</pre><p>Toto2 try to detect:</p><pre>CPU will overload soon</pre><p>This forms the foundation of basic predictive monitoring. From here, you can continue exploring the other features and capabilities provided by Toto 2.0.</p><p>The article was first published on <a href="https://www.shamimbhuiyan.ru/">https://www.shamimbhuiyan.ru</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d087f8ca8041" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How OceanBase Works Under the Hood]]></title>
            <link>https://medium.com/@shamim_ru/how-oceanbase-works-under-the-hood-eca28f15c354?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/eca28f15c354</guid>
            <category><![CDATA[distributed-sql-database]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[oceanbase]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Fri, 27 Feb 2026 10:56:44 GMT</pubDate>
            <atom:updated>2026-02-27T11:04:19.496Z</atom:updated>
            <content:encoded><![CDATA[<p>Two <a href="https://www.shamimbhuiyan.ru/blogs/quick-start-with-the-relational-distributed-database:-oceanbase">years ago</a>, I introduced <a href="https://en.oceanbase.com/">OceanBase</a> as a distributed relational database designed for strong consistency, horizontal scalability, and multi–data center deployment.</p><p>Since then, the system has evolved significantly. It now supports vector search and includes major architectural improvements. That progress motivated me to revisit OceanBase and explore its internals at a deeper level.</p><p>In this post, we’ll examine what actually happens inside OceanBase when you execute a simple SQL query — no marketing language, just implementation details.</p><p>We’ll cover:</p><ul><li>What happens when you write data</li><li>What happens when you read data</li><li>How replication works</li><li>How consistency is guaranteed</li></ul><p>Understanding how a distributed RDBMS operates under the hood is both practical and intellectually rewarding. The goal of this article is to break down those mechanics clearly and accurately, focusing purely on how the system works.</p><blockquote>Recently, much of the conversation has focused on higher-level abstractions like OpenClaw, Claude workers, and Claude skills. At some point, it started to feel repetitive. So I chose to pause the noise and return to core engineering — distributed databases, storage engines, replication mechanics.</blockquote><blockquote>If that sounds refreshing to you, keep reading.</blockquote><p>My setup is very simple as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/0*0sT_GHezXViGn2ZR.png" /></figure><h3>Basic concepts</h3><p>OceanBase DB documentations uses a lot of terms like OBServers, OBProxy, Zone and much more. These three components defines how an OceanBase cluster is physically organized. Let’s start from the OBServer.</p><ol><li>OBserver — The database node.</li></ol><p>In OceanBase, each physical machine runs a single process called: observer. OBServer is not just storage. It is the entire database engine running on one node. An OBServer includes:</p><ul><li>SQL engine (parser, optimizer, executor)</li><li>Transaction layer</li><li>Replication layer (Paxos)</li><li>Storage engine (MemTable + SSTables)</li><li>Log service (redo/WAL)</li><li>Resource management (multi-tenant isolation)</li></ul><p>Note that, OceanBase is a shared-nothing architecture. That means:</p><p>- Every OBServer is fully independent.</p><p>- No shared storage.</p><p>- No special hardware.</p><p>2. Zone — A Logical Group of OBServers</p><p>A Zone is not a machine. It is a logical grouping of OBServers. A zone represents a set of nodes with similar availability characteristics. In practice, this usually means:</p><ul><li>One datacenter</li><li>One availability zone</li></ul><p>If you deploy OceanBase across 3 datacenters:</p><p>DC1 → Zone A<br>DC2 → Zone B<br>DC3 → Zone C</p><p>Each zone may contains multiple OBServers as follows:</p><p>Zone A<br>- OBServer 1<br>- OBServer 2<br>Zone B<br>- OBServer 3<br>- OBServer 4<br>Zone C<br>- OBServer 5<br>- OBServer 6</p><p>3. OBProxy — The Smart Router</p><p>OBProxy is separate from OBServer. It is the access layer of OceanBase. Each SQL queries goes through the OBProxy.</p><p>OBProxy:</p><ul><li>Listens on MySQL protocol</li><li>Accepts client connections</li><li>Forwards SQL to correct OBServer</li><li>Automatically discovers cluster topology</li></ul><p>When your application sends SQL:</p><ol><li>OBProxy parses the statement.</li><li>Determines which Log Stream owns the data.</li><li>Finds the leader of that Log Stream.</li><li>Forwards the request directly to that OBServer.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*42KjBXXpD_O-wXj2.png" /></figure><p>After reviewing the core concepts, let’s dive into OceanBase’s storage and replication layers. Understanding these components will clarify how the read and write paths operate — and how replication is implemented under the hood.</p><p>4. Replication level: Log Stream</p><p>In OceanBase, a Log Stream is the basic unit of replication, consensus, and failover. In simple terms: A Log Stream is a group of data that shares the same replication log and leader.</p><blockquote>Think of it as a “data container” that moves and replicates together.</blockquote><p>Technically, A Log Stream is:</p><ul><li>A collection of table partitions (called tablets)</li><li>With one shared write-ahead log</li><li>Replicated using <a href="https://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos</a></li><li>Having one leader and multiple followers</li></ul><p>All data inside the same LS:</p><ul><li>Is replicated together</li><li>Fails over together</li><li>Moves together during load balancing</li></ul><p>You can think of every Log Stream as a small replicated database inside the cluster.</p><p>Each Log Stream:</p><ul><li>Has one leader</li><li>Has multiple followers</li><li>Owns a group of partitions (called tablets)</li><li>Replicates data using Paxos consensus</li></ul><p>5. Storage Level: WAL (Redo Log) and SStables (LSM Storage)</p><p>WAL (Redo Log) provides Durability + consensus of the data. On the other hand, SSTables gives the ability to Long-term data storage.</p><blockquote>If you familiar with Cassandra Database, you probably knows how SSTables works.</blockquote><p>SSTables are part of its LSM-tree storage engine. It is<strong> </strong>an immutable, sorted data file stored on disk. LSM storage engines idea is very simple:</p><ul><li>Writes go to memory first (Memtable).</li><li>Disk files are written sequentially and never updated in place.</li></ul><p>Now we are ready to go through the process how write/read works in Oceanbase db.</p><h3>What Happens When You Write Data?</h3><p>Let’s say you run:</p><pre>UPDATE accounts SET balance = balance - 100 WHERE id = 1;</pre><p>Here’s what happens step by step.</p><h4>Step 1 — Request Routing</h4><p>Your application connects through OBProxy.</p><p>OBProxy figures out which Log Stream owns this row and sends the request to the leader of that Log Stream. Only the leader can accept writes.</p><h4>Step 2 — Redo Log Is Created</h4><p>Before changing any data, OceanBase writes a redo log entry.</p><p>This redo entry contains:</p><ul><li>Transaction ID</li><li>Changed data</li><li>Log sequence number</li></ul><p>This is written to the Write-Ahead Log (WAL). If the server (OBServer) crashes after this point, the change can be recovered.</p><h4>Step 3 — Replication to Other Datacenters</h4><p>If OceanBase is deployed across 3 datacenters as shown earlier:</p><ul><li>The leader sends the redo log entry to follower replicas.</li><li>Followers write it to their own WAL.</li><li>Followers send acknowledgment back.</li></ul><p>The leader waits for a majority votes (2 out of 3).</p><p>Only after majority confirms:</p><ul><li>The transaction is considered committed.</li><li>The client receives success.</li></ul><h4>Step 4 — Data Goes to Memory (MemTable)</h4><p>After the redo log is safely replicated:</p><ul><li>The change is applied to an in-memory structure called MemTable.</li><li>MemTable keeps rows sorted by primary key.</li></ul><p>At this point:</p><ul><li>The data is durable (because WAL is replicated).</li><li>The data is visible to future reads.</li></ul><h4>Step 5 — Flush to SSTable</h4><p>When MemTable becomes large:</p><ul><li>It is frozen.</li><li>Written to disk as an SSTable.</li><li>SSTables are immutable and sorted.</li></ul><p>SSTables are built locally on each replica. The key point is that, they are not copied between servers. As a result, the write path looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I4aOn1yi7OD4V4yn.png" /></figure><h3>What Happens When You Read Data?</h3><p>Now let’s look at a simple read:</p><pre>SELECT balance FROM accounts WHERE id = 1;</pre><h4>Step 1 — Routing</h4><p>For strong consistency, the request goes to the Log Stream leader. OceanBase supports snapshot reads using a global timestamp service (GTS). Each read gets a snapshot timestamp.</p><h4>Step 2 — MVCC Snapshot</h4><p>OceanBase uses multi-version concurrency control (MVCC). Each row version has a commit timestamp.</p><p>When reading:</p><ul><li>The engine checks MemTable.</li><li>Then L0 SSTables.</li><li>Then L1 SSTables.</li><li>Then major SSTables.</li></ul><p>It selects the newest row version that is less than or equal to the snapshot timestamp.</p><p>This provides consistent reads even during concurrent writes.</p><p>Here is the high-level view of the read path:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tYQrLgEb7r_cyv9S.png" /></figure><p>Understanding OceanBase under the hood isn’t about memorizing components or internal module names. It’s about understanding the data flow — how reads, writes, and replication actually move through the system.</p><p>Once that flow becomes clear, the architecture stops feeling complex. The pieces connect naturally, and the design decisions start to make sense.</p><p>From here, the next logical step is to explore distributed query execution in more depth.</p><p>In the next part of this series, I’ll deploy a three-node OceanBase cluster and run targeted tests to explore its behavior in a real environment.</p><p>The article was first published on <a href="https://www.shamimbhuiyan.ru/">https://www.shamimbhuiyan.ru</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eca28f15c354" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From App Fatigue to AI Clarity: Building My Own Intelligent Wealth Dashboard]]></title>
            <link>https://medium.com/@shamim_ru/from-app-fatigue-to-ai-clarity-building-my-own-intelligent-wealth-dashboard-6420021123ee?source=rss-c3b65fc9a1f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6420021123ee</guid>
            <category><![CDATA[ai-wealth-management]]></category>
            <category><![CDATA[portfolio-dashboard]]></category>
            <category><![CDATA[passive-income-streams]]></category>
            <category><![CDATA[asset-tracking-app]]></category>
            <dc:creator><![CDATA[Shamim Bhuiyan]]></dc:creator>
            <pubDate>Thu, 04 Dec 2025 01:58:58 GMT</pubDate>
            <atom:updated>2025-12-04T01:58:58.577Z</atom:updated>
            <content:encoded><![CDATA[<p>Managing wealth today can feel like a logistical puzzle. As an investor, I found myself using a mobile app for stocks, a separate wallet for crypto, complex Excel sheets for real estate, and Google Docs for strategy. This fragmentation was more than an annoyance — it wasted my time and clouded my decision-making. Simply calculating my net worth required logging into multiple platforms and manually stitching the data together.</p><p>While some unified dashboards exist, they often have significant drawbacks. Many are expensive subscriptions, and others require uploading sensitive financial data to third-party servers — a major concern for me. I needed a solution that was centralized, intelligent, and truly private. So, I decided to build one.</p><p><strong>My Solution: A Unified Financial View</strong></p><p>I created a tool to address this fragmentation by offering a single, unified view. This goes beyond just displaying data; it enables actionable management powered by Generative AI. I built it as an open-source project to show how developers can use modern APIs to create “on-demand” financial advisors that operate locally on your device.</p><blockquote>You can view the project and code here: <a href="https://github.com/srecon/beyond9to6/tree/main/assest_monitor"><strong>Wealth Dashboard Project on GitHub</strong></a></blockquote><blockquote>Play with the <a href="https://wealtholio.vercel.app/">demo</a>.</blockquote><p><strong>How My Dashboard Works</strong></p><p>This privacy-focused web application tracks, visualizes, and analyzes an entire financial life. Here is the main dashboard view, which provides an immediate overview of net worth, liabilities, and key income metrics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*7iRHwyTCpAPIIhUM.png" /></figure><p>The tool aggregates diverse assets — from crypto and stocks to real estate and gold — alongside liabilities and income streams. It uses Google’s Gemini API to function as a personal financial advisor, providing detailed analysis of net worth, diversification, and progress toward goals.</p><p><strong>1. Holistic Data Integration</strong><br>The tool solves the data silo problem by accepting information from multiple sources. You can upload complex Excel files with multiple sheets or connect directly to Google Sheets. Its “Smart Asset Classification” automatically sorts inputs into categories like:</p><ul><li><strong>Foundation Assets:</strong> Cash, Real Estate</li><li><strong>Growth Assets:</strong> Stocks, Gold</li><li><strong>High-Risk Assets:</strong> Crypto, Startups</li></ul><p>This automatic sorting separates Capital Assets (which build Net Worth) from Income Sources (which generate Cashflow), providing a clear picture of financial health.</p><p><strong>2. Visualizing Risk: The Wealth Pyramid</strong><br>A key strength of this approach is making abstract concepts visual. The dashboard features a Wealth Pyramid that illustrates risk structure from the bottom up. Instead of a list of numbers, you get an immediate visual of your financial stability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/0*EECZzlgU71HHWmqo.png" /></figure><p>If the pyramid is top-heavy (for example, too much crypto and not enough cash), the AI can flag this as a critical risk. You can also track future plans and see your top assets by value, making it easy to understand your financial standing at a glance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/0*PDspnFRBxbEG-FZD.png" /></figure><p><strong>3. A “Privacy First” Architecture</strong><br>The most critical aspect of the design is its security model. In an era of frequent data breaches, it uses a Client-Side Data Processing architecture. Your data never leaves your machine. Financial records are processed locally and stored only in your browser. When you ask for advice, only anonymized, aggregated context (like “70% in high-risk assets”) is sent to the AI API for analysis. This gives you the power of a large AI model without exposing private bank details to the cloud.</p><p><strong>The Tech Stack</strong><br>The project is built on a modern technology stack that balances speed and intelligence:</p><ul><li><strong>Frontend Speed (React &amp; Vite):</strong> The application runs as a fast, responsive Single Page Application (SPA).</li><li><strong>The AI Engine (Gemini API):</strong> This model is optimized for complex reasoning. It interprets data to assess risk exposure, suggest diversification, and evaluate progress toward goals.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*UbgpRLGadSIltfgf.png" /></figure><p><strong>Passive Income Focus</strong><br>Beyond net worth, a core feature is calculating and visualizing all income streams. This helps you see the complete picture of your earnings, from salary and deposits to royalties and business income, making it easier to plan for financial freedom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/0*SCH-7L456E-rK4hV.png" /></figure><p><strong>Current Limitations and Future Outlook</strong><br>This project is a working prototype and, like any new tool, has room to grow. Current limitations include:</p><ul><li><strong>Static Data Reliance:</strong> To maintain privacy, it relies on manual imports from Excel or CSV files rather than live connections to banks.</li><li><strong>Desktop Focus:</strong> The complex charts are currently best viewed on larger screens, though mobile responsiveness is a goal.</li></ul><p><strong>Conclusion</strong><br>I built this project to solve my own problem and to demonstrate how AI can be applied to everyday challenges. By tackling financial fragmentation — unifying brokers, crypto wallets, and spreadsheets into a single dashboard — this approach offers the power of a private, intelligent financial assistant without the high costs or privacy compromises.</p><p>The final step is getting that expert analysis. The AI can review your consolidated data — including liabilities and asset allocation — to provide professional-grade insights on demand, all while keeping your raw data private.</p><p>The article was first published on <a href="https://www.shamimbhuiyan.ru/">https://www.shamimbhuiyan.ru</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6420021123ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>