<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhruv Sachdev on Medium]]></title>
        <description><![CDATA[Stories by Dhruv Sachdev on Medium]]></description>
        <link>https://medium.com/@dhruvhsachdev?source=rss-85eb0660359e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*O5uD3e1AEuvkJUu7XyIKcw.jpeg</url>
            <title>Stories by Dhruv Sachdev on Medium</title>
            <link>https://medium.com/@dhruvhsachdev?source=rss-85eb0660359e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:55:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhruvhsachdev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I built CUTBACK to fix a video without recording it all again]]></title>
            <link>https://dhruvhsachdev.medium.com/i-built-cutback-to-revise-videos-when-their-facts-change-3bd3809b5b8d?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bd3809b5b8d</guid>
            <category><![CDATA[ai-editor]]></category>
            <category><![CDATA[video-editor-software]]></category>
            <category><![CDATA[graph-database]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[falkordb]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 17:36:16 GMT</pubDate>
            <atom:updated>2026-10-18T21:14:14.108Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever finished recording a video, only to realise that one of the facts in it needs to change?</p><p>Maybe a price was wrong. Maybe a product needs an accessory you thought was included. You can replace the sentence where you mention it, but that sentence might also support a chart, a comparison, and the recommendation at the end.</p><p>Now you have to find all of them. Then record new audio, update the graphics, check the captions, and make sure the ending still makes sense.</p><p>That was the problem I wanted to work on with CUTBACK.</p><p>I built it for the FalkorDB hackathon’s <strong>Agents That Act on Connected Data</strong> track. It takes a video and its source material, connects the claims in the video to their evidence, and uses those connections to decide which scenes need to change when a fact changes.</p><p>The output is another playable video. I can compare it with the original and inspect why each scene was selected for an edit.</p><h3>One missing adapter changes the recommendation</h3><p>I used a short comparison of two fictional speakers to make the problem concrete.</p><p>Orbit Mini starts at $79. Forma One costs $119 with everything required to use it. The original source sheet says Orbit’s required power adapter is included, so the video recommends Orbit as the cheaper option. The difference is $40.</p><p>Then I introduce a revised source sheet. The adapter is actually sold separately for $49.</p><p>The new calculation is straightforward:</p><pre>Orbit Mini: $79 + $49 = $128<br>Forma One:              $119<br>Difference:              $9</pre><p>Forma becomes the cheaper choice. Updating the $79 label alone would leave the video recommending the wrong speaker.</p><p>CUTBACK follows the adapter assumption to the cost comparison and the verdict. In the revised film, the chart shows $128 against $119, and the recommendation changes to Forma. The narration and captions use the new figures too.</p><p>The uploaded and revised cost scenes in CUTBACK.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zciE9TalK_XA-WCcwQGDPQ.png" /><figcaption><em>The original chart treats Orbit as a $79 complete purchase. The revised chart includes the $49 adapter.</em></figcaption></figure><p>I kept the products and source sheets fictional so the entire example could be uploaded, inspected, and shared with the repository.</p><h3>Starting with an upload and its sources</h3><p>CUTBACK starts with an empty project. I upload the MP4 and attach the original source sheets used by the video. The revised source is supplied separately, after reviewing the original claims.</p><p>Keeping those inputs separate matters. The initial graph needs to describe the evidence behind the uploaded film. The later source change gives the agent a specific fact to reconsider and a reason for changing it.</p><p>A new project with the video and original source material selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nKQBVXzzCE3SfvwZjLG28Q.png" /><figcaption><em>The two inputs are visible before analysis: the film and the source material behind it.</em></figcaption></figure><p>The backend extracts the audio and runs Whisper locally to produce timestamped speech passages. It also samples video frames. The OpenAI model receives those passages, the frames, and the source text, then proposes facts and the claims that depend on them.</p><p>There are several different objects here.</p><p>A <strong>source</strong> is the uploaded video or the supplied document. <strong>Evidence</strong> is a particular speech passage, sampled frame, or document passage. A <strong>fact</strong> is an underlying assertion, such as “the adapter is included.” A <strong>claim</strong> is something the video says on the strength of those facts, such as “Orbit costs $40 less.” A <strong>scene</strong> is the interval of the film where a claim appears.</p><p>The latest fresh-upload run produced seven facts and five claims across four scenes. Those counts belong to that run; the extraction is model-generated and can vary.</p><p>I added a review step before letting the agent act. Each fact shows its quoted passage and whether it came from the transcript, a video frame, or my source material. Speech timestamps let me jump back into the original film.</p><p>Extracted facts, source quotations and the original video.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vo8WVZxSREP3c5dU3RighA.png" /><figcaption><em>A fact has a source to check. It is still an extracted assertion, not an independently verified truth.</em></figcaption></figure><p>For text evidence, the server checks that the quoted passage occurs in the cited source. It also rejects links to missing facts or scenes.</p><p>These checks catch broken references. They cannot establish that a source is correct or that every inferred dependency is sensible. I still need to review those connections.</p><h3>Using FalkorDB to choose the edit scope</h3><p>The upload version stores this structure in FalkorDB:</p><pre>Source ─CONTAINS→ Evidence ─SUPPORTS→ Fact<br>Fact ─SUPPORTS→ Claim ─APPEARS_IN→ Scene</pre><p>The adapter fact can support several claims. A claim can appear in several scenes. When I change the fact, I need to follow those relationships and collect the affected scenes.</p><p>This is the traversal used by the application’s impact tool:</p><pre>MATCH p=(f:Fact {id:$factId})<br>        -[:SUPPORTS|APPEARS_IN*1..4]-&gt;(s:Scene)<br>RETURN s.id AS id,<br>       [n IN nodes(p) | n.title] AS path</pre><p>It starts at the selected fact and follows outgoing SUPPORTS and APPEARS_IN relationships. The *1..4 allows paths containing one to four relationships; the current fact-to-claim-to-scene paths use two.</p><p>I return the path as well as the scene ID because I want to explain the result. “Rebuild the verdict” is more useful when I can see which assumption and claim caused it to be included.</p><p>The application deduplicates scene IDs when multiple paths reach the same scene.</p><p>Each upload gets a separate graph named cutback_import_&lt;project ID&gt;. That keeps a rehearsal’s facts and exports out of the next project’s history. A small index graph holds the project list.</p><p>The backend uses the <a href="https://github.com/FalkorDB/falkordb-ts">FalkorDB JavaScript client</a>. In the store, the selected fact is passed separately as a query parameter:</p><pre>const result = await db.selectGraph(graphName).query(impactQuery, {<br>  params: { factId },<br>  TIMEOUT: 10000,<br>});</pre><p>The returned scene IDs become the permitted edit set. The server rejects a plan that adds an unrelated scene or omits an affected one. It queries FalkorDB again immediately before rendering and checks the plan against that result.</p><p>I tested this by removing a dependency edge in an isolated test graph. The corresponding scene disappeared from the allowed set.</p><p>The graph therefore changes what the application can do, as well as what its interface displays.</p><h3>Opening the same graph in FalkorDB Browser</h3><p>I wanted to show the stored relationships outside my own interface, so I also used <a href="https://github.com/FalkorDB/falkordb-browser">FalkorDB Browser</a>, the visual client.</p><p>In my local setup, the upload app connects to the database on port 6397, and Browser opens on 6398.</p><p>Browser’s connection form uses localhost:6379 because it reaches the database inside the container. The Node application uses the host-mapped port.</p><p>After selecting the project’s graph, I can run:</p><pre>MATCH p=(f:Fact {id:&#39;f3&#39;})<br>        -[:SUPPORTS|APPEARS_IN*1..4]-&gt;(s:Scene)<br>RETURN p</pre><p>Here, f3 is the adapter-included fact in my latest upload. Fact IDs can differ between runs. Returning p gives Browser the complete paths to draw.</p><p>The adapter fact and its dependency paths in FalkorDB Browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HqIwLB5RkqYNSI68iYNLHA.png" /><figcaption><em>This run’s graph reaches the cost scene, verdict, and closing scene. The extra closing-scene dependency exposed an extraction issue described below.</em></figcaption></figure><p>For a list of distinct affected scenes, I use:</p><pre>MATCH (f:Fact {id:&#39;f3&#39;})<br>      -[:SUPPORTS|APPEARS_IN*1..4]-&gt;(s:Scene)<br>RETURN DISTINCT s.id AS sceneId,<br>                s.title AS scene</pre><p>These are read-only queries. Opening the graph and inspecting the result does not make a model request.</p><h3>Giving the agent tools and a boundary</h3><p>The editing workflow uses one model-driven agent with four MCP tools:</p><ul><li>trace_impact queries FalkorDB for the affected scenes and dependency paths.</li><li>read_affected_scenes supplies the relevant scenes, claims, facts, and source passages.</li><li>calculate performs addition and subtraction, using cents for the price comparison.</li><li>submit_edit_plan submits the proposed scene titles, narration, and reasons.</li></ul><p>The MCP client and server communicate through an in-process transport. The client discovers the tool schemas, and the application supplies those definitions to the OpenAI Responses API.</p><p>When the model requests a function call, the application dispatches it through the MCP client and returns the tool result.</p><p>In the completed fresh-upload run, the activity log records impact tracing, a scene read, two calculator calls, and plan submission. The agent adds 7,900 and 4,900 cents, then compares the result with 11,900 cents.</p><p>I enforce the order where it matters. The agent must trace the impact before reading the affected scenes, and it must perform both steps before submitting a plan. The submitted scene IDs must match the graph result exactly. The planning loop also has a five-turn limit.</p><p>For the speaker film, the final chart values and narration are compiled from the corrected prices using its editable layouts. That keeps the displayed total, spoken total, and winner consistent.</p><p>General uploads still need a person to review the proposed prose. Matching the graph’s edit scope does not prove that every sentence is correct.</p><p>Once the plan is ready, I review it and approve the render.</p><h3>Rebuilding the scenes inside the video</h3><p>The goal was to carry a source change through the film itself.</p><p>If the complete price changes, the cost chart needs a new value and bar length. If the cheaper product changes, the verdict needs a different speaker illustration and recommendation. The narration and captions have to agree with both.</p><p>I built the speaker film around editable scene layouts. The renderer takes the revised values, calculates the totals and winner, and uses the result to generate the visuals and narration for the affected scenes.</p><p>That gives the comparison a consistent appearance while allowing its argument to change.</p><p>The revised verdict says:</p><blockquote><em>Forma One costs $9 less to get started. That makes it our lower-cost pick.</em></blockquote><p>The original Orbit verdict and revised Forma verdict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zciE9TalK_XA-WCcwQGDPQ.png" /><figcaption><em>The revised film recommends Forma and shows its $9 advantage using the same visual style as the original.</em></figcaption></figure><p>The uploaded file’s SHA-256 fingerprint identifies the speaker film’s editable layouts and cut points. This prevents an unrelated file with the same filename from receiving the wrong layout. A fresh project’s facts and dependencies still come from analysis of its inputs.</p><p>Other uploaded videos use editorial replacement scenes with local synthetic narration. This prototype does not reconstruct arbitrary footage, clone a speaker’s voice, or change a filmed person’s lip movements.</p><p>For rendering, Sharp converts SVG layouts into PNG frames. Local speech synthesis produces the narration, and FFmpeg assembles H.264 video with AAC audio. The export also includes SRT captions. Caption timing is estimated within segments; I have not implemented word-level alignment.</p><p>Combining retained footage with newly rendered scenes also requires compatible media streams. I normalise the clips to a 1280 × 720 canvas at 24 frames per second, with AAC audio at 48 kHz in stereo, before concatenating them. Videos with another aspect ratio are fitted inside that canvas.</p><p>Scenes outside the affected set retain their uploaded footage and audio through that normalisation.</p><p>Each completed export also records the source change and the scenes it rebuilt in FalkorDB, so I can inspect the edit later. The original upload and earlier exports remain available.</p><h3>A correct price can still produce a bad cut</h3><p>One problem appeared before I got to the model’s reasoning: the initial automatic segmentation cut through the middle of the cost scene.</p><p>The first three-segment split ended the opening segment at about 11.32 seconds. But the authored film’s cost scene starts at about 6.917 seconds.</p><p>Preserving that first segment would preserve several seconds of the old price graphic, even if the later replacement scene showed the right total.</p><p>I fixed the bundled film’s segmentation by using its actual four scene boundaries. That lets the renderer replace the complete cost scene and the complete verdict.</p><p>The fresh-upload run exposed a second, related problem.</p><p>Whisper placed the verdict passage between 20.52 and 26.10 seconds, while the closing scene begins at 25.75 seconds. My extractor currently attaches a whole passage to any scene it overlaps. That gave the closing scene some verdict text as input to the model.</p><p>The resulting graph linked the recommendation to the closing scene. FalkorDB returned three distinct scenes, and the renderer rebuilt all three. The closing narration remained the same because its editable layout contains the scope and fictional-product disclosure.</p><p>That run followed its stored graph, but the graph was broader than the intended edit. The earlier aligned example selected only the cost and verdict scenes. I cannot treat that two-scene result as a guarantee for fresh extraction.</p><p>This is the next part I want to improve: word-level timing, better claim-to-scene checks, and an editor for reviewing dependency edges.</p><p>The current review screen lets me inspect the connections and adjust fact wording, but it does not yet let me repair individual edges.</p><h3>Keeping projects separate and controlling API costs</h3><p>Each project needs its graph, source media, and exports to stay together. I gave every upload its own graph and media directory, with a project index for finding it again.</p><p>Starting another upload creates an empty project. Previous attempts stay behind the Projects button.</p><p>During development, I also kept the upload workflow in a separate Git worktree with its own app port, database port, Browser port, data directory, and Docker volume. That let me test changes while keeping a working version available. Separating only the code would not have protected its database or media files.</p><p>I also wanted to avoid paying to extract the same information during every rendering experiment.</p><p>For the supplied speaker film, I added a way to reuse its reviewed transcript, facts, and source change in a separate project. The application aligns the saved claims with the authored scene boundaries and queries FalkorDB for the edit scope again. Rendering can then proceed from that saved analysis without another model request.</p><p>I was working with $5 of API credit, so the expensive stages needed to be explicit.</p><p>Transcription, narration, playback, and rendering run locally. Analysis and agent planning use the API. I added a persisted $1 application allowance, estimated reservations before requests, and no automatic retries or fallback to another paid provider. The allowance is a local estimate, not my OpenAI account balance.</p><p>The backend also prevents overlapping jobs on the same project. If the process restarts during analysis, planning, or rendering, it marks the interrupted job as failed and retains earlier exports. This implementation assumes a single API process.</p><p>There were ordinary setup problems too.</p><p>Earlier in the build, a FalkorDB volume was mounted at /data while the image wrote to /var/lib/falkordb/data. I corrected the mount and checked revision persistence after recreating the container.</p><p>A nearly full disk interrupted npm installation and caused Docker I/O errors. Recovering disk space and reinstalling dependencies was necessary before application debugging could continue.</p><p>I also found a reporting bug while building the authored renderer. Restoring a previously rendered clip could report zero edits because the renderer had reused a cached file. But that clip could still differ from the preceding version.</p><p>I separated cache reuse from the comparison with the previous cut: whether a file needs rendering and whether a scene has changed are separate questions.</p><h3>Checking the result and the remaining work</h3><p>I checked the MP4s by decoding them and playing them in the browser. I also transcribed the revised audio locally to confirm that the spoken prices and recommendation matched the intended change.</p><p>The automated checks cover source quotations, graph references, edit-scope enforcement, MCP calls, project isolation, budget reservations, and browser playback.</p><p>For the editable speaker layout, I tested other adapter prices too. A $40 adapter produces a tie at $119, while a $20 adapter leaves Orbit cheaper at $99. The renderer is calculating the outcome from the supplied values.</p><p>There are still clear limits.</p><p>Each export applies one correction to the original upload; edits are not cumulative. Frame sampling can miss something visible between samples. The model can propose an unnecessary dependency, as the closing-scene example showed. Exact visual matching is available for the supplied editable film, while other footage gets editorial replacement scenes.</p><p>I want to extend the editable-layout approach to more video formats and make the evidence review more useful before adding more autonomous behaviour. A person should be able to inspect and repair the claim connections that will control an edit.</p><p>The moment I wanted from this project is visible in the two verdicts. The original film recommends Orbit. I change one assumption in its source material, follow the dependencies, and the revised film recommends Forma.</p><p>The explanation for that change is still there to inspect, from the source passage to the exported scene.</p><p>The code, demo video, synthetic source sheets, graph model, and local setup instructions are in the <a href="https://github.com/Dhruv-Sachdev1313/cutback">CUTBACK repository</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bd3809b5b8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The release still returned the right 503. It had become harder to debug.]]></title>
            <link>https://dhruvhsachdev.medium.com/the-release-still-returned-the-right-503-it-had-become-harder-to-debug-52f22687fe01?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/52f22687fe01</guid>
            <category><![CDATA[signoz]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[otel]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 16:34:42 GMT</pubDate>
            <atom:updated>2026-07-26T16:34:42.671Z</atom:updated>
            <content:encoded><![CDATA[<p>I started DebugProof with a small question: can a release pass its functional tests and still be unsafe to operate?</p><p>To test that, I built two versions of a checkout service. Both versions return the same 503 Service Unavailable response when the orders database times out. From the API client’s point of view, they behave exactly the same. The difference only appears when someone tries to investigate the failure.</p><p>Version 1 creates a database span, marks it as an error, records useful attributes, and writes a log connected to the trace. Version 2 returns the same response but drops the database span and writes a generic checkout failed log without trace context.</p><p>A normal API test sees no regression. DebugProof scores the first release 100/100 and the second 40/100.</p><h3>Turning an incident into a test</h3><p>DebugProof is an incident-centered release gate. It injects the same controlled incident into a known-good baseline and a candidate release, then compares the evidence they send to SigNoz.</p><p>I wanted to test the questions an on-call engineer would actually ask:</p><ul><li>Did the request fail?</li><li>Which dependency caused it?</li><li>Is there a useful log connected to the trace?</li><li>Where was the request time spent?</li><li>Is there a metric I can use in a dashboard or alert?</li></ul><p>I put these expectations into a YAML file that I call a diagnosability contract:</p><pre>name: checkout-database-timeout<br>service: checkout-service<br>scenario: database_timeout<br><br>probe:<br>  method: POST<br>  path: /checkout<br>  headers:<br>    X-DebugProof-Fault: database_timeout<br>  json:<br>    order_id: order-demo-42<br>    sku: signoz-hoodie<br>    quantity: 1<br>  expected_status: 503<br><br>evidence:<br>  - id: dependency-localized<br>    signal: traces<br>    check: span_attributes<br>    required: true<br>    expect:<br>      span_name: db.orders.lookup<br>      status: error<br>      attributes:<br>        db.system.name: postgresql<br>        server.address: orders-db<br>        error.type: TimeoutError<br><br>  - id: error-correlated<br>    signal: logs<br>    check: correlated_log<br>    required: true<br>    expect:<br>      severity: error<br>      body_contains: orders database timed out<br>      require_trace_id: true</pre><p>The contract isn’t part of OpenTelemetry. It is application policy. OpenTelemetry produces the traces, logs, and metrics; SigNoz stores and queries them; DebugProof decides whether the evidence is good enough.</p><p>OpenTelemetry calls traces, metrics, and logs different telemetry signals. In practice, I found that the useful part was not the presence of each signal by itself. It was the relationship between them. A log saying checkout failed exists, but it is much less useful when it cannot be connected to the failing trace. The <a href="https://opentelemetry.io/docs/concepts/signals/">OpenTelemetry signals documentation</a> is a good reference for the underlying model.</p><h3>Instrumenting the two releases</h3><p>The demo service is a FastAPI application. The baseline creates a manual child span around the database operation:</p><pre>with telemetry.tracer.start_as_current_span(&quot;db.orders.lookup&quot;) as span:<br>    span.set_attributes(<br>        {<br>            &quot;db.system.name&quot;: &quot;postgresql&quot;,<br>            &quot;db.operation.name&quot;: &quot;SELECT&quot;,<br>            &quot;server.address&quot;: &quot;orders-db&quot;,<br>            &quot;debugproof.probe.id&quot;: probe_id,<br>            &quot;debugproof.scenario&quot;: scenario,<br>        }<br>    )<br><br>    try:<br>        raise OrdersDatabaseTimeout(&quot;orders database timed out&quot;)<br>    except OrdersDatabaseTimeout as exc:<br>        span.set_attribute(&quot;error.type&quot;, &quot;TimeoutError&quot;)<br>        span.record_exception(exc)<br>        span.set_status(Status(StatusCode.ERROR, str(exc)))<br>        logger.exception(&quot;orders database timed out&quot;)<br>        raise</pre><p>The candidate deliberately removes that span. It also emits its log after clearing the active trace context:</p><pre>with _without_trace_context():<br>    logger.error(<br>        &quot;checkout failed&quot;,<br>        extra={<br>            &quot;debugproof.probe.id&quot;: probe_id,<br>            &quot;debugproof.scenario&quot;: scenario,<br>        },<br>    )</pre><p>This creates the failure I wanted to test. The application behavior stays the same, while the investigation experience becomes worse.</p><p>Each probe gets a unique debugproof.probe.id. I add it to spans and logs so DebugProof can find the exact telemetry generated by one run. I intentionally do not add it to metrics. A unique probe ID on every metric series would create unnecessary cardinality.</p><h3>Using SigNoz as the evidence store</h3><p>I ran SigNoz locally with Docker. The services exported telemetry over OTLP/HTTP to port 4318, while the SigNoz UI and query API were available on port 8080.</p><p>For my self-hosted setup, the important environment variables were:</p><pre>export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318<br>export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf<br>export OTEL_TRACES_EXPORTER=otlp<br>export OTEL_METRICS_EXPORTER=otlp<br>export OTEL_LOGS_EXPORTER=otlp</pre><p>Self-hosted SigNoz does not require an ingestion key for this local OTLP endpoint. The <a href="https://signoz.io/docs/ingestion/self-hosted/overview/">SigNoz self-hosted ingestion guide</a> documents the available endpoints and configuration.</p><p>I started both releases on different ports:</p><pre>SERVICE_VERSION=v1 DEBUGPROOF_BROKEN=false \<br>  uv run uvicorn demo_app.main:app --port 8001</pre><pre>SERVICE_VERSION=v2 DEBUGPROOF_BROKEN=true \<br>  uv run uvicorn demo_app.main:app --port 8002</pre><p>In SigNoz, the difference was easy to see. The baseline trace had this shape:</p><pre>POST /checkout<br>└── db.orders.lookup</pre><p>The candidate only had:</p><pre>POST /checkout</pre><p>The dashboard still showed failures for both versions. That turned out to be a useful part of the demo: metrics correctly told me that checkout was failing, but they could not tell me why the candidate was failing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oY6K1eqyAJ1_kR6tRD7ptA.png" /><figcaption>dashboard grouped by service.version</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fvD8JfLxmcR9OhRGsYwCyw.png" /><figcaption>V1 traces</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3vLq8Fl7mDPyL7vykIEtnw.png" /><figcaption>V2 traces</figcaption></figure><p>The Logs Explorer showed the same loss of context. Version 1 had orders database timed out with a trace ID and span ID. Version 2 had checkout failed without a useful connection to the trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*vARTvyzgqioPvoo7hmam-Q.png" /></figure><h3>The Query Builder detail that cost me time</h3><p>DebugProof reads the evidence back through the SigNoz Query Builder API. I created a dedicated service account with the managed viewer role and generated an API key. SigNoz documents this flow in its <a href="https://signoz.io/docs/manage/administrator-guide/iam/service-accounts/">service accounts guide</a>.</p><p>My first real query failed with:</p><pre>field `debugproof.probe.id` not found</pre><p>The attribute was definitely present in ClickHouse. I had already inspected the stored span and could see it in attributes_string. The problem was the field context.</p><p>My original filter used:</p><pre>debugproof.probe.id = &#39;...&#39;</pre><p>The working filter was explicit:</p><pre>service.name = &#39;checkout-service&#39;<br>AND resource.service.version = &#39;v1&#39;<br>AND attribute.debugproof.probe.id = &#39;...&#39;</pre><p>The same rule applied to selected fields. Custom span fields needed fieldContext: &quot;attribute&quot;, while service.version used the resource context. The <a href="https://signoz.io/docs/userguide/query-builder-v5/">Query Builder v5 guide</a> explains these field contexts, but this only clicked for me after seeing the ambiguous-field warning from my own query.</p><p>I also had to handle the real v0.133 response envelope. Rows arrived nested inside data.results[].rows[].data, so a parser that only looked for a top-level rows array silently missed the actual span fields.</p><h3>Waiting for all three signals</h3><p>The other issue was timing. My traces appeared before the metric export. The first working gate showed the baseline at 80/100 because DebugProof queried the metric too early.</p><p>In this application, spans and logs use batch processors, and metrics export every five seconds. I changed the collector to poll SigNoz for up to 35 seconds and refresh the query end time on every attempt. That matters because a fixed end time can exclude telemetry that arrives while the gate is waiting.</p><p>This was the final command:</p><pre>uv run debugproof run \<br>  --contract contracts/database-timeout.yaml \<br>  --baseline-url http://localhost:8001 \<br>  --candidate-url http://localhost:8002 \<br>  --backend signoz \<br>  --output debugproof-report.json</pre><p>And this was the result:</p><pre>Baseline v1:  PASS 100/100<br>Candidate v2: FAIL 40/100</pre><pre>Regressions:<br>- Database dependency could not be localized<br>- Useful error log was no longer correlated<br>- Request latency could no longer be attributed</pre><pre>DEPLOYMENT BLOCKED</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mC6ZcTgFx2e3Lx-329VPhg.png" /></figure><p>The command exits with a non-zero status because blocking the candidate is the expected result.</p><h3>What I took away from this</h3><p>The strongest lesson was that telemetry presence is a weak test. The candidate still had a failed request span, an error status, a log, and a failure metric. A dashboard could detect the incident. The missing part was the path from detection to explanation.</p><p>DebugProof is narrower than a general trace-testing framework such as Tracetest. Its focus is release comparison during a known incident, including the connections between traces, logs, and metrics. I think the two approaches can work together.</p><p>My next step is to run these contracts in GitHub Actions and post a Debuggability Passport on each pull request. I also want to detect when telemetry changes would break existing SigNoz dashboards, alerts, or SLOs.</p><p>For now, the project answers one question that I did not know how to test before this hackathon: when the next release fails, did we preserve enough evidence to find out why?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52f22687fe01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Instrumenting 5minpastebin.com with OpenTelemetry and SigNoz]]></title>
            <link>https://dhruvhsachdev.medium.com/instrumenting-5minpastebin-com-with-opentelemetry-and-signoz-4dc17228b560?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dc17228b560</guid>
            <category><![CDATA[otel]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[logs]]></category>
            <category><![CDATA[signoz]]></category>
            <category><![CDATA[trace]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Sun, 19 Jul 2026 07:41:25 GMT</pubDate>
            <atom:updated>2026-07-19T08:04:13.904Z</atom:updated>
            <content:encoded><![CDATA[<p>I wanted to add observability to <a href="http://5minpastebin.com/">5minpastebin.com</a> , my small open source pastebin that stores text in Redis and expires it after five minutes.</p><p>The app is deliberately boring:</p><ul><li>Go API</li><li>Echo for routing</li><li>Redis for storage</li><li>templ and HTMX for the web UI</li><li>one create endpoint</li><li>one read endpoint</li><li>one health endpoint</li></ul><p>There is enough going on to make traces, metrics, and logs useful, but not so much code that the instrumentation gets lost inside framework noise.</p><p>The goal was not “send something to a dashboard”. I wanted to understand where the OpenTelemetry SDK belongs in a Go service, where framework instrumentation stops, where manual spans make sense, and how to avoid leaking paste contents or paste IDs into telemetry.</p><p>The backend I had in mind was SigNoz, but I did not want SigNoz-specific code in the app. The app exports OTLP. SigNoz, the OpenTelemetry Collector, and many other backends can receive OTLP.</p><p>For the screenshots in this post, I used SigNoz locally on Docker. I am deliberately not turning this into a SigNoz installation guide, because that would make the post much longer and the official docs will stay more current than anything I write here. The only local details that matter for the app are the OTLP endpoints: `4317` for OTLP/gRPC and `4318` for OTLP over HTTP/protobuf.</p><p>One local setup gotcha: finish SigNoz’s first-run account and organization setup before sending telemetry. In my local run, the collector only reloaded into the real traces/metrics/logs pipeline after the org existed. Before that, the app could start normally, but exporter flushes failed with connection resets.</p><h3>The mental model I used</h3><p>OpenTelemetry in Go has a clean split:</p><ul><li>The API is what instrumented code uses.</li><li>The SDK is what records and exports telemetry.</li><li>Exporters send telemetry somewhere.</li><li>Resources describe the process producing the telemetry.</li><li>Propagators move trace context across process boundaries.</li></ul><p>That split matters. Importing `go.opentelemetry.io/otel` does not magically export anything. If the global tracer provider or meter provider is not set, most instrumentation quietly becomes no-op.</p><p>So I added a small package called `internal/observability`. The rest of the app does not know about SigNoz. It just records spans, metrics, and logs.</p><pre>func main() {<br> shutdownTelemetry, err := observability.Init(context.Background())<br> if err != nil {<br> fmt.Fprintf(os.Stderr, &quot;failed to initialize OpenTelemetry: %v\n&quot;, err)<br> os.Exit(1)<br> }<br> defer func() {<br> ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)<br> defer cancel()<br> if err := shutdownTelemetry(ctx); err != nil {<br> slog.Error(&quot;failed to shutdown OpenTelemetry&quot;, &quot;error&quot;, err)<br> }<br> }()<br>apiServer := server.NewServer()<br> // …<br>}</pre><p>The shutdown is not decorative. The trace, metric, and log exporters batch data. If the process exits without shutdown, the last few spans or logs can disappear.</p><h3>Config first, vendor later</h3><p>The app turns telemetry export on when either `OTEL_ENABLED=true` is set or an OTLP endpoint is configured.</p><p>For a local SigNoz or local OpenTelemetry Collector over gRPC:</p><pre>export OTEL_ENABLED=true<br>export OTEL_SERVICE_NAME=5minpastebin<br>export OTEL_EXPORTER_OTLP_PROTOCOL=grpc<br>export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317<br>export OTEL_EXPORTER_OTLP_TIMEOUT=5000</pre><p>For OTLP over HTTP/protobuf:</p><pre>export OTEL_ENABLED=true<br>export OTEL_SERVICE_NAME=5minpastebin<br>export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf<br>export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318<br>export OTEL_EXPORTER_OTLP_TIMEOUT=5000</pre><p>For SigNoz Cloud, the endpoint and ingestion key come from SigNoz:</p><pre>export OTEL_ENABLED=true<br>export OTEL_SERVICE_NAME=5minpastebin<br>export OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.&lt;region&gt;.signoz.cloud:443<br>export OTEL_EXPORTER_OTLP_HEADERS=signoz-ingestion-key=&lt;key&gt;<br>export OTEL_RESOURCE_ATTRIBUTES=service.version=$(git rev-parse - short HEAD)</pre><p>For my local screenshots I used the HTTP/protobuf path:</p><pre>export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf<br>export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318</pre><p>One small Go-specific detail: the official Go exporters already read OTLP env vars such as `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, and `OTEL_EXPORTER_OTLP_TIMEOUT`. I still keep my own `OTEL_ENABLED` switch because I do not want a local dev run to spam exporter errors when no collector is running.</p><p>I also honor `OTEL_SDK_DISABLED=true` in my wrapper. The OpenTelemetry Go docs currently call out that the Go SDK does not support that env var by itself, so if you want it to work, your setup code has to check it.</p><h3>The SDK setup</h3><p>This is the central setup function. I create providers for all three signals:</p><ul><li>tracer provider</li><li>meter provider</li><li>logger provider</li></ul><p>I also install W3C trace context propagation.</p><pre>otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(<br> propagation.TraceContext{},<br> propagation.Baggage{},<br>))</pre><p>For this app, propagation is not exciting yet because there is only one service. Still worth doing. The first time another service calls 5minpastebin with a `traceparent` header, the incoming request can join the existing trace instead of starting a disconnected one.</p><p>The resource setup is where I attach service identity:</p><pre>res, err := resource.New(ctx,<br> resource.WithFromEnv(),<br> resource.WithTelemetrySDK(),<br> resource.WithHost(),<br> resource.WithOS(),<br> resource.WithAttributes(<br> attribute.String(“service.name”, cfg.ServiceName),<br> attribute.String(“deployment.environment.name”, cfg.Environment),<br> ),<br>)</pre><p>`service.name` is the one attribute I always want to see in SigNoz. Without it, multiple services become a pile of anonymous telemetry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*To_RM1LCfICJUcchWf6-Cg.png" /></figure><h3>Echo request traces</h3><p>For Echo, I used the official contrib instrumentation:</p><pre>e.Use(otelecho.Middleware(“5minpastebin”))</pre><p>That creates the server span for each request. A `POST /create_link` request gets a trace. A `GET /:id` request gets a trace. The middleware handles the framework-level work, so I do not need to hand-roll a span around every route just to know a request happened.</p><p>I did add child spans inside the two paste operations:</p><pre>ctx, span := otel.Tracer(<br> “github.com/Dhruv-Sachdev1313/5minpastebin.com/internal/server”,<br>).Start(c.Request().Context(), “paste.create”)<br><br>defer span.End()<br><br>content := c.FormValue(“content”)<br>span.SetAttributes(<br> attribute.Int(“paste.content_bytes”, len(content)),<br> attribute.Int(“paste.ttl_seconds”, 300),<br>)</pre><p>The parent span answers “what HTTP request was this?” The child span answers “what was my app doing inside that request?”</p><p>I kept the attributes boring on purpose. Content length is safe. TTL is safe. Paste content is not safe. Paste IDs are also not safe because the ID is the access token for the paste.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hpCjgslnOeYFf66C6NulzA.png" /></figure><h3>Metrics I actually wanted</h3><p>There are two kinds of metrics here.</p><p>First, generic request metrics:</p><pre>pastebin.http.server.requests<br>pastebin.http.server.duration</pre><p>These use route templates, not raw URLs:</p><pre>attrs := metric.WithAttributes(<br> attribute.String(“http.route”, route),<br> attribute.String(“http.request.method”, method),<br> attribute.Int(“http.response.status_code”, statusCode),<br>)</pre><p>That detail is easy to miss. If I used `/abc123` as a label value, every paste would create a new time series. Route `/:id` is the useful label. The raw paste ID is both high-cardinality and private.</p><p>Then there are app metrics:</p><pre>pastebin.pastes.created<br>pastebin.pastes.read<br>pastebin.paste.content.bytes<br>pastebin.redis.keys</pre><p>`pastebin.pastes.read` has outcomes like `hit`, `miss`, and `error`. That is more useful than just counting requests to `/:id`, because a lot of reads can be expired links.</p><p>For Redis key count, I used an observable gauge:</p><pre>gauge, err := m.meter.Int64ObservableGauge(<br> “pastebin.redis.keys”,<br> metric.WithDescription(“Current number of keys in Redis.”),<br> metric.WithUnit(“{key}”),<br>)</pre><p>The callback calls `DBSIZE`:</p><pre>return s.redis.DBSize(ctx).Result()</pre><p>This is a reasonable metric only because this app owns its Redis DB. If you share Redis DB 0 with other apps, `DBSIZE` no longer means “active pastes”. It means “all keys in this Redis database”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n-CXpR2Yf-rxZfvSTETOMQ.png" /></figure><h3>Redis spans need context</h3><p>This was the part that made the tracing model click for me.</p><p>Before instrumentation, my database interface did not accept request context:</p><pre>SetKeyWithExpiration(string, string) error<br>GetData(string) (string, error)</pre><p>That is fine for basic Redis calls, but not for tracing. The active span lives in `context.Context`. If Redis does not receive that context, the Redis span cannot become a child of the HTTP request span.</p><p>So the interface became:</p><pre>SetKeyWithExpiration(context.Context, string, string) error<br>GetData(context.Context, string) (string, error)</pre><p>The Redis client itself is instrumented with go-redis’ `redisotel` package:</p><pre>if err := redisotel.InstrumentTracing(rdb, redisotel.WithDBStatement(false)); err != nil {<br> slog.Warn(“failed to instrument Redis tracing”, “error”, err)<br>}<br>if err := redisotel.InstrumentMetrics(rdb); err != nil {<br> slog.Warn(“failed to instrument Redis metrics”, “error”, err)<br>}</pre><p>I passed `WithDBStatement(false)` because raw Redis commands can include keys and values. This app should not put paste data or paste tokens into spans.</p><h3>Logs: stdout first, OTel second</h3><p>I did not want to choose between normal container logs and OpenTelemetry logs. The app now uses `slog`, and the handler fans records out to stdout and, when enabled, to the OpenTelemetry logs pipeline.</p><pre>handlers := []slog.Handler{stdout}<br>if cfg.LogsEnabled {<br> handlers = append(handlers, otelslog.NewHandler(<br> “github.com/Dhruv-Sachdev1313/5minpastebin.com”,<br> otelslog.WithSource(true),<br> ))<br>}<br><br>slog.SetDefault(slog.New(fanoutHandler{handlers: handlers}))</pre><p>There is one thing to be honest about: Go logs in OpenTelemetry are newer and less settled than traces and metrics. The OpenTelemetry Go docs describe log collection through bridges such as `slog`, and the direct-to-collector workflow is still marked experimental. I am fine with that here because stdout logs still exist even if I turn OTel logs off.</p><p>For local logs, I add trace and span IDs when a log happens inside a span:</p><pre>spanContext := trace.SpanContextFromContext(ctx)<br>if spanContext.IsValid() {<br> record.AddAttrs(<br> slog.String(“trace_id”, spanContext.TraceID().String()),<br> slog.String(“span_id”, spanContext.SpanID().String()),<br> )<br>}</pre><p>That gives me trace-log correlation even before I open SigNoz.</p><p>The request logger also avoids raw URIs:</p><pre>route := c.Path()<br>if route == “” {<br> route = “unknown”<br>}</pre><p>Logging `/health` is fine. Logging `/somePasteToken` is not.</p><h3>A small bug I found while adding telemetry</h3><p>The route order used to register `/:id` before `/health`. With Echo, that means `/health` can be treated as a paste ID if the catch-all route wins first.</p><p>The fixed order is:</p><pre>e.POST(“/create_link”, s.createLinkHandler)<br>e.GET(“/health”, s.healthHandler)<br>e.GET(“/:id”, s.serveContentWithIdHandler)</pre><p>This is one of the underrated parts of instrumentation work. You stare at the request path long enough to notice small routing mistakes.</p><h3>What I expect to see in SigNoz</h3><p>After starting the app with telemetry enabled, I create a paste and open the generated link once.</p><p>For `POST /create_link`, I expect:</p><ul><li>an HTTP server span from Echo</li><li>a `paste.create` child span</li><li>a Redis `SETEX` span</li><li>`pastebin.pastes.created` incremented with `paste.outcome=success`</li><li>a request log with the same trace ID</li></ul><p>For `GET /:id`, I expect:</p><ul><li>an HTTP server span</li><li>a `paste.view` child span</li><li>a Redis `GET` span</li><li>`pastebin.pastes.read` incremented with `hit` or `miss`</li></ul><p>For Redis:</p><ul><li>client metrics from `redisotel`</li><li>`pastebin.redis.keys` reporting the current key count</li></ul><p>In my local run, the trace table showed exactly the shape I wanted: Echo request spans like `POST /create_link` and `GET /:id`, manual spans like `paste.create` and `paste.view`, and Redis spans like `setex`, `get`, `ping`, and `dbsize`. The logs also carried the same trace IDs, which made the log/trace correlation easy to verify without guessing.</p><h3>The privacy line I am trying not to cross</h3><p>Telemetry is data. It deserves the same suspicion as any other data path.</p><p>For this app, I do not record:</p><ul><li>paste contents</li><li>paste IDs</li><li>raw request URLs for paste reads</li><li>raw Redis statements</li></ul><p>I do record:</p><ul><li>route templates</li><li>status codes</li><li>durations</li><li>content size in bytes</li><li>hit/miss/error outcomes</li><li>Redis key count</li></ul><p>That is enough to debug the service without turning the observability backend into a second paste store.</p><h3>Sources I checked</h3><p>- OpenTelemetry Go docs: <a href="https://opentelemetry.io/docs/languages/go/">https://opentelemetry.io/docs/languages/go/</a><br>- OpenTelemetry Go exporters: <a href="https://opentelemetry.io/docs/languages/go/exporters/">https://opentelemetry.io/docs/languages/go/exporters/</a><br>- OpenTelemetry Go instrumentation: <a href="https://opentelemetry.io/docs/languages/go/instrumentation/">https://opentelemetry.io/docs/languages/go/instrumentation/</a><br>- SigNoz Go instrumentation guide: <a href="https://signoz.io/docs/instrumentation/opentelemetry-golang/">https://signoz.io/docs/instrumentation/opentelemetry-golang/</a><br>- go-redis `redisotel`: <a href="https://pkg.go.dev/github.com/redis/go-redis/extra/redisotel/v9">https://pkg.go.dev/github.com/redis/go-redis/extra/redisotel/v9</a><br>- Echo `otelecho`: <a href="https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho">https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho</a><br>- `otelslog`: <a href="https://pkg.go.dev/go.opentelemetry.io/contrib/bridges/otelslog">https://pkg.go.dev/go.opentelemetry.io/contrib/bridges/otelslog</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dc17228b560" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Solana — Part 1]]></title>
            <link>https://dhruvhsachdev.medium.com/introduction-to-solana-part-1-236835c416c3?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/236835c416c3</guid>
            <category><![CDATA[solana-network]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 07:36:03 GMT</pubDate>
            <atom:updated>2026-04-13T07:36:03.525Z</atom:updated>
            <content:encoded><![CDATA[<p>This series is an attempt to walk you into the world of Solana blockchain not by overwhelming you, but by letting things unfold naturally.</p><p>I’ll assume you already have a rough sense of what blockchains and smart contracts are. We won’t start from first principles, but we also won’t rush past the basics that actually matter when you begin building.</p><p>This first piece is about grounding and understanding what Solana is, why it exists, and how a few core pieces fit together.</p><h3><strong>So, why Solana?</strong></h3><p>Most blockchains ask you to make a trade-off: decentralization, security, or scalability (also called as Blockchain Trilemma), pick two and compromise on the third.</p><p>Solana is an attempt to bend that constraint.</p><p>At a surface level, two things stand out:</p><ul><li>It is fast! Transactions are confirmed in seconds, often faster.</li><li>It is inexpensive as well ,fees are tiny compared to many other chains.</li></ul><p>That combination matters more than it first appears. When you’re building applications especially consumer-facing ones, you need interactions to feel immediate and cheap. Otherwise, the product never quite feels like a product; it feels like infrastructure.</p><p>Underneath this speed is a different way of thinking about time and ordering in distributed systems, most notably through something called Proof of History. We’ll come back to that later, because it deserves its own space.</p><p>For now, it’s enough to say: Solana is designed so that the network doesn’t waste time agreeing on <em>when</em> something happened as it already has a shared, cryptographically provable sense of time.</p><h3><strong>Wallets and cryptography (what’s actually happening)</strong></h3><p>Before you can do anything on Solana, you need a wallet. And a wallet, at its core, is not an app — it’s a pair of cryptographic keys.</p><ul><li>A <strong>private key</strong>, which must remain secret</li><li>A <strong>public key</strong>, which can be shared freely</li></ul><p>When you “own” a wallet, what you really own is the private key.</p><p>When you send a transaction, you’re not just telling the network what to do, but you’re proving that the request came from you. This proof is created using a digital signature scheme. In Solana’s case, that’s Ed25519.</p><p>Here’s the flow in plain terms:</p><ul><li>You create a transaction</li><li>You sign it with your private key</li><li>The network uses your public key to verify that signature</li></ul><p>If the verification passes, the network accepts that <em>you authorized this action</em> without ever seeing your private key.</p><p>That’s the entire trust model.</p><p>If you have the Solana CLI installed, generating this keypair locally is as simple as:</p><pre>solana-keygen new</pre><p>Registering the public key generated from this gives you your identity on the network. Everything that follows builds on top of this.</p><h3>Accounts and the System Program</h3><p>Now comes a subtle but important shift in thinking.</p><p>On Solana, everything is an <strong>account</strong>.</p><p>Your wallet is an account. Programs are accounts. Data lives inside accounts.</p><p>But just generating a keypair isn’t enough. The network doesn’t automatically “know” about your account until it is created on-chain.</p><p>This is where the System Program (Solana) comes in.</p><p>It’s one of Solana’s built-in programs, the kind that every node understands. You can think of it as the entry point for creating and managing accounts.</p><p>The System Program is responsible for:</p><ul><li>Creating new accounts (linked to public keys)</li><li>Allocating storage space for those accounts</li><li>Assigning ownership of accounts to programs</li></ul><p>That last point “<strong>ownership”</strong> is crucial.</p><p>An account in Solana isn’t just a container of data. It is always controlled by a program. Only the owning program is allowed to modify the account’s data.</p><p>So when you create an account, you’re also deciding <strong>which program gets to control it</strong>.</p><p>This design is what enables Solana’s performance model. Instead of every contract holding its own state internally, state is externalized into accounts, and programs operate on them.</p><p>We’ll go deeper into accounts in the next piece, because this is where Solana really starts to feel different from other chains.</p><h3><strong>Where this leaves us</strong></h3><p>At this point, you have just enough to orient yourself:</p><ul><li>A wallet is a cryptographic identity</li><li>Transactions are signed proofs of intent</li><li>Everything lives as an account</li><li>The System Program helps bring those accounts into existence</li></ul><p>That’s the skeleton of Solana.</p><p>In the next part, we’ll slow down around “<strong>accounts”, </strong>how they store data, how ownership works in practice, and why this model is both powerful and slightly unintuitive at first.</p><p><strong>Helpful resources</strong></p><p>If you want to explore while reading:</p><ul><li><a href="https://beta.solpg.io/">Solana Playground </a>— a quick way to experiment without setup</li><li><a href="https://solana.com/developers/cookbook">Solana Cookbook</a> — practical patterns and examples</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=236835c416c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC’21 Week 10 and Beyond!]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-week-10-and-beyond-f0f0fd09d99c?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0f0fd09d99c</guid>
            <category><![CDATA[google-summer-of-code]]></category>
            <category><![CDATA[gsoc2021]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[chaoss]]></category>
            <category><![CDATA[dependencies]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Wed, 25 Aug 2021 08:41:26 GMT</pubDate>
            <atom:updated>2021-08-25T10:00:04.334Z</atom:updated>
            <content:encoded><![CDATA[<p>Week 10 was the last week of the coding phase 2 and a lot happened in this week. We completed wrapping up the Libyear program as a worker and it is working perfectly!</p><h3><strong>Week 10 summary</strong></h3><p>We have discussed in previous blogs about the libyear worker, how it is being build and advantages of our libyear worker over other existing libyear calculating tools listed on Libyear website. I did mention in my last blog that we have built our own parsers and functions to calculate libyear. This week we made it into a complete worker and added function to identify the file type and call the right parser. Of course we faced a lot of bugs, but this time it was relatively easy to handle them as I had some experience while making a new model to get Scorecard data.</p><p>We also added parsers for Poetry and Poetry.lock files. We added support for Conda as Pip dependencies can be embedded in conda environment files too. This also completes the PYPI part of Libyear Worker.</p><h3>What Next?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*RSMhBZ2OYsn0OUx8g5WPng.gif" /></figure><p>Libyear Worker is a very evolving worker and we have plans to extend it to support different package managers as well! We can also improve a lot on the regular expressions that we use to get dependency data and requirement string from the files.</p><p>These 10 weeks of Google Summer of Code have undoubtedly been one of the best learning experience of my life not only in terms of my technical knowledge but I have also improved a lot on my communication skills.I am really grateful to my mentors <a href="https://github.com/sgoggins">Sean Goggins</a> and <a href="https://github.com/vinodkahuja">Vinod Ahuja</a> for giving me this opportunity and guiding me throughout. I would love to keep contributing to CHAOSS in every way possible!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0f0fd09d99c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSOC’21@ CHAOSS- Week 8&9]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-chaoss-coding-period-week-8-9-f9455e55ee7a?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9455e55ee7a</guid>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[gsoc2021]]></category>
            <category><![CDATA[chaoss]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 09 Aug 2021 08:21:45 GMT</pubDate>
            <atom:updated>2021-08-09T11:29:11.564Z</atom:updated>
            <content:encoded><![CDATA[<p>You must have read in the previous blogs that we are currently working on a new worker for augur to calculate Libyear. Weeks 8 and 9 have been really productive as we have made good progress in that direction.</p><h3><strong>Working of the Worker-</strong></h3><p>We can divide Worker into 3 parts —</p><p><strong>1- Parsers — </strong>Parsers parse the package files and get the list of dependencies and their version representations. Currently, we have only build parsers for the PYPI package manager and supporting file formats are — system.py, requirement.txt, pipfile, pipfile.lock, and poetry. I would be adding the support for conda soon. For these parsers, formats like TOML and JSON are easily handled, as we can parse them easily using python code. But the tricky part was parsing setup.py and requirement.txt. These parsers heavily rely on pattern string matching using regular expressions. Setup.py was the most tricky one as it has a lot more than just dependencies. Conda parsers would be parsing YAML files, so I am expecting it to be straightforward.</p><p><strong>2- Getting meta data for packages — </strong>This part is doing a lot of things. Firstly we get the dependency name and spec string from the parsers. Then it checks and cleans the spec string according to the way it is represented. For example, people might represent their dependencies in this way —</p><pre>“wheel”,</pre><pre>“psycopg2-binary==2.8.6”,</pre><pre>“click&lt;8.0,&gt;=5.1”,</pre><pre>“dask&gt;=2021.6.0” </pre><p>As you can see above, there are different kinds of spec strings for the dependencies. When there is no version specified we assume it to be the latest, when we have a format like ` &lt;8.0,&gt;=5.1 `, this considers version just before `8.0`, I will explain shortly the way we get the list of all the versions. So we get the current cleaned version according to the way it is represented. Once we have the cleaned version now we would need current version’s release date, latest version and its release date to calculate Libyear. As I have mentioned before, right now we are only supporting PYPI, so we get all the data from a PYPI url, which takes dependency name and version(if needed) as parameters and gives back a JSON response. This response has a lot of meta data, more than we need. I have also tried my best to minimize the call to PYPI as we don’t want to bother their servers as well as it also reduces the risk of getting a bad response i.e. ERROR 404. We have the list of releases with their release dates and time which we fetch according to the versions.</p><p>Now there are scenarios where the release date or version doesn’t exist. This is fairly common for really old PYPI packages. I have added error handling for this, if we do not find the release date then it automatically sets it to 1970–01–01 00:00:00 and sets the Libyear to -1. If we see -1 as Libyear, we would know that there would be some problem with the data and by this approach we also do not miss any dependencies in the files.</p><p><strong>3- Libyear Calculator — </strong>As the name suggests, this calculates the Libyear. We get all the release dates and it checks for the data, if we do not have the latest version or release dates, it returns -1 as its value. If the file does not specify any version or just have `&gt;=`, we assume the latest version, in this case we return Libyear as 0. For all other scenarios, we calculate Libyear by subtracting the current release date from the latest release date and dividing it by 365. This subtraction is done by using a python library `dateutil.parser`</p><p>This is how the current code is working. This code is written separately from augur and not as a worker because it may get way more complicated and time-consuming if we directly start with the worker. Augur already does a lot of things so sometimes debugging may get difficult and also for every small change we have to rebuild the entire instance to see it working. So it is always recommended to build a separate program and then wrap it up as a worker. So this is what I am currently doing, wrapping everything as a worker. I hope this would work as a worker too, fingers crossed!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*oB4qrXe3JovhQvFsa_muKg.gif" /><figcaption>and toes too please 🤣</figcaption></figure><p>`</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9455e55ee7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC’21 @ CHAOSS: Coding Period Week 6&7]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-chaoss-coding-period-week-6-7-4c3b3ac5173b?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c3b3ac5173b</guid>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[chaoss]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <category><![CDATA[gsoc2021]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 26 Jul 2021 08:50:16 GMT</pubDate>
            <atom:updated>2021-07-26T11:20:27.787Z</atom:updated>
            <content:encoded><![CDATA[<p>Week 6 was the start of Phase 2 of our Google Summer of Code. As I mentioned in my previous blogs, we are now working to build a new Worker in Augur for calculating the libyear which is a measure of dependency freshness. These weeks were a lot about researching and deciding on the architecture for the worker.</p><p><strong>But what is Libyear and dependency freshness?</strong></p><p>This is a good question, dependency freshness is a fancy way of telling how updated are your dependencies. Dependencies have a version, all developers keep updating them which fixes bugs, makes it more stable and, improves a lot of other things. So it is always a good idea to keep them up to date or at least keeping them close to the latest version. Libyear is just a unit to measure dependency freshness. It is simple, it looks at the latest version of a dependency and its release date, then it looks at the version that is being used by us and its release date. It is calculated as the days between the latest release and our current version release date and divides it by 365 to give us a number that would represent a Libyear. It is considered healthy to keep the libyear less than 1. We have some tools for libyear calculation but they come with their own set of limitations.</p><p><strong>What are those limitations?</strong></p><p>Well, to explain those limitations let’s know a little about the tools. So right now we have 8 different tools listed on the libyear website, all these tools support different languages and their package files. For example, the tool for pypi would calculate libyear for python repositories. These tools look for package files and parse them to know about dependencies and their versions. So the first limitation is the tool’s support for the files. The pypi tool would only look at requirement.txt files, but in more mature python project like augur, they use setup.py to store the list of dependencies and their versions.</p><p>Another limitation is that these tools are written in different languages which support different packages. Implementing them directly would be a problem as it would require more dependencies in augur, which in turn may make augur installation even more complicated. Right now to implement the scorecard we have made a necessary condition to install GO on our local machines while installing augur. Just going on adding dependencies would be a bad idea.</p><p><strong>Well, then what is the solution?</strong></p><p>The solution has two alternatives either to start making a new tool from scratch to calculate Libyear, or for the time being use additional dependencies for the tools. We are yet to decide on this. In the meanwhile, I also looked at how libraries.io is solving the problem, and how I got to know about it is a very fun story. It started with looking at their main repository and I couldn’t find anything for days, all the functions were taking dependencies and versions as their parameters but I didn’t found anywhere how they got the list of dependencies. So I decided to contact their top contributors. The first contributor had his twitter handle on his GitHub page and I sent out a message to him on twitter. The second one had his email-id, so I sent across an email with my queries. The third one had nothing listed, no email no twitter, just had that he worked for fedora. So I searched his name on LinkedIn with fedora and I finally found him. Guess what, he was the one who replied. Crazy right?</p><p><strong>How is Library.io getting the dependencies?</strong></p><p>Library.io uses a ruby library called <strong>bibliothecary </strong>which parses all kinds of package files to get the dependencies. This library is built by libraries.io and seems to be using regular expressions to look for dependencies. To use this library we would have to introduce ruby dependencies for Augur or the option would be to write something of our own like library.io.</p><p>I am trying to implement our parser for augur in python, if it doesn’t work, we may have to use bibliothecary directly as ‘Working code is better than no code’. Let&#39;s see how it turns out!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c3b3ac5173b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC’21 @ CHAOSS: Coding Period Week 4 & 5]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-chaoss-coding-period-week-4-5-fb5e80b96cc9?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb5e80b96cc9</guid>
            <category><![CDATA[chaoss]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[gsoc]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 12 Jul 2021 08:16:23 GMT</pubDate>
            <atom:updated>2021-07-12T08:34:22.678Z</atom:updated>
            <content:encoded><![CDATA[<p>Before getting into my progress for weeks 4 and 5, I would brief you about my first 3 weeks so that it gets you in the flow. I was working on extending the dependency worker to collect scorecard data. We had a bug in which it didn’t generate any logs and I figured it might be because the Deps worker not working at all. Well yes, we did resolve the bugs, and the Scorecard model is completely working now.</p><h3>But how did we solve the buggy situation?</h3><p>This bug led to another bug which led to another bug and so on. At first, we only had to figure out why the worker is not working, it was because of some database versioning issue, it couldn’t find the necessary table to populate dependencies which led to not getting switched. It was a simple resolve by just making a new instance of Augur and connecting it to a new database which generated the schema and worked perfectly. Now all I could see for the scorecard model was ERROR logs.</p><p>Before we used to take the relative_repo_path and modify it a little bit and give it to the generate_scorecard function. This was easy as it did not require much modification but not a great way as it generated problems. So I refactored the scorecard model to query the repo table and pass the URL to the repository. But still, it generated ERROR logs.</p><p>In the process to fix the bug, I used the most efficient method which was to add lots of logging everywhere with variables formatted to figure out what is happening inside:). It took some time to figure out that when our subprocess calls the scorecard function it returned an empty list. I was bewildered by looking at this as I never expected this could be an error. I started adding more logging almost after every step to see what’s happening with the data. After 2-3 days of trying everything from my side, I reached out to my mentor and planned a meeting. Just looking at a glance, he told me what could be going wrong. In augur, some repositories have ‘.git’ at the end of the URL which was not supported by the scorecard. Crazy right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*zXuJbrGYgbmvnnpKThFJDA.gif" /></figure><p>I could have never imagined this issue and I am so thankful to my mentor, he just figured it out in one glance. We did face more errors while inserting data into the database, So we changed the entire table and now the scorecard is working perfectly with Augur!</p><p><strong>Something bad happened</strong> - Do you know about those small green blocks that you get on your Github profile when you make a commit? Since I was using a deprecated version of git on my local machine, all my commits did not generate any of those small green blocks on GitHub!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*Xjm925Zdw8tiMphrnrovOg.gif" /><figcaption>When I figured no green blocks!</figcaption></figure><p>Well for now I have updated my git version and started working on implementing Libyear to calculate dependency freshness. We are building a new Worker for this I feel it’s going to be `legen`</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*D3gFJTIKisDkh-5rgX7ODQ.gif" /><figcaption>Legendary !!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb5e80b96cc9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC’21 CHAOSS — Week 3]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-chaoss-week-3-eacca81a3886?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/eacca81a3886</guid>
            <category><![CDATA[chaoss]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[augur]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 28 Jun 2021 12:07:50 GMT</pubDate>
            <atom:updated>2021-06-28T12:26:02.894Z</atom:updated>
            <content:encoded><![CDATA[<h3>GSoC’21 CHAOSS — Week 3</h3><p>My week 3 was a lot about bugs, but before getting into the bugs I would brief about my work from week 1 and 2 so that it gets you in the flow. In my Week 1 as I mentioned in my first blog, it started by extending the current Deps Worker to collect OSSF- Scorecard data. By the end of the week 1, it was working for me and my week 2 was about making it work for everyone.</p><p>In my week 2 most of the issues were resolved yet there was one particular about setting the environment variables for scorecard that remained unsolved. So setting the environment variables is not a big deal, there are couple of ways to do it but getting the GitHub key from the config file was a deal. It took me a while to understand how to get it and use it in my Deps Worker Class. All the individual worker classes has a base Worker parent class which handles getting the key from the config file, it was that simple.</p><p>I added the script to set the variables in the format required by the scorecard with a check that if it is not already present in the current environment. For sure this has some tradeoffs but seems like the best way out there currently. I also realised that for someone who is running multiple Augur instances or wants to rebuild Augur again would have a Scorecard repo already in the specified directory. This is handled now, Scorecard would not be cloned every time we build Augur.</p><h3><strong>What about the fun part — The bugs?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*QH-O7_mwyJLCcNXSZfjxOw.png" /></figure><p>Well yes , despite doing all this, we have a bug. When I stared the augur backend, I realised that my database table is still not getting populated. I looked for the log file and I could not find it anywhere. This means that our deps worker is not generating any logs. I looked if any environment variable is getting set but it did not exist. Even if we have something wrong with my Scorecard function, it should have generated the variable and logs. This implies that Deps Worker was not running at all. This was a blocking issue, it seems I might have missed some small step somewhere. I am hopeful that it would get resolved this week and I can finally make a complete pull request.</p><p><strong>What Next ?</strong></p><p>After getting the scorecard done, our plan is to start implementing OSS-FUZZ or Libyear based on the discussion I would have with my mentor this week. Both come with their own set of challenges but this is what make the project more interesting :). So this was my week 3 of Google Summer of Code at CHAOSS. A lot was not accomplished as planned for this week but I am hopeful that we can cover it up!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eacca81a3886" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GSoC’21@ CHAOSS- Coding Period -Week 1&2]]></title>
            <link>https://dhruvhsachdev.medium.com/gsoc21-chaoss-coding-period-week-1-2-158d11de8096?source=rss-85eb0660359e------2</link>
            <guid isPermaLink="false">https://medium.com/p/158d11de8096</guid>
            <category><![CDATA[community-health]]></category>
            <category><![CDATA[google-summer-of-code]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[chaoss]]></category>
            <dc:creator><![CDATA[Dhruv Sachdev]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 08:31:27 GMT</pubDate>
            <atom:updated>2021-06-21T09:13:57.834Z</atom:updated>
            <content:encoded><![CDATA[<p>Before getting to what I worked on , it would make more sense if I explained my project in brief. My project is about building a shared data resource focused on Dependencies, Risk and Vulnerabilities. Dependencies are basically piece of code that your project depends on. So, if there is something wrong with the dependency, then that would mean there might be something wrong with your project too.</p><p>So my Week 1 was mostly about design discussion with my Mentor and getting started. I started by extending the current Dependency Worker in Augur to collect OSSF Scorecard data. Scorecard is an open-source project written in ‘go’ programming language. So to make it work , first thing that came to my mind was to use ‘Subprocess’ in python (since augur is in python). The current Value Worker in augur also calls an external Go-based open-source project ‘SCC’ to calculate the code complexity of any project. It uses subprocess too, but we are using it in different ways!</p><p><strong>So what does Scorecard do?</strong></p><p>Basically, scorecard takes an input which is any GitHub project as an argument and does all the work and give us 16 tests with name of the test , pass/fail status and score of that particular test.</p><p>In augur we get the data, clean out as per our needs , and generate a python list of lists which in turn get inserted into the database. Well, by the end of week 1 it was working for me , but wouldn’t work for anyone else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*UyKqhZjgPpxjaSzTdFBb7A.jpeg" /><figcaption>I didn’t want my project to end up like this 🙈</figcaption></figure><h3><strong>Why would it not work for anyone else if it is working for me?</strong></h3><p>Well, this is what my Week 2 was about and some schema changes. So there is a list of reasons that it is only working for me.</p><p>1- They might not have Go installed — Scorecard is a project based on Go, to run it obviously we do need Go locally. So for this I added a check in augur’s Installation script to check if ‘Go’ exisits. Just a note , if anytime in future you get into checking things like this , <strong>never use ‘which go’. </strong>Instead use something like —<strong> command -v go &gt;/dev/null 2&gt;&amp;1 || { echo &gt;&amp;2 &quot;We require &#39;go&#39; to run Scorecard project to get data, Please install GO first....Aborting&quot;; exit 1; } . </strong>This is not true just about Go but for any package check.</p><p>2- They might not have scorecard cloned locally and built — We need Scorecard to generate data, so for this I added some more scripts to clone and build scorecard project to get ready to use by deps worker.</p><p>3- We need path to Scorecard — It was working for me because I knew where my project was cloned. This information is necessary because in Subprocess we have to pass a cwd(current working directory) path i.e. path to scorecard project for us. This is handled by cloning the project in a fixed directory and passing that path to subprocess.</p><p>4- Getting the Environment Variables — Scorecard need some environment variables which is mostly your GitHub API key so that it can make necessary API calls to get data. Since I had it all set for me locally , it was working for me. I am still figuring out ways to make it happen for everyone , maybe I will tell you the solution that I came up with in the next week’s blog :)</p><p>So this was it, my Week 1&amp;2 of Google Summer of Code at CHAOSS. I am learning a lot and very excited to see this completely working in augur!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=158d11de8096" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>