<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sendil Sadasivam on Medium]]></title>
        <description><![CDATA[Stories by Sendil Sadasivam on Medium]]></description>
        <link>https://medium.com/@senthilsivam?source=rss-a3654f10d790------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ojq5AskRLL_I-LL2DGQrxA.jpeg</url>
            <title>Stories by Sendil Sadasivam on Medium</title>
            <link>https://medium.com/@senthilsivam?source=rss-a3654f10d790------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:13:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@senthilsivam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Headless Data Architecture: Stop Copying Data Every Time]]></title>
            <link>https://medium.com/@senthilsivam/headless-data-architecture-stop-copying-data-every-time-e937d43214ce?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/e937d43214ce</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[headless-architecture]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[lake-house]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 15:53:17 GMT</pubDate>
            <atom:updated>2026-11-19T15:53:17.699Z</atom:updated>
            <content:encoded><![CDATA[<p>by Sendil Sadasivam</p><p>Most data platforms I review have the same problem, and it rarely shows up on an architecture slide. The same customer table exists in five places: a Kafka topic, a raw S3 dump, a warehouse staging schema, a “clean” warehouse schema, and a feature store export someone set up two years ago. Each copy has its own schema, its own access rules, and its own idea of what “current” means. When a regulator asks you to delete one customer’s data, you open a ticket for every copy.</p><p>That is not a tooling failure. It is what happens when storage comes bundled with the engine that queries it. If you buy a warehouse, your data lives in the warehouse’s format behind its access layer. If a second team wants Spark, Flink, or DuckDB, they make a copy.</p><p>Headless data architecture is the name for undoing that bundle. You keep the data in open formats with one catalog and one set of policies, and compute engines plug in as interchangeable “heads”. The idea is not new. What has changed recently is that the pieces are mature enough to build it without much custom glue.</p><h3>What “headless” actually means</h3><p>The term was popularised by Adam Bellemare at Confluent. In his <a href="https://www.confluent.io/blog/shift-left-headless-data-architecture-part-1/">Headless Data Architecture, Part 1</a> he describes it as the separation of data storage, management, optimisation, and access from the services that write, process, and query it. The analogy is a headless server: the machine runs fine, but you bring your own monitor and keyboard. Here, you bring your own query or processing engine (Trino, Flink, Spark, a warehouse) and point it at the data.</p><p>Three things move out of the engine and into a shared layer:</p><ol><li>Storage: files in object storage, in open formats such as Parquet, organised by an open table format like Apache Iceberg or Delta Lake.</li><li>Metadata and catalog: which tables exist, where they are, their schemas, and their current snapshot.</li><li>Governance: who can read or write what, enforced once, not re-implemented in each engine.</li></ol><p>What stays in the “head” is compute: query planning, execution, caching, and the user experience on top.</p><p>A useful test: if you switched off your main warehouse tomorrow, could another engine read the same tables, with the same permissions, without copying anything? If yes, you are headless. If no, you have a warehouse with an export job.</p><h3>Why now</h3><p>Three years ago, this design meant writing a lot of glue. Several things have changed since.</p><p>Open table formats have matured. The <a href="https://iceberg.apache.org/spec/">Apache Iceberg spec</a> gives you ACID commits, schema evolution, hidden partitioning, and time travel on plain object storage. The v3 spec, <a href="https://opensource.googleblog.com/2025/08/whats-new-in-iceberg-v3.html">adopted by the Iceberg community in 2025</a>, adds binary deletion vectors, row lineage, and a variant type for semi-structured data. Deletion vectors matter most here: they make row-level deletes and CDC far less painful, and that has been one of the main reasons teams kept mutable data inside a warehouse.</p><p>The catalog became an open API. The Iceberg REST catalog protocol means an engine does not need to know whose catalog it is talking to. <a href="https://polaris.apache.org/blog/2026/02/19/apache-polaris-graduates-to-top-level-project/">Apache Polaris graduated to an Apache Top-Level Project in February 2026</a> as a vendor-neutral implementation of that spec. <a href="https://github.com/unitycatalog/unitycatalog">Unity Catalog</a> is open source under the LF AI &amp; Data Foundation and <a href="https://unitycatalog.io/docs/api/iceberg/">exposes an Iceberg REST API</a> as well. AWS Glue, Snowflake Open Catalog, and others speak the same protocol.</p><p>Engines now read and write open tables properly. It is no longer “Spark writes, everyone else reads, carefully”. <a href="https://duckdb.org/2026/05/29/new-iceberg-features">DuckDB supports inserts, updates, deletes, and </a><a href="https://duckdb.org/2026/05/29/new-iceberg-features">MERGE INTO</a> (with caveats, see below) against tables in an Iceberg REST catalog. <a href="https://clickhouse.com/docs/reference/engines/database-engines/datalake">ClickHouse can attach a REST catalog as a database</a>. <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database">Snowflake can link a whole external Iceberg REST catalog</a> and <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-externally-managed-writes">write to externally managed Iceberg tables</a>. <a href="https://trino.io/docs/current/connector/iceberg.html">Trino</a>, <a href="https://iceberg.apache.org/docs/latest/flink/">Spark, and Flink</a> have had solid Iceberg support for a while.</p><p>Streams and tables are converging. Confluent’s <a href="https://docs.confluent.io/cloud/current/topics/tableflow/overview.html">Tableflow</a> materialises Kafka topics as Iceberg or Delta Lake tables and publishes them to external catalogs. Iceberg support <a href="https://www.confluent.io/blog/tableflow-is-now-generally-available/">went GA in March 2025</a>, and Delta Lake with Unity Catalog <a href="https://www.confluent.io/blog/tableflow-delta-lake-databricks-unity-catalog-ga/">followed in October 2025</a>. Kafka Connect sinks and Flink jobs can do the same job. <a href="https://newsroom.ibm.com/2026-03-17-ibm-completes-acquisition-of-confluent,-making-real-time-data-the-engine-of-enterprise-ai-and-agents">IBM completed its acquisition of Confluent in March 2026</a>, which is worth keeping in mind when you assess vendor roadmaps.</p><h3>A reference architecture</h3><p>Here is the vendor-neutral shape I use when sketching this for a client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tl6gw8P9nkY65lHN2qKT9w.png" /></figure><p><em>Figure 1. Sources publish once to governed streams, tables are derived from them, and every engine attaches through the stream or the catalog, with no copies between engines.</em></p><p>Read it from left to right:</p><ul><li>Sources publish once, to a stream with a registered schema. That schema is the contract.</li><li>Materialisation turns each stream into an append-only or upsert table. This can be Tableflow, a Kafka Connect Iceberg sink, or a Flink job.</li><li>Tables live in your object storage, in an open format, registered in a catalog that speaks the Iceberg REST protocol.</li><li>Governance hangs off the catalog, so the permission check happens once, not separately in each engine.</li><li>Heads attach to the stream (for low latency) or to the catalog (for tables). Nothing is copied to attach a new head.</li></ul><p>The key line in the diagram is the one that is missing: there is no arrow from one engine’s storage into another engine’s storage.</p><h3>Streams and tables: the two headless forms</h3><p>Bellemare’s <a href="https://www.confluent.io/blog/shift-left-headless-data-architecture-part-2/">Part 2</a> makes a point I agree with: a headless data product is a stream and its matching table, not one or the other.</p><ul><li>Streams give you incremental, low-latency access. Use them for order processing, fraud checks, dispatch, or anything that needs to react within seconds.</li><li>Tables give you efficient bulk reads. Use them for reporting, feature engineering, model training, and ad hoc analysis.</li></ul><p>The producer writes only to the stream. The table is derived from it. That avoids a dual write to Kafka and a table, which is hard to get right and is one of the most common sources of drift I see.</p><p>Two practical notes from experience:</p><ol><li>Be explicit about table semantics. An append-only event table and an upserted “current state” table are different products. Name them differently and document which one a consumer should use. CDC streams almost always need both.</li><li>Freshness is a number, not an adjective. A table materialised from a stream might lag by seconds or minutes, depending on commit intervals. Publish that lag as part of the data product’s contract, because consumers will assume “real time” otherwise.</li></ol><h3>Shift left: where the work moves</h3><p>Headless pairs naturally with what Confluent calls <a href="https://www.confluent.io/blog/shift-left-headless-data-architecture-part-2/">shift left</a>: moving cleansing, schema enforcement, and basic structuring upstream to the producing team, instead of rebuilding it in bronze and silver layers for every consumer.</p><p>This is the part that is organisational, not technical. If the source team publishes messy, unversioned events, a headless layer simply gives five engines equal access to messy data. Shift left only works if producing teams accept that the published stream is a product with an owner, a schema, and compatibility rules. Schema registries and data contracts are how you make that enforceable, and Bellemare’s earlier piece on <a href="https://www.confluent.io/blog/implementing-streaming-data-products/">streaming data products and data contracts</a> is a good practical primer.</p><p>My opinion: do not shift everything left. Shift left only the work that is common to all consumers (types, keys, PII tagging, deduplication). Keep work that is specific to one consumer (a finance team’s revenue rules, an ML team’s feature logic) with that consumer. Pushing domain-specific logic into producers recreates the bottleneck the central data team used to be.</p><h3>The catalog is the real platform</h3><p>In a headless design, the catalog does the job the warehouse used to do. It decides which tables exist, which snapshot is current, and, increasingly, who is allowed to see what. Pick it with the same care you would put into choosing a database.</p><p>Questions I ask when evaluating one:</p><ul><li>Does it implement the Iceberg REST spec fully, including commits, not just reads?</li><li>Does it vend credentials? Short-lived, scoped storage credentials handed to engines at query time are much safer than giving every engine broad bucket access. Polaris, Unity Catalog, and <a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-vended-credentials">Snowflake’s catalog integrations</a> support this pattern.</li><li>Where is policy enforced? A catalog can grant table-level access, but row filters and column masks still depend on each engine honouring them. Check this for every head you plan to attach, not just the main one.</li><li>Can it federate? In large organisations you will end up with more than one catalog. A catalog that can front others, or be fronted, reduces pain later.</li><li>What does leaving look like? If the catalog’s metadata can only be exported in a proprietary form, you have moved your lock-in, not removed it.</li></ul><h3>Benefits</h3><p>When it works, the benefits are concrete:</p><ul><li>One copy, many engines. New use cases attach to existing tables instead of starting a new pipeline.</li><li>One place for policy and deletion. Compliance work becomes a catalog and table operation, not a hunt across systems.</li><li>Engine choice becomes a workload decision. Use Flink for streaming, DuckDB for a notebook, a warehouse for BI, and switch later without a migration.</li><li>Better negotiating position. When storage and format are yours, compute vendors compete on price and performance.</li></ul><h3>Trade-offs and pitfalls</h3><p>Governance gets harder before it gets easier. A warehouse gives you one engine enforcing one policy model. With five heads, you have five enforcement points. Until your catalog and engines agree on fine-grained controls, assume the lowest common denominator applies, and restrict write access to a small number of trusted engines.</p><p>Small files and table maintenance are your job now. Streaming writes produce many small files and many snapshots. Without compaction, snapshot expiry, and orphan file cleanup, query performance drops and storage costs grow. The <a href="https://iceberg.apache.org/docs/latest/maintenance/">Iceberg maintenance guide</a> covers the routines; someone has to own running them. Managed services can take this on; confirm exactly what they maintain.</p><p>Catalog lock-in is the new warehouse lock-in. Open formats do not help if every engine is wired to a catalog that only one vendor controls. Prefer engines and catalogs that talk the open REST protocol, and test a second engine early.</p><p>Format fragmentation. Iceberg and Delta Lake both work well, but mixing them adds friction. Bridges like <a href="https://docs.delta.io/latest/delta-uniform.html">Delta UniForm</a> help, but pick a primary format and treat the other as an interoperability concern.</p><p>Cost moves, it does not disappear. You save on duplicate storage and pipelines, but you pay for object storage requests, maintenance compute, catalog operations, and egress if engines run in different regions or clouds. Model this per workload before you promise savings.</p><p>Feature parity across engines is uneven. Iceberg v3 features, row-level deletes, and specific partition transforms are supported at different levels in different engines. DuckDB’s own docs, for example, <a href="https://duckdb.org/docs/1.5/core_extensions/iceberg/writing_to_iceberg">list limitations on updates and deletes</a>. Check the support matrix for every engine that will write, not just read.</p><p>Operational ownership. Someone must be on call for the catalog, the materialisation jobs, and maintenance. In a bundled warehouse the vendor carried that; here, your platform team does.</p><h3>When not to go headless</h3><p>Headless is not a default. I would not recommend it when:</p><ul><li>You have one engine and no plans for a second. A single warehouse with good modelling is simpler and probably cheaper.</li><li>Your team is small. If two data engineers run everything, the operational overhead of catalog, maintenance, and multi-engine governance will eat their time.</li><li>Latency needs are sub-second for analytics. Table commits add lag. For truly real-time serving, use the stream or a purpose-built serving store.</li><li>Your governance model depends on rich, engine-specific features (complex row-level security, dynamic masking) that your other engines cannot enforce yet.</li><li>Data volumes are modest. At small scale, the copies are cheap.</li></ul><h3>A pragmatic adoption path</h3><p>Big-bang platform rewrites tend to fail. This is the sequence I recommend.</p><p>Phase 1: Open tables under your current engine (1 to 3 months). Move one or two high-value datasets to Iceberg or Delta in your own object storage, still managed by your current warehouse or lakehouse. Goal: prove the format and set up maintenance.</p><p>Phase 2: Introduce a REST catalog and a second head (2 to 4 months). Register those tables in an Iceberg REST catalog. Attach a second engine for a real use case (Trino for ad hoc SQL, DuckDB for analysts, Spark for ML). Goal: prove that two engines can share one copy with consistent permissions.</p><p>Phase 3: Streams as the front door (3 to 6 months). For new or high-change sources, publish to a governed stream with a registered schema, and materialise tables from it. Start with sources where freshness matters. Goal: retire at least one copy pipeline.</p><p>Phase 4: Data products and shift left (ongoing). Formalise stream and table pairs as owned data products with contracts, SLAs on freshness, and documented semantics. Move universal cleansing upstream.</p><p>Phase 5: Decommission and consolidate (once Phase 3 sources are stable). Retire duplicate copies and legacy staging layers. This is where the cost savings actually show up, so track it explicitly.</p><p>Give each phase an exit check. If the second engine cannot meet its use case, or maintenance costs exceed the savings, fix that before expanding.</p><h3>Checklist</h3><p>Before calling a dataset “headless”, check that:</p><ul><li>Data is stored in an open table format in storage you control.</li><li>Tables are registered in a catalog that speaks the Iceberg REST protocol (or an equivalent open API).</li><li>At least two engines read the same table without copying it.</li><li>Access policy is defined once, and you have verified how each engine enforces it.</li><li>Engines get scoped, short-lived storage credentials, not broad bucket keys.</li><li>Compaction, snapshot expiry, and orphan cleanup run on a schedule with an owner.</li><li>Producers publish a versioned schema with compatibility rules.</li><li>Freshness and table semantics (append vs. upsert) are documented.</li><li>Deletion requests can be fulfilled with a single, auditable operation.</li><li>You know the cost per workload, including storage requests and maintenance.</li></ul><h3>Conclusion</h3><p>Headless data architecture is less a new technology than a decision about ownership. You own the data, the format, and the catalog. Vendors compete to provide the compute. The building blocks are now good enough to make that decision practical.</p><p>The difficult parts are not the formats. They are governance across engines, table maintenance, and getting producing teams to treat their data as a product. Start small, prove that two engines can share one copy safely, and expand only when the numbers back it up.</p><p>I am collecting patterns like this, with decision guides and reference diagrams, in the <a href="https://data-architecture-playbook.onrender.com/">Data Architecture Playbook</a>. I also write about data and AI architecture on <a href="https://sendilsiva.substack.com/">Substack</a>. If you have run a headless platform in production, I would like to hear what broke first.</p><h3>References</h3><ul><li><a href="https://www.confluent.io/blog/shift-left-headless-data-architecture-part-1/">Adam Bellemare (Confluent), Shift Left: Headless Data Architecture, Part 1, 17 Oct 2024</a></li><li><a href="https://www.confluent.io/blog/shift-left-headless-data-architecture-part-2/">Adam Bellemare (Confluent), Shift Left: Headless Data Architecture, Part 2, 25 Oct 2024</a></li><li><a href="https://www.infoworld.com/article/3479722/a-developers-guide-to-the-headless-data-architecture.html">Adam Bellemare (InfoWorld), A developer’s guide to the headless data architecture, 31 Jul 2024</a></li><li><a href="https://www.confluent.io/blog/implementing-streaming-data-products/">Adam Bellemare (Confluent), Data Products, Data Contracts, and Change Data Capture, 21 Feb 2024</a></li><li><a href="https://docs.confluent.io/cloud/current/topics/tableflow/overview.html">Confluent, Tableflow in Confluent Cloud (documentation), last updated 29 Oct 2026</a></li><li><a href="https://www.confluent.io/blog/tableflow-is-now-generally-available/">Confluent, Tableflow Is GA: Unifying Apache Kafka Topics with Apache Iceberg and Delta Lake Tables, 19 Mar 2025</a></li><li><a href="https://www.confluent.io/blog/tableflow-delta-lake-databricks-unity-catalog-ga/">Confluent, Tableflow for Delta Lake and Databricks Unity Catalog Is Now Generally Available, 29 Oct 2025</a></li><li><a href="https://newsroom.ibm.com/2026-03-17-ibm-completes-acquisition-of-confluent,-making-real-time-data-the-engine-of-enterprise-ai-and-agents">IBM, IBM Completes Acquisition of Confluent, 17 Mar 2026</a></li><li><a href="https://iceberg.apache.org/spec/">Apache Iceberg, Spec, n.d.</a></li><li><a href="https://iceberg.apache.org/docs/latest/maintenance/">Apache Iceberg, Maintenance, n.d.</a></li><li><a href="https://iceberg.apache.org/docs/latest/flink/">Apache Iceberg, Flink: Getting Started, n.d.</a></li><li><a href="https://opensource.googleblog.com/2025/08/whats-new-in-iceberg-v3.html">Google Open Source Blog, What’s new in Apache Iceberg v3?, 7 Aug 2025</a></li><li><a href="https://polaris.apache.org/blog/2026/02/19/apache-polaris-graduates-to-top-level-project/">Apache Polaris, Apache Polaris Graduates to Top Level Project!, 19 Feb 2026</a></li><li><a href="https://github.com/unitycatalog/unitycatalog">Unity Catalog, unitycatalog/unitycatalog (GitHub repository), n.d.</a></li><li><a href="https://unitycatalog.io/docs/api/iceberg/">Unity Catalog, Iceberg REST Catalog API, n.d.</a></li><li><a href="https://duckdb.org/2026/05/29/new-iceberg-features">DuckDB, New DuckDB-Iceberg Features in v1.5.3, 29 May 2026</a></li><li><a href="https://duckdb.org/docs/1.5/core_extensions/iceberg/writing_to_iceberg">DuckDB, Writing to Iceberg, n.d.</a></li><li><a href="https://clickhouse.com/docs/reference/engines/database-engines/datalake">ClickHouse, DataLakeCatalog, n.d.</a></li><li><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-catalog-linked-database">Snowflake, Use a catalog-linked database for Apache Iceberg tables, n.d.</a></li><li><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-externally-managed-writes">Snowflake, Write support for externally managed Apache Iceberg tables, n.d.</a></li><li><a href="https://docs.snowflake.com/en/user-guide/tables-iceberg-configure-catalog-integration-vended-credentials">Snowflake, Use catalog-vended credentials for Apache Iceberg tables, n.d.</a></li><li><a href="https://trino.io/docs/current/connector/iceberg.html">Trino, Iceberg connector, n.d.</a></li><li><a href="https://docs.delta.io/latest/delta-uniform.html">Delta Lake, Universal Format (UniForm), n.d.</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e937d43214ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Claude Code Skill Progression: Novice to Expert]]></title>
            <link>https://medium.com/@senthilsivam/claude-code-skill-progression-novice-to-expert-5c3443f15ff5?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c3443f15ff5</guid>
            <category><![CDATA[vibe-coding]]></category>
            <category><![CDATA[claude-code]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Wed, 16 Sep 2026 00:20:50 GMT</pubDate>
            <atom:updated>2026-09-16T00:20:50.723Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SxnFeUO66lBP8LQySz-VZA.png" /></figure><p>Based on Anthropic’s analysis of 400,000 Claude Code sessions (<a href="https://www.xda-developers.com/anthropic-analyzed-400000-claude-code-sessions-turns-out-theres-a-right-way-to-use-it/">XDA Developers</a>), the gap between novice and expert usage comes down to four skills: <strong>delegation, context-setting, verification, and recovery</strong>. Below is a practical progression through four levels, each with concrete behaviors to adopt and drills to build them.</p><h3>Level 1: Novice — Learn to Let Go</h3><p><strong>Current pattern:</strong> Micromanaging every step (which file, which function, which command), asking generic verification questions (“are you sure?”), abandoning sessions the moment something breaks.</p><p><strong>Target behaviors to build:</strong></p><ul><li>Stop specifying implementation steps. Instead of “open auth.py, change line 42 to use bcrypt,&quot; say &quot;switch password hashing to bcrypt across the auth module.&quot;</li><li>Replace “please double-check” with at least one concrete check per task (even a simple “run the test suite and show me the output”).</li><li>When something fails, read the error before deciding to restart — resist the urge to scrap the session immediately.</li></ul><p><strong>Drill:</strong> Take a task you’d normally over-specify. Write only the goal, the relevant files/modules to consider, and any hard constraints (e.g., “don’t break the public API”). Let Claude choose the implementation path. Compare the outcome to your usual approach.</p><p><strong>Self-check metric:</strong> Are you making &gt;80% of execution decisions yourself? If so, you’re still capping Claude’s autonomy — aim to hand more of “how” over.</p><h3>Level 2: Intermediate — Context Over Control</h3><p><strong>Current pattern:</strong> Giving goals but thin context; mixing a few targeted checks in with generic ones; recovering from failures inconsistently.</p><p><strong>Target behaviors to build:</strong></p><ul><li>Front-load context: relevant architecture, prior decisions, constraints, and what “done” looks like — before Claude starts, not after it goes wrong.</li><li>For every verification, ask about at least one specific mechanic (“did you actually call the retry decorator?” instead of “does this work?”).</li><li>When a session goes sideways, diagnose <em>why</em> (misunderstood scope? wrong assumption? missing context?) before deciding whether to redirect or restart.</li></ul><p><strong>Drill:</strong> Before starting a task, write a 3–4 sentence brief covering: the end goal, the constraint that matters most, and one thing that would prove success. Hand that to Claude verbatim as your first message.</p><p><strong>Self-check metric:</strong> Track your own output length/action count per prompt over a week. Intermediate users sit between novice (~5 actions, ~600 words) and expert (~12 actions, ~3,200 words) — if you’re stuck near novice numbers, your briefs are too thin.</p><h3>Level 3: Advanced — Design for Verification</h3><p><strong>Current pattern:</strong> Delegation feels natural; verification is still reactive rather than built into the task upfront.</p><p><strong>Target behaviors to build:</strong></p><ul><li>Before assigning a task, decide what evidence would prove it’s done right — a failing test, a rendered UI diff, a printed config state — and hand Claude that verification method along with the task.</li><li>Ask about specific state or edge cases relevant to the domain, not just “does it run.”</li><li>When recovering a troubled session, give Claude the specific evidence of what failed (test output, error trace) rather than just saying “that’s wrong, try again.”</li></ul><p><strong>Drill for each task type:</strong></p><p>Task type Verification to build in Bug fixWrite/point to a test that reproduces the bug; require it to pass afterwardUI changeProvide a reference image or spec; ask Claude to describe or screenshot the diffConfig/infra changeHave Claude print or diff the resulting state before you accept it Data/logic change Ask about cardinality, edge cases, or a specific computed value</p><p><strong>Self-check metric:</strong> Can you name, before Claude starts, the exact artifact (test result, diff, printed state) that will tell you if it worked? If not, you’re still verifying reactively.</p><h3>Level 4: Expert — Steer, Verify, Recover as One Loop</h3><p><strong>Current pattern (target state):</strong> Domain expertise drives precise direction; verification anticipates edge cases; troubled sessions get redirected rather than abandoned.</p><p><strong>Target behaviors to build:</strong></p><ul><li>Treat domain knowledge — not coding ability — as your main lever. You should be able to say what a <em>correct</em> result looks like even in areas where you wouldn’t personally write the code.</li><li>Anticipate failure modes before starting: “this will break if inputs are unsorted” or “watch for a duplicate join key” — and design the test/check around that specific risk.</li><li>When a session fails repeatedly on the same issue, stop retrying the same instruction. Diagnose the misunderstanding, supply the missing context or constraint, and redirect rather than restart from zero.</li><li>Give Claude enough room that a single prompt can trigger ~12+ actions and a substantial (page-plus) output — a sign you’re not bottlenecking with unnecessary check-ins.</li></ul><p><strong>Drill:</strong> Deliberately let a session run into a failure (e.g., a flaky test or ambiguous requirement). Practice diagnosing root cause from the failure evidence alone, then send one corrective message that supplies the missing context — instead of restarting the session.</p><p><strong>Self-check metric (recovery skill, the biggest expert differentiator):</strong></p><p>OutcomeNoviceExpertRecovers to verified success after a failure~4%~15%Abandons the session~19%~5–7%</p><p>If you’re still abandoning troubled sessions roughly 1 in 5 times, recovery — not delegation or verification — is your next growth area.</p><h3>Quick Reference: The Four Skills in One Loop</h3><ol><li><strong>Frame</strong> — Give the goal, context, and constraints; skip the step-by-step instructions.</li><li><strong>Release</strong> — Let Claude own the majority of implementation decisions.</li><li><strong>Verify</strong> — Attach a concrete, specific check (test, diff, printed state) to every task before it starts.</li><li><strong>Recover</strong> — When it goes wrong, diagnose from the evidence and redirect; don’t default to restarting.</li></ol><p>Progressing through novice → intermediate → advanced → expert is really about moving each of these four skills from reactive/generic to proactive/specific.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c3443f15ff5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rowboat: An AI Coworker That Remembers Your Work]]></title>
            <link>https://medium.com/@senthilsivam/rowboat-an-ai-coworker-that-remembers-your-work-0b439d0bff27?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b439d0bff27</guid>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Tue, 11 Aug 2026 11:51:42 GMT</pubDate>
            <atom:updated>2026-08-11T11:55:11.079Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QuxrqRaY-BgGDDJJ6m5FPQ.png" /></figure><p>Most AI coding assistants are helpful only when they receive enough context. Developers frequently have to explain the project architecture, business requirements, previous decisions, coding conventions, and current task before asking the assistant to generate code.</p><p><a href="https://github.com/rowboatlabs/rowboat">Rowboat</a> takes a different approach. It is a desktop AI coworker that continuously organizes work context into a living knowledge graph. This accumulated memory can then be used to suggest actions, create reminders, draft content, and help coding agents generate code based on the requirements.</p><h3>Building Context in the Background</h3><p>Rowboat indexes information from sources such as emails, meetings, Slack conversations, notes, and previous assistant interactions. It stores the resulting knowledge locally as interconnected Markdown notes.</p><p>Unlike tools that reconstruct context for every request, Rowboat maintains long-lived knowledge:</p><ul><li>Context accumulates over time.</li><li>Relationships between people, projects, requirements, and decisions remain explicit.</li><li>Knowledge can be inspected and edited.</li><li>Information is stored locally as plain Markdown.</li><li>Existing context can be passed to coding agents when required.</li></ul><p>This reduces the need to repeatedly explain the same project background.</p><h3>Coding with Existing Work Context</h3><p>Rowboat’s <strong>Code Mode</strong> can start multiple coding agents using Claude Code or OpenAI Codex. It can provide these agents with relevant information from the user’s accumulated work context.</p><p>For example, a developer could request:</p><blockquote><em>“Create the customer-address update API based on the requirements discussed with the team.”</em></blockquote><p>Rowboat can retrieve related notes, meeting discussions, decisions, and requirements and provide them to the coding agent. The agent can then examine the codebase and generate an implementation with less manual context-setting.</p><p>Code Mode can help with:</p><ul><li>Creating application features</li><li>Generating APIs and integrations</li><li>Fixing defects</li><li>Refactoring existing code</li><li>Writing unit and integration tests</li><li>Preparing technical documentation</li><li>Running parallel coding tasks</li><li>Connecting requirements and discussions to implementation</li></ul><p>Watch the official <a href="https://www.youtube.com/watch?v=et5yQABJ3xI">Apps-to-Code demonstration</a> to see how Rowboat connects work context with code generation.</p><h3>Suggestions, Reminders, and Background Agents</h3><p>Rowboat supports background agents that can run on a schedule or respond to events such as receiving a new email. These agents can search the web, operate the built-in browser, connect to external tools, and initiate coding activities through Claude Code or Codex.</p><p>Background agents could be configured to:</p><ul><li>Review important emails and prepare responses</li><li>Summarize daily project activity</li><li>Remind developers about pending tasks</li><li>Identify requirements that have not been implemented</li><li>Generate code from approved requirements</li><li>Check whether documentation needs to be updated</li><li>Prepare meeting summaries and action items</li><li>Monitor project events and suggest the next action</li></ul><p>Because the agents use Rowboat’s accumulated knowledge, their recommendations can reflect the user’s existing projects and working context.</p><h3>Integrating External Tools</h3><p>Rowboat supports external services through the <a href="https://modelcontextprotocol.io/">Model Context Protocol</a>. MCP allows Rowboat to connect with tools such as GitHub, Slack, Linear, Jira, databases, search services, and internal enterprise applications.</p><p>It also supports Composio integrations and optional services for web search, voice input, and voice output.</p><p>This makes it possible to build workflows that combine organizational context with development activities. For example, Rowboat could retrieve a requirement from a project-management system, locate related decisions in its knowledge graph, and provide everything to a coding agent for implementation.</p><h3>Local and Hosted AI Models</h3><p>Rowboat allows users to choose their preferred AI model setup. It supports:</p><ul><li>Local models through <a href="https://ollama.com/">Ollama</a></li><li>Local models through <a href="https://lmstudio.ai/">LM Studio</a></li><li>Hosted models using the user’s own provider and API key</li><li>Claude Code or Codex for coding-agent workflows</li></ul><p>Models can be changed without losing the accumulated knowledge because the underlying information remains in the local Markdown vault.</p><h3>Installing Rowboat</h3><p>Rowboat is available for macOS, Windows, and Linux.</p><ol><li>Visit the official <a href="https://www.rowboatlabs.com/downloads">Rowboat download page</a>.</li><li>Download the version for your operating system.</li><li>Install and open the desktop application.</li><li>Select and configure your preferred AI model.</li><li>Connect only the applications and services that Rowboat should access.</li><li>Create separate workspaces for different projects.</li><li>Enable Code Mode if you want to use Claude Code or Codex.</li><li>Configure background agents for scheduled or event-driven activities.</li></ol><p>Users who prefer to inspect individual versions can access the <a href="https://github.com/rowboatlabs/rowboat/releases/latest">latest GitHub release</a>.</p><h3>Optional Configuration</h3><p>Additional capabilities can be enabled through local configuration files:</p><ul><li><strong>Google services:</strong> Follow the official <a href="https://github.com/rowboatlabs/rowboat/blob/main/google-setup.md">Google setup guide</a> to connect Gmail, Google Calendar, and Google Drive.</li><li><strong>Voice input:</strong> Add a Deepgram API key to ~/.rowboat/config/deepgram.json.</li><li><strong>Voice output:</strong> Add an ElevenLabs API key to ~/.rowboat/config/elevenlabs.json.</li><li><strong>Web research:</strong> Add an Exa API key to ~/.rowboat/config/exa-search.json.</li><li><strong>Composio tools:</strong> Add a Composio API key to ~/.rowboat/config/composio.json.</li><li><strong>External tools:</strong> Configure compatible MCP servers for the services Rowboat needs to access.</li></ul><p>Each API-key file follows this structure:</p><pre>{<br>  &quot;apiKey&quot;: &quot;&lt;your-api-key&gt;&quot;<br>}</pre><p>The latest configuration information is maintained in the project’s <a href="https://github.com/rowboatlabs/rowboat/blob/main/README.md">official README</a>.</p><h3>Local-First and User-Controlled</h3><p>Rowboat stores its accumulated knowledge locally as plain Markdown instead of locking it inside a proprietary hosted system. Users can inspect, edit, back up, move, or delete the information.</p><p>However, connecting emails, repositories, calendars, browsers, and development tools gives an AI assistant access to sensitive information. Teams should therefore:</p><ul><li>Connect only required accounts and repositories.</li><li>Use separate workspaces for unrelated projects.</li><li>Protect API keys and configuration files.</li><li>Review generated code before merging it.</li><li>Restrict the commands and tools available to background agents.</li><li>Avoid indexing confidential information unless organizational policies permit it.</li></ul><h3>A Persistent Partner for Development Work</h3><p>Rowboat is more than a code generator. It provides a persistent working memory that connects conversations, meetings, emails, requirements, notes, and development activities.</p><p>Its most valuable capability is not that it eliminates context entirely. It reduces the need for developers to repeatedly provide context by maintaining knowledge that grows with their work. Combined with Claude Code or Codex, this can help Rowboat suggest actions, remind users about commitments, and generate code aligned with previously captured requirements.</p><p>Useful resources:</p><ul><li><a href="https://www.rowboatlabs.com/">Rowboat website</a></li><li><a href="https://github.com/rowboatlabs/rowboat">Rowboat GitHub repository</a></li><li><a href="https://www.rowboatlabs.com/downloads">Download Rowboat</a></li><li><a href="https://github.com/rowboatlabs/rowboat/blob/main/README.md#installation">Installation and configuration guide</a></li><li><a href="https://github.com/rowboatlabs/rowboat/blob/main/google-setup.md">Google integration setup</a></li><li><a href="https://github.com/rowboatlabs/rowboat/releases/latest">Latest releases</a></li><li><a href="https://www.youtube.com/watch?v=et5yQABJ3xI">Apps-to-Code demo</a></li><li><a href="https://www.youtube.com/watch?v=7xTpciZCfpw">Knowledge Graph demo</a></li><li><a href="https://discord.gg/wajrgmJQ6b">Rowboat community on Discord</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b439d0bff27" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying LLMs: From Static Weights to Open-Source Customization]]></title>
            <link>https://medium.com/@senthilsivam/demystifying-llms-from-static-weights-to-open-source-customization-52f9a3d03284?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/52f9a3d03284</guid>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[weights-and-biases]]></category>
            <category><![CDATA[open-weight-model]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Sun, 09 Aug 2026 04:52:35 GMT</pubDate>
            <atom:updated>2026-08-09T04:52:35.471Z</atom:updated>
            <content:encoded><![CDATA[<p>If you have ever wondered what actually happens under the hood when you prompt a Large Language Model (LLM), you aren’t alone. As generative AI transforms technology, understanding its core mechanics helps bridge the gap between “magic” and engineering.</p><p>Let’s break down the fundamentals of how LLMs work, how they learn, and what it truly means when you download an open-weights model.</p><h3>1. What Are Parameters? (The Brain’s Network)</h3><p>At the heart of any LLM are <strong>parameters</strong> — billions of numerical values divided into two categories:</p><ul><li><strong>Weights (W):</strong> Numbers that determine the connection strength between different artificial neurons.</li><li><strong>Biases (B):</strong> Adjustable offsets added to inputs to dictate how easily a neuron fires.</li></ul><p>When people talk about a model being “8B” or “70B,” they are referring to the total parameter count. More parameters generally mean a higher capacity for complex reasoning, nuances, and factual knowledge, but they also require significantly more computing power to run.</p><figure><img alt="A visualization of a dense neural network. The glowing lines are the weights, and the nodes include an internal calculation for the bias before passing data forward." src="https://cdn-images-1.medium.com/max/1024/1*2uLSkgFN3n0CSOlACtZ5jw.jpeg" /><figcaption>A visualization of a dense neural network. The glowing lines are the <strong>weights</strong>, and the nodes include an internal calculation for the <strong>bias</strong> before passing data forward.</figcaption></figure><h3>2. Static Weights vs. Dynamic Activations</h3><p>One common misconception is that a model adapts its internal wiring while you chat with it. In reality, <strong>weights are completely frozen</strong> once training is finished.</p><p>Think of an LLM’s weights like a massive, permanent network of plumbing pipes and valves. When you type a prompt, your text is converted into numbers and flows through this fixed structure. The unique response you get is driven by <strong>temporary activation states</strong> generated on the fly for your specific query. Once the response is delivered, those temporary values disappear, leaving the base model unchanged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ubG8o1fhPJtMwAG8kFQKA.jpeg" /><figcaption>A contrast of phases. Left: Training, where massive data actively reshapes the network connections. Right: Inference, where the user prompt (blue stream) flows through the <em>fixed</em> network to generate output.</figcaption></figure><h3>3. The Power of Training Data</h3><p>If the architecture and weights remain fixed, what makes models different? <strong>The data.</strong></p><p>In machine learning, data dictates the model’s worldview. If you feed a model medical journals, it thinks like a clinician; if you feed it code repositories, it thinks like a developer.</p><p>But how do models learn logic and reasoning from code when raw source files lack README files or architectural explanations? AI labs solve this using multi-stage pipelines:</p><ul><li><strong>Rich Context Data:</strong> Training corpora mix raw code with Q&amp;A platforms (like Stack Overflow), commit histories, and documentation.</li><li><strong>Synthetic Reasoning Traces:</strong> Models are trained on “Chain-of-Thought” data that explicitly maps out step-by-step problem-solving before writing code.</li><li><strong>Reinforcement Learning (RL):</strong> Models generate code, run it against automated unit tests, and receive rewards or penalties based on whether the logic actually works.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AsEDsuDcDsDLbvHAcl3UqA.jpeg" /><figcaption>A training graph. The blue Training Loss drops continuously. The red Validation Loss drops initially but eventually rises, indicating overfitting. Training must stop at the dotted ‘Early Stopping Point’.</figcaption></figure><h3>4. How Models Learn: Loss and Backpropagation</h3><p>Models don’t come pre-programmed with correct answers; they learn automatically through trial and error:</p><ol><li><strong>The Forward Pass:</strong> The model makes a prediction.</li><li><strong>Loss Calculation:</strong> A loss function measures how wrong the prediction was compared to the training data.</li><li><strong>Backpropagation &amp; Gradient Descent:</strong> Using calculus, the model calculates a “gradient” to see how tweaking specific weights and biases would reduce the error, adjusting them slightly via an optimizer.</li></ol><p>During training, engineers monitor <strong>Training Loss</strong> and <strong>Validation Loss</strong>. When validation loss hits a plateau, the model has converged. If you train past this point, the model begins memorizing noise — a dangerous trap known as <em>overfitting</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lIyCiz159H3HT79HC0nENA.jpeg" /><figcaption>A conceptual visualization of LoRA. The large blue block (the base model) is frozen. Specialized data (golden stream) only interacts with the tiny, attachable green adapters, customizing the model efficiently.</figcaption></figure><h3>5. What Are Open-Weights Models?</h3><p>Unlike closed, proprietary APIs where companies keep weights locked behind server walls, <strong>open-weights models</strong> (like Meta’s Llama or Google’s Gemma) give you access to the raw mathematical files (such as .safetensors).</p><p>With open weights, you can run models locally behind your own firewall for data privacy or <strong>fine-tune</strong> them. Using techniques like <strong>LoRA (Low-Rank Adaptation)</strong>, you can freeze the massive base weights, attach tiny trainable adapter layers, and teach the model your custom domain data on a consumer GPU in just a few hours.</p><h3>Final Thoughts</h3><p>LLMs are ultimately massive matrices of static numbers shaped by petabytes of curated data, optimized through calculus, and brought to life by dynamic prompt activations. Whether you are running a lightweight model locally or building a multi-agent platform, demystifying these layers turns black-box AI into a predictable, customizable engineering tool.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=52f9a3d03284" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Role of Graphs and Semantics in Next-Generation AI]]></title>
            <link>https://medium.com/@senthilsivam/the-role-of-graphs-and-semantics-in-next-generation-ai-ed967b3f1852?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed967b3f1852</guid>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 12:44:47 GMT</pubDate>
            <atom:updated>2025-08-20T12:44:47.509Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IocmHA9F-XQfgiG0Nh4Xjw.png" /></figure><p>Today’s generative AI models are reshaping how organizations extract meaning and value from data. Yet, as these systems tackle ever more complex domains, simply scaling models or training on bigger datasets isn’t enough. True intelligence — and trustworthy, actionable AI — requires a deep understanding of context, relationships, and intent.</p><p>This is where semantic models and graph-based representations come into play. By explicitly encoding the meaning of entities, the connections between them, and the logic that governs their interactions, semantic and graph technologies provide the foundational knowledge structures that modern AI needs.</p><p>This section explores how these approaches unlock new levels of reasoning, accuracy, and domain-aligned insight for GenAI, making AI not just more powerful — but also more reliable, explainable, and aligned with real-world goals.</p><p>Gen AI models learning from semantic models or graph-based models is their ability to understand and represent the relationships between different objects or entities. This structured representation helps move beyond simple text generation to context-aware, logically sound outputs.</p><h4>1. Understanding Relationships</h4><ul><li>Semantic models and knowledge graphs encode explicit relationships — such as parent/child, part/whole, connections, dependencies — between entities.</li><li>Gen AI models use this structure to generate outputs that respect domain logic, accurately connect concepts, and answer complex questions involving relationships.</li></ul><h4>2. Contextual Awareness</h4><ul><li>Graphs and semantic layers provide rich context: models can understand not only what entities exist, but also how they fit together in the broader domain.</li><li>This context boosts the model’s ability to interpret ambiguous queries, resolve references, and generate more accurate, context-sensitive responses.</li></ul><h4>3. Reasoning and Inference</h4><ul><li>Semantic models (especially with logical frameworks like OWL) allow Gen AI to perform reasoning — infer unstated facts or discover new knowledge based on rules and relationships.</li><li>For example, if “A is part of B” and “B is part of C”, the model can infer “A is related to C”.</li></ul><h4>4. Domain Alignment and Governance</h4><ul><li>Semantic models enforce standardized definitions, terminologies, and business logic. This helps Gen AI models produce outputs that are consistent with enterprise rules, policies, and vocabularies — minimizing errors and misinterpretation.</li><li>Enables auditability and traceability, as Gen AI outputs can be mapped back to exact definitions and relationships in the semantic model.</li></ul><h4>5. Enhanced Query and Insight Generation</h4><ul><li>By grounding natural language queries in these models, Gen AI can turn vague or complex questions into precise analytic actions — automatically generating queries, reports, or actionable insights that align with business rules.</li></ul><h4>6. Safer, More Compliant Generation</h4><ul><li>With a semantic model as reference, Gen AI can avoid making up facts (“hallucinations”), check outputs against business rules, and help with compliance by marking sensitive links or forbidden combinations.</li></ul><h4>7. Interoperability and Integration</h4><ul><li>Graph/semantic structures enable models to access, link, and harmonize data across different sources, applications, and organizations, supporting broader interoperability.</li></ul><p>In addition to relationship understanding, semantic models and graphs provide:</p><ul><li>Rich metadata and constraints (like which values are allowed, data types, and units of measure).</li><li>Hierarchies and taxonomies for organizing concepts (e.g., category trees).</li><li>Event and process modeling (how things change and interact over time).</li><li>Abstraction and modularity — they make complex domains easier to explain, extend, and maintain.</li><li>Better signal extraction for discovery, personalization, and automated decision-making.</li></ul><p>Let’s go a level deeper into how Gen AI models get trained and leverage graph-based models (like knowledge graphs or ontologies):</p><h4>Graph-Based Models: Structure and Training Data</h4><ul><li>Graph models (knowledge graphs, property graphs, ontology-based schemas) represent entities as nodes and relationships as edges, possibly with attributes (properties), types, and rules.</li><li>They encode not just data, but meaning and connections, such as “Medication treats Disease” or “Employee is part of Department.”</li></ul><h4>How these models become training inputs:</h4><p>The graph data can be exported as sequences of triples (subject-predicate-object), paths, or subgraphs. Additional metadata, context, and constraints might be extracted.</p><p>For large language models and Gen AI, this graph-informed data is often converted into text, tabular, or structured formats suitable for model training.</p><h4>Training Approaches</h4><h4>a) Supervised Fine-Tuning</h4><ul><li>Gen AI models can be fine-tuned using sentences, Q&amp;A pairs, and natural language explanations generated or guided by the graph.</li><li>The ground truth for relationships and facts comes directly from the graph structure, improving factual and contextual quality.</li></ul><h4>b) Graph Embedding</h4><ul><li>Nodes and edges in a graph are transformed into <em>graph embeddings</em>: dense vector representations capturing the graph’s structure, proximity, and semantics.</li><li>These embeddings can be injected into transformer architectures or used in multi-modal Gen AI to enhance learning about entity relations and graph topology.</li></ul><h4>c) Retrieval-Augmented Generation (RAG)</h4><ul><li>During inference (not always training), Gen AI queries a knowledge graph in real time to fetch up-to-date relationships, facts, and context, grounding its outputs with the most relevant information.</li><li>Graph search results are embedded or translated to text to inform the generation.</li></ul><h4>d) Prompt Engineering with Semantic Annotation</h4><ul><li>Graph-based models can auto-generate prompts, completion templates, or context for Gen AI. This constrains output, guides logic, and ensures adherence to domain standards.</li></ul><h4>What Does the Model Learn?</h4><ul><li>Entities and Concepts: The kinds of things that exist in the domain (products, diseases).</li><li>Relationships and Hierarchies: Which entities are related, what parent-child/taxonomy they follow, how their properties change.</li><li>Constraints/Rules: Permissible actions, domain-specific restrictions, temporal/causal relationships.</li><li>Reasoning Paths: How to logically traverse relationships to answer complex queries or infer new knowledge.</li></ul><h4>Technical Integration</h4><ul><li>Pre-processing: Graph data is prepared via serialization (e.g., RDF triples, CSV exports, API) and annotated for context.</li><li>Embedding/Alignment: Special neural layers (like Graph Neural Networks or attention modules) ingest graph structure alongside sequence/text input.</li><li>Hybrid Models: Some architectures combine LLMs with GNNs (Graph Neural Networks) or use knowledge-enhanced vectors in transformer pipelines.</li><li>Evaluation: During/after training, models are tested on their ability to generate, retrieve, or explain outputs that match graph-derived truth.</li></ul><h4>Benefits for Gen AI Training</h4><ul><li>Less Hallucination: Factual grounding reduces made-up information.</li><li>Better Reasoning: Pathfinding, relationship chain traversal, logical inference.</li><li>Domain Alignment: Outputs reflect organizational or expert knowledge, not just web-scale associations.</li><li>Explainability: Models can justify outputs based on explicit graph relationships.</li></ul><h4>Examples</h4><ul><li>Medical Gen AI: Trained with medical ontologies (like SNOMED CT, UMLS), learns accurate disease–treatment–symptom chains and guidelines.</li><li>Enterprise Assistant: Uses company knowledge graph; learns about org structures, policies, customer relationships, business logic.</li></ul><p>Here are illustrative code snippets (Python, using popular libraries) that demonstrate the main stages by which Gen AI systems can train on and use graph-based semantic models. This will include:</p><ol><li>Exporting graph data as triples</li><li>Graph embeddings for use with neural models</li><li>Retrieval-Augmented Generation with knowledge graph search</li><li>Prompt engineering with context from a graph</li></ol><h4>Exporting Graph Data as Triples</h4><p>Suppose you have a graph in NetworkX or RDFlib:</p><pre># RDFlib for triple extraction (RDF graph)<br>from rdflib import Graph<br>g = Graph()<br>g.parse(&quot;your_graph_file.ttl&quot;, format=&quot;ttl&quot;) # Load an RDF file<br>triples = []<br>for subj, pred, obj in g:<br>triples.append((str(subj), str(pred), str(obj)))<br>print(triples[:3])<br># Output: [(subject1, predicate1, object1), (subject2, predicate2, object2), …]</pre><h4>Generating Graph Embeddings</h4><p>Use a library like stellargraphor PyKEENfor graph node embeddings:</p><pre>from stellargraph import StellarGraph<br>from stellargraph.data import BiasedRandomWalk<br>from gensim.models import Word2Vec<br>import networkx as nx<br># Example with NetworkX<br>G_nx = nx.read_edgelist(&quot;your_edgelist.txt&quot;)<br>G = StellarGraph.from_networkx(G_nx)<br>rw = BiasedRandomWalk(G)<br>walks = rw.run(<br>nodes=G.nodes(),<br>length=32,<br>n=10,<br>p=0.5,<br>q=2.0<br>)<br>str_walks = [[str(n) for n in walk] for walk in walks]<br>model = Word2Vec(str_walks, vector_size=128, window=5, min_count=0, sg=1)<br>embedding = model.wv[&#39;entity_id&#39;]<br>print(embedding.shape)<br># Output: (128,)</pre><h4>Retrieval-Augmented Generation (RAG) with Knowledge Graph Lookup</h4><p>A Gen AI system can use a search API before calling an LLM:</p><pre>def knowledge_graph_lookup(subject, predicate):<br># Fake function for example<br># In production, query a triple store (e.g., SPARQL endpoint)<br>if (subject, predicate) == (&quot;Flu&quot;, &quot;treatedBy&quot;):<br>return [&quot;Paracetamol&quot;, &quot;Rest&quot;]<br>return []<br># Integrating KG search with an LLM prompt<br>query = &quot;What treats the Flu?&quot;<br>meds = knowledge_graph_lookup(&quot;Flu&quot;, &quot;treatedBy&quot;)<br>llm_prompt = f&quot;The Flu is treated by: {&#39;, &#39;.join(meds)}.\nAnswer this question: {query}&quot;<br># Call your Gen AI model with llm_prompt as context</pre><h4>Prompt Engineering with Contextual Graph Information</h4><p>Supplying graph-informed context to an LLM:</p><pre># Assume we&#39;ve traversed a graph to get related entities<br>context = &quot;Entity: Diabetes\nSymptoms: Frequent urination, Thirst\nTreatments: Insulin, Diet&quot;<br>question = &quot;What are common treatments for Diabetes?&quot;<br>prompt = f&quot;{context}\n\nQuestion: {question}\nAnswer:&quot;<br># Feed `prompt` to your Gen AI model for grounded completion</pre><h4>Integration with Model Training</h4><p>If using graph embeddings as features in a neural model:</p><p>Note:</p><ul><li>In real systems, the process may be much more complex and optimized, including advanced APIs for graph search, fine-tuning LLMs on synthetic Q&amp;A from graphs, and storing/querying multi-hop reasoning paths.</li><li>But the code above conveys the core principles: extraction, embedding, retrieval, and integration with a language model pipeline. Let me know if you want code for a specific framework or end-to-end workflow!</li></ul><p>Here’s a breakdown of the process:</p><h4>Knowledge Graph to Triples</h4><ul><li>A knowledge graph is typically represented as a collection of triples:</li></ul><p>(subject, predicate, object)</p><p>Example: (“Flu”, “treatedBy”, “Paracetamol”)</p><ul><li>This structure captures entities (nodes) and the relationships (edges) between them, forming a graph of interconnected facts.</li></ul><h4>Triple Conversion and Embedding</h4><p>Each triple, or node (entity/relation), is converted into a vector representation (embedding)using specialized graph embedding techniques.</p><p>Methods like Node2Vec, TransE, Graph Neural Networks, or transformer-based approaches generate dense vectors that capture semantic and relational meaning.</p><p>These vectors can be stored in a vector database(e.g., Pinecone, Weaviate, Milvus, FAISS).</p><p>Each entry typically includes:</p><ul><li>The original triple or entity/relationship text</li><li>Its vector embedding Optional metadata (source, context, type)</li></ul><h4>Prompt-Time Lookup</h4><p>When a user provides a prompt or question (e.g., “How is Flu treated?”), the system:</p><p>Generates an embedding for the prompt (using the same embedding model).</p><ul><li>Searches the vector database for the closest (most semantically similar) knowledge graph entries/triples/entities — this is a nearest neighbor search.</li><li>The retrieved graph context (relevant facts or relationships) is then: Used to enrich the promptas context before sending it to the Gen AI model. Or, sometimes, directly assembled as an answer.</li></ul><h4>Grounded Generation</h4><ul><li>The Gen AI model uses this retrieved, semantically-structured information alongside the prompt to produce accurate, context-aware, and grounded outputs — drawing not just from its own pre-training, but also from explicit knowledge encoded in the graph.</li></ul><h4>Summary Table</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*oT2_5hHdRInBLoTnQ9oshw.png" /></figure><h4>Key Benefits</h4><ul><li>Factual grounding:Reduces hallucinations, aligns answers with curated domain knowledge.</li><li>Semantic matching:Finds deeper connections than just keyword search.</li><li>Extendibility:Easy to update/add knowledge without retraining the LLM.</li></ul><p>If your data is shared as a YAML file using the Semantic Modeling Language (SML) format, here’s how it typically plays into Gen AI workflows and knowledge integration:</p><h4>Nature of SML YAML File</h4><ul><li>The SML YAML configuration describes business entities, relationships, metrics, hierarchies, and calculation logic in a structured, human- and machine-readable way.</li><li>Example snippet:</li></ul><pre>entity: Customer<br>attributes:<br>  - id: integer<br>  - name: string<br>  - segment: string<br>metrics:<br>  total_orders: sum(Order.amount)<br>  average_order_value: avg(Order.amount)<br>relationship:<br>  - Customer has_many Order</pre><h4>Workflow for AI/Analytics Integration</h4><h4>a) Parsing and Extraction</h4><ul><li>The YAML file is parsed to extract entities, metrics, relationships, and rules.</li><li>This information forms a domain schema — effectively a business ontology for your company or project.</li></ul><h4>b) Semantic Layer Construction</h4><ul><li>The parsed SML definitions are loaded into a semantic layer engine (or an orchestration pipeline).</li><li>Analytics/AI/LLM platforms use this semantic layer as the authoritative context for generating, interpreting, and governing queries and outputs.</li></ul><h4>c) Optionally: Conversion for RAG or Embeddings</h4><ul><li>If you want to use SML config for Retrieval-Augmented Generation (RAG):</li><li>Entities, relationships, and metrics can be serialized into structured facts(e.g., as subject-predicate-object), and further embedded as vectors for semantic search.</li><li>Example triple: (“Customer”, “has_many”, “Order”) Or (“total_orders”, “is_metric_on”, “Order”)</li><li>These can be stored in a vector database for fast similarity search during prompt handling.</li></ul><h4>Grounding Gen AI or BI</h4><p>When a user asks a question (“What is a customer’s average order value?”), the system:</p><p>Looks up the relevant metric definition and relationships from the parsed SML config (or its embedded/triple representation).</p><p>Uses this information to generate:</p><ul><li>An accurate SQL query</li><li>A business-aligned response or tutorial</li><li>A context- or metric-rich prompt for an LLM.</li></ul><h4>Benefits of Using SML YAML for Gen AI and Analytics</h4><ul><li>Clarity and Governance:Business definitions are transparent, version-controlled, and easy to review.</li><li>Consistency:Guarantees every tool or AI agent refers to a single source of truth for business logic.</li><li>Scalability:Easily extendable as you define new entities or update metrics.</li><li>Interoperability:SML models can be converted to other formats or used as bridges between BI, analytics, and AI tools.</li></ul><h4>Example: Prompt and Code Usage</h4><p>Parsing SML YAML in Python:</p><pre>import yaml<br># Load SML YAML<br>with open(&#39;business_model.yaml&#39;, &#39;r&#39;) as file:<br>sml_model = yaml.safe_load(file)<br>print(sml_model[&#39;metrics&#39;][&#39;average_order_value&#39;])<br># Output: avg(Order.amount)</pre><p>Enriching LLM prompt:</p><p>llm_prompt = f”In our analytics model, ‘average_order_value’ means {sml_model[‘metrics’][‘average_order_value’]}. Provide an explanation using this definition.”</p><h4>Key Points</h4><ul><li>SML YAML isn’t a knowledge graph — it’s a declarative schema/config language optimized for analytics, but can be programmatically converted into knowledge graph structures if needed.</li><li>Primary usage is as a backbone for semantic layers in BI/AI, but it can be adapted for RAG and context-rich LLM prompting.</li><li>It helps the Gen AI agent generate answers, SQL queries, or explanations grounded in the true intent and logic of your business.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed967b3f1852" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Smaller Datasets Can Still Produce Effective LLMs]]></title>
            <link>https://medium.com/@senthilsivam/how-smaller-datasets-can-still-produce-effective-llms-ab418458a799?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab418458a799</guid>
            <category><![CDATA[llm-tuning]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Mon, 09 Jun 2025 14:39:32 GMT</pubDate>
            <atom:updated>2025-06-09T14:39:32.342Z</atom:updated>
            <content:encoded><![CDATA[<p>Despite the common perception that Large Language Models (LLMs) require massive datasets, AI engineering principles and recent research show that smaller datasets — if used strategically — can still yield highly effective models. Here are the primary ways this is achieved:</p><p><strong>1. Fine-Tuning Pre-Trained LLMs</strong></p><p>Most LLMs are first pre-trained on vast, general datasets. Fine-tuning adapts these models to specific tasks or domains using much smaller, focused datasets. This process leverages the general linguistic and world knowledge the model already possesses, allowing it to quickly specialize with limited new data<a href="#fn1">[1]</a><a href="#fn2">[2]</a><a href="#fn3">[3]</a>.</p><p>Fine-tuning can be enhanced with parameter-efficient techniques (like LoRA or Prefix Tuning), which update only a subset of the model’s parameters, reducing the risk of overfitting and computational costs<a href="#fn3">[3]</a>.</p><p><strong>2. Transfer Learning</strong></p><p>Transfer learning enables practitioners to start with a robust, pre-trained LLM and adapt it to new, niche tasks. This approach dramatically reduces the data needed for effective training, as the model already understands language structure and general concepts<a href="#fn2">[2]</a><a href="#fn4">[4]</a><a href="#fn5">[5]</a>.</p><p><strong>3. Data Quality and Curation</strong></p><p>With small datasets, the quality and relevance of data become paramount. Carefully curated, high-quality, and representative samples can compensate for lower quantity, ensuring the model learns the most important patterns for the target application<a href="#fn4">[4]</a><a href="#fn6">[6]</a>.</p><p>Human expertise is often leveraged to select, clean, and annotate these datasets, further increasing their value<a href="#fn6">[6]</a>.</p><p><strong>4. Data Augmentation</strong></p><p>Techniques such as synonym replacement, back-translation, random insertion, and random deletion artificially expand the training set. This helps the model generalize better and reduces overfitting, even when the original dataset is small<a href="#fn4">[4]</a><a href="#fn6">[6]</a><a href="#fn5">[5]</a>.</p><p><strong>5. Active Learning</strong></p><p>Active learning prioritizes the most informative or uncertain examples for training. By focusing on data points that will most improve the model, active learning maximizes the impact of each labeled example, making small datasets go further<a href="#fn4">[4]</a><a href="#fn6">[6]</a>.</p><p><strong>6. Data-Efficient Sampling and Selection</strong></p><p>Recent methods like Ask-LLM and Density sampling use the model itself to assess the quality and diversity of training examples, selecting only the most valuable data. Studies show that models trained on these curated subsets can match or even outperform those trained on full datasets, while converging faster and using fewer resources<a href="#fn7">[7]</a><a href="#fn8">[8]</a>.</p><p><strong>7. Distillation and Rationale-Based Training</strong></p><p>·“Distilling step-by-step” is a novel approach where rationales (explanations) generated by a large LLM are used to train smaller models. This technique enables smaller models to learn complex reasoning with less data, sometimes outperforming larger models trained on more data<a href="#fn9">[9]</a><a href="#fn10">[10]</a>.</p><p><strong>8. Iterative Hyperparameter Tuning and Model Selection</strong></p><p>Careful tuning of learning rate, batch size, and training epochs can optimize performance on small datasets. Starting with smaller models and scaling up as needed helps avoid overfitting and ensures efficient use of limited data<a href="#fn4">[4]</a><a href="#fn11">[11]</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*6LYC-lAru2KUd82PjPE2sg.png" /></figure><p><strong>In essence:</strong><br>AI engineering has developed a suite of strategies — fine-tuning, transfer learning, data augmentation, active learning, and more — that allow LLMs to be effectively trained or adapted using smaller, high-quality datasets. These approaches make it possible to deploy powerful, domain-specific language models even when large-scale data collection is impractical</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab418458a799" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing LFMs with Graph Transformers]]></title>
            <link>https://medium.com/@senthilsivam/comparing-lfms-with-graph-transformers-bf75a9902463?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf75a9902463</guid>
            <category><![CDATA[lfm]]></category>
            <category><![CDATA[graph-transformer]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Wed, 23 Apr 2025 06:16:11 GMT</pubDate>
            <atom:updated>2025-04-23T06:16:11.745Z</atom:updated>
            <content:encoded><![CDATA[<p>To understand the differences between Liquid Foundation Models (LFMs) and Graph Transformers, it’s essential to first grasp what each model type is designed to handle and their underlying architectures.</p><p>1. Design Philosophy</p><p>- LFMs: These models are built on a custom design space that combines elements from Liquid Time-constant Networks, deep signal processing layers, and state-space models. The focus is on creating a flexible architecture that can be optimized for specific tasks, moving away from the one-size-fits-all approach of many traditional models.</p><p>- Graph Transformers: Designed specifically for graph-structured data, Graph Transformers extend the principles of attention mechanisms used in Transformers to graph data structures. Their primary goal is to efficiently process and learn from relationships in graph data, such as social networks or molecular structures.</p><p>2. Handling Data Structure</p><p>- LFMs: While LFMs can handle various types of data (including but not limited to graph data), they do not inherently assume any specific data structure. They process data sequentially through their customized layers, making them versatile for multiple data types.</p><p>- Graph Transformers: These models are tailored for graph data, explicitly considering the node-to-node relationships and graph topology in their computation. They use graph convolutional layers or similar constructs to propagate information across the graph, making them particularly effective for tasks involving complex relational data.</p><p>3. Computational Efficiency</p><p>- LFMs: Known for their efficiency in computation, LFMs can perform operations in a more linear fashion compared to Transformers, facilitating better parallelization and reducing computational complexity.</p><p>- Graph Transformers: Although designed to efficiently handle graph data by avoiding the full matrix multiplication inherent in standard graph convolutional networks, they still face challenges related to the complexity of graph operations, especially for very large graphs.</p><p>4. Application Scope</p><p>- LFMs: Due to their general-purpose design, LFMs are applicable across a broad spectrum of tasks including natural language processing, time-series analysis, and more.</p><p>- Graph Transformers: Primarily focused on tasks involving graph-structured data, these models excel in areas like recommendation systems, network analysis, drug discovery, etc., where understanding the graph structure is crucial.</p><p>Summary</p><p>While both LFMs and Graph Transformers represent advanced model architectures, they cater to different needs and data structures. LFMs offer a versatile, efficient model applicable to a wide range of tasks, whereas Graph Transformers specialize in handling complex relational data inherent in graph structures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf75a9902463" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Surface Attacks in Cloud Environment]]></title>
            <link>https://medium.com/@senthilsivam/surface-attacks-in-cloud-environment-614736293547?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/614736293547</guid>
            <category><![CDATA[cloud-security]]></category>
            <category><![CDATA[mitigation-strategies]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 05:34:57 GMT</pubDate>
            <atom:updated>2025-01-24T05:34:57.902Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Surface Attacks in Cloud Environments</strong></p><p>Surface attacks in cloud environments target vulnerabilities in the exposed components of a cloud infrastructure. These attacks can exploit misconfigurations, weak access controls, and other security gaps to gain unauthorized access to sensitive data or disrupt services.</p><p>Here are some common types of surface attacks in cloud environments:</p><p><strong>1. Insecure Network Configurations:</strong></p><ul><li><strong>Misconfigured Security Groups:</strong> Incorrectly configured security groups can expose instances to unnecessary network traffic.</li><li><strong>Weak Network Access Controls:</strong> Inadequate access controls can allow unauthorized access to network resources.</li><li><strong>Lack of Network Segmentation:</strong> Failure to segment networks can lead to lateral movement within the environment.</li></ul><p><strong>2. Weak Access Controls:</strong></p><ul><li><strong>Insufficient Identity and Access Management (IAM):</strong> Weak password policies, lack of multi-factor authentication, and excessive permissions can compromise access.</li><li><strong>Insecure API Keys and Secrets:</strong> Improperly stored or exposed API keys and secrets can be exploited.</li><li><strong>Lack of Role-Based Access Control (RBAC):</strong> Failing to implement RBAC can grant unnecessary privileges to users.</li></ul><p><strong>3. Insecure Data Storage and Handling:</strong></p><ul><li><strong>Data Exposure:</strong> Sensitive data left exposed in storage buckets or databases can be accessed by unauthorized individuals.</li><li><strong>Data Breaches:</strong> Insufficient data protection measures can lead to data breaches.</li><li><strong>Lack of Data Encryption:</strong> Sensitive data stored unencrypted is vulnerable to attacks.</li></ul><p><strong>4. Misconfigured Cloud Services:</strong></p><ul><li><strong>Insecure Storage Buckets:</strong> Publicly accessible storage buckets can expose sensitive data.</li><li><strong>Weak Database Security:</strong> Poor database configurations and lack of security patches can leave databases vulnerable.</li><li><strong>Insecure Server Configurations:</strong> Misconfigured servers can expose vulnerabilities.</li></ul><p><strong>5. Injection Attacks:</strong></p><ul><li><strong>SQL Injection:</strong> Malicious SQL queries can compromise databases.</li><li><strong>Cross-Site Scripting (XSS):</strong> Malicious scripts can be injected into web applications.</li><li><strong>Command Injection:</strong> Malicious commands can be executed on servers.</li></ul><p><strong>6. Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) Attacks:</strong></p><ul><li>Overwhelming a cloud service with traffic to render it inaccessible.</li></ul><p><strong>Mitigation Strategies:</strong></p><ul><li><strong>Implement Strong Access Controls:</strong> Use strong password policies, multi-factor authentication, and RBAC to limit access.</li><li><strong>Secure Network Configurations:</strong> Configure security groups and network access controls properly to restrict access.</li><li><strong>Protect Data:</strong> Encrypt sensitive data, implement data loss prevention (DLP) measures, and regularly back up data.</li><li><strong>Monitor and Log Activity:</strong> Monitor cloud environments for unusual activity and log all events for forensic analysis.</li><li><strong>Stay Updated:</strong> Keep cloud services and software up-to-date with the latest security patches.</li><li><strong>Regular Security Assessments:</strong> Conduct regular security assessments to identify and mitigate vulnerabilities.</li><li><strong>Use Cloud Security Tools:</strong> Leverage cloud security tools to automate security tasks and detect threats.</li></ul><p>By following these best practices, organizations can significantly reduce the risk of surface attacks in cloud environments and protect their valuable data and applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=614736293547" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hardware Software Modules (HSM)]]></title>
            <link>https://medium.com/@senthilsivam/hardware-software-modules-hsm-669102b7031c?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/669102b7031c</guid>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 05:32:02 GMT</pubDate>
            <atom:updated>2025-01-24T05:32:02.936Z</atom:updated>
            <content:encoded><![CDATA[<p>This HSMs are used to accelerate the hardware to support encryption and decryption.</p><p>Lets do a closer look at what is HSM</p><p>A Hardware Security Module (HSM) is a dedicated hardware device or appliance that provides secure storage, management, and execution of cryptographic operations and sensitive data. HSMs are used to enhance the security of cryptographic key management, digital signatures, encryption, and other security-related functions. They are particularly useful in applications where strong security is critical, such as financial transactions, digital identity management, and secure communications.</p><p>Here’s how Hardware Security Modules work:</p><ol><li><strong>Physical Isolation:</strong> HSMs are designed to be physically tamper-resistant and tamper-evident. They are often housed in secure enclosures with mechanisms to detect physical tampering, such as opening the device’s casing.</li><li><strong>Key Management:</strong> One of the primary functions of an HSM is to securely generate, store, and manage cryptographic keys. Keys stored in an HSM are protected from unauthorized access, and many HSMs support key lifecycle management, including key generation, rotation, and destruction.</li><li><strong>Cryptographic Operations:</strong> HSMs perform a wide range of cryptographic operations, such as encryption, decryption, digital signatures, and hashing. These operations are executed within the secure hardware environment of the HSM, which protects sensitive data from being exposed during the processing.</li><li><strong>Secure Execution Environment:</strong> HSMs create a secure execution environment that isolates cryptographic operations from the host system. This isolation prevents unauthorized applications or processes from accessing sensitive keys or data.</li><li><strong>Key Wrapping and Unwrapping:</strong> HSMs support the secure wrapping (encrypting) and unwrapping (decrypting) of keys. This allows keys to be securely transferred between different HSMs or stored in backup locations without exposing the actual key material.</li><li><strong>Random Number Generation:</strong> HSMs provide high-quality random number generators (RNGs) that are used for various cryptographic operations, such as key generation and encryption initialization vectors.</li><li><strong>Authentication and Access Control:</strong> HSMs enforce strict authentication and access control mechanisms. Users and applications need to authenticate themselves before they can use the HSM’s cryptographic capabilities. Role-based access control ensures that only authorized personnel can perform specific operations.</li><li><strong>Auditing and Logging:</strong> HSMs maintain comprehensive audit logs of all operations performed within the device. This helps ensure accountability and traceability for security-sensitive actions.</li><li><strong>Remote Management:</strong> Many HSMs support remote management capabilities, allowing administrators to configure, monitor, and manage the HSM device remotely.</li><li><strong>Backup and Recovery:</strong> HSMs often provide mechanisms for securely backing up and restoring cryptographic keys, allowing organizations to recover from hardware failures or other emergencies.</li></ol><p>Overall, Hardware Security Modules provide a secure and trusted environment for managing cryptographic keys and performing sensitive cryptographic operations. They are used to protect against a wide range of threats, including unauthorized access, tampering, and key theft, ensuring the confidentiality, integrity, and authenticity of sensitive data and communications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=669102b7031c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Liquid Clusters in Databricks]]></title>
            <link>https://medium.com/@senthilsivam/liquid-clusters-in-databricks-0f503034a156?source=rss-a3654f10d790------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f503034a156</guid>
            <category><![CDATA[liquid-clustering]]></category>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[orderz]]></category>
            <category><![CDATA[optimise-cluster]]></category>
            <dc:creator><![CDATA[Sendil Sadasivam]]></dc:creator>
            <pubDate>Sun, 05 Jan 2025 06:08:09 GMT</pubDate>
            <atom:updated>2025-01-05T06:08:09.786Z</atom:updated>
            <content:encoded><![CDATA[<p>Liquid clusters in Databricks are a specialized type of Kubernetes cluster designed to optimize data processing and analysis workloads. They differ from normal Kubernetes clusters in several key aspects:</p><p><strong>1. Data-Centric Architecture:</strong></p><ul><li><strong>Optimized for Delta Lake:</strong> Liquid clusters are specifically designed to work with Delta Lake, a popular open-source storage layer for large datasets. They leverage Delta Lake’s features like ACID transactions, time travel, and schema evolution to provide a robust and efficient data management platform.</li><li><strong>Data-Aware Scheduling:</strong> The cluster’s scheduling algorithms are optimized to consider data locality and access patterns. This ensures that data processing tasks are executed on nodes that have the necessary data readily available, minimizing data movement and improving performance.</li></ul><p><strong>2. Enhanced Concurrency and Scalability:</strong></p><ul><li><strong>Row-Level Concurrency:</strong> Liquid clusters support row-level concurrency for Delta tables, allowing multiple users and applications to simultaneously read and write data without conflicts. This significantly enhances data availability and enables more efficient data pipelines.</li><li><strong>Dynamic Scaling:</strong> Liquid clusters can dynamically scale up or down based on workload demands. This allows organizations to optimize resource utilization and reduce costs by only paying for the resources they need.</li></ul><p><strong>3. Simplified Data Layout Management:</strong></p><ul><li><strong>Liquid Clustering:</strong> Liquid clusters introduce a new data layout optimization technique called “Liquid Clustering.” This allows data to be dynamically clustered based on changing query patterns and access needs, eliminating the need for manual data partitioning and re-clustering.</li><li><strong>Adaptive Query Optimization:</strong> The cluster’s query optimizer can adapt to the changing data layout and access patterns, ensuring that queries are executed efficiently and effectively.</li></ul><p><strong>4. Integration with Databricks Ecosystem:</strong></p><ul><li><strong>Seamless Integration:</strong> Liquid clusters seamlessly integrate with other Databricks services, such as Databricks SQL, Databricks Notebooks, and Databricks Machine Learning, providing a unified platform for data engineering, data science, and machine learning workloads.</li><li><strong>Unified Management:</strong> Databricks provides a unified management interface for both Liquid clusters and normal Kubernetes clusters, simplifying cluster provisioning, monitoring, and maintenance.</li></ul><p>In summary, Liquid clusters in Databricks offer several advantages over traditional Kubernetes clusters for data-intensive workloads. Their data-centric architecture, enhanced concurrency, simplified data layout management, and seamless integration with the Databricks ecosystem make them a powerful and efficient platform for building and deploying data-driven applications</p><h4>How Data Layout is determined</h4><p>Data layout is determined by a combination of factors and techniques, with the primary goal of optimizing query performance and resource utilization. Here’s a breakdown:</p><p><strong>1. Data Format:</strong></p><ul><li><strong>Delta Lake:</strong> Databricks strongly recommends using Delta Lake as the default storage format for its data lakes. Delta Lake is an open-source storage layer that provides ACID transactions, time travel, and schema evolution, enabling efficient data management and analysis.</li><li><strong>Other Formats:</strong> While Delta Lake is preferred, Databricks also supports other formats like Parquet, ORC, JSON, CSV, and Avro. The choice of format can influence data layout considerations.</li></ul><p><strong>2. Data Layout Techniques:</strong></p><ul><li><strong>Partitioning:</strong> Data can be partitioned based on one or more columns (e.g., date, country, category). Partitioning improves query performance by reducing the amount of data that needs to be scanned during a query.</li><li><strong>Clustering (ZORDER):</strong> The ZORDER BY clause can be used to cluster data based on one or more columns. This technique improves data locality and query performance, especially for range queries and joins.</li><li><strong>Liquid Clustering:</strong> Introduced in Databricks, Liquid Clustering dynamically adjusts data layout based on query patterns and access needs. It eliminates the need for manual partitioning and re-clustering.</li></ul><p><strong>3. Databricks Optimization:</strong></p><ul><li><strong>Databricks Runtime:</strong> The Databricks Runtime includes optimizations for data processing and analysis, such as optimized libraries, pre-configured clusters, and performance tuning recommendations.</li><li><strong>Predictive Optimization:</strong> For Unity Catalog managed tables, Databricks recommends enabling predictive optimization. This feature automatically runs OPTIMIZE commands to improve data layout and reduce storage costs.</li></ul><p><strong>4. Data Layout Considerations:</strong></p><ul><li><strong>Query Patterns:</strong> Analyze query patterns to identify frequently accessed columns and data ranges. This information can be used to determine the most effective partitioning and clustering strategies.</li><li><strong>Data Volume and Velocity:</strong> The volume and velocity of data can also influence data layout decisions. For high-volume, high-velocity data, efficient data ingestion and processing are critical.</li></ul><h4>Key Factors determining data layout in Liquid Clusters</h4><p>In Databricks Liquid clusters, data layout is primarily determined by <strong>Liquid Clustering</strong>, a dynamic optimization technique that adapts to changing query patterns and access needs. Here are the key factors that influence Liquid Clustering’s decisions:</p><p><strong>Query Patterns:</strong></p><ul><li><strong>Frequent Filters:</strong> Columns frequently used in filter conditions are strong candidates for clustering. Liquid Clustering analyzes query logs to identify these high-cardinality columns.</li><li><strong>Join Keys:</strong> Columns involved in frequent joins are also prioritized for clustering to improve join performance.</li><li><strong>Range Queries:</strong> For range queries (e.g., WHERE date BETWEEN &#39;2023-01-01&#39; AND &#39;2023-12-31&#39;), the relevant columns are considered for clustering.</li></ul><p><strong>Data Characteristics:</strong></p><ul><li><strong>Data Skew:</strong> If data is skewed (e.g., most values concentrated in a few categories), clustering on skewed columns can improve data locality and query performance.</li><li><strong>Data Volume and Velocity:</strong> For high-volume, high-velocity data, efficient data ingestion and processing are critical. Liquid Clustering adapts to these factors by dynamically adjusting data layout.</li></ul><p><strong>Workload Changes:</strong></p><ul><li><strong>Evolving Queries:</strong> As query patterns change over time, Liquid Clustering dynamically adjusts the data layout to maintain optimal performance.</li><li><strong>New Data Ingestion:</strong> New data ingestion can impact data layout. Liquid Clustering continuously monitors data changes and adapts accordingly.</li></ul><p><strong>Resource Constraints:</strong></p><ul><li><strong>Storage Costs:</strong> Liquid Clustering considers storage costs when determining the optimal data layout. It avoids unnecessary data movement or replication.</li><li><strong>Compute Resources:</strong> The available compute resources on the cluster influence data layout decisions. Liquid Clustering aims to optimize data layout for efficient query execution within the available resources.</li></ul><p><strong>Key Points:</strong></p><ul><li>Liquid Clustering is an automated process that continuously analyzes query patterns and data characteristics to optimize data layout.</li><li>It eliminates the need for manual partitioning and re-clustering, simplifying data management and reducing administrative overhead.</li><li>By dynamically adapting to changing workloads, Liquid Clustering ensures that data is always organized in a way that maximizes query performance and minimizes resource utilization.</li></ul><p>Liquid Clustering is a powerful feature that can significantly improve the performance and efficiency of data processing workloads in Databricks. By automating data layout optimization, Liquid Clustering simplifies data management and frees up data engineers to focus on other tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f503034a156" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>