<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Denis Burakov on Medium]]></title>
        <description><![CDATA[Stories by Denis Burakov on Medium]]></description>
        <link>https://medium.com/@deburky?source=rss-993c6d118a59------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xrXnPlbIPKFSf_I4hxcqPQ.jpeg</url>
            <title>Stories by Denis Burakov on Medium</title>
            <link>https://medium.com/@deburky?source=rss-993c6d118a59------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:38:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@deburky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[I inherited a Redshift cluster nobody could explain, so I wrote its permissions as code]]></title>
            <link>https://deburky.medium.com/i-inherited-a-redshift-cluster-nobody-could-explain-so-i-wrote-its-permissions-as-code-b0a0856a680d?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0a0856a680d</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[supabase]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[redshift]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 09:43:50 GMT</pubDate>
            <atom:updated>2026-10-31T09:43:50.415Z</atom:updated>
            <content:encoded><![CDATA[<h4>Meet pgsesame: YAML permissions for Postgres, Supabase and Redshift</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0oxHpQCTITo6ZNXNZkj9lA.png" /><figcaption>pgsesame.</figcaption></figure><p>Soon after I inherited a Redshift cluster, someone asked me who could see the customer emails. I literally had no idea — every permission was added by hand, a state inherited across several team leads. Each grant probably made sense at some point. Overall, it was a total mess.</p><p>So I wrote down what the access should be, in one file, and built a tool that makes the database match it. That tool is <a href="https://github.com/almostly/pgsesame">pgsesame</a>. You declare the access you want, review the SQL it plans and apply it, much as Terraform works for infrastructure. pgsesame is open source and runs on PostgreSQL, Supabase and Amazon Redshift.</p><h3>Why hand-written grants are bad news</h3><p>A manual GRANT is generally a bad practice. When tables are dropped and recreated, someone needs to manually apply it again. When someone leaves the company, another manual step is needed. New joiner? More manual grants on the way.</p><p>pgbedrock for PostgreSQL and redtape for Redshift tried to fix this problem, but both are now unmaintained. Running redtape against a production Redshift cluster showed me what such a tool has to get right:</p><ul><li>Read the real catalog without crashing</li><li>Issue only the difference</li><li>Revoke only when asked</li><li>Cover what people declare</li></ul><p>Neither of the tools managed to achieve these goals. pgsesame is built around these lessons.</p><h3>From the mess to a spec</h3><p>Below’s the whole workflow on a local PostgreSQL 16 with pgsesame 0.4. The database starts this way: an analyst role given to alice , and bob, who got SELECT on the customers table directly from someone at some point.</p><h4>sesame import</h4><p>Most teams don’t start from an empty database, so the first step is sesame import. It reads what the database grants today and writes it into the spec (the YAML file that declares who can do what). The first plan then has nothing to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jIjn9Urrjw8wh5utAVIKOw.png" /><figcaption>sesame import.</figcaption></figure><p>sesame import picks up roles, memberships, grants, column grants, default privileges, ownership (on Redshift, it also imports masking configs). Passwords deliberately stay out. If you have a large database, you can pass --schema to look at grants in select schemas only.</p><blockquote>On Redshift, import needs to see every grant, so import refuses to run without a superuser or the ACCESS SYSTEM TABLE permission.</blockquote><h4>sesame plan</h4><p>From now on, the file is the source of truth. Here I moved bob into analyst instead of keeping the one-off grant, gave analysts marts.daily_sales and added an etl_bot user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jQQ1EpQtGcuqOcnDNFAkXw.png" /><figcaption>sesame plan.</figcaption></figure><p>sesame plan reads the live catalog and prints the SQL needed to match the spec. When there are changes, plan exits with code 2, which CI can pick up.</p><blockquote>Revokes show up in the plan, but they won’t run unless you pass --allow-revoke.</blockquote><h4>sesame apply</h4><p>sesame apply runs the plan in one transaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4fZxq1ll4XtqMb4PTMxlIw.png" /><figcaption>sesame apply.</figcaption></figure><p>Run plan right after and there’s nothing to do.</p><h4>sesame plan (later drift)</h4><p>Say someone handles an urgent request by hand while you’re out of the office. The next plan shows those grants as drift (access the spec doesn’t declare) and plans to revoke them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*52ae28Z8R_WsOhasjBsebw.png" /><figcaption>sesame plan (drift).</figcaption></figure><p>It’s your call now: either revoke them or add them to the spec.</p><h4>sesame show</h4><p>You can also save a plan as a change set. Anyone can review it with sesame show. sesame apply then runs exactly those statements, or refuses if the database changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0h6zJC6E00JJfSSFlWdipQ.png" /><figcaption>sesame show.</figcaption></figure><p>The same file covers column grants, row-level security, ownership and default privileges.</p><h3>Other databases</h3><h4>Masked data on Redshift</h4><p>If you have sensitive data sitting in your warehouse, you’ve probably heard of masking: who sees the real emails, and who sees ***. Redshift&#39;s dynamic data masking can do this, but it&#39;s not straightforward. Each policy is attached to a role at a numbered priority, and the highest priority wins. Done improperly, the wrong people see raw data.</p><p>In pgsesame, you can declare what each role should see:</p><pre>masking:<br>  policies:<br>    redact: {type: varchar(256), using: &quot;&#39;***&#39;::varchar(256)&quot;}<br>    email_domain: {type: varchar(256), using: &quot;regexp_replace(value, &#39;^[^@]+&#39;, &#39;***&#39;)&quot;}<br>  columns:<br>    crm.customers.email:<br>      mask: redact  # everyone else<br>      unmasked: [pii_reader]  # the raw value<br>      roles:<br>        support: email_domain  # their own mask</pre><p>pgsesame takes care of the priorities for you. The default mask goes on PUBLIC, role-specific masks go above it in the order you choose; unmasked roles go on top. After apply, the next plan is empty. You need a superuser or the sys:secadmin role to plan masking.</p><p>You also don’t need a stored password to connect. pgsesame can sign in with temporary IAM credentials to a provisioned Redshift cluster or a Serverless workgroup, or go through the Redshift Data API when the database isn’t reachable over the network. RDS and Aurora PostgreSQL work in the same way: pass --rds with the instance or cluster name and pgsesame finds the endpoint.</p><h4>Supabase</h4><p>On October 30, 2026, Supabase stops automatically granting new tables in public to its Data API roles (anon, authenticated and service_role). From then on, every new table needs an explicit grant. That’s exactly what a spec is for, and sesame plan --migration writes the plan as a migration file for supabase db push.</p><p>Why should you care? Anyone with your project’s anon key talks to the database as anon, so row-level security is the only thing between them and your rows. pgsesame warns when a table is open to the API with row-level security off. Here’s a todos table created in the SQL editor of a local Supabase:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ALh43LhUGDsS5BB1O_gYdQ.png" /></figure><p>It also warns about default privileges that hand new tables to an API role. If you imported a spec before October 30, it still has Supabase’s old defaults in public, and applying it would bring them back. The warning tells you which ones to drop.</p><h4>Access changes as pull requests</h4><p>Once the spec lives in a repository, access changes become pull requests. Someone who needs access opens a pull request, a reviewer checks the plan and approves, and the merge applies it. pgsesame ships a GitHub Action for this:</p><pre>on:<br>  pull_request:<br>  push:<br>    branches: [main]<br><br>jobs:<br>  plan:<br>    runs-on: ubuntu-latest<br>    permissions: {contents: read, pull-requests: write}<br>    env: {SESAME_DSN: &quot;${{ secrets.SESAME_DSN }}&quot;}<br>    steps:<br>      - uses: actions/checkout@v4<br>      - uses: almostly/pgsesame@v0.5.0<br>        with: {command: plan, spec: permissions.yaml}<br><br>  apply:<br>    if: github.event_name == &#39;push&#39;<br>    needs: plan<br>    runs-on: ubuntu-latest<br>    environment: production<br>    env: {SESAME_DSN: &quot;${{ secrets.SESAME_DSN }}&quot;}<br>    steps:<br>      - uses: almostly/pgsesame@v0.5.0<br>        with: {command: apply}</pre><p>On a pull request, the plan is posted as a single comment that updates on each push. On merge, apply runs the change set the plan job saved, or refuses if the database changed in between.</p><p>Now the spec tells you who has access, and the pull requests tell you who approved it and when. Next time someone asks who can see the customer emails, the answer is right in front of them.</p><h3>Try it</h3><p>pgsesame is at 0.4 and still alpha. It’s tested against PostgreSQL 14 to 18, Supabase, Redshift Serverless and redshift-local (a local Redshift emulator from my oblako project), on Python 3.10 to 3.13.</p><p>To try it:</p><pre>uvx pgsesame --help                    # try it without installing<br>uv tool install &quot;pgsesame[redshift]&quot;   # Redshift through IAM or the Data API<br>uv tool install &quot;pgsesame[rds]&quot;        # RDS and Aurora through IAM or the Data API</pre><p>Not using Python in your CI? There’s a Docker image at ghcr.io/almostly/pgsesame. The code is Apache-2.0 at <a href="https://github.com/almostly/pgsesame">github.com/almostly/pgsesame</a>.</p><p>If you’ve fought with grants on Postgres or Redshift, I’d like to hear what matters and how pgsesame can help — issues and pull requests are welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0a0856a680d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Serverless ML Feature Store on AWS with Feast]]></title>
            <link>https://aws.plainenglish.io/a-serverless-ml-feature-store-on-aws-with-feast-06a06cd8abaf?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/06a06cd8abaf</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 17:14:35 GMT</pubDate>
            <atom:updated>2026-09-21T17:14:35.709Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Setting up Feast for batch and real-time ML inference</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTxccuYlFESxMP9leGKoWg.jpeg" /><figcaption><a href="https://feast.dev/">Feast — The Open Source Feature Store for Machine Learning</a>.</figcaption></figure><p>A couple of years ago I sat in a talk on feature stores at PyCon in Berlin. The thing I remember is when the speaker said he deliberately won’t name a feature store to use. He spent the next hour talking about the general problem instead, but for me that was exactly why I came to listen.</p><p>So I did the review myself over the next years, looking into SageMaker Feature Store, Hopsworks, and a few others. Each had its pros and cons, and none was obviously wrong, but none was what I wanted.</p><p>What I saw in Feast was almost aesthetic. Its Python just looked right: clear, well-written feature definitions that read like what I would expect from an open-source store. I already had Redshift as the offline store, DynamoDB as the online one, and dbt for the transforms, so I decided to give it a try.</p><p>The worked example in this piece is a fraud model on card transactions. The first question with features is how to serve the exact same numbers to training and to inference jobs without train-serve skew.</p><h3>Small history of Feast</h3><p>Feast was built at Gojek in 2018, with Google Cloud, to stop a dozen ML teams from re-implementing the same features, and open-sourced in January 2019 on the back of the Kubeflow community. In 2020 Tecton, the commercial feature-platform company founded by the people who had built Uber’s Michelangelo, became by far its largest contributor, and Feast’s original creator joined them. That is usually the moment an open-source project quietly turns into a single vendor’s on-ramp. Feast did not, because in the same year it was placed under the Linux Foundation’s AI and Data umbrella, so governance sat with the foundation rather than with any one company.</p><p>When Tecton later refocused on its managed product, maintenance passed to engineers at Affirm, Red Hat, J.P. Morgan and independent contributors, who steered the project toward a 1.0. Feast became a neutral, self-hostable registry and serving layer.</p><h3>Feast and Redshift</h3><p>Feast’s Redshift offline store operates through two AWS interfaces:</p><ul><li>the redshift-data API, the HTTP control plane that submits SQL and polls for results, so there is no connection to pool and no driver to configure,</li><li>S3, because Feast stages the result of a historical-features join by having Redshift UNLOAD it to a bucket and reading the Parquet back.</li></ul><p>The entire configuration lives in feature_store.yaml. On Redshift Serverless it looks like this:</p><pre>project: feast_ml<br>provider: aws<br>registry: s3://feast-production/registry/registry.db<br>offline_store:<br>  type: redshift<br>  region: eu-west-1<br>  workgroup: analytics-serverless<br>  database: dev<br>  s3_staging_location: s3://feast-production/staging<br>  iam_role: arn:aws:iam::111122223333:role/redshift-feast-unload<br>online_store:<br>  type: dynamodb<br>  region: eu-west-1<br>entity_key_serialization_version: 3</pre><p>On a provisioned cluster you name the cluster and a database user, and Feast asks redshift:GetClusterCredentials for a temporary password. On Serverless there is neither cluster nor user (only the workgroup), and Feast authenticates through the Data API against your own IAM identity.</p><p>Permissions come in three layers, and each fails differently:</p><ul><li><strong>IAM for the caller.</strong> It needs redshift-data:ExecuteStatement, DescribeStatement, GetStatementResult, redshift-serverless:GetCredentials, and the easy one to miss, redshift-data:DescribeTable, which Feast uses to read source schemas. Without it, feast apply fails on AccessDenied even though ExecuteStatement works,</li><li><strong>SQL grants in Redshift.</strong> The IAM identity maps to its own database user, separate from whoever built the mart. That user needs USAGE on the feature schema and SELECT on its tables. Otherwise get_historical_features fails with permission denied for schema feature_store while every IAM policy looks correct. In production, have the dbt build grant it,</li><li><strong>The UNLOAD role.</strong> Redshift assumes a separate role to write join results to s3_staging_location. That role must be attached to the namespace, must trust redshift.amazonaws.com, and must have write access to the staging prefix. As in my Redshift ML piece, the failure is asynchronous: the statement submits, then fails when Redshift cannot reach S3.</li></ul><p>feast apply does three things. It validates the definitions against the sources, writes them to registry.db in S3, and creates or updates one DynamoDB table per online feature view. It never touches feature data, so it is a fast deploy step, not a data job. Because the registry is a single file, run it from CI rather than from laptops, or two people will overwrite each other.</p><blockquote>The online store is optional. If you only want to use it for training, set online_store: null.</blockquote><h3>dbt and Feast</h3><p>Feast is a registry, the typed catalog of feature views and the services that pin them, plus two read paths: the offline <em>point-in-time</em> join for training and the online lookup for serving. The logic that produces a feature lives outside of it, in dbt.</p><p>If a feature needs logic, the logic is a dbt model first. Every feature view points at a warehouse table keyed by an entity id with an event-timestamp column, and nothing else. On a real platform that table is a reviewed dbt mart, one model to one view, so a new feature means a new mart before it can mean a new view. In this example it is feature_store.transactions, one row per card transaction.</p><p>An entity is the join key. Here it is the transaction itself:</p><pre>from feast import Entity, ValueType<br><br>transaction = Entity(<br>    name=&quot;transaction&quot;,<br>    join_keys=[&quot;event_id&quot;],<br>    value_type=ValueType.STRING,<br>    description=&quot;A single card transaction, keyed by its event id.&quot;,<br>)</pre><p>Keying on the transaction keeps the example simple. In production you would usually key on the card or the customer.</p><p>A feature view is a typed window over one table. The RedshiftSource carries a schema and a table, since it was already materialized:</p><pre>from datetime import timedelta<br>from feast import FeatureView, Field<br>from feast.infra.offline_stores.redshift_source import RedshiftSource<br>from feast.types import Float64, Int64<br><br>transactions_source = RedshiftSource(<br>    name=&quot;transactions_source&quot;,<br>    schema=&quot;feature_store&quot;,<br>    table=&quot;transactions&quot;,<br>    timestamp_field=&quot;event_timestamp&quot;,<br>)<br>fraud_features = FeatureView(<br>    name=&quot;fraud_features&quot;,<br>    entities=[transaction],<br>    ttl=timedelta(days=3650),<br>    schema=[<br>        Field(name=&quot;order_price&quot;, dtype=Float64),<br>        Field(name=&quot;card_bin&quot;, dtype=Int64),<br>        Field(name=&quot;billing_zip&quot;, dtype=Int64),<br>        Field(name=&quot;billing_latitude&quot;, dtype=Float64),<br>        Field(name=&quot;billing_longitude&quot;, dtype=Float64),<br>    ],<br>    online=True,<br>    source=transactions_source,<br>)</pre><p>A model version then pins its exact feature set as a feature service, and both training and inference reference the service:</p><pre>from feast import FeatureService<br><br>fraud_model_v1 = FeatureService(<br>    name=&quot;fraud_model_v1&quot;,<br>    features=[fraud_features, geo_mismatch],  # geo_mismatch is the ODFV defined later<br>)</pre><p>Feast does not have a special treatment for labels. For training, you pass get_historical_features an entity dataframe: the rows you want features for, each with an event_id and an event_timestamp. The label goes in that dataframe as an extra column, and Feast carries it through the point-in-time join untouched. Registering the label as a feature view would leak it, because any model could then read the outcome as an input.</p><pre>entity_df = pd.read_sql(<br>    &quot;SELECT event_id, event_timestamp, is_fraud FROM feature_store.labels&quot;,<br>    conn,<br>)<br>training_df = store.get_historical_features(<br>    entity_df=entity_df,<br>    features=store.get_feature_service(&quot;fraud_model_v1&quot;),<br>).to_df()</pre><h3>AWS deployment</h3><p>We ran the five steps against a real Redshift Serverless workgroup, on the same fraud dataset, all 116,705 transactions. It is the scaffold that ships with this piece, driven straight from its Makefile: make deploy, make load-data, make apply, then the loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJufURpbtGwVAUACcLDyuw.jpeg" /><figcaption><em>The offline store: a Serverless namespace and workgroup, no cluster to size. The query sparklines are the </em><em>get_historical_features UNLOAD running.</em></figcaption></figure><p>get_historical_features submits the point-in-time join over the Data API and UNLOADs the result to the staging prefix. The training set, the validation split, and the batch to score all land in S3 as the join’s output, which is exactly where it breaks if the UNLOAD role cannot write.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EA0PWVllVeFAQ39f41cwVA.jpeg" /><figcaption><em>Everything the offline store produced, as S3 objects: the join output plus the train/validation/batch splits.</em></figcaption></figure><p>materialize then writes the latest row per entity into DynamoDB, and get_online_features reads it back in milliseconds. Here is that online store after the run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XppMb57axl7QSWByCzdKxA.jpeg" /><figcaption><em>fraud_demo.fraud_features after </em><em>make apply and materialize: one item per entity, the feature values serialized by Feast.</em></figcaption></figure><p>The training set feeds a SageMaker XGBoost job. The model then goes behind a Serverless Inference endpoint for per-request scoring and a Batch Transform for batch inference tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MLHneVp8Z41qTQIJwlU-kA.png" /><figcaption><em>A Serverless endpoint for real-time decisions and batch transform jobs.</em></figcaption></figure><p>The fraud score below is calibrated so each cut carries a known false-positive rate, so you pick the cut by the alarm rate you are willing to pay and read off the recall it buys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qjrxo1QwJv9dD6-0A1jHzg.png" /><figcaption><em>The 0–1000 fraud score with its decision bands. At the 2% false-alarm cut (score 900) the model works a small review queue rather than a flood of good customers.</em></figcaption></figure><h3>The online store</h3><p>The online store is a cache of precomputed feature values. materialize (or materialize_incremental) reads each feature view from the offline store, takes the latest value per entity key as of now, and writes it into DynamoDB as one item per (feature view, entity). get_online_features(entity_rows=[{&quot;event_id&quot;: ...}]) is then a handful of key-value lookups.</p><p>On a real run this is what lands in DynamoDB:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aLqx3Lz5pP7Ywjxz0HuRRQ.jpeg" /><figcaption><em>One item expanded: </em><em>entity_id, </em><em>event_ts, and a </em><em>values map holding each materialized feature.</em></figcaption></figure><p>Point-in-time joins and history live offline for training. Online, you only get the latest row per entity.</p><h3><strong>On-demand feature view (ODFV)</strong></h3><p>Feast mostly serves rather than transforms, and parity comes from computing once and serving twice. Our features are built by dbt into Redshift tables. The offline store reads those rows for training datasets, and materialization copies the same rows online.</p><p>The one place Feast does run code is the <strong>on-demand feature view (ODFV)</strong>: a Python function registered in the feature repo that Feast executes. During get_historical_features it runs over the training dataframe and in the online path, get_online_features, it runs per request. Same function object.</p><p>The ODFV is ideal when a feature has to be computed at decision time by combining the incoming request with stored, batch-computed features.</p><p>An ODFV takes two kinds of input: existing feature views (the materialized values) and a RequestSource, fields that only exist at decision time and arrive in the request itself.</p><p>A classic fraud signal is a good example: the billing location is stored with the transaction, while the location the request actually came from is only known at decision time, geolocated from the IP. The distance between them is this feature:</p><pre>scoring_request = RequestSource(<br>    name=&quot;scoring_request&quot;,<br>    schema=[<br>        Field(name=&quot;ip_latitude&quot;, dtype=Float64),<br>        Field(name=&quot;ip_longitude&quot;, dtype=Float64),<br>    ],<br>)<br><br>@on_demand_feature_view(<br>    sources=[fraud_features, scoring_request],<br>    schema=[Field(name=&quot;billing_ip_distance_km&quot;, dtype=Float64)],<br>)<br>def geo_mismatch(inputs: pd.DataFrame) -&gt; pd.DataFrame:<br>    lat1, lon1 = np.radians(inputs[&quot;billing_latitude&quot;]), np.radians(inputs[&quot;billing_longitude&quot;])<br>    lat2, lon2 = np.radians(inputs[&quot;ip_latitude&quot;]), np.radians(inputs[&quot;ip_longitude&quot;])<br>    dlat, dlon = lat2 - lat1, lon2 - lon1<br>    a = np.sin(dlat / 2) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin(dlon / 2) ** 2<br>    out = pd.DataFrame()<br>    out[&quot;billing_ip_distance_km&quot;] = 6371.0 * 2 * np.arcsin(np.sqrt(a))<br>    return out</pre><ul><li>The scoring service calls the feature server (feast serve is a ready-made HTTP endpoint, or the SDK in-process) with the event_id plus the request fields,</li><li>Feast fetches the stored billing location from DynamoDB, runs the ODFV on the merged row, and returns the full feature vector including billing_ip_distance_km; the model scores,</li><li>At training time, the same ODFV runs over the historical dataframe, with the historical IP location fed in as the request-source column.</li></ul><p>I measured that call against real DynamoDB, which you can find in the code for this article. The lookup took most time: a get_online_features read is a DynamoDB GetItem, single-digit milliseconds in-region, and Feast adds almost nothing over the raw GetItem. The geo_mismatch ODFV adds a few milliseconds on top, and those milliseconds are Feast’s on-demand-transform plumbing.</p><p>ODFVs execute pandas inside the serving process, so the feature server is a Python service. A cold pandas import in a scale-to-zero Lambda takes seconds. Under concurrent load the lever is Feast’s async online path (get_online_features_async, DynamoDB over aiobotocore): it interleaves many reads on one event loop and gives several times the throughput on a busy feature server, though it does nothing for a single call&#39;s latency.</p><h3>What Feast isn’t designed for</h3><p>ODFVs cannot do windowed aggregation over raw events at request time, say, the “sum of transactions in the last 7 days” kind of feature. Feast has no streaming compute engine. Its streaming story is a push source instead: you compute the aggregate somewhere else, a Lambda on the event stream, Flink, or a frequent batch job, and push the finished value into the online store, where it is served like any other feature.</p><p>That leaves three options:</p><ul><li><strong>Heavy history</strong> (per-card velocity, merchant risk, device and address history): computed by dbt batch in Redshift, served by materialize then a DynamoDB lookup,</li><li><strong>Request-derived</strong> (billing-vs-IP distance, amount-vs-typical, request flags): computed by an ODFV, the same code offline and online, per request in the feature server,</li><li><strong>Truly fresh aggregates</strong> (“charges on this card in the last hour”): computed by an upstream stream or batch job and pushed into the online store.</li></ul><p>For a fraud-service design this is compatible with serving precomputed scores behind an API: the online store plus ODFV path only becomes necessary when the scorer wants features computed at decision time rather than looked up.</p><h3>Bonus: SUPER type for transforms</h3><p>There is a second design I considered, specific to Redshift as the offline store. The idea is to land the request and response JSON in Redshift, keep the payload as one SUPER column, and define features as dbt SQL views over it.</p><p>This works because SUPER parses the JSON once at write time into a shredded binary form, so reading a nested field is navigation, not parsing. Measured on our own cluster, it is roughly 14x faster than json_extract_path_text over varchar, and about 300x faster on navigation-heavy PartiQL queries.</p><p>Here is what that looks like:</p><pre>-- Raw table: one SUPER column holds the whole request/response payload<br>CREATE TABLE raw.scoring_events (<br>    arrived_at  TIMESTAMP,<br>    payload     SUPER<br>);<br><br>-- JSON_PARSE shreds the JSON once, at write time<br>INSERT INTO raw.scoring_events<br>SELECT GETDATE(), JSON_PARSE(&#39;{<br>    &quot;request&quot;:  {&quot;event_id&quot;: &quot;tx-1042&quot;, &quot;order_price&quot;: 129.90,<br>                 &quot;billing&quot;: {&quot;latitude&quot;: 52.51, &quot;longitude&quot;: 13.41},<br>                 &quot;items&quot;: [{&quot;sku&quot;: &quot;A1&quot;}, {&quot;sku&quot;: &quot;B7&quot;}]},<br>    &quot;response&quot;: {&quot;fraud_score&quot;: 912}<br>}&#39;);<br><br>-- dbt model: features are dot-navigation into SUPER, no string parsing<br>-- models/features/scoring_request_features.sql<br>{{ config(materialized=&#39;view&#39;) }}<br><br>SELECT<br>    payload.request.event_id::varchar AS event_id,<br>    arrived_at AS event_timestamp,<br>    payload.request.order_price::float8 AS order_price,<br>    payload.request.billing.latitude::float8 AS billing_latitude,<br>    payload.request.billing.longitude::float8 AS billing_longitude,<br>    GET_ARRAY_LENGTH(payload.request.items) AS n_items,<br>    payload.response.fraud_score::int AS fraud_score<br>FROM {{ source(&#39;raw&#39;, &#39;scoring_events&#39;) }}</pre><p>What I like about Feast is that it never has to become the system that does everything. It gives the features a consistent definition and two ways to get at them: historically for training, and immediately for inference. The ODFV is the one place Feast computes anything, and that is exactly where it belongs: in the few cases where the feature cannot exist until the request arrives.</p><p>That ended up being the reason Feast made sense to me after looking at the alternatives. It does not ask you to throw away the warehouse or build a second computation engine; it simply gives the existing pieces a contract. The architecture is deliberately boring, and that makes it a good fit for small teams that do not want to engineer everything from scratch.</p><p>You can find the code for this article in my <a href="https://github.com/deburky/aws-samples/tree/main/redshift-feast">GitHub</a>.</p><p>I also found these resources helpful in case you’d like to learn more about Feast:</p><ul><li><a href="https://aws.amazon.com/blogs/opensource/getting-started-with-feast-an-open-source-feature-store-running-on-aws-managed-services/">Getting started with Feast, an open source feature store running on AWS Managed Services</a></li><li><a href="https://docs.feast.dev/tutorials/tutorials-overview/real-time-credit-scoring-on-aws">Real-time credit scoring on AWS</a></li></ul><p><strong>Before you go</strong></p><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us">here</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=06a06cd8abaf" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/a-serverless-ml-feature-store-on-aws-with-feast-06a06cd8abaf">A Serverless ML Feature Store on AWS with Feast</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Unofficial Guide to Redshift ML]]></title>
            <link>https://deburky.medium.com/the-unofficial-guide-to-redshift-ml-f02852f4da47?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/f02852f4da47</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[redshift]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 07:01:04 GMT</pubDate>
            <atom:updated>2026-09-10T07:01:04.938Z</atom:updated>
            <content:encoded><![CDATA[<h4>What the docs say vs. how it went</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O3iB-G-4tkp7DVSr3keq6w.jpeg" /><figcaption>Amazon Redshift ML — Create, train, and deploy machine learning (ML) models using familiar SQL commands.</figcaption></figure><p>There is an easy way to train and score a machine learning (ML) model without the data ever leaving your warehouse using SQL alone. The feature that drives this is Amazon Redshift ML, and it has two hard prerequisites: RA3 node types and a supported region.</p><p>Redshift ML is not a new feature. AWS previewed it at re:Invent in late 2020 and made it generally available in 2021, aimed at analysts and SQL-fluent engineers rather than ML specialists: the idea was that anyone who could write a query could train and call a model, with Amazon SageMaker doing the heavy lifting underneath. Years on, it is mature and quietly present on most RA3 clusters and Serverless workgroups, which is part of why it is worth learning about.</p><h3><strong>Before you start</strong></h3><p>You need a few things in place:</p><ul><li>An RA3 cluster (or Redshift Serverless) in a region that supports Redshift ML,</li><li>superuser, or the CREATE MODEL privilege, on the target database,</li><li>An IAM admin on hand, because the one hard step, building the role, is an IAM change.</li></ul><p>Fresh and free-tier accounts often have a SageMaker training-instance quota of zero in some regions, so check Service Quotas for “ml.m5.xlarge for training job usage” as well as Redshift permissions (not available on the free tier).</p><p>One query against the system view confirms if you have it:</p><pre>select model_name, model_state<br>from svv_ml_model_info;</pre><p>This just returns an empty catalog of models.</p><p>That empty result is what started this whole exercise for me, and switching the feature on turned out to be far more interesting than the documentation of Redshift ML capabilities.</p><h3>The Promise: A Model Behind a SQL Function</h3><p>The pitch for Redshift ML is genuinely appealing. You write a CREATE MODEL statement, point it at a table, name a target column, and Redshift orchestrates a SageMaker training job on your behalf.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sK8D2YW4OLAIIjGElBtmbw.jpeg" /><figcaption>Amazon Redshift ML.</figcaption></figure><p>When training finishes, Amazon SageMaker Neo compiles the trained model into optimized code that Redshift runs as an ordinary in-database SQL function. From then on, scoring a row is just calling that function inside a query, with no endpoint to call out to. The model lives where the data lives.</p><p>That is the part worth paying attention to, because everything that follows was friction between me and that outcome. And the first piece of friction had nothing to do with SQL at all.</p><h3>The First Wall: No Role to Pass</h3><p>My first CREATE MODEL attempts did not fail on syntax, it was IAM. CREATE MODEL needs an IAM role that Redshift can hand to SageMaker so the training job can run, and our cluster had no such role. It had no default IAM role configured at all, and while plenty of roles were attached to the cluster over the years, none of them trusted sagemaker.amazonaws.com.</p><p>My first instinct was to reuse what we already had. We run ETL through a dedicated role, which can read and write the buckets, so I tried that. Redshift came back with a message that told me exactly what was wrong:</p><pre>Could not assume role ... allows principal<br>&#39;sagemaker.amazonaws.com&#39; to assume the role</pre><p>The role could talk to S3, but its trust policy never named SageMaker as a principal, so SageMaker could not assume it. I then tried the other documented path, role chaining, where the cluster role assumes a separate SageMaker execution role. That failed too, because the assume-role permission linking the two was never set up. Neither shortcut worked, and both failures pointed at the same missing piece. This was not a Redshift problem or a SageMaker problem.</p><h3>The Redshift ML IAM Role</h3><p>I defined the role in CloudFormation, redshift-ml-role. The important detail is the trust policy. A single role has to be assumable by two different services here, because Redshift assumes it to unload the training set to S3, and SageMaker assumes the very same role to run the training job. So both principals go in the trust policy:</p><pre>AssumeRolePolicyDocument:<br>  Version: &quot;2012-10-17&quot;<br>  Statement:<br>    - Effect: Allow<br>      Principal:<br>        Service:<br>          - redshift.amazonaws.com<br>          - sagemaker.amazonaws.com<br>      Action: sts:AssumeRole</pre><p>For the permissions policy, the getting-started guide takes the easy road and tells you to attach AmazonS3FullAccess and AmazonSageMakerFullAccess. I chose instead to mirror the scoped Redshift ML policy that AWS documents for exactly this case, granting only the actions the training flow actually exercises.</p><p>Concretely, that meant the role could:</p><ul><li>run and describe SageMaker jobs (sagemaker:*Job*) plus the explicit model and endpoint lifecycle actions,</li><li>publish training metrics with cloudwatch:PutMetricData,</li><li>pull the training container from ECR and write its logs (ecr:* pull actions and logs:*),</li><li>pass itself, and only itself, to SageMaker.</li></ul><p>That last one deserves a look, because a role that can create SageMaker jobs also needs to hand a role to those jobs, and left unbounded that is a privilege-escalation hole. The iam:PassRole grant is fenced to passing this role to SageMaker and nothing else:</p><pre>- Effect: Allow<br>  Action: iam:PassRole<br>  Resource: !GetAtt RedshiftMLRole.Arn<br>  Condition:<br>    StringEquals:<br>      iam:PassedToService: sagemaker.amazonaws.com</pre><p>The S3 access is scoped the same way, to the SageMaker artifact bucket sagemaker-eu-west-1-111122223333 where the training set and model artifacts live, plus read access to the public redshift-downloads bucket that Redshift ML uses internally. No wildcard bucket access, no AmazonS3FullAccess. It took longer to write than the two managed policies would have, and I would make the same choice again.</p><h3>Associating the Role and Waiting for In-Sync</h3><p>Writing the role is only half of it. The trust policy says who is allowed to assume the role, but the cluster still has to be told the role exists and is usable. Those are two separate facts, and you need both. Associating the role is a single call:</p><pre>aws redshift modify-cluster-iam-roles \<br>  --cluster-identifier dwh-cluster \<br>  --add-iam-roles arn:aws:iam::111122223333:role/redshift-ml-role</pre><p>The association is not instant. The role moves through an ApplyStatus that you have to watch until it reads in-sync, and only then will CREATE MODEL accept it. Attaching the role without the SageMaker principal in the trust policy would have left me back at the first wall, and having the trust policy without associating the role would have left Redshift unaware of it. Both had to be true at once.</p><h3>The Cheapest Real Test</h3><p>With the role in place I wanted the smallest honest test. The trap here is the default CREATE MODEL, the full AUTO path, which runs SageMaker Autopilot end to end and then stands up a persistent real-time inference endpoint. That endpoint bills by the hour whether you query it or not, which is a poor thing to leave behind after an experiment.</p><p>To save money, I turned AUTO off and pinned the model type to XGBOOST. That combination runs a single SageMaker training job and compiles the result into a local <strong><em>in-database function</em></strong>. For a tiny training set the job itself costs cents, and MAX_RUNTIME caps how long it can run before SageMaker stops it. Inference afterward is free, because it executes inside Redshift.</p><pre>create model public.rml_test_model<br>from public.rml_test_train<br>target label<br>function rml_test_predict<br>iam_role &#39;arn:aws:iam::111122223333:role/redshift-ml-role&#39;<br>auto off<br>model_type xgboost<br>objective &#39;binary:logistic&#39;<br>preprocessors &#39;none&#39;<br>hyperparameters default except (num_round &#39;50&#39;)<br>settings (<br>  s3_bucket &#39;sagemaker-eu-west-1-111122223333&#39;,<br>  max_runtime 1800<br>);</pre><p>That statement did not arrive fully formed. It is the residue of a short sequence of errors, each one teaching me a required clause. With AUTO off, Redshift refused the statement until I supplied HYPERPARAMETERS, because there is no Autopilot to choose them for me. Then it objected to preprocessing, which the manual path will not infer either, so I set PREPROCESSORS ‘none’ and prepared the features myself.</p><p>And it rejected my first training table outright for being too small, since Redshift ML wants at least 500 rows to train on. One more thing worth doing here: select only the feature columns and the target in the FROM. If you feed SELECT *, a row id gets trained as a feature and the generated function then expects it too.</p><p>XGBOOST is only one of the choices. With AUTO on, Autopilot picks the algorithm and covers regression, binary, and multiclass classification. With AUTO off you name it yourself: XGBOOST, MLP (a neural network), or LINEAR_LEARNER for supervised problems, and K-MEANS for unsupervised clustering.</p><blockquote>You can also bring your own pre-trained model, either compiled into a local in-database function or served from a SageMaker endpoint for remote inference. Whichever you pick, SHOW MODEL prints the model’s details once it exists: its state, model type, estimated training cost, and the name and exact argument types of the generated function, which is what you need to call it correctly.</blockquote><p>From there I polled the model catalog, the same view that had returned nothing at the start:</p><pre>select model_name, model_state<br>from svv_ml_model_info;</pre><p>This time there was a row. To my amazement, the model existed and was training. The role assumption had worked, Redshift had unloaded the training set to S3, and SageMaker had accepted the job. Which is exactly where the story gets interesting.</p><h3>The Second Wall (The Real One)</h3><p>CREATE MODEL succeeded, but the training job failed.</p><p>This is the failure mode nobody warns you about, because everything up to this point reports success. The SQL statement returns, the model shows up in the catalog, and the actual work happens asynchronously inside SageMaker, out of view of your Redshift session. To find out why it failed I had to go read the SageMaker training job’s FailureReason, and it said this:</p><pre>The execution role ... does not have permission to<br>perform ec2:DescribeSubnets</pre><p>An EC2 permission, on what I thought was a data-plane operation. The reason is a design choice in how Redshift ML isolates training. Rather than run the job on the open internet, Redshift ML launches the SageMaker training job inside the cluster’s own VPC. It passes the cluster’s subnet and security group as the job’s VpcConfig, so the job can reach the cluster and reach S3 through private networking.</p><p>A SageMaker training job that runs inside a VPC has to attach elastic network interfaces to those subnets to get there, and to do that its execution role needs permission to inspect the network and manage its own ENIs. My carefully scoped policy, the one that mirrored the AWS documentation so faithfully, had none of those EC2 actions, because the documented scoped policy leaves them out. The policy was correct for training on the public network and silently incomplete for training inside a VPC, which is the only way Redshift ML ever trains.</p><h3>Closing the Second Wall</h3><p>The fix is a dedicated statement on the role covering exactly the VPC networking actions SageMaker needs to stand up and tear down its interfaces. There is no way to opt out of needing them: Redshift ML passes the cluster’s VpcConfig to every training job unconditionally, so every job it launches lives inside the VPC and pays this ENI tax.</p><pre>- Effect: Allow<br>  Action:<br>    - ec2:CreateNetworkInterface<br>    - ec2:CreateNetworkInterfacePermission<br>    - ec2:DeleteNetworkInterface<br>    - ec2:DeleteNetworkInterfacePermission<br>    - ec2:DescribeNetworkInterfaces<br>    - ec2:DescribeVpcs<br>    - ec2:DescribeDhcpOptions<br>    - ec2:DescribeSubnets<br>    - ec2:DescribeSecurityGroups<br>  Resource: &quot;*&quot;<br>  Condition:<br>    StringEquals:<br>      aws:RequestedRegion: !Ref AWS::Region</pre><p>The Resource: &quot;*&quot; on that block bothered me, so it is worth being straight about why it stays. The EC2 Describe* actions simply do not support resource-level ARNs in IAM; AWS requires them to be granted against *. And the ENI create and delete actions target interfaces that SageMaker generates on the fly, with IDs that do not exist when I am writing the template, so I could not scope to them even if I wanted to. I could have hardcoded the specific production subnet and security group, but this is an environment-parameterized CloudFormation template meant to deploy across dev, staging, and production, and baking one environment&#39;s network into it would be its own kind of wrong.</p><p>So instead of pretending to a resource scope the actions cannot honor, I bound the statement with an aws:RequestedRegion condition pinned to the stack&#39;s own region. It is not the tightest policy imaginable. It is the honest one for a cross-environment template, and I would rather name that tradeoff than hide it behind a wildcard with no condition at all.</p><h3>Do It Yourself</h3><p>To deploy this yourself, you can use a CloudFormation template, deployed with AWS SAM.</p><p>SAM creates the role with both principals in the trust policy, the self-fenced iam:PassRole, and the VPC networking block, and it can also stand up a Redshift Serverless namespace and workgroup with the role already attached, so a single sam deploy is fully sufficient.</p><p>A short SQL walkthrough then runs CREATE MODEL with AUTO OFF, polls until the model is Ready, and scores rows with the generated function. The role is the hard-won part: the template makes it repeatable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCGrGJj-MM6NJdxndZ-MAA.png" /><figcaption>Deployment of Redshift ML with AWS SAM.</figcaption></figure><p>Two things I learned deploying it fresh, both now baked into the code. The region has to have SageMaker training-instance quota, or CREATE MODEL fails asynchronously with ResourceLimitExceeded. And one grant is easy to miss: to let a non-owner (say, someone in the query editor) call the model, you grant on the model itself, not the generated function, GRANT EXECUTE ON MODEL &lt;name&gt; TO &lt;role&gt;, since granting on the function is refused.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Aiprw5a55isSmRhASuDPA.jpeg" /><figcaption>Inference results in Amazon Redshift Query Editor V2.</figcaption></figure><h3>The Final Shape</h3><p>Once that permission deployed to the cluster’s role, the last barrier was gone. I re-ran the same CREATE MODEL, and this time the SageMaker training job attached its ENIs, reached the cluster and S3 over the VPC, and ran through to completion. The catalog view that had returned zero rows at the very start now reported Model is Ready, and the SageMaker job history showed the training run as Completed. The generated function was there too, ready to score rows directly in SQL:</p><pre>select id, rml_test_predict(feature_1, feature_2, feature_3) as prediction<br>from public.some_scoring_table;</pre><p>As for whether it is any good: the F1 that SHOW MODEL reported is the number to read here, and it is worth being honest about it even when it is mediocre. The point of this exercise was the pipeline, not the leaderboard, but a guide that stops at “Model is Ready” without ever looking at the score has proved the plumbing and skipped the machine learning.</p><p>The scored rows stay in the warehouse. The prediction is a scalar SQL function, so the results are just query output; to keep them you materialize a normal Redshift table (CREATE TABLE … AS SELECT … predict(…)), which lives on Redshift-managed storage, not an external file. No endpoint, no per-prediction SageMaker charge, no data leaving the warehouse. The training was the only billed event, a single small job capped by MAX_RUNTIME, and everything after it runs inside Redshift.</p><p>Then leave the account as you found it. Dropping the endpoint was never the risk here, since there is no endpoint, but CREATE MODEL does leave residue: the model and its function, the throwaway training table, and the training set plus model artifacts it unloaded into the S3 bucket, which nothing deletes for you.</p><pre>drop model if exists public.rml_test_model;<br>drop table if exists public.rml_test_train;</pre><pre>aws s3 rm s3://sagemaker-eu-west-1-111122223333/redshift-ml/ --recursive</pre><p>I do not see this as a replacement for Amazon SageMaker AI, and neither should you.</p><p>I tried this out of curiosity more than need. Redshift ML shows up in every AWS certification prep guide, usually as a one-line fact to memorize, that you can train a model with CREATE MODEL, and after reading that line for the third time I wanted to know what actually happens when you run it, and how the SageMaker machinery I already knew looks from the warehouse side.</p><p>The answer turned out to be more instructive than the exam materials suggest: almost none of the difficulty was in the machine learning part. The hard parts were, as always on AWS, the wiring: IAM trust between two services assuming one role, and a VPC permission no getting-started guide mentions.</p><p>Code for the article is available on <a href="https://github.com/deburky/aws-samples/tree/main/redshift-ml">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f02852f4da47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-Driven Data Pipelines on AWS]]></title>
            <link>https://aws.plainenglish.io/event-driven-data-pipelines-on-aws-c1a387e20743?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1a387e20743</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 06:51:24 GMT</pubDate>
            <atom:updated>2026-07-06T06:51:24.382Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build automated data ingestion pipelines with Lambda, dlt, and Iceberg</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tXQU2IbwZF7L4nEtKNrrA.jpeg" /><figcaption>Event-Driven Data Pipelines on AWS.</figcaption></figure><p>Most data pipelines start the same way: files land in a bucket, and someone<br>writes a nightly cron job to process them. It works until the schema drifts, a<br>file lands late, or the volume grows past what a single scheduled run can chew through in its window. Then you spend your mornings rerunning jobs by hand.</p><p>We had a bucket filling up with JSON, thousands of small files, each one a nested record with fields that changed over time as upstream teams added attributes. We wanted two things: the data normalized into flat, queryable tables that could evolve as the JSON evolved, and the whole thing to react on its own when new files arrived, without a schedule to babysit.</p><p>This is the event-driven version of that pipeline: S3 Inventory to catch what<br>landed, an event to kick things off, <a href="https://dlthub.com/product/dlt">dlt</a> to normalize the JSON into an<br>evolving schema, and an S3 Table as the queryable result. This article is the<br>design we settled on and the gotchas we hit along the way.</p><h4>Why event-driven, and why not just a cron job</h4><p>A scheduled job assumes you know when the data arrives and how much of it there will be. Neither is usually true. Files arrive when the upstream system decides to send them, and the volume on any given day is whatever it is. A cron job that runs at 2 a.m. either fires when there is nothing to do, or misses a burst that landed at 2:15.</p><p>Reacting to the arrival of data instead of the clock removes both problems. The pipeline runs when there is work, scales to the work that showed up, and there is no window to miss. The cost of that is a bit more plumbing, which is what the rest of this article is about.</p><h4>What we’re building</h4><p>The architecture is a straight line with two small forks. Files land in a source bucket. S3 Inventory produces a manifest of what is there. The arrival of that manifest triggers an orchestrator, which runs a dlt job that normalizes the JSON and writes it to an S3 Table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vbwwj_ku994qKqEaryi6xA.png" /><figcaption>The event-driven pipeline. Two ways to catch the S3 event, one dlt pipeline, and your choice of compute to run it.</figcaption></figure><p>The two forks are the interesting part, and both are decisions you make once and forget:</p><ul><li><strong>How you catch the S3 event </strong>depends on whether you own the bucket.</li><li><strong>What compute runs dlt</strong> depends on how much data you have.</li></ul><h4><strong>The trigger</strong></h4><p>If the S3 bucket is created inside your own CloudFormation or SAM stack, wiring the trigger is trivial. You attach an S3 event notification straight to your Lambda:</p><pre>Events:<br>  InventoryLanded:<br>    Type: S3<br>    Properties:<br>      Bucket: !Ref InventoryBucket<br>      Events: s3:ObjectCreated:*<br>      Filter:<br>        S3Key:<br>          Rules:<br>            - Name: suffix<br>              Value: manifest.json</pre><p>The moment the bucket is <strong>not</strong> in your stack, this stops working. We were pointing at a bucket owned by another team, and adding an S3 notification through CloudFormation kept failing: the notification configuration wants to modify the bucket, and CloudFormation cannot modify a bucket it does not manage.</p><p>The fix is to stop trying to touch the bucket at all. Enable EventBridge on the bucket once (a one-line bucket property the owning team sets), then catch the event with an EventBridge rule in your own stack. Your stack never modifies someone else’s resource, so there is nothing to fail:</p><pre>InventoryRule:<br>  Type: AWS::Events::Rule<br>  Properties:<br>    EventPattern:<br>      source: [&quot;aws.s3&quot;]<br>      detail-type: [&quot;Object Created&quot;]<br>      detail:<br>        bucket:<br>          name: [&quot;external-inventory-bucket&quot;]<br>        object:<br>          key: [{ suffix: &quot;manifest.json&quot; }]<br>    Targets:<br>      - Arn: !GetAtt Orchestrator.Arn<br>        Id: orchestrator</pre><p>This way nothing in your stack reaches across an account boundary. The rule of thumb we ended up with: own the bucket, use a notification; someone else owns it, use EventBridge.</p><h4><strong>Why S3 Inventory instead of listing the bucket</strong></h4><p>You could skip the inventory and just react to every ObjectCreated event. For a bucket with thousands of small files arriving in bursts, that is thousands of Lambda invocations, and you lose the ability to process a batch as one unit.</p><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html">S3 Inventory</a> gives you a scheduled manifest, a single file listing every object in the bucket (or a prefix), delivered as CSV or Parquet. You react to one manifest instead of thousands of individual events, and the manifest is your work list: read it, and you know exactly which files to normalize. It is the difference between processing a batch and being paged by every file.</p><h4><strong>The dlt processor: normalizing an evolving schema</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58dudFCy6yPTpAfqFFEeoQ.png" /><figcaption><a href="https://dlthub.com/docs/intro">https://dlthub.com/docs/intro</a></figcaption></figure><p>This is where dlt earns its place. The whole reason we did not hand-write the normalization is that the JSON schema was not stable. Fields appeared, nested objects grew, and a hand-rolled parser would have broken every time.</p><p>Here is the exact transformation, and it is the reason we did not hand-write this. A source file holds a nested record:</p><pre>{<br>  &quot;application_id&quot;: 1042,<br>  &quot;applicant&quot;: {&quot;state&quot;: &quot;CA&quot;, &quot;income&quot;: 4200},<br>  &quot;flags&quot;: [&quot;kyc_passed&quot;, &quot;docs_ok&quot;],<br>  &quot;pd&quot;: 0.068<br>}</pre><p>You hand a list of these records to a dlt pipeline and call run. That is the whole normalizer, three lines:</p><pre>import dlt<br><br>pipe = dlt.pipeline(<br>    pipeline_name=&quot;applications&quot;,<br>    destination=dlt.destinations.filesystem(bucket_url=&quot;file:///tmp/out&quot;),<br>)<br>pipe.run(records, table_name=&quot;applications&quot;)</pre><p>That single run call infers the schema and promotes the nested applicant object into columns on a flat parent table:</p><pre> application_id     pd   applicant__state   applicant__income<br>           1042  0.068                 CA                4200<br>           2087  0.142                 TX                3100</pre><p>and breaks the flags list out into a child table, linked back to its parent by a generated key:</p><pre>      value    _dlt_parent_id    _dlt_list_idx<br> kyc_passed    LJQ7Io4xqY7MEQ                0<br>    docs_ok    LJQ7Io4xqY7MEQ                1<br> kyc_passed    jP7uoGFvzx_d7w                0</pre><p>No schema defined up front, no parser to maintain. And when next week’s files add a channel field, dlt adds the column instead of failing the run. That automatic unnesting and schema evolution is the whole reason it is in the pipeline; for messy upstream JSON it is worth a lot.</p><h4><strong>The Lambda handler, piece by piece</strong></h4><p>The whole Lambda is small, so it is worth walking through the real code. It has one job: an inventory manifest.json arrives, and it turns the JSON that manifest points to into rows in the S3 Table.</p><p><strong>Follow the manifest to the files.</strong> A manifest does not contain the data; it points to inventory data files, gzipped CSVs that list every object in the source bucket.</p><p>So the first step reads the manifest and follows it to the object keys:</p><pre>def _object_keys_from_manifest(bucket, manifest_key):<br>    &quot;&quot;&quot;Get objects from the manifest.json.&quot;&quot;&quot;<br>    manifest = json.loads(<br>        s3.get_object(Bucket=bucket, Key=manifest_key)[&quot;Body&quot;].read()<br>    )<br>    source_bucket = manifest[&quot;sourceBucket&quot;]<br><br>    keys = []<br>    for entry in manifest[&quot;files&quot;]: # each is a gzipped CSV of object rows<br>        raw = s3.get_object(Bucket=bucket, Key=entry[&quot;key&quot;])[&quot;Body&quot;].read()<br>        text = gzip.decompress(raw).decode()<br>        for row in csv.reader(io.StringIO(text)):<br>            keys.append(urllib.parse.unquote_plus(row[1])) # The object Key<br>    return source_bucket, keys</pre><p>The handler then reads each of those JSON objects into a list of records, which is what dlt takes.</p><p>This is the part that matters, and it is where the earlier duckdb detour is gone. dlt needs to land its normalized output somewhere to resolve the schema, but that somewhere does not have to be a database. Point dlt’s filesystem destination at the function’s /tmp as Parquet, then read it straight back with pyarrow:</p><pre>def _normalize(records):<br>    &quot;&quot;&quot;Normalize records with dlt.&quot;&quot;&quot;<br>    shutil.rmtree(&quot;/tmp/edp_out&quot;, ignore_errors=True)<br>    pipe = dlt.pipeline(<br>        pipeline_name=&quot;edp&quot;,<br>        destination=dlt.destinations.filesystem(bucket_url=&quot;file:///tmp/edp_out&quot;),<br>        dataset_name=&quot;raw&quot;,<br>        pipelines_dir=&quot;/tmp/dlt_pipelines&quot;,<br>    )<br>    pipe.run(<br>        records,<br>        table_name=TABLE,<br>        write_disposition=&quot;append&quot;,<br>        loader_file_format=&quot;parquet&quot;<br>    )<br>    files = glob.glob(f&quot;/tmp/edp_out/raw/{TABLE}/*.parquet&quot;)<br>    tbl = pq.read_table(files)<br>    return tbl.select([c for c in tbl.schema.names if not c.startswith(&quot;_dlt&quot;)])</pre><p>dlt does the unnesting and schema inference, writes Parquet, and pyarrow reads it. No One Lambda detail makes this work: Lambda’s filesystem is read-only except /tmp, so dlt’s state and output both point there:</p><pre>os.environ.setdefault(&quot;HOME&quot;, &quot;/tmp&quot;)<br>os.environ.setdefault(&quot;DLT_DATA_DIR&quot;, &quot;/tmp/.dlt&quot;)</pre><p><strong>Write to the S3 Table, evolving the schema.</strong> The flat pyarrow table goes into the managed S3 Table through its Iceberg REST catalog, a SigV4-signed endpoint.</p><p>pyiceberg uses its <strong>pyarrow</strong> S3 FileIO (so we do not bundle s3fs into our Lambda), and on an existing table we union_by_name before appending, so a batch that grew a new field evolves the table instead of failing:</p><pre>def _catalog():<br>    &quot;&quot;&quot;Connect to Iceberg Catalog.&quot;&quot;&quot;<br>    return RestCatalog(&quot;s3tables&quot;, **{<br>        &quot;uri&quot;: f&quot;https://s3tables.{REGION}.amazonaws.com/iceberg&quot;,<br>        &quot;warehouse&quot;: WAREHOUSE,<br>        &quot;rest.sigv4-enabled&quot;: &quot;true&quot;,<br>        &quot;rest.signing-name&quot;: &quot;s3tables&quot;,<br>        &quot;rest.signing-region&quot;: REGION,<br>        &quot;py-io-impl&quot;: &quot;pyiceberg.io.pyarrow.PyArrowFileIO&quot;,  # no s3fs<br>    })<br><br># ... in the handler:<br>cat = _catalog()<br>ident = (NAMESPACE, TABLE)<br>if cat.table_exists(ident):<br>    table = cat.load_table(ident)<br>    with table.update_schema() as update:<br>        update.union_by_name(tbl.schema)  # evolve for any new fields<br>else:<br>    table = cat.create_table(ident, schema=tbl.schema)<br>table.append(tbl)</pre><p>That is the whole processor: read the manifest, read the JSON, normalize, append. The result is queryable from Athena the moment it lands.</p><h4><strong>Pick your compute: the same dlt job, three runtimes</strong></h4><p>Here is the design decision that keeps this pipeline from boxing you in. dlt is just Python. The normalization logic does not care what runs it, so we kept the dlt code in one module and made the runtime a thin wrapper around it. That means you choose the compute by data size, not by rewriting the pipeline:</p><ul><li><strong>Lambda</strong>: small, frequent batches. Cheapest, fastest to start, no servers. Fits inside the 15-minute limit.</li><li><strong>Fargate</strong>: medium batches. No time limit, a container you already built, still nothing to manage.</li><li><strong>Glue</strong>: large or Spark-scale batches, or when you want native Glue Data Catalog integration.</li></ul><p>Lambda is a good start when batches were small. The one thing worth noting is packaging. dlt, pyiceberg, and pyarrow together sit right against Lambda’s 250 MB unzipped limit, and our first build came out at 252 MB, just over. Two moves keep it in a zip and off a container. Let pyiceberg use its <strong>pyarrow S3 FileIO</strong> instead of s3fs, which drops botocore and aiobotocore, and let Lambda provide <strong>boto3</strong> from the runtime instead of bundling it. That lands the package around <strong>216 MB</strong>, and AWS SAM’s python-uv build method installs it fast and reproducibly.</p><p>When a batch stops fitting inside Lambda’s <strong>15-minute timeout</strong>, the same dlt module moves to Fargate or Glue with no code change, because the pipeline logic never knew what was running it.</p><h4><strong>Deploying it with SAM and uv</strong></h4><p>The whole thing is one SAM stack: the source bucket with its inventory configuration and the manifest notification, the S3 Table, and the function. The function is a plain zip, not a container, built with SAM’s python-uv build method so the dependencies resolve and install with uv:</p><pre>NormalizeFunction:<br>  Type: AWS::Serverless::Function<br>  Metadata:<br>    BuildMethod: python-uv<br>  Properties:<br>    CodeUri: lambda/<br>    Handler: handler.handler<br>    Runtime: python3.12<br>    Architectures: [arm64]<br>    Events:<br>      InventoryManifest:<br>        Type: S3<br>        Properties:<br>          Bucket: !Ref SourceBucket<br>          Events: s3:ObjectCreated:*<br>          Filter:<br>            S3Key:<br>              Rules:<br>                - {Name: prefix, Value: inventory/}<br>                - {Name: suffix, Value: manifest.json}</pre><p>The python-uv build method is the piece that matters here. It is a SAM beta feature, so you opt in with--beta-features (or beta_features = true in samconfig.toml), and SAM installs the function’s dependencies with uv instead of pip. The function’s requirements.txt is just dlt, pyiceberg, and pyarrow .</p><p>Two commands ship it:</p><pre>sam build --beta-features  # PythonUvBuilder resolves and installs with uv<br>sam deploy --beta-features</pre><p>Because SAM owns the source bucket, it wires the manifest notification straight to the function, which is the “own the bucket” trigger from earlier so no EventBridge needed.</p><p>We ran it end to end on a real AWS account. A manifest listing 120 JSON files, whose fields drifted from three to six across the set, landed <strong>120 rows</strong> in the S3 Table in about six seconds, with the schema carrying every field the JSON had grown and nulls where earlier files did not have them yet. The scheduled daily inventory then fires the same path on its own.</p><p>You can find the code for this article in <a href="https://github.com/deburky/aws-samples">this GitHub repo</a>.</p><h4><strong>What we’d do differently</strong></h4><p>The pattern itself is not complicated: catch the event, read the manifest, normalize with dlt, write to an S3 Iceberg Table. Two things are worth deciding up front: how you handle duplicates, and how you package the function.</p><p>The first is duplicates. Event-driven pipelines deliver the same event twice eventually; S3 notifications and EventBridge are both at-least-once. dlt’s three write dispositions handle this differently:</p><ul><li><strong>append</strong> just adds the incoming rows. Simplest and lightest, but does nothing about duplicates, so a redelivered event means duplicate rows unless something downstream collapses them.</li><li><strong>replace</strong> overwrites the table with this run’s output. Since our inventory is a full snapshot, this is the clean fix: a redelivered manifest rewrites the same state, so duplicates never accumulate.</li><li><strong>merge</strong> replaces existing rows for a primary_key instead of adding new ones. The direct at-least-once fix, but heavier: to replace a parent and its nested rows, dlt adds a _dlt_root_id column to every child table.</li></ul><p>We shipped append: batches are small and duplicate manifests rare, so collapsing the odd redelivery at query time beats carrying merge on every write. On a busier bucket, replace is the default we would reach for.</p><p>The second is packaging. dlt, pyiceberg, and pyarrow together sit right at Lambda’s 250 MB zip limit, and our first build spilled over it. Using pyarrow’s S3 FileIO instead of s3fs and letting the runtime provide boto3 pulled it back under, which is worth knowing before you reach for a container image you do not actually need.</p><h4><strong>References</strong></h4><ul><li><a href="https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-python-uv.html">Building Python Lambda functions with uv in AWS SAM</a></li><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html">Amazon S3 Inventory</a></li><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables.html">Amazon S3 Tables</a></li><li><a href="https://dlthub.com/docs">dlt (data load tool)</a></li><li><a href="https://py.iceberg.apache.org/">PyIceberg</a></li></ul><p><em>Thousands of developers share what they’re building, learning, and discovering across our publications every month. One account connects you to our entire network of publications and communities. </em><a href="https://plainenglish.io/write-for-us"><strong><em>Explore more at plainenglish.io</em></strong><em>.</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1a387e20743" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/event-driven-data-pipelines-on-aws-c1a387e20743">Event-Driven Data Pipelines on AWS</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying Airflow on EC2: A Production Guide]]></title>
            <link>https://aws.plainenglish.io/deploying-airflow-on-ec2-a-production-guide-7b54528cc9d2?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b54528cc9d2</guid>
            <category><![CDATA[ml-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 15:31:01 GMT</pubDate>
            <atom:updated>2026-06-17T15:31:01.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to run Airflow on EC2, behind an ALB, deployed with CloudFormation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrVP8lMVCY-SBXAod0qMhw.png" /><figcaption>Deploying Airflow on EC2 may be harder than it seems.</figcaption></figure><p>We needed a scheduler for our batch ML pipelines. Something the team could check each morning to see which jobs ran, which failed, and why. MWAA was the first thing we looked at, and for a few minutes it seemed like the obvious choice until we looked at the pricing. The smallest MWAA environment (a Small instance), runs around $360 a month. For a team running a handful of nightly scoring jobs with no 24/7 on-call requirements, that number is hard to defend.</p><p>We decided to run Airflow on EC2 instead: single instance, Docker, behind an ALB, managed with CloudFormation. The setup itself is straightforward. Getting it to deploy in a production account where AWS::EarlyValidation::ResourceExistenceCheck fires before any resource is provisioned is where it got interesting.</p><p>This article is the setup guide we wished existed when we started.</p><h3>Why EC2?</h3><p>The AWS-native option for managed Airflow is MWAA. It handles the infrastructure, scaling, and upgrades for you, but that convenience comes at a price. Here’s what it actually costs per month in eu-west-1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dZoiBx8XV0cIaaQRpmylgA.png" /><figcaption>Pricing comparison between MWAA and EC2 Airflow.</figcaption></figure><p>MWAA makes sense when you need high availability, auto-scaling workers, or genuinely zero ops overhead. For a data team running nightly batch jobs with a handful of DAGs and no SLA requirements, the cost difference is difficult to justify.</p><blockquote>Prices approximate as of mid-2025 for eu-west-1. Check <a href="https://aws.amazon.com/managed-workflows-for-apache-airflow/pricing/">AWS pricing</a> for current rates.</blockquote><h3>What We’re Building</h3><p>The architecture is deliberately simple. A single EC2 instance runs Airflow in Docker, sits behind an Application Load Balancer, and pulls secrets from SSM Parameter Store. The whole stack is defined via CloudFormation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WnnzEXzh1xbEfTDgK2hETA.png" /><figcaption>Airflow EC2 deployment on AWS.</figcaption></figure><p>A custom entrypoint starts the Airflow scheduler and webserver as separate processes inside the container. There’s no Celery or Redis in the background: just EC2. The EC2 only accepts traffic from the ALB: port 8080 is never directly exposed. Access goes through a VPN, so there’s no public internet exposure at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tGoUMcVnMz299tsvQSsyA.png" /><figcaption>A high-level overview of the deployment pipeline.</figcaption></figure><p>The deploy pipeline covers six steps in order: IAM permissions first, then the platform stack (EC2 and ALB), artifact uploads, Docker build, DAG sync, and finally the job stack with Lambda and Step Functions.</p><h3>Before You Deploy: Seed SSM Parameters</h3><p>By default, Airflow auto-generates a random admin password on first start and writes it to a local file on the EC2 instance. That works, but it means you have to shell into the instance every time you need the password, which defeats the purpose of having a clean deploy pipeline.</p><p>Instead, we store the admin password and the webserver secret key in SSM Parameter Store before the first deploy. The user data script reads them at boot and passes them explicitly to airflow users create, so the password is always retrievable with a single AWS CLI call and never hidden on the instance.</p><pre>aws ssm put-parameter \<br>  --name &quot;/your-path/production/airflow-admin-password&quot; \<br>  --value &quot;YourPassword&quot; --type SecureString<br><br>SECRET_KEY=$(python3 -c &quot;import secrets; print(secrets.token_hex(32))&quot;)<br>aws ssm put-parameter \<br>  --name &quot;/your-path/production/airflow-webserver-secret-key&quot; \<br>  --value &quot;$SECRET_KEY&quot; --type SecureString</pre><h3>The Docker Image</h3><p>The Dockerfile extends the official Airflow image and swaps in a custom entrypoint:</p><pre>FROM apache/airflow:2.9.3<br><br>COPY requirements.txt /requirements.txt<br>RUN pip install --no-cache-dir -r /requirements.txt<br><br>COPY entrypoint.sh /entrypoint.sh<br>RUN chmod +x /entrypoint.sh<br><br>ENTRYPOINT [&quot;/entrypoint.sh&quot;]</pre><p>The entrypoint handles everything the container needs on boot: directory setup, database migration, admin user creation, then starts the scheduler and webserver as separate background processes.</p><pre>#!/usr/bin/env bash<br>set -euo pipefail<br><br>admin_username=&quot;${AIRFLOW_ADMIN_USERNAME:-admin}&quot;<br>admin_password=&quot;${AIRFLOW_ADMIN_PASSWORD:?AIRFLOW_ADMIN_PASSWORD is required}&quot;<br><br>mkdir -p \<br>  /opt/airflow/dags \<br>  /opt/airflow/include \<br>  /opt/airflow/logs/scheduler \<br>  /opt/airflow/plugins<br><br>airflow db migrate<br><br>airflow users create \<br>  --username &quot;${admin_username}&quot; \<br>  --password &quot;${admin_password}&quot; \<br>  --firstname Admin --lastname User \<br>  --role Admin --email admin@example.com \<br>  || airflow users reset-password \<br>    --username &quot;${admin_username}&quot; \<br>    --password &quot;${admin_password}&quot;<br><br>airflow scheduler &amp;<br>scheduler_pid=&quot;$!&quot;<br><br>airflow webserver &amp;<br>webserver_pid=&quot;$!&quot;<br><br>trap &#39;kill &quot;${scheduler_pid}&quot; &quot;${webserver_pid}&quot; 2&gt;/dev/null || true; wait || true&#39; SIGINT SIGTERM<br><br>wait -n &quot;${scheduler_pid}&quot; &quot;${webserver_pid}&quot;</pre><p>Running scheduler and webserver as separate processes rather than using airflow standalone gives clean signal handling. The trap catches SIGINT and SIGTERM from Docker and shuts both processes down gracefully. wait -n exits as soon as either process dies, which surfaces failures immediately rather than keeping a half-alive container running.</p><h3>The CloudFormation Template</h3><h4>Parameters</h4><p>Declare all parameters that reference AWS resources as Type: String, not as typed AWS parameters:</p><pre>Parameters:<br>  VpcId:<br>    Type: String  # NOT AWS::EC2::VPC::Id<br><br>  AirflowAmiId:<br>    Type: String  # NOT AWS::EC2::Image::Id — pre-resolved in Makefile<br><br>  KeyName:<br>    Type: String<br>    Default: &quot;&quot;  # no SSH key in production — use SSM Session Manager</pre><p>Also resolve the AMI ID before calling cloudformation deploy rather than using a dynamic reference in the template:</p><pre>AIRFLOW_AMI_ID=$$(aws ssm get-parameter \<br>  --name &quot;/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64&quot; \<br>  --query &quot;Parameter.Value&quot; \<br>  --output text)<br><br>aws cloudformation deploy \<br>  --parameter-overrides &quot;AirflowAmiId=$$AIRFLOW_AMI_ID&quot; ...</pre><blockquote><strong>Why plain strings?</strong> Production accounts often have AWS::EarlyValidation::ResourceExistenceCheck enabled, a hook that validates every resource reference at changeset creation, before anything is provisioned. Typed parameters trigger extra permission checks your CI user may not have. Dynamic references like {{resolve:ssm:…}} aren’t resolved yet at that point, so the hook sees a raw string and fails. Plain strings with pre-resolved values sidestep both problems.</blockquote><h3>Security Groups</h3><p>The EC2 security group only accepts traffic from the ALB. Nothing talks directly to port 8080.</p><pre>AirflowSecurityGroup:<br>  SecurityGroupIngress:<br>    - IpProtocol: tcp<br>      FromPort: 8080<br>      ToPort: 8080<br>      SourceSecurityGroupId: !Ref AlbSecurityGroup<br><br>AlbSecurityGroup:<br>  SecurityGroupIngress:<br>    - IpProtocol: tcp<br>      FromPort: 80<br>      ToPort: 80<br>      CidrIp: !Ref AirflowWebIngressCidr  # 10.0.0.0/8</pre><h3>ALB</h3><p>Set UnhealthyThresholdCount to at least 5. Airflow takes 2–3 minutes to start, and the default threshold of 2 will cause the ALB to deregister the instance before Airflow has a chance to come up.</p><pre>AirflowTargetGroup:<br>  HealthCheckPath: /health<br>  HealthCheckIntervalSeconds: 30<br>  UnhealthyThresholdCount: 5</pre><h3>EC2 User Data</h3><p>The boot script pulls both secrets from SSM and starts the Airflow container. The entrypoint handles the rest: database migration and user creation happen inside the container. The poll loop waits for the database to be ready before the container’s user creation step runs:</p><pre>#!/bin/bash<br>set -e<br>yum install -y docker &amp;&amp; systemctl start docker<br><br>AIRFLOW_ADMIN_PASSWORD=$(aws ssm get-parameter \<br>  --name &#39;/your-path/production/airflow-admin-password&#39; \<br>  --with-decryption --query Parameter.Value --output text)<br><br>AIRFLOW_WEBSERVER_SECRET_KEY=$(aws ssm get-parameter \<br>  --name &#39;/your-path/production/airflow-webserver-secret-key&#39; \<br>  --with-decryption --query Parameter.Value --output text)<br><br>docker run -d --name airflow --restart unless-stopped -p 8080:8080 \<br>  -e AIRFLOW__WEBSERVER__SECRET_KEY=&quot;$AIRFLOW_WEBSERVER_SECRET_KEY&quot; \<br>  -e AIRFLOW_ADMIN_PASSWORD=&quot;$AIRFLOW_ADMIN_PASSWORD&quot; \<br>  your-image</pre><p>Passing secrets via -e flags makes them visible in docker inspect. For a VPN-only instance with a small team this is an acceptable tradeoff, but if you need stricter isolation, mount them from a file you delete after boot.</p><h3>The Deploy Pipeline</h3><p>A few things about the deploy order and CI setup matter more than they look.</p><p><strong>Lint before deploying.</strong> Run cfn-lint on your template before every deploy. It catches misconfigured resource types, invalid parameter types, and missing required fields before CloudFormation sees the template.</p><pre>pip install cfn-lint<br>cfn-lint cloudformation/platform.yaml</pre><p><strong>Deploy the IAM stack first.</strong> If your CI user’s permissions are managed by a separate CloudFormation stack, it needs to deploy before the platform stack. Otherwise the platform deploy runs before the CI user has the permissions CloudFormation needs to create resources. We broke the circular dependency between the two stacks by computing the bucket ARN from its naming convention rather than reading it from the platform stack’s outputs.</p><p><strong>Make sure the CI user has these permissions.</strong> CloudFormation uses the caller’s credentials to create and describe resources, it doesn’t assume a service role unless you configure one. These are the permissions that aren’t obvious until a deploy fails mid-way:</p><pre>- elasticloadbalancing:DescribeTargetHealth<br>- elasticloadbalancing:ModifyTargetGroup  # needed when EC2 is replaced<br>- elasticloadbalancing:RegisterTargets<br>- elasticloadbalancing:DeregisterTargets<br>- ec2:DescribeInstances<br>- ec2:DescribeImages</pre><blockquote><strong>ModifyTargetGroup is the one to watch.</strong> You need it not just at initial creation but any time CloudFormation replaces the EC2 instance: for example when you update the AMI or rename a logical resource ID. If the deploy fails mid-rollback because of this, the stack lands in UPDATE_ROLLBACK_FAILED. Run continue-update-rollback to recover it before redeploying.</blockquote><h3>Accessing Airflow</h3><p>Once the stack is up, get the ALB URL from the stack outputs:</p><pre>aws cloudformation describe-stacks --stack-name your-stack \<br>  --query &quot;Stacks[0].Outputs[?OutputKey==&#39;AirflowAlbUrl&#39;].OutputValue&quot; \<br>  --output text</pre><p>Open that URL in a browser and you’ll land on the Airflow login page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5hMNYbDdIROrD-YQbqv4Bg.png" /></figure><p>Use admin as the username and retrieve the password from SSM:</p><pre>aws ssm get-parameter \<br>  --name &quot;/your-path/production/airflow-admin-password&quot; \<br>  --with-decryption --query Parameter.Value --output text</pre><h3>What We’d Do Differently</h3><p>The manual SSM seeding step is the biggest footgun in this setup. It’s a one-time manual action that happens outside the deploy pipeline, which means it’s easy to forget when deploying to a new environment, and there’s no obvious failure when you do. We’d replace it with a CloudFormation custom resource backed by a Lambda that generates and stores the secret on stack create, or move to AWS Secrets Manager with auto-generation. Either way, the secret exists before the instance boots.</p><p>We didn’t add HTTPS and don’t need to. Our company runs a VPN and Airflow is only accessible through it, so traffic is already encrypted at the network layer. If you need to expose Airflow without a VPN, note that ACM doesn’t issue certificates for *.elb.amazonaws.com: you’ll need a custom domain and Route53 before you can add an HTTPS listener.</p><p>On instance sizing: t3.medium with 4GB handles light workloads well. Once you get to 20+ DAGs running concurrently, the scheduler starts to feel the memory pressure. t3.large is the sensible next step. And since the instance type is a CloudFormation parameter, it’s a one-liner change:</p><pre>AirflowInstanceType:<br> Default: t3.large</pre><p>The infrastructure side of this is straightforward once you know what to look for. The problem is you only learn about the problems by hitting them in production.</p><p>If you hit a production CloudFormation failure I didn’t cover, drop it in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b54528cc9d2" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/deploying-airflow-on-ec2-a-production-guide-7b54528cc9d2">Deploying Airflow on EC2: A Production Guide</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Fine-Tuned OpenAI gpt-oss-20b to Talk to Claude Code]]></title>
            <link>https://ai.plainenglish.io/how-i-fine-tuned-openai-gpt-oss-20b-to-talk-to-claude-code-e30ce160bd75?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/e30ce160bd75</guid>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Fri, 17 Apr 2026 00:49:09 GMT</pubDate>
            <atom:updated>2026-04-17T00:49:09.200Z</atom:updated>
            <content:encoded><![CDATA[<h4>Fine-tune on Apple Silicon, quantize to 10 GB, serve locally through vLLM-mlx</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FauafEQJpMeitigKn06_MA.png" /></figure><p>When OpenAI released gpt-oss-20b, its first open-weights GPT model since GPT-2, it came with something unusual: a structured output format called Harmony, complete with named channels for reasoning and tool calls. The question was whether gpt-oss-20b trained on real Claude Code conversations could be wired back into Claude Code itself: file operations, searches, multi-step coding tasks, all running on device instead of going to the API.</p><p>This guide describes how to do exactly that on a Mac Mini M4 with 32 GB of unified memory: fine-tune gpt-oss-20b on your own Claude Code history, quantize the result to 10 GB, and connect it to Claude Code through a thin compatibility bridge. The result is a local model that handles tool calls and coding questions through the same interface you already use, with inference running entirely on device. In the appendix you will also find the Colab example of tuning with transformers API.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgHmcUsK-r_ZWQMJnjDAJA.png" /></figure><h3><strong>What You’ll Build</strong></h3><p>By the end of this guide you will have four pieces working together.</p><ol><li>The first is a locally fine-tuned version of gpt-oss-20b, trained on Harmony tool calls and a concise response style via LoRA, then quantized to 4-bit for a footprint of around 10 GB.</li><li>The second is a vllm-mlx server running on port 8080, serving OpenAI-compatible chat completions from that model.</li><li>The third is a format bridge running on port 8082, which translates Claude Code’s Anthropic-format /v1/messages requests into the OpenAI format that vllm-mlx expects, then strips Harmony tokens from the response before returning clean text.</li><li>The fourth is Claude Code itself, pointed at the bridge by setting ANTHROPIC_BASE_URL, unaware that inference is happening locally.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NZQloLyegHYnS8XRauq1aQ.png" /></figure><h3><strong>Prerequisites</strong></h3><p>This guide assumes a <strong>Mac Mini M4 with 32 GB unified memory</strong>. The Q4 base model uses roughly 12 GB of RAM during both training and inference, which should fit comfortably on 16 GB machines as well. Work from the gpt-oss-claude-code directory and install <a href="https://github.com/astral-sh/uv">uv</a> for Python environment management along with mlx-lm and vllm-mlx. If Hugging Face requires authentication for the base model, set HF_TOKEN in ../.env.</p><h3><strong>Step 1: Prepare Your Dataset</strong></h3><h4><strong>What you need</strong></h4><p><a href="https://developers.openai.com/cookbook/articles/openai-harmony">Harmony</a> is gpt-oss’s structured chat format. It uses special tokens to mark where turns start (&lt;|start|&gt;), separate roles from content (&lt;|message|&gt;), and route output into named channels. The analysis channel carries chain-of-thought reasoning. The final channel carries the user-visible answer. Stop tokens (&lt;|end|&gt; and &lt;|return|&gt;) close each turn. Understanding this structure determines how to prepare training examples.</p><p>Training expects a JSONL file where each line contains a messages array. The last message must come from assistant and carry two fields: thinking, which maps to the analysis channel, and content, which maps to the final channel. The format makes the channel assignment explicit at the data level:</p><pre>{<br>  &quot;messages&quot;: [<br>    {&quot;role&quot;: &quot;developer&quot;, &quot;content&quot;: &quot;You are a helpful assistant...&quot;},<br>    {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;What is a closure?&quot;},<br>    {<br>      &quot;role&quot;: &quot;assistant&quot;,<br>      &quot;thinking&quot;: &quot;CS concept. Explain with concrete example.&quot;,<br>      &quot;content&quot;: &quot;A closure is a function that remembers variables...&quot;<br>    }<br>  ]<br>}</pre><p>If you feed raw messages to mlx_lm directly, it applies the chat template again on top of Harmony tokens and corrupts them silently. The prepare script renders the template exactly once via tokenizer.apply_chat_template() and saves the result as {“text”: “…”}, so training sees the exact token sequence the model should imitate for Claude Code interactions.</p><p>The training data teaches two things in parallel: a direct, concise response style without bullet lists or unnecessary hedging, and structured tool calls using &lt;tool_call&gt; JSON blocks for read, write, grep, and bash operations. When used through Claude Code via the bridge, Claude Code handles its own tool-calling layer. The &lt;tool_call&gt; format becomes the primary interface when using the minicode agent described later, which parses these calls and executes them in an agentic loop.</p><h4><strong>Mining your own Claude Code conversations</strong></h4><p>If you use Claude Code regularly, you already have authentic training data. Claude Code stores full conversation history, including tool_use and tool_result blocks in Anthropic API format, as JSONL files under ~/.claude/projects/. The extract_training_data.py script reads all of these files, reconstructs conversation threads by following parentUuid links, filters for threads containing at least one real tool call (Bash, Read, Write, Edit, Glob, or Grep), and slices long threads into overlapping windows of up to 12 turns. Tool results are truncated at 800 characters.</p><p>Most meaningful tool output (file reads, search results, error messages) fits well within that limit; the main thing lost is the tail of very long file reads, which rarely contains the information that drives the next action anyway. The output is Anthropic-format messages ready for prepare_dataset.py.</p><pre>make extract   # writes custom-data/extracted.jsonl</pre><p>Real interactions in our training data capture multi-turn patterns, error recovery, and the natural rhythm of tool use in ways that are difficult to hand-craft. Up to 500 examples are sampled, deduplicated by first user message, and shuffled. The extracted dataset used for this project is published at <a href="https://huggingface.co/datasets/deburky/gpt-oss-claude-code">deburky/gpt-oss-claude-code</a> on Hugging Face if you want a starting point before mining your own history.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyMDHWKXpNVn9f4CrcVUDA.png" /></figure><h4><strong>Run it</strong></h4><pre>make prepare</pre><p>Or with a custom input path:</p><pre>uv run python scripts/prepare_dataset.py - input custom-data/my_examples.jsonl</pre><p><strong>Output</strong>: data/train.jsonl and data/valid.jsonl at roughly an 80/20 split.</p><h3><strong>Step 2: LoRA Fine-Tune on Q4</strong></h3><h4><strong>Training directly on Q4</strong></h4><p>The base model is mlx-community/gpt-oss-20b-MXFP4-Q4, around 11 GB on disk and 14 GB peak memory during training. Training LoRA adapters directly on the Q4 model works well in practice. The quantization noise is modest enough that the adapters, which remain in full precision, compensate for it without requiring a larger Q8 base (which we also tested but got worse results). This keeps memory usage low across the entire pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BkAeWoiJA783JpVwCCW9PA.png" /></figure><h4><strong>Run it</strong></h4><pre>make finetune</pre><p>This runs LoRA fine-tuning with the following configuration:</p><pre>model: mlx-community/gpt-oss-20b-MXFP4-Q4<br>train: true<br>data: ./data<br>adapter_path: adapters<br><br>seed: 0<br>batch_size: 1<br>iters: 200<br>learning_rate: 1e-05<br>weight_decay: 0.01<br>steps_per_report: 10<br>steps_per_eval: 50<br>save_every: 50<br>val_batches: 25<br>max_seq_length: 2048<br>grad_checkpoint: true<br><br>lora_parameters:<br>  rank: 8<br>  scale: 20.0<br>  dropout: 0.1<br>  keys:  # Add all tunable layers <br>    - &quot;self_attn.q_proj&quot;<br>    - &quot;self_attn.k_proj&quot;<br>    - &quot;self_attn.v_proj&quot;<br>    - &quot;self_attn.o_proj&quot;<br>    - &quot;mlp.router&quot;<br>    - &quot;mlp.experts.gate_proj&quot;<br>    - &quot;mlp.experts.up_proj&quot;<br>    - &quot;mlp.experts.down_proj&quot;</pre><p>The LoRA keys cover both attention projections and the MoE routing and expert layers, giving the adapters influence over both attention patterns and token routing. Gradient checkpointing lowers peak RAM during the backward pass at the cost of slightly slower training.</p><p>Watch validation loss as training progresses. In practice the curve bottoms out around iteration 150, so it is worth picking a checkpoint rather than defaulting to the last one:</p><pre>make fuse ADAPTER_ITER=0000150</pre><p>For real style transfer away from the base model’s default voice, plan for 200 or more examples and five to ten epochs.</p><p><strong>Output</strong>: LoRA adapters in adapters/ as 0000050_adapters.safetensors, 0000100_adapters.safetensors, and so on.</p><h3><strong>Step 3: Fuse Adapters, Then Quantize to Q4</strong></h3><h4><strong>Why this order matters</strong></h4><p>Fusing LoRA directly into a quantized model stacks quantization error: the weights are already lossy, and quantizing again compounds that loss. The correct order is to first merge the adapters into clean bf16 weights and then apply fresh quantization to the result.</p><p>Running make fuse with the --dequantize flag produces a bf16 intermediate (roughly 40 GB for a 20B parameter model). Running make quantize afterward applies 4-bit quantization to those clean weights rather than to already-quantized ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6POi_5NXPO33p3HqPtz8vw.png" /></figure><h4><strong>Run it</strong></h4><pre>make fuse<br>make quantize</pre><p>The quantize step uses --q-bits 4 for 4-bit weights and --q-group-size 64, matching the base model&#39;s native quantization. Each group of 64 weights shares one scale factor, tracking local weight distributions more accurately than a larger group would. Output is at approximately 11 GB.</p><h3><strong>Step 4: Serve MLX and the Bridge</strong></h3><h4><strong>What the bridge does</strong></h4><p>vllm-mlx speaks the OpenAI chat completions format at /v1/chat/completions. Claude Code speaks the Anthropic messages format at /v1/messages. The bridge in scripts/bridge.py is a few hundred lines of standard library Python. It listens on port 8082 for Anthropic-format requests, translates them to the OpenAI format and forwards them to MLX on port 8080, waits for the response, extracts the content from Harmony’s final channel by splitting on &lt;|channel|&gt;final&lt;|message|&gt; and trimming stop tokens, and returns clean JSON to Claude Code.</p><p>If the bridge returns raw Harmony tokens, check that strip_cot() in bridge.py is correctly extracting the final channel. This usually means the model’s output structure changed or stop tokens are not being caught.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wXs3GLbTJh3uZMYrTwtT4A.png" /></figure><h4><strong>Run it</strong></h4><pre>make serve</pre><p>This starts both processes. Override ports if needed:</p><pre>make serve MODEL=model-finetuned-q4 MLX_PORT=8080 PROXY_PORT=8082</pre><p>Smoke tests:</p><pre>make curl-test # non-streaming /v1/messages<br>make curl-stream # streaming /v1/messages<br>make test-model # hit MLX directly</pre><h3><strong>Step 5: Point Claude Code at Your Model</strong></h3><pre>make claude</pre><p>Under the hood this runs:</p><pre>ANTHROPIC_BASE_URL=http://localhost:8082 \<br>ANTHROPIC_API_KEY=local \<br>claude</pre><p>The dummy API key satisfies the client library; the bridge ignores it. From this point Claude Code uses your local model for every request. Tool calls go through Claude Code’s own tool-calling layer, which does not understand Harmony syntax, so the model’s native Harmony tool-call training is masked here. The bridge strips the analysis channel and returns only final, giving Claude Code a single clean assistant string.</p><h4><strong>Path B: minicode (Harmony-Native Agent)</strong></h4><p>The bridge approach in Steps 4 and 5 translates between two formats so that Claude Code can drive the model without modification. There is a second path that skips the translation entirely.</p><p>The repo includes scripts/minicode.py, a roughly 700-line agent that speaks Harmony natively. It parses tool calls in the model’s &lt;|start|&gt;assistant to=TOOL format (read, write, bash, glob, and others) and executes them directly. The agent loop reads the model’s tool calls, executes them, feeds results back into the next turn, and repeats until the model emits a plain text response with no further tool calls.</p><p>This path is architecturally closer to what the model actually learned. Fine-tuning trained it to emit Harmony tool syntax; the bridge approach masks that and substitutes Claude Code’s own tool-calling layer. If you are building a custom agent, evaluating what the fine-tuning actually produced, or want to understand Harmony tool calls directly, minicode is the more informative starting point.</p><pre>make minicode</pre><h3><strong>What This Taught Me</strong></h3><ol><li>Harmony’s channel structure is fragile in a specific way: double-templating corrupts it silently. The model loads, training runs, and loss decreases, but the token sequence is wrong from the start. Rendering once with apply_chat_template() and saving as raw text is the only safe approach.</li><li>The fuse-then-quantize order matters more than it might seem. Fusing LoRA into a Q4 model while keeping it quantized compounds the error in both directions. The intermediate bf16 step costs 40 GB of temporary disk space but produces a cleaner quantized result.</li><li>The concern about quantization noise degrading LoRA training on a Q4 base turns out to be mostly theoretical. Adapters stay in full precision throughout and absorb the error without needing a larger Q8 starting point. This is consistent with LoftQ (<a href="https://medium.com/arxiv.org/abs/2310.08659">Li et al., 2023</a>) and Q-BLoRA (<a href="https://arxiv.org/html/2407.17029">Shen et al., 2025</a>), which both found that quantization-aware adapter training outperforms the alternative of training at higher precision and requantizing afterward.</li><li>Data preparation was key. Real conversations contain multi-turn patterns, natural error recovery, and a rhythm of tool use that is hard to fabricate synthetically. But raw logs need curation. We removed or stripped examples where the model looped on failed tool calls and streamlined multi-tool exchanges into focused single-tool examples to teach clean call-then-summarize patterns.</li></ol><p><strong>Models</strong>: The fine-tuned Q4 MLX model for Apple Silicon is at <a href="https://huggingface.co/deburky/gpt-oss-claude-mlx">deburky/gpt-oss-claude-mlx</a> (~10 GB). The merged bf16 transformers model for use on CUDA or via bitsandbytes is at <a href="https://huggingface.co/deburky/gpt-oss-claude-code">deburky/gpt-oss-claude-code</a> (~39 GB, load in 4-bit for practical inference).</p><p><strong>Data</strong>: Training data extracted from Claude Code conversations is at <a href="https://huggingface.co/datasets/deburky/gpt-oss-claude-code">deburky/gpt-oss-claude-code</a>.</p><p>Repo: <a href="https://github.com/deburky/language-models/tree/main/gpt-oss-claude-code">GitHub</a>.</p><ul><li><a href="https://huggingface.co/deburky/gpt-oss-claude-mlx">deburky/gpt-oss-claude-mlx · Hugging Face</a></li><li><a href="https://huggingface.co/datasets/deburky/gpt-oss-claude-code">deburky/gpt-oss-claude-code · Datasets at Hugging Face</a></li></ul><h3><strong>Appendix: Fine-Tuning gpt-oss on Colab</strong></h3><p>If you do not have a Mac Mini M4, the same fine-tuning can be done on Colab Pro (~$10 per month) with a G4 or H100 GPU with 80 GB VRAM. The pipeline differs from the local path in that it uses transformers and PEFT rather than mlx_lm, but the result is the same: a LoRA-adapted gpt-oss-20b available for inference.</p><p>The notebook (colab/colab-finetune.ipynb) handles the full pipeline: load the base model, train LoRA adapters, merge them, and push to Hub.</p><p>The OpenAI guide below gives a great overview of fine-tuning gpt-oss with transformers API.</p><p><a href="https://developers.openai.com/cookbook/articles/gpt-oss/fine-tune-transfomers">Fine-tuning with gpt-oss and Hugging Face Transformers</a></p><h4><strong>Loading the model on Colab</strong></h4><p>The base model (openai/gpt-oss-20b) ships with Mxfp4Config quantization built into its config. This is a custom format that transformers supports but cannot save back.</p><pre>from transformers import AutoModelForCausalLM, Mxfp4Config<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    &quot;openai/gpt-oss-20b&quot;,<br>    attn_implementation=&quot;eager&quot;,<br>    torch_dtype=torch.bfloat16,<br>    quantization_config=Mxfp4Config(dequantize=True),<br>    use_cache=False,<br>    device_map=&quot;auto&quot;,<br>)</pre><h4><strong>LoRA via PEFT</strong></h4><p>Use target_parameters to explicitly include the MoE expert layers, which all-linear alone does not reach:</p><pre>from peft import LoraConfig, get_peft_model<br><br>peft_config = LoraConfig(<br>    r=8,<br>    lora_alpha=16,<br>    target_modules=&quot;all-linear&quot;,<br>    target_parameters=[<br>        &quot;7.mlp.experts.gate_up_proj&quot;,  &quot;7.mlp.experts.down_proj&quot;,<br>        &quot;15.mlp.experts.gate_up_proj&quot;, &quot;15.mlp.experts.down_proj&quot;,<br>        &quot;23.mlp.experts.gate_up_proj&quot;, &quot;23.mlp.experts.down_proj&quot;,<br>    ],<br>)<br>peft_model = get_peft_model(model, peft_config)</pre><p>Three evenly spaced expert layers are targeted to keep VRAM within Colab limits; the MLX path targets all layers.</p><p>Train with TRL’s SFTTrainer on your {“text”: “…”} JSONL as usual.</p><h4><strong>Merging and pushing to Hub</strong></h4><p>After training, save_pretrained will raise NotImplementedError: reverse_op if Mxfp4Config is still attached to the model config. The fix is to merge the adapters onto CPU and strip the quantization config before saving:</p><pre>import gc<br>import json<br>import os<br><br>from huggingface_hub import HfApi<br>from safetensors.torch import save_file<br><br>merged = peft_model.merge_and_unload()<br>merged = merged.cpu()<br><br>os.makedirs(&quot;gpt-oss-claude-code&quot;, exist_ok=True)<br>state_dict = merged.state_dict()<br><br>max_shard = 4 * 1024**3<br>shards, current, current_size = [], {}, 0<br>weight_map, total_size = {}, 0<br><br>for k, v in state_dict.items():<br>    size = v.nbytes<br>    if current_size + size &gt; max_shard and current:<br>        shards.append(current)<br>        current, current_size = {}, 0<br>    current[k] = v.contiguous()<br>    current_size += size<br>    total_size += size<br>if current:<br>    shards.append(current)<br><br>for i, shard in enumerate(shards):<br>    fname = f&quot;model-{i+1:05d}-of-{len(shards):05d}.safetensors&quot;<br>    save_file(shard, f&quot;gpt-oss-claude-code/{fname}&quot;)<br>    for k in shard:<br>        weight_map[k] = fname<br><br>with open(&quot;gpt-oss-claude-code/model.safetensors.index.json&quot;, &quot;w&quot;) as f:<br>    json.dump(<br>        {&quot;metadata&quot;: {&quot;total_size&quot;: total_size}, &quot;weight_map&quot;: weight_map},<br>        f,<br>        indent=2,<br>    )<br>config = merged.config<br>if hasattr(config, &quot;quantization_config&quot;):<br>    delattr(config, &quot;quantization_config&quot;)<br>config.save_pretrained(&quot;gpt-oss-claude-code&quot;)<br>tokenizer.save_pretrained(&quot;gpt-oss-claude-code&quot;)<br><br>del merged, state_dict<br>gc.collect()<br><br>api = HfApi()<br>api.create_repo(&quot;deburky/gpt-oss-claude-code&quot;, exist_ok=True)<br>api.upload_folder(<br>    folder_path=&quot;gpt-oss-claude-code&quot;,<br>    repo_id=&quot;deburky/gpt-oss-claude-code&quot;,<br>    repo_type=&quot;model&quot;,<br>)</pre><p>This produces a clean bf16 model of roughly 39 GB across 13 shards, loadable anywhere with standard transformers.</p><h4><strong>Running inference on the Hub model</strong></h4><p>The merged bf16 model at 39 GB sits close to the 40 GB A100 memory ceiling with almost no headroom for activations. Loading in 4-bit with bitsandbytes brings the footprint to around 10 GB and drops inference latency from over 100 seconds to one to three seconds per response:</p><pre>from transformers import BitsAndBytesConfig<br><br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,<br>    bnb_4bit_quant_type=&quot;nf4&quot;,<br>    bnb_4bit_compute_dtype=torch.bfloat16,<br>    bnb_4bit_use_double_quant=True,<br>)<br>model = AutoModelForCausalLM.from_pretrained(<br>    &quot;deburky/gpt-oss-claude-code&quot;,<br>    quantization_config=bnb_config,<br>    device_map=&quot;cuda&quot;,<br>)</pre><p>The eval notebook (colab/colab-eval-4bit.ipynb) runs knowledge Q&amp;A and tool-calling tasks against the 4-bit Hub model and saves a markdown report.</p><p><strong>A message from our Founder</strong></p><p>Hey, Sunil here. I wanted to take a moment to thank you for reading until the end and for being a part of this community. Did you know that our team run these publications as a volunteer effort to over 3.5m monthly readers? We don’t receive any funding, we do this to support the community.</p><p>If you want to show some love, please take a moment to follow me on LinkedIn, TikTok, Instagram. You can also subscribe to our weekly newsletter. And before you go, don’t forget to clap and follow the writer️!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e30ce160bd75" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/how-i-fine-tuned-openai-gpt-oss-20b-to-talk-to-claude-code-e30ce160bd75">How I Fine-Tuned OpenAI gpt-oss-20b to Talk to Claude Code</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Shot Classification with Amazon Bedrock Batch Inference]]></title>
            <link>https://aws.plainenglish.io/zero-shot-classification-with-amazon-bedrock-batch-inference-0daa976459cd?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/0daa976459cd</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[amazon-bedrock]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 09:58:51 GMT</pubDate>
            <atom:updated>2026-04-12T09:58:51.652Z</atom:updated>
            <content:encoded><![CDATA[<h4>Classify thousands of records without infrastructure at half the cost</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o98mJcOWrBkRNidcXghGcw.jpeg" /></figure><p>Amazon Bedrock offers a batch inference API that lets you classify thousands of records without managing any infrastructure. You prepare your prompts as JSONL files, upload them to S3, and Bedrock runs them against a foundation model. This article walks through building a zero-shot text classification pipeline with Claude 3 Haiku.</p><p>Because each record is a standard Claude Messages API payload, batch jobs can carry anything the API accepts: text, base64-encoded images, multi-turn conversations, system prompts, or tool definitions. This makes it useful for workloads like embedding generation, entity extraction, LLM-as-judge evaluations, and text categorization. Batch inference is priced at a <a href="https://aws.amazon.com/bedrock/pricing/">50% discount</a> compared to on-demand.</p><h3>How batch inference works</h3><p>The core idea is simple: every request you would normally send to the Claude Messages API gets written as a single line in a JSONL file. Bedrock reads each line, sends it to the model, and writes the response to an output file on S3.</p><p>Each JSONL line has two fields:</p><ul><li>recordId, a string you use to match results back to your input.</li><li>modelInput, the full Claude API request body, including the prompt, temperature, max tokens, and any other parameters.</li></ul><p>Because modelInput is a standard Messages API payload, it can carry anything the API accepts: text, base64-encoded images, multi-turn conversations, system prompts, or tool definitions. The only constraints are file size (1 GiB per shard, 5 GiB per job).</p><p>Here is what a single line looks like for our classification task:</p><pre>{<br>  &quot;recordId&quot;: &quot;0&quot;,<br>  &quot;modelInput&quot;: {<br>    &quot;anthropic_version&quot;: &quot;bedrock-2023-05-31&quot;,<br>    &quot;messages&quot;: [<br>      {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: [<br>          {<br>            &quot;type&quot;: &quot;text&quot;,<br>            &quot;text&quot;: &quot;Classify this question into exactly one category.\nCategories: ml, programming, devops, tool_use, math_science, chat\n\n...\n\nInput:\nid=0 | What is Maximum Likelihood Estimation?&quot;<br>          }<br>        ]<br>      }<br>    ],<br>    &quot;max_tokens&quot;: 64,<br>    &quot;temperature&quot;: 0.1,<br>    &quot;top_p&quot;: 0.9<br>  }<br>}</pre><h3>The pipeline</h3><p>The Python script does five things:</p><ol><li>Loads a dataset of questions from HuggingFace.</li><li>For each question, builds a Claude API request with a zero-shot classification prompt and wraps it as a {&quot;recordId&quot;, &quot;modelInput&quot;} pair.</li><li>Writes these pairs as JSONL shards and uploads them to S3. The sharding logic estimates bytes per record and splits files to stay well under the 1 GiB limit.</li><li>Calls CreateModelInvocationJob, passing the S3 input prefix, an output prefix, the model ID, and an IAM role ARN.</li><li>After the job completes, downloads the .jsonl.out files from S3 and parses the model responses into a DataFrame.</li></ol><h3>IAM setup</h3><p>Batch inference requires two sets of IAM permissions.</p><p><strong>A service role</strong> (AmazonBedrockBatchRole) that Bedrock assumes to read input from and write output to S3. Its trust policy allows bedrock.amazonaws.com to assume it, and its permissions grant s3:GetObject, s3:PutObject, and s3:ListBucket on your batch bucket.</p><p><strong>Your IAM user or role</strong> needs four things:</p><ul><li>bedrock:CreateModelInvocationJob on the foundation model ARN and on model-invocation-job/* in your account. These resources are region-specific, so if you work across regions (e.g. us-east-1 and eu-central-1), list both.</li><li>bedrock:GetModelInvocationJob and bedrock:ListModelInvocationJobs to poll job status.</li><li>iam:PassRole on the service role ARN, conditioned on iam:PassedToService: bedrock.amazonaws.com.</li><li>S3 read/write on your batch bucket.</li></ul><p>A common gotcha: on-demand model access (InvokeModel) and batch inference (CreateModelInvocationJob) are separate account-level approvals. If you request Bedrock access through the AWS console or a support case, make sure the approval covers batch inference explicitly. You will know it is missing if you get a ValidationException: Your account is not authorized error when submitting a job, even though single InvokeModel calls work fine.</p><h3>The classification task</h3><p>We classify 284 questions from a coding assistant dataset into six categories: ml, programming, devops, tool_use, math_science, and chat. The prompt asks Claude to return only the category name, nothing else. With temperature=0.1 and max_tokens=64, responses are short and deterministic.</p><p>The dataset is publicly available on HuggingFace:</p><pre>import re<br>from datasets import load_dataset<br><br>ds = load_dataset(&quot;deburky/gpt-oss-claude-code&quot;)<br><br>def extract_q(text):<br>    user = re.search(r&quot;user&lt;\|message\|&gt;(.*?)&lt;\|end\|&gt;&quot;, text, re.DOTALL)<br>    return user.group(1).strip() if user else &quot;&quot;<br><br>rows = []<br>for i, row in enumerate(ds[&quot;train&quot;]):<br>    q = extract_q(row[&quot;text&quot;])<br>    if q:<br>        rows.append({&quot;id&quot;: str(i), &quot;text&quot;: q[:500]})<br><br>df = pd.DataFrame(rows)<br>print(f&quot;{len(df)} questions loaded&quot;)</pre><p>The labeled dataset (284 examples with ground-truth labels) allows us to evaluate accuracy after the batch job completes by joining on recordId.</p><pre>&quot;&quot;&quot;Parse Bedrock batch output shards into a labeled DataFrame.&quot;&quot;&quot;<br><br>import json<br>import boto3<br>import pandas as pd<br><br>BUCKET = &quot;your-batch-bucket&quot;<br>REGION = &quot;eu-central-1&quot;<br>OUTPUT_PREFIX = &quot;zero-shot/output/&quot;<br><br>s3 = boto3.client(&quot;s3&quot;, region_name=REGION)<br>paginator = s3.get_paginator(&quot;list_objects_v2&quot;)<br><br>rows = []<br>for page in paginator.paginate(Bucket=BUCKET, Prefix=OUTPUT_PREFIX):<br>    for obj in page.get(&quot;Contents&quot;, []):<br>        if not obj[&quot;Key&quot;].endswith(&quot;.jsonl.out&quot;):<br>            continue<br>        body = s3.get_object(Bucket=BUCKET, Key=obj[&quot;Key&quot;])[&quot;Body&quot;]<br>        for line in body.iter_lines():<br>            if not line:<br>                continue<br>            rec = json.loads(line)<br>            content = rec.get(&quot;modelOutput&quot;, {}).get(&quot;content&quot;, [])<br>            text = &quot;&quot;.join(p.get(&quot;text&quot;, &quot;&quot;) for p in content).strip()<br>            rows.append({<br>                &quot;recordId&quot;: rec[&quot;recordId&quot;],<br>                &quot;predicted&quot;: text,<br>            })<br><br>results = pd.DataFrame(rows).drop_duplicates(subset=[&quot;recordId&quot;], keep=&quot;first&quot;)<br>print(f&quot;{len(results)} predictions parsed&quot;)<br>print(results.head())</pre><p>After the shards are uploaded to S3, submit the job. You can do this programmatically with create_model_invocation_job, passing the S3 input and output prefixes, the model ID, and the service role ARN:</p><pre>import boto3<br><br>BUCKET = &quot;your-batch-bucket&quot;<br>REGION = &quot;eu-central-1&quot;<br>MODEL_ID = &quot;anthropic.claude-3-haiku-20240307-v1:0&quot;<br>ROLE_ARN = &quot;arn:aws:iam::your-account-id:role/AmazonBedrockBatchRole&quot;<br><br>bedrock = boto3.client(&quot;bedrock&quot;, region_name=REGION)<br><br>resp = bedrock.create_model_invocation_job(<br>    jobName=&quot;batch-inference&quot;,<br>    roleArn=ROLE_ARN,<br>    modelId=MODEL_ID,<br>    inputDataConfig={<br>        &quot;s3InputDataConfig&quot;: {<br>            &quot;s3Uri&quot;: f&quot;s3://{BUCKET}/zero-shot/input/&quot;,<br>            &quot;s3InputFormat&quot;: &quot;JSONL&quot;,<br>        }<br>    },<br>    outputDataConfig={<br>        &quot;s3OutputDataConfig&quot;: {<br>            &quot;s3Uri&quot;: f&quot;s3://{BUCKET}/zero-shot/output/&quot;,<br>        }<br>    },<br>)<br><br>job_arn = resp[&quot;jobArn&quot;]<br>print(&quot;Job ARN:&quot;, job_arn)</pre><h3>Running from console</h3><p>If you prefer clicking over scripting, you can submit the job directly from the AWS console. Go to <strong>Amazon Bedrock → Batch inference → Create batch inference job</strong>.</p><p>Give it a name, select Claude 3 Haiku as the model, pick <strong>InvokeModel</strong> as the invocation type, and point the input and output fields to your S3 prefixes. Hit create and wait — Bedrock will process every JSONL line and write the results back to the output location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*70T_Ks_s08w1QIsSrTR4Ow.png" /></figure><p>The Python script for batch inference job can be found on <a href="https://github.com/deburky/language-models/tree/main/bedrock-batch-inference">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0daa976459cd" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/zero-shot-classification-with-amazon-bedrock-batch-inference-0daa976459cd">Zero-Shot Classification with Amazon Bedrock Batch Inference</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Python Typing Looks Fragmented (And the Mental Model That Makes It Click)]]></title>
            <link>https://deburky.medium.com/why-python-typing-looks-fragmented-and-the-mental-model-that-makes-it-click-b99b15a7b935?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/b99b15a7b935</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Sat, 11 Apr 2026 10:39:19 GMT</pubDate>
            <atom:updated>2026-04-11T10:39:19.444Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Python’s typing feels strange, how it fits together, and how to reason about it without memorizing syntax</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZI2yhedN298ZV3GSfvIZ5g.png" /></figure><p>Python’s type system often feels fragmented. TypeVar, Protocol, TypedDict, Literal, Final, TYPE_CHECKING, from __future__ import annotations. Individually, each feature makes sense. Taken together, they can feel like a pile of unrelated ideas.</p><p>But they aren’t.</p><p>Python typing is a deliberately non-intrusive layer designed to <em>describe</em> Python, not to control it. Once you see that, everything clicks.</p><p>For a long time, it wasn’t even clear whether Python could be typed at all. That uncertainty eventually led Guido van Rossum to develop <a href="https://www.infoq.com/news/2014/08/python-type-annotation-proposal/">mypy</a>.</p><h3><strong>Python Typing Is Descriptive, Not Enforced</strong></h3><p>The most important rule: Python’s type hints do not change runtime behavior. They exist for humans, editors, and static analyzers. Python itself mostly ignores them.</p><p>This is why Python typing looks the way it does.</p><h3><strong>Why Unions Are Not Enough</strong></h3><pre>def identity(x: int | str) -&gt; int | str:<br>    return x</pre><p>This says: input can be int or str, and output can be int or str.</p><p>But it does <em>not</em> say: if input is int, output is int; if input is str, output is str.</p><p>Unions describe possibilities, not relationships.</p><h3><strong>“Same Type Flows Through”: Generics and Type Variables</strong></h3><p>To express relationships, Python needs generics.</p><pre>def identity[T](x: T) -&gt; T:<br>    return x</pre><p>Now the type checker knows that identity(1) returns int and identity(&quot;hi&quot;) returns str. Whatever type goes in, the same type comes out.</p><p>Generics can also be constrained:</p><pre>def concat[S: (str, bytes)](x: S, y: S) -&gt; S:<br>    return x + y</pre><p>Pick one type (str or bytes) and use it consistently everywhere. Unions can&#39;t express that.</p><p>Before Python 3.12, generics had to be expressed using TypeVar.</p><h3><strong>Structural Typing with </strong><strong>Protocol</strong></h3><p>Python has always been duck-typed: “If it walks like a duck…”</p><p>Protocol lets static typing model that.</p><pre>from typing import Protocol<br><br><br>class Closable(Protocol):<br>    def close(self) -&gt; None: ...</pre><p>Anything with a .close() method satisfies this type. No inheritance required. Behavior matters, class hierarchy does not.</p><p>Protocols can’t enforce interfaces at runtime: they rely on what’s observable when the type checker runs. Typing describing Python, not controlling it.</p><h3><strong>TypedDict vs dataclass: Shape vs Behavior</strong></h3><p>People mix these up constantly.</p><p>TypedDict describes the shape of a dictionary:</p><pre>class User(TypedDict):<br>    id: int<br>    name: str</pre><p>At runtime it’s still a dict, which neither has methods nor behavior. Use it for JSON, API payloads, external data.</p><p>dataclass creates a real Python class:</p><pre>@dataclass<br>class User:<br>    id: int<br>    name: str</pre><p>Methods, invariants, behavior. Use it for internal domain models and business logic.</p><p>The mental model:</p><ul><li>External data → TypedDict,</li><li>Internal objects → dataclass.</li></ul><h3><strong>Partial Dictionaries with total=False</strong></h3><pre>class PartialUser(TypedDict, total=False):<br>    id: int<br>    name: str</pre><p>All keys become optional, values stay typed. The default (total=True) requires every key.</p><pre># total=True (default)<br>{&quot;id&quot;: 1}            # type error: missing &quot;name&quot;<br><br># total=False<br>{}                   # valid<br>{&quot;id&quot;: 1}            # valid<br>{&quot;name&quot;: &quot;Alice&quot;}    # valid</pre><p>Useful for PATCH updates, partial configs, filters. This is how dicts actually get used in real code: typing adapting to existing Python patterns rather than imposing new ones.</p><h3><strong>Literal Types: Enum-Like Without Enums</strong></h3><p>Sometimes you want specific allowed values, not just a type.</p><pre>from typing import Literal<br><br>def open_file(mode: Literal[&quot;r&quot;, &quot;w&quot;, &quot;a&quot;]) -&gt; None:<br>    ...</pre><p>&quot;r&quot; is allowed, &quot;x&quot; is rejected. No runtime object, no class, no overhead. A type-only enum.</p><h3><strong>Final: Don’t Touch This</strong></h3><p>Final means: this name must not change or be overridden.</p><pre>MAX_SIZE: Final = 1024</pre><p>Reassignment becomes a type error. It also applies to methods:</p><pre>class Base:<br>    def run(self) -&gt; None: ...<br>    run: Final</pre><p>Subclasses cannot override run. This is about API contracts, not runtime enforcement.</p><h3><strong>Why TYPE_CHECKING Exists</strong></h3><p>Some code should exist only for type checkers, not at runtime.</p><pre>from typing import TYPE_CHECKING<br><br>if TYPE_CHECKING:<br>    from heavy_module import BigType</pre><p>TYPE_CHECKING is False at runtime. Always. Type checkers treat it as True. You can import heavy or circular dependencies without paying for them when the code actually runs. No magic, just a boolean and a convention.</p><h3><strong>Why __future__ import annotations</strong></h3><p>Annotations used to be evaluated immediately:</p><pre>def f(x: SomeType): ...</pre><p>If SomeType wasn&#39;t defined yet, this crashed. The future import fixes that:</p><pre>from __future__ import annotations</pre><p>Annotations are stored as strings, not evaluated. Forward references work, import cycles go away, runtime cost drops.</p><p>The __future__ mechanism isn&#39;t specific to typing. from __future__ import print_function did the same thing for the Python 2 to 3 transition: opt into tomorrow&#39;s behavior today.</p><h3><strong>Why Python Typing Looks Fragmented</strong></h3><p>Because it had to satisfy four constraints at once:</p><ol><li>No runtime enforcement</li><li>Backward compatibility</li><li>Incremental adoption</li><li>Zero performance cost</li></ol><p>You can’t build one unified system under those rules. So Python built small composable tools instead: relationships (TypeVar), shapes (TypedDict), behavior (Protocol), constants (Final), exact values (Literal). Each one solves one problem.</p><h3><strong>The Point</strong></h3><p>Python typing works because it describes Python as it is, not as some other language would prefer it to be. Once that clicks, the system stops looking ad-hoc.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b99b15a7b935" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Counting Words to Training GPT: the MLE Thread]]></title>
            <link>https://deburky.medium.com/from-counting-words-to-training-gpt-the-mle-thread-5adb91ba86a9?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/5adb91ba86a9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Wed, 08 Apr 2026 18:13:07 GMT</pubDate>
            <atom:updated>2026-04-08T18:13:07.471Z</atom:updated>
            <content:encoded><![CDATA[<h4>How a 250-year-old idea powers everything from extractive summarizers to large language models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*J5TfYYIW1HCeechBmKYTkQ.png" /></figure><p>In 1922, Ronald Fisher introduced a principle he called maximum likelihood estimation (MLE). The idea was older: Laplace had used it in 1774, Gauss had applied it to least squares in 1809. But Fisher formalized the theory, proved consistency and efficiency, and gave it a name. A century later, the same principle powers both a simple word-frequency summarizer and foundation language models like ChatGPT or Claude. The statistical machinery underneath both is identical.</p><p>MLE is the quiet workhorse of modern NLP. Understanding it properly, from counting words in a document to training billion-parameter transformers, is one of the most clarifying exercises you can do as a practitioner. This article walks through that arc, with working code at every step.</p><h3><strong>The Summarizer That Looks Like Black Magic</strong></h3><p>Here is a spaCy-based extractive summarizer, stripped to its essentials:</p><pre>from collections import Counter<br>from heapq import nlargest<br>from string import punctuation<br><br>import spacy<br>from spacy.lang.en.stop_words import STOP_WORDS<br><br>def summarize_text(text, nlp, summary_sentences=3, top_keywords=5):<br>    doc = nlp(text)<br><br>    # Step 1: Extract keywords (nouns, verbs, adjectives, proper nouns)<br>    pos_tags = {&quot;PROPN&quot;, &quot;ADJ&quot;, &quot;NOUN&quot;, &quot;VERB&quot;}<br>    stopwords = set(STOP_WORDS)<br>    keywords = [<br>        token.text for token in doc<br>        if token.text not in stopwords<br>        and token.text not in punctuation<br>        and token.pos_ in pos_tags<br>    ]<br><br>    # Step 2: Count keyword frequencies<br>    freq = Counter(keywords)<br>    max_freq = freq.most_common(1)[0][1]<br><br>    # Step 3: Normalize frequencies to [0, 1]<br>    for word in freq:<br>        freq[word] = freq[word] / max_freq<br><br>    # Step 4: Score each sentence by summing normalized keyword frequencies<br>    sent_scores = {}<br>    for sent in doc.sents:<br>        for word in sent:<br>            if word.text in freq:<br>                sent_scores[sent] = sent_scores.get(sent, 0) + freq[word.text]<br><br>    # Step 5: Select top-scoring sentences<br>    top_sents = nlargest(summary_sentences, sent_scores, key=sent_scores.get)<br>    return &quot; &quot;.join(s.text.strip() for s in top_sents)</pre><p>When you first see this, it feels like it shouldn’t work. The entire algorithm is a word counter with a ranking step. Yet on well-structured and logically reinforcing text, the results are remarkably good.</p><p>Take this short passage:</p><blockquote>The Federal Reserve raised its benchmark interest rate by a quarter point on Wednesday, the tenth increase since March 2022. Policymakers said inflation remains elevated and further rate increases may be necessary to bring it back to the 2 percent target. The labor market has remained resilient despite tighter financial conditions, with unemployment holding near historic lows. Chair Jerome Powell said the committee would proceed carefully and remains data-dependent. Inflation has slowed from its 2022 peak but continues to run above levels the Fed considers acceptable. Rate-sensitive sectors like housing have already felt the impact, with mortgage rates doubling over the past two years.</blockquote><p>With summary_sentences=2, the output is:</p><blockquote>Policymakers said inflation remains elevated and further rate increases may be necessary to bring it back to the 2 percent target. The Federal Reserve raised its benchmark interest rate by a quarter point on Wednesday, the tenth increase since March 2022.</blockquote><p>The two sentences with the highest combined keyword score both contain “inflation” and “rate,” the words with the highest document frequency. The algorithm surfaced the key message through counting alone.</p><p>The reason it works is that it is performing maximum likelihood estimation, whether the author intended it or not.</p><h3><strong>MLE: The Idea Behind the Curtain</strong></h3><p>Maximum likelihood estimation answers a simple question: given observed data, what parameter values make those observations most probable?</p><p>Consider a corpus where the word “credit” appears 400 times in 1,000,000 words. The MLE estimate for the probability of “credit” is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u4TF8yOFBRZPJesWxv4AOQ.png" /></figure><p>This is not necessarily the best estimate of how likely “credit” is to appear in all possible texts. In a corpus of cooking recipes it would be near zero. But it is the estimate that maximizes the probability of observing exactly 400 occurrences in a million words. That is what “maximum likelihood” means: find the parameters that make the training data most likely.</p><p>The summarizer does exactly that. When it counts keyword frequencies and divides by the maximum count, it is computing a scaled version of P(word | document), the MLE of each word’s probability under the document’s own unigram language model. The ranking is identical to what you would get from raw MLE probabilities, since dividing by the max is a monotonic transformation.</p><p>The sentence scoring step then asks: which sentences have the highest total probability mass under this document-level unigram model?</p><p>Sentences packed with the document’s most characteristic words float to the top. It is finding the sentences that are most <strong>representative</strong> of the document’s overall word distribution.</p><h3><strong>From Unigrams to Bigrams: Where Context Enters</strong></h3><p>The summarizer uses a unigram model: P(word). Each word is treated independently. This is a blunt instrument, because language is sequential. “Not good” and “very good” contain the same word “good” but carry opposite meanings.</p><p>The natural extension is the bigram model, which conditions each word on the previous one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mHfwQmNHCvIS6am1nPx7Rw.png" /></figure><p>Take three sentences:</p><ul><li>&lt;s&gt; I am Sam &lt;/s&gt;</li><li>&lt;s&gt; Sam I am &lt;/s&gt;</li><li>&lt;s&gt; I do not like green eggs and ham &lt;/s&gt;</li></ul><p>The bigram probabilities fall out directly from counting:</p><ul><li>P(I | &lt;s&gt;) = 2/3 = 0.67 (2 of 3 sentences start with “I”)</li><li>P(am | I) = 2/3 = 0.67 (“I am” appears twice, “I” appears 3 times)</li><li>P(do | I) = 1/3 = 0.33</li><li>P(Sam | am) = 1/2 = 0.50</li><li>P(&lt;/s&gt; | Sam) = 1/2 = 0.50</li></ul><p>Each probability is a ratio of counts: how often did this specific bigram occur, divided by how often the first word occurred at all? This ratio is called a <strong>relative frequency</strong>, and it is the MLE estimate for the bigram parameter.</p><p>The general n-gram formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sS6TFiS1M92i50d_hjcGMg.png" /></figure><p>Count the n-gram. Divide by the count of the (n-1)-gram prefix. That is MLE for n-gram models. It is the same counting-and-normalizing logic as the summarizer, just applied to sequences instead of individual words.</p><p>David MacKay, in <em>Information Theory, Inference, and Learning Algorithms</em> (2003), defines a language model as a prior probability distribution P(x) that “specifies the probability of the next character/word given the context and the known grammar and statistics of the language.”</p><p>The summarizer’s frequency table is exactly that: a monogram language model in MacKay’s sense, with zero context. The bigram model extends context by one word. LLMs extend it to the entire preceding sequence. The architecture changes dramatically at each step; the underlying goal (estimating P(next token | context) from data) does not.</p><h3><strong>The Jump to LLMs</strong></h3><p>A large language model like GPT is, at its core, an n-gram model with two radical upgrades.</p><p>First, instead of a fixed context window of 1 or 2 or 5 words, language transformers condition on the entire preceding sequence. The model estimates P(next token | all previous tokens), not P(wn | wn-1).</p><p>Second, instead of storing literal counts in a table (which would be astronomically large for full-context conditioning), the model parameterizes the probability distribution with a neural network. The transformer’s weights are the parameters, and training with cross-entropy loss is equivalent to maximizing the log-likelihood of the training corpus.</p><p>So the training objective of GPT is literally MLE. You are finding the neural network weights theta that maximize:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qG0am4O5eOsEoDzK2zZU9w.png" /></figure><p>The spaCy summarizer estimates P(word | document) with a frequency table. GPT estimates P(word | all previous words) with 175 billion parameters. Same statistical principle with a vastly different expressive power.</p><p>This is why the summarizer works on well-structured text but fails on subtle or surprising content. Its unigram model has no notion of word order, negation, or contextual meaning. It exploits a property of coherent writing: that important themes manifest as repeated vocabulary.</p><h3><strong>Beyond Summarization: spaCy for Named Entity Recognition</strong></h3><p>Extractive summarization is only one application of spaCy. When I worked at Amazon, I needed to extract brand names from product titles at scale. Millions of listings with titles like “Samsung Galaxy S24 Ultra 256GB Phantom Black” or “Nike Air Max 90 Essential Running Shoe.” The task was classification: given a product title, identify the brand.</p><p>The obvious modern approach would be to send every title through an LLM. But at millions of records, the cost was prohibitive. spaCy’s named entity recognition (NER) model, specifically its ORG entity label, provided a cheap and fast baseline:</p><pre>import re<br>from typing import Optional<br><br>import spacy<br><br>def extract_brand(text: str, nlp) -&gt; Optional[str]:<br>    if not text:<br>        return None<br><br>    doc = nlp(text)<br><br>    # Use spaCy ORG entities as brand candidates<br>    orgs = [ent.text for ent in doc.ents if ent.label_ == &quot;ORG&quot;]<br><br>    for org in orgs:<br>        if len(org) &lt; 2 or len(org) &gt; 30:<br>            continue<br>        # Skip common non-brand patterns<br>        if re.search(r&quot;\b(inc|corp|ltd|llc|company)\b&quot;, org, re.IGNORECASE):<br>            continue<br>        return org<br><br>    # Fallback: pattern matching for &quot;Brand + Product&quot; structure<br>    match = re.search(r&quot;^([A-Z][a-zA-Z0-9&amp;\&#39;\.\-\s]{1,15})\s+[A-Z]&quot;, text)<br>    if match:<br>        return match[1].strip()<br><br>    return None</pre><p>The hybrid approach, spaCy NER first with regex fallback, handled the volume at a fraction of the cost of LLM inference. It was not perfect. Some brands were missed, some non-brands were captured. But as a baseline that processes thousands of records per second on CPU, it earned its keep.</p><p>A similar pattern (spaCy ORG entities as a first-pass extractor, refined with scoring heuristics) is described in the open-source <a href="https://github.com/annis-souames/brand-ner">brand-ner</a> project by Souames and Mohammedi, whose <a href="https://hal.science/hal-03528324v1">accompanying paper</a> takes it further with a proper sequence labeling model. Worth reading if you need to push past the baseline.</p><p>The same MLE logic applies here. spaCy’s NER model was trained with MLE on annotated text, learning P(entity label | token features) from large corpora. The pre-trained en_core_web_sm model already knows that capitalized words at the start of product titles are often organizations. It learned that from counts and gradients, the same way GPT learned to complete sentences.</p><h3><strong>The Unifying View</strong></h3><p>Here is the thread that runs through all of it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIXey9qSElvKUbTgrjMgew.png" /></figure><p>Every row is the same game: observe data, estimate a conditional probability distribution, use it for prediction. The difference is in what you condition on and how many parameters you use to represent the distribution.</p><p>Laplace used the principle in 1774, Gauss applied it to least squares in 1809, and others (Thiele, Edgeworth) explored related ideas in between. None of them named it or proved its properties. Fisher did, between 1912 and 1922: he gave MLE its name, proved consistency and efficiency, and connected it to the information matrix. A century later, it remains the training objective for the most powerful language models ever built.</p><p>The summarizer counting word frequencies in a for loop and GPT processing a trillion tokens through attention layers are both doing the same thing Fisher described: finding the parameters that make the observed data most probable.</p><p>Understanding this connection is not just intellectually satisfying. When you know that a model like Claude is fitting a full-context MLE model, you understand why it needs so much data and compute, and why it hallucinates when pushed beyond its training distribution.</p><p>The code for this article can be found on <a href="https://github.com/deburky/language-models/tree/main/spacy-nlp">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5adb91ba86a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meet Kumo: A Free, No-Account Local AWS Emulator]]></title>
            <link>https://aws.plainenglish.io/meet-kumo-a-free-no-account-local-aws-emulator-4a171b917401?source=rss-993c6d118a59------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a171b917401</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Denis Burakov]]></dc:creator>
            <pubDate>Sun, 05 Apr 2026 16:06:16 GMT</pubDate>
            <atom:updated>2026-04-05T16:06:16.906Z</atom:updated>
            <content:encoded><![CDATA[<h4>How I replaced LocalStack with a 32MB Go container that needs no account, no token, and no registration</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6KWlLSp6OWCzoUqX5ck2KQ.png" /><figcaption>From LocalStack to Kumo.</figcaption></figure><p>I was using LocalStack for testing my AWS workflows locally, and for a while it worked well enough.</p><p>When I needed to go beyond the Community Edition to test things like AppConfig, ECS, and ECR, I moved to the Pro image. That turned out to be more friction than value. AppConfig did not match production variants, ECS did not provide direct container access, and ECR took a long time to set up correctly with Lambda.</p><p>SageMaker and Bedrock were behind the Ultimate tier entirely, so I still had to fall back to Local Mode in SageMaker to run experiments in a separate container.</p><p>The tool was not doing what I needed, and then its pricing model changed in a way that made the situation harder to ignore.</p><h4><strong>What Happened to LocalStack</strong></h4><p>LocalStack launched in 2016 as a fully free, Apache 2.0-licensed AWS emulator, growing to 25,000 GitHub stars by 2020 and becoming a default tool for local development and CI/CD pipelines. In 2019 it added a paid Pro tier while keeping core services free in a Community Edition, an arrangement that held for several years and built a large base of contributors and dependents.</p><p>In December 2025, LocalStack announced the end of the Community Edition. Effective March 23, 2026, all users, including those on the free Hobby tier, must now create an account and authenticate with a token just to start a container. The prior community image was archived and receives no further updates or security patches. The founders framed this as necessary to sustain the project, citing the “scope, security requirements, and operational complexity of maintaining high-fidelity AWS emulation,” and a $25 million Series A in late 2024 added context to the monetization direction.</p><p>The announcement drew significant criticism. Developers noted they had spent years contributing bug reports, code, and documentation under the assumption the tool would remain freely usable. Comments in the <a href="https://news.ycombinator.com/item?id=47493657">Hacker News thread</a> captured the mood:</p><blockquote>“…took advantage of a community that expected their freely offered labor to not be commercialized”</blockquote><blockquote>“I totally get wanting to monetize but removing the free product entirely feels like such a betrayal.”</blockquote><p>LocalStack initially proposed CI credit restrictions on free tiers, then reversed that after backlash, but the account requirement stands. Current pricing starts at $39/month on the Base tier (annual) for 55+ services, with a free Hobby tier covering 30+ services for non-commercial use, account required.</p><p>This led me to cancel my Pro license to look for alternatives, and I found Kumo ☁️.</p><h4><strong>What Is Kumo</strong></h4><p>Kumo is an open-source AWS emulator written in Go by Siva Chari.</p><p><a href="https://github.com/sivchari/kumo">GitHub - sivchari/kumo: A lightweight AWS service emulator written in Go</a></p><p>The name means “cloud” in Japanese (雲), and it was originally called awsim before being proactively renamed in March 2026 to avoid potential AWS trademark issues. It runs as a single binary or a 32MB Docker container, requires no account, no token, and no registration, and is MIT-licensed.</p><pre>docker run -p 4566:4566 ghcr.io/sivchari/kumo:latest</pre><p>That is the entire setup. Kumo listens on port 4566, the same default as LocalStack, so many existing configurations drop in without changes. At the time of writing it supports 71 AWS services across storage, compute, messaging, security, networking, and more.</p><h4><strong>What We Tested</strong></h4><p>Rather than running Kumo manually and writing tests against it separately, we used docker-py to manage the entire lifecycle from a single Python script: pull the image, start the container, wait for the port, run the tests, and stop the container on exit whether or not the tests passed.</p><p>We tested six services: S3, SQS, DynamoDB, RDS, ECR, and API Gateway. The script uses PEP 723 inline metadata so it can be run directly with uv run without any manual dependency setup.</p><p><strong>S3, SQS, and DynamoDB</strong></p><p>These three work out of the box with standard boto3 calls, pointing at http://localhost:4566 with dummy credentials. Rather than using a credentials dict, the script creates a boto3.Session upfront so the service name is passed as a proper literal and mypy stays happy across the board.</p><pre>import socket<br>import time<br><br>import boto3<br>import boto3.session<br>import docker<br><br>IMAGE = &quot;ghcr.io/sivchari/kumo:latest&quot;<br>PORT = 4566<br>ENDPOINT = f&quot;http://localhost:{PORT}&quot;<br><br>SESSION = boto3.session.Session(<br>    region_name=&quot;us-east-1&quot;,<br>    aws_access_key_id=&quot;test&quot;,<br>    aws_secret_access_key=&quot;test&quot;,<br>)<br><br>def test_s3() -&gt; None:<br>    client = SESSION.client(&quot;s3&quot;, endpoint_url=ENDPOINT)<br>    client.create_bucket(Bucket=&quot;test-bucket&quot;)<br>    client.put_object(Bucket=&quot;test-bucket&quot;, Key=&quot;hello.txt&quot;, Body=b&quot;hello kumo&quot;)<br>    body = client.get_object(Bucket=&quot;test-bucket&quot;, Key=&quot;hello.txt&quot;)[&quot;Body&quot;].read()<br>    assert body == b&quot;hello kumo&quot;<br>    print(&quot;S3 ✓&quot;)<br><br>def test_sqs() -&gt; None:<br>    client = SESSION.client(&quot;sqs&quot;, endpoint_url=ENDPOINT)<br>    q = client.create_queue(QueueName=&quot;test-queue&quot;)<br>    url = q[&quot;QueueUrl&quot;]<br>    client.send_message(QueueUrl=url, MessageBody=&quot;hello kumo&quot;)<br>    resp = client.receive_message(QueueUrl=url)<br>    assert resp.get(&quot;Messages&quot;, [])[0][&quot;Body&quot;] == &quot;hello kumo&quot;<br>    print(&quot;SQS ✓&quot;)<br><br>def test_dynamodb() -&gt; None:<br>    client = SESSION.client(&quot;dynamodb&quot;, endpoint_url=ENDPOINT)<br>    client.create_table(<br>        TableName=&quot;test-table&quot;,<br>        KeySchema=[{&quot;AttributeName&quot;: &quot;id&quot;, &quot;KeyType&quot;: &quot;HASH&quot;}],<br>        AttributeDefinitions=[{&quot;AttributeName&quot;: &quot;id&quot;, &quot;AttributeType&quot;: &quot;S&quot;}],<br>        BillingMode=&quot;PAY_PER_REQUEST&quot;,<br>    )<br>    table = SESSION.resource(&quot;dynamodb&quot;, endpoint_url=ENDPOINT).Table(&quot;test-table&quot;)<br>    table.put_item(Item={&quot;id&quot;: &quot;1&quot;, &quot;value&quot;: &quot;hello&quot;})<br>    assert table.get_item(Key={&quot;id&quot;: &quot;1&quot;})[&quot;Item&quot;][&quot;value&quot;] == &quot;hello&quot;<br>    print(&quot;DynamoDB ✓&quot;)</pre><p>For testing other services (API Gateway, ECR, RDS, STS, IAM), two small fixes were needed. RDS, STS, and IAM use the AWS Query protocol, and Kumo strict-matches the Content-Type header without the ; charset=utf-8 suffix that boto3 appends, causing a MissingTargetHeader error. API Gateway requires pointing the client at http://localhost:4566/apigateway instead of the root, since Kumo registers its routes under that prefix rather than at /restapis/ as the real AWS API does.</p><h4><strong>Reproducing a LocalStack ML Workflow with Kumo</strong></h4><p>To go beyond individual service tests, we reproduced the full ML prediction workflow from an earlier article of mine using Kumo as the emulator.</p><p><a href="https://aws.plainenglish.io/develop-aws-ml-workloads-locally-with-localstack-and-sam-24bdc0de81aa">Develop AWS ML Workloads Locally with LocalStack and SAM</a></p><p>That article deploys a containerized scikit-learn Lambda behind API Gateway with SAM local, using LocalStack for the surrounding AWS services. We replaced LocalStack with Kumo and kept everything else the same: the SAM template, the Docker-based Lambda, and the make start workflow.</p><p>The swap came down to three files. The docker-compose.yml went from the LocalStack image with its environment variables and volume mounts to a single line:</p><pre>services:<br>  kumo:<br>    image: ghcr.io/sivchari/kumo:latest<br>    ports:<br>      - &quot;4566:4566&quot;</pre><p>The sts_example.py, which creates an IAM role and assumes it via STS, needed two adjustments specific to Kumo. IAM is routed under /iam/ rather than the root path, and both IAM and STS use the Query protocol, which triggers the charset issue described earlier. Rather than applying the fix ad hoc, we extracted a reusable KumoSession class into src/kumo_compat.py:</p><pre>class KumoSession:<br>    &quot;&quot;&quot;Boto3 session wrapper pre-configured for Kumo.&quot;&quot;&quot;<br><br>    def client(self, service_name: str) -&gt; Any:<br>        &quot;&quot;&quot;Return a boto3 client with Kumo compatibility applied.&quot;&quot;&quot;<br>        prefix = _PATH_PREFIXES.get(service_name, &quot;&quot;)<br>        endpoint_url = self._base_url + prefix<br>        boto3_client = cast(Any, self._session).client(service_name, endpoint_url=endpoint_url)<br>        if service_name in _QUERY_PROTOCOL_SERVICES:<br>            boto3_client.meta.events.register(<br>                f&quot;before-send.{service_name}.*&quot;, _strip_charset<br>            )<br>        return boto3_client</pre><p>With KumoSession patch in place, everything becomes straightforward:</p><pre>session = KumoSession()<br>iam = session.client(&quot;iam&quot;)   # /iam prefix + charset hook applied automatically<br>sts = session.client(&quot;sts&quot;)   # charset hook applied automatically</pre><p>After those changes, make start ran cleanly: Kumo started, the models trained, SAM built the container image, and the prediction endpoint returned a correct response.</p><h4>Wrapping Up</h4><p>Kumo is early but functional, and for the services tested here it works well as a drop-in for local development and CI pipelines without requiring an account or any authentication.</p><p>If you were relying on LocalStack’s Community Edition and want a no-auth replacement, Kumo is worth a look. Covering 71 services in a 32MB container with a single docker run command is a strong foundation, the project is actively maintained, and the fact that the full test script can be run with just uv run test_kumo.py makes it straightforward to drop into any pipeline.</p><p>The code used in this article is available on <a href="https://github.com/deburky/aws-samples">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a171b917401" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/meet-kumo-a-free-no-account-local-aws-emulator-4a171b917401">Meet Kumo: A Free, No-Account Local AWS Emulator</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>