<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stefan Achtsnit on Medium]]></title>
        <description><![CDATA[Stories by Stefan Achtsnit on Medium]]></description>
        <link>https://medium.com/@stefan.achtsnit_41940?source=rss-3bee25fb233c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*9SxmlBd_QZU3ADzNUSC2QQ.png</url>
            <title>Stories by Stefan Achtsnit on Medium</title>
            <link>https://medium.com/@stefan.achtsnit_41940?source=rss-3bee25fb233c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:47:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@stefan.achtsnit_41940/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Act on Your Behalf: A five-minute token for a STAC export]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/act-on-your-behalf-a-five-minute-token-for-a-stac-export-955d420c1fc4?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/955d420c1fc4</guid>
            <category><![CDATA[security-token-exchange]]></category>
            <category><![CDATA[keycloak]]></category>
            <category><![CDATA[earth-observation]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Sun, 08 Nov 2026 13:35:15 GMT</pubDate>
            <atom:updated>2026-11-08T13:51:50.904Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAQ5iQAUy34vL_Jk7GnqIA.png" /></figure><p>Alice writes a short script on an Earth observation (EO) platform<br>running on Kubernetes. It reads a few records from a STAC catalogue and<br>writes a GeoParquet file to S3.</p><p>Her normal login also lets her provision managed services, connect to<br>a Kubernetes cluster or virtual cluster, and access other data in her<br>team’s private object storage. Giving an export script that full access<br>is a large grant of trust for a small task. The export needs to read the<br>catalogue and write one result.</p><p>Using a five-minute token, Alice’s script submitted a Job in the STAC<br>namespace to export three catalogue records to GeoParquet in S3. The token narrowed the script’s Kubernetes permissions, while the Job’s S3 access still depended on the workspace credentials passed through a Kubernetes Secret.</p><p>The <a href="https://github.com/EOEPCA/workspace/blob/main/docs/getting-started/stac-fastapi-pgstac.ipynb"><strong>STAC with managed PostgreSQL notebook</strong></a> follows that export from token request to checked file. The same access problem applies to a scheduled pipeline or an agent working on an EO task.</p><h3>The namespace profile Alice needs</h3><p>Alice has administrator access to ws-alice/vc0. She deploys STAC FastAPI PgSTAC, connects it to the managed catalog PostgreSQL database, and registers three NAIP Items. These are catalogue records; the setup does not copy the source imagery.</p><p>STAC is already connected to the running PostgreSQL database when the export starts, so the exporter reads the registered Items through STAC and writes the result to object storage. The namespace profile governs the Kubernetes operations that start the exporter; STAC’s database login and the S3 output credentials still have their own permissions.</p><p>To submit the export with fewer permissions, she needs a namespace access profile. The <a href="https://github.com/EOEPCA/workspace/blob/main/setup/examples.yaml"><strong>workspace declaration</strong></a> defines demo with the Kubernetes edit role in stac-fastapi-pgstac. It also binds the managed PostgreSQL connection into that namespace as managed-postgres. The full declaration is in the linked example.</p><pre>spec:<br>  users: [alice]<br>  vclusters:<br>    vc0:<br>      groups:<br>        - name: demo<br>          namespace: stac-fastapi-pgstac<br>          role: edit</pre><p><a href="https://provider-datalab.versioneer.at/"><strong>Provider Datalab</strong></a> prepares these services and access settings through Crossplane: managed databases, virtual clusters, and Keycloak clients and role mappings. <a href="https://github.com/EOEPCA/rm-workspace-api"><strong>Workspace API</strong></a> uses the published connection references to return service credentials and request token exchange. Keycloak issues the token; Kubernetes and the data services enforce the permissions they accept.</p><p>Workspace API resolves ready service connections from the Secret references published by Provider Datalab in status.connections.</p><p>Alice can inspect the PostgreSQL connection in the workspace UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*U8Rl9FVZLcth4P7g.png" /></figure><h3>Requesting the five-minute token</h3><p>The <a href="https://github.com/EOEPCA/rm-workspace-api/blob/main/docs/workspace-tokens.md"><strong>Workspace token API</strong></a> supports two flows. A request without scope returns a workspace service-account token. A request with configured profile scopes exchanges the caller&#39;s token for one that keeps the user&#39;s identity and selected runtime roles.</p><p>Alice requests the demo profile:</p><pre>curl --get \<br>  -H &#39;Authorization: Bearer &lt;user-access-token&gt;&#39; \<br>  --data-urlencode &#39;scope=ws-alice:vclusters_vc0_demo&#39; \<br>  https://api.example.org/workspaces/ws-alice/token</pre><p>The caller must be authorized to issue the token and already hold ws_api, the requested profile roles, and the required audiences. Administrator status does not bypass these checks. Workspace client credentials stay on the API server.</p><p>The exchanged token keeps Alice’s identity, ws_api, and vcluster_vc0_demo, without her workspace or cluster administrator roles. The relevant claims look like this; the user ID is a placeholder, and other identity and time claims are omitted:</p><pre>{<br>  &quot;sub&quot;: &quot;&lt;alice-user-id&gt;&quot;,<br>  &quot;aud&quot;: [&quot;workspace-api&quot;, &quot;ws-alice&quot;],<br>  &quot;resource_access&quot;: {<br>    &quot;ws-alice&quot;: {<br>      &quot;roles&quot;: [&quot;ws_api&quot;, &quot;vcluster_vc0_demo&quot;]<br>    }<br>  }<br>}</pre><p>Selecting a connection profile alone would leave her token permissions intact. The exchange uses a workspace client whose baseline role mapping contains only ws_api. Each configured profile has an optional client scope mapped to its runtime role. The broader interactive scope is blocked during exchange, and Workspace API validates the returned identity, audiences, scopes, and roles. Changing the audience alone would not remove administrator roles belonging to that same client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f_uLx4wmtAasam7n_B1BMQ.png" /><figcaption>Alice keeps her administrator access. The notebook uses the restricted token for the export setup and submission.</figcaption></figure><h3>From the notebook to the Job</h3><p><a href="https://provider-storage.versioneer.at/latest/"><strong>Provider Storage</strong></a> supplies the workspace’s S3 connection, including its endpoint and credentials. Alice runs a notebook cell that uses the exchanged token to request s3:primary from Workspace API. That cell creates the Kubernetes Secret catalog-export-storage with the returned storage credentials. It uses the restricted Kubernetes connection to apply the Secret, install the export script and a suspended CronJob template, then start one Job.</p><p>Kubernetes supplies the storage credentials to the export Pod through envFrom. Neither Alice&#39;s login token nor the exchanged token runs inside the Pod. The notebook completes the handoff before it starts.</p><p>Replacing the submitting token does not change those copied values. The Secret must be updated when credentials rotate. New export runs pick up the rotated credentials at startup; a running container keeps its existing values.</p><p>The notebook checks the permissions and verifies the exported file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*6acrDXKTXaUVHVYYAwLT5Q.png" /></figure><p>The two denied operations are permission checks. The permitted export is an actual completed Job: the notebook downloads its GeoParquet output and checks the contents and checksum. The hourly schedule stays suspended. The checks cover one small metadata export.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lctts5mrrjZ23hSvJj1ngA.png" /><figcaption>The notebook checks permitted and denied operations, then submits a real export Job.</figcaption></figure><h3>The S3 credentials still belong to the workspace</h3><p>Alice narrowed what her notebook could do in Kubernetes. The Job still received the workspace’s existing S3 credentials. Selecting s3:primary did not create a policy for exports/naip.parquet or limit writes to one output prefix. The requested filename is a destination, not an enforced access rule.</p><p>The retained ws_api role can also resolve other service connections. A namespace profile does not reduce that permission to one connection. This matters if Alice hands the token to another process: its Kubernetes permissions are smaller, but it can still request the service credentials that role permits.</p><p>The Workspace can act as the Security Token Service (STS), exchanging a workspace token for task credentials based on Provider Datalab’s declared access settings. For this export, those credentials would be limited to the permitted inputs, output location, operations, and lifetime. The API would also have to prevent the caller from retrieving a broader connection. A narrow key is of little help if the same caller can fetch wider credentials.</p><p>For S3, the client would receive the access key, secret key, session token, and expiry, with the storage backend enforcing the task’s policy. We covered this storage-side exchange in <a href="https://versioneer.at/blog.html?post=2026-09-09-stac-should-explain-data-access-without-guesswork">STAC Should Explain Data Access Without Guesswork</a>. Other backends need their own supported mechanism. The notebook does not yet perform that exchange or apply a per-task IAM policy. A check of that extension would need to show that the export can read its inputs and write its result, while access to another task’s data or broader credentials fails.</p><h3>The role and the runtime</h3><p>The exchanged token expires after five minutes. An accepted Job continues after that, and an enabled CronJob can keep creating Jobs. The credentials inside the Pod have their own lifetime too. The export template therefore sets a ten-minute Job deadline and permits one retry; the schedule stays suspended. Stopping the work means stopping active Jobs and any schedule that could start more.</p><p>The namespace role also needs care. Kubernetes RBAC grants add together, so other bindings to Alice’s identity or retained groups could add access again. The built-in edit role allows Secret access and workload creation in the namespace. It fits this submission example, but is too broad for a worker that only checks a result. Unrelated credentials should stay outside a namespace that the worker can edit.</p><p>A virtual cluster does not provide every other boundary. In vCluster’s shared-node model, tenant Pods run on the host cluster’s nodes. The platform team still needs host-side quotas, admission rules, and enforced network policies. ResourceQuota and LimitRange can bound requests and workload limits; they do not turn a namespace into reserved compute capacity.</p><h3>The catalogue outlives the export</h3><p>Alice can discard an exporter attempt while another colleague continues to use the same catalogue. The PostgreSQL service and source archive have a different lifetime from the export Job. Copying them for each attempt adds storage and transfer work; deleting the worker must leave shared data intact.</p><p>This is where the managed-service contract helps the team. An engineer declares the services and connections the application needs. The platform team maintains the operators, capacity, access rules, and recovery procedures. A shared catalogue needs an owner, retention rules, backups, and a checked restore procedure. Changes to its schema need coordination with the people who still use it.</p><p>To try the example, start with the <a href="https://eoepca.readthedocs.io/projects/workspace/en/latest/usage/notebook-guide/"><strong>Workspace notebook guide</strong></a>. Change a spatial filter or add a check for missing assets, then have a colleague run the saved process from a fresh environment. Check the result, stop the work, and remove the attempt’s resources while leaving the catalogue available.</p><h3>Where an agent helps</h3><p>An agent could help Alice try a Python library and a SQL query for the same regional export. Both attempts would read the same input snapshot, write to separate locations, and use the same result checks. An engineer could inspect either attempt’s code, dependencies, and command output, then continue from a failed step.</p><p>The platform enables these patterns with separate working environments and shared managed services. A vCluster provides the Kubernetes API for Jobs and services; engineers and agents can use kubectl, Helm, and normal clients within their grants. The exporter, service requests, connection references, and checks can stay with the code.</p><p>For Alice’s export, a saved run would include the source revision, image digest, dependency locks, input snapshot, and expected Item IDs. A colleague could repeat it from a terminal or continuous integration, while an agent could call the same steps when trying a change. A fresh run would still need ready services and valid grants; saved connection references do not preserve the services or the data behind them.</p><p>There is also an identity gap to close. The notebook’s token keeps Alice’s subject, but does not identify a separate worker acting for her. Operational records need to connect the submitting client and selected profile to the Job, input data, and output, without storing secrets. <a href="https://www.rfc-editor.org/rfc/rfc8693.html#section-4.1"><strong>OAuth token exchange</strong></a> already defines act for the current actor and may_act for a party authorized to act.</p><p><a href="https://www.keycloak.org/securing-apps/token-exchange#_token_exchange_delegation"><strong>Keycloak’s Token Exchange Delegation</strong></a> is a preview feature, disabled by default. Its client delegation flow checks delegation permissions and user consent, and can keep the user in sub while identifying the client in act. The notebook&#39;s profile exchange does not use that flow. The worker would still need S3 credentials limited to its task.</p><p>A submitted cloud Job can already continue after the laptop closes. An agent that keeps making decisions needs a remote worker and valid access for that work. Bring it in while the task needs decisions or code changes. Once the exporter works, a regular Job can run it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=955d420c1fc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[STAC Should Explain Data Access Without Guesswork]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/stac-should-explain-data-access-without-guesswork-10c6053cd551?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/10c6053cd551</guid>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[data-catalog]]></category>
            <category><![CDATA[metadata]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Sun, 18 Oct 2026 19:57:52 GMT</pubDate>
            <atom:updated>2026-10-18T19:57:52.463Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oJ1MSgB1IyJXOT8y3K36WA.png" /></figure><p>You query a STAC API, filter by area and cloud cover, and find the scene you want. The metadata looks right. You pass the asset URL to rasterio or rioxarray and try to read a band.</p><p>403 Forbidden.</p><p>Now you are reading provider documentation instead of satellite data. An API key might be missing. Perhaps the URL must be signed first, or the storage driver expects credentials from a cloud account. The search took a few lines of Python; now you have to work out how to configure the storage reader.</p><p>We put considerable care into formats, storage, and catalog metadata for Earth observation (EO). Getting credentials into the reader deserves the same attention as finding the right scene.</p><p>In our EOEPCA+ and EarthCODE work, we want STAC to describe the access flow alongside the bands, dates, and locations.</p><h3>The gap between the catalog client and the reader</h3><p>The <a href="https://stacspec.org/en/about/stac-spec/">SpatioTemporal Asset Catalog (STAC) specification</a> lets different providers describe their data in a common structure. A client can search by place and time, inspect the results, and extract an asset’s href.</p><p>A notebook can use <a href="https://pystac-client.readthedocs.io/en/stable/usage.html">pystac-client</a> to find scenes, then pass the items to <a href="https://odc-stac.readthedocs.io/en/latest/">odc-stac</a> or <a href="https://stackstac.readthedocs.io/en/latest/">stackstac</a> for lazy reads into xarray. For local processing, <a href="https://github.com/stac-utils/stac-asset">stac-asset</a> can download the linked files. Some helpers choose a client from the asset URL, but a familiar hostname cannot reliably describe the required authentication flow.</p><p>Rasterio and rioxarray read through GDAL; fsspec uses backends such as s3fs. Their storage credentials are separate from the catalog client&#39;s HTTP session.</p><p>For an S3 asset, GDAL’s <a href="https://gdal.org/en/stable/user/virtual_file_systems.html#vsis3-aws-s3-files">/vsis3/ handler</a> accepts settings such as AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN, along with the endpoint and region. A signed-in catalog session does not populate those settings. A notebook may then require a setup cell that obtains a token, prepares S3 credentials, and connects the reader&#39;s refresh mechanism.</p><p>A Cloud Optimized GeoTIFF (COG) makes this more demanding. The reader uses <a href="https://docs.ogc.org/is/21-026/21-026.html">HTTP range requests</a> to fetch selected parts of the image. Opening one asset can mean many storage requests. A Zarr workflow can likewise read many separate chunk objects. The access setup has to remain valid throughout those reads.</p><p>For a short notebook, a helper function may be enough. Move the analysis to a Dask or Ray cluster and the same setup has to work across many processes, possibly for hours. A token that was valid when the job started may expire before the last worker opens its next file. Updating credentials in the notebook does not necessarily update a running worker’s storage client.</p><p>We want shared tooling to handle that handoff from href to read, using access instructions published with the metadata.</p><h3>Public reads, shared capacity, real bills</h3><p>Anonymous public access avoids much of this setup. The <a href="https://registry.opendata.aws/sentinel-2-l2a-cogs/">Sentinel-2 COG archive on AWS</a>, for example, combines STAC discovery with reads that require no AWS account. For a workshop, a new research idea, or a public map, that is a welcome place to start. Many EO collections are still offered this way.</p><p>Public reads generate request and data-transfer bills alongside storage costs. Moving data between cloud regions or out over the internet can incur egress charges. <a href="https://aws.amazon.com/s3/pricing/">S3 pricing</a> separates these charges, and the hosting or sponsorship arrangement determines who pays them.</p><p>Reading a small part of a COG saves bytes compared with downloading the whole image. A large Dask job can still send millions of range requests across cloud boundaries. Retries and repeated reads increase both the request count and the traffic. The provider still has to budget for all of those reads, even when each worker requests only the pixels it needs.</p><p>A researcher inspecting two scenes can share a service with a production pipeline running hundreds of workers. Both workloads may be legitimate, but where bandwidth, request capacity, or gateway resources are shared, the larger job can leave less room for everyone else. This is the noisy-neighbor problem.</p><p>A provider can apply IP-based rate limits to anonymous traffic. But a university or cloud platform may put many unrelated users behind one address. That is a poor boundary for deciding whose job should slow down. With a project or workload identity, the provider has a better basis for allocating capacity, diagnosing a retry loop, or contacting its owner.</p><p>Public access does not necessarily mean fair access. The <a href="https://documentation.dataspace.copernicus.eu/Quotas.html">Copernicus Data Space Ecosystem</a> uses account quotas to share free resources fairly, reducing download speed when a user exceeds their allowance. <a href="https://www.ecmwf.int/en/about/media-centre/focus/2026/minutes-matter-how-ecmwf-delivers-time-critical-data-global-scale">ECMWF</a> also uses queues, throttling, and fair-use limits to prevent congestion and support equitable access to its data services.</p><h3>Requester Pays changes who gets the bill</h3><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html">S3 Requester Pays</a> addresses the delivery costs directly. The bucket owner pays for storage; the requester pays for requests and downloads. S3 authenticates the request to identify the account to charge, and the client must explicitly accept the charges.</p><p>For a provider hosting a large archive, this can make a substantial difference. A heavy user takes on its delivery costs instead of passing them to the host.</p><p>For that workshop or small research group, the extra step may be harder to justify. Someone must arrange a billing account, configure the tools, and accept the financial responsibility. A collection can be openly licensed and still be difficult to use within a particular institution’s account and budget rules.</p><p>Requester Pays changes who funds requests and downloads. Sharing capacity fairly still requires separate controls, and users still need a billing account and configured tools.</p><p>Sometimes the provider is willing to fund public reads and simply wants to know which project is making them. Authentication can support that. A service can remain free to read while applying a quota per project or workload. Restricted collections may require additional permissions, but identifying a caller already has value for an open collection.</p><h3>Keep a stable link and decide access when it is used</h3><p>A client can discover a stable asset URL first and request a temporary signed URL when it is ready to read. <a href="https://planetarycomputer.microsoft.com/docs/concepts/sas/">Microsoft Planetary Computer</a> uses a signing service for Azure Blob assets. Its planetary_computer.sign helper obtains a Shared Access Signature (SAS) token and adds it to the asset URL. The reader then uses that URL to fetch data directly from storage.</p><p>Planetary Computer’s public signing service accepts anonymous requests. The signature authorizes storage access, so a signed URL does not necessarily identify the person using it. The service can still vary rate limits and token lifetimes, including when a subscription key is supplied. It is an example of open data served through an access step, rather than a requirement for every reader to have an account.</p><p>In our own work, we often put a redirect in front of storage, for example in <a href="https://github.com/versioneer-tech/package-r">packageR</a>. The STAC asset contains a stable public URL. When a client requests it, a small service creates a short-lived presigned URL and returns a temporary HTTP redirect. The client follows it, and object storage serves the bytes.</p><p>This lets us keep the published link while applying access rules at the redirect service. We can expose one collection or asset, introduce project-based limits, or require authentication for a particular resource. The bucket can stay private, and a large raster does not have to pass through the application server. Temporary signed URLs stay out of the catalog.</p><p>The redirect service can become a bottleneck when many workers open assets at once. Readers must also follow redirects correctly, preserve range requests, and obtain a new signed URL when the old one expires. Testing a browser download does not establish that the same flow works through a geospatial driver.</p><p>An anonymous redirect also remains anonymous. To account for usage by project, we authenticate the request before issuing the storage link. That distinction matters when deciding what the access step is meant to achieve.</p><h3>Let STAC describe the access conversation</h3><p>OAuth 2.0, HTTP bearer tokens, and AWS Signature Version 4 (SigV4) are already widely used. What is missing across providers is a consistent description of how to connect them for a particular asset.</p><p>An S3 credential helper still needs to know where to get a token, whether to exchange it, and which permissions to request. Those instructions belong with the asset metadata, so each notebook can use the same integration.</p><p>The <a href="https://github.com/stac-extensions/authentication">STAC Authentication extension</a> provides fields for that description. auth:schemes defines access schemes; auth:refs connects assets and links to those schemes. Its token-exchange work describes the token endpoint, the input token type, and available scopes. Schemes can refer to the source of an input token, so a client can follow the exchange steps from the document.</p><p>For HTTP bearer access, the result is a token the reader sends in an Authorization: Bearer header. The extension remains at proposal maturity, and describing a flow does not mean every client can already execute it.</p><p>In EOEPCA+ and EarthCODE, we need STAC metadata to tell clients how to obtain S3 credentials. We have also learned from colleagues at <a href="https://developmentseed.org/blog/2026-08-19-foss4g/">Development Seed</a>. They work on similar access challenges in <a href="https://docs.maap-project.org/en/ogc/science/NISAR/NISAR_DEM.html">NASA Earthdata workflows</a> and <a href="https://docs.source.coop/data-proxy">Source Cooperative</a>: connecting a user’s login to the tools that read the data.</p><p>Our starting point is a short-lived access token, for example issued by Keycloak in JSON Web Token (JWT) format. The reader expects S3 credentials, so we need a way to connect the two.</p><p>We have identified two flows. Both preserve the S3 request format, but they put the conversion from identity to storage access in different places.</p><blockquote><strong><em>Update, 18 October 2026:</em></strong><em> We have now submitted a pull request following the discussion in </em><a href="https://github.com/stac-extensions/authentication/issues/47"><em>STAC Authentication issue #47</em></a><em>. The aim is to standardize how STAC describes the credential flows needed for SigV4 requests. That would let credential helpers configure standard S3 tools such as AWS CLI, Boto3, rclone, and GDAL from the catalog’s access metadata.</em></blockquote><h3>Exchange the token at STS</h3><p>The familiar route is a Security Token Service (STS). Public clouds offer this too: <a href="https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html">AWS STS</a>, for example, exchanges identity-provider tokens for temporary access credentials. The client receives AccessKeyId, SecretAccessKey, and SessionToken, together with an expiry time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*CgukrGsnuvolYGV88wwMMA.png" /></figure><p>The S3 client uses those temporary credentials to sign requests with SigV4.</p><p>The <a href="https://github.com/source-cooperative/data.source.coop/tree/main/adrs">Source Cooperative ADRs</a> explicitly describe this exchange pattern.</p><p>An STS design also creates a place to set a permission ceiling. A platform token can have a longer lifetime and broader scope, while the storage session is shorter and limited to the job. A user with write access could request read-only credentials for a processing run. The gateway then enforces that ceiling along with the current resource permissions.</p><p>The identity provider does not have to describe every storage operation in the original token, and the job does not carry all of its owner’s permissions into every worker. The cost is an additional service and an exchange whenever credentials are obtained or refreshed. The provider also has to define and document which roles clients can assume.</p><h3>Put the JWT in the S3 session-token field</h3><p>This flow skips STS and uses an S3-compatible gateway built to validate JWT access tokens. Native AWS S3 does not accept an arbitrary JWT as its session token.</p><p>For us, skipping STS is appealing: there is one fewer component to operate and one fewer credential exchange to manage. We also control which scopes clients can request and receive from our OpenID Connect (OIDC) provider, Keycloak. That lets us issue tokens with the scopes the job needs directly from the identity provider.</p><p>We learned the credential pattern from <a href="https://supabase.com/docs/guides/storage/s3/authentication#session-token">Supabase’s S3 session-token support</a>: the project reference identifies the tenant in the access-key ID field, while the project’s anonKey fills the secret-access-key field. The user&#39;s JWT goes in the session-token field. Storage validates the JWT and applies that user&#39;s access policies.</p><p>We adapted the idea for our EOEPCA+ and EarthCODE work. SigV4 requires SecretAccessKey to derive its HMAC signing key. We use a fixed signing value per tenant so standard S3 clients can sign requests and the gateway can verify them. Our credential profile is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rs6vR-t5y6ZI6S1NzDeLZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*Q6wDxjUOChuqTzy9bXgl3A.png" /></figure><p>The gateway can reject an unknown tenant or a mismatch between the access-key ID and the requested tenant before validating the JWT. It then verifies the SigV4 signature and the JWT’s signature, issuer, audience, and expiry. The verified claims and access policies must authorize the requested tenant, operation, and asset.</p><p>In this profile, the fixed signing value is not secret, so SigV4 adds no independent proof of identity or protection against request forgery. HTTPS protects requests in transit; access depends on the validated JWT and the gateway’s authorization checks.</p><p>With either flow, we can keep using AWS CLI, Boto3, rclone, and GDAL against the configured S3 endpoint. A credential helper can obtain a token, exchange it at STS or build the direct JWT credentials, and return the usual key, secret, and session token. AWS CLI and Boto3 can run the helper command specified by credential_process in an AWS profile. When the helper also returns an expiry time ( Expiration), the tool can rerun it automatically before the credentials expire. The helper integration differs by tool, but token handling can stay out of the analysis code.</p><h3>When all the workers start together</h3><p>A cluster launch can create a thundering herd at STS when every worker exchanges its token at once. The direct JWT flow removes that exchange; the gateway can validate self-contained JWTs locally using cached issuer public keys.</p><p>It does not remove the identity provider. Workers still require valid access tokens, and a badly coordinated refresh can send them all back to Keycloak at once. We still have to reuse credentials within their scope, cache them until refresh is due, and spread refresh requests over time. STS clients can use those techniques too.</p><p>Every worker needs a refresh path that reaches its data reader. For GDAL, test that path with a dataset kept open across token expiry. Range reads should reuse credentials without another token exchange.</p><p>Both flows can verify credentials without a session database. Permission checks, quotas, and usage tracking may still require state.</p><h3>Connect STAC to the credential provider</h3><p>Metadata access also has to scale. Indexed, paginated searches let clients select inputs without listing an archive. Catalog access and asset access may require different permissions, so the access description has to cover both.</p><p>The S3 metadata we are proposing would tell a credential helper where to get a token, whether to exchange it at STS, and which credential profile to use. The reader’s credential provider would then call the helper to obtain and refresh credentials.</p><p>A provider should publish the access flow once. Notebooks and workers should obtain and renew credentials through the same shared integration.</p><p><em>Originally published at </em><a href="https://versioneer.at/blog.html?post=2026-09-09-stac-should-explain-data-access-without-guesswork"><em>https://versioneer.at/blog.html?post=2026-09-09-stac-should-explain-data-access-without-guesswork</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10c6053cd551" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Before the Code, Before the Portal]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/before-the-code-before-the-portal-2e2b8b87e665?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e2b8b87e665</guid>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[data-access]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 21:35:19 GMT</pubDate>
            <atom:updated>2026-06-28T21:36:20.689Z</atom:updated>
            <content:encoded><![CDATA[<p>Earth observation teams should not have to download a directory just to find out what is inside it.</p><p>They should not have to write Python only to check whether a generated GeoTIFF looks right. They should not have to build a custom portal only to show a few output files. And they should not need broad bucket credentials just to connect a dashboard to one useful result.</p><p>This is a common gap in EO workflows.</p><p>The data is already in object storage. The processing job has finished. The workspace contains the expected files. But the next question is still practical: what is here, what looks useful, what can be shared, and what is ready to move toward publication?</p><p>A bucket can store EO data. It cannot, by itself, make that data easy to inspect, share, or reuse.</p><p>That is where <a href="https://package-r.versioneer.at/latest/"><strong>packageR</strong></a> fits.</p><blockquote>packageR is a browser-based first-look and sharing layer for object-storage-backed workspace data. It helps users browse folders, inspect files, preview supported assets, create scoped shares, expose public outputs, and connect selected results to catalog workflows.</blockquote><p>The main idea is simple: do not turn every small data question into code, downloads, or a new portal.</p><p>This is the first article in a series on Versioneer platform tooling for Earth science data lifecycles. We start with packageR because many lifecycles begin with a simple question: what is in this workspace, and can it be inspected or shared safely?</p><h3>Use case 1: check the output after a processing run</h3><p>Imagine an EO team has just run a flood mapping, snow cover, burn scar, or land surface temperature workflow. The output folder contains a Cloud Optimized GeoTIFF, a thumbnail, logs, QA artifacts, checksums, a style file, and maybe a STAC JSON file.</p><p>The first task is not deep analysis. The first task is orientation.</p><p>Does the output exist? Does the raster open? Does the preview roughly match the expected area? Are the QA files present? Is this the final product or an intermediate file? Is there enough here to justify opening a notebook, VS Code, QGIS, xarray, GDAL, or a validation workflow?</p><blockquote>packageR helps at that moment. A user opens a browser view over mounted workspace storage, navigates the product folder, checks file details, previews supported image assets, and decides what deserves deeper work.</blockquote><p>That is different from analysis. packageR does not replace Jupyter, QGIS, xarray, GDAL, Dask, or domain-specific viewers. It helps users decide when those tools are worth opening.</p><p>This matters because EO outputs are rarely single files. A result can include COGs, STAC JSON, GeoParquet indexes, thumbnails, QA layers, provenance, logs, style files, licenses, and Zarr stores. A raw object listing only shows that files exist. packageR helps people understand the folder well enough to choose the next step.</p><p>Cloud-native formats make this more useful. A <a href="https://www.cogeo.org/"><strong>Cloud Optimized GeoTIFF</strong></a> is still a GeoTIFF, but it is organized so clients can request only the parts of the file they need. packageR supports that direction by keeping data in storage and using presigned access where configured, instead of forcing every byte through the browser application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ee-66qiJe3s-Jj_c.png" /></figure><h3>Use case 2: a no-code view inside DataLab</h3><p>The point is not to avoid DataLab.</p><p>The point is to avoid making every first question start with code.</p><p>A DataLab is the workbench around the data. It can provide JupyterLab, VS Code Server, terminals, dashboards, storage access, credentials, quotas, and collaboration rules. That is where deeper analysis, validation, scripting, and production work belong.</p><p>But before opening a notebook or terminal, users often need a plain view of the workspace.</p><p>This is how packageR fits into the EOEPCA+ work. The EOEPCA Workspace Building Block brings together object storage, interactive runtimes, and collaboration tooling. It gives users browser access to data, editors, terminals, and secure share links within the same workspace context.</p><blockquote>Inside a DataLab session, packageR is the first-look component. Users can browse mounted storage, inspect generated outputs, preview supported files, and create scoped links. If that is enough, no code was needed. If the result needs deeper work, the analysis tools are already next to the data.</blockquote><p>The same pattern appears in <a href="https://provider-datalab.versioneer.at/"><strong>Provider Datalab</strong></a>, where a workspace can include VS Code Server, terminals, a storage browser, object-storage access, and tools such as awscli and rclone.</p><p>The split is simple: packageR gives the first look. DataLab gives the deeper workbench. Object storage remains the durable foundation.</p><h3>Use case 3: a simple publishing facade</h3><p>Sometimes a result does not need a full publication workflow yet. It just needs to be visible.</p><p>A team may want to expose a COG, thumbnail, GeoJSON file, style file, or small validation result. A dashboard may need a stable URL. A colleague may need to review a few outputs without getting access to the full workspace.</p><p>packageR can act as a simple publishing facade for that.</p><p>In a workspace deployment, a public share can be mapped with a rule such as publish=/home. The important part is the boundary. Users do not all write into one shared public directory. Each user writes into a scoped area below /home.</p><p>Alice can publish Alice’s files. Bob can publish Bob’s files. Neither user needs permission to overwrite the other person’s public outputs.</p><p>This matches the packageR access model. The feature documentation describes static default shares and user-default access rules, where non-admin users can be restricted to a sanitized username path below a configured home base path. The <a href="https://package-r.versioneer.at/latest/generated/usecases/public-sharing/"><strong>public-sharing walkthrough</strong></a> shows the same separation in practice.</p><blockquote>For users, the workflow is simple: put selected outputs into the publish area, and packageR exposes them through the public facade.</blockquote><p>No local download. No separate upload into a portal. No broad bucket credentials for consumers. No shared write area where one user can accidentally overwrite another user’s published files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LCjVTLI-ygF-eTWK.png" /></figure><p>This is not the final governed publication gate. It is the useful middle ground between private workspace output and a formal published data product.</p><h3>Use case 4: stable public URLs for dashboards</h3><p>The publishing facade becomes more useful when another application needs the data.</p><p>A dashboard, map client, or lightweight data viewer often wants a stable URL. But the actual bytes should still come from object storage where possible.</p><p>This is exactly the handoff needed by <a href="https://hub.eox.at/"><strong>EOxHub</strong></a> and applications built with <a href="https://eodash.org/"><strong>eodash</strong></a>. eodash is a dashboarding library, not a storage layer. It can render EO dashboards, but the browser still needs web-reachable references to STAC, services, COGs, GeoTIFFs, GeoJSON, FlatGeobuf, and similar assets.</p><p>A browser cannot mount a workspace path such as /home/alice/result.tif. It also should not receive broad S3 credentials. It needs HTTP URLs with the right access behavior.</p><p>packageR helps create that bridge.</p><p>Authenticated file requests and public share requests can return S3-compatible presigned URLs with ?presign=true. With ?followRedirect=true, packageR redirects to the generated presigned URL using HTTP 307 while preserving the request method.</p><p>That means an eodash-based dashboard can point to a stable packageR public asset URL. At request time, packageR can generate a temporary object-store URL and redirect the browser to storage. The dashboard should keep the stable facade URL, not the temporary presigned target.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mqQy072mFXPL4KwY.png" /></figure><p>The EOxHub story makes this concrete. In EOxHub Workspaces, packageR is packaged as the web file browser around workspace storage. A workspace can expose a public path that maps to a path in an object storage bucket. When an application requests an asset under that public path with presigning and redirect enabled, packageR can generate the presigned object-store URL and return the redirect automatically.</p><blockquote>An eodash-based dashboard can then access published outputs while the bucket itself remains private. The asset is reachable through a controlled facade, but the bucket is not turned into a public bucket.</blockquote><p>That is the friendly story: packageR is not only a backend component. It is already part of a real workspace pattern where users can place geospatial outputs in a public area, get stable URLs, and connect those assets to applications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*rOB4bnusnFAgJeC_.png" /></figure><h3>Use case 5: move from shared files to cataloged products</h3><p>A public folder is useful. A catalog is better when a result becomes reusable.</p><p>For Earth observation, <a href="https://stacspec.org/en/"><strong>STAC</strong></a> gives teams a common way to describe spatiotemporal assets so they can be indexed, discovered, and used by shared tooling.</p><p>packageR can bridge the file view and the catalog view. Shares can carry catalog metadata. A public catalog endpoint can read a configured Parquet catalog with DuckDB and return STAC Feature or FeatureCollection JSON. Matching asset links can be rewritten to packageR public-share URLs with ?presign=true&amp;followRedirect=true, so catalog discovery connects back to object-storage-backed access.</p><p>This gives teams a gradual path. First, users inspect the result in packageR. Then selected files move into a controlled publish area. A dashboard can use stable public URLs. If the result becomes more than a quick shared output, the same assets can be described through STAC and made discoverable through catalog tools. The <a href="https://package-r.versioneer.at/latest/generated/usecases/catalog-stac/"><strong>Catalog &amp; STAC walkthrough</strong></a> shows that in practice.</p><blockquote>That is the transition from “here is a folder” to “here is a product candidate.”</blockquote><h3>What packageR is, and what it is not</h3><p>packageR started as a fork of <a href="https://github.com/filebrowser/filebrowser"><strong>File Browser</strong></a>. It keeps familiar ideas such as users, shares, a file tree, and browser-based inspection.</p><p>The packageR-specific part is that data can live in object storage while appearing as a regular file tree, often through FUSE or another platform storage integration. packageR then adds presigned access, supported previews, public sharing, and catalog workflows such as STAC.</p><p>The boundary is important.</p><p>packageR is mainly for browsing, inspection, curation, sharing, preview, and catalog access. Large uploads and downloads should usually go directly to the bucket with S3-compatible tools.</p><p>It can preview TIFF and GeoTIFF files in the browser through a geotiff.js renderer. It detects Parquet mainly for catalog and STAC workflows. Zarr preview is not implemented today.</p><p>That is fine. packageR does not need to be a universal EO visualization engine. Its job is to remove friction from the first questions:</p><ul><li>What is here?</li><li>Can I inspect it?</li><li>Can I expose this subset?</li><li>Can an application consume it?</li><li>Can this move toward a cataloged product?</li></ul><p>When the question becomes scientific interpretation, validation, time series analysis, large-scale processing, or production automation, users should move into the right tool. The difference is that they can do so after they know which data is worth opening.</p><h3>A bucket is not enough</h3><p>Object storage is the right foundation for EO data, but it is not the user experience.</p><p>A bucket can hold the files. It cannot explain which output is useful. It cannot preview a GeoTIFF for a quick check. It cannot safely expose one user’s public result while protecting another user’s files. It cannot give a dashboard a clean public facade. It cannot turn a folder into a STAC-readable product candidate.</p><p>packageR fills that practical layer.</p><p>It helps teams inspect data before writing code. It helps workspace users publish selected outputs without copying them somewhere else. It gives applications stable facade URLs while still allowing object-storage-backed access through presigned redirects. It connects shared files to catalog workflows when a result is ready to become more than a folder.</p><p>packageR is not the platform. It is the small, useful layer between workspace storage and the next decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e2b8b87e665" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Lego Bricks to Composable Governance: Why Agentic Data Platforms Need a Resource Model]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/from-lego-bricks-to-composable-governance-why-agentic-data-platforms-need-a-resource-model-8871e7b1f036?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8871e7b1f036</guid>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[composable-infrastructure]]></category>
            <category><![CDATA[data-governance]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Sun, 17 May 2026 21:20:46 GMT</pubDate>
            <atom:updated>2026-05-18T07:18:38.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kJhwGYLkNEluc53Xt3uCKA.png" /></figure><p>In the <a href="https://medium.com/@stefan.achtsnit_41940/building-data-supply-chains-with-lego-bricks-why-earth-observation-needs-composable-infrastructure-38600b920bb6">earlier post</a> on building data supply chains with Lego bricks, we argued that Earth Observation does not only have a scale problem. It has a plumbing problem. Petabytes of data, countless sensor streams, and increasingly capable processing environments are already available. The hard part is how data is stored, shared, governed, reused, and turned into dependable products across teams. The proposal was simple: stop rebuilding the same infrastructure patterns again and again, and start composing reusable building blocks instead. Storage, Data Labs, registries, providers, controllers, and compositions become the Lego bricks of a modern EO data supply chain.</p><p>That argument still stands. But something has changed since then. The next users of these building blocks will not only be researchers, engineers, data stewards, and platform teams. Increasingly, they will also be agents.</p><blockquote>Agents will request workspaces. Agents will inspect data. Agents will suggest access changes. Agents will trigger processing. Agents will prepare publications. Agents will help debug failed workflows. Agents will eventually become normal participants in the data supply chain.</blockquote><p>This does not make the platform problem smaller. It makes it more visible. Because the moment an agent can do something useful, it can also do something risky.</p><h3>The first instinct: put a gateway in front of everything</h3><p>The agent ecosystem is already discovering this. The Model Context Protocol, or MCP, has become one of the main ways to connect AI applications to external systems, tools, workflows, and data sources. That is useful because it reduces integration friction. Instead of every agent, tool, and data source having its own custom integration, MCP gives the ecosystem a common shape.</p><blockquote>But once agents can call tools, the next question appears immediately: who is allowed to call which tool, under which identity, with which parameters, for which task, and with what audit trail?</blockquote><p>That is why MCP gateways are emerging. They sit between agents and MCP servers, adding authentication, authorization, tool filtering, observability, and policy enforcement. This is not surprising. It is the normal platform story repeating itself: first we connect things, then we realize connected things need rules.</p><p>For external APIs, the gateway pattern is often the right answer. If a tool is outside our control, a governed proxy gives us a boundary. It can authenticate, filter, rate-limit, log, and observe tool access. That is valuable.</p><p>But for platform capabilities we own, there is a better pattern. <strong>Do not only govern the API call. Govern the intent</strong>.</p><h3>A gateway governs a call. A resource governs a lifecycle.</h3><p>This distinction matters.</p><ul><li>A gateway can answer a question like: May this agent call create_bucket with these parameters?</li><li>A resource model can ask a richer question: May this actor declare this storage intent, in this namespace, under this policy, with this lifecycle, for this owner, and with these access consequences?</li></ul><p>The second question is much closer to what platform teams actually need.</p><p>Creating a bucket is rarely just creating a bucket. In a real data platform, it may also mean choosing a backend, creating credentials, applying lifecycle rules, granting access, exposing connection details, recording ownership, making storage discoverable, linking it to a workspace, checking quota policies, and reporting whether everything is ready. That is not a single tool call. <strong>It is a lifecycle object</strong>.</p><p>This is why the Kubernetes resource model is so important. A Kubernetes resource is not just a request. It is a durable object with metadata, desired state, observed state, references, labels, events, RBAC, admission, audit, and reconciliation around it. Once a platform capability is expressed as a resource, it can be reviewed, validated, watched, reconciled, audited, and composed with other resources.</p><blockquote>This is the point where the Lego analogy needs an upgrade. Lego bricks are useful because they are reusable. But Lego systems work because they also have a connection model. The studs fit. The geometry is predictable. Pieces can be combined without negotiating a new interface every time.</blockquote><p>A data platform needs the same thing:</p><ul><li>The bricks are storage, workspaces, registries, catalogs, credentials, snapshots, policies, and controllers.</li><li>The connection model is the Kubernetes resource model.</li></ul><p>The next step is composable governance.</p><h3>From composable infrastructure to composable governance</h3><p>In the previous posts, our <a href="https://provider-storage.versioneer.at/">storage resource</a> was the simple story. A team declares a storage claim. Behind the scenes, the platform creates buckets, users, policies, credentials, and Kubernetes Secrets. The team does not need to think about IAM policies or backend-specific storage details. It declares what it needs, and the platform composes the rest.</p><p>That remains a strong example. But the next step is not only that storage becomes composable. Governance around storage must become composable too.</p><p>Imagine an agent helping a research team prepare a new sea-ice analysis environment. In a naive tool-calling model, the agent might call separate tools to create a bucket, create an access key, create a workspace, mount a dataset, create a database, grant access to another team, publish a preview, and update a catalog.</p><p>Every tool call now needs its own policy logic. Every tool needs to know something about ownership, namespace, quota, data sensitivity, access rules, and audit requirements. The more useful the agent becomes, the more the policy surface grows.</p><p>This is where sprawl returns. Not infrastructure sprawl this time, but governance sprawl.</p><p>A Kubernetes-native model avoids much of this by changing the action surface. The agent does not call every backend directly. It declares a small number of platform resources.</p><p>For example:</p><pre>apiVersion: pkg.internal/v1beta1<br>kind: Storage<br>metadata:<br>  name: team-sea-ice<br>  namespace: research<br>spec:<br>  principal: team-sea-ice<br>  buckets:<br>    - bucketName: bucket-sea-ice<br>      discoverable: true<br>  bucketAccessGrants:<br>    - bucketName: bucket-sea-ice<br>      grantee: team-ocean-model<br>      permission: Read<br>      grantedAt: &quot;2025-10-01T12:17:00Z&quot;</pre><p>This is not just a YAML version of an API call. It is a <strong>platform contract</strong>.</p><p>The resource can be reviewed before it is applied. RBAC can decide who may create it. A policy engine can validate the fields. Crossplane can compose it into backend-specific resources. Controllers can reconcile it over time. Status can show whether the bucket, credentials, and grants are actually ready.</p><p>The same idea applies to a cloud workspace, what we call a Data Lab:</p><pre>apiVersion: pkg.internal/v1beta2<br>kind: Datalab<br>metadata:<br>  name: team-sea-ice<br>spec:<br>  users:<br>    - alice<br>    - bob<br>  sessions:<br>    - name: default<br>      state: started<br>  data:<br>    readOnlyMount: true<br>  databases:<br>    pg0:<br>      names:<br>        - analysis<br>      storage: 20Gi<br>  registry:<br>    enabled: true<br>  security:<br>    policy: baseline</pre><p>Again, the agent is not manually wiring every component. It is asking for a governed workspace. The platform decides how that request becomes namespaces, storage mounts, credentials, database instances, quotas, security defaults, and runtime access.</p><p>This is the difference between giving an agent a toolbox and giving it a platform contract.</p><h3>Crossplane is the composition layer, not the whole story</h3><p><a href="https://www.crossplane.io/">Crossplane</a> is important here because it gives us a practical way to define these higher-level APIs and compose them into lower-level resources. An XRD defines the shape of a custom API, while a Composition defines how that API creates other resources.</p><p>That is the platform pattern Versioneer is leaning into: expose reusable capabilities as Kubernetes resources, while keeping implementation details behind a versioned, inspectable, composable contract.</p><ul><li>A <a href="https://provider-storage.versioneer.at/">storage resource</a> does this for S3-compatible storage. It packages bucket creation, access policies, credentials, lifecycle rules, and cross-user sharing into one spec.</li><li>A <a href="https://provider-datalab.versioneer.at/">Data Lab resource</a> does this for governed cloud sandboxes. It packages VS Code Server, terminals, storage access, optional vclusters, managed data services, registries, quotas, and security defaults behind one claim.</li></ul><p>These are not isolated tools. They are examples of a pattern. Storage becomes a resource. A lab becomes a resource. Credentials become a resource output. Access becomes declared state. A database becomes part of a composed workspace. A registry becomes part of the same platform contract.</p><p>The more capabilities are expressed this way, the more the platform can reason about them.</p><h3>Policy becomes another brick</h3><p>The real power appears when policy is no longer an afterthought.</p><blockquote>In many systems, governance is bolted on after the interface exists. First there is an API. Then there is a gateway. Then there is an exception list. Then there is a special rule for one team, another rule for one agent, and another rule for one workflow.</blockquote><p>That can work for a while, but it does not compose well.</p><p>A Kubernetes-native approach lets policy participate directly in the lifecycle of the resource. Tools such as <a href="https://kyverno.io/">Kyverno</a> can validate or mutate resources before they become part of the cluster state. That means policies can be written against the platform object itself.</p><p>A discoverable bucket may require metadata about data sensitivity. A write grant to another team may require a reason. A Data Lab with elevated permissions may require a short lifetime. An agent-created workspace may require a budget class.</p><p>These rules are not hidden inside a prompt, a script, or a gateway-specific plugin. They become part of the platform.</p><p>This is composable governance: policy becomes another building block that can be combined with storage, workspaces, identity, registries, catalogs, and lifecycle controllers.</p><h3>Status is the platform’s memory</h3><p>The other half of the resource model is status.</p><p>This is where the platform stops being just a provisioning interface and starts becoming operationally useful. In Earth Observation, this matters a lot. As argued in the earlier processing post, EO workflows are often less limited by raw compute than by the coordination layer around processing: state, orchestration, reproducibility, validation, and productization.</p><p>Real operational workflows are not one clean pipeline. They involve pauses, retries, quality checks, updated inputs, partial reruns, product releases, and humans in the loop.</p><p>Agents will need to operate in that messy reality. They cannot rely on “the workflow ran” as proof that the system is ready. They need a way to inspect what the platform actually observed.</p><p>That is what status can provide.</p><pre>status:<br>  storage:<br>    mountedBuckets:<br>      - name: bucket-sea-ice<br>        quota: 500Gi<br>        used: 137Gi<br>  services:<br>    postgres:<br>      ready: true<br>      backup:<br>        lastRun: &quot;2026-05-17T02:00:00Z&quot;<br>        status: &quot;Succeeded&quot;<br>      slo:<br>        availability: &quot;99.72% / 99.9%&quot;<br>        latency: &quot;97.8% &lt; 100ms / 95% &lt; 100ms&quot;</pre><p>This is not a metrics database. Metrics, logs, traces, and large datasets belong in systems designed for those purposes.</p><blockquote>But status can summarize operational meaning. A human can read it. A controller can react to it. A GitOps system can wait on it. An agent can reason over it.</blockquote><p>This is why status is not decoration. It is the memory of the platform object.</p><h3>The practical middle ground: governed sessions</h3><p>Governance should not become a loose collection of rules spread across scripts, prompts, APIs, gateways, dashboards, and human memory. Composable governance does not mean building one giant control plane that owns everything. It means exposing important platform capabilities as clear, reusable resource contracts. Behind these contracts, the platform can still use object storage, Kubernetes, Crossplane, Kyverno, GitOps, registries, databases, caches, vector stores, observability systems, and external identity providers. But users and agents should not have to wire all of that together themselves.</p><p>The practical middle ground is the Data Lab session. A session is an always-available or on-demand cloud sandbox for a team, a user, or a project. Humans can use it directly through terminals, VS Code, notebooks, dashboards, and APIs. They can also start agents inside the same session. These agents do not need a separate special environment. They work in the same sandbox as the user or team that started them, with the same identity boundaries, storage access, network limits, and platform rules.</p><p>The session gives humans and agents a ready-made working environment. It provides compute, memory, file and object storage, and access to higher-level services such as Postgres, Redis, MongoDB, Qdrant, registries, catalogs, or other data services. But the important part is that this environment can also include its own Kubernetes API interface, through a dedicated namespace or a separate vcluster.</p><blockquote>This Kuberntes cluster is where new processes are started. When a human submits a processing job, deploys a temporary service, runs a validation task, or starts a workflow engine, it leverages the Kubernetes API. When an agent does the same, it acts through the same API interface. This is a key simplification: agents do not need a new platform-specific execution model. They already know how to interact with Kubernetes concepts such as resources, namespaces, jobs, services, status, and logs. The platform can therefore expose a familiar Kubernetes interface, while still keeping the real host cluster protected.</blockquote><p>This creates a simple split: the team gets freedom inside the session, while the platform keeps control around it. The outer platform governs the session before it even starts. Operators can define which teams may create sessions, which session types they may use, how long sessions may live, how much CPU, memory, GPU, and storage they may consume, whether internet access is allowed, which datasets may be mounted, which managed services may be added.</p><p>The same Kubernetes-native controls still apply around the sandbox. RBAC controls who may create or change sessions. ResourceQuotas and LimitRanges bound consumption. NetworkPolicies limit communication. Admission control and Kyverno can enforce allowed images, security settings, labels, service accounts, and other rules. GitOps can add review, history, and rollback. That is the useful middle ground:<strong> self-service for the team, but governance for the platform</strong>.</p><p>For agents, this is especially important. An agent should not be trusted just because it is useful. But it should also not be blocked from useful work because direct backend access would be too risky. A governed session gives it a bounded place to act. The agent can use the workspace, data access, services, network paths, and additional Kubernetes capabilities like a vcluster that the platform has explicitly made available.</p><blockquote>Side note: this also makes it easier to introduce your own customized agents. These agents do not need to learn a completely new platform model. They can build on concepts that already exist and are well documented: Kubernetes resources, namespaces, RBAC, vclusters, Kyverno policies, quotas, services, jobs, logs, and status. The platform does not create a new world for agents. It lets agents work in the existing cloud-native world, but inside a governed boundary. More on that in a follow-up post.</blockquote><p>MCP gateways are useful when agents need governed access to external tools and APIs. But for platform capabilities we own, the stronger pattern is to model the capability as a resource, controlled from the outside by the same Kubernetes-native governance model that already protects the human team.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8871e7b1f036" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Earth Observation Has a Processing Problem — But Not the One We Think]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/earth-observation-has-a-processing-problem-but-not-the-one-we-think-5deefe3ae47e?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5deefe3ae47e</guid>
            <category><![CDATA[data-processing]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[data-orchestration]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Wed, 28 Jan 2026 20:31:57 GMT</pubDate>
            <atom:updated>2026-01-28T20:39:29.406Z</atom:updated>
            <content:encoded><![CDATA[<h3>Earth Observation Has a Processing Problem — But Not the One We Think</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pZCL_baKwJkLuiqTe-SDCg.png" /><figcaption>Compute is no Longer the Limiting Factor. Coordination Is.</figcaption></figure><p>For a long time, I was convinced that the main bottleneck in Earth Observation was simply compute. If only we had bigger clusters, more storage, and easier access to the cloud, everything else would fall into place. That belief shaped an entire generation of EO engineering. Without the luxury of an always-available HPC system, I gradually moved in the cloud from carefully curated machines and shared clusters to elastic cloud infrastructure — from pets to cattle, from local NFS shares to virtually unlimited object storage, from hand-written bash scripts to distributed frameworks.</p><p>And to be fair, it worked. Frameworks like Spark, Dask, Ray made it possible to <a href="https://www.nature.com/articles/s41597-025-04513-y">scale EO processing across clouds</a> and process entire archives in parallel. Tasks that once required a week-long batch run suddenly finished before lunch. Spinning up a cluster became a credit card operation instead of a procurement process. From a technical perspective, this is real progress, and the engineering maturity of EO processing stacks has improved dramatically.</p><p>And yet, something strange happened. Despite faster pipelines and better infrastructure, many EO projects still struggle to become dependable operational products. They work beautifully in demonstrations, but friction appears the moment they meet real business processes. The uncomfortable conclusion is this: we mostly solved the compute problem. The remaining gap is about orchestration, state, and productization — not how fast we process data, but how we manage processes and trust results over time.</p><h3>The scientific mental model</h3><p>Research workflows typically assume a clean, almost idealized structure. Data is ingested, a well-defined processing chain runs end-to-end, and a reproducible result is produced. The entire system can be expressed as one coherent graph: deterministic, automated, self-contained. This mindset is deeply embedded in our tools. Distributed frameworks encourage describing everything as one large directed acyclic graph. If all steps are encoded correctly, the system simply executes the graph and produces the answer. For science, this works remarkably well because it optimizes for clarity, repeatability, and performance. But business reality rarely behaves like a single graph.</p><h3>The operational reality</h3><p>Operational EO environments are messy by design. Satellite acquisitions do not arrive like a clean benchmark dataset. Scenes are delayed, orbits are missing, and half of a tile may be covered by clouds. Atmospheric conditions change between passes. A preprocessing chain behaves differently because an auxiliary DEM or calibration file was updated. A cloud mask turns out to be too aggressive and needs to be recomputed. A customer asks for the same product but only for a smaller AOI or with slightly different thresholds.</p><p>None of this fits neatly into a single, one-shot pipeline. What starts as a tidy workflow on paper quickly becomes a living process. Some scenes are reprocessed, others skipped. Certain steps require manual quality control — inspecting mosaics, fixing masks, adjusting parameters, and triggering partial reruns. Products are regenerated for new time windows or updated inputs. The system spends most of its time not computing, but waiting, branching, retrying, and coordinating.</p><p>What looked like one pipeline turns into a long-lived process with pauses, retries, forks, and decisions — not a straight line, but a state machine; not fully automated, but human-in-the-loop; not executed once, but repeatedly, in many small variations across seasons, regions, and customers. At this point the limiting factor is no longer compute. It is coordination over time, and that is where many EO systems start to feel brittle.</p><h3>Industry already figured this out</h3><p>If you zoom out from EO and look at where the broader software and data industry invests its money, the priorities are revealing. Billions have flowed into <a href="https://temporal.io/blog/what-is-durable-execution">durable workflow and orchestration</a> systems. Platforms such as Airflow, Temporal, AWS Step Functions, Azure Durable Functions, Prefect, and Dagster exist for one reason: real processes span days or months, involve humans, and must survive failures. Stateless batch jobs are not enough. Systems need memory.</p><p>At the same time, analytics platforms like Databricks and Snowflake became multi-billion-dollar companies not because they execute SQL slightly faster, but because they solved trust problems. Transactional semantics, lineage, time travel made it possible to treat data as reliable products rather than loose files. Underneath, table formats like Iceberg, Delta Lake, and Hudi introduced <a href="https://www.min.io/learn/open-table-format">database-like guarantees on top of object storage</a>. More recently, the surge of interest in agentic AI is essentially another attempt to tackle orchestration: systems that can decide what to do next, react to exceptions, and <a href="https://www.anthropic.com/engineering/building-effective-agents">coordinate complex tasks</a> that previously required manual glue code.</p><p>Seen together, the pattern is clear. The market is investing in coordination, state, and reproducibility, not in faster map algebra. These are exactly the challenges we face when trying to run EO workflows operationally.</p><h3>The data problem behind the workflow problem</h3><p>There is another layer that often goes unnoticed. In structured data engineering, one of the biggest breakthroughs of the last decade was surprisingly simple: <a href="https://docs.databricks.com/aws/en/delta/history">versioned datasets</a> and the ability to <a href="https://docs.snowflake.com/en/user-guide/object-clone">take snapshots</a>, branch, experiment safely, and roll back. Time travel in Iceberg or Delta Lake sounds like a convenience feature, but in practice it completely changes how teams work. Experiments become cheap, reproducibility becomes trivial, and audits become possible. You no longer argue about which input was used; you simply reference the snapshot ID.</p><p>This idea has proven so valuable that the industry is now pushing it even further. Beyond versioned tables, there are significant investments into making <a href="https://neon.com/blog/practical-guide-to-database-branching">full database forking and cloning</a> a first-class primitive as well. Modern data platforms increasingly support instant copies of entire databases or environments for testing, experimentation, or customer-specific variants. Instead of exporting and re-importing data, teams simply branch the whole system state and work in isolation.</p><p>EO workflows rarely have that luxury. We still tend to treat object storage as a pile of files. Intermediate results are overwritten, derived products drift silently, and reproducing last quarter’s output often means hoping nothing changed in the meantime. For science this is <a href="https://medium.com/@stefan.achtsnit_41940/towards-a-pragmatic-storage-ecosystem-for-earth-sciences-71f9bbeda5b8">inconvenient</a>. For operations it is unacceptable. This is why concepts such as content-addressed storage, deduplication, bucket versioning, and data forking are becoming more relevant. Systems like LakeFS or DVC bring Git-like semantics to large binary assets, allowing raster cubes and derived products to be snapshotted and branched just like code or tables. Once you have that, experimentation stops being risky and trust increases dramatically.</p><h3>Where EO often gets stuck</h3><p>A pattern I see repeatedly is that teams invest enormous effort into optimizing processing engines while treating orchestration and data semantics as secondary concerns. We debate whether Spark or Dask is faster, benchmark cluster sizes, and tweak chunking strategies. Meanwhile, the actual blockers in production are much simpler and more fundamental: can we re-run only one step, prove which inputs generated this result, pause a workflow and resume tomorrow, allow multiple teams to collaborate safely on the same data, or publish something as immutable and auditable? These are not performance questions. They are system design questions that require different primitives — workflow engines, versioned storage, packaging, lineage, and governance. In other words, product infrastructure.</p><h3>A shift in mindset</h3><p>Perhaps the biggest change required is conceptual. Instead of thinking in terms of pipelines that process data, we need to <a href="https://martinfowler.com/articles/data-mesh-principles.html">think in terms of products that evolve over time</a>. A product has versions, owners, and releases. It can be audited and reproduced months later. It is not just the result of a single execution, but the outcome of many coordinated steps across people and systems. This perspective aligns much more closely with how modern software is built and operated. Data engineering already moved there. Software engineering has lived there for decades. EO is only now starting to make that transition.</p><h3>The opportunity ahead</h3><p>This is actually encouraging. It means the next wave of innovation in EO will likely not come from slightly better algorithms or marginally faster compute. Those improvements matter, but they are incremental. The real leverage lies in better orchestration, stronger data semantics, and clearer packaging of results as portable, immutable artifacts. When those pieces are in place, reproducibility stops being a best practice and becomes a property of the system, and trust stops relying on documentation and starts relying on architecture.</p><p>At that point, an organization can say with confidence that a product was generated from exactly this data state, with exactly this workflow, and can be reproduced anytime. That sentence is what businesses actually buy — not clusters, not frameworks, not faster jobs, but dependable products. If Earth Observation wants to close the gap between research and operations, this is where the focus needs to shift.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5deefe3ae47e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Cost of Redundancy]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/the-cost-of-redundancy-522874a985da?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/522874a985da</guid>
            <category><![CDATA[data-storage]]></category>
            <category><![CDATA[data-redundancy]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 22:49:27 GMT</pubDate>
            <atom:updated>2025-11-01T16:21:01.576Z</atom:updated>
            <content:encoded><![CDATA[<p>In the <a href="https://medium.com/@stefan.achtsnit_41940/towards-a-pragmatic-storage-ecosystem-for-earth-sciences-71f9bbeda5b8">previous blog post</a>, we explored how building a data ecosystem — rather than a single monolithic data platform — creates flexibility, scalability, and resilience. We argued that storage design should start from how data assets are consumed, not merely how they are written. The proposed model consists of distinct layers, each optimized for its specific role within the data lifecycle. We also noted that different usage patterns — from long-term preservation to real-time analysis — demand varying levels of performance, accessibility, and durability guarantees.</p><p>Naturally, this raises a pragmatic question:</p><blockquote>If data exists across multiple layers, does that mean we’re paying for the same storage several times?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*QBC59_yjCgG5mlb281CPOA.png" /></figure><p>At first glance, this may seem inefficient. But in reality, <strong>redundancy is not duplication — it’s design</strong>. Each layer provides a distinct set of guarantees, and what may appear to be the same data twice — or simply stored in a different format, such as losslessly compressed for archival — in fact reflects different responsibilities within the broader ecosystem.</p><h3>Guarantees Come First</h3><p>Each storage class or layer exists to deliver specific guarantees. Understanding what you pay for — and why — is the foundation of cost-efficient architecture design.</p><ul><li><strong>Durability</strong> — ensures that data remain intact over time, typically achieved through multiple physical copies or erasure coding across different hardware or regions.</li><li><strong>Availability</strong> — guarantees that data can be accessed whenever needed, backed by redundant infrastructure and failover mechanisms.</li><li><strong>Disaster resilience</strong> — protects against full site or zone failures through multi-AZ or multi-region replication, reducing the risk of catastrophic loss.</li><li><strong>Performance and locality</strong> — keeps data close to compute and ensures low-latency access, relying on premium hardware, caching, and intelligent placement.</li></ul><p>Each storage class or layer exists to deliver specific guarantees. Understanding what you pay for is the key to designing cost-efficient architectures.</p><h3>Redundancy Is Already Built In</h3><p>Even a single bucket in most cloud environments is redundant by design. Services such as AWS S3, OTC, or OVHCloud typically store each object across <strong>multiple availability zones</strong> — that is, geographically separated data centers within the same region. You already pay for duplication — it’s what ensures durability and availability.</p><p>If that level of resilience isn’t required, most providers offer <strong>Single-AZ storage</strong> at a lower price. The trade-off is clear: reduced cost means reduced protection. A system failure, maintenance window, or even a rare disaster could make the data temporarily or permanently unavailable.</p><p>The <a href="https://www.reuters.com/world/blaze-destroys-servers-europes-largest-cloud-services-firm-2021-03-10">OVHcloud Strasbourg data-centre fire in 2021</a> remains a sobering example of this risk: an entire facility was destroyed, leading to total data loss for customers who hadn’t replicated their content elsewhere.</p><h3>Different Layers, Different Guarantees</h3><p>When designing a multi-layer data ecosystem, we’re not duplicating cost — we’re aligning storage to purpose.</p><p>Archival storage provides long-term durability, immutability, and cost-efficiency. It’s optimized for preservation, not performance, and serves as the disaster-resilient backbone of the ecosystem. Because it already safeguards against data loss across regions or storage classes, other tiers can afford to trade redundancy for efficiency.</p><p>If temporary unavailability during a data center outage or maintenance window is acceptable, a single–data center setup may be sufficient for the other storage layers.</p><p>For on-demand processing, it may even be more cost-effective to re-provision data from other layers when needed rather than maintaining expensive built-in replication. These environments are typically short-lived, and their resilience comes from automation and reproducibility, not redundancy.</p><p>No single storage configuration can meet all these goals simultaneously. When comparing offerings, it is essential to read carefully through the provider’s price list and the associated capabilities and guarantees — as similar-sounding services can differ significantly in durability, availability, and performance levels.</p><p>The ecosystem approach ensures that each layer contributes exactly the right level of <strong>performance, resilience, and cost-efficiency</strong> for its role — together forming a coherent, dependable foundation for <strong>scalable data operations</strong>.</p><blockquote>Paying more doesn’t mean paying twice — it means paying for stronger SLAs, faster access, or more resilience.</blockquote><p>In such an ecosystem, every layer has a <strong>clear purpose</strong>: you’re not paying for the same storage multiple times, but <strong>once for durability, once for performance, and once for compute locality</strong> — each at the most efficient price point.</p><h3>The Takeaway</h3><p>Efficiency doesn’t come from minimizing copies; it comes from understanding why each copy exists. Some data will live for decades in immutable form, some will be cached near compute for fast processing, and others will move dynamically between tiers as part of an automated lifecycle.</p><p>With good metadata and version tracking, all layers can operate as a single, intelligent system where cost, performance, and reproducibility remain in balance by design.</p><p><strong>Redundancy, in this sense, isn’t waste — it’s architecture</strong>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=522874a985da" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Towards a Pragmatic Storage Ecosystem for Earth Sciences]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/towards-a-pragmatic-storage-ecosystem-for-earth-sciences-71f9bbeda5b8?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/71f9bbeda5b8</guid>
            <category><![CDATA[data-versioning]]></category>
            <category><![CDATA[composable-infrastructure]]></category>
            <category><![CDATA[cloud-workspace]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 21:58:00 GMT</pubDate>
            <atom:updated>2025-10-31T22:57:21.545Z</atom:updated>
            <content:encoded><![CDATA[<p>At <a href="https://versioneer.at/">Versioneer</a>, we share a single vision: to make it effortless for teams to publish, discover, and use each other’s data.</p><p>Such an ecosystem should feel like a true commodity:</p><ul><li>Provisioning is self-service and policy-driven, so getting started takes minutes, not weeks — with <a href="https://versioneer-tech.github.io/provider-storage">storage</a>, <a href="https://provider-datalab.versioneer.at/">compute</a> <a href="https://github.com/EOEPCA/workspace">workspaces</a>, access controls and catalog registries are automatically provisioned and pre-wired, making the system easy to operate, safe to use, and ready to scale — if needed across clouds.</li><li>Producers can <a href="https://medium.com/@stefan.achtsnit_41940/standardizing-the-eo-data-supply-chain-enabling-reproducible-data-collaboration-at-scale-78c83fe81c3d">ship</a> well-described, versioned datasets; consumers can easily find, <a href="https://github.com/versioneer-tech/package-r">preview</a>, and integrate them — allowing everyone to focus on creating value rather than fighting infrastructure.</li><li>Governance is built in, not bolted on, ensuring trust, traceability, and compliance by design.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*k2PHKDyalSsRahKwmlYG6Q.png" /></figure><p>The world’s system and software engineers have already done a remarkable job delivering the low-level tooling for large-scale data access and transformation. Infrastructure can now be abstracted and automated like never before. What remains challenging, however, is the <strong>plumbing</strong> — how all these components are stitched together into something that feels cohesive, consistent, and easy to use.</p><p>This is where Versioneer supports the journey — by building <a href="https://medium.com/@stefan.achtsnit_41940/building-data-supply-chains-with-lego-bricks-why-earth-observation-needs-composable-infrastructure-38600b920bb6">composable infrastructure</a> that blends data-engineering expertise with platform-engineering practices to enable what we call a <strong>pragmatic data mesh</strong>. We don’t claim to have a silver-bullet solution, but through our <a href="https://github.com/versioneer-tech">open-source tooling</a> and professional services, we try to help organizations of all kinds — from enterprise departments to cross-project collaborations and open research initiatives — create shared, governed environments where data can flow freely and meaningfully between teams working toward common goals.</p><h3>The Context: Research Domains and Their Complexity</h3><p>The domains we typically operate in with our partners — such as <a href="https://eox.at/">EOX </a>and others in the Earth sciences — share two defining characteristics.</p><ul><li>First, they are <strong>research-oriented</strong>, where the primary goal is to publish and share ideas in a FAIR (Findable, Accessible, Interoperable, Reusable) way — often without knowing exactly how or by whom the data will be used. This already contrasts with the common notion of a <a href="https://martinfowler.com/articles/data-mesh-principles.html?utm_source=chatgpt.com#DataAsAProduct">data product</a>, which assumes a clearly defined consumer or business outcome.</li><li>Second, the data itself is <strong>complex and heterogeneous</strong>. It is rarely tabular or uniform; instead, it combines structured and unstructured components, legacy formats, domain-specific software, and diverse computational environments.</li></ul><p>In such contexts, the goal is not to chase a mythical next-generation storage system but to <strong>combine and orchestrate existing technologies</strong> so that they collectively serve a broad and diverse community — from global archives to small research teams. The real task is not to reinvent storage, but to align it with how scientists and engineers actually work and how data is supposed to be used.</p><blockquote>Storage design should begin with how data are consumed, not how they are written.</blockquote><h3>From Formats to Access Patterns</h3><p>Across Earth observation and modelling domains, a few recurring usage patterns appear consistently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*h_XA_AJMsap93-dTaQmqBw.png" /></figure><p>1. Archival storage provides long-term durability and efficient push/pull access. It forms the immutable foundation for provenance, reproducibility, and reliable reprocessing.</p><p>2. Visualization encompasses both static and dynamic display. Simple map or image previews rely on range-readable formats such as Cloud-Optimized GeoTIFF (COG), while interactive visualization — involving filtering, temporal slicing, or layer composition — benefits from columnar or chunked layouts such as GeoParquet or Zarr.</p><p>3. Exploration and analysis take place in interactive environments such as notebooks or workspaces, where users control parallelism, thread counts, and caching. This pattern emphasizes direct, flexible access and responsiveness to user-driven queries.</p><p>4. Processing and preparation cover the repetitive and automated workloads — from training and inference to staging and subset creation. These workflows benefit from predictable chunk keys, co-located caches, and efficient materialization of filtered or regional subsets close to compute, often as temporary or shareable “views.”</p><blockquote>Different usage patterns demand different performance and durability guarantees.</blockquote><p>These categories are not technology choices but <strong>behavioral contracts between users and storage</strong>.<em> </em>The challenge is to support all of them without fragmenting infrastructure or enforcing disruptive conversions.</p><h3>Object Storage as the Common Ground</h3><blockquote>Object storage is unchallenged as the common commodity layer, with many mature and reliable offerings across clouds and on-premise platforms. Choose the best balance between cost and guarantees, keeping infrastructure simple and immutable while allowing downstream systems to evolve and optimize dynamically.</blockquote><p>Object storage has quietly become the universal substrate across clouds and institutions. It scales elastically, handles immutability by design, and already provides fairness and reliability at global scale.</p><p>This principle was already well articulated years ago in “<a href="https://medium.com/pangeo/step-by-step-guide-to-building-a-big-data-portal-e262af1c2977">A Step-by-Step Guide to Building a Big Data Portal</a>” — and it remains entirely true today:<br>“Put your data in cloud object storage, make it self-describing, and you essentially have your portal.”</p><p>Initiatives like <a href="https://source.coop/">Source Cooperative</a> demonstrate how this foundation can be used pragmatically: datasets remain in the formats chosen by their providers and are exposed via standard HTTP or S3 endpoints.</p><p>This approach treats object storage as the <strong>lowest common denominator</strong> — the point of stability where different communities meet. Higher-level services, catalogs, and visualization layers sit on top but never interfere with the raw data path. Contributors are guided toward best practices, not forced into conversions. Innovation can then happen independently on both sides: producers keep their pipelines, and consumers develop new ways of accessing the same bytes.</p><h3>Virtualization Instead of Conversion</h3><blockquote>Support users in creating FAIR data through education, tooling, and stewardship guidance — but remain pragmatic about what can ultimately be delivered. Emphasize progress over perfection. By adding virtualized access later, long-term flexibility and interoperability can be maintained, even as technologies and formats evolve.</blockquote><p>A growing family of tools demonstrates how to adapt to existing data rather than replace it. Kerchunk and VirtualZarr turn traditional NetCDF or HDF5 archives into virtual Zarr-like structures by mapping byte ranges through lightweight reference files.</p><p>For the user, the dataset behaves like a cloud-native array — fully chunked and parallelizable — yet the data never move. These technologies matter because most Earth science data will remain in older formats for a long time. They shift transformation to the <strong>consumer side</strong>: notebooks, visualization services, or analysis platforms virtualize data on demand, using the same storage underneath.</p><p>This keeps infrastructure simple and immutable while allowing downstream systems to optimize dynamically.</p><h3>Optimization Within, Not Across, Formats</h3><blockquote>Each format is only ever optimal for specific consumption patterns. A layout tuned for time-series analysis may perform poorly for visualization or machine learning. Accept these trade-offs and adhere to open standards without altering the original data for data sharing.</blockquote><p>Every format has its own internal tuning knobs — chunk size, compression codec, column-group size, pyramid structure, and more. Within each domain, those can be tuned carefully to match usage patterns: Zarr chunk dimensions to expected slicing, GeoParquet row groups to filter operations, COG overviews to zoom levels.</p><p>But there will never be a single perfect configuration for all workloads. Each format can only ever be great for a specific <strong>consumption pattern</strong>. A dataset optimized for notebook analysis might perform poorly for dynamic visualization or model training. Accepting those trade-offs is central to the pragmatic approach: tune locally, design globally.</p><h3>Versioning Beyond the Format</h3><blockquote>Embrace the natural evolution of data with the right tooling while remaining cost-aware: preserve immutability, and when updates occur, store only the modified objects instead of duplicating entire datasets.</blockquote><p>Data products naturally evolve — through new corrections, reprocessings, or the incorporation of timelier inputs. Some modern formats, such as Zarr, are beginning to experiment with built-in versioning models; for instance, <a href="http://icechunk.io/">Icechunk </a>introduces Git-like commits and branching directly within array stores. While powerful, these mechanisms remain limited to their respective ecosystems. The broader challenge of lineage and reproducibility extends across all formats — from TIFF and Parquet to NetCDF.</p><p>A metadata-level versioning layer, independent of any specific storage or format, can bridge this gap. By maintaining digests, manifests, and relationships, it provides transparent provenance across datasets and transformations. Excellent tooling for versioning structured and unstructured data exists — from <a href="https://dvc.org/">DVC</a> and <a href="https://git-annex.branchable.com/">Git-Annex</a> to <a href="https://lakefs.io/">LakeFS</a>. Combined with <strong>immutable </strong>object storage, these approaches form a reproducible substrate for scientific collaboration.</p><h3>Openness, Fairness, and Responsible Use</h3><blockquote>A sustainable data ecosystem maintains openness and fairness by enabling effortless exploration, isolating intensive workloads to dedicated environments, and replacing rigid access limits with a culture of transparent, cost-conscious accountability.</blockquote><p>A healthy data ecosystem must balance open access with fair use. Users should be encouraged to explore data directly — listing, inspecting, and sampling through standard streaming interfaces — without being forced to download gigabytes of data upfront. Yet shared access inevitably introduces the familiar noisy-neighbor problem: a single over-enthusiastic user running large parallel downloads can degrade service for everyone else.</p><p>The solution lies in segmentation by purpose. Lightweight, exploratory access should remain frictionless, while sustained, high-volume processing is redirected to dedicated environments — cloud workspaces, institutional HPC clusters, or staging areas with caching and replication.</p><p>Fairness then becomes a matter of policy and economics, not restriction. Cloud providers already distribute load globally and charge transparently for usage; scientific infrastructures can follow the same principle. By <strong>simplifying the provisioning</strong> of storage and compute environments — enabling them to spin up on demand and shut down automatically when idle — platforms can foster openness by default while making costs or quotas visible to those who consume resources most heavily.</p><p>The challenge thus shifts from performance contention to <strong>cost awareness</strong> — from throttling curiosity to encouraging responsibility.</p><h3>Closing Thought</h3><p>The future of Earth science data handling will not be defined by a single new format or platform, but by the ability to combine the best of what already exists. To meet diverse consumption patterns, we at Versioneer believe in<strong> building a storage ecosystem</strong> rather than a monolith — one that can evolve alongside the community it serves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*JmfbmrvGblhbUgNPLWDYbw.png" /></figure><p><strong>Update:</strong> For a reflection on cost considerations within the storage ecosystem, see <a href="https://medium.com/@stefan.achtsnit_41940/the-cost-of-redundancy-522874a985da">this follow-up post</a>.</p><p>Within each format, optimization will continue, but real progress lies in connecting these approaches coherently: ensuring that archives remain trustworthy, visualizations stay responsive, analyses scale gracefully, and everyone — from students to researchers — can participate on equal terms.</p><p>Our mission is not to reinvent storage, but to make it work together for the long tail of science — through a pragmatic approach that delivers on the true promise of a data mesh: making data collaboration seamless, reproducible, and scalable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71f9bbeda5b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Data Supply Chains with Lego Bricks: Why Earth Observation Needs Composable Infrastructure]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/building-data-supply-chains-with-lego-bricks-why-earth-observation-needs-composable-infrastructure-38600b920bb6?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/38600b920bb6</guid>
            <category><![CDATA[supply-chain-data]]></category>
            <category><![CDATA[crossplane]]></category>
            <category><![CDATA[composable-infrastructure]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Wed, 01 Oct 2025 20:06:11 GMT</pubDate>
            <atom:updated>2025-10-01T20:42:19.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*LCZr7QFQCn67jv3srbde7A.png" /></figure><p>Earth Observation has always been about scale. Petabytes of satellite imagery, countless sensor streams, and the ever-growing demand to turn raw data into actionable products. But for many teams, the real bottleneck isn’t access to the satellites or sensors. It’s the <strong>plumbing </strong>— how data is stored, shared, governed, and reused across groups.</p><p>Think about the typical workflow: one team downloads a large archive, extracts a subset for their analysis, and maybe, after some emailing and manual transfers, another team gets access to a derivative copy. Every step introduces duplication, inconsistency, and delays. Instead of focusing on science or application building, researchers and engineers often find themselves reinventing the same infrastructure patterns.</p><p>That’s where <strong>composable infrastructure</strong> comes in — a way to assemble reusable building blocks instead of starting from scratch every time. In the cloud-native world, these building blocks aren’t abstract ideas. They’re real, standardized units: providers, controllers, and compositions that can be combined just like Lego bricks.</p><h3>From Buckets to Collaboration: A Simple Story</h3><p>Imagine a team of researchers working in some on-demand cloud workspace. The workspace gives them elastic compute power that integrates seamlessly with their usual tools. When the team spins up their workspace, they also declare a storage claim. Instantly, a dedicated bucket called team-sea-ice is provisioned for them.</p><p>This bucket becomes their <strong>sandbox</strong>: a safe space to upload raw satellite imagery, calibration files, and generated data products. They don’t have to worry about IAM roles or bucket policies — the system automatically creates the bucket, the IAM user, policies, and all the necessary access credentials. Those credentials are handed back as a Kubernetes Secret in their namespace, ready to be mounted into notebooks, workflows, or analysis pipelines.</p><p>But soon, the team discovers something interesting: a subset of their data is relevant to another group working on climate models. Instead of duplicating data manually, they mark their bucket as <strong>discoverable</strong>. With a simple declarative request, the other group asks for access, providing a reason. The first team then grants <strong>ReadOnly</strong> rights, and suddenly the second group has direct, governed access to the same dataset.</p><p>Now, reverse the roles: the scientists working on ocean models produce a refined product — for example, a downscaled time series for a region of interest. Instead of keeping it private, they decide to <strong>share it back</strong>. By granting <strong>ReadWrite</strong> permissions to their collaborators in the sea-ice team, they enable them not only to consume the data but also to contribute new observations and validation results.</p><p>This creates the foundations of a <strong>data supply chain</strong>: each team enriches the product at their step, and the permission model ensures <strong>clarity, accountability, and traceability</strong>. No endless email threads, no ad-hoc data copies scattered across laptops. Just <strong>in-place sharing</strong>, with access controlled declaratively.</p><blockquote>Side note: Many organizations share data directly through raw storage resources like S3 buckets or large NFS folders. That approach is common and works at small scale. But in most cases, it’s worth adding a software layer on top — to provide governance, metadata, and smoother collaboration. That’s a story for another blog post, and it’s also at the core of what we focus on at <a href="https://versioneer.at/">Versioneer</a>. The key point is that <strong>even higher-level platform concepts still rely on the same fundamental storage and access primitives</strong> described here — they’re just building on top of them.</blockquote><h3>Why This Matters</h3><p>This kind of collaboration unlocks three critical capabilities:</p><ol><li><strong>Scalability</strong><br>No more manually wrangling terabytes of data. Buckets, policies, and credentials scale automatically with demand, so teams can focus on science instead of infrastructure.</li><li><strong>Reproducibility</strong><br>Every request, grant, and permission is captured <strong>declaratively</strong>. This means workflows are auditable, traceable, and trustworthy — ensuring results can be repeated and verified.</li><li><strong>Collaboration at Speed</strong><br>Scientists, developers, and analysts can share data products in real time. No waiting for IT tickets or ad-hoc workarounds — just seamless, governed access that accelerates discovery.</li></ol><p>These aren’t abstract ideals. They’re the real pain points in Earth Observation projects — and they’re solved by treating infrastructure as composable building blocks.</p><h3>The Lego Analogy: Building Blocks All the Way Down</h3><p>So how do we make this work? Enter <a href="https://www.crossplane.io/"><strong>Crossplane</strong></a>.</p><p>Crossplane is the orchestration layer that turns cloud APIs into Kubernetes resources. But the real magic comes from its concept of <strong>compositions</strong>. A composition is like a Lego block built from smaller Lego blocks. Instead of exposing every low-level detail, it packages a set of infrastructure resources into a higher-level abstraction.</p><blockquote>With Crossplane, providers and compositions, we can finally treat infrastructure as Lego.</blockquote><p>For example:</p><ul><li>At the <strong>lowest level</strong>, we have individual providers like provider-aws, provider-minio, or provider-otc. Each of them manages the raw building blocks—buckets, IAM or bucket policies, and secrets—based on their platform-specific concepts.</li><li>On <strong>top of these</strong>, we add provider-kubernetes. This acts as the glue layer, binding policies and credentials back into Kubernetes-native resources. It makes them observable and lets us state dependencies.</li><li>And <strong>above all</strong>, we define our provider-storage<strong> composition</strong>. This is the Lego block that users actually interact with. From their perspective, they don’t need to think about IAM roles, access policies, or secret wiring. They simply declare a Storage resource—and everything else is provisioned, connected, and exposed automatically:</li></ul><pre>apiVersion: pkg.internal/v1beta1<br>kind: Storage<br>metadata:<br>  name: team-sea-ice<br>spec:<br>  buckets:<br>    - bucketName: bucket-sea-ice<br>      discoverable: true</pre><p>Buckets, IAM users, and policies are automatically provisioned, and the corresponding access credentials are surfaced as a Kubernetes Secret. This Secret can then be consumed directly or further exposed through additional APIs or user interfaces on top.</p><p>The important point is that all of these concepts <strong>manifest inside the Kubernetes cluster itself</strong>. Buckets, access policies, and secrets are not abstract cloud artifacts hidden behind proprietary consoles — they are exposed as <strong>first-class Kubernetes resources</strong>. That means:</p><ul><li>They are accessed and managed through the <strong>standard Kubernetes API layer</strong>, the same way you would handle Pods, Deployments, or Services.</li><li>They become instantly usable with <strong>common Kubernetes tooling</strong> like kubectl, GitOps workflows (Flux, ArgoCD), or CI/CD pipelines.</li><li>They are fully <strong>auditable and observable</strong>: every request, status, and event flows through Kubernetes’ reconciliation model and can be logged, traced, or monitored.</li><li>They are <strong>securable</strong> using Kubernetes-native mechanisms like <strong>RBAC</strong>, meaning access to infrastructure resources can be controlled with the same fine-grained policies already used for cluster workloads.</li><li>They are <strong>orchestrated by Crossplane</strong>: compositions define how low-level resources come together into higher-level abstractions, and Crossplane’s controllers continuously reconcile the declared state against the actual state in the cloud or storage backend.</li></ul><p>In other words, provider-storage takes the complexity of distributed cloud resources and brings them into the same declarative, Kubernetes-native control plane. Infrastructure is no longer a separate world that requires its own scripts and manual steps — it becomes part of the same fabric where applications run, governed by the same APIs and policies.</p><p>It’s <strong>Lego on top of Lego</strong>: low-level providers build basic blocks, and compositions snap them together into something teams can use immediately.</p><h3>Platform Engineering in Action</h3><p>This is platform engineering at its core. Instead of every team writing Terraform scripts, managing cloud credentials, and debugging IAM policies, we provide a curated set of reusable, versioned building blocks. Teams just claim them, like renting Lego kits.</p><ul><li>Need a new storage bucket? Claim a Storage object.</li><li>Need to collaborate with another group? Add a bucketAccessRequest.</li><li>Want to grant access? Add a bucketAccessGrant.</li></ul><pre>apiVersion: pkg.internal/v1beta1<br>kind: Storage<br>metadata:<br>  name: team-ocean-model<br>spec:<br>  buckets:<br>    - bucketName: ocean-model-private<br>    - bucketName: ocean-model-shared<br>      discoverable: true<br>  bucketAccessRequests:<br>    - bucketName: bucket-sea-ice<br>      reason: &quot;Consume sea-ice indices for downstream modeling&quot;<br>      requestedAt: &quot;2025-10-01T12:07:00Z&quot;<br>  bucketAccessGrants:<br>    - bucketName: ocean-model-shared<br>      grantee: sea-ice<br>      permission: ReadWrite<br>      grantedAt: &quot;2025-10-01T12:12:00Z&quot;</pre><p>Everything else — the plumbing, the credentials, the policies — is handled automatically. Interested in learning more? Explore the <a href="https://provider-storage.versioneer.at/latest/how-to-guides/usage_concepts">Usage &amp; Concepts </a>in the provider-storage documentation.</p><p>This approach means Earth Observation platforms such as <a href="https://hub.eox.at/">EOxHub Workspaces</a>, developed by our colleagues at EOX, can scale effectively. Every new project, every new team doesn’t start from zero. They inherit a reliable, declarative system that lets them focus on science and data, not infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/1*eWr7U0zSqqDYznme4CrPOA.png" /></figure><h3>Building the Future of EO Collaboration</h3><p><strong>The lesson here is simple:</strong> Earth Observation isn’t just about satellites or sensors. It’s about the supply chain of data products that connects raw data to end users. And that supply chain needs building blocks — reusable, composable, trustworthy.</p><p>When the Lego bricks are efficiently orchestrated and stacked correctly — starting with cloud providers at the foundation and building up to high-level compositions such as <strong>Versioneer’s offerings</strong> alongside open-source contributions:</p><ul><li><strong>Storage</strong> — buckets, policies, credentials</li><li><strong>Data Labs</strong> — reproducible analysis workspaces</li><li><strong>Registries</strong> — OCI-based data product packaging</li></ul><p>…teams gain the freedom to collaborate faster, scale their work, and innovate without being slowed down by infrastructure complexity.</p><p>These building blocks aren’t just technical choices. They’re <strong>strategic enablers</strong>. They make it possible for teams to stop reinventing the wheel and start building the future together. And in Earth Observation, where collaboration and data sharing are the lifeblood of progress, that might just be the biggest breakthrough of all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38600b920bb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standardizing the EO Data Supply Chain: Enabling Reproducible Data Collaboration at Scale]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/standardizing-the-eo-data-supply-chain-enabling-reproducible-data-collaboration-at-scale-78c83fe81c3d?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/78c83fe81c3d</guid>
            <category><![CDATA[data-packaging]]></category>
            <category><![CDATA[supply-chain-data]]></category>
            <category><![CDATA[open-container-initiative]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 13:24:11 GMT</pubDate>
            <atom:updated>2025-07-16T13:24:11.322Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3MB3NnNZrqK5ShJHw8ugIA.jpeg" /></figure><p>As reliance on Earth Observation (EO) data intensifies — spanning sectors from agricultural insurance to climate monitoring and emergency response — the reliability and reproducibility of data access have become more critical than ever. As EO data pipelines grow in complexity and significance, the challenge extends beyond simple storage: how can it be ensured that data reaches its consumers in a state they can trust, reuse, and act upon?</p><blockquote>Reproducible Data Collaboration is the defining challenge of the EO data supply chain.</blockquote><p>Much like its industrial namesake, a data supply chain encompasses every stage a dataset moves through — from raw ingestion, calibration, fusion, and annotation to packaging, versioning, and delivery. The goal is to ensure that by the time a dataset arrives at its point of use — whether that’s a human analyst, a machine learning model, or an application running on an edge device — it is not just accessible, but fully reproducible, context-rich, and verifiably complete.</p><h3>The File-Based Legacy</h3><p>Across much of the broader Earth data ecosystem — including but not limited to Earth Observation (EO) — data sharing is still dominated by a file-based mindset. This includes not only EO datasets like satellite imagery or land cover classifications, but also meteorological and environmental datasets such as wind fields, temperature grids, and atmospheric reanalysis products. Widely used formats like NetCDF and GRIB, often used for gridded environmental and meteorological time series, are passed around as individual files via FTP, object storage, shared folders, or presigned links. This model has clear advantages: it’s intuitive, compatible with legacy tools, and deeply ingrained in scientific and operational workflows. But as Earth data scales in volume, variety, and strategic importance, this approach begins to falter. What it lacks is structure: not just in how data is described, but in how it is versioned, grouped, delivered, and verified — within and across organizational boundaries.</p><h3>Discovery Isn’t Delivery</h3><p>Recognizing this, the community has already begun to move toward more structured approaches. One of the most impactful developments in recent years has been the rise of the <a href="https://stacspec.org/">SpatioTemporal Asset Catalog</a> (STAC) — a specification for describing EO datasets in a standardized, JSON-based format. STAC catalogs, collections, and items provide a machine- and human-readable way to represent data holdings, enabling clients to search by time, space, or metadata and resolve assets through href links. With STAC, data providers can expose rich, navigable catalogs instead of ad hoc folders, and clients can explore these catalogs to find the data they need.</p><p>STAC also complements other domain-specific conventions such as the Climate and Forecast (CF) metadata conventions used in NetCDF, enabling multi-disciplinary interoperability. Together, these frameworks enable more precise, machine-readable descriptions of datasets that span remote sensing, meteorology, and environmental science.</p><p>This was — and remains — a major milestone. STAC decouples metadata from storage and aligns the EO world with web-native discovery practices. But STAC alone doesn’t define how datasets are packaged, delivered, or verified. Each STAC Item may point to multiple assets, but there is no guarantee that those assets arrive together, unchanged, or in a form ready for downstream use. What STAC solves is discovery; what remains is to solve delivery, integrity, and packaging.</p><h3>Two Emerging Paths: Structured Access vs Reproducible Packaging</h3><p>While many EO tools and users still think in terms of files — save, load, inspect — the underlying nature of data has changed. Formats like Zarr, partitioned Parquet, and chunked NetCDF organize data across many files, folders, or object keys. These are not monolithic blobs but logical datasets: collections whose structure matters just as much as their content. For example, a Zarr store consists of binary chunk files organized in a folder tree, with JSON metadata files describing array shapes, data types, and chunking. Partitioned <a href="https://geoparquet.org/">GeoParquet</a> datasets use directory paths to encode attributes like year, region, or product type, and open table formats such as <a href="https://iceberg.apache.org/">Apache Iceberg</a> are emerging as preferred backends for Parquet datasets — offering transactional semantics, scalable metadata handling, and support for dynamic, appendable datasets.</p><p>The relevance of this topic is underscored by efforts at major public agencies. ESA’s <a href="https://eopf.copernicus.eu/">Earth Observation Processor Framework</a> (EOPF) for CSC Data Processors Re-engineering efforts is enabling interoperable distribution of analysis-ready Copernicus Sentinel datasets— offering harmonized processing pipelines, metadata via STAC, and output in cloud-native formats such as Zarr. Meanwhile, NASA’s <a href="https://www.earthdata.nasa.gov/about/esdis/eosdis">EOSDIS system</a> — including its network of DAACs and the Common Metadata Repository (CMR) — continues to evolve in support of standardized discovery and access to large EO data holdings. These initiatives emphasize transparency and reproducibility, often shaped by strong community input and shared concerns: how to structure and describe folder-based data, how to bundle and compress it, where to locate metadata such as STAC Items, and whether flat or nested layouts are more effective for cloud-native workflows. At the same time, they must balance the long-term sustainability goals of archival systems with the need to deliver data that is as fresh and as fast as possible.</p><p>This is where the split becomes clear. Different workflows require different relationships with data:</p><ul><li>For interactive exploration and analytics, sophisticated data access strategies and storage systems have continued to evolve — from domain specific client-side tooling such as GDAL (with Cloud-Optimized GeoTIFF support) to cross domain fsspec-based analytics stacks and embedded engines like <a href="https://duckdb.org/">DuckDB</a> (with its spatial extension), all of which can be orchestrated within large-scale processing frameworks like Apache Spark and Dask or full-featured spatial backends such as Apache Sedona and TileDB. Together, they enable fast, queryable, mutable, and fine-grained access to dynamic datasets. Drawing on decades of database, HPC, and cloud-native innovation, many domains are now converging toward shared conventions for large-scale spatial and temporal data access.</li><li>In contrast, downstream workflows such as machine learning, regulatory audits, offline validation, or on-device inference require more than just access to data — they demand reproducible packaging. This means datasets must be versioned, portable, self-contained, and verifiably complete, so they can be reused without relying on external systems or reinterpreting metadata. Several packaging standards address these needs. <a href="https://en.wikipedia.org/wiki/BagIt">BagIt</a> ensures data integrity through directory-based manifests and checksums and is widely used in digital preservation. <a href="https://www.researchobject.org/ro-crate">RO-Crate</a>, often published via RoHub, wraps data and metadata in a JSON-LD format, supporting provenance and FAIR principles. The broader <a href="https://www.researchobject.org/initiative/research-object-model">Research Object model</a> captures full analysis context, including code and workflows. Other solutions exist, especially for tabular data, but they have remained confined to specific domains without gaining wider adoption.</li></ul><p>Support for both modes demands fundamentally different mechanisms. While access technologies have matured through collaborative efforts across disciplines, packaging remains fragmented, lacking a common foundation. Within this gap, a compelling opportunity has emerged for the Earth data community to adopt standardized, cross-domain packaging practices.</p><h3>What OCI Could Offer</h3><p>The <a href="https://opencontainers.org/">Open Container Initiative</a> (OCI), best known for its role in standardizing Docker images, provides exactly the kind of model needed for structured, reproducible delivery. An OCI artifact consists of compressed layers of content, a config blob with structured metadata, and a manifest that links everything together. These artifacts can be pushed and pulled from registries like Docker Hub, AWS ECR, Quay.io, or Harbor — part of a vast and mature ecosystem that already powers the global software supply chain. The ability to reuse these ubiquitous registries and client tools across domains significantly lowers the barrier to adoption.</p><p>Notably, OCI registries can act as access gateways, enforcing access control at the artifact level without requiring direct exposure of underlying storage backends (e.g., S3, GCS, Azure Blob, NFS, Ceph, IPFS). OCI also provides support for content-addressable storage, layered packaging, deduplication, cryptographic attestation, and version control — all of which align closely with Earth data distribution needs.</p><p>In the EO context, OCI artifacts can encapsulate not only the data but also its organization, versioning, and lineage. Each artifact may contain STAC metadata in the config, sensor-specific folders as layers, and annotations or derived products linked via OCI referrers. Importantly, a docker pull or oras pull is not merely a download — it is a registry-to-runtime synchronization of a complete, content-addressable, verifiable data package.</p><p>This framing is further strengthened by recent advancements in the OCI 1.1 specification. OCI 1.1 formally introduces the concept of non-container artifacts — such as datasets, models, or configuration bundles — as first-class citizens within OCI registries. It extends the OCI Image Specification to support fully flexible media types, enabling domain-specific communities to standardize around meaningful conventions without altering the core registry mechanics.</p><h3>The Structure of an OCI Artifact</h3><p>The OCI specification comprises three core components:</p><ul><li><strong>Image Specification</strong> — Defines the structure of artifacts and their metadata.</li><li><strong>Distribution Specification</strong> — Governs the protocol for pushing and retrieving artifacts from registries.</li><li><strong>Runtime Specification</strong> — Describes how artifacts are executed; this is less relevant in the context of data packaging.</li></ul><p>Each OCI artifact consists of:</p><ul><li><strong>Config Blob</strong> — In EO contexts, this can hold structured metadata such as spatial extent, temporal coverage, data lineage, or sensor type.</li><li><strong>Layers</strong> — The actual data payloads, stored as compressed binary blobs and addressed via cryptographic hashes. Layers may be organized by space, time, modality, or processing level.</li><li><strong>Manifest</strong> — The linking structure that ties together the config and layers. It specifies digests, media types, and ordering.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*fHFog97v9PyoSiPbfie3Wg.png" /><figcaption>OCI artifacts example</figcaption></figure><p>Artifacts can be published, discovered, pulled, and verified using standard tools such as the ORAS CLI, Docker, or compatible SDKs. This introduces a robust, interoperable mechanism for data delivery — supporting structured metadata, referrers, artifact linking, and layered reuse to promote modularity and efficiency.</p><h3>Comparing Delivery Patterns</h3><p>While the technical structure of OCI artifacts addresses the challenge of reproducible packaging, real-world data sharing rarely happens in isolation. It is shaped by organizational constraints, access policies, deployment topologies, and human workflows. In other words, packaging alone is not enough — it must align with how datasets are actually shared and consumed across teams and systems.</p><p>Four dominant sharing models can currently be identified:</p><ul><li><strong>Direct team-based access</strong> — Data is made available through shared storage within a trusted environment, such as mounted network drives or shared object storage buckets.</li><li><strong>Presigned URLs</strong> — Temporary, expiring links are issued to allow time-limited access to specific files or folders, without exposing broader storage permissions.</li><li><strong>Peer-to-peer staging</strong> — The consumer sets up their own storage space like a bucket, and the provider delivers data by copying or syncing files into it, typically using a loose folder structure.</li><li><strong>Scoped credential delegation </strong>— Access is granted through temporary, limited-scope credentials, often using mechanisms like AWS STS or OAuth2-based delegation flows.</li></ul><p>OCI-based packaging can be viewed as a structured enhancement of peer-to-peer staging. When you request an OCI artifact from a registry, the process resembles a “push” from that OCI registry into your (local or remote) OCI runtime — a controlled, versioned, and verifiable transfer that replaces the loosely defined copy-and-paste semantics of traditional file sharing.</p><h3>Evaluation in Practice</h3><p>To evaluate this approach, we applied OCI packaging to the Panoptic Agricultural Satellite Time Series (PASTIS) <a href="https://github.com/VSainteuf/pastis-benchmark">benchmark dataset</a>, which integrates Sentinel-1 radar, Sentinel-2 optical, SPOT very high-resolution (VHR) imagery, and semantic labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*syraAGMDSkEOlbuoHmP73Q.png" /><figcaption>Structure of the PASTIS benchmark dataset</figcaption></figure><p>Two intentionally contrasting partitioning strategies were explored:</p><ul><li><strong>PASTIS-2433</strong>: The dataset was split into 2,433 per-patch subsets. Each patch was packaged into a separate TAR archive and added as an individual layer in a single OCI artifact. This resulted in an artifact with 2,433 layers, each approximately 30–35 MB in size. Metadata for each patch was embedded in the config blob.</li><li><strong>PASTIS-t4</strong>: The dataset was partitioned into four large, tile-based subsets, each covering a distinct spatial region. Each tile was packaged as a separate TAR-based layer within the artifact, with each layer approximately 15–20 GB in size. Tile-level metadata was recorded in the config blob.</li></ul><p>The benchmark results, initial setup instructions, and partitioning scripts have been made publicly available via the <a href="https://research.versioneer.at/paper-oci-supply-chain">Versioneer research site</a>.</p><p>We published these artifacts to multiple OCI-compatible registries and confirmed that, even under realistic conditions, they could be reliably pushed, pulled, deduplicated, and inspected using standard OCI tooling. Large layers (20 GB and above) were accepted without issue. Thousands of small layers were handled efficiently. Metadata annotations remained intact, and content integrity was successfully verified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*P-mjm-iF6Frja_yofPiZKw.png" /><figcaption>Results from OCI Experiments</figcaption></figure><p>These experiments demonstrate that OCI-based packaging can accommodate both fine-grained and coarse-grained partitioning strategies. Importantly, integrity checks, digest validation, and artifact-level attestations were all supported and reproducible — capabilities that are essential in workflows where trust, traceability, and regulatory compliance are required.</p><h3>From Files to Packages</h3><p>These findings resonate with a broader trend across the data ecosystem:</p><blockquote>There is an increasing demand for materialized and curated datasets — well-defined, harmonized data packages that are immediately usable for downstream analysis or model training.</blockquote><p>Platforms like <a href="https://huggingface.co/">Hugging Face Datasets</a> have demonstrated the effectiveness of this paradigm by offering versioned datasets with structured metadata and reproducibility guarantees. Domain-specific initiatives like the <a href="https://eotdl.com/">Earth Observation Training Data Lab</a> (EOTDL) are now emerging to bring similar capabilities to the EO community.</p><p>The proposed approach leveraging OCI artifacts for packaging doesn’t replace files. It wraps them. It augments them. It provides a framework for delivery, provenance, and collaboration. It respects the legacy of file-based workflows while enabling the structure and integrity that the EO data supply chain increasingly demands.</p><p>With OCI artifacts, a standard, portable, and verifiable method exists to distribute EO datasets across domains, workflows, and infrastructures. When combined with metadata discovery via STAC and delivered through existing tooling, data is not just discoverable, but also distributable and reproducible in its delivery.</p><p>As the OCI ecosystem continues to evolve, new tooling — ranging from general-purpose clients like <a href="https://oras.land/">ORAS</a> and Docker to domain-specific interfaces — will become increasingly relevant for Earth Observation workflows. At the same time, community-driven efforts to formalize metadata conventions, annotation keys, and media types will be essential to ensure semantic interoperability across domains. The machine learning community is already advancing similar OCI-based standards for model packaging, reflecting a broader shift toward reproducible, domain-aware artifact ecosystems.</p><p>Looking further ahead, the foundational principles of OCI — layered storage, content-addressable integrity, and manifest-based composition — position registries to become more than distribution endpoints. With support for partial access via HTTP range requests, and growing adoption of formats like GeoParquet, Zarr, and Cloud-Optimized GeoTIFF, OCI-based infrastructure could serve as a general-purpose, versioned storage backend for dynamic, queryable data assets.</p><p>This shift opens the door to building future-ready data supply chains — where structured metadata, reproducible packaging, and scalable access converge to support transparent, trusted, and efficient Earth data collaboration at scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=78c83fe81c3d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop shipping raw files. Start shipping data packages.]]></title>
            <link>https://medium.com/@stefan.achtsnit_41940/stop-shipping-raw-files-start-shipping-data-packages-ce5aa3fe0a5c?source=rss-3bee25fb233c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce5aa3fe0a5c</guid>
            <category><![CDATA[layered-data-storage]]></category>
            <category><![CDATA[supply-chain-data]]></category>
            <category><![CDATA[oci-artifacts]]></category>
            <dc:creator><![CDATA[Stefan Achtsnit]]></dc:creator>
            <pubDate>Mon, 19 May 2025 17:39:12 GMT</pubDate>
            <atom:updated>2025-05-19T18:40:57.929Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*Y1YkCdgTNCxd_6OUDBvEng.png" /></figure><p>Ever struggled to grant fine-grained access to your data lake or S3 bucket?<br><strong>Data supply chains need more than downloads.</strong> Move beyond credentials — data distribution shouldn’t be a permission problem.</p><p>The <strong>Open Container Initiative (OCI)</strong> solved this challenge for code. With open standards, battle-tested tooling, and ubiquitous support across cloud and on-prem environments, it’s now ready for <strong>data</strong> too.</p><p><strong>OCI artifacts</strong> bring versioning, integrity, and reuse to data pipelines — making datasets <strong>discoverable, verifiable, and ready to use</strong>, just like containers make software easy to deploy.</p><p>Ideal for <strong>scalable analytics</strong> and <strong>reproducible ML inference</strong> — from <strong>Earth Observation</strong> to <strong>any kind of AI workload</strong>.</p><p>📄 Read the paper: <a href="https://research.versioneer.at/paper-oci-supply-chain">https://research.versioneer.at/paper-oci-supply-chain</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce5aa3fe0a5c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>