<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tigris Data on Medium]]></title>
        <description><![CDATA[Stories by Tigris Data on Medium]]></description>
        <link>https://medium.com/@tigris-data?source=rss-49b902974434------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*J28tngVx-EMKwTumqzdMFA.png</url>
            <title>Stories by Tigris Data on Medium</title>
            <link>https://medium.com/@tigris-data?source=rss-49b902974434------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 10:48:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tigris-data/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using Hugging Face datasets with Tigris]]></title>
            <link>https://tigris-data.medium.com/using-hugging-face-datasets-with-tigris-0507d44575ce?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/0507d44575ce</guid>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tigris]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Tue, 29 Jul 2025 00:00:57 GMT</pubDate>
            <atom:updated>2025-08-28T14:41:45.935Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*co7gIt9NZHqXXqjQ.jpg" /></figure><p>One of the most popular ways to share datasets is via <a href="https://huggingface.co/datasets">Hugging Face’s dataset platform</a>. You can even stream larger-than-laptop datasets, but there are no guarantees on throughput nor availability. When you’re developing a toy model, this might not be an issue. But as you mature your model, and combine your custom datasets with public datasets, it’s critical to save your own copy.</p><p>The ability to reproduce the state of your model at a given time has become critical, and even legally required, as models are integrated into healthcare, legal, and other compliance heavy domains. Why did the AI agree to <a href="https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-ex1">sell a car for $1</a>? Or <a href="https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/">delete a production database</a>?</p><p>As we develop models, they’re going to make mistakes. It’s challenging to debug across scattered datasets, especially public ones outside your control. Centralizing your datasets in a common store is a good first step on your way to full dataset version control. Just make sure you think about additional costs- HuggingFace dataset streaming is free, but private stores can quickly rack up egress fees.</p><p>Today we’re going to learn how to import <a href="https://huggingface.co/datasets">Hugging Face datasets</a> into Tigris so that you can use them for whatever you need.</p><blockquote><strong>NOTE</strong></blockquote><blockquote>In production workloads, we recommend that you use <a href="https://www.tigrisdata.com/docs/libraries/lancedb/">LanceDB’s multimodal lakehouse</a> to store your training datasets; but if you’re just getting started then this is way more than enough.</blockquote><h3>Prerequisites</h3><p>Here’s what you need to get started:</p><ul><li>A local Python development environment (our blog has a guide on <a href="https://www.tigrisdata.com/blog/dev-containers-python/">using development containers</a> to set one up).</li><li>A Tigris account from <a href="https://storage.new/">storage.new</a>.</li><li>A Tigris bucket and access keys with the Editor permission on that bucket.</li></ul><blockquote>For instructions on setting up your environment manually, visit our original post at: <a href="https://www.tigrisdata.com/docs/get-started/?utm_source=blog-post-huggingface-datasets">https://www.tigrisdata.com/blog/huggingface-datasets/</a></blockquote><h3>Importing a dataset</h3><p>Now let’s import the <a href="https://huggingface.co/datasets/mlabonne/FineTome-100k">FineTome-100k</a> dataset to Tigris. The process is surprisingly straightforward thanks to Hugging Face datasets’ built-in support for S3-compatible storage.</p><p>First, let’s look at the helper module that sets up our Tigris connection:</p><pre>import os<br>import s3fs<br>from dotenv import load_dotenv<br>from typing import Dict, Tuple<br><br>def setup() -&gt; Tuple[Dict[str, str], s3fs.S3FileSystem]:<br>    load_dotenv()<br><br>    storage_options = {<br>        &quot;key&quot;: os.getenv(&quot;AWS_ACCESS_KEY_ID&quot;),<br>        &quot;secret&quot;: os.getenv(&quot;AWS_SECRET_ACCESS_KEY&quot;),<br>        &quot;endpoint_url&quot;: os.getenv(&quot;AWS_ENDPOINT_URL_S3&quot;),<br>    }<br><br>    # Create the S3 filesystem<br>    fs = s3fs.S3FileSystem(**storage_options)<br><br>    # Test write access<br>    bucket_name = os.getenv(&quot;BUCKET_NAME&quot;)<br>    fs.write_text(f&quot;/{bucket_name}/test.txt&quot;, &quot;this is a test&quot;)<br>    fs.rm(f&quot;/{bucket_name}/test.txt&quot;)<br><br>    return (storage_options, fs)</pre><p>The import script uses Hugging Face datasets’ save_to_disk method with our Tigris storage options:</p><pre>import os<br>import tigris<br>from datasets import load_dataset<br>from dotenv import load_dotenv<br><br>def main():<br>    storage_options, fs = tigris.setup()<br><br>    bucket_name = os.getenv(&quot;BUCKET_NAME&quot;)<br>    dataset_name = os.getenv(&quot;DATASET_NAME&quot;)<br><br>    # Load the dataset from Hugging Face<br>    dataset = load_dataset(dataset_name, split=&quot;train&quot;)<br><br>    # Save directly to Tigris<br>    dataset.save_to_disk(<br>        f&quot;s3://{bucket_name}/datasets/{dataset_name}&quot;,<br>        storage_options=storage_options<br>    )<br><br>    print(f&quot;Dataset {dataset_name} is now in Tigris at {bucket_name}/datasets/{dataset_name}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Run the import script:</p><pre>uv run scripts/import-to-tigris.py</pre><p>That’s it! The dataset is now stored in Tigris and ready to use from anywhere.</p><h3>Reading and processing datasets from Tigris</h3><p>Once your dataset is in Tigris, you can load it from anywhere using the same storage options. Here’s an example that loads the dataset, applies a filter, and saves the filtered version back to Tigris:</p><pre>import os<br>import tigris<br>from datasets import load_from_disk<br><br>def remove_blue(row):<br>    &quot;&quot;&quot;<br>    Example transformation that removes conversations mentioning &quot;blue&quot;.<br>    You can implement any filtering or transformation logic here.<br>    &quot;&quot;&quot;<br>    for conv in row[&#39;conversations&#39;]:<br>        if &quot;blue&quot; in conv[&#39;value&#39;]:<br>            return False  # remove the row<br>    return True  # keep the row<br><br>def main():<br>    storage_options, fs = tigris.setup()<br><br>    bucket_name = os.getenv(&quot;BUCKET_NAME&quot;)<br>    dataset_name = os.getenv(&quot;DATASET_NAME&quot;)<br><br>    # Load dataset from Tigris<br>    dataset = load_from_disk(<br>        f&quot;s3://{bucket_name}/datasets/{dataset_name}&quot;,<br>        storage_options=storage_options<br>    )<br><br>    # Apply filtering<br>    filtered_ds = dataset.filter(remove_blue)<br><br>    # Save filtered dataset back to Tigris<br>    filtered_ds.save_to_disk(<br>        f&quot;s3://{bucket_name}/no-blue/{dataset_name}&quot;,<br>        storage_options=storage_options<br>    )<br><br>    print(f&quot;Filtered dataset saved to {bucket_name}/no-blue/{dataset_name}&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Run the processing script:</p><pre>uv run scripts/read-from-tigris.py</pre><h3>Conclusion</h3><p>You did it! Your copy of those datasets are safely stored in your own bucket. You have centralized your datasets and are on the path to versioning them.</p><p>We love Hugging Face for providing models and datasets to the world for free, and we want you to keep using them to develop your own models. However, as you start maturing and complying with regulations, making your own copy ensures no one tampers with the data. And that your bandwidth won’t suddenly drop mid training job, or lag across regions. Tigris dynamically places your datasets where you need them so you can scale fearlessly to any cloud with an internet connection.</p><figure><a href="https://www.tigrisdata.com/docs/get-started/?utm_source=blog-post-huggingface-datasets"><img alt="" src="https://cdn-images-1.medium.com/max/806/1*RtclM1fdonqxIG6mJbm5qA.png" /></a></figure><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/huggingface-datasets/"><em>https://www.tigrisdata.com</em></a><em> on July 29, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0507d44575ce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We made unhateable IAM. Here’s how to use it.]]></title>
            <link>https://tigris-data.medium.com/we-made-unhateable-iam-heres-how-to-use-it-aa0a0df92684?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa0a0df92684</guid>
            <category><![CDATA[iam-policy]]></category>
            <category><![CDATA[features]]></category>
            <category><![CDATA[iam-roles]]></category>
            <category><![CDATA[tigris]]></category>
            <category><![CDATA[updates]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Thu, 17 Jul 2025 00:00:03 GMT</pubDate>
            <atom:updated>2025-08-27T20:10:19.300Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_EMCheQ6ZAw-WelRrKNDQ.png" /></figure><p>We made IAM you can’t hate. Simplified permissions, an easy way to list access keys attached to a given policy, and a VSCode style editor experience that feels like your local development environment: all in our new IAM Policy Builder in the Tigris Console.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZabMm5uhZfQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZabMm5uhZfQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZabMm5uhZfQ%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4642f6cab4910c04c55d056e89a3f75e/href">https://medium.com/media/4642f6cab4910c04c55d056e89a3f75e/href</a></iframe><p>It should be a code smell that there are <a href="https://github.com/iann0036/iamlive">several</a> <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/access-analyzer-policy-generation.html">tools</a> for constructing <a href="https://en.wikipedia.org/wiki/Principle_of_least_privilege">least privilege policies</a>, all of which require you to have overprivileged entities and then cut back privileges, rather than making a least privilege policy from the start. I can personally list eight different ways to give access to an S3 bucket; there may be more. Why is it so hard to do the right thing?</p><p>The feedback loop of IAM policy development leads to even Senior Engineers™ having reactions like this:</p><figure><img alt="Tweet: If one had to properly configure AWS IAM roles to get into Heaven, everyone would be in Hell." src="https://cdn-images-1.medium.com/max/590/1*pivtMYDR9Mnx9f2thdBrjw.jpeg" /><figcaption><a href="https://x.com/glcst/status/1909957508682129883">Link to the source</a></figcaption></figure><figure><img alt="Tweet: Sorry, you don’t have permission to do this." src="https://cdn-images-1.medium.com/max/595/1*z3y7OlSG7cT_hBCzt8mIhw.jpeg" /><figcaption><a href="https://bsky.app/profile/cr3ative.co.uk/post/3ltpfm2hsq22w">Link to the source</a></figcaption></figure><figure><img alt="Tweet: user, role, permission, policy, policy document, resource — these entities are poorly defined and either more or less complex than they need to be, haven’t decided which. The explosion of permission names and the laughable practice of using wildcard matching in policies suggests that bit is in desperate need of simplification, perhaps by decomposing permissions into a tuple of entity type and a tight set of action verbs." src="https://cdn-images-1.medium.com/max/592/1*tCmjAnBisBWsT1_tdfus5w.jpeg" /><figcaption><a href="https://triangletoot.party/@donaldball/114857960806541467">Link to the source</a></figcaption></figure><p>How do you simplify IAM whilst maintaining a strong security posture? We started by removing two of the easiest to misuse components: IAM Users and IAM Roles. On Tigris, you’re a member of an organization, but that’s as far as “users” go. You don’t assume roles to get temporary credentials that you then need to refresh in the middle of longer running jobs. We don’t have compute instances, so we don’t need instance profiles for policy attachments to access your data from an instance. It’s only access keys with policy attachments. That’s the beauty of using a product that does one thing, and does it well.</p><p>“But static keys will get leaked and my data ransomed!” What you really want to do is ensure that a given access key has only enough permissions to get its job done for a given amount of time and no more. And then that key loses access automatically. It doesn’t have to be so complex, truly.</p><h3>Example: Running a Training Job on a Newer Cloud Provider</h3><p>A Suspicious Person™ in a trench coat shuffles past you on the streets of San Francisco and surreptitiously looks you up and down, sees the wad of computer cables poking from your backpack, and pulls you aside: “GPUs for 35 cents an hour?” They whip open their trench coat and dazzle you with a full selection of cloud prices so cheap you swear that they’re a front for the mafia. But you work at a startup, and runway is runway. How can you take advantage of these low, low prices without exposing yourself to hackery and leakery?</p><p>Do you load your data into their storage? Surely not, if they even have storage. You don’t want anything especially long-lived on those trench-coat GPUs, and you certainly don’t want to initially overprovision access to then cut down to least privilege. It should be tightly scoped from the start: mint access keys with minimum permissions to do the job, then delete them when you’re done.</p><p>So we do that: your training job on those absurdly affordable GPUs gets permission to:</p><ul><li>Read only access to one dataset so that the job cannot modify any datasets in use by other jobs</li><li>Read only access to the base model collection so that the job can’t corrupt any of your models</li><li>Write only access to the finetuned model collection so that all the job can do when it’s done is submit its work</li></ul><p>If an access key with these permissions is leaked, your attack surface is still quite small: your precious finetuned models are safe. The one dataset allocated to the job and base model collection can be read, but not altered. Other jobs are unaffected. Let’s take a look at building such a policy for your training job in the new IAM Policy Builder, and adding a time based restriction:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fijk9ZZdxeXA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dijk9ZZdxeXA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fijk9ZZdxeXA%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae038482551457bcd3370e62c2205815/href">https://medium.com/media/ae038482551457bcd3370e62c2205815/href</a></iframe><p>And here’s the policy JSON for that:</p><pre>{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Sid&quot;: &quot;WikipediaReadOnly&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::contoso-training-datasets-wikipedia-2025-07-01&quot;,<br>        &quot;arn:aws:s3:::contoso-training-datasets-wikipedia-2025-07-01/*&quot;<br>      ]<br>    },<br>    {<br>      &quot;Sid&quot;: &quot;BaseModelsReadOnly&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::contoso-base-models&quot;,<br>        &quot;arn:aws:s3:::contoso-base-models/*&quot;<br>      ]<br>    },<br>    {<br>      &quot;Sid&quot;: &quot;FinetunedModelsWrite&quot;,<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [<br>        &quot;s3:GetObject&quot;,<br>        &quot;s3:ListBucket&quot;,<br>        &quot;s3:PutObject&quot;,<br>        &quot;s3:PutObjectAcl&quot;,<br>        &quot;s3:AbortMultipartUpload&quot;,<br>        &quot;s3:ListMultipartUploadParts&quot;,<br>        &quot;s3:CompleteMultipartUpload&quot;<br>      ],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::contoso-finetuned-models&quot;,<br>        &quot;arn:aws:s3:::contoso-finetuned-models/*&quot;<br>      ]<br>    }<br>  ]<br>}</pre><p>When you make policies in our editor, you can use other policies as a starting point so that you don’t have to summon each statement into existence with sheer might of your left click button. We even give you some of the creature comforts that you get in your editor of choice: error squiggles to let you know when something is wrong, syntax highlighting so you can visually distinguish the brackets, and schema validation so you can’t create a policy that doesn’t work.</p><p>Imagine this same situation, but with the AWS IAM structure.</p><p>And even in correctly and fully implementing this complex process, there’s still inconsistencies that can bite back: tags based on policies on iam:PassRole are known to be unreliable. Complex doesn’t always mean secure.</p><h3>Burying the Lede: Linked Access Keys</h3><p>Being able to build policies with a combination of JSON and button clicking is nice, but we’re burying the lede here: we added something you cannot easily do in other storage providers. On Tigris, you can list all the access keys that have a given policy attached. Then if you do have a key leak or want to investigate the permissions of keys you found on some vintage cron job, it’s all there in the Dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_yP_DdQ3AqFJnis7ozNIHA.png" /></figure><p>This is nontrivial to assemble on other platforms due to the overhead involved with IAM Roles and IAM Users. It’s possible but not the most pleasant. In case you need it, here’s the incantation for AWS:</p><pre>POLICY_ARN=&quot;arn:aws:iam::123456789012:policy/YourPolicyName&quot;<br><br>for user in $(aws iam list-entities-for-policy \<br>    --policy-arn &quot;$POLICY_ARN&quot; \<br>    --query &#39;PolicyUsers[].UserName&#39; --output text); do<br>  echo &quot;User: $user&quot;<br>  aws iam list-access-keys --user-name &quot;$user&quot; \<br>    --query &#39;AccessKeyMetadata[].AccessKeyId&#39; --output text<br>done</pre><p>This really helps when your production environment looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vcOD5uSDFpTA9CQ4f0SCTw.png" /></figure><h3>Conclusion</h3><p>IAM doesn’t have to be hellish; it can be just as good as your local development setup. Tigris’ IAM Policy Builder blends a tile based GUI with a VSCode like editor experience to give you the best of both worlds. You can start with a pasted example policy, customize it, and know what you’re writing actually works. Give it a try, we’re sure that you’ll come to learn to love IAM in a way you never have before.</p><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/iam-policy-builder/"><em>https://www.tigrisdata.com</em></a><em> on July 17, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa0a0df92684" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with WarpStream on Tigris]]></title>
            <link>https://tigris-data.medium.com/getting-started-with-warpstream-on-tigris-d0a129f99c15?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0a129f99c15</guid>
            <category><![CDATA[tigris]]></category>
            <category><![CDATA[warpstream]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Thu, 10 Jul 2025 00:00:11 GMT</pubDate>
            <atom:updated>2025-08-27T15:30:17.581Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5niWDSH8H5Wzeb_8.jpg" /></figure><p><a href="https://www.warpstream.com/">WarpStream</a> is a <a href="https://kafka.apache.org/">Kafka®</a>-compatible streaming platform that stores data directly in object storage instead of relying on local disks. This architecture unlocks virtually unlimited retention and simplifies operations. Because WarpStream is zone-aligned, it avoids costly inter-AZ network traffic by ensuring that agents and object storage communicate within the same availability zone. However, in multi-cloud architectures where clients and agents communicate across clouds, egress fees from cross-cloud traffic can quickly dominate your cloud bill as you scale.</p><p>Tigris is a globally distributed, multi-cloud object storage platform with native S3 API support and zero egress fees. It dynamically places data in the region where it’s being accessed-eliminating cross-cloud data transfer costs without sacrificing on performance.</p><p>By combining WarpStream with Tigris, you get a bottomless, durable, and globally aware message queue. Your data is stored efficiently, close to where it’s consumed, without incurring hidden transfer fees or needing to plan for regional bucket placement.</p><p>Before we get started, let’s cover the moving parts:</p><h3>The Moving Parts</h3><p>Apache Kafka is a durable, distributed messaging system. In Kafka, Producers write records into Topics hosted by Brokers that are read by Consumers. Used by 80% of Fortune 500 companies, Kafka connects applications and data stores using a real-time publish/subscribe model with strong durability guarantees. One of the main differences between Kafka and a traditional queueing system is <em>persistence.</em> Data written to Kafka is retained, even after it is read by a Consumer, which enables interesting patterns such as multiple consumers reading from the same topic. The main downside is that Kafka relies on local storage, meaning that your Kafka Brokers require fast disks and intra-cluster replication to ensure durability and availability.</p><p>WarpStream is fully compatible with the Kafka protocol, but it fundamentally reimagines the architecture. Instead of using local disks, it writes all data directly to object storage, completely decoupling compute and storage. While Apache Kafka 3.6.0 introduced Tiered Storage to enable more flexible scaling of compute and storage, it stops short of full decoupling-brokers still require local disk, retain state, manage replication, and serve all client traffic.</p><p>With WarpStream, the compute layer is completely stateless. The data does not need to be replicated between nodes since durability is handled entirely by the object store. This architecture is much simpler to operate: you don’t need to set up and manage all of Kafka’s dependencies (Zookeeper, the JVM, etc). WarpStream also ships an easy to use command line utility that helps you administrate your message queue and test functionality.</p><p>In addition, we’ll use Docker, the universal package format for the Internet. Docker lets you put your application and all its dependencies into a container image so that it can’t conflict with anything else on the system.</p><p>Today we’re going to deploy a WarpStream cluster backed by Tigris into a Docker container so you can create your own bottomless durable message queue. This example will use Docker compose, but it will help you understand how to create your own broker so you can deploy it anywhere.</p><h3>Prerequisites</h3><p>Make sure you have the following installed on your computer:</p><ul><li><a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a> or another similar app like <a href="https://podman-desktop.io/">Podman Desktop</a>.</li><li>The <a href="https://aws.amazon.com/cli/">AWS CLI</a>.</li><li><a href="https://docs.warpstream.com/warpstream/reference/cli-reference">Warpstream’s CLI</a>.</li></ul><p>You will need the following accounts:</p><ul><li>A Tigris account from <a href="https://storage.new/">storage.new</a>.</li><li>A Warpstream account from <a href="http://console.warpstream.com/login">console.warpstream.com</a>.</li></ul><h3>Building a compose file</h3><p>Clone the <a href="https://github.com/tigrisdata-community/warpstream-tigris">warpstream-tigris</a> demo repo to your laptop and open it in your favourite editor, such as VS Code.</p><pre>services:<br>  warp:<br>    # Grab the latest copy of the warpstream agent for your computer<br>    image: public.ecr.aws/warpstream-labs/warpstream_agent:latest<br>    # Run warpstream in &quot;playground&quot; mode for testing<br>    command:<br>      - playground<br>      - -advertiseHostnameStrategy<br>      - custom<br>      - -advertiseHostnameCustom<br>      - warp<br>    environment:<br>      # this is a no-op as it will default on the custom advertised hostname defined above, but you can change this if you want to use a different hostname with Kafka<br>      - WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE=warp<br>    healthcheck:<br>      # Wait for the Agent to finish setting up the demo before marking it as healthy<br>      # to delay the diagnose-connection command from running for a few seconds.<br>      test: [&quot;CMD&quot;, &quot;sh&quot;, &quot;-c&quot;, &quot;sleep 10&quot;]<br>      interval: 5s<br>      timeout: 15s<br>      retries: 5</pre><p>Open a new terminal in your development container and make sure WarpStream is up and running:</p><pre>warpstream kcmd --bootstrap-host warp --type diagnose-connection</pre><p>This should return output like the following:</p><pre>running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092<br><br><br>Broker Details<br>---------------<br>  warp:9092 (NodeID: 1547451680) [playground]<br>    ACCESSIBLE ✅<br><br><br>GroupCoordinator: warp:9092 (NodeID: 1547451680)<br>    ACCESSIBLE ✅</pre><p>Excellent! Create a new topic with warpstream kcmd:</p><pre>warpstream kcmd --bootstrap-host warp --type create-topic --topic hello</pre><p>This should return output like the following:</p><pre>running create-topic sub-command with bootstrap-host: warp and bootstrap-port: 9092<br><br>created topic &quot;hello&quot; successfully, topic ID: MQAAAAAAAAAAAAAAAAAAAA==</pre><p>Perfect! Now let’s make it work with Tigris. Create a .env file in the root of the repository:</p><pre>cp .env.example .env<br>code .env</pre><p>Create a new bucket at <a href="https://storage.new/">storage.new</a> in the Standard access tier. Copy its name down into your notes. Create a new <a href="https://storage.new/accesskey">access key</a> with Editor permissions for that bucket. Copy the environment details into your.env file:</p><pre>## Tigris credentials<br>AWS_ACCESS_KEY_ID=tid_access_key_id<br>AWS_SECRET_ACCESS_KEY=tsec_secret_access_key<br>AWS_ENDPOINT_URL_S3=https://t3.storage.dev<br>AWS_ENDPOINT_URL_IAM=https://iam.storage.dev<br>AWS_REGION=auto</pre><p>Then fill in your Warpstream secrets from the console, you need the following:</p><ul><li>Cluster ID from the virtual clusters list (begins with vci_)</li><li>Bucket URL (explained below)</li><li>Agent key from the agent keys page for that virtual cluster (begins with aks_)</li><li>Cluster region from the admin panel (such as us-east-1)</li></ul><p>If your bucket is named xe-warpstream-demo, your bucket URL should look like this:</p><pre>s3://xe-warpstream-demo?region=auto&amp;endpoint=https://t3.storage.dev</pre><p>Altogether, put these credentials in your .env file:</p><pre>## Warpstream credentials<br>WARPSTREAM_AGENT_KEY=aks_agent_key<br>WARPSTREAM_BUCKET_URL=&#39;s3://xe-warpstream-demo?region=auto&amp;endpoint=https://t3.storage.dev&#39;<br>WARPSTREAM_DEFAULT_VIRTUAL_CLUSTER_ID=vci_cluster_id<br>WARPSTREAM_REGION=us-east-1</pre><p>Edit your docker-compose.yaml file to load the .env file and start warpstream in agent mode:</p><pre># docker-compose.yaml<br>services:<br>  warp:<br>    image: public.ecr.aws/warpstream-labs/warpstream_agent:latest<br>    command:<br>      - agent<br>    environment:<br>      WARPSTREAM_DISCOVERY_KAFKA_HOSTNAME_OVERRIDE: warp<br>      WARPSTREAM_DISCOVERY_KAFKA_PORT_OVERRIDE: 9092<br>      WARPSTREAM_REQUIRE_AUTHENTICATION: &quot;false&quot;<br>    env_file:<br>      - .env</pre><p>Then restart your development container with control/command shift-p “Dev Containers: Rebuild Container”. Test the health of your Broker:</p><pre>warpstream kcmd --bootstrap-host warp --type diagnose-connection</pre><p>You should get output like this:</p><pre>running diagnose-connection sub-command with bootstrap-host: warp and bootstrap-port: 9092<br><br><br>Broker Details<br>---------------<br>  warp:9092 (NodeID: 1415344910) [warpstream-unset-az]<br>    ACCESSIBLE ✅<br><br><br>GroupCoordinator: warp:9092 (NodeID: 1415344910)<br>    ACCESSIBLE ✅</pre><p>It’s working! Create a topic and publish some messages:</p><pre>warpstream kcmd --bootstrap-host warp --type create-topic --topic hello<br>warpstream kcmd --bootstrap-host warp --type produce --topic hello --records &quot;world,,world&quot;</pre><p>This should create the topic hello and two messages with world in them. You should get output like this:</p><pre>result: partition:0 offset:0 value:&quot;world&quot;<br>result: partition:0 offset:1 value:&quot;world&quot;</pre><p>Now let’s read them back:</p><pre>warpstream kcmd --bootstrap-host warp --type fetch --topic hello --offset 0</pre><p>You should get output like this:</p><pre>consuming topic:&quot;hello&quot; partition:0 offset:0<br>result: partition:0 offset:0 key:&quot;hello&quot; value:&quot;world&quot;<br>result: partition:0 offset:1 key:&quot;hello&quot; value:&quot;world&quot;</pre><p>It works! You’ve successfully put data into a queue and fetched it back from the queue. From here you can connect to your broker on host warp and port 9092. All your data is securely backed by Tigris and you can access it from anywhere in the world.</p><figure><a href="https://www.tigrisdata.com/docs/get-started/?utm_source=blog-post-warpstream"><img alt="" src="https://cdn-images-1.medium.com/max/806/1*CchBOWhuHJEZg0csPzZYLw.png" /></a></figure><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/warpstream/"><em>https://www.tigrisdata.com</em></a><em> on July 10, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0a129f99c15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small Objects, Big Gains: Benchmarking Tigris Against AWS S3 and Cloudflare R2]]></title>
            <link>https://tigris-data.medium.com/small-objects-big-gains-benchmarking-tigris-against-aws-s3-and-cloudflare-r2-d8bddc26df0b?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8bddc26df0b</guid>
            <category><![CDATA[benchmarking]]></category>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[upgrade]]></category>
            <category><![CDATA[tigris]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Tue, 08 Jul 2025 00:00:56 GMT</pubDate>
            <atom:updated>2025-08-27T15:17:59.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1jdZLp_2CY09oRFVhkVdw.png" /></figure><blockquote><strong>NOTE</strong></blockquote><blockquote>Due to a high volume of figures, tables, and graphs which do not format nicely on Medium.com, for the best reading experience please read this post on the Tigris blog at: <a href="https://www.tigrisdata.com/blog/benchmark-small-objects/">https://www.tigrisdata.com/blog/benchmark-small-objects/</a></blockquote><p>One of Tigris’s standout capabilities is its performance when storing and retrieving small objects. To quantify this advantage, we benchmarked Tigris against two popular object stores-AWS S3 and Cloudflare R2-and found that Tigris consistently delivers higher throughput and lower latency. These gains let you use a single store for everything from tiny payloads to multi-gigabyte blobs without sacrificing efficiency.</p><p>Under the hood, Tigris accelerates small-object workloads by (i) inlining very small objects inside metadata records, (ii) coalescing adjacent keys to reduce storage overhead, and (iii) caching hot items in an on-disk, LSM-backed cache.</p><h3>Summary</h3><p>Our benchmarks reveal that Tigris significantly outperforms both AWS S3 and Cloudflare R2 for small object workloads. Our benchmarks show Tigris achieves <strong>sub-10ms</strong> read latency and <strong>sub-20ms</strong> write latency, while sustaining <strong>4 x throughput</strong> than S3 and <strong>20 x throughput</strong> than R2 for both operations.</p><p>To ensure our findings are reproducible, we outline the full benchmarking methodology and provide links to all artifacts.</p><figure><a href="https://www.tigrisdata.com/docs/get-started/?utm_source=blog-post-benchmarks"><img alt="" src="https://cdn-images-1.medium.com/max/877/1*6iNXrxSJMmjFJ7IhDV_THA.png" /></a></figure><h3>Benchmark Setup</h3><p>We used the <a href="https://en.wikipedia.org/wiki/YCSB">Yahoo Cloud Serving Benchmark (YCSB)</a> to evaluate the three systems. We <a href="https://github.com/pingcap/go-ycsb/pull/307">added support</a> for S3-compatible object storage systems (such as Tigris and Cloudflare R2), which was merged shortly after publish.</p><p>All experiments ran on a neutral cloud provider to avoid vendor-specific optimizations. Table 1 summarizes the test instance:</p><p><em>Table 1: Benchmark host configuration.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*MsABWIdY8ewiSKFWXbHoFg.jpeg" /><figcaption><em>Table 1: Benchmark host configuration.</em></figcaption></figure><h4>YCSB Configuration</h4><p>We benchmarked a dataset of 10 million objects, each 1 KB in size. You can view our configuration in the <a href="https://github.com/tigrisdata-community/ycsb-benchmarks">tigrisdata-community/ycsb-benchmarks</a> GitHub repo, specifically at <a href="https://github.com/tigrisdata-community/ycsb-benchmarks/blob/main/results/10m-1kb/workloads3">results/10m-1kb/workloads3</a>.</p><p>Our buckets were placed in the following regions per provider:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*GRy0jyfnKXiiaSaV908uKw.jpeg" /></figure><h3>Results</h3><p>Using YCSB we evaluated two distinct phases: (i) a bulk load of 10 million 1 KB objects and (ii) a mixed workload of one million operations composed of 80% reads and 20% writes.</p><h4>Loading 10 million objects</h4><p>Figure 1 (below) plots the end-to-end ingestion time. Tigris finishes the load in <strong>6711 s</strong>, which is roughly <strong>31 % faster than S3 (8826 s)</strong> and <strong>an order of magnitude faster than R2 (72063 s)</strong>.</p><p>Latency drives this gap. As shown in Figure 2, R2’s p90 PUT latency tops <strong>340 ms</strong> whereas Tigris stays below <strong>36 ms</strong> and S3 below <strong>38 ms</strong>. Table 2 summarizes the key statistics.</p><p><em>Table 2: Load-phase latency and throughput metrics.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wnY61SU32uQ7hqPbUuwV2g.jpeg" /><figcaption><em>Table 2: Load-phase latency and throughput metrics.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iu1RzzsWw29dYpvmnSsxjg.png" /><figcaption><em>Figure 1: Total load time for loading 10 M 1 KB objects.</em></figcaption></figure><p>R2 takes more than 300ms to write a single object which explains the slowness of the data load.</p><p>While comparing Tigris latency to S3, it is still better but not the same margin as compared to R2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tqp91hE23tExZBOqON-zSQ.png" /><figcaption><em>Figure 2: PUT p90 latency during load phase.</em></figcaption></figure><h3>1 million operations (20% write, 80% read)</h3><p>This is the <em>run</em> phase of the YCSB benchmark. As a reminder, it is a 20% write and 80% read workload totaling 1 million operations.</p><h4>Read throughput</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9yYes5Yie9q66f3HdyRKw.png" /><figcaption><em>Figure 3: Read throughput during mixed workload (Tigris vs R2).</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HDI8q5pin_dWyIm9zpFGBw.png" /><figcaption><em>Figure 4: Read throughput during mixed workload (Tigris vs S3).</em></figcaption></figure><p>Throughput traces for all three providers remain stable — useful for capacity planning — but the absolute rates diverge sharply. Tigris sustains <strong>≈3.3 k ops/s</strong>, nearly <strong>4 × S3 (≈ 892 ops/s)</strong> and <strong>20 × R2 (≈ 170 ops/s)</strong>. This headroom lets applications serve real-time workloads directly from Tigris.</p><h4>Read latency</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuGjw8dHPQN4F7ShWVJ5Mg.png" /><figcaption><em>Figure 5: Read p90 latency during mixed workload (Tigris vs R2).</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stbPPjr3_agK4m8t5IdWFw.png" /><figcaption><em>Figure 6: Read p90 latency during mixed workload (Tigris vs S3).</em></figcaption></figure><p>Latency follows the same pattern. Tigris keeps p90 below <strong>8 ms</strong>; S3 settles around <strong>42 ms</strong>, and R2 stretches beyond <strong>199 ms</strong>. At sub-10 ms, reads feel closer to a key-value store than a traditional object store.</p><h4>Write throughput</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f9wtJSHNRWS3kPbVfkXqgA.png" /><figcaption><em>Figure 7: Write throughput during mixed workload (Tigris vs R2).</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xiSS6XgUIGrVQJTu6JqgWg.png" /><figcaption><em>Figure 8: Write throughput during mixed workload (Tigris vs S3).</em></figcaption></figure><p>Write throughput shows the same spread. Tigris delivers <strong>≈ 828 ops/s</strong>, close to <strong>4 × S3 (224 ops/s)</strong> and <strong>20 × R2 (43 ops/s)</strong>, giving plenty of margin for bursty ingest pipelines.</p><h4>Write latency</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I9sHiUJn1O5TiFlojwl-GQ.png" /><figcaption><em>Figure 9: Write p90 latency during mixed workload (Tigris vs R2).</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBAUOpkB1nmhD2y_hUXVIA.png" /><figcaption><em>Figure 10: Write p90 latency during mixed workload (Tigris vs S3).</em></figcaption></figure><p>Write-side tail latency tracks proportionally: <strong>&lt; 17 ms</strong> for Tigris, <strong>≈ 41 ms</strong> for S3, and <strong>&gt; 680 ms</strong> for R2 — an order-of-magnitude gap that can make or break user-facing workloads.</p><p>To summarize:</p><p><em>Table 3: Read and throughput metrics.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*XOkamXgGuvaAsjucNMjdIA.jpeg" /></figure><p><em>Table 4: Update and throughput metrics.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*K6CHGEbBRSlp55MqcH6lJw.jpeg" /></figure><h3>Conclusion</h3><p>Tigris outperforms S3 and comprehensively outperforms R2 for small object workloads. The performance advantage stems from Tigris’s optimized architecture for small objects. While S3 and R2 struggle with high latency on small payloads (R2’s p90 PUT latency reaches 340ms), Tigris maintains consistent low latency through intelligent object inlining, key coalescing, and LSM-backed caching.</p><p>These results demonstrate that Tigris can serve as a unified storage solution for mixed workloads, eliminating the need to maintain separate systems for small and large objects. Whether you’re storing billions of tiny metadata files or streaming gigabytes of video data, Tigris delivers optimal performance across the entire object size spectrum.</p><p>You can find the full benchmark results in the <a href="https://github.com/tigrisdata-community/ycsb-benchmarks">ycsb-benchmarks</a> repository.</p><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/benchmark-small-objects/"><em>https://www.tigrisdata.com</em></a><em> on July 8, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8bddc26df0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Standardizing Python Environments with Development Containers]]></title>
            <link>https://tigris-data.medium.com/standardizing-python-environments-with-development-containers-fdac863a32b2?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdac863a32b2</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[containers]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Thu, 03 Jul 2025 00:00:10 GMT</pubDate>
            <atom:updated>2025-08-07T21:28:40.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*27G7VWfdwvdkA6E6Ojaekw.png" /></figure><p>If you’re working in AI, you’re probably working in Python. Maybe you have a webapp in whatever JS framework is popular right now, but most of the core tooling in AI is built in and around Python. So maybe it’s time for a Go programmer like me to figure out how the production Python gets made.</p><p>Last week I rediscovered <a href="https://containers.dev/">Development Containers</a>. When you use them, you do all your development in a container instead of on your machine directly. This container is defined using a<a href="https://containers.dev/implementors/json_schema/">devcontainer.json</a> file and when you create a development container it&#39;s rebuilt from scratch every time. This means that when you get your build working in development, it won&#39;t just work on your machine. It&#39;ll work on anyone&#39;s machine.</p><p>Having to use Python shouldn’t be that big of a deal, my first programming language was Python, but there’s one small problem that has resulted in me thinking that I have been cursed by an elder deity: Python environment management tools randomly break for me. I’ve never been able to figure out why this happens, but in the last three years I have not been able to consistently have basic editing, testing, or other project management tooling work reliably. I’ve spent hours debugging weird SIGBUS errors that nobody else can recreate and other things that go way above and beyond normal debugging of issues.</p><p>The biggest thing that breaks is the language server in my editor. If I can’t get the language server working, I don’t know what I’m allowed to do with any given thing in the file without having to have a bunch of documentation tabs open. This combined with Python not having <a href="https://pkg.go.dev/">a standard documentation site like Go does</a> means that figuring out what I can do is not easy.</p><p>Making things worse, there’s as many ways to manage Python as there are grains of sand on the planet. Starting to use Python means you get to make a lot of lovely decisions:</p><ul><li>What environment manager are you using? Conda? Virtualenv? uv? Anaconda? Miniconda? Homebrew? Pipenv?</li><li>Which version of Python does your project depend on? Many big libraries like tensorflow do deep monkey patching of Python for performance reasons and as a result they can’t work on newer versions of the interpreter.</li><li>How are you installing your dependencies? Pip? Pip3? Uv?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*4VOX04JbnIu7RS9aWHe-0Q.png" /><figcaption><a href="https://xkcd.com/927/">Standards — XKCD</a></figcaption></figure><p>There has to be some kind of middle path. We should be able to have nice things like the ability to just open a git repo and get a working development environment, right?</p><h3>How it works</h3><p>When you package your app in a Docker image, you make a Dockerfile manifest with a base image and then list out all the changes you make to that base image to get things working. This could be anything from copying your source code into the image, building that code, installing dependencies, or anything else that boils down to copying files and running commands. When you define a development container, you make a devcontainer.json manifest that specifies the base image you&#39;re working from and any <a href="https://containers.dev/features">features</a> you want to add to it.</p><p>For example, let’s consider what you need to do in order to get a <a href="http://node.js/">Node.js</a> environment working. Here’s a sample devcontainer.json file for working with Node:</p><pre>{<br>  &quot;name&quot;: &quot;Node&quot;,<br>  &quot;image&quot;: &quot;mcr.microsoft.com/devcontainers/base:bookworm&quot;,<br>  &quot;features&quot;: {<br>    &quot;ghcr.io/devcontainers/features/node:1&quot;: {},<br>    &quot;ghcr.io/devcontainers-extra/features/neovim-apt-get:1&quot;: {}<br>  },<br>  &quot;postCreateCommand&quot;: &quot;npm ci&quot;<br>}</pre><p>This tells your editor to make a copy of <a href="https://github.com/devcontainers/images/tree/main/src/base-debian">Microsoft’s base Debian image</a> with <a href="https://github.com/devcontainers/features/tree/main/src/node">Node</a> and <a href="https://github.com/devcontainers-extra/features/tree/main/src/neovim-apt-get">neovim</a> automatically installed. It also installs all of your Node dependencies so that all you need to do to get up and running is:</p><ol><li>Open the repo in a development container</li><li>Open a terminal</li><li>Run npm run start</li><li>There is no step 4.</li></ol><p>Just imagine what that workflow could give you. Spinning people up would be a walk in the park.</p><h4>What about Python?</h4><p>You’re probably sitting there asking yourself “yeah, that’s cool, but what about Python?” Python presents a lot of challenges for development use because there’s so many variables at play. If you know what you’re doing, this is fine and manageable. If you don’t, you end up in pip hell. You don’t want to be in pip hell with me.</p><p>One of the big things that development containers give teams that have a mix of Python experts and non-Python experts is the ability to just have a known working setup for people to fall back on in case they aren’t an expert in Python environment metaphysics. It’s great for people like me who care about the end result, but do not care at all how things go about getting done as long as it works (for some reasonable definition of “works”). Even better, you can define editor configuration settings and a list of extensions specifically for that project, meaning that you really can just open a new repo and get up and running within seconds.</p><p>This editor preconfiguration means you can fix problems like “What version of Python do I need?” or “How do I just install the dependencies?” forever. Take <a href="https://github.com/tigrisdata-community/huggingface-datasets-with-tigris">tigrisdata-community/huggingface-datasets-with-tigris</a> for example. Its<a href="https://github.com/tigrisdata-community/huggingface-datasets-with-tigris/blob/main/.devcontainer/devcontainer.json">devcontainer.json</a> answers that question for you:</p><pre>{<br>  // ...<br>  &quot;postCreateCommand&quot;: &quot;uv python install &amp;&amp; uv venv &amp;&amp; uv sync&quot;,<br>  &quot;remoteEnv&quot;: {<br>    &quot;UV_LINK_MODE&quot;: &quot;copy&quot;,<br>    &quot;UV_PYTHON&quot;: &quot;3.10&quot;<br>  }<br>  // ...<br>}</pre><p>When you create a development container with this manifest, it does the following:</p><ol><li>Installs Python 3.10.x with <a href="https://docs.astral.sh/uv/">uv</a></li><li>Creates a <a href="https://docs.astral.sh/uv/pip/environments/#using-python-environments">Python virtual environment</a> for all of your dependencies</li><li>Installs all of the Python dependencies</li></ol><p>And then you can run the code with uv run and things Just Work™. All of that complicated dependency management becomes your environment&#39;s problem. Even better, take a look at <a href="https://github.com/tigrisdata-community/huggingface-datasets-with-tigris/blob/5d32918c5d890b924b46703074e9966249406032/.devcontainer/devcontainer.json#L33-L51">this part of the manifest</a>:</p><pre>{<br>  // ...<br>  &quot;customizations&quot;: {<br>    &quot;vscode&quot;: {<br>      &quot;extensions&quot;: [<br>        &quot;ms-python.python&quot;,<br>        &quot;ms-python.vscode-pylance&quot;,<br>        &quot;tamasfe.even-better-toml&quot;,<br>        &quot;ms-toolsai.jupyter&quot;,<br>        &quot;ms-toolsai.vscode-jupyter-cell-tags&quot;,<br>        &quot;ms-toolsai.jupyter-renderers&quot;,<br>        &quot;ms-toolsai.vscode-jupyter-slideshow&quot;,<br>        &quot;ms-python.debugpy&quot;,<br>        &quot;ms-toolsai.jupyter-keymap&quot;,<br>        &quot;amazonwebservices.aws-toolkit-vscode&quot;<br>      ],<br>      &quot;settings&quot;: {<br>        &quot;python.defaultInterpreterPath&quot;: &quot;./.venv/bin/python&quot;<br>      }<br>    }<br>  }<br>  // ...<br>}</pre><p>This makes VS Code install every extension you need to get a working development environment and that python.defaultInterpreterPath setting is the cherry on top that makes the language server integration work. This lets you simply clone a repo and get a working language server.</p><h3>Conclusion</h3><p>I realize this sounds like a fairly simple thing, and let’s be honest, it should be this simple, but it’s taken me three years of experimentation, toil, and suffering to get to the point where you really can just clone a repo and get working language server integration. If you have also been suffering trying to get Python installed so you can vibe code your way to an IPO, give development containers a try.</p><p>This even works if you use <a href="https://github.com/features/codespaces">GitHub Codespaces</a>, meaning that you don’t even need to install a copy of VS Code to work on the project.</p><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/dev-containers-python/"><em>https://www.tigrisdata.com</em></a><em> on July 3, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdac863a32b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[mount -t tigrisfs]]></title>
            <link>https://tigris-data.medium.com/mount-t-tigrisfs-eda42ae6b45d?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/eda42ae6b45d</guid>
            <category><![CDATA[updates]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[tigris]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 00:00:41 GMT</pubDate>
            <atom:updated>2025-08-06T21:27:17.747Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TsAO3QZvmZFJ76auVmrVRQ.png" /></figure><p>At Tigris we put your big data close to your compute so you don’t have to do it yourself. However, there’s been a small problem with that: most of the programs that are built to process that data such as AI training, document indexing, and other kinds of workloads expect to read data from a filesystem.</p><p>Not to mention, big data means big data. Bigger than ram. Bigger than your disk. Bigger than any one machine can have on any amount of disks. Sometimes even bigger than human minds can imagine. What if that data was as easy to access as your code folder, but had unlimited storage?</p><p>We’re proud to announce the immediate availability of <a href="https://github.com/tigrisdata/tigrisfs">tigrisfs</a>, the native filesystem interface for Tigris. This lets you mount Tigris buckets to your laptops, desktops, and servers so you can use data in your buckets as if it was local. This bridges the gap between the cloud and your machine.</p><blockquote>Internally, tigrisfs is a fork of <a href="https://github.com/yandex-cloud/geesefs">geesefs</a>, another project that converts object storage buckets into mountable storage. geesefs has good performance and makes it easy to access the same bucket from the S3 API and the filesystem without obfuscating object names like juicefs. We have extended geesefs to leverage Tigris-specific features that improve throughput and latency. With tigrisfs you can use the S3 API or the filesystem interchangeably without having to worry about name mangling. tigrisfs is the canonical filesystem implementation for Tigris.<br> — Ovais Tariq, CEO @ Tigris Data</blockquote><h3>Your data: everywhere</h3><p>Let’s imagine that you have big data in your stack. Not just big, but <em>unimaginably big</em>, we’re talking about data the size of Wikipedia, the entire Linux Kernel Mailing List archives, and the entire git history for all the big open source projects. Not to mention small datasets like every scientific paper from arxiv. tigrisfs lets you mount the same dataset in the same place on every machine in your cluster. Imagine just reading from/mnt/tigris/datasets/raw/lkml, processing the data around a bit, and then writing it to/mnt/tigris/datasets/massaged/lkml for the downstream analysis to run. We&#39;ll go into more detail about this in the near future, keep an eye out for that!</p><p>The really cool part about this is that it lets you have a global filesystem on your local machine. All your data is just there and waiting to be used. If you write that massaged dataset to /mnt/tigris/datasets/massaged/lkml on one machine, it&#39;s instantly available to any other machine in the cluster. Any time it&#39;s used, it&#39;ll be seamlessly cached on the device so that it&#39;s hot&#39;n&#39;ready for action! It&#39;s like having a ReadWriteMany Kubernetes volume, but without having to set up Ceph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*fIKRdgXu9KKCklm9SmPI7A.jpeg" /></figure><p>If you’re dealing with anything bigger than ram, tigrisfs is a great fit.</p><p>One of the neat parts about tigrisfs is that using it means you can deal with your files using either the S3 API or the filesystem API. This is in contrast to other tools like JuiceFS which break files into blocks and obfuscate the filenames, meaning you need to spend time and energy reverse-engineering how the block → data mapping works. With tigrisfs you can PUT an object into your bucket with the S3 API, and then open the file in your favorite text editor. This unlocks any number of fun integrations, including:</p><ul><li>Using inotifywait to process data as it&#39;s created in a bucket by your analytics pipeline</li><li>Backing up your home folder with rsync in a cronjob</li><li>Using tools like gzcat to read compressed data without having to decompress it</li><li>Storing TLS certificates across the cluster so that one machine can renew it, and it’ll roll out to the rest of the machines instantly</li><li>Reading your training datasets directly from disk instead of having to set up object storage with the datasets library</li><li>Reading a raw video out of one bucket and compressing it for global distribution into another bucket using ffmpeg</li></ul><p>Let’s say you want to edit your secret plans in your Linux VM on your MacBook. First, upload it to Tigris with aws s3 cp:</p><pre>$ aws s3 cp secretplans.txt s3://pitohui<br>upload: ./secretplans.txt to s3://pitohui/secretplans.txt</pre><p>Then you can view it like normal with the shell:</p><pre>xe@pitohui:~ $ cat /mnt/tigris/pitohui/secretplans.txt<br>- world domination via the use of hypnodrones<br>- make there be such a thing as a free lunch<br>- create more paperclips</pre><p>And now you can do whatever you want! You can even do backups of your home folder with a single command:</p><pre>xe@pitohui:~ $ rsync -av ~ /mnt/tigris/pitohui</pre><p>The cloud’s the limit!</p><h3>Getting started with tigrisfs</h3><p>If you want to get started, all you need is an aarch64/x86_64 Linux system, a Tigris bucket, and a keypair.</p><h4>Installing tigrisfs</h4><p><strong>One-liner install</strong></p><p>This will install the latest version of tigrisfs and its dependencies.</p><pre>curl -sSL https://raw.githubusercontent.com/tigrisdata/tigrisfs/refs/heads/main/install.sh | bash</pre><p><strong>Or, install using package manager</strong></p><p>If you’re using a package manager, you can install tigrisfs by following the instructions below.</p><ul><li>Download the package from <a href="https://github.com/tigrisdata/tigrisfs/releases">the most recent release</a></li><li>Install the package using your package manager<br>- Debian/Ubuntu: sudo apt install ./tigrisfs-version.deb <br>- Alma Linux/Fedora/Red Hat/Rocky Linux: sudo dnf install ./tigrisfs-version.rpm</li></ul><h4>Mounting the filesystem</h4><p>We are going to assume that you have a bucket called pitohui and you want to mount it to/mnt/tigris/pitohui.</p><p>Open /etc/default/tigrisfs in your favorite text editor as root and uncomment the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY variables and paste in the access key you got from the Tigris dashboard.</p><p><strong>Using the command line</strong></p><p>First, create the directory you want to mount the bucket to:</p><pre>mkdir -p /mnt/tigris/pitohui</pre><p>Then mount the bucket:</p><pre>tigrisfs pitohui /mnt/tigris/pitohui</pre><p>This gives me permission to do whatever I want such as touching grass:</p><pre>$ touch /mnt/tigris/pitohui/grass<br>$ stat /mnt/tigris/pitohui/grass<br>  File: /mnt/tigris/pitohui/grass<br>  Size: 0            Blocks: 0       IO Block: 4096   regular empty file<br>Device: 80h/128d     Inode: 1631     Links: 1<br>Access: (0644/-rw-r--r--)  Uid: ( 1000/   xe)   Gid: ( 1000/  xe)<br>Context: system_u:object_r:fusefs_t:s0<br>Access: 2025-04-07 20:15:07.549222957 +0000<br>Modify: 2025-04-07 20:15:07.549222957 +0000<br>Change: 2025-04-07 20:15:07.549222957 +0000<br> Birth: -</pre><p>And make sure it exists in Tigris:</p><pre>$ aws s3 ls s3://pitohui | grep grass<br>2025-04-07 16:15:07       0 grass</pre><p>Perfect!</p><p><strong>Using systemd</strong></p><p>If you’re in an environment with systemd, mount your bucket with systemctl enable --now:</p><pre>sudo systemctl enable --now tigrisfs@pitohui.service</pre><p>Your bucket will be available at /mnt/tigris/pitohui. If you need things to be writable by your user account, edit the OPTS line based on your account&#39;s information. For example on my MacBook&#39;s Oracle Linux VM:</p><pre>$ id<br>uid=1000(xe) gid=1000(xe) groups=1000(xe),10(wheel) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023</pre><p>My user id (uid) is 1000 and my group id (gid) is 1000, so to give my user permissions, I need this OPTS line:</p><pre># Mount options<br>OPTS=&quot;-o allow_other --gid=1000 --uid=1000&quot;</pre><p>This gives me permission to do whatever I want such as touching grass:</p><pre>$ touch /mnt/tigris/pitohui/grass<br>$ stat /mnt/tigris/pitohui/grass<br>  File: /mnt/tigris/pitohui/grass<br>  Size: 0           	Blocks: 0      	IO Block: 4096   regular empty file<br>Device: 80h/128d    	Inode: 1631    	Links: 1<br>Access: (0644/-rw-r--r--)  Uid: ( 1000/ 	 xe)   Gid: ( 1000/ 	xe)<br>Context: system_u:object_r:fusefs_t:s0<br>Access: 2025-04-07 20:15:07.549222957 +0000<br>Modify: 2025-04-07 20:15:07.549222957 +0000<br>Change: 2025-04-07 20:15:07.549222957 +0000<br> Birth: -</pre><p>And make sure it exists in Tigris:</p><pre>$ aws s3 ls s3://pitohui | grep grass<br>2025-04-07 16:15:07      	0 grass</pre><p>Perfect!</p><p>If you are looking to crank up performance, there are a few configuration options that you can tweak. Take a look at the <a href="https://www.tigrisdata.com/docs/training/tigrisfs/#maximizing-performance">documentation</a> for more details.</p><h3>Under the hood</h3><p>tigrisfs is a fork of <a href="https://github.com/yandex-cloud/geesefs">geesefs</a>, a high performance FUSE filesystem adaptor for object storage. We have extended geesefs to leverage Tigris-specific features that improve throughput and latency.</p><p>tigrisfs is a high performance FUSE filesystem adaptor for object storage based on <a href="https://github.com/yandex-cloud/geesefs">geesefs</a>, which is a fork of <a href="https://github.com/kahing/goofys">goofys</a>. GeeseFS solves performance problems which FUSE file systems based on S3 typically have, especially with small files and metadata operations. It solves these problems by using aggressive parallelism and asynchrony.</p><h4>Improvements over GeeseFS</h4><p>Our initial release zeroed-in on hardening the codebase for production, focusing on two areas:</p><ul><li><strong>Security hardening<br>- </strong>Replaced the bundled, legacy AWS SDK that contained known CVEs<br>- Upgraded every dependency to its latest secure version</li><li><strong>Reliability upgrades<br>- </strong>Eliminated all race conditions flagged by the Go race detector (now mandatory in tests)<br>- Fixed every linter warning and added lint checks to CI<br>- Dramatically expanded the test-suite and made the extended tests a default part of CI</li></ul><h4>Tigris-specific improvements</h4><p>We also shipped a few features that lean on Tigris internals:</p><ul><li><strong>POSIX semantics</strong> — permissions, special files, and symlinks now behave just like they do on a local disk</li><li><strong>Turbo-charged small files</strong> — listing a directory automatically batch-fetches and caches tiny objects in a single round-trip</li><li><strong>Smart prefetch</strong> — directory listings kick off background fetches so the nextcat or grep feels instant</li></ul><p>In essence, tigrisfs bridges the gap between the Linux kernel and Tigris. It translates filesystem calls into S3 API calls so that you can explore your bucket with the shell and bridge the gap between the old world of servers and shells with the new world of dynamic infinity in the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTSJI1T19mKiVpgVWrhLGw.png" /></figure><h4>Benchmarks</h4><p>Benchmarking filesystems is kind of annoying, and networked filesystems can be even more annoying to benchmark. Most of the time, you end up making a lot of assumptions about the system state and network configuration. Here are the specs of our benchmarking machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*KifjIp8HXdW95QSPeb0Txw.jpeg" /></figure><p>Our benchmarks are done with the <a href="https://github.com/axboe/fio">flexible i/o tester fio</a>. Note that we are using direct I/O to avoid <a href="https://en.wikipedia.org/wiki/Page_cache">page caching</a> being an issue.</p><p><strong>Read performance</strong></p><p>Here is the command we used to test read performance on a bucket:</p><pre>fio --name=read_throughput \<br>    --directory=/mnt/test-tigrisfs-bucket \<br>    --numjobs=4 \<br>    --size=4G \<br>    --time_based \<br>    --runtime=120s \<br>    --ramp_time=2s \<br>    --ioengine=libaio \<br>    --direct=1 \<br>    --verify=0 \<br>    --bs=1M \<br>    --iodepth=1 \<br>    --rw=read \<br>    --group_reporting=1</pre><p>This has fio run for 2 minutes with 2 seconds of ramp-up time (during which, the results are not counted in the statistics) trying to read up to 4 gigabytes of data per thread (job) in one megabyte blocks. This reads a total of 16 gigabytes of data. The test was run in permutations of thread count and block size to see if the limitations are on tigrisfs, the Tigris service, and the network card of the machine.</p><p>And we got these results for each permutation of the test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*aU1v_s_H-wLbG-uOTCHOTw.jpeg" /></figure><blockquote><strong>NOTE</strong></blockquote><blockquote>The throughput numbers with the asterisk next to them could theoretically be faster, but at this point we saturated out the network card on the test machine.</blockquote><p><strong>Write performance</strong></p><p>Here is the command we used to test write performance:</p><pre>fio --name=write_throughput \<br>    --directory=/mnt/test-tigrisfs-bucket \<br>    --numjobs=8 \<br>    --size=4G \<br>    --time_based \<br>    --runtime=120s \<br>    --ramp_time=2s \<br>    --ioengine=libaio \<br>    --direct=1 \<br>    --verify=0 \<br>    --bs=4M \<br>    --iodepth=1 \<br>    --rw=write \<br>    --group_reporting=1</pre><p>This has fio run for 2 minutes with 2 seconds of ramp-up time (during which the results are not counted in the statistics) trying to write up to 4 gigabytes of data per thread (job) in four-megabyte blocks. This writes a total of 16 gigabytes of data. The test was run in permutations of thread count and block size to see if the limitations are on tigrisfs, the Tigris service, and the network card of the machine.</p><p>And we got these results for each permutation of the test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*ZZcdjNcx4cx2el2scpKubg.jpeg" /></figure><p>Needless to say, this represents being able to read and write multiple DVDs of data per second per machine with tigrisfs. This combined with the caching that tigrisfs uses means that it should be more than sufficient for anything you can throw at it.</p><h3>When should I use tigrisfs?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*en7AMvouoYrHc1-lPQQlGw.jpeg" /></figure><p>Personally, I use tigrisfs all the time on my own machines. One of the main things I use it for is running <a href="https://www.tigrisdata.com/blog/anubis/">analytics across honeypot logs</a> so that I can fight off evil scrapers and save the internet.</p><p>In general, tigrisfs can be slower than the native disk for files that aren’t cached yet, but it more than makes up for it by allowing you to make the location of your files irrelevant. All you need to do is run tigrisfs, and you have a single global namespace for your data across all your machines.</p><p>tigrisfs is written in Go and is <a href="https://github.com/tigrisdata/tigrisfs">open source on GitHub</a>. We welcome any and all contributions to make it even better!</p><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/tigrisfs/"><em>https://www.tigrisdata.com</em></a><em> on July 1, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eda42ae6b45d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Time Travel with DuckLake and Tigris]]></title>
            <link>https://tigris-data.medium.com/data-time-travel-with-ducklake-and-tigris-fdcb77d23e05?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdcb77d23e05</guid>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Wed, 25 Jun 2025 00:00:51 GMT</pubDate>
            <atom:updated>2025-06-30T20:20:41.806Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCMZbPYnCt8NP-0SAerYUg.png" /></figure><p>You’ve got your tunes in high gear, your editor is open, and you’re working on recreating some database tables with your AI agent in an epic duo. Your AI agent says “run this SQL query?” and you click yes. The tests pass, you make PR that’s quickly stamped and merged, when suddenly your pager goes off. And again. And again. You’ve just broken the analytics database and everything is on fire. What do you do?</p><p>If you’re using most database engines, this is a priority zero “stop the world and restore from backups” shaped problem. This is especially annoying with analytics databases because many times those databases aren’t just bigger than ram, they’re bigger than your local disk, and sometimes even bigger than any disk in any computer can ever be. But further, if your AI starts renaming columns or combining data in interesting ways, it can be quite the mess to untangle, even impossible. In such cases, this is an XK-class end-of-the-project scenario, which is triple-plus ungood.</p><p>However, you read <a href="https://www.tigrisdata.com/blog/ducklake/">our last post on Ducklake</a> and have been storing your analytics data in Tigris so you can get that sweet, sweet global performance. How do you go back to the past where everything Just Worked? Turns out it’s easy, no DeLorean required. All you have to do is reset the timeline with a couple simple commands.</p><h3>Ducklake and you</h3><p><a href="https://ducklake.select/">Ducklake</a> is an analytics data lakehouse that lets you import SQL and NoSQL data so you can run SQL queries on it. One of the really cool parts about Ducklake is that when you do any INSERT or DELETE into Ducklake tables, you create a new snapshot of the database that you can roll back to. Ducklake turns your SQL database into an append-only-log.</p><p>As an example, let’s create a Ducklake database backed by Tigris, insert some data and see what happens. First, <a href="https://duckdb.org/docs/installation/">install DuckDB</a> and then set up the Ducklake extension:</p><pre>INSTALL ducklake;<br>LOAD ducklake;</pre><p>Then attach to a new Ducklake database in Tigris:</p><pre>ATTACH &#39;ducklake:timetravel_demo.ddb&#39;<br>AS delorean<br>  ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>  );</pre><p>Note: This creates the Ducklake metadata on the local filesystem which is fine for demos like this, but for production use we suggest putting your Ducklake metadata in the cloud with a Postgres database or <a href="https://ducklake.select/docs/stable/duckdb/usage/choosing_a_catalog_database">one of the other backends Ducklake supports</a>.</p><p>Now that we have the database, let’s create a simple table and throw some data in it:</p><pre>CREATE TABLE IF NOT EXISTS delorean.youtube_videos<br>  ( id TEXT NOT NULL PRIMARY KEY<br>  , title TEXT NOT NULL<br>  , channel TEXT NOT NULL<br>  );<br><br>INSERT INTO delorean.youtube_videos<br>  ( id, title, channel )<br>VALUES<br>  ( &#39;WcSCYzI2peM&#39;, &#39;Delfino Plaza (Super Mario Sunshine) - Mario Kart World&#39;, &#39;SiIvaGunner&#39; )<br>, ( &#39;W4AcveHnDzg&#39;, &#39;Retribution for the Eternal Night ~ Imperishable Night (Beta Mix) - Touhou 8: Imperishable Night&#39;, &#39;SiIvaGunner&#39; );</pre><p>Awesome, let’s see what the bucket looks like:</p><pre>$ aws s3 ls s3://xe-ducklake/delorean/main/youtube_videos/<br>2025-06-25 10:28:34       1175 ducklake-0197a77d-905b-7624-8e32-c80c69470e52.parquet</pre><p>Interesting, Ducklake created a parquet file named after the table we inserted the data into, let’s see what it looks like:</p><pre>FROM &#39;s3://xe-ducklake/delorean/main/youtube_videos/ducklake-0197a77d-905b-7624-8e32-c80c69470e52.parquet&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*4DMRdwr_pS1fYztxISGQtw.jpeg" /></figure><p>This is the key to how Ducklake works. Every time you write to one of its tables, it puts those rows you added into a parquet file in object storage. Let’s see the changes we made to the delorean database:</p><pre>FROM ducklake_snapshots(&#39;delorean&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*zcfSbg_FE9CxY2OhxMt1Hw.jpeg" /></figure><p>The first change is creating the main schema, and then you can see that while I was working on this article I made the youtube_videos table, messed up the schema, dropped it, recreated it, and then inserted information about <a href="https://youtu.be/W4AcveHnDzg">epic tunes</a> into the table. To really show off this time travel power though, let&#39;s delete the data and then add other data into the mix:</p><pre>DELETE FROM delorean.youtube_videos;<br><br>INSERT INTO delorean.youtube_videos<br>  ( id, title, channel )<br>VALUES<br>  ( &#39;jhl5afLEKdo&#39;, &#39;Hatsune Miku World is Mine / ryo（supercell)&#39;, &#39;Hatsune Miku&#39; )<br>, ( &#39;sqK-jh4TDXo&#39;, &#39;Machine Love (feat. Kasane Teto)&#39;, &#39;Jamie Page&#39; );</pre><p>So what happened to the database? Here’s what the table looks like now:</p><pre>SELECT * FROM delorean.youtube_videos;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/1*qK_uS9M0maHoZnCjrxyPlw.jpeg" /></figure><p>But if you look in the bucket, the old data is still there:</p><pre>FROM &#39;s3://xe-ducklake/delorean/main/youtube_videos/*.parquet&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*shpSTQjHJrfjjienUum-fA.jpeg" /></figure><p>How do you get the old data back? Well for one, we can time travel <em>directly in SQL queries</em>! Let’s look at the database snapshots again and try to figure out what happened:</p><pre>FROM ducklake_snapshots(&#39;delorean&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*akkdri4VIo69NXwZHrUg4Q.jpeg" /></figure><p>So it looks like the table was added to in snapshot 4, the data was deleted in snapshot 5, and the new data comes in at snapshot 6. Let’s get the superset of the data at snapshot 4 AND snapshot 6:</p><pre>SELECT * FROM delorean.youtube_videos AT (VERSION =&gt; 4)<br>UNION ALL<br>SELECT * FROM delorean.youtube_videos AT (VERSION =&gt; 6) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*iEWZRby0OQjVzwuicsDj_w.jpeg" /></figure><p>You can see the power here right? The data is still safely stored in your bucket, so deletes <em>don’t matter</em>. It may be more inconvenient to access the data, but you can also time travel for <em>the entire database at once</em>:</p><pre>ATTACH &#39;ducklake:timetravel_demo.ddb&#39;<br>AS delorean_past<br>  ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>  , SNAPSHOT_VERSION 4<br>  );<br><br>SELECT * FROM delorean_past.youtube_videos;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*DdD9SrJmjXZs-lRTnv2Tzg.jpeg" /></figure><h3>Advanced temporal mechanics</h3><p>You’re not limited to just running queries against the past, you can also connect to the database at a given point in time. This combined with making a local fork of the database lets you get into <em>advanced</em> temporal mechanics.</p><p>Let’s make a local copy of the database to debug the AI agent’s changes. Connect to the database in the past before the AI model messed things up:</p><pre>ATTACH &#39;ducklake:timetravel_demo.ddb&#39;<br>AS delorean_past<br>  ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>  , SNAPSHOT_VERSION 4<br>  );</pre><p>Cool, then let’s make a local copy of it at that point in time:</p><pre>ATTACH &#39;ducklake:timetravel_local_copy.ddb&#39;<br>AS local_delorean<br>    ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>    );<br><br>COPY FROM DATABASE delorean TO local_delorean;<br>DETACH delorean;<br>DETACH local_delorean;</pre><p>The magic part of that is the COPY FROM DATABASE instruction. That makes a copy of the database locally so you can debug the AI agent&#39;s change and prevent a future timeline from coming to pass the way the current one did. Then for the cherry on top, attach the local database with the same name as the remote one so that your agent is none the wiser:</p><pre>ATTACH &#39;ducklake:timetravel_local_copy.ddb&#39;<br>AS delorean<br>    ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>    );</pre><p>Et voila! We have successfully forked the timeline and can now make any change we want without affecting the main timeline. Test it by running a SQL query:</p><pre>SELECT * FROM delorean.youtube_videos</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*Z9mBXS-gQLTR9HCmONuXwg.jpeg" /></figure><p>The really cool part is that we did all that <em>without</em> affecting the data in Tigris:</p><pre>SELECT * FROM &#39;s3://xe-ducklake/delorean/main/youtube_videos/*.parquet&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/1*b1DAJsz3qTla28y-b8xOAQ.jpeg" /></figure><p>When we made a copy of the data lake to hack on locally, all we were copying was the schemata of the database and references to objects in Tigris. At some level, the tables don’t logically exist, they’re really just a bunch of rules and references that Ducklake uses to rebuild your database on the fly! And because the tables quack like SQL tables enough, the illusion is maintained!</p><p>Another really cool thing is that every INSERT or UPDATE operation results in discrete parquet files being put into Tigris:</p><pre>$ aws s3 ls s3://xe-ducklake/delorean/main/youtube_videos/<br>2025-06-25 10:28:34       1175 ducklake-0197a77d-905b-7624-8e32-c80c69470e52.parquet<br>2025-06-25 10:41:49        975 ducklake-0197a789-b1b2-797e-8bd5-a20905d2d73f.parquet</pre><p>These parquet files are written to once and NEVER updated. This means that as your analytics pipelines or developers all over the world access things, they’re automatically fast and local thanks to Tigris’ global performance.</p><h3>Victory achieved!</h3><p>Now you can turn your analytics pipeline back on and go on with hacking up a storm. Next time you do those changes with your AI agent though, make sure to test them against a backup of the data lake just in case things go pear-shaped again. Ensure your AI isn’t renaming columns, deleting tables, changing the structure, etc. Ideally your schema changes should only add columns and never remove them. Your database tables should be treated as a public API.</p><p>To make a local backup of your data lake so your agent can break whatever the tokens deem worthy without breaking prod:</p><pre>ATTACH &#39;ducklake:timetravel_local_copy.ddb&#39;<br>AS local_delorean<br>    ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>    );<br><br>COPY FROM DATABASE delorean TO local_delorean;<br>DETACH delorean;<br>DETACH local_delorean;<br><br>ATTACH &#39;ducklake:timetravel_local_copy.ddb&#39;<br>AS delorean<br>    ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>    );</pre><p>And to attach to Ducklake in read-only mode so that it can’t break anything if it wanted to:</p><pre>ATTACH &#39;ducklake:timetravel_demo.ddb&#39;<br>AS stone_tablet<br>  ( DATA_PATH &#39;s3://xe-ducklake/delorean&#39;<br>  , READ_ONLY -- &lt;- attaches the ducklake database in read-only mode<br>  );</pre><p>And then you can go back to fearless vibe coding to make your dreams come true in the form of B2B SaaS! All your data will be safe in the cloud and fast to load anywhere in the world, even if you need to time travel a bit to get things working again.</p><figure><a href="https://www.tigrisdata.com/docs/get-started/https://www.tigrisdata.com/docs/get-started/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoRS2o-rJPherGYNqgM4eg.png" /></a></figure><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/ducklake-time-travel/"><em>https://www.tigrisdata.com</em></a><em> on June 25, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdcb77d23e05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get your data ducks in a row with DuckLake]]></title>
            <link>https://tigris-data.medium.com/get-your-data-ducks-in-a-row-with-ducklake-8da37aef7778?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/8da37aef7778</guid>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Tue, 10 Jun 2025 00:00:05 GMT</pubDate>
            <atom:updated>2025-06-16T16:12:47.552Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*38mTrDVAgmdwYWs5.jpg" /></figure><p>There’s a new data lake on the market: <a href="https://ducklake.select/">DuckLake</a>. It’s worth the hype because it gives you an open standard that not only enables you to run queries on your data lakes from anywhere, it outright encourages you to run your query, metadata, and storage layers separately in whatever platform works best for you. We’ve been thinking about DuckDB as a solution for Small Data, but now with the limitless storage capability of object storage, it can support massive scale datasets. Big Data doesn’t have to be complicated.</p><p>One of the key changes with DuckLake is moving the metadata layer out of object storage and into a dedicated database. But why would an object storage company like Tigris be excited about putting <em>less</em> data in object storage?</p><p>Iceberg and Delta Lake both store their metadata in files on object storage: Delta Lake uses a transactional log file within the object storage itself, whereas Iceberg uses an external catalog that can be stored in a DB or in object storage. The trouble comes when you handle concurrent writes. Someone has to “win” and object storage systems don’t let you lock files without coordination via DynamoDB or another service. DuckLake lets you handle concurrent writes via the concurrent write support of your database engine of choice, you get all of the benefits of a normal database without having to have all your data in one disk. You get all the advantages of a database by actually using a database instead of <a href="https://www.e6data.com/blog/iceberg-metadata-evolution-after-compaction#:~:text=,understanding%20the%20iceberg%20file%20layout">poorly emulating one with a bunch of flat files</a>!</p><p>But if you really want to take advantage of that flexibility and choice that DuckLake offers, you need to unstick all the data in your lakehouse from $BigCloud’s storage where every gigabyte of egress to a tool in another cloud has a cost. The metadata layer being separate from the data also eliminates another point where other solutions can get your provider to dig you in the cost department. Storing all your actual data in Tigris also gives you global performance and the ability for your data to be accessible and fast from anywhere.</p><blockquote><em>DuckLake lets you horizontally scale your queries with a serverless architecture on any cloud, and Tigris makes it economical to do so because Tigris has no egress fees.</em></blockquote><p>Let’s say that I have an ETL pipeline processing customer data and storing it in a data lake on a Big Cloud provider like GCS. I have a fleet of Agents all using the customer data to answer questions during Support workflows. If I’ve deployed my Agents on an Agentic platform, I won’t be guaranteed to run my queries within the same Big Cloud as my storage. I’d have to pay by the gigabyte if I had the gall to run that query from anywhere but Google Cloud. I can’t magically teleport my laptop to be inside Google Cloud’s network (yeah, they have a VPN product, but you still have to pay egress for that) and my neocloud workloads definitely aren’t in Google Cloud, so I’d have to restrict all of my planning to find ways to lock myself deeper into the clutches of that ecosystem.</p><p>If we want to live in this bright multicloud future where we <em>can</em> use the best tool for the job, we need to make upfront choices about where and how we store data. When you use DuckLake with its metadata in a Postgres database running somewhere, you can connect to that data lake from any developer laptop, cloud instance, or serverless worker. This allows you to take the concepts of <a href="https://www.tigrisdata.com/blog/nomadic-compute/">Nomadic Compute</a> to your data, meaning that you can leverage the cost savings of neo-clouds to manage your data from anywhere no matter how big or small it is. Your data will automatically move around to where it’s used without having to worry about egress fees and cost management.</p><h3>DuckLake is a data lakehouse for big and small data</h3><p>DuckLake lets you import data from SQL databases and object storage, then stores that aggregated data in object storage so that queries are nice and quick. This lets you store and run queries on data that is bigger than your local device can ever store, everything is queried on demand and you only operate on the data you actually use. When you use this with Tigris, we have <a href="https://www.tigrisdata.com/docs/pricing/">no egress fees</a>, meaning that you don’t have to worry about the bill when you make a big query.</p><p>DuckLake has two basic parts:</p><ul><li>Metadata in a SQL database (MySQL, Postgres, DuckDB, or SQLite)</li><li>Data storage in object storage or on the local filesystem</li></ul><p>Here’s how you declare a table in DuckLake:</p><pre>INSTALL ducklake;<br>LOAD ducklake;<br><br>ATTACH<br>  &#39;ducklake:postgres:dbname=postgres host=localhost&#39;<br>AS cloud<br>  ( DATA_PATH &#39;s3://xe-ducklake/data/&#39;<br>  );<br><br>CREATE TABLE IF NOT EXISTS cloud.customers<br>  ( id INTEGER NOT NULL<br>  , username VARCHAR UNIQUE NOT NULL<br>  , email_address VARCHAR UNIQUE NOT NULL<br>  , created_at TIMESTAMP NOT NULL<br>  , updated_at TIMESTAMP<br>  , deleted_at TIMESTAMP<br>  , usage_kind VARCHAR<br>  );</pre><p>That’s it. No setting up massive infrastructure, no bargaining with the powers that be for API access, and no overhead and churn to maintain the setup requiring expensive consultants to emit utterances that no human should have to comprehend. You just declare the table and <em>you’re done</em>. There are no more steps.</p><p>Don’t want to store the metadata in Postgres? You don’t need to:</p><pre>INSTALL ducklake;<br>LOAD ducklake;<br><br>ATTACH<br>  &#39;ducklake:metadata.ducklake&#39;<br>AS cloud<br>  ( DATA_PATH &#39;./data&#39;<br>  );</pre><p>This is how I play with DuckLake on my MacBook. I’m only dealing with data <a href="https://www.tigrisdata.com/blog/anubis/">much smaller than ram</a>, but the best part about this in my book is that I can fit all of the moving parts in one part of my mental context. This leaves me able to think about what I’m doing without having to think too much about how I have to do it. If I need more space, I can just farm it out to Tigris:</p><pre>ATTACH<br>  &#39;ducklake:metadata.ducklake&#39;<br>AS cloud<br>  ( DATA_PATH &#39;s3://xe-ducklake/hoshimi-miyabi-hacking/&#39;<br>  );</pre><p>But realistically, you probably want to store the DuckLake metadata in Postgres so you can connect to the same data lake from another computer, production instances, or serverless infrastructure. Once your lake is set up, you can import the data from another Postgres database:</p><pre>INSTALL postgres;<br>LOAD postgres;<br><br>ATTACH<br>  &#39;dbname=dunda-prod user=readonly host=dunda-prod-pg.dunda.ti.do&#39;<br>AS prod<br>  ( TYPE postgres<br>  , READ_ONLY<br>  );<br><br>INSERT INTO<br>  cloud.customers<br>    ( id<br>    , username<br>    , email_address<br>    , created_at<br>    , updated_at<br>    , deleted_at<br>    )<br>SELECT<br>  id<br>, username<br>, email_addresses<br>, created_at<br>, updated_at<br>, deleted_at<br>FROM<br>  cloud.users<br>WHERE<br>  deleted_at IS NOT NULL;</pre><p>But what about your telemetry usage data in Tigris? You can import that too. Let’s say your data roughly looks like this:</p><pre>{<br>  &quot;created_at&quot;: &quot;2025-06-09 20:38:40&quot;,<br>  &quot;user_id&quot;: 1337,<br>  &quot;action_verb&quot;: &quot;CreateDocument&quot;,<br>  &quot;metadata&quot;: {<br>    &quot;document_id&quot;: &quot;019755f9-7965-78ee-8677-b21ee8fa9bb6&quot;,<br>    &quot;org_id&quot;: &quot;019755f9-f6b0-763b-9882-b773cfaf828c&quot;<br>  }<br>}</pre><p>Then make your table:</p><pre>CREATE TABLE cloud.telemetry<br>  ( created_at TIMESTAMP NOT NULL<br>  , user_id INTEGER NOT NULL<br>  , action_verb VARCHAR NOT NULL<br>  , metadata MAP(STRING, STRING) NOT NULL<br>  );</pre><p>And load it directly from the cloud to the cloud:</p><pre>INSERT INTO<br>  cloud.telemetry<br>    ( created_at<br>    , user_id<br>    , action_verb<br>    , metadata<br>    )<br>SELECT<br>  *<br>FROM<br>  READ_JSON<br>    ( &#39;s3://dunda-prod-telemetry/new/**/*.jsonl&#39;<br>    , columns =<br>      { created_at: &#39;TIMESTAMP NOT NULL&#39;<br>      , user_id: &#39;INTEGER NOT NULL&#39;<br>      , action_verb: &#39;VARCHAR NOT NULL&#39;<br>      , metadata: &#39;MAP(string, STRING) NOT NULL&#39;<br>      }<br>    , format = &#39;nd&#39;<br>    );</pre><h4>Lakehouses in anger</h4><p>As for how you’d use this in production, let’s think about one of the most common issues when making a Software-as-a-Service business: users that sign up and then never do anything with your product. Typically you’d want to send them an email to let them know about some feature of the product that will entice them to re-engage with your product. Typically when you’re trying to figure out when to send these re-engagement emails, you will need to combine non-relational telemetry observations (such as when your product emits events like CreatedDocument or StartFunction) with relational data (such as user or organization details from your database). Historically, figuring out when to send these requires a lot of “database churn” with fields like the last time a user logged into the platform, or putting all of those events into your database in the first place. These make your SRE team unhappy, but they do <em>work</em>. We can do better though!</p><p>DuckLake lets you just run the query on your data lakehouse directly. Assume that a perfectly spherical import script runs every so often to sync data from Postgres and Tigris to your lakehouse. In order to mark customers to be selected for a re-engagement email, you can write a single query:</p><pre>-- If users have no telemetry activity but their account was created a week ago, set them to Reengage.<br>-- When the re-engagement is sent but there&#39;s no activity after a week, set them to GaveUp.<br>-- Otherwise, if the user has activity then set them to Active.<br><br>WITH last_event AS (<br>    SELECT<br>        c.id,<br>        c.created_at                         AS customer_created_at,<br>        COUNT(t.*)                           AS evt_cnt,              -- 0 = never seen<br>        MAX(t.created_at)                    AS last_evt_at,          -- NULL if no events<br>        arg_max(t.action_verb, t.created_at) AS last_evt_verb      -- DuckDB  v0.10+; picks verb of the latest row<br>    FROM cloud.customers AS c<br>    LEFT JOIN cloud.telemetry AS t<br>           ON t.user_id = c.id<br>    GROUP BY c.id, c.created_at<br>)<br>, desired AS (<br>    SELECT<br>        id,<br>        CASE<br>            WHEN evt_cnt = 0<br>                 AND customer_created_at &lt; CURRENT_TIMESTAMP - INTERVAL &#39;7 days&#39;<br>            THEN &#39;Reengage&#39;<br><br>            WHEN evt_cnt &gt; 0<br>                 AND last_evt_verb = &#39;Reengaged&#39;<br>                 AND last_evt_at   &lt; CURRENT_TIMESTAMP - INTERVAL &#39;7 days&#39;<br>            THEN &#39;GaveUp&#39;<br><br>            WHEN evt_cnt &gt; 0<br>            THEN &#39;Active&#39;<br><br>            ELSE usage_kind            -- leave as‑is for very new sign‑ups, etc.<br>        END AS new_usage_kind<br>    FROM last_event<br>)<br>UPDATE cloud.customers AS c<br>SET    usage_kind = d.new_usage_kind<br>FROM   desired AS d<br>WHERE  c.id = d.id<br>  AND  c.usage_kind IS DISTINCT FROM d.new_usage_kind;</pre><p>This lets you seamlessly weave relational data from your databases with non-relational data in object storage. All of that data we inserted gets stored in Tigris so that you only have to do your big imports <em>once</em>. Throw this into a serverless function, make it run every day as a cronjob, and then you can have another worker find customers that need to re-engage and send them gentle encouragement.</p><h3>How it works</h3><p>DuckLake has two basic parts:</p><ul><li>Metadata in a SQL database (MySQL, Postgres, DuckDB, or SQLite)</li><li>Data storage in object storage or on the local filesystem</li></ul><p>When you create a DuckLake backed by Postgres, it creates a bunch of tables for you:</p><pre>thoth=&gt; \dt<br>                       List of relations<br> Schema |                 Name                  | Type  | Owner<br>--------+---------------------------------------+-------+-------<br> public | ducklake_column                       | table | thoth<br> public | ducklake_column_tag                   | table | thoth<br> public | ducklake_data_file                    | table | thoth<br> public | ducklake_delete_file                  | table | thoth<br> public | ducklake_file_column_statistics       | table | thoth<br> public | ducklake_file_partition_value         | table | thoth<br> public | ducklake_files_scheduled_for_deletion | table | thoth<br> public | ducklake_inlined_data_tables          | table | thoth<br> public | ducklake_metadata                     | table | thoth<br> public | ducklake_partition_column             | table | thoth<br> public | ducklake_partition_info               | table | thoth<br> public | ducklake_schema                       | table | thoth<br> public | ducklake_snapshot                     | table | thoth<br> public | ducklake_snapshot_changes             | table | thoth<br> public | ducklake_table                        | table | thoth<br> public | ducklake_table_column_stats           | table | thoth<br> public | ducklake_table_stats                  | table | thoth<br> public | ducklake_tag                          | table | thoth<br> public | ducklake_view                         | table | thoth</pre><p>All of these tables are used to create the mappings of what tables point to which data. This data can be stored on your local laptop for quick hacking or in object storage like Tigris for when your data gets much bigger than any one computer can possibly store.</p><p>Additionally, any time you make a change to your DuckLake, it creates a snapshot of the entire state of the world so that you can roll back changes when you mess things up. This means that you can import data, run big analytics queries, and more without any fear of losing data. All the data will be safely stored in Tigris and you can do whatever queries you want from anywhere in the world without having to pay egress fees.</p><p>Let’s say you need to look at data about a specific customer from a week ago:</p><pre>SELECT<br>  *<br>FROM<br>  cloud.customers<br>WHERE<br>  email_address LIKE &#39;%@conto.so&#39;<br>AT ( TIMESTAMP =&gt; now() - INTERVAL &#39;1 week&#39; );</pre><p>And bam, every one of your customers at Contoso has all their information show up. Need to make sure that the re-engagement emails got sent out back then? You can just do that without having to do a mass scan of object storage.</p><p>Honestly this is one of the coolest parts of DuckLake in my book. Sure it’s not as theoretically performant as it would be if all the data was directly on my laptop, but being able to operate on extremely large datasets more than makes up for the downsides. Your postgres database won’t let you roll back a change that someone messed up a week ago, for that you’d have to restore from backup.</p><h3>The cloud’s the limit</h3><p>Of course, this isn’t just limited to your developer laptop. You can access your DuckLake datasets from the <a href="https://duckdb.org/docs/stable/clients/node_neo/overview">Node.js</a> client too! Take a gander:</p><pre>import duckdb, { DuckDBConnection } from &quot;@duckdb/node-api&quot;;<br><br>console.log(&quot;duckdb version&quot;, duckdb.version());<br><br>const connection = await DuckDBConnection.create();<br><br>for (const statement of [<br>  `INSTALL postgres`,<br>  `LOAD postgres`,<br>  `INSTALL ducklake`,<br>  `LOAD ducklake`,<br>  `INSTALL httpfs`,<br>  `LOAD httpfs`,<br>  `INSTALL json`,<br>  `LOAD json`,<br>  `CREATE OR REPLACE PERSISTENT SECRET tigris (<br>    TYPE s3,<br>    PROVIDER config,<br>    KEY_ID &#39;tid_access_key_id&#39;,<br>    SECRET &#39;tsec_secret_access_key&#39;,<br>    REGION &#39;auto&#39;,<br>    ENDPOINT &#39;t3.storage.dev&#39;,<br>    URL_STYLE &#39;vhost&#39;,<br>    SCOPE &#39;s3://xe-ducklake&#39;<br>  )`,<br>  `CREATE OR REPLACE SECRET postgres_localhost (<br>    TYPE postgres,<br>    HOST &#39;localhost&#39;,<br>    PORT 5432,<br>    DATABASE postgres,<br>    USER &#39;root&#39;,<br>    PASSWORD &#39;hunter2&#39;<br>  )`,<br>  `ATTACH &#39;ducklake:postgres:dbname=postgres host=localhost&#39; as cloud (DATA_PATH &#39;s3://xe-ducklake/data/&#39;)`,<br>]) {<br>  await connection.run(statement);<br>}</pre><p>Now you can SELECT, INSERT, UPDATE, and DELETE to your heart&#39;s content! Just make sure that the statements that create secrets are put safely into environment variables. This is super useful for making sure that your import jobs run in Lambda or for distributing analytics jobs across an entire fleet.</p><p>Imagine a world where you have an API that creates resources and you want to charge users for those resources. You could imagine a world where that data gets written to Tigris based on the time it’s written, such as:</p><pre>s3://dunda-prod-usage/2025/06/10/13/*.jsonl</pre><p>And then your lambda jobs could take the data from the last hour (remember: time is “consistent enough” in values over one hour), massage it into DuckDB, and then update it into your production database with what’s changed! Remember, you can attach Postgres databases the same way you can attach a DuckLake! Throw in some <a href="https://www.tigrisdata.com/docs/buckets/object-lifecycle-rules/">lifecycle rules</a> to delete the raw usage data after 6 months and add some batching to your uploading and you can get all of the fun of a fancy streaming analytics setup with nearly none of the bloat of having to reinvent the entire world!</p><h3>The new era of databases</h3><p>Personally, I think that DuckLake is really cool and this post barely scratches the surface of what you can do. It’s a SQL database without limits embedded right into your app. No more having to resize database instances for the data team. No more complicated import pipelines to get data from object storage into a database engine. You just query your data and then you’re done with it. This is seriously one of the most underrated data science superpowers and if you play your cards right, you can scale fearlessly.</p><p>As I dig into DuckLake some more, I’m going to show you some demos that really play into Tigris and DuckLake working together to make data analytics work great on big and small scales. Join along for the ride!</p><figure><a href="https://www.tigrisdata.com/docs/get-started/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoRS2o-rJPherGYNqgM4eg.png" /></a></figure><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/ducklake/"><em>https://www.tigrisdata.com</em></a><em> on June 10, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8da37aef7778" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Eliminating Data Migration Risk with Shadow Buckets on Tigris]]></title>
            <link>https://tigris-data.medium.com/eliminating-data-migration-risk-with-shadow-buckets-on-tigris-3f25341722f4?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f25341722f4</guid>
            <category><![CDATA[object-storage-services]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[features]]></category>
            <category><![CDATA[object-storage]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Tue, 03 Jun 2025 00:00:14 GMT</pubDate>
            <atom:updated>2025-06-16T15:54:09.997Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-pP8g7I6zQFiET9G-XhLmw.png" /></figure><p>Data migrations are risky. Data is sticky; once you put it somewhere, you need to spend a lot of time, energy, and egress fees trying to move it somewhere else. Let’s say you want to move all your data from The Big Cloud™ to another cloud. The gold standard process usually goes something like this:</p><ol><li>Figure out every service that could be using object storage and identify which buckets they use</li><li>Group those buckets by access tier (Standard, Infrequent Access, Glacier, etc)</li><li>Copy over all the data in several rounds to be sure that everything is migrated</li><li>Shut down everything</li><li>Copy over any remaining data</li><li>Swap out access keys and configuration</li><li>Start everything up</li><li>Hope it works</li></ol><p>This is a very stressful migration. It’s doable, but very very risky because there’s little ability to fall back to your old storage provider. This risk is usually so great that people (rightly) don’t want to even attempt to move object storage providers. It’s too easy to accidentally lose data in the process.</p><p>Sure, you can do a blue green deployment, and test in lower environments. But there’s almost always some legacy microservice, long forgotten but somehow in the critical path. During a migration I led, there was a service with a monthly job that delivered data to a customer. We only found out that service was never migrated when the customer complained weeks after the migration. No one wants this, and migrations don’t have to be a one way door.</p><p>At Tigris, we realize that this data inertia may make it hard for you to adopt Tigris for your object storage needs. So we implemented <a href="https://www.tigrisdata.com/docs/migration/">shadow bucket migration</a> to eliminate many of the risks associated with migrating object storage providers. When you set up shadow bucket migration, all the data on your old provider shows up in Tigris like it was there in the first place. When you access the data from Tigris, it gets copied over on demand.</p><p>Here’s what migrating to Tigris looks like:</p><ol><li>Create new buckets</li><li>Enable shadow bucket migration with write-through</li><li>Move services over one at a time</li><li>If you encounter problems, revert them to the old object storage provider</li><li>Once everything is moved over and you feel confident, remove the shadow bucket configuration</li></ol><p>We call this the “two way door” migration strategy.</p><h4>Migration Workflow Differences: Tigris vs other clouds</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBOgbBHnxxI7A7_2e1pnwg.png" /></figure><h3>Two Way Door Migration</h3><p>When folks talk about an incremental migration between object storage providers, they usually talk about migrating data in phases, either bucket by bucket or service by service. There’s other strategies you can use like making your application write data to both providers instead of one, or a read-through strategy where data is moved as it’s accessed.</p><p>Tigris shadow bucket migration reads data from your old cloud storage service as it’s accessed. If you enable <a href="https://www.tigrisdata.com/docs/migration/#enable-data-migration-in-the-tigris-dashboard">write-through support</a>, then any new data you write to Tigris seamlessly gets copied over to your old cloud storage service. This means that you can revert should you need to.</p><p>This write-through option further reduces the risk of moving to another object storage provider because undoubtedly you will forget about some long-untouched workload that nobody has documented anywhere. Write-through makes sure that the data keeps going to the old destination. This means that old workloads keep getting new data and gives you enough time to track it down with audit logs.</p><h4>How it works</h4><p>One of the best ways to think about Tigris is that it’s a multi-tiered cache system. When a request comes in, the data is pulled from:</p><ul><li>SSD cache</li><li>The local FoundationDB cluster</li><li>The local regional block storage cluster</li><li>Another regional block storage cluster in another region</li></ul><p>If any one of those fails to contain the data, Tigris moves on to the next tier of caching and tries again. If none of them succeed, Tigris returns a 404.</p><p>When you enable shadow bucket migration, you add a fifth tier of caching: your remote object storage service of choice. There’s additional niceties like regularly listing objects to know what’s available, but that’s pretty much it.</p><h3>An example</h3><p>Let’s say you have a bucket in Hetzner object storage. This bucket has over a terabyte of old data, backups, and other important data that you want to migrate over to Tigris. Here’s how you set up shadow bucket migration:</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DwQmZraGjqvw&amp;type=text%2Fhtml&amp;schema=youtube&amp;display_name=YouTube&amp;src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FwQmZraGjqvw%3Ffeature%3Doembed" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/14655eb61ca19c64c2595491b1a4c0a8/href">https://medium.com/media/14655eb61ca19c64c2595491b1a4c0a8/href</a></iframe><p>Create your bucket, punch your Hetzner object storage credentials into the settings, hit save, then try and download things. It’s trivial. Everything moves over on demand.</p><h4>Caveats</h4><p>In order to have the best experience with shadow bucket migration, make sure to do the following things:</p><ul><li>Make sure that your access credentials have the right IAM permissions for the buckets in your other cloud.</li><li>Make sure that objects in the Glacier access tier have been restored from Glacier into standard.</li><li>Make sure to take it easy, cloud migration can be stressful!</li></ul><h3>Conclusion</h3><p>We hope that this will make it easier for you to choose Tigris for your object storage needs. Shadow bucket migration helps you overcome the data inertia effect so that you can seamlessly upgrade your single-cloud data into becoming multi-cloud data by default.</p><h4>When should I use Tigris?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NmL3ywf7yRLojQgdsjzAjw.png" /></figure><figure><a href="https://www.tigrisdata.com/docs/get-started/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NoRS2o-rJPherGYNqgM4eg.png" /></a></figure><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/shadow-bucket/"><em>https://www.tigrisdata.com</em></a><em> on June 3, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f25341722f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fending off scrapers with Tigris and DuckDB]]></title>
            <link>https://tigris-data.medium.com/fending-off-scrapers-with-tigris-and-duckdb-9fffa78961d3?source=rss-49b902974434------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fffa78961d3</guid>
            <category><![CDATA[scraping]]></category>
            <category><![CDATA[anubis]]></category>
            <dc:creator><![CDATA[Tigris Data]]></dc:creator>
            <pubDate>Thu, 29 May 2025 00:00:55 GMT</pubDate>
            <atom:updated>2025-07-18T19:09:18.726Z</atom:updated>
            <content:encoded><![CDATA[<p>By day I work on DevRel at Tigris Data, and by night I’m <a href="https://void.rehab/notes/a6tp1qqquesrjpy5">a virtual anime person that makes software deployed by the United Nations</a>. A side project of mine took off recently, a Web AI Firewall Utility named <a href="https://anubis.techaro.lol/">Anubis</a>. Anubis blocks abusive scrapers from taking out web services by asserting that clients that claim to act like browsers do, in fact, act like browsers. As usage has grown (yes, the UN actually uses Anubis), we’ve gotten creative about detecting bots, including setting honeypots that give us more detailed data on their requests. These honeypot logs are sent to Tigris, and we analyze them using DuckDB to find patterns and improve bot detection.</p><p>In this article I’m going to be calling those abusive scrapers “AI scrapers”, but to be clear I’m not really sure if they are for training generative AI or not. However, the worst offender has been Amazon’s Alexa team so it’s pretty easy to take those two points and draw a line between them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qnk_yfjJEA2A0KJDI7VVeQ.png" /></figure><p>At a high level, Anubis has a big old set of rules in <a href="https://anubis.techaro.lol/docs/admin/policies">your bot policy file</a>. If clients match a rule, they either get allowed through, blocked, or selected for secondary screening. By default, Anubis is meant to instantly work by stopping all the bleeding and letting administrators sleep without downtime alerts waking them up. This means that it’s overly paranoid and aggressively challenges everything, similar to Cloudflare’s “I’m under attack” mode.</p><p>My intent was that admins would start out with Anubis being quite paranoid and then slowly lessen the paranoia as they find better patterns and match out ways to do things. Users tend to use Anubis in its default configuration, but this default configuration interferes with RSS feed readers and other “good bots”. Some users also have opinions about their privacy that lead them to disable things like cookie storage, JavaScript execution, and other fundamental parts of how the internet works. All of these interfere with how Anubis works.</p><p>I’d like to have Anubis throw challenges less often than it does right now, but there are a few key problems that interfere with this:</p><ul><li>Anubis is open source software, a critical part of the setup. Anubis is open source so that adoption is easier and so that users can trust that I am not doing malicious things in the name of security. It’s an asset as much as it is a liability because it means that attackers can (and have been) studying the open source code in order to bypass it better.</li><li>In order to show the challenge page less often, you need data. A lot of data. More specifically you need a general “shape” of what “known good” and “known bad” clients look like. For privacy, Anubis does not collect this data. This is intentional. You can look in <a href="https://github.com/TecharoHQ/anubis/">the code</a> to confirm for yourself should you desire.</li><li>Data that can be used to improve bot detection is sensitive- it can be used to identify individual users. Though tools like Tigris are equipped to handle sensitive data, me personally, I don’t want that responsibility right now.</li></ul><p>Finally, the biggest problem is that I don’t really know how to analyze data. I’m getting better at it, but generally if it’s a bit too big for me to either fit into a single SQLite database or excel sheet, it becomes a problem. I don’t have an analytics team (yet), so an easy-to-use toolchain for small data seems appropriate. Enter: DuckDB, a lightweight SQL DB that’s meant for exploratory queries with minimal overhead.</p><p>From here the rest is just wiring things up, getting access to the data I need, and figuring out how to analyze it.</p><h3>Everything’s a big data problem</h3><p>As the sacred texts have foretold:</p><blockquote><em>Given enough time, every problem becomes a big data problem.</em></blockquote><p>My sites are popular, sure, but I need more data than I can get myself. After experimentation, I set up some honeypot servers. I had a TLS terminating reverse proxy in my <a href="https://github.com/Xe/x">giant bag of experimental code</a>, so I slapped some logic on the side of it to dump information about every request to Tigris:</p><ul><li>Basic request metadata (method, path, query string)</li><li>HTTP header metadata (names and contents of HTTP headers, just in case there’s a pattern there)</li><li>Other connection-level metadata (TLS session details, other metadata like IP addresses)</li></ul><p>I set up two honeypots: one that gets talked about, and one that does not. Both had TLS certificates via Let’s Encrypt. It’s commonly thought that AI scrapers use the TLS certificate transparency log in order to find new targets to scrape. To attract the AIs, I ensured that the second honeypot had a cgit server with the source code to the Linux kernel. For some reason the AI scrapers seem to love requesting the “git blame” route. I won’t reveal all my tricks, but rest assured, the honeypot was loaded with AI Brawndo: it’s got what AI’s crave.</p><p>I set the trap and let it sit for a few weeks. I posted publicly about the known server, and I kept the other a secret outside of its listing in the TLS certificate transparency logs. I also asked ChatGPT, Gemini, Qwen, and DeepSeek about the git server, just to see if I managed to get lucky and get them thrown into the training dataset.</p><p>Spoiler alert: I got lucky.</p><h3>Peering at the data</h3><p>In Tigris, the honeypot log data is a giant pile of .jsonl files. For lack of a better idea, I named them after <a href="https://uuid7.com/">UUIDv7</a> values so that alphabetic sorting would result in each file being in temporal order. I set up <a href="https://github.com/tigrisdata/tigrisfs">tigrisfs</a> on my Fedora workstation and then started running grep to look for some patterns like what the OpenAI crawler does. Turns out that crawler is particularly noisy:</p><pre>{<br>  &quot;request_date&quot;: {<br>    &quot;seconds&quot;: 1748104928,<br>    &quot;nanos&quot;: 106832015<br>  },<br>  &quot;response_time&quot;: {<br>    &quot;nanos&quot;: 12817879<br>  },<br>  &quot;host&quot;: &quot;[redacted]&quot;,<br>  &quot;method&quot;: &quot;GET&quot;,<br>  &quot;path&quot;: &quot;/glibc.git/plain/locale/setlocale.c&quot;,<br>  &quot;query&quot;: {<br>    &quot;id&quot;: &quot;b647f210e61ab339fbb75dd9873daf7cb8f12665&quot;<br>  },<br>  &quot;headers&quot;: {<br>    &quot;Accept&quot;: &quot;*/*&quot;,<br>    &quot;Accept-Encoding&quot;: &quot;gzip, br, deflate&quot;,<br>    &quot;From&quot;: &quot;gptbot(at)openai.com&quot;,<br>    &quot;User-Agent&quot;: &quot;Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)&quot;,<br>    &quot;X-Openai-Host-Hash&quot;: &quot;589806218&quot;,<br>    &quot;X-Real-Ip&quot;: &quot;[redacted but in an Azure ASN]&quot;<br>  },<br>  &quot;remote_ip&quot;: &quot;20.171.207.44&quot;,<br>  &quot;ja4&quot;: &quot;t13d1011h2_61a7ad8aa9b6_3fcd1a44f3e3&quot;,<br>  &quot;request_id&quot;: &quot;0197032c-5b6a-7c7f-944d-8b2cfdb87258&quot;<br>}</pre><p>Other searches for openai got me here:</p><pre>{<br>  &quot;request_date&quot;: {<br>    &quot;seconds&quot;: 1747130654,<br>    &quot;nanos&quot;: 998119913<br>  },<br>  &quot;response_time&quot;: {<br>    &quot;nanos&quot;: 3273033<br>  },<br>  &quot;host&quot;: &quot;[redacted]&quot;,<br>  &quot;method&quot;: &quot;GET&quot;,<br>  &quot;path&quot;: &quot;/robots.txt&quot;,<br>  &quot;headers&quot;: {<br>    &quot;Accept&quot;: &quot;text/plain&quot;,<br>    &quot;Accept-Encoding&quot;: &quot;gzip, deflate, br&quot;,<br>    &quot;Cache-Control&quot;: &quot;no-cache&quot;,<br>    &quot;User-Agent&quot;: &quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot&quot;,<br>    &quot;X-Envoy-Expected-Rq-Timeout-Ms&quot;: &quot;13916&quot;,<br>    &quot;X-Openai-Internal-Caller&quot;: &quot;webcrawler-robots-txt&quot;,<br>    &quot;X-Openai-Originator&quot;: &quot;webcrawler-robots-txt&quot;,<br>    &quot;X-Openai-Originator-Env&quot;: &quot;prod&quot;,<br>    &quot;X-Openai-Product-Sku&quot;: &quot;unknown&quot;,<br>    &quot;X-Openai-Traffic-Source&quot;: &quot;user&quot;,<br>    &quot;X-Real-Ip&quot;: &quot;[redacted but in an Azure ASN]&quot;,<br>    &quot;X-Request-Id&quot;: &quot;577763b8-200d-49c4-985f-bd7cdaf1964b&quot;<br>  },<br>  &quot;request_id&quot;: &quot;0196c91a-2116-71ca-8a8a-429b4b14de76&quot;,<br>  &quot;response_code&quot;: 404<br>}</pre><p>This is plenty of metadata to be able to write a good set of rules for OpenAI in particular:</p><pre>- name: openai-crawler<br>  action: DENY<br>  expression:<br>    all:<br>      - userAgent.contains(&quot;https://openai.com/gptbot&quot;)<br>      - &#39;&quot;X-Openai-Host-Hash&quot; in headers&#39;<br>      - &#39;&quot;From&quot; in headers&#39;<br>      - headers[&quot;From&quot;] == &quot;gptbot(at)openai.com&quot;<br><br>- name: openai-robots-txt-fetcher<br>  action: DENY<br>  expression:<br>    all:<br>      - userAgent.contains(&quot;OAI-SearchBot&quot;)<br>      - &#39;&quot;X-Envoy-Expected-Rq-Timeout-Ms&quot; in headers&#39;<br>      - &#39;&quot;X-Openai-Internal-Caller&quot; in headers&#39;<br>      - &#39;&quot;X-Openai-Originator&quot; in headers&#39;<br>      - &#39;&quot;X-Openai-Traffic-Source&quot; in headers&#39;</pre><p>Or even to write a ruleset that matches clients that only partially look like OpenAI:</p><pre>- name: challenge-sus-openai-crawler<br>  action: CHALLENGE<br>  expression:<br>    all:<br>      - userAgent.contains(&quot;https://openai.com/gptbot&quot;)<br>      - &#39;!(&quot;From&quot; in headers)&#39;<br>      - &#39;!(&quot;X-Openai-Host-Hash&quot; in headers)&#39;<br><br>- name: challenge-sus-openai-robots-txt-fetcher<br>  action: CHALLENGE<br>  expression:<br>    all:<br>      - userAgent.contains(&quot;OAI-SearchBot&quot;)<br>      - &#39;!(&quot;X-Envoy-Expected-Rq-Timeout-Ms&quot; in headers)&#39;<br>      - &#39;!(&quot;X-Openai-Internal-Caller&quot; in headers)&#39;<br>      - &#39;!(&quot;X-Openai-Originator&quot; in headers)&#39;<br>      - &#39;!(&quot;X-Openai-Traffic-Source&quot; in headers)&#39;</pre><p>This should be all I need to make more educated guesses about how crawlers work so I can make it easier for website operators to interfere with the quality of service that the scraper bots get. In case you’re wondering, here’s how many requests OpenAI has made to my server:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*9G6cd2_YyEreZFVBrI382A.png" /></figure><p>All of these requests account for four hours of compute time on my server. During OpenAI’s most aggressive scrape, the server crashed and rebooted because it ran out of ram.</p><p>But as the scrapers evolve, so must my methods. I needed to dig deeper to make more educated guesses about how crawlers work and improve detection.</p><h3>Importing the logs to MotherDuck with DuckDB</h3><p>Now that I have data, how do I find patterns so that we can better detect the AI scrapers as they evolve? I’m more of an SRE than a data analyst, but I have a lot of experience with SQL. Normally I’d use SQLite for this because <a href="https://sqlite.org/lang_select.html">their documentation has the most lovely diagrams</a>, but SQLite has very poor support for nested data, such as HTTP headers. DuckDB is based on SQLite but adds extensions for things like arbitrary key/value pairs and strongly typed structures. This lets me take the JSON logs that the honeypots produce and turn them into something I can query.</p><p>MotherDuck offers a cloud SaaS built on DuckDB that gives you a notebook, storage, sharing, and hybrid execution across cloud and local data. The notebooks are pretty convenient, especially if I want to have fresh data as it gets written to Tigris. Having fresh data becomes important when I want to analyze traffic from a machine currently experiencing a heavy volume of scraping traffic.</p><p>Connecting your Tigris bucket o’ logs to MotherDuck is pretty simple: in your notebook, you use your Tigris URL:</p><pre>from &#39;s3://your-tigris-bucket/path/to/data&#39;</pre><p>And set the endpoint to the Tigris high performance endpoint in your secret config:</p><pre>CREATE OR REPLACE SECRET tigris<br>  (      TYPE  s3<br>  ,  PROVIDER  config<br>  ,    REGION  &#39;auto&#39;<br>  ,  ENDPOINT  &#39;t3.storage.dev&#39;<br>  , URL_STYLE  &#39;vhost&#39;<br>  );</pre><p>It’s also pretty easy to run DuckDB locally if your data is small enough to fit on your laptop. I’m pretty pro having the same toolchain for local development as I do in the cloud, so that’s a little nicety.</p><p>Now that we have the data loaded into a notebook, we can take a look at those honeypot logs. The JSON logs contains nested key-value pairs in two basic shapes:</p><ol><li>Known shapes, such as the timestamps and durations</li><li>Unknown shapes, such as request headers and query strings</li></ol><p>DuckDB lets me handle both of them. Here’s the SQL table I made after reading the <a href="https://duckdb.org/docs/stable/sql/data_types/overview">DuckDB schema docs</a> and converting <a href="https://github.com/Xe/x/blob/master/proto/relayd.proto">relayd’s HTTP request schema</a> by hand:</p><pre>CREATE TABLE requests<br>  ( request_date      STRUCT(&quot;seconds&quot; BIGINT, &quot;nanos&quot; BIGINT)<br>  , response_time     STRUCT(nanos BIGINT)<br>  , host              VARCHAR<br>  , &quot;method&quot;          VARCHAR<br>  , path              VARCHAR<br>  , headers           MAP(STRING, STRING)<br>  , remote_ip         VARCHAR<br>  , ja4               VARCHAR<br>  , request_id        UUID UNIQUE<br>  , request_protocol  VARCHAR<br>  , alpn              VARCHAR<br>  );</pre><p>Then I imported all of the data <a href="https://www.tigrisdata.com/docs/quickstarts/duckdb/#configure-duckdb-to-use-tigris">through DuckDB’s object storage functions</a>:</p><pre>INSERT INTO<br>  requests<br>    ( request_date  , response_time<br>    , host       , &quot;method&quot;<br>    , path       , headers<br>   , remote_ip  , ja4<br>   , request_id , request_protocol<br>    , alpn<br> )<br>SELECT<br>  *<br>, CASE<br>    WHEN ja4 LIKE &#39;t_______h1\_%&#39; ESCAPE &#39;\&#39; THEN &#39;HTTP/1.1&#39;<br>    WHEN ja4 LIKE &#39;t_______h2\_%&#39; ESCAPE &#39;\&#39; THEN &#39;HTTP/2.0&#39;<br>    WHEN ja4 LIKE &#39;q_______h3\_%&#39; ESCAPE &#39;\&#39; THEN &#39;HTTP/3.0&#39;<br>    ELSE NULL<br>  END AS alpn<br>, CASE<br>    WHEN alpn IS NOT NULL THEN alpn<br>    ELSE &#39;HTTP/1.1&#39;<br>  END AS request_protocol<br>FROM<br>  READ_JSON<br>    ( &#39;s3://relayd-logs/halone.within.lgbt/*.jsonl&#39;<br>    , columns =<br>      {  request_date: &#39;struct(&quot;seconds&quot; bigint, &quot;nanos&quot; bigint)&#39;<br>      , response_time: &#39;struct(nanos bigint)&#39;<br>      ,          host: &#39;varchar&#39;<br>      ,      &quot;method&quot;: &#39;varchar&#39;<br>      ,          path: &#39;varchar&#39;<br>      ,       headers: &#39;map(string, string)&#39;<br>      ,     remote_ip: &#39;varchar&#39;<br>      ,           ja4: &#39;varchar&#39;<br>      ,    request_id: &#39;uuid&#39;<br>      }<br>    , format = &#39;nd&#39;<br>    );</pre><h3>Off to the races</h3><p>Now that we have the data, we can start to look for patterns. Let’s start by looking at patterns involved with my own browser’s User-Agent string:</p><pre>SELECT COUNT(DISTINCT remote_ip)<br>  FROM requests<br>  WHERE headers[&#39;User-Agent&#39;] = &#39;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*yLeuPtRunzU_fDaOFjZ-8A.png" /></figure><p>Cool! 185 IP addresses. Let’s see if there’s any other things that pop up. Here’s Google Chrome hits:</p><pre>SELECT headers[&#39;User-Agent&#39;], COUNT(*) AS hits<br>  FROM requests<br>  WHERE headers[&#39;User-Agent&#39;] LIKE &#39;%Chrome/%&#39;<br>  GROUP BY headers[&#39;User-Agent&#39;]<br>  HAVING COUNT(*) &gt;= 76<br>  ORDER BY hits DESC;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*f64asXXtdZdbTk_yY9O_Mg.png" /></figure><p>So, let’s look at a random Google Chrome for Android request:</p><pre>{<br>  &quot;request_date&quot;: {<br>    &quot;seconds&quot;: 1746331998,<br>    &quot;nanos&quot;: 774116562<br>  },<br>  &quot;response_time&quot;: {<br>    &quot;nanos&quot;: 1002352<br>  },<br>  &quot;host&quot;: &quot;halone.within.lgbt&quot;,<br>  &quot;method&quot;: &quot;GET&quot;,<br>  &quot;path&quot;: &quot;/&quot;,<br>  &quot;headers&quot;: {<br>    &quot;Accept&quot;: &quot;text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7&quot;,<br>    &quot;Accept-Encoding&quot;: &quot;gzip, deflate, br, zstd&quot;,<br>    &quot;Accept-Language&quot;: &quot;en-US,en;q=0.9&quot;,<br>    &quot;Priority&quot;: &quot;u=0, i&quot;,<br>    &quot;Sec-Ch-Ua&quot;: &quot;\&quot;Chromium\&quot;;v=\&quot;136\&quot;, \&quot;Google Chrome\&quot;;v=\&quot;136\&quot;, \&quot;Not.A/Brand\&quot;;v=\&quot;99\&quot;&quot;,<br>    &quot;Sec-Ch-Ua-Mobile&quot;: &quot;?1&quot;,<br>    &quot;Sec-Ch-Ua-Platform&quot;: &quot;\&quot;Android\&quot;&quot;,<br>    &quot;Sec-Fetch-Dest&quot;: &quot;document&quot;,<br>    &quot;Sec-Fetch-Mode&quot;: &quot;navigate&quot;,<br>    &quot;Sec-Fetch-Site&quot;: &quot;cross-site&quot;,<br>    &quot;Upgrade-Insecure-Requests&quot;: &quot;1&quot;,<br>    &quot;User-Agent&quot;: &quot;Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Mobile Safari/537.36&quot;,<br>    &quot;X-Real-Ip&quot;: &quot;[mobile phone IP]&quot;<br>  },<br>  &quot;remote_ip&quot;: &quot;[mobile phone IP]&quot;,<br>  &quot;ja4&quot;: &quot;t13d1516h2_8daaf6152771_d8a2da3f94cd&quot;,<br>  &quot;request_id&quot;: &quot;0196997f-9a36-71c2-9662-6e7bf807b43d&quot;<br>}</pre><p>Something interesting about Chrome is that it sets a bunch of headers other than the User-Agent string and you can use those headers to pretty reliably identify Chrome.</p><p>Either way, these let you build a library of “known good” patterns so you can build fingerprinting methods on HTTP requests. There’s prior art here in the form of <a href="https://github.com/FoxIO-LLC/ja4/blob/main/technical_details/JA4H.md">JA4H</a>, but in testing against this dataset I didn’t have a sufficiently high match rate against malicious clients.</p><h3>Labeling</h3><p>One of the great things about this dataset is it’s all HTTP requests, which are a well known structure with many components. Patterns of interest could be based on the IP range the requests came from, HTTP headers, or other bits of metadata.</p><p>Every request has a unique ID, so it’s pretty easy to match requests into rules by creating a little table:</p><pre>CREATE TABLE rule_matches<br>  ( request_id TEXT<br>  , rule_name TEXT<br>  );<br><br>CREATE UNIQUE INDEX<br>  rule_matches_request_id_rule_name<br>ON<br>  rule_matches(request_id, rule_name);</pre><p>And then using INSERT INTO ... SELECT to fill it with data:</p><pre>INSERT INTO rule_matches (request_id, rule_name)<br>  SELECT request_id, &#39;chrome-without-sec-ch-ua&#39;<br>  FROM requests<br>  WHERE headers[&#39;User-Agent&#39;] LIKE &#39;%Chrome/%&#39;<br>    AND NOT map_contains(headers, &#39;Sec-Ch-Ua&#39;);<br><br>SELECT COUNT(*) FROM rule_matches WHERE rule_name=&#39;chrome-without-sec-ch-ua&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*2k3zm0EbikTdsuAh5KVmsA.png" /></figure><pre>INSERT INTO rule_matches (request_id, rule_name)<br>  SELECT request_id, &#39;generic-browser&#39;<br>  FROM requests<br>  WHERE headers[&#39;User-Agent&#39;] LIKE &#39;%Mozilla/%&#39;<br>    OR headers[&#39;User-Agent&#39;] LIKE &#39;%Opera/%&#39;;<br><br>select COUNT(*) from rule_matches where rule_name=&#39;generic-browser&#39;;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/282/1*4dzhPchHzuGLJc_tD1dCaA.png" /></figure><p>I’ve converted over a lot of the other <a href="https://github.com/TecharoHQ/anubis/tree/main/data">core/stdlib request matchers</a> and have found that they seem to work well against this dataset.</p><h3>Conclusion</h3><p>Though this starter dataset is much smaller than ram, as I collect more data, I’ll be more confident in the patterns I find. I’m working on an Ingress Controller named <a href="https://git.xeserv.us/Techaro/hythlodaeus">hythlodaeus</a> that helps my homelab and production clusters collect a filtered version of this request metadata. Then I can import that data into duckdb, find more patterns, and refine my filtering logic.</p><p>As AI Scrapers evolve, Anubis has to stay a couple steps ahead by spotting new patterns in the data. Lightweight tools like Tigris and DuckDB make it easy to query data and ultimately make the internet a better place by stopping bots.</p><p><em>Originally published at </em><a href="https://www.tigrisdata.com/blog/anubis/"><em>https://www.tigrisdata.com</em></a><em> on May 29, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fffa78961d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>