<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Koby Bass on Medium]]></title>
        <description><![CDATA[Stories by Koby Bass on Medium]]></description>
        <link>https://medium.com/@kobybass?source=rss-c3072716bcb4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*cZ_IAOwpB3zUjnWbz5lf7Q.jpeg</url>
            <title>Stories by Koby Bass on Medium</title>
            <link>https://medium.com/@kobybass?source=rss-c3072716bcb4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:43:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kobybass/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Typescript AWS Policies]]></title>
            <link>https://kobybass.medium.com/typescript-aws-policies-6ba631d7c56d?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ba631d7c56d</guid>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[typescript]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[cdktf]]></category>
            <category><![CDATA[policy]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Fri, 07 Mar 2025 19:26:17 GMT</pubDate>
            <atom:updated>2025-03-07T19:26:17.256Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6oiWIIIcUzJYfIme" /><figcaption>Photo by <a href="https://unsplash.com/@mattpopovich?utm_source=medium&amp;utm_medium=referral">Matt Popovich</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>IAM policies should be easy, reusable, and concise.</strong> Stop dealing with JSON strings — here’s how to make AWS access control painless with TypeScript.</p><p>After working with AWS for a while, I’ve found myself writing the same IAM policy patterns over and over. Whether you’re using CDK, Terraform, or just the AWS policy creation often involves copying JSON snippets and tweaking them for your specific resources.</p><p>I wanted a more TypeScript-friendly way to handle this common task, so I built @cdklib/aws-policy—a simple library that makes writing AWS policies simpler.</p><p>It’s designed to work with any TypeScript project, whether you’re using infrastructure-as-code tools or creating resources dynamically (such as tenant provisioning).</p><h3>The Problem with Traditional Approaches</h3><p>If you’ve worked with AWS IAM policies in TypeScript, you’re probably familiar with awkward patterns like these:</p><p><strong>Approach 1: JSON.stringify a raw object</strong></p><pre>const bucketPolicy = JSON.stringify({<br>  Version: &quot;2012-10-17&quot;,<br>  Statement: [<br>    {<br>      Effect: &quot;Allow&quot;,<br>      Action: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>      Resource: [&quot;arn:aws:s3:::my-bucket&quot;, &quot;arn:aws:s3:::my-bucket/*&quot;],<br>    },<br>  ],<br>});</pre><p><strong>Approach 2: Template literals</strong></p><pre>const bucketName = &quot;app-assets&quot;;<br>const policyJson = `{<br>  &quot;Version&quot;: &quot;2012-10-17&quot;,<br>  &quot;Statement&quot;: [<br>    {<br>      &quot;Effect&quot;: &quot;Allow&quot;,<br>      &quot;Action&quot;: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>      &quot;Resource&quot;: [<br>        &quot;arn:aws:s3:::${bucketName}&quot;,<br>        &quot;arn:aws:s3:::${bucketName}/*&quot;<br>      ]<br>    }<br>  ]<br>}`;</pre><p><strong>Approach 3: CDK policies with duplicated statement wrappers</strong></p><pre>new iam.PolicyDocument({<br>  statements: [<br>    new iam.PolicyStatement({<br>      effect: iam.Effect.ALLOW,<br>      actions: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>      resources: [`arn:aws:s3:::${bucketName}`, `arn:aws:s3:::${bucketName}/*`],<br>    }),<br>  ],<br>});</pre><h3>Issues with these approaches:</h3><ul><li>No TypeScript intellisense for action names or effect types</li><li>Duplication of Version and Statement wrapper boilerplate (which never changes)</li><li>Error-prone when modifying policies for multiple resources</li><li>Inconsistent approaches across your codebase</li></ul><h3>Introducing @cdklib/aws-policy</h3><p>At its core, @cdklib/aws-policy lets you:</p><ol><li>Create policies with TypeScript instead of JSON</li><li>Get intellisense and type checking for your policy statements</li><li>Build reusable policy templates with parameters</li><li>Easily import AWS examples into your codebase</li></ol><h3>Installation</h3><pre>npm install @cdklib/aws-policy</pre><h3>Basic Usage</h3><pre>import { AwsPolicy } from &quot;@cdklib/aws-policy&quot;;<br><br>const bucketPolicy = AwsPolicy.from(<br>  {<br>    Effect: &quot;Allow&quot;,<br>    Action: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>    Resource: [&quot;arn:aws:s3:::my-bucket&quot;, &quot;arn:aws:s3:::my-bucket/*&quot;],<br>  },<br>  {<br>    Effect: &quot;Deny&quot;,<br>    Action: &quot;s3:DeleteObject&quot;,<br>    Resource: &quot;arn:aws:s3:::my-bucket/*&quot;,<br>  }<br>);<br>const policyJson = bucketPolicy.toJson();</pre><p>With this approach:</p><ul><li>The Version field is automatically added</li><li>TypeScript provides autocompletion and validation</li><li>Policies are easier to modify and maintain</li></ul><h3>Importing AWS Examples</h3><p>Many times, you just want to copy an example from the AWS docs and use it in your code.</p><p>With @cdklib/aws-policy, you can paste policy statements directly into TypeScript:</p><pre>const policy = AwsPolicy.from({<br>  Effect: &quot;Allow&quot;,<br>  Action: &quot;s3:ListBucket&quot;,<br>  Resource: &quot;arn:aws:s3:::example_bucket&quot;,<br>});</pre><h3>Reusable Policy Templates</h3><p>Often, you need similar policies with minor differences. This library allows you to create templates for reuse:</p><pre>import { AwsPreparedPolicy } from &quot;@cdklib/aws-policy&quot;;<br><br>const s3BucketPolicy = new AwsPreparedPolicy&lt;{ bucketName: string }&gt;(<br>  ({ bucketName }) =&gt; ({<br>    Effect: &quot;Allow&quot;,<br>    Action: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>    Resource: [<br>      `arn:aws:s3:::${bucketName}`,<br>      `arn:aws:s3:::${bucketName}/*`,<br>    ],<br>  })<br>);<br>const userDataPolicy = s3BucketPolicy.fill({ bucketName: &quot;user-data&quot; });<br>const appAssetsPolicy = s3BucketPolicy.fill({ bucketName: &quot;app-assets&quot; });</pre><p>This eliminates duplicate code and keeps policies type-safe.</p><h3>Combining Policies</h3><p>You can also combine multiple policy templates:</p><pre>const combinedPolicy = AwsPreparedPolicy.combine(<br>  s3ReadPolicy,<br>  lambdaInvokePolicy<br>);<br><br>const policy = combinedPolicy.fill({<br>  bucketName: &quot;my-bucket&quot;,<br>  functionName: &quot;my-function&quot;,<br>});</pre><p>This ensures consistency while keeping policies modular and reusable.</p><h3>Integration with CdkConfig</h3><p>If you’re using the @cdklib/config library I explained in my <a href="https://koby.one/blog/DevOps/cdk-config/">previous post</a>, you can create policies that use the CDK scope to access configuration:</p><pre>import { AwsPreparedPolicy } from &quot;@cdklib/aws-policy&quot;;<br>import { awsConfig } from &quot;./config/aws&quot;;<br><br>// Define a policy that includes scope as a parameter<br>const s3BucketPolicy = new AwsPreparedPolicy&lt;{<br>  scope: Construct;<br>  bucketName: string;<br>}&gt;(({ scope, bucketName }) =&gt; {<br>  // Get config values from scope<br>  const { accountId } = awsConfig.get(scope);<br><br>  return {<br>    Effect: &quot;Allow&quot;,<br>    Action: [&quot;s3:GetObject&quot;, &quot;s3:ListBucket&quot;],<br>    Resource: [`arn:aws:s3:::${bucketName}`, `arn:aws:s3:::${bucketName}/*`],<br>    Principal: {<br>      AWS: `arn:aws:iam::${accountId}:root`,<br>    },<br>  };<br>});<br><br>// Provide scope and parameters<br>const policy = s3BucketPolicy.fill({<br>  scope: myApp,<br>  bucketName: &quot;app-assets&quot;,<br>});</pre><h3>Closing Thoughts</h3><p>The @cdklib/aws-policy library is a small but powerful utility that makes working with AWS IAM policies easier in TypeScript projects.</p><p>The @cdklib/aws-policy library is a small but powerful utility that makes working with AWS IAM policies easier in TypeScript projects.</p><p>The library is open source and available on GitHub, where you can find more examples and documentation. Feel free to use it, modify it, or build on it to fit your needs.</p><p>For more details and CDK libraries, check out the <a href="https://github.com/kobybum/cdk-libs">@cdklib project readme</a>.</p><p>This post is also available on <a href="https://koby.one/blog/DevOps/aws-policy">my blog</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ba631d7c56d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dev Environments — Why and How]]></title>
            <link>https://kobybass.medium.com/dev-environments-why-and-how-bed2192a9758?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/bed2192a9758</guid>
            <category><![CDATA[argo-cd]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[development]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 22:51:14 GMT</pubDate>
            <atom:updated>2024-03-18T22:51:14.094Z</atom:updated>
            <content:encoded><![CDATA[<h3>Dev Environments — Why and How</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EZbECbXyAET0vUY0" /><figcaption>Cloud Env = Happy Dev :)</figcaption></figure><h3>Introduction</h3><p>In this article, we will discuss why you need a developer environment in your organization, what you should strive for, and how to go about implementing one.</p><p>This is quite a large topic to cover, so we will not go into any implementation. Instead, I will try to guide you through the thought process of integrating a good tool while utilizing known solutions, like a puzzle.</p><p>Recently, while working at <a href="https://layerxsecurity.com/">LayerX</a>, our dev environment tool Raftt was acquired and stopped maintaining the product. We had to make a decision — swap to a different solution, or build one in-house.</p><h3><em>What we want</em></h3><p>As a developer, I want to be able to run a copy of my application as I develop it. I don’t care where it runs or how it works. It should run all the services, wire them up, and make me feel like I’m serving it locally.</p><p>The problem is, the application often consists of multiple services, frontend and databases. You may think docker is a good solution, but as your codebase and system complexity grows, your machine will not be able to handle it, and will start to sound like an apache helicopter taking off.</p><h3>Requirements</h3><ul><li>Each developer should have a separate environment</li><li>Run in the cloud to allow scaling the system</li><li>Hot reload — When changes are made, they should be reflected in the environment</li><li>Port forwarding — the system should <em>feel</em> as though it’s running locally</li><li>Database seeding — The database should be pre-loaded with application data, and we should be able to run migrations against it.</li></ul><h4>Existing solutions</h4><p>I always prefer to use a SaaS solution when possible, and advice you to do the same. At the time of writing, both <a href="https://www.okteto.com/">Okteto</a> (expensive), or <a href="https://tilt.dev/">Tilt</a> (runs locally) are viable solutions — please take a look at them before starting to develop and maintain you in-house solution.<br><strong>Keep in mind</strong>, that an in-house solution may be superior, as it allows you to make assumptions to optimize the process. All solutions require a lot of work to integrate!</p><h4>A word about cost</h4><p>You might feel discouraged from running these environments in the cloud — it’s expensive and unnecessary. Or, you might feel like this is unnecessary, and you can just serve your services using node.</p><p>Consider how your devs will thrive with such a tool. No more discrepancies between environments, your devs can verify everything E2E while developing without a hassle. You have less bugs, deliver faster features. You encourage E2E features, ownership and confidence within your R&amp;D.</p><p>Still think it’s not worth it? Contrast the cost with the developer salary, and how much time they’re wasting on bugs that could’ve been found before deployment. I’m not exaggerating when saying a dev environment is one of the most important tools for your R&amp;D.</p><h3>Designing a Solution</h3><p>Let’s make a few assumptions that will help us create a solution that is not as complicated as Okteto -</p><ul><li>Our backend consists of single language. We’ll focus on nodejs in this article.</li><li>We have a working Kubernetes with autoscaling to utilize for remote compute</li><li>We have built docker images of our base commit from main, which lack our code but have the base tools and most packages to run the app (node and node_modules).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruiZCpr0IQiQNZLP5hHMZw.png" /><figcaption>Solution, outlined in two flows</figcaption></figure><h4>Initial Setup</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5_vDPRczKEG31f5qokINCw.png" /><figcaption>Initial setup flow</figcaption></figure><ul><li>Deploy a database (optional) and seed it. You can skip this if you don’t want to seed the database, but it’s recommended you implement this at some point to create a better developer experience.</li><li>Deploy the latest docker images</li><li>Setup port forwarding</li></ul><h4>Local Changes Syncing</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXzhTbeHkg0CoNMQX8ZKzA.png" /><figcaption>Syncing local changes</figcaption></figure><p>After the initial setup, we should have a working replica of the staging environment. However, we lack the local changes we made.</p><p>Every time we save, and initially, we’d like to sync those local changes to our environment. This consists of:</p><ul><li>Watching for changes</li><li>Building the app (using webpack or any other build tool)</li><li>Sync packages (copy the local package.json file and run npm)</li><li>Hot reload — Copy the built file main.js to the remote, and reload the process. This can be achieved with nodemon.</li></ul><h3>Implementation Details</h3><p>Implementing a solution will change between companies, depending on your tech stack. Our tech stack includes some useful tools that made the job easier, which you may decide to utiliize -</p><h4>Tool Usability</h4><p>Aim for a good developer experience — The developer should be able to run dev up and dev down or something similar. Provide good errors, and help the developer understand if something is wrong.</p><p>For example, if a project fails to bulild, it should be easily diagnosed using a log file and something like dev status .</p><p>Commands you should consider (look at okteto for more) —</p><ul><li>dev status — Provide status for deploying, building, syncing and port forwarding</li><li>dev port-forward — Ensure and fix the port forwarding</li><li>dev sync — Force the syncing process to restart, in case of build errors and such.</li></ul><p>To achieve this, you want some sort of CLI component to spawn the environment. Then, decide between a web UI and a CLI to debug and use the tool.</p><h4>ArgoCD / Flux</h4><p><a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a> allows you to manage your application on Kubernetes using GitOps. Luckily, we utilized this tool from day one. We already had all our cluster definitions, and creating a ‘dev’ environment was as easy as tuning the resources and HPAs.</p><p>If you’d like to learn more about ArgoCD and why it’s amazing, <a href="https://www.youtube.com/watch?v=MeU5_k9ssrs">I recommend this tutorial</a>.</p><p><strong><em>If you’d like</em></strong> to use GitOps for your solution, you could use the following flow —</p><ul><li>When spawning an environment, <strong>create a branch</strong> for the developer in the GitOps repo.</li><li>Make the necessary changes to the <strong>YAML definitions</strong> — Reduce the resources, scale down the deployments, and duplicate any secrets. You may want to replace the container command (from node to nodemon for example)</li><li>Create an ArgoCD <a href="https://argo-cd.readthedocs.io/en/stable/user-guide/application-set/">ApplicationSet</a> on the remote cluster to track the remote branch, for example dev/koby . Deploy the application set to a separate namespace per developer, for example dev-koby</li><li><strong>ArgoCD will deploy</strong> the changes. If you’re using seeding, you might want to apply a subset of the app (DBs only) first, seed, then deploy the entire app.</li><li><strong>Port forward</strong> using a kubernetes client (or kubectl).</li><li>When deleting the environment, simply delete the ApplicationSet. ArgoCD will take care of the rest.</li></ul><h4>Change Management</h4><p>Take inspiration from your language build tools to watch for changes. We use nx monorepo, so we had the watch functionality out of the box.</p><ul><li>Watch functionality — Detect what <strong>project</strong> has changed</li><li>Detect if package syncing is needed</li><li>Copy the files to the remote container (using kubernetes client or kubectl)</li><li>Reload the process (prefer existing tools like nodemon) — If you decide to kill the process, it may kill the container so be careful</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bed2192a9758" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python contextvars and multithreading]]></title>
            <link>https://kobybass.medium.com/python-contextvars-and-multithreading-faa33dbe953d?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/faa33dbe953d</guid>
            <category><![CDATA[multithreading]]></category>
            <category><![CDATA[elastic-apm]]></category>
            <category><![CDATA[concurrency]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Thu, 08 Jul 2021 00:14:02 GMT</pubDate>
            <atom:updated>2021-07-08T14:00:59.200Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why can’t they be friends?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GfNGxWf_a23-VhXb" /><figcaption>Photo by <a href="https://unsplash.com/@paolocandelo?utm_source=medium&amp;utm_medium=referral">paolo candelo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What is it?</h3><h4>contextvars</h4><p>Sometimes in our code we find that every function relies on a single parameter, like the current user_id or access_token . In single threaded code, this can be solved by a global variable or the Singleton pattern.</p><p>For most applications, concurrency is required to support multiple users at the same time, or just make things faster. contextvars solves this by creating consistent variable per thread:</p><pre>import contextvars<br>from threading import Thread</pre><pre>current_user = contextvars.ContextVar(&quot;Name of current user&quot;)</pre><pre>def handle_request(user):<br>    current_user.set(user)<br>    print_hello()</pre><pre>def print_hello():<br>    print(f&quot;Hello, {current_user.get()}&quot;)</pre><pre>Thread(target=handle_request, args=(&quot;koby&quot;, )).start()<br>Thread(target=handle_request, args=(&quot;world&quot;, )).start()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*L8FC3m7-bafjdVWDFY4jMQ.png" /><figcaption><strong>Running the script, we see context vars are passed between functions.</strong></figcaption></figure><p>Notice we no longer need to pass the user down to print_hello. In a real scenario, there may be a call chain of 10+ functions, and only the last function needs the user.</p><h4>ThreadPoolExecutor</h4><p>A common way to run your code concurrently is ThreadPoolExecutor . For example, if we want to say hello and at the same time offer our user sympathies for our terrible service:</p><pre>import time<br>from concurrent.futures import ThreadPoolExecutor, as_completed</pre><pre>def say_hello():<br>    time.sleep(1)<br>    return &quot;Hello, user&quot;</pre><pre>def offer_sympathies():<br>    time.sleep(1)<br>    return &quot;Very sorry, user&quot;</pre><pre>with ThreadPoolExecutor() as exec:<br>    tasks = (say_hello, offer_sympathies)<br>    futures = (exec.submit(task) for task in tasks)<br>    for future in as_completed(futures):<br>        print(future.result())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*JOpSlrZUeR-0XGOj02BA_g.png" /><figcaption><strong>Using </strong><strong>ThreadPoolExecutor we saved our user a second and made the user very confused.</strong></figcaption></figure><h3>ThreadPoolExecutor and contextlib don’t play nice</h3><p>Unfortunately, at the time of writing, the executor does not propagate context variables our of the box. So if we want to pass a user down a long chain of functions using contextvars , using an executor will break the chain:</p><pre>import contextvars<br>import time<br>from concurrent.futures import ThreadPoolExecutor, as_completed</pre><pre>current_user = contextvars.ContextVar(&quot;Name of current user&quot;)</pre><pre>def say_hello():<br>    time.sleep(1)<br>    return f&quot;Hello, {current_user.get()}&quot;</pre><pre>def offer_sympathies():<br>    time.sleep(1)<br>    return f&quot;Very sorry, {current_user.get()}&quot;</pre><pre>current_user.set(&quot;koby&quot;)</pre><pre>with ThreadPoolExecutor() as exec:<br>    tasks = (say_hello, offer_sympathies)<br>    futures = (exec.submit(task) for task in tasks)<br>    for future in as_completed(futures):<br>        print(future.result())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*B6Y5kWq808h1aOROOXojog.png" /><figcaption><strong>Rather than greeting the user, we raise an exception.</strong></figcaption></figure><p>Terrible. Our current_user variable is lost inside the thread! Luckily, we can fix this relatively easily, using two executor parameters. From the <a href="https://docs.python.org/3/library/concurrent.futures.html">python docs</a>:</p><blockquote><strong>initializer</strong> is an optional callable that is called at the start of each worker thread; <strong>initargs</strong> is a tuple of arguments passed to the initializer.</blockquote><p>Using this, all we have to do is provide the original context to the initializer function, and set all the context variables in the new thread. This can be done using a simple function:</p><pre>def set_context(context):<br>    for var, value in context:<br>        var.set(value)</pre><pre>set_context(contextvars.copy_context())</pre><p>Using this function, our code will work as expected:</p><pre>import contextvars<br>import time<br>from concurrent.futures import ThreadPoolExecutor, as_completed</pre><pre>current_user = contextvars.ContextVar(&quot;ID of current user&quot;)</pre><pre>def say_hello():<br>    time.sleep(1)<br>    return f&quot;Hello, {current_user.get()}&quot;</pre><pre>def offer_sympathies():<br>    time.sleep(1)<br>    return f&quot;Very sorry, {current_user.get()}&quot;</pre><pre>current_user.set(&quot;koby&quot;)</pre><pre>def set_context(context):<br>    for var, value in context:<br>        var.set(value)</pre><pre>with ThreadPoolExecutor() as exec:<br>    tasks = (say_hello, offer_sympathies)<br>    futures = (exec.submit(task) for task in tasks)<br>    for future in as_completed(futures):<br>        print(future.result())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*c5CDbCnOPjXziwqlKxfung.png" /><figcaption><strong>Current user is properly passed to child threads</strong></figcaption></figure><h4>Other Options</h4><p>Instead of explicitly calling copy_context and passing the arguments every time, you could make a subclass to help you:</p><pre>class ContextExecutor(ThreadPoolExecutor):<br>    def __init__(self):<br>        self.context = contextvars.copy_context()<br>        super().__init__(initializer=self._set_child_context)</pre><pre>    def _set_child_context(self):<br>        fo<em>r</em> var, value in self.context.items():<br>            var.set(value)</pre><p>Or, if you prefer a functional approach, a wrapper function:</p><pre><a href="http://twitter.com/contextlib">@contextlib</a>.contextmanager<br>def ContextExecutor():<br>    parent_context = contextvars.copy_context()<br>    with ThreadPoolExecutor(<br>        initializer=set_context,<br>        initargs=(parent_context, )<br>    ) as executor:<br>        yield executor</pre><p>All of these method assume the context is not changed inside the with statement, which can cause race conditions and should be avoided.</p><h3>Wrapping Up</h3><p>If you use contextvars, using an executor will break the context between threads. You can either move to asyncio which propagates context out of the box, or make a simple wrapper to pass context variables.</p><p>Even if you don’t use contextvars, there’s a good change a library you’re using is passing around arguments behind your back for you convenience using contextlib. Executors will break the library behavior and you won’t be sure why.</p><p>For example, <a href="https://github.com/elastic/apm-agent-python">elastic APM python client</a> uses contextlib to pass transaction information between functions, and breaks when using executors.</p><p>Hope this helped, if you have a better solution or an opensource library feel free to share. When I was researching this it was harder than it should be, so hopefully this saved you time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=faa33dbe953d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ORMs vs Plain SQL in Python]]></title>
            <link>https://kobybass.medium.com/orms-vs-plain-sql-in-python-2ba5362bca21?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba5362bca21</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[sqlalchemy]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Sun, 08 Sep 2019 19:16:24 GMT</pubDate>
            <atom:updated>2019-09-08T19:28:29.983Z</atom:updated>
            <content:encoded><![CDATA[<h4>Trading developer time for clean code and optimizations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tvxJ2cwGG8yS-UiA" /><figcaption>Photo by <a href="https://unsplash.com/@cadop?utm_source=medium&amp;utm_medium=referral">Mathew Schwartz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>As part of the Marketing R&amp;D team at Lightricks, we encountered performance and clean code issues when using Django ORM for our database management. We decided to move to a plain SQL approach in newer projects to facilitate database optimizations and better boundaries in our code.</p><p>In this article I’d like to discuss using ORMs vs writing plain SQL queries.</p><p>In the next article, if you decide to use plain SQL, I’ll explain how we use Flyway as our migration tool for schema management in a CI/CD workflow.</p><h3>ORM vs Plain SQL</h3><p>Working with python, it’s common to use an SQL abstraction like Django ORM or SQL alchemy. While these approaches work well for new applications, they quickly fail at scaling your data or your team.</p><p>When working with large applications, optimizations to the database structure and queries often need to be made. For example, as our database grows we may want to <a href="https://www.postgresql.org/docs/10/ddl-partitioning.html">partition it</a> into multiple sub-tables. Or perhaps we need control of our query to make database specific optimizations.<br>We encountered both problems, and had to write most of the select queries ourselves, and redefine tables created by Django.</p><p>On the surface, using an SQL abstraction is very tempting:</p><ul><li><strong>Tests</strong> are very clean and don’t require extra code.</li><li><strong>Record inserts </strong>are very easy to implement, assuming we are passing ORM objects in our code, since we just need to call .save()</li><li><strong>SQL Abstraction</strong> — ability to easily switch between database implementations, without writing any SQL.<br>No need to write SQL queries</li><li><strong>Migrations</strong> are automatically generated and come with a built-in migration tool.</li></ul><p>This saves a lot of development time when you first start the project, but comes at a cost:</p><h4>Coupling</h4><p>Since ORM models already contain all the data you need, your team will be tempted to use it outside of database related code.</p><p>For example, say you’re fetching data for a Todo list, you’ll probably create the model instance in the parsing code and start passing it around to other functions. However, ORM models are often mutable, and have information that has nothing to do with most of your code. Any function that receives the model can change it, which makes code hard to debug and test.</p><p>Instead, if you create proper boundaries with immutable objects(using a NamedTuple / attrs / dataclasses ), you’ll make your code much simpler and easier to test, at the cost of a few more lines of code.</p><h4>Schema/Query Optimizations</h4><p>While ORMs work fine when you first start out, as your data grows you’ll want to optimize the table structure and queries:</p><ul><li>Partition tables by timestamp/country/client etc.</li><li>Use engine specific tricks to increase performance tenfold.</li></ul><p>When you encounter these problems, you’ll need to write SQL queries manually anyway, and write migrations yourselves, so using ORMs suddenly doesn’t save that much time.</p><h3><strong>Summary</strong></h3><p>If you’re planning on working in a small group on a small project, an ORM approach will save you plenty of time. But if you anticipate big amounts of data and scaling your app to more developers, I suggest you try using plain SQL instead.</p><p>If you do use an ORM, I suggest you only use the database objects in separate data layer files, to make code more functional, cleaner and easier to test.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CR3fOEPXNsCVRnad" /><figcaption>Photo by <a href="https://unsplash.com/@hagalnaud?utm_source=medium&amp;utm_medium=referral">Anna Samoylova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Please feel free to leave your thoughts below, as this is written from personal experience and getting input is always neat.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ba5362bca21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding Dead Python Files with Snakefood]]></title>
            <link>https://kobybass.medium.com/finding-dead-python-files-with-snakefood-6c75a3e82294?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c75a3e82294</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[refactoring]]></category>
            <category><![CDATA[dependencies]]></category>
            <category><![CDATA[code]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Wed, 12 Dec 2018 22:16:12 GMT</pubDate>
            <atom:updated>2018-12-12T22:16:12.503Z</atom:updated>
            <content:encoded><![CDATA[<p><em>And cleaning them up.</em></p><p>TLDR;<strong> </strong><em>Download snakefood and look at </em><a href="https://gist.github.com/kobybum/a162be138b2d19e3eac20547a7f10fa3"><strong><em>this gist</em></strong></a><em>.</em></p><p>Recently I started working on a python project with a large and mature codebase. During development, the codebase was evolving and some parts were left behind, unused.</p><p>To clean things up, I wanted to find all unused python modules in the project.</p><p><a href="https://github.com/jendrikseipp/vulture"><strong>Vulture</strong></a> is a great tool to find unused code inside your files, but I wanted to find whole files that weren’t being used. For this, I needed some dependency analysis tool.</p><p><a href="https://coverage.readthedocs.io/en/v4.5.x/"><strong>Coverage</strong></a><strong> </strong>allows us to see which code is being used by running our project, but we have many entrypoints and code flows.</p><p><a href="http://furius.ca/snakefood/"><strong>Snakefood</strong></a><strong> </strong>is a command line tool that analyses dependencies, and can great graphs from them. I decided to use the dependency analysis to find which files weren’t included by any of our code.</p><p>A summary is available in <a href="https://gist.github.com/kobybum/a162be138b2d19e3eac20547a7f10fa3"><strong><em>this gist</em></strong></a><strong><em>, </em></strong>and I explain the details below.</p><h3>How It Works</h3><p>To generate a dependency file, we run sfood with the internal flag -ion our directory. The internal flag will exclude any files outside of our project. Since this can take some time, we cache results to a temporary file:</p><pre>sfood -i &lt;dir&gt; &gt; /tmp/out.deps</pre><p>The dependency file contains many dependency lines, like the following:</p><pre># file main.py depends on common/redis.py<br>((&#39;/tmp&#39;, ‘project/main.py’), (&#39;/tmp&#39;, ‘project/common/redis.py’))</pre><p>We’re interested in which files are being dependant on, using pipes we can create a command that parses those:</p><pre>cat /tmp/out.deps | \ <br>    grep -v test | \<br>    cut -d&quot;&#39;&quot; -f8 | \ <br>    sort | \<br>    uniq &gt; /tmp/required.txt</pre><ul><li>Split the dependency by &#39; and get the 8th field, the filename.</li><li>Sort filenames and get unique filenames</li><li>Save result for later comparison.</li></ul><p>We can generate a list of all our python files using some bash scripting:</p><pre>find . -name &#39;*.py&#39; | \<br>    grep -v &quot;__init__.py&quot; | \<br>    grep -v &quot;test&quot; | \<br>    sort &gt; /tmp/modules.txt</pre><p>Comparing those files, we get what we wanted, the list of files that aren’t dependant on:</p><pre>&gt; diff /tmp/modules.txt /tmp/required.txt</pre><pre>project/main.py<br>project/common/unused.py</pre><p>Filtering out our entry point project/main.py and framework files, we get our list of dead files.</p><h3>EOF</h3><p>That’s it. Hope it helped someone, wasn’t enough information about this when I was searching. Comment below if you know a better way, I’ll be glad to hear it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c75a3e82294" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro to Crypto Arbitrage]]></title>
            <link>https://kobybass.medium.com/lessons-from-crypto-arbitrage-8739b62544bc?source=rss-c3072716bcb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8739b62544bc</guid>
            <category><![CDATA[arbitrage]]></category>
            <category><![CDATA[bitcoin]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Koby Bass]]></dc:creator>
            <pubDate>Sun, 14 Oct 2018 02:34:49 GMT</pubDate>
            <atom:updated>2018-10-14T09:55:46.009Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vcKcjl-kfX6Gqqz8" /><figcaption>Photo by <a href="https://unsplash.com/@silverhousehd?utm_source=medium&amp;utm_medium=referral">Andre Francois</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>An Intro to Cryptocurrency Arbitrage</h3><h4>Approaches and common pitfalls to profit trading cryptocurrencies</h4><p>About six years ago, I went out to launch with a good friend of mine to escape the awful food at our workplace. He casually mentioned he made good money buying and selling cryptocurrencies on a local exchange.</p><p>At the time, I did not know much about how markets and cryptocurrency exchanges work. Naturally, I was skeptical of the idea of making easy money. But the numbers were real, and coming from a programming background I offered to automate the trading. Since then, I’ve had a lot of experience in the field, and would like to share some lessons and common pitfalls I’ve encountered during this time.</p><p>This article is intended as an overview of the field. I will explain general concepts, strategies, pitfalls and ways of automating trading tasks.</p><h3>How Markets Work — General Concepts</h3><h4>Order Books</h4><p>On every public market, there are buyers and sellers. People write down how much they’re willing to buy and sell, and at which price. This information is saved on the <strong>Order Book</strong>, which contains all the open orders people made.</p><p>If two people agree on a price, a trade is made and the broker gets an agreed upon <strong>fee</strong>. The rest of the order book contains the open orders which did not find a match. A typical Bitcoin to USD order book will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*XUfTDyIgvdetrntZpPnZfw.png" /><figcaption>BTC/USD order book from yobit</figcaption></figure><h4>Spread</h4><p>Every listing includes the price, amount (In Bitcoin) and price per Bitcoin the person is willing to pay. As you can see, there’s a price difference between the buy and sell orders. This difference is referred to as the <strong>spread</strong>. Generally, the more volume the pair has, the smaller the spread will be.</p><h4>Makers and Takers</h4><p>People who buy from available offers on the order book are called <strong>takers</strong>, because they take liquidity from the order book, and increase the spread.</p><p>Conversely, a <strong>maker </strong>is someone who puts down his own offers and waits for others to take them. Makers often get much smaller fees than takers, because they reduce the spread.</p><p>If the spread is big enough to cover the fees, you could potentially make money by putting both buy and sell orders. This is called <strong>market making</strong>, and essentially it’s the technique my friend used 6 years ago.</p><h4>Arbitrage</h4><p>In the cryptocurrency world in particular, there are a lot of different exchanges with different order books. Right now, I can buy Bitcoin at cex.io at a rate of 6200$, and sell it on bitfinex for 6300$, making a whopping 2%! Easy money, right? This price difference is called <strong>arbitrage</strong>, and is not as easy as it seems. I will explain why in the next section.</p><h4>Pairs</h4><p>A <strong>pair</strong> is a pair of two coins which are traded with one another. For example, common pairs are BTC/USD, which means selling bitcoin for US Dollars, or ETH/EUR, which means selling Ethereum for euros.</p><h4>Price delays</h4><p>Usually, larger exchanges calibrate their price quicker, and smaller exchanges follow after them. This means, that you could predict the price on a small exchange by looking at a bigger one if you’re fast enough.</p><p>This requires a more complex platform, because you need to be aware of prices on multiple exchanges and react quickly enough, so it’s only possible with software.</p><h4>Triangular Arbitrage</h4><p>Sometimes there’s a price difference between pairs in the same exchange.<br>For example, if we can buy Bitcoin on BTC/USD for 1000$, sell on BTC/EUR for 900€, and exchange the Euros for 1020$. In reality, the difference is smaller and you have to account for the triple fee for the transaction.</p><p>Knowing this, let’s review the options for making money exploiting price arbitrage:</p><ol><li>Exploit arbitrage between exchanges, also called <strong>inter-exchange arbitrage</strong>.</li><li>Using <strong>Triangular Arbitrage</strong></li><li><strong>Market making</strong> on a pair with high <strong>spread</strong>.</li><li>Exploiting <strong>price delays</strong>.</li></ol><h3>Intra-Exchange Arbitrage Pitfalls</h3><p>People often see the price difference between exchanges and think they found their golden ticket. However, making money from this difference is not trivial often a reason for it. This is especially true when dealing with FIAT currencies, such as US Dollar, Russian Rubble and the like. Let’s review what we need to do to profit from this:</p><ol><li>Buy bitcoin on cex.io at 6200$ and pay the fee of 0.16%, which leaves us with roughly 6215$.</li><li>Pay bitcoin withdrawal fee and transaction cost on cex.io, which is negligible.</li><li>Move the bitcoin from cex.io to bitfinex. This requires a bitcoin transaction, which can take roughly 10 minutes.</li><li>Sell the bitcoin on bitfinex for 6300$, which after the 0.2% fee is roughly 6290$.</li></ol><p>Most people think the deal ends here, and we just made more than 1% in 10 minutes. However, there’s a subtle this is the ideal situation and several things could go wrong, losing you money:</p><ol><li>During the transaction period the price may change drastically, and when it does you can lose a lot of your money. This time can be reduced by working with faster coins like Litecoin or Dash, but then you have to make additional trades which may take time and more fees.</li><li>When scaling this to larger sums of money, your orders will drive the buy prices higher, because demand goes up. On small exchanges, this change will be significant and you’ll find it’s not profitable to buy more than a certain amount.</li><li>Fiat(USD, EUR, RUR etc.) currencies are valued differently across exchanges. This is caused by different fees on deposits and withdrawals, because you have to make a bank transfer to withdraw this money. If an exchange has a 5% higher withdrawal fee on USD for example, the price of all coins in USD will be 5% higher on that exchange.<br>This means you’ll have to pay the fee to get your money back, and is the main factor for the price difference between exchanges.</li></ol><p>This makes arbitrage more complicated than it first seems. So before attempting to make money this way, think of the steps you’ll have to take, the risks and fees in each step, and validate the risk is worth the reward.</p><h4>Avoiding transaction times</h4><p>One way to avoid the transaction times, is keep a balance of both BTC and USD on two exchanges. Assuming you found a price gap after factoring in the fees, you could make two opposite trades — buy bitcoin at the cheaper exchange, and sell on the same amount on the expensive exchange. This will net you an immediate profit.</p><p>You may fail to execute the opposite trade at a good price since the market is constantly moving, but it reduces the risk significantly. If you have several coins you’re willing to hold, I recommend doing this.</p><h3>Let’s Trade!</h3><h4>Manual trading</h4><p>So you found an exploitable price gap in an exchange or between exchanges. You now need to open an account on each exchange and insert some money into it.</p><p>Sites provide a web interface to manage your orders and balance. If you’re getti with dizzy from using too many sites, consider using a platform like <a href="https://www.coinigy.com/">Coinigy</a>, and trade on multiple exchanges from the same site.</p><p>Unfortunately, trading manually can take a lot of time and you will find you’re often overrun by bots. Due to the nature of the order book, the order with the highest buy price is executed first, so bots which can place multiple orders every seconds can always outbid you by a penny and your order will not be executed.</p><p>Additionally, when the price moves, it can move <strong>FAST</strong>. I’ve seen coins go up and down upwards of 15% of their price in 5 minutes, and unless you’re the flash you’ll struggle to keep up with the changes at the volatile times,</p><h4>Automating trading</h4><p>To avoid the downfalls of manual trading, we can use a bot to run our strategies. A good in depth overview of available free and paid bot options can be found <a href="https://blockonomi.com/bitcoin-trading-bots/">here</a>.</p><p>However, most of the available solutions work on the same exchange and ignore prices on other exchanges.</p><h4>Don’t put all your eggs in one basket</h4><p>One of the greatest risks when trading on multiple exchanges, is that the smaller and more questionable exchanges can make the best revenues. However, exchanges are often hacked or downright scam people, and I would advise to split your money somewhat evenly between multiple exchanges to reduce risks, and avoid keeping your money on exchanges if you don’t need to.</p><h3>Writing your own platform</h3><p>My approach to solve the problems with manual trading was to automate the entire process by writing my own platform in python. The bot would watch the prices on various exchanges, check for possible profits, and finally execute trades to make some money using the API provided by the exchange.</p><p>Doing this has many technical difficulties. For starters, there’re <strong>no standards</strong> for the APIs, meaning each exchange can provide an entirely different interface for automatic trading.<br>There’s a great open-source library which tries to solve this by providing a common RESTful interface called <a href="https://github.com/ccxt/ccxt"><strong>CCXT</strong></a><strong> </strong>and I highly recommend it. However, it’s still limited because some exchanges don’t provide certain functions.</p><p>Exchanges also <strong>block </strong>you if you make too many requests. Some exchange support websockets with less limits, but many don’t. This is a problem, since we want low delays and a lot of information to scale our operation to many coins and exchanges.</p><p>To solve this, I split my architecture into microservices. One microservice, the <strong>Fetcher</strong>, was responsible for fetching public information from the exchanges, such as the public order book and transaction records. A second microservice, the <strong>Trader</strong>, was responsible for running private operations which required an API key.</p><p>All the requests the fetcher made were tunneled through proxies, allowing me to make more concurrent requests without getting blocked. Additionally, I implemented websocket clients for the larger exchanges to allow me to get live price information.</p><p>The trader, however, couldn’t use proxies because all operations were private and authenticated using the same API key. Here, the challenge was mostly about reducing the amount of API calls you make to the absolute minimum.</p><p>The final problem is one of <strong>scale</strong>. When working with a lot of exchanges, you’ll have a lot of technical difficulties. Sometimes exchanges are under a DDoS attack, under maintenance or just plain crashing. <br>You should ensure your program is robust enough to not lose your money in this situation, and have a good monitoring system to allow you to debug those problems.</p><h3>Conclusion</h3><p>Hopefully this article served it’s purpose, and you have a good idea of how cryptocurrency exchanges work, and how you could approach making money on them.</p><p>Next time you find a crazy gap in prices between cryptocurrencies, try to include the trade, deposit and withdrawal fees, as well as the time it will take to make the profit in your calculation. If you find a meaningful gap, I encourage you to try to exploit it.</p><p>Remember that the profits don’t scale linearly, and as you pour more money into the system your ROIs will diminish, since you there’s a finite amount of bargains out there.</p><p>Be advised that the profits are getting lower and lower as more players join the game, and the market is often heavily manipulated and unpredictable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8739b62544bc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>