<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alec Clowes on Medium]]></title>
        <description><![CDATA[Stories by Alec Clowes on Medium]]></description>
        <link>https://medium.com/@aclowes?source=rss-a9e5e1cea66a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*TXCvuU7x8g3edUDK</url>
            <title>Stories by Alec Clowes on Medium</title>
            <link>https://medium.com/@aclowes?source=rss-a9e5e1cea66a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:36:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@aclowes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Faster API development with django-dataclasses]]></title>
            <link>https://medium.com/roivant-technology/faster-api-development-with-django-dataclasses-5d33c366f23c?source=rss-a9e5e1cea66a------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d33c366f23c</guid>
            <category><![CDATA[rest-api]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[typescript]]></category>
            <dc:creator><![CDATA[Alec Clowes]]></dc:creator>
            <pubDate>Tue, 11 Jan 2022 19:13:02 GMT</pubDate>
            <atom:updated>2022-01-11T19:13:02.101Z</atom:updated>
            <content:encoded><![CDATA[<p>Our new <a href="https://pypi.org/project/django-dataclasses/">django-dataclasses</a> library allows you to:</p><ul><li>Define your API schema using Python dataclasses</li><li>Use the dataclasses as request and response bodies in Django views</li><li>Generate a JavaScript API client from an OpenAPI export</li></ul><p>Why did we build this if there are many great libraries for building APIs in Python already?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*PRejZ_mkyqu9sPMNzLCTsg.jpeg" /><figcaption>Sniffing out a new development paradigm. Painting by <a href="https://archive.org/stream/cu31924001182181/#page/n502/mode/1up">G. Stubbs</a>, 1802.</figcaption></figure><h3>Why not…</h3><h4><strong>Use FastAPI?</strong></h4><p>I like the <a href="https://docs.djangoproject.com/en/4.0/topics/db/models/">Django ORM</a> and the overall quality of the Django documentation. <a href="https://fastapi.tiangolo.com/">FastAPI</a> does not recommend a particular ORM, but I think it is often used with SQLAlchemy which in my opinion is more complicated to use, has less usable documentation, and requires the use of a separate tool (<a href="https://alembic.sqlalchemy.org/en/latest/">Alembic</a>) to manage database migrations.</p><h4><strong>Use pydantic?</strong></h4><p><a href="https://pydantic-docs.helpmanual.io/">pydantic</a> does almost exactly the same thing as dataclasses-jsonschema, and when used with FastAPI it is a similar solution to Django + django-dataclasses. However, pydantic does not do schema validation; if you pass it a float it will silently converted to an integer as <a href="https://pydantic-docs.helpmanual.io/usage/models/#data-conversion">documented here</a>.</p><h4><strong>Use Django REST framework?</strong></h4><p><a href="https://www.django-rest-framework.org/">Django REST framework</a> (DRF) is powerful but complicated if you use the the class-based views and override methods like <em>get_object</em> or <em>filter_query</em>. The REST paradigm adds complexity such as serializers which accept different fields on GET versus POST.</p><h4><strong>Use gRPC?</strong></h4><p>I prefer <a href="https://grpc.io/">gRPC</a> for API development and I have used it with the Django ORM, but some projects require JSON serialization. The HTTP/2 streaming features, compact protobuf serialization, and code generation features are great if you don’t need JSON.</p><h3>What it is</h3><p>django-dataclasses glues together the <a href="https://github.com/s-knibbs/dataclasses-jsonschema">dataclasses-jsonschema</a> library and Django, allowing you to use Python <a href="https://docs.python.org/3.8/library/dataclasses.html">dataclasses</a> to define the request and response schemas for a Django view. An example illustrates this best:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c545a2c7b1ac773f94566d90f380682/href">https://medium.com/media/3c545a2c7b1ac773f94566d90f380682/href</a></iframe><p>Most of the value enabled by django-dataclasses is implemented by the dataclass-jsonschema library, so kudos to <a href="https://github.com/s-knibbs">Simon Knibbs</a> for his work in that package.</p><p>The type annotations on the view function tell the django-dataclasses wrapper which schemas to use for the request and response.</p><h3>The simplicity of procedure calls</h3><p>The RPC (Remote Procedure Call) paradigm is based on actions as opposed to the object state used in REST (REpresentational State Transfer). The RPC approach is valuable it encourages developers to make API methods that do one thing well, whereas REST encourages all actions on an object to be grouped into a single method. For example, a hypothetical REST endpoint for a user object like /user/1 could support changing the username or the password, and the developer may expect API clients to do one or the other but not both. A bug of this type could be an issue with either approach, but RPC encourages these actions to be implemented as two separate methods.</p><p>RPC is also better suited to payloads which contain information about multiple related objects because there is no presumption that the payload represents a single object type. A strict REST approach would require decomposing this into separate endpoints, which is inefficient. Similarly, REST objects are often one-to-one with database models, which makes changing either difficult. RPC has no opinion on the contents of the payload, thereby encourages endpoints tailored to a specific client need and not mapped directly to a single database model. <a href="https://graphql.org/">GraphQL</a> is based on a similar learning, but does not try anticipate the client’s needs at all!</p><h3>Sharing the API definition</h3><p>Having written the API, why should clients redefine the request and response schemas and associated URLs? django-dataclasses provides an export of your schemas in the OpenAPI format using a management command:</p><pre>./manage.py openapi_export &gt; api.json</pre><p>The export can be passed to a tool like <a href="https://github.com/ferdikoomen/openapi-typescript-codegen">openapi-typescript-codegen</a> to generate a JavaScript client library that provides the TypeScript types and methods for calling the API:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1340013b4e2b49200c7147d41cd7dd18/href">https://medium.com/media/1340013b4e2b49200c7147d41cd7dd18/href</a></iframe><p>To use the client in your code, you write something like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e16128da42b38a9628e365b85813109/href">https://medium.com/media/7e16128da42b38a9628e365b85813109/href</a></iframe><h3>Conclusion</h3><p>API development is constantly improving as new tools remove boilerplate code and support better type checking. I hope you find the django-dataclasses library useful, and contribute by sharing issues or creating merge requests. Thanks!</p><ul><li><a href="https://pypi.org/project/django-dataclasses/">https://pypi.org/project/django-dataclasses/</a></li><li><a href="https://gitlab.com/roivant/oss/django-dataclasses">https://gitlab.com/roivant/oss/django-dataclasses</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d33c366f23c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/roivant-technology/faster-api-development-with-django-dataclasses-5d33c366f23c">Faster API development with django-dataclasses</a> was originally published in <a href="https://medium.com/roivant-technology">Roivant Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building dag-workflows]]></title>
            <link>https://medium.com/roivant-technology/building-dag-workflows-16d5c40f7789?source=rss-a9e5e1cea66a------2</link>
            <guid isPermaLink="false">https://medium.com/p/16d5c40f7789</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alec Clowes]]></dc:creator>
            <pubDate>Mon, 23 Aug 2021 18:38:46 GMT</pubDate>
            <atom:updated>2021-08-23T18:38:46.369Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building dag-workflows: a Python Workflow Execution Tool</h3><p>At Roivant, we use technology to ingest and analyze large datasets to support our mission of bringing innovative therapies to patients.</p><p>In this post, we’ll walk through the decision-making process that led to building our own <strong>workflow orchestration tool</strong>. We’ll talk about our needs and goals, the current product landscape, and the Python package we decided to build and open source.</p><p>We hope you’ll enjoy the discussion and find something useful in both our approach and the tool itself.</p><figure><img alt="Generic looking server racks" src="https://cdn-images-1.medium.com/max/1024/1*ywd_QK_ptvoPt5Tk0PAYBA.jpeg" /><figcaption>Someone needs to tell these computers what to work on. Photo by Manuel Geissinger from Pexels.</figcaption></figure><h3>Our needs</h3><p>We started our journey by looking at our past experiences and reading up on new projects. Since the mid-2010s, tools like Apache Airflow and Spark have completely changed data processing, enabling teams to operate at a new scale using open-source software. While these tools were a huge improvement, teams now want workflow tools that are self-service, freeing up engineers for more valuable work.</p><p>An article from Google engineer Adler Santos on <a href="https://cloud.google.com/blog/topics/developers-practitioners/datasets-google-cloud-introducing-our-new-reference-architecture">Datasets for Google Cloud</a> is a great example of one approach we considered: use Cloud Composer to abstract the administration of Airflow and use templating to provide guardrails in the configuration of directed acyclic graphs (DAGs). The deep analysis of features by Ian McGraw in <a href="https://medium.com/arthur-engineering/picking-a-kubernetes-orchestrator-airflow-argo-and-prefect-83539ecc69b">Picking a Kubernetes Executor</a> is a good template for reviewing requirements and making a decision based on how well they are met.</p><p>We compiled our desired features for data processing:</p><ul><li><strong>Python compatible:</strong> We use Python because data science libraries like Pandas and web frameworks like Django are easy to use and well documented.</li><li><strong>Asynchronous tasks:</strong> A tool for executing longer-running code, which is to say almost anything that cannot be completed during an API response.</li><li><strong>Triggered on-demand or on a schedule:</strong> Tasks should be triggered with API calls, by external events, or on a cron-like schedule, and we need a way to check on execution status.</li><li><strong>Support for inter-task dependencies:</strong> The ability to run tasks in parallel in a workflow makes the overall workflow faster and localizes failures.</li><li><strong>Low overhead:</strong> Tools that are easy to set up and learn, keep developers happy, and reduce the support burden.</li><li><strong>Kubernetes native:</strong> Because we deploy our applications on Kubernetes, it is convenient to have a tool that is designed for container-based orchestration.</li><li><strong>Decentralized:</strong> Defining workflows and writing orchestration code alongside the API services, data models, and scientific computing code to make dependency management, shared data models, and change management easier.</li><li><strong>Easy to test:</strong> Writing tests for common use cases should be easy, such as running SQL against a database.</li></ul><h3>Solutions considered</h3><p>We reviewed existing tools looking for something that would meet our needs. Here is a summary of our research:</p><ul><li><a href="https://man7.org/linux/man-pages/man8/cron.8.html"><strong>Cron</strong></a> is the old standby, but it does not parallelize your workflow or run across multiple servers. It’s nicely agnostic on testing, monitoring, and pretty much everything else.</li><li><a href="https://docs.celeryproject.org/"><strong>Celery</strong></a> is an established Python message-based task execution framework. It is has grown to support scheduled tasks and workflows, but the queue-based core means looking up task status and building workflows is unnecessarily difficult.</li><li><a href="https://airflow.apache.org/"><strong>Airflow</strong></a> is the market leader, popularizing the concept of DAGs in the Python community. It is a heavyweight and centralized solution, however; most developers don’t want to run it locally. It also is not Kubernetes-native: Cloud Composer abstracts the administration of Airflow but uses a legacy Celery worker to run a Kubernetes Pod. The flexibility afforded by the Python DAG definition makes enforcing common patterns difficult.</li><li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/"><strong>Kubernetes Job/CronJob</strong></a> is easy to define and a good starting point but doesn’t handle workflows. It is agnostic to what gets run, so testing is not addressed.</li><li><a href="https://docs.dagster.io/"><strong>Dagster</strong></a> addresses many of the same testing and orchestration issues, but it mixes the orchestration metadata into the functional code, which means there is more to learn.</li><li><a href="https://docs.getdbt.com/"><strong>dbt</strong></a> is focused on advanced SQL templating, and for orchestration, they recommend their proprietary cloud or Airflow. It’s a lot to learn for a partial solution.</li><li><a href="https://meltano.com/"><strong>Meltano</strong></a> glues together Singer and dbt. Again, it uses Airflow for orchestration.</li><li><a href="https://argoproj.github.io/argo-workflows/"><strong>Argo Workflows</strong></a> implements workflows within Kubernetes, which is very promising. However, it lacks data-centric testing support and a simple local runner for development.</li><li><strong>Serverless approaches</strong> like AWS Lambda + Glue are popular but hard to simulate locally. They also split the codebase awkwardly along the needs of each component.</li></ul><h3>Our decision</h3><p>While there were many options available, none of them seemed quite right for us. Most tools were either too complicated or lacked clean Kubernetes integration. Airflow, for instance, has both shortcomings. Dagster has native Kubernetes support but a steep learning curve. It’s also opinionated about passing data and defining workflows in code, which is in conflict with our desired simplicity.</p><p>At this point, we decided to build our own lightweight wrapper for running workflows. Our vision was a tool that runs locally during development and deploys easily onto Kubernetes, with data-centric features for testing and validation. We determined there would be three main components to design: the workflow definition, the task execution, and the testing support.</p><p>We’ll introduce each of these elements in the next section in a short tutorial on using the tool we named <strong>“workflows”</strong>.</p><h3>A brief introduction</h3><p>The command line and module are workflows but the package is installed as dag-workflows like this:</p><pre>pip install dag-workflows</pre><h4>Defining DAGs and Tasks</h4><p>There are two predominant patterns for defining tasks and grouping them into a DAG. Tools like Airflow, Celery, and Dagster, define the DAG using Python code. Tools like Kubernetes and dbt use YAML. We like YAML because it is more readable and helps enforce a single way of doing things, making the configuration options clearer and easier to manage across teams. The proliferation of tools like <a href="https://github.com/chriscardillo/gusty">Gusty</a> that turn YAML into Airflow DAGs suggests many see a similar advantage. Most peculiar is the way Google’s <a href="https://github.com/GoogleCloudPlatform/public-datasets-pipelines/blob/main/README.md">Public Datasets Pipelines</a> uses Jinga to generate the Python code from YAML.</p><p>We follow the pattern of grouping individual tasks into a DAG by representing each task as a file in a folder representing the DAG. Like Gusty and other tools, we put the YAML configuration in a comment at the top of each file. The individual task files can be.sql, .py, or .yaml files. A SQL task looks like this:</p><pre>/*<br> description: Make a table<br> type: sql<br> inputs: []<br> */<br> CREATE TABLE IF NOT EXISTS example AS<br> SELECT &#39;test&#39; AS name, 1 AS value;</pre><p>And a Python task should have a run method that looks like this:</p><pre>&quot;&quot;&quot;<br>description: Copy a file<br>type: python<br>inputs:<br>- other_task.sql<br>&quot;&quot;&quot;<br>import shutil</pre><pre>def run(context):<br>    shutil.copy(&#39;file_A&#39;, &#39;file_B&#39;)</pre><p>You’ll notice that the YAML has a field called inputs; this is where you list the tasks which are predecessors and should run first.</p><h4>Running the DAG</h4><p>After writing your tasks, the next step is to run them. We designed workflows to support multiple execution models, two of which handle scheduling and parallelization:</p><ul><li>The <strong>Kubernetes scheduler</strong> runs each task as a Kubernetes Pod. It triggers DAGs on a cron-like schedule and runs tasks as their dependencies are satisfied. It uses a PostgreSQL database to keep track of DAG and task runs.</li><li>The <strong>worker scheduler</strong> uses the same scheduling logic but runs tasks as a subprocess. Multiple workers can be run to parallelize execution.</li><li>For <strong>local development</strong>, the command line workflows run interface is used to run tasks locally. The same interface is used by the schedulers when running a single task.</li><li><strong>Third-party executors</strong> such as good ol’ cron and Airflow can use CLI runner as well. In Python, they can introspect the DAG object and call the execute method on tasks directly.</li></ul><figure><img alt="Diagram of the scheduler and runner components of workflows." src="https://cdn-images-1.medium.com/max/1024/1*11Z8W8BFAQ377lNYlZYlrQ.png" /></figure><p>To run the local executor, use the command line. The first argument is a configuration file which, at minimum, tells workflows what folder to look in for DAGs:</p><pre>workflows dags.config run dag_name</pre><p>To run the worker or Kubernetes schedulers, you need to provide a cron-like schedule for each DAGs in a YAML file, along with executor specific configurations like this:</p><pre>description: Example DAG<br>schedule: &#39;5 */4 * * *&#39;  # every four hours at five after<br>worker_config:<br>    kube_image: &quot;my_image:latest&quot;</pre><p>The scheduler requires access to a PostgreSQL database and is run from the command line like this. The scheduler type to use is specified in the last argument:</p><pre>export DATABASE_URL=postgres://localhost/workflows<br>workflows dags.config schedule kubernetes</pre><h4>Writing Tests</h4><p>An important requirement for us was easy testing of tasks. To support testing, we built a pytest fixture that supports running a task or DAG, and handles test database setup and teardown in the special case of SQL tasks. Our fixture utilizes pytest-django to create the database, and while you can choose to use <a href="http://xn--djangoprojet-yrb.com/">Django</a> with workflows, it is not required.</p><p>This example test covers a SQL task. It gets the task, sets up the input tables with test data, and executes the task. It asserts that the output matches the expected values:</p><pre>from example_project import config<br></pre><pre>def test_calculate_trades(workflows):<br>    task = config.DAGS[&quot;trader_bot&quot;].get_task(<br>        &quot;calculate_trades.sql&quot;)<br>    initial_data = {<br>        &quot;stock_prices&quot;: [<br>            {&quot;date&quot;: &quot;2020-01-01&quot;, &quot;close&quot;: 34},<br>            {&quot;date&quot;: &quot;2020-01-02&quot;, &quot;close&quot;: 43}<br>        ],<br>        &quot;limits&quot;: [{&quot;max_bid&quot;: 100}],<br>    }<br>    workflows.setup_task_inputs(task, initial_data)</pre><pre>    results = workflows.execute_task(task)</pre><pre>    assert len(results[&quot;new_trades&quot;]) == 2<br>    assert results[&quot;new_trades&quot;][0][&quot;limit_price&quot;] == 134</pre><h3>Learn More</h3><p>Thanks for taking the time to read about workflows! Here’s some suggested reading that might be of interest.</p><ul><li>See <a href="https://dag-workflows.readthedocs.io/en/latest/using_workflows.html">Using Workflows</a> on Read the Docs for a comprehensive introduction. In particular, read about writing custom <a href="https://dag-workflows.readthedocs.io/en/latest/task_types.html">task types</a> to support your use cases and using <a href="https://dag-workflows.readthedocs.io/en/latest/output_types.html">outputs</a> to perform data validations.</li><li>Read <a href="https://medium.com/roivant-technology/introducing-the-roivant-technology-blog-201972bbe49f">Introducing the Roivant Technology Blog</a></li><li>Work with us! View our <a href="https://www.roivantcareers.com/job/senior-software-engineer-vant-technology-technology-us-ny-new-york-151-west-42nd-s-4-1746959/">open positions</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16d5c40f7789" width="1" height="1" alt=""><hr><p><a href="https://medium.com/roivant-technology/building-dag-workflows-16d5c40f7789">Building dag-workflows</a> was originally published in <a href="https://medium.com/roivant-technology">Roivant Technology</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>