https://github.com/aisrael/datu

My software engineering journey has certainly taken me places. From hacking on Win32 drivers in C/C++, to Web apps in Ruby on Rails, to mobile apps, to “Digital Transformation” and “Big Data”, to Cloud infrastructure, then DevOps, and, most recently, data engineering and ML/AI.

Of course, through all that I’ve encountered and worked with a variety of data file formats, from the ubiquitous CSV, to non-technical friendly .XLS/.XLSX, to JSON, YAML, and then industry heavyweights like Parquet, Avro, ORC, and rookies like f3.

What I found remarkable was, when I wanted to quickly convert a sample .parq file to .avro, there didn’t seem to be a widely-used, simple CLI tool to do so.

I thought to myself, surely there must be something like ImageMagick, but for data files instead of images? (The closest I’ve found is Miller, but it only handles text-based formats (e.g. CSV, TSV, JSON).

So, me being me and me loving writing tools in Rust, I set about to write one.

The hardest part was choosing a name. All the TLAs I wanted were already taken on crates.io — “dtf”, “dfu”, “dmt” (data multi-tool). Once I found “dtfu” available, I started using that, then realized with a small change, another, cooler name was also available — thus, datu.

A type-safe pipeline in Rust

Also me being me, I couldn’t just write this as a straightforward collection of convert_avro_to_parquet(), convert_parquet_to_avro(), etc. functions.

Instead, I decided to model it internally as a pipeline — after all, pretty much all of DevOps, data/ML/AI engineering these days are pipelines all the way down.

I also didn’t just want to model this as some ‘generic’ pipeline that had some ‘catch-all’ context or state that could be passed around any/all steps, essentially a glorified Hash or dict that could contain anything.

Having read enough articles about how Rust lets you write type-safe state machines where the compiler itself prevents illegal state transitions, I thought to myself: we should be able to write a type-safe pipeline in Rust, where the compiler itself can prevent illegal stage transitions.

I looked at existing crates if they could be useful, such as deltaflow and streamweave, and while impressive, I thought they were a bit overkill for all I needed to do, for now.

I took me a while to express what I wanted and how I wanted it to work, but eventually came up with something that works and let me move on.

That is, given a

trait Step {
  type Input;
  type Output;

  fn execute(self, input: Self::Input) 
    -> Result<Self::Output>
}

Then one can have a

impl Step for FileReader {
  Input = String
  Output = Vec<String>

  ...

And

impl Step for Printer {
  Input = Vec<String>
  Output = ()

  ...

And the Rust compiler itself will prevent you from composing a pipeline that goes Printer |> FileReader!

That was probably the hardest part, and the part that I wrote the most code in by hand.

Because once that fell into place, everything else could be written by Cursor.

Vibe-coding a data file extraction and transformation tool

At $WORK we use Cursor (and Claude, and pretty much every AI provider) exhaustively. It’s not a “nice to have”, but a baseline requirement.

Naturally, I also subscribe to Cursor for my personal projects.

For the past couple of years prior to this, though, I’d been using Cursor (and Windsurf, and GitHub Copilot, etc.) as just amped-up autocomplete.

That is, I’d start typing, and watch gleefully as the AI magically completed my thoughts, sometimes suggesting entire functions/methods based on something I had just written.

I started writing datu fully expecting a similar workflow.

However, this time around I think I had a couple of things going my way:

1. Apache Arrow have published a full-featured, quite mature Rust crate that can handle just about anything I needed
2. Cursor, and other AI tools, have gotten to the point where, yes, it can understand simple one-line instructions and translate that to entire architectural rewrites across dozens of files in the codebase.

In other words, once I’d hand-coded (with ample AI assistance) the initial Parquet to CSV conversion pipeline, I admittedly pretty much vibe-coded the rest.

(At first, I ‘practiced’ by writing parq, a Rust equivalent to parq-cli, written in Python. Once I saw how Arrow + Cursor could get me that far, I became confident I could write a more powerful and flexible tool.)

“Add a --select option to the convert command, that uses a SelectColumnStep to only propagate the selected columns. Use a SelectColumnRecordBatchReader that wraps an underlying RecordBatchReader and projects the schema and all RecordBatches.”

“Add a test for SelectColumnStep”

“Add support for converting Parquet to Avro”

“Add full read & write support and conversion for both Parquet and Avro”

“Add aschema command, based on parq.”

“Add a head command, based on parq.”

“Add a tail command, based on parq.”

“Add an --output argument to the schema, head, and tail commands to allow output as JSON or YAML.”

“Write more cucumber tests in features for all the commands and arguments we have now.”

“Add support for json-pretty output.”

“Add support for ORC.”

“For all the .feature files that output JSON or YAML, add a check And the file should be valid JSON or And the file should be valid YAML, and use the corresponding JSON or YAML parser to validate the output”

I mean, I’m paraphrasing most of the above prompts (except the last, which I added/instructed just now), but they illustrate pretty much how it all went.

The AI dutifully added code, making sure unit tests and .feature files were updated and passed, and the README.md was also updated as we went along.

It even went so far as correctly detecting, identifying, and trying to fix errors like:

The Avro-to-YAML conversion was failing because userdata5.avro has Unicode
paragraph separators (U+2029) in the comments field. Without quoting, these
produced invalid YAML.

(To be fair, it didn’t fully fix the error the first round. I had to tell Cursor “the avro to yaml conversions are still failing”, then it rewrote some stuff and fixed the tests. But that’s maybe just because I didn’t have “After major changes, make sure all linting and tests pass” in my AGENTS.md)

Would I have caught that? Maybe in minutes or a couple of hours.

Cursor wrote code in seconds, found the bug in seconds, and (after another prompt), fixed the code in seconds.

Am I cheating? Does this feel hollow?

Am I ashamed? Do I feel, as James Randall says, that programming now feels hollowed out?

Maybe, to a certain extent?

I mean, it feels like cheating?

It feels like I didn’t really solve all the interesting puzzles of how to use the Arrow Rust crate properly?

But at the same time, over the course of several 1–2 hour sessions over several nights, I now have a tool that can convert across several popular file formats used in data engineering, examine their schema, and display their first/last lines, among other things.

It’s a tool that I, and others in my field can use now, instead of maybe in a couple of more weeks if I didn’t have Cursor and had to implement everything by hand.

If I could do it all over again, would I do it any different? No. My dopamine-hit from figuring out the API and code by myself has been replaced by just watching in amazement as the AI understands my intent, sees the existing patterns I’ve already placed, and writes the code, most of the time close to how I would’ve written it myself.

When I see it writing code that I think could be better, I simply tell it.

“Consolidate all the Write*Args into a single WriteArgs since they all have the same structure.”

“I rewrote the Step trait (so execute() takes the Input), fix all the implementations and make sure all linting and tests pass.”

The experience went from “being an experienced engineer finding pleasure in puzzling about how to write code”, to “being an experienced engineer wearing a product manager and architect hat, finding pleasure in directing a smart, but inexperienced junior engineer in how to write code as if I had written it.”

Is that so bad?

Is this the future?

I dunno. All I know is I’ve stopped writing commit messages.

And I have a working tool.

YAML (short for “YAML Ain’t Markup Language”) is a human-readable data serialization format designed to be easy to read and write, which has become widely popular, especially for configuration files.

Whether deploying to the cloud, configuring CI/CD pipelines, or setting up local development environments, you’ve probably run into YAML.

YAML’s popularity and flexibility are strengths — but they’re also double-edged swords.

If you’re like me, you know how even small misconfigurations when writing or templating YAML can lead to serious consequences: broken builds, failed deployments, or worse, production incidents.

The root of many of these issues is simple: we often don’t validate our YAML.

Unlike some data serialization formats, such as Protobuf or Avro, YAML is typically written and shipped without guaranteeing that it conforms to an expected structure. A missing key or an incorrect type might silently pass through the pipeline only to cause problems downstream.

Given how central YAML is to infrastructure and automation today, having a way to validate that it conforms to a specific schema isn’t just a nice-to-have — it’s essential for reliability and safety.

What about JSON Schema?

JSON Schema is an existing and mature specification for defining the structure and validation rules of JSON documents. With it, you can describe required fields, enforce data types, apply string patterns, and much more. It’s widely adopted and well-supported across tools and languages.

Because YAML is a superset of JSON, in theory, you can use JSON Schema to validate YAML files by first parsing them into JSON.

Using these JSON-first tools, however, just felt off. These tools typically convert YAML into JSON before validation — a process that may introduce inefficiencies in both performance and fidelity by stripping away YAML-specific features.

Many of these validators are written in higher-overhead languages like JavaScript or Python, which can struggle with performance and memory usage, especially when embedded in larger systems or working with large configuration files.

yaml-schema

Inspired by this, I set out to write a YAML native schema specification and validation tool.

I chose to implement it in Rust with an eye towards memory safety and performance, especially when dealing with larger YAML documents.

Admittedly, memory safety in a CLI tool may not be such an essential requirement. However, I also designed yaml-schema to be usable as a library embedded in other tools, where memory usage, safety and performance may be paramount.

yaml-schema is released as open source under the MIT License, available at https://github.com/yaml-schema/yaml-schema

For now, the best way to install and use it is through cargo, Rust’s package manager:

cargo install yaml-schema

Should install the ys binary in your path.

Basic Examples

Empty or true

A YAML schema that’s empty or simply states

true

will accept any YAML data. That is, according to the schema, all of the following documents are valid

42

Foo

an:
  - arbitrarily
  - nested
data: structure

false

Conversely, a YAML schema that simply says false will reject any YAML data, so all the above examples will fail to validate.

Strings

A YAML schema that states

type: string

Will accept "I'm a string" but reject 42.

Strings can also specify minimum/maximum length constraints:

type: string
minLength: 2
maxLength: 3

Will validate "one" but not "a" or "abcd".

Finally, strings can be validated against a regular expression pattern:

type: string
pattern: "^(\\([0-9]{3}\\) ?)?[0-9]{3}-[0-9]{4}$"

Will accept 867–5309 and (519) 867–5309 but reject 5555–1212 or 123-456-7890

Numbers

type: number and type: integer behave alike, but type: integer only accepts whole numbers, while type: number will accept floating point numbers.

Both can accept range constraints: minimum / exclusiveMinimum and maximum / eclusiveMaximum

That is, given a schema that specifies

type: integer
minimum: 0
exclusiveMaximum: 100

Then it should accept 0, 1, and 99. But it will not validate -1, 100 nor 101.

Enums

YAML schema can validate against a fixed set of values. For example, the schema

enum:
  - red
  - amber
  - green

Will accept "red", "amber", and "green", but reject 42 or "blue"

Objects

yaml-schema’s utility is best demonstrated when we start validating objects.

A basic schema of

type: object

Will accept any YAML hash, such as

# Unlike JSON, YAML allows numeric keys, which are treated as strings
0.01: cm
1: m
1000: km

We can refine this by specifying property names and their respective schemas:

type: object
properties:
  number:
    type: number
  streetName:
    type: string
  streetType:
    enum: [Street, Avenue, Boulevard]

The schema above will accept

number: 1600
streetName: Pennsylvania
streetType: Avenue

But it will not accept

number: "1600" # here, the quotes turn 1600 into a string
streetName: Pennsylvania
streetType: Avenue

Lists

YAML schema can also validate arrays or lists

type: array
items:
  type: number

The above schema should accept

- 1
- 2
- 3
- 5
- 8

But not

- 1
- 2
- "3" # a single non-number cause the entire array to be invalid
- 5
- 8

Composition

Finally, we can combine schemas usingallOf, anyOf or oneOf. They behave similarly, but anyOf will accept the data as long as any one of the subschemas successfully validates. oneOf will only accept the data if one, and only one of the subschemas successfully validates. Finally, allOf will require all the subschemas to successfully validate.

That is, given a schema

anyOf:
  - type: number
    multipleOf: 3
  - type: number
    multipleOf: 5

Then that should accept 10 or 15.

If it were oneOf instead of anyOf, then it would reject 15 since it’s both a multiple of 3 and 5.

If it were allOf, then it would accept 15 and 30, but not 10 or 9.

But wait, there’s more!

The complete set of schema validation rules are expressed as Cucumber features at https://github.com/yaml-schema/yaml-schema/tree/main/features

yaml-schema is a work-in-progress, but we hope you’ll try it out and give feedback, suggestions, or feature requests.

Better yet, if you’re so inclined, open an issue or submit a pull request to https://github.com/yaml-schema/yaml-schema and help improve the tool for everyone!

I tried out Cursor AI-assisted code editor for real the other day.

I’ve been using GitHub Copilot even during technical preview. If you’re already impressed by GitHub Copilot, you’ll be blown away by Cursor.

Python is not my go-to scripting language, so I’m far from being a Python expert.

But, during a live call with colleagues I was able to create a tool to sample objects in your S3 buckets then use AWS Comprehend to check if they (potentially) contain PII.

In less than 30 minutes, from scratch!

So, I wrote a poor man’s version of AWS Macie: https://github.com/aisrael/gumshoe

What really funny was, I started out by writing one line at the top of the script about what I intended it for, and Cursor happily suggested list_s3_buckets() then get_s3_bucket_object() functions as we went along.

I especially liked that Cursor could detect exceptions in the program output, explain the error, and suggest a possible code fix!

Then, you can highlight any part of your code and ask it to write more code to add a feature, or refactor whole parts of it.

When I decided to add program arguments, first I asked Cursor to add an ArgParser to accept aws-region, bucket-name, etc. arguments, and it did it.

Then I asked Cursor to refactor the script, comprising several functions, into a class, and it did so flawlessly!

It even figured out which arguments I intended to turn into instance variables, so I wouldn’t have to keep passing them around as function parameters, and generated the appropriate __init__() code!

My friend @kixpanganiban was right! This is like when you first went from Notepad to VSCode (or, from VSCode to JetBrains’ Rubymine/PyCharm/RustRover).

AI-assisted programming is here to stay, and I can’t see myself programming any other way now.

(Can’t wait ‘till JetBrains gets the same level, or better, of GenAI though… their code introspection and navigation capabilities are still top notch.)

One of the most remarkable features of Elixir, which it owes to its foundation on Erlang and the BEAM VM, is having the process as a core abstraction.

Elixir improves usability on top of raw processes with GenServer, then takes it up to higher levels with Agents, Applications, and GenStage.

When I was beginning to use Elixir, though, GenServers were one of those things that seemed like magic me. While I ended up using them extensively, how they actually worked and how they related to the supervision tree were a mystery.

So here we’ll attempt to unriddle how Elixir GenServers really work by building up an approximation of how they’re implemented starting all the way from Erlang processes. We won’t be covering everything about processes, the Supervisor, or Elixir Applications — just enough to better understand how GenServers work under the hood.

At the end of this exercise, we will have written a “prototype GenServer” module, which we’ll call ProtoServer, which we can use just as a regular GenServer and provides the basic GenServer.call syntax and functionality.

Processes

Erlang processes are how it can achieve massive, reliable concurrency.

Let’s look at a textbook example just to see how they’re written in Erlang, then we’ll switch to Elixir from thereon.

-module(spawn).
-export([start/0, greet/1]).

greet(Arg1) ->
   io:format("Hello, ~p!~n", [Arg1]).
start() ->
   Pid = spawn(?MODULE, greet, ["world"]),
   io:fwrite("~p",[Pid]).

If we write the above code out to spawn.erl, we can execute it using the erl shell:

$ erl
Erlang/OTP 24 [erts-12.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Eshell V12.2  (abort with ^G)
1> c('spawn').
{ok,spawn}
2> spawn:start().
<0.86.0>Hello, "world"!
ok

So, spawn can take a Module, a function name, and a list of arguments. It then executes the specified function (passing it the list of arguments) as an independent process.

The equivalent of the above in Elixir would be something like:

defmodule Spawn do
  def greet(name), do: IO.puts "Hello, #{name}"
end

:erlang.spawn(Spawn, :greet, ["world"])

Elixir provides a much simpler spawn/1 function, which can be used if we just want to execute some code in the background (aka “fire and forget”):

spawn(fn -> IO.puts("Hello in the background!") end)

We can see for ourselves that this is in the background if we make it wait for a few milliseconds:

iex> spawn(fn -> :timer.sleep(500); IO.puts("Hello in the background!") end)
#PID<0.107.0>
Hello in the background!

spawn_link/1 is an alternative to spawn/1 that propagates or bubbles up errors to the parent process. We won’t go into it much here, but just know that with spawn_link creates a ‘link’ between the parent process and the spawned child process, such that if the child process fails (or raises an error), the error “bubbles up” to the parent.

Let’s take this further by writing a function that calls itself (a recursive infinite loop):

iex> defmodule Loop do
  def loop() do
    IO.puts("Looping...")
    :timer.sleep(1000)
    loop()
  end
end

iex> pid_loop = spawn(&Loop.loop/0)
Looping...
#PID<0.117.0>
Looping...
Looping...
Looping...
...

Note that spawn/1 returns a PID or process identifier. As you can see, our Loop.loop/0 function never exits and runs continuously in the background.

iex> inspect(pid_loop)
"#PID<0.118.0>"

Now if all we could do was execute code in the background, then we wouldn’t have much over naked threads.

If all you need is to execute code in the background, and perhaps determine if it completed successfully or inspect and handle any failures, then you should be using Tasks rather than spawn/1. See https://elixir-lang.org/getting-started/processes.html#tasks

Inter-Process Communication

Elixir processes let us do so much more than low-level threads because processes aren’t just a means to execute code concurrently — they also provide messaging (signals in Erlang).

To exit our infinite looping function earlier, we just send it the exit signal:

Looping...
Looping...
iex> Process.exit(pid_loop, 0)
true

iex>

Now let’s write another function that loops indefinitely, but this time we’ll demonstrate how to send and receive meaningful messages:

iex> defmodule Receiver do
  def loop() do

    # receive blocks, awaiting a message
    receive do
      {:greet, name} -> IO.puts("Hello, #{name}")
      msg -> IO.puts "Received unknown message: #{inspect(msg)}"
    end
    loop()
  end
end

iex> pid = spawn(&Receiver.loop/0)
#PID<0.119.0>

When the receive block is executed, the process waits until a message is sent to it, pattern matches on the received message, then executes the matching branch. Receiver then loops back on itself, awaiting the next message:

iex> send(pid, {:greet, "world"})
Hello, world
{:greet, "world"}

iex> send(pid, "foo")
Received unknown message: "foo"
"foo"

We now have the beginnings of a persistent, background actor.

Mailbox

Elixir signalling isn’t just some elaborate way to call functions in the background. When we say Elixir has messaging, we mean that each process actually has a mailbox that can act as a message queue!

We can see this by adding an artificial delay in the receiver, then sending several messages one after the other:

iex> defmodule Mailbox do
  def loop() do
    receive do
      {:greet, name} -> greet(name)
      msg -> IO.puts "Received unknown message: #{inspect(msg)}"
    end
    loop()
  end
  def greet(name) do
    IO.puts("Hello, #{name}")
    :timer.sleep(1000)
  end
end

iex> Enum.each(1..5, fn i -> send(pid, {:greet, ["world #{i}"]}) end)
Hello, world 1
:ok
Hello, world 2
Hello, world 3
Hello, world 4
Hello, world 5

Here we can see that even after the Enum.each/2 loop has returned after sending all messages, the messages are held in the mailbox until receive is called. Once all messages have been received, the next call to receive blocks awaiting another message.

State

In object oriented programming, persistent objects are used to hold state in the heap, until they’re explicitly freed, garbage collected, or the program exits.

Since Elixir is functional, rather than having persistent “objects” occupying memory on the heap and responding to method calls, we have processes that run concurrently and respond to messages. But what about persistent state?

Simple. Function calls can take arguments, and there’s no reason a function can’t recursively call itself with arguments as well!

First let’s write a simple countdown function:

defmodule Count do
  # call the loop with initial state
  def down do
    IO.puts("Counting down...")
    loop(5)
  end

  # this recursively calls itself with its own 'state'
  def loop(n) do
    if n > 0 do
      IO.puts("#{n}")
      :timer.sleep(1000)
      loop(n-1)
    end
  end
end

pid = spawn(&Count.down/0)
IO.puts("spawn(&Count.down/0) -> #{inspect(pid)}")
:timer.sleep(5000)

If we write this to, say, count_down.exs and run it with elixir count_down.exs, we get the output:

elixir count_down.exs
Counting down...
5
spawn(&Count.down/0) -> #PID<0.100.0>
4
3
2
1

As we can see, even while the process loops in the background, it maintains an ever decreasing count as ‘state’.

Now let’s see a more useful example, a counter that can receive meaningful messages and modify its own state:

defmodule Counter do
  # call the loop with initial state of 0
  def init do
    IO.puts("Initial state: 0")
    message_loop(0)
  end

  # Recursively calls itself with its own 'state', and receive messages
  def message_loop(n) do
    receive do
      :inc -> IO.puts("#{n} + 1 -> #{n + 1}"); message_loop(n+1)
      :dec -> IO.puts("#{n} - 1 -> #{n - 1}"); message_loop(n-1)
      :print -> IO.puts("Current state: #{n}"); message_loop(n)
      msg -> raise "Received unknown message: #{inspect(msg)}"
    end
  end
end

pid = spawn(&Counter.init/0)
IO.puts("spawn(&Counter.init/0) -> #{inspect(pid)}")
send(pid, :inc) # 0 + 1 -> 1
send(pid, :inc) # 1 + 1 -> 2
send(pid, :inc) # 2 + 1 -> 3
send(pid, :dec) # 3 - 1 -> 2
send(pid, :inc) # 2 + 1 -> 3
send(pid, :print) # Current state: 3

When we run this using elixir counter.exs, we get the expected output:

Initial state: 0
spawn(&Counter.init/0) -> #PID<0.100.0>
0 + 1 -> 1
1 + 1 -> 2
2 + 1 -> 3
3 - 1 -> 2
2 + 1 -> 3
Current state: 3

Receiving responses

If all our background process could do was print out its own state, then it’s not quite as useful as a proper object, right? We also need a way for our object to respond to messages, and perhaps return its own state (or something related to its internal state).

Recall that we can send/2 a message to background process’ mailbox. Well, the current process or caller also has a mailbox and can also receive messages!

For us to receive a response to a sent message we’ll need to:

1. Tell the receiver how to send back messages to the caller, by sending along the caller’s process id. We can retrieve the current process id using self/0.
2. Setup a receive block in the calling process to await and handle the message sent back.

Let’s build upon our Counter example to see how this goes. First, we’ll modify our receive_loop so that it can respond to a message with a tuple of {:get, pid} where the pid is the caller’s process id:

  def receive_loop(n) do
    receive do
      :inc -> IO.puts("#{n} + 1 -> #{n + 1}"); receive_loop(n+1)
      :dec -> IO.puts("#{n} - 1 -> #{n - 1}"); receive_loop(n-1)
      {:get, caller} -> send(caller, {:counter, n}); receive_loop(n)
      msg -> raise "Received unknown message: #{inspect(msg)}"
    end
  end

Next, instead of just sending :print, we send {:get, self()}. Right after sending that message, in the calling process, we also set up a receive block to await the response from Counter:

send(pid, {:get, self()})
receive do
  {:counter, n} -> IO.puts("Got response from counter: #{n}")
  msg -> raise "Received unknown message: #{inspect(msg)}"
end

Now when we run our modified counter, the output, as expected, is:

Initial state: 0
spawn(&Counter.init/0) -> #PID<0.100.0>
0 + 1 -> 1
1 + 1 -> 2
2 + 1 -> 3
3 - 1 -> 2
2 + 1 -> 3
Got response from counter: 3

Demonstrating that Counter is indeed sending a reply back to the main process that called it.

Abstraction

We now have the beginnings of a proper send & receive framework. But before we can extract it to a reusable module, let’s abstract away the spawn, send, and receive bits, as so callers only interact with Counter.

First, let’s add a start() function to Counter that just calls spawn and returns the counter’s PID:

  # spawn the Counter in the background and return its pid
  def start() do
    spawn(&init/0)
  end

Next, let’s generalize Counter’s receive_loop and have it expect that all messages are a tuple of the caller’s PID and the actual message. Upon receiving a message, it’ll delegate the handling to a handle_msg function, which will return the new state. It will then send the new state as a reply to the caller, then recursively loop back on itself.

  def receive_loop(state) do
    receive do
      {from, msg} ->
        new_state = handle_msg(msg, state)
        send(from, new_state)
        receive_loop(new_state)
      msg -> raise "Received unknown message: #{inspect(msg)}"
    end
  end

We can now express Counter’s logic more clearly in handle_msg. What we’re really doing is also abstracting away the send and receive logic from the core of Counter.

  def handle_msg(:inc, n), do: n + 1
  def handle_msg(:dec, n), do: n - 1
  def handle_msg(:get, n), do: n

Finally, we’ll add ‘wrapper’ functions to Counter so callers also won’t need write a receive block:

  # Increment the counter and get the new count
  def inc(pid) do
    send(pid, {self(), :inc})
    receive do
      resp -> IO.puts("Counter.inc() -> #{resp}")
    end
  end

  # Decrement the counter and get the new count
  def dec(pid) do
    send(pid, {self(), :inc})
    receive do
      resp -> IO.puts("Counter.dec() -> #{resp}")
    end
  end

  # Just get the current count
  def get(pid) do
    send(pid, {self(), :inc})
    receive do
      resp -> IO.puts("Counter.get() -> #{resp}")
    end
  end

With this version of Counter, the calling code is now simplified:

pid = Counter.start()
IO.puts("Counter.start() -> #{inspect(pid)}")
Counter.inc(pid) # 1
Counter.inc(pid) # 2
Counter.inc(pid) # 3
Counter.dec(pid) # 2
Counter.inc(pid) # 3
Counter.get(pid) # 3

Named Processes

The only thing remaining from a caller’s perspective is that we’re still handling the PID. Fortunately, there’s a way to register a name for a PID using Process.register/2.

Let’s rewrite our Counter.start to use the module name (conveniently available as __MODULE__) as the registered name of the process:

  # spawn the Counter in the background and return its pid
  def start() do
    pid = spawn(&init/0)
    Process.register(pid, __MODULE__)
    pid
  end

With that, we can now simplify all our inc, dec, and get calls by using Process.whereis/1 to retrieve the PID, like the following:

  def inc() do
    pid = Process.whereis(__MODULE__)
    send(pid, {self(), :inc})
    receive do
      resp -> IO.puts("Counter.inc() -> #{resp}")
    end
  end

With that, our calling code is vastly simplified!

Counter.start() # We can discard the returned PID
Counter.inc() # 1
Counter.inc() # 2
Counter.inc() # 3
Counter.dec() # 2
Counter.inc() # 3
Counter.get() # 3

ProtoServer

Let’s wrap up everything we’ve learned and abstract away the underlying Process calls and message sending and receiving into a reusable module.

We want our ProtoServer to behave just like a GenServer. Meaning:

• When we call ProtoServe.start_link/3 (or start/3) we want to pass in our module, the initial state, and accept some options
• If the name: option is given, use that as the process name, so we don’t have to track the PID
• When the process is started, call our module’s init/1 callback with the initial state
• Allow init/1 to return either :ok and the initial state, or :error and some error. On :ok, proceed to the message loop. On :error, immediately raise the error
• In the message loop, on a :call message, call our handle_call/3 callback with the message, the caller PID, and the current state

Whew! That’s a lot to unpack, but don’t worry, it’s actually quite easy to implement and follow.

We want our module to take care of all the necessary “boilerplate” above, and be as simple to use by, well, use.

In Elixir, when we use another module, we allow that module module to “inject” any code into the current module using Elixir macros. Explaining elixir macros fully is out of scope for now — it’s enough to know that to make use work, we just need to implement a __using__ macro!

Let’s go ahead and start writing our ProtoServer and implement start_link/3:

# A prototype GenServer
defmodule ProtoServer do

  # Just link GenServer.start_link/3, spawns the calling
  # calling module, and returns the PID. Also, support the
  # optional `name:` option 
  def start_link(module, state, options \\ %{}) do
    pid = spawn_link(module, :on_spawn, [state])

    if Keyword.has_key?(options, :name) do
      Process.register(pid, options[:name])
    end
    pid
  end

So, start_link merely calls spawn_link, but we provide our own on_spawn function that will implement the ‘magic’ of calling the module’s init/1 callback. We also support the optional name: option and register the process using that name, to make it easier to call our ProtoServer module later on.

Now we can’t just def on_spawn in ProtoServer itself, because when we spawn_link we need to pass in the calling module. Otherwise, we’d be writing spawn_link(ProtoServer, :on_spawn, ...) which would make it difficult to call the our module’s init/1 callback function!

Instead, this is where we rely on the __using__ macro to write on_spawn in the calling module itself! It’s easier to show than explain:

  defmacro __using__(_opts) do
    quote do
      def on_spawn(args) do
        case init(args) do
          {:ok, initial_state} -> receive_loop(initial_state)
          {:error, reason} -> raise reason
        end
      end

In ProtoServer.__using__, we use a quote block and write the on_spawn/1 startup function as if we were writing it in the calling module itself!

That is, when we write, for example:

defmodule Counter do
  use ProtoServer

It’s as if we had written:

defmodule Counter do
  def on_spawn(args) do...

In on_spawn/1, we call init/1 with the same initial arguments, then depending on whether init/1 returns :ok or :error, we either proceed to call the receive_loop or fail.

We use the same quote block in the __using__ macro to write out receive_loop/1 as if we were writing it in the calling module:

      # Recursively calls itself with its own 'state', 
      # and receive messages
      def receive_loop(state) do
        receive do
          {:call, from, msg} -> case handle_call(msg, from, state) do
            {:reply, response, new_state} -> send(from, response); receive_loop(new_state)
            {:no_reply, new_state} -> receive_loop(new_state)
            result -> raise "Received unknown result: #{inspect(result)}"
          end
          msg -> raise "Received unknown message: #{inspect(msg)}"
        end
      end

The receive_loop is almost similar to the one we wrote earlier, except this time we handle the :call message and delegate to the module’s handle_call/3 callback. We then send back a response, do nothing, or raise an error depending on whether handle_call/3 returns :reply, :no_reply, or an unknown return value.

That’s all we need in our __using__ macro! Now we just need to implement ProtoServer.call/2. For call/2, we provide a convenient way to pass in a symbol (the module name) rather than a PID, and useProcess.whereis/1 ourselves:

def call(to, msg) do
    case to do
      pid when is_pid(pid) -> send(pid, {:call, self(), msg})
      name when is_atom(name) -> send(Process.whereis(name), {:call, self(), msg})
    end
    receive do
      resp -> resp
    end
  end

Our final implementation of our fully featured Counter can be written with use ProteServer exactly as if we had used a GenServer instead. You can try it out for yourself by replacing ProtoServer with GenServer in the following code!

defmodule Counter do
  use ProtoServer

  def start_link() do
    ProtoServer.start_link(__MODULE__, 0, name: __MODULE__)
  end

  def init(initial_value) do
    IO.puts("init(#{initial_value})")
    {:ok, initial_value}
  end

  def handle_call(msg, _from, state) do
    resp = handle_msg(msg, state)
    {:reply, resp, resp}
  end

  def handle_msg(:inc, n), do: n + 1
  def handle_msg(:dec, n), do: n - 1
  def handle_msg(:get, n), do: n

  def inc() do
    resp = ProtoServer.call(__MODULE__, :inc)
    IO.puts("Counter.inc() -> #{resp}")
  end

  def dec() do
    resp = ProtoServer.call(__MODULE__, :dec)
    IO.puts("Counter.inc() -> #{resp}")
  end

  def get() do
    resp = ProtoServer.call(__MODULE__, :get)
    IO.puts("Counter.inc() -> #{resp}")
  end
end

The implementation of ProtoServer.cast/2 is left as an exercise to the reader.

This article ran a bit longer than I wanted, and covered more than a few topics — but I hope that you were able to follow along and now have a deeper understanding of GenServer and greater confidence in using it!

Cheers!

Level Up Coding

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the Level Up Coding publication
• 🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job

ProtoServer: Implementing Elixir’s GenServer from scratch was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

While there are existing guides on how to package a Phoenix application using Docker, most of them deal with ‘regular’ Phoenix apps, not umbrella projects. In this guide, we’ll take our earlier Phoenix 1.6 umbrella project, build a minimal release binary, package it as a Docker image and deploy it as a Docker container.

The accompanying source code to this article is also at https://github.com/aisrael/elixir-phoenix-typescript-react

Now that we’ve seen how to create a Phoenix 1.6 project with TypeScript and React, let’s now explore how to release, package, and deploy it using Docker.

What/Why/How to Release?

An Elixir or Erlang/OTP Release is a self-contained artifact that allows us to run our Elixir application, with minimal dependencies and no source code. I like to think of it as something like a binary executable that contains everything needed to run our application, and nothing more.

In production environments, we don’t really need or want the entire Elixir/Erlang and Node toolchains. Some of those weigh in at dozens, even hundreds of MB and when deploying at scale, every MB saved matters.

Prior to Elixir v1.9, we would need to use Distillery to package and release our Elixir or Phoenix app. Starting with v1.9, however, Elixir provided the ability to mix release our app without any additional tooling.

We can still use Distillery, especially if we want some of its additional features such as hot code upgrades. Since we’re also looking to deploy our application as a container, however, then hot code upgrading becomes less of a concern.

I won’t go too much into how containers have become the preferred means of packaging and deploying server-side applications and services, or why Docker is the de facto standard, having popularized and made containers accessible in the beginning.

But for any large-scale Web application or service, chances are you’ll want to deploy them as containers using an orchestration platform such as Kubernetes, so this is where we’re headed in this guide.

Phoenix Umbrella Project Releases

The Phoenix framework helpfully provides us with a helper task, mix phx.gen.release that can even generate a Dockerfile that serves as a great starting point for us!

However, if we try to run mix phx.gen.release from the root of a Phoenix umbrella project, we are greeted with:

$ mix phx.gen.release
** (Mix) mix phx.gen.release is not supported in umbrella applications.

Run this task in your web application instead.

So, let’s try it in apps/hello_web instead.

$ cd apps/hello_web

This time, we’ll also supply the --docker option to get a Dockerfile we can work with. This creates a few files we’ll be needing:

hello_web$ mix phx.gen.release --docker
* creating rel/overlays/bin/server
* creating rel/overlays/bin/server.bat
* creating rel/overlays/bin/migrate
* creating rel/overlays/bin/migrate.bat
* creating lib/hello_web/release.ex
* creating Dockerfile
* creating .dockerignore

...

As well as printing out a few more helpful instructions. At this point, though, if we try mix release like the instructions suggest, while it will complete successfully and the resulting executable does run — it doesn’t actually run our Web application.

Going Back to Our Root

We need our entire umbrella project, so we need to go back to working from the project root. First, move the generated Dockerfile and .dockerignore files into the project root

hello_web$ cd ../..
$ mv apps/hello_web/Dockerfile apps/hello_web/.dockerignore .

From this point forward, we’ll also want to make sure our MIX_ENV is set to prod, because we are trying to build a release for production:

$ export MIX_ENV=prod

Now, if we try to mix release again from the root, we run into:

$ mix release
** (Mix) Umbrella projects require releases to be explicitly defined with a non-empty applications key that chooses which umbrella children should be part of the releases:

    releases: [
      foo: [
        applications: [child_app_foo: :permanent]
      ],
      bar: [
        applications: [child_app_bar: :permanent]
      ]
    ]

Alternatively you can perform the release from the children applications

Mix is telling us that we need to configure our umbrella app and tell it what the child applications are.

To do that, we’ll have to edit mix.exs and add something like the following lines (in bold) to the project definition:

  def project do
    [
      apps_path: "apps",
      version: "0.1.0",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      aliases: aliases(),
      releases: [
        hello_umbrella: [
          applications: [
            hello: :permanent,
            hello_web: :permanent
          ]
        ]
      ]
    ]
  end

At this point, mix release will complete and build a more or less self-contained executable:

$ MIX_ENV=prod mix release
==> hello
Generated hello app
==> hello_web
Compiling 4 files (.ex)
Generated hello_web app
Release hello_umbrella-0.1.0 already exists. Overwrite? [Yn] y
* assembling hello_umbrella-0.1.0 on MIX_ENV=prod
* using config/runtime.exs to configure the release at runtime

Release created at _build/prod/rel/hello_umbrella!

# To start your system
    _build/prod/rel/hello_umbrella/bin/hello_umbrella start

If we do what it says now and try to _build/prod/rel/hello_umbrella/bin/hello_umbrella start, we’ll quickly discover some runtime configuration errors.

Rather than explain how to address them at this point, let’s skip that for now and try to build our Docker image so we focus on running our Phoenix application as a container.

server: true

One small, but important thing that’s not very well documented and might cause you some level of head-scratching and frustration (it did me) is that somewhere inside config/runtime.exs, there’s a section that goes

  # ## Using releases
  #
  # If you are doing OTP releases, you need to instruct Phoenix
  # to start each relevant endpoint:
  #
  # config :hello_web, HelloWeb.Endpoint, server: true

Since we are using OTP releases, we will have to comment that last line out. If you don’t everything else below will seem to work, but when you actually try to navigate to your Web application you’ll just keep getting no response!

So make sure you uncomment the last line, as follows:

config :hello_web, HelloWeb.Endpoint, server: true

Tweaking the Dockerfile

While Phoenix was helpful enough to generate a Dockerfile for us, it also assumes a regular Phoenix project rather than an umbrella project.

Release Builder Stage

Notably, since we’re coming from a project that uses React, we’ll still need npm but only in the builder stage. So, in the Dockerfile where it says # install build dependencies we’ll want to add npm, like so:

# install build dependencies
RUN apt-get update -y && apt-get install -y build-essential git npm \
  && apt-get clean && rm -f /var/lib/apt/lists/*_*

Next, the generated Dockerfile only copies mix.exs and mix.lock from the project root into the build container. Because we have an umbrella project, we also need to copy in mix.exs for all child apps.

We can compress all the steps by simply going COPY . . but that defeats the purpose of the multiple, explicit steps in the generated Dockerfile, namely, being able to cache earlier steps that involve infrequently changed files.

Let’s add mix.exs for all the child apps:

COPY apps/hello/mix.exs ./apps/hello/mix.exs
COPY apps/hello_web/mix.exs ./apps/hello_web/mix.exs

Next, the generated Dockerfile tries to copy over some folders such as priv and assets from the project root, but in our case, those folders actually reside under apps/hello_web/. We’ll need to modify the following lines accordingly:

# COPY priv priv
COPY apps/hello_web/priv apps/hello_web/priv

And

# COPY assets assets
COPY apps/hello_web/assets apps/hello_web/assets

Now, once again when compiling the assets, we’ll want to modify the Dockerfile to run the command under apps/hello_web:

# compile assets
RUN cd apps/hello_web && mix assets.deploy

Almost there! Now, where a typical Phoenix app has all its source under lib, for an umbrella project we want everything under apps, so replace the line COPY lib lib with:

# Compile the release
COPY apps apps

Runner Image Stage

Now we’re done with the release builder stage, we just need a couple of adjustments in the final image stage.

In particular, the generated Dockerfile was for hello_web but the actual release name is hello_umbrella:

# Only copy the final release from the build stage

COPY --from=builder --chown=nobody:root /app/_build/prod/rel/hello_umbrella ./

Finally, the generated Dockerfile uses the CMD /app/bin/server, which is problematic.

First since the file app/bin/server doesn’t actually exist. Rather, what we’ll find there is app/bin/hello_umbrella.

Second, Docker best practices tell us to prefer ENTRYPOINT and not just CMD —this lets us specify program arguments at run time.

So, finally, we replace the last line with:

ENTRYPOINT [ "/app/bin/hello_umbrella" ]

For a quick summary of all the changes in the Dockerfile, see this commit.

At this point, we should be able to successfully build and tag our Docker image using:

$ docker build -t hello .

Up and Running!

After our Docker image has completed building, if we just try to run it using docker run hello we should just see:

Usage: hello_umbrella COMMAND [ARGS]

The known commands are:

    start          Starts the system
    ...

Now, if we try docker run hello start we’ll get a runtime error telling us that the environment variable DATABASE_URL is missing, and that it probably should look like ecto://USER:PASS@HOST/DATABASE.

DATABASE_URL

Recall that in our earlier article we used Docker Compose to run Postgres. While Postgres should be listening on localhost relative to our workstation, within Docker, a container by default can’t just refer to the localhost of the host.

Instead, we need to run our container as part of the same Docker internal network that Docker Compose used to run Postgres. To get this network, we have to issue the command docker network ls, which should give us something like:

$ docker network ls
NETWORK ID     NAME                              DRIVER    SCOPE
ddd978e072b8   bridge                            bridge    local
0e787c1bff92   hello_umbrella_default            bridge    local

So our network name is hello_umbrella_default.

Next, we’ll need to figure out what the hostname of the Postgres server is within Docker — by default, that’s just container name, which we can get by issuing docker ps:

$ docker ps
CONTAINER ID   IMAGE           COMMAND                  CREATED         STATUS         PORTS                    NAMES
c69d93a2f9bb   postgres:14.1   "docker-entrypoint.s…"   4 seconds ago   Up 3 seconds   0.0.0.0:5432->5432/tcp   hello_umbrella-postgres-1

So, our Postgres hostname is hello_umbrella-postgres-1.

Using the default credentials (and the development database), our complete DATABASE_URL now becomes:

export DATABASE_URL=ecto://postgres:postgres@hello_umbrella-postgres-1/hello_dev

SECRET_KEY_BASE

With DATABASE_URL out of the way, if we try to run our Docker image again, we’ll get another error message saying environment variable SECRET_KEY_BASE is missing.

The SECRET_KEY_BASE is used by Phoenix to sign/encrypt cookies and other secrets. In development, it’s generated at project creation and hard-coded in config/dev.exs.

In production, however, we don’t want secrets to be committed to source code, and, ideally should only be accessible to operations, if at all.

For now, all we need to do is supply some value, at least 64 bytes long, encoded as Base64. Since we already have Elixir, we can use it generate a suitably encoded random value:

$ export SECRET_KEY_BASE=$(elixir --eval 'IO.puts(:crypto.strong_rand_bytes(64) |> Base.encode64(padding: false))')

Hello, Phoenix from Docker!

Let’s put together our final docker run command.

Those of you more experienced with Docker can probably craft a docker-compose.yml file that avoids having to type out the command below by hand, but the purpose of this guide was to educate the reader about the individual configuration items that are needed to run a Phoenix release as a container.

To recap, we need:

• To specify the network, --net hello_umbrella_default
• The DATABASE_URLvariable
• The SECRET_KEY_BASE variable

We also need to publish the port that our Phoenix app is serving on, 4000 to our localhost, using -p 4000:4000.

The final docker run command thus looks like:

$ docker run -p 4000:4000 --net hello_umbrella_default -e DATABASE_URL=$DATABASE_URL -e SECRET_KEY_BASE=$SECRET_KEY_BASE hello start

If we navigate to http://localhost:4000 now, we should be greeted with our “Welcome to Phoenix with TypeScript and React!” landing page.

I hope this is helpful to people exploring releasing and deploying their Elixir+Phoenix applications as containers.

In the future I hope to find time to write about adding a GraphQL API into the mix.

This is the 2022 update to my earlier article on building up a modern Web application stack.

The code is also up on Github: aisrael/elixir-phoenix-typescript-react.

The release of Phoenix 1.6 brings with it a couple of notable improvements, such as HEEx templating. To me, though, the most remarkable thing that improves developer experience was dropping webpack in favor of esbuild.

Not only does esbuild promise faster build times, it supports TypeScript out of the box, and adding React is also so much easier!

The entire process has become so streamlined that in this article we can skip entire sections from its predecessors.

With that out of way, let’s go ahead and create a new project using Elixir and Phoenix 1.6 with TypeScript and React.

Prerequisites

This guide will require you to have the following, with indicated versions as of this writing:

• Erlang (24.2+)
• Elixir (1.13+)
• npm (7.6.0+)
• git
• Docker with Docker Compose

On macOS, the easiest way to get started is to install Homebrew, then use Homebrew to install Node, Git, Docker, and asdf.

$ brew install node git homebrew/cask/docker asdf

We can then use asdf to install both Erlang and Elixir:

$ asdf plugin add erlang
$ asdf install erlang 24.2
$ asdf global erlang 24.2

$ asdf plugin add elixir
$ asdf install elixir 1.13.1
$ asdf global elixir 1.13.1

Phoenix

For those already familiar with Elixir and Phoenix, you may wish to skip ahead straight to integrating TypeScript and React with Phoenix 1.6.

If you haven’t already done so, let’s install Phoenix following the Phoenix installation instructions.

First we’ll want to get the Hex package manager:

$ mix local.hex
Are you sure you want to install "https://repo.hex.pm/installs/1.13.1/hex-1.0.1.ez"? [Yn]

Then install the Phoenix application generator:

$ mix archive.install hex phx_new
Resolving Hex dependencies...
Dependency resolution completed:
New:
phx_new 1.6.5
* Getting phx_new (Hex package)
All dependencies are up to date
Compiling 11 files (.ex)
Generated phx_new app
Generated archive "phx_new-1.6.5.ez" with MIX_ENV=prod
Are you sure you want to install "phx_new-1.6.5.ez"? [Yn]

You can check if Phoenix installation went well using mix phx.new --version:

$ mix phx.new --version
Phoenix installer v1.6.5

Generate the Phoenix app

$ mix phx.new hello --umbrella

This will generate a Phoenix umbrella project named hello_umbrella in the current directory, with the following directory structure:

.
├── apps
│   ├── hello
│   └── hello_web
└── config

The two Elixir apps are /apps/hello (the domain model) and /apps/hello_web (the Web application).

Each app will have its own dependency configuration, though the entire umbrella project will have a shared dependency library (in /deps) for all apps.

All child apps will also share the same root configuration in the /config folder.

We start with an umbrella project because it helps organise code as the application grows in size and complexity. I’ve also found that it’s easier to refactor an umbrella project down to a single app project than it is to go the other way around.

If you think you only need a single app project, simply use mix phx.new hello (without the --umbrella option).

PostgreSQL, MySQL, or None

Phoenix by default uses Postgres for its database.

If you want to use MySQL rather than Postgres, then you’ll need to generate your Phoenix app using

mix phx.new hello_react --umbrella --database mysql

If you won’t be needing a database or only wish to follow along without one, then create your Phoenix app using

mix phx.new hello_react --umbrella --no-ecto

The rest of this guide, however, assumes the default which is Postgres.

Git

At this point, we should start using Git to track our changes. If you don’t need to use Git, feel free to skip to the next section.

$ cd hello_umbrella

~/hello_umbrella$ git init
Initialized empty Git repository in /Users/aisrael/hello_umbrella/.git/

~/hello_umbrella$ git add -A
~/hello_umbrella$ git commit -m "mix phx.new hello --umbrella"

Docker Compose

We’ll be needing a PostgreSQL server to run our Phoenix app. For local development and testing purposes I’ve found that using Docker, or specifically, Docker Compose makes dealing with service dependencies a breeze.

Create the following docker-compose.yml file in the project root:

version: "3"
services:
  postgres:
    image: postgres:14.1
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: hello_dev

Note that we configure PostgreSQL (using the POSTRES_* environment variables) to work with the generated Phoenix app defaults.

Then, to run Postgres in the background you only need to go:

$ docker compose up -d
[+] Running 2/2
⠿ Network hello_umbrella_default             Created
⠿ Container hello_umbrella_postgres_1        Started

For other Docker Compose commands please just visit https://docs.docker.com/compose/reference/

If you can’t or don’t want to use Docker & Docker Compose, you’ll have to install PostgreSQL by hand on your local workstation. Make sure to configure it with the same defaults generated by mix phx.new, or, modify the respective config.exs files with the appropriate configuration.

Welcome to Phoenix!

At this point we should be able to run our Phoenix application. From the project root (you may wish to run this in a new terminal window or tab), just go:

~/hello_umbrella$ mix phx.server

(If you get a prompt asking if you want to install rebar3, just go ahead and say yes.)

Now if we visit http://localhost:4000 we should be able to see the familiar “Welcome to Phoenix!” page:

TypeScript in Phoenix 1.6

The best thing about Phoenix 1.6 is that it ditched webpack in favor of esbuild.

Not only does this promise shorter build times, esbuild supports TypeScript out of the box!— no further tooling or configuration required!

Welcome to Phoenix with TypeScript

To demonstate TypeScript in action, let’s create a new TypeScript module apps/hello_web/assets/js/greeter.ts:

function greet(name: string): string {
  return "Welcome to " + name + " with TypeScript!";
}

export default greet;

Then, in apps/hello_web/assets/js/app.js, add the following lines toward the end:

import greet from "./greeter";

document.querySelector("section.phx-hero h1").innerHTML = greet("Phoenix");

Refresh the page at localhost:4000 and you should now see it say “Welcome to Phoenix with TypeScript!”:

That’s it! No need for npm or fiddling with tsconfig.json and webpack.config.js!

React

Now we can add React into the mix. For this, we’ll now need npm to fetch and install the necessary packages.

First, make sure we’re in the apps/hello_web/assets directory:

~/hello_umbrella$ cd apps/hello_web/assets

From there, we’ll issue:

npm install --save react react-dom
npm install --save-dev @types/react @types/react-dom

Our First Component

Let’s rename greeter.ts to greeter.tsx (similar to how you would rename a regular .js file to .jsx).

Then, replace the contents of greeter.tsx with a Greeter React component:

import React from "react";

interface GreeterProps {
    name: string;
}

const Greeter: React.FC<GreeterProps> = (props: GreeterProps) => {
    const name = props.name;
    return (
        <section className="phx-hero">
          <h1>Welcome to {name} with TypeScript and React!</h1>
          <p>Peace-of-mind from prototype to production</p>
        </section>
    );
};

export default Greeter;

Welcome to Phoenix with TypeScript and React

Next, edit the file apps/hello_web/lib/hello_web/templates/page/index.html.heex and replace the entire section that goes:

<section class="phx-hero">
  <h1><%= gettext "Welcome to %{name}!", name: "Phoenix" %></h1>
  <p>Peace of mind from prototype to production</p>
</section>

With just

<div id="greeting"/>

Now, we’ll need to rename apps/hello_web/assets/js/app.js to app.jsx, and replace the last two lines we added earlier with:

import React from "react";
import ReactDOM from "react-dom";

import Greeter from "./greeter";

const greeting = document.getElementById("greeting");
ReactDOM.render(<Greeter name="Phoenix" />, greeting);

(We need to rename it from .js to .jsx to avoid a syntax error in the last line.)

Finally, we’ll need to edit config/config.exs to tell esbuild to look for app.jsx instead of app.js. Look for the section starting with # Configure esbuild and replace the line that goes

~w(js/app.js --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*),

with

~w(js/app.jsx --bundle --target=es2017 --outdir=../priv/static/assets --external:/fonts/* --external:/images/*)

At this point, we might have to restart our Phoenix server (press Ctrl+C then (a)bort) and restart it again with mix phx.server.

When we reload http://localhost:4000, we should then be greeted with:

Voila!

I think that Phoenix 1.6 with esbuild provides one the best developer experiences I’ve ever encountered lately in setting up one of the most modern and productive Web app stacks around, and I hope this article has convinced you of how easy it is, especially compared to before.

Now get out there and build your next unicorn Web app!

Phoenix 1.6 With TypeScript and React was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

An updated guide to building up a modern Web application stack

This is an updated edition of my earlier guide, published in December, 2019 on DEV.to.

The code is also up on Github: aisrael/elixir-phoenix-typescript-react

While the essential approach has remained unchanged, somebody filed a ticket on my repo to upgrade it to Phoenix 1.5.

Rather than take the earlier codebase and upgrade it Phoenix 1.5 (which I’ve managed to do for a couple of other projects), I decided instead to update the original article for 2020.

Let’s jump right back in.

Prerequisites

This guide assumes you already have the following set up:

• Elixir (1.10.3), which means you’ll also need
• Erlang (23.0.2)
• npm (@6.14.5)
• git (not really required but you’ll want to anyway)
• Docker Compose (for running PostgreSQL)

If you don’t have Elixir (and Erlang) yet, I highly recommend using asdf to manage Elixir/Erlang versions.

Install asdf according to your platform’s instructions, then use asdf to install both Erlang and Elixir:

$ asdf plugin add erlang
$ asdf install erlang 23.0.2
$ asdf global erlang 23.0.2

$ asdf plugin add elixir
$ asdf install elixir 1.10.3
$ asdf global elixir 1.10.3

Phoenix

(If you’re already experienced with Elixir Phoenix applications, you may wish to skip ahead to the Typescript and React bits.)

If you haven’t already done so, let’s install Phoenix following the Phoenix installation instructions.

First we’ll want to get the Hex package manager:

$ mix local.hex
Are you sure you want to install "https://repo.hex.pm/installs/1.10.0/hex-0.20.5.ez"? [Yn] y
* creating /Users/aisrael/.asdf/installs/elixir/1.10.3/.mix/archives/hex-0.20.5

Then the Elixir Mix archive:

$ mix archive.install hex phx_new 1.5.3
Resolving Hex dependencies…
Dependency resolution completed:
New:
phx_new 1.5.3
* Getting phx_new (Hex package)
All dependencies are up to date
Compiling 10 files (.ex)
Generated phx_new app
Generated archive "phx_new-1.5.3.ez" with MIX_ENV=prod
Are you sure you want to install "phx_new-1.5.3.ez"? [Yn] Y
* creating /Users/aisrael/.asdf/installs/elixir/1.10.3/.mix/archives/phx_new-1.5.3

You can check if Phoenix installation went well using mix phx.new --version:

$ mix phx.new --version
Phoenix v1.5.3

Generate the Phoenix app

$ mix phx.new hello_react --umbrella

This will generate an Elixir + Phoenix umbrella app named hello_react_umbrella in the current directory with the following directory structure:

.
├── apps
│ ├── hello_react
│ └── hello_react_web
├── config
└── deps

The two Elixir apps are /apps/hello_react and apps/hello_react_web.

Each app will its own dependency configuration, though the entire umbrella project will have a shared dependency library (in /deps) for all apps.

All child apps also share the same root configuration in the /config folder.

We start with an umbrella app because it makes it easier to organize code as the application gets bigger and more complex. Besides, we’ve found that it’s easier to refactor an umbrella app project to a single app project than it is to go the other way around.

PostgreSQL, MySQL, or ` — no-ecto`

Phoenix by default uses PostgreSQL for its database.

If you want to use MySQL rather than PostgreSQL, then you’ll need to generate your Phoenix app using

mix phx.new hello_react --umbrella --database mysql

If you won’t be needing a database or only wish to follow along without one, then create your Phoenix app using

mix phx.new hello_react --umbrella --no-ecto

The rest of this guide, however, assumes the default which is Postgres.

Git

At this point, we should start using Git to track our changes. If you don’t need to use Git, feel free to skip to the next section.

$ cd hello_react_umbrella
~/hello_react_umbrella$

Then

$ git init
Initialized empty Git repository in ~/hello_react_umbrella/.git/
$ git add -A
$ git commit -m “mix phx.new hello_react --umbrella”

Docker Compose

Since we’ll be needing a PostgreSQL server to run our Phoenix app, for local development and testing purposes we’ve found that using Docker, specifically, Docker Compose makes dealing with service dependencies a breeze.

Create the following docker-compose.yml in the project root:

version: "3"
services:
  postgres:
    image: postgres:11.5
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: hello_react_dev

Note that we configure PostgreSQL (using the POSTGRES_* environment variables) to work with the generated Phoenix app defaults.

Then, to run Postgres in the background you only need to go:

$ docker-compose up -d
Creating network "hello_react_umbrella_default" with the default driver
Creating hello_react_umbrella_postgres_1 ... done

Since Docker Compose is beyond the scope of this article, for other Docker Compose commands please just visit:

• https://docs.docker.com/compose/reference/overview/

If you can’t or don’t want to use Docker & Docker Compose, you’ll have to install PostgreSQL by hand on your local workstation. Make sure to configure it with the same defaults generated by mix phx.new, or, modify the respective config/*.exs files with the appropriate credentials.

Node

Before we can run our Phoenix application, we need to initialise the generated CSS and Javascript assets using npm install:

~/hello_react_umbrella$ cd apps/hello_react_web/assets
~/hello_react_umbrella/apps/hello_web/assets$ npm install

Welcome to Phoenix!

At this point we should be able to run our Phoenix application. From the project root (you may wish to run this in a new terminal window or tab):

$ mix phx.server

(If you get a prompt asking if you want to install rebar3, just go ahead and say yes.)

Now if we visit http://localhost:4000 we should be able to see the familiar “Welcome to Phoenix!” page:

Typescript

We’re ready to start adding Typescript to the frontend.

First, make sure we’re back in apps/hello_react_web/assets/. Add the Typescript libraries using:

$ npm install --save-dev typescript ts-loader source-map-loader @types/phoenix

Typescript Configuration

Afterwards, let’s ask Typescript to generate a default tsconfig.json for us:

$ ./node_modules/.bin/tsc --init
message TS6071: Successfully created a tsconfig.json file.

We need to change a few things from the Typescript defaults. Here’s a minimal tsconfig.json with some of the necessary changes:

{
  "compilerOptions": {
    "target": "es5",
    "module": "ESNext",
    "allowJs": true,
    "jsx": "react",
    "outDir": "./dist/",
    "strict": true,
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true
  },
  "exclude": [
    "/node_modules/**/*",
  ]
}

Configure Webpack

Next, we’ll need to tell Webpack to recognise .ts files along with .js files:

Open apps/hello_react_web/assets/webpack.config.js and change the first module rule to:

    rules: [
      {
        test: /\.(j|t)s$/,
        exclude: /node_modules/,
        use: [
          {
            loader: "babel-loader"
          },
          {
            loader: "ts-loader"
          }
        ]
      },

Additionally, add an outermost "resolve" key after "module" as follows:

  resolve: {
    extensions: [".ts", ".js"]
  },

app.js

When we generated our Phoenix app, it created apps/hello_react_web/assets/js/app.js with an import css from "../css/app.css"; line.

This causes problems when that file is parsed by Typescript. You can see a lengthy discussion and several workarounds for this at this Stackoverflow question and on this page.

The simplest (though not the best) way to fix this before proceeding (so Webpack will continue to process our app.css) is to change that line to use require:

const _css = require("../css/app.css");

Since we don’t actually use the variable (we only need it so Webpack can generate our app.css file properly), we prepend its name with an underscore to suppress the “unused variable” warning that Typescript would otherwise emit.

Welcome to Phoenix with Typescript!

To demonstrate Typescript in action, we’ll create a new Typescript module apps/hello_react_web/assets/js/hello.ts:

function greet(name: string): string {
  return "Welcome to " + name + " with Typescript!";
}

export default greet;

Then, in assets/js/app.js add the following lines toward the end:

import greet from "./hello";

document.querySelector("section.phx-hero h1").innerHTML = greet("Phoenix");

Refresh the page at localhost:4000 and you should now see it say “Welcome to Phoenix with Typescript!”.

React

Let’s go ahead and add React roughly following the guide at: https://www.typescriptlang.org/docs/handbook/react-&-webpack.html

First, add the necessary packages:

$ npm install --save react react-dom
$ npm install --save-dev @types/react @types/react-dom

Once again we need to reconfigure webpack.config.js.

First, let’s recognize *.jsx and *.tsx files:

    rules: [
      {
        test: /\.(j|t)sx?$/,
        exclude: /node_modules/,

Also:

  resolve: {
    extensions: [".ts", ".tsx", ".js", ".jsx"]
  },

Our First Component

First let’s rename apps/hello_react_web/assets/js/hello.ts to apps/hello_react_web/assets/js/hello.tsx (similar to how you would name a React JS file .jsx from plain .js):

$ mv apps/hello_react_web/assets/js/hello.ts apps/hello_react_web/assets/js/hello.tsx

Then, replace the contents of hello.tsx with:

import React from "react";

interface GreeterProps {
  name: string;
}

const Greeter: React.FC<GreeterProps> = (props: GreeterProps) => {
  const name = props.name;
  return (
    <section className="phx-hero">
      <h1>Welcome to {name} with Typescript and React!</h1>
      <p>Peace-of-mind from prototype to production</p>
    </section>
  );
};

export default Greeter;

Welcome to Phoenix with Typescript and React

Next, in apps/hello_react_web/lib/hello_react_web/templates/page/index.html.eex, remove the section:

<section class="phx-hero">
  <h1><%= gettext "Welcome to %{name}!", name: "Phoenix" %></h1>
  <p>Peace-of-mind from prototype to production</p>
</section>

And replace it with simply:

<div id=”greeting”></div>

Then, in apps/hello_react_web/assets/js/app.js, replace the last few lines with:

import React from "react";
import ReactDOM from "react-dom";

import Greeter from "./hello";

const greeting = document.getElementById("greeting");
ReactDOM.render(<Greeter name="Phoenix" />, greeting);

Finally (you may neeed to restart Phoenix using mix phx.server and wait a seconds for Webpack to compile everything), when we reload localhost:4000 we should see “Welcome to Phoenix with Typescript and React!`

Now you’re ready to rock building your next unicorn Web app using one of the most modern and productive Web app stacks around.

In the future, I hope to expound upon this by adding a React UI framework and an article on how to add GraphQL to you Phoenix + React app.

Coding Interview Questions + Land Your Dev Job | Skilled.dev

Elixir, Phoenix, Typescript, and React — 2020 Edition was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

After seeing 8F3E’s article, How Fast Is Golang? which compares the performance of Python vs. Go using a Sieve of Eratosthenes as a microbenchmark, I thought it might be fun to see how Rust stacks up.

So, as an exercise to flex my Rust muscles before taking on a much harder challenge, I decided to fork 8F3E’s repo into my own: https://github.com/aisrael/sieve-of-eratosthenes

I then ported the Go code to Rust code, almost verbatim:

    let mut primes: Vec<i32> = Vec::new();

    for i in 2..=n {
        primes.push(i);
    }

    {
        let mut i = 0;
        while i < primes.len() {
            let factor = primes[i];
            sieve(&mut primes, factor);
            i += 1;
        }
    }

    for i in 0..primes.len() {
        println!("{}", primes[i])
    }

And

fn sieve(primes: &mut Vec<i32>, factor: i32) {
    let mut i = 0;
    while i < primes.len() {
        let value = primes[i];
        if value != factor {
            if value % factor == 0 {
                primes.remove(i);
            }
        }
        i += 1;
    }
}

In most cases I had to use a while loop rather than a for loop, because in Rust, for loops take one of two forms:

for i in 0..list.len() {
    if primes[i] != 0 {
        let factor = primes[i];
        sieve(&mut primes, factor);
    }
}

That won’t work if you mutate the vector and change its length (since the list.len() is evaluated only once, at the beginning of the loop). It will compile, but give us a run time error.

We might’ve also tried:

for factor in primes {
    sieve(&mut primes, factor);
}

But this won’t even compile!

The reason is all about why Rust is considered a language that helps you write ‘safer’ and more robust code. In this particular case, Rust sees that we’re mutating the primes vector but we’re also trying to iterate over it which can lead to hard to debug errors later on.

For instance, Rust if you try to do this:

sieve(&mut primes, primes[i]);

Rust complains with:

28 |                 sieve(&mut primes, primes[i]);
   |                 ----- -----------  ^^^^^^ immutable borrow occurs here
   |                 |     |
   |                 |     mutable borrow occurs here
   |                 mutable borrow later used by call

In this case, we have to introduce another, immutable, variable just to ‘hold’ the value of primes[i] which we can then pass to the sieve() function.

So, while Rust may feel like it’s getting in the way by being strict, the workaround is just to be more explicit, telling Rust that, “Yes, we know what we’re doing.”

EDIT: I rewrote the Rust implementation to more closely match the Go implementation

So, now instead of having to use while, the Rust code looks like:

    for i in 0..primes.len() {
        let factor = primes[i];
        if factor != 0 {
            sieve(&mut primes, factor);
        }
    }

See the changes in this commit: 567b05a.

I’ve also updated the results section after re-running the benchmarks.

Building

A quick note before just running the microbenchmarks: if we simply use rustc or even cargo using just no arguments, the Rust compiler will compile the resulting binary without optimisations and with debugging information.

This executable will be slower than the executable built with Go, because the Go compiler performs optimisations by default.

Hence, to be fair we need to pass flags to rustc to build the executable as if we had used Cargo and ran cargo build --release. I also took care to name the Rust binary differently from the Go binary:

rustc -C debuginfo=0 -C opt-level=3 sieve.rs -o bin/sieve-rs

Running

I ran and timed the Rust executable using zsh’s built-in time function:

time bin/sieve-rs 10000

Which, on my 2019 16" Macbook Pro gives

bin/sieve-rs 10000  0.01s user 0.00s system 14% cpu 0.075 total

I then wrote up a Python script that could run both the Go and Rust executables, giving them different and progressively increasing parameters, then capturing the user time output for each:

python bench.py

We can even specify minimum and maximum limits, and the benchmark script will try to compute the ‘steps’ in between by using powers of 10. For example, if given:

MIN=100 MAX=10000 python bench.py

Then the benchmark is run for both Go and Rust for 100, 200, …, 800, 900, 1000, 2000, …, 8000, 9000, then 10000 primes.

The bench.py script also overwrites the timing_data.csv in the testing folder. I then modified 8F3E’s original Jupyter notebook script to just use the CSV column headers for the graph labels.

Like 8F3E I didn’t bother running each n multiple times to get the average, since the trend was clear with one pass.

Results

I’ll let the graphs speak for themselves (lower is better):

Final Words

Sure, microbenchmarks are fun and, when judiciously applied and carefully analysed, can provide some useful insight into a very specific choice between, say, one implementation or another.

In the real-world though, with large applications and complex systems, language or compiler level differences are overshadowed by a host of other factors: algorithms, program architecture, I/O, infrastructure, system architecture, and so on.

So please don’t take these results too seriously.

On the other hand, the reason why we use Rust is precisely what I mentioned earlier — while there’s added effort and cognitive load on the developer because we have to worry about ownership and mutability, this in the long run helps us write more performant code (since there’s no overhead for garbage collection) and more importantly, more reliable code.

How Fast Is Rust? was originally published in The Startup on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve been following along, in Part 1: Closures, we first recollected our understanding of Rust closures.

Then in Part 2: Futures, we flexed our understanding of the Tokio runtime, the Future trait and the futures crate.

The Async you’ve been Awaiting for

Let’s pretend we don’t know anything and jump right in and try to write our first async function. To do this, we simply prepend the async keyword to our fn declaration:

async fn async_hello() {
    debug!("Hello, asynchronously!");
}

That’s all it takes, really!

But what happens if we try to call that function directly? In our main() let’s try writing:

fn main() {
    // ...

    async_hello();
}

We get a compiler warning:

warning: unused implementer of `core::future::future::Future` that must be used
   --> src/main.rs:169:5
    |
169 |     async_hello();
    |     ^^^^^^^^^^^^^^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: futures do nothing unless you `.await` or poll them

Rust is telling us that async_hello() is returning a Future that is unused. (You may recall we’ve seen this warning before, when we tried to simply call our returns_delayed_future() in rt.enter().)

But what happens if we try to .await it just like it says?

Now we get a compiler error, since our main() is not async:

error[E0728]: `await` is only allowed inside `async` functions and blocks
   --> src/main.rs:169:5
    |
102 | fn main() {
    |    ---- this is not `async`
...
169 |     async_hello().await;
    |     ^^^^^^^^^^^^^^^^^^^ only allowed inside `async` functions and blocks

The error mentions async blocks, let’s try wrapping that line in an async {}:

async {
    async_hello().await;
}

If that block is at the end of fn main() { ... }, we immediately run into:

error[E0308]: mismatched types
   --> src/main.rs:169:5
    |
102 |   fn main() {
    |             - expected `()` because of default return type
...
169 | /     async {
170 | |         async_hello().await;
171 | |     }
    | |     ^- help: try adding a semicolon: `;`
    | |_____|
    |       expected `()`, found opaque type

Now if we add a semicolon after the block as it says:

async {
    async_hello().await;
};

We once again get:

warning: unused implementer of `core::future::future::Future` that must be used
   --> src/main.rs:169:5
    |
169 | /     async {
170 | |         async_hello().await;
171 | |     };
    | |______^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: futures do nothing unless you `.await` or poll them

But this time, at the end of the async block.

We’re just running around in circles!

async -> Future

The key thing to remember is that whether we use async for a block or a function, both times we get a Future.

We already know what to do from Part 2 when we tried to call a function that returns a Future directly in main().

For a Future to be executed or polled, we need to pass our future to the Tokio runtime. Thus we need to write:

let mut rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async_hello());

So we get the output we desire:

08:10:29 [DEBUG] (1) Hello, asynchronously!

Now let’s try that async block again. First we’ll print something directly in the block:

let async_block = async {
    debug!("in async_block");
};
rt.block_on(async_block);

That works as expected. We can even ‘capture’ surrounding variables (Rust has a different and specific definition of async closure though and as of this writing, it’s a feature still deemed unstable):

{
    let x = 42;
    let async_capture = async {
        debug!("in async_capture, x => {}", x);
    };
    rt.block_on(async_capture);
}

You can think of the above code as equivalent to:

{
    let x = 42;
    let async_capture = future::lazy(|_| {
        debug!("in async_capture, x => {}", x);
    });
    rt.block_on(async_capture);
}

To recap we can think of async {...} as ‘desugaring’ to future::lazy(|_| {...}). They’re not exactly the same but for practical purposes they exhibit the same behavior.

Async Functions

Similarly, we can now begin to understand async fn as merely ‘sugar’ that returns a Future<Output=T>.

That is, the following functions are practically equivalent:

async fn async_returns_i32() -> i32 {
    42
}

fn returns_future_i32() -> impl Future<Output = i32> {
    future::ready(42)
}

So the way to read an async fn f() -> T is simply as being equivalent to an ordinary fn f() -> impl Future<Output = T>.

We can even return an async block as a Future:

fn returns_async_block_i32() -> impl Future<Output = i32> {
    async { 42 }
}

All three above functions when called behave pretty much exactly the same way.

From Futures to Async-Await

Up until now, we’ve been calling rt.block_on() multiple times.

Suppose our code looks like this:

rt.block_on(future::lazy(|_| debug!("in rt.block_on()")));
let r0 = rt.block_on(future::ready("Hello from rt.block_on()"));
debug!("{}", r0);
let r1 = rt.block_on(returns_impl_future_i32());
debug!("returns_impl_future_i32() -> {}", r1);
let r2 = rt.block_on(returns_dyn_future_i32());
debug!("returns_dyn_future_i32() -> {}", r2);
let r3 = rt.block_on(returns_future_result());
debug!("returns_future_result() -> {}", r3.unwrap());
let r4 = rt.block_on(returns_future_result_dyn_error());
debug!("returns_future_result_dyn_error() -> {}", r4.unwrap());
let r5 = rt.block_on(returns_delayed_future()));
debug!("returns_delayed_future() -> {}", r5))
let r6 = rt.block_on(wait_a_sec(future::ready(42)));
debug!("wait_a_sec(future::ready(42)) -> {}", r6));
rt.block_on(async_hello());
let async_block = async {
    debug!("in async_block");
};
rt.block_on(async_block);
let x = 42;
let async_capture = async {
    debug!("in async_capture, x => {}", x);
};
rt.block_on(async_capture);
let r7 = rt.block_on(async_returns_i32());
debug!("async_returns_i32 -> {}", r7);
let r8 = rt.block_on(returns_future_i32());
debug!("returns_future_i32 -> {}", r8);
let r9 = rt.block_on(returns_async_block_i32());
debug!("returns_async_block_i32 -> {}", r9);

We can rewrite all the above using future combinators to return a single future chain, or we can now truly start making use of the power and convenience async and await provide us.

As we’ve seen, an async {} and a future::lazy(|_| {…}) are practically interchangeable.

So first, we need to surround all those lines starting with future::lazy in a single async {...} block which we will block_on().

We then simply replace all the inner calls to rt.block_on(f) with f.await

The above code now becomes:

rt.block_on(
    async {
        debug!("in rt.block_on()");
        let r0 = future::ready("Hello from rt.block_on()").await;
        debug!("{}", r0);
        let r1 = returns_impl_future_i32().await;
        debug!("returns_impl_future_i32() -> {}", r1);
        let r2 = returns_dyn_future_i32().await;
        debug!("returns_dyn_future_i32() -> {}", r2);
        let r3 = returns_future_result().await;
        debug!("returns_future_result() -> {}", r3.unwrap());
        let r4 = returns_future_result_dyn_error().await;
        debug!("returns_future_result_dyn_error() -> {}", r4.unwrap());
        let r5 = returns_delayed_future().await;
        debug!("returns_delayed_future() -> {}", r5);
        let r6 = wait_a_sec(future::ready(42)).await;
        debug!("wait_a_sec(future::ready(42)) -> {}", r6);
        async_hello().await;
        let async_block = async {
            debug!("in async_block");
        };
        async_block.await;
        let x = 42;
        let async_capture = async {
            debug!("in async_capture, x => {}", x);
        };
        async_capture.await;
        let r7 = async_returns_i32().await;
        debug!("async_returns_i32 -> {}", r7);
        let r8 = returns_future_i32().await;
        debug!("returns_future_i32 -> {}", r8);
        let r9 = returns_async_block_i32().await;
        debug!("returns_async_block_i32 -> {}", r9);
    });

Like rt.block_on(), .await works just as well whether for a Future or an async block or an async fn.

#[tokio::main]

Now if all our main() function does is equivalent to the following:

fn main() {
    let mut rt = tokio::runtime::Runtime::new().unwrap();
    rt.block_on(async {
        // do async stuff
    });
}

Then Tokio lets us pull another trick to further simplify our code and make it as if Rust had an asynchronous runtime built-in.

It uses a custom Procedural Macro that allows us to attach the #[tokio::main] attribute macro to our main() function, which lets Tokio ‘inject’ all the boilerplate code into the resulting binary.

This is how Tokio lets us simply rewrite the above as:

#[tokio::main]
async fn main() {
    // do async stuff
}

Which is now how most contemporary examples of Rust asynchronous code starts.

Async Error Handling

As with Futures, we also need to handle errors gracefully. Fortunately, Rust already provides us with a way to reduce boilerplate via the ? operator for error handling, which works just as well with .awaited Results.

That is, suppose we have a function that may return an error:

fn fallible() -> Result<(), Box<dyn Error>> {
    let _f = std::fs::File::open("foo.txt")?;
    Ok(())
}

We can call it directly from main() as such:

let _ = fallible().unwrap();

If we don’t care much for graceful error handling (we should, but this is just an exercise), we can avoid unwrapping by also changing main() to return a Result<(), Box<dyn Error>>:

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // ...

    let _ = fallible()?;

    Ok(())
}

Now we can easily turn fallible asynchronous, and we can just as easily .await?:

async fn fallible() -> Result<(), Box<dyn Error>> {
    let _f = std::fs::File::open("foo.txt")?;
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    // ...
    let _ = fallible().await?;
    Ok(())
}

The key to error handling in Rust is being able to make sense of both the Result success and error types.

Because there are so many error types in Rust, I’ve found that using crates such as anyhow (https://github.com/dtolnay/anyhow) make dealing with return values and error handling a lot less confusing.

Post Scriptum

I originally started to write an article on how to implement an HTTP proxy in Rust from scratch, when I realized that there was so much material to cover and so decided to break it up into a series of articles instead.

This is the first part, which I’ve also long intended to write, where I attempt to bring the reader into a better understanding of Rust closures, futures, async-await and error handling.

We’ve come a long way—and yet we have barely scratched the surface of all the nuances and intricacies of all these.

However, it was never my intent to dive deep into all of that. In particular, we haven’t really looked into move closures, or when you’ll need to specify the Send marker trait, or Fn vs FnOnce vs FnMut.

These are important concepts, though the beginner in Rust asynchronous programming hopefully won’t need to worry about them.

Instead, I hope that, like me, as you’ve progressed in this rather long series you’ve come to a more practical intuition of how all these concepts relate and work together to provide a safe asynchronous programming environment that is truly performant, since there are no hidden costs of thread management, cloning, referencing counting, or garbage collection.

As the maxim goes Explicit is Better Than Implicit. All of the above are available in Rust, but only if the programmer expressly chooses to use them.

Because of that, while a Rust programmer has to juggle with a lot more things than developers in other languages, its a trade-off we’re willing to do when we need the safety and performance that Rust can provide.

You can checkout the accompanying source code up on GitHub: https://github.com/aisrael/rust-closures-futures-async-await

In a future article or series, I will refer to these posts as I write about using Rust’s hyper HTTP client/server library to build an HTTP proxy.