I am not going to lie: I love programming in C.

I know it’s a little bit irrational, but programming in C just feels right. Perhaps that’s because I did spend a lot of time programming with C as I contributed to the Linux kernel. Those were good times because I had the opportunity to learn from some of the best C programmers in the world. Hacking on the Linux kernel also taught me the connection between C programs and the machine code it compiles to, which I guess also influenced my thinking.

However, C is an unsafe language. It’s easy to make mistakes. But I spent so many hours on debugging C programs that at some point I ended up being reasonably good at it. I was often able to trace back what happened just from Linux kernel register dump (or “oops” as they call it) and fix a bug that I was not able to reproduce. I also learned to write C in a way that minimized the mistakes. I felt productive with C.

C is also a low level language. When you’re programming in C, you end up writing a lot of code just to do the simplest of tasks. There are no built-in data types so I wanted a hash table or a linked list, I usually ended up rolling my own. And I almost never used a binary search tree because writing one yourself took forever to get right. So perhaps C wasn’t perfect after all?

I did end up writing quite a bit of C++ professionally on two different occasions. The modernization effort that went into C++11 and subsequent versions made a big difference. You now had access to a variety of data structures and algorithms and with things like classes and templates, you ended up writing much less code. But C++ inherited the unsafety of C, which meant you still got to spend a bunch of time debugging your code.

Rust’s memory safety and complexity

When Rust appeared in 2015, I was really drawn to its promise of memory safety. I had done some Haskell programming around that time and felt that both languages — although designing for completely different use cases — shared the “if it compiles, it works” mindset. The fact that the borrow checker would be able to eliminate a class of issues that you’d encounter in C all the time felt magical. But after exploring the language for some time, I gave up because the tooling was changing all the time, and — quite frankly — I could not wrap my head around the borrow checker. Perhaps I was thinking in C too much.

Fast-forward to 2021, I started working with Rust professionally. I knew I didn’t want to use C or C++ and Glauber — my co-founder — had good experiences with Rust, so we went with that. I was probably fighting with the borrow checker for two to three months until things started to click. I was thinking in C too much. I then discovered the `Send` and `Sync` traits and how they really help you with multi-threaded programs. The tooling and the ecosystem had also really grown up since 2015. After the initial learning curve, programming in Rust became really productive!

But Rust is a complex language. The more I write Rust, the more I feel that there is too much complexity. When you use traits or async Rust, things start to feel complicated. Perhaps I am still thinking in C too much.

Zig as the modern C?

I think sometime in 2022 I discovered Zig because of two projects: Bun and TigerBeetle. As I had done with Rust in 2015, I started to explore it. I quickly discovered that I really like Zig because it feels like C! But just as with Rust, after initial exploration, I gave up on it because the tooling kept changing all the time and the ecosystem just wasn’t there for me.

In the summer of 2023, I wanted to give Zig another chance, partly because Glauber would not shut about it. I decided I would write a small proof-of-concept, a clone of SQLite in Zig. As I started the work, I discovered that Zig is a low level language. Not low level in the way C is, but much lower level than Rust. But I figured that’s an acceptable price to pay, because Zig just feels right. As I was working on a database, I needed a way to perform I/O efficiently. I was reading the TigerBeetle source code and then discovered Michell’s libxev and even contributed to it a bit. But after three months of weekend hacks, I ended up converting the code to Rust, and felt productive again.

Why I am not ready to switch to Zig just yet

So why did I end up ditching Zig in favor of Rust, although I still like Zig a lot?

For starters, comptime feels like a hack. This is probably irrational, but I just don’t like comptime very much. In fact, I much would prefer the legacy preprocessor macros C if I can’t have generics. But I think I should have generics in Zig.

You still end up chasing SIGSEGVs. I don’t know if that was me thinking too much in Rust, but I ended up writing lots of silly bugs that ended up in segmentation faults that I then chased with the debugger. I assume this would get better over time if I wrote more Zig, of course, but I’ve grown to trust the borrow checker.

You also end up leaking memory. Again, this is probably me thinking too much in Rust, but I also ended up writing code that leaked memory all the time. Of course, Zig has really nice tooling here to catch them quickly, so perhaps not such a big issue.

There are no books on Zig. I really, really need a good technical book to learn a programming language, but there are none for Zig. In contrast, there’s lots of great books on Rust and specific topics on Rust available. It just feels very hard to learn Zig.

Ecosystem is perhaps not there. There are two large Zig projects, Bun and TigerBeetle, and they’re great source trees to learn from. But there does not yet seem to be a large ecosystem to tap into like there is with Rust. Tooling is also perhaps not there yet. The toolchain Zig itself provides is great, but it still keeps changing a lot like Rust back in 2015. But more importantly, third-party tools like Github Copilot

Industry investment is not there. Rust is used across our industry. For me, the tipping points were when Microsoft started writing parts of Windows in Rust and Linux kernel added support for Rust. These two things alone mean that Rust is likely going to stay around for a very long time. In contrast, Zig does not yet have the level of industry backing right now. It does not mean that Zig won’t eventually get there, but Rust is a much safer (and boring?) option right now.

Summary

I really, really like Zig. But I am not ready to switch to it just yet. Rust is not perfect, but it’s now my go-to language for systems programming. I feel productive with it. Zig may be in the situation Rust was in 2015 and all it needs is more time. But for me, I will be waiting for the first book to come out, and then get back to it. I am also hoping that maybe they will dumb down the language for people like me, and provide an alternative to comptime. But until then, I will remain a happy Rustacean.

Over the years, I have learned a lot about how CPUs execute code and what their internals look like because I have worked on projects such as the Linux kernel and ScyllaDB, which are close to metal. I’ve even played a bit with Verilog in a futile attempt to build my own RISC-V core.

Unlike CPUs, GPUs have been mostly a black box to me, despite some exposure to them. I do remember playing around with NVIDIA RIVA 128 or something similar with DirectX when they were still 3D graphics accelerators and I have also tried to keep up with the times and did some basic shader programming on a contemporary GPUs. But I never really went deep with GPUs and am, therefore, thinking in CPU-centric terms.

However, because of the explosion of AI workloads, including large language models (LLMs), GPUs are becoming essential for modern compute. AI workloads are massive applications of tensor operations such as matrix addition and multiplication, which is a job for a GPU. But how does the modern GPU execute them and why are they much more efficient than running the workloads on a CPU?

Defining kernels to run on GPU

To dive into the world of GPUs, let’s look at CUDA, which is NVIDIA’s programming language that extends C to exploit data parallelism on GPUs. While most of AI programming today happens with high-level libraries such as PyTorch or TensorFlow, CUDA is still great to know to build intuition around how GPUs operate.

In CUDA, you write code for CPU (host code) or GPU (device code). CPU code is just mostly plain C, but CUDA extends the language in two ways: it allows you to define functions for GPUs, called kernels, and also provides a way to launch the kernels on the GPU. If you are wondering why GPU functions are called kernels, that’s because what you define in CUDA is executed in parallel unlike a function, which is executed serially.

If we take the example of vector addition where you perform an element-wise addition from vectors A and B in an output vector C, you would have something like the following as the kernel (the thing that runs on the GPU):

__global__
void vecAddKernel(float *A, float *B, float *C, int n) {
    int i = blockDim.x * blockIdx.x + threadIdx.x;
    if (i < n) C[i] = A[i] + B[i];
}

The __global__ keyword in CUDA marks a C function as a kernel that runs on the GPU. The input A and B vectors and the output C vector are passed as parameters. All the vectors are of n elements in size.

The part where the kernel is doing actual vector addition is:

if (i < n) C[i] = A[i] + B[i];

But what are the variables blockDim, blockIdx, and threadIdx in the kernel used to calculate the element index i and the bounds check here?

Launching kernels on GPU from CPU

To answer the question, let’s first jump to the host code that is needed to launch the kernel, which would look something like this:

void vecAdd(float *A, float *B, float *C, int n) {
    float *A_d, *B_d, *C_d;
    int size = n * sizeof(float);

    cudaMalloc(&A_d, size);
    cudaMalloc(&B_d, size);
    cudaMalloc(&C_d, size);

    cudaMemcpy(A_d, A, size, cudaMemcpyHostToDevice);
    cudaMemcpy(B_d, B, size, cudaMemcpyHostToDevice);

    vecAddKernel<<<ceil(n/256.0), 256>>>(A_d, B_d, C_d, n);

    cudaMemcpy(C, C_d, size, cudaMemcpyDeviceToHost);

    cudaFree(A_d);
    cudaFree(B_d);
    cudaFree(C_d);
}

As there are no particular keywords, it’s just a C function with the same parameters. The first thing you probably notice is the cudaMalloc, cudaMemcpy, and cudaFree function calls, which are part of CUDA’s heterogeneous memory management. Remember, this function executes on the CPU, which has access to the host memory. But GPUs have their memory, so you must allocate memory on the GPU and copy the input vectors. When the GPU kernel completes, you need to copy the results back to the host memory space and free up the memory used on the GPU.

Also, to launch the kernel, the CUDA syntax looks like a weird templatized function call, but what are the two additional configuration parameters? The first configures the number of blocks in a grid, and the second specifies the number of threads in a block. But what on earth is a grid? In CUDA, a kernel executes on a grid, which is blocks of threads. When we launch a kernel, we determine the grid with the two configuration parameters.

And if we go back to the kernel definition, we can see the unique variables blockDim, blockIdx, and threadIdx in use, which is the part where we define what part of the grid the kernel executes.

__global__
void vecAddKernel(float *A, float *B, float *C, int n) {
    int i = blockDim.x * blockIdx.x + threadIdx.x;
    if (i < n) C[i] = A[i] + B[i];
}

The blockDim variable specifies block dimensions because CUDA allows blocks to be three-dimensional. We only have a one-dimensional block for our vector example, so that part is straightforward. On the other hand, the blockIdx variable tells us which block the kernel is executing. Similarly, the threadIdx variable tells us which hardware thread executes the kernel.

In the kernel launch, we had the following, which means there are n/256 blocks, rounded up to the nearest integer and 256 threads per block.

void vecAdd(float *A, float *B, float *C, int n) {
    // ...
    vecAddKernel<<<ceil(n/256.0), 256>>>(A_d, B_d, C_d, n);
    // ...
}

For example, if the vector size n is 1000, we have four blocks and 256 threads per block, 1024 hardware threads. And that’s where the boundary check i < n in the kernel definition comes in handy: the first 1000 threads perform addition, but the remaining 24 threads are essentially a no-op.

So how is the execution on the GPU different from the CPU?

If we were to implement the same vector addition on a CPU, we would start out with something as follows, which has an explicit loop to iterate over all the n elements, performing an addition for every loop iteration:

void vecAdd(float *A, float *B, float *C, int n) {
    for (int i = 0; i < n; i++) {
        C[i] = A[i] + B[i];
    }
}

You might then attempt to use SIMD instructions to speed up the function either by manually unrolling the loop and using SIMD instructions via compiler intrinsics or assembly. Or you would turn on your compiler auto-vectorization optimization flags to make the compiler do that for you. But either way, the loop in function would always run serially across a batch of elements.

However, for GPUs, we specify parallelism as part of the kernel launch and only define the serial part that runs on GPU hardware threads of the vector addition in CUDA. That is, on the GPU, there is no loop for vector addition. Instead, every vector element addition runs its own thread. And now if you’re a CPU-centric programmer, you’re probably wondering how that can ever be fast.

That’s because the cost of a GPU hardware thread is minimal compared to operating system threads you would use on the CPU. The GPUs internally have a large register file compared to CPUs, meaning switching between threads is almost free because there’s no need to save and restore registers. Furthermore, thread scheduling happens at the hardware layer, which means you don’t need to transition into privileged kernel code for scheduling. What this all adds up is that you can expect a GPU thread creation and scheduling to take a few GPU cycles, whereas a POSIX thread, for example, can take microseconds to create and schedule.

Furthermore, GPUs optimize for parallelism further with warps, which is a set of 32 hardware threads, which all execute the same instruction at a time. So now you can imagine how 32 ALUs are executing the per-element addition in parallel, sharing the same control logic. For CPU-centric programmer, think of this as if there was a really large SIMD instruction (The warp control flows can also diverge, but the GPU thread scheduler handles that as well, by executing just parts of the warp at a time, depending on what control flow they take.)

However, a key difference to SIMD, if there is a cache miss on the GPU, execution can quickly switch to other warps and proceed in parallel while the memory loads are in-flight. In other words, GPUs optimize for high throughput where you throw a massive amount of hardware threads at a parallel data problem, whereas CPUs optimize for low latency execution of individual instructions.

tl;dr;

If you’re a CPU-centric programmer, one way to think about execution model of GPUs is that instead of looping over a data set, you essentially break down the job in lots and lots of threads, which work on a subset of the data. Of course, there are some key differences on a GPU compared to a CPU. First of all, you define a kernel instead of a function and specify parallelism explicitly when the kernel is launched. But also, context switching between GPU threads is orders of magnitude cheaper than OS threads, which is why you can just throw lots and lots of threads at a problem. Finally, GPUs further optimize thread execution by grouping a bunch of threads as warps, which execute using shared control logic on parallel ALUs. And if there is a cache miss, the GPU can quickly switch to another warp while the memory load happens in the background.

The fastest access to your database is when your database is embedded in your application. However, most contemporary databases such as PostgreSQL and MySQL use a client-server model where the client executes SQL remotely over a network write protocol. But there’s one database engine that’s built different: SQLite. It implements a full-blown SQL relational database engine that you can just drop into your application for microsecond-level SQL queries.

Unfortunately, SQLite is really built around the concept of a local filesystem, which makes it hard to build applications that access a shared database. That’s why when we started working towards making SQLite easier to use for applications, we built a server in our fork, libSQL, that allows remote SQL execution over HTTP, much like what you see with other database such as Neon or PlanetScale. The server approach is great for serverless applications as well, because the serverless platforms such as like Cloudflare Workers or AWS Lambda don’t have enough persistent state to bring over SQLite.

But there is another way to make SQLite work for applications that need a shared database: replication. LiteFS is a popular SQLite replication sidecar that works via a special user space filesystem that let’s applications just use SQLite, with LiteFS dealing with replication in the background. But since remote SQL execution with HTTP is important for serverless, it would be awesome to have both options.

So how does it work?

First, if you’re working with SQLite you might have a database with a users table with two columns, name and email:

$ sqlite3 hello.db
SQLite version 3.42.0 2023-05-16 12:36:15
Enter ".help" for usage hints.
sqlite> CREATE TABLE users (name TEXT, email TEXT);
sqlite> INSERT INTO users VALUES ('Pekka Enberg', 'penberg@iki.fi');

To access the local database file, you can use a new libsql Rust crate as follows:

use libsql::{Database, params};

fn main() -> anyhow::Result<()> {
    // Path to the database file.
    let db_file = "hello.db";

    // Open the database file.
    let db = Database::open(db_file)?;

    // Create a connection to the database.
    let conn = db.connect()?;

    // Prepare a SQL statement for execution.
    let query = conn.prepare("SELECT name, email FROM users")?;

    // Execute the prepared SQL statement.
    let rows = query.query(&params![])?;

    // Fetch the first row from the result set.
    let row = rows.next()?.unwrap();

    // Extract columns from the row...
    let name = row.get::<&str>(0).unwrap();
    let email = row.get::<&str>(1).unwrap();

    // ...and print them out.
    println!("Name: {}, email: {}", name, email);

    Ok(())
}

If you run the program, you can see the first row of the users table printed out:

penberg@vonneumann libsql % cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 3.79s
     Running `/Users/penberg/src/libsql/libsql-examples/rust/target/debug/libsql-example`
Name: Pekka Enberg, email: penberg@iki.fi

If you are a Rust developer, the example program might look familiar to you. And that’s because the API is inspired by rusqlite, a popular Rust API for SQLite. The same program using rusqlite would look as follows:

use rusqlite::Connection;

fn main() -> anyhow::Result<()> {
    let conn = Connection::open("hello.db")?;
    let mut query = conn.prepare("SELECT name, email FROM users")?;
    let mut rows = query.query([])?;
    let row = rows.next()?.unwrap();
    let name: String = row.get(0).unwrap();
    let email: String = row.get(1).unwrap();
    println!("Name: {}, email: {}", name, email);
    Ok(())
}

So far so good, but how does the libsql crate provide replication?

Well, first we need the libSQL server running:

penberg@vonneumann sqld % cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.91s
     Running `target/debug/sqld`
           _     _
 ___  __ _| | __| |
/ __|/ _` | |/ _` |
\__ \ (_| | | (_| |
|___/\__, |_|\__,_|
        |_|

Welcome to sqld!

version: 0.17.2
commit SHA: be9a6efc48948179427fcbee4d4a8e391a943ac6
build date: 2023-08-08

This software is in BETA version.
If you encounter any bug, please open an issue at https://github.com/libsql/sqld/issues

config:
 - mode: standalone
 - database path: data.sqld
 - extensions path: <disabled>
 - listening for HTTP requests on: 127.0.0.1:8080
 - grpc_tls: no
2023-08-08T12:27:58.227665Z  WARN sqld: No authentication specified, the server will not require authentication
2023-08-08T12:27:58.228429Z  INFO sqld::http: listening for HTTP requests on 127.0.0.1:8080
2023-08-08T12:27:58.228456Z  WARN sqld: No server heartbeat configured

Then we need to create the schema and insert data into into like we did with plain SQLite.

(We’re going to just borrow the Turso CLI for this because sqld does not ship with a shell yet.)

penberg@vonneumann ~ % turso db shell http://localhost:8080
Connected to http://localhost:8080

Welcome to Turso SQL shell!

Type ".quit" to exit the shell and ".help" to list all available commands.

→  CREATE TABLE users (name TEXT, email TEXT);
→  INSERT INTO users VALUES ('Pekka Enberg', 'penberg@iki.fi');

We can just switch our simple program to synchronize the database file from the server by changing the code to:

use libsql::{Database, Opts, params};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let db_file = "hello.db";
    let opts = Opts::with_http_sync("http://localhost:8080".to_owned());
    let db = Database::open_with_opts(db_file, opts).await?;
    let conn = db.connect()?;
    db.sync().await?;
    let query = conn.prepare("SELECT name, email FROM users")?;
    let rows = query.query(&params![])?;
    let row = rows.next()?.unwrap();
    let name = row.get::<&str>(0).unwrap();
    let email = row.get::<&str>(1).unwrap();
    println!("Name: {}, email: {}", name, email);
    Ok(())
}

And now if we run the program, the libsql crate will synchronize its database file (that starts out as empty) from the server and perform SQL queries locally:

penberg@vonneumann libsql-sync % cargo run
    Updating git repository `https://github.com/libsql/libsql/`
   Compiling libsql-sync-example v0.1.0 (/Users/penberg/src/libsql/libsql-examples/rust/libsql-sync)
    Finished dev [unoptimized + debuginfo] target(s) in 2.43s
     Running `/Users/penberg/src/libsql/libsql-examples/rust/target/debug/libsql-sync-example`
Name: Pekka Enberg, email: penberg@iki.fi

It’s not just Rust either. We’re building language bindings for JavaScript, Python, and Go on top of the libsql Rust crate. For example, with JavaScript you can do the following to synchronize your database file and query locally:

import Database from 'libsql-experimental';

const options = { syncUrl: "http://localhost:8080" };
const db = new Database('hello.db', options);

db.sync();

const row = db.prepare("SELECT * FROM users WHERE id = ?").get(1);

console.log(`Name: ${row.name}, email: ${row.email}`);

(If you’re a JavaScript developer and that looks familiar to you, that’s because it’s inspired by better-sqlite3, although we’re switching to sqlite-async to support write delegation later.)

So how fast is it? Well, as fast as SQLite. The micro-benchmarks in the libsql crate, for example, show that with prepared statements, you can perform a SQL query in just 200 nanoseconds:

penberg@vonneumann core % cargo bench
     Running benches/benchmark.rs (/Users/penberg/src/libsql/libsql/crates/target/release/deps/benchmark-a95c7258e88dbd24)
Gnuplot not found, using plotters backend
libsql/SELECT * FROM users LIMIT 1
                        time:   [190.57 ns 190.84 ns 191.16 ns]
                        thrpt:  [5.2313 Melem/s 5.2401 Melem/s 5.2475 Melem/s]
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  1 (1.00%) high severe
libsql/SELECT * FROM users LIMIT 100
                        time:   [190.76 ns 190.95 ns 191.13 ns]
                        thrpt:  [5.2320 Melem/s 5.2370 Melem/s 5.2422 Melem/s]
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

If that sounds insanely fast, that’s because SQLite is running like a bat out of hell and in the micro-benchmark it’s really just reading from DRAM (well, really your CPU caches). For real use cases, you can still expect microsecond-level query latency, though.

There’s still heaps of work to do. We want the libsql crate to be the main crate people consume for local database files, local replicas, but also remote SQL execution over HTTP. We essentially want people to be able to have their SQLite database wherever they need it, inside the application, on an edge replica, laptop, or wherever they’re running their application and need their database. We’ll also be obviously integrating this in the Turso CLI when things are more stable.

Microsecond-level SQL query latency with libSQL local replicas was originally published in Turso blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Turso is an database for the edge, which is why it departs from how most databases operate. First and foremost, Turso works both locally and remotely by relying on libSQL, our fork of SQLite. When working remotely, the database is accessed through HTTP and WebSockets, as opposed to a traditional TCP-based wire protocols, which makes Turso accessible from almost any environment, including edge functions.

But also, we’re developing Turso control plane as a REST API because we believe infrastructure should be programmable. In fact, early on, we made a decision to make the whole UI initially just be a CLI, to make sure we didn’t take short-cuts in the API design. Of course, we’re planning to add a web UI down the road, but the current developer experience with just a CLI is surprisingly smooth and simple!

We also put a lot of thought in Turso to make the creation and operating geographically dispersed replicas easy, so you can, for example, programmatically create a replica in Australia when you see increased traffic coming from that area and destroy it right after. The CLI plays the role of a reference implementation for the API, and together with the reference documentation that we plan to publish soon, should help developers understand how to integrate Turso database operations into their workflows.

But there’s a more fundamental reason to making the CLI open source: we strongly believe in the transparency and hackability that open source brings. In fact, the replication code for libSQL and rest of the data plane that powers Turso’s replicas is all open source, and we intend to keep developing new features in the core database in open source because as Glauber, the CEO my co-founder keeps pointing out: it’s just in our DNA. That’s also why sponsoring Drizzle and working with the team towards bringing a great experience with Drizzle on Turso.

So with that out of the way, go check out the Turso CLI on Github:

https://github.com/chiselstrike/turso-cli

If you have questions/comments/bug reports, please drop by our Discord or leave an issue on Github!

Turso CLI is now open source was originally published in Turso blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

We at ChiselStrike are building a backend-as-a-service product and run most of our infrastructure on AWS using tools like Kubernetes and Terraform, which are arguably the industry standard for infrastructure. Although everybody talks about how Kubernetes, for example, is complex and hard, most companies out there use it. It’s hardly surprising to folks like me who lived the pre-cloud computing days. You literally had to file an infrastructure ticket to get anything done and sometimes it take multiple weeks to get it done. So being able to fire up instances on the cloud with automated cluster management and provisioning, it feels pretty darn amazing knowing the history.

However, infrastructure is still hard, and you need expert infrastructure engineers to work on it. As a developer, I still often resort to just asking our infrastructure folks to do things for me. Instead of filing a an infrastructure change request ticket, I ping them on Slack and wait for an hour or maximum a day, instead of waiting for weeks. Don’t get me wrong, I would like to contribute more, but whenever people start to talk about EKS storage classes or Helm charts, I want to run away screaming. And I think I should know some of this stuff because if you want to talk about Linux kernel namespaces or cgroups, I feel right at home. There’s something about infrastructure tooling that’s just foreign to us developers.

Developer friendly infrastructure?

Fly.io has made a big splash recently on no-nonsense infrastructure. When I first heard about them, I was sceptical. Infrastructure is tons and tons of hard work, so how could anyone actually compete with industry giants like AWS, Kubernetes or Terraform? But the more I read about their architecture the more it made sense. To put my money where my mouth is, I decided to give it a go and see how I could deploy ChiselStrike applications on Fly.io.

The first thing I did was go to the Fly.io web site and sign up. The experience was actually bit confusing on the pricing. I am on a “Hobby Plan”, but still they asked for my credit card (which I entered). I have so far not been charged with anything, but am unable to find pricing information to confirm that’s the case. After signing up, I installed the flyctlcommand line tool, which lets users provision and control deployments.

From a developer perspective, there are two main abstractions on Fly.io: applications and machines. Applications are the high-level unit of deployment whereas machines are a lower-level interface for provisioning micro-VMs. For my own use case, I decided to just focus on applications, although I did play around a bit wit the machine interface too.

For deployments, Fly.io uses Firecracker-based micro-VMs, which supposedly boot in some hundreds of milliseconds and are automatically suspended when inactive, which reduces cost. The way you deploy an application to Fly.io infrastructure is via a Dockerfile, which is internally consumed by Fly.io.

Deploying ChiselStrike apps on Fly.io

The first thing I needed to do was to make it easy to build a Docker image of a ChiselStrike applications. After some trial and error, this turned out to be pretty easy. As ChiselStrike can already be consumed via npm, I just needed to make sure I copy all the project contents to the image and then run npm install in the container, and launch npm run dev:

FROM node:18-slim
COPY . /opt/app
WORKDIR /opt/app
RUN npm install
EXPOSE 8080/tcp
CMD npm run dev -- -- --api-listen-addr 0.0.0.0:8080

The resulting image is slightly bigger than I had hoped for, but part of the reason is that ChiselStrike uses npx internally so the final image needs Node.js. However, I consider that to be a bug and once that’s fixed, the final image only needs an OS and ChiselStrike, and the OS part can be also shrunk with Alpine Linux, for example.

So how does the deploying a ChiselStrike application to Fly.io look like in practice?

You first create a ChiselStrike app with:

npx create-chiselstrike-app my-backend

The command creates a project template with a “hello world” endpoint.

Once you have signed up to Fly.io, deploying is as simple as:

fly launch --generate-name --region fra --now

That’s it! No additional configuration or setup needed. As long as you have a working Dockerfile, Fly.io deals with the rest. As a developer, that’s a pretty magical experience!

Summary

Although I absolutely fell in love with Fly.io, the APIs and documentation are still bit rough around the edges. As I was trying to get gRPC working, I had to consult random discussion forum posts and the age-old procedure of trial-and-error to get things working. I did hit random DNS errors when attempting to ssh to the Fly.io machines and OpenSSL failures while testing. I also hit a really weird error when lauching an app on macOS that didn’t happen on Linux.

That said, Fly.io really does feel like it’s going to make infrastructure easy for us developers. It’s very easy to get started and the interfaces are very discoverable. I am personally looking forward to playing around with Fly.io more in the future, and maybe doing a more complete proof-of-concept for deploying ChiselStrike applications.

Fly.io makes infrastructure easy for developers was originally published in Turso blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Latency drives modern infrastructure architecture

Latency is becoming essential across various use cases in modern applications. For example, in e-commerce, going beyond single-digit millisecond latencies is believed to hurt the bottom line: a decade ago, Amazon reported that every 100 milliseconds in response time cost them 1% of sales. And it’s not just e-commerce; the need for low latency is becoming more important for OLAP and analytics workloads as well:

Kishore Gopalakrishna on Twitter: "Latency is becoming more and more important for OLAP/analytics workloads as well - especially when it's user-facing.At LinkedIn, "Who Viewed My Profile" had 99th pct latency of < 100ms with an average latency of ~5ms https://t.co/6DQbOsGfJF / Twitter"

Latency is becoming more and more important for OLAP/analytics workloads as well - especially when it's user-facing.At LinkedIn, "Who Viewed My Profile" had 99th pct latency of < 100ms with an average latency of ~5ms https://t.co/6DQbOsGfJF

But how is this need for speed driving infrastructure change today? Let’s start from the constraints.

Geographical latency is a fundamental limit, which infrastructure needs to hide from the end-user. For example, according to Cloudflare, it takes up to 150 milliseconds for a packet sent from New York to arrive in London¹, but we don’t want users to experience that.

Content delivery platforms such as Vercel and Netlify allow the deployment of web application assets across multiple geographically distributed locations closer to the end-users, eliminating the geographical latency and making our websites snappy! More generally, edge computing platforms such as Cloudflare and Fastly have the infrastructure to deploy your code on the edge, eliminating geographical latency. For example, Cloudflare Workers provides an abstraction for deploying code that automatically runs everywhere. This deploy once, run everywhere abstraction of edge computing is pretty powerful, but it also creates an infrastructure problem: edge locations are closer to the end-user but have little capacity. On the other hand, data centers that power the cloud can be far away but have more storage and compute capacity. How do we reconcile that in our infrastructure?

One infrastructure impact is that the virtualization technology on edge must be extremely lightweight to support massive multitenancy. With cloud computing, we’re used to reasoning about virtual machines or containers, but as it turns out, they’re too heavy-weight for edge deployments. Kenton Varda has an excellent presentation on how Cloudflare uses the V8 JavaScript engine’s capabilities for lightweight virtualization:

Fine-Grained Sandboxing with V8 Isolates

With V8 providing virtualization, modern application deployments on edge effectively become either JavaScript or WebAssembly serverless functions. However, the next frontier in edge computing infrastructure is solving database access, which is still a problem.

The database access problem has two parts:

• Serverless functions have no state, which complicating database access.
• Storage capacity at large will likely remain in the data centers powering cloud computing.

The lack of state in serverless functions means that with traditional databases, every access has to establish a connection to the database, which is slow. Prisma, for example, attempts to address this problem with a proxy that handles the connection pooling and solutions such as pg_bouncer also solve the problem for Postgres:

Karthik Ranganathan on Twitter: "Finally, the issue of persistent connections. In addition to tools like Prisma listed by @penberg, there are tools in the PostgreSQL ecosystem like pg_bouncer which can also cut connection establishment latency that we're looking at. 5/ / Twitter"

Finally, the issue of persistent connections. In addition to tools like Prisma listed by @penberg, there are tools in the PostgreSQL ecosystem like pg_bouncer which can also cut connection establishment latency that we're looking at. 5/

However, although a proxy or pg_bouncer approach solves the connection pooling issue (and likely will be integrated into future serverless runtimes), the issue of some database access latency remains as databases are deployed into the cloud, but application logic is increasingly running on the edge.

We at ChiselStrike have been exploring an option of going even further to drive down latency: integrating the relational database engine (in our case SQLite) directly to the serverless runtime:

GitHub - chiselstrike/chiselstore: SQLite + Little Raft = 🚀

An embedded database management system solves the issue of latency, but it’s not a complete solution as you still need to replicate the data and storage capacity is limited on edge locations. ChiselStore is using Raft consensus protocol to replicate data, but it’s lacking optimizations needed to making it fast. And although we have ideas how to integrate an edge database approach with a centralized database that has higher storage capacity, we have not started the work.

Database vendors are also tacking this problem via an approach dubbed as serverless databases. For example, CockroachDB released their serverless offering end of last year, and YugaByte is working on their serverless offering:

JavaScript is not available.

In summary, while there’s lots of exciting work happening in serverless database space, my belief is that combining the low latency of the edge with the cloud’s storage capacity will be one of the big frontiers to explore in edge and serverless computing infrastructure in the years to come and will transform how we think about serverless and data!

Footnotes:

¹ As various people on Twitter point out, the geographical latency between New York and London can be as low as 30–50 milliseconds if you are willing to pay for it.

Latency is driving edge and serverless computing infrastructure was originally published in Turso blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The network stack architecture is under a revolution because the network is becoming faster than the CPU. Since the breakdown of Dennard scaling in 2006, the single-threaded performance of a CPU has stagnated [Rupp, 2014]. Meanwhile, NICs are getting faster and faster. 10 GbE NICs are a commodity today, but at high-end NICs are already at 200 GbE and improving further. The faster the NIC becomes, the smaller the time budget to process an individual packet is. For example, on a 10 GbE NIC, the time between two 1538-byte packets is 1230 ns [Corbet, 2015]. However, with 200 GbE, the time between such packets is as already low as 61 ns. These high packet rates impose a significant challenge to both the hardware and OS network stack architecture. In short, the traditional in-kernel network stack design is inadequate to keep up with the high rate of packets.

In this article, we first look at the deficiencies of the POSIX sockets API and their in-kernel implementations. We then discuss kernel-bypass networking and programmable packet processing, including offloading to SmartNICs, to understand how the network stack is changing to meet the needs of contemporary hardware and workloads.

POSIX sockets

POSIX sockets are the standard programming interface for networking. First appeared in 1983 as BSD sockets, POSIX sockets have been adopted by most commodity operating systems from Linux to Windows. In the POSIX socket model, applications create a socket, which represents a flow, and use that socket’s file descriptor to send and receive data over the network. For example, a server application that wants to accept TCP connections first creates a socket with the socket() operation, then binds the socket to an interface or an address with the bind() operation and starts listening to incoming connections with the listen() operation. When there is a new connection, the server application establishes a new connection with the accept() operation and then sends and receives data over the connection with the sendmsg() and recvmsg() operations.

The overheads of in-kernel network stacks are widely known [Han et al., 2012; Pesterev et al, 2012; Rizzo, 2012; Jeong, 2014; Lin et al., 2016; Høiland-Jørgensen et al., 2018]. Most in-kernel network stacks implement POSIX socket operations as system calls. That is, for both control plane operation, such as socket(), and data plane operations, such as sendmsg(), applications transfer control to the kernel using a system call. System calls are a problem for network-intensive applications because they have significant overheads (e.g., context switch and CPU cache pollution) [Soares and Stumm, 2010] and, recently, from OS mitigations for the Meltdown attack. Also, the POSIX socket interface is oblivious of multicore CPU and multi-queue NIC architecture. The application can access a socket on a different CPU that is managing the packet queue, which requires the OS to move packets between CPU cores.

The socket API also pushes the OS to adopt a design, which demands dynamic memory allocation and locking. When a packet arrives on the NIC, the OS first wraps the packet in a buffer object, called a socket buffer (skb) in Linux and network memory buffer (mbuf) in FreeBSD. The allocation of the buffer object puts much stress on the OS dynamic memory allocator. Once allocated, the OS then passes the buffer object down the in-kernel network stack for further processing. The buffer object lives until the application consumes all the data it holds with the recvmsg() system call. As the buffer object can be forwarded between CPU cores and accessed from multiple threads, locks must be used to protect against concurrent access.

Kernel-bypass networking

Kernel-bypass networking eliminates the overheads of in-kernel network stacks by moving protocol processing to userspace. The packet I/O is either handled by the hardware, the OS, or by userspace, depending on the specific kernel-bypass architecture in use. For example, RDMA provides interfaces for directly accessing the memory of a remote machine, bypassing the OS for data plane operations altogether. In other words, an application receives messages in an RDMA-managed memory region without any inference from the OS. For Ethernet, the OS can dedicate the NIC to an application (e.g., DPDK), which programs it from userspace, or the OS can continue to manage the NIC by allowing applications to map NIC queues to their address space (e.g. Netmap). Either way, packets flow from the NIC to userspace with minimal interference by the OS.

With the OS limiting itself to managing packet I/O, userspace is responsible for implementing the rest of the network stack. In practice, this means that userspace must at least implement the TCP/IP protocol suite and provide interfaces for applications to access messages carried over by the protocols. Various userspace network stacks exist, but none of them have become a standard. Development and testing of the stacks are therefore fragmented, which limits their usefulness. Also, while it is possible to implement POSIX sockets API as a library [Hruby et al., 2014], most userspace stacks provide their interfaces, which limits adoption and compatibility.

Programmable packet processing

Programmable packet processors are emerging as another technique to address the limitations of the in-kernel network stack. They allow execution of user-defined code either in the OS or the hardware. XDP is Linux’s programmable packet processor. It allows a user-defined eBPF program to process a packet before it enters the in-kernel network stack. The eBPF program can either process the packet in full, perform some preprocessing and forward it to the in-kernel stack, or, with AF_XDP, forward the packet to userspace memory buffer after processing. Also, some SmartNICs, such as the Netronome Agilio CX, are capable of running eBPF programs directly on hardware. Offloads packet processing from the CPU to the NIC can reduce packet processing latency and improve energy-efficiency.

Applications can leverage XDP by offloading some parts of their request processing in an eBPF program. At a high-level, an eBPF program for XDP could look as follows:

#include <linux/bpf.h>
#include <linux/if_ether.h>

SEC("xdp_prog")
int xdp_program(struct xdp_md *ctx)
{
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    int action = XDP_DROP; 
    struct ethhdr *eth = data;
    if (ethernet_packet_matches(eth)) {
        action = XDP_PASS;
    }
    return action;
}

In the above example, xdp_program() is the entry point for the eBPF program. The XDP subsystem passes the start and end pointers of a packet to process. The value returned by xdp_program() determines the action to take on the packet. In our example, the program drops packets by default with XDP_DROP. If the ethernet_packet_matches() function returns true, the program returns the XDP_PASS action instead, which allows the packet to enter the in-kernel network stack. For a detailed tutorial on XDP, please refer to Andy Gospodarek's and Jesper Dangaard Brouer's talk XDP for the Rest of Us.

Summary

POSIX sockets API and the traditional in-kernel network stacks are not suitable for leveraging the hardware today. Kernel-bypass networking is a promising technique to address the issues, but they still lack standard interfaces and protocol implementations for applications to take advantage of them. Programmable packet processing is an exciting emerging technique for applications that can offload their request processing in part or in full to either in-kernel virtual machine or SmartNIC.

References

Jonathan Corbet. (2015). Improving Linux networking performance. https://lwn.net/Articles/629155/

Sangjin Han, Scott Marshall, Byung-Gon Chun, and Sylvia Ratnasamy. (2012). MegaPipe: a new programming interface for scalable network I/O. In Proceedings OSDI’12.

Tomas Hruby, Teodor Crivat, Herbert Bos, and Andrew S. Tanenbaum. (2014). On sockets and system calls minimizing context switches for the socket API. In Proceedings TRIOS’14.

Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. 2018. The eXpress data path: fast programmable packet processing in the operating system kernel. In Proceedings CoNEXT ‘18.

Eun Young Jeong, Shinae Woo, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. (2014). mTCP: a highly scalable user-level TCP stack for multicore systems. In Proceedings of NSDI’14.

Xiaofeng Lin, Yu Chen, Xiaodong Li, Junjie Mao, Jiaquan He, Wei Xu, and Yuanchun Shi. (2016). Scalable Kernel TCP Design and Implementation for Short-Lived Connections. In Proceedings ASPLOS ‘16.

Aleksey Pesterev, Jacob Strauss, Nickolai Zeldovich, and Robert T. Morris. (2012). Improving network connection locality on multicore systems. In Proceedings of EuroSys ‘12.

Luigi Rizzo. (2012). Netmap: a novel framework for fast packet I/O. In Proceedings of USENIX ATC’12.

Karl Rupp. (2018). 42 Years of Microprocessor Trend Data. https://www.karlrupp.net/2018/02/42-years-of-microprocessor-trend-data/

Livio Soares and Michael Stumm. (2010). FlexSC: Flexible System Call Scheduling with Exception-Less System Calls. In Proceedings of OSDI ‘10.

In this article, we will first outline the architecture of classical trap-and-emulate hypervisors that were invented in the 1970s. We then describe how hypervisors evolved from the 1970s to today’s cloud computing era. Finally, we will look at future trends that affect hypervisor design.

(This article was inspired by an awesome talk on Amazon’s Nitro project by Anthony Liguori, which I highly recommend everyone interested in hypervisors and cloud infrastructure to watch.)

Architecture

A hypervisor is a piece of system software that provides virtual machines (VMs), which users can use to run their OS and applications on. The hypervisor provides isolation between VMs, which run independent of each other, and also allows different VMs to run their own OS. Like other virtualization techniques, hypervisors provide multitenancy, which simplifies machine provision and administration. One of the main criticisms against hypervisors is that they tend to usually be heavy-weight compared to other virtualization techniques like containers (Morabito et al., 2015). However, it’s also possible to build hypervisors that are light-weight (Manco et al., 2017) and also make the guest OS more light-weight when running in under a hypervisor (Madhavapeddy, 2013).

A hypervisor can be decomposed into two major parts: the virtual machine monitor (VMM) and the device model. The VMM is responsible for setting up VMs and handling traps (a.k.a VM exits) caused by the guest OS executing privileged instructions like I/O access. The device model, on the other hand, is responsible for implementing I/O interfaces for all the devices like networking cards, storage, and so on, the hypervisor supports. Hypervisor architecture is illustrated in the following diagram.

(The terms hypervisor and VMM are often used interchangeably. However, we refer to hypervisor as the combination of a VMM and a device model.)

Virtual machine monitor (VMM)

A VMM must satisfy three properties (Popek and Goldberg, 1973):

• Equivalence property states that program execution has identical observable behavior on bare metal and under VMM, except for timing and resource availability, which are difficult to preserve because of shared physical hardware.
• Efficiency property states that the majority of program instructions are executed directly on a physical CPU without interference from the hypervisor.
• Resource control property states that the VMM manages all hardware resources. Virtual machines require permission from the hypervisor to directly access hardware.

As a side note, it’s worth noting that emulators satisfy both equivalence and resource control properties, but does not satisfy the efficiency property.

The KVM subsystem in the Linux kernel (and other OS’es it has been ported to), for example, provides the building blocks for implementing a VMM. The KVM subsystem is effectively a portable abstraction over CPU hardware virtualization capabilities, which can be leveraged by userspace applications like QEMU to implement a VMM or a full hypervisor.

Device Model

The device model is the part of a hypervisor, which provides I/O interfaces for virtual machines. While the VMM is responsible for handling traps, it delegates I/O requests to the appropriate device model. Examples of device models are virtualized NICs and storage devices. Device models can either provide interface for a real hardware device or a paravirtualized device. The device model can be implemented either using software, like the virtio family of drivers, or in hardware, using SR-IOV, for example.

I/O Virtualization

To implement a device model, I/O virtualization is needed. The two approaches for I/O virtualization are software-based and hardware-assisted.

Software-based I/O virtualization implements I/O interfaces in software to allow sharing the same physical devices across multiple virtual machines. Software-based I/O virtualization can be implemented on top of various different backends. For example, a software-based storage device can be layered on top of a block device or a filesystem. One issue with software-based approach is that the device model uses the same CPU resources that the vCPUs, which reduces available CPU capacity and causes jitter.

Hardware-assisted I/O virtualization implements I/O interfaces in hardware. This approach requires hardware support for sharing the same physical device across multiple virtual machines. SRV-IO is a PCI extension, which allows a physical PCI function to be partitioned into multiple virtual PCI functions.

Evolution

The semantics of a trap-and-emulate VMM was formalized in the early 1970s (Popek and Goldberg, 1973) and made popular again in the mid-1990s for running commodity OS’es on multicore machines (Bugnion et al., 1997). However, the most popular machine architecture at the time, Intel x86, was not virtualizable because some of it’s privileged instructions did not trap.

The VMware hypervisor, which targeted x86, was first released in 1999. It used binary translation to replace privileged instructions to trap into the hypervisor, while still running unprivileged instructions directly on the physical CPU, which solved x86’s virtualization issues (Adams and Agesen, 2006). This allowed the VMware hypervisor to run unmodified commodity OS’es on x86 hardware in virtual machines without the performance penalty of emulation.

The Xen hypervisor released first in 2003 took a different approach to solving the x86 virtualization issue. Instead of binary translation, they modified the source code of the guest OS to trap to the hypervisor instead of executing non-trapping privileged instructions.

Intel and AMD released x86 CPUs with virtualization extensions in 2005 and 2006, which made classic trap-and-emulate virtualization possible. KVM, initially developed for Linux, implements a kernel subsystem that in combination with QEMU’s device model provides a full hypervisor. Initially, the KVM project provided software-based device model that emulated full hardware devices, but later acquired paravirtualized I/O device model when the virtio device model was introduced.

Future

The classic hypervisor architecture has stood the test of time but there some trends that affect hypervisor design.

Hardware virtualization is becoming more wide-spread. For example, the Amazon Nitro project (talk by Anthony Liguori) takes an unconventional approach to hypervisor design, which replaces all of the software-based device model with hardware virtualization as illustrated in this diagram. Amazon’s Nitro hypervisor also uses a custom designed VMM that leverage’s Linux KVM.

Operating systems have also started to evolve to accommodate hypervisors better. Unikernels are an interesting OS design approach that packages the OS and the application into one bundle, which runs in the same CPU protection level (Madhavapeddy, 2013). This eliminates the traditional separation between kernel and user space, which reduces context switch and system call overheads at the expense of losing some OS functionality. The basic idea was already pioneered earlier in the form of library OSes, but the much simpler device model of a hypervisor compared to bare metal made the idea much more feasible for real world use.

Light-weight virtualization is becoming more and more important as the use of cloud computing grows. Containers are excellent technology for providing light-weight virtualization. However, containers are unable to provide the full isolation capabilities of VMs, and have various security problems because containers share the same host OS and have access to the large OS system call interface (Manco et al., 2017). Hypervisors can be slimmed down significantly (Manco et al., 2017) and unikernels provide even larger opportunity to optimize the hypervisor if we relax the equivalence property requirement of VMMs (Williams, 2016).

Serverless computing is a new computing model, better described as Functions as a Service (FaaS), that allows application developers to deploy functions instead of applications to a managed platform. One approach to serverless computing is to use hypervisors and unikernels for packaging and deploying the functions (Koller and Williams, 2017).

Energy efficiency is another important future direction for hypervisor design. Communications technology, which cloud computing is a large part of, is forecasted to consume around 20% of global electricity by 2030, or as much as 50% in the worst case (Andrae and Edler, 2015)! The energy overhead of a hypervisor can be extremely high depending on workload. One experiment reports between 59% and 273% energy overhead for KVM (Jin et al., 2012)!

Kernel-bypass networking has become important recently because NICs are getting faster and traditional TCP/IP and POSIX socket abstraction is proving to have high overheads (Han et al., 2012; Young et al., 2014; Yasukata et al., 2016). Hypervisors that implement the device model using I/O paravirtualized effectively introduce another layer to the networking data path, which increases networking overheads. In Linux, the vhost architecture is one solution to the problem. Vhost moves the virtio paravirtualized I/O device model from QEMU (which is the VMM userspace kernel) to the host kernel (which also hosts the KVM module), which eliminates the exit from host kernel to userspace VMM. Another solution is full hypervisor kernel-bypass using hardware NIC virtualization introduced by the Arrakis project (Peter et al., 2014).

Summary

The hypervisor architecture invented in the 1970s has stood the test of time. The x86 architecture quirks meant that the first successful hypervisors had to resort into binary translation to handle privileged instructions. Binary translation solutions were followed by paravirtualization (popularized by Xen) but hypervisor architectures were consolidated to the classic model as Intel and AMD added virtualization extensions to the x86 architecture.

Although containers have recently become a very popular virtualization technique, emerging computing paradigms like serverless computing could make hypervisors an attractive technique again. Light-weight hypervisor designs, unikernels, and hardware-assisted virtualization all reduce hypervisor overheads, which also makes hypervisors more competitive against containers.

References

Keith Adams and Ole Agesen. 2006. A comparison of software and hardware techniques for x86 virtualization. In Proceedings of the 12th international conference on Architectural support for programming languages and operating systems (ASPLOS XII). ACM, New York, NY, USA, 2–13. DOI: https://doi.org/10.1145/1168857.1168860

Anders S. G. Andrae and Tomas Edler. 2015. On Global Electricity Usage of Communication Technology: Trends to 2030. In Challenges, 6(1):117–157, 2015. DOI: http://dx.doi.org/10.3390/challe6010117

Edouard Bugnion, Scott Devine, and Mendel Rosenblum. 1997. Disco: running commodity operating systems on scalable multiprocessors. In Proceedings of the sixteenth ACM symposium on Operating systems principles (SOSP ‘97), William M. Waite (Ed.). ACM, New York, NY, USA, 143–156. DOI: http://dx.doi.org/10.1145/268998.266672

Sangjin Han, Scott Marshall, Byung-Gon Chun, and Sylvia Ratnasamy. 2012. MegaPipe: a new programming interface for scalable network I/O. In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation (OSDI’12). USENIX Association, Berkeley, CA, USA, 135–148.

Yichao Jin, Yonggang Wen, and Qinghua Chen. 2012. Energy efficiency and server virtualization in data centers: An empirical investigation. In Proceedings IEEE INFOCOM Workshops, Orlando, FL, 2012, pp. 133–138. DOI: http://dx.doi.org/10.1109/INFCOMW.2012.6193474

Eun Young Jeong, Shinae Woo, Muhammad Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: a highly scalable user-level TCP stack for multicore systems. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI’14). USENIX Association, Berkeley, CA, USA, 489–502.

Ricardo Koller and Dan Williams. 2017. Will Serverless End the Dominance of Linux in the Cloud?. In Proceedings of the 16th Workshop on Hot Topics in Operating Systems (HotOS ‘17). ACM, New York, NY, USA, 169–173. DOI: https://doi.org/10.1145/3102980.3103008

Anil Madhavapeddy, Richard Mortier, Charalampos Rotsos, David Scott, Balraj Singh, Thomas Gazagnaire, Steven Smith, Steven Hand, and Jon Crowcroft. 2013. Unikernels: library operating systems for the cloud. In Proceedings of the eighteenth international conference on Architectural support for programming languages and operating systems (ASPLOS ‘13). ACM, New York, NY, USA, 461–472. DOI=http://dx.doi.org/10.1145/2451116.2451167

Filipe Manco, Costin Lupu, Florian Schmidt, Jose Mendes, Simon Kuenzer, Sumit Sati, Kenichi Yasukata, Costin Raiciu, and Felipe Huici. 2017. My VM is Lighter (and Safer) than your Container. In Proceedings of the 26th Symposium on Operating Systems Principles (SOSP ‘17). ACM, New York, NY, USA, 218–233. DOI: https://doi.org/10.1145/3132747.3132763

Roberto Morabito, Jimmy Kjällman, and Miika Komu. 2015. Hypervisors vs. Lightweight Virtualization: A Performance Comparison. In Proceedings of the 2015 IEEE International Conference on Cloud Engineering (IC2E ‘15). IEEE Computer Society, Washington, DC, USA, 386–393. DOI: http://dx.doi.org/10.1109/IC2E.2015.74

Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe. 2014. Arrakis: the operating system is the control plane. In Proceedings of the 11th USENIX conference on Operating Systems Design and Implementation (OSDI’14). USENIX Association, Berkeley, CA, USA, 1–16.

Gerald J. Popek and Robert P. Goldberg. 1973. Formal requirements for virtualizable third generation architectures. In Proceedings of the fourth ACM symposium on Operating system principles (SOSP ‘73). ACM, New York, NY, USA, 121-. DOI: http://dx.doi.org/10.1145/800009.808061

Dan Williams and Ricardo Koller. 2016. Unikernel monitors: extending minimalism outside of the box. In Proceedings of the 8th USENIX Conference on Hot Topics in Cloud Computing (HotCloud’16). USENIX Association, Berkeley, CA, USA, 71–76.

Kenichi Yasukata, Michio Honda, Douglas Santry, and Lars Eggert. 2016. StackMap: low-latency networking with the OS stack and dedicated NICs. In Proceedings of the 2016 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC ‘16). USENIX Association, Berkeley, CA, USA, 43–56.