SurrealKV, the new key-value storage built by the SurrealDB team, is backed by a powerful data structure that enables it to have several important features:

• Performance
• Low memory usage
• Versioning
• Multiple concurrent readers and writers

This data structure is the Versioned Adaptive Radix Trie, or VART. In this post I’ll explain this data structure by building up to it from a simple concept to the powerhouse that VART is. Afterwards, we’ll look at how this structure is uniquely suited to the way SurrealDB works.

Let’s dive in!

ART — Intuition

The ART data structure, or Adaptive Radix Tree, was first proposed in a 2013 research paper as a potential solution to enhance the efficiency of key-value lookup operations. Traditional key-value data structures, such as hash tables and binary search trees, often face performance bottlenecks when dealing with large datasets or dynamic workloads. ART was designed to address these limitations by leveraging a more adaptive and efficient approach to key storage and retrieval.

The OG Prefix Tree (Trie)

As the name suggests, this data structure is an evolution of a classical prefix tree (a.k.a trie).

A trie is a search tree that is used to store specific keys from a set. A trie is built up such that each node in the tree is a letter, and so words sharing a common prefix will share a root path. The advantage of this structure is that it is relatively efficient to look up a key, a search being only O(m), where m is the length of the key in question.

Let’s look at an example:

Each edge node (or leaf as it’s usually called), will contain a value associated with the key up to that leaf.

So far so good, right?

Radix Trie

One thing that’s clear with the above example is that we can improve it with one some simple modification — combining nodes with only one child with that child. This way we don’t have to traverse multiple nodes where only one hop is needed. This trick hugely reduces both the memory footprint and the lookup time in the suffix tree.

A lookup tree with nodes combined is such a fashion is called a Radix Trie. Applying it to our example:

Much simpler, isn’t it? But can we do better?

Let’s Adapt

One further optimization that we can apply is by using nodes of different sizes.

Think about it — a node has to have an array of pointers that connect it to the child nodes. Even if we only have one letter in each node, then the maximal amount of pointers we’ll need is not small (the number of letters+numbers+symbols), and the more of those pointers that are empty, the more wasted space we have.

Knowing this, the logical leap is to use several sizes nodes (specifically 4, 16, 48, 256), each containing a different number of pointers. This way the percentage of unused pointers is kept relatively low, and both memory usage and lookup time improves.

Our example doesn’t change much, but the smaller boxes indicate the fact that we are now using smaller nodes, using up less space.

But how good is it, really?

This original paper on ART reports some impressive numbers.

Memory usage

Compared to various other tree structures of differing sizes (the s parameter is the number of pointers in a normal radix tree node), ART provides an exponential advantage, as can be seen:

This means that ART is not saving a little bit of space, it’s 1000 times better than the biggest trees!

Runtime

In addition to having a smaller memory footprint, the lookup time also rivals optimized lookup trees (FAST) or hash tables (HT), all while having a much less memory.

A key point here is the dense and sparse modes. ART is more efficient the “closer” the keys are together and therefore the less nodes need to be created. In this benchmark, the authors inserted a range of 16 million keys for the dense example, and chose 16M random keys for the sparse one.

As we’ll immediately see, this is one reason this structure suites Surreal so well.

How SurrealKV uses Versioned ART under the hood

In a previous post, I looked at SurrealKV, the new key-value storage engine that the Surreal team announced. This engine uses Versioned ART, or VART, as the underlying storage engine.

Why is this the appropriate data structure?

In this section I aim to convince you that not only is this the appropriate data structure for SurrealDB’s use-case, it is uniquely suited for it.

In order to do that I’d like to review how SurrealDB stores its data as key-value pairs. Each key is constructed in hierarchical fashion, a record for example will be saved with a key of the format:

* namespace_name
* database_name
* table_name
* object_key

Now let’s imagine how the ART will look. Because of the use of shared prefixes we know that all the records will be children of the same node, something like this:

This in turn means that accessing multiple pieces of data will require less reads, as the relevant node is already accessed!

More than that, since Surreal’s keys are dense (in the sense that they are usually grouped under similar nodes), this takes advantage of ART’s performance benefits as well as it’s reduced memory usage.

Why version?

As I’ve discussed in the SurrealKV post, querying past versions of the data is essential for multiple use cases when using a database, such as preventing data loss, undoing changes, tracking updates and so on.

Surreal enables this by introducting a new VERSION keyword that can be added to queries:

-- Example from Surreal's blog post
SELECT * FROM user VERSION d'2024-08-12T11:03:00Z'
[[{ id: user:john, name: 'John v1' }]]

Behind the scenes this is accomplished using a versioned ART that allows for querying the same key across different versions. They’ve expanded upon the subject in their post.

How do they do it?

Now that we understand the data structure behind the storage engine, expanding it for multiple versions is almost trivial.

Instead of each leaf containing a single value, it will now contain a list of values and the dates they were inserted. In such a manner, accessing the latest version is the default, and the ability to filter for certain insertion times becomes trivial.

Under the hood, there are several more aspects that make this technique efficient.

1. VART utilizes a Copy-on-Write (CoW) technique, which means that it can hold several copies of the entire tree in-memory, with shared pointers to unchanged data (utilizing Rust’s Arc struct, which is an Atomic Reference Counted Pointer, only changed data creates a new element, with all previous pointers using the same underlying data).
2. By creating snapshots of the data while a transaction is running, we can ensure that each transaction works on a seperate copy of the data and does not interfere with other transactions. The transaction that finishes first will successfully commit it’s version to the main tree and transactions that will try to change the same nodes as the finished transaction will see a different version when they try to commit and fail.
3. On the other hand, two transactions that don’t change the same nodes can run and complete concurrently without interfering with each other, making this data structure able to handle multiple transactions at the same time.

Conclusion

In this post we’ve looked in-depth at the VART data structure and how it powers SurrealKV, SurrealDB’s new storage engine. We’ve seen how it improves upon classical search trees and how it is uniquely suited for SurrealDB’s needs.

We now also understand how this allows SurrealKV the ability to “time-travel” by saving multiple version of the values and accessing the correct one.

I hoped you enjoyed this post, see you next time!

Interested in learning more about how SurrealDB works? All my posts are available here.

Have specific questions or requests? Let me know in the comments.

Please clap and follow for more, I’ll be continuing my deep dives into SurrealKV and SurrealDB.

Disclosure as per Medium’s AI policy: in this post I’ve used Microsoft Copilot for editing.

For the past year, I’ve been using Obsidian, the personal knowledge management app, for almost everything I’ve been doing, from task management, through meeting notes and up to planning my Medium posts. In this post, I’d like to share with you the one key insight I understood about working with this wonderful tool.

First, What is Obsidian? Obsidian isn’t just another note-taking app; it’s a digital garden for your ideas, thoughts, and knowledge. Think of it as a personal knowledge base that interlinks your notes, creating a web of information that’s tailored to your mind. Imagine a place where your thoughts can evolve, connect, and grow organically, transforming isolated notes into a cohesive network of ideas. Whether you’re a student, a writer, or just someone who loves to keep your thoughts organized, Obsidian offers a unique approach to managing your knowledge.

In this post I’d like to address a key point that in my opinion isn’t discussed enough — You need to personalize your vault. More than that — You should do it every single day. I’ll show this concept through several personal examples.

Why?

Personalizing your vault is the secret sauce to making Obsidian truly work for you. Your vault should reflect how your mind processes information and how you best work. Customizing the layout, choosing the right plugins, and organizing your notes in a way that feels intuitive can significantly enhance your productivity and creativity.

Personalizing your vault transforms Obsidian from a generic note-taking app into a powerful personal knowledge management system that’s uniquely yours, one that you feel comfortable working with every day. Simply copying templates from blogs or YouTube channels might be a great source of inspiration, but if you don’t modify them to match you how you feel is best, you will feel like there is a high cost to using all of these wonderful tools.

Now, let’s delve into how you can start customizing and get the most out of your digital garden.

How?

Before we begin discussing several aspects of personalizing your vault, you must understand that this is a dynamic and open-ended process. The vault I’m using today is the 200th iteration of the vault I started with one year ago. Every day you should look at how you work, what suits you and what doesn’t, and make small changes to make your work slightly easier, smooth over a pain point, and automate a small corner. Those small changes build up over time and bloom into a beautiful design that fits you personally.

Lines of thought

Here are a few general aspects that make up your vault. In each you have many choices that can shape and change the way you work, and I’ll show a few examples further on in the post.

Start by identifying what kind of structure suits you best. Are you a visual thinker? Utilize the Excalidraw to draw out your notes. Prefer a more linear approach? Keep your folders and tags well-organized. The beauty of Obsidian lies in its flexibility — it can adapt to whatever system you prefer.

Although Obsidian was originally meant to be a text-based editor, there are now many ways to lay out your thoughts more visually and dynamically. The built-in solution by Obsidian is Obsidian Canvas, but the community-driven Excalidraw plugin is a popular alternative. I won’t focus on a specific plugin but on the general ideas. It’s worth noting that I have separate workflows for different tasks, and so my vault contains all of these — text, canvas and Excalidraw.

Next, explore plugins that can add functionality tailored to your needs. There's a vast array of options, from calendar and task management plugins to those that enhance your writing experience. Don’t be afraid to experiment and find what enhances your workflow the most.

Perfect your vault by taking the time to create templates for recurring notes. This can streamline your process, making it easier to maintain consistency and focus.

Lastly, build a connected ecosystem of notes that interact with each other. When your notes interact with one another, you’re no longer dealing with isolated bits of information. Instead, you have a dynamic, interconnected web of knowledge that mirrors the complexity of your thoughts and ideas. This holistic approach to note-taking ensures that your ideas are always accessible, interconnected, and ready to be expanded upon.

My notes have formed complex connections over time, this diagram shows how my templates link to each other. While this seems like a bit of chaos, it’s actually really helpful to be able to jump between all the relevant pieces of information when viewing a note.

Examples

In this section, I’ll demonstrate these three aspects through several personal examples, trying to show how they have built up over time.

Daily Notes

My daily note is the center of my workflow in obsidian. I start each day by opening a new note, and the content is mostly dynamically updated to reflect what happened that day.

Daily notes are the prime example of a template that needs to be personalized. Not just for every person, but for the way you’re currently using Obsidian. As a concrete example, my daily note has changed significantly when I become a team leader.

Here’s how the note structure compares:

All sections are created dynamically with help from plugins such as Dataview, Rollover Daily Todos and QuickAdd.

Dataview

This plugin can query your notes and extract specific information out of them. For example, my meeting summary section is a table that is created by querying all the summaries of the meetings that happened today and are in the meetings folder.

Rollover daily todos

This nifty little plugin does exactly what it sounds like — it takes all the uncompleted tasks from the previous day and pastes them into the current daily note. This way I don’t have to worry about losing tasks from previous days.

QuickAdd

This plugin is all about automating your vault, which I love. It enables you to create macros that run as a command or when pressing a hotkey combination. One of the primary uses I have is for jotting down quick logs of what I’m doing or what I need to remember from anywhere in my vault and having them appear as a list in a section of the daily note I can refer to later.

While using this extensively as a team member, I’ve realized that as a team lead most of what I need is encapsulated in the meeting summeries and I didn’t use this feature any longer, changing my template to reflect that.

Meeting Notes

A meeting is a largely structure affair.

My meeting notes have a few pre-built sections for attendees, points raised during the meeting, tasks, and a summary. Each section was added as I needed it.

My meeting notes started as plain text files with only titles, but over time I added more advanced features:

1. Creating a list of meeting in the daily note, which shows the summary from the metadata
2. Creating pages for people, which show the meetings they attended
3. Pulling all uncompleted tasks from all meetings so as to never miss a task

For each of these, I added a small section, slowly building up the form the meeting note has today. This form has been static for a few months now and I’m quite pleased with it.

Brainstorming \ Planning

For me, brainstorming is a visual exercise. There are usually multiple components and the interaction between them is what sparks new ideas.

I could use a text note and references to brainstorm, and there are times that I found that laying out my thought more linearly makes sense and helps me organize them, but for the most part I find that using a visual tool is much more productive.

You’ve actually seen parts of a brainstorming note already, all the excalidraws were a part of the planning for this post:

My brainstorming notes actually don’t have a template, even though Excalidraw drawings support it. I like the flexibility of the visual method, and the change here happened through using different plugins. I started with Obsidian’s built in canvas, added the X-Mind plugin to it, and only then moved to Excalidraw. I’m still using both, as I have existing brainmaps that I update, but almost all new notes are in Excalidraw.

Summary

Obsidian is a great tool that can be used in almost endless ways.

In the fast-paced world of digital note-taking, continuously evolving your Obsidian vault is the key to unlocking its full potential. This post delves into the importance of regularly updating and customizing your vault, transforming it from a static repository into a dynamic, living system. By constantly refining your vault, you ensure that it grows and adapts alongside your ever-changing needs and ideas. From creating interconnected ecosystems of notes to leveraging plugins for enhanced functionality, we’ve seen practical strategies to keep your vault vibrant and relevant.

I hope that in this post I convinced you that upgrading your obsidian notes a little bit at a time is crucial for maintaining the vault’s value as a productive and helpful tool.

Have ideas how to make the process better? Want to share your thoughts? Leave a comment so that everyone can benefit!

Want to see more Obsidian content? Please Clap and follow.

Not sure what SurrealDB 2.0 is and what the hype is about? Read my introduction here.

Hidden in SurrealDB’s announcement is a gem that can easily be overlooked — the introduction of a new key-value store, built by the Surreal team, called SurrealKV.

In this post I’d like to discuss two burning questions I had when reading the post —

1. Why did they bother building an entire storage engine? (hint — there are a lot of reasons)
2. How does it work?

Why a new engine?

SurrealDB uses a key-value (KV) storage under the hood to store and query all of it’s data, including both metadata (e.g. table definitions) and records (the actual rows in the tables).

For a more in depth look on how they do it, have a look at this post.

Previously, SurrealDB had several options for a KV backend, including RocksDB (for local, persistant storage), TiKV (for distributed clusters) and in-memory (for small and/or temporary instances). These seem to account for all major use-cases, why do we need a new one?

To answer this question we need to look at what a database offers us. On the surface level, a database allows us to simply store and retrieve data, but if we go deeper it can provide so much more. There are several important aspects of a database that are expected of a modern and mature tool:

1. ACID — the ability to perform multiple actions on the database atomically (either all succeed or all fail)
2. Versioning — made a mistake? don’t worry, just see how the data looked in the past. This is a new feature that SurrealKV provides
3. SurrealDB specific improvements — by using a data structure uniquely suitable for how SurrealDB stores it’s values, by using an adaptive radix trie.

And finally, one more to align with SurrealDB’s vision — the entire storage engine can be embedded alongside SurrealDB itself for various application needs.

Versioning

The highlight of the new engine is it’s ability to time travel.

What I mean by that is that it enables querying different versions of the data by specifying the timestamp you want SurrealDB to show you. Querying past versions is extremely useful for many applications: preventing data loss, undoing changes, tracking updates and so on.

Surreal enables this by introducting a new VERSION keyword that can be added to queries:

-- Example from Surreal's blog post
SELECT * FROM user VERSION d'2024-08-12T11:03:00Z'
[[{ id: user:john, name: 'John v1' }]]

Behind the scenes this is accomplished using a versioned trie data structure that allows for querying the same key across different versions. The implementation of this is a whole new project created by the team called vart, which we’ll explain in the next section.

SurrealDB specific improvements

I’ve written in the past about how SurrealDB stores it’s objects, but the gist of the matter is that it always builds it’s keys as a list of hierarchal values. For example, saving a table will generate a key with the structure:

* namespace_name
* database_name
!tb <- A constant value
table_name

And creating an object in that table will have a key of the structure:

* namespace_name
* database_name
* table_name
* object_key

SurrealKV utilizes a prefix-tree (or trie) as the underlying data structure. What this enables is that related objects in the database are “closer” in the stored data. For example, all the objects for the same table have the same prefix (up to the table name), and accessing multiple keys from the same table becomes quicker (you only need to filter for the prefix of the table name once and then you have all the keys).

By building the new store in such a way, it is uniquely suited for SurrealDB’s use case and can unlock better performance for the entire project.

Embedding

Lastly, SurrealDB prides itself on being able to be embedded directly, both into other project, on websites or on-device. Building a storage engine that aligns with that vision, can be deployed anywhere SurrealDB is and doesn’t require any external services is a huge milestone that enables SurrealDB to move forwart with their vision of being the new database for the web.

How is it built?

The second most important questions after why is how.

In future posts we’ll dive into each element in more detail, but for now we’ll look at a brief overview.

The use of SurrealKV is done on three layers:

1. SurrealDB translates all the commands it recieves into key-value format, either for saving or retrieving.
2. SurrealKV recieves those keys and values and makes sure they are saves in an atomic manner and that no two concurrent actions can impact each other.
3. VART is the underlying code that saves the keys as an efficient, versioned Adaptive Radix Trie

All of these layers work together to provide an efficient and reliable database experience.

Radix what?

The underlying data structure, adaptive radix tree (ART), is a variation on a prefix-tree (or trie), a data structure that allows for efficient storage of lookup of keys. This is doubly so for Surreal, as we’ve seen.

What ART improves over the classic trie is in two aspects:

1. Combining nodes if there is only one child. This way we don’t generate a long length of nodes for an extremely unique key
2. Using different sizes of nodes for a different number of child nodes, allowing for optimized storage.

All in all, ART provides a storage solution that is efficient in both storage space and lookup time. It’s a really cool data structure that I’ll probably do an entire post on in the near future.

Summary

SurrealDB 2.0 introduces SurrealKV, a new storage engine for the database, that both improves and enables future enhancements to the product,, while introducing two new features: version queries and an embedded store alongside SurrealDB.

In following posts we’ll be looking more deeply into the technology powering this new engine, I hope you’ll join me on this journey.

Interested in learning more about how SurrealDB works? All my posts are available here.

Have specific questions or requests? Let me know in the comments.

Please clap and follow for more, I’ll be continuing my deep dives into SurrealKV and SurrealDB.

Haven’t heard about Surreal’s new major version? Checkout my overview here.

All my SurrealDB posts are available here.

In the new version, one of the most important, and in fact the first feature on the blog post, is a brand new parser for SurrealQL, the query language for the database. SurrealQL is a SQL-esque query language, with various features that allow for querying the complex and unique structures SurrealDB provides, such as relations and nested fields.

Why the change?

The previous version of Surreal used the nom Rust crate, which allows for building extremely complex parsers by combining smaller functions (for example, a function for identifying the word SELECT and another to identify the name of the columns, and so on). Surreal used this method to parse all SurrealQL queries (for a more in-depth look have a look at my previous post on the subject).

This approach, while useful and efficient on it’s own, had several drawbacks which made is less than ideal for Surreal’s use case, as the team explains in the announcement:

It [the new parser] addresses many of the shortcomings of the previous version, offering better error messages and overcoming the limitations of system stack constraints, ensuring that performance remains consistent even as query complexity increases … the new parser is an optimised recursive descent parser with a separate lexing step.

Whoa, that’s a lot. Let’s break this down.

The new parser has several important components we’ll dive into:

1. Seperating the lexing and parsing steps
2. Custom error handling
3. Memory management for complex queries

The way they managed to do all this is by rebuilding the parser from scratch.

A Guiding Example

Just to have a taste of what this looks like, let’s have a look at a code example (don’t know rust? Don’t worry, everything is explained). I’m using the same example as the previous post to help comparing the two — the DEFINE TABLE statement, the documentation for which is:

Now let’s look at the relevant code (don’t worry, no need to know Rust to understand this in a high level):

Looking at the code, it’s a pretty straightforward translation to the docs, and the code structure makes it pretty easy to navigate and understand, with each subsection being it’s own separate function. For example, the PERMISSIONS option has a separate function that is called:

 t!("PERMISSIONS") => {
    self.pop_peek();
    res.permissions = ctx.run(|ctx| self.parse_permission(ctx, false)).await?;
  }

For those interested in diving deeper, the code for parsing permissions is here.

Now let’s use the example to understand the changes in this version.

Seperate lexing and parsing

Almost all common compilers and interpreters nowadays divide the process into two distinct steps:

• Lexing: seperating the text into tokens (usually be whitespace), each token having a meaning in the language.
• Parsing: given a stream of tokens, convert them into an internal representation of the statement that can be acted upon

Running the lexing step before the actual parsing optimized the parser itself as it doesn’t need to consider irrelevant elements of text (such as whitespaces, linebreaks, indentations etc.) and can focus on the translation itself.

In our example, the result of separating the lexing is that we don’t need to think about separating the text, we can just ask for tokens, for example:

 let name = self.next_token_value()?;

Custom Error handling

Nom has an adaptive error handling system that can be used in various cases, but in this case the team probably wanted either more control, an easier way to return human-readable errors, or both.

The solution is using a custom errors.

In the example above we abbreviated the error cases, but the function has several places where they are used, for example when parsing the table type:

The unexpected! macro is a custom macro by SurrealDB that can output the kind of error message we all know and love, such as:

"Unexpected token `DUMMY`, expected `NORMAL`, `RELATION`, or `ANY`"

If you want to dive deeper, the unexpected! macro itself is defined here.

The way Surreal utilizes the powerful Rust macros to create a powerful error system while maintaining readably and easy-to-write code is interesting in and of itself, I might have a post on that in the future.

Memory management and stack depth

The last part of the new parser is the fact that it can better handle more complex queries.

The previous parser needed to build all possible combinations of the query before parsing, which incurred a heavy cost as queries got more and more complex. This new approach, on the other hand, builds only the necessary parts of the parser that are needed, and discards the parts of the parser that were already run on this query, the result of which is that only the current section being parsed in held in memory, so that the memory footprint remains more or less constant.

The downside of running extremly deep recursive functions is that the function stack calls (on the stack of the program) can cause the dreaded stack overflow error (the namesake of the famous site). In order to address this issue, they use a rust crate called reblessive, which creates a dynamic stack on the application heap and calles each recursive call in a seperate future.

We see this in the premissions example:

// ctx in the reblessive stack, which runs the recursive function
// as a future that will resolve without overflowing the stack.

res.permissions = ctx.run(|ctx| self.parse_permission(ctx, false)).await?;

Using this technique, Surreal minimizes the memory and stack footprint and allows for more complex queries and further flexability in the future.

Recap

In this post, we briefly looked at the elements that make up the new SurrealQL parser, from the new syntax to the way it better handles memory management.

This was by no means a comprehensive view, if you’re interested, I encourage you to look at the source code, it is very well organized and easy to navigate.

Further reading

External crates

1. Nom — the parser library used in the previous versions
2. Reblessive — A heap allocated runtime for deeply recursive algorithms.

SurrealDB

1. parse_define_table
2. unexpected! macro

Rust

1. Macros
2. Async/Await

Interested in more SurrealDB deep dives? See all my posts here.

Have specific questions or requests? Let me know in the comments.

Please like and follow for more, I’ll be continuing my deep dives into the new features in version 2.0.

Not sure what SurrealDB is and what the hype is about? Read my introduction here.

Last week the SurrealDB team announced version 2.0, with major changes encompassing internal changes such as a new parser and a caching layer, as well as exposing new syntax and an experimental GraphQL API. For a full review written by the team you can read their awesome post here.

This version has many amazing features, and instead of running down the list of features like they did, I would like to discuss how the features show how the project is becoming more mature and why I think this marks the beginning of the tool becoming more generally accepted.

I will categorize the features into two groups:

• Changing Surreal to use more industry standards
• Addressing the community’s major pain points of performance and snapshots

Welcoming existing standards

For a long time, SurrealDB was the king of inventing new concepts. This was not like any other database anyone has ever seen. This had both positive and negative implications. On one hand — people were excited to be able to model more complex data in a simpler way with powerful querying syntax. On the other hand, however, the learning curve for Surreal was quite steep. You had to learn both the data model and the query language before being able to work properly with the DB, which put off many.

In this release, Surreal’s direction moves more into industry standards that all of us working with data have come to both accept and love. In my opinion, the feature that shows this most clearly is the GraphQL API, but it also shines through many of the smaller features as well.

GraphQL

Since it’s invention, GraphQL has quickly become one of the world’s favorite API standards, one of the main reasons being the fact that it moves the power into the hands of the data consumers (be it the frontend site or different services) by enabling each consumer to request exactly the data it needs.

This reason is also why this is such a good fit for SurrealDB — which aims to be an all-in-one backend solution. By welcoming GraphQL with open arms, Surreal leaps forward in several key aspects:

1. The learning curve dramatically decreases: GraphQL’s language is more generally known and is less complex than SurrealQL’s more niche features.
2. SurrealDB can now be a drop-in replacement! I can’t understate the importance of this. The hardest part of using a new technology is that fact that our application usually have existing code to request data. For many, this is GraphQL, which means adopting SurrealDB into new projects has never been easier!

Other Features

Many other features also nod to the fact that Surreal is accepting the standard terminology, making it easier for experienced data practitioners to use the project. I’ll quickly go through them:

1. Define access moves to a more accepted access terms instead of Surreal’s tokens. The new access statement also accepts known methods such as JWT.
2. Nested field definition as literals. Experience teaches us that the more strict the data validation, the less bugs we’ll have. Enabling people to define complex types while maintaining strong typing makes the usage much more similar to any type-safe language or DB.
3. Making UPSERT explicit. While in previous version UPDATE could also insert new data, in this version the UPDATE statement acts like it does in any other database and in order to create or update we need to explicitly declare we wish to do it. This will reduce bugs for people who come from other database solutions by a significant margin.

All of these features show a direction, in my opinion, that will make SurrealDB a more widespread and accepted solution.

User pain points

The second important aspect of this version is that it solves many of the major pain points users had head on.

Performance

One of the most common complaints about Surreal is that it sacrificed speed for flexibility, with user complaining it is slower that classical DBs such as Postgres. While not publicly available yet (they are now working on making them stable), many features in this release aim to make SurrealDB faster and more reliable. Let’s look at some of them:

1. New parser: I’ve previously covered the way SurrealQL has been parsed in the previous version. The parsing has been rebuilt from the ground up to be more efficient and more flexible. I’ll write a more in-depth post on the subject in the near future.
2. A new caching layer for faster queries without multiple round-trips to the underlying storage.
3. Better Indices, that now have better algorithms, build asynchronously and can be rebuilt, all of which allows for faster queries on more data.

Snapshots

Ever accidentally deleted important data? Updated a field you shouldn’t have? Many databases have a concept of versioning, which allows querying previous iterations of the data. SurrealDB now joins the club by introducing their own storage solution — SurrealKV.

SurralKV is a replacement for the previous storage layers (RocksDB, TiKV, etc.) and allows for snapshot queries, which means you can see how the data looked on a specific time.

Conclusion

I see this version as a first huge step in a good direction.

This version showed how Surreal can balance both it’s own identity and the world’s best practices. I believe in the coming months and years we’ll see the product mature and become a product we can all look forward to using.

Interested in more SurrealDB deep dives? See all my posts here.

Please like and follow for more, I’ll be continuing my deep dives into the new features in version 2.0.

For the last two years, I’ve been the team lead for a team in a large organization. During this time, I’d devoted a considerable amount of time to building a framework of principles that guide me in my job as both a manager, a friend and a member of the organization who is committed to it’s continued growth and success.

This article will focus on three points that I believe are the most relevant and that I wish someone had discussed with me when I entered my position. The article is meant not only for team leads, but also towards anyone who would like to take a step back and consider how, perhaps, we can create a work environment that is both better to work in and is more productive.

I am always looking to learn, as so feel free to comment your thoughts on the matter.

Without further ado, let’s get into it.

Ego

Ego, in the sense that I want to talk about, is “an inflated feeling of pride in your superiority to others” (vocabulary.com).

Everyone has some ego, and while it can be a driving force for positive change, it can often be a great hindrance to mutual projects, as people seek to put themselves above the shared goal. As a team lead, one of my main jobs has been to work both with my team members and other coworkers, while carefully balancing the need to provide value with the need to make the workers feel fulfilled, so as to not let everyone’s ego get in our way.

I’ve had projects go either way. I’ve had projects where all participants were entirely on board and completely dedicated to our work, as well as less successful projects where the ego of other teams didn’t let us move forwards in our own projects.

There are a few methods that can improve the situation:

1. Convince the upper management of a shared goal. Higher mangers are a resources I wish I’d employed more earlier. They can negotiate on your behalf, find resources and compromises that I could not, and back you up when you make your case (especially when you have data to show, read on for more on the that subject).
2. Remove toxic team members, either entirely or from a specific project. There are times where one or two people are the catalysts for the friction in the project, and so removing them from the work will do wonders.
3. Invest time into personal meeting, both inside your team and outside of it, so that you can discuss this issue frankly. From my experience, people would like to improve, and constructive feedback on this subject can go a long way

Separation of concerns

Often times, the team lead has two roles, both of which are critical to the team’s success:

1. The Manager — a person responsible for directing the team and organizing both the technical, logistical and administrative work
2. The Tech Lead — the most knowledgeable person on the team that oftentimes also has legacy knowledge about past team activities.

The reason for this is clear and perfectly reasonable. Oftentimes, one of the senior team members is the one promoted to management, and it is only natural to put him or her in charge of the team they know best. Nevertheless, this put the team in it’s entirety in a fragile state that I believe should be avoided and worked upon.

This state stems from the fact that a team cannot depend on a single person, especially the team lead. As a lead, I find that most of my time is spent in meeting, both inside the team and outside of it, and I am therefor not free to help on the technical issues the team faces. This has gotten to a point where when I take a vacation, the team’s ability to solve issues is noticeably degraded.

I have taken steps to remedy this situation, investing considerable time and effort into the process. Here are a few tips how to do so in your own team:

• Let the team make technical decisions. I have met many leads who want to make all decisions, down to the technical details. Having the team make the decisions, on the other hand, both ensures the development of team members, gives them the satisfaction of knowing they are responsible for their entire project, and improve the technical knowledge of the entire team.
• Have a tech lead\architect role in your team, both as a means of promoting a successful team member and ensuring the longevity of the team’s knowledge. The way to create such a role is by having a senior teammate start attending more technical meetings, until they have a broad enough understanding of the team’s projects that they can support the team as a technical knowledge base.
• Give your team the freedom to make mistakes. As a lead, the most important currency I can invest is time (both mine and the team’s), and letting the team know I choose to invest our time in the way they choose and that I have their back if we need to change in the future is the way I found most enhances the teammates’ will to expand their knowledge.
• Give teammates a broad view. In some team, the workers focus entirely on the technical job at hand, while the lead also is responsible for coordinating with other teams, the product owners, the IT devs and so on. I found that letting each team member have a complete, end to end, ownership of their projects (from research to coding to monitoring to deploying) lets them become better at both their job and as teammates.

Start Speaking the Data Language

We’re living in 2023, there is absolutely no reason to keep arguing without data to back up our claims. As a team lead, I found that bringing facts is one of the most reliable ways to convince people, both managers and fellow leads.

Put some time into building data infrastructure in your team, it will benefit you in the long run. Some of the features may include:

1. A data map, such as OpenMetadata, that allows you to clearly define where your data exists, what it contains and how to access it.
2. A dashboard system, such as Redash, that allows your to show any and all interested parties insights regading your team in a user-friendly visual interface. I have also found that it is mostly an investment that is very beneficial in the long run, as it frees the team from constant report generation.
3. Well defined logs and metrics. Having the state of your systems explained well, both their health and the value they bring, is a huge asset.

We team leaders are investors no less than any VC. We invest the time and resources of our team, and we need to do so wisely and carefully.

I hope that I convinced you that you should invest some of your time into making the people under you better and enhancing the data infrastructure, both long term investments that together, I believe, will let us create a better tomorrow.

Hot tip: All stories in this series are here

There is an old programming saying that it is usually better to have simple code that is easily explainable than obscure code that is more performant. This is definitely one of the exceptions.

The only way I know how to describe the way Surreal stores it’s data is as brilliant, simply brilliant.

In order to understand how the data is saved, let’s first look at what needs to be in storage. There are two categories of items to store:

1. Metadata — the structure of tables, indices, scopes, etc.
2. Data — the values of the objects retained by Surreal

As the underlying storage is a key-value store, both are stored in a similar way, but the values stored for each are different.
In order to better demonstrate how Surreal stores data, I built a small application to read the RocksDB file that Surreal stores it’s data in (if the server is started with a file path).

Running the RocksDB Viewer

First, we start both the server and SQL prompt:

surreal start rocksdb://local.db --user root --pass root

surreal sql --conn http://0.0.0.0:8000 --user root --pass root --ns test --db test

Second, we insert some data:

-- create new tables and records
CREATE user:oricc SET name="Ori";
CREATE user:tobie SET name="Tobie";
CREATE repo:surrealdb SET name="SurrealDB";

-- Add some relations
RELATE user:tobie->contributes->repo:surrealdb;
RELATE user:ori->contributes->repo:surrealdb;

then, when we run the app, we get the following results (the left column is the key, the right is the value):

Let’s break down what we see, starting with the metadata.

The first rows describe the objects in the database:

Surreal has several types of objects, here we can see the namespace, database, and tables that are defined in this server.

Now the question becomes what the keys and values represent. In order to understand that, we’ll go back to the code.

Understanding Keys

Surreal has an entire directory (lib/src/key) which deals with creating the keys for the various different objects. For this example, we’ll look at our old friend, the table.

The struct defining the table key has the following structure:

At first, this struct looks very odd. What are all those variables with the unconventional names? what are we representing here? Look closer, does it look familiar? Let me give you another hint.

When creating the key it’s values are initiated as follows:

Yes! This struct is directly mapped to the key for the tables that we can see in the image above!

How the struct is turned into the key is another interesting point. Notice in the struct definition, the struct macro-derives the Key macro, which automatically creates the serialization and deserialization methods for the struct using the msgpack format. MessagePack is a protocol for efficiently compressing data into a minimalist format, Surreal uses the serde binding for the protocol (rmp_serde) so that structs can be easily mapped to and from the serialized data. For those interested in the macro itself, you can have a look at the code.

Understanding Values

The second element of the metadata is the values.

In order to understand what the value of the metadata stores we need to look back to the previous post and review the table definition statement:

Now compare the struct to the value for the “contributes” table:

��contributes�����None�None�None�None

This is an exact map of the table definition, again using the Store macro to create the MessagePack methods on the struct that translate it to the value we see in the RocksDB.

What this allows us to do is to always have the entire definition for each of the objects in the database, and act accordingly.

Item storage

For any non-metadata items we need to save in the key-value store, we can easily see that they are very similar to the metadata:

Each item is stored with the entire hierarchy (Namespace -> Database -> table -> ID) and the value is the MessagePacked object itself (notice the name and Ori inside the value).

That’s it for storing the values themselves.

The next question is why the keys have this format.

Breaking down the key structure

The structure of the keys is actually the reason this storage design is so brilliant. In order to understand why, we first need to understand how access to key-value databases is done. There are two methods of accessing a KV store.

1. Key-based — we look up a specific key and get the value for it
2. Scan — we look up a range of keys and retrieve all the values.

By design, key-based lookups are extremely efficient, while scans are a little slower. For that reason, we want to have the range we are scanning be as small as possible.

The way Surreal achieves this is by building the keys in a hierarchy that allows us to translate each key type into a key range in the underlying KV storage.

Let’s go back to our trusty example — the table.

As we saw, the table name is a concatenation of the namespace, database and table name, in this order. Therefor, if we want to find all tables in the test namespace and test database, we would look for all keys of the form /*test*test!tb[table_name] . This can be translated into a range by looking for any character where the table name should be. In technical terms this means looking for all hex values from 0x00 to 0xff.

This is exactly what Surreal does!

Each item (table, fields, etc.) has two functions called prefix and suffix that allow Surreal to map the ranges for this item.

These functions are used to iterate the range and find all relevant keys (for example, where querying the info for a database).

This same method is used to query all records of a table (because the record keys can be iterated given the table name), and is the base that makes Surreal so fast!

Conclusion

Whoa! We looked at a lot of details under-the-hood today.

Let’s recap:

1. Surreal stores all data (both the database structure and the data itself) in a Key-Value store
2. The keys are built using a concatenation of all the previous database layers (for example, table keys include the database and namespace)
3. This structure allows Surreal to iterate over all the items inside an object (e.g. tables in a database or rows in a table)
4. The values are a serialized using MessagePack in order to save storage.

This is the heart and soul of Surreal, and is the reason Surreal is both so flexible and so performant, both of which make SurrealDB such an innovative project and a database for the future.

Want to read more? All SurrealDB stories are available here.

Please clap if you liked the content and follow for more content. Let me know in the comments what else you would like a deep dive into.

See you next time!

A full list of posts in the series can be found here.

In the previous post we looked at an overview of SurrealDB and it’s architecture. In this post we will start looking under the hood of SurrealDB — with the code responsible for parsing the language Surreal speaks, which is commonly referred to as SurQL (Pronounced as circle).

SurQL

The language itself is similar to classical SQL in structure, but it has a few unique features to allow for SurrealDB’s full abilities.

The way Surreal parses the queries passed to it is by using the nom crate. Nom is a parsing library that allows for building parsers incrementally by chaining functions, while also using Rust’s strong type system to maintain the memory and type safety Rust guarantees as part of it’s programming model.

Basically, you can write functions to parse small pieces of text that return a Rust type, and compose them in larger and larger functions that eventually can parse your entire language, much like building up with Lego pieces into a larger and larger build.

Let’s look at an example from inside Surreal itself — the table definition.

As with any Rust code, the data type is the language we build our code around. Let’s start by looking at the table definition type.

Just by looking at the type we can understand a lot about the code that we need to write. We will need internal functions to parse sub-types (Ident, View and Permissions). As well as seeing the view definition is optional, and we will need to handle the cases where it does and doesn't exist separately.

For reference, the official docs give us the full syntax (abbreviated here to only show the table definition):

We can see that the type and syntax overlap exactly, but to me the type is easier to understand as it destructures the statement into it’s components in a clear and concise way.

First, let’s try and describe the statement as a tree of statements. The beauty of nom is that it allows us to write each leaf in the tree as a separate parser and then combine them all into a large parser.

The translation into code is straightforwards:

The function itself is a bit long, but it shows us exactly how awesome Nom is.

Let’s start from the top:

fn table(i: &str) -> IResult<&str, DefineTableStatement> 

Each parser in Nom takes the string to parse as an input, and returns a special type of Result (as the parsing can fail). When the parsing succeeds, the parser returns a tuple — both the remaining text to process (and that text is passed on to additional functions) and the parsed Rust type that is created from this parser, in our case the table definition.

The parsing itself is done in only 6 lines of this function (lines 4–9), which also demonstrate how to use the result of a parser, for example (line 8):

let (i, name) = ident(i)?;

• i — the remaining text
• name — the object of type Ident that the parser returned

Reading the parsing lines we can easily understand the allowed syntax for this command.

Lines 12–43 show us the extreme usefulness of the fact that Nom returns rust types. In those lines we are using idiomatic Rust, and are guaranteed all the memory safety as well as correctness promises Rust gives.

In order to gain a better understanding of the power of composite parsers, let’s dive even deeper into the table options parser.

Again, the options themselves are guaranteed to be a certain set of commands via a use of a Rust type:

The really interesting part comes at the parsing stage. As we can have multiple options that come in different orders, we need to check for alternatives, each alternative a separate parser. Luckily, nom makes this almost trivial.

First we define a series of parsers, one for each option (notice that all return a variant of the enum we defined above):

And then we simply wrap them is a general parser for all options that can return any of the alternatives, using the build in alt parser nom provides, which does exactly what we want:

For those interested, a good exercise is to look even further down at the view parser and understands how it fits in to the general table definition parser.

But wait, we are not done. How do we get all the options for the table? Nom again comes to the rescue!

Look again at the definition parser:

let (i, opts) = many0(table_opts)(i)?;

We used another nom parser called many0, which allows us to iterate and return a list (of type Vec<DefineTableOption>) of all the matching patterns in a sequence until the match fails, allowing for the option that there are no matches (the complementary many1 method requires at least one match or the entire parser will fail).

All parsers in the project follow the same pattern, combining smaller parsers into increasingly large ones that can deal with the large and complex statements that make up SurQL.

Conclusion

In this post we looked at how SurrealDB uses Nom to parse SurQL, it’s SQL-like query language, using the example of the table definition to study the amazing power of combination parsers to create complex ones with relative ease.

Now that you understand one, understanding all other parsers in SurrealDB should not be a problem. In addition, here are a few tips to not get lost in the depth of the parsers:

1. Start with the structs, they will tell you what should be returned as the entire statement and allow you to get a overhead view of the parser.
2. Open the docs, if they exist, to see how the parser translates into the full query.
3. don’t worry if you don’t understand a parser all the way down, sometimes knowing the type returned from some sub-parser (for example, the view parser above) is enough.

In the next post we’ll talk about the core of SurrealDB and one of the most brilliant and exciting parts of it — the data model in the Key-Value stores backing the database.

See you then!

Additional Resources

• A great explanation about nom parsers: https://naiveai.hashnode.dev/practical-parsing-nom
• Nom repo: https://github.com/Geal/nom
• SurQL in the official docs: https://surrealdb.com/docs/surrealql/statements

A full list of posts in the series can be found here.

In the previous post we talked about the features of SurrealDB, in this post we’ll start looking at the code itself from the top down.

We’ll start with the general architecture design and then look at the file structure of the project.

Architecture

SurrealDB is built with one of the most time-tested and well-known architectures for any project and for servers in particular — the 2-tier architecture.

2-Tier Architecture

This architecture is based on the idea that we need to separate our code into two logical parts (called layers):

1. API (Application Programming Interface)— the client facing part of the application
2. DAL (Data Access Layer)— the part of the code that handles our custom code and talks to external systems

What separates these two layers is the language they speak. What I mean by this is that each layer uses different data models:

1. The API talks in the language of the users, and translates it to the internal models
2. The Business Logic (BL) translates our models and communicates with external services using their language, returning the results back to the API layers using the internal models

Each tier only calls the tier below it and doesn’t know about the other tiers.

SurrealDB Architecture and File Structure

SurrealDB’s implementation of the 2-tier architecture is shown in it’s file structure.

|-- src            // API layer
    |-- net       
    |-- cli
    |-- rpc
    ...

|-- lib/src       // BL
   |-- sql 
   |-- fnc
   ...

   |-- kvs  
      |-- indexdb
      |-- rocksdb
      |-- tikv
      ...
   

The src folder includes all the API code, each subfolder representing a different way to access the DB (REST, websockets, CLI, etc…)

The lib folder includes both the BL.

• The kvs subfolder includes all the different storage options for the data (memory, TiKV, RocksDB, etc…)
• The rest of the subfolders are the business logic, the custom code the makes SurrealDB function

Knowing the file structure of the project and understanding how the different parts talk to each other is always the best way to getting started on a new project, as it allows us to start jumping around the code base without getting lost.

Code Examples

Let’s look at some examples from the code itself.

API Layer

As we said previously, the API is responsible for communicating with the clients, and allowing the internal code to run without knowing the origin of the request, allowing it to be easily switched out. This is the reason SurrealDB can be used in several modes, including both as a server and as an embedded database.

The heart of this function is line 15, which calls the method of the BL that runs the operations requested by the user. The rest of the code wraps around this line and translates the user language into the server’s language and back into the user response format.

Business and Data Layer

The data layer both parses runs the application code and accesses the underlying data itself.

In SurrealDB, each file in lib is responsible for both the parsing and computing of a specific functionality. For example, the select statement has it’s own file, all the code related to the geometry is in another, etc.

Let’s look for example at defining a database.

We will have a more in-depth look at the query language itself in the next post, but let’s look at the example in the context of the layer architecture.

The code to handle the database definition is divided into four parts:

1. Model — The internal model that the database knows how to handle
2. Parsing — The method that translates the user’s language (SurQL) into the internal model
3. Display Format — For displaying the internal model in the user’s language
4. Compute — Finally, the method to execute the code represented in the internal model

As you can see, these functions can be seen as translations between the different languages and between the different layers.

Now that we understand the high-level architecture, we can look at the code itself.

Conclusion

See, not so bad after all, if we just know how each piece fits into the larger architecture.

Now that we understand the high-level overview of the code, in the following posts we will dive into one subject at a time and look at the internal implementation.

Next up — understanding how Surreal parses the query language (we’ve seen a small example above).

See you there!

A full list of posts in the series can be found here.

One of the most hyped and anticipated Database technologies that has arisen recently is a product created by two individuals and that promises to be the DB for the future — that is none other than Surrealdb.

The code was open sourced a few months ago, and I’m proud to say I contributed a feature to this amazing project. What I’ve discovered while writing code in this ecosystem, is that the internal design of Surrealdb is not documented, and so I had to figure out where anything goes by debugging and clicking references in VSCode, not an ideal situation by any measure.

As an attempt to make the code more accessible to the community as a whole, and to allow more people to contribute, in this series of posts I will do my best to break the code design, focusing at one part at a time.

This post is an overview of Surrealdb, on the next posts we will dive into the code itself.

What is Surrealdb?

According to the documentation:

SurrealDB is an innovative NewSQL cloud database, suitable for serverless applications, jamstack applications, single-page applications, and traditional applications.

Let’s break that down.

NewSQL — a database that offers a NoSQL experience (with no structured data constraints), whole also maintaining the ACID guarantees (transactional execution, data consistency and rollback capability).

Serverless Cloud — surrealdb can be deployed on a distributed architecture and can run on all major providers, enabling the database to scale as needed.

Application Backend — surrealdb has additional features that allow it to be used directly from a customer facing client as well as from a custom server. These features include:

• An advanced premissions model
• Live updates via Websockets
• A REST API
• And more…

SurrealDB data model

Sound awesome! What can you store in this DB?

Basically — Anything.

SurrealDB supports three types of data paradigms, all working together to create a modern data model:

1. Unstructured data — Arbitrary objects with no defined structure, including nested references to other objects in the database
2. Tabular (structured) data — like classical SQL databases, storing rows with a defined and enforced structure
3. Graph connections — arbitrary connections between objects using a graph syntax, which allows the connection itself to have properties that can be queried

This means that we can save whatever data we want, decide if we want it to have a structure, and connect different objects into a graph that we can query.

All data is stored in groups that for simplicity’s sake are called “tables”, even when the data is unstructured.

Example Time

All operations on the database are pretty much intuitive for anyone who has dealt with any kind of database before.

(The example is based on the official documentation):

Working with unstructured data:

UPDATE person SET
	waist = <int> "34.59",
	height = <float> 201,
	score = <decimal> 0.3 + 0.3 + 0.3 + 0.1
;

Or with adding structure:

-- Create a schemafull table
DEFINE TABLE user SCHEMAFULL;

-- Specify fields on the user table
DEFINE FIELD name ON TABLE user TYPE object;
DEFINE FIELD name.first ON TABLE user TYPE string;
DEFINE FIELD name.last ON TABLE user TYPE string;
DEFINE FIELD email ON TABLE user TYPE string ASSERT is::email($value);

-- Add a unique index on the email field preventing duplicate values
DEFINE INDEX email ON TABLE user COLUMNS email UNIQUE;

On top of this data we can build our graph:

-- Add a graph edge between user:tobie and article:surreal
RELATE user:ori->write->article:welcome-to-wonderland
	SET time.written = time::now()
;

-- Add a graph edge between specific users and developers
LET $from = (SELECT users FROM site:medium);
LET $devs = (SELECT * FROM user WHERE tags CONTAINS 'surrealdb');
RELATE $from->like->$devs UNIQUE
	SET time.connected = time::now()
;

And all of this can be done via a REST API, custom client libraries in various languages, or websockets (see the docs).

Conclusion

I’ve only touched the surface of the functionality of this amazing tool, it can do so much more.

I encourage you to read the docs, or watch this short Youtube video if you only want a taste:

Future posts will dive into the code base itself (which is open source and can be found here) and will be focused on the community, with an aim of enabling more people to get their hands dirty and contribute.

Follow to get notified when the posts go live!