Rust Beyond 101
Rust is an awesome language that provides some great abstractions and an awesome compiler to help you build safe code.
This tutorial guides you through a practical example of building an API for parsing and validating medium volumes of json data. It takes a practical approach to allow you to build and extend the APIs yourself firm in the knowledge that you are building performant APIs addressing the scale you require.
The tutorial will demonstrate the following:
- Initial Project
- Add CLI
- Add Config Loading via YAML
- JSON Generation
- Defining our Error Handling
- Add JSON Schema Generation
- JSON Validation
- Create a Webservice
- Dockerise the Service
- Benchmarking
- Documentation
- Testing
Install Rust
Follow the standard instructions here: https://www.rust-lang.org/learn/get-started
# curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# cargo --version
cargo 1.70.0-nightly (15d090969 2023-03-21)After this installs you should have a working cargo CLI.
Initial Project
Now lets initialise the project. Note: tree is not normally installed in linux systems so you may need to install it via your package manager (eg brew).
# cargo init largejson
Created binary (application) package
# cd largejson
# tree .
.
├── Cargo.toml
└── src
└── main.rs
2 directories, 2 filesThis creates two files Cargo.toml is the definition of our dependencies and build. src/main.rs is our initial source code.
Our initial application is super simple hello world.
fn main() {
println!("Hello, world!");
}And our Cargo.toml is simple no dependencies definition.
[package]
name = "largejson"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]The toml file defines the name of the package, the version and the rust edition we need to use.
Lets build it and try it out.
# cargo build
Compiling largejson v0.1.0 (...largejson)
Finished dev [unoptimized + debuginfo] target(s) in 0.33s
# tree target
target
├── CACHEDIR.TAG
└── debug
├── build
├── deps
│ ├── largejson-76dec941f82a2ca6
│ ├── largejson-76dec941f82a2ca6.10idt00h1jkemqpk.rcgu.o
│ ├── largejson-76dec941f82a2ca6.1m0htgtn2l6ludtj.rcgu.o
│ ├── largejson-76dec941f82a2ca6.2ukio9dei4i8qlsv.rcgu.o
│ ├── largejson-76dec941f82a2ca6.3a8vu3im9ngri3pk.rcgu.o
│ ├── largejson-76dec941f82a2ca6.3alb8s9x9k77b7tx.rcgu.o
│ ├── largejson-76dec941f82a2ca6.3ed4kjyz3clop3pr.rcgu.o
│ └── largejson-76dec941f82a2ca6.d
├── examples
├── incremental
│ └── largejson-2p62db7upgl2q
│ ├── s-gp3go42z7y-1d1zqba-3df6bcx9ax93i
│ │ ├── 10idt00h1jkemqpk.o
│ │ ├── 1m0htgtn2l6ludtj.o
│ │ ├── 2ukio9dei4i8qlsv.o
│ │ ├── 3a8vu3im9ngri3pk.o
│ │ ├── 3alb8s9x9k77b7tx.o
│ │ ├── 3ed4kjyz3clop3pr.o
│ │ ├── dep-graph.bin
│ │ ├── query-cache.bin
│ │ └── work-products.bin
│ └── s-gp3go42z7y-1d1zqba.lock
├── largejson
└── largejson.d
8 directories, 21 filesWe can see the build artefacts are created in the target/debug directory. And in particular we can se the resultant binary command.
# ./target/debug/largejson
Hello, world!
# ls -lh target/debug/largejson
-rwxr-xr-x 1 bengreene staff 412K 26 Sep 21:00 target/debug/largejsonAnd we can see the file is “quite” small at 412K. We can also build a release version of this file which will make the compiler work harder and turn on some more optimisations. However on such a simple program release does very little.
# cargo build --release
Compiling largejson v0.1.0 (...largejson)
Finished release [optimized] target(s) in 0.45s
# ls -lh target/release/largejson
-rwxr-xr-x 1 bengreene staff 409K 26 Sep 21:06 target/release/largejsonYou can compare with Github here:
Add CLI
The library we are going to use to build our CLI interface is called Clap. It is a derivative of Click (python) and Clikt (Kotlin).
Now we use cargo to add the clap dependency with the derive dependency.
# cargo add clap@~4.3 --features derive
Updating crates.io index
Adding clap ~4.3 to dependencies.
Features as of v4.3.0:
+ color
+ derive
+ error-context
+ help
+ std
+ suggestions
+ usage
- cargo
- debug
- deprecated
- env
- string
- unicode
- unstable-doc
- unstable-styles
- unstable-v5
- wrap_help
Updating crates.io indexAnd we can observe the update to the Cargo.toml file where it show the clap dependency has been added.
When we installed clap we used a tilde dependency. This installs version 4.3 and the upward versions in the 4.x.x range.
clap = { version = "~4.3", features = ["derive"] }Now we update our main application to include clap parsing.
use std::path::PathBuf;
use clap::{Parser, Subcommand};
/// Application definition to defer to set of commands under [Commands]
#[derive(Parser)]
#[command(author, version, about, long_about = None)]
struct Args {
#[command(subcommand)]
command: Commands,
}
/// Commands to run inside this program
#[derive(Debug, Subcommand)]
enum Commands {
/// Generate messages
Generate {
/// filename to write content to
filename: String,
/// Number of records
#[arg(long, default_value_t = 1)]
count: u32,
},
/// Show schema for object
Schema,
/// Show schema for Vec of object
SchemaList,
/// Validate file against schema
Validate {
/// filename to read content from
filename: String,
},
/// Receive json file via http
Receive {
/// Sets a custom config file
#[arg(short, long, value_name = "FILE")]
config: PathBuf,
},
}
fn main() {
let args = Args::parse();
match args.command {
Commands::Generate { filename, count } => todo!(),
Commands::Schema => todo!(),
Commands::SchemaList => todo!(),
Commands::Validate { filename } => todo!(),
Commands::Receive { config } => todo!(),
};
}Now lets build and run this (there is a single command to do that).
# cargo run
warning: unused variable: `filename`
--> src/main.rs:50:30
|
50 | Commands::Generate { filename, count } => todo!(),
| ^^^^^^^^ help: try ignoring the field: `filename: _`
|
= note: `#[warn(unused_variables)]` on by default
warning: unused variable: `count`
--> src/main.rs:50:40
|
50 | Commands::Generate { filename, count } => todo!(),
| ^^^^^ help: try ignoring the field: `count: _`
warning: unused variable: `filename`
--> src/main.rs:53:30
|
53 | Commands::Validate { filename } => todo!(),
| ^^^^^^^^ help: try ignoring the field: `filename: _`
warning: unused variable: `config`
--> src/main.rs:54:29
|
54 | Commands::Receive { config } => todo!(),
| ^^^^^^ help: try ignoring the field: `config: _`
warning: `largejson` (bin "largejson") generated 4 warnings (run `cargo fix --bin "largejson"` to apply 4 suggestions)
Finished dev [unoptimized + debuginfo] target(s) in 0.07s
Running `target/debug/largejson`
Application definition to defer to set of commands under [Commands]
Usage: largejson <COMMAND>
Commands:
generate Generate messages
schema Show schema for object
schema-list Show schema for Vec of object
validate Validate file against schema
receive Receive json file via http
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print versionThere were warnings this time (we will ignore them for the moment). However the application builds and runs and presents us with some nice CLI parsing and documentation.
We can then try running one of the subcommands.
# cargo run schema
...
Finished dev [unoptimized + debuginfo] target(s) in 0.02s
Running `target/debug/largejson schema`
thread 'main' panicked at 'not yet implemented', src/main.rs:51:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceUpdating git:
Add Config Loading via YAML
We will now add the functions to load config from a Yaml file. The library ot use for this is called Figment. It provides similar functions to Pydantic or Hoplite.
This time we can going to install it by adding it directly to the Cargo.toml file. Lets these lines to the end of our dependencies. In addition to Figment we are going to add log and serde.
figment = { version = "0.10.10", features = ["yaml", "env"] }
serde = { version ="~1.0", features = ["derive"] }
log = "~0.4"Now we need to udpate our arg execution path for Command::Receive
Commands::Receive{ config } => {
info!("Starting {NAME} for {VERSION}");
let config: MyConfig = MyConfig::figment(config)
.extract()
.expect("Config file loaded");
info!("Loaded config {:?}", config);
println!("Loaded config as {:#?}", config);
},However if you just add this you will see it erroring out. Now lets start watching our source code and rebuilding it as we update the application. Before we can do that we need to install cargo watch.
# cargo install cargo-watch
Updating crates.io index
...
Replacing .../.cargo/bin/cargo-watch
Replaced package `cargo-watch v8.1.1` with `cargo-watch v8.4.1` (executable `cargo-watch`)
# cargo watchWe add a new file to our src called src/webservice.rs. This will eventually describe our webservice, but initially it will describe our configurations.
use std::path::Path;
use figment::{Figment, providers::{Yaml, Format}};
use serde::Deserialize;
#[derive(Deserialize, Debug, Clone)]
pub struct WebServicePrefixConfig {
pub name: String,
pub version: String,
}
#[derive(Deserialize, Debug, Clone)]
pub struct WebServiceConfig {
/// Prefix of the served API
pub prefix: WebServicePrefixConfig,
}
#[derive(Deserialize, Debug)]
pub struct MyConfig {
/// Config of my web service
pub webservice: WebServiceConfig,
}
impl MyConfig {
// Note the `nested` option on both `file` providers. This makes each
// top-level dictionary act as a profile.
pub fn figment<P: AsRef<Path>>(path: P) -> Figment {
Figment::new().merge(Yaml::file(path))
}
}From just adding this file it will not be included in the build unless it is pulled in as a module. The best way to do this is via adding a src/lib.rs file. This file describes a library (internal) for the largejson package and is a conventient and clean way to define our functionality. Technically we do not need to use src/lib.rs as we can map all modules directly into src/main.rs but some of the benchmarking and test tools we will use later cannot access functions inside our applications but can access them inside our libraries so this becomes good practice to map our functions to src/lib.rs. A final benefit of this approach is that it makes it easy to build multiple CLI applications with shared functions. Thus we create src/lib.rs as follows.
pub mod webservice;
/// Name of the Crate
pub const NAME: &str = env!("CARGO_PKG_NAME");
/// Version of the Crate
pub const VERSION: &str = env!("CARGO_PKG_VERSION");We have also added two const definitions for NAME and VERSION to capture env vars from the compile process. We grab some env vars defined by the rust compiler: CARGO_PKG_NAME and CARGO_PKG_VERSION.
Finally add some additional dependencies to the top of our main.rs file.
use std::path::PathBuf;
use clap::{Parser, Subcommand};
use largejson::webservice::MyConfig;
use log::info;
use largejson::{NAME, VERSION};We should now be able to compile our application again and run it. Before we do so lets create a simple config file myconfig.yaml
webservice:
prefix:
name: test
version: v0Lets run it with the watch but this time we also run the resulting command and provide it with arguments.
# cargo watch -x "run receive --config myconfig.yaml"
...
Finished dev [unoptimized + debuginfo] target(s) in 0.63s
Running `target/debug/largejson receive --config myconfig.yaml`
Loaded config as MyConfig {
webservice: WebServiceConfig {
prefix: WebServicePrefixConfig {
name: "test",
version: "v0",
},
},
}
[Finished running. Exit status: 0]We can see that the configuration has been pretty printed (ie the :#?) out from the println! macro but not the info! macro.
We may have the log macros in place but we do not have a logging framework in place so our logs do not get output. So we will add a logging library with the following dependency.
env_logger = "~0.10"Now add the following lines as the first lines inside our main function.
let log_level = env_logger::Env::default().default_filter_or("info");
env_logger::Builder::from_env(log_level).init();Now as our application runs it will also show the info! logs (using standard formatting not pretty formatting.
Updating git:
JSON Generation
Our next task is to generate Json output from our application and use it to write to files. We start by creating a new file for managing the schema objects and writing files called src/schema.rs with the following content.
use std::fs::File;
use serde::{Deserialize, Serialize};
// Describe a person using a simple data structure
#[derive(Serialize, Deserialize)]
pub struct Person {
/// Name of the Person
pub name: String,
/// Persons age
pub age: u32,
}
// Write a JSON file with with [count] copies of the [Person] object
///
/// * `filename` - A string slice that naem fo the file to generate
/// * `count` - a unsigned int to define how many copies of [Person] to generate
pub fn write_records(filename: &str, count: u32) {
let mydata: Vec<_> = (0..count).map(|x| Person{ name: format!("name-{:08}", x), age: x, }).collect();
let file = File::create(filename).unwrap();
serde_json::to_writer(file,&mydata).expect("Write JSON to file");
}NOTE: in the above code our function create and to_writer can generate errors BUT we simply use unwrap() or expect("") to catch these and trigger a panic (exit of application). The unwrap and expect are interchangable.
And add the following libraries to your dependencies.
serde_json = "~1.0"Remember to add the following to your src/ib.rs file to include the src/schema.rs in your build.
pub mod schema;Your build should now compile but you will note that the write_records function is not used yet. Lets update our src/main.rs to use it with the following code.
Commands::Generate { filename, count } => {
println!("Creating filename {filename} and writing {count} records");
write_records(&filename, count);
},and update your import to include write_records
use largejson::{webservice::MyConfig, schema::write_records};Now after building you can test it out.
# ./target/debug/largejson generate apple.json --count 100
Creating filename apple.json and writing 100 records
# ls -lh apple.json
-rw-r--r-- 1 bengreene staff 3.3K 27 Sep 17:17 apple.jsonSo far we have loaded config via yaml and written json content to a file based on a Struct.
Updating git:
Defining our Error Handling
Error handling is a key part of any application and especially so in Rust where error handling is very explicit. We can improve the errors we generated within the write_records function. First we will create a custom error object in a file called src/error.rs and again we need to include this into our modules by adding it to our src/lib.rs file.
/// Error type for handling errors on Sample
#[derive(Debug)]
pub enum MyError {
/// Error with a message
Message(&'static str),
/// Error when we call cancel
Cancelled,
/// Serde error including a message
Serde(&'static str),
/// IO error including a message
Io(&'static str),
/// Error with JSON validation
JsonValidation(Vec<String>),
/// Validation error
/// Improve this to carry an object to describe the details of the validation failure
ValidationError(),
}The MyError is an Enum that allows us to structure each option with in it to encode custom content. Now that add the import of MyError to the top of the src/schema.rs file (eg use crate::error::MyError;)and update your write_records function as follows.
// Write a JSON file with with [count] copies of the [Person] object
///
/// * `filename` - A string slice that naem fo the file to generate
/// * `count` - a unsigned int to define how many copies of [Person] to generate
pub fn write_records(filename: &str, count: u32) -> Result<(), MyError> {
let mydata: Vec<_> = (0..count).map(|x| Person{ name: format!("name-{:08}", x), age: x, }).collect();
let file = File::create(filename)?;
serde_json::to_writer(file,&mydata)?;
Ok(())
}The ? operator is a shortcut to capture an Err object and return that in the case of a error. Note we are also required to return an Ok for the positive case. The () returns an execution context rather than an actual object. In more complex examples you will return something from the function rather than simply completing execution.
We are not complete yet. If you look at your compilation output you will note that we have the following error the trait `From<serde_json::Error>` is not implemented for `MyError`. This indicates we desire the Err return type of write_records to be of type MyError but we have a type serde_json::Error and do not know how to convert them from one to the other. Let's add that capability to our MyError class. Add the following content to our error.rs file.
impl From<serde_json::Error> for MyError {
fn from(_value: serde_json::Error) -> Self {
Self::Serde("UNKNOWN &value.to_string()")
}
}
impl From<std::io::Error> for MyError {
fn from(_value: std::io::Error) -> Self {
Self::Io("UNKNOWN")
}
}Our application should almost be updated apart from the fact we changed the return type of write_records to return a Result.
Modify the return type of your main function to be Result<(), MyError>. After this add the ? operator to the call site of write_records. And add the Ok(()) as the final return.
What is the purpose of Result? The result definition is a return type contract that declares the return to be either of type () or when failing to be of type MyError. Additionally when the result is received the compiler will lint (and optionally enforce) that the Result has not been unpacked. The ? is syntax sugar to make it easy to consume Result objects and bubble errors upwards in the call stack providing we can handle the types of returned value. In our case originally we could not handle types of serde_json::Error or std::io::Error until we defined coercion/casting definitions for these using From and Into.
Updating git:
Add JSON Schema Generation
Next we will add Json Schema generation to our application. For that we will use Json Schema and therefore need to use theschemars library to help us generate JsonSchema from our structures (equivalent of data class in Python in Kotlin). Lets add this to our dependencies.
schemars = "~0.8"First we update our src/main.rs to reference a new function schema_person_string which we will build into our src/schema.rs file.
Commands::Schema => println!("{}", schema_person_string()?),And then reference that function by adding the import to top of our main.rs.
use largejson::{webservice::MyConfig, schema::{schema_person_string}};Lets build our schema write function into our src/schema.rs file.
/// Create and return the JsonSchema for the [Person] object
pub fn schema_person_string() -> Result<String, MyError> {
let my_schema = schema_for!(Person);
let my_schema_json = serde_json::to_string_pretty(&my_schema)?;
let my_schema_string = my_schema_json;
// println!("{my_schema_string}");
Ok(my_schema_string)
}Resolve the errors you see (eg error: cannot find macro `schema_for` in this scope) by adding the following use statement.
use schemars::{schema_for, JsonSchema};We also added JsonSchema at the same time (we need that soon). By inspeciting the compiler log we can see that the schema_for! macro needs the Person structure to implement the JsonSchema trait.
A trait is a contract we express against our Structures to declare the methods they implement or when used as inputs to a function it expresses the functions that the objects passed must implement. This is similar to classes BUT it is not the same.
Now back to our code. We need to add the JsonSchema trait capability to the Person structure. We can implement this manually or we can utilise the derive macro that is provided by the schemars library. So we update the derive statement of our Person structure to include JsonSchema in addition to the Serialise and Deserialise macros. The line will now look like this.
#[derive(JsonSchema, Serialize, Deserialize)]At this point your application should now build and we should be able to try it out.
# ./target/debug/largejson schema
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Person",
"type": "object",
"required": [
"age",
"name"
],
"properties": {
"age": {
"description": "Persons age",
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"name": {
"description": "Name of the Person",
"type": "string"
}
}
}If we read through the documentation for schemars we can add additional annotations to express more rules to the schema definition.
Get Benjamin Greene’s stories in your inbox
Join Medium for free to get updates from this writer.
Our next step is to add a schema generator for a list of Person objects. This will be similar to our schema_person_string function but we can build a generic funtion called schema_string to handle this for any type rather than creating a specific function for each type. We add this function call to our src/main.rs including the use reference.
Commands::SchemaList => println!("{}", schema_string::<Vec<Person>>()?),Then we add the function into src/schema.rs. Note the type reference on the function is <MyType: JsonSchema> which expresses that the type MyType is any type that implements the JsonSchema trait. If we were to use a parameter to the function that uses the MyType object then we could call the function and allow the compiler to resolve the type of MyType. This is not the case in this particular function therefore we need to explicitly provide the type of MyType at the function call site. In the above example (added to the src/main.rs the MyType is Vec<Person.
/// Dynamic generate of JsonSchema
///
/// Can use this to generate the JsonSchema for any [Serialisable] object rather than writing one function for each object.
/// This function can be used instead of [stdout_schema_list] and [schema_person_string]
pub fn schema_string<MyType: JsonSchema>() -> Result<String, MyError> {
let my_schema = schema_for!(MyType);
let my_schema_json = serde_json::to_string_pretty(&my_schema)?;
let my_schema_string = my_schema_json;
Ok(my_schema_string)
}And then to testing.
# ./target/debug/largejson schema-list
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Array_of_Person",
"type": "array",
"items": {
"$ref": "#/definitions/Person"
},
"definitions": {
"Person": {
"type": "object",
"required": [
"age",
"name"
],
"properties": {
"age": {
"description": "Persons age",
"type": "integer",
"format": "uint32",
"minimum": 0.0
},
"name": {
"description": "Name of the Person",
"type": "string"
}
}
}
}
}As a follow up we could remove the original schema_person_string and replace uses with our new schema_string function.
Updating github:
JSON Validation
Next we will add Json validation to our application. For that we will use our Json Schema and therefore need a include jsonschemas library to help us read Json Schema and perform validation. Lets add that to our dependencies.
jsonschema = "~0.17"Like for our schema command lets add a validate_with_schema function to src/schema.rs and map that function into our src/main.rs not forgetting to add the relevant use statement.
Commands::Validate { filename } => validate_with_schema(&filename, 2)?,Add the function to our src/schema.rs file.
pub fn validate_with_schema(filename: &str, error_limit: usize) -> Result<(), MyError> {
let schema_str = schema_string::<Vec<Person>>()?;
let file = File::open(filename)?;
let reader = BufReader::new(file);
match validate(&schema_str, reader, error_limit) {
Ok(data) => {
if let Value::Array(array) = &data {
let length = array.len();
println!("Found array of length {length}")
}
let _my_content: Vec<Person> = serde_json::from_value(data)?;
Ok(())
},
Err(err) => Err(err),
}
}
pub fn validate<R: Read>(schema: &str, reader: R, error_limit: usize) -> Result<Value, MyError> {
let schema_value = serde_json::from_str(schema)?;
let compiled_schema = JSONSchema::compile(&schema_value)?;
let data_value = serde_json::from_reader(reader)?;
{
let validation_result = compiled_schema.validate(&data_value);
if let Err(errors) = validation_result {
let error_stringified: Vec<_> = errors.take(error_limit).map(|value| {
format!("Error: instance: {:?}, kind: {:?}, instance_path:: {}, schema_path: {}",
value.instance, value.kind, value.instance_path, value.schema_path)
}).collect();
return Err(MyError::JsonValidation(error_stringified));
};
}
Ok(data_value)
}You will have to add some use statements to the top of our file to solve the type references. Once that is done your application should be ready to test except for the error: the trait `From<ValidationError<’_>>` is not implemented for `MyError`. We can fix that by adding a From implementation to our MyError structure. Add this to your src/error.rs.
impl From<ValidationError<'_>> for MyError {
fn from(_value: ValidationError<'_>) -> Self {
Self::ValidationError()
}
}Now lets create our test file (data_good.json) that matches to the schema and then test it out.
[
{"name":"name-00000000","age":0},
{"name":"name-00000001","age":1},
{"name":"name-00000002","age":2},
{"name":"name-00000003","age":3},
{"name":"name-00000004","age":4},
{"name":"name-00000005","age":5},
{"name":"name-00000006","age":6},
{"name":"name-00000007","age":7},
{"name":"name-00000008","age":8},
{"name":"name-00000009","age":9},
{"name":"name-00000010","age":10},
{"name":"name-00000011","age":11}
]Then we run our new command
# ./target/debug/largejson validate data_good.json
Found array of length 12Now lets create a faulty json (data_bad.json) file
[
{"name":"name-00000000","age":0},
{"name":"name-00000001","age":1},
{"name":"name-00000002","age":2},
{"name":"name-00000003","age":3},
{"name":"name-00000004","age":4},
{"name":"name-00000005","age":5},
{"name":"name-00000006"},
{"name":"name-00000007","age":7},
{"name":"name-00000008","age":8},
{"name":"name-00000009","age":"9"},
{"name":"name-00000010","age":10},
{"name":"name-00000011","age":11}
]And then we try again.
# ./target/debug/largejson validate data_bad.json
Error: JsonValidation(["Error: instance: Object {\"name\": String(\"name-00000006\")}, kind: Required { property: String(\"age\") }, instance_path:: /6, schema_path: /required", "Error: instance: String(\"9\"), kind: Type { kind: Single(Integer) }, instance_path:: /9/age, schema_path: /properties/age/type"])Updating github:
Create a Webservice
We have all the functional logic necessary for our webservice. Let us start to build that out. First we reference our xxx function for starting our webservice to the src/main.rs. We can update our Receive match arm as follows.
Commands::Receive{ config } => {
info!("Starting {NAME} for {VERSION}");
let config: MyConfig = MyConfig::figment(config)
.extract()
.expect("Config file loaded");
info!("Loaded config {:?}", config);
println!("Loaded config as {:#?}", config);
http_receive_json(config.webservice)?
},Then we start building our webservice.rs by adding the following functions.
handle_rejection: provide a nice error message to logging and as a return from the webservicereceive_binary_review: read our incoming file and validate it against our JSON Schema. This is a naive implementation. We may wish to move the compile step outside the http path and consider utilisation of streaming APIs so we do not need to assemble the whole file in memory ahead of processing it.http_service_cancellable: A function to define our webservice including the routing, paths, server and logging definitions.http_receive_json: a wrapper function to provide a vanilla function interface to our CLI logic and encapsulate the async runtime and execution of our webservice function wtihin the runtime.
use std::{
path::Path,
convert::Infallible,
};
use figment::{Figment, providers::{Yaml, Format}};
use futures::{Stream, StreamExt};
use log::info;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tokio_util::{sync::CancellationToken, bytes::Buf};
use warp::{hyper::StatusCode, Rejection, Reply, reply::json, reject::Reject, Filter};
use crate::{
NAME,
schema::{schema_string, Person, validate},
tokio_tools::run_in_tokio,
error::MyError,
};
//... ORIGINAL webservice.rs file content goes here
#[derive(Serialize)]
struct ValidationReply {
size: usize,
length: usize,
validate: bool,
}
async fn handle_rejection(err: Rejection) -> std::result::Result<impl Reply, Infallible> {
let (code, json_message) = if err.is_not_found() {
(StatusCode::NOT_FOUND, json(&"Not Found".to_string()))
} else if err.find::<warp::reject::PayloadTooLarge>().is_some() {
(StatusCode::BAD_REQUEST, json(&"Payload too large".to_string()))
} else if let Some(e) = err.find::<MyError>() {
match e {
MyError::Message(_) => todo!(),
MyError::Cancelled => todo!(),
MyError::Serde(_) => todo!(),
MyError::Io(_) => todo!(),
MyError::JsonValidation(errors) => {
let myval = serde_json::json!( { "status:": "validation failed","errors": errors});
(StatusCode::BAD_REQUEST, json(&myval))
},
MyError::ValidationError() => todo!(),
}
} else {
eprintln!("unhandled error: {:?}", err);
(
StatusCode::INTERNAL_SERVER_ERROR,
json(&"Internal Server Error".to_string()),
)
};
Ok(warp::reply::with_status(json_message, code))
}
async fn receive_binary_review(
mut body: impl Stream<Item = Result<impl Buf, warp::Error>> + Unpin + Send + Sync
) -> Result<impl Reply, Rejection> {
// https://github.com/seanmonstar/warp/issues/448
let error_limit = 20;
let mut file_content = Vec::new();
let mut chunk_tot = 0;
while let Some(buf) = body.next().await {
let mut buf = buf.unwrap();
while buf.remaining() > 0 {
let chunk = buf.chunk();
let chunk_len = chunk.len();
// println!("getting chunk of len = {chunk_len}");
chunk_tot += chunk_len;
file_content.extend_from_slice(chunk);
buf.advance(chunk_len);
}
}
let schema_str = schema_string::<Vec<Person>>()?;
match validate(&schema_str, &file_content[..], error_limit) {
Ok(data) => {
let length = if let Value::Array(array) = &data {
array.len()
} else {
0
};
let reply = ValidationReply{
size: chunk_tot,
length: length,
validate: true,
};
Ok(warp::reply::json(&reply))
},
Err(err) => Err(err.into()),
}
}
// Marker trait to indicate MyError is a planned rejection type
impl Reject for MyError {}
pub async fn http_service_cancellable(ct: CancellationToken, config: WebServiceConfig) -> Result<(), MyError> {
// Setup http server
let download_route = warp::path("files")
.and(warp::fs::dir("./files/"));
let upload_route_review = warp::path("review")
.and(warp::post())
.and(warp::body::stream())
.and_then(receive_binary_review);
let weblog = warp::log(NAME);
let combined = download_route
.or(upload_route_review)
.recover(handle_rejection)
.with(weblog);
let prefix_path = warp::path(config.prefix.name.clone())
.and(warp::path(config.prefix.version.clone()));
let router = prefix_path
.and(combined);
let (addr, server) = warp::serve(router)
.bind_with_graceful_shutdown(
([0, 0, 0, 0], 8080),
async move {ct.cancelled().await},
);
info!("Server started on port {}", addr);
Ok(server.await)
}
pub fn http_receive_json(config: WebServiceConfig) -> Result<(), MyError> {
let ct = CancellationToken::new();
run_in_tokio(http_service_cancellable(ct, config))
}As we add this we now need to add some more dependencies to enable our webservice. We will use tokio to provide an async runtime, tokio-util to extend tokio’s functionality and warp provides a powerful (and fast) webservice based on the hyper library. futures provides a definition of future objects to wait for using tokio.
tokio = { version = "~1.32", features = ["full"] }
futures = "~0.3"
tokio-util = "~0.7"
warp = "~0.3.5"This function utilises some utility functions to make running code in tokio a little simpler whilst allowing us to control the threading. Create a new file src/tokio_utils.rs and reference it as a module from your src/lib.rs.
//! Module to handle easy sending functions to tokio
//!
//! The provides two functions one function run_in_tokio creates and sends the function to tokio.
//! The second function run_in_tokio_with_cancel allows the creation of a CancellationToken which can be used to shut down the tokio async.
use crate::error::MyError;
use futures::Future;
use log::{info, error};
use tokio_util::sync::CancellationToken;
/// run async function inside tokio instance on current thread
pub fn run_in_tokio<F, T>( my_function: F) -> F::Output
where F: Future<Output = Result<T, MyError>>
{
info!("starting Tokio");
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("Runtime created in current thread")
.block_on(my_function)
}
/// Run async with cancellability via CancellationToken
pub fn run_in_tokio_with_cancel<F, T>(cancel: CancellationToken, my_function: F) -> F::Output
where F: Future<Output = Result<T,MyError>>
{
run_in_tokio(async {
tokio::select!{
_ = cancel.cancelled() => {
error!("Token cancelled");
Err(MyError::Cancelled)
},
z = my_function => {
eprintln!("Completed function");
z
},
}
})
}We can now test out our service to see it working with the data_good.json.
# curl -v -d@data_good.json http://localhost:8080/test/v0/review
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /test/v0/review HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 449
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 40
< date: Fri, 06 Oct 2023 16:46:43 GMT
<
* Connection #0 to host localhost left intact
{"size":449,"length":12,"validate":true}We can see the good response returns a json object with information relating to the size, number of records and validation outcome.
And we also try with the data_bad.json file.
# curl -v -d@data_bad.json http://localhost:8080/test/v0/review
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> POST /test/v0/review HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.1.2
> Accept: */*
> Content-Length: 443
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 400 Bad Request
< content-type: application/json
< content-length: 326
< date: Fri, 06 Oct 2023 16:46:56 GMT
<
* Connection #0 to host localhost left intact
{"errors":["Error: instance: Object {\"name\": String(\"name-00000006\")}, kind: Required { property: String(\"age\") }, instance_path:: /6, schema_path: /required","Error: instance: String(\"9\"), kind: Type { kind: Single(Integer) }, instance_path:: /9/age, schema_path: /properties/age/type"],"status:":"validation failed"}We can see the negative status code accompanied by our ValidationError object as a json reply showing a status string and an errors list showing the first 20 error records defined by error_limit.
Updating github:
Dockerise the Service
Now that we have a working web service our next step is to compile this application into a Docker container to allow us to deploy it within a container execution environment (eg Kubernetes, etc).
Create a Dockerfile to describe the container build process with the following content.
ARG BASEIMAGE=debian:bookworm-slim
FROM rust:1-bookworm AS chef
RUN cargo install cargo-chef
WORKDIR app
FROM chef as planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json
FROM chef AS build
COPY --from=planner /app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json
# Build application
COPY . .
RUN cargo build --release
# We do not need the Rust toolchain to run the binary!
FROM ${BASEIMAGE} as runtime
WORKDIR app
COPY --from=build /app/target/release/largejson /usr/local/bin/
EXPOSE 8080We now add a Makefile to our build to provide some nicer ergonomics.
TAG ?= 0.1.0
IMAGE_NAME=myimages/largejson
BASEIMAGE=debian:bookworm-slim
ifeq ($(shell command -v podman 2> /dev/null),)
DOCKER=docker
else
DOCKER=podman
endif
.PHONY: build crds
all:
@echo making everything
.ONESHELL:
docker-build:
{ \
$(DOCKER) build . -t $(IMAGE_NAME) -f Dockerfile --build-arg BASEIMAGE=$(BASEIMAGE); \
$(DOCKER) image ls $(IMAGE_NAME); \
}
docker-build-distroless: BASEIMAGE=gcr.io/distroless/cc-debian12
docker-build-distroless: IMAGE_NAME=myimages/largejson-distroless
docker-build-distroless: docker-build
cargo-doc:
@cargo doc --no-deps --document-private-items
@open target/doc/largejson/index.htmlWhen creating the Makefile definition remember Makefile is OLD so uses a tab delimiter as the indent for the task definitions. Now lets try our our Docker build process. This build process uses a tool called chef to generate a recipe.json file which is derived from the dependencies of your application. If the dependencies change then the recipe.json file updates and therefore triggers a refresh of the build process. However, if the dependencies are not updated then parts of the build (dependency fetch) can be optimised out leaving a faster build process during development.
# make docker-build
{ \
podman build . -t myimages/largejson -f Dockerfile; \
podman image ls myimages/largejson; \
}
[1/4] STEP 1/3: FROM rust:1-bookworm AS chef
[1/4] STEP 2/3: RUN cargo install cargo-chef
--> Using cache dcd773c82ac7a568257f86408b84d25d6dacedc8768bcd6f2fbe6c842527e2f7
--> dcd773c82ac7
[1/4] STEP 3/3: WORKDIR app
--> Using cache 2bab0f06ea7e3fc019eda037411b58fd9f94e5b3ccd9ab46d8f01b4ec6ebe4d1
--> 2bab0f06ea7e
[2/4] STEP 1/3: FROM 2bab0f06ea7e3fc019eda037411b58fd9f94e5b3ccd9ab46d8f01b4ec6ebe4d1 AS planner
[2/4] STEP 2/3: COPY . .
--> d8a8d2ee8aab
[2/4] STEP 3/3: RUN cargo chef prepare --recipe-path recipe.json
--> 9b195cbfd789
[3/4] STEP 1/5: FROM 2bab0f06ea7e3fc019eda037411b58fd9f94e5b3ccd9ab46d8f01b4ec6ebe4d1 AS build
[3/4] STEP 2/5: COPY --from=planner /app/recipe.json recipe.json
--> a433087ea6e9
[3/4] STEP 3/5: RUN cargo chef cook --release --recipe-path recipe.json
Updating crates.io index
Downloading crates ...
...
[4/4] STEP 1/4: FROM debian:bookworm-slim AS runtime
[4/4] STEP 2/4: WORKDIR app
--> Using cache 4145a0c73519793fb4169bdd7618a44058073f773242302fca4ae00083e3b9c3
--> 4145a0c73519
[4/4] STEP 3/4: COPY --from=build /app/target/release/largejson /usr/local/bin/
--> aecc572bcba2
[4/4] STEP 4/4: EXPOSE 8080
[4/4] COMMIT myimages/largejson
--> 7a524365b508
Successfully tagged localhost/myimages/largejson:latest
7a524365b508895538dcaf45492585bee3fdb0d49b352757ee2e58b87ceaa64b
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/myimages/largejson latest 7a524365b508 Less than a second ago 118 MB
# docker image ls myimages/largejson
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/myimages/largejson latest 7a524365b508 35 seconds ago 118 MB
# docker run -it myimages/largejson
oot@7340c4206287:/app# /usr/local/bin/largejson
Application definition to defer to set of commands under [Commands]
Usage: largejson <COMMAND>
Commands:
generate Generate messages
schema Show schema for object
schema-list Show schema for Vec of object
validate Validate file against schema
receive Receive json file via http
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print versionOur final image came out at 118MB which is not too bad considering the base image (debian:bookworm-slim) is 100MB. A simple optimisation is to change the base image to gcr.io/distroless/cc-debian12. This base image is 35.3 MB resulting in a final docker image of 52.7 MB. Note that we need to match the version of the build system to the version of the runtime in order to match up the usage of glibc.
# make docker-build-distroless
...
Successfully tagged localhost/myimages/largejson-distroless:latest
9b8c939db51e82f9b42b8b986fb819b30fca5a8b053c63260d043e2849bb36ce
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/myimages/largejson-distroless latest 9b8c939db51e 30 seconds ago 52.7 MBIf we wanted to futher reduce this we could consider building a statically compiled executable and using FROM SCRATCH. This article provides a good overview of that https://kerkour.com/rust-small-docker-image.
update github:
Benchmarking
Now that we have a basic webservice that implements the functionality we desire the next step is to test this. In particular we are interested in the number of threads needed for running this application and the impact threads have upon its ability to implement its API.
Rust provides a useful library called Criterion that provides an excellent benchmarking suite for benchmarking function performance. For testing we will also add teh reqwest library. This is similar to the request library commonly used in python. So we add the following to our Cargo.toml file.
[dev-dependencies]
criterion = { version = "~0.5", features = ["async_tokio"] }
reqwest = { version = "~0.11" }
[[bench]]
name = "validate"
harness = falseThis will add some libraries to our development dependencies and declare a benchmark to create. Now we need to write our benchmark. All benchmarks are written in the benches directory. Create the new file src/validate.rs under the benches directory with the following content.
use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId};
use largejson::schema::{schema_person_string, Person, schema_string, validate};
pub fn schema_benchmark(c: &mut Criterion) {
c.bench_function("schema generate Person", |b| b.iter(schema_person_string));
c.bench_function("schema Generate Vec<Person>", |b| b.iter(schema_string::<Vec<Person>>));
}
pub fn validate_benchmark(c: &mut Criterion) {
c.bench_function("schema generate Person", |b| b.iter(schema_person_string));
c.bench_function("schema Generate Vec<Person>", |b| b.iter(schema_string::<Vec<Person>>));
let error_limit = 2;
let schema = schema_string::<Vec<Person>>().expect("got schema");
let mut group = c.benchmark_group("validate");
for size in [
0,
1,10,100,1000,10000,
// 1,2,3,4,5,6,7,8,9,
// 10,20,30,40,50,60,70,80,90,
// 100,200,300,400,500,600,700,800,900,
// 1000,2000,3000,4000,5000,6000,7000,8000,9000,
// 10000,
// 100000,
// 1000000,
].iter() {
let my_data_vec: Vec<_> = (0..*size).map(|x| Person{ name: format!("name-{:08}", x), age: x, }).collect();
let my_data_string = serde_json::to_string_pretty(&my_data_vec).expect("to string");
let my_data_bytes = my_data_string.as_bytes();
group.bench_with_input(
BenchmarkId::from_parameter(size),
size,
|b, &_size| {
b.iter(|| validate(&schema, my_data_bytes, error_limit).expect("validated") );
}
);
}
group.finish();
}
criterion_group!(benches, validate_benchmark, schema_benchmark);
criterion_main!(benches);Criterion access our code as a library so can only reference interfaces that are publically exposed. If you have embedded functions into your library that you later want to benchmark you will need to resolve them as public methods.
Now we run our benchmark.
# cargo bench
Finished bench [optimized] target(s) in 0.30s
Running unittests src/lib.rs (target/release/deps/largejson-8235f941187fe6f8)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running unittests src/main.rs (target/release/deps/largejson-d6b15ad3714eb711)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Running benches/validate.rs (target/release/deps/validate-0df08bfa549f0c7c)
schema generate Person time: [2.0284 µs 2.0362 µs 2.0440 µs]
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) low mild
Benchmarking schema Generate Vec<Person>: Collecting 100 samples in estimated 5.0032 s (1.4M iteration
schema Generate Vec<Person>
time: [3.3991 µs 3.4179 µs 3.4366 µs]
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
validate/0 time: [15.307 µs 15.399 µs 15.491 µs]
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
validate/1 time: [21.147 µs 21.230 µs 21.308 µs]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
validate/10 time: [28.465 µs 28.569 µs 28.683 µs]
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
validate/100 time: [94.432 µs 94.674 µs 94.904 µs]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
validate/1000 time: [732.02 µs 733.66 µs 735.97 µs]
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) high mild
6 (6.00%) high severe
validate/10000 time: [7.7367 ms 7.7745 ms 7.8124 ms]
schema generate Person #2
time: [1.9457 µs 1.9572 µs 1.9684 µs]
Benchmarking schema Generate Vec<Person> #2: Collecting 100 samples in estimated 5.0144 s (1.5M iterat
schema Generate Vec<Person> #2
time: [3.4117 µs 3.4163 µs 3.4213 µs]
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severeThis will create and/or update the criterion report in this file target/criterion/report/index.html.
which provides some nice performance diagrams such as the following exerpts.
Lets add a more complex benchmark to replicate our users accessing the service via http. Add the new benchmark definition to your Cargo.toml file
[[bench]]
name = "http"
harness = falseNow we create the benchmark definition in the files benches/http.rs with the following types:
- ParamVariant (Enum describing the variable we are going to test over)
- PostParams (Definition of the parameters for a specific test and the variant)
And the following functions:
- rt_multithread (create a async runtime with n threads)
- gen_data (generate our dataset to test with)
- http_post_fetch (definition of our post client)
- http_post_parallel (run n copies of http_post_fetch in parallel)
- post_bench_paramed (generic test function to test across all our parameters)
- bench_concurrent (Specific test to vary the concurrent users)
- bench_server (Specific test to vary the server threads)
use std::{time::Duration, fmt::Display};
use criterion::{criterion_group, criterion_main, Criterion, BenchmarkId, Throughput};
use futures::future::join_all;
use largejson::{schema::Person, webservice::{http_service_cancellable, WebServicePrefixConfig, WebServiceConfig}, error::MyError};
use reqwest::{Client, Response};
use serde::Serialize;
use tokio::runtime::{Runtime, self};
use tokio_util::sync::CancellationToken;
#[derive(Debug, PartialEq, Clone)]
enum ParamVariant {
ServerThreads,
ClientThreads,
DataLength,
ConcurrentUsers,
}
#[derive(Debug)]
struct PostParams {
num_threads_server: usize,
num_threads_client: usize,
data_length: usize,
num_concurrent: usize,
variable: ParamVariant,
}
impl Display for PostParams {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{:?}", self.variable)?;
if self.variable != ParamVariant::ServerThreads {
write!(f, "-server:{}", self.num_threads_server)?;
}
if self.variable != ParamVariant::ClientThreads {
write!(f, "-client:{}", self.num_threads_client)?;
}
if self.variable != ParamVariant::DataLength {
write!(f, "-length:{:07}", self.data_length)?;
}
if self.variable != ParamVariant::ConcurrentUsers {
write!(f, "-concurrent:{:03}", self.num_concurrent)?;
}
Ok(())
}
}
fn rt_multithreaded(threads: usize) -> Runtime {
if threads == 0 {
runtime::Builder::new_current_thread()
.enable_io()
.enable_time()
.build().unwrap()
} else {
runtime::Builder::new_multi_thread()
.worker_threads(threads)
.thread_name(format!("threads-{threads}"))
.thread_stack_size(3 * 1024 * 1024)
.enable_io()
.enable_time()
.build().unwrap()
}
}
fn gen_data(size: usize) -> Result<Vec<Person>, MyError> {
Ok((0..size).map(|index| Person{
name: format!("name-{:08}", index),
age: u32::try_from(index).unwrap(),
}).collect())
}
async fn http_post_fetch<MyObj: Serialize >(client: &Client, my_content: &Vec<MyObj>) -> Result<Response, reqwest::Error> {
client.post("http://localhost:8080/test/v0/review")
.json(my_content)
.send()
.await
}
async fn http_post_parallel<MyObj: Serialize>(client: &Client, my_content: &Vec<MyObj>, concurrent_count: usize) {
let list_of = (0..concurrent_count).map(|_index| {
async move {
let resp = http_post_fetch(client, my_content);
match resp.await {
Ok(_msg) => {
},
Err(e) => println!("ERRORED {}", e),
};
}
});
let _mye = join_all(list_of).await;
}
fn post_benchmark_paramed(c: &mut Criterion,
group_name: &str, variable: &ParamVariant, bench_name: &str,
server_threads: Vec<usize>, client_threads: Vec<usize>, message_lengths: Vec<usize>, message_concurrents: Vec<usize>,
) {
let config = WebServiceConfig{
prefix: WebServicePrefixConfig {
name: "test".to_owned(),
version: "v0".to_owned(),
},
};
let client = reqwest::Client::new();
let mut group = c.benchmark_group(group_name);
for num_threads_server in &server_threads {
let server_rt = rt_multithreaded(*num_threads_server);
let ct = CancellationToken::new();
let server_jh = server_rt.spawn(
http_service_cancellable(ct.clone(), config.clone())
);
for data_length in &message_lengths {
let my_data = gen_data(*data_length).unwrap();
for num_threads_client in &client_threads {
let client_rt = rt_multithreaded(*num_threads_client);
for num_concurrent in &message_concurrents {
group.throughput(Throughput::Elements((data_length*num_concurrent) as u64 ));
let my_var = match *variable {
ParamVariant::ServerThreads => num_threads_server,
ParamVariant::ClientThreads => num_threads_client,
ParamVariant::DataLength => data_length,
ParamVariant::ConcurrentUsers => num_concurrent,
};
group.bench_with_input(
BenchmarkId::new(format!("{}-{}", bench_name,PostParams{ num_threads_server: *num_threads_server, num_threads_client: *num_threads_client, data_length: *data_length, num_concurrent: *num_concurrent, variable: (*variable).clone() }), my_var),
// BenchmarkId::new(bench_name, PostParams{ num_threads_server: *num_threads_server, num_threads_client: *num_threads_client, data_length: *data_length, num_concurrent: *num_concurrent, variable: (*variable).clone() }),
my_var,
|b, &s| {
// Insert a call to `to_async` to convert the bencher to async mode.
// The timing loops are the same as with the normal bencher.
b.to_async(&client_rt).iter(|| http_post_parallel(&client, &my_data, *num_concurrent));
}
);
}
client_rt.shutdown_timeout(Duration::from_secs(3));
}
}
ct.cancel();
server_rt.shutdown_timeout(Duration::from_secs(3));
}
group.finish();
}
pub fn bench_concurrent(c: &mut Criterion) {
let server_threads = vec!(1);
let client_threads = vec!(5);
let message_lengths = vec!(1000);
let message_concurrents = vec!(1,2,10,20,200);
post_benchmark_paramed(c, "Users", &ParamVariant::ConcurrentUsers, "post", server_threads, client_threads, message_lengths, message_concurrents);
}
pub fn bench_server(c: &mut Criterion) {
let server_threads = vec!(1,2,5,10);
let client_threads = vec!(5);
let message_lengths = vec!(1000);
let message_concurrents = vec!(200);
post_benchmark_paramed(c, "Server Threads", &ParamVariant::ServerThreads, "post", server_threads, client_threads, message_lengths, message_concurrents);
}
criterion_group!(benches, bench_concurrent, bench_server);
criterion_main!(benches);We can run these benchmarks using the following commands:
# cargo bench Users
...
# cargo bench Server
...And when complete view your results in the test report.
A useful feature of Criterion is that it provides a comparison with previous benchmark runs as a regression allowing you to compare performance across changes in your functions.
Updating github:
Documentation
Rust provides some great support to allow you to generate your API and/or application documentation within your source code. You can try this out with the following command.
# cargo doc --no-deps --document-private-items
Documenting largejson v0.1.0 (.../RustBeyond101/largejson)
warning: unresolved link to `Serialisable`
--> src/schema.rs:44:54
|
44 | ...JsonSchema for any [Serialisable] object rather than writing one function for each object.
| ^^^^^^^^^^^^ no item named `Serialisable` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
= note: `#[warn(rustdoc::broken_intra_doc_links)]` on by default
warning: unresolved link to `stdout_schema_list`
--> src/schema.rs:45:43
|
45 | /// This function can be used instead of [stdout_schema_list] and [schema_person_string]
| ^^^^^^^^^^^^^^^^^^ no item named `stdout_schema_list` in scope
|
= help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
warning: `largejson` (lib doc) generated 2 warningsYou can then open the documentation at this location target/doc/largejson/index.html providing a page such as this.
Rust uses two forms of the documentation syntax to provide documents for the object that is enclosing the docs (//! content here) and prefixing the object using either (/// content here or /** content here */).
Work through your functions in the webservice.rs and add documentation. eg. we document the parameters in our WebServicePrefixConfig object and also provide references to WebServiceConfig.
/// Configuration for our webservice prefix. Typically provided as part of the [WebServiceConfig]
#[derive(Deserialize, Debug, Clone)]
pub struct WebServicePrefixConfig {
/// Name of our webservice
pub name: String,
/// Version of our webservice
pub version: String,
}Similarly we can provide short and long form documentation seperated by an empty comment line. The short form is visible where a summary is needed and then when we descend to the detail views we can see the long form.
/// Define the negative rejection responses for our service.
///
/// This needs additional work to fully expand all the TODO: sections
/// The key part of this response is that we provide an explict response for the JsonValidation error to provide a
/// limited summary of the validation errors to allow the client to correct any issues.
async fn handle_rejection(err: Rejection) -> std::result::Result<impl Reply, Infallible> {Much more information is available on using rust doc here at the official book: https://doc.rust-lang.org/rustdoc/index.html
One particularly interesting add on to rust doc is the Aquamarine library which provides embedded diagrams using https://mermaid.js.org/.
Updating github:
Testing
Rust provides a rich integrated testing definition that allowing you to create your tests along with you code and additionally you can also include tests within your documentation which is ideal for providing code snippits and API demonstration code.
We will start by adding a test onto the end of our schema.rs file. We define a tests module within schema.rs and then enable it only when test is configured (via macro).
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn validate_vec() {
let schema = schema_string::<Vec<Person>>().expect("got schema");
println!("my schema is {schema}");
let error_limit = 2;
let size = 220;
let my_data_vec: Vec<_> = (0..size).map(|x| Person{ name: format!("name-{:08}", x), age: x, }).collect();
let my_data_string = serde_json::to_string_pretty(&my_data_vec).expect("to string");
let my_data_bytes = my_data_string.as_bytes();
let validated = validate(&schema, my_data_bytes, error_limit).expect("validated");
let validated_typed: Vec<Person> = serde_json::from_value(validated).expect("conversion to Explicit Type");
assert!(validated_typed.len() == usize::try_from(size).unwrap());
println!("There are {} records", validated_typed.len());
}
}This simple test constructs a schema using our schema_string function then uses creates some sample data and then passes both to the validation function and confirms we get a positive reply. We can then test it as follows.
# cargo test
Blocking waiting for file lock on build directory
Finished test [unoptimized + debuginfo] target(s) in 0.64s
Running unittests src/lib.rs (target/debug/deps/largejson-9af8e76592ca58d5)
running 1 test
test schema::tests::validate_vec ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
Running unittests src/main.rs (target/debug/deps/largejson-3012c4929a2e03d5)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests largejson
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sOur next step is to add an example of how to use our schema_string function into the docs and also double this up as a test. Add the following lines to the long form of our comment for schema_string.
///
/// ```
/// # use largejson::error::MyError;
/// # use largejson::schema::Person;
/// # use largejson::schema::schema_string;
/// # fn main() -> Result<(), MyError> {
/// let person_schema = schema_string::<Person>()?;
///
/// assert!(person_schema.len() > 0);
/// println!("Schema = {}", person_schema);
/// # Ok(())
/// # }
/// ```This provides a minimal example where the Person object has its schema generated and we check that we get a non-empty reply back.
We can run this using the same test command as above and view it as part of our documentation.
# cargo test
Finished test [unoptimized + debuginfo] target(s) in 0.24s
Running unittests src/lib.rs (target/debug/deps/largejson-9af8e76592ca58d5)
running 1 test
test schema::tests::validate_vec ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
Running unittests src/main.rs (target/debug/deps/largejson-3012c4929a2e03d5)
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
Doc-tests largejson
running 1 test
test src/schema.rs - schema::schema_string (line 49) ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.92s
# cargo doc --no-deps --document-private-items
Documenting largejson v0.1.0 (.../RustBeyond101/largejson)
Finished dev [unoptimized + debuginfo] target(s) in 1.46sNow inspecting the relevant part of our documentation we can observe our inline example.
Updating github:
Conclusion
Rust has some useful libraries to allow us to quickly create a functioning webservice that we could easily deploy as a microservice. The Criterion benchmarking tools allow us to quickly establish performance comparisons for functions in our code which will allow us to be more confident about our production readiness.
This tutorial only covered creating a validation endpoint and does not include any storage tier. That would an interesting next step to this tutorial using libraries such as Diesel.
