<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by György Balázsi on Medium]]></title>
        <description><![CDATA[Stories by György Balázsi on Medium]]></description>
        <link>https://medium.com/@gyorgybalazsi?source=rss-42425608db34------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Ib-sY24ZLem9S1rR.jpg</url>
            <title>Stories by György Balázsi on Medium</title>
            <link>https://medium.com/@gyorgybalazsi?source=rss-42425608db34------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 23:06:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gyorgybalazsi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to iterate over Rust struct fields]]></title>
            <link>https://medium.com/rust-masterclass/how-to-iterate-over-rust-struct-fields-65e7331a9b00?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/65e7331a9b00</guid>
            <category><![CDATA[procedural-macros]]></category>
            <category><![CDATA[macro]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[traits]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 22:17:42 GMT</pubDate>
            <atom:updated>2025-01-26T22:17:42.388Z</atom:updated>
            <content:encoded><![CDATA[<p>Programming is fun; metaprogramming is fun squared (I repeat myself). The basic use case for manipulating Rust code programmatically is to iterate over struct fields, e.g., for deriving traits. This can be done using procedural macros, which are more complex but also more interesting and more useful than declarative macros.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1V1EkgbIEtOhvTkO" /><figcaption>Photo by <a href="https://unsplash.com/@jjying?utm_source=medium&amp;utm_medium=referral">JJ Ying</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Disclaimer: I didn’t invent the wheel or the method to iterate over struct fields in Rust. My contribution to the latter topic is sharing a trick to reverse-engineer and better understand the <a href="https://solana.com/developers/courses/program-optimization/rust-macros#example-of-a-procedural-macro">Example of a procedural macro</a> tutorial, published in the Solana Development Courses collection. (Or any other code example, for that matter.)</p><p>If you find the original explanation crystal clear, this blog post is not for you. If you can use some help to understand it, bear with me.</p><h3>Overview</h3><p>The goal is to create a macro, which automatically derives the Describe trait for any struct, like this:</p><pre>use example_macro::Describe;<br> <br>#[derive(Describe)]<br>struct MyStruct {<br>    my_string: String,<br>    my_number: u64,<br>}<br> <br>fn main() {<br>    MyStruct::describe();<br>}</pre><p>With the following result:</p><pre>MyStruct is a struct with these named fields: my_string, my_number.</pre><p>In other words, the describe function should print a message containing the name of the struct (“MyStruct”) and a list of its field names, separated by commas (“my_string, my_number”).</p><p>Such a function can be easily created manually for every struct. But we don’t want to do it manually.</p><p>Procedural macros for trait derivation work like this:</p><ol><li>The compiler converts the input source code, that is, the source code of the struct for which we want to derive the trait, into a <a href="https://doc.rust-lang.org/proc_macro/struct.TokenStream.html">TokenStream</a>. A TokenStream value is a sequence of <a href="https://doc.rust-lang.org/beta/proc_macro/enum.TokenTree.html">token trees</a>, where “token” is the basic building block of the Rust syntax (like the “struct” keyword, the “MyStruct” identifier, curly braces, field name identifiers and types, colons, commas, etc). The token stream is not a simple sequence of tokens but rather a sequence of token trees because it contains token groups wrapped into pairs of different brackets (“(…)”, “{…}”, “[…]”, “&lt;…&gt;”), potentially nested. This conversion happens under the hood, and we don’t have to care about it during the implementation. We will see later how the TokenStream value for our struct gets printed.</li><li>What we have to implement is the macro function that takes as input the struct TokenStream and produces another TokenStream for the “impl” block, which, in our case, defines the describe function for the Describe trait. The macro function performs the following steps: a) using the <a href="https://crates.io/crates/syn">syn crate</a>, it converts the TokenStream input to the corresponding Abstract Syntax Tree (aka AST), which in the case of a derive macro has the <a href="https://docs.rs/syn/latest/syn/struct.DeriveInput.html">DeriveInput</a> data type; b) it extracts the necessary information from the AST (in our case the struct name and the sequence of field names); c) using the <a href="https://crates.io/crates/quote">quote crate</a>, it assembles the TokenStream which will be returned by the macro function.</li><li>The TokenStream of the “impl” block, returned by the macro function, will be inserted into the AST of the source file by the compiler.</li></ol><p>Please note that in the case of a “derive” macro, the generated code is added to the module or block containing the struct (as it contains a trait implementation for the struct). In the case of other kinds of macros, the input code is replaced by the code generated by the macro function.</p><h3>Reverse-engineering and illustrating the example</h3><p>All this is explained in the <a href="https://solana.com/developers/courses/program-optimization/rust-macros#example-of-a-procedural-macro">Example of a procedural macro</a> tutorial.</p><p>What can help to better understand the example code is the illustration of the input TokenStream and the DeriveInput AST. The illustration of the TokenStream value is just interesting, but the illustration of the DeriveInput AST is actually needed to understand how the macro function extracts the struct name and the field names from the AST.</p><p>For the illustration, we want to add some debug calls to the original code example. But this isn’t possible if we simply declare the syn crate as a dependency.</p><p>Because in the syn crate, the Debug trait implementations are feature-gated, to debug some values, we have to specify the dependency in the Cargo.tomlfile like this:</p><pre>syn = { version = &quot;2.0&quot;, features = [&quot;full&quot;, &quot;extra-traits&quot;] }</pre><p>With this in place, we can add the necessary dbg! macro calls to the example code:</p><pre>use proc_macro::{self, TokenStream};<br>use quote::quote;<br>use syn::{parse_macro_input, DeriveInput, FieldsNamed};<br><br><br>#[proc_macro_derive(Describe)]<br>pub fn describe(input: TokenStream) -&gt; TokenStream {<br>    <br>    dbg!(&amp;input);<br>    let DeriveInput { ident, data, .. } = dbg!(parse_macro_input!(input));<br> <br>    let field_names: String = match data {<br>        syn::Data::Struct(s) =&gt; match s.fields {<br>            syn::Fields::Named(FieldsNamed { named, .. }) =&gt; {<br>                let idents = named.iter().map(|f| &amp;f.ident);<br>                format!(<br>                    &quot;a struct with these named fields: {}&quot;,<br>                    quote! {#(#idents), *},<br>                )<br>            }<br>            _ =&gt; panic!(&quot;The syn::Fields variant is not supported&quot;),<br>        },<br>        _ =&gt; panic!(&quot;The syn::Data variant is not supported&quot;),<br>    };<br> <br>    let expanded = quote! {<br>        impl #ident {<br>            fn describe() {<br>               println!(&quot;{} is {}.&quot;, stringify!(#ident), #field_names);<br>            }<br>        }<br>    };<br> <br>    expanded.into()<br>}</pre><p>The first dbg! call will print the input token stream. This is interesting but not inevitable for the understanding, but while we are at it, let’s see what it looks like:</p><pre>[describe_derive/src/lib.rs:10:5] &amp;input = TokenStream [<br>    Ident {<br>        ident: &quot;struct&quot;,<br>        span: #0 bytes(53..59),<br>    },<br>    Ident {<br>        ident: &quot;YourStruct&quot;,<br>        span: #0 bytes(60..70),<br>    },<br>    Group {<br>        delimiter: Brace,<br>        stream: TokenStream [<br>            Ident {<br>                ident: &quot;my_string&quot;,<br>                span: #0 bytes(77..86),<br>            },<br>            Punct {<br>                ch: &#39;:&#39;,<br>                spacing: Alone,<br>                span: #0 bytes(86..87),<br>            },<br>            Ident {<br>                ident: &quot;String&quot;,<br>                span: #0 bytes(88..94),<br>            },<br>            Punct {<br>                ch: &#39;,&#39;,<br>                spacing: Alone,<br>                span: #0 bytes(94..95),<br>            },<br>            Ident {<br>                ident: &quot;my_number&quot;,<br>                span: #0 bytes(100..109),<br>            },<br>            Punct {<br>                ch: &#39;:&#39;,<br>                spacing: Alone,<br>                span: #0 bytes(109..110),<br>            },<br>            Ident {<br>                ident: &quot;u64&quot;,<br>                span: #0 bytes(111..114),<br>            },<br>            Punct {<br>                ch: &#39;,&#39;,<br>                spacing: Alone,<br>                span: #0 bytes(114..115),<br>            },<br>        ],<br>        span: #0 bytes(71..117),<br>    },<br>]</pre><p>You can observe that the TokenStream consists of three TokenTree elements. The first two are simple identifiers with the “struct” keyword and the struct name, and the third is a group of tokens, making up the code block specifying the struct fields.</p><p>The second dbg! call does two things: a) prints the result of parsing the input token stream, and b) returns the original value so that the dbg!(...) call doesn’t change the working of the original code.</p><p>The printed DeriveInput AST looks like this:</p><pre>[describe_derive/src/lib.rs:11:43] parse_macro_input!(input) = DeriveInput {<br>    attrs: [],<br>    vis: Visibility::Inherited,<br>    ident: Ident {<br>        ident: &quot;YourStruct&quot;,<br>        span: #0 bytes(60..70),<br>    },<br>    generics: Generics {<br>        lt_token: None,<br>        params: [],<br>        gt_token: None,<br>        where_clause: None,<br>    },<br>    data: Data::Struct {<br>        struct_token: Struct,<br>        fields: Fields::Named {<br>            brace_token: Brace,<br>            named: [<br>                Field {<br>                    attrs: [],<br>                    vis: Visibility::Inherited,<br>                    mutability: FieldMutability::None,<br>                    ident: Some(<br>                        Ident {<br>                            ident: &quot;my_string&quot;,<br>                            span: #0 bytes(77..86),<br>                        },<br>                    ),<br>                    colon_token: Some(<br>                        Colon,<br>                    ),<br>                    ty: Type::Path {<br>                        qself: None,<br>                        path: Path {<br>                            leading_colon: None,<br>                            segments: [<br>                                PathSegment {<br>                                    ident: Ident {<br>                                        ident: &quot;String&quot;,<br>                                        span: #0 bytes(88..94),<br>                                    },<br>                                    arguments: PathArguments::None,<br>                                },<br>                            ],<br>                        },<br>                    },<br>                },<br>                Comma,<br>                Field {<br>                    attrs: [],<br>                    vis: Visibility::Inherited,<br>                    mutability: FieldMutability::None,<br>                    ident: Some(<br>                        Ident {<br>                            ident: &quot;my_number&quot;,<br>                            span: #0 bytes(100..109),<br>                        },<br>                    ),<br>                    colon_token: Some(<br>                        Colon,<br>                    ),<br>                    ty: Type::Path {<br>                        qself: None,<br>                        path: Path {<br>                            leading_colon: None,<br>                            segments: [<br>                                PathSegment {<br>                                    ident: Ident {<br>                                        ident: &quot;u64&quot;,<br>                                        span: #0 bytes(111..114),<br>                                    },<br>                                    arguments: PathArguments::None,<br>                                },<br>                            ],<br>                        },<br>                    },<br>                },<br>                Comma,<br>            ],<br>        },<br>        semi_token: None,<br>    },<br>}</pre><p>For us now, the important fields of the DeriveInput value are the “ident” and the “data” fields, which contain the identifier (name) and the data (field definitions) of the struct.</p><p>The field_names string will contain the description fragment with the field names, separated by commas.</p><p>It is constructed by two nested match expressions:</p><ul><li>The outer match panics if the AST doesn’t represent a struct definition (that is, if the value of the data field is not a Data::Struct variant).</li><li>The inner match panics if the struct is a <a href="https://doc.rust-lang.org/book/ch05-01-defining-structs.html#using-tuple-structs-without-named-fields-to-create-different-types">tuple struct</a>, which doesn’t have named fields, or a <a href="https://doc.rust-lang.org/book/ch05-01-defining-structs.html#unit-like-structs-without-any-fields">unit-like struct</a>, which doesn’t have fields at all (that is, if the value of the fields field is not a Fields::Named variant).</li></ul><p>Otherwise, the inner branch creates the idents iterator which iterates over the field names.</p><p>The macro function uses the <a href="https://crates.io/crates/quote">quote crate</a>’s quote! macro in two places to create a token stream from a combination of source code and AST elements. The quote! macro is instructed to evaluate variables with the # prefix.</p><ul><li>The first quote! macro call assembles the description fragment with the field names, separated by commas. The `quote! {#(#idents), *}` part expresses the iteration based on the idents iterator. The # token does two things: a) the inner # instructs the macro to evaluate the idents variable and b) the #(...),* expresses the iteration with the comma being the separator character and the star being the repetition specifier (similarly to the iteration syntax used in declarative macros).</li><li>The second quote! macro call will assemble the “impl” block which will be returned by the macro function.</li></ul><p>There are some nuances I glossed over, but you will understand these by experimenting with more macro examples.</p><h3>Crate structure</h3><p>For this example to work, you need more than just the file containing the macro function definition.</p><p>You need three crates for a) declaring the trait, b) specifying the macro function, and c) declaring the struct and the trait derivation.</p><p>See the <a href="https://doc.rust-lang.org/book/ch19-06-macros.html#how-to-write-a-custom-derive-macro">How to Write a Custom derive Macro</a> chapter of the Rust Book for instructions about how to set up and link the three crates for the derive macro to work.</p><p>This chapter contains a simpler derive macro example, which only uses the struct name and not the field names.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65e7331a9b00" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rust-masterclass/how-to-iterate-over-rust-struct-fields-65e7331a9b00">How to iterate over Rust struct fields</a> was originally published in <a href="https://medium.com/rust-masterclass">Rust Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to create the Fibonacci (and other recursive sequences) in a declarative way with Rust macros]]></title>
            <link>https://medium.com/rust-masterclass/how-to-create-the-fibonacci-and-other-recursive-sequences-in-a-declarative-way-with-rust-macros-ec982d0b5f31?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec982d0b5f31</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[macro]]></category>
            <category><![CDATA[fibonacci-sequence]]></category>
            <category><![CDATA[domain-specific-languages]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Sat, 18 Jan 2025 11:04:05 GMT</pubDate>
            <atom:updated>2025-01-18T13:49:26.061Z</atom:updated>
            <content:encoded><![CDATA[<p>Programming is fun, metaprogramming is fun squared. E.g. declaring the Fibonacci sequence like this: “a[n] = 0, 1; …; a[n-2] + a[n-1]”. This is possible in Rust using macros.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFHoN5OsHkn39YLKcB7gRQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@_christianlambert?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Christian Lambert</a> on <a href="https://unsplash.com/photos/selective-focus-photography-of-green-succulent-pWRHIMCL41E?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p><a href="https://doc.rust-lang.org/book/ch19-06-macros.html">Macros</a> in Rust consist of code generating other code. Writing code that generates code is also known as <a href="https://en.wikipedia.org/wiki/Metaprogramming">metaprogramming</a>.</p><h3>Using macros</h3><p><em>Using</em> macros vs <em>writing</em> macros represents two different levels of Rust mastery.</p><p>Macro usage starts at the very first step of learning Rust, namely with the “Hello, World!” program:</p><pre>fn main () {<br>    println!(&quot;Hello, World!&quot;);<br>}</pre><p>The exclamation mark in the name of “printl!” indicates that it is a macro, rather than a function. With this simple example it may not be obvious why a macro is used rather than a function, like in some other programming languages. One of the reasons will be obvious if we print parameterized rather than simple text:</p><pre>fn main () {<br>    let hello = &quot;Hello&quot;;<br>    let world = &quot;World&quot;;<br>    println!(&quot;{}, {}!&quot;, hello, world);<br>}</pre><p>The effect of this program is the same as the effect of the previous one. The difference is that we have separated text formatting from content. This makes it obvious, that the println! expression is variadic, which is not possible for functions in Rust. That’s why we use a macro rather than a function.</p><p>Another example you will encounter very early in your Rust journey is vector creation. <a href="https://doc.rust-lang.org/std/vec/struct.Vec.html">Vector</a> in Rust is a fairly complex data type. The complexity is needed to make it variable in length (as opposed to fixed-size arrays). Consequently, if we want a user-friendly syntax for creating vectors from elements, we need variadic expressions. That is, we need a macro, rather than a function:</p><pre>fn main () {<br>    let vec: Vec&lt;u8&gt; = vec![1,2,3];<br>    println!(&quot;Vec: {:?}&quot;, vec);<br>}</pre><p>This macro expression has exactly the same effect as the following multi-step code:</p><pre>fn main () {<br>    let mut vec: Vec&lt;u8&gt; = Vec::new();<br>    vec.push(1);<br>    vec.push(2);<br>    vec.push(3);<br>    println!(&quot;Vec: {:?}&quot;, vec);<br>}</pre><h3>Writing macros</h3><p>You can get to a relatively high level in your Rust journey only by using, and never writing macros. After a while, though, you may start to be interested in how the macros we use daily are written.</p><p>Another way to look at macros is the angle of Domain Specific Languages (DSLs). If you think about it this way, you can say that the vec![1,2,3] expression is based on a simple DSL for vector creation. And you may think that if it is possible to create in Rust a DSL for vector creation, maybe it is possible to create more complex DSLs in a similar way.</p><p>And you are right. In the remainder of this blog post, I will explain how you can create a simple DSL for creating recursive sequences. The most famous recursive sequence is the Fibonacci sequence. Creating the Fibonacci in our DSL will look like this:</p><p>a[n] = 0, 1; …; a[n-2] + a[n-1]</p><h3>Declarative macros are based on pattern-matching</h3><p>Rust has various types of macros. One family is called “declarative”, the other is called “procedural”. The simpler of the two families is the declarative kind, which is enough for our DSL. So for now, I will ignore the more complex procedural family.</p><p>Declarative macros are based on pattern matching, in a somewhat similar way to the <a href="https://doc.rust-lang.org/book/ch06-02-match.html">match</a> control flow construct. The similarity only extends to the surface structure, and not to the syntax of the match expressions. Some differences are the following: in declarative macros</p><ul><li>the matched fragments are typically tied to metavariables, which are used in the generated code;</li><li>the matched fragments are labeled by <a href="https://veykril.github.io/tlborm/decl-macros/minutiae/fragment-specifiers.html">fragment specifiers</a>, which express the fragment’s syntactic function (e.g. block, identifier, item definition, type, etc.);</li><li>declarative macros can capture repetition.</li></ul><p>The example for declarative macros mentioned in the <a href="https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming">Rust Book</a> is a simplified version of the vec! macro mentioned above:</p><pre>macro_rules! vec {<br>    ( $( $x:expr ),* ) =&gt; {<br>        {<br>            let mut temp_vec = Vec::new();<br>            $(<br>                temp_vec.push($x);<br>            )*<br>            temp_vec<br>        }<br>    };<br>}</pre><p>The “macro_rules!” expression means, that we invoke the “macro_rules” macro, which is implemented in the compiler.</p><p>The “vec” part is the name of the macro we are defining.</p><pre>( $( $x:expr ),* ) =&gt; {\*...*\}</pre><p>The dollar sign means two things, based on the context: repetition or metavariable. In this example, the outer dollar sign, together with the brackets, the optional separator, and the repetition specifier ( $(…),*), signals repetition, and the inner dollar sign signals a metavariable.</p><p>The above part means, that the pattern we want to capture consists of zero or more expressions in a series, and the expressions are divided by commas. The expression values are tied to the $x metavariable in every cycle. The “zero or more” part is expressed by the “*” repetition specifier. Instead of the asterisk, also the “?” or the “+” repetition specifier can be used, meaning, as usual, “zero or one” or “one or more” repetitions respectively.</p><pre>$(<br>    temp_vec.push($x);<br> )*</pre><p>This is also a repetition, a repetition in the generated code. This is the counterpart of the repetition in the match expression. The number of cycles in the two repetitions is the same.</p><p>The two actions that happen in tandem in the match cycle and in the code generation cycle are the following:</p><ul><li>In the match cycle, the $x metavariable picks up a value;</li><li>In the code generation cycle, the value of the $x metavariable gets pushed onto the temporary vector.</li></ul><p>Together with the empty vector creation and the returning of the temporary vector, the generated code expresses the same multistep vector creation process we have shown earlier.</p><h3>The recursive sequence DSL</h3><p>Now let’s see, how we can implement our DSL for Fibonacci (or other recursive sequence) declaration.</p><p>The following is a simplified version of the <a href="https://veykril.github.io/tlborm/decl-macros/macros-practical.html">Fibonacci example in The Little Book of Rust Macros</a>. (If you look at the more complex example in the book, you will see that our DSL is weaker than what is presented there: our example cannot handle definitions where the continuation expression allows calculations involving element indices.)</p><p>So again, this is the definition we want to parse in a way that the method can be generalized to other recursive definitions:</p><p>a[n] = 0, 1; …; a[n-2] + a[n-1]</p><p>We saw earlier, during the explanation of the vec! macro, that collecting the sequence of initial values is easy.</p><p>Evaluating the continuation expression is also easy because it’s a valid Rust expression — provided we specify the value of a as a vector, and n as usize (the integer type which can be a vector index).</p><p>The variable names a and n are indicated by the leading a[n]= part. The specific letters do not matter. The only thing that matters is that the leading part of the definition and the continuation expression must be consistent. So the following definition should yield the same result as the previous one:</p><p>f[i] = 0, 1; …; f[i-2] + f[i-1]</p><p>So this is what we have to achieve in the generated code:</p><ul><li>Collect the initial values of the sequence into a vector and bind this vector to the vector name indicated in the leading part of the definition (in the examples above a and f ).</li><li>Bind the length of the vector to the index variable indicated in the leading part of the definition (in the examples above n and i). (The indices of the sequence elements in the continuation expression are calculated backward from the length of the vector.)</li><li>Implement a rolling update of the vector so that it always contains the same number of last elements of the sequence (in our examples n and i ).</li><li>At each recursive step, evaluate the continuation expression as a Rust expression.</li><li>Ignore the “; …;” part — it’s just a filler separating the significant portions of the sequence declaration, following the traditional sequence notation, which indicates an undefined number of elided sequence elements. (The traditional notation would use commas instead of semicolons in the filler. In the original example in The Little Book of Macros, it’s explained that we cannot use commas here because the Rust macro system couldn’t distinguish between the comma as the separator of the expression sequence and the comma as part of the literal filler.)</li></ul><p>We haven’t discussed yet what Rust data type we want to use as a return value of the sequence creation.</p><p>The straightforward approach is that the generated code creates and returns a struct on which the <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html">Iterator</a> trait is implemented so that we can collect an arbitrary number of leading elements of the iterator into a vector. After having accomplished this, we can use our DSL like this:</p><pre>let fib = recurrence![a[n] = 0, 1; ...; a[n-2] + a[n-1]];<br>let res: Vec&lt;u64&gt; = fib.take(10).collect();<br>assert_eq!(res, vec![0, 1, 1, 2, 3, 5, 8, 13, 21, 34]);</pre><p>Based on the explanation of the pattern matching above and some familiarity with the Iterator trait, the following implementation should be easy to understand:</p><pre>macro_rules! recurrence {<br>    ( $seq:ident[$idx_var:ident] = $($inits:expr),+; ...; $recur:expr ) =&gt; {<br>        {<br>        // Collect the initial elements of the sequence to a vector<br>        let mut vec: Vec&lt;u64&gt; = Vec::new();<br>        $( vec.push($inits); )*<br><br>        // Specify the iterator struct we want to return<br>        struct Recurrence {<br>            mem: Vec&lt;u64&gt;,<br>            pos: usize<br>        }<br><br>        // Implement the `next` function on the iterator struct<br>        impl Iterator for Recurrence {<br>            type Item = u64;<br>            fn next(&amp;mut self) -&gt; Option&lt;u64&gt; {<br>                // Iterate over the initial elements of the sequence<br>                if self.pos &lt; self.mem.len() {<br>                    let next_val = self.mem[self.pos];<br>                    self.pos += 1;<br>                    Some(next_val)<br>                // Compute the continuation and rolling update of the last `n` elements<br>                } else {<br>                    // Bind the vector length to the index variable (`n`)<br>                    let $idx_var = &amp;self.mem.len();<br>                    // Bind the last `n` elements of the sequence<br>                    // to the vector variable (`a`)<br>                    let $seq = self.mem.clone();<br>                    // Evaluate the continuation expression<br>                    let next_val = $recur;<br>                    // Rolling update<br>                    self.mem.push(next_val);<br>                    self.mem.remove(0);<br>                    // End of rolling update<br>                    self.pos += 1;<br>                    Some(next_val)<br>                }<br>            }<br>        }<br>        Recurrence {<br>            mem: vec,<br>            pos: 0<br>        }<br>    }<br>        };<br>}<br><br>#[test]<br>fn test_recurrence() {<br>    let fib = recurrence![a[n] = 0, 1; ...; a[n-2] + a[n-1]];<br>    let res: Vec&lt;u64&gt; = fib.take(10).collect();<br>    assert_eq!(res, vec![0, 1, 1, 2, 3, 5, 8, 13, 21, 34]);<br><br>    // Lucas numbers, see https://en.wikipedia.org/wiki/Lucas_number<br>    let lucas = recurrence![a[n] = 2, 1; ...; a[n-1] + a[n-2]];<br>    let res: Vec&lt;u64&gt; = lucas.take(8).collect();<br>    assert_eq!(res, vec![2, 1, 3, 4, 7, 11, 18, 29]);<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec982d0b5f31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rust-masterclass/how-to-create-the-fibonacci-and-other-recursive-sequences-in-a-declarative-way-with-rust-macros-ec982d0b5f31">How to create the Fibonacci (and other recursive sequences) in a declarative way with Rust macros</a> was originally published in <a href="https://medium.com/rust-masterclass">Rust Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does the Daml API support all CRUD operations? It depends on who is asking.]]></title>
            <link>https://medium.com/daml-masterclass/does-the-daml-api-support-all-crud-operations-it-depends-on-who-is-asking-90769e0cacfd?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/90769e0cacfd</guid>
            <category><![CDATA[daml]]></category>
            <category><![CDATA[smart-contracts]]></category>
            <category><![CDATA[crud]]></category>
            <category><![CDATA[ledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Sat, 02 Sep 2023 09:45:48 GMT</pubDate>
            <atom:updated>2023-09-11T07:15:57.578Z</atom:updated>
            <content:encoded><![CDATA[<p>From a technical POV, we only have Create and Read (and confusingly call these collectively “update”). Businesswise, consuming exercise is like Update and Delete. This combo is also known as the UTXO model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3FDAv1BcemVbUzcB9kihYw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@grimstad?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Håkon Grimstad</a> on <a href="https://unsplash.com/photos/hteXWSF9jA4?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><em>(I wrote about what </em><a href="https://www.digitalasset.com/developers"><em>Daml</em></a><em> ledgers contain recently in my blog posts </em><a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-its-time-to-show-it-to-everyone-9bf1eedbd131"><em>A Daml ledger tells a story — it’s time to show it to everyone</em></a><em> and </em><a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d"><em>A Daml ledger tells a story–how does the action happen?</em></a><em> If you are not closely familiar with the platfrom I recommend you take a peak at these pieces before reading this one.)</em></p><p>At an abstract level, a Daml ledger is a data storage system. All data storage systems must support the CRUD operations, where the acronym stands for Create, Read, Update and Delete, namely for data entries.</p><p>Is this true for Daml as well?</p><p>As the title indicates, the answer is yes or no, depending on whether we put the question form a business or from a technical point of view.</p><h3>What does a Daml ledger (projection) contain?</h3><p>Before answering the question, a small detour concerning terminology:</p><p>So far, I spoke about a “Daml ledger”, but strictly speaking, there is no such thing as a full, all-encompassing Daml ledger.</p><p>Physically only exist projections of an imaginary or “virtual” ledger. The projections are projections for certain ledger parties, and contain all the information said ledger party is privy to. (This behavior is called “need-to-know-basis privacy”.) The magic of the Daml platform is that all ledger projections behave exactly as if the all-encompassing full Daml ledger existed in reality, and the projections were derived from that.</p><blockquote>The magic of the Daml platform is that all ledger projections behave exactly as if the all-encompassing full Daml ledger existed in reality, and the projections were derived from that.</blockquote><p>In the rest of this blog post, I will write casually “Daml ledger”, but it always means that “a Daml ledger projection for a ledger party, pertaining to a virtual full ledger”.</p><p>Coming back to the original question: what does a ledger contain?</p><p>The answer is that a Daml ledger contains an event graph, which consists of “create” and “exercise” events as nodes, and two kinds of relationships as edges between the events:</p><ul><li>“consequence” relationships pointing from top-level exercise events, requested directly by ledger partes to their consequences (in the Daml code listed as the “do” block of the choice body)</li><li>“consuming exercise” relationships; the edge points from the “exercise” event to the preexisting “create” event, which was “consumed” by the exercise.</li></ul><p>I wrote about the contents of Daml ledgers in more detail in my blog post <a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d">A Daml ledger tells a story–how does the action happen?</a> In this blog post, you can see the following diagram, depicting the event graph of a simple ticket purchase use case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZG-nN0CEHjPUcpEx4L_Ig.jpeg" /><figcaption>The event graph of a simple ticket purchase use case</figcaption></figure><p>Usually we don’t use these precise expressions when we speak about the contents of a Daml ledger, but use the following abstractions:</p><ul><li>By “active contract” we mean “create” events, pointing to which no consuming exercise relationship was recorded so far.</li><li>By “archived contract” we mean “create” events, pointing to which a consuming exercise relationship was already recorded.</li></ul><blockquote>“Active contract” and “archived contract” are abstractions over the details of the event graph.</blockquote><p>The practical consequence for a contract (or “create” event) to be “archived” (or being the target of a “consuming exercise” relationship) is that it cannot be used in any way in future transactions. “Used” means here: exercise the choices contained in it, or being referenced by any transaction.</p><h3>In the UTXO model we don’t have data entry Update or Delete</h3><p>A Daml ledger evolves in an append-only manner. This means that no event already recorded on the ledger can be modified or deleted afterwards.</p><p>(This is true in the normal course of ledger transactions. In contrast, we have a “garbage collection” mechanism called “<a href="https://docs.daml.com/canton/usermanual/pruning.html">pruning</a>”, done by admins, in order to delete archived contracts from the ledger for resource management and GDPR compliance.)</p><p>In particular, a consuming exercise neither deletes nor modifies its target. The “consuming exercise” relationship is recorded on the exercise event as a property specifying the contract id of its target “create” event.</p><p>This model, where a ledger can only grow, is also called the UTXO model introduced by Bitcoin. (The acronym stands for Unspent Transaction Output.) There is a substantial difference though between the Bitcoin UTXO model and the Daml ledger model: in a Daml ledger an exercise is not necessarily consuming.</p><h3>Ledger update is different from data entry Update</h3><p>Please note that we use the expression “update” for Daml ledgers, but its meaning is different from the data entry Update the CRUD acronym refers to.</p><p>We call collectively “ledger update” the recording of events on the ledger, reading the contents of the ledger, and even getting the ledger time. Such kind of ledger update is expressed in the Daml smart contract language by the <a href="https://docs.daml.com/daml/stdlib/Prelude.html#type-da-internal-lf-update-68072">Update</a> monad.</p><h3>Updating and deleting business objects</h3><p>Business object updates are expressed on a Daml ledger as a combination of “consuming exercise” and “create” events.</p><p>A “create” event recorded on a Daml ledger expresses a certain state of a business object at a particular time.</p><p>The change of the business object state is expressed by the combination of two events and a relationship recorded on the ledger:</p><ul><li>an “exercise” event together with a “consuming exercise” relationship targeting the said state, and</li><li>a new “create” event with the modified properties.</li></ul><p>The two events are recorded in an atomic transaction, so for the outside world the change appears to be an update of the business object. The ordered list of events to be recorded atomically on the ledger is expressed in the “do” block of the corresponding choice body in the Daml code.</p><p>The atomicity of the ledger update is achieved in a simple way: the graph fragment expressing all the intended changes either gets appended to the ledger as a whole or nothing is appended to the ledger. (“Simple” means here simple on the surface. The mechanism of checking if all conditions hold for the update under the hood is quite involved.)</p><p>One example for such a business object “update” or state change is the transfer of the cash in the ticket purchase use case above. The “9_0. Cash” event represents the issuing of an IOU by the bank for the ticket buyer. This event gets consumed by the “11_2. Transfer” event, which is a consequence of the buyer accepting the ticket offer. The transfer event, in an atomic way, also spawns another cash creation event, this time for the event organizer, as a compensation for the purchased ticket.</p><p>The continuity of the business objects across state changes can be maintained by optional <a href="https://docs.daml.com/canton/usermanual/contract_keys.html">contact keys</a>.</p><p>Business object deletions are expressed by simply recording a “consuming exercise” relationship targeting the original “create” event. This is the case in the above example with the “10_0. TicketOffer” event.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90769e0cacfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daml-masterclass/does-the-daml-api-support-all-crud-operations-it-depends-on-who-is-asking-90769e0cacfd">Does the Daml API support all CRUD operations? It depends on who is asking.</a> was originally published in <a href="https://medium.com/daml-masterclass">DAML Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Discovering the Transformer architecture]]></title>
            <link>https://medium.com/beneficial-intelligence/discovering-the-transformer-architecture-cda3eef84d3a?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/cda3eef84d3a</guid>
            <category><![CDATA[machine-translation]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[transformer-architecture]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Thu, 17 Aug 2023 19:13:57 GMT</pubDate>
            <atom:updated>2023-08-18T07:52:15.726Z</atom:updated>
            <content:encoded><![CDATA[<p>Many Large Language Models or LLMs (like ChatGPT, BERT, LLaMA etc.) are built on the Transformer architecture, invented (originally for machine translation) in 2017 by researchers working with Google Brain and Google Research. This blog post will walk you through three steps to discover it: 1) Ramp-up: Understanding language models in general with a toy model 2) RTFM: Understand the original paper 3) Talk is cheap, show me the code: Play with the Google Trax library which contains the Transformer source code.</p><h3>Introduction and methodology</h3><h4>What are LLMs for?</h4><p>From the perspective of practical applications, LLMs can be thought of as the generalization and amalgamation of machine translation, grammar/style suggestion and web search.</p><p>Despite the fact that they are referred to as “generative” AI, LLMs don’t only excel in generating new content, but also in interpreting and restructuring existing text. (I don’t even like to use them for text generation, see my earlier post <a href="https://medium.com/beneficial-intelligence/i-want-to-use-chatgpt-as-interpreter-rather-than-generative-ai-6eea453428ae">I want to use ChatGPT as “interpreter” rather than “generative” AI</a>.)</p><p>As an illustration, I asked ChatGPT to explain what the phrase “Attention Is All You Need” (the title of the original Transformer paper, as you will see later) means outside of AI research. It gave the following, pretty impressive (though to my taste a little bit too verbose) answer:</p><blockquote>Outside of the realm of AI research, the phrase “Attention is all you need” can be interpreted more broadly as a statement about the importance of focusing one’s attention or directing one’s awareness in various contexts. While it was originally coined in the context of the “Transformer” model in AI research, it can be applied to convey the idea that by giving careful attention and consideration to a particular subject or task, you can achieve significant results or understanding.</blockquote><blockquote>In everyday life, the phrase might be used to emphasize the idea that paying close attention to details, relationships, or specific aspects of a situation can lead to effective problem-solving, insightful comprehension, or successful outcomes. It could also be used as a motivational statement, encouraging individuals to prioritize what truly matters and focus their efforts on the most essential aspects.</blockquote><blockquote>For example, in a personal or professional context:</blockquote><blockquote>1. <strong>Learning and Studying</strong>: “Attention is all you need” could be a motto for effective study habits, encouraging students to concentrate fully on their studies to grasp complex concepts and excel academically.</blockquote><blockquote>2. <strong>Relationships:</strong> The phrase might be used to convey the idea that genuine attention, active listening, and empathy are crucial for building and maintaining strong relationships.</blockquote><blockquote>3. <strong>Productivity:</strong> In a work setting, it could suggest that concentrating on the most critical tasks and avoiding distractions can lead to higher productivity and accomplishments.</blockquote><blockquote>4. <strong>Problem-Solving:</strong> The phrase could remind individuals that focusing on the root causes or essential factors of a problem can lead to more effective and efficient solutions.</blockquote><blockquote>5. <strong>Mindfulness:</strong> In the context of mindfulness and meditation, the phrase might underscore the importance of being fully present in the moment and directing one’s attention to the here and now.</blockquote><blockquote>Overall, while the phrase originates from AI research, its underlying message about the significance of attention can be applied to various aspects of life where concentration, awareness, and focus are key to achieving meaningful results.</blockquote><h4>Thesis, antithesis, synthesis</h4><p>I find the arc of development which was leading up to the invention of the Transformer architecture pretty interesting.</p><p>I can imagine that Hegel, the influential 19th century German philosopher would describe this path as an example of the triad pattern “thesis”, “antithesis” and “synthesis” he liked to use to describe the “<a href="https://plato.stanford.edu/entries/hegel-dialectics/">dialectical</a>” evolution of the “Spirit”. Using this triad pattern, the stages of the development would be the following:</p><p><strong>Thesis</strong>: Convolutional Neural Networks or CNNs are suitable for processing singleton inputs, e.g. in the case of image recognition. CNNs scan images using filters to extract features and pooling layers to distill information.</p><p><strong>Antithesis:</strong> Recurrent Neural Networks or RNNs can process streams (or as in the AI literature are called sequences) of information by applying the same function to all elements of the stream one by one, usually maintaining some inner state and also taking the previously outputted predictions as input.</p><p><strong>Synthesis:</strong> The Transformer architecture can process streams of information without the constraint of sequentiality, applying clever tricks called “positional encoding” and “self-attention”.</p><p>The “thesis”, “antithesis”, “synthesis” pattern is, of course, an abstraction, and “<a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">all non-trivial abstractions, to some degree, are leaky</a>”. The real story is rather a big dependency graph than a clear sequence of these steps. The related <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">Wikipedia article</a> gives a good overview of the related work before and after the Transformer architecture.</p><h4>Why sticking to the original paper and code?</h4><p>There are quite a few publications out there which offer alternative explanations and visualizations for the Transformer architecture rather than sticking to the original paper.</p><p>Granted, the paper is quite dense and some background knowledge helps to understand it. But to my taste, “innovative” explanations and visualizations are detractions from the original intellectual achievement without real added value.</p><p>Additionally, wrestling with the primary source flexes your muscles for digesting difficult AI papers more than taking the shortcut.</p><p>I want to offer you some help to find your way to the original concept. Paradoxically, my help will become useless as soon as you get there.</p><h4>Prerequisite knowledge</h4><p>In the following, I suppose that you understand, at least in broad strokes, how neural networks are trained with backpropagation. If this is not the case, you can catch up by reading e.g. Chris Olah’s blog post <a href="https://colah.github.io/posts/2015-08-Backprop/">Calculus on Computational Graphs: Backpropagation</a> or any material you find with a web search.</p><h3>1) Ramp-up: a toy language model</h3><p>Before the Transformer architecture and LLMs became cool, in 2015 Andrej Karpathy wrote a blog post with the title <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">The Unreasonable Effectiveness of Recurrent Neural Networks</a>.</p><p>(The title is a playful allusion to the paper <a href="https://en.wikipedia.org/wiki/The_Unreasonable_Effectiveness_of_Mathematics_in_the_Natural_Sciences">The Unreasonable Effectiveness of Mathematics in the Natural Sciences</a> written by physicist Eugene Wigner. As Google suggestions indicate, some other authors also used this formula for a catchy title.)</p><p>Karpathy’s blog post is not a singular mile stone in the AI literature, it rather references a lot of papers containing prior work he draws on, and indicates trends which he finds promising, e.g. that “The concept of attention is the most interesting recent architectural innovation in neural networks.”</p><blockquote>“The concept of attention is the most interesting recent architectural innovation in neural networks.”</blockquote><p>It’s also an enjoyable read, because it reflects the author’s passion for playing with tools the deep learning community develops. Apart from stop words, the single most frequent word in the post is “fun” (occurs ten times).</p><p>The author is also not the first researcher who experimented with character level language models, but his joy of doing it easily infects the reader to the point of forgetting to question their usefulness. (A language model is an application which predicts the next element of a series, based on the previously predicted elements. A character level language model takes a character stream as input and outputs another character stream. The author of another paper mentions text compression as a “serious” application of character level language models — and makes it clear that other, more effective methods already exist for this.)</p><blockquote>A language model is an application which predicts the next element of a series, based on the previously predicted elements.</blockquote><p>The following diagram, taken from the posts, depicts the various patterns of input and output of an RNN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OrRZ9FpVhiBk950smZkXAA.png" /></figure><p>Input vectors (which encode words, characters, images etc. as elements of an input stream) are represented by red rectangles, output vectors (also encoding some output, either a singleton element or elements of an output stream) are represented by blue rectangles, and the RNN’s hidden state (meaning different from the input and output, also a vector) is represented by green rectangles.</p><p>The arrows in the diagram represent learned functions (in fact, matrices are learnt, which are multiplied by the vectors and become functions due to the multiplication) which don’t change for the entire prediction process.</p><p>The columns of the diagram don’t represent different network elements, but different stages of the prediction process, during which only the contents of the data represented by rectangles change.</p><p>The use case of machine translation and language model, collectively also called <a href="https://en.wikipedia.org/wiki/Transduction_(machine_learning)">transduction</a>, is represented by the fourth and fifth pattern. (The difference is that in the fourth pattern the input sequence length can be different from the output sequence length.)</p><p>This representation of the RNN structure is quite sketchy. Another diagram from another paper (<a href="https://arxiv.org/abs/1508.04025">Effective Approaches to Attention-based Neural Machine Translation</a>) displays the fourth pattern (machine translation) in more detail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slX0aOE1vaCLvl1_U66eDw.png" /></figure><p>Here it is properly emphasized that the prediction process consists of two phases, an encoding phase (blue rectangles) and a decoding phase (burgundy rectangles). Again, the columns having the same color do not represent different network elements but different stages of the prediction process. The different colors indicate that the encoder component of the network (one set of blue rectangles) can be different from the decoder component (one set of burgundy rectangles) of the network (which is the case, as will will see later, with the Transformer architecture).</p><p>The structure and working mechanism of Karpathy’s toy example, the character level language model is represented by the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Ry5cnm5EwKTk2R82g6jXA.png" /></figure><p>The output layer vectors contain a probability distribution of the possible continuations of the output stream. The input vectors are in the toy example “one-hot” vectors, instead of the <a href="https://en.wikipedia.org/wiki/Word_embedding">word embedding</a> vectors of realistic applications.</p><p>The diagram displays the inner state and output vectors of the network before training, after initializing the “W_xh”, “W_hh” and “W_hy” “functions” (in fact matrices) with random values. It is expected that after training the network, in the output vectors the probabilities marked with green will be the highest, because the vectors will represent in this way the expected continuations of the character stream.</p><p>To be clear: this toy example is totally useless, its only purpose is to illustrate in general how a language model is expected to work.</p><p>In the remainder of the blog post, in the section “Fun with RNNs”, the author shows examples of actually useless but interesting “gibberish generators” (this is how I characterize them, not the author) trained on</p><ul><li>Paul Graham essays (Paul Graham is a computer scientist, essayist, founder of Y Combinator and Hacker News),</li><li>works of Shakespeare,</li><li>the Wikipedia,</li><li>an algebraic geometry text book,</li><li>the Linux source code and</li><li>baby names.</li></ul><p>All these generators spit out nonsense text which amusingly remind of their training datasets.</p><h3>2) RTFM: Attention Is All You Need</h3><p>I wrote so detailed about Karpathy’s toy character-level language model to prepare you for the real thing, which is the Transformer architecture invented by Google researchers, published two years after Karpathy’s blog post.</p><p>The Transformer architecture was described in the paper <a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>.</p><p>At one level, the title is a playful allusion to pieces of everyday wisdom.</p><p>At the factual level of the paper’s topic, it’s meaning is summed up in the abstract like this (structured as a pointed list by me):</p><ul><li>[Prior art:] The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism.</li><li>[Innovation:] We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.</li><li>[Benefit:] Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.</li></ul><p>So the attention mechanism in itself is not the innovation of the authors, the innovation is in the “all you need” part. We will focus in the following paper walkthrough on how this “all you need” innovation leads to better quality, more parallelizable computations and reduced training time.</p><p>(Parallelization is a technical detail but an important one: a higher degree of parallelization means horizontal scalability, which means that the computations can be made faster by applying more computing power to them, and utilizing the parallel computation capabilities of GPUs. Which translates into the increased volume of the training set, which translates into the higher quality of the model.)</p><h4>“Attention” has a special meaning</h4><p>In the context of Natural Language Processing or NLP research in AI, the term “attention” has a special meaning, different from the everyday use of the word.</p><p>The usefulness of attention follows from the fact that natural language texts are not signal streams, but rather serialized graph structures which are represented by linguists as phrase structures or dependency grammars.</p><p>Attention can be thought of as the “deserialization” mechanism for the serialized text graphs.</p><blockquote>Natural language texts are not signal streams, but rather serialized graph structures. Attention can be thought of as the “deserialization” mechanism for the serialized text graphs.</blockquote><p>The key technical term which explains the special meaning of “attention” is “dependency”, which is a key word also in this paper. (To be honest, I cannot tell if the word “dependency” has the same meaning here as in <a href="https://en.wikipedia.org/wiki/Dependency_grammar">linguistics</a>, but for sure it’s something similar.)</p><p>In the context of the challenge at hand, “dependency” and “attention” means the following:</p><p>The network is supposed to spit out the result of computation one token at a time (roughly word by word). For computing the next word in the output sentence, often not the next unused word in the input sentence is relevant (see “dependency”), but one or more words before or after it, possibly at a longer distance. The attention mechanism helps the model to focus on the relevant word or words for generating the next output word. Moreover, different words can be relevant from different perspectives. That’s why the Transformer model operates with several attention “heads”. One illustration for this is the following diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W6ty_PxAqdPJJwVKuRkotw.png" /></figure><h4>Why is the architecture called “Transformer”?</h4><p>I have no idea. I couldn’t find any clue for this neither in the paper nor in any online source.</p><p>When I asked ChatGPT, it pretended to give a definitive answer, but the anser is, unfortunately, not convincing:</p><blockquote>The Transformer model architecture is called “transformer” because it utilizes a specific type of attention mechanism called “self-attention” to transform and process input data in a highly parallelizable and efficient manner. (…) The name “transformer” reflects the transformative nature of this attention mechanism in capturing complex patterns and dependencies within sequences. (… Blah, blah, blah…)</blockquote><p>If you manage to find out the real answer, please let me know.</p><h4>The model architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CUeVcFFBclKsH504JraLQg.png" /></figure><p>This is the famous diagram illustrating the Transformer model architecture. I will structure the remainder of this section by questions you may have about this diagram.</p><h4>What do the two parts of the diagram illustrate?</h4><p>The left-hand module is the encoder, the right-hand module is the decoder.</p><p>The encoder takes the input sequence and transforms it into a representation called generally Context (the expression is not used in this paper). The Context vector then is passed to the decoder, which generates the output.</p><p>The relationship between encoder and decoder can be understood from the diagram above which I also insert here. The blue rectangles represent the encoder, the burgundy rectangles represent the decoder. Please note that the diagram shows an RNN which processes information sequentially, which is not true fro the Transformer, so the diagram does not fully fit to the Transformer architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*slX0aOE1vaCLvl1_U66eDw.png" /></figure><h4>What is the input to the encoder?</h4><p>The input is a representation of the entire input sentence. This is the major innovation of the Transformer architecture as opposed to RNNs, where the input is a stream of words, one word at a time.</p><h4>Why is the input to the decoder “Outputs (shifted right)”?</h4><p>This can also be understood, mutatis mutandis, from the above diagram. The decoder takes as input not only the context vector from the encoder but also the previously generated output, shifted, meaning that the first element is the “end of sentence” token.</p><h4>What is the output of the decoder?</h4><p>It’s a sequence of probability distributions for the elements of the output over the words of the dictionary.</p><h4>What is input and output embedding?</h4><p>In NLP applications, words are represented by vectors, which magically encode the meaning of the words. By “magically” I mean that if you look at the numbers contained in the vectors, you cannot attach any meaning to them, still, vectors for words with similar meanings stack together in the vector space, and differences of vector pairs can express analogies.</p><p>Word embedding is a complex topic in itself, but has been used for a much longer time than the Transformer architecture. This e.g. is a blog post by Chris Olah from which you can understand what word embeddings are for are how they can be created: <a href="https://colah.github.io/posts/2014-07-NLP-RNNs-Representations/">Deep Learning, NLP, and Representations</a>.</p><h4>What is positional encoding?</h4><p>Positional encoding is a representation of the position of a word in the sentence. It’s needed because the model doesn’t contain any recurrence or convolution.</p><p>Positional encoding is a vector having the same length as the word embedding. The vector members are computed by sine and cosine functions of different wavelength, corresponding to word position and vector element index, ad described in section 3.5 of the paper.</p><p>The positional encoding vector is added to the word embedding vector of the inputs of the encoder and decoder module.</p><h4>Why do Multi-Head Attention layers have three-way fork shaped inputs?</h4><p>The three branches of the fork represent three learned matrices. The matrices are multiplied with the input vector and the products magically encode a Query vector and a set of Key-Value pair vectors. By “magically” I mean, similarly to word embeddings, that the numbers in the vectors won’t convey any meaning in themselves, but the Query vector behaves as if it was expressing a question (e.g. Who, Where, When, Why etc), and the Key-Value pairs behave as if they were expressing potential answers to the question.</p><p>The computation of attention for a certain word mimics a query against a key-value store in a differentiable way (differentiability is needed for backpropagation). First the query and the key vector is combined which returns a probability distribution of the relevance of the keys for the query (mimics picking the right key corresponding to the query), than this probability distribution is multiplied with the value vector which returns a weighted combination of the possible values (mimics returning the value corresponding to the chosen key).</p><p>The computation of one attention “head” and the ensemble of heads as a multi-head attention layer is represented by these diagrams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zhrw6YNdu5E5WjvEfJv3bw.png" /></figure><p>The scaled dot-product attention is almost the same as described in the previous paper <a href="https://arxiv.org/abs/1409.0473">Neural Machine Translation by Jointly Learning to Align and Translate</a>, just the scaling factor is added. (This is also a difficult paper, I don’t want to pretend that I fully understand it, but the diagrams can help to understand what’t going on.)</p><h4>Which components come from where in the decoder multi-head attention layer?</h4><p>I mean the right side of this part of the diagram:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQqD0XQhMKsCr-xkVEjA7w.png" /></figure><p>The answer is that the Query component comes from the decoder itself, and the Key and Value component comes from the encoder.</p><p>The intuition for this is that a “question” is asked by the decoder as to what the next word of the output should be, and it tries to find the answer to this in the context outputted by the encoder.</p><h4>Why does one of the multi-head attention layers in the decoder say “Masked Multi-Head Attention”?</h4><p>Masking is used during training to pretend that the input is a perfectly but partially translated sentence by masking out the remainder of the input sentence.</p><h4>What do the “Add &amp; Norm” and the “Feed Forward” layers do?</h4><p>They add residual connections and normalization in order to improve learning efficiency.</p><h4>What does “Nx” mean next to the encoder and decoder block?</h4><p>This means that the same structure is repeated N times. In the case of the Transformer paper N=6.</p><p>The first layer combines words, the second layer combines pairs of words, the third layer pairs of pairs of words etc.</p><h4>[You can ask me:] Where can I find additional insights and material?</h4><p>You can find additional insights and material all over the Internet, e.g. in this StackExchange thread: <a href="https://stats.stackexchange.com/questions/421935/what-exactly-are-keys-queries-and-values-in-attention-mechanisms">What exactly are keys, queries, and values in attention mechanisms?</a> I especially recommend this lecture by prof Pascal Poupart mentioned in one of the answers: <a href="https://www.youtube.com/watch?v=OyFJWRnt_AY">CS480/680 Lecture 19: Attention and Transformer Networks</a>.</p><h3>3) Talk is cheap, show me the code!</h3><p>You can play around with the Transformer architecture using its source code. The code is now available as part of the <a href="https://trax-ml.readthedocs.io/en/latest/index.html">Trax</a> library which is maintained by the <a href="https://research.google/teams/brain/">Google Brain</a> team.</p><h4>Translation with varying temperature</h4><p>You can try the English-German translation with a pre-trained Transformer network in a Google Colab <a href="https://colab.research.google.com/github/google/trax/blob/master/trax/intro.ipynb">notebook</a>. There is one parameter you can play with, “temperature”. Temperature is actually a parameter of the softmax layer. Higher temperature yields more diverse results.</p><p>This English input sentence e.g.: “It is nice to learn new things today!” is translated differently into German with different temperatures:</p><ul><li>With the default temperature=0.0 value it will produce: “Es ist schön, heute neue Dinge zu lernen!”</li><li>With temperature=0.5, the translation will be: “Es ist schön, heute Neues zu lernen!”</li></ul><p>As you can see, the translation with higher temperature follows less strictly the words of the original (doesn’t contain “Dinge” which is the equivalent of “things”) but sounds more German.</p><h4>Implementation source code</h4><p>The source code of the Transformer model can be found in <a href="https://github.com/google/trax/blob/1bb3b89427f669f2f0ec84633952e21b68964a23/trax/models/transformer.py#L267">this</a> file in the <a href="https://github.com/google/trax/tree/1bb3b89427f669f2f0ec84633952e21b68964a23">Trax</a> Github repo.</p><p>The Transformer function returns the following model, where the comments mark the encoder and decoder block :</p><pre>return tl.Serial(<br>      tl.Select([0, 1, 1]),  # Copies decoder tokens for use in loss.<br><br>      # Encode.<br>      tl.Branch([], tl.PaddingMask()),  # tok_e masks tok_d tok_d<br>      _Encoder(),<br><br>      # Decode.<br>      tl.Select([2, 1, 0]),  # Re-orders inputs: tok_d masks vec_e .....<br>      tl.ShiftRight(mode=mode),<br>      out_embedder,<br>      _Dropout(),<br>      tl.PositionalEncoding(max_len=max_len, mode=mode),<br>      tl.Branch([], tl.EncoderDecoderMask()),  # vec_d masks ..... .....<br>      [_EncDecBlock() for _ in range(n_decoder_layers)],<br>      tl.LayerNorm(),<br>      tl.Select([0], n_in=3),  # Drops masks and encoding vectors.<br><br>      # Map vectors to match output vocab size.<br>      tl.Dense(output_vocab_size),<br>  )</pre><p>The Transformer layers are combined with the Serial and Branch <a href="https://trax-ml.readthedocs.io/en/latest/notebooks/layers_intro.html#Layers-combine-into-layers.">combinators</a> for sequential and parallel computations respectively.</p><p>You can dig deep in the source code to understand the layers. E.g. the encoder block is defined like this (but as you can see, you have to dig even deeper to find out what _EncBlock is etc.):</p><pre>def _Encoder():<br>    encoder = tl.Serial(<br>        in_embedder,<br>        _Dropout(),<br>        tl.PositionalEncoding(max_len=max_len, mode=encoder_mode),<br>        [_EncBlock() for _ in range(n_encoder_layers)],<br>        tl.LayerNorm(),<br>    )<br>    return tl.Cache(encoder) if mode == &#39;predict&#39; else encoder</pre><p>A documentation for the Trax layers can be found <a href="https://trax-ml.readthedocs.io/en/latest/notebooks/layers_intro.html#">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cda3eef84d3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/beneficial-intelligence/discovering-the-transformer-architecture-cda3eef84d3a">Discovering the Transformer architecture</a> was originally published in <a href="https://medium.com/beneficial-intelligence">Beneficial Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I want to use ChatGPT as “interpreter” rather than “generative” AI]]></title>
            <link>https://medium.com/beneficial-intelligence/i-want-to-use-chatgpt-as-interpreter-rather-than-generative-ai-6eea453428ae?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/6eea453428ae</guid>
            <category><![CDATA[thinking-fast-and-slow]]></category>
            <category><![CDATA[andrej-karpathy]]></category>
            <category><![CDATA[barbie]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Sat, 12 Aug 2023 10:51:36 GMT</pubDate>
            <atom:updated>2023-08-12T12:04:43.119Z</atom:updated>
            <content:encoded><![CDATA[<p>I don’t want ChatGPT to ghost write form me. I want to use it as “System 1.5”, as an assistant interpreter, rather than outsourcing my slowly and deliberately thinking “System 2” to it.</p><h3>Bullshit generator</h3><p>Closing his <a href="https://www.youtube.com/watch?v=bZQun8Y4L2A">State of GPT</a> conference presentation, Andrej Karpathy, one of the leading figures in AI development today, entertained his audience with an “inspiring” message generated by the platform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OqIqauGODhE2-EZIX0m2Aw.png" /></figure><p>His thought process might have been: if it’s called “generative” AI, the best way to demonstrate its capabilities is to let it <em>generate</em> something.</p><p>I don’t know about you, but to my taste this “inspire the audience” message is pure bullshit. Not a good selling point for me.</p><h3>Prompt engineering</h3><p>The presentation otherwise is very informative and enjoyable, explaining what ChatGPT does and how it is trained and fine tuned, and giving practical advice on prompting it to “think” step by step with instructions of the style “what would you tell a task contractor if they can’t email you back?”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fcNt1cVIHN6RtCFtUGYTQQ.png" /></figure><h3>Thinking, Fast and Slow</h3><p>The distinction between the two “systems” of human thinking, the fast, automatic “System 1” on one hand, and the slow, deliberate “System 2” on the other hand, introduced by Daniel Kahneman in his book <a href="https://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman-ebook/dp/B005MJFA2W/">Thinking, Fast and Slow</a>, is a recurring topic of theorizing about what different AI systems do.</p><p>Here, Karpathy uses the analogy to explain his recommendation for the “Tree of Thought” method of “prompt engineering”, comparing the idea to tree search algorithms used in game-playing AI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l5juOI0J6U0DDkQwE2q0wQ.png" /></figure><h3>Do I want ChatGPT to think instead of me?</h3><p>At this point, the conscious consumer awakens in me and asks: do I really want to outsource my “System 2” to ChatGPT? Or to any software for that matter?</p><p>There are some advantages of doing so:</p><ul><li>I can make it write an “essay” on “any topic” in no time.</li><li>(Unfortunately no other advantage comes to my mind.)</li></ul><p>On the other hand, why I write anything on any topic? There are a couple of reasons:</p><ul><li>I want to understand things better.</li><li>I want to flex my reasoning and composing muscles.</li><li>I want other people to understand things from a point of view, which is relevant at the same time for them and for me.</li></ul><p>None of these points justifies using a shortcut instead of doing the work myself.</p><p>In fact, this conclusion is in line with Karpathy’s advice: I would never ask a task contractor (wether they can email back or not) to ghost write for me.</p><h3>System 1.5, the “interpreter”</h3><p>On the other hand, there are some cases when I want to comb through and interpret material which is not as well structured as I would like it to be. Understanding and structuring text is also flexing some muscles, but, just like lifting heavy objects, can require too much time and effort for the result.</p><p>So what if I thought about a “System 1.5” lying halfway through between “System 1” and “System 2”, working slowly but passively, doing the hard work of structuring and interpreting existing material produced by others?</p><p>It turns out that ChatGPT does a decent job with that, yielding the sweet spot for me. Let’s look at an example.</p><p>Creating a Deep Neural Network for sentiment analysis is an introductory exercise in Natural Language Processing. One popular training field is the <a href="https://www.tensorflow.org/datasets/catalog/imdb_reviews">IMDB review</a> dataset in TensorFlow. The aim of the exercise is to classify movie reviews into positive and negative ones.</p><p>With ChatGPT we can do more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*CpEyJxMyZgYn_XqpVmquMw.jpeg" /></figure><p>Let’s look at a <a href="https://www.imdb.com/review/rw9212352/?ref_=tt_urv">review</a> about the Barbie movie (fell free to skip this version, later you will see a much more digestible version):</p><blockquote>The first thing you need to know about Barbie is that Ryan Gosling is the best part of the entire film. For the duration he is over the top, dumb and hilarious. Every scene he was in had me laughing and I enjoyed just about everything he did in the film.</blockquote><blockquote>Margot Robbie is a great choice for Barbie and as usual puts in a fantastic performance. She hit pretty much every mark in my opinion and I can’t see anyone taking her place in this role.</blockquote><blockquote>Where the film falls flat unfortunately is not the characters, actors or even humour for that matter. It’s the incessant political lecturing.</blockquote><blockquote>It’s no surprise that this film takes a strong feminist angle. That’s to expected. Barbie was a doll that portrayed many different professions in an attempt to show young girls that they could strive for me than what they were told they could do. But that was the 1950s. This is 2023.</blockquote><blockquote>By no means is the world a perfect representation of equality today, but it is so vastly superior in that category than it ever has been that the theme of the film falls a little flat. Telling young girls to go after their dreams and not be beholden to any stereotypes is a positive, wonderful message and one that could’ve been portrayed so much better. But instead, the film teeters on the edge of a radical feminist position.</blockquote><blockquote>I’ve seen a lot of people say the film hates men. I disagree with this. I don’t think it hates men. Instead, it takes a very simplistic, black and white view of Western culture and assumes that women’s lives are overall difficult and men’s lives are overall quite easy. When in reality, life is tough. For everyone. At least 99% of us. Claiming otherwise is unjust.</blockquote><blockquote>However, it’s not just the political stance it takes that brings the film down. It’s how incessant it is. They are without any shame pushing a political message through the screen down your throat every 10 minutes. It’s unbearable at times. Going to the cinema or watching a film should be an enjoyable experience, not a lecture.</blockquote><blockquote>This film could’ve been beloved by all genders and ages. Both men and women love Margot Robbie and Ryan Gosling. This feels like an easy lay up for the filmmakers but by going the political route they end up having many jokes go over kids heads and alienate many of the people who where genuinely excited to see this film.</blockquote><blockquote>I give it a 6/10 because of Gosling and most of the humour. It was enjoyable overall. I think Hollywood needs to stop employing multi millionaire actors to lecture the general public on how to live. It’s embarrassing.</blockquote><p>It’s long and I don’t know in advance how valuable it is for me. The score is 6/10 which is a hint that it can be useful for me, because I want to learn both positive and negative opinions about the film. But is the result worth the effort?</p><p>Responding to the obvious prompt (“Separate the positive and negative elements of the following Barbie movie review”), ChatGPT does the favor of separating the positive and negative elements of it:</p><blockquote>Positive Elements:</blockquote><blockquote>- Ryan Gosling’s performance is described as the best part of the film, over-the-top, dumb, and hilarious.<br>- Every scene with Ryan Gosling had the reviewer laughing and enjoying his performance.<br>- Margot Robbie is praised as a great choice for Barbie, delivering a fantastic performance and hitting all the right marks.<br>- The theme of encouraging young girls to pursue their dreams and challenge stereotypes is acknowledged as a positive and wonderful message.</blockquote><blockquote>Negative Elements:</blockquote><blockquote>- The film is criticized for its incessant political lecturing, which is seen as a drawback.<br>- The strong feminist angle of the film is noted, and while the message is positive, the film is criticized for teetering on the edge of a radical feminist position.<br>- The reviewer disagrees with the film’s portrayal of men and women’s lives in a simplistic, black and white view of Western culture.<br>- The film’s excessive and constant pushing of a political message is deemed unbearable, turning the viewing experience into a lecture.<br>- The decision to go the political route is seen as alienating and resulting in many jokes going over kids’ heads.</blockquote><blockquote>Mixed Elements:</blockquote><blockquote>- While the reviewer enjoys the performances of Gosling and most of the humor, they give the film a 6/10 rating.<br>- The potential for the film to be beloved by all genders and ages is acknowledged, but the political approach is seen as a missed opportunity.<br>- The idea of Hollywood employing millionaire actors for societal lectures is criticized, suggesting it’s embarrassing.</blockquote><blockquote>Overall, the review highlights positive performances by the actors, especially Ryan Gosling and Margot Robbie, but criticizes the film’s heavy-handed political messaging and its potential to alienate and detract from the enjoyment of the viewing experience.</blockquote><p>Now the text is much easier to comb through and I can easily find the relevant points for me (with which I may or may not agree).</p><p>Following a hint given by Karpathy, I also asked ChatGPT to structure the answer as a JSON document (“Put the positive and negative elements of the following Barbie movie review into a JSON document”). The result is pretty good, with the format being machine-readable for further processing:</p><pre>{<br>  &quot;Positive_Elements&quot;: {<br>    &quot;Ryan_Gosling&quot;: {<br>      &quot;Description&quot;: &quot;Best part of the film, over-the-top, dumb, hilarious&quot;,<br>      &quot;Scenes&quot;: &quot;Every scene with him had the reviewer laughing and enjoying&quot;<br>    },<br>    &quot;Margot_Robbie&quot;: {<br>      &quot;Description&quot;: &quot;Great choice for Barbie, fantastic performance&quot;,<br>      &quot;Opinion&quot;: &quot;Hit every mark, irreplaceable in the role&quot;<br>    },<br>    &quot;Message&quot;: {<br>      &quot;Description&quot;: &quot;Encouraging young girls to pursue dreams and challenge stereotypes&quot;,<br>      &quot;Opinion&quot;: &quot;Positive, wonderful message&quot;<br>    }<br>  },<br>  &quot;Negative_Elements&quot;: {<br>    &quot;Political_Lecturing&quot;: &quot;Incessant and overbearing political lecturing&quot;,<br>    &quot;Feminist_Angle&quot;: {<br>      &quot;Description&quot;: &quot;Strong feminist angle, expected due to Barbie&#39;s history&quot;,<br>      &quot;Opinion&quot;: &quot;Teeters on the edge of a radical feminist position&quot;<br>    },<br>    &quot;Simplistic_Portrayal&quot;: {<br>      &quot;Description&quot;: &quot;Simplistic view of gender roles and equality&quot;,<br>      &quot;Opinion&quot;: &quot;Disagreement, life is tough for everyone&quot;<br>    },<br>    &quot;Excessive_Politics&quot;: {<br>      &quot;Description&quot;: &quot;Excessive and constant pushing of a political message&quot;,<br>      &quot;Opinion&quot;: &quot;Unbearable, turns viewing into a lecture&quot;<br>    },<br>    &quot;Alienation&quot;: {<br>      &quot;Description&quot;: &quot;Political approach alienates viewers&quot;,<br>      &quot;Opinion&quot;: &quot;Jokes go over kids&#39; heads, alienates excited audience&quot;<br>    }<br>  }<br>}</pre><h3>Conclusion</h3><p>If you didn’t try ChatGPT so far because of thinking that the “generative AI” genre is not relevant to you, you can try using it as an interpreter assistant.</p><p>You can even think about business models where the structuring and analysis of a large amount of text can be valuable.</p><p>Based on the example above, I offer one idea to IMDB for free: extending its website with the option to see the movie reviews in this structured format.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6eea453428ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/beneficial-intelligence/i-want-to-use-chatgpt-as-interpreter-rather-than-generative-ai-6eea453428ae">I want to use ChatGPT as “interpreter” rather than “generative” AI</a> was originally published in <a href="https://medium.com/beneficial-intelligence">Beneficial Intelligence</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bridging the business-IT gap — how I can help thrive your digital business]]></title>
            <link>https://gyorgybalazsi.medium.com/bridging-the-business-it-gap-how-i-can-help-thrive-your-digital-business-ffc73d78144a?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffc73d78144a</guid>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 12:37:16 GMT</pubDate>
            <atom:updated>2023-09-03T20:51:31.206Z</atom:updated>
            <content:encoded><![CDATA[<h3>Bridging the business-IT gap — how I can support your digital business</h3><p>I am a tech/business consultant and developer, with experience and toolset for solution design, development, metaprogramming, data analysis, documentation, product management, product ownership, content marketing, consulting.</p><p>In this blog post, I will share the top challenges with bridging the gap between the execution of business strategy vs IT solutions I saw in my projects so far, and the way I can help you to tackle them.</p><p>A preview of upcoming attractions:</p><ul><li>Challenge #1: Integrate your multiparty business model ← smart contract based transaction systems</li><li>Challenge #2: Separate the signal from the noise within the apparent paradox of seeing the AI hype and doomerism at the same time ← understanding the fundamentals, business model design</li><li>Challenge #3: Showcasing your company’s expertise in the blogosphere ← content marketing, coaching</li><li>Challenge #4: Without a second opinion next to your vendors’ claims and recommendations, you are flying blind ← unbiased assessment of trade-offs</li><li>Challenge #5: POCs are not always useful ← POC design and evaluation</li><li>Challenge #6: Finding experts for a niche, e.g. smart contract development ← metaprogramming</li><li>Challenge #7: Not many programming languages are business friendly ← Rust</li><li>Challenge #8: Business/legal stakeholders and IT experts don’t speak the same language ← show, don’t tell</li><li>Challenge #9: The potential of digital collaboration tools is not fully exploited ← hacks for seamless idea/information transfer</li><li>Challenge #10: Eliciting early feedback from business stakeholders ← demo hacks</li><li>Challenge #11: Missing the forest for the trees with ticket management software ← solution design</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dt7dSziNaEOgCuSh2qTPfg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@boliviainteligente?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">BoliviaInteligente</a> on <a href="https://unsplash.com/photos/AXNZb4FEkh8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Challenge #1: Integrate your multiparty business model</h3><p>Nowadays more often than not, services are bundled, combining multiple service providers’ contributions. So business workflows should be composed, but implementing such composed workflows, spanning silos within and across organizations is far from being trivial.</p><h4>How I can help: smart contract based transaction systems</h4><p>Several platforms are on the market offering business workflow composition, labelled as “blockchain”, “DLT”, “smart contract”, “interoperability”, “distributed application platform” etc.</p><p>One of these platforms is the <a href="https://www.digitalasset.com/developers">Daml</a>/<a href="https://www.canton.network/">Canton</a> platform, offered by Digital Asset. I’m a certified Daml developer. I’ve been working for more than two years on Daml projects in partnership with <a href="https://rtledgers.com/">Rethink Ledgers</a>, a blockchain consultancy and developer company.</p><p>I’m also an advocate and educator of Daml, see my latest (and so far the best) blog post about it: <a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d">A Daml ledger tells a story–how does the action happen?</a></p><p>Besides Daml, I can help you with other blockchain platforms like <a href="https://ethereum.org/en/">Ethereum</a>, <a href="https://corda.net/">Corda</a> and the <a href="https://corda.network/">Corda network</a>, <a href="https://www.hyperledger.org/use/fabric">Hyperledger Fabric</a> and others.</p><h3>Challenge #2: Separate the signal from the noise within the apparent paradox of seeing the AI hype and doomerism at the same time</h3><p>The AI arms race kicked into overdrive recently. At the same time, AI doomerism became part of mainstream media and scientific discourse. Some products like ChatGPT and other language models offer huge benefits, and sometimes prove to be disappointingly dumb. How can you separate the signal from the noise within this whirlwind of apparently contradicting trends?</p><h4>How I can help: understanding the fundamentals, business model design</h4><p>This topic needs to be approached from two angles at the same time: assessing the potential of the technology, and being able to spot the business opportunities the technology provides.</p><p>I study intensively the mathematical fundamentals and practical challenges of the AI technology. I’ve completed the <a href="https://coursera.org/share/e10c8eae428664da5db86ac2e6bfe678">Deep Learning specialization</a> by DeepLearning.AI, and I’m working towards my <a href="https://coursera.org/share/e780204eceae77cfac57f9ae21b26708">Tensorflow developer certification</a>.</p><p>See my recent blog post <a href="https://medium.com/beneficial-intelligence/discovering-the-transformer-architecture-cda3eef84d3a">Discovering the Transformer architecture</a> and others in my <a href="https://medium.com/beneficial-intelligence">Beneficial Intelligence</a> collection.</p><p>I also have experience and a toolset for business strategy and business model design.</p><p>Currently I put my two cents in two trends and one missing puzzle piece:</p><ul><li>Trend #1: The combination of smart contract based transaction networks and AI. These two disciplines are like yin and yang. Smart contracts offer strong legal foundations for the collaboration of independent parties. AI offers effective private strategies within the limits drawn by the common frameworks.</li><li>Trend #1: Graph neural networks, currently used e.g. to building fraud detection systems and forecasting. Large language models are dumb because they don’t have a world model. Knowledge graphs offer just that extension.</li><li>Missing piece: Intelligibility of smart contract code. Smart contracts are already quite good at encoding rights and obligations, and the underlying transaction systems are good at enforcing these rights and obligations. What is missing is the legal prose equivalent of the smart contract code. Without this, it’s hard to speak about a “meeting of minds” which is an indispensable element of contract law. I hope that language models can help with generating the correct prose equivalent of smart contract code.</li></ul><h3>Challenge #3: Showcasing your company’s expertise in the blogosphere</h3><p>Being able to express your vision and ideas about business and technology topics clearly and concisely has undeniable benefits and can even impact your bottom line.</p><h4>How I can help: content marketing, mentoring</h4><p>There are some business executives who are also excellent bloggers. If you are one of them, I want to learn from you. If not, maybe I can help you with either writing for your company or coaching you for better writing.</p><p>Before entering the IT industry, I worked for more than a decade as a political journalist. At the time when mobile content became a thing, initially in the form of text messages and WAP sites, I joined a telco company for writing short content, and I also became an expert of expressing ideas in a short, clear and concise way.</p><p>I am a <a href="https://medium.com/daml-masterclass">blogger</a>, published several blog posts on the <a href="https://blog.digitalasset.com/developers/author/daml-community-member-gy%C3%B6rgy-bal%C3%A1zsi-1">Digital Asset blog</a>, and did content marketing for other clients.</p><h3>Challenge #4: Without a second opinion next to your vendors’ claims and recommendations, you are flying blind</h3><p>If you have chosen a certain IT platform to support your business strategy, it certainly aligns with your main objectives. On the other hand, for the implementation details there are always different options with their specific trade-offs where the optimal solution for you may or may not perfectly align with your vendors’ priorities and incidental blind spots. If you don’t have in-house experts to uncover these trade-offs in an unbiased way, you are vulnerable to your vendor’s product strategy and cost-benefit expectations.</p><h4>How I can help: unbiased assessment of trade-offs</h4><p>I am an independent consultant so I can work with you for your best interest.</p><h3>Challenge #5: POCs are not always useful</h3><p>POC stands for “proof of concept”. In order for it to be useful, a POC needs to challenge some concept relevant to your project and needs to be evaluated properly.</p><p>This is theory. In real-life projects, for various reasons, sometimes demos which yield nothing else than show something that works, are passed off as POCs.</p><h4>How I can help: POC design and evaluation</h4><p>A POC, in order to be useful, needs to be designed so that it highlights some trade-offs important for the project, and yields material to evaluate those trade-offs from a business perspective.</p><h3>Challenge #6: Finding experts for a niche, e.g. smart contract development</h3><p>Example: it’s a challenge to find experts for a certain paradigm of smart contract development because it’s sometimes too “legal” for programmers and too “IT” for lawyers. I mean that smart contract paradigm which aims at expressing and enforcing rights and obligations by data recorded on a shared virtual ledger, together with computer code linked to pieces of data.</p><h4>How I can help: metaprogramming</h4><p>One example: a large part of a smart contract model is the implementation of a static data model. That part of the code is actually boilerplate and can be generated from any machine-readable representation of the business requirements. In this way, you can concentrate your scarce resources to the parts of the model which really matter, the implementation of the ledger updates.</p><h3>Challenge #7: Not many programming languages are business friendly</h3><p>In many cases, the choice of the programming language(s) used in your project is predetermined by some external factor. If this is not the case and you have freedom in choosing your language, there are at least the following factors to consider:</p><ul><li><strong>Expressivity</strong>. Meaning, how you can represent high level real-life and business concepts in your data types. The highest level of expressivity in programming languages currently is the support for “<a href="https://en.wikipedia.org/wiki/Algebraic_data_type">algebraic data types</a>” (ADTs). Trade-off: ADTs are supported by functional languages, but these can be too slow and memory hungry for some applications.</li><li><strong>Speed</strong>. There are many high-level programming languages out there but they are not famous for speed.</li><li><strong>Safety.</strong> Systems programming languages are fast, but some of them lack the required type safety (see null pointers), memory safety or thread safety.</li></ul><h4>How I can help: Rust</h4><p>Rust is a popular programming language which is expressive, fast and safe at the same time. It borrows concepts from both functional and older systems programming languages.</p><p>The only objection against using Rust that it’s not very easy to learn, because of some novel concepts it introduced for memory management.</p><p>I am intensively learning Rust, and I can help your organization embrace it.</p><p>See my blog post about why Rust is a suitable client language for a Daml ledger: <a href="https://medium.com/daml-masterclass/rust-daml-a-biz-friendly-smart-contract-platform-deserves-a-biz-friendly-client-language-ddb74922a263">Rust + Daml: a biz-friendly smart contract platform deserves a biz-friendly client language</a></p><h3>Challenge #8: Business/legal stakeholders and IT experts don’t speak the same language</h3><p>When business stakeholders, legal experts and IT experts use the same words like “transaction”, “platform”, “service” etc. sometimes they mean different things and may not even notice it.</p><h4>How I can help: show, don’t tell</h4><p>During the years, I developed a natural “comparative linguistics” for the language of different communities working together in business/IT projects.</p><p>On top of reflecting on the use of language, other communication tools like diagrams, design documents and demos help a lot to establish a “ubiquitous language” (see Domain Driven Design) for the project.</p><h3>Challenge #9: The potential of digital collaboration tools is not fully exploited</h3><p>Digital collaboration tools potentially have the power of seamless idea and information transfer from requirements to solution design to implementation in a consistent, reproducible way. The sobering reality is often that this information flow is scattered with manual steps, the need for ad hoc interpretation and version mismatch. Excel is still the single most often used digital tool in the office, and is often used as a free-form drawing canvas recording unstructured information.</p><h4>How I can help: hacks for seamless idea/information transfer</h4><p>It’s not what tools you use but how you use them.</p><p>“Digital” should mean machine-readable, self-explanatory and automatically updated.</p><p>The solution can be as simple as a human-editable and machine-readable, well-structured Excel layout, <a href="https://miro.com/marketplace/mindmapdownloader/?backUrl=%2Fmarketplace%2F">extracting</a> Miro mind maps to structured CSV, or as advanced as Jetbrain’s <a href="https://www.jetbrains.com/mps/">Meta Programming System</a>, a platform for creating business-oriented domain specific languages.</p><h3>Challenge #10: Eliciting early feedback from business stakeholders</h3><p>Humans interact with your application via a UI. Such interactions are necessary from the early stages of your project for eliciting feedback from the business stakeholders. But building a full-fledged UI at the time when the overall functionality of the system is still fluid means either that you will have to rewrite your UI many times which is wasted effort or that you get trapped in a UI which doesn’t fit to the final version of your application.</p><h4>How I can help: demo hacks</h4><p>There are many ways to demonstrate the functionality of your system between the two bad extremes of showing incomprehensible JSONs coming back from Postman transactions and building UI too early.</p><p>Some examples are feeding filtered JSON result of API calls into Python Pandas dataframes which look nice in Google Colab or Jupyter notebooks, or using the Neo4J graph database UI to demonstrate Daml <a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d">event graphs</a>.</p><h3>Challenge #11: Missing the forest for the trees with ticket management software</h3><p>It’s a fulfilling experience to mark tasks completed for the individual contributors, and watch burn-down charts for people in charge for projects. This might be the reason for the undeserved popularity of ticket management software beyond its proper application domain, which is, well, ticket management.</p><p>Granted, the work won’t do itself, and projects need visibility about the meaningful units of work, about their dependencies, and about who will do them and by when. But focusing only on the “tickets” means we mistake the desired business result for the price of getting there, and give the false impression that there is a one-to-one correspondence between business requirements and implementation todo items.</p><h4>How I can help: solution design</h4><p>As an intermediary work between properly elicited business requirements and implementation tasks, solution design documents need to be written and kept up-to-date with the changing requirements and insights gained during implementation. This document should be the anchor place for todo items for the work to be done.</p><p>For the documentation part there are various options:</p><ul><li>The implementation can be as simple as having a solution design document published as a Google Docs/Sheet and personal todo items attached to the appropriate sections.</li><li>You can use a proper product management software where the feature roadmap takes the center stage.</li><li>Or you can even use your familiar ticket management software, with the tickets linked to a proper solution design document shared with all stakeholders.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffc73d78144a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Daml ledger tells a story–how does the action happen?]]></title>
            <link>https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c2b5c2f866d</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[neo4j]]></category>
            <category><![CDATA[monads]]></category>
            <category><![CDATA[daml]]></category>
            <category><![CDATA[smart-contracts]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Mon, 17 Jul 2023 10:16:00 GMT</pubDate>
            <atom:updated>2023-07-19T07:27:31.405Z</atom:updated>
            <content:encoded><![CDATA[<p>In this follow-up post I will show you how the action happens on a Daml ledger and how the screenplay is written for that. TL;DR: the key is the Update monad. You’ll also learn how to build a time machine by graph query.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnV28qd_V_7GuuITiuMc6w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@jakobowens1?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jakob Owens</a> on <a href="https://unsplash.com/photos/CiUR8zISX60?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In my previous blog post <a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-its-time-to-show-it-to-everyone-9bf1eedbd131">A Daml ledger tells a story — it’s time to show it to everyone</a>, I explained how a Daml ledger tells a story via the event graph it stores and how the event graph can be visualized and analyzed via a graph database, in our example Neo4J, using the <a href="https://neo4j.com/docs/cypher-manual/current/introduction/cypher_overview/">Cypher</a> graph query language.</p><p>In this follow-up post I will look more closely into one kind of event out of the two kinds which are recorded on the ledger, namely choice exercises.</p><h3>Recap: visualizing the event graph</h3><p>As a recap, I show again the visualization of the event graph, this time using a Daml model a little bit more complex than the Skeleton model used in the previous post (although, of course, this “more complex” model is still oversimplified compared to any real-life business model).</p><p>This time, let’s look into a Daml model which implements a ticket offer/purchase business model. The three actors of the model are in the following roles in the legal relationships expressed by the three kinds of contracts, “ticket offer”, “ticket agreement” and “cash”:</p><ul><li>Actor #1 is the buyer in the “ticket offer” relationship, owner in the “ticket agreement” relationship and original owner in the “cash” relationship.</li><li>Actor #2 is event organizer in the “ticket offer” and the “ticket agreement” relationships and new owner in the “cash” relationship.</li><li>Actor #3 is issuer in the two versions of the “cash” relationship.</li></ul><p>The structure of the Daml model can be represented, using the model <a href="https://docs.daml.com/tools/visual.html">visualizer</a> feature of the Daml platform, in the following way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*AoaqH2KzSHG1qhLoSkioKw.png" /><figcaption>The visual representation of the ticket offer/purchase Daml model</figcaption></figure><p>The event graph visible to Actor #1 after purchasing a ticket is the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*usec3y5qbWKsDQVbJlZgNg.png" /><figcaption>Actor #1’s ledger projection</figcaption></figure><p><strong>The introduction:</strong> In offset 9 a Cash contract was created, signed by Actor #3, the issuer of the Cash contract. In offset 10 a TicketOffer contract was created, signed by Actor #2, the event organizer.</p><p><strong>The real action: </strong>In offset 11, Actor #1 exercised the Accept choice on the TicketOffer contract, specifying the contract id of the Cash contract created in offset 9 as input, namely the money which is meant to pay for the ticket. Offset 11 now forms a non-trivial transaction tree, unlike the transaction tree formed by the only consequence of the Give choice in my previous blog post:</p><ul><li>One consequence of accepting the offer by Actor #1 is transferring the cash to the event organizer (itself having a consequence, creating a new Cash contract for the event organizer, and at the same time recording a consuming relationship with the original version of the Cash contract)</li><li>The other consequence is getting created a TicketAgreement contract for Actor #1.</li></ul><h3>Aside: contracts are “items on the ledger”?</h3><p>As an aside, I want to draw your attention to an abstraction in the Daml parlance which is fine to use — as long as you exactly understand what it actually means. Otherwise it can lead to serious misunderstandings.</p><p>The <a href="https://docs.daml.com/concepts/glossary.html#contract">Daml docs</a> say that “Contracts are items on a ledger.”</p><p>They go on: “When a contract is created on a ledger, it becomes active. But that doesn’t mean it will remain active forever: it can be archived.”</p><p>From this formulation the false conclusion can be drawn that being active or archived are properties of isolated “items” or things, namely contracts, “on the ledger”.</p><p>From what you’ve learnt about the ledger event graph, you can see that this is actually not the case. A contract, “active” or “archived”, is actually a relationship (or lack of relationship) between events recorded on the ledger.</p><p>The expression “active contract” is shorthand for a creation event recorded on the ledger for which no consuming exercise event was recorded on the ledger. The expression “archived contract” is shorthand for the totality of a creation event and a consuming exercise event pointing to that creation event, recorded on the ledger.</p><h3>Atomic “delivery vs payment”</h3><p>In general, the above pattern is called “delivery vs payment”, or “DvP” for short. Another name for the pattern is “atomic swap”.</p><p>The challenge in this use case is that we want to guarantee for both the seller and the buyer that they cannot remain without compensation: it cannot happen that the buyer pays the money but doesn’t get the goods or the seller delivers the goods and doesn’t get the money.</p><p>One way to achieve such a guarantee is to involve an escrow agent trusted by both the seller and the buyer, but Daml knows better: as long as the “money” asset and the goods asset live on the ledger, the atomic swap of these can be guaranteed (if not, promises or IOU’s for both can be swapped).</p><p>The way how the Daml ledger model supports atomic swaps is actually simple: a transaction tree is either appended to the ledger as a whole or not, there is no third option. As the transaction tree contains both the creation of new contracts and the consuming exercise on previously created contracts, the swap as a whole either happens or not.</p><p>Achieving this “simplicity”, of course, under the hood is not simple. You can see how many Canton system elements are needed for it to work from the section of the docs describing <a href="https://docs.daml.com/canton/architecture/overview.html#transaction-processing-in-canton">Transaction Processing in Canton</a>.</p><h3>Need-to-know basis</h3><p>As an aside, we can demonstrate Daml’s need-to-know-basis subtransaction privacy feature by looking at the ledger projection of Actor #3. As expected, it only contains the events for which Actor #3 is a stakeholder, namely the events around the creation and transfer of Cash contracts — the issuer of the Cash contracts doesn’t need to know anything about what the money buys:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z24hK_kZHDoHdW_um6ELew.png" /><figcaption>Actor #3’s ledger projection</figcaption></figure><h3>Time machine</h3><p>If you wish to see the ledger history in state machine style, as a series of active contract set (ACS) snapshots, the Cypher graph query language comes in handy.</p><p>For this still simple ledger you can tell without query, simply by inspection that (in the ledger projection of Actor #1):</p><ul><li>Before the 11_0 Accept event, the 9_0 Cash and the 10_0 TicketOffer contracts were active. If you imagine the event graph before adding the snapshot 11 transaction tree, these were the creation events for which no consuming exercise event was recorded on the ledger.</li><li>After recording the 11_0 Accept event, the 11_3 Cash and the 11_4 TicketAgreement contracts constitute the ACS. Now these are the creation events for which no consuming exercise event was recorded on the ledger.</li></ul><p>For a larger ledger, simple inspection doesn’t work, so it’s useful to know how to query the ACS snapshot.</p><p>The simpler query is the one for the current ACS, expressing in the <a href="https://neo4j.com/docs/cypher-manual/current/introduction/cypher_overview/">Cypher</a> graph query language that we are looking for create events for which no consuming exercise was recorded on the ledger:</p><pre>MATCH (c:created) <br>WHERE NOT ()-[:CONSUMING]-&gt;(c) <br>RETURN c</pre><p>For getting the series of historic ACS sets at a certain snapshot, we have to query for the create events which happened at or before that snapshot, for which no consuming exercise was recorded on the ledger before or at that snapshot. The below example shows the query for the ACS at snapshot 10, and all other snapshots can be queried the same way:</p><pre>MATCH (c:created WHERE c.offset&lt;=10)<br>WHERE COUNT{<br>    (e:exercised WHERE e.offset&lt;=10)-[:CONSUMING]-&gt;(c)<br>}=0<br>RETURN c</pre><h3>How the screenplay is written</h3><p>Now let’s see how the screenplay is written for the action of our story.</p><p>The full code of the underlying Daml model and the Daml script for the purchase is the following:</p><pre>module Main where<br><br>import DA.Assert ((===))<br>import Daml.Script<br><br>template TicketOffer<br>  with<br>      organizer : Party<br>      buyer : Party<br>      price : Decimal<br>  where<br>      signatory organizer<br>      observer buyer<br>      choice Accept : ContractId TicketAgreement<br>        with <br>          cashId : ContractId Cash<br>        controller buyer<br>        do <br>          cash &lt;- fetch cashId<br>          cash.amount === price<br>          exercise cashId Transfer with<br>              newOwner = organizer<br>          create TicketAgreement with<br>              organizer; owner = buyer<br><br>template Cash <br>  with <br>    issuer : Party <br>    owner : Party <br>    amount : Decimal <br>  where <br>    signatory issuer <br>    observer owner<br>    choice Transfer : ContractId Cash <br>      with<br>        newOwner : Party <br>      controller owner<br>      do <br>        create this with owner = newOwner <br><br>template TicketAgreement <br>  with <br>    organizer : Party <br>    owner : Party <br>  where <br>    signatory organizer, owner <br><br>setup : Script (ContractId TicketAgreement)<br>setup = script do <br>  alice &lt;- allocatePartyWithHint &quot;Alice&quot; (PartyIdHint &quot;Alice&quot;)<br>  ticketWizard &lt;- allocatePartyWithHint &quot;TicketWizard&quot; (PartyIdHint &quot;TicketWizard&quot;)<br>  scroogeBank &lt;- allocatePartyWithHint &quot;ScroogeBank&quot; (PartyIdHint &quot;ScroogeBank&quot;)<br>  aliceId &lt;- validateUserId &quot;alice&quot;<br>  ticketWizardId &lt;- validateUserId &quot;ticketwizard&quot;<br>  scroogeBankId &lt;- validateUserId &quot;scroogebank&quot;<br>  createUser (User aliceId (Some alice)) [CanActAs alice]<br>  createUser (User ticketWizardId (Some ticketWizard)) [CanActAs ticketWizard]<br>  createUser (User scroogeBankId (Some scroogeBank)) [CanActAs scroogeBank]<br><br>  cashCid &lt;- submit scroogeBank $ createCmd (Cash scroogeBank alice 10.0)<br>  offerCid &lt;- submit ticketWizard $ createCmd (TicketOffer ticketWizard alice 10.0)<br>  submit alice $ exerciseCmd offerCid (Accept cashCid)</pre><p>The important part for us now is the formulation of the “Accept” choice on the TicketOffer contract, so let’s highlight that:</p><pre>template TicketOffer<br>  ...<br>    choice Accept : ContractId TicketAgreement<br>        with <br>          cashId : ContractId Cash<br>        controller buyer<br>        do <br>          cash &lt;- fetch cashId<br>          cash.amount === price<br>          exercise cashId Transfer with<br>              newOwner = organizer<br>          create TicketAgreement with<br>              organizer; owner = buyer</pre><p>In order to fully understand what this choice really means, let’s consider that the template choice syntax is a shorthand of something a little bit more complex.</p><p>As the <a href="https://docs.daml.com/daml/intro/4_Transformations.html">Daml docs</a> put it:</p><blockquote>[Choices are] permissioned functions that result in an Update. Using choices, authority can be passed around, allowing the construction of complex transactions…</blockquote><blockquote>As a do block always returns the value of the last statement within it, the whole do block returns an Update, but the return type on the choice is just a ContractId Contact. This is a convenience. Choices always return an Update so for readability it’s omitted on the type declaration of a choice.</blockquote><p>How a choice is an Update function can be fully understood if we convert it back into a real Update function and plug it back into the choice body:</p><pre>accept: TicketOffer -&gt; Accept -&gt; Update (ContractId TicketAgreement)<br>accept TicketOffer{..} Accept{..} =  do<br>  cash &lt;- fetch cashId<br>  cash.amount === price<br>  exercise cashId Transfer with<br>      newOwner = organizer<br>  create TicketAgreement with<br>      organizer; owner = buyer</pre><p>Note that the “TicketOffer” record data type stands for the payload of the template with the same name, and the “Accept” record data type stands for the input data type of the choice with the same name.</p><p>After having “outsourced” the “Accept” choice into the “accept” function, we can simply plug the function invocation into the template’s choice body like this:</p><pre>template TicketOffer<br>  ...  <br><br>    choice Accept : ContractId TicketAgreement<br>        with <br>          cashId : ContractId Cash<br>        controller buyer<br>        do <br>          accept TicketOffer{..} Accept{..}</pre><p>After this substitution you will see that the compiler accepts the new code and the script result will be the same as before, so the two pieces of code express the same.</p><h3>Update is a monad</h3><p>As you can see from the signature of the “outsourced” choice function above, its return type is “Update (ContractId TicketAgreement)”.</p><p>What is this Update?</p><p>Update is a parametric type. It describes some kind of “context” which “contains” an instance of some type, in this case “ContractId TicketAgreement”. (In this case, the “contained” type is parametric itself: ContractId is a parametric type which becomes a specific type if we add a record data type which is a template payload type.)</p><p>The meaning of Update is a ledger update which may or may not be successful. The meaning of its type parameter is that this is the type of the data which gets returned from a ledger update — if the update is successful.</p><p>(The returned data can be “nothing”, e.g. from the archival of a contract, expressed by the unit data type which is written as an empty tuple. But it has to be specified even in this case. Returning this kind of “nothing” from a successful ledger update is fundamentally different from returning “nothing but an error message” from an unsuccessful ledger update.)</p><p>Returning a piece of data from a successful ledger update means that this piece of data gets recorded in the “exercise_result” field of the choice exercise events. For creation events the return value is always the contract id of the created contract, recorded in the “contract_id” field of the event. If the update is not successful, an error message gets returned to the caller client application and nothing recorded on the ledger.</p><p>The whole list of ledger updates is listed in the <a href="https://docs.daml.com/daml/reference/updates.html">Reference: Updates</a> chapter of the Daml docs. The list entails creation, choice exercise (including archive), fetch a contract payload by contract id, fetch contract id and payload by contract key, check contract key visibility, assertion, getting the ledger time, etc.</p><p>What does it mean that Update is a monad?</p><p>The concept of “monad” is central to functional programming. It may sound strange at first, but once you understand it, you will see that it’s quite useful. There are a lot of explanations out there, I list here some such explanations I found useful:</p><ul><li><a href="http://learnyouahaskell.com/a-fistful-of-monads">A Fistful of Monads </a>— Learn You a Haskell for Great Good!</li><li><a href="https://www.youtube.com/watch?v=ZhuHCtR3xq8">Brian Beckman: Don’t fear the Monad</a> (YouTube video)</li><li><a href="https://www.youtube.com/watch?v=gHiyzctYqZ0">Category Theory 10.1: Monads</a> by Bartosz Milewski (YouTube video)</li></ul><p>The Update monad is somewhat analogous to the IO monad in Haskell, so you may want to take a look at how this is <a href="http://learnyouahaskell.com/input-and-output">explained</a> in the Learn You a Haskell for Great Good! book.</p><p>In a nutshell: a monad is a parametric family of types (type class) which has functions for expressing the composition of functions with side effects (like IO in Haskell or ledger update in Daml) in a meaningful way so that return values can be passed over to the next steps of the function chain. “In a meaningful way” means that adhering to certain rules which basically express associativity.</p><h3>The “do” block is syntactic sugar</h3><p>You may have noticed that in the Daml choice body the consequences of the choice exercise are nicely listed in a “do” block.</p><p>This “do” block notation in Daml choice bodies is convenient because it reads like a list of todo items, just as instructions in non-functional, procedural languages.</p><p>This is also something which is fine to use, as long as you exactly understand what it actually means. Otherwise, it can lead to serious misunderstandings.</p><p>The fundamental difference between a “do” block and a list of instructions in procedural languages is the following:</p><ul><li>the “do” block consists of actions which may or may not succeed</li><li>the “do” block as a whole will fail if any of its elements fail</li><li>the elements of the “do” block must belong to the same side effect category (that is monad), in our case ledger update</li><li>the “do” block returns the return value its last element.</li></ul><p>The backwards pointing arrow syntax is used for assigning action return values to variables, which can be used in later actions of the block.</p><p>It turns out that the “do” block notation in Daml choice bodies is the same as the <a href="http://learnyouahaskell.com/a-fistful-of-monads#do-notation">“do” notation</a> in Haskell for monads, and is syntactic sugar for the native &gt;&gt;= (“bind” operator to pass a monadic value to a pure -&gt; monad function to receive a monadic value) and &gt;&gt; (“then” operator) operators.</p><p>The above discussed, outsourced function version of the “Accept” choice can be rewritten without the “do” block sugar in the following way:</p><pre>accept&#39;: TicketOffer -&gt; Accept -&gt; Update (ContractId TicketAgreement)<br>accept&#39; TicketOffer{..} Accept{..} =<br>  fetch cashId &gt;&gt;=<br>    \cash -&gt; cash.amount === price &gt;&gt;<br>      exercise cashId (Transfer with newOwner = organizer) &gt;&gt;<br>        create TicketAgreement with organizer = organizer, owner = buyer</pre><p>Just like in Haskell, “do” blocks in Daml are not only used in choice bodies but can be used in any monad. We use “do” blocks in Script all the time. You can even use it, if you wish to show off your functional programming savvy with the list monad e.g. like this (but your PR will be probably rejected):</p><pre>let baseList = [1,2,3]<br>let modifiedList = do<br>      x &lt;- baseList<br>      [x,x,x]<br><br>-- modifiedList = [1,1,1,2,2,2,3,3,3]</pre><h3>How is it guaranteed that the Update monad adheres monadic laws?</h3><p>It is fundamental to understand that the monadic laws are not checked by the compiler. If the programmer declares that a certain type is an instance of the monad type class (called Action in Daml), the compiler only uses this information for type checking. The programmer themselves need to make sure that the underlying implementation is correct in that it adheres the monad laws.</p><p>This is guaranteed e.g. for the IO monad in Haskell how the language interacts with the operating system. And this is guaranteed for the Update monad in Daml how <a href="https://docs.daml.com/canton/architecture/overview.html#transaction-processing-in-canton">transactions are processed by Canton</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c2b5c2f866d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-how-does-the-action-happen-6c2b5c2f866d">A Daml ledger tells a story–how does the action happen?</a> was originally published in <a href="https://medium.com/daml-masterclass">DAML Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Daml ledger tells a story — it’s time to show it to everyone]]></title>
            <link>https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-its-time-to-show-it-to-everyone-9bf1eedbd131?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bf1eedbd131</guid>
            <category><![CDATA[smart-contracts]]></category>
            <category><![CDATA[ledger]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[daml]]></category>
            <category><![CDATA[neo4j]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Wed, 05 Jul 2023 10:13:48 GMT</pubDate>
            <atom:updated>2023-07-05T10:13:48.706Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Daml ledger tells a story — it’s time to show it to everyone</h3><p>A Daml-based application seems to be an ordinary state machine. Sure, the application UI only displays the current state of the system and the transition options. But the unique feature of a Daml ledger is that it stores not only the state of the system but also the state transitions. For explaining the idea of Daml ledgers and the story told by a specific ledger (and for audit, <a href="https://www.youtube.com/watch?v=GXhBEj1ZtE8">graph neural network</a> based fraud detection, etc.), it’s a good idea to show the full picture.</p><p>As a preview of upcoming attractions: at the end of this blog post you will understand what this event graph means and how it can be created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pylxzYcYCBvsu4uB-J1Asg.png" /><figcaption>The event graph of a Daml ledger</figcaption></figure><h3>The Daml ledger‘s event graph</h3><p>The summary of the working mechanism of a Daml ledger, based on the <a href="https://docs.daml.com/concepts/ledger-model/index.html">Daml ledger model</a> is the following:</p><ul><li>The ledger stores an event graph, containing creation and exercise event nodes, as well as “consuming” and “consequence” edges.</li><li>The event graph grows in an append-only manner during normal operation, which guarantees atomic transaction handling</li><li>The current state of the ledger (also called the active contract set, ACS for short) consists of such creation events for which no consuming exercise was recorded on the ledger.</li><li>Ledger parties have access to a subset of the event graph containing the events they are entitled to see. This guarantees sub-transaction privacy.</li><li>The ledger can be pruned under certain circumstances, removing events which are not needed for operation.</li></ul><p>Let’s see these points in more detail.</p><p><strong>The ledger stores an event graph.</strong></p><p><em>(Please not that the Daml documentation speaks about the ledger </em><a href="https://docs.daml.com/concepts/local-ledger.html#causality-graphs"><em>causality graph</em></a><em> which is a related but somewhat different concept, and is important for validating transaction requests which I don’t cover in this blog post.)</em></p><p>The nodes of the graph are <strong>creation events</strong> for the creation of contracts, and <strong>exercise events</strong> for exercising choices on previously created contracts. (Plus some other kinds of events which I omit from this account for clarity.)</p><blockquote>For those of you who are not closely familiar with what Daml contracts and choices are, here is a short detour.</blockquote><blockquote>In the Daml code, the equivalent of contract creation and choice exercise is the contract template and the choices on these templates.</blockquote><pre>type AssetId = ContractId Asset<br><br>template Asset<br>  with <br>    issuer : Party<br>    owner  : Party<br>    name   : Text<br>  where<br>    ensure name /= &quot;&quot;<br>    signatory issuer <br>    observer owner<br>    choice Give : AssetId<br>      with<br>        newOwner : Party<br>      controller owner<br>      do create this with <br>        owner = newOwner<br>    </pre><blockquote>The above code is an Asset contract template. The data content (payload) of the contract instance created from this template will be what is specified in the upper “with” block.</blockquote><blockquote>Contracts represent legal relationships, so the payload must contain at least one (and usually more than one) ledger party. (Ledger parties are the on-ledger representations of the actors of the legal relationships, namely people of organizations.) In this case the contract represents a legal relationship between an issuer and an owner. It’s good practice to choose field names for the parties which indicate the roles of the parties in the relationship.</blockquote><blockquote>Such an Asset contract can be created by any party existing on the ledger. There are some restrictions though:</blockquote><blockquote>Anyone who submits the contract creation request to the ledger, identified as a ledger party, must specify themselves as the issuer of the Asset. This is indicated by the “signatory issuer” clause. The party id of the submitter will be stored not only in the “issuer” field of the created contract instance, but also in the “signatories” list which is created by the platform.</blockquote><blockquote>The Asset name must be a text, which is not empty. This is indicated by the `ensure name /= “”` clause.</blockquote><blockquote>The creator of the contract can specify any existing ledger party as “owner”, including themselves. The owner doesn’t have any obligations as a consequence of being mentioned in the contract (of course, as they didn’t sign anything so far). Any ledger party mentioned as owner can see the contract creation in their ledger views. (This is specified by the “observer owner” clause. Similarly to the above mentioned “signatories” filed, there is an “observers” field on contract instances, created by the platform. In this case the party specified as owner will be filled in.)</blockquote><blockquote>The owner party is granted a right by this contract, specified in the “choice” block. They can give it to any other ledger party, by exercising the “Give” choice, specifying a new owner. (The required input data for the choice are specified in the “with” block under the choice name.) The “do” block specifies the explicit consequences of exercising the choice: the creation of another instance from the same template, with almost the same data as the underlying contact, just changing the owner to the new owner.</blockquote><blockquote>I wrote above that the “do” block contains the explicit consequences of the choice exercise. This is suggests that there can be some implicit consequence of the choice exercise. And there is indeed, in the default case. If it is not indicated otherwise, a choice exercise “consumes” the underlying contract, meaning rendering it unavailable for further use. The creator of the Daml model can change this default behavior by putting the “nonconsuming” keyword in front of the “choice” keyword.</blockquote><p>The creation and exercise events don’t store the choices associated with them on the ledger, just the payload of the contract and the input data of the choice exercise (plus some metadata).</p><blockquote>Another short detour about how to know what rights and obligations are associated with contract instances:</blockquote><blockquote>You may ask: how anyone inspecting the ledger can know what rights and obligations are associated with the contract instances? The answer is that from inspecting the ledger alone they cannot. They can only know this by also inspecting and understanding the source code of the Daml model.</blockquote><blockquote>From which arises the next question: how can they know that a certain contract instance belongs to a certain Daml model? The answer is that every Daml model has a unique id, called package id, which is the hash of its source code + its version + its dependencies. Every contract instance on the ledger contains the package id of the Daml model it belongs to.</blockquote><p>The edges of the graph are the “consuming” and “consequence” relationships starting at exercise events and pointing to other creation or exercise events. We can picture these relationships as arrows for ourselves. The “consuming” arrow points to previously created contracts, “consumed” by the choice exercise. The “consequence” arrow points to the events listed in the “do” block of the choice body in the source code.</p><p><strong>The event graph grows in an append-only manner during normal operation.</strong></p><p>When receiving a transaction request, the ledger performs some general checks and examines if all the necessary authorizations were given in advance. (It doesn’t wait for any ledger party to authorize the request — if such post-hoc authorization is needed, then the Daml model needs to contain appropriate contract templates for that).</p><p>If the request is valid and well-authorized, the corresponding event nodes and relationship edges are added to the event graph. If anything is missing, nothing gets added to the ledger. This guarantees atomic transaction composition.</p><p><strong>The current state of the ledger (also called the active contract set, ACS for short) consists of such creation events for which no consuming exercise was recorded on the ledger.</strong></p><p>When we speak about “active contracts” and “archived contracts”, this is an abstraction. The “archival” of the contract doesn’t change anything on the creation event of it. It just adds an “arrow” to the ledger pointing back to the previously recorded creation event, from a consuming exercise event.</p><p>You may be tempted to think that “active contracts” are on-ledger, while “archived contracts” are off-ledger objects. This couldn’t be further from the truth — until, of course, the ledger gets pruned (see later).</p><p>All this means that a Daml ledger is an <a href="https://martinfowler.com/eaaDev/EventSourcing.html">event sourcing</a> system. In other words, the current state of the system must be computed from the whole event graph, which is usually slower than just retrieve the state snapshot from a database. Speeding up this computation is a challenge. One option is to use the <a href="https://docs.daml.com/json-api/production-setup/query-store.html">query store</a> feature of the Daml JSON API. Another option can be to store the event graph in a graph database which is optimized for fast graph query (see later).</p><p><strong>Ledger parties have access to a subset of the event graph containing the events they are entitled to see. This guarantees sub-transaction privacy.</strong></p><p>Such subsets of the event graph are called “<a href="https://docs.daml.com/concepts/ledger-model/ledger-privacy.html#projections">ledger projections</a>”. The magic of Daml is that the consistency of projections guarantees the consistency of the whole virtual ledger. (Here I am simplifying things, because consistency is actually a property of the causality graph.)</p><p><strong>The ledger can be pruned under certain circumstances, removing events which are not needed for operation.</strong></p><p>This serves two purposes: 1) some kind of “garbage collection” 2) is necessary to comply with the “<a href="https://gdpr.eu/right-to-be-forgotten/">right to be forgotten</a>” mandate of GDPR. See the details of ledger pruning <a href="https://docs.daml.com/canton/usermanual/pruning.html">here</a>.</p><h3>Why we usually cannot see the whole event graph</h3><p>For obvious reasons, the UI of Daml applications usually only display the “active contracts”, or in the case of the <a href="https://docs.daml.com/tools/navigator/index.html">Navigator</a> UI, on demand also the “archived contracts” (and we already know how these are abstractions).</p><p>Though exercise events are the links between ledger states, the dynamic elements which tell the story of the ledger, they remain the “missing links” in this way.</p><p>I hope you are convinced by the reasoning put forward in the previous sections that the event graph is worth displaying. Let’s see how this goes.</p><h3>Displaying the event graph</h3><p>The Daml docs use causality graphs for explaining the Daml ledger model. Below is an example. Single arrows together with the “Exe” keyword signal “consuming” relationships, double arrows “consequence” relationships. (Nonconsuming exercises were signalled with the “ExeN” keyword.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SOkWAloYCF6wueG-ymxWLg.png" /><figcaption>See the original graph <a href="https://docs.daml.com/concepts/ledger-model/ledger-structure.html?_ga=2.3900281.810738926.1687970499-459617840.1687970499#ledgers">here</a></figcaption></figure><p>The direction of the arrows is not crucial but can convey a subtle message.</p><p>This is a causality graph, so the single arrow means that the creation of a contract must happen before a choice exercise on that contract.</p><p>For event graphs, I prefer the reversed direction of the “consuming” relationships. On graph representations of directed graphs, the edges are usually so represented that the set of the ending nodes is a property of the starting node. And this is how the Daml platform actually returns the event graph.</p><p>Drawing the “consuming” edge starting at the consumed contract may suggest that the creation event of the “consumed” contract is changed afterwards. As we saw earlier, this is not the case. The creation events, once recorded, are never changed because the event graph grows in an append-only way.</p><p>A similar graph representation of the Daml ledger can be seen in Daml Studio on the “Script result” pane, choosing the “Show transaction view” option. The below graph represents the ledger after three events, happening with Asset contracts explained in previous sections. The events are:</p><ol><li>Alice creates an Asset contract instance with the name “TV”, specifying herself as issuer and owner.</li><li>Alice gives the TV asset to Bob.</li><li>Bob gives the TV asset back to Alice.</li></ol><p>The same as expressed in Daml Script:</p><pre>  aliceTV &lt;- submit alice do<br>    createCmd Asset with<br>      issuer = alice<br>      owner = alice<br>      name = &quot;TV&quot;<br><br>  bobTV &lt;- submit alice do<br>    exerciseCmd aliceTV Give with newOwner = bob<br><br>  submit bob do<br>    exerciseCmd bobTV Give with newOwner = alice</pre><p>After these three events, the ledger graph looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGe6fMIrM_-uyD8xPsFStg.png" /></figure><p>There is a lot going on here, I won’t explain every detail just what is important for us now.</p><p>Transactions are marked with the TX keyword, a growing number and a timestamp. The number stands for the combination of a unique, randomly chosen transaction id and a growing index called “ledger offset”. We can see here three transactions, corresponding to the three events described above.</p><p>Events are marked with a compound id, consisting of the transaction id and an index as suffix. E.g. event #2:1 is the second event of the third transaction. Event descriptions also contain the actor, the “creates” or “exercises” keyword together with the template name or the choice name, and the contract payload orexercise inputs in the corresponding “with” blocks.</p><p>Transactions always have a top level action, which is directly requested by a ledger party. These are marked with zero suffix.</p><p>The list of consequences of top-level exercise actions is marked with the “children” keyword. (Create actions, naturally, don’t have consequences. The list of consequences can be empty when a choice only consumes its contract, or is only used to signal some information to a client application without changing the ledger.) The list of children in these cases is always a singleton list, because the “do” block of the “Give” choice on the “Asset” template only contains the creation of an “Asset” template with a modified content.</p><p>“Consuming” relationships are marked with the “consumed by” purple clauses. My warning regarding the direction of the “consumed” arrow previously is also valid here. Marking the creation events themselves as “consumed by” may lead to the false assumption that the creation events on the ledger are modified afterwards by the consuming exercise, which is not the case.</p><p>This graph view is nice for small ledgers and small contracts, but can quickly become unintelligible. An additional limitation of it is that it can be displayed for displaying a series of hypothetical ledger submissions expressed in Daml Script against Daml Studio’s script running service, and not for displaying the actual content of a running ledger.</p><h3>What a specific party can see on the ledger?</h3><p><em>(Please note that “the ledger” is an abstract concept. You may be tempted to ask “What is on the ledger?”. This question is not a meaningful one. You can ask instead: “What a specific party can see on the ledger?”.)</em></p><p>The full view of the ledger for a specific party is available through the <a href="https://docs.daml.com/app-dev/services.html#read-from-the-ledger">transaction service</a>, with the GetTransactionTrees subscription. This command returns the stream of <a href="https://docs.daml.com/app-dev/grpc/proto-docs.html#transactiontree">transaction trees</a> as a gRPC message stream.</p><p>The ledger view can be returned as a message stream because the ledger grows in a pure additive manner. The ledger view after the first transaction was the first message, the ledger view after the first two transactions was the first two messages, etc.</p><p>For prototyping, the message stream can be fetched by e.g. <a href="https://aristanetworks.github.io/openmgmt/examples/gnoi/grpcurl/">gRPCurl</a>. The returned JSON representation of the first two transactions from the above mentioned three transactions in Alice’s view looks like this:</p><pre>{<br>  &quot;transactions&quot;: [<br>    {<br>      &quot;transaction_id&quot;: &quot;1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09&quot;,<br>      &quot;command_id&quot;: &quot;ce783485-56eb-4823-8086-2a109ff32651&quot;,<br>      &quot;effective_at&quot;: &quot;2023-03-07T14:02:49.476346Z&quot;,<br>      &quot;offset&quot;: &quot;000000000000000007&quot;,<br>      &quot;events_by_id&quot;: {<br>        &quot;#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0&quot;: {<br>          &quot;created&quot;: {<br>            &quot;event_id&quot;: &quot;#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0&quot;,<br>            &quot;contract_id&quot;: &quot;00a87fa6e78e7c059f0f192eea08bdb38cd1339103bee0116339dce93abc9caa7dca01122083a06b269dab95a5d1abd78bb35cdad628d4f0c0b8a7751c01fb56b6d734d527&quot;,<br>            &quot;template_id&quot;: {<br>              &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>              &quot;module_name&quot;: &quot;Main&quot;,<br>              &quot;entity_name&quot;: &quot;Asset&quot;<br>            },<br>            &quot;create_arguments&quot;: {<br>              &quot;record_id&quot;: {<br>                &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>                &quot;module_name&quot;: &quot;Main&quot;,<br>                &quot;entity_name&quot;: &quot;Asset&quot;<br>              },<br>              &quot;fields&quot;: [<br>                {<br>                  &quot;label&quot;: &quot;issuer&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;party&quot;: &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>                  }<br>                },<br>                {<br>                  &quot;label&quot;: &quot;owner&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;party&quot;: &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>                  }<br>                },<br>                {<br>                  &quot;label&quot;: &quot;name&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;text&quot;: &quot;TV&quot;<br>                  }<br>                }<br>              ]<br>            },<br>            &quot;witness_parties&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;agreement_text&quot;: &quot;&quot;,<br>            &quot;signatories&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;metadata&quot;: {<br>              &quot;created_at&quot;: &quot;2023-03-07T14:02:49.476346Z&quot;,<br>              &quot;driver_metadata&quot;: &quot;CiYKJAgBEiCbJmDtJrpjE9ux5M2GS268rbcgvEt1/T6OgDm+U9VPgw==&quot;<br>            }<br>          }<br>        }<br>      },<br>      &quot;root_event_ids&quot;: [<br>        &quot;#1220bb3313bff96d52932e0e088e60cdbc3e4165292d89c0c76a588f1e9dc911bb09:0&quot;<br>      ]<br>    }<br>  ]<br>}<br>{<br>  &quot;transactions&quot;: [<br>    {<br>      &quot;transaction_id&quot;: &quot;1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3&quot;,<br>      &quot;command_id&quot;: &quot;cdbc4694-35b7-4493-8350-6cff66d43c84&quot;,<br>      &quot;effective_at&quot;: &quot;2023-03-07T14:02:54.749560Z&quot;,<br>      &quot;offset&quot;: &quot;000000000000000008&quot;,<br>      &quot;events_by_id&quot;: {<br>        &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0&quot;: {<br>          &quot;exercised&quot;: {<br>            &quot;event_id&quot;: &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0&quot;,<br>            &quot;contract_id&quot;: &quot;00a87fa6e78e7c059f0f192eea08bdb38cd1339103bee0116339dce93abc9caa7dca01122083a06b269dab95a5d1abd78bb35cdad628d4f0c0b8a7751c01fb56b6d734d527&quot;,<br>            &quot;template_id&quot;: {<br>              &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>              &quot;module_name&quot;: &quot;Main&quot;,<br>              &quot;entity_name&quot;: &quot;Asset&quot;<br>            },<br>            &quot;choice&quot;: &quot;Give&quot;,<br>            &quot;choice_argument&quot;: {<br>              &quot;record&quot;: {<br>                &quot;record_id&quot;: {<br>                  &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>                  &quot;module_name&quot;: &quot;Main&quot;,<br>                  &quot;entity_name&quot;: &quot;Give&quot;<br>                },<br>                &quot;fields&quot;: [<br>                  {<br>                    &quot;label&quot;: &quot;newOwner&quot;,<br>                    &quot;value&quot;: {<br>                      &quot;party&quot;: &quot;Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>                    }<br>                  }<br>                ]<br>              }<br>            },<br>            &quot;acting_parties&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;consuming&quot;: true,<br>            &quot;witness_parties&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;child_event_ids&quot;: [<br>              &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1&quot;<br>            ],<br>            &quot;exercise_result&quot;: {<br>              &quot;contract_id&quot;: &quot;00027c63797ce8dce9e793ae740770334c72e499c9e7b9c7bc72b2ab7cf978c07aca01122056a1884001af49b183f1a7a3cba394e616dc68e881f17f5f76c07f68c8f4179c&quot;<br>            }<br>          }<br>        },<br>        &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1&quot;: {<br>          &quot;created&quot;: {<br>            &quot;event_id&quot;: &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:1&quot;,<br>            &quot;contract_id&quot;: &quot;00027c63797ce8dce9e793ae740770334c72e499c9e7b9c7bc72b2ab7cf978c07aca01122056a1884001af49b183f1a7a3cba394e616dc68e881f17f5f76c07f68c8f4179c&quot;,<br>            &quot;template_id&quot;: {<br>              &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>              &quot;module_name&quot;: &quot;Main&quot;,<br>              &quot;entity_name&quot;: &quot;Asset&quot;<br>            },<br>            &quot;create_arguments&quot;: {<br>              &quot;record_id&quot;: {<br>                &quot;package_id&quot;: &quot;e44c38bd3be067c6f812661268514b1570c9bd03be97fda6f6a59e714b00dde9&quot;,<br>                &quot;module_name&quot;: &quot;Main&quot;,<br>                &quot;entity_name&quot;: &quot;Asset&quot;<br>              },<br>              &quot;fields&quot;: [<br>                {<br>                  &quot;label&quot;: &quot;issuer&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;party&quot;: &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>                  }<br>                },<br>                {<br>                  &quot;label&quot;: &quot;owner&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;party&quot;: &quot;Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>                  }<br>                },<br>                {<br>                  &quot;label&quot;: &quot;name&quot;,<br>                  &quot;value&quot;: {<br>                    &quot;text&quot;: &quot;TV&quot;<br>                  }<br>                }<br>              ]<br>            },<br>            &quot;witness_parties&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;agreement_text&quot;: &quot;&quot;,<br>            &quot;signatories&quot;: [<br>              &quot;Alice::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;observers&quot;: [<br>              &quot;Bob::122035081beb0fd9cfd581a260e2448390a9c4c3b4e51e3351b7bdb191202b8dfb67&quot;<br>            ],<br>            &quot;metadata&quot;: {<br>              &quot;created_at&quot;: &quot;2023-03-07T14:02:54.749560Z&quot;,<br>              &quot;driver_metadata&quot;: &quot;CiYKJAgBEiCcT0sakwjtEG60q0pCsNfx5ZvaZxPIwRkynyzeLWsh6A==&quot;<br>            }<br>          }<br>        }<br>      },<br>      &quot;root_event_ids&quot;: [<br>        &quot;#1220921fce3e510617cbc78d70f3b1a4392dd4815a2314b4ec04676d645d181bffc3:0&quot;<br>      ]<br>    }<br>  ]<br>}</pre><p>As expected, the first transaction contains one event, the creation of an Asset contract by Alice. The second transaction contains two events, 1) the consuming exercise if the “Give” choice on the previously created contract (identified by contract id), 2) the creation of a new Asset contract with modified content.</p><p>Please note again that the first “created” event is never changed after having recorded on the ledger. The fact that the corresponding contract was “archived” is recorded on the “exercised” event by setting the value of the “consuming” field to “true”.</p><h3>Show, don’t tell</h3><p>Now finally we have arrived at the point where we can create a nice visualization of the event graph.</p><p>After some transformation (see the <a href="https://colab.research.google.com/drive/1gE-NM73f2CRhfdJnYoVmii2DSaCX_33I?usp=sharing">source code</a> on Google Colab) we can create from the transaction trees JSON the Cypher code to create the graph:</p><pre>CREATE (Event_7_0:created {offset: 7, event_id: &quot;7_0&quot;, template: &quot;Asset&quot;, issuer: &quot;Alice&quot;, owner: &quot;Alice&quot;, name: &quot;TV&quot;})<br>CREATE (Event_8_0:exercised {offset: 8, event_id: &quot;8_0&quot;, choice: &quot;Give&quot;, newOwner: &quot;Bob&quot;})<br>CREATE (Event_8_1:created {offset: 8, event_id: &quot;8_1&quot;, template: &quot;Asset&quot;, issuer: &quot;Alice&quot;, owner: &quot;Bob&quot;, name: &quot;TV&quot;})<br>CREATE (Event_9_0:exercised {offset: 9, event_id: &quot;9_0&quot;, choice: &quot;Give&quot;, newOwner: &quot;Alice&quot;})<br>CREATE (Event_9_1:created {offset: 9, event_id: &quot;9_1&quot;, template: &quot;Asset&quot;, issuer: &quot;Alice&quot;, owner: &quot;Alice&quot;, name: &quot;TV&quot;})<br>CREATE (Event_8_0)-[:CONSEQUENCE]-&gt;(Event_8_1)<br>CREATE (Event_9_0)-[:CONSEQUENCE]-&gt;(Event_9_1)<br>CREATE (Event_8_0)-[:CONSUMING]-&gt;(Event_7_0)<br>CREATE (Event_9_0)-[:CONSUMING]-&gt;(Event_8_1)</pre><p>Which, in the <a href="https://neo4j.com/">Neo4j</a> graph database creates the following graph you’ve already seen in the introduction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pylxzYcYCBvsu4uB-J1Asg.png" /><figcaption>The event graph of a Daml ledger</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bf1eedbd131" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daml-masterclass/a-daml-ledger-tells-a-story-its-time-to-show-it-to-everyone-9bf1eedbd131">A Daml ledger tells a story — it’s time to show it to everyone</a> was originally published in <a href="https://medium.com/daml-masterclass">DAML Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rust + Daml: a biz-friendly smart contract platform deserves a biz-friendly client language]]></title>
            <link>https://medium.com/daml-masterclass/rust-daml-a-biz-friendly-smart-contract-platform-deserves-a-biz-friendly-client-language-ddb74922a263?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddb74922a263</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ledger]]></category>
            <category><![CDATA[daml]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[smart-contracts]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Tue, 11 Apr 2023 09:27:12 GMT</pubDate>
            <atom:updated>2023-04-11T22:17:47.729Z</atom:updated>
            <content:encoded><![CDATA[<p>TL;DR Daml is an expressive language and so is the Rust code generated from the Daml model. The reason for the similarity is that both languages adopt basic design choices from Haskell.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*trUyXWahAXJfDqN9" /><figcaption>Photo by <a href="https://unsplash.com/@vespir?utm_source=medium&amp;utm_medium=referral">James Forbes</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A Daml model implements the legal architecture of a business model. So using Daml poses a double challenge:</p><ol><li>For the business partners collectively to agree on the common rules of the game and implement them in the form of a Daml model. The Daml model, deployed on the Daml/Canton platform will constitute the shared virtual ledger, used by the business partners as a transaction platform and system of record with need-to-know basis privacy. On the positive side, the fact that all members of a business community use exactly the same piece of software, reduces the risk for the advantages of the collaboration to be overshadowed by costly coordination problems leading to something which economists call “<a href="https://en.wikipedia.org/wiki/Diseconomies_of_scale">diseconomies of scale</a>”.</li><li>For the business partners individually to integrate the Daml/Canton node they operate with their own systems.</li></ol><p>Both challenges need the collaboration of business stakeholders, legal experts and developers in order to make sure that the application serves the business purpose for which it is created, it’s legally correct and fulfills the non-functional requirements like processing speed and throughput.</p><p>In this blog post, I will focus on point #2, from the angle of business-friendliness. By this, I mean how easy it is for developers and business stakeholders to reason about wether the application really does what it is supposed to do.</p><h3>Client-side support for Daml</h3><p>Theoretically, you can write client applications for the Daml/Canton platform in any language which supports gRPC codegen, because the platform exposes a gRPC API, defined by <a href="https://github.com/digital-asset/daml/tree/main/ledger-api/grpc-definitions">these</a> protobuf docs.</p><p>In practice though, it’s tedious to write your client-side datatypes which you want to use as a representation of the Daml datatypes, based directly on the code generated from the protobuf files. This is the reason why some higher-level alternatives exist.</p><p>Digital Asset (DA), creators of the Daml/Canton platform offer the following options (you can find further details about all these in the <a href="https://docs.daml.com/">Daml docs</a>):</p><ul><li><strong>A JSON API which is language-independent.</strong> It exposes a small number of endpoints for querying, creating and manipulating smart contracts on the ledger and for some housekeeping. The API is deceptively simple — but this means that the complexity is transferred into the JSON payload. In order to be able to submit a valid request, the client application has to create a JSON which is a Daml-compatible representation of the data to be submitted. For some more involved data types, this is far from being trivial. The JSON API exposes a core set of features of the gRPC API. It doesn’t provide advanced features like retrieving the full transaction stream or a former snapshot of the active contract set. But on the positive side, it provides a cache-like feature (called query store) which makes it faster to retrieve the active contract set than just pulling it from the ledger every time. (A little bit of context as an explanation: a Daml ledger is an event source system, meaning it actually doesn’t store contracts, but contract creation and exercise events (where an exercise may or may not consume the underlying contract), from which — without caching — the active contract set needs to be computed every time we need it. The concept is similar to what is called “unspent transaction output” or UTXO in the Bitcoin world.) The JSON API can be used as a REST or as a websocket API.</li><li><strong>A Python binding.</strong> Using Python’s “async/await” syntax, it’s easy to express ledger interactions. The challenge is again the Daml-compatible representation of the input/output data. The library supports a syntax based on Python (frozen) dictionaries and lists, more or less identically to the Daml-compatible JSON syntax, but nothing more. Using more involved Daml data types can quickly spiral out of control in terms nesting levels and complexity. For this reason, it’s good practice to write (nested) classes for the data types, but this is not supported in any way by the Python library provided by DA. The number of classes needed can be high for a more complex Daml model, and has to be updated manually every time the Daml model changes. (As a personal aside: I like to use the Python binding because it’s compatible with Colab and Jupyter notebooks which is a convenient way of demonstrating some ledger behavior. The output of queries can be displayed conveniently as Pandas dataframes. UI widgets can be used for fast prototyping UI events. I also used the Python bindings to upload data to a ledger from Excel files, building on the Excel reading feature of Pandas.)</li><li><strong>A TypeScript binding to be used against the JSON API.</strong> There is a basic library which can be used independently from any UI framework, and a React library built on top of it. Its main advantage over the Python library is codegen. Through the codegen feature of the Daml platform, we can generate an up-to-date set of TypeScript types from the Daml model. The limitation of TypeScript codegen is thought that TypeScript types are mere type annotations, and don’t provide constructor functions for typed objects. But nonetheless, if you want to write single page applications against Daml using a TypeScript/JavaScript based UI framework, this library is a useful companion.</li><li><strong>A Java binding to be used agains the gRPC API.</strong> It offers the full feature set of the gRPC API, and also supports codegen. The generated code contains Java classes corresponding to the data types contained in the underlying Daml model.</li><li><strong>Scala binding.</strong> It still exists on Maven, but the Daml docs say it’s deprecated and there is no documentation for it.</li></ul><p>And last but not least, on top of the client solutions provided by DA, we have an <strong>unofficial </strong><a href="https://crates.io/crates/daml"><strong>Rust binding</strong></a>. In the rest of this blog post I will examine this one from the angle of business friendliness. And before that, I will explain how Daml is very business-friendly.</p><h3>The secret sauce of Daml’s business friendliness</h3><p>The most expressive way of representing real-world data is the so-called <a href="https://en.wikipedia.org/wiki/Algebraic_data_type">algebraic data type</a> (ADT). An ADT allows for a limited set of options, each option potentially being a compound piece of data, containing a set of values of specified types. It’s similar to the enum datatype present in many languages with the difference that the variants can potentially hold one or more typed fields rather than being a single text label.</p><p>An example can be a data type expressing book attributes. Let’s say a piece of book attribute data can express</p><ul><li>either the number of pages,</li><li>or the list of authors,</li><li>or the book’s title,</li><li>or the book’s publishing details, consisting of the year of publishing and the publisher.</li></ul><p>We want to use this data type so that we can rest assured that it’s expressing some book attributes, and we also want to be able to unwrap the individual details from the piece of data.</p><p>Functional languages usually (or maybe all of them?) support ADTs, paired with pattern matching for extracting the individual details from it. The data type is called “algebraic” because the set of options is conceptually similar to addition, and the bundle of fields within one variant is conceptually similar to multiplication. (If you want to know what exactly “conceptually similar” means you can consult a category theory text book or watch <a href="https://www.youtube.com/watch?v=w1WMykh7AxA">this video</a>.)</p><p>Daml, being a functional language, more explicitly a descendant of Haskell, supports ADTs, called “variant” in the docs. (So “variant” has two meanings: it’s a label for the whole data type and also for one element of it.) The aforementioned book attribute data type can be expressed in Daml (almost, but not entirely in the same way as in Haskell) like this:</p><pre>data BookAttribure = Pages Int<br>                   | Authors [Text]<br>                   | Title Text<br>                   | Published with year: Int, publisher: Text<br>  deriving (Eq, Show, Ord)</pre><p>The “BookAttribute” name is the type constructor, it specifies the name of the data type. This is what we use in e.g. function signatures. The pipe operators divide the options for being a book attribute under the umbrella. The “Pages”, “Authors”, “Title” and “Published” names are the data constructors, which are used to specify pieces of data of type BookAttribute. The data types accompanying the data constructors specify their field value types, with or without specifying field names. If we specify more than one field for a constructor it’s mandatory to specify field names in Daml (which is not manadatry in Haskell). (Under the hood, the value constructors are functions, mapping their inputs to a BookAttribute value. This is the reason why they are not namespaced under the wrapper datatype, and must be unique within a module.) The “deriving” clause is necessary so that we can check for equality, display and order pieces of BookAttribute data using default operators, without having to write explicitly these operators for this data type.</p><p>In order to mimic a real-world business model more closely, let’s create a Daml contract template with a field which contains a set of book attributes (we don’t use isolated pieces of book attributes, right?). The template below is a slightly modified version of the default skeleton template which is loaded when you start a new Daml project. The Asset can be thought of as a very simple version of an NFT, representing a real-world book, and can be given to somebody else by its owner.</p><pre>template Asset<br>  with<br>    issuer     : Party<br>    owner      : Party<br>    id         : Text<br>    attributes : Set BookAttribute<br>  where<br>    ensure id /= &quot;&quot;<br>    signatory issuer<br>    observer owner<br>    choice Give : AssetId<br>      with<br>        newOwner : Party<br>      controller owner<br>      do create this with<br>           owner = newOwner</pre><p>So this kind of expressiveness and type safety is what I mean when I speak about Daml being business-friendly. It turns out that the Rust types generated by the Rust Daml binding are as expressive as the original Daml data types. You will appreciate this similarity when we compare it with the alternative client language options.</p><h3>Expressive generated Rust types</h3><p>The generated Rust type for the Asset template payload, which we need to create an Asset contract instance looks like this:</p><pre>#[derive(Eq, PartialEq, Ord, PartialOrd, Clone, Debug)]<br>pub struct Asset {<br>    pub issuer: DamlParty,<br>    pub owner: DamlParty,<br>    pub id: DamlText,<br>    pub attributes:<br>        super::super::super::daml_stdlib_da_set_types::da::set::types::Set&lt;BookAttribute&gt;,<br>}</pre><p>(The code contains the full module path to the Set datatype, which is ugly but necessary, because the code generator doesn’t declare this path in a “use” statement.)</p><p>The BookAttribute data type used in the definition of the “attributes” field is the following:</p><pre>#[derive(Eq, PartialEq, Ord, PartialOrd, Clone, Debug)]<br>pub enum BookAttribute {<br>    Pages(DamlInt64),<br>    Authors(DamlList&lt;DamlText&gt;),<br>    Title(DamlText),<br>    Published(Published),<br>}</pre><p>Digging deeper into the Published struct:</p><pre>#[derive(Eq, PartialEq, Ord, PartialOrd, Clone, Debug)]<br>pub struct Published {<br>    pub year: DamlInt64,<br>    pub publisher: DamlText,<br>}</pre><p>Apart from the curly braces and some other syntax elements, pretty similar, right? One notable difference is that in Rust an enum variant, unlike a Daml ADT variant, cannot have fields itself, but instead can store a struct data type which wraps the fields. Another difference is that, as we will see shortly, in Rust enum variants are namespaced under the enum type, which in general gives more freedom: two different enums can have identical variants without causing a name clash.</p><p>The similarity is not a coincidence. Rust is not a functional language per se, but its creators <a href="https://doc.rust-lang.org/stable/book/ch13-00-functional-features.html">adopted</a> some design choices from functional languages. Among these is the expressive <a href="https://doc.rust-lang.org/stable/book/ch06-01-defining-an-enum.html">enum</a> datatype.</p><p>An important aside: A more fundamental similarity between Daml (Haskell) and Rust is, also based on ADT and enum respectivly, the lack of a null value. Neither of these languages contain a universal “nothing” value, which in other languages constitutes a crack on the shield of type checking, as a result of the infamous “<a href="https://doc.rust-lang.org/stable/book/ch06-01-defining-an-enum.html#the-option-enum-and-its-advantages-over-null-values">billion dollar mistake</a>”. Instead, the lack of a value is always a typed “nothing”, expressed by an ADT/enum variant, called <em>Nothing</em> (of type <em>Maybe a</em>) in Haskell, <em>None</em> (of type <em>Optional a</em>) in Daml, and <em>Option&lt;a&gt;::None</em> in Rust.</p><p>Now let’s see how we can specify values in Daml vs Rust.</p><p>The following piece of Daml data, a possible value for an Asset contract payload (in Daml, the name of a contract template is at the same time the name of the record data type holding the contract payload): …</p><pre>import DA.Set qualified as S<br><br>let asset = Asset with<br>        issuer = alice<br>        owner = alice<br>        id = &quot;abc123&quot;<br>        attributes = S.fromList [ Pages 104<br>                                , Authors [&quot;Lewis Carroll&quot;, &quot;John Tenniel&quot;]<br>                                , Title &quot;Alice&#39;s Adventures in Wonderland&quot;<br>                                , Published 1865 &quot;Macmillan&quot;]</pre><p>… can be expressed using the generated Rust types as follows:</p><pre>fn main() {<br><br>    let attributes_map : DamlGenMap&lt;BookAttribute, ()&gt; = DamlGenMap::from([<br>        (BookAttribute::Pages(104), ()),<br>        (BookAttribute::Authors(DamlList::from([&quot;Lewis Carroll&quot;.to_string(), &quot;John Tenniel&quot;.to_string()])), ()),<br>        (BookAttribute::Title(&quot;Alice&#39;s Adventures in Wonderland&quot;.to_string()), ()),<br>        (BookAttribute::Published(Published{ year: 1865, publisher: &quot;Macmmillan&quot;.to_string() }), ())<br>    ]);<br>    let attributes: Set&lt;BookAttribute&gt; = Set::new(attributes_map);<br>    let asset:Asset = Asset::new(&quot;alice&quot;, &quot;alice&quot;, &quot;abc123&quot;, attributes );<br><br>    dbg!(asset);<br>}</pre><p>The construction of the set of book attributes is a little bit involved. Under the hood, a set is a map which maps its elements to the unit value, and there is no constructor function which hides this implementation detail. (This is <a href="https://discuss.daml.com/t/is-there-a-way-to-specify-a-daml-set-value-directly-using-the-generated-typescript-and-java-code/6402/">true</a> for other client languages, see later.)</p><p>Otherwise the syntax for creating a fairly complex piece of data is clean and readable.</p><h3>Future vision: UI widgets generated dynamically from the generated Rust types</h3><p>This section is short, because I didn’t try anything from what I write here, I’m just speculating.</p><p>It would be great to be able to generate and update UI widgets dynamically, based on the Daml model/generated client code, in a similar fashion as <a href="https://discuss.daml.com/t/how-do-you-generate-dynamically-in-navigator-the-ui-input-widgets/6396">Navigator</a> works currently. This would alleviate the burden of updating the UI code each time the Daml model changes.</p><p>Another concern is sometimes that teams use use two sets of generated code: TypeScript for the UI, and Java for the middleware. If we could generate the UI widgets from the generated code used for the middleware, we could simplify the technology stack.</p><p>My hunch is that all this will be pretty much doable, maybe not tomorrow but soon, as Rust is on its way be become a viable UI language. Let’s see some hints:</p><p>As the generated Rust code for ADT’s contains enums and structs, we need a way to enumerate enum variants to populate dropdown lists representing enum variants dynamically. The <a href="https://crates.io/crates/strum">strum</a> crate does just this: “Strum is a set of macros and traits for working with enums and strings easier in Rust.”</p><p>Rust is not a widely used GUI language, but there are some libraries already. The website <a href="https://www.areweguiyet.com/">Are we GUI yet?</a> e.g. lists some of these libraries.</p><h3>Inside-out programming for harmonized applications</h3><p>Let’s take a step back for a moment and take a look at the purpose we are serving with Daml applications.</p><p>By “inside-out” I mean that start thinking at the core of the application progressing towards the end users.</p><p>The core of a shared transaction system and virtual system of record is the common Daml model.</p><p>The fact that we can create a seamless fit between the common Daml code and the client applications, means that we create harmony along this pipeline:</p><p>BUSINESS MODEL &gt;&gt; DAML MODEL &gt;&gt; GENERATED CODE &gt;&gt; CLIENT-SIDE DATA REPRESENTATION &gt;&gt; UI</p><p>Which means at the end of the day that e.g. users of client systems of different members of the business community use harmonized applications, without the need to actually harmonizing their systems other than connecting to the same Daml model.</p><h3>What about the alternative client languages?</h3><p>Now let’s go back to the topic of the expressiveness and type safety of the client-side code.</p><p>You will appreciate the expressiveness and type safety of the generated Rust types if you compare them to the way other client library languages express the same kind of data, namely a set of variants.</p><p>For the<strong> JSON API</strong>, the following payload is needed. You can observe that both the variant and the set datatype representation is a bit involved. It’s needless to say that on this JSON object there is not type checking whatsoever, so you will only know if the JSON is not compatible with the Daml model when the ledger API rejects the submission.</p><pre>{<br>    &quot;issuer&quot;: &quot;Alice::1220026465cd981d159dae404a92f96477a3aab5eb5cc37e924363c43b7203074d47&quot;,<br>    &quot;owner&quot;: &quot;Alice::1220026465cd981d159dae404a92f96477a3aab5eb5cc37e924363c43b7203074d47&quot;,<br>    &quot;id&quot;: &quot;abc123&quot;,<br>    &quot;attributes&quot;: {<br>      &quot;map&quot;: [<br>        [<br>          {<br>            &quot;tag&quot;: &quot;Pages&quot;,<br>            &quot;value&quot;: 104<br>          },<br>          {}<br>        ],<br>        [<br>          {<br>            &quot;tag&quot;: &quot;Authors&quot;,<br>            &quot;value&quot;: [<br>              &quot;Lewis Carroll&quot;,<br>              &quot;John Tenniel&quot;<br>            ]<br>          },<br>          {}<br>        ],<br>        [<br>          {<br>            &quot;tag&quot;: &quot;Title&quot;,<br>            &quot;value&quot;: &quot;Alice&#39;s Adventures in Wonderland&quot;<br>          },<br>          {}<br>        ],<br>        [<br>          {<br>            &quot;tag&quot;: &quot;Published&quot;,<br>            &quot;value&quot;: {<br>              &quot;year&quot;: 1865,<br>              &quot;publisher&quot;: &quot;Macmillan&quot;<br>            }<br>          },<br>          {}<br>        ]<br>      ]<br>    }<br>  }</pre><p>For the <strong>Python binding</strong>, you have to produce the corresponding structure using Python dicts and lists, so I will skip that for now.</p><p>The generated code for the <strong>TypeScript binding</strong> doesn’t contain classes (so we don’t get value constructors with it). It contains type aliases which can be used for type checking. The type alias mirroring the BookAttribute Daml ADT looks like as follows:</p><pre>export declare type BookAttribute =<br>  |  { tag: &#39;Pages&#39;; value: damlTypes.Int }<br>  |  { tag: &#39;Authors&#39;; value: string[] }<br>  |  { tag: &#39;Title&#39;; value: string }<br>  |  { tag: &#39;Published&#39;; value: BookAttribute.Published }</pre><p>Using this type annotation, we can declare the Asset payload as follows:</p><pre>let attributesMap : damlTypes.Map&lt;BookAttribute, {}&gt; = damlTypes.emptyMap();<br><br>let pages: BookAttribute = {tag: &quot;Pages&quot;, value: &quot;104&quot;};<br>attributesMap = attributesMap.set(pages, {});<br><br>let authors: BookAttribute = {tag: &quot;Authors&quot;, value: [&quot;Lewis Carroll&quot;, &quot;John Tenniel&quot;]};<br>attributesMap = attributesMap.set(authors, {});<br><br>let title: BookAttribute = {tag: &quot;Title&quot;, value: &quot;Alice&#39;s Adventures in Wonderland&quot;};<br>attributesMap = attributesMap.set(title, {});<br><br>let published: BookAttribute = {tag: &quot;Published&quot;, value: {year: &quot;1865&quot;, publisher: &quot;Macmillan&quot;}};<br>attributesMap = attributesMap.set(published, {});<br><br>let attributes : Set&lt;BookAttribute&gt; = { map : attributesMap };<br><br>const asset: Main.Asset = {<br>    issuer: &quot;alice&quot;,<br>    owner: &quot;alice&quot;,<br>    id : &quot;abc123&quot;,<br>    attributes : attributes<br><br>}</pre><p>The BookAttribute type annotation will prevent us from declaring variants where the tag string is not the name of an allowed BookAttribute variant, and the value is not of the corresponding type.</p><p>Java doesn’t know the notion of ADT. The code generated for the <strong>Java binding</strong> applies a workaround: the equivalent of the Daml type constructor (BookAttribute) is represented by an abstract class, and the Daml value constructors (Pages, Authors, Title, Published) are represented by classes which extend BookAttribute.</p><pre>public abstract class BookAttribute extends Variant&lt;BookAttribute&gt; {...}<br><br>public class Authors extends BookAttribute {...}<br><br>public class Pages extends BookAttribute {...}<br><br>public class Published extends BookAttribute {...}<br><br>public class Title extends BookAttribute {...}</pre><p>You can find a detailed description of this implementation in the <a href="https://docs.daml.com/app-dev/bindings-java/codegen.html#variants-a-k-a-sum-types">Daml docs</a>. (Actually, I have chosen BookAttribute as an illustration so that I can refer to this section of the docs.)</p><p>This workaround of course is not type safe: nothing prevents us from creating by hand another subclass extending BookAttribute, like Isbn.</p><pre>public class Isbn extends BookAttribute {...}</pre><p>The Java compiler will accept an Isbn value as BookAttribute. The following code will compile, and you will only know that it’s not compatible with the Daml model (because Isbn is not a valid value constructor for the BookAttribute Daml type) when the Daml ledger API rejects the submission which contains it.</p><pre>public class Main {<br><br>  public static void main(String[] args) {<br><br>      Pages pages = new Pages(104L);<br>      Authors authors = new Authors(List.of(&quot;Lewis Carroll&quot;, &quot;John Tenniel&quot;));<br>      Title title = new Title(&quot;Alice&#39;s Adventures in Wonderland&quot;);<br>      Published published = new Published(1865L, &quot;Macmillan&quot;);<br>      Isbn isbn = new Isbn(&quot;9780440000754&quot;);<br>      Map&lt;BookAttribute, Unit&gt; attribuesMap = Map.of(<br>              pages, Unit.getInstance(),<br>              authors, Unit.getInstance(),<br>              title, Unit.getInstance(),<br>              published, Unit.getInstance(),<br>              // Delete the line below so that<br>              // Daml accepts the input.<br>              isbn, Unit.getInstance()<br>      );<br>      Set&lt;BookAttribute&gt; attributes = new Set&lt;&gt;(attribuesMap);<br>      Asset asset = new Asset(&quot;alice&quot;, &quot;alice&quot;, &quot;abc123&quot;, attributes);<br><br>      System.out.println(asset);<br>    <br>}<br>}</pre><p>With that said, this lack of type safety, though not elegant, is not likely to cause any problems in practice, because we don’t have any good reason to extend BookAttribute by hand. If the set of variants changes in the Daml model, we apply codegen again which will generate the correct new set of subclasses. (I ignore here the criminal case when a malicious agent is able to infiltrate your organization and wants to cause a DOS situation exploiting this gap.)</p><h3>Conclusion</h3><p>When you make a decision about which client language/library to use, there are several factors to consider.</p><p>Probably there are limitations encoded in the context in the first place: architecture decisions, technology stack used by the connected systems and/or the project, preferred languages of the team, etc. Given the fact that Python, Java and JavaScript/TypeScript are among the most widely used languages currently, there is a good chance that your project will choose one of these as a client language.</p><p>Another consideration is that the current version of the Daml Rust binding doesn’t support (<a href="https://github.com/fujiapple852/rust-daml-bindings/issues/285">yet</a>) the latest Daml-LF version which is needed to use Daml interfaces. (Daml-LF is the underlying “raw” language which actually encodes Daml data, produced by the Daml compiler.) So if you need Daml interfaces you currently cannot use the Rust binding (unless you are willing to put sum effort into extending it).</p><p>But if speed, memory safety, thread safety, expressiveness, type safety and <a href="https://survey.stackoverflow.co/2022/#section-most-loved-dreaded-and-wanted-programming-scripting-and-markup-languages">cool factor</a> are on top of the list of your non-functional requirements and you don’t need Daml interfaces (or are willing to put in some extra effort), the Rust binding can be a viable choice.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddb74922a263" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daml-masterclass/rust-daml-a-biz-friendly-smart-contract-platform-deserves-a-biz-friendly-client-language-ddb74922a263">Rust + Daml: a biz-friendly smart contract platform deserves a biz-friendly client language</a> was originally published in <a href="https://medium.com/daml-masterclass">DAML Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we reverse Daml transactions?]]></title>
            <link>https://medium.com/daml-masterclass/can-we-reverse-daml-transactions-704eb1f7324c?source=rss-42425608db34------2</link>
            <guid isPermaLink="false">https://medium.com/p/704eb1f7324c</guid>
            <category><![CDATA[daml]]></category>
            <category><![CDATA[transactions]]></category>
            <category><![CDATA[reversal]]></category>
            <dc:creator><![CDATA[György Balázsi]]></dc:creator>
            <pubDate>Mon, 13 Mar 2023 10:33:20 GMT</pubDate>
            <atom:updated>2023-03-13T11:49:26.499Z</atom:updated>
            <content:encoded><![CDATA[<p>TLDR; The short answer: no. A little bit longer answer: it depends. An even longer answer: Reversing business transactions, implemented in a Daml model, is as easy or difficult as finding the correct legal construct for the reversal. Non-functional requirements can further limit the range of viable options.</p><h3>Daml transactions vs business transactions</h3><p>An important distinction: Daml transactions are not the same as business transactions.</p><p>A Daml transaction is a bundle of ledger actions where the actions are accepted or rejected in an “all or nothing” manner. The way the actions are bundled is determined by the Daml model. The results of the actions on the ledger, once accepted, are called events. The actions can belong to one of four types: contract creation, exercise of a choice on an existing contract, contract fetching and key assertion.</p><p>The term “business transaction” on the other hand, is a vague term. Usually, it refers to a bundle of business actions, where the meaning of “action” and “bundle” is also vaguely defined. The intent behind calling a set of business actions a business transaction may be similar to what we call a Daml transaction, but there is no guarantee for a correct translation.</p><h3>What “reversal” means — Daml transactions</h3><p>Similarly to the difference between Daml transactions and business transactions, there is a difference between speaking about reversing a Daml transaction or about reversing a business transaction.</p><p>The Daml transaction case is simple: a Daml transaction cannot be reversed.</p><p>A Daml ledger is a growing array of committed transactions. Nothing can be deleted from the ledger. Any right or obligation previously recorded on the ledger in the form of one or more Daml contracts, can be eliminated only by recording consuming events with regards to those contracts. Of course, whether or not such a consuming event can be recorded on the ledger depends on the Daml model and the set of active contracts.</p><p>Let’s see an example, the ledger which is the result of the setup script of the skeleton template (the Daml model you get by default after creating a new Daml project by issuing the commanddaml new &lt;your-project-name&gt;). The ledger aggregates the result of three changes: 1) Alice creating a TV asset, 2) Alice giving the asset to Bob, 3) Bob giving the asset back to Alice. All these changes are recorded by not changing or deleting previously recorded events but adding new events, some of which consume previously created contracts.</p><p>The nodes of the graph below represent events recorded on the ledger. In the top row you can see the top level events, which were directly requested by ledger parties. The events situated in the same column and having the same id prefix are the events which belong to the same atomic transaction. The blue nodes stand for choice exercises which don’t appear in the active contract set but constitute the glue between contracts; the red nodes stand for contract creation events for such contracts which were later declared consumed by choice exercise events so they were previously but are not currently part of the active contract set; the green node stands for the creation of the contract which currently constitutes the active contract set:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*inbxGZVUCfHBTYlUvwzm9w.jpeg" /><figcaption>See more details in <a href="https://discuss.daml.com/t/drawing-the-transaction-graph-of-a-running-daml-ledger/6249">this Daml Forum post</a></figcaption></figure><h3>What “reversal” means — business transactions</h3><p>The exact meaning of “reversing” a business transaction, similarly to the very notion of a business transaction, is vague. We might mean by that e.g. the “restoration of the original state” in some sense, where “original” means some previous point in time, and “state” is a set of parameters which need to be specified.</p><p>Whether of not this is possible on a Daml ledger, and if possible how difficult it is, depends on whether or not the “restoration of the original state” can be defined in correct legal terms as the creation/elimination of rights and obligations, and if yes how difficult it is.</p><h3>The Daml constraints are the constraints of contract law</h3><p>A Daml model implements the legal architecture of a business model.</p><p>On a Daml ledger we can only represent relationships between ledger parties in accordance with the offer-acceptance pattern of contract law.</p><p>The principles of legal rights and obligations are expressed in Daml terms as follows (see more info in the Daml docs section linked below):</p><ul><li>Obligations need consent.</li><li>Consent is needed to take away on-ledger rights.</li><li>On-ledger obligations cannot be unilaterally escaped.</li></ul><p><a href="https://docs.daml.com/concepts/ledger-model/ledger-integrity.html">Integrity - Daml SDK 2.5.5 documentation</a></p><p>This means that if a business transaction established/eliminated some rights and/or obligations for some ledger party or parties, the reversal, that is the elimination/re-establishment of these rights/obligations respectively can also happen only by observing these rules.</p><h3>Easy use case #1: give back the asset</h3><p>There are some use cases where implementing a correct legal model (and a viable Daml model accordingly) for the reversal of business transactions is not too difficult. Some additional off-ledger complexity may arise as a result, as we will see. I will briefly describe three such use cases.</p><p>We can notice that the bare-bones use case implemented by the skeleton template used above as illustration already contains the possibility to reverse the business transaction which consists of giving away the asset: the new owner can give it back to the original owner.</p><p>We can also notice that the decision about reversing the business transaction is totally in the hands of the new owner though. The old owner can take no on-ledger action to facilitate this. This “good faith” based reversal option may or may not be what your business model actually requires.</p><h3>Easy use case #2: result contract with „reverse” choice</h3><p>If you want to give control about the reversal of the business transaction in the hands of the old owner of the asset, you can extend the Daml model with a contract expressing the result of giving away the asset, containing an option for the old owner to reverse it by consuming the given asset and creating an identical one for the old owner.</p><p>The challenge here is that the new owner may give the asset away to a third ledger party before the old owner decides to take it back. In this case the reversal will be rejected by the ledger.</p><p>A more advanced version can be that the result contract with the reversal option locks the asset, preventing the new owner from giving it away (the locking can be easily implemented by including a fetch step in the give choice) — but in this case giving away the asset actually doesn’t give control about it to the new owner. From a business perspective though, this can be ok, if the ownership of the asset yields some other benefits to the owner. A middle ground can be that the reversal option and the locking effect has an expiration or is deleted/released by some trigger.</p><h3>Easy use case #3: purely additive delta events</h3><p>Yet another use case where the reversal of the business transaction is easy to implement when we record delta events on the ledger rather than states. This is what happens e.g. in double-entry bookkeeping where we record credit and debit events. In this case the effect of a debit or credit event can be offset by recording a reverse event: credit for debit and debit for credit. The controller of the reversal of the business transaction may be the same ledger party which was the controller of the original transaction.</p><p>While this pattern is easy to implement in a Daml model, creates additional complexity on the client side. The balance of accounts is not stored on the ledger, but needs to be computed every time by aggregating the debit and credit events by a client application. This can be a problem when a large number of accounts needs to be processed within a short period of time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=704eb1f7324c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/daml-masterclass/can-we-reverse-daml-transactions-704eb1f7324c">Can we reverse Daml transactions?</a> was originally published in <a href="https://medium.com/daml-masterclass">DAML Masterclass</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>