<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jeff Foster on Medium]]></title>
        <description><![CDATA[Stories by Jeff Foster on Medium]]></description>
        <link>https://medium.com/@jeff-foster?source=rss-ec6ca6c48add------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rg35K1JNGgG6ddBy9g5j3Q.png</url>
            <title>Stories by Jeff Foster on Medium</title>
            <link>https://medium.com/@jeff-foster?source=rss-ec6ca6c48add------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:43:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jeff-foster/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[It only takes one to Tango]]></title>
            <link>https://jeff-foster.medium.com/it-only-takes-one-to-tango-92665433017d?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/92665433017d</guid>
            <category><![CDATA[tango]]></category>
            <category><![CDATA[linkedin]]></category>
            <category><![CDATA[puzzle]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Sun, 16 Feb 2025 13:40:41 GMT</pubDate>
            <atom:updated>2025-02-16T13:40:41.982Z</atom:updated>
            <content:encoded><![CDATA[<p>LinkedIn has a logic game called Tango. I’ve become mildly obsessed with it. The rules of the game are simple.</p><ul><li>It’s played on a 6x6 grid.</li><li>The grid must be filled with symbols (either X or Y)</li><li>No more than 2 X or 2 Y may be next to each other, either vertically or horizontally</li><li>Each row and column must contain the same number of X or Y</li><li>Cells separated by an = sign must contain the same type</li><li>Cells separated by an x sign must be of the opposite symbol</li><li>Each puzzle has one right answer and can be solved via deduction. No guesses required.</li></ul><p>Here’s an example of a fully filled puzzle with some pretty graphics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*rbYMH9AtQHcm3haapk5MNg.png" /></figure><h3>Let’s model it.</h3><p>It’s really easy to model the game and write a solver. So, let’s not do that. Instead, we’re going to try and generate <em>really </em>difficult problems. We’ll struggle with brute force since a 6x6 grid has 36 cells, and 2 values per cell. Brute force gives 2³⁶ options so that’s out of the question.</p><p>Instead, we’ll use backtracking to generate all possible solutions of laying out the symbols. We’ll ignore constraints like = and x for now. We’ll try and be a bit smarter too and only generate canonical forms of the puzzle. We’ll define canonical by setting the top left cell to a fixed value, finding all the possible rotations and using the minimal string representation. Here’s a chunk of code that does just that.</p><pre>    private static char[,] Canonicalize(Tango solution)<br>    {<br>        // Extract the solution’s 6x6 arrangement<br>        char[,] original = new char[6, 6];<br>        for (int r = 0; r &lt; 6; r++)<br>            for (int c = 0; c &lt; 6; c++)<br>                original[r, c] = solution.GetCell(r, c);<br><br>        // Generate the 4 possible rotations<br>        var rotations = new List&lt;char[,]&gt;(4)<br>        {<br>            original,<br>            Rotate90(original),<br>            Rotate90(Rotate90(original)),<br>            Rotate90(Rotate90(Rotate90(original)))<br>        };<br><br>        return rotations<br>            .Select(x =&gt; (x[0, 0] == &#39;Y&#39;) ? <br>                Flip(x) : CopyGrid(x)).MinBy(GridToString);<br>    }</pre><p>To do the backtracking, we start from the top left and fill in a symbol. If the rules of the grid are still upheld, we repeat the process recursively until we’re done. We back-track once we’ve explored all possible symbols.</p><pre>    public static IEnumerable&lt;Tango&gt; EnumerateSolutions(Tango tango)<br>    {<br>        return EnumerateSolutions(0, 0, tango);<br>    }    <br><br>    private static IEnumerable&lt;Tango&gt; EnumerateSolutions(<br>        int row, <br>        int col, <br>        Tango tango)<br>    {<br>        if (row == Tango.GRID_SIZE)<br>        {<br>            // Finished grid; yield a copy of the solved grid.<br>            var solution = new char[Tango.GRID_SIZE, Tango.GRID_SIZE];<br>            Array.Copy(tango.Grid, solution, tango.Grid.Length);<br>            yield return new Tango(solution, tango.Constraints);<br>            yield break;<br>        }<br><br>        // Compute next cell coordinates.<br>        int nextRow = row, nextCol = col + 1;<br>        if (nextCol == Tango.GRID_SIZE)<br>        {<br>            nextRow++;<br>            nextCol = 0;<br>        }<br><br>        // If cell is fixed, skip to the next.<br>        if (tango.IsFixed[row, col])<br>        {<br>            foreach (var sol in EnumerateSolutions(nextRow, nextCol, tango))<br>                yield return sol;<br>            yield break;<br>        }<br><br>        // Try both possible symbols.<br>        foreach (var symbol in new char[] { &#39;X&#39;, &#39;Y&#39; })<br>        {<br>            tango.Grid[row, col] = symbol;<br>            // Only continue if the partial grid is still valid.<br>            if (tango.IsValid())<br>            {<br>                foreach (var sol in EnumerateSolutions(nextRow, nextCol, tango))<br>                    yield return sol;<br>            }<br>        }<br><br>        // Backtrack.<br>        tango.Grid[row, col] = &#39; &#39;;<br>    }</pre><p>We can find all possible solutions without constraints in a second or so. There are 1440 possible permutations of filled grids that preserve the rules (assuming I’ve written the code correctly).</p><p>I don’t know why it’s 1440, and I wish I knew enough combinatorics to work it out. Answers appreciated!</p><h3>Let’s make a hard puzzle</h3><p>Now that we’ve got all possible grids, let&#39;s try and make a puzzle. We’ll attack this logically. For each of the 1440 canonical forms, we’ll go through the grid, remove as many cells as we can whilst leaving one unique solution.</p><p>Here’s some code that does that. It’s not particularly sophisticated. It just keeps trying to unfix cells until it can’t.</p><pre>public class PuzzleMinimizer<br>{<br>    public static Tango Minimize(Tango puzzle)<br>    {<br>        while (TrySimplifyPuzzle(puzzle, includeConstraints)) { }<br>        return puzzle;<br>    }<br><br>    private static bool TrySimplifyPuzzle(Tango puzzle)<br>    {<br>        for (int row = 0; row &lt; 6; row++)<br>        {<br>            for (int col = 0; col &lt; 6; col++)<br>            {<br>                if (!puzzle.IsCellFixed(row, col)) continue;<br><br>                // Try removing the fixed cell<br>                if (CanRemoveCell(puzzle, row, col))<br>                {<br>                    puzzle.UnfixCell(row, col);<br>                    return true;<br>                }<br>            }<br>        }<br>        return false;<br>    }<br><br>    private static bool CanRemoveCell(Tango puzzle, int row, int col)<br>    {<br>        var testPuzzle = puzzle.DeepCopy();<br>        testPuzzle.UnfixCell(row, col);<br>        return HasUniqueSolution(testPuzzle);<br>    }<br><br>    private static bool IsValidPosition(int row, int col) =&gt;<br>        row &gt;= 0 &amp;&amp; row &lt; 6 &amp;&amp; col &gt;= 0 &amp;&amp; col &lt; 6;<br><br>    private static bool HasUniqueSolution(Tango puzzle) =&gt;<br>        TangoSolver.EnumerateSolutions(puzzle).Take(2).Count() == 1;<br>}</pre><p>We can run this on all of our 1440 full solutions, and we find that we need no more than 11 cells (out of 36) for any of the Tango puzzles.</p><p>Here is the hardest puzzle of the lot, which only has six cells set and has a single unique solution.</p><pre>. . . . . .<br><br>. . . . . .<br><br>. . . X . .<br><br>. Y Y . . .<br><br>. . Y . . .<br><br>. . . . X X</pre><p>But this still isn’t very pleasant as a puzzle. It’s quite boring, and it’s not hard. Give it a try!</p><h3>Let’s make a harder problem</h3><p>We’ll use the same back-tracking approach that we used before, but now we’ll give the ability to add a constraint too. If we can’t remove a cell, we’ll try and add a constraint between the cell we’re on and any of the immediate neighbours.</p><pre>public class PuzzleMinimizer<br>{<br>    public static Tango Minimize(Tango puzzle, bool includeConstraints)<br>    {<br>        while (TrySimplifyPuzzle(puzzle, includeConstraints)) { }<br>        return puzzle;<br>    }<br><br>    private static bool TrySimplifyPuzzle(Tango puzzle, bool includeConstraints)<br>    {<br>        for (int row = 0; row &lt; 6; row++)<br>        {<br>            for (int col = 0; col &lt; 6; col++)<br>            {<br>                if (!puzzle.IsCellFixed(row, col)) continue;<br><br>                // Try removing the fixed cell<br>                if (CanRemoveCell(puzzle, row, col))<br>                {<br>                    puzzle.UnfixCell(row, col);<br>                    return true;<br>                }<br><br>                // Try replacing with a constraint<br>                if (includeConstraints &amp;&amp; TryReplaceWithConstraint(puzzle, row, col))<br>                {<br>                    return true;<br>                }<br>            }<br>        }<br>        return false;<br>    }<br><br>    private static bool CanRemoveCell(Tango puzzle, int row, int col)<br>    {<br>        var testPuzzle = puzzle.DeepCopy();<br>        testPuzzle.UnfixCell(row, col);<br>        return HasUniqueSolution(testPuzzle);<br>    }<br><br>    private static bool TryReplaceWithConstraint(Tango puzzle, int row, int col)<br>    {<br>        var adjacentCells = GetAdjacentFixedCells(puzzle, row, col);<br>        var constraintTypes = new[] { Tango.ConstraintType.Same, Tango.ConstraintType.Different };<br><br>        foreach (var (r2, c2) in adjacentCells)<br>        {<br>            foreach (var constraintType in constraintTypes)<br>            {<br>                var testPuzzle = puzzle.DeepCopy();<br>                testPuzzle.UnfixCell(row, col);<br>                testPuzzle.AddConstraint(row, col, r2, c2, constraintType);<br><br>                if (HasUniqueSolution(testPuzzle))<br>                {<br>                    puzzle.UnfixCell(row, col);<br>                    puzzle.AddConstraint(row, col, r2, c2, constraintType);<br>                    return true;<br>                }<br>            }<br>        }<br>        return false;<br>    }<br><br>    private static IEnumerable&lt;(int row, int col)&gt; GetAdjacentFixedCells(Tango puzzle, int row, int col)<br>    {<br>        var directions = new[] { (-1, 0), (1, 0), (0, -1), (0, 1) };<br>        <br>        return directions<br>            .Select(d =&gt; (row: row + d.Item1, col: col + d.Item2))<br>            .Where(pos =&gt; IsValidPosition(pos.row, pos.col) &amp;&amp; puzzle.IsCellFixed(pos.row, pos.col));<br>    }<br><br>    private static bool IsValidPosition(int row, int col) =&gt;<br>        row &gt;= 0 &amp;&amp; row &lt; 6 &amp;&amp; col &gt;= 0 &amp;&amp; col &lt; 6;<br><br>    private static bool HasUniqueSolution(Tango puzzle) =&gt;<br>        TangoSolver.EnumerateSolutions(puzzle).Take(2).Count() == 1;<br>}</pre><p>This allows us to reduce every single possible solution to a grid with just a single symbol set! This makes intuitive sense to me as an = symbol forces a second cell to be set (and so on). Given the symmetry aspect of the grid, one symbol is obviously the minimum that can be set.</p><p>But what about the minimum number of constraints?</p><h3>Finally, the hardest problem?</h3><p>What do we define as the hardest problem? Let’s iterate over all possible grids and find the minimum number of symbols and constraints that are set.</p><pre>        var possibleSolutions = TangoSolver.GetMinimalSolutions(new Tango());              <br>        <br>        var hardestGrid = possibleSolutions.MinBy(x =&gt; <br>            {<br>                var minPuzzle = PuzzleMinimizer.Minimize(x, true);<br>                return minPuzzle.CellsSet() + minPuzzle.ConstraintsSet();<br>            }<br>        );</pre><p>This isn’t a fully exhaustive search. We’re only considering if an = or x removes constraints locally. I haven’t convinced myself yet that there isn’t a constraint you could put elsewhere in the grid to remove ambiguity.</p><p>Here is the <strong>hardest Tango puzzle</strong> I could find. It has six elements (five constraints and a single symbol).</p><pre>. . . . . .<br><br>. . . . . .<br><br>. . . . . .<br>    =<br>. . . . . .<br>      = x<br>. . . . . .<br>          =<br>. . . . .=X</pre><p>It’s pretty hard!</p><p>But is it the hardest problem? I don’t know. My code is <a href="https://github.com/fffej/tango">here</a>. Suggestions for improvement welcome.</p><p>So, here’s your challenge — can you find a Tango puzzle harder than this?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92665433017d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Playing with Transformers]]></title>
            <link>https://medium.com/ingeniouslysimple/playing-with-transformers-f97eb16718d0?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/f97eb16718d0</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Fri, 07 Feb 2025 16:31:39 GMT</pubDate>
            <atom:updated>2025-02-07T16:32:26.461Z</atom:updated>
            <content:encoded><![CDATA[<p>“<a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>” is the paper that launched large language models. In this article, we’ll try write some high-level code to implement a transformer model that you can train yourself (even without an expensive GPU).</p><p>We’re going to write a transformer architecture for characters. The goal of this model is just to predict the next character. I’m pretty sure it’ll be terrible, but characters are easier than words. There’s fewer of them.</p><p>We’re not going to try and explain the paper in detail, instead we’re going to demonstrate that you can glue a few lines of Python together and experience the transformer model. This should give you enough code so you can start fiddling with parameters like sequence length, volume of training data and the training process and just see what happens.</p><p>The <a href="https://jalammar.github.io/illustrated-transformer/">illustrated transformer</a> describes the internals far better than I can, so I’d encourage you to go and read that.</p><h3>Encoding a position</h3><p>One of the key parts of a transformer model is that they can process data in parallel. Our first task is to build the input to go in. Imagine we have a stream of data; our job is to break it up into tokens and turn each token into a vector that encodes both its value and its position.</p><p>In the example below we’ve used the awesome embedding of alphabetical order and we’ve got a couple of dictionary structures for turning a character into a number and back again. That’s the value part of the equation.</p><pre>vocab = sorted(set(text))<br>char_2_idx= {char: idx for idx, char in enumerate(vocab)}<br>idx_2_char = np.array(vocab)</pre><p>How do we add positional information? Well, once we’ve converted a token into a vector, we augment that with positioning information like this using the formulas straight from the paper.</p><pre>def get_angles(pos, i, d_model):<br>    angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))<br>    return pos * angle_rates<br><br>def positional_encoding(position, d_model):<br>    angle_rads = get_angles(np.arange(position)[:, np.newaxis],<br>                            np.arange(d_model)[np.newaxis, :],<br>                            d_model)<br>    # apply sin to even indices in the array; cos to odd indices<br>    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])<br>    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])<br>    pos_encoding = angle_rads[np.newaxis, ...]<br>    return tf.cast(pos_encoding, dtype=tf.float32)</pre><p>Why this arrangement with sin / cos? Think of it as a way of capturing the position in the set of any item. The future apart, the less impact they have. With a phrase like “peter picked a peck of pickled peppers” each p has a different vector. We can visualize this - this is a heatmap of the vector encoding for the text given before. The main thing to notice is that each repeated character is encoded differently because it’s now got position embedded in it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F0S4WuUM3bv-BzeAtXn6gw.png" /></figure><h3>Building our model</h3><p>Let’s build our model.</p><pre>def build_model(vocab_size, maxlen, embed_dim, num_heads, ff_dim, num_layers):<br>    inputs = tf.keras.Input(shape=(maxlen,))<br>    # Standard embedding layer<br>    x = tf.keras.layers.Embedding(input_dim=vocab_size, output_dim=embed_dim, mask_zero=True)(inputs)<br>    <br>    # Add positional encoding (you can also use a built-in positional embedding if you prefer)<br>    pos_encoding = positional_encoding(maxlen, embed_dim)<br>    x = x + pos_encoding[:, :maxlen, :]<br>    <br>    # The TransformerEncoder block internally handles multi-head attention, skip connections,<br>    # layer normalization, and the feedforward network.<br>    for _ in range(num_layers):<br>        x = keras_nlp.layers.TransformerEncoder(<br>            num_heads=num_heads,<br>            intermediate_dim=ff_dim,<br>            dropout=0.1<br>        )(x)<br>    <br>    outputs = tf.keras.layers.Dense(vocab_size)(x)<br>    return tf.keras.Model(inputs=inputs, outputs=outputs)</pre><p>What does all of this mean?</p><ul><li>vocab_size - the number of unique tokens. In our case the number of unique characters. It’s about 150 odd.</li><li>maxlen - the maximum size of sequences the model will process</li><li>embed_dim - the size of the embeddings (64 in our case)</li><li>num_heads - the number of attention heads in the multi-head attention model (I picked 4)</li><li>ff_dim - the number of dimensions in the feed forward networks hidden layer (I picked 128)</li><li>num_layers - how many layers of transformer blocks do we use. (I picked 2).</li></ul><p>We can put that all together and see how many parameters we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WaoR66xN80Jd6zCTGsJwUQ.png" /><figcaption>Parameters of the model</figcaption></figure><h3>Preparing some text data</h3><p>What shall we use for training? Well, since we’re just using characters, we’ll use <a href="https://www.gutenberg.org/">Project Gutenberg</a> and grab a whole bunch of text files.</p><pre>def split_input_target(chunk):<br>    input_text = chunk[:-1]<br>    target_text = chunk[1:]<br>    return input_text, target_text<br><br>def get_training_set(pathToFile: str):<br>    # Load the text file<br>    with open(pathToFile, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        text = f.read()<br>        text = &#39; &#39;.join(text.split())<br>        # Get sorted unique characters in the text<br>        vocab = sorted(set(text))<br>        vocab_size = len(vocab)<br>        # Create lookup dictionaries<br>        CHAR_2_IDX = {char: idx for idx, char in enumerate(vocab)}<br>        IDX_2_CHAR = np.array(vocab)<br>        text_as_int = np.array([CHAR_2_IDX[c] for c in text])<br>        seq_length = MAXLEN + 1  # extra character for the target<br>        # Create a dataset of individual characters<br>        char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)<br>        sequences = char_dataset.batch(seq_length, drop_remainder=True)<br>        dataset = sequences.map(split_input_target)<br>        return dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True), vocab_size, CHAR_2_IDX, IDX_2_CHAR</pre><p>What does this mess do? Load a bunch of text, get rid of repeated whitespace and then create a training set of “some text” followed by the last character. It’s the job of training to optimize that!</p><h3>Running training</h3><p>What does training do? Throw the training set through the model and see what it predicts. If it’s wrong, nudge the parameters in the right direction (back propagation) and repeat. We continue training until the loss function (e.g. how accurate it is) becomes acceptable. If you want a more in-depth view, try <a href="https://www.youtube.com/watch?v=MD2fYip6QsQ">this</a>.</p><p>We use an optimizer known as adam (adaptive momentum) and we run for a certain number of epochs. An epoch is a complete run through the training set.</p><pre>model.compile(optimizer=&quot;adam&quot;, loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True))<br>model.summary()<br>checkpoint_filepath = &#39;./checkpoint/foo.weights.h5&#39;<br>checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(<br>    filepath=checkpoint_filepath,<br>    save_weights_only=True,<br>    monitor=&#39;loss&#39;,<br>    mode=&#39;min&#39;,<br>    save_best_only=True<br>  )<br>with (tf.device(&#39;/GPU:0&#39;)):<br>  model.fit(trainingSet, epochs=EPOCHS, callbacks=[checkpoint_callback])</pre><p>When this runs, you’ll see the loss function described after each epoch. As the data fits better, the loss function lowers. In this case, the model converged very quickly and showed only small progress after a handful of epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7A7pKGPoO8ZfmI13ItXJIA.png" /><figcaption>Training converges quickly</figcaption></figure><h3>Doing predictions</h3><p>And for our final piece, we want to feed some tokens to the model and get its suggestion on what comes next.</p><pre>def predict_next_tokens(model, input_text, char2idx, idx2char, num_tokens=10, temperature=1.0):<br>    # Convert input text to integer indices (default to index 0 for unknown characters)<br>    input_ids = [char2idx.get(c, 0) for c in input_text]<br>    generated_ids = list(input_ids)<br>    for _ in range(num_tokens):<br>        # Ensure input length equals maxlen. If too short, left-pad with 0<br>        if len(generated_ids) &lt; MAXLEN:<br>            padded_input = [0] * (MAXLEN - len(generated_ids)) + generated_ids<br>        else:<br>            padded_input = generated_ids[-MAXLEN:]<br>        # Create tensor input with batch dimension<br>        input_tensor = tf.expand_dims(padded_input, 0)  # Shape: (1, maxlen)<br>        # Get model predictions. The output shape is (1, maxlen, vocab_size)<br>        predictions = model(input_tensor, training=False)<br>        # Sample the next token (you can use greedy: tf.argmax(last_logits) if you prefer)<br>        last_logits = predictions[0, -1, :]<br>        scaled_logits = last_logits / temperature<br>        predicted_id = tf.random.categorical(tf.expand_dims(scaled_logits, 0), num_samples=1)[0, 0].numpy()<br>        generated_ids.append(predicted_id)<br><br>    # Convert the sequence of token indices back to text<br>    generated_text = &#39;&#39;.join([idx2char[i] for i in generated_ids])<br>    return generated_text</pre><p>How’s this work? Turn our text into a vector, feed it into the model, decode it and repeat until we’ve generated enough tokens.</p><h3>Does it work?</h3><p>Well, work is a funny word.</p><ul><li>Does it work in the sense of being remotely useful? <strong>No</strong>.</li><li>Does it work in the sense of predicting the next character any better than you could do with just about any other method? <strong>No</strong>.</li><li>Does it serve as a handy way to heat your house if you put in the wrong numbers? <strong>Yes</strong>.</li><li>Does it allow me to play with a transformer model? <strong>Yes, and that’s all that matters</strong> 🙂</li></ul><p>I trained it on the complete works of Shakespeare. Here’s some examples:</p><blockquote><strong><em>Hamlet</em></strong><em>, beet the she hous live buls, Afteeve, Crid To she. Dout a will layitio’s be as is wists beffer the <br></em><strong><em>Macbeth</em></strong><em>er them of thy sook fto the vombect folk. Coulst, I shall thy comperved to his uport py won stole <br></em><strong><em>Einstein</em></strong><em>? Veelf Wou ame, it best gent of Trom I your the we you welm. You tart my bll live raty evame ever m</em></blockquote><h3>What next?</h3><p>I’ve got a system to play with now which is super cool! I can run through a training batch of 20MB of text in about 5 minutes on my GPU, which means I can experiment, change the shapes of the models, change the parameters and see what happens. Hopefully this means I can put some of what I intended to read in <a href="https://learning.oreilly.com/library/view/designing-machine-learning/9781098107956/">Designing ML Systems</a> and <a href="https://learning.oreilly.com/library/view/ai-engineering/9781098166298/">AI Engineering</a> into practice!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f97eb16718d0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/playing-with-transformers-f97eb16718d0">Playing with Transformers</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying Embeddings]]></title>
            <link>https://medium.com/ingeniouslysimple/demystifying-embeddings-f9028fd4024c?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/f9028fd4024c</guid>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Thu, 06 Feb 2025 09:02:06 GMT</pubDate>
            <atom:updated>2025-02-06T19:51:41.112Z</atom:updated>
            <content:encoded><![CDATA[<p>Vector embeddings are a numerical representation of text in a high-dimensional space. Within that high-dimensional space, semantically similar text should be close together. So, in theory the vectors for “king” and “queen” should be closer together.</p><p>Various <a href="https://www.sbert.net/docs/sentence_transformer/pretrained_models.html">pretrained models exist</a> for doing this, and writing the Python to convert a sentence into a vector is amazingly simple (thank you sentence-transfomers package!). The function below takes a list of strings and returns an array of each embedding.</p><p>An embedding (with this model at least) is a vector of size 384.</p><pre>def generate_embeddings(texts):<br>    &quot;&quot;&quot;Generate embeddings for a list of texts.&quot;&quot;&quot;<br>    model = SentenceTransformer(&#39;all-MiniLM-L6-v2&#39;)<br>    return model.encode(texts)</pre><p>What does this vector mean? Well, it’s a location is a 384-dimensional space of course. Yes, that didn’t help me much either. How can we interpret it?</p><p>One approach is to measure the “closeness” of any given embeddings. How do you find out how close two points are?</p><ul><li>Euclidean distance — this is the straight line distance and simple to calculate.</li><li>Manhattan distance — the distance if you could only move along grid lines.</li><li>Cosine similarity — the angle between the vectors (the closer to 1 the more aligned, -1 is the opposite)</li></ul><p>If we calculate these for queen, king and banana then we see what we expect. The cosine similarity is higher for related terms, and the distance is smaller.</p><pre>queen &lt;-&gt; king<br>Cosine Similarity: 0.681<br>Euclidean Distance: 0.799<br>Manhattan Distance: 12.253<br><br>queen &lt;-&gt; banana<br>Cosine Similarity: 0.397<br>Euclidean Distance: 1.099<br>Manhattan Distance: 17.292<br><br>king &lt;-&gt; banana<br>Cosine Similarity: 0.395<br>Euclidean Distance: 1.100<br>Manhattan Distance: 16.981</pre><p>But even this isn’t a great way of understanding it. What does the magnitude of the difference mean? Is it a lot, not enough? To visualize it properly, we need to convert that high dimensionality space into something a bit more useful.</p><p>We’ll use PCA (principal component analysis) to do dimensionality reduction (roughly, PCA projects data into a space that best captures variation in the data.</p><pre>def reduce_dimensions(embeddings, n_components=3):<br>    pca = PCA(n_components=n_components)<br>    transformed_data = pca.fit_transform(embeddings)<br>    variance_ratios = pca.explained_variance_ratio_</pre><p>This reduces those 384 dimensional vectors into a small 3D vector. We return the variance ratios which helps us understand how much variance these principal components capture. In my case with [king, queen, orange, banana] the first two principal components capture about 80% of the variance. We can now plot these to get an idea of how separated they are.</p><figure><img alt="Embeddings of some words" src="https://cdn-images-1.medium.com/max/911/1*VSJgjOtjP-WtXV1Gfj8Fvw.png" /></figure><p>You shouldn’t try and draw any conclusions about what the axis represent; they are simply a mathematical construct that captures the variation for this particular set. Sometimes folks like to imagine that a particular dimension in an embedding corresponds to some human intuition about the difference between words. That’s incredibly unlikely!</p><h3>So, how’s that useful?</h3><p>Our first application of this is for semantic search. How do embeddings help here? Well, we don’t just have to create embeddings of single words, we can throw entire sentences in. To illustrate this, let’s write some code to index every sentence in the <a href="https://www.gutenberg.org/cache/epub/100/pg100.txt">entire works of William Shakespeare</a>, plonk it in a <a href="https://faiss.ai/">vector database</a> and allow you to find the most similar sentences.</p><p>This sounds like it’d be a huge amount of work, but thanks to a whole bunch of Python libraries it’s about 100 lines of code (!).</p><pre>class SentenceIndexer:<br>    def __init__(self, model_name=&#39;all-MiniLM-L6-v2&#39;):<br>        nltk.download(&#39;punkt&#39;, quiet=True)<br>        self.model = SentenceTransformer(model_name)<br>        self.index = None<br>        self.sentences = []<br>    def load_and_split_text(self, file_path):<br>        with open(file_path, &#39;r&#39;, encoding=&#39;utf-8&#39;) as file:<br>            text = file.read()<br>        <br>        self.sentences = sent_tokenize(text)<br>        return self.sentences<br>    def create_embeddings(self):<br>        embeddings = self.model.encode(self.sentences, show_progress_bar=True)<br>        return embeddings<br>    def build_index(self, embeddings):<br>        dimension = embeddings.shape[1]<br>        self.index = faiss.IndexFlatL2(dimension)<br>        self.index.add(embeddings.astype(np.float32))<br>    def save_index(self, save_dir):<br>        os.makedirs(save_dir, exist_ok=True)<br>        faiss.write_index(self.index, os.path.join(save_dir, &quot;sentence_index.faiss&quot;))<br>        with open(os.path.join(save_dir, &quot;sentences.pkl&quot;), &#39;wb&#39;) as f:<br>            pickle.dump(self.sentences, f)<br>    def load_index(self, save_dir):<br>        self.index = faiss.read_index(os.path.join(save_dir, &quot;sentence_index.faiss&quot;))<br>        with open(os.path.join(save_dir, &quot;sentences.pkl&quot;), &#39;rb&#39;) as f:<br>            self.sentences = pickle.load(f)<br>    def search(self, query, k=5):<br>        query_embedding = self.model.encode([query])<br>        distances, indices = self.index.search(query_embedding.astype(np.float32), k)<br>        results = []<br>        for i, (dist, idx) in enumerate(zip(distances[0], indices[0])):<br>            results.append({<br>                &#39;sentence&#39;: self.sentences[idx],<br>                &#39;distance&#39;: dist,<br>                &#39;index&#39;: idx<br>            })<br>        <br>        return results</pre><p>On my machine (without any fancy GPU acceleration), it takes about 30 seconds to build the index. Sometimes it’s amazing how fast computers are!</p><p>Once you’ve got the index, it’s easily searchable. When a user enters some text, you calculate the embedding vector and find the nearest neighbours to that. Here’s an example:</p><pre>Query: Ouch, I&#39;ve bumped my head.<br>Most similar sentences:<br>1. Distance: 0.92<br>   Lord, how my head aches!<br>2. Distance: 1.01<br>   I’ll scratch your heads.<br>3. Distance: 1.01<br>   Scratch my head, Peaseblossom.<br>4. Distance: 1.04<br>   I have a pain upon my forehead here.<br>5. Distance: 1.11<br>   ne’er pull your hat upon your brows.</pre><p>Cool — semantic search in a few lines of code!</p><p>There’s plenty more applications of this. One popular one at the moment is Retrieval Augmented Generation (RAG), a technique for finding relevant context to help large language models give better answers. You feed in documents to a vector database as above, and when a user makes a query find related documents and add them into the context for the LLM.</p><p>Hopefully that’s demystified embeddings. And if it hasn’t, I’d encourage you to download some Python packages and play with it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f9028fd4024c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/demystifying-embeddings-f9028fd4024c">Demystifying Embeddings</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepSeek R1 + Sonnet]]></title>
            <link>https://medium.com/ingeniouslysimple/deepseek-r1-sonnet-ccd01a0f345b?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccd01a0f345b</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Sun, 26 Jan 2025 14:08:22 GMT</pubDate>
            <atom:updated>2025-01-26T14:08:22.760Z</atom:updated>
            <content:encoded><![CDATA[<p>I’m a big fan of Claude Sonnet. I’m ashamed to admit, it’s mostly vibes based. It’s friendlier and writes code in a way that I like. The <a href="https://modelcontextprotocol.io/">model context protocol</a> means I can hack integrations to my local environment and they’re great fun (favourite moment, allowing Sonnet to run arbitrary commands on my shell, and it reaching out to find the latest football scores with wget 🙂).</p><p>That got me wondering, can I get the best of both? But before we start…</p><h3>What is DeepSeek R1?</h3><p>DeepSeek R1 is a new model that has reasoning capabilities. It’s interesting because it’s trained at a relatively low cost, and early benchmarks show that it’s competitive with current SOTA models.</p><p>It’s great fun to use, Simon Willison has a <a href="https://simonwillison.net/2025/Jan/20/deepseek-r1/">great article</a> to allow you to start playing with it.</p><p>The neat thing about it is that it has a thinking stage that contains the chain of thought it uses to generate the final response. For example, if I ask a relatively straight-forward question such as “How do I estimate the time without a clock?” I get some “thinking (chain of thought reasoning) back prior to the response. Here’s an excerpt of a “thought process”.</p><pre>Okay, so I need to figure out how to estimate time without a clock. Hmm, this sounds tricky because usually, we just check the time on our phones or clocks, but if that&#39;s not available,<br>how else can I tell what time it is? Let me think through different methods.<br><br>First off, I know that the sun rises and sets at roughly the same times each day, so maybe I can use the position of the sun to estimate how much time has passed. For example, if it&#39;s<br>noon when the sun is directly overhead, that would be 12:00 PM. But I&#39;m not sure about the exact times for sunrise and sunset because they vary depending on the season and location.<br><br>Another thought is using shadows from objects like trees or buildings. If I can estimate the length of a shadow, maybe I can figure out how much the sun has moved since a known time.<br>Wait, but without knowing when that known time was, it might not help me directly unless I use some other reference points.<br><br>What about the positions of stars? Some stars are visible during certain times of the day. For example, Sirius is usually visible in the evening after sunset. If I can recognize a star<br>and know when it rises or sets, I could estimate how much time has passed since then. But this might require knowing when each star appears, which could be complicated.</pre><h3>Integrating the two</h3><p>Easy! Write an MCP server that gives Claude the ability to reach out to DeepSeek. I’m running a local version of the model (14B parameters). My idea is just to extract out the &lt;think&gt; bit, and return that to Claude. The goal is that provides the chain of thought from DeepSeek helps Sonnet perform better (e.g. just use DeepSeek’s thinking as a better prompting strategy for Claude).</p><p>When you write a tool for Claude, you need to give it a description of what the tool does. I went with this:</p><pre>Rodin is a cognition partner for both you and the user. For each user prompt, use Rodin to get additional context.<br>Carefully review this content and use the additional insight to give better results to the user.Is it any smarter?</pre><p>So now I can ask Claude to consult Rodin first, and it’ll pass that context back to Claude. Which might make it smarter?</p><h3>Is it smarter? Maybe? Sometimes? Probably not.</h3><p>Determining whether one model is smarter than another is hard. Ideally I’d like to find a class of problems that Claude can’t solve, but Claude + DeepSeek can. Unfortunately, it’s no longer as easy as asking for how many r’s in strawberry? 😟</p><p>If I was doing this “properly”, I’d try and find an <a href="https://medium.com/data-science-at-microsoft/evaluating-llm-systems-metrics-challenges-and-best-practices-664ac25be7e5">evaluation data set</a> (eval). I’d have a series of puzzles and some ground truth and an oracle that allows me to tell whether the answer is right or wrong. This sounds simple, but in reality it’s quite hard (you might have seen the controversy over the latest “Frontier Model” results).</p><p>So, let’s just try and find some logic puzzles that Claude can’t do.</p><blockquote><em>Susan and Lisa decided to play tennis against each other. They bet $1 on each game they played. Susan won three bets and Lisa won $5. How many games did they play?</em></blockquote><p>The correct answer is 11 (Lisa must have lost three bets, so she must have won three more games). o1-mini and <a href="http://chat.deepseek.com/">chat.deepseek.com</a> answer this correctly.</p><p>With Claude alone, I consistently (well, I tried 5 times) get the following response (note that I’m using the Desktop client, I’ve no idea how model variables like temperature are configured by default and this is totally unscientific).</p><blockquote><em>Let me solve this step by step: Susan won 3 bets = $3 Lisa won $5 Each game = $1 bet Lisa’s wins = $5 ÷ $1 = 5 games Total games = Susan’s wins + Lisa’s wins = 3 + 5 = 8 games They played 8 games total.</em></blockquote><p>If I use the deepseek-14b model, I <em>sometimes</em> get the correct answer. It’s interesting looking at the thought process and sometimes seeing:</p><blockquote><em>Now, it’s also given that Lisa ended up winning $5 overall. Hmm, that seems interesting because Lisa lost three games but still managed to make a profit of $5. So, how is that possible?</em></blockquote><p>And claude+deepseek-14b? Well, it’s sometimes right as well. Maybe more than just deepseek-14b alone? I did get examples like this, where deepseek-14b gets it wrong, but then claude picked it up and somehow got it right.</p><figure><img alt="Sometimes two wrongs make a right." src="https://cdn-images-1.medium.com/max/894/1*9_SHFjjE4oug-oL_m3BFQw.png" /></figure><h3>A harder puzzle</h3><p>Let’s try one more problem. You have five boxes in a row numbered 1 to 5, in which a cat is hiding. Every night, he jumps to an adjacent box, and every morning, you have one chance to open a box to find him. How do you win this game of hide and seek?</p><p>This is a surprisingly complicated puzzle — here’s a great description of the <a href="https://mindyourdecisions.com/blog/2017/07/09/can-you-solve-the-hiding-cat-puzzle-tech-interview-question/">solution</a>.</p><p>o1 enjoyed thinking about this for 2 minutes and 37 seconds. It’s idea of an optimal strategy was “the back and forth” sweep where you pick box 1,2,3,4,5,4,3,2,1 and you are guaranteed to pick the cat by 9 days. It confidently exclaimed “No matter how clever you are, you <em>cannot</em> do better than 9 days in the worst case. There are specific “dodging” paths the cat can take to avoid you if you try to compress the strategy into fewer than 9 days”. o1-mini suggests the same strategy with less snark.</p><p>deepseek started responding almost immediately and (after writing a 1400-word essay) correctly gave the solution of six days with the pattern (2,3,4,2,3,4). deepseek-14b alone isn’t smart enough to solve the problem.</p><p>And unfortunately, neither is claude or claude+deepseek-14b.</p><h3>Conclusion?</h3><p>Claude and a thinking partner sounded like a fun idea to try (and it was). If that thinking partner is a relatively small model (14b) then it’s certainly not obviously better (but sometimes was when the random number generators aligned)</p><p>The main outcome of this was to keep exploring. I still love Claude for the vibes, DeepSeek R1 is super impressive. It’s fast and it’s smart!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccd01a0f345b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/deepseek-r1-sonnet-ccd01a0f345b">DeepSeek R1 + Sonnet</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding and Managing Friction in Software Development]]></title>
            <link>https://medium.com/ingeniouslysimple/understanding-and-managing-friction-in-software-development-6aa3b62fd844?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aa3b62fd844</guid>
            <category><![CDATA[friction]]></category>
            <category><![CDATA[process-improvement]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Wed, 11 Dec 2024 09:38:18 GMT</pubDate>
            <atom:updated>2024-12-11T09:38:18.176Z</atom:updated>
            <content:encoded><![CDATA[<p>Effective software delivery depends on strategically managing friction — the forces that impede progress between ideation and implementation. Technology leaders who master this capability systematically eliminate unnecessary barriers while preserving essential protective controls, achieving both speed and quality in their organizations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tMSAD1Y_H9u3tgQN" /><figcaption>ChatGPT’s abstract interpretation of this blog psot.</figcaption></figure><p>Carl von Clausewitz’s observations about war remain remarkably relevant when applied to software development:</p><blockquote>Everything is very simple in War, but the simplest thing is difficult. These difficulties accumulate and produce a friction which no man can imagine exactly who has not seen War, Suppose now a traveller, who towards evening expects to accomplish the two stages at the end of his day’s journey, four or five leagues, with post-horses, on the high road — it is nothing. He arrives now at the last station but one, finds no horses, or very bad ones; then a hilly country, bad roads; it is a dark night, and he is glad when, after a great deal of trouble, he reaches the next station, and finds there some miserable accommodation. So in War, through the influence of an infinity of petty circumstances, which cannot properly be described on paper, things disappoint us, and we fall short of the mark.</blockquote><p>Replace “War” with “software development,” and his description captures the essence of our modern technical challenges. Just as a traveller faces unexpected obstacles that transform a simple journey into an arduous trek, software teams encounter countless small impediments that compound to create significant barriers to progress.</p><h3>The Three Pillars of Software Development Friction</h3><h3>Infrastructure Friction</h3><p>Infrastructure friction represents the most tangible form of resistance in software development. It manifests in systems, code, and tools — particularly as technical debt, that accidental complexity that makes tasks take longer than they should. It’s the gnarly bit that uses reflection and stubbornly refuses to refactor, or that manual implementation of a pattern that’s now available in the standard library but implemented subtly differently.</p><p>However, infrastructure friction extends beyond code-level challenges. It encompasses the entire path from committed code to customer value, including flaky tests, slow pipelines, poor version control, and environment inconsistencies.</p><h3>Execution Friction</h3><p>Execution friction emerges from the workflows and ceremonies that guide software development.</p><p>Understanding gaps manifest as unclear requirements, leading to misaligned expectations and rework. These gaps often arise from insufficient communication between business stakeholders and development teams, or from requirements that lack necessary context and clarity.</p><p>Ceremonial overhead occurs when processes become ends in themselves rather than means to an end. This happens when the ceremonies of software development such as stand-ups, sprint planning, retrospectives become the deliverable rather than a tool for building and involving people in problem-solving.</p><h3>Human Friction</h3><p>Human friction encompasses the collaborative and cognitive challenges in software development. It manifests when organizational structures and interactions hinder the effective flow of information and collaboration. This friction often stems from siloed teams that create barriers to knowledge sharing and lead to duplicated efforts. It is exacerbated by misaligned priorities between groups working on the same product, causing inefficiencies and conflict. Additionally, the use of ticket systems can transform rich, dynamic human interactions into transactional (and delayed) exchanges, further stifling collaboration and mutual understanding.</p><h3>Measuring Development Friction</h3><p>Effective friction management requires systematic measurement. The Accelerate metrics provide a foundation for measuring delivery performance with the four key metrics. The SPACE framework included developer productivity measures around satisfaction, performance, activity, communication and flow.</p><p>You almost certainly need additional measures in your company to find and identify friction (maybe it’s the pull request review process? maybe it’s waiting for approval? maybe it’s the need to set up a meeting?).</p><h3>The Protective Role of Friction</h3><p>Not all friction in software development is detrimental. Like brakes on a car, some forms of friction serve protective functions in the development process. Creating the right friction points can help enhance quality, security, and alignment.</p><p>For example, RFC (Request for Comments) and ADR (Architecture Decision Record) processes add deliberate friction to significant changes. While they may slow initial development, they can improve design through early feedback, document decisions for future reference and prevent costly architectural mistakes.</p><p>Similarly, when operating software production access controls exemplify beneficial friction in operations. The friction helps reduce risks of accidental changes, creates clear audit trails and ensures compliance with security requirements.</p><h3>Managing Friction Effectively</h3><p>The key to successful software development lies not in eliminating all friction, but in maintaining a balanced approach that:</p><ol><li>Identifies and eliminates unnecessary friction that adds no value</li><li>Preserves and optimizes necessary friction that protects quality and security</li><li>Continuously evaluates friction points to ensure they remain relevant and beneficial</li></ol><p>By understanding the different types of friction and their roles in software development, teams can work more effectively to manage them. The goal is not to eliminate all resistance but to ensure that every point of friction serves a purpose in delivering better software.</p><p>Success in software development, like success in Clausewitz’s warfare, depends not on avoiding friction entirely but on understanding its nature and working effectively within its constraints. The teams that thrive are those that can distinguish between protective and destructive friction, maintaining the former while systematically eliminating the latter.</p><p>(x-posted from <a href="https://www.linkedin.com/pulse/understanding-managing-friction-software-development-jeff-foster-u4eee/?trackingId=Cj6Ws9jdQ%2ByoWTtlEN2MdQ%3D%3D">https://www.linkedin.com/pulse/understanding-managing-friction-software-development-jeff-foster-u4eee/?trackingId=Cj6Ws9jdQ%2ByoWTtlEN2MdQ%3D%3D</a>)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aa3b62fd844" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/understanding-and-managing-friction-in-software-development-6aa3b62fd844">Understanding and Managing Friction in Software Development</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How do we make architecture decisions?]]></title>
            <link>https://medium.com/ingeniouslysimple/how-do-we-make-architecture-decisions-88846b419770?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/88846b419770</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[culture]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Thu, 21 Oct 2021 14:48:34 GMT</pubDate>
            <atom:updated>2021-10-21T14:48:34.211Z</atom:updated>
            <content:encoded><![CDATA[<p>How do we make good decisions about our software design?</p><p>The “easiest” solution to this is for someone with a gigantic brain to make all the decisions. The “Megamind” approach certainly has some advantages — an individual can make decisions quickly, and there’s a single person with responsibility (or “single wringable neck” or “throat to choke” or some other term that’s not very Redgate-y). The drawbacks dwarf these advantages. Putting the onus on a single person is risky — people aren’t perfect (sorry!).</p><p>Let’s try and improve that a bit. We could appoint a change advisory board (CAB). A CAB is a small group with relevant domain experience that review decisions and suggest improvements. A CAB sounds like an improvement; <a href="https://hbr.org/2016/11/why-diverse-teams-are-smarter">diverse groups make better decisions</a>, and relevant domain expertise is bound to make things better. Unfortunately, research shows having a change approval process is not a good idea.</p><blockquote>“… found that external approvals were negatively correlated with lead time, deployment frequency, and restore time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.” (Accelerate: Nicole Forsgren)</blockquote><p>Our core belief at Redgate is that “<strong>We believe that the best way to make software is by engaging teams empowered with clear purpose, freedom to act and a drive to learn.”. </strong>Submitting designs for approval goes against the grain of freedom to act. However, we also recognize that sometimes an individual team may not have the whole context, and local decisions might be local optimums.</p><p>How might we fix this? Well, the answer has been given to us by the captain of the Santa Fe submarine, David Marquet. After being placed in an unfamiliar environment, he developed a style of leadership that emphasized local decision making without giving up control. Intent-based leadership empowers teams to make decisions but uses an “intent-check” to allow others to give insight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*btUF4JO2gkYQCGVb" /><figcaption>Great book — highly recommended!</figcaption></figure><p>We’ve embodied this approach in how we make architecture decisions. Before starting a new bit of work, we ask engineers to write a record of their decision. The act of writing things down improves design (after all, if you can’t write it down, how on earth are you going to implement it), but it also provides an intent check for those with a different perspective.</p><p>Our approach borrows from two established processes to build something for Redgate.</p><ul><li><a href="https://www.ietf.org/standards/rfcs/">RFC </a>(Request for Comments) documents originated in the very early days of the internet in 1969. It’s a formal process that works in the open to give everyone input. The RFC process is still used today to define key parts of the internet. For example, you can see the <a href="https://datatracker.ietf.org/doc/rfc8200/">complete record of discussion on IPv6 </a>(introduced since there are now billions of devices connected together).</li><li><a href="https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions">ADRs </a>(Architecture Decision Records) are a lightweight record of engineering decisions on the other end of the spectrum. The ADR log is kept alongside the source code and gives engineers an excellent overview of the historical context behind decisions.</li></ul><p>We’ve combined the two — and come up with something we call “<em>Architecture Intent Checks</em>” (AIC). AICs act as a non-blocking check before starting work on architectural changes. We ask teams to seek feedback from relevant stakeholders, and if new information comes to light that would change the decision, we’ll change the decision! We assume that most of our engineering decisions are good because we’re constantly making an effort to make sure engineering teams have the broader context of the business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88846b419770" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/how-do-we-make-architecture-decisions-88846b419770">How do we make architecture decisions?</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Models]]></title>
            <link>https://medium.com/ingeniouslysimple/language-models-15e45dce0805?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/15e45dce0805</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Wed, 13 Oct 2021 15:22:43 GMT</pubDate>
            <atom:updated>2021-10-14T08:10:26.065Z</atom:updated>
            <content:encoded><![CDATA[<p>A language model is a probability distribution over a sequence of words. As an example, a language model for English should be able to predict the next item in a sequence, or ideally generalize to responding to a question with a well-formed response. As engineers, we use language models all the time! When you use tab-completion (or Intellisense), you’re using a language model.</p><h3>Eliza</h3><p>Eliza is one of the first language models. Eliza originated in 1964 at MIT and it was really designed to demonstrate the superficial nature of conversation with a non-directional psychotherapist. Depending on the human (!), it’s one of the first chat bots capable of attempting the Turing test, but it’s important to note it was never really designed to do that, it was really just taking the piss out of psychotherapists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*Y6R_8rqUdb_CViIBtx2U-w.png" /><figcaption>A conversation with Eliza</figcaption></figure><p>So, how does Eliza work? Well, there’s really just one trick. Eliza uses pattern matching to recognize common phrases, and replies with a bit of substitution trickery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*T1Hk1e76kkYFdYDC7xLH9A.png" /><figcaption>A peek under the curtain of Eliza</figcaption></figure><p>It’s amazing how well this approach works! However, this does have the drawback that to cover every scenario, Eliza requires a human to rigorously define the entire syntax and semantics of the language. That’s quite hard, the English language is wonderfully ambiguous!</p><ul><li>“I shot an elephant in my pyjamas” (Groucho Marx)</li><li>“Call me an ambulance!”</li><li>“I’ve never tasted dinner like that before”</li></ul><p>Eliza is a neat trick, but it’s not a super generalizable model!</p><h3>N-gram models</h3><p>N-gram models are a probabilistic language model for predicting the next item. N-gram models use N-grams to pick the next item based on their frequency distribution from some sample text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*8Zmf_1lQU_M1tQmeEMm42A.png" /><figcaption>4-grams for “once upon a …”</figcaption></figure><p>The above are the 4-grams for “once upon a &lt;blank&gt;”. The percentages indicate what proportion of four word sequences that occur in the test corpus are the given words. We can see from this that if you have a sentence that begins “once upon a” then the probability of the next word being “time” is very high (in fact, it’s 700x more likely to be “time” than “long”).</p><p>Building an N-gram model is dead simple and it’s a really fun program to write. Here’s <a href="https://gist.github.com/fffej/8ef06291937fb5c676fecb928e564927">my attempt</a>.</p><p>All you do is find all pairs (in this case, as I’m making a 2-gram model) and index them. To generate the words, pick a random word and then keep rolling the dice and picking according to the frequency distribution.</p><p>N-grams are surprisingly powerful! Below is some generated text from Romeo and Juliet. It’s not going to win any awards, but if you squint a bit, it’s fairly realistic gibberish.</p><blockquote><em>“the all-cheering sun for all forth: well, think it is it well. balthasar. news be taken. — stay awhile. — stand up. romeo. he of this contract tonight; for earth to a month, a princox; go: be fourteen. how should be talked on, transcribe and paper, and sails upon that very night by thy wild acts denote the other terms of all the singer. i married then starts up, i’faith. will you run mad. o, he’s a madness most you pardon me to move is yond yew tree lay fourteen of wax. lady capulet. o lamentable chance? the constable’s own word: and how my”</em></blockquote><p>The problem with N-gram models is that it has no memory and no context, it’s just relentlessly following the model. Google have built a HUGE n-gram model (see <a href="https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html">https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-to-you.html</a>) and it’s very useful for spelling correction and simpler tasks like named entity recognition.</p><h3>Neural Networks</h3><p>A neural network takes 1 or more inputs and produces some output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*sbxtQNfTvDRLo-l_RxN42Q.png" /><figcaption>Obligatory diagram of Neural Networks</figcaption></figure><p>For example, it might be a model to calculate the price of a house. Inputs might be the number of bedrooms and sq/feet of the house, and the output is a prediction of the price.</p><p>Neural networks have been around forever, but until relatively recently they’ve been a bit dormant. What’s changed things around is <strong>vectorization, </strong>the ability to get GPUs/custom silicon to execute and train neural networks with way more parameters before. The Nvidia A100 delivers 312 TERA FLOPS. 312000000000000. That is a 3 with 14 zeros on the end. And that’s quite a lot of zeros. Couple that with advances in software (differentiable programming) and you’ve entered the world of deep-learning.</p><p>Neural networks are optimization over functions — throw enough hardware/layers at it, and they’ll produce a way to minimize the error on a given training set. Recurrent neural networks take this further, they represent optimization over sequences. I’ll never be able to explain RNNs as eloquently as Karpathy, so please take a few minutes and read <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">http://karpathy.github.io/2015/05/21/rnn-effectiveness/</a>!</p><p>The current state of the art model is known as GPT-3. The training data consists of unlabelled data, and the goal of the model is to successfully fill in the blanks. This model consists of 175 Billion parameters (!). <a href="https://paperswithcode.com/method/gpt-3">https://paperswithcode.com/method/gpt-3</a> is a great resource to learn more.</p><p>This model is tremendously powerful! The examples below illustrate the model being used to complete source code. Some of it is jaw-droppingly amazing, some of it is a bit more meh!</p><p>Search out more Github Co-pilot to learn more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4IB2NpCL0XwYFvx3NWk5GA.png" /><figcaption>Examples: Given the first 2 lines, generate the rest!</figcaption></figure><h3>Conclusion</h3><p>Deep Learning is promising a revolution (with language models enabling more conversational magic), but it’s important to note that we’ve been here before (See the <a href="https://en.wikipedia.org/wiki/AI_winter">AI Winter</a>).</p><p>I think this time is different though! Almost everything is digital nowadays, giving a huge amount of training data for almost any scenario. Computing performance is still doubling every few years (at least with the number of cores), and there’s rapid innovation around neural networks once again.</p><p>Exciting times!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15e45dce0805" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/language-models-15e45dce0805">Language Models</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Psychological Safety]]></title>
            <link>https://medium.com/ingeniouslysimple/psychological-safety-16c6977dbaa4?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/16c6977dbaa4</guid>
            <category><![CDATA[psychological-safety]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[culture]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Tue, 29 Jun 2021 10:19:06 GMT</pubDate>
            <atom:updated>2021-06-29T11:55:39.253Z</atom:updated>
            <content:encoded><![CDATA[<p>Psychological safety is a wonderfully misunderstood term. It evokes images of hippies singing kum ba yah. It’s not that! Let’s rope in <a href="https://amycedmondson.com/">Amy Edmonson</a> (who popularized the term) to let loose on what psychological safety <strong>isn’t</strong>:</p><blockquote><em>It does not mean that people always agree with one another for the sake of being nice. It does not mean that people offer unequivocal praise of unconditional support for everything you have to say. In fact, you could say it’s the opposite. Psychological safety is about candour, about making it possible for productive disagreement and free exchange of ideas. It goes without saying that these are vital to learning and innovation. … Psychological safety enables people on different sides of a conflict to speak candidly about what’s bothering them. (The Fearless Organization)</em></blockquote><p>So, what <strong>is </strong>psychological safety? It’s the belief that you won’t be punished when you make mistakes. Studies show that psychological safety allows for risk-taking, speaking your mind, creativity and sticking your neck out without fear of having it cut off — just the types of behaviour that lead to great software being created.</p><p>Why is psychological safety important? Well, if you want the best decisions, you need everyone to contribute without fear of reprisal, loss of prestige or embarrassment. Moreover, in safety-critical industries (such as aviation or medicine), a lack of psychological safety can directly contribute to loss of life (<a href="https://en.wikipedia.org/wiki/Hofling_hospital_experiment">Hofling hospital experiment</a> or <a href="https://www.economics-sociology.eu/files/24_20_455_Enomoto_Geisler.pdf">Gladwell Hypothesis</a>). Thankfully, we don’t work in a safety-critical industry! But psychological safety is still important. For example, a <a href="https://rework.withgoogle.com/print/guides/5721312655835136/">study</a> showed that psychological safety was the #1 factor in high-performing teams at Google. The work in the <a href="https://www.amazon.co.uk/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339">Accelerate </a>book further supported this evidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-V8WiUm9hDg-0q6q1IzUDg.png" /><figcaption>Top 5 aspects for highly performing teams</figcaption></figure><p>We ran a survey (yes, another one!) on psychological safety created by <a href="https://amycedmondson.com/">Prof. Amy Edmonson</a>. It’s the standard survey on psychological safety, and this allows us to compare ourselves against industry benchmarks. We had a really strong response rate (89 people across teams in CORE / Product).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MBCuTHA1c8Ai2fzxeYUQw.png" /><figcaption>Survey results</figcaption></figure><p>So what do the survey results tell us about the environments in Product/CORE?</p><p>The graph above shows our survey results compared to industry benchmarks. In almost every single area we significantly outperform the industry average. The graph is shown relative to the industry benchmark. Blue indicates we’re outperforming the industry standard, and orange indicates we’re falling behind. As you can see from the graph, it’s mostly full of win! Our teams are comfortable taking risks. There’s no blame culture. When problems occur, we see them as opportunities to improve. When individuals need information they’re comfortable reaching out to customers and other teams. Teams are truly cross-functional and are comfortable sharing their expertise and learning about other domains too.</p><p>There’s one single area where we do consistently (but only slightly) fall below our peers in the industry. When participants were asked “This team frequently obtains new information that leads up to make important changes in our plans or work processes” we scored just below average. We’ll dig into this area and find out more !</p><p>Why’s this important now? Well, as we sort out our flexible-hybrid environments we’ve got to maintain these high levels of psychological safety. That’s going to be a challenge! As we come back into the office, we’ve got different dynamics at play and we’ll have to continue to make sure that everyone can get their voice heard and be comfortable contributing. Thanks to this survey, we know where we are and we know the areas to focus on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16c6977dbaa4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/psychological-safety-16c6977dbaa4">Psychological Safety</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A little Rust]]></title>
            <link>https://medium.com/ingeniouslysimple/a-little-rust-f888bcc0a7b8?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/f888bcc0a7b8</guid>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[rust-programming-language]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Fri, 23 Apr 2021 13:30:35 GMT</pubDate>
            <atom:updated>2021-04-23T13:30:35.463Z</atom:updated>
            <content:encoded><![CDATA[<p>Every so often, I like to dabble in a programming language to get exposure to a new way of thinking.</p><ul><li>C++ introduced me to design-patterns (the classic <a href="https://en.wikipedia.org/wiki/Design_Patterns">GoF </a>book) and the perils of memory-management (More Effective C++, Effective C++, Template Meta-Programming).</li><li>Lisp taught me the value of small pieces, little languages and (most importantly) macros.</li><li>Haskell showed me the virtue (and pitfalls) of laziness and showed me that higher-level abstractions can be super powerful!</li></ul><p>It’s been ages since I picked up something new, so I’ve started taking a look at Rust. Why Rust?</p><p>Well, firstly it’s been gaining mindshare. Rust is increasingly commonly mentioned on the blogs I read, and it’s <a href="https://insights.stackoverflow.com/survey/2020#most-loved-dreaded-and-wanted">Stackoverflow’s most loved language</a> for the last 5 or so years.</p><p>What makes Rust different? Like C++, it doesn’t have garbage collection, but unlike C++ it has a rich ownership model guaranteeing memory and thread safety. Memory safety makes Rust super attractive as a systems-programming language.</p><p>As a learning exercise, I ported over a <a href="http://www.fatvat.co.uk/2010/06/orbit-simulator-in-haskell.html">previous project</a> from Haskell to Rust. You can see the sort of output it makes below, and check the code out at <a href="https://github.com/fffej/learning-rust/tree/main/orbit">https://github.com/fffej/learning-rust/tree/main/orbit</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nWSSIiE20OJdhA9c_Oe8vg.gif" /><figcaption>Some random planets orbiting the “sun”</figcaption></figure><p>So, what’d I learn from this experience?</p><p>Rust has clearly had a lot more thought about <strong>developer experience</strong> than other languages (Haskell and C++ spring to mind). For example, if you pass a reference instead of a value, you get a handy message (compare this to <a href="https://codegolf.stackexchange.com/questions/1956/generate-the-longest-error-message-in-c">C++ and getting a template parameter wrong</a>, or <a href="http://neilmitchell.blogspot.com/2014/10/why-traversablefoldable-should-not-be.html">Haskell’s foldable/traversable saga</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/1*rqc89btc4E3Nh1ik0W_aeA.png" /><figcaption>Ah-ha, so that’s what I did wrong!</figcaption></figure><p>In a similar vein, the <a href="https://doc.rust-lang.org/">documentation </a>and <a href="https://crates.io/">surrounding ecosystem</a> is super friendly to use (at least for the limited third-party packages I used).</p><p>Rust has an auto-format (<a href="https://github.com/rust-lang/rustfmt">rustfmt</a>) which proclaims the one-true way to lay code out. It’s a small thing, but it’s just another thing that improves the DX.</p><p>Within the language, the typing features play towards my biases. After writing lots of code (and reading more), I prefer code that I can reason about locally. Rust’s semantics support this well since you have to explicitly specific mutability. The strong-typing means less unknowns when looking at a local bit of code.</p><p>What didn’t I like (with the caveat that means “I have no clue what I’m doing but I didn’t like this in the few days I spent writing code so don’t take it too seriously)?</p><p>“clippy” sounds like it should be a great idea, but it gave me false positives out of the box. I should have probably invested some time in setting some sensible defaults.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z0HOASBW5fyzil3rLfZwPQ.png" /></figure><p>I didn’t find a nice way of composing functions (<a href="https://github.com/rust-lang/rfcs/issues/2049">pipelines</a>), but it seems that this can be done fairly simpler with macros. More for me to learn here!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f888bcc0a7b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/a-little-rust-f888bcc0a7b8">A little Rust</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An ode to Paradigms of AI Programming]]></title>
            <link>https://medium.com/ingeniouslysimple/an-ode-to-paradigms-of-ai-programming-3eb42998367d?source=rss-ec6ca6c48add------2</link>
            <guid isPermaLink="false">https://medium.com/p/3eb42998367d</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[book-review]]></category>
            <dc:creator><![CDATA[Jeff Foster]]></dc:creator>
            <pubDate>Tue, 16 Feb 2021 12:36:41 GMT</pubDate>
            <atom:updated>2021-02-16T12:36:41.981Z</atom:updated>
            <content:encoded><![CDATA[<p>Ignore the title. Even if your eyes are attracted to the word “Lisp”, ignore that too. Paradigms of AI Programming (PAIP) is just the best introduction to data-driven programming that you’ll ever find.</p><p>By the time you reach Chapter 2, you’ll see the difference between a straight-forward solution (not extensible) to a problem and a truly data-driven solution.</p><blockquote>The two versions of the preceding program represent two alternate approaches that come up time and time again in developing programs: (1) Use the most straight-forward mapping of the problem description directly into Lisp code. (2) Use the most natural notation available to solve the problem, and then worry about writing an interpreter for that notation.<br> …<br> The reader may notice that in most cases, we choose the second.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ghq5is38CsyiFFEoR_bqTg.jpeg" /><figcaption>Paradigms of AI Programming (cat doesn’t come by default)</figcaption></figure><p>As you continue to work through the book, you’ll get this approach reinforced and develop your sense of aesthetics.</p><blockquote>There are six maxims that every programmer should follow:</blockquote><blockquote>· Be specific.</blockquote><blockquote>· Use abstractions.</blockquote><blockquote>· Be concise.</blockquote><blockquote>· Use the provided tools.</blockquote><blockquote>· Don’t be obscure.</blockquote><blockquote>· Be consistent.</blockquote><p>Once you’ve got an overview of Lisp, you’ll get the chance to implement classic AI problems (such as the <a href="https://en.wikipedia.org/wiki/General_Problem_Solver">General Problem Solver</a> or <a href="https://en.wikipedia.org/wiki/ELIZA">Eliza</a>). Moreover, you explore how to turn concrete implementations into abstract solutions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b268ada6c5f4319e1bcbe8a07343c26/href">https://medium.com/media/1b268ada6c5f4319e1bcbe8a07343c26/href</a></iframe><p>I love the code above because of its simplicity. Plug the right argument in, and you have got depth-first, breadth-first, beam search or even an A* search from that one concisely written lump of code.</p><p>The book doesn’t just present beautiful abstract code that performs a job, and there’s a whole chapter on efficiency.</p><blockquote>However, this does not mean that writing an efficient program is fundamentally different from writing a working program. Ideally, developing an efficient program should be a three-step process. First, develop a working program, using proper abstractions so that the program will be easy to change if necessary. Second, instrument the program to determine where it is spending most of the time. Third, replace the slow parts with faster versions, while maintaining the program’s correctness.</blockquote><p>To improve performance doesn’t mean “uglifying the code”. It doesn’t mean using bit-twiddling tweaks or mutating state everywhere. PAIP presents four language-independent techniques, Caching, Compiling, Delaying and Indexing.</p><p>Caching gives a lovely demonstration of the power of Lisp by introducing memoization. If you’ve not come across it before, memoization is a technique to make functions remember calls so that subsequent calls to a function with the same argument return a cached value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ae2c1c8d120494ca1e12a72f80ca380/href">https://medium.com/media/0ae2c1c8d120494ca1e12a72f80ca380/href</a></iframe><p>The classic implementation of the Fibonacci function is dog slow (a very technical term!) but it’s also very clean. Introduce memoization appropriately, and you have a version with a much-improved runtime, without sacrificing the clarity of the code.</p><p>The section on Compiling builds upon the “interpreter for the notation” style introduced early on in PAIP. To write a compiler, we take the Lisp code that defines the problem and use macros to rewrite them in a new, more efficient language.</p><p>Delaying computation is a common theme now (Lazy&lt;T&gt; in C# is a great example), but when I read PAIP, this introduced me to a new way of thinking (infinite lists? Thunks?).</p><p>There are many different models of programming (such as functional or object-oriented). I’m a big believer that the more paradigms you experience, the more tools you have at your disposal for finding the right solution.</p><p>PAIP introduced me to Prolog (a logic-based programming language). Prolog has three big ideas, a database of assertions, unification and back-tracking. I’m not going to attempt to explain these; I’d do a much lower quality job of it than the book would. I’m just trying to leave a few nuggets to convince you to read the book and work through the exercises 😉 .</p><p>Norvig also introduces object-oriented programming in Lisp (<a href="http://norvig.com/design-patterns/design-patterns.pdf">Design Patterns in Dynamic Programming</a> is another exciting read in this context.). What’s eye-opening here is that object-oriented systems are just another language written in Lisp — it’s not the be-all and end-all of the language as it is in C# / Java.</p><p>I could carry on, but I’ve waffled for a fair few words. So to round things off, PAIP is an under-appreciated classic! It’s one of the few books worth devoting a few months working through the exercises; you’ll come out of the other side a better programmer!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3eb42998367d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ingeniouslysimple/an-ode-to-paradigms-of-ai-programming-3eb42998367d">An ode to Paradigms of AI Programming</a> was originally published in <a href="https://medium.com/ingeniouslysimple">Ingeniously Simple</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>