<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Morten Forfang on Medium]]></title>
        <description><![CDATA[Stories by Morten Forfang on Medium]]></description>
        <link>https://medium.com/@mfo_47746?source=rss-a9883a9f4a82------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*s5dJHmcEZo3p3Rt3</url>
            <title>Stories by Morten Forfang on Medium</title>
            <link>https://medium.com/@mfo_47746?source=rss-a9883a9f4a82------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:47:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mfo_47746/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quantum computing in machine learning (part 3)]]></title>
            <link>https://medium.com/compendium/quantum-computing-in-machine-learning-part-3-abe8b0078d2c?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/abe8b0078d2c</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[qiskit]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[quantum-computing]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 09:35:10 GMT</pubDate>
            <atom:updated>2025-04-29T09:43:15.380Z</atom:updated>
            <content:encoded><![CDATA[<h3><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-1-21eb44dfa142"><strong>Read Quantum computing in machine learning part 1 here </strong></a><strong>— and </strong><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-2-e3cc55294ce6"><strong>part 2 here.</strong></a></h3><p>In the setup in the paper the authors use a microwave pulse generator to manipulate the qubit state. And the state we might visualize as being a point on a sphere, as I mentioned. The sphere is called a Bloch sphere. The Schrodinger equation then computes results over time. I mentioned that you can do two things. You can measure. And you can rotate the sphere. That is to say, if you choose a particular qubit state you can use the pulse generator to make it so that the Schrodinger rotates the point around a chosen axis.</p><p>Let’s look into how we could write code for this. Code that you could send to a vendor that has a physical lab as we described and who could run the code, rotate the Bloch sphere, that is do the multiplication approximation by a quantum computer. So that you could plug that into your algorithm for updating the model weights of your large language model.</p><p>Previously we mentioned that a qubit state is defined as a combination of two parameters governing state 0 and state 1. They are called alpha and beta by convention. The sphere we work on is just a visualization. The state, described with alpha and beta, corresponds to a point on the sphere. Let’s say we want to put the point 45 degrees “north” — up the z axis — and at Greenwich longitude — at x = 1 and y = 0. With some math, that would correspond to some specific alpha and beta values. That would be the qubit state for that point on the sphere. Using a so-called RY gate, that point will by Schrodinger equation start rotating around the Y axis by itself. This is the code (<a href="https://github.com/computas/2503qubit-simple-bloch-sphere/blob/main/single-sphere.py">github</a>):</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br>from mpl_toolkits.mplot3d import Axes3D<br><br><br>omega = 2 * np.pi                     <br>t_vals = np.linspace(0, 1, 200)       <br><br><br>alpha0 = np.sqrt((2 + np.sqrt(2)) / 4)<br>beta0 = np.sqrt((2 - np.sqrt(2)) / 4)<br><br><br>alpha_t = alpha0 * np.exp(-1j * omega * t_vals / 2)<br>beta_t = beta0 * np.exp(1j * omega * t_vals / 2)<br><br><br>def bloch_coords(alpha, beta):<br>    x = 2 * np.real(np.conj(alpha) * beta)<br>    y = 2 * np.imag(np.conj(alpha) * beta)<br>    z = np.abs(alpha)**2 - np.abs(beta)**2<br>    return x, y, z<br><br><br>x_vals, y_vals, z_vals = [], [], []<br>for a, b in zip(alpha_t, beta_t):<br>    x, y, z = bloch_coords(a, b)<br>    x_vals.append(x)<br>    y_vals.append(y)<br>    z_vals.append(z)<br><br><br>u, v = np.mgrid[0:2*np.pi:60j, 0:np.pi:30j]<br>xs = np.cos(u) * np.sin(v)<br>ys = np.sin(u) * np.sin(v)<br>zs = np.cos(v)<br><br><br>fig = plt.figure(figsize=(8, 8))<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br><br><br>ax.plot_surface(xs, ys, zs, color=&#39;lightblue&#39;, alpha=0.1, linewidth=0)<br><br><br>ax.plot(x_vals, y_vals, z_vals, color=&#39;purple&#39;, label=&#39;Qubit state trajectory&#39;)<br>ax.scatter(x_vals[0], y_vals[0], z_vals[0], color=&#39;green&#39;, s=50, label=&#39;Start (t=0)&#39;)<br>ax.scatter(x_vals[-1], y_vals[-1], z_vals[-1], color=&#39;blue&#39;, s=50, label=&#39;End (t=1)&#39;)<br><br><br>ax.set_title(r&quot;$\alpha = \sqrt{\frac{2 + \sqrt{2}}{4}}, \quad \beta = \sqrt{\frac{2 - \sqrt{2}}{4}}$&quot;)<br><br>ax.set_xlabel(&quot;X&quot;)<br>ax.set_ylabel(&quot;Y&quot;)<br>ax.set_zlabel(&quot;Z&quot;)<br>ax.legend()<br>ax.set_box_aspect([1, 1, 1])<br><br>plt.show()</pre><p>which outputs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/1*FrWnu-0cjA-3H9cRkkgNVw.png" /></figure><p>The blue and (occluded) green dot represent the start and the point after a full rotation. The violet line illustrates the trajectory on the qubit state sphere.</p><p>If you wanted to run this on physical hardware, let Schrodinger do the computation for you, you could write the same in qiskit, and submit it to a vendor that accepts qiskit code. IBM could run the code on an Eagle quantum computer for example. The vendor would then use the microwave pulse generator to do the “RY gate”, i.e. configure the qubit state and it’s energy so that Schrodinger would rotate the point around the Y axis. The code looks like this (<a href="https://github.com/computas/2503qubit-simple-bloch-sphere/blob/main/qiskit-single-sphere.py">github</a>):</p><pre>from qiskit import QuantumCircuit, transpile<br>from qiskit_aer import Aer<br>from qiskit.visualization import plot_bloch_multivector<br>import matplotlib.pyplot as plt<br>import numpy as np<br><br><br>qc = QuantumCircuit(1)<br>qc.ry(np.pi / 4, 0)  <br><br><br>backend = Aer.get_backend(&#39;statevector_simulator&#39;)<br>transpiled_qc = transpile(qc, backend)<br>job = backend.run(transpiled_qc)<br>result = job.result()<br>statevector = result.get_statevector()<br><br><br>fig = plot_bloch_multivector(statevector)<br><br><br>plt.title(r&quot;$\alpha = \sqrt{\frac{2 + \sqrt{2}}{4}}, \quad \beta = \sqrt{\frac{2 - \sqrt{2}}{4}}$&quot;)<br><br>plt.show()</pre><p>That about wraps it up for now. You’ve followed me from a Lora substitute for fine-tuning a large language model, to writing a small relevant snippet that you could run on an actual quantum machine today. Thanks for your time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abe8b0078d2c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-3-abe8b0078d2c">Quantum computing in machine learning (part 3)</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quantum computing in machine learning (part 2)]]></title>
            <link>https://medium.com/compendium/quantum-computing-in-machine-learning-part-2-e3cc55294ce6?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3cc55294ce6</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[quantum-computing]]></category>
            <category><![CDATA[qubit]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 09:34:39 GMT</pubDate>
            <atom:updated>2025-04-29T09:54:42.153Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-1-21eb44dfa142">Read Quantum computing in machine learning part 1 here</a>!</h4><h4>Still here?</h4><p>For those few of my readers who are lingering, the question I’d scratch my head over if I was you would be the magical part. The part where I wrote “let nature i.e. physics do the calculation”. That is the long known, but still head-spinning, part in my opinion.</p><p>The part is the quantum computing promise. Here is how I think about it.</p><p>At the core of the magic, we have that nature is really good at solving a particular type of equation. The equation is the engine of reality, the Schrodinger equation. It basically says that the change of things depends on the state of things. It is akin to saying that if a ball is high in the air and about to fall down, the change in height — the velocity — is much less than when the ball is halfway down. Then the ball moves fast. That kind of reasoning. Using that analogy, we can say that both the velocity and height is dependent on time. If I throw the ball up in the air, there is a formula that gives me both height and velocity at any time afterwards. The Schrodinger equation is that kind of formula, only that it works on quantum states and energy instead. For our purposes the magic part is that it is a computation engine. You nudge reality to be in a particular state, you start the stopwatch and you observe after a time lapse. What you observe is the computation of the formula over time. The engine is blindingly fast, has truly negligible footprint and is high fidelity.</p><p>That means that if we can express any type of computational problem on the form that the Schrodinger equation requires, we have a physical quantum-scale computer that can solve it. Classic computers have gotten really good at many problems so it will not solve all or even a majority of computing problems probably, but it might solve some that the classic approach struggles with. Like fine-tuning a large language model capturing all the subtleties in the data, efficiently and effortlessly at the same time.</p><p>We want a quantum version of a classical bit that is minimal, it has two states. And which fits how the Schrodinger equation works. This later equation forces our hand , and the simplest way to satisfy these constraints is to define a qubit state as a combination of two parameters each governing their state, state 0 and state 1. A qubit state can be thought of as a point on a sphere. The north pole represents 1 and the south pole represents 0. Any other place represents a combination of the the two states, and that is the quantum part — the qubit state can be a combination of the two.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*TSfrfUackv2VQw24WG78LQ.png" /><figcaption>A <a href="https://www.nature.com/articles/s41534-024-00840-x">realization of qubits</a> (© Bal M. et al., nature.com 2024)</figcaption></figure><p>As I mentioned initially, we want to do element-wise multiplication using Schrodinger, nature’s own computation engine. With qubits. Physically, we can only do two things.</p><p>We can rotate the point on the sphere. The microwave pulse generator puts everything in place. And Schrodinger does the rotation. That’s what nature does.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*nif8Xh_G5q0MFfybkJNlvw.png" /><figcaption>The suspension and principles of how a <a href="https://www.nature.com/articles/s41467-024-50333-w">microwave pulse generator </a>works (Bao Z. et al., © nature.com 2024)</figcaption></figure><p>And we can read where the point is. When we read, we might get different results each time, even though the state is the same. That is part of how quants work. The state is defined and not uncertain, but quantum states are such that they aren’t locked into one outcome. Such states hold many possibilities at once. It is hard to wrap one’s mind around, but that is the quantum states’s true form. When we measure, we force a choice and get one particular result.</p><p>So to approximate a multiplication, we code the numbers as start angles on the sphere, we wait for a short time (nanoseconds) for Schrodinger to do its work and then we read. We repeat many times, say a thousand times as I mentioned before. After having read a thousand times we have a fairly good idea of the probability of each state. And it transpires that this probability is correlated with the multiplication of the two numbers. Given quite a set of assumptions and constraints. But that is the essence.</p><p>Earlier I argued that quality wise we might preserve more of the nonlinear, subtle parts of the data with a quantum multiplication, than with a classic lora approach where these are lost. To achieve that we need to represent the vectors we are multiplying in different qubits. So the multiplication we just went through with the rotations need to be done over several qubits. The reason why that preserves better the subtle parts of the data is because the qubits effect each other when rotating. The are linked. The effect, called entanglement, takes into account a fuller interaction between the numbers in the vectors we’re multiplying. And that in sum makes the fine tuning make a better, higher resolution decision surface. That decision surface is what makes the large language model choose a more appropriate word over others when producing text.</p><h4><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-3-abe8b0078d2c">Read Quantum computing in machine learning part 3 here!</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3cc55294ce6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-2-e3cc55294ce6">Quantum computing in machine learning (part 2)</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Quantum computing in machine learning (part 1)]]></title>
            <link>https://medium.com/compendium/quantum-computing-in-machine-learning-part-1-21eb44dfa142?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/21eb44dfa142</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[lora]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[quantum-computing]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 09:34:06 GMT</pubDate>
            <atom:updated>2025-04-29T09:36:20.780Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve been following quantum computing with a stifled yawn over years. Yes, I understand it has revolutionary potential, but we are far away from anything actual and useful. Move on.</p><p>I’m an artificial intelligence guy. Recently <a href="https://www.scmp.com/news/china/science/article/3305761/first-encounter-chinese-ai-meets-quantum-power-and-gets-smarter-faster">a headline</a> caught my attention; “<em>Chinese AI meets quantum power and gets smarter, faster</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*8Cqhaa67Mll3RDhfYdpb4A.png" /><figcaption>from the quoted article, © scmp.com 2025</figcaption></figure><p>Hm. Has the goal post sneaked appreciably closer?</p><p>Lora is a learning strategy that comes about because it is often too computationally expensive to fine tune a large model. You typically want to tailor a large language model like Falcon or Llama for a specific problem. So instead of updating the full set of weights in the model — which would be very expensive — you approximate the weight updates themselves using smaller matrices. This reduces computational cost while often retaining most of the quality.</p><p>Key here is that we’re working on an approximation. Another approximation might suffice as well. One other approximation comes through a trick. Some variants of Lora approximate the model weights using element-wise multiplication. It turns out that if you choose that particular approximation, nature has a built-in computer that can do something similar. That built-in computer is how quants, as in quantum physics, behave. That is the trick.</p><p>So, the promise here is that the heavy lifting in fine-tuning a large language model (say) you can leave to nature. It does it really fast and cheap. In fact it does it trillions of times everywhere you care to look.</p><p>Part of the problem is that until now quantum computers have just been a promise. Some time in the future. It is still in the future, but we’re starting to see actual working implementations of this and similar ideas. They still do not pay off in a practical architecture for most problems, but with actual realizations we’re a quantum (duh!) leap closer to turning computing upside down.</p><p>Let’s dig a bit into how this works. The paper and authors behind the headline (Kong X et al, “Quantum-Enhanced LLM Efficient Fine Tuning”, <a href="https://arxiv.org/pdf/2503.12790">https://arxiv.org/pdf/2503.12790</a>) use a physical technology that is fairly close to what you might rent from IBM and to some extent Google. If you were to build one, the chip itself (a so-called Josephson circuit) contains a set of qubits — the quantum equivalent of classical bits. The chip measures millimeters, but you need to freeze it, getting it working at a couple of thousandths above absolute zero temperature (-273 degrees celsius). And you need a fairly large microwave pulse generator. And Helium gas. And cables. And control equipment. In short, you’d need a small building. And you’d get a substantial electricity bill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ne9zmHC9DvOKnOdJt-1vYw.png" /><figcaption>The <a href="https://www.nytimes.com/2023/06/14/science/ibm-quantum-computing.html">IBM Eagle</a> (left, © nytimes.com) and the Google Willow at <a href="https://blog.google/products/google-cloud/google-cloud-next-2025-sundar-pichai-keynote/#research-advances">Google Next 2025</a> (right, © google.com)</figcaption></figure><p>What about the time it would take? The trick I mentioned is such that you prepare the numbers you need for fine tuning in a very peculiar way, punch those numbers into that large microwave pulse generator, trigger the generator, let nature i.e. physics, do the calculation and read the result. The trick is such that to read the result of a single multiplication (a vector product) you need to read it many times. Perhaps a thousand or thereabouts. Each round takes some tens of microseconds. An eternity in modern computing. So with a thousand rounds we’re spending some tens of milliseconds. Fine-tuning a large language model requires updating hundreds of millions of parameters or thereabouts, and you would need to do it in many batches. The paper isn’t precise on their exact setup here, but you’re looking at hours to weeks for a single batch probably. This is orders of magnitude slower than if you were to do the training by CPU. Which nobody does, you use GPUs that are much faster. But just for comparison.</p><p>So that was physical realization, cost and time.</p><p>What about quality? In Lora you make a smaller set of numbers (a weight matrix) behave like a bigger set of numbers. Often the loss of quality is negligible, and the reduced computational footprint is substantial. But that’s not always. If the problem domain shows great variety or there is subtlety in the data the approach might fail. For example, if you with a language model try to be good at legal texts, poetry and slang at the same time. Or if you translate between very different languages. The trick I mentioned, has the advantage of being able to capture many independent features. Or non-linear behavior between them. Which Lora struggles with. The trick where we use nature’s quantum engine to approximate vector multiplication is not encumbered in this way.</p><p>So there are quality gains. But the practical outlook is challenging. If it is big, expensive and ridiculously slow, why am I writing about it?</p><p>Because now it is possible. You could rent the hardware and do this today. Some years back it was a pipe dream. It was not something you could do. Fast forward another 5 years?</p><p>That is why it caught my attention. It is all coming together. We’re not where everybody uses it yet, but we’re there where it is possible to do for a normal corporation. That is a crucial inflection point I’m thinking.</p><p>If you’ve read to here you can move on now. At least for the vast majority of you. I hope it was worth your time.</p><h4><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-2-e3cc55294ce6"><strong>Read Quantum computing in machine learning part 2 here!</strong></a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21eb44dfa142" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/quantum-computing-in-machine-learning-part-1-21eb44dfa142">Quantum computing in machine learning (part 1)</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How fast is generative cloud AI?]]></title>
            <link>https://medium.com/compendium/how-fast-is-generative-cloud-ai-6e367319f045?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e367319f045</guid>
            <category><![CDATA[gemini]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Thu, 03 Oct 2024 08:18:38 GMT</pubDate>
            <atom:updated>2024-10-07T08:26:20.089Z</atom:updated>
            <content:encoded><![CDATA[<p>The tech titans provide a variety of large language models. Clients try to balance quality, cost and latency. You generally want fast response, top response quality and minimal cost.</p><p>Many applications suffer from long response times. Which model do you choose to get better response times? If your choice of model improves response times how does the quality fare? And do you save money?</p><p>In this short article we focus on response times. We look at Google Gemini⁷ and OpenAI GPT⁸ non-streaming chat completion calls and measure what seems to influence latency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qq6Co80I4SAE_Kvp9qiPVg.jpeg" /></figure><h3>What do the vendors promise?</h3><p>OpenAI doesn’t guarantee anything w.r.t. response times³. Azure’s OpenAI services and Google provides uptime guarantees for OpenAI/Gemini respectively, but no guarantees on the maximum response time for an individual call to the api⁴ ⁵ ⁶.</p><p>Microsoft does however provide a fairly good advisory on what might influence latency¹. Here we learn</p><ul><li><strong>Model type, prompt length </strong>and<strong> response length </strong>are what matters</li><li><strong>Response length trumps prompt length </strong>particularly when the lengths are large<strong>.</strong></li></ul><h3>Empirical findings</h3><p>So how does this play out in practice?</p><p>Let’s find out. For each model, Gemini and GPT variants, I timestamp the same chat completion query⁹ . Since the advisory indicates response length is an important parameter, I ask the model to reply with lengths 10, 100 and 1000 words. The actual response lengths in bytes will vary considerably, albeit within a certain band.</p><p>I’ve plotted some results below. The latency is in ms. Length is in bytes. The query/prompt itself is about 100 bytes.</p><h4>Does response length matter?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*_8S8QH4ieJnguVsaq5h19Q.png" /></figure><p>I observe:</p><ul><li>Broadly speaking response length correlates with response latency. That fits well with the advisory from Microsoft.</li><li>Response latency is considerable, particularly for the more advanced models. Some take more than 10 seconds routinely.</li><li>For some models the response length matters a lot. For <em>gemini-1.5-pro</em> it matters most, whereas for <em>gemini-1.5-flash</em> it matters least.</li></ul><h4>Does query length matter?</h4><p>The advisory states that the prompt length also matters, albeit much less than response length. Let’s investigate that as well.</p><p>Looking at <em>Gpt-4o </em>below, the red line is the same as from previous figure, being the same query of about 100 bytes . Comparing that with a query with length of about 25 kilobytes, the green dotted line shows us that for short responses the latency is comparable, whereas for long queries, the latency is worse by a double digit percentage if the query is long.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*U4GGgZ_3lWbrSpvgTVavUw.png" /></figure><p>Not all models are like this however. For the <em>Gemini-1.5-pro</em> query length doesn’t seem to matter much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*961kY519v1itkaJsZIq8-A.png" /></figure><p>Large language models are stochastic in nature. The results above are only a single sample. See the gitlab repository⁹ for graphs of all the models and run the Go code yourself to see if your results compare with mine.</p><h3>Summary</h3><p>What did we find out?</p><p>Response length matters. The longer the response, the longer the the time it takes to get an answer. For the largest and newest models from both vendors this is most pronounced.</p><p>Query (or prompt) length does not matter much. Ask a question of a 100 bytes or 25 kilobytes, the answer comes back at roughly the same speed.</p><p>The response times in absolute numbers are strikingly large. For a response length of about 7000 bytes, <em>Gpt-4o</em> will come back to you in 20+ seconds, and with <em>Gemini-1.5-pro</em> you might wait half a minute.</p><h3>References</h3><p>[1] “Performance and latency”, <a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/latency">https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/latency</a></p><p>[2] “Making GPT API responses faster”, <a href="https://www.taivo.ai/__making-gpt-api-responses-faster/">https://www.taivo.ai/__making-gpt-api-responses-faster/</a></p><p>[3] “Is there an SLA for latency guarantees on the various engines?”, <a href="https://help.openai.com/en/articles/5008641-is-there-an-sla-for-latency-guarantees-on-the-various-engines">https://help.openai.com/en/articles/5008641-is-there-an-sla-for-latency-guarantees-on-the-various-engines</a></p><p>[4] “Gemini for Google Cloud Service Level Agreement (SLA)”, <a href="https://cloud.google.com/products/gemini/sla?hl=en">https://cloud.google.com/products/gemini/sla?hl=en</a></p><p>[5] “Gemini Online Inference on Vertex AI Service Level Agreement (SLA)”, <a href="https://cloud.google.com/vertex-ai/generative-ai/sla?hl=en">https://cloud.google.com/vertex-ai/generative-ai/sla?hl=en</a></p><p>[6] “Service Level Agreements (SLA) for Online Services”, <a href="https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1">https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1</a></p><p>[7] “Innovate faster with enterprise-ready AI, enhanced by Gemini models”, <a href="https://cloud.google.com/vertex-ai">https://cloud.google.com/vertex-ai</a></p><p>[8] “The most powerful platform for building AI products”, <a href="https://openai.com/api/">https://openai.com/api/</a></p><p>[9] <a href="https://gitlab.com/yarc68000/llm_benchmark">https://gitlab.com/yarc68000/llm_benchmark</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e367319f045" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/how-fast-is-generative-cloud-ai-6e367319f045">How fast is generative cloud AI?</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Secure chat on own live data (part 1)]]></title>
            <link>https://medium.com/compendium/secure-chat-on-own-live-data-part-1-89c54f319941?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/89c54f319941</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[retrieval-augmented]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[chat]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 06:27:01 GMT</pubDate>
            <atom:updated>2024-08-12T06:27:01.292Z</atom:updated>
            <content:encoded><![CDATA[<h3>Challenge</h3><p>Chatting or searching with the help of generative AI on your own data is popular.</p><p>The solutions are typically <em>Retrieval Augmented Generation (RAG) Apps</em>⁹ in some form. How to secure these, both with respect to app and data access, confidentiality and inappropriate response content is often a challenge. How to secure these on <em>live</em> data, i.e. things that may have changed just now, is an additional challenge. This series of articles reviews common solutions and offers practical ways to plug the holes.</p><p>Consider the common need to chat or search across multiple private data sources.</p><p><em>“How do we make sure the chat is only on data the user has access to?”.</em></p><p><em>“Does the data access reflect recent changes to access policy?”</em></p><p><em>“Does the chat reflect recent changes to source content?”</em></p><p>Some clients come with such concerns. Or requirements if you will. At Computas, an independent consultancy, we provide tailored solutions, that being custom code or products with configuration. What do we suggest for clients that have the type of concerns exemplified above?</p><p>Let’s assume for this article that the requirements are</p><ul><li>The generative AI based queries (e.g. search or chat) is to span data sources from several vendors and/or products. Let’s call this requirement “<em>Sources</em>”.</li><li>When a query is submitted, the solution shall only take into account data the specific logged-in user doing the query has access to. We name this data source “<em>Access control</em>”.</li><li>If there is a change in access, the response to a query, must take this into account immediately. We name this “<em>Access Freshness</em>”.</li><li>If there is a change in the content of a data source, response to a query, must take this into account immediately. We name this “<em>Data Freshness</em>”.</li></ul><p>The title speaks of “secure”. In this part and part 2 we deal with how to secure access to and confidentiality of the data sources. There is a lot more to securing RAG solutions⁸. We dive into that in part 3. See the bottom of the article to get to the latter parts.</p><h3>Alternatives</h3><p>When considering chat or search on own data, many clients reach for one of the many generative AI chat/search-as-a-service products out there, and by extension do not deal directly with the intricacies of a RAG. It is hard however to find mature products that cater to the above requirements.</p><p>Let’s look at some common alternatives.</p><p>Consider a primary data source product which provides its own LLM based chat/search/summarization solution. Atlassian Intelligence for Confluence¹ for example. You get access to data constrained by what the logged in user has access to. When access permissions change to a document, the search is reindexed with low, but not necessarily immediate latency². You do not get a service that spans data sources from different vendors or sources. So in terms of requirements the alternative supports a single <em>source</em>, supports <em>access control</em> at user level, <em>access freshness</em> and <em>content freshness</em> is near immediate.</p><p>You do however get multiple data sources and security with an enterprise search provider. Consider Elastic’s generative AI offering, the Relevance Engine³. You have built in access control tailored to the specific user doing the query. The access reflects the state of affairs at the last import and index however. So if there has been changes since then at the source, that is not reflected in the search. And there are non-trivial issues with mapping the semantics of the access constraints at the source with the access control model in the search engine. Summing up, this alternative supports multiple data <em>sources</em>, <em>access control</em> at user level and both access constraints and content refreshes when new content is imported and reindexed. For the purposes of this article we categorize that as <em>cached access and content freshness.</em></p><p>What about open cloud AI platform products?</p><p>For example, you might reach for Google Vertex AI Agent Builder. This is a platform where you quickly create a chat or search solution. Via connectors (“data stores” in their parlance) you provide data for an LLM enabling the chat. You can create your own connector, where you connect to a 3rd party data source, upload the data to a store within the confines of the Agent Builder product. There is control of access to the data, but at upload time, i.e. prior to indexing of the data⁴. If there is change to access permissions at the source it will not be reflected before a new import is performed. And there are possibilities of configuration mismatches between access constraint definitions at import time and constraint enforcement at the time of query. Additionally, if the data at the source changes, that is not reflected before new data has been imported to the data store. Summing up, this alternative supports multiple data <em>sources</em>, data source <em>access control</em> is in general at the level of logging in by a system user and access and content <em>freshness</em> we categorize as cached.</p><p>Let’s reach for Azure AI studio instead. The setup is very similar to the one above. In general you connect to a data source. You import data. The studio indexes the source data and makes it available to an LLM enabling the chat. You get slightly better support for authentication to third party data sources, but it remains a challenge that access to data is checked and is retrieved at construction time, and it is disjoint from the access privileges that the logged in user consuming the services might have. In general we summarize the alternative as we did with the Google alternative.</p><p>Finally, you might consider an end-to-end suite that caters for both multiple sources, user interface and everything in between. E.g. Microsoft 365 copilot. It does chat and search. It does multiple data sources. It makes sure that it only uses data that you as a logged-in user have access to⁵ ⁶. So, that’s it you might say. Not so fast. Also here, changes to the data at source might not be reflected if there is a while since last time the underlying Microsoft Graph imported data⁷. And changes to user access since the last import will not be reflected either. And then there is the broader issue of whether Microsoft 365 Copilot and the Graph product is a good fit for the client’s use case. It is, as it is with many other solutions, a considerable vendor lock-in. Many design decisions have been made for you, both in terms of data model, sharing and user interface. Summing up, the alternative supports multiple data <em>sources</em>, data source <em>access control</em> is at a user level and access and content <em>freshness</em> we categorize as cached.</p><p>We sum up our informal alternative categorization;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*M7dmgFU_BvDyt0zB_ChPKA.png" /></figure><p>Now, as you see, the alternatives could all do better at some or several requirements. Could we devise an approach which spans <em>multiple data sources</em>, enforces data source <em>access constraints</em> at the level of the logged in user doing the query and ensures that both access and content <em>freshness</em> is near immediate?</p><p>Yes there is. We’ll cover that in part 2 and part 3. Get access <a href="https://share-eu1.hsforms.com/1_BUMjTA5QKCxUNEZWEt_fAf7ofu">here</a>.</p><h3>References</h3><p>[1] “Supercharge your workflows with Atlassian AI”, <a href="https://www.atlassian.com/software/confluence/resources/guides/best-practices/atlassian-ai">https://www.atlassian.com/software/confluence/resources/guides/best-practices/atlassian-ai</a></p><p>[2] “Page Restrictions Performance Considerations”, “<a href="https://confluence.atlassian.com/confkb/page-restrictions-performance-considerations-200705378.html">https://confluence.atlassian.com/confkb/page-restrictions-performance-considerations-200705378.html</a></p><p>[3] “What is Elasticsearch Relevance Engine (ESRE)?”, <a href="https://www.elastic.co/guide/en/esre/current/learn.html">https://www.elastic.co/guide/en/esre/current/learn.html</a></p><p>[4] “Use data source access control”, <a href="https://cloud.google.com/generative-ai-app-builder/docs/data-source-access-control">https://cloud.google.com/generative-ai-app-builder/docs/data-source-access-control</a></p><p>[5] “Microsoft Graph connectors overview” <a href="https://learn.microsoft.com/en-us/graph/connecting-external-content-connectors-overview">https://learn.microsoft.com/en-us/graph/connecting-external-content-connectors-overview</a></p><p>[6] “Microsoft Graph connectors SDK connector OAuth API”, <a href="https://learn.microsoft.com/en-us/graph/custom-connector-sdk-contracts-connectoroauth">https://learn.microsoft.com/en-us/graph/custom-connector-sdk-contracts-connectoroauth</a></p><p>[7] “Set up Microsoft Graph connectors in the Microsoft 365 admin center”, <a href="https://learn.microsoft.com/en-us/microsoftsearch/configure-connector">https://learn.microsoft.com/en-us/microsoftsearch/configure-connector</a></p><p>[8] “Mitigating Security Risks in Retrieval Augmented Generation (RAG) LLM Applications”, <a href="https://cloudsecurityalliance.org/blog/2023/11/22/mitigating-security-risks-in-retrieval-augmented-generation-rag-llm-applications">https://cloudsecurityalliance.org/blog/2023/11/22/mitigating-security-risks-in-retrieval-augmented-generation-rag-llm-applications</a></p><p>[9] “Build a Retrieval Augmented Generation (RAG) App”, <a href="https://python.langchain.com/v0.2/docs/tutorials/rag/">https://python.langchain.com/v0.2/docs/tutorials/rag/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89c54f319941" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/secure-chat-on-own-live-data-part-1-89c54f319941">Secure chat on own live data (part 1)</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[It Will All Be Swept Away]]></title>
            <link>https://medium.com/compendium/it-will-all-be-swept-away-4ebe0f12f130?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ebe0f12f130</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[genai]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Tue, 05 Sep 2023 08:06:32 GMT</pubDate>
            <atom:updated>2023-09-12T09:20:19.385Z</atom:updated>
            <content:encoded><![CDATA[<p>In this short text, I speculate that we’re often too modest in what we in IT think will be impacted by large language models (LLMs). Over some few years, I think the impact on all aspects of information architecture will be profound. That means all, or quite much, of interfaces in general, data stores, data transformers, pipelines, schedulers, data formats and the like. It will all be swept away and rewritten. Well, much of it anyway.</p><p>Let me illustrate with an example from the old days.</p><h3>In the old days, before computers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nHQo0emZscF9bVsj" /></figure><p>Before the advent of information systems, if you wanted to know something, let’s say the company file on a property they own, you needed to speak to someone at corporate records. Let’s call him Smith.</p><p>You: <em>“Smith, I need the file on Sodor street. You know, our property there. In the Commodore project, they’re referring to a tenant named Alderson. I need to know the particulars.”</em></p><p>Smith: <em>“Here it is. I’ll send it over. Alderson was a tenant in the early seventies it seems. We have a file on them, I’m sending over a copied page with highlights. And after the war, our holdings at Sodor street were part of the old company back then. It so happens that Alderson owned a small share of that company, and they used the office on the 3rd floor. The one with the lift. I send you a copy of the relevant pages there as well.”</em></p><h3>Then we built all this IT information infrastructure</h3><p>Years later, with the advent of the database it was different. You logged in. You queried the property record table yourself. Then you joined the query with the client record table. After a bit of struggling with the cranky user interface, you had your report for the Commodore project. But perhaps you missed the ancient bits about the client having used the office on the third floor after the war. Maybe it wasn’t digitized information. Or maybe it was in a table you didn’t know about and which the intranet search engine hadn’t picked up.</p><h3>It will all be swept away. Well, a large chunk at least</h3><p>I’m using this example to illustrate that man has had to adapt to machines to be efficient at information work. Not the other way around. It is worth contemplating, now that we have these new generative, large language model based (LLMs) artificial intelligences (AIs). Because, in a sense, we are about to have a veritable army of Smiths. Let’s call them “AI Smith”s. Well educated. Excellent memory. Polite. Quick on the uptake. Efficient. Never tiring.</p><p>If we had all these Smiths before we started with information systems, would we have built these systems the same way? I think not. I think that we built them with the cranky, brittle interfaces and schemas because it was the best we had and they were better than a single, fallible human. If we had Smiths in abundance, we perhaps would never have built any of the IT infrastructure at all, we just would’ve kept asking Smith for information.</p><h3>Away with the user interface</h3><p>We made the user interface, the screen, the responsive design, the accessibility features, the compulsory fields to be filled out, the validation before you could press “submit”, all of it, because Smith was not there to talk to. Now Smith is about to be. And if not all, then quite much of existing user interfaces will be swept away and built anew. The machine will be made to suit man, and not the other way around. Of course, some problem domains are visual in nature, and the visual surface will not disappear, but so much form filling and brittle validation are there because there was no better alternative, and I’m guessing it will be replaced with natural language and dialogue.</p><figure><img alt="Google contact center customer experience with generative AI" src="https://cdn-images-1.medium.com/max/623/0*_69JsOSA59yPa8VY" /><figcaption>Google contact center customer experience with generative AI © Google 2023</figcaption></figure><p>From an <a href="https://www.youtube.com/watch?v=h-zFjQKKf0I&amp;t=12s">example of early work in this direction</a> I see a natural extension from crude shopping assistance chatbots to a dialogue that has the style of an old friend you’re hanging out with, but where advertisements, online sales, advice, multimedia and shipping are all intertwined in a fluid dialogue. I’m not saying that all interfaces will be like this, but you can see in this example, how the old shopping cart is replaced with something that resembles how you would navigate the matter over a dinner table, or with colleagues in a meeting room. Gone is the shopping cart and the item catalogue. It is Apple’s Siri, but where all artificiality is gone and hadn’t it been for the demi godlike abilities it has of knowing everything, representing a million agencies and being a global logistics operator, it could’ve been that old friend you’re speaking to. Whether this is dystopian or utopian, it is at least what user experience professionals (and data privacy specialists) increasingly will be dealing with. I wager a good chunk of all canvas based, forms based and select and submit like interfaces will be swept away.</p><h3>Away with the database</h3><p>If we had “AI Smith”, remembering all the records and, as old human Smith did, associating accurately across records and history, why would we need the SQL database? The database is not a perfect solution for answering human queries. It was just the best we had.</p><p>My take is that over some years, all of those databases where the need ultimately comes from human queries, will be scrapped or rewritten. Perhaps not all. And very gradually over a generation perhaps. But still, quite a lot of them. And they are many.</p><p>We see glimpses of how this will happen. How <a href="https://cloud.google.com/blog/products/data-analytics/whats-new-with-data-analytics-and-ai-at-next23">natural language may and will replace query languages</a>. Surely, for some challenges, the precision that the query language gives you will trump other needs, but for many human-centric needs, as the example with Smith at records show, the semantics and back and forth dialogue that humans excel at, will be more important.</p><p>And on a deeper level, maybe the database itself, not just the interface will change. Combining your own data with an LLM is a hot topic that <a href="https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data">vendors</a> and <a href="https://python.langchain.com/docs/use_cases/question_answering/">open source communities</a> are racing to address. Present approaches, both via so-called fine tuning and through few-shot learning, have significant <a href="https://www.quora.com/What-are-the-challenges-of-fine-tuning-and-training-chat-GPT-models-for-specific-use-cases-and-how-can-they-be-overcome">disadvantages</a>. How to store and learn from a steady stream of new information is an <a href="https://proceedings.mlr.press/v162/zhmoginov22a.html">open research area</a>. The ways large language models will impact data management are clearly unknown but <a href="http://raulcastrofernandez.com/papers/llm_db_vision_vldb23-11.pdf">some argue</a> it will be disruptive, where key industry challenges will be revisited for new solutions and patterns.</p><h3>Away with JSON, XML, MOF, BPM, EDI, REST etc</h3><p>And if you’ve replaced your database with “AI Smith”, what about the interfaces, the APIs, internal to the application you used in the office to query the records? In many cases there are of course legitimate needs for precise, brittle, defined, formal communication protocols. Particularly when the original source is a sensor for example, or when the end-user is a machine. The protocols are painful nevertheless. They break down. Semantics are lost for every transform. New needs arise and the work to formalize and incorporate new expressions and new meanings take enormous effort.</p><p>If at one end you have an information producer that originally was human and the content is language based, and at the other end you have a human consumer, then perhaps all those defined, brittle protocols in the middle of information architectures are more pain than gain?</p><p>Here as well, my take is that over some years, a good chunk of all interfaces and communication protocols will be scrapped and rewritten as well. I don’t know exactly how many at the end, and not how fast. But there will be an enormity.</p><p>Research gives us some indication on how component communication and interfaces may look in the future. Firstly, you got the obvious, the communication between machine and a human, which now, thanks to LLMs may increasingly be happening in a language suitable for man. Then, you’ve got communication between LLM empowered machine components, where for example, new <a href="https://www.politico.com/newsletters/digital-future-daily/2023/03/02/psst-when-ais-talk-among-themselves-00085282">more efficient negotiating strategies may be pursued</a>. Here the language is still human language. But it clearly doesn’t need to be. If the human is out of the loop, but a deep neural network is talking to another, altogether new, efficient <a href="https://arxiv.org/pdf/2006.02419.pdf">communication forms may emerge</a> with alien semantics and syntax, but that functionally is a better fit.</p><p>This concludes my speculation. There are exciting times ahead. And if I’m right, no shortage of refactoring work for developers (like us at <a href="https://computas.com/">Computas</a>) and probably for Smith as well. <a href="https://cloud.google.com/blog/products/application-development/introducing-duet-ai-for-developers">AI Smith</a> that is!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ebe0f12f130" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/it-will-all-be-swept-away-4ebe0f12f130">It Will All Be Swept Away</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Large Language Models + Robots = the next “big thing”]]></title>
            <link>https://medium.com/compendium/large-language-models-robots-the-next-big-thing-e6eac7cdf165?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6eac7cdf165</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Fri, 11 Aug 2023 09:55:41 GMT</pubDate>
            <atom:updated>2023-08-15T09:20:02.919Z</atom:updated>
            <content:encoded><![CDATA[<p>The importance of good datasets cannot be overstated in AI.</p><p>There is groundbreaking work going on that combines natural language ability and robotics. The potential is huge, imagine having a robot that understands, thinks, plans and responds in real-time to natural language input. Or vice-versa, a ChatGPT look-a-like which learns that all that it hallucinates doesn’t correspond to physical reality and thus corrects itself, in the same way as we humans do. Among all the actors in the robotics field, some focus their work on the combination of large language models (LLMs) and robotics. Microsoft and OpenAI works on this (e.g. Vemprala et al, <a href="https://arxiv.org/pdf/2306.17582.pdf">https://arxiv.org/pdf/2306.17582.pdf</a>) as does Google. They are churning out a steady stream of high quality papers and accompanying labs and physical installations to be at the forefront of this next potential “big thing” in artificial intelligence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m3vlxneAo50hBQYOZbT_ug.jpeg" /></figure><p>A key ingredient in previous quantum leaps in AI have been the establishment of datasets. These have been used in competitions, making results in papers reproducible, and for focusing research efforts on certain tasks. <em>What sets Google apart, is their pursuit focused on establishing benchmarks with accompanying datasets</em> for LLM + Robotics.</p><p>In October last year Google robotics and Stanford published dataset related work on whether a set of robot actions corresponds to a language label (Nair et al. 2022, <a href="https://arxiv.org/pdf/2109.01115.pdf">https://arxiv.org/pdf/2109.01115.pdf</a> ). The dataset addresses whether a set of robot actions fulfill a task, e.g. “open the left drawer”? Through crowdsourcing and video, thousands of episodes like this constitute a dataset that forms the foundation for connecting robotic action with task semantics.</p><p>In a parallel effort, during the development of the SayCan robot, Google over a year amassed close to 70000 manually assisted robot behavior demonstrations using 10 robots as well as more than a quarter of a million autonomous episodes. This extensive dataset made it possible to ground large language models to physical reality, in turn enabling robots to plan and act flexibly on commands entailing multiple steps for success (Ahn et. al, 2022 Google Robotics and Everyday Robots (<a href="https://arxiv.org/pdf/2204.01691.pdf">https://arxiv.org/pdf/2204.01691.pdf</a> ).</p><p>Also in October 2022, Google robotics published work on how to establish datasets for connecting language with robotic behavior. These are benchmarks for the AI and robotic community. As mentioned, high quality, public datasets tied to a task type are a critical enabler for making progress in AI. In this round, Google robotics’ dataset contains more than half a million labeled demonstrations. With this they build robots that can perform close to a hundred thousand different behaviors specified in natural language. (Lynch et al. 2022, Google robotics, <a href="https://arxiv.org/pdf/2210.06407.pdf">https://arxiv.org/pdf/2210.06407.pdf</a> ). Much of this work was on so-called short-horizon behavior. This prepared the ground for combining this tactical perspective with longer horizon planning.</p><p>With Palm-E in march ’23 (Driess et al. 2023, Google research and robotics, TU Berlin, <a href="https://palm-e.github.io/assets/palm-e.pdf">https://palm-e.github.io/assets/palm-e.pdf</a> ) we start to see the emergence of robots that combine language and sensory input as a new multimodal “language”. Some of the “words” in the sentence are sensory input. This is used as input in using and training a model. The output is text that can be interpreted and executed by a robot. This groundbreaking methodical work relies among other things, on the aforementioned dataset work of Lynch et al. The net result is a fairly long step towards machine learning models that are natural language speakers and at the same time better grounded and relating to and affecting the physical world.</p><p>Although a year old, this clip sets the context and explains fairly well where we’re headed; <a href="https://www.youtube.com/watch?v=dCPHGwW9SOk">https://www.youtube.com/watch?v=dCPHGwW9SOk</a> .</p><p><a href="https://computas.com/">Computas</a> has long experience with AI, we presently work with clients on large language models (LLMs), and we’re really good at working methodically with datasets. Talk to us if you have interesting challenges.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6eac7cdf165" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/large-language-models-robots-the-next-big-thing-e6eac7cdf165">Large Language Models + Robots = the next “big thing”</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When and why using (Go) function calls in GPT]]></title>
            <link>https://medium.com/compendium/when-and-why-for-go-function-calls-in-gpt-ec4b8b2b1c7d?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec4b8b2b1c7d</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[ethics]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Wed, 02 Aug 2023 07:14:04 GMT</pubDate>
            <atom:updated>2023-08-04T10:14:00.901Z</atom:updated>
            <content:encoded><![CDATA[<p>The new generative AIs are being used for all kinds of purposes. It is a time for experimentation. And caution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIGXeYHAlTwBUDPXBNweRg.jpeg" /></figure><p>At <a href="https://computas.com/en/">Computas</a>, one of the things we’re using OpenAI’s GPT models for is information transformation. For some clients we transform semi structured information with a lot of prose into structured information. Similar to what we did with Natural Language Understanding (NLU), entity extraction (NER) and slot-filling prior to the arrival of these new powerful LLMs. Frequently, the need is to use that structured information to do some background processing. It might be something that changes state in some backend system. Or do a search. Or some enrichment. The result from this processing we in some projects need to put back into prose for human consumption.</p><p>Let’s say for example you have voice or text messages representing purchasing orders. A message may say <em>“Hi, we don’t need the screws after all. We’ll do it with clout nails instead. So perhaps 250 30 mm. And the same with 50 mm. Both 2.8. Send it with the same package if you please. Dick”</em>. The challenges here are many; what has actually been ordered here? What types of items. How many. What size. And which existing order is being modified with what exactly? GPT gives us an entirely new solution space for this, which shows great promise.</p><p><strong>Function calls</strong></p><p>One task for GPT could be to ask it to fill out a structured form representing an order. The form could be a schema where salient fields are “item_type”, “amount” and “size”. With that information in hand a background process could map these fields to a formally defined item registry where a unique identifier pins down the item in question. This could then be used by an order system further downstream. OpenAI’s GPT API gives us some interesting choices when we want to find the information nuggets in a message. We can use vanilla prompt engineering and ask GPT to reply in a structured way;</p><pre>Read this order; &quot;Hi, we don&#39;t need the screws after all. We&#39;ll do it with clout nails instead. So perhaps 250 30 mm. And the same with 50 mm. Both 2.8. Send it with the same package if you please. Dick&quot;<br><br>Please find each item, the amount, size and unit and put each  on a line on the form <br><br>item type : size : unit : amount</pre><p>where GPT (ChatGPT in this example) answers</p><pre>Item type: Clout nails<br>Size: 30 mm<br>Unit: 2.8 (assumed to be the diameter or gauge of the nails)<br>Amount: 250<br><br>Item type: Clout nails<br>Size: 50 mm<br>Unit: 2.8 (assumed to be the diameter or gauge of the nails)<br>Amount: 250</pre><p>As you can see, this free form prose comes with challenges if you’re to use this as input to a backend function. Neither did GPT produce single colon separated lines per item as we requested, nor is the Unit field suitable.</p><p>Recently, OpenAI has provided us with a <a href="https://platform.openai.com/docs/guides/gpt/function-calling">function call interface</a> as an alternative. This is a sequence of API calls where you with the initial prompt provide a function signature. <a href="https://gist.github.com/yarcyarc/dee9ed209e62a59b818a5a4be013a6ce">Illustrating</a> in Go, and using Alexander Baranov’s dependable <a href="https://github.com/sashabaranov/go-openai">go-openai</a> library we define <em>functions</em>;</p><pre>resp, err := client.CreateChatCompletion(<br>  context.Background(),<br>  openai.ChatCompletionRequest{<br>   Model:     openai.GPT3Dot5Turbo0613,<br>   Messages:  messages,<br>   Functions: []openai.FunctionDefinition{*functions},<br>  },<br> )</pre><p>GPT decides itself whether some of the text fits the function signature(s) and might respond with a filled out parameter section for the call. We, the developer, decide how to act upon this, possibly making a backend call with the parameters provided, here illustrated with the <em>callFunction()</em></p><pre>if resp.Choices[0].FinishReason == &quot;function_call&quot; {<br><br>  messages = append(messages, resp.Choices[0].Message)<br><br>  functionToCall := resp.Choices[0].Message.FunctionCall.Name<br>  args := resp.Choices[0].Message.FunctionCall.Arguments<br>  functionResponse := callFunction(functionToCall, args)<br><br>  message3 := openai.ChatCompletionMessage{<br>   Role:    openai.ChatMessageRoleFunction,<br>   Content: functionResponse,<br>   Name:    functionToCall,<br>  }<br><br>  messages = append(messages, message3)<br><br> }</pre><p>Then you provide the result of the backend call to GPT.</p><pre>resp, err = client.CreateChatCompletion(<br>  context.Background(),<br>  openai.ChatCompletionRequest{<br>   Model:    openai.GPT3Dot5Turbo0613,<br>   Messages: messages,<br>  },<br> )</pre><p>GPT then responds in this third round with a natural language response where the background call’s response is baked in as prose. A python version is in the OpenAI <a href="https://platform.openai.com/docs/guides/gpt/function-calling">doc</a>. The technicalities of the procedure in python has been well covered by various <a href="https://github.com/rizerphe/local-llm-function-calling">blogs</a> and <a href="https://towardsdatascience.com/the-power-of-openais-function-calling-in-language-learning-models-a-comprehensive-guide-cce8cd84dc3c">articles</a>.</p><p><strong>Why and when?</strong></p><p>So what are the pros and cons of using this new way of calling backend services? What we find is that the pros are</p><ul><li>Function calls are a succinct way of filling out function parameters for a backend call in a dialogue. There might be less prompt engineering and clutter.</li><li>You might have a better chance of filling out required parameters and valid json.</li><li>You get a natural language response back to the end-user out of the box.</li></ul><p>There are some cons, or at least things to be cautious about however</p><ul><li>GPT isn’t given a lot of information about the call. As the snippet above shows, you get to describe the function and the parameters, but it leaves ample room for misinterpretation. Given a <a href="https://platform.openai.com/docs/quickstart/adjust-your-settings">temperature</a> above zero, the responses are stochastic and you might find that a particular GPT response doesn’t capture the meaning of the function response sufficiently.</li><li>Similarly, you might find that GPT is better at populating the parameters you need by providing context and guidance in the prompts rather than as part of the function call specification</li><li>Since the function call is prepared for you by GPT, you might gradually slide towards less human validation and less checks before you call backend functions that may have real world consequences. You need to be careful.</li><li>You might want to call more than one function as a response to a prompt. The API gives you limited <a href="https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb">freedom</a>. You might want more flexible semantics.</li><li>In some cases you might not need to provide a natural language response after the function call. For the message example provided above and with the client we work with there, that need is not prominent.</li><li>You need to use <a href="https://platform.openai.com/docs/models/models">specialized models</a> that are quickly deprecated.</li></ul><p>The API and technology is in great flux of course. Next month there might be new, even better ways of dealing with these classic NLP challenges.</p><p>Happy coding :-)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec4b8b2b1c7d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/when-and-why-for-go-function-calls-in-gpt-ec4b8b2b1c7d">When and why using (Go) function calls in GPT</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical security at the nexus of Kubernetes, Devops and Cloud]]></title>
            <link>https://medium.com/compendium/practical-security-at-the-nexus-of-kubernetes-devops-and-cloud-3415acbfcf76?source=rss-a9883a9f4a82------2</link>
            <guid isPermaLink="false">https://medium.com/p/3415acbfcf76</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[devsecops]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Morten Forfang]]></dc:creator>
            <pubDate>Thu, 15 Oct 2020 07:32:44 GMT</pubDate>
            <atom:updated>2020-10-15T07:32:44.535Z</atom:updated>
            <content:encoded><![CDATA[<h3>Practical Security at the Center of Kubernetes, DevOps and Cloud</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t5BJNbhZeRWnYfJlrj27EA.png" /><figcaption>A not so untypical Kubernetes solution</figcaption></figure><p>Many of our clients are building solutions in the cloud based on Kubernetes. They have a modern software development methodology, so agile practices are a given to them, and if they haven’t already done so, they’re trying to implement DevOps. But each of these three concepts –Kubernetes containers, cloud and DevOps– are <em>competence intensive disciplines</em>, with a considerable flux. They require constant attention and effort to stay on top of the latest updates.</p><p>And then you have security. Security by itself is a rapidly evolving field, also requiring constant attention. But trying to secure something that is built on rapidly changing technology is quite a challenge. It’s therefore no surprise that many of our clients struggle to cope with all the new vulnerabilities being discovered every day, and updates and changes to the security mechanisms and best practices to secure their solutions.</p><p>One possible strategy to handle all these challenges is to enforce compliance with a standard, a set of principles, or maybe a list of requirements that the company has set up. Alternatively, or in addition, one might do security assessments, perhaps even on a regular basis, either internally or by an external partner.</p><p>For a software development team, the results of such assessments often only add to the headache. The team now has a list of known vulnerabilities and non-compliances. Usually, the list is long and in many cases of the check-list variety. But where should the team start? Even if you just look at a single item, what would be sufficient effort to be able to cross it off the list? Which of these items need to be checked off, and which can left on the list as an issue you can live with? And are you out of the woods by the time you’re done with the list? Often not. New features have been added. The supply chain has changed. You need to decide when to start again. As the exercise takes effort away from other things — like creating functionality — the team has a built-in incentive to do security reviews or assessments seldom and with minimal effort.</p><p>There is a better way to go about this.</p><p>At <a href="https://computas.com/en">Computas</a>, when we work with project cybersecurity or advise software teams, we strive to make the list short and prioritized. And even more important: list items are not of the check-list variety; we provide recipes for what to do more than stating a goal without saying how to get there or how much effort is sufficient.</p><p>Our practice framework, which we’ve termed “6C”, is inspired by <a href="https://kubernetes.io/docs/concepts/security/overview/">Kubernetes “4C” of Cloud Native Security</a>, and includes the following:</p><ul><li>it’s an <strong>evolving, operational</strong> framework,</li><li>it focuses on the nexus <strong>cloud, DevOps </strong>and<strong> container</strong>, and has peripheral focus on topics that relate to these three core areas,</li><li>it stems from <strong>practical work</strong> with security in <strong>software development</strong>.</li><li>it may fit into various security management and risk framework <strong>standards</strong> as operational capacity.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruhfhzmFkvU7ELbqa-UCWg.png" /><figcaption>How the “6C” cybersecurity framework relates to other security topics. Green is core focus, yellow is peripheral, and red not a focus.</figcaption></figure><p>Given a focus on subject-matter, cf. the figure above, we typically do a broad, brief survey of the different “C”s when assisting a project:</p><ul><li><strong>Core</strong>: These are <em>foundational security issues</em> where we work through project setup, ownership, organization, networks, monitoring, culture, competence, prevention vs. containment, disaster management, how the project handles risk and incidents, separation of duty, identity and roles. This C’s slogan is <em>“Know what you’re doing. Own up.”</em></li><li><strong>Chain</strong>: This is where we focus on <em>process security</em>: pipelines, doing things over and over, automation, and toil. Frequent review topics are software delivery pipeline, supply chain (3rd party dependencies), infrastructure-as-code, testing, operations/DevOps, the risk+incidents+backlog nexus and sustained quality. This C’s slogan is <em>“Do it all, every time. Sleep well.”</em></li><li><strong>Cluster</strong>: Here we work with security on <em>nodes, network and VMs</em>. Frequent topics are cluster hardening, API access, image management, zero-trust, defence-in-depth, mesh, secrets, logging, monitoring, isolation, attack surface and service account/access token management. This C’s slogan is “<em>Secure the VMs.</em>”</li><li><strong>Container</strong>: This is often about <em>Dockerfiles</em>, dependencies,<em> images</em>, repositories, hardening and privileges. This C’s slogan is “<em>Secure the images.</em>”</li><li><strong>Code:</strong> This is about the <em>internals of the business application itself</em>. Focus areas are API vulnerabilities, resilience, confidentiality and information integrity, application logic, clients, authentication and authorization. This C’s slogan is <em>“Secure the application.”</em></li><li><strong>Components</strong>: Here we work through security aspects for important <em>run- and build-time components</em> in the architecture. These are typically managed cloud components like databases, API management, Pub/Sub, firewalls, builders and SaaS services that the solution depends on. Since this is about components, core topics center on <em>interfaces and component life-cycle</em>. Often this means access control, authentication, governance, encryption, information integrity and confidentiality. This C’s slogan is <em>“Secure the components.”</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*bofXAPZm9fRac4FPabPflA.png" /><figcaption>Excerpt from Kubernetes “4C”, which inspired Computas “6C” (<a href="https://kubernetes.io/docs/concepts/security/overview/">https://kubernetes.io/docs/concepts/security/overview/</a>)</figcaption></figure><p>While the topic list may seem exhausting, there are three ways we <em>constrain the scope</em>. <strong>(a)</strong> As mentioned, the “6C”s focus is on <em>cloud, containers and DevOps</em>. Other topics, as the figure above shows, are dealt with in a more cursory manner. <strong>(b)</strong> As mentioned we often do a <em>broad and brief</em> — meaning days and not weeks — assessment of the state of security of a project before we possibly do “deep dives” into particular areas. This leads to <strong>(c)</strong> a <em>prioritization</em> of issues. We work on the most important ones.</p><p>The outcome of a brief assessment as mentioned in (b) may be an advisory, i.e. a presentation or a report. The contents are not <em>compliance goals</em> referring to a standard or a company check-list, but a set of recommended <em>actions</em>. Do this, implement this, improve this, etc. Often the recommended actions are more geared towards process — e.g. put in place a way to decide whether incidents are to be transformed into product backlog items or introduce a step that breaks the build if an image dependency is not whitelisted — rather than fix individual security flaws. If a team chooses to do a “deep dive”, we might arrange for a temporary extension of the team, allowing a security expert working with the team for a time. Here the focus is more on operational capacity, <em>doing things</em>, than writing down actions for someone else to do.</p><p>I hope that was inspiring. Below, I’ve included some of the resources we often rely on.</p><p>—</p><p>Chain:</p><ul><li>“<em>DevOpsSec</em>” by Jim Bird, <a href="https://learning.oreilly.com/library/view/devopssec/9781491971413/">https://learning.oreilly.com/library/view/devopssec/9781491971413/</a></li><li><a href="https://www.microsoft.com/en-us/securityengineering/devsecops">https://www.microsoft.com/en-us/securityengineering/devsecops</a></li></ul><p>Core:</p><ul><li><a href="https://docs.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns">https://docs.microsoft.com/en-us/azure/security/fundamentals/best-practices-and-patterns</a></li><li><a href="https://cloud.google.com/blog/products/networking/google-cloud-networking-in-depth-three-defense-in-depth-principles-for-securing-your-environment">https://cloud.google.com/blog/products/networking/google-cloud-networking-in-depth-three-defense-in-depth-principles-for-securing-your-environment</a></li></ul><p>Components:</p><ul><li><a href="https://azure.microsoft.com/mediahandler/files/resourcefiles/security-best-practices-for-azure-solutions/Azure%20Security%20Best%20Practices.pdf">https://azure.microsoft.com/mediahandler/files/resourcefiles/security-best-practices-for-azure-solutions/Azure%20Security%20Best%20Practices.pdf</a></li></ul><p>Cluster:</p><ul><li><a href="https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/">https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster/</a></li><li><a href="https://www.cisecurity.org/benchmark/kubernetes/">https://www.cisecurity.org/benchmark/kubernetes/</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster">https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster</a></li><li><a href="https://docs.microsoft.com/en-us/azure/aks/concepts-security">https://docs.microsoft.com/en-us/azure/aks/concepts-security</a></li><li><a href="https://www.cvedetails.com/vulnerability-list/vendor_id-15867/product_id-34016/Kubernetes-Kubernetes.html">https://www.cvedetails.com/vulnerability-list/vendor_id-15867/product_id-34016/Kubernetes-Kubernetes.html</a></li></ul><p>Container:</p><ul><li><a href="https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf">https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-190.pdf</a></li><li><a href="https://github.com/aquasecurity">https://github.com/aquasecurity</a></li><li>“<em>Container Security</em>” by Liz Rice, <a href="https://learning.oreilly.com/library/view/container-security/9781492056690/">https://learning.oreilly.com/library/view/container-security/9781492056690/</a></li></ul><p>Code:</p><ul><li><a href="https://github.com/OWASP/wstg/tree/master/document">https://github.com/OWASP/wstg/tree/master/document</a></li><li><a href="https://mobile-security.gitbook.io/mobile-security-testing-guide/">https://mobile-security.gitbook.io/mobile-security-testing-guide/</a></li><li><a href="https://owasp.org/www-project-proactive-controls/">https://owasp.org/www-project-proactive-controls/</a></li><li><a href="https://www.microsoft.com/en-us/securityengineering/sdl/practices">https://www.microsoft.com/en-us/securityengineering/sdl/practices</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3415acbfcf76" width="1" height="1" alt=""><hr><p><a href="https://medium.com/compendium/practical-security-at-the-nexus-of-kubernetes-devops-and-cloud-3415acbfcf76">Practical security at the nexus of Kubernetes, Devops and Cloud</a> was originally published in <a href="https://medium.com/compendium">Compendium</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>