<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Madhawa Perera on Medium]]></title>
        <description><![CDATA[Stories by Madhawa Perera on Medium]]></description>
        <link>https://medium.com/@madhawacperera?source=rss-51210a3a6f0d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*ODTjTBy-6Z6ECDilZW4Pgg.jpeg</url>
            <title>Stories by Madhawa Perera on Medium</title>
            <link>https://medium.com/@madhawacperera?source=rss-51210a3a6f0d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:31:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@madhawacperera/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[No Internet, No Cloud: Train and Run Browser-Based Keyword Detection with TensorFlow.js]]></title>
            <link>https://madhawacperera.medium.com/no-internet-no-cloud-train-and-run-browser-based-keyword-detection-with-tensorflow-js-a81c19a3b0ac?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/a81c19a3b0ac</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[keywords-detection]]></category>
            <category><![CDATA[model-training]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[tensorflowjs]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Wed, 22 Apr 2026 08:50:36 GMT</pubDate>
            <atom:updated>2026-04-23T08:59:17.517Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*f3eW0dNBCGVO-Hi-LgZ5zA.gif" /></figure><p>Recently, I explored building a browser based WebXR experience where simple actions like start, pause, and resume could be controlled by voice, without relying on internet connectivity or cloud APIs every time the user spoke. Button presses can sometimes be inconvenient in XR, especially when the user is already immersed and their hands are busy, like teleoperating a humanoid using your own hands ;) This is why a lightweight local keyword detector was needed, so the app could react to short spoken commands directly in the browser.</p><p>So the goal was to have a few triggers that could be controlled by voice, for example to start, pause, and resume an operation. We did not want to transcribe full sentences. We were not trying to understand grammar, punctuation, or long form speech. We only wanted to detect a small set of fixed words that map to specific app actions. This article is about how TensorFlow.js was used to achieve that.</p><h4>Why not full speech-to-text?</h4><p>For full speech to text, larger models and heavier runtime setups are usually needed. Depending on the stack, that can involve ONNX Runtime Web, Transformers.js, web workers, model sharding, and more aggressive performance tuning. That is valid for transcription heavy applications, but it is often overkill for keyword triggering.</p><p>For keyword spotting in browser apps, TensorFlow.js with the Speech Commands model is a great fit. It already gives us a compact pretrained audio feature pipeline and a transfer-learning path. That means we can quickly adapt the model to custom words like start, pause, train in-browser, export model files, and run fully offline in a listener page.</p><p>So in this article, I’ll take you through how I developed a simple web app to collect examples for custom keywords, train a small transfer head, download the model files, and then load and run them in the production page without internet access.</p><h4>What is transfer learning?</h4><p>Before going into the details, for those who are new to transfer learning, it means you do not train a model from scratch. Instead, you reuse a model that has already learned useful patterns from a large dataset, then train a smaller task specific layer on top for your own labels.</p><p>In this case, the pretrained Speech Commands model already captures many low level speech characteristics: frequency patterns, temporal changes, and rough phonetic structure in short audio windows. We keep that knowledge and only train the final classifier for our custom keywords. You might then be thinking, why not fine tune it instead? Let’s do a quick comparison.</p><p>Transfer learning usually freezes most or all of the pretrained feature extractor and trains only a new classification head. A classification head is the final part of the model that takes the extracted features and maps them to our output labels, such as new keywords. In architecture terms, the backbone stays fixed and only the top task specific layers are updated. This is computationally cheap and data efficient. Fine tuning means you unfreeze some pretrained layers, sometimes all of them, and continue gradient based optimisation end to end, typically with a smaller learning rate. That can improve task accuracy when more data and compute are available, but it also risks overfitting and catastrophic forgetting when the dataset is small. So transfer learning is “add a new smart last layer”. For us transfer learning gave 3 main advantages:</p><ul><li>it needs much less training data, and in practice around 25 recordings per keyword can often be enough</li><li>it trains much faster in the browser</li><li>it is usually more stable than training a small model from scratch with limited data</li></ul><p>That’s a bit of background, so let’s dive into this demo.</p><h4>Prerequisites</h4><p>Before we jump into the implementation, let’s first download the thirdparty files this demo expects into a folder. So we can load script/model files from local paths. I named the folder as vendor. Below is what’s inside that folder.</p><pre>── vendor<br>    ├── speech-commands-model<br>    │   ├── group1-shard1of2<br>    │   ├── group1-shard2of2<br>    │   ├── metadata.json<br>    │   └── model.json<br>    ├── speech-commands.min.js<br>    └── tf.min.js</pre><p>If you are new to this stack, here is the quick breakdown of what these files do:</p><ul><li>tf.min.js is the bundled core Tensorflow.js ML runtime (tensors, kernels, training/inference execution). .min.js means minified JavaScript.</li><li>TensorFlow.js Speech Commands (speech-commands.min.js) is a higher-level helper library from the TensorFlow.js models repo. It provides keyword-spotting APIs like speechCommands.create(…), createTransfer(…), collectExample(…), train(…), and listen(…).</li><li>The files under speech-commands-model are pretrained model assets (topology, weight manifest, metadata, and weight shards) used by that Speech Commands library.</li></ul><p>So, I will refer to Speech Commands asboth a library API layer and an associated pretrained model family. We need the library file to get the browser keyword-recognition API, and we need the model files so the recognizer has pretrained weights to transfer from.</p><p>To download these file you can use below curl commands. For this part, internet access is only needed once.</p><pre>mkdir -p vendor/speech-commands-model<br>curl -L &quot;https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@4.22.0/dist/tf.min.js&quot; -o vendor/tf.min.js<br>curl -L &quot;https://cdn.jsdelivr.net/npm/@tensorflow-models/speech-commands@0.5.4/dist/speech-commands.min.js&quot; -o vendor/speech-commands.min.js<br>curl -L &quot;https://storage.googleapis.com/tfjs-models/tfjs/speech-commands/v0.4/browser_fft/model.json&quot; -o vendor/speech-commands-model/model.json<br>curl -L &quot;https://storage.googleapis.com/tfjs-models/tfjs/speech-commands/v0.4/browser_fft/group1-shard1of2&quot; -o vendor/speech-commands-model/group1-shard1of2<br>curl -L &quot;https://storage.googleapis.com/tfjs-models/tfjs/speech-commands/v0.4/browser_fft/group1-shard2of2&quot; -o vendor/speech-commands-model/group1-shard2of2<br>curl -L &quot;https://storage.googleapis.com/tfjs-models/tfjs/speech-commands/v0.4/browser_fft/metadata.json&quot; -o vendor/speech-commands-model/metadata.json</pre><h4>Development Guide</h4><p>There are 2 ways to follow this demo.</p><p><em>First, if you want to quickly get it running and test it yourself, I recommend checking the </em><a href="https://github.com/madhawap/keywords-detection-example"><em>GitHub link</em></a><em>, downloading the repo, and either reading the “How to run and test this” section below or the README in the GitHub repo. This way, you can train a model and test it in less than 10 minutes.</em></p><p><em>The second way is to follow the guide below, where things are explained step by step. If you are new to this, it is recommended to read and implement it (you can copy paste the code), as it will give you a better understanding of how it works.</em></p><p>So we need 4 files altogether as below.</p><ul><li>trainkeyword.html + trainkeyword.js for collecting samples and training.</li><li>keywordlistener.html + keywordlistener.js for loading the saved model and detecting words live.</li></ul><p>The final project repo structure looks like below.</p><pre>├── keywordlistener.html<br>├── keywordlistener.js<br>├── README.md<br>├── trainkeyword.html<br>├── trainkeyword.js<br>└── vendor<br>    ├── speech-commands-model<br>    │   ├── group1-shard1of2<br>    │   ├── group1-shard2of2<br>    │   ├── metadata.json<br>    │   └── model.json<br>    ├── speech-commands.min.js<br>    └── tf.min.js</pre><p>Both pages use the same base Speech Commands model from local vendor folder, so everything can run offline after assets are available.</p><h4>Training page: collect examples</h4><p>Now the first task is to build a labeled dataset for our custom commands/keywords. I wanted to type keywords (e.g. start, pause, resume), click a record button, and say each word, and do this repeatedly so each label has enough examples. Each click captures one short microphone window. We also need to. record ‘background_noise’ examples, because the classifier must learn what to ignore, not only what to detect.</p><p>With that plan in mind, the first question was: how do we capture audio when Record button is pressed? The good news is that the app does not directly call getUserMedia(…) in this demo. TensorFlow.js Speech Commands handles that internally. On first use during data collection or listening, it requests microphone permission, captures audio through the browser’s media and audio APIs, runs the audio through its built in feature pipeline, and computes BROWSER_FFT input features. BROWSER_FFT here refers to the pretrained browser speech-command frontend setup provided by TensorFlow.js Speech Commands. It already knows how to convert incoming audio into useful frequency-based features, so we do not start from zero. BROWSER_FFT is used because it is the default, browser-optimized frontend for TensorFlow.js Speech Commands. FFT stands for Fast Fourier Transform, a math operation that converts audio from time domain (waveform over time) into frequency domain. Speech models work better when they can analyze these frequency patterns.</p><p>Before any button click, above setup already happens during page load (see app() in trainkeyword.js) . We create a base recognizer from the pretrained BROWSER_FFT model, then create a transfer recognizer on top of it.</p><pre>let baseRecognizer; // wraps the pretrained 18-word model.<br>baseRecognizer = speechCommands.create(&#39;BROWSER_FFT&#39;, undefined, LOCAL_MODEL_URL, LOCAL_METADATA_URL);<br>await baseRecognizer.ensureModelLoaded();<br><br>const name = (document.getElementById(&#39;modelName&#39;).value || &#39;my-keywords&#39;).trim();<br>transfer = baseRecognizer.createTransfer(name);</pre><p>Then, on Record button click, our first data-collection action is recordKeyword(…) calling await transfer.collectExample(name). collectExample(name) captures one window, converts it into spectrogram-like time-frequency features, and stores that sample in the transfer recognizer’s in-memory dataset under the label/name you passed (for example start , pause, resume or _background_noise_). A spectrogram is frequency content over time (like a heatmap: x-axis time, y-axis frequency, color intensity). During collection, the model is learning from these compact time-frequency patterns. Repeating this process builds the dataset for the next step: transfer.train(…).</p><pre>async function recordKeyword(idx) {<br>  const nameInput = document.getElementById(&#39;kw&#39; + idx);<br>  const name = (nameInput.value || &#39;&#39;).trim();<br>  try {<br>    await transfer.collectExample(name);<br>  } catch (err) {<br>    setStatus(`Failed to record: ${err.message}`);<br>    return;<br>  } finally {<br>    btn.classList.remove(&#39;recording&#39;);<br>  }</pre><p>So flow is: Record button -&gt; audio captured -&gt; converted into model-friendly features -&gt; stored in the transfer recognizer dataset.</p><h4>Training page: Train transfer head</h4><p>After enough examples are collected (usually 15–25 per keyword), we move from “data collection” to “learning.” The plan in this step is to click a button named Train, run multiple passes over the collected examples, and update the transfer head so the custom labels become distinguishable.</p><p>When we press Train button our code calls transfer.train(…). We only need to write that one API call and provide options (like epochs and callbacks). TensorFlow.js + Speech Commands handle the training loop, batching, gradient updates, and loss/accuracy computation internally. Note that the dataset is already attached to the transfer recognizer instance.</p><p>Again, if you are new, what is an epoch? One epoch means one full pass over our current training dataset. If you set epochs: 25, the model sees the whole dataset 25 times (in shuffled mini batches) and gradually adjusts the weights on each pass.</p><pre>try {<br>    await transfer.train({<br>      epochs,<br>      callback: {<br>        onEpochEnd: (epoch, logs) =&gt; {<br>          lastEpochMsg =<br>            `Epoch ${epoch + 1}/${epochs}  —  ` +<br>            `acc: ${(logs.acc * 100).toFixed(1)}%  loss: ${logs.loss.toFixed(3)}`;<br>          setStatus(lastEpochMsg);<br>        },<br>      },<br>    });<br>  } catch (err) {<br>    setStatus(`Training failed: ${err.message}`);<br>    toggleButtons(true);<br>    return;<br>  }</pre><p>Now you might be wondering, where does this model training computation run? TensorFlow.js runs on a backend selected at runtime. The browser can execute kernels on: webgl for GPU execution through WebGL, webgpu for GPU execution through WebGPU, when available, wasm for CPU execution through WebAssembly or cpu as a plain JavaScript fallback. Who configures this? Well, we can explicitly configure it in the app code with tf.setBackend(…), but if we do not set one, TensorFlow.js chooses the best available backend from what is bundled and supported by the browser and device.</p><h4>Training page: Save model as downloadable files</h4><p>Now this part is also mostly facilitate by APIs. When training finishes, we save with:</p><pre>await transfer.save(&#39;downloads://&#39; + name);</pre><p>TensorFlow.js writes two files, and our code writes a third sidecar file, so altogether three files are downloaded as below.</p><ol><li>&lt;name&gt;.json — model topology + weights manifest. This is the model “blueprint” (layer graph/config) and references to where the weight bytes live.</li><li>`&lt;name&gt;.weights.bin — raw learned weight tensors. This is the numeric parameter data used during inference and training continuation.</li><li>&lt;name&gt;.metadata.json — custom sidecar metadata written by your code. This contains `{ modelName, wordLabels }`, so the listener knows the exact vocabulary order expected by the classifier.</li></ol><p>A sidecar file is a small extra file that travels with the main model files. The .json + .weights.bin files contain neural-network structure and numbers, but app specific details (like our keywords start, pause, resume and _background_noise_) are better stored in a separate metadata file.</p><p>In the listener page, which is discussed next, tf.io.browserFiles([modelFile, weightsFile]) is used to tell TensorFlow.js how to load the model topology and weight tensors from user selected files. It does not know the app level vocabulary semantics unless they are provided. So in our listen flow, that vocabulary is wordLabels, so it is persisted in &lt;name&gt;.metadata.json and re attached after loading. Without this sidecar, the model weights may load correctly, but label-to-index mapping can be missing/wrong in our UI logic. That means a score index might not map to the intended spoken word.</p><pre>&quot;modelName&quot;: &quot;my-keywords&quot;,<br>&quot;wordLabels&quot;: [&quot;_background_noise_&quot;, &quot;start&quot;, &quot;pasue&quot;, &quot;resume&quot;]<br>}</pre><h4>Listener page: load and run inference</h4><p>Now we can test the trained model. On the listener page, all three downloaded files (.json,.weights.bin,.metadata.json) need to be uploaded and loaded like this:</p><pre>transfer = baseRecognizer.createTransfer(name);<br>await transfer.load(tf.io.browserFiles([modelFile, weightsFile]));<br>transfer.words = metadata.wordLabels;</pre><p>After that, clicking Listen button starts continuous microphone inference:</p><pre>await transfer.listen(callback, {<br>probabilityThreshold: 0.75,<br>invokeCallbackOnNoiseAndUnknown: false,<br>overlapFactor: 0.5,<br>suppressionTimeMillis: 1000,<br>});</pre><p>While transfer.listen(…) is active, the microphone stream stays open and the library continuously runs inference until stopListening() is called. Audio is captured as PCM (Pulse Code Modulation), which is the raw digital representation of the sound wave as amplitude samples over time. Instead of processing one long recording, the stream is split into short, fixed size sliding windows. Windowing is handled internally using overlap. With overlapFactor: 0.5, each new window overlaps the previous one by 50%, so the window slides forward in smaller steps. This means the user can speak at any time, and because windows overlap continuously, a spoken word is likely to be fully or mostly captured in at least one of the nearby windows. The Speech Commands frontend converts each window into time frequency features using spectrogram like BROWSER_FFT tensors, which are then fed into the transfer model. The model returns one score per label, including _background_noise_, and the callback receives result.scores, selects the highest score, and maps it to a label using transfer.wordLabels().</p><p>This setup also helps avoid bad timing windows. A single spoken keyword usually spans multiple overlapping windows, so even if one window catches only part of the word, a nearby overlapping window will often capture a better aligned version. Then probabilityThreshold filters out weak or uncertain predictions, invokeCallbackOnNoiseAndUnknown: false suppresses noise and unknown callbacks, and suppressionTimeMillis: 1000 adds a cooldown so one utterance does not trigger repeatedly. So in practice, there is no need to manually align chunks to exact word boundaries in the our code. The library handles the continuous sliding windows, while options like overlap factor, threshold, and suppression control sensitivity and stability.</p><h4>How to run and test this</h4><p>Before opening the pages, run them through an HTTP server (do not open the files directly with file://). The browser audio/model loading flow is much more reliable when served over http://localhost, and mic permissions also behave more consistently.</p><pre>cd vendor<br><br># Option A (Python)<br>python3 -m http.server 8000<br><br># Option B (Node)<br>npx serve .</pre><p>Suggested run sequence:</p><p>1. Open trainkeyword.html. <a href="http://localhost:8000/trainkeyword.html%60">http://localhost:8000/trainkeyword.html</a> (or the port printed by your server)</p><p>2. Record each keyword multiple times with balanced counts per label.</p><p>3. Record _background_noise_ samples (fan noise, keyboard, room tone, breathing, silence, etc.).</p><p>4. Click Train, then Save Model (this downloads &lt;name&gt;.json, &lt;name&gt;.weights.bin, and &lt;name&gt;.metadata.json).</p><p>5. Open keywordlistener.html. http://localhost:8000/keywordlistener.html</p><p>6. Select all three downloaded files and click Load Model.</p><p>7. Click Listen and test live detections.</p><p>If you see false positives, raise `probabilityThreshold` and add more noise samples. If detections are missed, lower the threshold slightly and add more pronunciation/speaking-distance variation for each keyword.</p><h4>Limitations and what to improve next</h4><p>This demo is not a speech to text system. It is designed for keyword spotting. Accent variation, microphone quality, room acoustics, and speaking distance can all affect accuracy. If you want to take this further, collect data from multiple speakers and in multiple environments. You can also add a simple command state machine so accidental repeats are ignored at the application logic level.</p><p>If rich transcription or multilingual sentence understanding is needed, then a larger speech to text pipeline may be a better fit, such as Whisper with Transformers.js running in a web worker. But for lightweight command and control in browser based applications, transfer learning with TensorFlow.js is usually the right level of complexity. That is the main focus of this article, because the goal was to use it in a WebXR setting.</p><p>If you need a complete Next.js example with App Router, Tailwind CSS, and TypeScript, check the <a href="https://github.com/madhawap/keywords-detector">GitHub repo</a>. I also added a WebXR example in that repo, which works well with the model I trained. Note that it was trained only on my voice, so it may not work reliably for the start, pause, and resume keywords (in general), so train your own and test. Otherwise, it still works with the 18 word Speech Commands base model (_unknown_, zero, one, two, three, four, five, six, seven, eight, nine, up, down, yes, no, go, stop, right).</p><p>Hope this article helps, and see you in another one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a81c19a3b0ac" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Peeking Behind LangChain ChatModels: Tracing the Actual HTTP Calls]]></title>
            <link>https://madhawacperera.medium.com/peeking-behind-langchain-chatmodels-tracing-the-actual-http-calls-742cb5367aaf?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/742cb5367aaf</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[langchain-chatmodels]]></category>
            <category><![CDATA[gemini-api]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 11:34:50 GMT</pubDate>
            <atom:updated>2026-04-12T11:43:44.849Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UEb64n6aq-kB8qGrDIjfcA.png" /></figure><p>In this article, the goal is to explore what actually happens behind the scenes when a LangChain chat model is invoked. In the previous article we used ChatGoogleGenerativeAI with a simple code example. In this article I will investigate that same code to explore how ChatGoogleGenerativeAI ended up calling Gemini APIs.</p><p>If you have read the <a href="https://medium.com/@madhawacperera/model-integration-for-agentic-ai-applications-an-overview-of-langchain-chatmodels-45923ba01b41">previous article on LangChain ChatModels</a>, you already know what problem they solve and why an AI engineer would choose to use them in Agentic AI applications. In a nutshell, LangChain ChatModels provide a easy to use abstraction over different model providers, letting you focus on prompts, tools, and workflows rather than low level integration details.</p><p>At the core of this ChatModel abstraction is the BaseChatModel interface (<a href="https://reference.langchain.com/python/langchain-core/language_models/chat_models/BaseChatModel">https://reference.langchain.com/python/langchain-core/language_models/chat_models/BaseChatModel</a>). All chat based integrations in LangChain, including OpenAI, Anthropic, Google, and Ollama, implement this interface. This standardisation lets higher level systems work with different providers without changing the main application logic. So if you host your own model, for example with vLLM, and want to use LangChain or LangGraph, you can implement your own ChatModel too.</p><p>Importantly, BaseChatModel does not know anything about a specific provider. It does not know how OpenAI, Google, or Ollama APIs work. Instead, it defines what a chat model should be able to do, not how it does it. A specific ChatModel, such as ChatGoogleGenerativeAI, ChatOpenAI, or ChatOllama, builds on top of this abstraction and adds provider specific functionality. That is why you often need to install packages such as langchain-ollama or langchain-google-genai to get that provider specific implementation of BaseChatModel.</p><p>On a high level, these specific implementations do three main things.</p><ol><li>They introduce the configuration parameters required by that provider, such as model name, temperature, token limits, timeout settings, retry logic, and authentication details.</li><li>They implement the translation layer between LangChain message objects and the provider’s API format. LangChain works with objects like SystemMessage and HumanMessage, but the provider expects its own request structure.</li><li>They expose provider specific capabilities. For example, Google’s chat models support multimodal inputs such as images, audio, and video, as well as tool calling and structured outputs. These capabilities are exposed through the concrete ChatModel implementation, not through BaseChatModel itself.</li></ol><p>In this article, let’s inspect ChatGoogleGenerativeAI a little more deeply, so we can get an idea of how the others work too.</p><p>The first thing you should know is that to use this ChatGoogleGenerativeAI implementation, you need to install the langchain-google-genai package. That is what pip install langchain-google-genai does. So if you really want to, you can go to somewhere like .venv (your virtual environment) /lib/python3.xx (your version) /site-packages/langchain_google_genai/chat_models.py, and this will show you how the ChatGoogleGenerativeAI implementation is done.</p><p>In a nutshell, ChatGoogleGenerativeAI from the langchain-google-genai package maps LangChain types such as messages, tools, and model settings into arguments for the google-genai client which is in Google Gen AI SDK (<a href="https://googleapis.github.io/python-genai/index.html">https://googleapis.github.io/python-genai/index.html</a>). In practice, it invokes Client.models.generate_content or generate_content_stream, along with the async equivalents, passing the model, contents, and a config object, along with related fields, as defined by the SDK. The google.genai.Client then serializes those into the HTTP request body for the generateContent REST method, which is JSON over HTTPS. The google-genai library performs the actual HTTP I/O using httpx through httpx.Client or httpx.AsyncClient.</p><p>A quick note on httpx[3]. httpx is a Python HTTP client library, similar in purpose to requests[4], but with support for both synchronous and asynchronous APIs, connection pooling, and importantly, event hooks, which I plan to use to see what the final request sent to the Gemini API looks like.</p><h4>So this is what really happens:</h4><p>LangChain ChatModel (ChatGoogleGenerativeAI) -&gt; Google Gen AI SDK -&gt; httpx -&gt; HTTP API (Gemini)</p><p>Now that this is understood, I was curious to see what the final message sent to the Google Gemini API actually looks like (not just in terms of the request format, which can of course be checked in the Gemini API docs), in terms of what transformations happen to a LangChain ChatModel prompt before it reaches that stage. To do this, my plan is to use event_hooks. event_hooks is an httpx feature and a valid constructor argument for httpx.Client. I can inject standard httpx options through the Google Gen AI SDK’s HttpOptions.client_args, which LangChain surfaces under the name client_args. So whatever httpx sends, such as the method, URL, headers, and body, we are going to inspect. Inspecting that layer with httpx request event hooks means observing the request after the SDK has fully constructed it, but before httpx sends it over the network. This lets us see the final outgoing HTTP request exactly as it is about to be sent to the provider. See below for the code that implements this plan. Note that it is the same code from the previous <a href="https://madhawacperera.medium.com/model-integration-for-agentic-ai-applications-an-overview-of-langchain-chatmodels-45923ba01b41">article</a>, but with modifications to pass the event hooks so the HTTP requests can be traced.</p><pre>from __future__ import annotations<br><br>import json<br>import sys<br>import httpx<br>from typing import Any<br>from dotenv import load_dotenv<br>from langchain.agents import create_agent<br>from langchain_core.tools import tool<br>from langchain_google_genai import ChatGoogleGenerativeAI<br><br>load_dotenv()<br><br>_LOCATIONS = {&quot;mug&quot;: &quot;in the level 42 room B112&quot;, &quot;book&quot;: &quot;in the study room&quot;}<br><br><br>@tool<br>def get_object_location(object_name: str) -&gt; str:<br>    &quot;&quot;&quot;Look up where an object is stored by name.&quot;&quot;&quot;<br>    return _LOCATIONS.get(object_name.lower().strip(), &quot;unknown&quot;)<br><br><br>def log_outgoing_request(request: httpx.Request) -&gt; None:<br>    &quot;&quot;&quot;httpx hook: print method, URL, headers, and body for every outgoing call.&quot;&quot;&quot;<br>    print(&quot;\n&quot; + &quot;=&quot; * 72)<br>    print(&quot;WIRE: outgoing HTTP request (google-genai / httpx)&quot;)<br>    print(&quot;=&quot; * 72)<br>    print(f&quot;{request.method} {request.url}\n&quot;)<br>    print(&quot;--- headers ---&quot;)<br>    for key, value in request.headers.items():<br>        print(f&quot;  {key}: {value}&quot;)<br>    raw = request.content<br>    print(&quot;\n--- body ---&quot;)<br>    if not raw:<br>        print(&quot;(empty body)&quot;)<br>        return<br>    try:<br>        payload: Any = json.loads(raw.decode())<br>        print(json.dumps(payload, indent=2, ensure_ascii=False))<br>    except json.JSONDecodeError:<br>        print(raw.decode(errors=&quot;replace&quot;))<br><br><br>def log_response_line(response: httpx.Response) -&gt; None:<br>    print(<br>        f&quot;\nWIRE: response {response.status_code} for {response.request.method} &quot;<br>        f&quot;{response.request.url}\n&quot;<br>    )<br><br><br>def main() -&gt; None:<br>    model = ChatGoogleGenerativeAI(<br>        model=&quot;gemini-3.1-pro-preview&quot;,<br>        client_args={<br>            &quot;event_hooks&quot;: {<br>                &quot;request&quot;: [log_outgoing_request],<br>                &quot;response&quot;: [log_response_line],<br>            },<br>        },<br>    )<br><br>    agent = create_agent(<br>        model,<br>        [get_object_location],<br>        system_prompt=(<br>            &quot;You help find where the user stored things. &quot;<br>            &quot;Never guess from general knowledge. Use the tools you have available to you.&quot;<br>        ),<br>    )<br><br>    try:<br>        result = agent.invoke({&quot;messages&quot;: [(&quot;human&quot;, &quot;where is my mug?&quot;)]})<br>    except Exception as e:<br>        print(f&quot;\nInvoke failed ({type(e).__name__}): {e}&quot;, file=sys.stderr)<br>        sys.exit(1)<br><br>    print(&quot;\n&quot; + &quot;=&quot; * 72)<br>    print(&quot;LangChain messages (agent result)&quot;)<br>    print(&quot;=&quot; * 72)<br>    for message in result[&quot;messages&quot;]:<br>        print(repr(message))<br>        print()<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>When I run this code, below is the output I get. <strong><em>N.B. This will expose your API keys, so be cautious with it</em></strong>. Let’s quickly walk through it.</p><pre>========================================================================<br>WIRE: outgoing HTTP request (google-genai / httpx)<br>========================================================================<br>POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent<br><br>--- headers ---<br>  host: generativelanguage.googleapis.com<br>  accept: */*<br>  accept-encoding: gzip, deflate, zstd<br>  connection: keep-alive<br>  content-type: application/json<br>  x-goog-api-key: xxxxxxxxxxxxxxxxxx<br>  user-agent: langchain-google-genai/4.2.1-ChatGoogleGenerativeAI, google-genai-sdk/1.70.0 gl-python/3.14.3<br>  x-goog-api-client: google-genai-sdk/1.70.0 gl-python/3.14.3<br>  content-length: 573<br><br>--- body ---<br>{<br>  &quot;contents&quot;: [<br>    {<br>      &quot;parts&quot;: [<br>        {<br>          &quot;text&quot;: &quot;where is my mug?&quot;<br>        }<br>      ],<br>      &quot;role&quot;: &quot;user&quot;<br>    }<br>  ],<br>  &quot;systemInstruction&quot;: {<br>    &quot;parts&quot;: [<br>      {<br>        &quot;text&quot;: &quot;You help find where the user stored things. Never guess from general knowledge. Use the tools you have available to you.&quot;<br>      }<br>    ]<br>  },<br>  &quot;safetySettings&quot;: [],<br>  &quot;tools&quot;: [<br>    {<br>      &quot;functionDeclarations&quot;: [<br>        {<br>          &quot;description&quot;: &quot;Look up where an object is stored by name.&quot;,<br>          &quot;name&quot;: &quot;get_object_location&quot;,<br>          &quot;parameters&quot;: {<br>            &quot;properties&quot;: {<br>              &quot;object_name&quot;: {<br>                &quot;type&quot;: &quot;STRING&quot;<br>              }<br>            },<br>            &quot;required&quot;: [<br>              &quot;object_name&quot;<br>            ],<br>            &quot;type&quot;: &quot;OBJECT&quot;<br>          }<br>        }<br>      ]<br>    }<br>  ],<br>  &quot;generationConfig&quot;: {<br>    &quot;temperature&quot;: 1.0,<br>    &quot;candidateCount&quot;: 1<br>  }<br>}<br><br>WIRE: response 200 for POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent<br><br><br>========================================================================<br>WIRE: outgoing HTTP request (google-genai / httpx)<br>========================================================================<br>POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent<br><br>--- headers ---<br>  host: generativelanguage.googleapis.com<br>  accept: */*<br>  accept-encoding: gzip, deflate, zstd<br>  connection: keep-alive<br>  content-type: application/json<br>  x-goog-api-key: xxxxxxxxxxxxxxxxxx<br>  user-agent: langchain-google-genai/4.2.1-ChatGoogleGenerativeAI, google-genai-sdk/1.70.0 gl-python/3.14.3<br>  x-goog-api-client: google-genai-sdk/1.70.0 gl-python/3.14.3<br>  content-length: 1467<br><br>--- body ---<br>{<br>  &quot;contents&quot;: [<br>    {<br>      &quot;parts&quot;: [<br>        {<br>          &quot;text&quot;: &quot;where is my mug?&quot;<br>        }<br>      ],<br>      &quot;role&quot;: &quot;user&quot;<br>    },<br>    {<br>      &quot;parts&quot;: [<br>        {<br>          &quot;functionCall&quot;: {<br>            &quot;args&quot;: {<br>              &quot;object_name&quot;: &quot;mug&quot;<br>            },<br>            &quot;name&quot;: &quot;get_object_location&quot;<br>          },<br>          &quot;thoughtSignature&quot;: &quot;Esw+8vuL3Gg8Fi77VPvS83fkd8duEbmk3bCbeBkOLGEt...&quot;<br>        }<br>      ],<br>      &quot;role&quot;: &quot;model&quot;<br>    },<br>    {<br>      &quot;parts&quot;: [<br>        {<br>          &quot;functionResponse&quot;: {<br>            &quot;name&quot;: &quot;get_object_location&quot;,<br>            &quot;response&quot;: {<br>              &quot;output&quot;: &quot;in the level 42 room B112&quot;<br>            }<br>          }<br>        }<br>      ],<br>      &quot;role&quot;: &quot;user&quot;<br>    }<br>  ],<br>  &quot;systemInstruction&quot;: {<br>    &quot;parts&quot;: [<br>      {<br>        &quot;text&quot;: &quot;You help find where the user stored things. Never guess from general knowledge. Use the tools you have available to you.&quot;<br>      }<br>    ]<br>  },<br>  &quot;safetySettings&quot;: [],<br>  &quot;tools&quot;: [<br>    {<br>      &quot;functionDeclarations&quot;: [<br>        {<br>          &quot;description&quot;: &quot;Look up where an object is stored by name.&quot;,<br>          &quot;name&quot;: &quot;get_object_location&quot;,<br>          &quot;parameters&quot;: {<br>            &quot;properties&quot;: {<br>              &quot;object_name&quot;: {<br>                &quot;type&quot;: &quot;STRING&quot;<br>              }<br>            },<br>            &quot;required&quot;: [<br>              &quot;object_name&quot;<br>            ],<br>            &quot;type&quot;: &quot;OBJECT&quot;<br>          }<br>        }<br>      ]<br>    }<br>  ],<br>  &quot;generationConfig&quot;: {<br>    &quot;temperature&quot;: 1.0,<br>    &quot;candidateCount&quot;: 1<br>  }<br>}<br><br>WIRE: response 200 for POST https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent<br><br><br>========================================================================<br>LangChain messages (agent result)<br>========================================================================<br>HumanMessage(content=&#39;where is my mug?&#39;, <br>additional_kwargs={}, <br>response_metadata={}, id=&#39;5b2a6e44-9b1a-4c0f-9ad1-xxxxxxxxx&#39;)<br><br>AIMessage(content=[], <br>additional_kwargs={&#39;function_call&#39;: {&#39;name&#39;: &#39;get_object_location&#39;, &#39;arguments&#39;: &#39;{&quot;object_name&quot;: &quot;mug&quot;}&#39;}, &#39;__gemini_function_call_thought_signatures__&#39;: {&#39;22b88d36-0007-0000-0000-a0000cd3a123&#39;: &#39;Esw+8vuL3Gg8Fi77VPvS83fkd8duEbmk3bCbeBkOLGEt...&#39;}}, <br>response_metadata={&#39;finish_reason&#39;: &#39;STOP&#39;, &#39;model_name&#39;: &#39;gemini-3.1-pro-preview&#39;, &#39;safety_ratings&#39;: [], &#39;model_provider&#39;: &#39;google_genai&#39;}, id=&#39;lc_run--019d7bb6-6df8-7780-9510-xxxxxxxxx&#39;, <br>tool_calls=[{&#39;name&#39;: &#39;get_object_location&#39;, &#39;args&#39;: {&#39;object_name&#39;: &#39;mug&#39;}, &#39;id&#39;: &#39;22b88d36-0007-0000-0000-xxxxxxxxx&#39;, &#39;type&#39;: &#39;tool_call&#39;}], <br>invalid_tool_calls=[], <br>usage_metadata={&#39;input_tokens&#39;: 83, &#39;output_tokens&#39;: 129, &#39;total_tokens&#39;: 212, &#39;input_token_details&#39;: {&#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;reasoning&#39;: 109}})<br><br>ToolMessage(content=&#39;in the level 42 room B112&#39;, <br>name=&#39;get_object_location&#39;, id=&#39;53bce6ce-fb35-4ad3-bb12-xxxxxxxxx&#39;, <br>tool_call_id=&#39;22b88d36-0007-0000-0000-a0000cd3a123&#39;)<br><br>AIMessage(content=[{&#39;type&#39;: &#39;text&#39;, &#39;text&#39;: &#39;Your mug is located in the level 42 room B112.&#39;, &#39;extras&#39;: {&#39;signature&#39;: &#39;EoC+10vsK+5C6E47+RGqlDxZqUWZVCabYSEwQ...&#39;}}], <br>additional_kwargs={}, <br>response_metadata={&#39;finish_reason&#39;: &#39;STOP&#39;, &#39;model_name&#39;: &#39;gemini-3.1-pro-preview&#39;, &#39;safety_ratings&#39;: [], &#39;model_provider&#39;: &#39;google_genai&#39;}, id=&#39;lc_run--019d7bb6-7cc2-7480-a823-xxxxxxxxxxxx&#39;, <br>tool_calls=[], <br>invalid_tool_calls=[], <br>usage_metadata={&#39;input_tokens&#39;: 238, &#39;output_tokens&#39;: 74, &#39;total_tokens&#39;: 312, &#39;input_token_details&#39;: {&#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;reasoning&#39;: 58}})</pre><p>The first call to the provider is a POST request to <a href="https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent">https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-pro-preview:generateContent</a>. The body contains the opening turn: contents includes a single user message, “where is my mug?”, along with systemInstruction, tools (the get_object_location function declaration and its JSON Schema parameters), safetySettings (which is empty, meaning Google’s server side default safety filters will be used), and generationConfig (e.g., temperature and candidateCount).</p><p>Now here we can see some configuration fields that ChatGoogleGenerativeAI exposes even though we did not explicitly set them. For example, safety related behaviour can be configured if needed via safetySettings. One interesting detail is that temperature is set to 1.0, which controls how random versus deterministic the model’s completions are. In this code, it was not set explicitly, but for Gemini 3+ the default appears to be 1.0 when left unset. Similarly, candidateCount, which specifies how many different full answers the model should generate for the same prompt in one API call, also appears to default to 1.</p><p>Now, it is important to note that this is the first request, so there is no model response included yet. At this stage, the model only sees the user question and the tool definitions.</p><p>We then see the 200 response. Although the response body is not printed here, and this was explained in the paragraph before the conclusion, this is the point where Gemini returns a functionCall for get_object_location with object_name: “mug”, along with internal thoughtSignature data in Gemini 3.x. How do we know that? We can infer it by looking at the second API call. There, we can see that LangChain has carried forward this Gemini 3.x internal thoughtSignature data into the next request. LangChain keeps this in additional_kwargs, which can also be seen in the final LangChain result messages.</p><p>The second POST request to …:generateContent is the next model turn after the agent has run the Python tool. The contents array is now a multi turn transcript in Gemini’s format: (1) the user question, (2) a model role entry containing a functionCall part, along with a thoughtSignature on that part, and (3) a user role entry containing a functionResponse part whose response field wraps the tool output, that is, the returned location string. The same systemInstruction, tools, safetySettings and generationConfig are sent again because each generateContent request is stateless at the HTTP level. This means <em>the full conversation so far must be sent again in contents with each new request</em>. So this seems additional token usage, and that can be observed if you inspect the usage_metadata in the LangChain messages: the first call used 83 input tokens, while the second used 238 input tokens. The model’s reply to this second call becomes the final natural language answer to the original query.</p><p>The LangChain block at the bottom is the agent’s structured result, not the raw HTTP requests and responses. These messages were discussed in the previous <a href="https://madhawacperera.medium.com/model-integration-for-agentic-ai-applications-an-overview-of-langchain-chatmodels-45923ba01b41">article</a>. HumanMessage contains the user’s text. The first AIMessage has empty text content but includes tool_calls, which is LangChain’s way of representing that the model decided to call a tool with specific arguments. ToolMessage contains the value returned by get_object_location, linked through the corresponding tool_call_id. The second AIMessage contains the model’s final answer, which corresponds to the second API round trip.</p><p>Finally, I want to discuss why an httpx response hook is usually a poor place to read response.text. The response hook runs when httpx already has the response status and headers, but the body may still be a stream. For many Gemini calls, httpx does not materialise response.text until the body is explicitly read using response.read() or await response.aread(). If .text is accessed before that, it can raise ResponseNotRead. The other issue is that if the hook itself calls read() or aread(), it consumes the response stream. The google-genai SDK code that runs afterwards may then find that the stream has already been read, which can break its normal processing. So request hooks are a reliable place to log the outgoing JSON because Request.content is already available, but response hooks are not ideal for logging full response bodies, especially for streaming or chunked responses.</p><p>I hope this was a fun investigation into what is actually happening behind the widely used LangChain ChatModel abstraction by looking at the specific ChatGoogleGenerativeAI implementation. You can do a similar investigation with something like ChatOllama as well, especially if you want to explore how a locally running model interacts with a LangChain ChatModel abstraction. What matters most is understanding this flow and deciding whether it fits the kind of agentic AI solution you want to build. Looking at it early helps you judge the suitability of the framework before going too far down the implementation path.</p><p>See you in another article.</p><h4>References</h4><ol><li><a href="https://reference.langchain.com/python/langchain-core/language_models/chat_models/BaseChatModel">https://reference.langchain.com/python/langchain-core/language_models/chat_models/BaseChatModel</a></li><li><a href="https://googleapis.github.io/python-genai/index.html">https://googleapis.github.io/python-genai/index.html</a></li><li><a href="https://www.python-httpx.org/">https://www.python-httpx.org/</a></li><li><a href="https://pypi.org/project/requests/">https://pypi.org/project/requests/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=742cb5367aaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Integration for Agentic AI Applications: An Overview of LangChain ChatModels]]></title>
            <link>https://madhawacperera.medium.com/model-integration-for-agentic-ai-applications-an-overview-of-langchain-chatmodels-45923ba01b41?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/45923ba01b41</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[llm-integration]]></category>
            <category><![CDATA[chat-model]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Tue, 07 Apr 2026 12:20:40 GMT</pubDate>
            <atom:updated>2026-04-12T11:35:51.550Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a0IAQ80mAffmeVBNs80qgg.png" /></figure><p>When developing agentic applications, one of the core architectural decisions is how to integrate and run a Large Language Model (LLM) or a Vision Language Model (VLM). Agents combine these models with external tools to create systems that can reason about tasks, decide which tools to use, and iteratively work towards a solution.</p><p>For example, if a framework such as <a href="https://docs.langchain.com/oss/python/langgraph/overview">LangGraph</a> is used for agentic AI solution development, it still needs a way to integrate and interface with an LLM or VLM within that solution. This is where <strong>model integration</strong> <strong>layers</strong> come into play.</p><p>There are several frameworks and libraries designed to simplify this integration of models into Agentic AI solutions, including <strong>LangChain</strong> (e.g.from langchain.llms import OpenAI or from langchain_openai import ChatOpenAI), <strong>LiteLLM</strong> (e.g.from litellm import completion), <strong>LlamaIndex</strong> (e.g. from llama_index.llms.openai import OpenAI), etc. There are also provider-specific SDKs such as the OpenAI (from openai import OpenAI), Anthropic Python clients (from anthropic import Anthropic), Google Gemini API ( from google import genai), etc. In this article, the focus is narrowed to one such abstraction within LangChain framework, namely <strong>ChatModels</strong>, and how they fit into agentic application development.</p><h4><strong>A brief note on model providers</strong></h4><p>In most cases, AI engineers building applications interact with foundation models through provider APIs rather than running the models directly themselves. Common examples include OpenAI, Anthropic, Google, Hugging Face, and Ollama, which expose LLMs and VLMs through managed interfaces. They are known as <strong>model providers</strong>. (That said, in some cases models are deployed through self-hosted serving systems such as <a href="https://vllm.ai/">vLLM</a>, <a href="https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html">NVIDIA Triton Inference Server</a>, or <a href="https://huggingface.co/docs/text-generation-inference/index">Hugging Face Text Generation Inference (TGI)</a>, <a href="https://docs.sglang.io/">SGLang</a>, etc., particularly when more control over deployment, privacy, and infrastructure is needed.)</p><p>Each model provider defines its own interface for interacting with models, and different providers often represent tool calls and structured outputs in different ways. For example, if a model indicates that a tool should be called, the engineers may need to detect that intent from the model output, parse the response, and trigger the appropriate function or tool, such as an <a href="https://modelcontextprotocol.io/docs/getting-started/intro">MCP</a> tool. However, doing so can lead to tightly coupled implementations tied to specific providers, require handling different request and response formats, and force engineers to build capabilities such as tool calling, structured outputs, retries, streaming, and batching themselves. This can be useful for learning and for gaining low-level control, but it also increases the likelihood of errors and adds ongoing maintenance overhead.</p><p>Thus, while model providers may offer powerful models and APIs that engineers can use in their applications, it is equally important to consider abstraction frameworks that help reduce integration complexity. This is why frameworks like LangChain, LiteLLM, and LlamaIndex are used in developing agentic solutions. These frameworks help reduce having to write provider specific code, improve portability across model backends, and support modular and maintainable system design.</p><p>Now let’s look at a wrapper provided by the LangChain framework, called <strong><em>ChatModel</em></strong>, which can be used to integrate LLMs or VLMs into agentic AI solution designs.</p><h4>What is LangChain’s Chatmodel</h4><p>LangChain ChatModels provide a standardised interface for interacting with chat-oriented LLMs and VLMs, meaning models designed to work with conversational message inputs such as system, user, and AI assistant messages. They sit between the application logic, such as an agent or workflow, and the underlying model provider.</p><p>Before ChatModels, LangChain mainly exposed text-in, text-out LLM interfaces. These are still available today, but for many modern use cases they are now considered legacy. At the most basic level, a model predicts the next token, so one simple way to interact with it is through a text completion interface: a prompt string is passed in, and a text string is returned. See the below example for instance.</p><pre>from langchain_ollama.llms import OllamaLLM<br><br>model = OllamaLLM(model=&quot;gemma3:latest&quot;)<br>response = model.invoke(&quot;What is the capital of France?&quot;)<br>print(repr(response))<br># Output:<br># &quot;The capital of France is **Paris**....&quot;</pre><p>In this interaction pattern, the model behaves like a stateless text generator. The developer sends a prompt and receives raw text in return. While LangChain abstracts away much of the provider-specific code here, this interface still leaves the developer responsible for building message structures, handling tool-calling logic, and managing conversational state. This approach is simple, but it is often not ideal for agentic AI applications. For this reason, current LangChain documentation generally recommends using <strong>ChatModels</strong> for modern model-driven systems (11,12). See an example of using the ChatOllama ChatModel with the same prompt as in the previous example that did not use a ChatModel.</p><pre>from langchain_ollama import ChatOllama<br><br>model = ChatOllama(model=&quot;gemma3:latest&quot;)<br>response = model.invoke(&quot;What is the capital of France?&quot;)<br>print(repr(response))<br># Output:<br># AIMessage(content=&quot;The capital of France is **Paris**...., <br># additional_kwargs={}, <br># response_metadata={&#39;model&#39;: &#39;gemma3:latest&#39;, &#39;created_at&#39;: &#39;2026-02-05T01:37:16.566796Z&#39;, &#39;done&#39;: True, &#39;done_reason&#39;: &#39;stop&#39;, &#39;total_duration&#39;: 3725356208, &#39;load_duration&#39;: 233014833, &#39;prompt_eval_count&#39;: 16, &#39;prompt_eval_duration&#39;: 2539705667, &#39;eval_count&#39;: 40, &#39;eval_duration&#39;: 919510329, &#39;logprobs&#39;: None, &#39;model_name&#39;: &#39;gemma3:latest&#39;, &#39;model_provider&#39;: &#39;ollama&#39;}<br># id=&#39;lc_run--019d6596-2887-7f13-b5c0-88cfe360265f-0&#39;, <br># tool_calls=[], <br># invalid_tool_calls=[], <br># usage_metadata={&#39;input_tokens&#39;: 16, &#39;output_tokens&#39;: 40, &#39;total_tokens&#39;: 56})</pre><p>As shown above, and pay attention to the output, a ChatModel returns a structured response object rather than plain text. In this case, the response is wrapped in an AIMessage object. In practice, ChatModels are usually used with these <strong>role-based messages</strong>. Instead of passing a single string, the system engineers provides a sequence of messages, where each message has a role such as system, or human like below.</p><p>Please note that, if preferred, you can explicitly construct messages using <strong>SystemMessage</strong> and <strong>HumanMessage</strong> from LangChain instead of using tuple-based inputs using from langchain_core.messages import SystemMessage, HumanMessage. This approach is useful when you want more explicit control over message types and is often preferred in more complex workflows or when working with advanced features such as tool calling and message manipulation.</p><pre>from langchain_ollama import ChatOllama<br><br>model = ChatOllama(model=&quot;gemma3:latest&quot;)<br><br>messages = [<br>    (&quot;system&quot;, &quot;You are a helpful assistant provide very short and concise answers.&quot;),<br>    (&quot;human&quot;, &quot;What is the capital of France?&quot;),<br>]<br><br>&quot;&quot;&quot;<br>OPTIONAL<br>from langchain_core.messages import SystemMessage, HumanMessage<br>messages = [<br>    SystemMessage(content=&quot;You are a helpful assistant provide very short and concise answers.&quot;),<br>    HumanMessage(content=&quot;What is the capital of France?&quot;)<br>]<br>&quot;&quot;&quot;<br><br>response = model.invoke(messages)<br>print(repr(response))<br># AIMessage(content=&#39;Paris.&#39;, <br># additional_kwargs={}, <br># response_metadata={&#39;model&#39;: &#39;gemma3:latest&#39;, &#39;created_at&#39;: &#39;2026-02-05T01:44:14.279555Z&#39;, &#39;done&#39;: True, &#39;done_reason&#39;: &#39;stop&#39;, &#39;total_duration&#39;: 514253584, &#39;load_duration&#39;: 219611125, &#39;prompt_eval_count&#39;: 33, &#39;prompt_eval_duration&#39;: 263162834, &#39;eval_count&#39;: 3, &#39;eval_duration&#39;: 28395667, &#39;logprobs&#39;: None, &#39;model_name&#39;: &#39;gemma3:latest&#39;, &#39;model_provider&#39;: &#39;ollama&#39;},<br># id=&#39;lc_run--019d659c-94c2-7a00-9356-95b9f97646d3-0&#39;, <br># tool_calls=[], <br># invalid_tool_calls=[], <br># usage_metadata={&#39;input_tokens&#39;: 33, &#39;output_tokens&#39;: 3, &#39;total_tokens&#39;: 36})</pre><p>So, one of the fundamental building blocks of <strong>ChatModels</strong> is this <strong>role-based</strong> <strong>message</strong>. These messages are used to represent the input and output of models and to maintain the conversational context passed to the model. Each message carries the content itself together with any additional metadata needed to represent the state of the interaction (See output in the example above).</p><p>A Message object usually contains three main parts:</p><ul><li>Role — identifies the message type, such as system, human, or assistant (AI)</li><li>Content — the actual message content, which may include text and, for some models, other data types such as images or audio</li><li>Metadata — optional information such as response details, message IDs, token usage, or tool call data</li></ul><p>LangChain provides standard message classes that work across providers, which helps keep model interaction consistent even when the backend model changes. Below the those message Message types:</p><ul><li><strong>SystemMessage</strong> — tells the model how to behave and provides high-level context.</li><li><strong>HumanMessage —</strong> represents user input.</li><li><strong>AIMessage</strong> — represents the model’s response and may include not only text but also tool calls and response metadata.</li><li><strong>ToolMessage —</strong> represents the result returned by a tool after it has been called.</li></ul><p>You have already seen examples of <strong>HumanMessage</strong> and <strong>SystemMessage</strong> above. Here is an example using a <strong>ToolMessage</strong> to help illustrate the concept. Do not worry about from langchain.agents import reate_agent, as that is outside the scope of this article. It is used here only to keep the example shorter. create_agent is another abstraction provided by LangChain and, as far as this article is concerned, it is enough to treat it simply as a helper. The purpose of this example is to show what a <strong>ToolMessage</strong> looks like and how a tool call is handled by the ChatGoogleGenerativeAI ChatModel.</p><pre>from dotenv import load_dotenv<br>from langchain.agents import create_agent<br>from langchain_core.tools import tool<br>from langchain_google_genai import ChatGoogleGenerativeAI<br><br>load_dotenv()<br><br>_LOCATIONS = {&quot;mug&quot;: &quot;in the level 42 room B112&quot;, &quot;book&quot;: &quot;in the study room&quot;}<br><br><br>@tool<br>def get_object_location(object_name: str) -&gt; str:<br>    &quot;&quot;&quot;Look up where an object is stored by name.&quot;&quot;&quot;<br>    return _LOCATIONS.get(object_name.lower().strip(), &quot;unknown&quot;)<br><br><br>model = ChatGoogleGenerativeAI(model=&quot;gemini-3.1-pro-preview&quot;)<br><br>agent = create_agent(<br>    model,<br>    [get_object_location],<br>    system_prompt=(<br>        &quot;You help find where the user stored things. &quot;<br>        &quot;Never guess from general knowledge. Use the tools you have available to you.&quot;<br>    ),<br>)<br><br>result = agent.invoke({&quot;messages&quot;: [(&quot;human&quot;, &quot;where is my mug?&quot;)]})<br><br>for message in result[&quot;messages&quot;]:<br>    print(repr(message))<br>    print()</pre><p>Example output from the above code:</p><pre>HumanMessage(<br>content=&#39;where is my mug?&#39;, <br>additional_kwargs={}, <br>response_metadata={}, id=&#39;6f030b72-d326-442e-87ea-xxxxxxx&#39;)<br><br>AIMessage(<br>content=[], <br>additional_kwargs={&#39;function_call&#39;: <br>{&#39;name&#39;: &#39;get_object_location&#39;, &#39;arguments&#39;: &#39;{&quot;object_name&quot;: &quot;mug&quot;}&#39;}, <br>&#39;__gemini_function_call_thought_signatures__&#39;: {&#39;b1595291-701f-43dd-9906-xxxxxxxx&#39;: &#39;ErQDCraSDIhsEDAXZA==&#39;}}, <br>response_metadata={&#39;finish_reason&#39;: &#39;STOP&#39;, &#39;model_name&#39;: &#39;gemini-3.1-pro-preview&#39;, &#39;safety_ratings&#39;: [], &#39;model_provider&#39;: &#39;google_genai&#39;}, <br>id=&#39;lc_run--019xxxxxxxx&#39;, <br>tool_calls=[{&#39;name&#39;: &#39;get_object_location&#39;, &#39;args&#39;: {&#39;object_name&#39;: &#39;mug&#39;}, &#39;id&#39;: &#39;b1595291-701f-43dd-9906-xxxxxxxx&#39;, &#39;type&#39;: &#39;tool_call&#39;}], <br>invalid_tool_calls=[], <br>usage_metadata={&#39;input_tokens&#39;: 83, &#39;output_tokens&#39;: 119, &#39;total_tokens&#39;: 202, &#39;input_token_details&#39;: {&#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;reasoning&#39;: 99}})<br><br>ToolMessage(<br>content=&#39;in the level 42 room B112&#39;, <br>name=&#39;get_object_location&#39;, <br>id=&#39;4c789e9f-5bee-4ca0-xxxxxxxx&#39;, <br>tool_call_id=&#39;b1595291-701f-43dd-9906-xxxxxxxx&#39;)<br><br>AIMessage(<br>content=[{&#39;type&#39;: &#39;text&#39;, &#39;text&#39;: &#39;Your mug is in the level 42 room B112.&#39;, <br>&#39;extras&#39;: {&#39;signature&#39;: &#39;EoYCCrQDCraoMH9g==&#39;}}], <br>additional_kwargs={}, <br>response_metadata={&#39;finish_reason&#39;: &#39;STOP&#39;, &#39;model_name&#39;: &#39;gemini-3.1-pro-preview&#39;, &#39;safety_ratings&#39;: [], &#39;model_provider&#39;: &#39;google_genai&#39;}, <br>id=&#39;lc_run--019d667b-6ad4-7953-8682-8066789c6953-0&#39;, <br>tool_calls=[], <br>invalid_tool_calls=[], <br>usage_metadata={&#39;input_tokens&#39;: 228, &#39;output_tokens&#39;: 71, &#39;total_tokens&#39;: 299, &#39;input_token_details&#39;: {&#39;cache_read&#39;: 0}, &#39;output_token_details&#39;: {&#39;reasoning&#39;: 56}})</pre><p>The message sequence above shows how a ChatModel (ChatGoogleGenerativeAI in this example) manages the flow of an interaction when tool calling is involved. It begins with a <strong>HumanMessage</strong> containing the user query, <em>“where is my mug?”</em>. The first <strong>AIMessage</strong> does not return a final text answer straight away. Instead, it identifies that a tool is needed and returns a structured tool call for get_object_location with the argument {“object_name”: “mug”}. Once that tool is executed, its result is wrapped as a <strong>ToolMessage</strong>, in this case returning <em>“in the level 42 room B112”</em>. The model then receives that tool output as part of the ongoing message history and produces a final <strong>AIMessage</strong> with the natural language response, <em>“Your mug is in the level 42 room B112.”</em> The key advantage here is that the ChatModel handles the structured message passing, tool call representation, and context forwarding in a consistent way. As an engineer, there is no need to manually parse raw model text to detect tool usage or to handcraft the conversational state between steps. This makes agent workflows cleaner, less error-prone, and easier to maintain across different providers.</p><p>Further, this structure aligns more closely with how modern models are exposed through provider APIs and how agentic systems are typically built. It also enables access to per-message data such as <em>token usage</em>, <em>reasoning details, and other metadata</em> when needed. In addition, conversation context can be managed more effectively, for example by summarising or trimming messages (e.g. keeping only the most recent <em>n</em> messages). So, ChatModels not only abstract provider-specific logic, but also provide a consistent message flow and make analytics, monitoring, and debugging easier.</p><p>ChatModels also allow developers to pass model-specific parameters such as temperature, maximum tokens, and other parameters in a consistent way across providers. While the exact parameters supported may vary depending on the model provider, LangChain provides a unified interface for configuring them. For example, when using ChatOllama, you can set parameters such as temperature or token limits directly when initialising the model.</p><pre>from langchain_ollama import ChatOllama<br><br>model = ChatOllama(<br>    model=&quot;gemma3:latest&quot;,<br>    temperature=0.2,<br>    # other params...<br>)<br><br>from langchain_google_genai import ChatGoogleGenerativeAI<br><br>model = ChatGoogleGenerativeAI(<br>    model=&quot;gemini-3.1-pro-preview&quot;,<br>    temperature=0.8,<br>    max_tokens=None,<br>    timeout=None,<br>    max_retries=2,<br>    # other params...<br>)</pre><p>It is important to note that ChatModels abstract over different providers, and LangChain offers many provider-specific ChatModel implementations. This article used two examples, <strong>ChatOllama</strong> and <strong>ChatGoogleGenerativeAI</strong>. At the core of this ChatModel abstraction is the BaseChatModel interface. Chat-based integrations across providers such as OpenAI, Google, Ollama, etc., implement this common interface, which is why LangChain provides separate integration packages such as langchain-ollama(<em>e.g. pip install -qU langchain-ollama</em>)<em>,</em> langchain-google-genai(<em>e.g. pip install -U langchain-google-genai</em>), langchain-openai<em> (e.g. pip install -U langchain-openai</em>) for use in your dev environment.</p><p>LangChain has <a href="https://docs.langchain.com/oss/python/integrations/chat">a collection of ChatModel integrations</a> so developers can choose the provider that best fits their needs. However, it is also important to note that while ChatModels provide a unified interface, <em>not all features are supported uniformly across all providers or their LangChain integrations.</em> Some capabilities depend on the underlying model or API, and in certain cases may not yet be exposed through the specific ChatModel implementation. Therefore, it is recommended to refer to both the provider’s documentation and the corresponding LangChain integration documentation for the specific ChatModel you plan to use, in order to understand which features, parameters, and modalities are supported for a given model or backend.</p><p>So I encourage you to read further in the LangChain documentation, but hopefully this gives a high-level understanding of what a ChatModel is.</p><p>A common concern when using these abstractions is understanding what happens under the hood. For example, how user messages are formatted before they are sent, how is the content prepared for the model, and how much control does an engineer still have over customisation? To answer these questions, we would need to look more closely at ChatModels and explore a specific implementation (e.g. ChatOllama). However, this article is already getting longer, so if you would like to explore this further, you can read <a href="https://madhawacperera.medium.com/peeking-behind-langchain-chatmodels-tracing-the-actual-http-calls-742cb5367aaf?postPublishedType=initial"><strong><em>Explore ChatModel Abstraction</em></strong></a>, where I look at what happens inside a ChatModel, how a request is prepared, and how it is sent to the provider.</p><p>To conclude, the main takeaway from this article is that there are many ways to integrate an LLM or VLM into an agentic AI solution. In some cases, this can be done from the ground up by self-hosting models and writing custom integration code where needed. However, in both prototyping and production contexts, there are many frameworks that handle much of the model integration work. This allows AI engineers to focus more on the solution itself and the application logic they are building. One widely used framework is LangChain, and in this article we looked at its <strong>ChatModel</strong> abstraction, which helps integrate models exposed by many different providers.</p><p>References</p><ol><li><a href="https://docs.langchain.com/oss/python/integrations/providers/all_providers">https://docs.langchain.com/oss/python/integrations/providers/all_providers</a></li><li><a href="https://docs.langchain.com/oss/python/integrations/chat">https://docs.langchain.com/oss/python/integrations/chat</a></li><li><a href="https://docs.langchain.com/oss/python/langchain/messages">https://docs.langchain.com/oss/python/langchain/messages</a></li><li><a href="https://docs.langchain.com/oss/python/integrations/chat/ollama">https://docs.langchain.com/oss/python/integrations/chat/ollama</a></li><li><a href="https://docs.langchain.com/oss/python/integrations/chat/google_generative_ai">https://docs.langchain.com/oss/python/integrations/chat/google_generative_ai</a></li><li><a href="https://developers.llamaindex.ai/python/framework/integrations/llm/ollama/">https://developers.llamaindex.ai/python/framework/integrations/llm/ollama/</a></li><li><a href="https://docs.litellm.ai/docs/providers/gemini">https://docs.litellm.ai/docs/providers/gemini</a></li><li><a href="https://developers.openai.com/api/docs/quickstart?language=python">https://developers.openai.com/api/docs/quickstart?language=python</a></li><li><a href="https://modelcontextprotocol.io/docs/getting-started/intro">https://modelcontextprotocol.io/docs/getting-started/intro</a></li><li><a href="https://docs.langchain.com/oss/python/langgraph/overview">https://docs.langchain.com/oss/python/langgraph/overview</a></li><li><a href="https://docs.langchain.com/oss/python/integrations/llms/openai">https://docs.langchain.com/oss/python/integrations/llms/openai</a></li><li><a href="https://docs.langchain.com/oss/python/integrations/llms/ollama">https://docs.langchain.com/oss/python/integrations/llms/ollama</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45923ba01b41" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git Rebase in a Nutshell]]></title>
            <link>https://madhawacperera.medium.com/githacks-lets-talk-git-rebase-0233b0aa254b?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/0233b0aa254b</guid>
            <category><![CDATA[git-rebase]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[githack]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 05:57:05 GMT</pubDate>
            <atom:updated>2026-03-02T09:06:39.817Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x6xzazkAHstJlWncdOO3DQ.png" /></figure><p>Let’s look into the Git rebase in this article. If you’ve ever wondered how to keep your branch history clean and your commits in tip-top shape, this one’s for you!</p><p><strong><em>Note:</em></strong> <em>I aim to keep this article short for a quick read. If you’re already familiar with Git, this could be a refresher. And if you notice anything missing or unclear, please leave a comment so I can take a look!</em></p><h4>Scenario: your feature branch is behind main:</h4><p>Imagine you’re working on a feature branch called dev/new-feature, which is based off the main branch. You created the dev/new-feature branch after commits A and B were made to main. Meanwhile, the main branch has moved forward with some new changes (commits C and D), and you want to incorporate those updates into your dev/new-feature branch without losing your own work. Here’s where Git rebase become handy.</p><h4>But wait, why not just do a simple git pull from main?</h4><p>You might wonder, why not simply run git pull origin main to get the latest changes? Well, here are a few things that could happen:</p><ul><li><strong>Merge Commits:</strong> A simple git pull performs a git fetch followed by a git merge. This means if there are new changes in main, Git will create a new merge commit to combine your branch with main. This can clutter your history with extra commits that might not add significant value, making it harder to read.</li><li><strong>Conflict Resolution:</strong> If there are conflicting changes between your dev/new-feature branch and main, a merge will also prompt you to resolve those conflicts. While you can resolve conflicts during a merge, doing so may leave a messier commit history.</li><li><strong>Branch Divergence:</strong> If you frequently merge changes from main into your feature branch, you risk creating a complex commit history with lots of branches and merge commits, making it difficult to trace the evolution of your code over time.</li></ul><p>In contrast, using git rebase allows you to apply your changes on top of the latest commits from main in a clean, linear fashion. This way, your history remains tidy, and you incorporate all necessary updates without extra merge commits.</p><h4>So how to do git rebase?</h4><p>First, let’s fetch the latest changes from the remote repository:</p><pre>git fetch origin</pre><p>Now, let’s dev/new-feature onto main :</p><pre>git rebase origin/main</pre><p>If there are no conflicts (hurrah!), Git will replay your commits on top of the latest changes from main. However, if conflicts arise, Git will pause the process and highlight the conflicting files. If so open the conflicting files, resolve the conflicts, and then add them to the staging area using below command.</p><pre>git add &lt;conflicted-file&gt;</pre><p>After resolving all conflicts, continue with the rebase:</p><pre>git rebase --continue</pre><p>Finally, push the rebased changes to your remote branch:</p><pre>git push origin dev/new-feature --force-with-lease</pre><p><strong><em>Note: </em></strong>Using --force-with-lease is recommended because it protects against accidentally overwriting someone else’s work. It checks that the remote branch has not changed since you last pulled or fetched. If someone else has pushed changes, the push will be rejected, allowing you to pull the latest changes and address any conflicts.</p><p>If you’re rebasing and the remote branch hasn’t changed since your last fetch, you can also use a regular push:</p><pre>git push origin dev/new-feature</pre><p>Again, just be cautious, as this will fail if the remote branch has new commits since your last fetch.</p><h4>Finally, why not use git cherry-pick?</h4><p>You might wonder if git cherry-pick could do the job here. While cherry-pick lets you pull specific commits from main, it can also potentially clutter your history and create more complex merges later. Since you want to incorporate <strong>all changes</strong> from main, rebase is the better option here. It helps to maintain a clean linear history.</p><p>And there you have it! Git rebase helps keep your commit history linear and clean. Remember, practice makes perfect, so feel free to experiment in your own Git sandbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0233b0aa254b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Git Hack: Move a Single Commit Across Branches]]></title>
            <link>https://madhawacperera.medium.com/githacks-a-way-to-grab-changes-from-another-branch-7073dbc218a5?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7073dbc218a5</guid>
            <category><![CDATA[version-control]]></category>
            <category><![CDATA[cherrypick]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[githack]]></category>
            <category><![CDATA[git-cherry-pick]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 04:55:19 GMT</pubDate>
            <atom:updated>2026-04-12T11:44:50.218Z</atom:updated>
            <content:encoded><![CDATA[<p>Sometimes you may want to move commits from one branch to another. There is a super handy trick for that. Say you made a useful commit on one Git branch but wish you could just zap it onto another branch. Well, you can. There is no need to manually copy your changes. Git’s cherry pick feature has you covered.</p><p><em>I know there are a ton of tools, articles, and endless info out there about using Git to make your life easier. Even recently stumbled across </em><a href="https://gitbutler.com/"><em>GitButler</em></a><em> (a cool Git GUI tool), which is still in its early beta if you want to check it out. But this is for someone who still likes using the terminal. ;)</em></p><p>This is a quick guide to grab specific commits from one branch and drop them into another without all the mess of merging the entire branch.</p><h4><strong><em>Scenario</em></strong>:</h4><p>You just created a new branch called dev/ios-speech from main on your local machine, and now you want to bring in some commits from the remote dev/android-speech branch. Since dev/android-speech hasn’t been merged into main yet, you are going to need a quick way to pull specific commits over without the entire branch.</p><h4>Step 1: Switch to Your New Branch</h4><p>You’ve already created dev/ios-speech, so just make sure you’re on that branch:</p><pre>git checkout dev/ios-speech</pre><h4>Step 2: Fetch the dev/android-speech Branch (But Not All the Other Branches)</h4><p>You don’t need to pull everything from the remote, just the branch you need:</p><pre>git fetch origin dev/android-speech:dev/android-speech</pre><p>This grabs the dev/android-speech branch and creates a local copy of it.</p><h4>Step 3: Cherry-Pick the Commits You Want</h4><p>Now that you have the dev/android-speech branch locally, you can pick the specific commits you need. First, check out the commit history of dev/android-speech:</p><pre>git log dev/android-speech</pre><p>Find the commit hashes you want (e.g., a1b2c3d) and cherry-pick them onto your current branch:</p><pre>git cherry-pick a1b2c3d</pre><p>If you want multiple commits, just run the cherry-pick command for each one:</p><pre>git cherry-pick e4f5g6h</pre><p><strong>Need to undo a cherry-pick?</strong> No worries! If something went wrong, you can simply undo the last cherry-pick using:</p><pre>git cherry-pick --abort  # If you&#39;re still in a merge conflict</pre><p>Or, if the cherry-pick was applied but you want to remove it:</p><pre>git revert &lt;commit_hash&gt;</pre><p>This creates a new commit that undoes the changes from the cherry-picked commit.</p><h4>Step 4: Push Your Changes</h4><p>After successfully adding those commits, it is time to push your updated dev/ios-speech branch to the remote:</p><pre>git push origin dev/ios-speech</pre><h4>And that’s it!</h4><p>You have successfully grabbed commits from dev/andriod-speech branch. This way, you get just what you need without merging the entire dev/android-speech branch into your shiny new dev/ios-speech branch. Easy peasy! Plus, you now know how to backtrack if needed, because we all make mistakes 🎉</p><p>Let me know if you run into any issues, spot any missing pieces, or if you found better ways to improve this flow. After all, we are all constantly learning, right?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7073dbc218a5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building HoloLens 2 QR Code Tracking App with MRTK3 and Unity]]></title>
            <link>https://medium.com/ar-vr-journey/building-hololens-2-qr-code-tracking-app-with-mrtk3-and-unity-24c2cc806648?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c2cc806648</guid>
            <category><![CDATA[mrtk]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[qr-code]]></category>
            <category><![CDATA[hololens-2]]></category>
            <category><![CDATA[mrtk3]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 04:08:09 GMT</pubDate>
            <atom:updated>2024-01-03T04:50:38.202Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s been some time since I wrote a ‘how-to’ article. Recently, one of the interns on our team wanted to know how to use HoloLens 2 to track QR codes, so I thought of compiling a short article to share with everyone who wants this functionality in their mixed reality/augmented reality projects.</p><p>So let’s get started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gk5QfPlOunwdWiRR_clA_Q.gif" /><figcaption>Figure 1: QR code tracking with HoloLens 2.</figcaption></figure><h4>Prerequisites</h4><ul><li>Unity 2022.3.XX LTS (I used Unity 2022.3.8f1 LTS)</li><li>Your preferred C# IDE (VS 2022 or JetBrains Rider 2023)</li><li>Mixed Reality Toolkit 3 (MRTK3)</li><li>Microsoft HoloLens 2</li><li>Mixed Reality Feature Tool (MRFL) — Download from <a href="https://www.microsoft.com/en-us/download/details.aspx?id=102778">here</a>.</li></ul><p>Start by creating a new project with Unity Hub using the 3D template. Then, set the platform to <em>Universal Windows Platform</em> by navigating to <strong>File -&gt; Build Settings…</strong> Refer to Figure 2 below for guidance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*kIRNgeAxTC6-4FbObCpeRA.png" /><figcaption>Figure 2: Unity Build Settings</figcaption></figure><p>Alright, now that we have Unity ready, let’s proceed to install MRTK3. Note that MRTK3 has a new architecture compared to MRTK v2, which was a bit monolithic. MRTK3 is more modular (See Figure 3.2), allowing us to install only the features that are required for development independently and update them separately. Please refer to Figure 3.1 below for illustration of high-level MRTK3 architecture. For more details, visit the Microsoft documentation <a href="http://Mixed Reality Toolkit 3">here</a>. <em>Keep in mind that the steps discussed in this article may not work with MRTK v2 due to these changes in architecture</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s9qQH1_iTns1TDvv.png" /><figcaption>Figure 3.1: Reference -&gt; <a href="https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/architecture/architecture">https://learn.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk3-overview/architecture/architecture</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*GWMZ2FC1cmxlZ-pKUuqx6w.png" /><figcaption>Figure 3.2 MRTK v2 and MRTK3 feature based packages</figcaption></figure><p>Now, close Unity and open the <a href="https://www.microsoft.com/en-us/download/details.aspx?id=102778">Mixed Reality Feature Tool (MRFL)</a> and indicate the location of your Unity project. One thing to note is that when installing the required packages for MRTK3, I observed that attempting to select and install all of the required features together caused MRFL to crash. This might be a bug that could have been addressed in subsequent updates.</p><p>Under the “Discover Features” section in MRFL, expand the “MRTK3 (1 of 15)” collapsible list. Tick the following options: <em>MRTK Graphics Tools</em>, <em>MRTK UX Components</em>, and <em>MRTK UX Core Scripts</em>. Additionally, under “Platform Support (1 of 5)”, select the Mixed Reality OpenXR plugin (v1.10.0). See Figure 4 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x59nlhPz0fcvtBsAy95ZnA.png" /><figcaption>Figure 4: Mixed Reality Feature Tool (MRFL) feature selection</figcaption></figure><p>Once you have selected these features, click on “Get Features” button (See Figure 4) and follow the GUI instructions to install them within your Unity Project. Upon reopening the Unity project, you may get several prompts, and it might ask you to restart Unity. If prompted, please proceed with the restart. Then, validate the Unity project to ensure that all the necessary configurations are correctly set, as shown in Figure 5 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*7MnyXgoOVrOIUKlngebfAw.png" /><figcaption>Figure 5: MRTK3 project validation</figcaption></figure><p>Ensure to add <strong>Webcam </strong>capability to the Manifest. Navigate to <strong>Edit &gt; Project Settings &gt; Player</strong> window. Select the ‘Universal Windows Platform Settings’ tab (With Windows icon), expand the ‘Publishing Settings’ section, and locate <strong>Webcam </strong>the Capabilities list. Refer to Figure 6 below for guidance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*tNFe0VKzmOQm5Xgji_QGxg.png" /><figcaption>Figure 6</figcaption></figure><p>Done with the configurations. Now, I have created an empty object called <em>QRCodeReading</em> in the hierarchy and attached three scripts (components) to it. Two of them come with MRTK3, and one is a custom script. Look for the <em>XROrigin </em>and <em>ARMarkerManager</em> scripts, which come with the Mixed Reality OpenXR plugin package. To do this, go to the project window and use the search bar to type the script name, then change the search scope to “All” or “In Packages,” as shown in Figure 7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*Az3sV5klayzGujlCmu_1Uw.png" /><figcaption>Figure 7</figcaption></figure><p>Perform the same steps for the <em>XROrigin </em>script. Then, drag and drop both the <em>XROrigin </em>and <em>ARMarkerManager </em>scripts onto the <em>QRCodeReading </em>game object. Afterward, create a custom script — I named it <em>TestQRCodeDetection</em> (feel free to choose your own name). Refer to the <em>TestQRCodeDetection</em> code provided <a href="https://github.com/madhawap/HLQRTracking/blob/main/Assets/Scripts/TestQRCodeDetection.cs">here</a>. I have added comments in the code for clarity, so I won’t describe it here to keep this article concise and easy to follow. Add this script to the <em>QRCodeReading </em>game object as well.</p><p>Next, create another empty game object and name it <em>QRCodeMarker</em>. Again, search for the <em>ARMarker </em>script as described above (similar to how we looked for the <em>ARMarkerManager </em>and <em>XROrigin </em>scripts). Add the <em>ARMarker</em> script to the <em>QRCodeMarker </em>game object. Drag and drop it into the Project window to make it a Prefab. For best practices, it is advisable to create a Prefab folder inside the Assets folder and then drag the <em>QRCodeMarker </em>into that folder. Once you’ve done that, you can delete the <em>QRCodeMarker </em>from the Hierarchy window.</p><p>Finally, I added a <em>CanvasDialog </em>to display the contents of the QRCode (See Figure 8.1 and 8.2). This step is optional, as you may prefer not to visually show the QRCode contents and use other means, such as the Debug Log, to inspect the QR code contents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*Cdvah12TDajwA2ImIZuy0Q.png" /><figcaption>Figure 8.1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YwdjyU8Ni5xp6IhAHnzLbA.png" /><figcaption>Figure 8.2</figcaption></figure><p>Alright we are almost there. Few more things to do before we test. Drag and drop Main Camera gameobject to Camera feild in XROrigin, Add QRCodeReading gameobject to Origin Base GameObject field in XROrigin and Marker Manager field in TEstQRCodeDetection script. Then drag and drop the prefab we created above (named QRCodeMarker) to the Maker Prefab field in ARMarkerManager script and that’s it. In case I have missed something please see my configurations in the below screenshot.</p><p>Alright, we are almost there. A few more things to do before we test this HL2 app. Drag and drop the <em>Main Camera</em> game object to the ‘<em>Camera’ </em>field in <em>XR Origin </em>component. Add the <em>QRCodeReading </em>game object to the ‘<em>Origin Base GameObject’</em> field in ‘<em>XR Origin</em>’ component and the ‘<em>Marker Manager’</em> field in the ‘<em>Test QR Code Detection’</em> component. Then, drag and drop the prefab we created earlier (named <em>QRCodeMarker</em>) to the ‘<em>Marker Prefab’</em> field in the ‘<em>AR Marker Manager</em>’ component, and that’s it. In case I have missed something, please refer to my configurations in the screenshot in Figure 9 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*OwbSZkoj0Qh_Ogd5QUdgvw.png" /><figcaption>Figure 9</figcaption></figure><p>Now we are ready to test this on HoloLens 2. The good news is that we don’t need to build the app to test this. With the Mixed Reality OpenXR Plugin v1.10.0, QR code detection works with <em>Holographic Remoting</em> as well. If you need a QR code to test, you can use the one provided below. Alternatively, you can use an online tool (e.g. <a href="https://www.the-qrcode-generator.com/">qrcode-generator</a>) to generate your own QR code.</p><p>If you want to try out the full project, you can find it on my <a href="https://github.com/madhawap/HLQRTracking">GitHub</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*g8ow1DRPaEvPmSUVXzvDJw.png" /><figcaption>Figure 10</figcaption></figure><p>As always, please let me know if I have missed anything, if the steps described are clear, or if I have over complicated something (i.e., if there is a simpler way to achieve this). I wish you the best of luck with your amazing mixed reality/augmented reality projects, and I hope this short guide proves helpful.</p><p>If you enjoyed this article and found it helpful, you can buy me a virtual coffee ☕️😊;</p><figure><a href="https://www.buymeacoffee.com/madhawa"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Dpw8-hNGI2fDmosV4E8DVQ.png" /></a></figure><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c2cc806648" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ar-vr-journey/building-hololens-2-qr-code-tracking-app-with-mrtk3-and-unity-24c2cc806648">Building HoloLens 2 QR Code Tracking App with MRTK3 and Unity</a> was originally published in <a href="https://medium.com/ar-vr-journey">AR/VR Journey: Augmented &amp; Virtual Reality Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up custom Ubuntu version with Windows Subsystem for Linux (WSL)]]></title>
            <link>https://madhawacperera.medium.com/setting-up-custom-ubuntu-version-with-windows-subsystem-for-linux-wsl-c760a0d91635?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c760a0d91635</guid>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[wsl-2]]></category>
            <category><![CDATA[wsl]]></category>
            <category><![CDATA[windows-subsystem-linux]]></category>
            <category><![CDATA[wsl-setup]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 01:36:37 GMT</pubDate>
            <atom:updated>2024-01-15T03:26:32.534Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, I got a new Windows machine, and I wanted to set up WSL (Windows Subsystem for Linux). I realized it would be helpful to create a post serving as both a personal note for myself and a simple guide for others who might be interested.</p><p>So to begin the setup process, the first thing I did was open PowerShell (PS) as an administrator (See Figure 1 below). Well, of course, I made a few minor changes to the terminal’s appearance :).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*gkDlHVpLmwviqicjigTR5w.gif" /><figcaption>Figure 1: Windows PowerShell</figcaption></figure><p>I wanted to install WSL without any default Linux distributions that come with WSL. To do this, you should use the command wsl --install --no-distribution. (You can use wsl --install if you are happy with the default Linux distribution that comes with <a href="https://apps.microsoft.com/store/detail/windows-subsystem-for-linux/9P9TQF7MRM4R?hl=en-us&amp;gl=us">Microsoft Store Version</a>). Once you do that, you will get a similar message as shown in Figure 2 below. Restart your computer to make the installation effective. You can type the Restart-Computer command in PowerShell for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*v1KSUzoTQ0Qoqcr3MsSasA.gif" /><figcaption>Figure 2: Installation of WSL</figcaption></figure><p>Once you restart you can use the commands wsl --list or wsl --list verbose to check the Linux distribution that has been installed. However, since we decided not to use the default distribution, you should see the message “<em>Windows Subsystem for Linux has no installed distributions.</em>” To view the available Linux distributions that can be installed, simply use wsl --list --online. This will display a list of all the available distributions that you can install (refer to Figure 3 below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*J8yAmIRwGKXD-6G8LXeyEw.gif" /><figcaption>Figure 3: WSL Distribution List</figcaption></figure><p>Next, we can install the latest Ubuntu distribution. To do that, first, download the distribution you want to install. In my case, I wanted to install Ubuntu 22.04.2 (Jammy Jellyfish). As shown in below, you can use the Invoke-WebRequest command in PowerShell to download (Make sure to give a proper download file path). Alternatively, you can copy and paste the URI on the browser or use the curl command to download Ubuntu using the command curl.exe -O <a href="https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz">https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz</a>. The “-O” in the curl command here will save the file in the current working directory with the same file name as the remote file. On the other hand, “-o” allows you to specify a different file name or location. Choose the argument that suits your needs. Please note that curl will not work on Windows PS by default (though it works on Windows Command line). You have to use curl.exe with PS. In my case I used Invoke-WebRequest as below.</p><pre>Invoke-WebRequest -URI https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-root.tar.xz -OutFile /filepath/jammy-server-cloudimg-amd64-root.tar.xz</pre><p>Now that we have the distribution we want, we can start installing this on WSL. However, before doing that, I wanted to check which WSL version I am running. To do that, simply execute the command wsl -v or wsl --version. I found that I was on WSL 1, but I want to install Ubuntu Jammy Jellyfish with WSL 2. To do so, firstly go to the Ubuntu distribution download location, and then execute the following command as shown in Figure 4 below. Don’t forget the--version 2 bit at the end of that command. Here, for &lt;distribution_name&gt;, you can provide any name you like, e.g., Ubuntu22.04, and for &lt;path_to_downloaded_Ubuntu_Image&gt;, use the path to the downloaded Ubuntu image. See the below command.</p><pre>wsl --import &lt;distribution_name&gt; &lt;path_to_downloaded_Ubuntu_Image&gt; jammy-server-cloudimg-amd64-root.tar.xz --version 2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nl08rSd5N2_keh4EnEwXJQ.gif" /><figcaption>Figure 4: Installing WSL distribution</figcaption></figure><p>Once you see the message “<em>The operation completed successfully</em>” use the command wsl -l -v to see the installed distribution and corresponding WSL version. See Figure 5 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*0S_AbJZas1y23b1qy3u8qQ.gif" /><figcaption>Figure 5</figcaption></figure><p>Next, run wsl -d &lt;distribution_name&gt; or wsl --distribution &lt;distribution_name&gt; (e.g. wsl -d Ubuntu22.04 in my case) to enter the Ubuntu subsystem. Once you are inside the Ubuntu subsystem, you can execute lsb_release -a to see which version of Ubuntu is running, as shown in Figure 6 below. Here, ‘lsb’ stands for Linux Standard Base. I am not sure why I am seeing the “<em>No LSB modules are available</em>” message; however, I found online that running sudo apt install lsb-core may fix this. Please let me know if you know why this message was printed and if sudo apt install lab-core will fix this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*4sXt9IDzxdasFE0QV7yNeQ.gif" /><figcaption>Figure 6</figcaption></figure><p>One thing to notice is that you have logged in as the root user here. To test this, you can use the whoami command on the Linux terminal. However, if you want to create a new Linux user and grant them sudo privileges, you can use the following commands within the Ubuntu subsystem. However, this is an optional task.</p><p>To create a user in Linux, you can use the adduser &lt;username&gt; or useradd &lt;username&gt; command. Using adduser is recommended for most Linux distributions. Replace &lt;username&gt; with the desired username of the new user. The adduser command will prompt you to set a password and provide other user information interactively. Then, add the created user to the sudo group using the command sudo usermod -aG sudo &lt;username&gt;. After that, check if the user is now part of the sudo group by using groups &lt;username&gt; These steps will create a new user and grant them sudo privileges on the Linux system. See Figure 7 where I have demonstrated these steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*9_sVOIyoPLWSzL3gQqTrbw.gif" /><figcaption>Figure 7</figcaption></figure><p>One last thing you may want to do is to make this created user the default user for the WSL distribution you created. In other words, instead of logging in as the root user every time and then switching to this created user, you can set the created user as the default user for the distribution, so it automatically logs in with that user by default (and will prompt for the password). To do this within the Linux environment, you can use the following commands. Firstly use sudo to add the &quot;[user]&quot; line to the /etc/wsl.conf file by using sudo echo “[user]” &gt;&gt; /etc/wsl.conf. Again use sudo to add the &quot;default=&lt;username&gt;&quot; line to the /etc/wsl.conf file with the command sudo echo “default=&lt;username&gt;” &gt;&gt; /etc/wsl.conf. Replace &lt;username&gt; with the name of the created user. Then exit from the Linux system using the command exit, terminate the WSL distribution using the wsl -t &lt;distribution_name&gt; command. Finally log in again, and this time you will be logged in with the created user as the default. Refer to Figure 8 below for a demonstration of these steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*bUOB1rCcTVK-7RyN0-TX4g.gif" /><figcaption>Figure 8</figcaption></figure><p>That is it. So, we have managed to set up WSL on Windows with a custom Linux image. Optionally, we discussed how to create a new user for that distribution and make them the default user.</p><p>If this article helps you and want to thank me, you can buy me a virtual coffee by clicking the ‘Buy me a coffee’ button below. :)</p><figure><a href="https://www.buymeacoffee.com/madhawa"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Dpw8-hNGI2fDmosV4E8DVQ.png" /></a></figure><p>Thanks for the read and I will see you in another article soon. Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c760a0d91635" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HoloLens 2 Marker Tracking with Vuforia Engine and MRTK]]></title>
            <link>https://medium.com/ar-vr-journey/hololens-2-marker-tracking-with-vuforia-engine-and-mrtk-fb582c8f8ac0?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb582c8f8ac0</guid>
            <category><![CDATA[marker-tracking]]></category>
            <category><![CDATA[vuforia]]></category>
            <category><![CDATA[hololens-2]]></category>
            <category><![CDATA[mrtk]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Mon, 15 Feb 2021 12:41:51 GMT</pubDate>
            <atom:updated>2024-01-03T04:45:38.262Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zN_DPKaPQHG-uJXjSSluKA.jpeg" /></figure><p>It has been some time since I wrote my last article. I decided to break the silence by compiling another beginner guide for people who are interested in getting into the field of AR/<a href="https://arvrjourney.com/expert-view-3-ways-vr-is-transforming-learning-development-487a021e8334">VR </a>and XR. This article is centered around an easy but important task in <a href="https://arvrjourney.com/how-to-subtly-use-augmented-reality-to-survive-your-zoom-meetings-7722abfeabce">AR</a>, that is <a href="https://arvrjourney.com/create-lenskart-like-ar-app-using-ios-realitycomposer-and-realitykit-with-almost-no-coding-119b149b7310">AR </a>marker tracking. This is definitely not a new topic, rather an area that has evolved over time. In this article, I will guide you on how to set up <a href="https://arvrjourney.com/how-ar-vr-are-transforming-the-real-estate-industry-73098df2901f">AR </a>marker tracking with HoloLens 2 without writing a single line of code, but with configurations. If you follow the article until the end you will be able to develop a HoloLens application that recognizes any 2D marker and popup a hologram as shown in Figure 1 below. So let’s get started!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*frAcVkhUi_siuT6DqtnjCQ.gif" /><figcaption>Figure 1</figcaption></figure><h4>Prerequisites</h4><ol><li>Windows Computer.</li><li>Unity version 2019.4.X installed with the Universal Windows Platform (UWP) support. If you have Unity installed without this module you can add a new module using Unity Hub easily.</li><li>A HoloLens 2 device (or you can use the computer webcam as well).</li><li>Unity Mixed reality Tool Kit (MRTK). Download the Microsoft.MixedReality.Toolkit.Unity.Foundation unity package. The other packages are optional. Download from <a href="https://github.com/microsoft/MixedRealityToolkit-Unity/releases/download/v2.5.4/Microsoft.MixedReality.Toolkit.Unity.Foundation.2.5.4.unitypackage">here</a>.</li><li><a href="https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=Community&amp;rel=16">Visual Studio 2019 community version</a> installed with ‘Desktop development with C++’ and ‘Universal Windows Platform (UWP) development’. You can tick these two workloads during the installation process or after using Visual Studio Installer (This is only required if you want to build your application to HoloLens, otherwise optional).</li></ol><p>Firstly, open a new Unity 3D project (Refer Figure 2) and go to <strong>File → Build Settings…</strong> (or Press Ctrl + Shift + B) to open the build settings window. On the pop-up window, select the <strong>Universal Windows Platform (UWP)</strong> under the platform section and <strong>HoloLens</strong> under the Target Device as shown in Figure 3 below. Then hit ‘<strong>Switch Platforms</strong>’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*aJGlK6lpHsSiek9bZYchRA.png" /><figcaption>Figure 2</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*cMNeYtVaK4LFdkikDs3WUw.png" /><figcaption>Figure 3</figcaption></figure><p>Secondly, we have to install Vuforia Engine Package. As usual, go to <strong>Window → Package Manager</strong> to add a new Unity package. The only difference here is, Vuforia Engine Package is installed from Git URL with a tag. Therefore, in the Package Manager window, click on the add (<strong>+</strong>) icon and select ‘<strong>Add package from git URL…’</strong>. See Figure 4 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*JTjpy8nc8zUgnKY5xNjD1A.png" /><figcaption>Figure 4</figcaption></figure><p>Then enter the following URL to install the Vuforia Engine Unity package and click <strong>Add</strong>. git+https://git-packages.developer.vuforia.com#9.7.4 Please note that there will be no visual feedback or pop up screen with a loading bar or a message while the package is importing. This will happen in the background. It may take some time to complete, depending on the speed of your internet connection. So give it a good 3 to 4 minutes and take a walk and come back. Maybe it will not take that long if you have very fast internet connectivity. After a few minutes select the ‘<strong>In Project</strong>’ option to see that package has successfully installed. See Figure 5 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*HYM_LWfRSIjFAu4VN-0Gwg.png" /><figcaption>Figure 5</figcaption></figure><p>Alright, now we have to install the MRTK foundation package. I have given the download path under the prerequisites. As mentioned, <strong>Microsoft.MixedReality.Toolkit.Unity.Foundation.2.5.4.unitypackage</strong> is required and others are optional. After downloading it, come back to Unity and go to <strong>Assets → Import Package → Custom Package…</strong> Select the downloaded MRTK foundation unitypackage file and click open. After resolving the packages there will be a pop-up window as in Figure 6. Click <strong>‘All’</strong> and then <strong>‘Import’</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*cpIB9WckythzkndOqRCPXw.png" /><figcaption>Figure 6</figcaption></figure><p>Once the import is complete there will be a prompt as shown in Figure 7. Click ‘<strong>Apply</strong>’ to finish the installation. Next, there is a set of configurations that need to make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*rfAaMuHlyBBCDivnxT5PTQ.png" /><figcaption>Figure 7</figcaption></figure><p>Well, if you want to make these configurations faster go to <strong>Mixed Reality Toolkit →Utilities → Configure Unity Project</strong>. I like to do it manually. To do it manually, go to <strong>Edit → Project Settings…</strong> You will get the Project Settings window on your screen. Select <strong>Player</strong>. Select the <strong>UWP icon</strong> tab and under that go to <strong>XR Settings</strong> and enable the ‘<strong>Virtual Reality Supported</strong>’ option. You might get a window similar to Figure 7 above. If so hit the ‘<strong>Apply</strong>’ button again. Under Virtual Reality SDKs (When you click +, you will see these SDKs) ensure that <strong>Window Mixed Reality</strong> is included in the list and select that. The <strong>Depth Format</strong> should set to <strong>16-bit depth</strong> and enable <strong>Depth Buffer Sharing</strong>. Ensure that the <strong>Stereo Rendering Mode</strong> is set to ‘<strong>Single Pass Instanced</strong>’. See Figure 8 and add the changes as in the figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ISj2nTulhHe_lUpts0f82g.png" /><figcaption>Figure 8</figcaption></figure><p>Next, go to ‘<strong>Publishing Settings</strong>’ (above the XR Settings) and find the ‘<strong>Capabilities</strong>’ section under that. Under ‘<strong>Capabilities</strong>’ ensure that <strong>Internet Client</strong>, <strong>WebCam</strong>, <strong>Microphone</strong>, and <strong>SpatialPerception</strong> are selected. Further, under ‘<strong>Supported Device Families</strong>’, ensure that ‘<strong>Holographic</strong>’ is selected. See Figure 9 to get an idea about these settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSKF8cKKZQD55BklKKQ_yA.png" /><figcaption>Figure 9</figcaption></figure><p>Finally, go to the ‘<strong>Resolution and Presentation</strong>’ section and disable ‘<strong>Run in Background</strong>’. So that Vuforia pauses when the app is put into the background and can access the camera again when the app is resumed. Further ensure that in the <strong>Default Orientation </strong>dropdown, ‘<strong>Landscape Left</strong>’ is selected. See Figure 10 below. After completing these configs close the Project Settings window.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGwcqdaZIhtK22uKLuSaTA.png" /><figcaption>Figure 10</figcaption></figure><p>Now let’s start configuring our Unity scene. Go to the <strong>Mixed Reality Toolkit</strong> menu and click on ‘<strong>Add to Scene and Configure…</strong>’ as shown in Figure 11 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tg12c-YOOZ5iY2QywpOEw.png" /><figcaption>Figure 11</figcaption></figure><p>Next, while ‘<strong>MixedrealityToolkit</strong>’ game object is selected, go to the inspector panel. On the inspector window, under the <strong>MixedREalityToolkit component</strong>, set the configuration profile to the ‘<strong>DefaultXRSDKConfigurationProfile</strong>’. See Figure 12 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*x5FMAR_JicBQTKqOR2oXCA.png" /><figcaption>Figure 12</figcaption></figure><p>After that, expand the ‘<strong>MixedRealityPlaySpace</strong>’ game object in the Hierarchy panel. Find the <strong>Main Camera</strong> game object. While the Main Camera game object is selected, go to the inspector panel. Under that, click ‘<strong>Add Component</strong>’ and type ‘<strong>Vuforia Behaviour</strong>’ and click on it to add the component. See Figure 13 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*OnjoIyfPrQXEMPZj7cFjig.png" /><figcaption>Figure 13</figcaption></figure><p>Now go back to the Hierarchy panel and right-click on it. Find the ‘<strong>Vuforia Engine</strong>’ option and click on ‘<strong>Image Target</strong>’. See Figure 14 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*wnX-Ep9wXD3puPn9XOmUMA.png" /><figcaption>Figure 14</figcaption></figure><p>This will add the ‘<strong>Image Target</strong>’ game object to the Hierarchy panel. While this game object is selected, once again go to the inspector and under the ‘<strong>Image Target Behaviour</strong>’ component → <strong>Type, s</strong>elect the ‘<strong>From Database</strong>’ option. If you get a prompt saying there is no database select the ‘<strong>Import</strong>’ option and it will set the default Vuforia Target database. See Figure 15 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*KNZm8WEGCuS3GsX-kTITyA.png" /><figcaption>Figure 15</figcaption></figure><p>Alright, we are almost there. Now we have to do a few more steps to set up Vuforia. Select the <strong>Main Camera</strong> game object again and on the inspector, under the <strong>Vuforia Behaviour</strong> component. Then click the ‘<strong>Open Vuforia Engine</strong>’ configuration button. Under that, you will see a button called ‘<strong>Add License</strong>’. Click on that. See Figure 16 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*vbKPEsj6CIElilCwEtv5RQ.png" /><figcaption>Figure 16</figcaption></figure><p>You will be directed to the <strong>Vuforia Developer Portal</strong> as shown in Figure 17. If you do not have an account, please create one, it is free! However, keep in mind that if you are going to use this library for commercial applications, then you have to purchase their subscription. But for this tutorial, the free account is sufficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aKEjkQv9c4b5TwzXFNVG_g.png" /><figcaption>Figure 17</figcaption></figure><p>Alright, assuming that you successfully created an account, I am going to show you the rest of the steps. Once you log in, you will see an interface similar to Figure 18. Click on ‘<strong>Get Development Key</strong>’ and enter a License name and click ‘<strong>Confirm</strong>’ (Figure 19). Then you will be redirected to the Figure 18 page. Click on your newly created License name and copy the license key and go back to Unity and paste it under the <strong>App license key</strong> as shown in Figure 16.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pBErG3j7_aPse2D0--rT8Q.png" /><figcaption>Figure 18</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EwR99bih5be5h9GXMcqyAQ.png" /><figcaption>Figure 19</figcaption></figure><p>Alright, now you added the license. One last step before we test the application. Go to the Hierarchy panel on the Unity editor and select the ‘<strong>Image Target</strong>’ game object. While ‘<strong>Image target</strong>’ is selected, click on the ‘<strong>Add Target</strong>’ button under ‘<strong>Image Target Behaviour Component</strong>’. You will be redirected to the Vuforia Developer Portal as shown in Figure 20.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cn9BryVFGEY1JC4ipTH6Mw.png" /><figcaption>Figure 20</figcaption></figure><p>Now we are going to create our marker database. Click on ‘<strong>Add Database</strong>’ and give it a name and select ‘<strong>Device</strong>’ as shown in Figure 21 below. Next click on the database name you created (I have named mine as <strong>SampleDb</strong>) and you will be redirected to the page as shown in Figure 22.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*SdIVgszJewZlpIVT0lxYOw.png" /><figcaption>Figure 21</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aJOtYrq005akcqINI6cO_A.png" /><figcaption>Figure 22</figcaption></figure><p>Click ‘<strong>Add Target</strong>’ and follow the instruction to upload your Target image(or we call it a marker). Make sure you provide the width of your target. You can see my target marker in Figure 24 below. Once you upload your marker, Vuforia will give it a rating. See Figure 25. That shows how confidently Vuforia can identify your marker. 5 starts are the best.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*s-FxXJuU24QIc7FXdn7ilQ.png" /><figcaption>Figure 24: Photo Credits — Ekrulila at <a href="https://www.pexels.com/photo/painting-of-giraffe-3246665/">https://www.pexels.com/photo/painting-of-giraffe-3246665/</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oqXcE997sfMmN_4HJhYLWA.png" /><figcaption>Figure 25</figcaption></figure><p>After you finish these steps, select the database and click ‘<strong>Download Database</strong>’. Then select the ‘<strong>Unity Editor</strong>’ option as shown in Figure 26. Now you should have &lt;database_name&gt;.unitypackage file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*5h5TIZePl25o8VnRbBim0Q.png" /><figcaption>Figure 26</figcaption></figure><p>Now, go back to Unity and import the target database that you downloaded from the Vuforia developer portal. To do that, go to <strong>Assets → Import packages → Custom Package…</strong> and selected the &lt;database_name&gt;.unitypackage file. After you import the database successfully, select the ‘<strong>Target Image</strong>’ game object and go to the inspector panel. In the inspector panel, under the ‘<strong>Image Target Behaviour</strong>’ component click on the dropdown under ‘<strong>Database</strong>’. You should see the newly added database name. See figure 27 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*nTcMFEvfLE76CQ82BFa0tA.png" /><figcaption>Figure 27</figcaption></figure><p>After selecting the database, select the <strong>Image Target</strong> if you have more than one marker in your database. For this article, I only have a single marker. See Figure 28 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*brACrG7aZzyPDz4NDS679w.png" /><figcaption>Figure 28</figcaption></figure><p>If you are getting this far, then congratulations, you have successfully configured everything. One final step, which is to add the hologram that you need to show when HoloLens detect the target marker. To do that, go to the Hierarchy panel and right-click on the ‘<strong>Image Target</strong>’ game object, and create any 3D object you like. For this article, I simply created a <strong>cube </strong>and added a red color material to it. See Figure 29. Feel free to use your creativity and you can add videos, 3D models (created from other software such as Blender and downloaded from the internet), and even animations. I am not going to discuss those as this article is already getting longer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zkXBSae3zhUK7MeEEAX_AQ.png" /><figcaption>Figure 29</figcaption></figure><p>Now save the scene and hit the play button on Unity to test the application. If you have a webcam, it should automatically turn on and you can bring your marker in front of your webcam. See Figure 30.</p><p>Optionally, to rotate the cube as in Figure 30, you need to add a small C# script. If you like to try it, attach the sample script below (See Script1) to the cube. If you do not add this script you should still be able to see a static cube.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f14d5f08dc28daf98e5b0d573d116724/href">https://medium.com/media/f14d5f08dc28daf98e5b0d573d116724/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b0Gan2-HwYNadf1bXixgOQ.gif" /><figcaption>Figure 30</figcaption></figure><p>If you want to build this to HoloLens go to <strong>File → Build Settings…</strong> and click ‘<strong>Add Open Scenes</strong>’ and hit <strong>Build</strong>. Select a directory to save the build files. You will get a .sln file after a successful build. See my very brief article on <a href="https://arvrjourney.com/how-to-build-your-ar-application-to-your-hololens-2-ca0640a01246">How to build your AR Application to your HoloLens 2</a> which will guide you to install this into HoloLens 2 and test it out. Good luck! Please feel free to leave your comments and suggestions on this article. If this article helps you and want to thank me, consider buying me a virtual coffee by clicking the ‘Buy me a coffee’ button below.</p><figure><a href="https://www.buymeacoffee.com/madhawa"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Dpw8-hNGI2fDmosV4E8DVQ.png" /></a></figure><p>Thanks for the read and I will see you in another article soon. Cheers!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*yX6T_VDVDq7kqnJU" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fcc7833%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2FARVR%3Ffbclid%3DIwAR3o3Cek44YsQ5uIM2rk9fWCGFnhDthkHbC095-H0CcmYt9FKGpa5UJ2dps&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252Fcc7833%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/href">https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/href</a></iframe><figure><a href="https://www.facebook.com/groups/984060571735120/"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/ARVR"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://docs.google.com/forms/d/1YBHAY792wL3OslxTQRqcIbRICNeKdeYSiSr2HyVrptU/edit"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb582c8f8ac0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ar-vr-journey/hololens-2-marker-tracking-with-vuforia-engine-and-mrtk-fb582c8f8ac0">HoloLens 2 Marker Tracking with Vuforia Engine and MRTK</a> was originally published in <a href="https://medium.com/ar-vr-journey">AR/VR Journey: Augmented &amp; Virtual Reality Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tools for hosting Virtual Conferences 2020 — Highlights]]></title>
            <link>https://madhawacperera.medium.com/tools-for-hosting-virtual-conferences-2020-highlights-eab6a4b5e375?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/eab6a4b5e375</guid>
            <category><![CDATA[virtual-conference]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[conference-planning]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 15:04:30 GMT</pubDate>
            <atom:updated>2020-11-13T15:04:30.928Z</atom:updated>
            <content:encoded><![CDATA[<h3>Tools for hosting Virtual Conferences 2020 — Highlights</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zvNolBaGL6jgtIkNQ5X2UQ.gif" /><figcaption>Figure1: Facebook’s social VR — <a href="https://techcrunch.com/2016/10/06/vr-emoji/">Image Source</a></figcaption></figure><p>In 2020, I participated in several virtual conferences and volunteered in some. This includes SIGGRAPH 2020, ICMI 2020, ISWC 2020 and ISS 2020. This article is a brief overview of the tools I was introduced to and learnt, and my thoughts on them as a presenter, attendee and a student volunteer. In this article, I will discuss some platforms, what they were used for and some of their notable common features. Finally, I would offer some useful tips and suggestions, from my experience, for both organizers and participants of future virtual conferences and events.</p><p>Many think that virtual conferences are not as good as physical conferences. At least, that is what I have heard most of the time during conferences I attended. Well, if I compare my previous year visits to physical conferences with the virtual experience, of course, I miss travelling around the world and collecting all the souvenirs and swags. Nevertheless, I still enjoyed all the virtual conferences that I attended this year.</p><p>To me, both physical and virtual conferences have their own pros and cons. For example, in past years, most conferences did not allow pre-recorded videos or live online video presentations; physical participation was the norm. Things are changing for good. While all of us are still getting used to fully virtual conferences, my prediction is that in the future, conferences would be hybrid. This is considering the cost of time + money it incurs to host and visit a conference in person. I am positive that considering the experiences gained through physical and virtual events, eventually, we all will fit well within the new hybrid (both in-person and virtual participants) conferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4zc8YrIsqNYzh7zC5xeCDw.jpeg" /><figcaption>Figure 2 — Image credits: All the images displayed are taken from respective vendors</figcaption></figure><h4>Which tools are better?</h4><p>The most important topic in virtual conferences is which tools are better to use, and how to pick one over another. Well, the direct answer is there are many tools out there and all the conference that I attended, at least had one new software.</p><p>Let me share some of the tools that these conferences used in hosting attendees. In SIGGRAPH 2020, they used <a href="https://www.hubb.me/">hubb.me</a> and <a href="https://zoom.us/">Zoom</a> to deliver the conference. ICMI2020 used 3 platforms. Those were <a href="https://zoom.us/">Zoom</a>, <a href="https://spatial.chat/">SpatialChat</a> and <a href="https://ryver.com/">Ryver</a>. ISWC2020 provided a login to registered participants to their conference program webpage which included relevant links to each session. As per tools they used <a href="https://zoom.us/">Zoom</a>, <a href="https://www.microsoft.com/en-au/microsoft-365/microsoft-teams/group-chat-software">Microsoft Teams</a>, <a href="https://remo.co/">Remo</a> and <a href="https://slack.com/">Slack</a>. In ISS2020, they used <a href="https://zoom.us/">Zoom</a>, <a href="https://discord.com/">Discord</a> and <a href="https://heyhubbub.com/">hubbub</a>. Further, I am helping with another symposium which will be held in which we are planning to use <a href="https://www.webex.com/">Cisco Webex</a>.</p><p>Each of these tools has its own capabilities and limitations. This is why many conferences had to use more than one tool to host the entire conference. The vital fact to note is that all these conferences had a larger audience to mange. Thus, there were challenges which I am not going to discuss in this article.</p><h4>Purposes of some tools</h4><p>The reason to use multiple tools in one conference is to fill the gap in user networking. So most of the organizers wanted to have some kind of a tool that was fun + provided some opportunities for informal chat and networking. Currently, with Zoom, Webex and Google meet (all the video conferencing tools) cannot facilitate this due to various reasons. Of course, I am expecting in the future these software to provide event management features as plug-ins (extensions), so that participants can have a seamless experience where they don&#39;t need to switch between tools. However, for now, these tools are not offering such options.</p><p>Below I will categorize the tools by their use and discuss their features in brief.</p><p>All of the main conference sessions, paper sessions, workshops were hosted either on Zoom meetings or Zoom webinars (This could be done with Webex as well). Keynotes were mainly delivered in the format of Zoom webinars where attendees cannot view other attendees joined; Such that fewer interactions. Yet, there are features for raise hand (so that organiser could unmute the attendee, type questions etc). In ICMI2020 they even used breakout rooms on Zoom meetings. If the session is interaction heavy (like a workshop) it was conducted using Zoom meetings where each attendee can see each other, chat and talk.</p><p>Next, look at some program management software. <a href="https://www.hubb.me/">hubb.me</a> and <a href="https://ryver.com/">Ryver</a> can be used as tools to communicate users with relevant information and manage the sessions of the event. Simply put, if you have 10 sessions, and there are 10 different meeting links to those 10 sessions, then you need to have a platform you could share all these information with users. hubb. me has a calender like feature which allows users to create their own small schedule by adding the necessary session they want to attend. <a href="https://ryver.com/">Ryver</a> provides a similar feature which they call as &#39;tasks&#39;. User can create their own tasks to keep a list of things that they want to do during the conference. Otherwise, you can create your own conference website with user log-in which I think is alright (<a href="https://en.wikipedia.org/wiki/KISS_principle">KISS</a> principle).</p><p>Discord and Slack are of course good tools for networking. However, the way this has used in conferences, which is running a meeting on Zoom and asking participants to post a question on the relevant Discord or Slack channel could be problematic. But this is a good way to encourage to create a discussion thread based on a particular session (or research paper). I experienced that participants could raise questions on multiple platforms if the conference is using many, eg: some can ask questions on Zoom while some others ask on Slack or Discord. This was managed through a student volunteer who watched both these spaces and posted all the question on another space for the session chair to see. I think managing Q&amp;A using the video conference tool itself is much more appropriate. If the session is recorded Q&amp;A will also be recorded (assuming session chair reiterate all the question that are in chat).</p><p>I like Discord and Slack as they have video and voice calling features, which enable you to have a quick video chat with individual participants. I used this feature quite frequently when networking with people.</p><p>Another thing is recording and taking down minutes. ICMI2020 used <a href="https://otter.ai/login">otter.ai</a> to take minutes automatically. This is another way to keep a record of what happened during a session if in case a session is not recorded. This way, student volunteer can post the recorded Q&amp;A in the respective Discord or Slack channel afterwards. Discord and Slack channels could be open as many times as the organizer wants to. If someone has a question later on they can still contact the author if necessary.</p><p>Remo, Spatial Chat, Mozilla Hubs, Gather Town, Spatial.io, hubbub are mainly used for social gatherings and informal chats. However, these still can be used to host an entire conference as some of these tools have screen sharing, video and audio conferencing capabilities inbuilt (e.g. <a href="https://gather.town/">Gather</a> and <a href="https://hubs.mozilla.com/">Mozilla Hubs</a>). The problem is these tools are new hence not all participants would welcome them. This is something to keep in mind if you plan to use these platforms for the entirety of the conference. However, in future (when Facebook + Oculus create the next social VR platform) I am sure these spaces will become much more intuitive for many users just like we adapted to social media and to share things we do in our lives with the whole world.</p><p>I am not going into further details of these tools. But before I conclude I would like to highlight some perspectives as an attendee, presenter and a student volunteer.</p><h4>Conference Attendees perspective…</h4><p>As an attendee, of course, having to learn multiple tools is not a very pleasant experience (at least that is what I heard from many). Every conference organizer had to send several emails with instructions, tutorial videos etc. which means lots of homework to be done before attending the conference. If you don’t do your homework, well there are helpdesk, but still, the attendee has to face various difficulties such as browser not supporting some software, not enough memory to install software, OS-related security issues and whatnot. So it is very important to have a set of dedicated technical support staff before and during the conference. Well, student volunteers would be happy to help with these.</p><p>Next point is, try different tools first and see for yourself (as organizers) which tool would be more intuitive for your use and provide a better experience to the user. Do not rush to pick the tool. Always appoint an online-chair or virtual-chair (you can name it as you want) with a set of student volunteers/organizing committee members. Try some tools (there is a free trial in every software) and some are free as well. For example, if you take Ryver. Firstly a user needs an account. Next, on a browser, Ryver doesn’t allow a user to open it in two different tabs. If someone, like me, who has tons of tabs and a couple of browsers open, I lost the track of Ryver tab totally. The only option for me was to install Ryver desktop client on my computer. This may not be an option for everyone (I am not against Ryver, it is a great tool and it has nice features such as task scheduling which I didn’t see in another tool). However, lesser the trouble better it is because some participant might not want to install completely new software for a single conference.</p><p>Let’s look at some other requirements. Since the conference is online many attendees would want to access presenter slide deck or videos. So make sure the platform you choose has video sharing and file-sharing capabilities. Otherwise, of course, you have to have cloud storage (e.g. Microsoft OneDrive, Google Drive) where you can upload files and share links with participants. YouTube, Vimeo, GitHub, GitLab (for video and file sharing) are also options but as I mentioned earlier, lesser the number of tools used better it is.</p><h4>From a presenter perspective…</h4><p>As a presenter, I think (at least for me) virtual conferences gave me a positive experience (Well I am generally okay with public speaking though). I could prepare my video in a way that I clearly deliver my message. I avoided running out of time or missing presenting important points. It is a bit of work (to prepare a video) but you can try and give a clear picture of your work to the audience. Again, you could save all the question, feedback received which is another important thing. Again, I would like if I could concentrate on one platform while I am presenting and later, of course, answering on Discord or Slack.</p><h4>Student Volunteer (SV) Experience…</h4><p>Finally, as a student volunteer. Most importantly we (SVs) got to learn all these platforms and it is indeed handy to master a lot of virtual platforms these days. I myself experienced all the software that is listed in Figure 2 above and confident in helping someone with troubleshooting and setting up.</p><p>Further, SVs still get some perks such as free registration and some swags (I got a T-shirt that came all the way from Europe). I agree it is not a compensation to the experience of travelling though, but we cannot travel always even if you don’t have any restrictions due to various other reasons.</p><p>The best thing that I gained as a student volunteer is a network and the experience. You can meet from fellow PhD students to giants in the field. Well of course you get student volunteer perks like T-shirts and all the swags, free registration and a lot of fun meetings with other students from around the world. With virtual conferences, even though we miss the &#39;elevator pitch&#39;, no dinner or catch up at lunch, we still get the chance to send people we want to connect with a private message using all these tools. The chances are high that you could get in touch with them. You can drop them an email, or chat instantaneously. I think this is efficient and effective than sharing a physical business card which may or may not connect you with relevant people.</p><h4>In summary</h4><p>It would have been nice if there is a way to conduct the whole conference with a single point of entry. That means users can log into one software that has all the facilities combined (e.g. Features of Ryver, Zoom and Spatial Chat all in one software). Well, I understand this could be a bit heavy engineering task but I am sure such platforms will have immense benefit for virtual conferences.</p><p>Well, I was thinking what if conferences could provide VR Head Mount Displays (HMD) for participants (cost can be included in the registration pack) then the experience will be much nicer with platforms like <a href="https://spatial.io/">spatial.io</a> and Mozilla Hubs. If we put some thoughts in on how to manage cost and shipping, I don’t think this totally is a fancy (junk) idea. Since conference organizer doesn’t have to pay for expensive venues and don’t have to pay for travelling and hotel stay, if they can arrange some agreement with VR HMD manufacturers like Oculus, I believe they can try and provide a VR HMD for a lesser price than in the market.</p><p>With all these experiences and ideas in mind, looking forward to hybrid conferences in 2021…</p><p>If you like the article and the content, you can buy me a virtual coffee as everything else moves from reality towards virtual too 😊</p><figure><a href="https://www.buymeacoffee.com/madhawa"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Dpw8-hNGI2fDmosV4E8DVQ.png" /></a></figure><p>Cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eab6a4b5e375" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hubs: Let's meet virtually]]></title>
            <link>https://medium.com/ar-vr-journey/hubs-lets-meet-virtually-c9646f7a00ad?source=rss-51210a3a6f0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9646f7a00ad</guid>
            <category><![CDATA[virtual-conference]]></category>
            <category><![CDATA[mozilla]]></category>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[mozilla-hubs]]></category>
            <dc:creator><![CDATA[Madhawa Perera]]></dc:creator>
            <pubDate>Mon, 07 Sep 2020 16:04:45 GMT</pubDate>
            <atom:updated>2020-11-13T15:05:56.927Z</atom:updated>
            <content:encoded><![CDATA[<p>Hubs is the talk among several communities who organize virtual conferences (Of course there are other platforms as well), as it allows participants with <a href="https://arvrjourney.com/designing-for-a-modern-3d-world-a-ux-design-guide-for-vr-c6741a72daeb">VR</a> headsets as well as those who don’t, to be immersed in the virtual scene. Accessibility and experience are key in drawing people towards this technology. On top of all, you can create your own virtual spaces (like a building, lecture hall, etc.). This article has been sitting in my drafts for some time now but thought of giving it a go as many people (@ANU) show interest and me about this.</p><p>This article, as indicated by the topic, will help you to get an understanding of Mozilla Hubs and how and where you can use it and for what. Additionally, I will share my experience using Mozilla Hubs via browser and <a href="https://arvrjourney.com/">VR</a> headset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*8gEQcrb197Y6mWYeu6tsjQ.gif" /><figcaption><a href="https://hubs.mozilla.com/G729HTM/never-never-land/">https://hubs.mozilla.com/G729HTM/never-never-land/</a></figcaption></figure><h4>Introduction to Mozilla Hubs</h4><p>For those who do not know what Mozilla Hubs is, it is a virtual collaboration platform. Other teleconferencing tools such as WebEx, Zoom, Google Meet, Microsoft Teams, etc offer similar features. The best part of Mozilla Hub is that it runs in your browser without any installations and anyone can create their own 3D spaces. There are already ‘pre-build environments’ which you can use with a single click and invite others to join using a URL. Additionally, you can use Spoke (an online 3D scene editor), which is super simple and requires no coding skills, to create your own space as you please. See a quick video below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJpF8DmOGMZ8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJpF8DmOGMZ8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJpF8DmOGMZ8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2633b8f7db5f29336da6bd3e6cdfff76/href">https://medium.com/media/2633b8f7db5f29336da6bd3e6cdfff76/href</a></iframe><h4>Can I try this right now?</h4><p>Yes, it is as simple as clicking here <a href="https://hubs.mozilla.com/G729HTM/never-never-land/">https://hubs.mozilla.com/G729HTM/never-never-land/</a></p><p>Screen captures in this article are taken from a scene that I created using <a href="https://hubs.mozilla.com/spoke"><em>Spoke </em></a>for an event. Once you go to the above URL, you will see an interface as shown in Figure 1 below. Click on ‘Enter Room’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GhuXZ7hp-7r39lgqSoYQ5g.png" /><figcaption>Figure 1:Never-never land scene</figcaption></figure><p>Once you enter, you will be asked to pick an avatar. Well, this is the 3D character that is going to represent you inside the scene. (You can change this at any point so don&#39;t worry). I found a cool avatar from <a href="https://sketchfab.com/feed">Sketchfab</a> done by <a href="https://sketchfab.com/11fishem">MatthewAFisher</a> and that is shown in figure 2 below. You will get this same window to pick your avatar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*TXHIHBwWspsLeqJAeKo1sw.png" /><figcaption>Figure 2: Avatar that represents you</figcaption></figure><p>If you do not like the one that is shown there, click on ‘Browse Avatars’ and you will get a bunch of pre-designed avatars as shown in Figure 3 below. Pick one of those. If you do not like any of them, either you can create your own or find one online from various sources and copy-paste the URL to the GLB file. Or else download and upload if you already have GLB file clicking on ‘Create Avatar’. Do not spend too much time on this if this is your first time and just want to experience Hubs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQG7_X9yBHSLssJ9rxg6Ig.png" /><figcaption>Figure 3: Default Avatar options</figcaption></figure><p>Once finished, click <strong>Accept</strong> and click <strong>Enter on Screen</strong> if you are using your computer. See Figure 4 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*oGXNom_SJApmVrKwFR6jRQ.png" /><figcaption>Figure 4</figcaption></figure><p>Test your microphone and Audio and that&#39;s it. Click &quot;<strong>Enter Now</strong>&quot;.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*UI3StN76PZnfZRavvxDbeQ.png" /><figcaption>Figure 5</figcaption></figure><p>In Hubs, you get the first-person view on the browser as well. In order to give you a quick understanding of what is first-person vs the third person views, I have included the image in Figure 6 below. I extracted this figure from Oshita et al.&#39;s [1] paper on the Motion-capture-based avatar control framework in third-person view virtual environments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*R_iFdSKZYGkeGikt7mQyPw.png" /><figcaption>Figure 6: Oshita et al. Motion-capture-based avatar control framework in third-person view virtual environments [1]</figcaption></figure><p>Back to Hubs. Once you enter the room you can copy-paste URLs (e.g. youtube videos), 3D objects (in .GLB formats) on the screen and it will be shared with all the participants in a room. You can download free .glb files from many online sources like <a href="https://www.cgtrader.com/">CGTrader</a>, <a href="https://clara.io/">Clara.io</a>, <a href="https://grabcad.com/">Grabcad</a>, etc. or create your own (<a href="https://www.blender.org/">Blender</a> is a good tool for this and it&#39;s free) and add them. Optionally, Hubs allows you to browse and add 3D objects within the Hubs window from the 3D asset stores like Sktchfab, Google poly. So there are plenty of options for you.</p><p>Alright, you are all set now and inside the never-never land scene that I created. Figure 7 below shows how it will look on your screen/browser. If you followed the URL I shared above (here the link again <a href="https://hubs.mozilla.com/G729HTM/never-never-land/">https://hubs.mozilla.com/G729HTM/never-never-land/</a>) and followed all the steps mentioned, you will get here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CdV4CKoHzic7qTeG6RD_BQ.png" /><figcaption>Figure 7</figcaption></figure><p>Let’s have a quick look at the interface. What are those icons for? I created a very cluttered image below (See Figure 8 below). I know it is not super readable but I hope this is faster than naming them A, B, C .. and explaining separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r8Ol7VaZDK6s6qKs75fjEg.png" /><figcaption>Figure 8: Hubs user interface inside a scene</figcaption></figure><p>Now you are ready to invite participants. Click on the share button, you will get a link as well as a 6 digit number by default. If you have a <a href="https://arvrjourney.com/build-your-first-vr-game-with-oculus-quest-and-unity-part-1-3133449ceb12">VR</a> headset, put it on and go to the <strong>Browser </strong>and type ‘hub.link’ and you will be prompted to enter this code. Once you enter the code you can enter the scene using your VR headset. I think Hubs is better in this shape but it carries it&#39;s own disadvantages as well, which I will not go into describing in this article.</p><p>I found Hubs to be pretty cool! I used this for my tutorials and students really liked it as it, sort of, gave them a feeling of “playing a game”.</p><h4>Alright! How do I create my own room?</h4><p>This is even easier. Go to <a href="https://hubs.mozilla.com/#/">https://hubs.mozilla.com/#/</a> and click ‘Create Room’.</p><p>You will de be directed to Hub&#39;s default room. Which looks like below in Figure 9.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*EWpXJOwnokYV6SdA.jpeg" /><figcaption>Figure 9</figcaption></figure><p>Click on the <strong>Menu </strong>button click on <strong>Choose a Scene</strong> (See Figure 10 below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*W1rEDblCCfeoEDwDAKjg1g.png" /><figcaption>Figure 10</figcaption></figure><p>You will be asked to provide an email to log in as you need an Account to create a room. It is a quick process and you can use your email address to sign in. Once done, there are plenty of pre-designed scenes as shown in Figure 11 below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cC_nN-IqpP7iy_KeW3Ru9Q.png" /><figcaption>Figure 11</figcaption></figure><p>If you do not like any of those you can click on ‘create a scene with <a href="https://hubs.mozilla.com/spoke">Spoke</a>’. I will not talk about Spoke here but the GUI looks like below as shown in Figure 12. It is easier to understand if you have experience working with Unity otherwise, their welcome tutorial is sufficient to get going.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VjmvaLy79pqCxfu67Aeqrw.png" /><figcaption>Figure 12: Spoke</figcaption></figure><p>As per my observations, Hubs provide a very different user experience (UX) for users who are in a fully immersed environment (Using VR headsets) and using computer browsers. However, that doesn&#39;t say that Hubs only suit one set of users but the UX differs, so think about your audience if you chose to use Hubs for an event. There are pros and cons to using Hubs and if you want to read my experience in delivering online courses with Hubs wait for a separate article on that.</p><p>That is all I planned to discuss in this article. If you like the Article and the content, you can buy me a virtual coffee as everything else moves from reality towards virtual too 😊;</p><figure><a href="https://www.buymeacoffee.com/madhawa"><img alt="" src="https://cdn-images-1.medium.com/max/170/1*Dpw8-hNGI2fDmosV4E8DVQ.png" /></a></figure><p>Cheers!</p><h4>Reference</h4><p>[1] Oshita, M., 2006, June. Motion-capture-based avatar control framework in third-person view virtual environments. In <em>Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology</em> (pp. 2-es).</p><h3>Trending AR VR Articles:</h3><blockquote><a href="https://arvrjourney.com/how-to-subtly-use-augmented-reality-to-survive-your-zoom-meetings-7722abfeabce">1. How to use subtle AR filters to survive your Zoom meetings?</a></blockquote><blockquote><a href="https://arvrjourney.com/the-first-no-headset-virtual-monitor-91ffc578cca">2. The First No-Headset Virtual Monitor</a></blockquote><blockquote><a href="https://arvrjourney.com/augmented-reality-ar-is-the-future-of-restaurant-menu-4c66704b1142">3. Augmented reality (AR) is the future of Restaurant Menu?</a></blockquote><blockquote><a href="https://arvrjourney.com/creating-remote-mr-productions-2ed270630823">4. Creating remote MR productions</a></blockquote><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*yX6T_VDVDq7kqnJU" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fcc7833%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2FARVR%3Ffbclid%3DIwAR3o3Cek44YsQ5uIM2rk9fWCGFnhDthkHbC095-H0CcmYt9FKGpa5UJ2dps&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252Fcc7833%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/href">https://medium.com/media/1e1f2ee7654748bb938735cbca6f0fd3/href</a></iframe><figure><a href="https://www.facebook.com/groups/984060571735120/"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*2f7OqE2AJK1KSrhkmD9ZMw.png" /></a></figure><figure><a href="https://upscri.be/ARVR"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*v-PpfkSWHbvlWWamSVHHWg.png" /></a></figure><figure><a href="https://docs.google.com/forms/d/1YBHAY792wL3OslxTQRqcIbRICNeKdeYSiSr2HyVrptU/edit"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*Wt2auqISiEAOZxJ-I7brDQ.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9646f7a00ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ar-vr-journey/hubs-lets-meet-virtually-c9646f7a00ad">Hubs: Let&#39;s meet virtually</a> was originally published in <a href="https://medium.com/ar-vr-journey">AR/VR Journey: Augmented &amp; Virtual Reality Magazine</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>