<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Carson Kohlbrenner on Medium]]></title>
        <description><![CDATA[Stories by Carson Kohlbrenner on Medium]]></description>
        <link>https://medium.com/@carson.kohlbrenner?source=rss-ec8758ce8022------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*XBb98cYQ-chlq_RvSMDdOQ.jpeg</url>
            <title>Stories by Carson Kohlbrenner on Medium</title>
            <link>https://medium.com/@carson.kohlbrenner?source=rss-ec8758ce8022------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:25:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@carson.kohlbrenner/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Can humanoid robots rely on teleoperation for edge cases like self-driving cars can?]]></title>
            <link>https://medium.com/@carson.kohlbrenner/can-humanoid-robots-rely-on-teleoperation-for-edge-cases-like-self-driving-cars-can-bae2274847bd?source=rss-ec8758ce8022------2</link>
            <guid isPermaLink="false">https://medium.com/p/bae2274847bd</guid>
            <category><![CDATA[humanoid-robot]]></category>
            <dc:creator><![CDATA[Carson Kohlbrenner]]></dc:creator>
            <pubDate>Tue, 24 Mar 2026 04:18:31 GMT</pubDate>
            <atom:updated>2026-03-24T04:18:31.570Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jOnwddPydMEXaKgnLLGiDA.png" /><figcaption>A teleoperator controls a humanoid robot from a distance with a virtual reality (VR) headset. Source: ChatGPT</figcaption></figure><p>Last October, <a href="https://www.1x.tech/discover/neo-home-robot">1X Technologies</a> announced the first consumer ready humanoid robot with the purpose of automating house chores. <a href="https://time.com/7324233/figure-03-robot-humanoid-reveal/">Figure</a> and <a href="https://www.axios.com/2026/01/22/elon-musk-tesla-optimus-robots">Tesla</a> are closely following, with plans to relaease their humanoid robots for home-use by 2027.</p><p>As these robots begin to work in human environments with kids, elderly, and pets, safety becomes a critical concern. Humanoid robot safety is currently governed by <a href="https://www.iso.org/standard/73933.html">ISO 10218–1:2025</a>, which is intended for industrial regulation. Currently, <strong>no standardized safety framework exists for humanoid robots operating in people’s homes.</strong></p><p>Humanoid robot companies, inspired by the autonomous automotive industry, use humans in the loop to in the form of teleoperation to combat edge cases. For example, when <a href="https://medium.com/@hayekesteloo/1xs-20-000-neo-humanoid-robot-launches-with-a-catch-it-needs-humans-to-do-your-chores-77eda207fbb8">1X’s NEO robot encounters a task it cannot complete</a>, it will schedule an operator wearing a virtual reality headset to take over and do the task. Teleoperation may even be used as the primary control method for the robot, as in the case of <a href="https://www.businessinsider.com/tesla-optimus-robots-bartending-controlled-by-humans-2024-10">Tesla’s bartending Optimus</a>.</p><p>But what happens when the <strong><em>connection to the operator is lost</em></strong>?</p><p>Consider the scenario where a humanoid is folding laundry and loses connection to an operator. It may be sufficient in this case for the humanoid to freeze and wait for the operator to resume control. However, if the humanoid was pouring a hot pot of coffee, assisting someone with limited mobility, or carrying heavy objects down a flight of stairs, freezing may be <em>catastrophic</em>.</p><p>In this article, we will look at potential fallbacks ranging from hardware compliance, return states, and autonomous control for household humanoid robots when the connection to a human in the loop is lost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hum4g3rutQNBVIP6.png" /><figcaption>A humanoid robot is teleoperated to move a box. Source: <a href="https://arxiv.org/abs/2505.02833">[2505.02833] TWIST: Teleoperated Whole-Body Imitation System</a></figcaption></figure><h3><strong>Failing Safely</strong></h3><p>The robot should know if it is in a “potentially hazardous” configuration or task when the hand-off from human to robot is initiated due to a loss of connection. The ideal case is that the robot is able to safely continue performing the task autonomously until the human in the loop returns if it is deemed non-hazardous. If the task is potentially hazardous, the robot should immediately assume the safest state.</p><p>One common “safe state” is to lower the robot’s center of mass and have them sit [5]. This reduces the potential danger of the robot losing balance and falling on someone or something, but sitting may also pose a danger in itself. Take the example of a robot moving boxes into a garage or walking across a street at a crosswalk. If the human in the loop loses connection, having the robot sitting down in the way of large vehicles poses a serious hazard.</p><p>Instead, if the robot is capable of high-level reasoning, such as possible through vision-language-action (VLA) models, then the robot should identify possible locations to sit or stand that pose the lowest safety hazard and navigate to them. <strong>The robot should additionally avoid collisions wherever possible</strong> while in transit to the safe location and configuration until the human in the loop returns to minimize the risk of being a hazard.</p><p>Even with a sophisticated ability to reason and react to the environment, the robot may still be unable to avoid a collision. Robots with metal frames are particularly dangerous and can cause serious injury from minor collisions. For this reason, <strong>robots deployed in the home should always have a flexible exterior to absorb impact energy</strong> [6].</p><h3><strong>The Autonomy Boundary</strong></h3><p>The amount of edge cases that a home robot may face is vast, so identifying when the robot should resort to a safe state should depend on experience, not intuition.</p><p>Semantically describing all the possible tasks that a robot may face with a classical control model is infeasible, so future home robots will likely rely on learning skills from data. Instead of manually identifying tasks that are hazardous, <strong>a model should be trained to both identify when a task is in distribution and if it is confident in its ability to complete the task autonomously</strong>. If the task has been done hundreds to thousands of times with high success and low impact to other objects or people, then the robot should only then consider completing the task autonomously.</p><p>A contemporary challenge remains on how humanoid robots should avoid collisions with objects and people. Humans and animals use a combination of all their senses to be spatially aware of objects in their environment, but even the most sophisticated artificial perception systems have trouble tracking objects that move when outside the field of view of their onboard cameras. Luckily, world models such as <a href="https://arxiv.org/abs/2501.03575">Cosmos</a> [2] have begun to show that object permanence can be learned and may be the key to anticipating collisions before they happen.</p><p>Until such models can be deployed in real-time, humanoid robots should be equipped with distributed high coverage sensors that minimize blind spots and can detect objects in the environment for reactive avoidance.</p><h3><strong>Trust and Latency</strong></h3><p>When a task is being performed by a human teleoperator, they need to be aware of what the robot’s fallback state is should they lose connection and seamlessly be able to resume control. The robot’s confidence may directly impact a teleoperator’s decision to engage in a risky operation or trust the robot to complete the task autonomously. Additionally, it was recently reported that the state of the teleoperator diverges from the robot’s state due to latency, and VR teleoperation introduces an average of 400ms of latency [3]. <strong>This latency alone renders reactive control infeasible for a teleoperator</strong>, considering that human reaction time is around 213ms, where 130 ms accounts for just detecting the stimulus [4].</p><p>Humanoid robots should have a low-latency reactive controller that compliments the teleoperator’s control input to ensure the robot can better avoid collisions and the human teleoperator can provide coarse inputs to resolve reactive edge cases.</p><h3><strong>Conclusion</strong></h3><p>Latency makes reactive teleoperation unreliable, and connection loss can occur at any moment, making the human-in-the-loop model an incomplete safety solution on its own.</p><p>Robots must be able to identify hazardous situations, transition gracefully to a safe state, and resume tasks autonomously when confident in their ability to do so. Compliant hardware, distributed sensing, and low-latency reactive controllers are baseline requirements for operating safely around people. As deployment scales, the trust between teleoperator and robot will only be as strong as the robot’s ability to fail predictably and recover intelligently.</p><p>With companies like 1X, Figure, and Tesla planning to bring humanoid robots into homes within the next few years, solving these challenges belongs on the critical path to deployment.</p><p><em>Note: This article has not been peer reviewed and simply serves as a thought experiment for making household humanoid robots safer.</em></p><h3><strong>References</strong></h3><p>[1] Hogan, Neville. “Impedance control: An approach to manipulation.” 1984 American control conference. IEEE, 1984.</p><p>[2] Agarwal, Niket, et al. “Cosmos world foundation model platform for physical ai.” arXiv preprint arXiv:2501.03575 (2025).</p><p>[3] Sun, Zhenguo, et al. “MOSAIC: Bridging the Sim-to-Real Gap in Generalist Humanoid Motion Tracking and Teleoperation with Rapid Residual Adaptation.” arXiv preprint arXiv:2602.08594 (2026).</p><p>[4] Woods, David L., et al. “Factors influencing the latency of simple reaction time.” Frontiers in human neuroscience 9 (2015): 131.</p><p>[5] Scianca, Nicola, et al. “A behavior-based framework for safe deployment of humanoid robots.” Autonomous Robots 45.4 (2021): 435–456.</p><p>[6] Ikuta, Koji, Hideki Ishii, and Makoto Nokata. “Safety evaluation method of design and control for human-care robots.” The International Journal of Robotics Research 22.5 (2003): 281–297.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bae2274847bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bringing Tactile VLMs to Local GPUs]]></title>
            <link>https://medium.com/correll-lab/bringing-tactile-vlms-to-local-gpus-8f7ac082f306?source=rss-ec8758ce8022------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f7ac082f306</guid>
            <category><![CDATA[tactile]]></category>
            <category><![CDATA[vlm]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[gelsight]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Carson Kohlbrenner]]></dc:creator>
            <pubDate>Fri, 20 Dec 2024 09:28:02 GMT</pubDate>
            <atom:updated>2024-12-20T09:28:02.110Z</atom:updated>
            <content:encoded><![CDATA[<h4>Scaling the size of the OCTOPI tactile VLM down to a more digestible size</h4><p>Touch is a sensing modality first explored for robots back in the 80s that, until recently, only delivered high hopes and lackluster capabilities. Today, robotics labs around the world have regained interest in the modality, as recent advances in foundation models may finally bridge the gap between how humans and robots perceive the world through the sense of touch. Traditionally, tactile sensors could only be used to classify objects, detect force and slip, or inform a robot when it was in contact with an object in the environment. Today, tactile sensors are being used by robots to learn about, think about, and even plan their actions in their environment. The secret to this new capability? <strong>Tactile-vision-language models, </strong>or TVLMs for short<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b-EsE9ABPFnjFkS7D7FpxA.jpeg" /><figcaption>Fig. 1) This robot uses its transformer architecture to learn about an avocado by how it feels instead of just by how it looks. | Source: DALL•E 3</figcaption></figure><p>While TVLMs are powerful, they are not easy to use. Current TVLMs such as <a href="https://github.com/clear-nus/octopi">OCTOPI</a> and <a href="https://tactile-vlm.github.io/">TVL</a> have only been tested using setups with over 40 GiB of VRAM, almost twice as much as an RTX 4090! In this article, we will analyze the OCTOPI VLM model’s limiting factors of deployment and modify it to work on a single RTX 4090 GPU. This article is geared toward roboticists interested in running TVLMs locally who may not have a comprehensive background in VLMs. Follow along with the code from the <a href="https://github.com/clear-nus/octopi">original paper here</a>.</p><h3>The OCTOPI Model:</h3><p>The OCTOPI model is a TVLM that uses <em>tactile videos </em>to perceive and reason about its environment intelligently. Tactile videos are temporally constrained sequences of images taken by a <a href="https://www.gelsight.com/">GelSight</a> sensor. These tactile videos contain information about the surface properties of objects in contact and can provide contextual information to solve problems. The figure below is an example of using information from within a tactile video to determine if a bowl of rice is cooked or uncooked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ItCIsgRFysiFMCJt" /><figcaption>Fig. 2) OCTOPI reasons about a bowl of rice through touch. | Source: <a href="https://octopi-tactile-lvlm.github.io/">OCTOPI paper</a></figcaption></figure><p>The OCTOPI model consists of three main components: A visual-tactile encoder, a projection module, and a large-language model (LLM). The model’s power lies within its visual transformer architecture to encode the commonly present patterns in tactile data and project them in a way that its highly trained LLM can understand. The vast knowledge stored within the LLM is what reasons about the environment based on the context provided by the tactile encoder. OCTOPI uses the pre-trained <a href="https://drive.google.com/drive/folders/1AdLCaD_5CiNvsm_cm6IT7KDeXgZVSQlY?usp=sharing">CLIP ViT-L/14</a> as the visual encoder, two sequential linear layers with GELU activations for the projection module (inspired by <a href="https://arxiv.org/abs/2304.08485">LLAVA</a>), and <a href="https://huggingface.co/lmsys/vicuna-13b-v1.3">Vicuna</a> as the LLM.</p><p>Check out my latest post for a deeper dive into the OCTOPI model, how the dataset was collected, and how the model was evaluated:</p><p><a href="https://medium.com/correll-lab/paper-review-octopi-object-property-reasoning-with-large-tactile-language-models-0f856dcdf8be">Paper Review: Octopi: Object Property Reasoning with Large Tactile-Language Models</a></p><p>We will assume you are familiar with the fundamentals of the model going forward and look at training your own model on a local GPU!</p><h3><strong>Limiting Factor: The Large-Language Model (LLM)</strong></h3><p>Large language models (LLMs) are the limiting factor in a low GPU environment due to their massive size and computational complexity. These models often have billions (and maybe trillions) of parameters, and their transformer-based architectures involve intensive matrix multiplications and attention mechanisms that demand significant processing power. High-end GPUs provide the necessary parallelism and memory bandwidth to handle these operations efficiently. However, in low-GPU compute setups, limited VRAM becomes a critical constraint, as it cannot accommodate the large weights and activations of LLMs, leading to memory overflow or reliance on slower CPU processing.</p><p>The Vicuna LLM used in the original OCTOPI model has a 7 billion (7B) and a 13B parameter variant, which requires 24.7 and 48.0 GiB of VRAM, respectively. End-to-end finetuning, the final stage of training, requires that the full set of parameters be loaded onto our GPU for it to work. Loading every parameter constrains us from offloading our model layers to the CPU or utilizing tools like <a href="https://github.com/lyogavin/airllm">AirLLM</a> that perform layer-by-layer inference, so we must reduce the LLM size directly.</p><h4><strong>Solution Approach: Quantization</strong></h4><p>Quantization is one approach to reducing the size of the LLM that changes the number of bits of each weight in the model layers. Typically, model weights are initialized and trained at 32-bit precision. Using quantization, the number of bits can be reduced to 16, 8, 4, 3, and 2-bit precision. For a 7B parameter model, 16-bit precision will drop its memory size from about 28 to 13.5 GiB at a small accuracy loss in its responses. This can be taken even further, down to about 3.7 GiB at 2-bit precision (Badri, Hicham, and Appu Shaji. “Half-quadratic quantization of large machine learning models.” (2023)). Quantization of the LLM in the OCTOPI model can be done using the <a href="https://huggingface.co/docs/transformers/en/quantization/bitsandbytes">“Bits and Bytes (BnB)”</a> configurations included in the <em>transformers</em> library. To quantize the OCTOPI model’s LLM to 4-bit, we will first create a BnB configuration object as so,</p><pre>bnb_config = BitsAndBytesConfig(<br>                load_in_4bit=True,<br>                bnb_4bit_compute_dtype=torch.float16)</pre><p>and verify that we load in the LLM with our configuration object:</p><pre>llm = AutoModelForCausalLM.from_pretrained(model_path, <br>        device_map=device_map, <br>        offload_folder=configs[&quot;offload_dir&quot;], <br>        quantization_config=bnb_config)</pre><p>That’s it! Now, the LLM in OCTOPI will be small enough to run on your local RTX 4090 with 24GiB of GPU. This change can be propagated through each LLM call in the training and evaluation scripts. However, now that each layer in the LLM uses 4 bits, the encoder, projection layer, and data must also be converted to 4 bits to avoid matrix multiplication errors. Additionally, the end-to-end fine-tuning of our model uses <a href="https://github.com/microsoft/LoRA">low-rank adaptation (LoRA)</a> and is not compatible with quantized models without implementing <a href="https://arxiv.org/abs/2305.14314">QLoRA</a>. To save ourselves the headache of implementing these changes, let’s look at the second approach we can take.</p><h4><strong>Solution Approach: Parameter Reduction</strong></h4><p>The other option we can take to reduce the size of the OCTOPI model is to replace its LLM component entirely. Although this sounds far more difficult than quantization, it is actually very straightforward, with only a few steps and some background on how LLMs are deployed. In the OCTOPI codebase, there is a separate configuration file for all three steps of the training process. In each of these configuration files, there is a place to enter your model&#39;s name:</p><pre>model_type: vicuna-7b</pre><p>This will be used to identify the path to the tokenizer and model weights for your model. In the tactile feature alignment and end-to-end fine-tuning scripts, the tokenizer and weights are specified based on this <em>model_type</em> parameter:</p><pre>if configs[&quot;model_type&quot;] == &quot;vicuna-7b&quot;:<br>        tokenizer_path = &quot;lmsys/vicuna-7b-v1.5&quot;<br>        model_path = &quot;lmsys/vicuna-7b-v1.5&quot;<br>elif configs[&quot;model_type&quot;] == &quot;vicuna-13b&quot;:<br>        tokenizer_path = &quot;lmsys/vicuna-13b-v1.5&quot;<br>        model_path = &quot;lmsys/vicuna-13b-v1.5&quot;</pre><p>The <em>model_path</em> and <em>tokenizer_path</em> variables are Huggingface IDs, not relative paths where the model is stored. Huggingface allows you to bypass downloading the weights locally by importing them from its online repositories directly to the GPU in the following lines:</p><pre>llm = AutoModelForCausalLM.from_pretrained(model_path, <br>        device_map=device_map,  <br>        offload_folder=configs[&quot;offload_dir&quot;])<br><br>tokenizer = AutoTokenizer.from_pretrained(tokenizer_path, padding_side=&quot;left&quot;)</pre><p>The Phi-2 LLM from Microsoft and the Lamma 3.2 1B LLM from Meta are lightweight models known for achieving comparable performance to larger state-of-the-art models and can be loaded into our project. We will add entries for these models as so in each script that loads the models:</p><pre>if configs[&quot;model_type&quot;] == &quot;vicuna-7b&quot;: # Existing Model<br>        tokenizer_path = &quot;lmsys/vicuna-7b-v1.5&quot;<br>        model_path = &quot;lmsys/vicuna-7b-v1.5&quot;<br>elif configs[&quot;model_type&quot;] == &quot;vicuna-13b&quot;: # Existing Model<br>        tokenizer_path = &quot;lmsys/vicuna-13b-v1.5&quot;<br>        model_path = &quot;lmsys/vicuna-13b-v1.5&quot;<br>elif configs[&quot;model_type&quot;] == &quot;phi-2&quot;: # New Model<br>        tokenizer_path = &quot;microsoft/phi-2&quot;<br>        model_path = &quot;microsoft/phi-2&quot;<br>elif configs[&quot;model_type&quot;] == &quot;llama-3.2-1B&quot;: # New Model<br>        tokenizer_path = &quot;meta-llama/Llama-3.2-1B&quot;<br>        model_path = &quot;meta-llama/Llama-3.2-1B&quot;</pre><p>The final step is setting our model type in the the config file:</p><pre>model_type: phi-2</pre><p>or,</p><pre>model_type: llama-3.2-1B</pre><h3><strong>Training</strong></h3><p>OCTOPI leverages the pre-trained weights of the CLIP model and the LLM instead of training the individual model components from scratch. Instead, the weights are fine-tuned by minor adjustments to work more effectively with tactile data. This fine-tuning process takes three steps: encoder fine-tuning, tactile feature alignment, and end-to-end fine-tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yn76knyExPx9JtD_" /><figcaption>Fig. 3) The three-step process for fine-tuning the OCTOPI TVLM | Source: <a href="https://octopi-tactile-lvlm.github.io/">OCTOPI paper</a></figcaption></figure><ul><li>1) We fine-tuned our encoder for 60 epochs with a batch size of 2 and a learning rate of 0.001. The encoder took ≈20 minutes to train on an RTX 4090 and takes 1.2 GiB of VRAM.</li><li>2) We performed tactile alignment with both the Phi-2 and Llama 3.2 — 1B models using our trained encoder for 10000 epochs at a learning rate of 0.0002. Phi-2 alignment took ≈40 minutes, and Llama 3.2–1B took ≈32 minutes.</li><li>3) We trained LoRA for 3000 epochs on the Q and K projection modules in the attention layer of the LLMs with an alpha of 256 and rank of 128. Phi-2 end-to-end finetuning took ≈25 minutes, and Llama 3.2–1B took ≈18 minutes.</li></ul><h3><strong>Evaluation</strong></h3><p>To assess the effectiveness and performance of the models in tactile reasoning, we evaluated the models on new and unseen tactile videos. Zero-shot evaluations ensure that the models can perform on features not present during training, thus proving that the models have not overfitted. We evaluated the models on three categories that capture critical aspects of understanding and interpreting tactile data:</p><ul><li><strong>Property Comparison (PC): </strong>The model is tasked with comparing two tactile videos and deciding which one better exhibits a specific property. This task emulates a scenario where relative comparison is crucial. For example, a robot gripper squeezes two fruits and determines which one is more ripe based on their softness.</li><li><strong>Property Superlative Selection (PSS): </strong>The model must select the tactile video that best exemplifies a given property out of multiple options. This task evaluates the model’s ability to rank tactile interactions and identify extremes. Example: Given three tactile videos of three different fruits, the model must pick which fruit is the ripest based on the levels of softness.</li><li><strong>Property Object Matching (POM): </strong>Match object properties with their corresponding tactile videos. Example: The model is given tactile videos of three separate objects. It knows there is an orange, eraser, and hairbrush and must correctly match the objects to the correlated tactile videos.</li><li><strong>Object Property Description (OPD): </strong>Classify the roughness, hardness, and bumpiness properties of the object in a given tactile video.</li></ul><p>These tasks comprehensively evaluate the models’ capabilities in tactile reasoning from relative comparison to object-property association.</p><h3><strong>Results</strong></h3><p>Below are the evaluation results of the LLama 3.2 — 1B, Phi-2 — 2.7B, Vicuna — 7B, and 13B models. We expanded upon the OPD task in Fig. 5 to identify tactile characteristics most effectively captured by the TVLMs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8yprO1OasJCXGHZqYX7Hg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZtMkfDtXx33qdXM8" /><figcaption>Fig. 4) Performance results of each LLM model for the PC, PSS, and POM tasks. | Source: Own work</figcaption></figure><p><strong>PC Task: </strong>Llama 3.2 performed surprisingly badly at this task and got a lower score than if it had guessed randomly, and Phi-2 narrowly outperformed random guessing.</p><p><strong>PSS Task: </strong>Vicuna-7B again performs the best at identifying extremes in tactile data with a 74.7% success rate. Phi-2 — 2.7B also performs well, with 57.2%, while other models lag significantly behind.</p><p><strong>POM Task: </strong>Object matching seems to be the task with the lowest best score, with Vicuna-7B performing best at 44.4%. Llama 3.2 and Phi-2 performed exceptionally poorly at this task.</p><p><strong>OPD Task:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BsPbhN556dt9VCYBLivGIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C5z5-uP7nXwQpWbA" /><figcaption>Fig. 5) Performance results of each LLM model for the OPD task. | Source: Own work</figcaption></figure><p>Vicuna — 7B outperformed all the other models except for Vicuna — 13B on roughness classification, but the Llama 3.2–1B managed to fare surprisingly well. The Phi-2 model lags behind the other models despite having more parameters than the Llama 3.2 model, which might indicate that its initial training data was not as beneficial to the classification tasks.</p><h3>Conclusion</h3><p>We presented approaches for modifying the OCTOPI VLM to run on a local GPU, where the full model comes out to around &lt;8GiB at full 32-bit precision for the Llama 3.2 model and visual encoder. The Phi-2–2.7B LLM performed overall better than the Llama 1B model for the PC, PSS, and POM tasks but still significantly fell short of being practical. The Llama 3.2 model performed comparably to the full-sized Vicuna models for the OPD tasks, suggesting that Phi-2’s pre-trained knowledge was more useful for the reasoning tasks than the classification tasks and vice versa for Llama 3.2.</p><p>By showing how to make the OCTOPI model compatible with consumer-grade GPUs, we hope to encourage further experimentation and innovation in TVLMs. Future work will focus on optimizing these models for better performance in low-resource settings and exploring new architectures designed specifically for tactile perception.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f7ac082f306" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/bringing-tactile-vlms-to-local-gpus-8f7ac082f306">Bringing Tactile VLMs to Local GPUs</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paper Review: Octopi: Object Property Reasoning with Large Tactile-Language Models]]></title>
            <link>https://medium.com/correll-lab/paper-review-octopi-object-property-reasoning-with-large-tactile-language-models-0f856dcdf8be?source=rss-ec8758ce8022------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f856dcdf8be</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[tactile-sensing]]></category>
            <dc:creator><![CDATA[Carson Kohlbrenner]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 07:45:57 GMT</pubDate>
            <atom:updated>2024-11-05T07:45:57.710Z</atom:updated>
            <content:encoded><![CDATA[<h4>Enabling robots to understand and reason through touch!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*3wwboBqntPin2jMdRt1Q3g.gif" /><figcaption>Fig. 1) Using OCTOPI to reason which avocado is ripest through touch. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><p>Imagine moving through the world with hands that feel nothing — a delicate feather or a rough piece of sandpaper offering you no difference, no clue, no response. Without touch, you couldn’t feel the subtle give of a ripe avocado or sense when it’s still firm and ready to ripen just a day or two more. For humans, touch is central to our ability to make choices and navigate our environment. It’s an instinctive guide, a sense we don’t even realize is at play in everything we do, from holding a child’s hand to gauging the heat of a coffee cup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uWsOgGvIu5luk3MjYvnWJw.png" /><figcaption>Fig. 2) Reasoning about the winning car’s color is easy through a picture, but it is nearly impossible to gauge the temperature of a coffee mug in this setting without feeling it. | Source: Own Work.</figcaption></figure><p>But for robots, this sensory experience is still largely absent. To them, the world remains an enigmatic, intangible space — one they are “blind” to, missing the endless cues and textures that make the world around us. The development of tactile perception in robotics isn’t just about adding a sense; it’s about awakening a new dimension of intelligence and reasoning.</p><p>In this review, we will be covering “Octopi: Object Property Reasoning with Large Tactile-Language Models” by S. Yu et al. to learn how robots can reason through the sense of touch, as well as other related concepts and papers along the way. The code and dataset can be accessed from the original paper:</p><p><a href="https://doi.org/10.48550/arXiv.2405.02794">Octopi: Object Property Reasoning with Large Tactile-Language Models</a></p><h3><strong>Outline</strong></h3><ol><li><em>PhysiClear Dataset — </em>A diverse tactile dataset with object property descriptions.</li><li><em>OCTOPI Model — </em>A tactile extension to pre-trained large language models (LLMs).</li><li><em>Model Tuning</em> — Alignment of tactile data with large vision language models (LVLMs).</li><li><em>Training — </em>The metrics under which the model is tuned.</li><li><em>Results — </em>Performance of OCTOPI for various reasoning and classification tasks.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iwNIwOIX5XXAoYcyPVfR0Q.png" /><figcaption>Fig. 3) Octopi paper contributions and outline. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><h3>PhysiClear Dataset</h3><p>Vision-based tactile sensors are the most widely used tactile sensors in robotics research. These sensors use a camera to detect and capture the materials, properties, and dynamics of objects in contact with their deformable membrane. The cameras embedded within vision-based tactile sensors can record videos to capture continuous tactile data, referred to hereafter as tactile videos. More information about GelSight, the tactile sensor used for this study, can be found here:</p><p><a href="https://doi.org/10.3390/s17122762">GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force</a></p><p>The PhysiClear dataset is designed to capture three main properties primarily determined through touch:<em> hardness</em>, <em>roughness</em>, and <em>bumpiness</em>. The idea is that these three properties will contain enough information to help decision-making and reasoning about objects being touched. The dataset is comprised of 408 tactile videos of 74 different everyday objects that were manually annotated with the aforementioned properties.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0y9JbNgII1wKUzCtbE-QQ.png" /><figcaption>Tab. 1) Property annotations for each object in a tactile video. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><p>It should be noted that PhysiClear is not the only existing tactile dataset. This dataset focuses primarily on the three property categories instead of the dataset size. Here is a high-level overview of how this dataset compares to others available:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uBKREBFusJGW0CIRp2mhfA.png" /><figcaption>Tab. 2) Tactile dataset comparisons. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><h3>OCTOPI Model</h3><p>OCTOPI is a pre-trained LVLM fine-tuned on tactile videos. The OCTOPI model works by taking tasks and tactile videos from the user and responding in written language. It does this by first explaining tactile videos by object properties, then projecting the descriptions within the user-specified tasks. More specifically, the model projects the image embeddings of tactile videos into the word embedding of an LLM so that the LLM can understand tactile videos and apply its commonsense knowledge to reasoning. Three main components make the OCTOPI model work: the <em>tactile image encoder</em>, the <em>projection module</em>, and the <em>LLM</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ogZy3fBdezGwKB9s3YFuxw.png" /><figcaption>Fig. 4) OCTOPI model architecture with an example task. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><h4>The Tactile Video Encoder</h4><p>The tactile image encoder uses contrastive image-language pre-training (CLIP) to match features from an image embedding to a text embedding. I do a deep dive and tutorial into how to train CLIP with tactile images in my previous article found here:</p><p><a href="https://medium.com/correll-lab/training-clip-to-recognize-objects-with-tactile-sensors-8e4f480027a3">Training CLIP to Recognize Objects with Tactile Sensors</a></p><p>OCTOPI uses the base parameters from CLIP’s ViT-L/14 for the image encoder and is fine-tuned on the PhysiClear dataset. A difference between my previous tutorial and OCTOPI’s tactile encoder is that tactile images are sampled from videos instead of images using the ViFi-CLIP architecture to learn contextual features spread out through videos. This works using temporal pooling to get a video-wide understanding. ViFi-CLIP was introduced in <a href="https://doi.org/10.48550/arXiv.2212.03640">“Fine-tuned CLIP Models are Efficient Video Learners”</a> by Rasheed et al.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6RPNGzfNqMRSyhz5Lb8vg.png" /><figcaption>Fig. 5) ViFi-CLIP method of video-wide feature extraction used in OCTOPI. | Source: <a href="https://doi.org/10.48550/arXiv.2212.03640">ViFi-CLIP Paper</a></figcaption></figure><p><a href="https://doi.org/10.48550/arXiv.2212.03640">Fine-tuned CLIP Models are Efficient Video Learners</a></p><h4>The Projection Module</h4><p>The projection module takes the text embedding from the tactile image encoder and projects it into the text embedding of the LLM. The dimensions and shape of the text embeddings are inherently different, so a projection module is a learned neural network that is trained to transfer the features extracted from the encoder to the LLM with as little loss as possible. This is done via two fully connected linear layers with intermediate Gaussian error linear unit (GELU) activation to capture non-linearities in the projections.</p><h4><strong>The LLM</strong></h4><p>The open-source LLM used in this study was <a href="https://lmsys.org/blog/2023-03-30-vicuna/">Vicuna v1.5</a>. This model is attractive for research because it can be run locally with publically available parameters. In this paper, variants of OCOTOPI include using the 7 billion and 13 billion parameter versions of Vicuna v1.5 to evaluate model size on tactile reasoning. Although this model is still available as of the publication date, it should be noted that many open-source LLMs are available that could replace the Vicuna model used in OCTOPI with a little bit of modification.</p><h3>Model Tuning</h3><p>Tuning is a necessary component to make the OCTOPI model understand tactile videos effectively. The idea is that the base parameters of the encoder contain many general patterns that it could use when looking at tactile videos, but they should have less emphasis than patterns found to be frequently contained in tactile videos through tuning. The model is tuned using a three-step process that first fine-tunes the encoder, then aligns tactile features in the projection module, and finally end-to-end fine-tunes the LLM and projection module to be more similar in responses to the encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/0*-L6B03gPM1Uo9_4z.png" /><figcaption>Fig. 7) Tuning steps to align tactile features to natural language. The fire symbol represents tuning and the snowflake symbol means the model is frozen. | Source: <a href="https://octopi-tactile-lvlm.github.io/">OCTOPI GitHub</a></figcaption></figure><h4>Encoder Fine-Tuning</h4><p>Three extra classification heads were added to the tactile encoder to represent the <em>hardness</em>, <em>roughness</em>, and <em>bumpiness, </em>properties during fine-tuning. These classification heads were added to establish how cross-entropy loss would be evaluated during training. Visual Prompt Tuning (VPT) was used to tune the encoder heads and prompts to learn useful features from the tactile videos. More information on how VPT works can be found here:</p><p><a href="https://doi.org/10.48550/arXiv.2203.12119">Visual Prompt Tuning</a></p><h4><strong>End-to-End Fine-Tuning</strong></h4><p>To improve the coherence between the LLM’s responses and the tactile language annotations, low-rank adaption (LoRA) parameters are introduced to the LLM. Fine-tuning the entire weight matrices of an LLM can be expensive in terms of both computational resources and memory. LoRA introduces an efficient approach by only updating a low-rank subspace of these matrices during training. More information on LoRA can be found in its original paper:</p><p><a href="https://doi.org/10.48550/arXiv.2106.09685">LoRA: Low-Rank Adaptation of Large Language Models</a></p><p>The LLM, word embedding, and projection module can then be trained on a series of tasks while the tactile encoder is held frozen. The tasks used to obtain tactile feature alignment are described in the next section.</p><h3>Training</h3><p>The OCTOPI model was trained using a series of five tasks designed to strengthen property and material understanding of objects being touched. Each of the tasks consists of a curated question-and-answer pair that addresses a specific type of tactile understanding. The idea is that the OCTOPI model will be fine-tuned to provide answers to new tasks with unseen tactile videos under the same understanding and format as the premade question-and-answer pairs. Here is a brief description of the five tasks, a very in-depth breakdown of each is found in the original paper:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5eggeyf2ofuZoQtib20WkA.png" /><figcaption>Tab. 3) Tasks used to train and evaluate the OCTOPI model.</figcaption></figure><ol><li><strong>Object Property Description (OPD)</strong> —<strong> </strong>Describe a given tactile video in terms of its <em>hardness</em>, <em>roughness</em>, and <em>bumpiness.</em></li><li><strong>Property Comparison (PC)</strong> — Compare two given tactile videos for a specific property.</li><li><strong>Property Superlative Selection (PSS) </strong>— Pick the tactile video with the most of a given property out of three videos.</li><li><strong>Property Object Matching (POM)</strong> — Match the labels of three objects to three given tactile videos.</li><li><strong>Property Scenario Reasoning (PSR)</strong> — Reason about a given tactile video for a specific use case that involves commonsense knowledge.</li></ol><p>PSR is exciting compared to the other tasks because it uses a wider array of pattern recognition stored within the LLM’s pre-trained parameters to reason about tactile videos. This allows the LLM to use more knowledge about the world than what was present in the PhysiClear dataset. Here is an example of PSR:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/0*7HtSJ_DYYjdaNwKG.png" /><figcaption>Fig. 8) Example of PSR where OCTOPI takes tactile videos from the handle and bristles of a toothbrush and reasons which is which. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><h3>Results</h3><p>The OCTOPI model was evaluated on two main subjects, physical understanding and scenario reasoning. Physical understanding refers to how well the model can perform OPD, PC, PSS, and POM on new and unseen tactile videos. Scenario reasoning refers to how well the model can reason about objects based on their predicted physical properties, evaluated using PSR.</p><ul><li><strong>Physical Understanding: </strong>The model performed PC, PSS, and POM with significantly better accuracy than randomly guessing, but was still incorrect over 50% of the time with PC and POM. The authors of the original paper found that having the LLM first describe objects (OPD) before performing the tasks gave it a major boost for PSS and POM, but only a minor boost for PC. OCTOPI-13b performed approximately 10% better in all tests than OCTOPI-7b, suggesting the LLM size plays a large role in model accuracy.</li><li><strong>Scenario Reasoning: </strong>The model was given three scenario questions for PSR and judged by how well it identified the target property. For example, if the model was asked to choose the most easily grippable object when wet and slippery, the model was judged if it could select objects based on higher bumpiness and roughness predictions. The 7b model outperformed the 13b model in this evaluation and achieved about 20% higher accuracy than randomly guessing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*WMDTOTFITTxGv7hxvMi3MA.png" /><figcaption>Tab. 4) Physical reasoning results. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*7Qsf8CSlc_8ogCZRNs3TrA.png" /><figcaption>Tab. 5) Scenario reasoning results. | Source: <a href="https://arxiv.org/abs/2405.02794">Octopi Paper</a></figcaption></figure><h3>Conclusion</h3><p>We reviewed OCTOPI and showed how tactile reasoning can be achieved with LVLM models fine-tuned on tactile videos captured by GelSight sensors. The PhysiClear dataset was used to teach the OCTOPI model patterns contained within the <em>hardness</em>, <em>roughness</em>, and<em> bumpiness</em> properties of tactile videos.</p><p>Overall, the model performed… ok. While it was a big step forward in enabling robots to reason through the sense of touch, questionable assumptions were made in the training and evaluation process to judge their performance. It is unclear to me why hardness, roughness, and bumpiness were chosen to be the only labels to describe the contact properties of objects other than to be a proof of concept that this method can work. In the future, OCTOPI or other LVLMs like it may benefit gloriously from scaling the dataset to include more objects and richer annotations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f856dcdf8be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/paper-review-octopi-object-property-reasoning-with-large-tactile-language-models-0f856dcdf8be">Paper Review: Octopi: Object Property Reasoning with Large Tactile-Language Models</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training CLIP to Recognize Objects with Tactile Sensors]]></title>
            <link>https://medium.com/correll-lab/training-clip-to-recognize-objects-with-tactile-sensors-8e4f480027a3?source=rss-ec8758ce8022------2</link>
            <guid isPermaLink="false">https://medium.com/p/8e4f480027a3</guid>
            <category><![CDATA[tactile]]></category>
            <category><![CDATA[object-recognition]]></category>
            <category><![CDATA[gelsight]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Carson Kohlbrenner]]></dc:creator>
            <pubDate>Sun, 06 Oct 2024 15:27:06 GMT</pubDate>
            <atom:updated>2024-10-08T23:32:18.514Z</atom:updated>
            <content:encoded><![CDATA[<h4>An Open World Approach for Tactile Classification on the GelSight Sim2Real Dataset</h4><p>Tactile sensors are designed to give robots the sense of touch so that they can feel objects in their environment. In the past, convolutional neural networks (CNNs) were trained on large labeled datasets to recognize specific objects a tactile sensor is in contact with. Contrastive Language-Image Pre-training (CLIP) on the other hand is a more robust architecture for zero-shot classification on large unstructured datasets that will be explored in this tutorial as an alternate classification model to CNNs.</p><p>The code for this tutorial is available on Google Colab and can be <a href="https://colab.research.google.com/drive/1iYD2WDOGnPQSRmr0qXOLSl3eMh7S1IXt?usp=sharing">accessed here</a>. In this tutorial, we will modify the code from Matt Nguyen’s <a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP from Scratch</a> tutorial to train a CLIP embedding of tactile images and their labels. We will use this embedding for zero-shot classification and compare its accuracy to CNNs trained on the same data. The publicly available tactile dataset used in this tutorial was collected by <a href="https://ieeexplore.ieee.org/document/9756938">Chen et al. (2022)</a> and contains a mix of real and simulated data.</p><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></p><h4>Experimental Setup</h4><p>The dataset is comprised of 16,600 tactile images from a custom GelSight optical tactile sensor in contact with 18 unique shapes in random poses. In the <a href="https://ieeexplore.ieee.org/document/9756938">dataset paper</a>, cyclical generative adversarial networks (cGANs) were used to generate an additional 16,600 tactile images. Fig. 1 shows the experimental setup for collecting data and Fig. 2 shows examples of some of the real and simulated tactile images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VhHs6pGtcSVQfi8Svoas1A.png" /><figcaption>Fig. 1) Experimental setup for the tactile dataset. Image: <a href="https://ieeexplore.ieee.org/document/9756938">Dataset Paper</a>.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yr3q7JA-VVwWc4wOF4oOAw.png" /><figcaption>Fig. 2) Samples of each object captured by the real and simulated sensors. Images: <a href="https://github.com/RVSATHU/GelSight-Sim2Real?tab=readme-ov-file">Tactile Dataset</a>.</figcaption></figure><h4><strong>Import Libraries and Modules</strong></h4><pre># From Building CLIP from Scratch<br>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision.transforms as T<br>from torch.utils.data import Dataset, DataLoader<br>from datasets import load_dataset<br>import matplotlib.pyplot as plt<br>import numpy as np<br><br># Additional Libraries<br>import io<br>from PIL import Image<br>from IPython.display import clear_output</pre><p>In addition to all the necessary libraries from <a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP from Scratch</a>, we will also need to import io and PIL for handling the image data in the dataset. The <em>clear_output </em>function is also imported for displaying the loss curve cleanly during training.</p><pre>import io<br>from PIL import Image<br>from IPython.display import clear_output</pre><h4>Loading the Data</h4><p>The data is loaded in as a <em>Dataset</em> class so it can be passed into a <em>DataLoader</em> for training. This class will handle storing and augmenting the tactile images.</p><pre>class TactileDataset(Dataset):<br>    def __init__(self, train=True):<br>        <br>        self.dataset = load_dataset(&quot;carsonk10/tactile_data_full&quot;) # For both real and sim data<br>      <br>        self.transform = T.Compose([<br>            T.Resize((140, 140)),  # Resize the image to 224x224 pixels<br>            T.RandomHorizontalFlip(),<br>            T.RandomVerticalFlip(),<br>            T.RandomCrop((128, 128), pad_if_needed=True),<br>            T.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2),<br>            T.ToTensor()           # Convert the image to a PyTorch tensor<br>        ])<br>        self.split = &quot;train&quot; if train else &quot;test&quot;<br><br>        self.captions = {0:&quot;cone&quot;,<br>                         1:&quot;cylinder&quot;,<br>                         2:&quot;cylinder shell&quot;,<br>                         3:&quot;side of cylinder&quot;, <br>                         4:&quot;hole&quot;,<br>                         5:&quot;dots&quot;,<br>                         6:&quot;hexagon&quot;,<br>                         7:&quot;large sphere&quot;,<br>                         8:&quot;line&quot;,<br>                         9:&quot;moon&quot;,<br>                         10:&quot;pacman&quot;,<br>                         11:&quot;parallel lines&quot;,<br>                         12:&quot;prism&quot;,<br>                         13:&quot;random shape&quot;,<br>                         14:&quot;small sphere&quot;,<br>                         15:&quot;torus&quot;,<br>                         16:&quot;triangle&quot;,<br>                         17:&quot;wave&quot;}<br><br>    def __len__(self):<br>        return len(self.dataset[self.split])<br>    <br>    def __getitem__(self, i):<br>        img = self.dataset[self.split][i][&quot;image&quot;]<br>        img = Image.open(io.BytesIO(img))<br>        img = self.transform(img)<br><br>        label = self.dataset[self.split][i][&quot;label&quot;]<br>        caption = self.captions[label]<br>        cap, mask = tokenizer(caption)<br>        mask = mask.repeat(len(mask), 1)<br><br>        return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask, &quot;label&quot;: label}</pre><p>The dataset is first imported from Hugging Face,</p><pre>self.dataset = load_dataset(&quot;carsonk10/tactile_data_full&quot;) # For both real and sim data</pre><p>then separated into a train and test split that prevents tactile images from being used in both training and testing. There are a total of 2400 tactile images (1200 real and 1200 sim) for each object shape in contact. The training set contains 2000 tactile images that were randomly selected and the test set contains the remaining 400.</p><pre>self.split = &quot;train&quot; if train else &quot;test&quot;</pre><h4><strong>Image Augmentation</strong></h4><p>The original dataset contains consistent patterns that vary slightly throughout the tactile images such as the background pattern and color orientation. The goal is for the model to only pick up on the patterns associated with the shape of the objects in contact which can be isolated using image augmentation. Image augmentation is performed in this case by randomly applying transformations, cropping, and color adjustments to an image. The original images are 320x320 pixels in size. For faster computation, the images are reduced to 64x64 during augmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UJhal6nj7H5otXCFQ8hCsA.png" /><figcaption>Fig. 3) Image augmentation includes a random combination of transformations, cropping, and color jitter to diversify the dataset. Image: Own work.</figcaption></figure><pre>self.transform = T.Compose([<br>            T.Resize((140, 140)),<br>            T.RandomHorizontalFlip(),<br>            T.RandomVerticalFlip(),<br>            T.RandomCrop((128, 128), pad_if_needed=True),<br>            T.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.2),<br>            T.ToTensor()           # Convert the image to a PyTorch tensor<br>])</pre><p>The <em>self.transform</em> function is called every time an image is accessed through the <em>__get_item__ </em>function:</p><pre>def __getitem__(self, i):<br>        img = self.dataset[self.split][i][&quot;image&quot;]<br>        img = Image.open(io.BytesIO(img))<br>        img = self.transform(img)<br><br>        label = self.dataset[self.split][i][&quot;label&quot;]<br>        caption = self.captions[label]<br>        cap, mask = tokenizer(caption)<br>        mask = mask.repeat(len(mask), 1)<br><br>        return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask, &quot;label&quot;: label}</pre><h4>Training</h4><p>After the data loader is set up, we can train our model. This will create a new CLIP instantiation and then minimize the cross-entropy loss of the image and text embeddings. The loss is plotted during training to ensure the model is learning. I found that approx. 1–2 hours of training on an RTX 4060 GPU is enough time to at least get 50% zero-shot accuracy with the default hyperparameters in the code.</p><pre>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br><br>optimizer = optim.Adam(model.parameters(), lr=lr)<br><br>best_loss = np.inf<br>losses = []<br>f = plt.figure()<br>for epoch in range(epochs):<br>    for i, data in enumerate(train_loader, 0):<br>        img, cap, mask = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device), data[&quot;mask&quot;].to(device)<br>        loss = model(img,cap,mask)<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>    # Saves model if it performed better than the previous best<br>    print(f&quot;Epoch [{epoch+1}/{epochs}], Batch Loss: {loss.item():.3f}&quot;)<br><br>    # Save the loss<br>    losses.append(loss.item())<br>    if loss.item() &lt;= best_loss:<br>        best_loss = loss.item()<br>        torch.save(model.state_dict(), f&quot;models/{save_name}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*MXVnJ7iKQ9bRrva1Ogug0w.png" /><figcaption>Fig. 4) Cross entropy loss over a training period of 1000 epochs. Image: Own work.</figcaption></figure><h4>Zero-Shot Classification</h4><p>The trained model is now ready for classification.</p><pre># Loading Best Model<br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br>model.load_state_dict(torch.load(f&quot;models/{model_name}&quot;, map_location=device))<br><br>class_names = list(train_set.captions.values())<br><br>text = torch.stack([tokenizer(x)[0] for x in class_names]).to(device)<br>mask = torch.stack([tokenizer(x)[1] for x in class_names])<br>mask = mask.repeat(1,len(mask[0])).reshape(len(mask),len(mask[0]),len(mask[0])).to(device)<br><br>idx = np.random.randint(0,len(test_set))<br><br>img = test_set[idx][&quot;image&quot;][None,:]<br>plt.figure(figsize=(3,3))<br>plt.imshow(  img[0].permute(1, 2, 0)  ,cmap=&quot;gray&quot;)<br>plt.title(tokenizer(test_set[idx][&quot;caption&quot;], encode=False, mask=test_set[idx][&quot;mask&quot;][0])[0])<br>plt.show()<br>img = img.to(device)<br>with torch.no_grad():<br>  image_features = model.image_encoder(img)<br>  text_features = model.text_encoder(text, mask=mask)<br><br><br>image_features /= image_features.norm(dim=-1, keepdim=True)<br>text_features /= text_features.norm(dim=-1, keepdim=True)<br>similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)<br>values, indices = similarity[0].topk(5)<br><br># Print the result<br>print(&quot;\nTop predictions:\n&quot;)<br>for value, index in zip(values, indices):<br>    print(f&quot;{class_names[int(index)]:&gt;16s}: {100 * value.item():.2f}%&quot;)</pre><p>First, the class names are tokenized and an image is sampled from the test set:</p><pre>class_names = list(train_set.captions.values())<br><br>text = torch.stack([tokenizer(x)[0] for x in class_names]).to(device)<br>mask = torch.stack([tokenizer(x)[1] for x in class_names])<br>mask = mask.repeat(1,len(mask[0])).reshape(len(mask),len(mask[0]),len(mask[0])).to(device)<br><br>idx = np.random.randint(0,len(test_set))</pre><p>The text and image embeddings are then retrieved from the trained model for this sampled image:</p><pre>image_features = model.image_encoder(img)<br>text_features = model.text_encoder(text, mask=mask)</pre><p>Finally, the softmax of the cosine similarity of the embeddings is used to compare the image to all the possible captions. The top five captions with the highest similarity value are displayed as the CLIP model’s best prediction for the image caption:</p><pre>image_features /= image_features.norm(dim=-1, keepdim=True)<br>text_features /= text_features.norm(dim=-1, keepdim=True)<br>similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)<br>values, indices = similarity[0].topk(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*q6ieruEyMGqsKZJPqSqnRw.png" /><figcaption>Fig. 5) Zero-shot classification of a sampled image from the test split. Image: Own Work.</figcaption></figure><h4>Evaluation</h4><p>We can evaluate the performance of our model by getting the best zero-shot prediction from every image in our test set and compiling it in a confusion matrix. Fig. 6 shows the confusion matrix and parameters used for the highest-performing model I was able to train on this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*DDxOS6bwPWpiNUu1APa60g.png" /><figcaption>Fig. 6) Confusion matrix for the model trained with both real and simulated datasets. Image: Own work.</figcaption></figure><p>The dataset contained a lot of circular objects that look similar in the tactile images (cylinder, cylinder shell, moon, and pacman) and the model had the most trouble identifying the difference between them over any other objects.</p><p>The original paper using this dataset performed Sim2Real and Real2Sim classification with approximately 97% accuracy using CNNs. It is well known that CLIP undeperforms CNNs for smaller datasets, so using a pre-trained image/text encoder or fine-tuning a pre-trained CLIP model is likely to meet or exceed CNN performance. See for example:</p><p><a href="https://medium.com/correll-lab/extending-clip-to-lit-ee67f1c89c3b">Extending CLIP to LiT</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8e4f480027a3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/training-clip-to-recognize-objects-with-tactile-sensors-8e4f480027a3">Training CLIP to Recognize Objects with Tactile Sensors</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>