<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luc Nguyen on Medium]]></title>
        <description><![CDATA[Stories by Luc Nguyen on Medium]]></description>
        <link>https://medium.com/@lucnguyen_61589?source=rss-aee34c8a9754------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hiqzz2i5lVXUrYC7Z4SJRg@2x.jpeg</url>
            <title>Stories by Luc Nguyen on Medium</title>
            <link>https://medium.com/@lucnguyen_61589?source=rss-aee34c8a9754------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:41:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lucnguyen_61589/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Joystick to Wheels: Tracing One Command Through a Real ROS 2 Robot]]></title>
            <link>https://medium.com/@lucnguyen_61589/from-joystick-to-wheels-tracing-one-command-through-a-real-ros-2-robot-b1a15d1a5839?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/b1a15d1a5839</guid>
            <category><![CDATA[roses]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[physical-ai]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 06:29:55 GMT</pubDate>
            <atom:updated>2026-09-28T06:29:55.916Z</atom:updated>
            <content:encoded><![CDATA[<h4>What actually happens between pushing a stick and turning four wheels — Physical AI 101, Part 1 of 4</h4><p>AI can read images, write code, and make plans. But what happens when it has to do something in the real world, not just answer on a screen?</p><p>Tell a robot “Go to the next room,” and an LLM will understand the sentence without trouble. The hard part is what comes after. How does a sentence sitting in memory end up turning a real wheel? And once the wheels turn, how does the robot know it went the right way?</p><p>That gap is where robotics stopped feeling like software AI to me. So instead of starting with something big like VLAs or humanoids, I started with something much simpler: <strong>the joystick of a ROSMASTER M3 Pro</strong>.</p><p>In this post, I’ll follow one stick push through /joy, /cmd_vel, micro-ROS, and an STM32 to the motors, and then follow the measurements back up. Along the way, you&#39;ll see why the robot splits deciding from executing and why it needs feedback. You&#39;ll also see one trap that caught me: seeing a command in ROS doesn&#39;t mean it reached the motors.</p><p>It’s written for people who know AI or software but are new to robots. If you’ve already driven a mobile robot with ROS 2, you won’t find much new here.</p><h3>My setup</h3><p>Here’s the hardware I used:</p><ul><li><strong>Robot:</strong> Yahboom ROSMASTER M3 Pro (4 mecanum wheels + 6-axis arm)</li><li><strong>Main board:</strong> NVIDIA Jetson Orin &lt;TODO: NANO SUPER / NX SUPER&gt;</li><li><strong>Low-level controller:</strong> STM32 (firmware with a micro-ROS client)</li><li><strong>ROS:</strong> ROS 2 Humble</li><li><strong>Input device:</strong> gamepad-style joystick (shows up as /dev/input/js0)</li><li><strong>Jetson ⇔ STM32:</strong> serial connection</li></ul><p>The animation below also shows the order in which the signal passes through each part.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*h6PjWjoN4Jvx92gcASqVNQ.gif" /><figcaption><em>The Jetson Orin decides, the STM32 drives the motors, and a serial link connects the two.</em></figcaption></figure><p><em>This post is based on my own robot’s setup as of .</em></p><h3>The big picture: push the stick, and the robot moves left</h3><p>Written down, it sounds like nothing special. A toy car can do it. But while watching the robot move, I started to wonder:</p><blockquote><strong><em>Between the moment I push the stick and the moment the four wheels turn, what is actually happening inside?</em></strong></blockquote><p>At first I assumed it was simple, something like Joystick → Robot. It isn&#39;t. Between the stick and the wheels there&#39;s a long chain, and the signal crosses it one hop at a time before the robot slides sideways.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*lHEt0BH16pCus7EYTfh_tA.gif" /><figcaption><em>One stick push crosses five hops before a single wheel turns.</em></figcaption></figure><p>And that’s not the end. After the wheels turn, measurements from the encoders and the IMU have to travel back up to the Jetson. The robot has to do more than receive a command and drive: it also needs to know <strong>how it actually moved</strong>.</p><p>That’s the most important idea in this post, so let’s take the chain one hop at a time.</p><h3>Step 1: The joystick is not controlling the wheels yet</h3><p>The joystick is connected to the <strong>Jetson Orin</strong> on the robot.</p><blockquote><strong><em>Note — What is a Jetson?</em></strong><em> Think of it as a small Linux computer mounted on the robot. It has a CPU and a GPU, and it can run heavy workloads such as camera processing, computer vision, SLAM, navigation, and AI models.</em></blockquote><p>When you plug in the joystick, Linux sees it as an input device; on my robot, it shows up as /dev/input/js0. Pushing the stick changes the values from this device.</p><p>None of this touches the motors yet. Nothing moves forward until a program on the Jetson reads the joystick data and hands it to ROS.</p><h3>So what is ROS?</h3><p>ROS comes with a lot of vocabulary, such as <strong>nodes, topics, publish, and subscribe</strong>, but you don’t need all of it to follow this post. Start with a simpler problem instead: a robot runs many programs at the same time.</p><ul><li>A program that reads the joystick</li><li>A program that reads the LiDAR</li><li>A program that processes camera images</li><li>A program that estimates where the robot is</li><li>A program that plans paths</li><li>A program that sends commands to the hardware</li></ul><p>These programs need a way to exchange data, and ROS 2 solves most of that problem.</p><blockquote><strong><em>In short, ROS 2 is the foundation that lets the programs inside a robot exchange data.</em></strong></blockquote><p>Each program is called a <strong>node</strong>, and the channels that data flows through are called <strong>topics</strong>. For example, the node that reads the joystick publishes its data to a topic called /joy.</p><h3>Checking it: what’s inside /joy</h3><p>Here’s how to look:</p><pre>ros2 topic echo /joy</pre><p>Move the stick while this is running, and the axes values change. Press a button, and its buttons value flips from 0 to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*oEaytgiHQ86dMG3ZSQZhqw.gif" /><figcaption><em>Each stick axis and each button maps to one value in the </em><em>/joy message.</em></figcaption></figure><blockquote><strong><em>Note:</em></strong><em> The output above is an example. The number and order of </em><em>axes and </em><em>buttons depend on your controller.</em></blockquote><p>So the Jetson can read the joystick. What it still doesn’t know is <em>how fast</em> to drive. All it knows is where the stick is.</p><h3>Step 2: From /joy to /cmd_vel</h3><p>Next, a program has to translate the stick position into <em>speed</em>. On the ROSMASTER, the /joy_ctrl node does this job. For example:</p><ul><li><strong>Push slightly forward</strong> → drive forward at 0.2 m/s</li><li><strong>Push in a turning direction</strong> → rotate at 0.4 rad/s</li></ul><p>The result is published to a topic called /cmd_vel, short for <strong>command velocity</strong>. It&#39;s one of the central topics on any mobile robot. Its message type is geometry_msgs/msg/Twist, and three fields matter most:</p><ul><li><strong>linear.x</strong> — forward / backward</li><li><strong>linear.y</strong> — left / right (sideways)</li><li><strong>angular.z</strong> — rotation</li></ul><p>Here’s where it gets interesting: the layers below /cmd_vel <strong>don&#39;t need to know where the command came from</strong>. Today a person sends it with a joystick. Tomorrow Nav2 might send it, and someday an AI agent. The decision-maker changes, but everything below /cmd_vel stays the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*w026r-sydwA0gOSbhiKe7g.gif" /><figcaption><em>The source of </em><em>/cmd_vel changes; the path below it never does.</em></figcaption></figure><p>In other words, the joystick is only playing the role of <em>decision-maker</em>. The rest of the robot doesn’t care whether that decision came from a person or from an AI. All it needs is the answer to <strong>“How should I move right now?”</strong>, and that’s where robotics starts to connect to Physical AI.</p><h3>Step 3: We have /cmd_vel, so why aren&#39;t the wheels turning?</h3><p>Because /cmd_vel is still only data living on the Jetson. The Jetson doesn&#39;t take the value linear.x = 0.2 and push current into a motor itself. That job belongs to another chip downstream: the <strong>STM32</strong>.</p><p>If the Jetson is the “brain,” the STM32 is closer to the “hands and feet.” It reads the encoders, controls the motors, and runs the speed control loop. Put simply:</p><blockquote><strong><em>The Jetson decides </em>what<em> the robot should do.</em></strong><em> </em><strong><em>The STM32 handles </em>how<em> to make the motors do it.</em></strong></blockquote><p>Why not let the Jetson drive the motors directly? A motor control loop has to run on a tight, fixed schedule, and a Linux computer busy with cameras, SLAM, and AI models can’t promise that. Nobody wants an LLM that sometimes thinks for several seconds to also be in charge of keeping motor speed steady. Instead, the timing-critical work goes to a microcontroller that does nothing else. The cost is one more link in the chain, and, as you’ll see later, one more place for things to break.</p><h3>How does the Jetson send commands to the STM32?</h3><p>This is where <strong>micro-ROS</strong> comes in. The STM32 doesn’t run Linux or a full ROS 2 stack. Instead, its firmware includes a <strong>micro-ROS client</strong>.</p><p>On the Jetson side, you start micro_ros_agent, which bridges the two. The Agent converts ROS 2 messages into small packets for the serial link, and micro-ROS on the STM32 receives them and drives the motors. Measurements such as /odom_raw come back the other way along the same path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*D6FPRq-Pxo2vtnhGoHCffw.gif" /><figcaption><em>micro_ros_agent is the bridge: commands go down the serial link, and measurements come back up.</em></figcaption></figure><p>Here’s the command I use to start it:</p><pre>micro_ros_agent serial \<br>  --dev /dev/myserial \<br>  -b 2000000</pre><p>The command has now gone <strong>from a Linux program to a chip that controls the hardware directly</strong>. We’re close to the point where numbers turn into motion.</p><h3>Step 4: To “move left,” how should the four wheels turn?</h3><p>The ROSMASTER M3 Pro uses four <strong>mecanum wheels</strong>. Small rollers set at an angle around each rim let the robot move forward, backward, and rotate, and also <strong>slide sideways</strong>. It’s fun to watch.</p><p>But it gives the software one more problem to solve. Say /cmd_vel says &quot;move sideways to the left.&quot; Neither the front-left wheel nor the front-right wheel knows what &quot;move left&quot; means. Each motor only understands <strong>which direction to spin, and how fast</strong>. So the system has to work out:</p><blockquote><strong><em>To move the whole robot left, how should each of the four wheels turn?</em></strong></blockquote><p>This is called <strong>mecanum wheel inverse kinematics</strong>. The name sounds intimidating, but the idea is simple: you split the motion you want for the whole robot (vx, vy, ω) into a speed for each of the four wheels. Changing only the direction and speed of each wheel gives you forward, sideways, rotating, and diagonal motion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*WDMU54JH6pQQnu62r6I2GQ.gif" /><figcaption><em>The same four wheels produce four different motions; only each wheel’s speed and direction change.</em></figcaption></figure><p>You might think that’s the end of it. It isn’t.</p><h3>Telling a motor “spin at 100 RPM” doesn’t mean it will spin at 100 RPM</h3><p>This is where the gap between software and the physical world becomes hard to ignore.</p><p>In code, if you write x = 100, then x is 100, always. But tell a motor to spin at 100 RPM, and it might only reach 92 RPM. There are many possible reasons:</p><ul><li>The battery is a little low</li><li>The robot is a little heavy</li><li>The floor or the friction is different</li><li>One motor is slightly stronger than the other</li><li>A wheel is slipping</li></ul><p>So the robot can’t assume that a command it sent actually happened. It has to <strong>measure</strong>.</p><p>The ROSMASTER’s wheels have <strong>encoders</strong>, which report how far each wheel has turned and how fast it’s turning right now. If the actual speed is 92 RPM, the controller strengthens the signal to the motor. A moment later the speed reaches 99 RPM, closer to the target.</p><p>The most common way to make this adjustment is <strong>PID control</strong>. It looks at the error (the difference between the target and the actual speed) and nudges the signal until the two match.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*SYq-7p8sXr18_Q704mvEaw.gif" /><figcaption><em>The PID loop closes the gap between the commanded 100 RPM and what the motor actually does.</em></figcaption></figure><p>This loop runs continuously, and it’s what finally turns a command into motion: <strong>command in software → electrical signal → the motor turns → the wheel turns → the robot moves</strong>. The push on the stick has reached the real world.</p><h3>Step 5: Moving isn’t the same as moving correctly</h3><p>Tell the robot to drive straight. All four wheels turn, but the robot may slowly drift off course as the small errors from the previous section add up.</p><p>If the Jetson only looked at the command it sent, it would assume the robot is going straight. But that’s only what it <em>wants</em> to happen. The robot needs to know <strong>what’s actually happening</strong>.</p><p>In this setup, the STM32 sends motion data back to the Jetson on two topics:</p><ul><li><strong>/odom_raw</strong> — movement estimated from wheel rotation</li><li><strong>/imu/data_raw</strong> — IMU data such as angular velocity and acceleration</li></ul><p>Together with the commands going down, this closes the loop. Suppose the right-side motors spin 6% faster. /cmd_vel still says &quot;straight,&quot; yet the robot slowly curves to the left. The only way to notice is to look at the measurements coming back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Pq3Rm2I9fQo722BA8879Rw.gif" /><figcaption><em>A 6% speed difference is invisible in </em><em>/cmd_vel, but it shows up in the feedback.</em></figcaption></figure><p>The robot doesn’t only send commands; it also checks the results. That’s <strong>feedback</strong>, one of the most basic ideas in robotics.</p><h3>So, is this Physical AI yet?</h3><p>Not yet. In this example, the “smartest” part of the system is still… me. I look at the surroundings, decide where to go, and move the stick, and the robot follows. This setup is called <strong>teleoperation</strong>, or remote control.</p><p>But here’s the important point: <strong>everything below </strong><strong>/cmd_vel would still be needed, almost unchanged, if the human were replaced by an AI.</strong></p><p>Imagine you skip the joystick and say, “Go check meeting room 3.” A complete system might need to:</p><ol><li>Have an AI understand the request</li><li>Figure out where meeting room 3 is and where the robot is now</li><li>Plan a path</li><li>Generate and send /cmd_vel</li><li>Keep observing with its sensors while moving, and finish the task on arrival</li></ol><p>Steps 1–3 are where the AI lives. From step 4 onward, it’s the same machinery you’ve seen in this post.</p><h3>How is AI in a robot different from AI on a screen?</h3><p>We usually think of a chatbot as Input → Model → Output. Once there&#39;s an output, you&#39;re basically done.</p><p>In robotics, the AI’s output is only where the work begins. After it come planning, navigation, controllers, motors, mechanical parts, sensors, and a real-world environment that isn’t as well-behaved as data inside a computer:</p><ul><li>The camera might be too dark</li><li>The LiDAR might get confused by glass</li><li>Data might arrive late</li><li>Someone might suddenly step in front of the robot</li></ul><p>With a chatbot, if you don’t like the answer, you press <strong>Regenerate</strong>. If a robot drives off sideways, there’s no <strong>Ctrl+Z</strong> to put the whole room back.</p><p>That’s why, to me, the most interesting question in Physical AI isn’t only “Is the AI smart enough?” It’s this:</p><blockquote><strong><em>How do you pass a smart decision down to a physical system that is stable enough to carry it out, and how do you know whether the result in the real world matched the plan?</em></strong></blockquote><h3>Where I got stuck: /cmd_vel is published, but the robot doesn&#39;t move</h3><p>This is a problem I actually ran into on the ROSMASTER. I sent /cmd_vel directly from ROS, with a command like this:</p><pre>ros2 topic pub /cmd_vel geometry_msgs/msg/Twist \<br>  &quot;{linear: {x: 0.2, y: 0.0, z: 0.0}, angular: {x: 0.0, y: 0.0, z: 0.0}}&quot;</pre><p>From the ROS side, everything looked fine: the topic existed and the commands were flowing. Yet sometimes the robot didn’t move.</p><p>Only after tracing the layers one by one did I notice something important:</p><blockquote><strong><em>Warning:</em></strong><em> </em><strong><em>Seeing a command in ROS does not mean that command has reached the motors.</em></strong></blockquote><p>For the motors to move, every link in between has to work:</p><ul><li>The micro-ROS Agent is running and relaying commands</li><li>The STM32 is connected correctly</li><li>The firmware has created the right subscriber</li><li>The motor controller is working</li><li>The power supply is stable</li></ul><p>If any one of these breaks, the Jetson side looks perfectly clean while the robot sits still. Next time I hit this, I’d check in this order:</p><pre># 1. Is any node subscribed to /cmd_vel? (If Subscription count is 0, nothing downstream is receiving it)<br>ros2 topic info /cmd_vel --verbose</pre><pre># 2. Are the topics from the STM32 showing up? (checks the micro-ROS connection)<br>ros2 topic list | grep -E &quot;odom_raw|imu&quot;</pre><pre># 3. Is the feedback actually updating?<br>ros2 topic hz /odom_raw</pre><p>For example, if the micro-ROS Agent isn’t running, the trail looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*7k0Fwl9UKPfv90R0f36qCg.gif" /><figcaption><em>“Subscription count: 0” points straight at the missing link.</em></figcaption></figure><p>For someone coming from software, this is a real shift in thinking. There, it’s usually enough to check “Was the function called?” or “Did the API return a response?” In robotics, you also have to ask:</p><blockquote><strong><em>Did the real world do what the software told it to do?</em></strong></blockquote><h3>Conclusion</h3><p>This post followed one stick push down through /joy → /cmd_vel → micro-ROS → STM32 → motors, and then back up from the encoders and IMU to the Jetson.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*U5vMVjW7HeluQfdVeq7Ccg.gif" /><figcaption><em>The whole loop: the Jetson decides, the STM32 executes, and </em><em>/cmd_vel is the boundary.</em></figcaption></figure><p>The key lesson: <strong>a robot has to check what the real world did, not only what the software said.</strong> You don’t need to memorize every name in this post, but these four points are worth keeping:</p><ul><li><strong>Deciding and executing are separate.</strong> The Jetson (ROS 2) decides <em>what</em> to do; the STM32 handles <em>how</em> to move</li><li><strong>/cmd_vel is the boundary.</strong> Whether the layer above is a person, Nav2, or an AI, the machinery below stays mostly the same</li><li><strong>Things don’t move exactly as commanded.</strong> That’s why the robot keeps measuring with encoders and an IMU, and keeps correcting with PID and similar methods</li><li><strong>Visible in ROS ≠ moving in the real world.</strong> Debug by tracing the layers one by one</li></ul><p>Put another way, robotics runs on a loop: <strong>sense → decide → act → observe the result → decide again</strong>. Terms like SLAM, Nav2, and VLA are easier to place once you see them as parts that plug into this loop. And AI starts to leave the screen when its decisions are connected to it.</p><p>If you have a mobile robot at hand, run ros2 topic echo /joy and ros2 topic echo /cmd_vel side by side while you move the stick. Watching stick positions turn into speeds is the quickest way to make this post concrete.</p><h3>Next time: the arm has motors, so why can’t ROS move it?</h3><p>The ROSMASTER M3 Pro also has a 6-axis robotic arm. From the outside, everything seems to be there: the frame, the servos, the cables, even a calibration feature.</p><p>But when I checked the ROS graph, something odd showed up. The STM32 topics I expected for controlling the arm and reading its state weren’t there the way the wheel topics were. The mechanical parts are there, and so are the motors, but the control path from the “brain” to the arm wasn’t connected the way I thought.</p><p>If this post was the story of one path that works end to end, the next one is the opposite:</p><blockquote><strong><em>When one link in the chain is missing, how do you trace the system to find where it breaks?</em></strong></blockquote><p>To put it a little playfully: <strong>the arm has bones and muscles, but the nerves to the brain aren’t connected yet.</strong></p><p><strong>Part 2 of 4: The robot arm has bones, but no nerves yet</strong></p><h3>References</h3><ul><li><a href="https://category.yahboom.net/products/rosmaster-m3-pro">ROSMASTER M3 Pro (Yahboom official product page)</a></li><li><a href="https://www.yahboom.net/study/ROSMASTER-M3PRO">ROSMASTER M3 Pro tutorials (Yahboom)</a></li><li><a href="https://github.com/YahboomTechnology/ROSMASTER-M3PRO">YahboomTechnology/ROSMASTER-M3PRO (GitHub)</a></li><li><a href="https://docs.ros.org/en/humble/Concepts/Basic/About-Nodes.html">ROS 2 Humble documentation: Nodes</a></li><li><a href="https://docs.ros.org/en/humble/Concepts/Basic/About-Topics.html">ROS 2 Humble documentation: Topics</a></li><li><a href="https://github.com/ros2/common_interfaces/blob/rolling/geometry_msgs/msg/Twist.msg">Definition of geometry_msgs/msg/Twist (GitHub)</a></li><li><a href="https://github.com/ros-drivers/joystick_drivers">ros-drivers/joystick_drivers (the joy package that publishes </a><a href="https://github.com/ros-drivers/joystick_drivers">/joy)</a></li><li><a href="https://micro.ros.org/">micro-ROS official site</a></li><li><a href="https://github.com/micro-ROS/micro-ROS-Agent">micro-ROS/micro-ROS-Agent (GitHub)</a></li><li><a href="https://docs.nav2.org/">Nav2 documentation</a></li><li><a href="https://developer.nvidia.com/embedded/jetson-orin">NVIDIA Jetson Orin</a></li><li><a href="https://en.wikipedia.org/wiki/Mecanum_wheel">Mecanum wheel (Wikipedia)</a></li><li><a href="https://en.wikipedia.org/wiki/Proportional%E2%80%93integral%E2%80%93derivative_controller">PID controller (Wikipedia)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b1a15d1a5839" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speeding Up Speech-to-Text for Real-Time Conversations]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@lucnguyen_61589/speeding-up-speech-to-text-for-real-time-conversations-51af91de2fc0?source=rss-aee34c8a9754------2"><img src="https://cdn-images-1.medium.com/max/1970/1*xMQckX0k-7lM1BvzOwD7aw.png" width="1970"></a></p><p class="medium-feed-snippet">In certain scenarios, speed is the single most critical factor in speech-to-text performance. Take a real-time IVR (Interactive Voice&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/@lucnguyen_61589/speeding-up-speech-to-text-for-real-time-conversations-51af91de2fc0?source=rss-aee34c8a9754------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@lucnguyen_61589/speeding-up-speech-to-text-for-real-time-conversations-51af91de2fc0?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/51af91de2fc0</guid>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[speech-to-text-conversion]]></category>
            <category><![CDATA[voice-activity-detection]]></category>
            <category><![CDATA[deepgram]]></category>
            <category><![CDATA[speech-to-text-api]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Mon, 11 Aug 2025 23:58:19 GMT</pubDate>
            <atom:updated>2025-08-12T06:11:51.552Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Understanding Multimodal AI: Merging Text and Visual Data]]></title>
            <link>https://blog.gopenai.com/understanding-multimodal-ai-merging-text-and-visual-data-584a8cb055c5?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/584a8cb055c5</guid>
            <category><![CDATA[qwen2-vl]]></category>
            <category><![CDATA[vision-language-model]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[multimodal]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Thu, 20 Mar 2025 01:12:17 GMT</pubDate>
            <atom:updated>2025-03-20T04:54:55.223Z</atom:updated>
            <content:encoded><![CDATA[<p>In our previous <a href="https://medium.com/gopenai/revolutionizing-home-decor-a-vision-language-model-for-furniture-matching-145978a2a6b4">blog</a>, we explored how multimodal AI helps you find the perfect furniture for your home. Today, let’s explore what this model can do and take a deeper dive into how it understands images and text together.</p><h3>I. What is Qwen 2.5 VL?</h3><p>Qwen 2.5 VL is an open-source multimodal model developed by Alibaba. It understands both text and images, enabling tasks such as:</p><ul><li><strong>Advanced Visual Recognition</strong>: Identifying objects, landmarks, and products accurately.</li><li><strong>Object Localization</strong>: Detecting and highlighting objects within images.</li><li><strong>Document Parsing</strong>: Extracting structured information from invoices, bills, and forms.</li><li><strong>Long-Video Comprehension</strong>: Understanding specific actions and moments in videos.</li></ul><p>There are three main versions of Qwen 2.5 VL:</p><ul><li><strong>Qwen 2.5 VL 3B</strong> (suitable for quick tests)</li><li><strong>Qwen 2.5 VL 7B</strong> (good for practical applications)</li><li><strong>Qwen 2.5 VL 72B</strong> (high accuracy for advanced use-cases)</li></ul><p>You can easily run the 3B or 7B models on Google Colab with around 16GB of VRAM.</p><h4>Quick Start</h4><p>Install required packages:</p><pre>!pip install git+https://github.com/huggingface/transformers<br>!pip install qwen-vl-utils<br>!pip install faiss-cpu</pre><p>Load the model:</p><pre>from io import BytesIO<br>from urllib.request import urlopen<br>import librosa<br>from transformers import Qwen2AudioForConditionalGeneration, AutoProcessor<br><br>processor = AutoProcessor.from_pretrained(&quot;Qwen/Qwen2-Audio-7B-Instruct&quot;)<br>model = Qwen2AudioForConditionalGeneration.from_pretrained(&quot;Qwen/Qwen2-Audio-7B-Instruct&quot;, device_map=&quot;auto&quot;)</pre><h4>Examples</h4><p>Full code is available at <a href="https://github.com/easyautoml/teradata/blob/main/10_Multimodals/02_Qwen2.5%20VL_Models.ipynb">here</a></p><p><strong>Visual Question Answering</strong></p><pre>prompt = &quot;How many people are in the picture?&quot;<br>result, _ = generate_text_from_image(image, prompt)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Nh59F8YwEmu_eYt8IZRBMQ.png" /></figure><p>Results :</p><pre>There are three people in the picture.</pre><h4>Object Detection</h4><pre>PROMPT = &quot;&quot;&quot;<br>Please analyze the provided image and identify the location of the cat. <br>Return the bounding box coordinates as a JSON dictionary with keys ‘x’, ‘y’, ‘width’, and ‘height’, where (x, y) represents the top-left corner of the box and ‘width’ and ‘height’ represent its dimensions. <br>Only return the JSON output.<br>&quot;&quot;&quot;<br><br>result, image_w_h = generate_text_from_image(image, PROMPT)<br>json_string = result.replace(&quot;```&quot;, &quot;&quot;)[4:]<br>json_string = json.loads(json_string)<br>draw_bounding_box(image, json_string, &quot;cat&quot;, image_w_h)</pre><p><strong><em>Result:</em></strong></p><pre>```json<br>{<br>  &quot;x&quot;: 250,<br>  &quot;y&quot;: 300,<br>  &quot;width&quot;: 200,<br>  &quot;height&quot;: 200<br>}<br>```</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rIeYe-Lch8q4B5ixSre6Ew.png" /></figure><p><strong>Note</strong>: The model’s JSON outputs may change depending on the prompt. For consistent results in object detection, you might need to fine-tune the model.</p><h4>Documents Parsing</h4><p>Extract key information from bills:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ScZMlxWaJfGn3TagAxz5g.png" /><figcaption>Vietnamese Elictric Bill</figcaption></figure><pre>PROMPT = &quot;Please analyze the provided electric bill and extract total amount of bill. Structure the extracted data into a well-formatted JSON output with appropriate keys and values.&quot;<br>result, _ = generate_text_from_image(image, PROMPT)</pre><p>Output :</p><pre>```json<br>{<br>  &quot;bill_number&quot;: &quot;AE/18E&quot;,<br>  &quot;date&quot;: &quot;19/10/2018&quot;,<br>  &quot;company_name&quot;: &quot;Công Ty Điện Lực Vĩnh Long - Điện Lực Vũng Liêm&quot;,<br>  &quot;customer_number&quot;: &quot;01GTKT0001&quot;,<br>  &quot;customer_name&quot;: &quot;Số 59 Nam Kỳ Khởi Nghĩa, khu 1, thị trấn Vũng Liêm, huyện Vũng Liêm, tỉnh Vĩnh Long&quot;,<br>  &quot;customer_address&quot;: &quot;Số 1-10 xã Trung Ngãi - huyện Vũng Liêm&quot;,<br>  &quot;customer_phone&quot;: &quot;0300942001-021&quot;,<br>  &quot;customer_email&quot;: &quot;19001006&quot;,<br>  &quot;customer_id&quot;: &quot;66559&quot;,<br>  &quot;total_amount&quot;: 562903<br>}<br>```</pre><p>Qwen 2.5 VL does an impressive job with structured document parsing.</p><h3>II. How Qwen 2.5 VL Works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oWra_BU7DI-amO-qQN9HKQ.png" /><figcaption>Components in Qwen VL</figcaption></figure><p>Qwen 2.5 VL consists of three main components:</p><ul><li><strong>Vision Encoder</strong>: Efficiently processes visual inputs (images/videos) and converts them into structured visual tokens (features) using a Vision Transformer (ViT) architecture enhanced by windowed attention for efficiency.</li><li><strong>MLP-based Vision-Language Merger</strong>: Simplifies and transforms visual tokens from the vision encoder into compact representations, enabling the language model to easily integrate and process visual information alongside textual input.</li><li><strong>Large Language Model (LLM)</strong>: Processes textual inputs and generates text outputs, based on the pre-trained QwenLM, providing the language understanding and generation capabilities for multimodal reasoning tasks</li></ul><h4><strong>1. Vision Encoder</strong></h4><p>The Vision Encoder is crucial for helping the language model “see.” Qwen2.5 VL uses a <strong>Vision Transformer (ViT)</strong>, plus some enhancements for improved performance.</p><p><strong>Understanding ViT (Vision Transformer)</strong></p><p>The Vision Transformer (ViT) is a foundational model designed to process images effectively. Its goal is to extract meaningful features from images, making it easier for language models to understand visual data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmUklG1u7vP9myyDWVEJHA.png" /><figcaption>ViT architecutre (source : <a href="https://www.geeksforgeeks.org/vision-transformer-vit-architecture/"><em>Link</em></a>)</figcaption></figure><p>ViT works by splitting an input image into <strong><em>multiple</em></strong> <strong><em>patches</em></strong>. Each patch is flattened into a vector, and an extra special token called [CLS] (classification token) is added. The [CLS] token summarizes the entire image and serves as an overall image representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YQQKDGvmb6bMr0Ksa5IwDQ.png" /><figcaption>Training Dataset (Ex : <strong>ImageNet-1K</strong>)</figcaption></figure><p>During training, ViT uses labeled image datasets to learn and classify images. Once trained, ViT can provide powerful image embeddings, which the Vision Encoder uses to facilitate understanding and integration with language models</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SrPCxbIMTP7NkCIZP766cQ.png" /></figure><p>After training, ViT models can generate vision embedding vectors, which effectively capture key image features. These embeddings can be used for various downstream tasks, such as building classification models or image retrieval.</p><p>To better understanding about ViT model, you can download pre-trained ViT model with few line of code as following</p><pre>from transformers import ViTImageProcessor, ViTModel<br>from PIL import Image<br>import requests<br>import torch<br>from PIL import Image<br>import os<br><br>processor = ViTImageProcessor.from_pretrained(&#39;google/vit-base-patch16-224&#39;)<br>model = ViTModel.from_pretrained(&#39;google/vit-base-patch16-224&#39;)<br><br>def get_image_embedding(image_path):<br>    &quot;&quot;&quot;<br>    Extracts the CLS token embedding from a given image.<br>    &quot;&quot;&quot;<br>    # Load image<br>    image = Image.open(image_path).convert(&quot;RGB&quot;)<br><br>    # Preprocess image<br>    inputs = processor(images=image, return_tensors=&quot;pt&quot;)<br><br>    # Forward pass through model<br>    with torch.no_grad():<br>        outputs = model(**inputs)<br><br>    # Extract CLS token embedding (global image representation)<br>    cls_embedding = outputs.last_hidden_state[:, 0, :]  # Shape: [1, 768]<br><br>    return cls_embedding<br><br><br>image_path = &lt;image_url&gt;<br>embedding = get_image_embedding(image_path)</pre><p>One of the biggest drawbacks of traditional ViTs is that they force images into a fixed size before splitting into patches. Below, we explain how window-based attention helps solve this problem — especially for OCR</p><h4><strong>Windows Attentions</strong></h4><p>Traditional ViTs require input images to be resized to a fixed resolution (e.g., 224×224) before splitting them into a fixed number of patches. Unfortunately, such resizing can degrade performance in detail-oriented tasks like OCR, because tiny yet crucial features (like small text or digits) may be blurred or lost altogether.</p><p><strong>Qwen’s Approach<br></strong>To address this limitation, Qwen employs a <strong>window-based attention</strong> strategy. Instead of resizing the image globally, it preserves the image at its native resolution and divides it into local windows (e.g., 4×4 patches each). Qwen then applies self-attention <em>only</em> within these small, local windows — rather than across the entire image — ensuring that fine-grained details remain intact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hp0sd1uxG5gmLoKX7aViBw.png" /><figcaption>Compare windows creation</figcaption></figure><p><strong>This approach has two major advantages:</strong></p><ol><li><strong>It preserves fine-grained details<br></strong>Because there’s no global resizing, Qwen avoids blurring out small text or subtle features. For instance, when analyzing a scanned receipt for OCR, window-based attention keeps each line of text crisp and legible.</li><li><strong>Reduces Computation<br></strong>By confining self-attention to smaller windows, the model sidesteps the large, memory-heavy attention map that a full-resolution global approach would require. This makes training and inference more efficient, especially for high-resolution or high-DPI images.</li></ol><h4>2. Vision Language Merge</h4><p><strong>2.1 Motivation<br></strong>Modern Large Language Models (LLMs) — like Llama, Qwen, and GPT — mostly work with text. They map tokens (words or subwords) into one shared embedding space.</p><p>At the same time, a Vision Transformer creates <strong>different embeddings</strong> for images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TPJ-GkLBwKLegWCnELi9Vg.png" /><figcaption>Decoder Only LLM Model</figcaption></figure><p>Because these two embedding types do not match, <strong>Qwen2.5-VL</strong> solves the problem with an <strong>MLP-based Vision-Language Merge</strong> layer. This layer converts image embeddings into a space that aligns with text embeddings.</p><p><strong>2.2 MLP-based Vision-Language Merge</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u63e7aGgmQazUXJOSph6-g.png" /><figcaption>MLP-Based Vision Language Merge</figcaption></figure><p>This layer in the Qwen2.5-VL model serves two main purposes:</p><ul><li><strong>Reduce the dimensionality</strong> of vision embeddings to match the LLM’s embedding space.</li><li><strong>Align vision tokens with text tokens</strong> in a multimodal space.</li></ul><p><strong>Reduce the dimensionality</strong></p><p>Unlike ViT, which uses global self-attention and a [CLS] token to represent the whole image, Qwen2.5-VL’s Vision Encoder uses windowed self-attention. Because of this, there is no single token representing the entire image — instead, the model processes a sequence of patch embeddings.</p><p><strong><em>For example :</em></strong><br>With a 224×224 image the model generates 256 (16×16) patches. Each patch is a 1,024-dimensional vector, creating a large set of high-dimensional tokens that would be expensive for the LLM to handle directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-wb5U6K10hvOG2_2v2mtw.png" /><figcaption><strong>Projecting Vision Patches into LLM Embeddings</strong></figcaption></figure><p>To address this, Qwen2.5-VL applies an <strong>MLP-based projection</strong>. While it keeps the same number of patches, it <strong>reduces</strong> each patch’s embedding size (e.g., from 1,024 to 768), matching the LLM’s text embedding dimension. This makes vision embeddings more manageable for the LLM while still preserving multimodal alignment.</p><p><strong>Align vision tokens with text tokens</strong></p><p>Even after we shrink each patch to 768 dimensions, we still need to make sure they actually “talk” the same language as text tokens. Just matching sizes doesn’t mean an image of a dog knows it’s a dog. That’s where <strong>contrastive learning</strong> steps in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eff2xPLnSjIQLNYBi7_Now.png" /><figcaption>Contrastive Learning</figcaption></figure><p>With contrastive learning, the MLP-based layer learns to <strong>pull</strong> correct pairs (like “a cute dog” and an image of a cute puppy) closer together, and <strong>push</strong> wrong pairs (like “cat” versus that puppy image) farther apart.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Qe3AXzzNhtaou4duG7EKKQ.png" /></figure><p>It uses cosine similarity along with a contrastive loss to make the MLP-based layer adjust the vision embeddings. Correct pairs (like a “dog” text with a dog image) end up with higher similarity scores, while wrong pairs (like a cat image with “dog” text) get pushed to lower scores.</p><p><strong>2.3. Large Language Model</strong></p><p>Qwen2.5-VL builds on the pre-trained Qwen2.5 large language model and adds a Vision Encoder plus an MLP-based Vision-Text Merge layer. The Vision Encoder extracts features from images, while the MLP layer aligns these vision embeddings with the LLM’s text embeddings, creating a unified multimodal model.</p><p>After merging these components, the entire system is fine-tuned on tasks like OCR, Visual Question Answering (VQA), and Object Detection, ensuring strong real-world performance across a wide range of multimodal applications.</p><h3>Conclusion</h3><p>By combining a Vision Encoder (often ViT with windowed attention), an MLP-based Merger, and the pre-trained Qwen2.5 LLM, <strong>Qwen2.5-VL</strong> can “see” images, “read” text, and connect both modalities.</p><p>Whether you’re parsing receipts, detecting objects, or answering visual questions, Qwen2.5-VL offers a flexible and powerful multimodal framework.</p><p><strong>Source :</strong></p><ul><li><a href="https://github.com/QwenLM/Qwen2.5-VL">https://github.com/QwenLM/Qwen2.5-VL</a></li><li><a href="https://arxiv.org/abs/2502.13923">https://arxiv.org/abs/2502.13923</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=584a8cb055c5" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/understanding-multimodal-ai-merging-text-and-visual-data-584a8cb055c5">Understanding Multimodal AI: Merging Text and Visual Data</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Revolutionizing Home Decor: A Vision Language Model for Furniture Matching]]></title>
            <link>https://blog.gopenai.com/revolutionizing-home-decor-a-vision-language-model-for-furniture-matching-145978a2a6b4?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/145978a2a6b4</guid>
            <category><![CDATA[qwen2-5]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Wed, 19 Feb 2025 23:17:22 GMT</pubDate>
            <atom:updated>2025-03-17T06:04:02.032Z</atom:updated>
            <content:encoded><![CDATA[<p>Imagine walking into a furniture store with a photo of your room, and instantly knowing which chair complements your home’s style. This isn’t a far-off dream; it’s a project I’m diving into, combining AI with home decor to simplify furniture shopping.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xhAhgPiPyHzMylkB" /></figure><p>This project not only promises to streamline furniture shopping but also offers a fascinating dive into multimodal AI, merging visual and textual data in ways that can revolutionize user interaction with AI systems.</p><h3><strong>1. Overall</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/967/1*6164G2-a8hLDuGDHlJtb5Q.png" /></figure><p>To bring this concept to life, I’ve split the project into two main parts:</p><p><strong>Image-Based Description</strong><br>The first step involves creating a system where you can upload an image of your room. Along with your requirements (e.g., “Which chair should I buy for my new room?”), the system analyzes the style of the room and generates a description of potential products — covering aspects like color, material, and overall style.</p><p><strong>Product Retrieval</strong><br>In the second step, the system takes the generated description — such as “a sleek, modern, gray fabric chair” — and retrieves a list of matching products from an existing database or catalog. In other words, it filters through a product inventory to find the closest matches to the style and specifications the AI has identified.</p><h3><strong>2. Image-Based Description</strong></h3><p>With the goal of letting our AI “see” your room and suggest suitable furniture, we need a vision-language model capable of both analyzing images and generating concise text descriptions.</p><p>To generate concise room-style descriptions, we rely on the <strong>Qwen2-VL-2B-Instruct</strong> model. It accepts both text and images, letting the AI “see” your room and respond in natural language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*fq7Wox3ugppMQS_9nfsbew.png" /></figure><p>Below is the core code: it takes your image, prompts the model (e.g., “What chair fits my room?”), and returns a short description focusing on style-related keywords (color, material, etc.).</p><pre>from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor<br>from qwen_vl_utils import process_vision_info<br>import torch<br><br># Step 1 : Load Qwen VL 2B <br>model_name = &quot;Qwen/Qwen2-VL-2B-Instruct&quot;<br>qwen_model = Qwen2VLForConditionalGeneration.from_pretrained(<br>    model_name, <br>    torch_dtype=torch.float16, <br>    device_map=&quot;auto&quot;<br>)<br>qwen_processor = AutoProcessor.from_pretrained(<br>    model_name, <br>    min_pixels=256*28*28, <br>    max_pixels=1024*28*28<br>)<br><br>def generate_text_from_image(image_path, text, model, processor):<br>    # You can find the full source code in the shared link at the end of the <br>    # article.<br>    ...<br>    return output_text[0]<br><br># Step 2 : Generate text<br>prompt = &quot;&quot;&quot;<br>  I’m going to buy a chair for my house. <br>  Describe which chair would fit best, focusing on color, <br>  material, style, shape.<br>&quot;&quot;&quot;<br><br>image_path = &quot;./living_room.png&quot;<br><br>output = generate_text_from_image(image_path, prompt, qwen_model, qwen_processor)<br>print(output)  # e.g., &quot;Modern, upholstered, round, wooden legs&quot;</pre><p><strong>Input Image :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*4O7Hn_kcVRHiLiPp6sotjQ.png" /><figcaption>Input image for model</figcaption></figure><p><strong>Output from model :</strong></p><pre>Modern, upholstered, round, wooden legs</pre><p>To ensure model can understand the image I uploaded, I aso asked model describe about image I uploaded, and here is result, even this model only have 2B parameter but the output is really impressive</p><pre>The image depicts a modern living room with a minimalist and elegant design. <br>The room features light green walls and a white ceiling, creating a serene and <br>calming atmosphere. <br><br>The floor is covered with light-colored hardwood flooring, adding to the <br>room&#39;s brightness.<br><br>In the center of the room, there is a large, L-shaped sectional sofa <br>upholstered in a dark gray fabric. The sofa is adorned with several throw <br>pillows in various colors, including beige, brown, and orange, adding a touch <br>of warmth and personality to the space. <br><br>In front of the sofa, there is a round, wooden coffee table with a modern <br>design, complementing the overall aesthetic</pre><h3>3. Product Retrieval</h3><p>Now that we’ve described a product that fits the room’s style, the next step is finding the actual items that match that description.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*YBiFPpIgxnUnmQpVw3wjEw.png" /><figcaption>Semantic Search</figcaption></figure><p>A common approach is <strong>semantic search</strong>, which compares text-based requirements with a product’s text description. However, this method often fails if the description is incomplete — key details might be missing, leading to inaccurate matches.</p><p><strong>A New Approach: Image-Based Retrieval</strong></p><p>Instead of relying solely on text, why not compare your requirement directly against <strong>product images</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*ZhKokV1S8QBdFeiclYQO9A.png" /></figure><p>This approach captures details that might never appear in text descriptions, reducing the need for exhaustive metadata. By searching visually, we maintain all the product’s nuances and potentially achieve more precise results, especially when style or appearance is paramount.</p><h4>3.1 Overview of <strong>Image-Based Retrieval</strong></h4><p>When matching text-based requirements (e.g., <strong>“modern, upholstered, round, wooden legs”</strong>) to product images, we can use a <strong>multimodal embedding model</strong>. In this article, we’ll rely on <strong>CLIP</strong>, a model from OpenAI that accepts both text and images as input and produces corresponding embedding vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*mec6vRxYGSl7DBybGv-DQA.png" /></figure><p>This approach is similar to <strong>Retrieval-Augmented Generation (RAG)</strong>, but instead of retrieving relevant text, we retrieve relevant images. We split the system into two steps:</p><ol><li><strong>Preparation</strong><br>Convert all product images into embedding vectors and store them in a vector database.</li><li><strong>Retrieval</strong><br>Take a product description as input, generate an embedding, and retrieve the most similar images from the vector database.</li></ol><h4>3.2 Preparation</h4><p>In this step, we’ll load a CLIP model, generate embeddings for each product image, and store them in a FAISS index. This lets us efficiently retrieve similar images based on text or other image queries.</p><pre>import torch<br>import faiss<br>import numpy as np<br>from transformers import CLIPProcessor, CLIPModel<br><br>image_files = &quot;&lt;List of your image path files&gt;&quot;<br><br>def calculate_embedding(text, image_path):<br>    # You can find the full source code in the shared link at the end of the <br>    # article.<br>    ...<br>    return image_embedding, text_embedding<br><br># 1. Load CLIP<br>model_id = &quot;openai/clip-vit-base-patch32&quot;<br>clip_processor = CLIPProcessor.from_pretrained(model_id)<br>clip_model = CLIPModel.from_pretrained(model_id).to(<br>    &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>)<br><br># 2. Helper to create a FAISS index<br>def create_faiss_index(embeddings, image_paths):<br>    d = embeddings.shape[1]  # embedding dimension<br>    index = faiss.IndexFlatL2(d)  # L2 distance<br>    index.add(embeddings)<br>    index.image_paths = image_paths  # attach image paths for retrieval<br>    return index<br><br># 3. Compute image embeddings &amp; build index<br>all_image_embeddings = []<br>all_image_paths = []<br><br>for image_file in image_files:<br>    image_embedding, _ = calculate_embedding(&quot;&quot;, image_file)  # custom helper<br>    embedding_np = image_embedding.cpu().detach().numpy().reshape(1, -1)<br>    all_image_embeddings.append(embedding_np)<br>    all_image_paths.append(image_file)<br><br>image_embeddings_np = np.concatenate(all_image_embeddings, axis=0)<br>vector_db = create_faiss_index(image_embeddings_np.astype(&#39;float32&#39;), all_image_paths)</pre><p>Here is the list of products I prepared for this demo; it includes various types of chairs and cabinets. After running the code above, all of these product embeddings will be stored in the vector database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4-OXkKVnVcfaBYfD5wNHbA.png" /></figure><h4>3.3 Retrieval</h4><p>With our images stored as embeddings in a FAISS index, the next step is to find which products match a user query. In the code below, we take a text description — like <strong>“Modern, upholstered, round, wooden legs”</strong> — generate a CLIP embedding for it, then search for the most similar embeddings in our vector database.</p><pre>def retrieve_similar_images(image_path=None, text=None, top_k=2):<br>    _, _embedding = calculate_embedding(text=text, image_path=image_path)  # Produces a CLIP embedding<br>    query_embedding = _embedding.cpu().detach().numpy().reshape(1, -1)<br><br>    # Search for the top_k most similar items<br>    D, I = vector_db.search(query_embedding.astype(&#39;float32&#39;), k=top_k)<br><br>    retrieved_images = []<br>    for i in range(top_k):<br>        retrieved_image_path = vector_db.image_paths[I[0, i]]<br>        similarity_score = 1 - D[0, i] / 2  # Convert distance to a rough similarity measure<br>        retrieved_images.append((retrieved_image_path, similarity_score))<br><br>    return retrieved_images<br><br># Example usage:<br>query = &quot;Modern, upholstered, round, wooden legs&quot;<br>retrieved_images = retrieve_similar_images(text=query)</pre><p>In one pass, CLIP transforms the query into an embedding, FAISS locates the closest matches, we convert distances into intuitive scores, and deliver the top results. And here is result</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*Qjw61O29hxN5YAV9_rwkjA.png" /><figcaption>Top 2 retrieval</figcaption></figure><h3><strong>3. Conclusion</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*1CCi79-UUYLEUMpBuAIatg.png" /><figcaption>Final Results</figcaption></figure><p>By leveraging <strong>CLIP</strong> for product retrieval and <strong>Qwen2-VL</strong> for style-based insights, we’ve created a simple yet powerful two-step pipeline. Now, a user can walk into your shop with a photo of their room and a rough idea of what they need, and the system will suggest items that fit their style. You can easily expand this approach with filters — like budget — or even integrate another AI model to visualize the new furniture in the user’s space.</p><p>In the next chapter, we’ll delve deeper into how CLIP and Qwen actually work, covering their architectures and the reasons they excel at multimodal tasks.</p><p><strong>Source Code :</strong> <a href="https://github.com/easyautoml/teradata/blob/main/10_Multimodals/01_Qwen_CLIP_Models.ipynb">https://github.com/easyautoml/teradata/blob/main/10_Multimodals/01_Qwen_CLIP_Models.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=145978a2a6b4" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/revolutionizing-home-decor-a-vision-language-model-for-furniture-matching-145978a2a6b4">Revolutionizing Home Decor: A Vision Language Model for Furniture Matching</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Virtual Taigu Sama: AI-Generated Responses to Life’s Tough Questions]]></title>
            <link>https://medium.com/@lucnguyen_61589/creating-a-virtual-taigu-sama-ai-generated-responses-to-lifes-tough-questions-cceeadb1e374?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/cceeadb1e374</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[speech-to-text-api]]></category>
            <category><![CDATA[wav2lip]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[voice-cloning]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Wed, 06 Nov 2024 00:31:23 GMT</pubDate>
            <atom:updated>2024-11-06T00:31:23.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Taigu Sama, a Japanese YouTuber I admire, is known for helping people overcome life’s challenges, making them feel happier and more fulfilled. I had my own question for him, but knowing he receives so many, I wasn’t sure if he’d ever get to mine. That’s when I decided to use generative AI to mimic his voice and create a video answer to my question.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*yOMBwOrRolIe86g5b4Mk5g.png" /></figure><h3>About Taiku sama</h3><p>Taigu, the<strong><em> ‘Great Fool,’ </em></strong>merges spirituality, personal experience, and martial arts to spread compassion and wisdom in modern life.</p><p>With his Heart of Buddha Academy, Teramachi Project, and YouTube channel, he shares Buddhist insights to help people find purpose and inner peace.</p><p>His channel, <strong><em>Osho Taigu’s Heart of Buddha</em></strong>, provides comforting advice on handling stress and finding meaning, uplifting viewers to live more fulfilling lives.</p><p><strong>Result :</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9-n8PWhBOeM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9-n8PWhBOeM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9-n8PWhBOeM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9dfea947aea56e8a18d7ed91422cf74/href">https://medium.com/media/d9dfea947aea56e8a18d7ed91422cf74/href</a></iframe><h3>Overall Flows</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*DOyhcqrGhxK0MX0UpJueqg.png" /></figure><p><strong>Step 1: Fine-Tuning the Text-To-Speech Model</strong><br>The first step was creating audio that truly sounds like Taigu Sama. I initially tried Voice Cloning with just 6 seconds of audio, but it didn’t sound natural enough, so I chose to fine-tune the model.</p><p><strong>Step 2: Generating the Answer and Audio</strong><br>With the fine-tuned TTS model, I used ChatGPT combined with RAG to generate an answer that closely matches Taigu Sama’s speaking style. This approach helped incorporate Buddhist knowledge, phrases, and expressions he often uses. Then, I converted the generated response into audio in his voice.</p><p><strong>Step 3: Syncing Audio and Video</strong><br>Finally, I used Wav2Lip to align the video with the audio, giving the impression that Taigu Sama is actually speaking.</p><h4><strong>Step 1: Fine-Tuning the Text-To-Speech Model</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*ZYAvebfajKQ3cy1QDs3sng.png" /><figcaption>Step 1 : Fine Tuning Text To Speech model</figcaption></figure><p>To fine-tune the TTS model, I needed at least 30 minutes of audio and corresponding transcriptions from Taigu Sama.</p><p>Instead of manually listening and splitting the audio, I used the Whisper model, a powerful speech-to-text tool, to automate the transcription process.</p><p>This setup allowed me to create a dataset with paired audio and text, enabling the model to closely mimic Taigu Sama’s voice and speech patterns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*IwHnw8x5zxws57lF0GOIHg.png" /><figcaption>Fine-Tuning Data</figcaption></figure><h4>Step 2: Generating the Answer and Audio</h4><p>The next step was crafting a response that genuinely sounded like Taigu Sama. This meant incorporating Buddhist wisdom and the specific phrases he often uses.</p><p>To do this, I collected Taigu Sama’s previous answers and stored them in a Vector Database. With each new question, RAG (Retrieval-Augmented Generation) pulls relevant insights from this database and passes them to ChatGPT, which then generates a response in Taigu Sama’s style.</p><p>This approach ensures the answer feels realistic and resonates with his usual expressions and tone.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*UTsPUnlfQ2vAvaYWUtQQmQ.png" /><figcaption>Flow of generate answer</figcaption></figure><p>Here’s an example of a question and the generated answer. It closely captures Taigu Sama’s thoughtful approach, making it hard to distinguish from his real responses.</p><pre>Question = &quot;&quot;&quot;<br>I&#39;ve been in Japan for 10 years. Should I stay here or go back? <br>If I stay, I can provide my children with a better education, but it means I won&#39;t be able to spend as much time with my parents.<br>&quot;&quot;&quot;<br><br>Answer = &quot;&quot;&quot;<br>Your question reflects a heartfelt dilemma, one that many people face in today&#39;s world. <br>It’s natural to feel pulled between wanting to support your parents and ensuring your children’s future in a nurturing environment. <br>The importance of these values is deeply understood.<br><br>Throughout history, across cultures, this has been a universal question. <br>In Eastern thought, there’s a concept known as “filial piety,” which not only means showing respect and love for one’s parents <br>but also honoring their wishes by living a fulfilling life oneself. <br><br>This means considering what kind of life will bring true happiness to your parents, which is also an expression of this virtue.<br>Buddhism also speaks of “cause and effect,” suggesting that our actions and environment shape the future. If you can offer your children a stable life and excellent education in Japan, that achievement can contribute to the happiness your parents wish for you. Likewise, being close to your parents and directly supporting them might bring them peace and joy. Both are noble paths.<br><br>The key is to reflect on what matters most to you. How do you wish to show gratitude to your parents? What meaning do you find in raising your children in Japan? Taking time to consider these questions can guide your decision.<br>Rather than choosing one path over the other, finding balance may also be possible. For instance, while living in Japan, you could make regular trips to visit your parents or keep in close contact online to bridge the distance. With today’s technology, it’s easier to maintain strong connections, even from afar.<br><br>Ultimately, there is no “right” answer. Following the path that brings peace to your heart will, in turn, honor your parents and show love to your children. What matters most is believing that your choice will contribute to your family’s happiness and moving forward with that conviction.<br>Whether you prioritize your parents’ well-being or your children’s future, rest assured that any path you choose will be valuable. May you find peace in making the decision that feels right for you.<br>&quot;&quot;&quot;</pre><p>After the response is created, I use the fine-tuned TTS model to generate the audio in Taigu Sama’s voice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*wvcowYgbXhPIO0QQm8DWBw.png" /><figcaption>Generate audio</figcaption></figure><h4>Step 3: Generating the Video</h4><p>The <strong>Wav2Lip</strong> model allows us to create natural-looking lip-syncing by matching lip movements to the audio. Using the audio generated in Step 2, we select a video clip of the same length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*vDXj3GL2jI8G3sHpjyWtSg.png" /></figure><p><strong>Wav2Lip</strong> then aligns the lip movements with the spoken words, making it appear as though Taigu Sama is directly speaking to the viewer.</p><h3>Conclusion</h3><p>Using a mix of cutting-edge technologies — Speech-to-Text, Text-to-Speech, LLM models, and Wav2Lip — we’ve pushed the boundaries of what’s possible, creating something straight out of a sci-fi film. Picture receiving life advice from famous thinkers, interacting with digital teachers who sound like your favorite mentors, or even creating an AI-driven narrator for your stories. This blend of tech offers vast new opportunities for entertainment, education, and personalized experiences, bringing fictional ideas closer to reality.</p><p><strong>Reference :</strong></p><ul><li>Speech To Text : <a href="https://github.com/openai/whisper">https://github.com/openai/whisper</a></li><li>Text To Speech : <a href="https://github.com/coqui-ai/TTS">https://github.com/coqui-ai/TTS</a></li><li>Wav 2 Lip : <a href="https://github.com/Rudrabha/Wav2Lip">https://github.com/Rudrabha/Wav2Lip</a></li><li>OpenAI API : <a href="https://openai.com/index/openai-api/">https://openai.com/index/openai-api/</a></li><li>Taigu様 : <a href="https://taigu-gensho.com/">https://taigu-gensho.com/</a> (<a href="https://www.youtube.com/@osho_taigu">https://www.youtube.com/@osho_taigu</a>)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cceeadb1e374" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bring Your Voice to Life: Getting Started with Coqui XTTS V2]]></title>
            <link>https://blog.gopenai.com/bring-your-voice-to-life-getting-started-with-coqui-xtts-v2-e17c9e0e5ba7?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/e17c9e0e5ba7</guid>
            <category><![CDATA[voices]]></category>
            <category><![CDATA[coqui]]></category>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[voice-clone-ai]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 23:32:29 GMT</pubDate>
            <atom:updated>2025-03-17T06:03:54.254Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Text-to-speech </strong>technology has taken a major leap forward, with voice models now sounding more natural and human-like than ever before. What’s even more impressive is the ability to mimic specific voices — a feature known as “voice cloning.”</p><p>With just a 6-second sample of your voice, or anyone else’s, these models can generate audio that sounds remarkably close to the original speaker. Imagine generating speech from text in your own voice or that of someone you admire! In this article, I’ll explore how we use the XTTS model for voice cloning and take a deep dive into how this groundbreaking technology works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*u3TGFQnRdQxJLkGHokCHjA.png" /><figcaption>Voice clone with XTTS V2 Models</figcaption></figure><h3>1. Step-by-Step Guide</h3><p>Based on number of download, Coqui XTTS v2 seem to be one of the most model are using. To install it, you can following this article</p><h4>Install &amp; Run Your First Hello World</h4><p>In previous <a href="https://medium.com/@lucnguyen_61589/text-to-speech-for-beginners-f1d282f5c6bb">blog</a>, I talked detail about install coqui and some basic method in using Coqui models, if you are not familiar on how to using Coqui, please check it.</p><p><strong>Step 1:</strong> Install Coqui XTTS</p><pre>pip install TTS </pre><p><strong>Step 2:</strong> Launch XTTS V2 in Jupyter-Lab</p><pre>from TTS.api import TTS<br>from IPython.display import Audio<br>import librosa<br>import os<br><br># Load model<br>tts = TTS(&quot;tts_models/multilingual/multi-dataset/xtts_v2&quot;, gpu=True)<br><br>wav = tts.tts(&quot;お世話になっております！はじめまして, 私はCoqui XTTS　V2です&quot;,<br>        speaker=&quot;Kazuhiko Atallah&quot;,<br>        language=&quot;ja&quot;)<br><br>display(Audio(wav, rate=24000))</pre><h4>Clone Your Voice in Two Easy Steps</h4><p><strong>Step 1: </strong>Prepare a 6-Second Audio Sample in WAV Format</p><p><strong>Step 2: </strong>Generate Audio in Your Own Voice!</p><pre>wav = tts.tts(&quot;お世話になっております！はじめまして, 私はCoqui XTTS V2です&quot;,<br>  speaker_wav=[&lt;path of wav file&gt;],,<br>  language=&quot;ja&quot;<br>)<br><br>display(Audio(wav, rate=24000))</pre><h3>2. Inside XTTS: Unpacking the Key Components of Advanced Voice Cloning</h3><p>Let’s dive deeper into how the XTTS model works and explore the technologies that make this magic possible.</p><h4>Core Technologies Behind XTTS</h4><p>The XTTS V2 model performs text-to-speech conversion through several sophisticated steps. It starts by tokenizing the input text, then generates audio tokens using a GPT model, reconstructs the audio waveform via a HiFi-GAN decoder, and finally outputs natural-sounding speech.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cS34msVYGb74aG5nLBEz5Q.png" /></figure><p>The process relies on two key components:</p><ul><li><strong>GPT Model</strong>: This model transforms a sequence of text tokens into a sequence of audio tokens. These audio tokens represent key speech features, such as pitch, tone, and rhythm, that allow for the generation of natural-sounding speech from the textual input.</li><li><strong>HiFi-GAN Decoder</strong>: This component takes the audio tokens, which represent a mel-spectrogram (a detailed frequency representation of sound), and converts them into a raw audio waveform. The HiFi-GAN decoder plays a crucial role in ensuring the final output sounds clear, realistic, and high-quality.</li></ul><h4>Audio Tokens</h4><p>In traditional models like Tacotron-2, the common approach was to predict <strong>mel-spectrograms</strong> — a detailed frequency representation of audio — based on the input text. These spectrograms were then used to reconstruct speech.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*wAJY2IleEPqm230Y27GzMQ.png" /><figcaption>Tacotron2 flows</figcaption></figure><p>However, in XTTS V2, instead of predicting mel-spectrograms, the model predicts something called <strong>audio tokens</strong>. These are simplified, lower-dimensional representations of the audio, making the process more efficient. Think of audio tokens as small building blocks that represent the key features of sound but in a more compact way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*T3giKSa8MBU3ktS7-Cd7IA.png" /><figcaption>XTTS V2 models</figcaption></figure><p>If you’re curious about how VQ-VAE works in more detail, check out this <a href="https://erogol.com/2023/09/27/xtts-v1-notes">link</a> for a deeper dive.</p><h4>GPT Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*NXluHyKm-JW8l78vmsywpw.png" /></figure><p>The GPT model has three main parts:</p><ul><li><strong>Text Embedding</strong>: Converts words into numbers that the model can understand.</li><li><strong>Mel Embedding</strong>: Turns pieces of audio into numbers, allowing the computer to handle sound similarly to how it handles text.</li><li><strong>Condition Encoder</strong>: Adds extra information like the speaker’s style, emotion, or voice characteristics, influencing how the speech sounds.</li></ul><p>The model is trained on thousands of audio recordings from different speakers. By analyzing these recordings, it learns how words and sounds connect. It also picks up on emotions in speech, recognizing patterns like happiness or sadness in the way people speak.</p><p>The <strong>Transformer</strong> part of the model helps it understand the flow and context of speech, so it can generate more natural, expressive audio — almost like it “remembers” what was said earlier to make the speech sound smoother and more realistic.</p><h4>HifiGAN Decoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*Y5Eam6cSBXhiAUFIGq5lhA.png" /><figcaption>Hifigan Decoder components</figcaption></figure><p>The <strong>HiFi-GAN Decoder</strong> is trained on thousands of different voices, with the goal of producing high-quality, natural-sounding speech. The main target of this training is to ensure that the generated audio is indistinguishable from real human speech in both clarity and naturalness.</p><p>It performs two key tasks:</p><ul><li><strong>Speaker Encoder</strong>: This extracts unique features of a person’s voice — such as pitch and tone — acting like a voice fingerprint. It helps the model recognize who is speaking by identifying these specific features from different voices.</li><li><strong>Waveform Decoder</strong>: This component turns the processed audio features (the “instructions” for speech) into actual sounds. Trained with the target of reproducing real, high-quality speech, it generates lifelike audio that closely mimics human voices.</li></ul><h3>3. How XTTS Voice Cloning Works: A Step-by-Step Breakdown</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6CE1COzB50Ui5jfBW0LlRQ.png" /><figcaption>Inference Flows</figcaption></figure><p>The process starts by taking the reference audio and text. The audio is used to capture the speaker’s unique characteristics, while the text is converted into tokens. GPT then generates audio tokens based on these, and the HiFiGAN decoder transforms those tokens into speech, closely mimicking the speaker’s voice.</p><p>Before we dive into the step-by-step process, let’s load the pre-trained model with the following code.</p><pre>Load pre-trained model <br>import os<br>import torch<br>import torchaudio<br>from TTS.tts.configs.xtts_config import XttsConfig<br>from TTS.tts.models.xtts import Xtts<br><br>model_path = &quot;./models/tts_models--multilingual--multi-dataset--xtts_v2&quot;<br><br>config = XttsConfig()<br>config.load_json(os.path.join(model_path, &quot;config.json&quot;))<br><br>model = Xtts.init_from_config(config)<br><br># Load pretrain model<br>model.load_checkpoint(config, checkpoint_dir=model_path, use_deepspeed=False)<br>model = model.cuda()<br><br># Load three components<br>tokenizer = model.tokenizer<br>gpt = model.gpt<br>hifigan_decoder = model.hifigan_decoder</pre><p><strong>Step 1:</strong> Extract Speaker Embedding and Conditioning Latent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*YRMHKipwNb4jZWWeW7d6xA.png" /></figure><p>The reference audio is processed by two components: the <strong>Speaker Encoder</strong> and the <strong>Conditioning Encoder</strong>. This step extracts the <strong>Speaker Embedding</strong> (the unique voice characteristics) and the <strong>Conditioning Latent</strong> (additional details like style or emotion).</p><pre>def get_speaker_embedding(audio_path, sr = 22050, max_ref_length = 30):<br>    audio = load_audio(audio_path, sampling_rate=sr)<br>    audio = audio[:, : sr * max_ref_length].to(device)<br>    audio_16k = torchaudio.functional.resample(audio, sr, 16000).to(device)<br><br>    # Get speaker embedding from speaker_encoder<br>    speaker_embedding = hifigan_decoder.speaker_encoder.forward(audio_16k, l2_norm=True)<br><br>    return speaker_embedding<br><br><br>def get_cond_lattents(audio_path, sr=22050, max_ref_length=30, gpt_cond_len=6, gpt_cond_chunk_len=6):<br>    audio = load_audio(reference_audio_path, sampling_rate=sr)<br>    audio = audio[:, : sr * max_ref_length].to(device)<br>    <br>    gpt_cond_latents = model.get_gpt_cond_latents(audio, sr, length=gpt_cond_len, chunk_length=gpt_cond_chunk_len)<br><br>    return gpt_cond_latents<br><br><br>reference_audio_path = &quot;./voice/sample.wav&quot;<br><br>speaker_embedding = get_speaker_embedding(reference_audio_path)<br>condition_latents = get_cond_lattents(reference_audio_path)</pre><p><strong>Step 2 : </strong>Text Embedding Creation<br>The text is processed by the <strong>Text Embedding</strong> component, which converts it into a series of numerical “tokens” that the model can understand and use in the next steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/1*PSd3t4kYSB5Uq7zrJ8RC2w.png" /></figure><pre># Convert input text into id<br>text_tokens = torch.IntTensor(tokenizer.encode(input_text, lang=&quot;ja&quot;)).unsqueeze(0).to(device)<br><br># Padd stop and start token into text_tokens<br>text_inputs = F.pad(text_tokens, (0, 1), value=gpt.stop_audio_token)<br>text_inputs = F.pad(text_tokens, (1, 0), value=gpt.start_text_token)<br><br># Calculate text embedding<br>text_embedding = gpt.text_embedding(text_inputs) + gpt.text_pos_embedding(text_inputs)</pre><p><strong>Step 3 : </strong>Generates Audio Tokens<br>The GPT model combines the <strong>Text Embedding</strong> and the <strong>Conditioning Latent</strong> to generate a sequence of <strong>audio tokens</strong>. These tokens represent the sounds needed to create natural and expressive speech.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*6UcXDIh77noX_Qy2nQtwew.png" /></figure><pre>def generate_gpt_codes(condition_latents, text_embedding):<br>    # Concat text embedding and condition lattents<br>    embedding = torch.cat([condition_latents, text_embedding], dim=1)<br>    ...<br>    <br>    # Storage embeeding  into gpt inference<br>    gpt.gpt_inference.store_prefix_emb(embedding)<br>    <br>    # Generate audio tokens<br>    gen = gpt.gpt_inference.generate(<br>        gpt_inputs,<br>        bos_token_id=gpt.start_audio_token,<br>        pad_token_id=gpt.stop_audio_token,<br>        eos_token_id=gpt.stop_audio_token,<br>        max_length=gpt.max_gen_mel_tokens + gpt_inputs.shape[-1],<br>    )<br>    return gpt_codes<br><br># Step 3.1 Generate gpt_codes. <br>gpt_codes = generate_gpt_codes(condition_latents, text_embedding)<br><br>...<br># Step 3.2 from gpt_codes, generate gpt_latents<br>gpt_latents = gpt(<br>    text_tokens,<br>    text_len,<br>    gpt_codes,<br>    expected_output_len,<br>    cond_latents=condition_latents,<br>    return_attentions=False,<br>    return_latent=True,<br>)</pre><p><strong>Step 4 : </strong>Generate waveform using Hifigan<br>In this step, the <strong>HiFi-GAN decoder</strong> takes the audio tokens generated by the GPT model, along with the <strong>Speaker Embedding</strong>, and converts them into a raw audio waveform. This waveform represents the final speech in the cloned voice.</p><pre>wav = hifigan_decoder(gpt_latents, g=speaker_embedding).cpu().squeeze()<br><br>display(<br>    Audio(wav.detach().cpu().numpy(), rate=22400<br>))</pre><h3>Conclusion</h3><p>Voice cloning with XTTS V2 offers a powerful and efficient way to generate natural-sounding speech in a specific voice using just a few seconds of audio. By leveraging advanced models like GPT and HiFi-GAN, this technology creates lifelike audio through a streamlined process. With the steps outlined in this guide, you can start exploring how to replicate voices and unlock the full potential of voice cloning.</p><p><strong>Source Code :</strong></p><ul><li><a href="https://github.com/easyautoml/teradata/blob/main/09_TextToSpeech/02_XTTS_V2.ipynb">https://github.com/easyautoml/teradata/blob/main/09_TextToSpeech/02_XTTS_V2.ipynb</a></li></ul><p><strong>References :</strong></p><ul><li><a href="https://erogol.com/2023/09/27/xtts-v1-notes">https://erogol.com/2023/09/27/xtts-v1-notes</a></li><li><a href="https://arxiv.org/pdf/2207.09983">https://arxiv.org/pdf/2207.09983</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e17c9e0e5ba7" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/bring-your-voice-to-life-getting-started-with-coqui-xtts-v2-e17c9e0e5ba7">Bring Your Voice to Life: Getting Started with Coqui XTTS V2</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text-to-Speech for Beginners]]></title>
            <link>https://blog.gopenai.com/text-to-speech-for-beginners-f1d282f5c6bb?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1d282f5c6bb</guid>
            <category><![CDATA[text-to-speech]]></category>
            <category><![CDATA[tacotron2]]></category>
            <category><![CDATA[coqui]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Fri, 06 Sep 2024 00:44:41 GMT</pubDate>
            <atom:updated>2025-03-17T06:03:45.581Z</atom:updated>
            <content:encoded><![CDATA[<p>Combining a Text-to-Speech (TTS) model with a Large Language Model (LLM) is an exciting way to enhance your LLM’s capabilities. In this article, I’ll show you how to set up a TTS model on your local machine and explain how it works.<br>There are many TTS services available, such as Amazon Polly and Azure AI Speech, but most are not free. If you’re looking for a free TTS model that runs locally, Coqui is a great choice.</p><h3>1. Installing and Getting Started with Coqui</h3><pre>pip install TTS</pre><p><strong>Note:</strong> <em>If you want to work with Japanese, you need to install the mecab library on your system.</em></p><p>After installed TTS lib, Coqui offers two methods to run the model locally:</p><ol><li>Using the TTS command line</li><li>Using Python</li></ol><h4>Method 1 : TTS command line</h4><p><strong>Step 1 : <em>List all available open-source models</em></strong><br>(You can run the following command in your Conda environment)</p><pre>tts --list_models</pre><p><strong>Output: </strong><em>List name of pre-trained model</em></p><pre>Name format: type/language/dataset/model<br>1: tts_models/multilingual/multi-dataset/xtts_v2<br>2: tts_models/multilingual/multi-dataset/xtts_v1.1<br>...<br>41: tts_models/ja/kokoro/tacotron2-DDC<br>42: tts_models/tr/common-voice/glow-tts<br>...<br>70: tts_models/be/common-voice/glow-tts<br><br>Name format: type/language/dataset/model<br> 1: vocoder_models/universal/libri-tts/wavegrad<br> 2: vocoder_models/universal/libri-tts/fullband-melgan<br> 3: vocoder_models/en/ek1/wavegrad<br> 4: vocoder_models/en/ljspeech/multiband-melgan<br> ...<br> 14: vocoder_models/ja/kokoro/hifigan_v1<br> 15: vocoder_models/uk/mai/multiband-melgan<br> 16: vocoder_models/tr/common-voice/hifigan<br> 17: vocoder_models/be/common-voice/hifigan</pre><p>Coqui has published 70 pre-trained TTS models, including multilingual and single-language models like Japanese, English, French, and more. In addition to TTS models, Coqui has also released 17 vocoder models. In the next chapter, I will provide more details about vocoder models, but you don’t need to know about them at this stage.</p><p><strong>Step 2 : <em>Start TTS Services.</em></strong><br>From above list, start TTS service with models you want. In my case, I want to using single japanese model as following. After run this code, both TTS model and relevant vocoder model will be automaticlly downloaded into your local.</p><pre>tts-server --model_name=tts_models/ja/kokoro/tacotron2-DDC </pre><p><strong>Result : </strong>locahost:5002</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nX21jdP41uPrrx7LSuhzyQ.png" /></figure><p><a href="https://soundcloud.com/nguyen-manh-luc/jpaudio?si=bdef2a8edd2e4accbe1e3a9ed4a95ac7&amp;utm_source=clipboard&amp;utm_medium=text&amp;utm_campaign=social_sharing">Here</a> is audio it generated. Its sound really good!</p><h4>Method 2 : Using Python</h4><p>Another way to run a TTS model is by using Python.</p><pre>import torch<br>from TTS.api import TTS<br>import soundfile as sf<br>from IPython.display import Audio<br><br>tts = TTS(&quot;tts_models/ja/kokoro/tacotron2-DDC&quot;, gpu=False)<br>text = &quot;こんにちは！はじめまして、よろしくお願いいたします&quot;  <br>wav = tts.tts(text)<br>display(Audio(wav, rate=22050))</pre><h3>2. In-Depth Explanation of How It Works</h3><p>If you only want to use a pre-trained model, reading the guide here is sufficient. However, if you want to do more advanced tasks, like training the TTS model with your own voice, I highly recommend reading the next section.</p><p>As I mentioned earlier, after calling the TTS function in Python, the code will download the pre-trained model to your local environment. If you check your local files, you might find these two folders, inside that folder is pre-trained <strong>TTS model </strong>and <strong>Vocoder model</strong>:</p><pre>|- tts_models-ja-kokoro-tacotron2-DDC<br>|  |- model_file.pth<br>|  |- config.json<br>|  |- scale_stats.npy<br>|- vocoder_models-ja-kokoro-hifigan_v1<br>|  |- model_file.pth<br>|  |- config.json<br>|  |- scale_stats.npy</pre><h4>TTS Model and Vocoder Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TTtu008veZAJM9I-Zuwf1g.png" /><figcaption>Flows of generate audio from text</figcaption></figure><p><strong>TTS model</strong> are used to <strong>(1) <em>generate a mel-spectrogram</em></strong> from the input text. However, this mel-spectrogram cannot be played or heard directly; it needs to be <strong>(2) <em>converted into a waveform</em></strong>, a task performed by the <strong>Vocoder model</strong>.</p><h4>(1) Generate mel-spectrogram</h4><p>To help TTS model can generate mel-spectrogram, <strong>Tokenizer </strong>and <strong>AudioProcessor </strong>are two importance class help TTS model processing input and output data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dQG4yd_SHPkhixDenIgEKA.png" /><figcaption>More detail about generate mel-spectrogram</figcaption></figure><p>We’ll load components in Python, tokenize the text, generate a mel-spectrogram with the TTS model, and denormalize it using an audio processor.</p><p><strong>Step 1. Load Tokenizer, AudioProcessor and Models</strong><br>All settings for the Tokenizer, AudioProcessor, and TTS model are stored in the config file of TTS model, which is downloaded with the pre-trained models. Below are the details of the <strong><em>config.json</em></strong> file</p><pre>{<br>    ...<br>    &quot;precision&quot;: &quot;fp16&quot;,<br>    &quot;batch_size&quot;: 64,<br>    &quot;lr&quot;: 0.00005,<br>    ...<br><br>    &quot;audio&quot;: {<br>        &quot;fft_size&quot;: 1024,<br>        &quot;win_length&quot;: 1024,<br>        &quot;hop_length&quot;: 256,<br>        ....<br>    },<br>    ....<br>    &quot;use_phonemes&quot;: true,<br>    &quot;characters&quot;: {<br>        ...<br>        &quot;characters&quot;: &quot;ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!&#39;(),-.:;? &quot;,<br>        &quot;punctuations&quot;: &quot;!&#39;(),-.:;? &quot;,<br>        &quot;phonemes&quot;: &quot;ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz&quot;,<br>        ....<br>    },<br>    ....<br>}</pre><p>All components can be loaded with the following code:</p><pre>from TTS.config import load_config<br>from TTS.tts.utils.text.tokenizer import TTSTokenizer<br>from TTS.utils.audio import AudioProcessor<br>from TTS.tts.models.tacotron2 import Tacotron2<br>import numpy as np<br><br>device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br># Step 1. Load config file<br>config_path = os.path.join(tts_model_dir, &quot;config.json&quot;)<br>config = load_config(config_path)<br><br># Step 2. Load tokenizer<br>tokenizer, new_config = TTSTokenizer.init_from_config(config)<br><br># Step 3. Load audio processor<br>ap = AudioProcessor.init_from_config(config)<br><br># Step 4. Load TTS models<br>model = Tacotron2(new_config, ap, tokenizer, None)<br>model.load_checkpoint(config, os.path.join(tts_model_dir, &quot;model_file.pth&quot;), eval=True)<br>model = model.cuda()</pre><p>To quickly test if the tokenizer is working, try functions like the following:</p><pre>tts_model.tokenizer.phonemizer.phonemize(&quot;こんにちは&quot;)<br># Output : &#39;k|o|N|n|i|c|h|i|w|a&#39;<br><br>tokenizer.text_to_ids(&quot;こんにちは&quot;)<br># Output : [39, 43, 16, 42, 37, 31, 36, 37, 51, 29]<br><br>tokenizer.ids_to_text([39, 43, 16, 42, 37, 31, 36, 37, 51, 29])<br># Output : &#39;koNnichiwa&#39;</pre><p><strong>Step 2 : Processing text and generate mel-spectrogram</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G39Y80XETCbwoXahZJNIlQ.png" /><figcaption>Infernce Flows</figcaption></figure><p>After loading all components and confirming the tokenizer works, generate a mel-spectrogram in three simple steps</p><pre># Step 1. Convert text to ids with tokenizer<br>input_ids = tokenizer.text_to_ids(&quot;こんにちは&quot;)<br><br># Step 2. Feed to models<br># Convert to torch. Using CUDA if GPU are available<br>input_ids = np.asarray(input_ids, dtype=np.int32)<br>input_ids = torch.as_tensor(input_ids, dtype=torch.long, device=&quot;cuda&quot;)<br>input_ids = input_ids.unsqueeze(0)<br>output = model.inference(input_ids)[&#39;model_outputs&#39;][0]<br><br># Step 3. Denormalize values<br>mel_postnet_spec = output.detach().cpu().numpy()<br>mel_postnet_spec = ap.denormalize(mel_postnet_spec.T).T</pre><p>The output from model is 2 dimension array with shape is [78, 80]. 80 is number of mel frequency bands and 78 is the the time domain, when visualize it look like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*1RT13X_D1f_u0gcxaSO9UA.png" /></figure><h4>(2) Convert into waveform</h4><p>The next step is to convert the mel-spectrogram into a waveform using the Vocoder model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*bnDziHQPuy85yBPHfpw91g.png" /><figcaption>Flows of inference waveform</figcaption></figure><p>Similar to the TTS model, the Vocoder model has its own AudioProcessor, which normalizes the mel-spectrogram before feeding it into the model.</p><p>To generate the waveform, follow these two simple steps:</p><pre>from TTS.vocoder.models.gan import GAN<br>from TTS.vocoder.models import setup_model as setup_vocoder_model<br><br># Step 1. Load Vocoder and Audio Processor<br>vocoder_config = load_config(os.path.join(vocoder_model_dir, &quot;config.json&quot;))<br>vocoder_ap = AudioProcessor(verbose=False, **vocoder_config.audio)<br><br>vocoder_model = GAN.init_from_config(vocoder_config)<br>vocoder_model.load_checkpoint(vocoder_config, os.path.join(vocoder_model_dir, &quot;model_file.pth&quot;), eval=True)<br><br># Step 2. Normalize mel-spectrogram then inference<br>mel_spec = vocoder_ap.normalize(mel_postnet_spec.T)<br>mel_spec = torch.tensor(mel_spec).unsqueeze(0).to(&quot;cuda&quot;)<br><br>waveform = vocoder_model.inference(mel_spec).cpu().numpy().squeeze()</pre><p><strong>Output :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/1*i1c9njaci1NEMV4V2IhxKg.png" /></figure><h3>Conclusion</h3><p>In this article, we discussed how to use a pre-trained TTS model and provided a step-by-step understanding of how it works during inference.</p><p>I chose a single-speaker model to simplify the model’s architecture, making it easier to work with, especially given the limitations of my GPU for future steps like training the model on my own dataset.</p><p>In the next blog, I’ll dive deeper into the algorithm used by the model, the overall training process, and some basic knowledge of audio processing.</p><p>I’m still a beginner in this field and hope to connect with others who are interested in TTS models, especially for Japanese.</p><p><strong>Sample Code :</strong></p><p><a href="https://github.com/easyautoml/teradata/blob/main/09_TextToSpeech/01_HelloWorldTTS.ipynb">teradata/09_TextToSpeech/01_HelloWorldTTS.ipynb at main · easyautoml/teradata</a></p><p>My LinkedIn : <br><a href="https://www.linkedin.com/in/luc-nguyen-767a3394/">https://www.linkedin.com/in/luc-nguyen-767a3394/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1d282f5c6bb" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/text-to-speech-for-beginners-f1d282f5c6bb">Text-to-Speech for Beginners</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding the Basics of Reinforcement Learning]]></title>
            <link>https://blog.gopenai.com/understanding-the-basics-of-reinforcement-learning-a6ae303e4393?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6ae303e4393</guid>
            <category><![CDATA[gym]]></category>
            <category><![CDATA[teradata]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[bellman-equation]]></category>
            <category><![CDATA[rlhf]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Tue, 14 May 2024 00:37:54 GMT</pubDate>
            <atom:updated>2024-05-23T12:35:23.002Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you curious about a popular topic in machine learning called <strong>Reinforcement Learning from Human Feedback</strong> <strong>(RLHF)?</strong></p><p>Without a solid grasp of basic Reinforcement Learning, understanding RLHF can be quite challenging. While libraries like Huggingface and PaLM make RLHF easier to use, it’s essential to understand the core principles behind it. If you want to dive into the fundamentals, let’s explore <strong>RF</strong> together!</p><h3>1. Reinforcement learning</h3><p>Reinforcement learning (RL) is a type of machine learning where an agent learns to make decisions by trying different actions and receiving feedback in the form of rewards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*lgH-gmf6QzVlwTwMDxAgow.png" /></figure><p>This method involves learning what to do — how to map situations to actions — so as to maximize a numerical reward signal. The agent is not told which actions to take, but instead must discover which actions yield the most reward by trying them.</p><p>The best way to understand reinforcement learning and see how it works is through hands-on experience. In the next section, you will use reinforcement learning to play a game called Mountain Car. After that, we will delve deeper into a fundamental algorithm called Q-Learning, which is essential for making this process work effectively.</p><h3>2. Hands-on</h3><h4><strong>2.1 Mountain Car Game</strong></h4><p>This game is a popular test problem for reinforcement learning, the goal is to drive an underpowered car up a steep mountain road to reach the top.</p><p>The car’s engine is not strong enough to climb the hill in a direct route, so the car must learn to leverage potential energy by driving back and forth to build up enough momentum.</p><p>The episode ends if the car’s position reaches or exceeds 0.5 (the goal position on top of the right hill) or if the episode length reaches 200 steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*usAw4Uoyw9X8R9EKNws83A.png" /><figcaption>Moutain Car</figcaption></figure><p><strong>Action Space<br></strong><em>The action space is discrete, consisting of three possible actions:</em></p><ul><li><strong>0 :</strong> Accelerate to the left</li><li><strong>1</strong>: Don’t accelerate</li><li><strong>2</strong>: Accelerate to the right</li></ul><p><strong>Observation space <br></strong><em>The observation space is described by an array with 2 elements, corresponding to:</em></p><ul><li>Position of the car along the x-axis (limits ranging from -1.2 to 0.6)</li><li>Velocity of the car (limits ranging from -0.07 to 0.07)</li></ul><p><strong>Reward</strong><br>The goal is to reach the flag placed on top of the right hill as quickly as possible, as such the agent is penalised with a reward of<strong> -1</strong> for each timestep</p><p>For more details about this game, please visit this <a href="https://www.gymlibrary.dev/environments/classic_control/mountain_car/">link</a>.</p><h4><strong>2.2 Work Flows</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*za1Cij7DvEJFPJun3v2X9g.png" /></figure><p><strong>Step 1 : Discretize state<br></strong><em>In the Mountain Car game, the position and velocity of the car are continuous, meaning they can take any value within a certain range.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*pSBjil95qU3KuIk8FV4Wdw.png" /><figcaption>After convert state into discrete value</figcaption></figure><p>To simplify the continuous values of location and velocity into discrete categories, I’ll convert each into 20 distinct <strong><em>bins </em></strong>using the following function:</p><pre>def discretize_state(state):<br>    num_bins = 20<br><br>    x_min, v_min = -1.2, -0.07<br>    x_max, v_max = 0.6, 0.07<br><br>    bins = [np.linspace(x_min, x_max, num_bins), np.linspace(v_min, v_max, num_bins)]<br>    discretized_state = tuple(np.digitize(s, b) for s, b in zip(state, bins))<br>    <br>    return discretized_state</pre><p>This function divides the range of location and velocity into 20 evenly spaced intervals and assigns each continuous value to its corresponding interval, creating a discrete representation of the state.</p><p><strong>Step 2. Init Q-Table<br></strong><em>Next we will represent all combination of status and action with matrix, that matrix we call Q-Table, as its look like following image</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*pAUiB_ZfkOvqyoi3n40Y-w.png" /><figcaption>Q-Table</figcaption></figure><p>You don’t need to know why it’s called a Q-table just yet. Simply put, a Q-table is an array that represents different states. Each state in the table includes values (<strong><em>q-value</em></strong>) such as the potential reward an agent can expect if it goes left , and another value for if it goes right.</p><p>Here is the Python code used to initialize a random Q-table:</p><pre>q_table_size = [20, 20]<br>action_space = 2<br>q_table = np.random.uniform(low=-1, high=1, size=(q_table_size + [action_space]))</pre><p>Using this table, an agent can easily decide which action to take by looking at which action offers the higher potential reward according to the table.</p><p>Initially, all the values (<strong><em>q-value</em></strong>) in the Q-table are set <strong><em>randomly</em></strong>. Our goal is to update these values during the learning process so they gradually become more accurate, reflecting the real rewards the agent can expect to receive for different actions in different states.</p><p><strong>Step 3. Learning</strong></p><p><em>During the learning process, we let our agent play the game multiple times, which I refer to as multiple episodes. Throughout these episodes, we update the </em><strong><em>q-value</em></strong><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*n8ADYGt3FVbYLZcLJplaXA.png" /><figcaption>Detail flow of learning process</figcaption></figure><p>Here’s how the learning process works:</p><ul><li><strong>Select Action:</strong> The agent chooses actions based on the Q-table. Sometimes, actions are chosen randomly and sometimes based on the highest values in the Q-table. We will delve deeper into this balance between random and strategic choices in the next part.</li><li><strong>Reward and Status:</strong> After taking an <strong>action</strong>, the environment provides a reward and updates the state.</li><li><strong>q-value calculation</strong>: We use the reward and the new state to calculate a <strong>q-value. </strong>This <strong>q-value</strong> is an improved estimate of the reward that the agent can expect by taking an action, compared to a random choice.</li><li><strong>Q-Table Update</strong>: We update the Q-table with the newly calculated <strong><em>q-value</em></strong>.</li></ul><p>Here is the simplified code to demonstrate how the learning process works:</p><pre># Common functions<br>def calculate_q_value(current_state_qvalue, reward, next_state_qvalue):<br>    &quot;&quot;&quot;<br>    Calculate Q-value using Bellman equation<br>    &quot;&quot;&quot;<br>    ...<br>    return q_value<br><br><br>def select_action(q_table, state, epsilon=0):<br>    &quot;&quot;&quot;<br>    Selects the best action for a given state based on a Q-table with an option for random action choice.<br>    &quot;&quot;&quot;<br>    ...<br><br>    return action<br><br><br>def update_q_table(q_table, state, action, next_state, reward):<br>    &quot;&quot;&quot;<br>    Updates the Q-table based on the agent&#39;s action and the reward received.<br>    &quot;&quot;&quot;<br>    ...    <br>    return q_table</pre><p><strong>Learning Code :</strong></p><pre>episo = 5000<br>epsilon = 1.0<br>epsilon_decay = 0.9995<br>min_epsilon = 0.01<br><br>for e in range(0, episo):<br>    # Step 1. Get current state<br>    current_state = env.reset()<br>    done = False<br>    <br>    while not done:<br><br>        # Step 1. Take action by random or by q_table<br>        action = select_action(q_table, current_state, epsilon)<br>        <br>        # Step 2. Action<br>        next_state, reward, done, _  = env.step(action=action)<br>    <br>        # Step 3. Update q_table<br>        q_table = update_q_table(q_table, current_state, action, next_state, reward)<br>        <br>        current_state = next_state<br><br>    epsilon = max(min_epsilon, epsilon * epsilon_decay)</pre><p><strong>Step 4. Run mountain car using Q-Table</strong></p><p>After the learning process, we can now use the Q-table to control the car. <em>Here is the code:</em></p><pre>state = env.reset()<br><br>done = False<br><br>while not done:<br>    action = select_action(q_table, state, 0)<br>    <br>    state, reward, done, _ = env.step(action)<br>    <br>    env.render()</pre><p><strong><em>Result :</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*fAbUKjEO0iUoaH7tLTXY0g.gif" /><figcaption>Mountain Car using RF</figcaption></figure><p>You can find the full code at this <a href="https://github.com/easyautoml/teradata/blob/main/08_ReinforcementLearning/10_Q-Learning.ipynb">link</a></p><h3>3. Deep dive to RF</h3><h4>3.1 Calculate q-value</h4><p><em>The Bellman equation is fundamental, as it underpins the core mechanics of the process. Understanding this formula helps you grasp how the Q-learning method works and enables you to adjust key hyperparameters effectively.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*_skGlgKzWP3FmTgAFLSPIQ.png" /><figcaption>Bellman Equation</figcaption></figure><ul><li><strong>𝑄(𝑠,𝑎)</strong> : is the current q-value for the state-action pair <strong>(s, a)</strong>.</li><li><strong><em>α</em>:</strong> is the <strong>learning rate</strong>, which determines how much new information overrides old information.</li><li><strong><em>R</em>(<em>s</em>,<em>a</em>)</strong>: is the reward received after taking action <strong><em>a</em></strong> in state <strong>𝑠</strong>.</li><li><strong><em>γ</em></strong>: is the <strong>discount factor</strong>, which quantifies how much importance is given to future rewards.</li><li><strong>max <em>a</em>′​<em>Q</em>(<em>s</em>′,<em>a</em>′)</strong>: is the maximum predicted reward for the next state <strong><em>s</em></strong><strong>′</strong>, looking at all possible actions <strong><em>a</em></strong><strong>′</strong></li></ul><pre>def calculate_q_value(current_state_qvalue, reward, next_state_qvalue):<br>    &quot;&quot;&quot;<br>    Calculate Q-value using Bellman equation<br>    &quot;&quot;&quot;<br><br>    learning_rate = 0.1<br>    discount_factor = 0.9<br><br>    return (1 - learning_rate) * current_state_qvalue + learning_rate * (reward + discount_factor * next_state_qvalue)</pre><p><strong>Discount Factor</strong></p><p><em>To understand how the discount factor is used, let’s imagine the game is simplified as shown in the image.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*7lFwnRiE3l2U57nN22gtVQ.png" /><figcaption>Moutain Car in simplify</figcaption></figure><p><strong><em>We have two scenarios:</em></strong></p><ul><li><strong>Scenario 1: </strong>Move right twice. The car reaches the goal and receives a reward of 1.</li><li><strong>Scenario 2: </strong>Move left first, then right three times. The car still reaches the goal and receives a reward of 1.</li></ul><p>Obviously, Scenario 1 is better than Scenario 2 because it only requires two moves, while Scenario 2 requires four moves.</p><p>To help the agent understand that Scenario 1 is better than Scenario 2, we use the discount factor.</p><h4>3.2 Select Action</h4><p>Here are the details of the select_actionfunction.</p><pre>def select_action(q_table, state, epsilon=0):<br>    ...<br><br>    if np.random.random() &gt;= epsilon:<br>        # Take an random action by using q-value<br>        action = np.argmax(q_table[d_state])<br>    else:<br>        # Take an action by random<br>        action = np.random.choice([0, 1, 2])<br><br>    return action</pre><p>Code when we call select_action</p><pre>epsilon = 1.0<br>epsilon_decay = 0.9995<br>min_epsilon = 0.01<br><br>for e in range(0, episo):<br>  ...<br>  while not done:<br>    action = select_action(q_table, current_state, epsilon)<br>    ...<br>  epsilon = max(min_epsilon, epsilon * epsilon_decay)</pre><p>You can see that <strong>epsilon</strong> is initially set to 1 and is then reduced by multiplying it by 0.995 after each episode.</p><p>This means that at the beginning, the agent will take random actions, which helps it explore the environment. As <strong>epsilon</strong> decreases and approaches 0.001, the agent will rely less on random actions and more on the learned<strong> Q-values </strong>to choose the best action.</p><p>This strategy is called <strong><em>ϵ</em></strong><strong>-greedy</strong>. This method helps balance exploration and exploitation.</p><h3>Conclusion</h3><p>Using the Mountain Car game and the Q-learning method is one of the simplest ways to understand the fundamentals of Reinforcement Learning. However, in real-world applications and more complex games, we need more powerful algorithms for the computer to play the game effectively. In the next part, I will discuss Deep Q-Network (DQN), a new approach that addresses the challenges of continuous environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6ae303e4393" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/understanding-the-basics-of-reinforcement-learning-a6ae303e4393">Understanding the Basics of Reinforcement Learning</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Reinforcement Learning from Human Feedback]]></title>
            <link>https://blog.gopenai.com/deep-dive-into-reinforcement-learning-from-human-feedback-ae7d3d8ef38e?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae7d3d8ef38e</guid>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[feedback]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[reward-modeling]]></category>
            <category><![CDATA[rlhf]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Tue, 12 Mar 2024 05:57:19 GMT</pubDate>
            <atom:updated>2024-05-23T12:35:25.321Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous <a href="https://medium.com/@lucnguyen_61589/things-need-to-know-before-fine-tuning-llm-models-56f8d77abb55">post</a>, I covered how to build a Large Language Model (LLM), including training it with online data, using Supervised Fine-Tuning, and applying Reinforcement Learning from Human Feedback (RLHF) to make responses safe and human-like. This post will focus on RLHF, explaining its basics, development process, and how it operates.</p><h3>1. Overall about RLHF</h3><p>As mentioned previously, we use <strong>Reinforcement Learning from Human Feedback </strong>(<strong><em>RLHF</em></strong>) when we want our Large Language Model (LLM) to respond in ways that are polite, non-toxic, and helpful, especially when these requirements are difficult to define clearly.</p><p>For example, it’s not always straightforward to determine what constitutes a <strong><em>“toxic”</em></strong> response. In such situations, <strong>RLHF</strong> becomes essential. The core idea behind <strong>RLHF </strong>is to rely less on predefined rules and more on human judgment.</p><p>We gather feedback from people to identify which responses are more appropriate, less toxic, etc. This collected feedback is then used to train the model, enhancing its ability to generate better responses.</p><p>Todo : Split it into three Steps.</p><ul><li>Step 1. Collect Data From Human Feedback</li><li>Step 2 : Training Reward Model</li><li>Step 3 : Optimiza Policy</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TDjSd3U2IgTBUAGniBpvww.png" /></figure><p><strong>The process typically involves the following stages:</strong></p><ul><li><strong>Step 1 : Collect Data from Human Feedback</strong></li><li><strong>Step 2 : Training Reward Modeling (RM)</strong><br>At this stage, people review the LLM model’s answers to judge their quality, relevance, and how well they meet important standards like making sense, being accurate, and safe. Their feedback trains a <strong><em>reward model</em></strong>, which learns to identify high-quality responses according to what people think are good answers.</li><li><strong>Step 3 : Optimize Policy <br></strong>In the Optimize Policy stage, we use the reward model to guide adjustments to the AI. By employing reinforcement learning methods, such as Proximal Policy Optimization (PPO), we fine-tune the AI’s behavior. Here, the AI proposes answers, the reward model assesses them, and then PPO updates the AI to improve its answers based on these evaluations. This process aims to produce answers that not only score well according to the reward model but also align closely with human expectations.</li></ul><h3>2. Collect Data From Human Feedback</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*sd2NfpAVUgnJS2cxkwbhmw.png" /></figure><p>To collect this data, we could present the question to the model, have it generate two response options, and then select the option we think is better than the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*izYVHMbueDMWr7Ab54s-yg.png" /><figcaption>Anthropic/hh-rlhf dataset</figcaption></figure><p>This task can be time-consuming, so using publicly available data is an alternative. A popular and accessible dataset is Anthropic/hh-rlhf, available for download from Hugging Face.</p><pre>from datasets import load_dataset<br>dataset = load_dataset(&quot;Anthropic/hh-rlhf&quot;, data_dir=&quot;harmless-base&quot;)</pre><p>This dataset include two parts chosen and rejected. Here are one samples :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*77lYEProsKnaepK5qWedBg.png" /><figcaption>Sample of chosen and rejected</figcaption></figure><p>We have two conversations between a Human and an Assistant, both featuring the same question and answer.</p><p>However, when the human asks which cuss word is the Assistant’s favorite, we expect the model to respond with “<strong><em>I haven’t even thought about it</em></strong>” rather than answering with “<strong><em>ass</em></strong>.”</p><h3>3. <strong>Training Reward Modeling (RM)</strong></h3><p>A Reward Model is a model trained from human feedback data. It takes inputs that are the LLM model’s responses and outputs scores indicating how good or bad these responses are based on human preferences.</p><p>This model acts like a teacher, telling the AI which response is better than another. It is used during the reinforcement learning step to guide the AI in making better decisions.</p><p>Here are more details about creating reward models using HuggingFace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*yRFLGc7A8fwcjepKcN2xpw.png" /><figcaption>Flows of build Reward Model</figcaption></figure><p>To simplify the task of creating a Reward model, I’ll use the method supported by Hugging Face, which involves the following steps:</p><ul><li><strong>Step 1 : </strong><em>Load a pre-trained model and tokenizer</em></li><li><strong>Step 2 : </strong><em>Pre-process data using the tokenizer to format the data according to Hugging Face’s requirements.</em></li><li><strong>Step 3 : </strong><em>Create the Reward model using the RewardModel class. The inputs for this task are the model loaded in the first step and the data pre-processed in the second step.</em></li></ul><p><strong>Step 1 : Load Pretrained Model and Tokenizer<br></strong>In this task, I’ll use the model named <strong><em>“gpt2”</em></strong>, but you can choose other models like BERT, Llama or mistralai for this task.</p><pre>from transformers import AutoModelForSequenceClassification, AutoTokenizer<br><br>model = AutoModelForSequenceClassification.from_pretrained(&quot;gpt2&quot;, device_map={&quot;&quot;:0})<br>tokenizer = AutoTokenizer.from_pretrained(&quot;gpt2&quot;)</pre><p>If you’re not familiar with <strong>tokenizers</strong>, here’s a simple explanation:</p><blockquote>A tokenizer transforms text into a format understandable by a model. It takes text as input and produces a list of numbers (tokens) that represent each piece of the text, along with attention masks to highlight the parts to focus on.</blockquote><p><strong>Step 2 : Pre-Processing Data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*ExP3BAzMsfHcL3U51s_KzQ.png" /><figcaption>Flows of processing data</figcaption></figure><p>Using a tokenizer aids in data processing. The tokenizer takes text as input and produces output consisting of input_ids and attention_mask. We will process both chosen and rejected data</p><pre>def preprocess_examples(examples, tokenizer):<br>    preprocessed = {<br>        &quot;input_ids_chosen&quot;: [],<br>        &quot;attention_mask_chosen&quot;: [],<br>        &quot;input_ids_rejected&quot;: [],<br>        &quot;attention_mask_rejected&quot;: [],<br>    }<br>    <br>    for chosen_text, rejected_text in zip(examples[&quot;chosen&quot;], examples[&quot;rejected&quot;]):<br>        tokenized_chosen = tokenizer(chosen_text, truncation=True)<br>        tokenized_rejected = tokenizer(rejected_text, truncation=True)<br><br>        preprocessed[&quot;input_ids_chosen&quot;].append(tokenized_chosen[&quot;input_ids&quot;])<br>        preprocessed[&quot;attention_mask_chosen&quot;].append(tokenized_chosen[&quot;attention_mask&quot;])<br>        preprocessed[&quot;input_ids_rejected&quot;].append(tokenized_rejected[&quot;input_ids&quot;])<br>        preprocessed[&quot;attention_mask_rejected&quot;].append(tokenized_rejected[&quot;attention_mask&quot;])<br><br>    return preprocessed<br><br><br>dataset = dataset.map(preprocess_examples, batched=True, num_proc=1)<br>dataset = dataset.filter(lambda x: len(x[&quot;input_ids_chosen&quot;]) &lt;= 512 and len(x[&quot;input_ids_rejected&quot;]) &lt;= 512)</pre><p><strong>Step 3 : Build Reward Model<br></strong>The method I’m using for fine-tuning the reward model is full fine-tuning. You can use the PEFT method to reduce the required memory if necessary.</p><pre>from peft import LoraConfig, TaskType<br>from transformers import AutoModelForSequenceClassification, AutoTokenizer<br>from trl import RewardTrainer, RewardConfig<br><br># 1. Setting tokenizer<br>tokenizer.pad_token = tokenizer.eos_token<br>tokenizer.padding_side = &quot;right&quot;<br><br>model.config.pad_token_id = model.config.eos_token_id<br>model.config.use_cache = False<br><br># 2. Setting training config<br>training_args = RewardConfig(<br>    output_dir=&quot;output&quot;,<br>    per_device_train_batch_size=8,<br>    per_device_eval_batch_size=8,<br>    num_train_epochs=3,<br>    gradient_accumulation_steps=4,<br>    gradient_checkpointing=True,<br>    learning_rate=1.41e-5,<br>    report_to=&quot;tensorboard&quot;,<br>    remove_unused_columns=False,<br>    optim=&quot;adamw_torch&quot;,<br>    logging_steps=10,<br>    eval_steps=0.1,<br>    save_steps=0.25,<br>    bf16=True,<br>    evaluation_strategy=&quot;steps&quot;,<br>    max_length=2048,<br>)<br><br><br>trainer = RewardTrainer(<br>    model=model,<br>    args=training_args,<br>    tokenizer=tokenizer,<br>    train_dataset=dataset[&#39;train&#39;],<br>    eval_dataset=dataset[&#39;test&#39;]<br>)<br><br>trainer.train()<br>trainer.model.save_pretrained(&quot;./reward_model&quot;)</pre><p>Results after training :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*lmQTq2S1znfsweL5ZXUfgQ.png" /></figure><p>The accuracy I achieved after training this model is approximately 70%, which is quite decent for reward models.</p><h3>3. Fine-Tuning model with RF</h3><p>Now is time to make LLM Model response like human, if you still not familar with basic fundamental of RF such as Policy, Observation, Reward etc, I suggest reading this <a href="https://medium.com/@lucnguyen_61589/understanding-the-basics-of-reinforcement-learning-a6ae303e4393">blog </a>before continous this part.</p><h4>Overall Fine-Tuning Process</h4><h3>4. Conclusion</h3><p>Creating a reward model involves getting detailed feedback from people, using datasets like Anthropic/hh-rlhf, and applying smart methods to make the most of this feedback. The goal is to make AI respond like humans, helping it become a better partner for us.</p><p>Hitting 70% accuracy shows our approach works and can get even better. By improving these models, we make AI more helpful in our everyday lives, aiding our decisions and interactions.</p><p>Next, we’re not stopping here. Our upcoming post will explore Proximal Policy Optimization (PPO), a key technique for refining AI further. We’ll look into how PPO works and its importance. Stay with us as we delve deeper into making AI that truly benefits us, step by step.</p><p><strong>Source code : </strong><a href="https://github.com/easyautoml/teradata/blob/main/07_RLHF/01_RewardModel.ipynb">https://github.com/easyautoml/teradata/blob/main/07_RLHF/01_RewardModel.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae7d3d8ef38e" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/deep-dive-into-reinforcement-learning-from-human-feedback-ae7d3d8ef38e">Deep Dive into Reinforcement Learning from Human Feedback</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Constructing Knowledge Graphs: A Guide to Using OpenAI and Pyvis]]></title>
            <link>https://blog.gopenai.com/constructing-knowledge-graphs-a-guide-to-using-openai-and-pyvis-959eda240c4d?source=rss-aee34c8a9754------2</link>
            <guid isPermaLink="false">https://medium.com/p/959eda240c4d</guid>
            <category><![CDATA[knowledge-graph]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Luc Nguyen]]></dc:creator>
            <pubDate>Mon, 26 Feb 2024 03:29:55 GMT</pubDate>
            <atom:updated>2024-05-23T12:35:20.733Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Knowledge graphs have revolutionized the way we organize and analyze data. By visually depicting entities and their interconnections, they offer a powerful tool for data exploration and gaining insights. The integration of Large Language Models (LLMs) like OpenAI has further enhanced the automation potential for creating these intricate structures. This guide will clarify the concept of knowledge graphs, introduce basic terminology, the creation process, and the role of LLMs in automating their generation.</em></p><h3>Understanding Knowledge Graphs</h3><p>A knowledge graph organizes data into a visual network of nodes (entities such as people, places, or events) and edges (the relationships between them), with labels providing additional details on these relationships.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/563/1*dZSiIpyeJ4DbSn9eNV8bag.png" /><figcaption>Example of Knowledge Graph</figcaption></figure><p>For example, in a simple knowledge graph about hobbies:</p><ul><li><strong>Nodes </strong>could be you, a book, and a library.</li><li><strong>Edges</strong> connect you to the book (showing “owns”) and the book to the library (indicating “borrowed from”).</li><li><strong>Labels</strong> might specify “borrowed on January 1st” for the timeline.</li></ul><p>This setup helps visualize complex relationships in a straightforward, easy-to-understand way.</p><h3>When to Use Knowledge Graphs</h3><p><em>Use a knowledge graph in the following scenario:</em></p><ul><li>Your data is complex and interconnected, like a web of friends in a social network where everyone is connected in various ways.</li><li>You need to combine information from different places, similar to putting together pieces of a puzzle from different boxes.</li><li>You plan to frequently update or change how your data is organized, akin to rearranging books on a shelf without needing a new shelf.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*Qz9eF-G-1o63KfpS8Z82kg.png" /></figure><p>Imagine a social network where John is friends with Mary and works at Company X. Mary likes hiking and also works at Company X. A knowledge graph can easily show these connections, making it clear how John, Mary, and Company X are related, much like drawing a map that shows how different landmarks are connected by roads.</p><h3>Creating a Knowledge Graph with Pyvis</h3><p>While graph databases like Neo4J or Amazon Neptune are common tools for creating knowledge graphs, Python’s Pyvis library offers a simpler, more accessible approach. Here’s a basic example:</p><pre>from pyvis.network import Network<br><br># Create a new network<br>net = Network()<br><br># Add nodes<br>net.add_node(&quot;Mary&quot;, label=&quot;Mary&quot;)<br>net.add_node(&quot;John&quot;, label=&quot;John&quot;)<br>net.add_node(&quot;Teradata&quot;, label=&quot;Teradata&quot;)<br><br># Add edges to represent the working relationships<br>net.add_edge(&quot;Mary&quot;, &quot;Teradata&quot;, title=&quot;works at&quot;)<br>net.add_edge(&quot;John&quot;, &quot;Teradata&quot;, title=&quot;works at&quot;)<br><br># Save and show the network<br>net.show(&quot;index.html&quot;, local=False, notebook=False)</pre><p><strong>Result :</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*--FLuw4HKkmzW3ABtJuSNA.png" /></figure><h3>Automating Knowledge Graph Creation with OpenAI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BKyYUXrAXCuTHHXl4S87Wg.png" /><figcaption>Flows of create knowledge graph</figcaption></figure><p>The advent of LLMs like OpenAI has made it possible to automate the extraction of entities and relationships from textual data, significantly simplifying the creation of knowledge graphs. The process involves:</p><ol><li><strong>Extracting Nodes and Edges</strong>: Developing a function to identify entities and their connections within documents using OpenAI’s API.</li><li><strong>Parsing Data:</strong> Converting the extracted information into a structured format, such as a dictionary.</li><li><strong>Building the Graph</strong>: Utilizing Pyvis or similar tools to visually represent the parsed data as a knowledge graph.</li></ol><p><em>In this example, we will create a knowledge graph from documents related to </em><strong><em>Barack Obama, Donald Trump and Joe Biden</em></strong><em>.</em></p><p><strong>Step 1 </strong>: <strong>Extracting Nodes and Edges</strong></p><pre># This document was collected from Wikipedia.<br>documents = &quot;&quot;&quot;<br>    Barack Hussein Obama II (/bəˈrɑːk huːˈseɪn oʊˈbɑːmə/ (audio speaker iconlisten); born August 4, 1961)[1] is an American politician and attorney. <br>    He was the 44th president of the United States from 2009 to 2017. He was the first African-American president in U.S. history. <br>    As a member of the Democratic Party, he also served as member of the Illinois Senate from 1997 to 2004 and a United States senator from Illinois from 2005 to 2008.<br>    ...<br>    Joseph Robinette Biden Jr. (/ˈbaɪdən/ ⓘ BY-dən; born November 20, 1942) is an American<br>    ...<br>    Donald John Trump (born June 14, 1946) is an American politician, media personality<br>    ...<br>&quot;&quot;&quot;</pre><pre>from langchain.chains import LLMChain<br>from langchain.prompts import PromptTemplate<br>from langchain_openai import OpenAI<br>import json<br><br>instruction = &quot;&quot;&quot;<br>    You are a data scientist working for a company that is building a graph database. Your task is to extract information from data and convert it into a graph database.<br>    Provide a set of Nodes in the form [ENTITY_ID, TYPE, PROPERTIES] and a set of relationships in the form [ENTITY_ID_1, RELATIONSHIP, ENTITY_ID_2, PROPERTIES].<br>    It is important that the ENTITY_ID_1 and ENTITY_ID_2 exists as nodes with a matching ENTITY_ID. If you can&#39;t pair a relationship with a pair of nodes don&#39;t add it.<br>    When you find a node or relationship you want to add try to create a generic TYPE for it that  describes the entity you can also think of it as a label.<br>    Note that output must in dictionary type<br>    <br>    Example:<br>    Input : Alice lawyer and is 25 years old and Bob is her roommate since 2001. Bob works as a journalist. Alice owns a the webpage www.alice.com and Bob owns the webpage www.bob.com.<br>    Output : <br>    { &quot;Nodes&quot;: [&quot;alice&quot;, &quot;Person&quot;, {&quot;age&quot;: 25, &quot;occupation&quot;: &quot;lawyer&quot;, &quot;name&quot;:&quot;Alice&quot;}], [&quot;bob&quot;, &quot;Person&quot;, {&quot;occupation&quot;: &quot;journalist&quot;, &quot;name&quot;: &quot;Bob&quot;}], [&quot;alice.com&quot;, &quot;Webpage&quot;, {&quot;url&quot;: &quot;www.alice.com&quot;}], [&quot;bob.com&quot;, &quot;Webpage&quot;, {&quot;url&quot;: &quot;www.bob.com&quot;}],<br>      &quot;Edges&quot;: [&quot;alice&quot;, &quot;roommate&quot;, &quot;bob&quot;, {&quot;start&quot;: 2021}], [&quot;alice&quot;, &quot;owns&quot;, &quot;alice.com&quot;, {}], [&quot;bob&quot;, &quot;owns&quot;, &quot;bob.com&quot;, {}]<br>    }<br>&quot;&quot;&quot;<br><br># Create prompt template<br>prompt_template = PromptTemplate.from_template(&quot;&quot;&quot;<br>    {instruction}<br>    Here is document. {documents}<br>&quot;&quot;&quot;<br>)<br># Setting OpenAI API<br>llm = OpenAI(max_tokens=-1)<br><br># Create Chain<br>llm_chain = LLMChain(prompt=prompt_template, llm=llm)<br><br>result = llm_chain.run(instruction=instruction, documents=documents)</pre><p>Result from this function as following :</p><pre>{<br>  &quot;Nodes&quot;: [<br>    [&quot;barack_obama&quot;, &quot;Person&quot;, {&quot;name&quot;: &quot;Barack Obama&quot;}],<br>    [&quot;michelle_robinson&quot;, &quot;Person&quot;, {&quot;name&quot;: &quot;Michelle Robinson&quot;}],<br>    [&quot;malia_obama&quot;, &quot;Person&quot;, {&quot;name&quot;: &quot;Malia Obama&quot;}],<br>    ...<br>    [&quot;gun_control&quot;, &quot;Government Policy&quot;, {&quot;title&quot;: &quot;Gun Control&quot;}]<br>  ],<br>  &quot;Edges&quot;: [<br>    [&quot;barack_obama&quot;, &quot;married_to&quot;, &quot;michelle_robinson&quot;, {&quot;start&quot;: 1992}],<br>    [&quot;barack_obama&quot;, &quot;parent_of&quot;, &quot;malia_obama&quot;, {}],<br>    [&quot;barack_obama&quot;, &quot;parent_of&quot;, &quot;sasha_obama&quot;, {}],<br>    ...<br>    [&quot;barack_obama&quot;, &quot;proposed&quot;, &quot;gun_control&quot;, {&quot;start&quot;: 2012}]<br>  ]<br>}</pre><p><strong>Step 2:</strong> <strong>Parsing Data</strong></p><pre>def parser(result):<br>    &quot;&quot;&quot;<br>    Convert string to dictionary with &quot;Nodes&quot; and &quot;Edges&quot; as keys.<br>    &quot;&quot;&quot;<br>    return json.loads(result)</pre><p><strong>Step 3 : Building the Graph</strong></p><pre>from pyvis.network import Network<br><br># Define knowledge graph<br>net = Network(<br>    notebook=True, <br>    cdn_resources=&quot;remote&quot;, <br>    bgcolor=&quot;#222222&quot;,<br>    font_color=&quot;white&quot;,<br>    height=&quot;750px&quot;,<br>    width=&quot;100%&quot;,<br>    select_menu=True,<br>    filter_menu=True,<br>)<br><br># Create Nodes and Edges<br>for node in result_parsed.get(&quot;Nodes&quot;):<br>    net.add_node(node[0], label=node[0], title=str(node[2]))<br>   <br>for edge in result_parsed.get(&quot;Edges&quot;):<br>    from_node, title_edge, to_node = edge[0], edge[1], edge[2]<br>    try:<br>        net.add_edge(from_node, to=to_node, title=title_edge)<br>    except:<br>        net.add_node(to_node, label=to_node, title=str(to_node))<br>        net.add_edge(from_node, to=to_node, title=title_edge)</pre><p><strong><em>Result :</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*JJd72OygZyVctgrk92KKHg.png" /></figure><h3>Conclusion</h3><p>Knowledge graphs offer a powerful framework for structuring and analyzing complex data, making intricate relationships understandable and visually accessible. The integration of Large Language Models (LLMs) like OpenAI’s with visualization tools such as Pyvis significantly enhances our ability to automate the creation of these graphs.</p><p>This synergy simplifies handling large volumes of data and democratizes the process, enabling users from various domains to generate valuable insights efficiently. As we navigate an increasingly data-rich world, the combination of advanced computational models and user-friendly visualization tools will be pivotal in uncovering and understanding the hidden connections within our data, promising new avenues for exploration and discovery.</p><p><strong><em>Source Code :</em></strong></p><p><a href="https://github.com/easyautoml/teradata/blob/main/06_KnowledgeGraph/01_BasicOfKnowledGraph.ipynb">https://github.com/easyautoml/teradata/blob/main/06_KnowledgeGraph/01_BasicOfKnowledGraph.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=959eda240c4d" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/constructing-knowledge-graphs-a-guide-to-using-openai-and-pyvis-959eda240c4d">Constructing Knowledge Graphs: A Guide to Using OpenAI and Pyvis</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>