<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sijuade Oguntayo on Medium]]></title>
        <description><![CDATA[Stories by Sijuade Oguntayo on Medium]]></description>
        <link>https://medium.com/@sijpapi?source=rss-deb787286fa9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*q7ne4YHtvT7ZoP1uD29aaA.jpeg</url>
            <title>Stories by Sijuade Oguntayo on Medium</title>
            <link>https://medium.com/@sijpapi?source=rss-deb787286fa9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:10:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sijpapi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My Cosmos World Model Didn’t Know What to Do With a Physics Engine]]></title>
            <link>https://levelup.gitconnected.com/my-cosmos-world-model-didnt-know-what-to-do-with-a-physics-engine-255fa9b4820d?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/255fa9b4820d</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[large-world-model]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 14:39:59 GMT</pubDate>
            <atom:updated>2026-10-09T14:39:59.518Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cosmos <strong>World Model Didn’t Know What to Do With a </strong>Physics Engine</h3><p><em>Pointing NVIDIA’s Cosmos 3 at a MuJoCo simulator: what failed, what fine-tuning fixed, and what it didn’t</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxrzbkdrWf_niFQ9fFDIqA.png" /><figcaption><em>Top: MuJoCo frames of the gripper grasping and lifting the cube. Bottom: Cosmos 3’s prediction from the same first frame and actions, with a red object appearing from nowhere.</em></figcaption></figure><p>NVIDIA’s Cosmos 3 is an action-conditioned world foundation model. Given a frame and a sequence of robot actions, it predicts the video that follows; Cosmos calls this mode forward dynamics. In principle that makes it a stand-in for a simulator: a policy picks an action, Cosmos predicts what the camera sees next, and no physics engine is needed.</p><p>I wanted to use it that way, for pick-and-place, with data from a MuJoCo simulator that renders synthetic scenes. It didn’t work out of the box. Cosmos’s action-conditioning works on real camera footage and barely at all on my renders. Fine-tuning fixes most of that. Then a second limit appears that has nothing to do with the renders.</p><h4><strong>What kind of world model this is</strong></h4><p>Cosmos 3 is, first, a video generation model, and even more technically, a world foundation model: given a prompt or an image, it generates video. What makes it a world model for control is action-conditioning: it also takes the actions a robot will perform, and the video it outputs has to be the consequence of those actions. A video generator’s job is a plausible continuation but an action-conditioned model’s job is the continuation that follows from these particular actions, and if we change the actions the video must change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*37jedGxw0O684Mc5.png" /><figcaption>Nvidia Cosmos — <a href="https://mixpeek.com/blog/world-foundation-model">https://mixpeek.com/blog/world-foundation-model</a></figcaption></figure><p>Cosmos also commits to pixels. It predicts the next frames themselves, as compressed video latents that decode back to images. <strong>Not every world model does</strong>. Dreamer predicts a compact latent state, the reward and whether the episode ends. JEPA-style models go further and predict only the embedding of the future, on the argument that pixels make the model spend capacity on texture and lighting that don’t matter for deciding what to do.</p><p>The pixel route has advantages: we can watch the output and judge it, and the model can learn from any video, with or without actions. The cost is the objective: it rewards convincing video, and convincing video that ignores the actions scores well on it. Both failures in this article are about that objective and what it was learned from.</p><h4><strong>The first result looked right</strong></h4><p>The simulator’s actions are 7 numbers; Cosmos’s robot domains expect 10, and a small conversion bridges them. Fed one frame and a real grasp sequence, Cosmos returned a plausible-looking continuation. Before trusting it, I ran a control: the same frame with an all-zero “stay” action. We can measure a prediction’s motion by summing the differences between consecutive frames. The real grasp scored 0.631 and the no-op 0.638. Negating the action’s sign left the motion where it was. Cosmos was generating generic motion and ignoring the numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hvQN93jazOVCT0m42gdOag.gif" /><figcaption><em>The same frame, predicted with the real grasp action (left) and with no action at all (right).</em></figcaption></figure><p><strong>Scale</strong> My actions are in metres and radians. Sweeping a multiplier on them, the motion stayed at the noise floor up to 10× and rose sharply between 20× and 40×. Cosmos’s inverse-dynamics mode, asked what actions it infers from my real footage, reported vertical movements about 12.5× my raw values.</p><p><strong>The renderer</strong> Scaled into the responsive range, the response on my renders was still weak, the cube moved the wrong way, and at larger multipliers the model hallucinated, which is where the red object in the opening figure comes from. The test that settles it holds the action and scale fixed and swaps only the conditioning image for the real photo that ships with the checkpoint:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E2l57XQEKCTBJkkKAeeTKQ.jpeg" /></figure><p>On a real image, my converted actions produce a strong response that scales cleanly towards the model’s own ceiling. Swap back to my render with the same action, and it’s weak or hallucinated.</p><p>Every one of Cosmos 3’s action-conditioned domains (Bridge, DROID, UMI, Fractal and the rest) was trained on real camera footage. The model has learned how actions move pixels in real video, and it’s unable to apply that to a synthetic frame.</p><h4><strong>A bigger model fails differently</strong></h4><p>I was using Cosmos3-Edge, a distilled 4-billion-parameter checkpoint. Cosmos3-Nano has 16 billion and isn’t distilled. It took 30.6 GiB of a 46 GiB GPU to load, and on the same grasp test its motion scores were 0.649 and 0.885 against a no-op of 0.731: the same weak response, with a different failure. Instead of a phantom object, the gripper’s base detached from the arm and floated out of the frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BvjSUVT1j9ezGnvphCbD3g.png" /><figcaption><em>Cosmos3-Nano on the same first frame and actions. The gripper detaches and floats upward.</em></figcaption></figure><p>The domain gap is a property of what the whole model family was trained on, and size doesn’t close it.</p><h4><strong>Making the renders look like photographs helps a little</strong></h4><p>I added a post-processing pass to the simulator: Gaussian blur, sensor noise, vignetting, chromatic aberration and colour jitter, applied after rendering so the physics and recorded actions are untouched. On the clean render, the predicted cube motion pointed the wrong way at every scale; on the processed render it pointed the right way from 18× upward. But the magnitude still undershot the real motion by 2–3×, and at 60× the model hallucinated again. The flat shading is still there under the blur. It isn’t the fix.</p><h4><strong>Fine-tuning fixes direction, not magnitude</strong></h4><p>Inside Cosmos 3’s transformer, the only place action tokens can influence the predicted video is a set of attention projections: four modules in each of 28 layers, 112 in all.</p><p>That makes the fine-tune, which NVIDIA calls post-training, narrow. I put LoRA, i.e. a small set of extra trainable weights bolted onto a frozen model, on exactly those 112 modules: 3.2 million trainable parameters out of 3.37 billion, or 0.095%. Cosmos also looks up its action-projection weights by domain ID in a table of 32 slots; the shipped domains use 21. I gave my simulator slot 21 and trained its rows from scratch, so the pretrained domains never move and the new slot learns its own action scale from the data.</p><p>After an overnight run of 21,483 steps on 800 episodes, the predicted cube moved in the same direction as the real one in 84% of 25 held-out test episodes, up from 60%. The hallucinations are gone.</p><p>The ratio of motion between a real action and a no-op was 1.654 before training and 1.644 after. The model has learned which way things go, and it hasn’t learned to move more for a bigger action. Since 26× more steps than the pilot didn’t move that ratio, I read it as a capacity limit of a LoRA this narrow. The next lever is LoRA on the pathway’s MLP blocks as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*wBttTUiFHxNQaHkfAOo51g.gif" /><figcaption><em>The same real frame and real action sequence. Top: MuJoCo ground truth. Middle: stock Cosmos3-Edge. Bottom: after fine-tuning.</em></figcaption></figure><h4><strong>The second wall</strong></h4><p>To be an environment, a world model has to run on its own output: the policy picks an action, Cosmos predicts the next frame, the policy sees that frame and picks again. The policy here is a ResNet18 trained by behaviour cloning; it succeeds on 86–90% of real-simulator episodes, so it isn’t the weak link.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-djgmNz5NdKT_4Eizg4ew.jpeg" /><figcaption>Policy Model — Resnet18 with action prediction</figcaption></figure><p>I ran that loop for 128 steps in two ways.</p><p><strong>Anchored</strong>: Cosmos predicts each chunk from a real simulator frame, so errors can’t compound.</p><p><strong>Pure imagination</strong>: Cosmos sees frame 0 and only its own predictions after that.</p><p>Anchored tracks the real rollout throughout. Pure imagination holds for about 3 chunks, then the geometry distorts (the arm thins, objects blur into blobs) and, from chunk 4, the lighting washes out.</p><p>That comparison has a confound. The policy reacts to what it sees, so once Cosmos drifts it may pick different actions. To isolate the model, I ran a third condition: Cosmos predicting from its own output, driven by the real, correct action sequence throughout, with no policy at all. It drifts just as badly, from about the same chunk, with the same symptoms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*4FR-8sBseH2uP4CKMU15UA.gif" /><figcaption><em>From the same opening frame: the real rollout; Cosmos anchored each chunk; the policy and Cosmos in pure imagination; Cosmos in pure imagination with the real actions. The bottom two drift alike.</em></figcaption></figure><p>So the instability belongs to Cosmos’s self-conditioning. It shows up after the domain gap is fixed, so it’s a second, separate limit.</p><h4><strong>The obvious fix didn’t work</strong></h4><p>The diagnosis points at exposure bias: the model was only ever trained to continue from clean frames, and at inference it has to continue from frames carrying its own artefacts. One possible remedy, called scheduled sampling, is to train on our own imperfect outputs some of the time.</p><p>I trained a fresh adapter from the base model with the same recipe and two additions: about 40% of training windows got a perturbed conditioning frame, and periodically the loop rolled the current model forward to generate a conditioning frame and trained the next chunk from it. The run went 18 hours and 34,315 steps on 2,000 episodes.</p><p>On three episodes, in both loops, measured as pixel distance from the real frame at each step, the drift was no better than the first adapter’s and slightly worse on most. What that rules out is that a sparse dose of self-conditioning is enough. What’s left is a much higher ratio of self-generated conditioning frames, or more adapter capacity, or both: a narrower problem than the one I started with.</p><p><strong>What to expect if you try this</strong></p><ul><li>If your renderer isn’t photoreal, expect no transfer from a pretrained action-conditioned world model out of the box. A larger checkpoint won’t change that.</li><li>Budget for fine-tuning, and target it. The action-to-vision pathway here was 0.095% of the parameters, enough to go from hallucination to 84% direction-correctness.</li><li>Short, re-grounded prediction from a real frame works now. Long autoregressive self-simulation, the thing people mean by “virtual environment”, isn’t there yet for this model on this data.</li></ul><p>The code is on <a href="https://github.com/cydal/cosmos-policy/tree/main">GitHub</a>.</p><p>NVIDIA, “Cosmos 3: Omnimodal World Models for Physical AI”, arXiv:2606.02800, 2026. Checkpoints: <a href="https://huggingface.co/nvidia/Cosmos3-Edge">https://huggingface.co/nvidia/Cosmos3-Edge</a></p><p>The simulator and dataset: <a href="https://github.com/cydal/mujoco-env-dataset">https://github.com/cydal/mujoco-env-dataset</a></p><p>Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models”, arXiv:2106.09685, 2021.</p><p>Bengio et al., “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks”, NeurIPS 2015.</p><p>The Cosmos adapter, validation scripts, fine-tuning and closed-loop code: <a href="https://github.com/cydal/cosmos-policy">https://github.com/cydal/cosmos-policy</a></p><p>Elliot Sage., “AI Build &amp; Learn” <a href="https://github.com/sagecodes/ai-build-and-learn">https://github.com/sagecodes/ai-build-and-learn</a>, 2026</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=255fa9b4820d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/my-cosmos-world-model-didnt-know-what-to-do-with-a-physics-engine-255fa9b4820d">My Cosmos World Model Didn’t Know What to Do With a Physics Engine</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Weakly Supervised Segmentation: How Far Can a Few Labelled Pixels Take You?]]></title>
            <link>https://levelup.gitconnected.com/weakly-supervised-segmentation-how-far-can-a-few-labelled-pixels-take-you-10e8a736cead?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/10e8a736cead</guid>
            <category><![CDATA[image-annotation]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[semantic-segmentation]]></category>
            <category><![CDATA[weak-supervision]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 15:48:44 GMT</pubDate>
            <atom:updated>2026-10-05T15:48:44.192Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How spreading sparse point labels yields 98% of full-mask performance on aerial images</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*q8yKo6yCML60l0Mn" /></figure><p>Semantic segmentation assigns a class to every pixel in an image. The standard way to train it is with a full mask for every training image, i.e. a ground truth where every pixel already has its class. Producing those masks is slow, because someone has to trace the outline of every building, tree and car.</p><p>A much cheaper form of annotation is to click a few points on each thing in the image. This is point supervision, one of several forms of weakly supervised semantic segmentation, and Bearman et al. found it took about a tenth of the annotation time of full masks on PASCAL VOC. I wanted to know how far a model gets when the loss only sees those few labelled pixels, and what makes the most of them.</p><p>I ran this on the <a href="https://www.kaggle.com/datasets/deasadiqbal/private-data-1">ISPRS Potsdam aerial dataset</a>, which has six classes: impervious surfaces, building, low vegetation, tree, car and clutter. I varied four things: the number of points, the loss, spreading labels with superpixels, and where the points are placed. One of them clearly helped and one clearly hurt. Most of the rest were hard to separate from the variation between identical runs. To see how much the missing pixels cost, I also trained the same model on full masks.</p><p>The code and results are on <a href="https://github.com/cydal/partial-segmentation">GitHub</a>.</p><h3>The setup</h3><p>The model is DeepLabV3+, a standard encoder-decoder segmentation network, with an ImageNet-pretrained ResNet-50 backbone from segmentation-models-pytorch. It trains on 2,400 patches of 300×300 pixels, split 70/15/15 into train, validation and test, for 50 epochs with Adam at a learning rate of 1e-4. All scores are mean intersection over union (mIoU) on the test split, measured against the full ground-truth masks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*ZmkOYWG-hrI4a5VL.gif" /><figcaption><a href="https://arxiv.org/abs/1802.02611">https://arxiv.org/abs/1802.02611</a></figcaption></figure><p>The model still predicts every pixel. The trick is in the loss, which is only computed at the labelled pixels:</p><pre>p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1)<br>focal_weight = (1.0 - p_t) ** self.gamma<br>pixel_loss = -focal_weight * torch.log(p_t + 1e-8)<br>loss = (pixel_loss * point_mask).sum() / point_mask.sum().clamp(min=1)</pre><p>p_t is the probability the model gives the correct class. The focal weight reduces the loss on pixels the model is already confident about, and γ controls how strongly. At γ = 0 this is plain cross-entropy. point_mask is 1 at labelled pixels and 0 everywhere else, so unlabelled pixels contribute no gradient.</p><p>Points are simulated by sampling from each class in the ground-truth mask. Each training image gets its points once, before augmentation, and keeps them for the whole run, as it would with a human annotator. The densities are nested, i.e. the 10-point labels include the 5-point labels, which include the single point, so density comparisons differ only in the number of points. Not every class appears in every patch, so at 10 points per class an image has about 27 labelled pixels out of 90,000, roughly 0.03%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hD8Teq8P8HQeSw7XxiaJA.png" /><figcaption><em>Simulated point labels at 1, 10 and 50 points per class. The smaller sets are subsets of the larger ones.</em></figcaption></figure><p>To tell real differences from noise, I trained the reference setup (10 points per class, γ = 2, uniform sampling) three times with the same points and different training seeds. The runs scored 0.560, 0.559 and 0.544. So a gap of under about 1.5 points between two single runs tells us nothing, and I treat it that way below. Per-class IoU varies by up to 3 points between those runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lq_o2Etjm9yNq8zQ6u9zDg.png" /><figcaption><em>Test mIoU for every condition. The shaded band is the range of the three reference runs.</em></figcaption></figure><h3>More points help, with diminishing returns</h3><p>With 1, 5, 20 and 50 points per class, the model scored 0.491, 0.551, 0.569 and 0.572. The reference, at 10 points, sat between 0.544 and 0.560.</p><p>A single point per class is clearly not enough. By 5 points the model is already inside the reference band. Going to 20 or 50 adds only about a point above the top of the band, and 50 is no better than 20, so most of the gain comes from the first few clicks.</p><h3>Focal weighting: no measurable effect</h3><p>With 10 points per class, γ values of 0, 0.5 and 1 gave 0.561, 0.564 and 0.556, against 0.544–0.560 at γ = 2.</p><p>All four settings are within about half a point of the reference band, so I can’t separate any of them from noise. With so few labelled pixels per image, down-weighting the easy ones doesn’t seem to change much. Plain cross-entropy is a fine default here.</p><h3>Superpixel label propagation: the one clear win</h3><p>Each labelled point supervises one pixel. But the pixels around it usually belong to the same class, since a point on a roof is mostly surrounded by roof. So the next experiment spreads each point’s label to its neighbourhood.</p><p>It uses SLIC, which splits an image into a few hundred small regions of similar colour, called superpixels. Each point’s label is copied to every pixel in the superpixel it lands in. At 200 superpixels per image, each point then labels about 560 pixels instead of one. The expanded labels act as pseudo-labels for the unlabelled pixels around each point, without a separate stage to generate them.</p><p>This only helps if most superpixels contain a single class. Across 50 images, the median superpixel was essentially pure, i.e. all its pixels shared the point’s class, and 74% were at least 80% pure. Mean purity was lower, at 0.84, because a minority of superpixels straddle a class boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5zZG2XyqN-3mDj4Mq9F7Ig.png" /><figcaption><em>SLIC superpixels on a Potsdam patch. Most superpixels containing a point are pure; the impure ones sit on class boundaries.</em></figcaption></figure><p>With the same 10 points per class, SLIC expansion scored 0.603. That’s 4.3 points above the best reference run, and 3 points above the 50-point model (0.572). The only extra cost is running SLIC over each image.</p><p>Every class improved beyond the per-class spread of the reference runs. The largest gain was on cars, from 0.62–0.65 IoU to 0.71. A car is a compact object of fairly uniform colour, so it fits neatly into one or two superpixels.</p><h3>Cleaner labels were not better labels</h3><p>The last experiment asked where points should go: deep inside objects, or near their edges.</p><p>A point inside a region has clean surroundings, e.g. a point in the middle of a road is surrounded by road. A point near an edge has mixed surroundings, but it sits where the model has to make its hardest decisions. I tried three sampling strategies: uniform, biased towards boundaries, and biased towards interiors. To check they did what they claimed, I measured local purity, the share of an 11×11 patch around each point that has the point’s class:</p><ul><li><strong>Interior:</strong> 0.974, with 11% of patches containing a second class</li><li><strong>Uniform:</strong> 0.918, with 31%</li><li><strong>Boundary:</strong> 0.848, with 51%</li></ul><p>If cleaner labels made better models, interior sampling would win. It came last, at 0.532, about a point below the lowest reference run. Boundary sampling scored 0.553, inside the reference band, so it didn’t beat uniform either.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u2ya6B0vp5YlJPkdilUb1g.png" /><figcaption><em>Label purity against test mIoU. Uniform is shown as the range of the three reference runs.</em></figcaption></figure><p>My reading is that points deep inside large regions are mostly redundant. The model would get the middle of a large roof right anyway, so a label there adds little. Purity measures how trustworthy a label is, which isn’t the same as how much the model learns from it. Uniform sampling is a sensible default, and interior-biased sampling is worth avoiding.</p><h3>How close this gets to full masks</h3><p>To put these scores in context, I trained the same model with the same settings on the full masks, so every pixel contributes to the loss. It scored 0.617.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TSQs-kzgirI5OSkH_XA29A.png" /><figcaption><em>Test mIoU for point supervision as a share of full supervision (dashed line).</em></figcaption></figure><p>Ten points per class, about 0.03% of the pixels, reach 90% of that score. Fifty points reach 93%. With SLIC, the same 10 points reach 98%, and every class ends within about 3 points of its full-mask IoU. The remaining gap, 1.4 points, is smaller than the spread between the reference runs, so with one run each I can’t tell the SLIC model and the full-mask model apart.</p><h3>What I’d take away</h3><p>The results that hold up are:</p><ul><li><strong>Superpixel expansion</strong> was the biggest improvement, larger than using five times as many points, and brought 10 points per class to within noise of full masks.</li><li><strong>Interior points</strong>, the cleanest labels, gave the worst model at 10 points per class.</li><li><strong>More points help</strong> up to about 5 per class. Beyond that, the gains are around a point.</li><li><strong>Focal weighting</strong> made no difference I could measure.</li></ul><p>Clutter was the weakest class everywhere, never above 0.16 IoU even with full masks. It’s a catch-all for everything that isn’t one of the other five, so there’s no consistent appearance to learn.</p><h3>Limitations</h3><ul><li><strong>One set of points.</strong> Every run uses the same sampled points. A different draw would move the scores, and I haven’t measured by how much.</li><li><strong>One run per condition</strong>, except the reference setup. This includes the full-mask model, and gaps under about 1.5 points shouldn’t be trusted.</li><li><strong>Simulated annotators.</strong> Points come from ground-truth masks. Real annotators click differently, so the placement experiment is a proxy for their behaviour rather than a study of it.</li><li><strong>Fixed learning rate.</strong> Validation mIoU was still oscillating in the last epochs, and a scheduler might raise every score.</li></ul><h3>Notes</h3><ol><li>Bearman et al., “What’s the Point: Semantic Segmentation with Point Supervision”, ECCV 2016.</li><li>Code, configs and per-run results: <a href="https://github.com/cydal/partial-segmentation">https://github.com/cydal/partial-segmentation</a></li><li>Lin et al., “Focal Loss for Dense Object Detection”, ICCV 2017.</li><li>Achanta et al., “SLIC Superpixels Compared to State-of-the-Art Superpixel Methods”, IEEE TPAMI, 2012.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10e8a736cead" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/weakly-supervised-segmentation-how-far-can-a-few-labelled-pixels-take-you-10e8a736cead">Weakly Supervised Segmentation: How Far Can a Few Labelled Pixels Take You?</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Best Driving World Model Agent Was the Worst Driver]]></title>
            <link>https://briefs.aiadvances.org/my-best-driving-world-model-agent-was-the-worst-driver-58bc50adee3f?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/58bc50adee3f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[world-models]]></category>
            <category><![CDATA[latent-space]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 23:00:13 GMT</pubDate>
            <atom:updated>2026-10-01T23:00:13.527Z</atom:updated>
            <content:encoded><![CDATA[<h4>Reward shaping lessons from DreamerV3 in a simulated city, and what I only found by watching it</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4tD6NmOTdFU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4tD6NmOTdFU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4tD6NmOTdFU%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/770892f0c34c36cd55d34b7e80b4bbae/href">https://medium.com/media/770892f0c34c36cd55d34b7e80b4bbae/href</a></iframe><p>Going by the numbers, this car is the best model I trained. Its score was still rising when I stopped it at about 2 million steps, and its success rate had climbed for the whole run.</p><p>It also takes corners too fast, clips them, makes sudden U-turns and runs red lights. None of that showed up in the crash rate. I only found it because I watched it drive.</p><p>The agent is DreamerV3, and it learned every action inside its own model of the city. Nearly all the work went into reward shaping, i.e. adding penalties to steer how the car drives, and the fix that worked changed how often a penalty applies, not how large it is.</p><h4><strong>Learning to drive in a dream</strong></h4><p>Most reinforcement learning agents learn by trial and error: act, see the reward, adjust. DreamerV3 first learns a world model, i.e. a network that predicts what the car will sense next, what reward it will get and whether the episode will end, given what it senses now and what it does. The policy then practises inside that model. It imagines about 15 steps ahead, a critic scores each imagined future, and the policy learns from those futures without touching the real simulator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p5VF5OoN14MSaAJe.jpg" /><figcaption><a href="https://arxiv.org/abs/2301.04104">https://arxiv.org/abs/2301.04104</a> — Deepmind</figcaption></figure><p>Two things follow from that. The policy can only be as good as the model’s predictions. And we can look at those predictions directly: the viewer I built draws 3 imagined paths, each 40 steps ahead, on top of the car.</p><p>DreamerV3 uses one set of hyperparameters across very different tasks. It predicts rewards and values on a squashed (symlog) scale and normalises returns, so I never had to rescale the reward, even though it ranges from −0.1 per step to spikes of ±100. I used the <a href="https://github.com/danijar/dreamerv3">original code</a> mostly unmodified, with a thin adapter, on one NVIDIA T4 and DreamerV3’s smallest model (about 1M parameters).</p><h4><strong>The city</strong></h4><p><a href="https://github.com/cydal/Car-Navigation-Env">CarNav</a> builds a new city every episode, with buildings, parked cars, moving traffic, pedestrians and traffic lights. The car has to reach 3 waypoints in order within 1,000 steps. It senses 32 lidar beams, its own speed, yaw rate and slip, the direction to the next waypoint, the nearest traffic light and nearby objects. It controls throttle, steering and brake.</p><p>The reward adds up progress towards the waypoint, a bonus for each waypoint, a crash penalty, a one-time penalty of 50 for running a red light, and a cost of 0.1 every step. <strong>There’s no term for speed, comfort or lane-keeping.</strong></p><p>A scripted driver ships with the environment. On the simplest version of the task, with no traffic or lights, it scores +222 and completes the route 44% of the time. My first DreamerV3 run on that task was scoring in the 400s after 4 hours.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PSZ7PIlLOTJ3p5AN6m_Tog.jpeg" /><figcaption>Simulated City for Car Navigation</figcaption></figure><h4><strong>Nine hours, no waypoints</strong></h4><p>So I went straight to the full city, with traffic, pedestrians and lights, and a larger model. After 9 hours and 613 episodes, it hadn’t reached a single waypoint. The scripted driver completed routes on the same setup, so the task was solvable. The agent had learned to survive instead: most episodes ended at the time limit, with the car going nowhere in particular.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fpm82BF0vfHg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dpm82BF0vfHg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fpm82BF0vfHg%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a22a2ea36238f70eac3bfacaa3d56c60/href">https://medium.com/media/a22a2ea36238f70eac3bfacaa3d56c60/href</a></iframe><p>To find the hard part, I added one feature at a time, 4 hours each. Lights were fine. Pedestrians were harder but learnable. Traffic broke it: with only other cars added, the mean score over the last 40 episodes was +304 with 2 cars, +113 with 4 and −22 with 8.</p><p>How it failed was the interesting part. Nearly all its crashes were into buildings, which it had already learned to avoid in the empty city. The lidar reconstruction loss was the same with or without traffic, but the loss for predicting the next state was 25–35% higher. The policy learns only from imagined futures, so worse predictions make everything it learns worse, including skills that have nothing to do with other cars.</p><h4><strong>Best score, worst driver</strong></h4><p>Checkpoint-2’s building-crash rate was about a third of earlier runs’ (0.0009 against 0.0026–0.0031), because a clipped corner or a near miss doesn’t count as a crash. Average speed was where it showed: 14.8 m/s, against 9.7–13.7 for every earlier run, with a top speed of 22.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*9jWJFJK9NmDP_gJsWbxcDQ.png" /><figcaption><em>Three imagined futures (purple dotted lines), each 40 steps ahead, drawn from the world model’s own predictions.</em></figcaption></figure><h4><strong>Punishing it harder made it stop</strong></h4><p>My first fix changed three things at once: I tripled the red-light penalty, halved the per-step cost, and added a penalty on turning rate.</p><p>The car almost stopped driving. Its average speed fell to about 0.5 m/s, a quarter of episodes hit the time limit, and it reached no waypoints. The world-model losses and the policy’s entropy looked normal. The car had simply found that barely moving was the cheapest option.</p><p>Two things went wrong. The turn penalty taxed all turning, and every waypoint sits at an intersection. I added a threshold set from the scripted driver’s own turning: 90% of its yaw rates were under 0.91 rad/s, so I put the threshold at 1.0. Normal cornering is now free.</p><p>And I’d changed three things at once. Short 2-hour tests couldn’t separate them, because runs only diverged after 300–500k steps. So I ran one change at a time for about 950k steps each. The turn penalty alone tracked ahead of the baseline. The tripled red-light penalty alone was behind it at every matched step.</p><h4><strong>A penalty felt on every step</strong></h4><p>Checkpoint-3 kept only the thresholded turn penalty. It matched checkpoint-2s score and took corners more carefully.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-XMSVkDlSEI%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-XMSVkDlSEI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-XMSVkDlSEI%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/85dd4e65d22afe0e685a98285bb7ea9e/href">https://medium.com/media/85dd4e65d22afe0e685a98285bb7ea9e/href</a></iframe><p>The two experiments pointed to the fix. The turn penalty works because the car feels it on every step of a fast turn. A one-time red-light penalty is small next to a cost paid on every step, and tripling it held training back instead. In RL terms, the red-light penalty was sparse and the turn penalty dense. So checkpoint-4 adds a red-light penalty the car feels throughout its approach. While a light is red or yellow, any speed above is penalised.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyKhLCVYp-7Kgs4VwShHkA.png" /><figcaption>checkpoint 3 — score</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a7eNvVVfI0yPVuqdAB3h3g.png" /><figcaption>checkpoint 3 — avg speed</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FoWOfKu5FHv8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DoWOfKu5FHv8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FoWOfKu5FHv8%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d937f9251fd253099ed2ae5daddbe6f/href">https://medium.com/media/6d937f9251fd253099ed2ae5daddbe6f/href</a></iframe><p>To compare the last three checkpoints properly, I ran each on the same 300 test episodes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AU89QndeJzbvK-0edhzAXg.jpeg" /></figure><p>checkpoint-3 hit buildings less than checkpoint-2, but it drove just as fast and ran as many red lights. checkpoint-4 ran about a sixth as many red lights as checkpoint-3, hit buildings about half as often, and completed slightly more routes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucCPh40Dam21myockBLoUg.png" /><figcaption>checkpoint 4 — average speed</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BaOvS-t5-d2LwYABn1kwzA.png" /><figcaption>checkpoint 4 — score</figcaption></figure><p>It also slowed down everywhere, from 14.9 to 9.8 m/s, even though the penalty only applies near red lights. My guess is that lights are common enough in this city that approaching every intersection at a stoppable speed became the cheapest habit. The cost is time: episodes take twice as long, and 9% run out.</p><h4><strong>What’s still wrong</strong></h4><p>Pedestrians. A pedestrian crash ends 81 of checkpoint-4’s 300 test episodes, against 7 for the scripted driver, and none of the reward changes helped. Nothing makes the car yield at a crossing the way the braking penalty makes it stop at red lights, so that’s the next fix. The city also has no lanes, so lane discipline is out of scope.</p><p>What I’d carry into the next project:</p><ul><li><strong>With a world model, what the model can’t predict, the policy can’t learn. </strong>Traffic hurt obstacle avoidance because it hurt the model’s predictions. Moving beyond the 1M parameter count might help the model handle traffic better.</li><li><strong>How often a penalty applies matters more than how large it is. </strong>A dense penalty worked where a sparse one three times the size didn’t.</li><li><strong>Set thresholds from physics or a reference driver</strong>, not by guessing.</li></ul><p>The code is on <a href="https://github.com/cydal/Car-Navigation-Env">GitHub</a>.</p><p><strong>— -</strong></p><p><strong>## Notes</strong></p><p>1. Hafner et al., “Mastering Diverse Domains through World Models”, arXiv:2301.04104, 2023.</p><p>2. DreamerV3’s official code: <a href="https://github.com/danijar/dreamerv3">https://github.com/danijar/dreamerv3</a></p><p>3. The adapter, environment configs and checkpoints: <a href="https://github.com/cydal/dreamer-car-nav">https://github.com/cydal/dreamer-car-nav</a></p><p>4. Elliot Sage., “AI Build &amp; Learn” <a href="https://github.com/sagecodes/ai-build-and-learn">https://github.com/sagecodes/ai-build-and-learn</a>, 2026</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58bc50adee3f" width="1" height="1" alt=""><hr><p><a href="https://briefs.aiadvances.org/my-best-driving-world-model-agent-was-the-worst-driver-58bc50adee3f">My Best Driving World Model Agent Was the Worst Driver</a> was originally published in <a href="https://briefs.aiadvances.org/">AI Advances Briefs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Does It Mean for an AI to Understand the World?]]></title>
            <link>https://generativeai.pub/what-does-it-mean-for-an-ai-to-understand-the-world-593692930651?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/593692930651</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[world-models]]></category>
            <category><![CDATA[machine-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 00:58:26 GMT</pubDate>
            <atom:updated>2026-10-01T00:58:26.854Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Why knowing about the world and being able to predict it may be two very different things, and what that gap means for large language models, robotics, and the next generation of AI</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7YhOnSqALm2IIX_f" /></figure><p>Picture someone waiting to cross a side street. A car is approaching, and it isn’t clear whether it’s slowing down. They watch it for a moment, notice the front of the car dip as it brakes, and step off the kerb.</p><p>That decision involved more than seeing a car. The person had to judge how the car was moving, predict where it would be in a couple of seconds, and consider what would happen if they stepped out and it didn’t stop. In effect, they ran a short simulation of the future and acted on the result.</p><p>Now give the same kerb to a robot or a self-driving car. Detecting the car is the easy part. The system also needs to know where the car is going, how its speed is changing, and how the driver might react to the system’s own movement.</p><p>Yann LeCun often points out that a teenager can learn to drive in about 20 hours of practice, while autonomous driving systems train on far more data and still struggle with situations a learner driver handles without much thought. Whatever the teenager is doing, more data doesn’t seem to be the explanation.</p><p>I came across the passage below while reading James S. A. Corey’s <em>Babylon’s Ashes</em>:</p><blockquote>“What does resistance mean?” Natalia asked. Her face was serious and sober, and on so small a person, vaguely comic.</blockquote><blockquote>“It’s a measure of how electrons flow through something,” Prax said. “You see, we think about current flowing through wires kind of the same way we think of water flowing through pipes. It’s actually more complicated than that when you get down to quantum levels, but it’s a very, very good model.”</blockquote><blockquote>“And models are how we make sense of things,” Natalia said.</blockquote><p>Natalia’s last line stayed with me.</p><p>If models are how we make sense of things, what kind of model does a large language model have? Can it reason, and does it really understand the world it describes so fluently? The more I thought about this, the more it seemed to depend on another question: <strong>what would it mean for a machine to have a model of the world at all</strong>?</p><p>The term used for this in AI is <strong><em>world model</em></strong>. Loosely, a world model is an internal representation that lets a system predict how its environment will change, including in response to its own actions. Beyond that loose description, there’s surprisingly little agreement on what one needs to include.</p><p>In this article, we’ll look at what people mean by the term and why some researchers think it matters so much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FgMCcuCyamy0uvD3" /></figure><h4><strong>An old idea</strong></h4><p>The idea that minds build internal models of the world is older than computing. In 1943, the psychologist Kenneth Craik suggested that if an organism “carries a ‘small-scale model’ of external reality and of its own possible actions within its head, it is able to try out various alternatives, conclude which is the best of them, and react to future situations before they arise.” This is more or less what happened at the kerb.</p><p>AI picked up the idea in the early 90s; Richard Sutton’s Dyna architecture had an agent learn a model of its environment and use it to practise. In 2018, David Ha and Jürgen Schmidhuber published a paper titled “World Models”, in which agents learned compressed models of simple video games and, in one case, trained entirely inside their own imagined version of the game.</p><p>So the concept itself isn’t new; what has changed is that it’s now central to the debate about where AI goes next.</p><h4><strong>Four questions</strong></h4><p>One way to think about world models is as a series of questions an agent may need to answer:</p><ul><li>What is happening?</li><li>What will happen?</li><li>What would happen if I did something different?</li><li>What should I do?</li></ul><p>The first is perception, the second is prediction, the third is counterfactual reasoning, and the fourth is planning. Each builds on the one before it. Judea Pearl describes a similar ladder for causal reasoning (seeing, doing, and imagining) and argues that most of machine learning operates on the first rung.</p><p>Prediction and planning are different problems. A weather model may predict tomorrow’s rain accurately, but it can’t tell us what happens if we act, since nothing we do affects the weather. This is passive prediction: <em>what happens next</em>?</p><p>The car at the kerb is different because the driver reacts to the pedestrian. The useful question becomes: <em>what happens next if I step out</em>? This is action-conditioned prediction, and it’s much closer to the problem an intelligent agent faces.</p><p>The same distinction shows up in reinforcement learning. A model-free agent learns which actions pay off through trial and error, without building an explicit picture of how its environment works. A model-based agent learns the dynamics of the environment, i.e. how it changes in response to actions, and can plan by trying out options internally before acting.</p><p><strong>AlphaZero</strong> was given the rules of Go and chess and searched through them. Its successor, MuZero, was never given the rules. It learned its own model and planned with that.</p><p>Put together, this gives a rough sequence a world model might support: represent the environment, predict how it evolves, imagine alternatives, intervene, plan and act. This is a sketch rather than a definition. Some researchers would stop at prediction. Others would argue that without action and planning, it isn’t a world model at all.</p><h4><strong>What counts as a world?</strong></h4><p>Most coverage of world models is about robots and self-driving cars, so it’s easy to assume the “world” means physical reality. It doesn’t have to.</p><p>A world here is any environment in which an agent observes something, takes actions, and sees the consequences. A Go board is a world. So is a video game, a warehouse, a simulated factory floor or a financial market.</p><p>Chess has fixed rules and complete information. A market has neither, and it’s full of other agents who are modelling you back. Both are worlds in this sense, but they need very different models.</p><p>Physical AI gets most of the attention because it’s one of the hardest places to show that a world model is useful. A robot in a kitchen only sees part of the scene, a problem known as <em>partial observability</em>. Objects hide behind each other, physics doesn’t forgive errors, consequences can be delayed, and some mistakes can’t be undone.</p><p>A robot that drops a glass can’t retry with the same glass. Describing what’s in front of it isn’t enough; it needs to anticipate what happens when it reaches for the handle. Robotics is a demanding test for world models, but it isn’t the definition of one.</p><h4><strong>What should the model represent?</strong></h4><p>The options vary widely. A model could predict raw pixels, i.e. every detail of the next frame of video. It could predict a latent representation, i.e. a compressed set of numbers that describes the state of the scene without being an image of it.</p><p>It could track objects and their relationships, or the geometry of the space around it. OpenDriveLab’s ViDAR, which we’ll look at later in the series, takes past camera images and predicts future 3D point clouds. It predicts the shape of the road scene, not its colours or textures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l6BcqXyUIe4w2TKt" /><figcaption>Latent representation of input data and reconstructed</figcaption></figure><p>Behind these choices is a real disagreement. One view is that to understand the world, a model should be able to reconstruct it and generate what the future will look like. The other is that reconstruction is wasteful, and a model only needs to predict the aspects of the future that matter for what it’s trying to do.</p><blockquote><strong>To know we’ll get wet, we don’t need to predict where each raindrop lands.</strong></blockquote><p>Generative video has made this debate more concrete. Google DeepMind’s Genie 3, announced in August 2025, generates worlds a user can move around in real time at 720p and 24 frames per second, and they stay consistent for a few minutes.</p><p>NVIDIA markets its Cosmos models as “world foundation models”, large pretrained models intended to be fine-tuned for specific robotics and driving tasks. It’s tempting to look at a video model that produces a plausible next five seconds and conclude that it has a model of the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bpE_e5b5sW4M9NIa" /></figure><p>It might. But a video can look right while getting the physics wrong, and a model can capture what matters about the future without drawing any of it. MuZero never produces an image of the board. It predicts only what it needs to choose a move: <em>how good the position is and what to play next</em>.</p><p>Is a video prediction model automatically a world model? Does it become one once it can be steered with actions, as Genie can? I don’t think the field has settled this. The term currently covers systems that do quite different things.</p><h4><strong>Do LLMs already have one?</strong></h4><p>This is where I started, so it’s where I’m least willing to accept an easy answer.</p><p>The easy answer is that language models don’t understand the world and only predict text. That doesn’t hold up well against how they behave. LLMs encode a lot of structure about objects, people, events, physical concepts, social relationships, cause and effect, and the order in which things happen. Ask one what happens if you put a metal fork in a microwave, and it will tell you, correctly, and explain why.</p><p>There’s also evidence that something like a world model can emerge from sequence prediction alone. In a 2023 study often referred to as Othello-GPT, researchers trained a transformer only on sequences of legal moves in the board game Othello.</p><p>It was never shown a board. Yet they could read the state of the board from the model’s internal activations, and when they edited that internal state, the model’s predicted moves changed to match the edited board. The model appears to have built a representation of the board because that was the most useful way to predict the next move.</p><p>A 2024 study by Keyon Vafa and colleagues points the other way. They trained transformers on millions of turn-by-turn taxi routes through Manhattan. The models were very good at the task, producing valid routes between new pairs of locations 96–99% of the time. But when the researchers reconstructed the street map implied by the model, it bore little resemblance to Manhattan.</p><p>It had streets pointing in physically impossible directions and roads passing over other roads. The weakness showed as soon as the task changed slightly. When 1% of turns were replaced with forced detours, the share of valid routes from the best-performing model fell from 99% to 69%. At 10%, it fell to 8%.</p><p>Both studies used small models trained from scratch on narrow tasks, not frontier LLMs, so neither settles the question for systems like GPT or Claude. They do show why the question is hard. Doing well on a task isn’t strong evidence of a coherent model of the world behind it. A system can perform well by learning a set of shortcuts that work until something unusual happens.</p><p>The question I’m interested in is whether the knowledge LLMs have is the same kind of thing as a model that can simulate the consequences of actions. Knowing that forks spark in microwaves is knowledge about the world. Working out what happens in a situation nobody has written about needs something closer to what the pedestrian did at the kerb. I don’t know how much of the second LLMs have, and I’m wary of anyone who claims to be certain either way.</p><h4><strong>LeCun’s bet</strong></h4><p>The best-known argument that current approaches fall short comes from Yann LeCun. His position, set out most fully in a 2022 paper, is that text alone is too limited a source for learning how the world works. A four-year-old has taken in far more information through their eyes than any LLM has read, and most of what they know about gravity, objects and other people didn’t come from language.</p><p>On this view, a system that learns mainly by predicting the next token will hit a ceiling when asked to plan and act in the physical world, however large it gets.</p><p>His proposed alternative is a system that learns representations of the world from observation, predicts how those representations change (particularly in response to actions), and uses those predictions to plan towards goals.</p><p>His concrete proposal is the Joint Embedding Predictive Architecture (JEPA). Instead of predicting the next token or the pixels of the next frame, a JEPA model predicts the latent representation of a missing or future part of its input.</p><p>It’s trained with self-supervised learning, meaning there are no human labels and the data provides its own targets. The argument is that predicting in latent space lets the model ignore detail that can’t be predicted, like the exact movement of leaves in the wind, and focus on the structure of the scene. This puts LeCun firmly on the “predict what matters” side of the earlier debate.</p><p>He has also backed this outside of research papers. In November 2025, LeCun announced he was leaving Meta after a decade as its chief AI scientist, and co-founded Advanced Machine Intelligence Labs to build world models. In March 2026, the company announced it had raised $1.03 billion.</p><p>This is where the phrase “next paradigm” tends to come up: the idea that the next big step in AI may come less from better models of language and more from models that learn to predict and reason about environments. It’s an influential position, but there’s a counterargument. Frontier models are already trained on images, video, and audio as well as text, and increasingly on the results of their own actions in software environments.</p><p>It’s possible that a large enough multimodal model trained this way learns world dynamics as a side effect, the same way Othello-GPT learned the board. If that’s the case, world models may not be a separate paradigm at all but something the current one grows into.</p><p>This matters for the AGI debate. If general intelligence means an agent that can operate flexibly in environments it hasn’t seen before, it’s hard to picture that agent without some internal model of how those environments behave.</p><p>It would need to perceive its situation, keep track of state, anticipate possible futures, compare alternative actions, draw on past experience, act, and update its model when the results surprise it. This is one hypothesis about what intelligence requires. I find it plausible, but it isn’t a blueprint, and no architecture has yet been shown to deliver it.</p><p>It also raises a practical question: how would we know? If strong performance on a task can hide an incoherent map, as it did in Manhattan, we need better ways to test whether a system has a model of its world or just a very good set of habits.</p><h4><strong>The models we’ll explore</strong></h4><p>World models aren’t coming from one lab or one architecture. The idea shows up in model-based reinforcement learning, generative video, robotics, self-supervised representation learning, multimodal models, and simulation. Each answers a different version of the same question: <strong>what does an AI system need to predict about its environment</strong>?</p><p>In this series, we’ll work through several of these approaches. The list below is a starting point and will likely change as we go.</p><p><strong>Dreamer</strong> learns a compact model of its environment and trains its behaviour almost entirely inside it. The latest version learned to collect diamonds in Minecraft from scratch. Can an agent learn to act by imagining futures?</p><p><strong>Cosmos</strong> takes the generative route, producing video of possible physical futures. One practical use is generating synthetic training data for robots and vehicles, which raises the question of whether what’s learned from generated video transfers to the real world (the sim-to-real gap). Can a generative model become a useful simulator of the physical world?</p><p><strong>V-JEPA</strong> applies JEPA to video, learning by predicting the latent representations of masked parts of a clip. Can a model learn the structure of the world without reconstructing it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Uxkt0lPkJxld0FLI.png" /></figure><p><strong>MuZero</strong> models only what it needs to make decisions. How much of the world does an agent need to model to plan well?</p><p><strong>ViDAR</strong> forecasts future 3D geometry from camera video as a way to pre-train driving systems. Is predicting the shape of the future enough to teach a car what matters?</p><h4><strong>Back to the kerb</strong></h4><p>When I started looking into this, I thought about intelligence mainly in terms of what a system knows. The pedestrian at the kerb suggests that what a system expects to happen next may matter just as much.</p><p>There’s a lot here I don’t yet know. Should a world model necessarily represent pixels, objects or something more abstract? Does it need to generate the future, or only predict the parts that matter? Can it learn cause and effect by watching, or does it need to act and see what happens? Do LLMs already contain basic world models, or very good imitations of them? And how would we test, in a way that holds up, whether an AI has a model of the world at all?</p><p>That’s what I want to find out. The next article starts with Dreamer.</p><h4><strong>Notes</strong></h4><ul><li>Yann LeCun, “A Path Towards Autonomous Machine Intelligence”, OpenReview, 2022.</li><li>James S. A. Corey, <em>Babylon’s Ashes</em>(Orbit, 2016).</li><li>Kenneth Craik, <em>*The Nature of Explanation*</em> (Cambridge University Press, 1943), p. 61.</li><li>Richard S. Sutton, “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming”, <em>*Proceedings of the Seventh International Conference on Machine Learning*</em>, 1990.</li><li>David Ha and Jürgen Schmidhuber, “World Models”, arXiv:1803.10122, 2018.</li><li>Judea Pearl and Dana Mackenzie, <em>The Book of Why </em>(Basic Books, 2018).</li><li>Julian Schrittwieser et al., “Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model”, <em>Nature</em> 588, 2020.</li><li>Zetong Yang, Li Chen, Yanan Sun and Hongyang Li, “Visual Point Cloud Forecasting enables Scalable Autonomous Driving”, CVPR 2024.</li><li>Google DeepMind, “Genie 3: A new frontier for world models”, 5 August 2025.</li><li>NVIDIA, “Cosmos World Foundation Model Platform for Physical AI”, arXiv:2501.03575, 2025.</li><li>Kenneth Li et al., “Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task”, ICLR 2023.</li><li>Keyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon Kleinberg and Sendhil Mullainathan, “Evaluating the World Model Implicit in a Generative Model”, NeurIPS 2024.</li><li>LeCun confirmed his departure from Meta on 19 November 2025. AMI Labs announced its $1.03 billion raise in March 2026.</li><li>Danijar Hafner et al., “Mastering Diverse Control Tasks through World Models”, <em>Nature</em>, 2025.</li><li>Adrien Bardes et al., “Revisiting Feature Prediction for Learning Visual Representations from Video”, 2024; Mido Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning”, 2025.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*y0L2TXmV2FAUs1im.png" /></figure><p>This story is published under the <a href="https://generativeai.pub/">Generative AI</a> publication. Connect with us on <a href="https://www.linkedin.com/company/generative-ai-publication">LinkedIn</a> and follow <a href="https://www.zeniteq.com/">Zeniteq</a> to stay in the loop with the latest AI stories. Let’s shape the future of AI together!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*7Llu3ivIUhSi8jjy.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=593692930651" width="1" height="1" alt=""><hr><p><a href="https://generativeai.pub/what-does-it-mean-for-an-ai-to-understand-the-world-593692930651">What Does It Mean for an AI to Understand the World?</a> was originally published in <a href="https://generativeai.pub/">Generative AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LlamaParse vs PixelRAG: They Tie, Until the Question Mentions a Colour]]></title>
            <link>https://aiadvances.org/llamaparse-vs-pixelrag-they-tie-until-the-question-mentions-a-colour-6df0ea88747d?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/6df0ea88747d</guid>
            <category><![CDATA[vision-language-model]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llamaindex]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Mon, 28 Sep 2026 22:25:03 GMT</pubDate>
            <atom:updated>2026-09-28T22:25:03.896Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Comparing text extraction and vision models on 70 hard questions from FinanceBench, DocVQA and ChartQA</em></h4><p>There are two quite different ways to parse a complicated PDF for RAG, i.e. to turn it into something a language model can retrieve from and answer questions about.</p><p>The first is to convert the document into text that keeps its structure. LlamaParse is the best-known tool for this. It turns a PDF into Markdown, running OCR on scanned pages, so tables become rows of |-separated cells and headings become ## headers.</p><p>The second is to skip text extraction entirely: render each page as an image and give it to a vision model. This is the idea behind PixelRAG, and the argument for it is simple. No extraction step means nothing gets lost in extraction.</p><p>I ran both on 70 questions drawn from financial statements, scanned forms and charts. They tie on most documents. The vision pipeline pulls ahead when the answer depends on something that only exists visually, like the colour of a line. And even where their scores match, they fail in opposite ways, which matters more in production than the scores do.</p><p>The code and results are on <a href="https://github.com/Sij-Agentic/rag-benchmark">GitHub</a>.</p><h3>The two pipelines</h3><p>The pipelines differ only in what the answering model sees. Everything else is held constant:</p><ul><li><strong>Generator:</strong> Gemini 2.5 Flash at temperature 0</li><li><strong>Retrieval:</strong> Jina jina-embeddings-v5-omni-small text embeddings with a FAISS index, k=5</li><li><strong>Judge:</strong> Gemini 2.5 Flash at temperature 0, comparing each answer with the gold answer</li><li><strong>Prompt:</strong> answer strictly from the context, or say “Cannot determine from provided context”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kSd2ce92K7WltksOItIxJA.png" /><figcaption><em>The two pipelines share everything except what the model reads.</em></figcaption></figure><h3>Pipeline B: LlamaParse</h3><p>LlamaParse converts the PDF to Markdown. MarkdownNodeParser from LlamaIndex then splits it on structure (headers, tables) rather than character count, so a table stays in one chunk.</p><pre>parser = LlamaParse(api_key=llama_api_key, result_type=&quot;markdown&quot;)<br>docs = parser.load_data(str(pdf_path))</pre><pre>node_parser = MarkdownNodeParser()<br>nodes = node_parser.get_nodes_from_documents(docs)</pre><p>This used llama-parse 0.6.94 in its default Markdown mode. LlamaParse&#39;s higher tiers, such as Premium and Agentic, parse with vision or LLM models and may behave quite differently. I&#39;ll come back to this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1JL5Sk7JlCXOUyUM" /><figcaption>LlamaParse — <a href="https://www.llamaindex.ai/blog/designing-a-visual-document-intelligence-workflow-with-llamaparse">https://www.llamaindex.ai/blog/designing-a-visual-document-intelligence-workflow-with-llamaparse</a></figcaption></figure><h3>Pipeline C: PixelRAG-style</h3><p>PixelRAG is an open-source RAG framework that retrieves and reads page images rather than extracted text. This pipeline uses its pixelshot renderer² to turn each page into an image at 150 DPI. Pages are retrieved by a text embedding of their PyPDF text, and the retrieved page images are sent to Gemini.</p><pre>prompt_parts = [<br>    &quot;Answer the following question based strictly on the provided document page images.&quot;,<br>    &quot;Read the visual content carefully including any tables, charts, or layouts.&quot;,<br>    &quot;If the answer cannot be determined, say &#39;Cannot determine from provided context.&#39;&quot;,<br>    f&quot;\nQuestion: {question}\n&quot;,<br>]<br>for chunk in retrieved_chunks:<br>    img_data = base64.b64encode(chunk.image_path.read_bytes()).decode()<br>    prompt_parts.append({&quot;inline_data&quot;: {&quot;mime_type&quot;: &quot;image/jpeg&quot;, &quot;data&quot;: img_data}})<br><br>response = client.models.generate_content(model=&quot;gemini-2.5-flash&quot;, contents=prompt_parts)</pre><p>It&#39;s PixelRAG-<em>style</em> because it doesn&#39;t use PixelRAG&#39;s retriever, which embeds the screenshots themselves. In this benchmark retrieval turned out not to matter (see below), so that wouldn&#39;t have changed the results.</p><p>I also ran a naive baseline, Pipeline A: PyPDF text in 1,000-character chunks. It needs a text layer, so it only appears on the financial documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3__T6tSqAYBgxIfT.png" /><figcaption><a href="https://github.com/StarTrail-org/PixelRAG">https://github.com/StarTrail-org/PixelRAG</a></figcaption></figure><h3>The benchmark: FinanceBench, DocVQA and ChartQA</h3><p>The questions come from three public datasets where extraction is hard: 19 from FinanceBench (financial filings), 30 from DocVQA (scanned forms, letters and adverts) and 20 from ChartQA (bar and line charts). The DocVQA and ChartQA documents are image-only.</p><pre>prompt_parts = [<br>    &quot;Answer the following question based strictly on the provided document page images.&quot;,<br>    &quot;Read the visual content carefully including any tables, charts, or layouts.&quot;,<br>    &quot;If the answer cannot be determined, say &#39;Cannot determine from provided context.&#39;&quot;,<br>    f&quot;\nQuestion: {question}\n&quot;,<br>]<br>for chunk in retrieved_chunks:<br>    img_data = base64.b64encode(chunk.image_path.read_bytes()).decode()<br>    prompt_parts.append({&quot;inline_data&quot;: {&quot;mime_type&quot;: &quot;image/jpeg&quot;, &quot;data&quot;: img_data}})<br><br>response = client.models.generate_content(model=&quot;gemini-2.5-flash&quot;, contents=prompt_parts)</pre><p>It&#39;s PixelRAG-<em>style</em> because it doesn&#39;t use PixelRAG&#39;s retriever, which embeds the screenshots themselves. In this benchmark retrieval turned out not to matter (see below), so that wouldn&#39;t have changed the results.</p><p>I also ran a naive baseline, Pipeline A: PyPDF text in 1,000-character chunks. It needs a text layer, so it only appears on the financial documents.</p><h3>The benchmark: FinanceBench, DocVQA and ChartQA</h3><p>The questions come from three public datasets where extraction is hard: 19 from FinanceBench (financial filings), 30 from DocVQA (scanned forms, letters and adverts) and 20 from ChartQA (bar and line charts). The DocVQA and ChartQA documents are image-only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TYo8fwsckaCXuGZ69e55Ng.png" /><figcaption><em>The benchmark corpus.</em></figcaption></figure><p>Two things about this setup matter.</p><p>First, <strong>it measures reading, not retrieval.</strong> Each question gets its own index built from a single document, usually one page long, so retrieval is trivially correct. What’s being compared is how well each pipeline reads the page.</p><p>Second, the questions are clustered. The 30 DocVQA questions come from 12 images and the 20 chart questions from 10 charts, so one answer moves a dataset score by 3–5 points.</p><p>Each pipeline was run three times. Scores are from run 1 unless I give a range. Two of LlamaParse’s FinanceBench questions crashed out of memory while embedding very long tables on my 15 GB GPU. That’s my hardware rather than LlamaParse, so they’re excluded.</p><h3>Overall results</h3><p>The vision pipeline leads overall, 81.4% to 75.0%, but the gap isn’t statistically significant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFgqLoJFyl3tVjqAbe7YVQ.png" /><figcaption><em>Run 1 accuracy by dataset. Two LlamaParse questions that crashed out of memory are excluded.</em></figcaption></figure><p>On the 68 questions both completed, both were right on 45 and both wrong on 6. Only LlamaParse was right on 6, and only the vision pipeline on 11. A sign test on those 17 disagreements gives p = 0.33, well within what chance could produce. None of the per-dataset gaps is significant either.</p><p>The more interesting question is <em>where</em> they disagree, and why.</p><h3>Charts: a tie, until the question mentions a colour</h3><p>The chart gap (45% vs 70%) is the largest in the benchmark, and it would be easy to conclude that vision models are simply better at charts. The data says something more specific.</p><p>Half of the ChartQA questions refer to a data series by its colour, e.g. “What’s the lowest value of the yellow line?” Splitting on that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTA4uNTQsrNyIhzNngqFww.png" /><figcaption><em>ChartQA accuracy, split by whether the question refers to a colour.</em></figcaption></figure><p>Without a colour reference, they score 8/10 each. With one, LlamaParse gets 1/10 and the vision pipeline 6/10. Almost the entire chart gap is the colour questions.</p><p>The reason is straightforward. When LlamaParse converts a chart to Markdown, it keeps the axis labels, values and legend entries. The colour of each line doesn’t survive. So when a question asks about “the yellow line”, nothing in the Markdown says which series that is, and the model refuses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*So9uiCrpX3MAPsKTi2wVPQ.png" /><figcaption><em>“What’s the lowest value of yellow line?” The Markdown has the values but not the colours.</em></figcaption></figure><p>The same happened outside charts. A DocVQA advert asked for the name of the “choco fills” advertised. The brand name appears only in the artwork. LlamaParse refused; the vision pipeline answered “Dark Fantasy”, which is correct.</p><p>When chart values are labelled, though, LlamaParse’s default mode reads them about as well as a vision model. It correctly counted two points above 50 on a “disapprove” line, citing October at 51 and November at 53, where the vision pipeline miscounted.</p><p>With 10 questions per group this is a small sample (p = 0.18 on the chart disagreements). But it’s a clear pattern with a clear cause, which I find more convincing than the p-value. A vision model has an advantage when the question points at something that only exists visually. Otherwise the advantage mostly disappears.</p><h3>They fail in different ways</h3><p>This is the finding I think matters most in practice. Both pipelines had the same instruction to say “Cannot determine” if they couldn’t answer, but their wrong answers break down very differently:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qF0uEC7wy4X9YgGW1l-wvQ.png" /><figcaption><em>Wrong answers in run 1, split into refusals and wrong values.</em></figcaption></figure><p>12 of LlamaParse’s 17 errors were refusals. 11 of the vision pipeline’s 13 errors were wrong values.</p><p>Part of this is the colour effect. When the information isn’t in the Markdown at all, LlamaParse’s model can see it’s missing and says so. The vision model has the information in front of it, misreads it, and gives no sign that anything went wrong:</p><blockquote><strong><em>Q:</em></strong><em> Nike FY2019 total current assets </em>(gold: $16,525M)</blockquote><blockquote><strong><em>LlamaParse:</em></strong><em> $16,525 million</em></blockquote><blockquote><strong><em>Vision:</em></strong><em> $16,798 USD millions</em></blockquote><p>That’s a plausible number in the right format, and it’s wrong. It’s the kind of error usually called a hallucination, though here the correct figure was on the page and the model misread it.</p><p>It isn’t a clean rule. LlamaParse confidently said a letter was sent to “Bill” when it was Paul, and the vision pipeline refused an Amcor question that LlamaParse got right. But the tendency is clear enough to plan around.</p><p>A refusal is visible. You can detect it, log it, retry with another pipeline or send it to a person. A confidently wrong number is much harder to catch. So even if two pipelines scored exactly the same, I wouldn’t treat them as interchangeable, particularly for anything financial.</p><h3>Financial tables: LlamaParse is at least as good, and steadier</h3><p>On FinanceBench both got 15 right in run 1, out of 17 for LlamaParse and 19 for vision. That’s a tie, or at most a slight edge to LlamaParse.</p><p>The bigger difference is consistency. Across three runs, LlamaParse scored 88–94% and the vision pipeline 68–79%, with three of its answers changing between runs even at temperature 0. Reading a dense table from an image seems to be less stable than reading it as Markdown.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*auytC4UxQlHrwY_94luz4A.png" /><figcaption><em>FinanceBench accuracy across three runs.</em></figcaption></figure><p>Naive PyPDF scored 78.9% here. These filings have a proper text layer and were trimmed to the relevant pages, and on documents like that plain extraction is a strong baseline worth measuring first.</p><p>It does fail in a characteristic way, though. Asked for 3M’s FY2018 capital expenditure:</p><ul><li><strong>Naive</strong> refused. The figure 1,577 was in the retrieved chunk, but character splitting had separated it from its row label, “Purchases of property, plant and equipment”.</li><li><strong>LlamaParse</strong> answered $1,577 million. The Markdown row | Purchases of property, plant and equipment (PP&amp;E) | (1,577) | (1,544) | (1,522) | keeps label and value together.</li><li><strong>Vision</strong> answered 1,577, reading the row straight off the page.</li></ul><p>LlamaParse and the vision pipeline both solve the table-structure problem, one by encoding rows and columns explicitly and the other by reading the grid.</p><h3>Cost and speed</h3><p>Most of the difference is at ingestion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mZ_FF2wfXWrP_l9V7boVZQ.png" /><figcaption><em>Ingestion is the first, uncached pass over each document; per-query figures are means over three runs. Gemini 2.5 Flash pricing as of 28 September 2026.</em></figcaption></figure><p>Rendering a page locally is free and takes about half a second. LlamaParse is a paid API call that took about 17 seconds per document on the first pass, after which the output is cached. The 123 pages here would cost roughly $0.46–$1.54 to parse, depending on which pricing tier the default mode corresponds to, which LlamaParse doesn’t publish.³ It also means every document goes to a third-party service, although both pipelines send the retrieved pages to Gemini anyway.</p><p>Per query, they’re about even. The page images used fewer input tokens than the Markdown, but the vision model wrote longer answers, so the Gemini cost came out almost identical.</p><h3>Which one to use</h3><ul><li><strong>If questions depend on visual properties</strong> such as colour, legends, logos or artwork, use a vision pipeline. This is the one clear gap, and it has a clear cause.</li><li><strong>For dense financial tables with a text layer</strong>, LlamaParse was at least as accurate and more consistent. Try plain PyPDF first; on clean filings it may be enough.</li><li><strong>For scanned forms and letters</strong>, either works (86.7% vs 90.0%).</li><li><strong>If a confidently wrong answer is worse than no answer</strong>, prefer LlamaParse. Its failures are mostly refusals you can detect.</li><li><strong>If ingestion cost or speed is the constraint</strong>, use a vision pipeline.</li></ul><h3>Limitations</h3><ul><li><strong>Sample size.</strong> 70 questions from clustered sources, and none of the head-to-head gaps is significant.</li><li><strong>Reading, not retrieval.</strong> On a real corpus, where the right page has to be found among thousands, results could differ, and PixelRAG’s own retriever would matter.</li><li><strong>The judge.</strong> Gemini judged Gemini, which risks bias. I also found two correct answers marked wrong: LlamaParse’s “$5,466.312 million” for Netflix against a gold answer of $5,466, and the vision pipeline’s “(1,577)” in accounting parentheses for 3M in one run. Treat the scores as approximate.</li><li><strong>LlamaParse’s mode.</strong> It ran in default Markdown mode through the older v1 client. The Premium and Agentic tiers might close the colour gap, which is the biggest caveat on the chart results.</li><li><strong>One vision model.</strong> Only Gemini 2.5 Flash was tested.</li></ul><h3>What’s next</h3><p>The obvious next experiment is a hybrid. Since the two fail differently, combining them should help, either by routing colour questions to the vision model or by giving the model both the Markdown and the page image.</p><p>A cheaper version would use one to check the other. In the Nike example, the correct figure was in LlamaParse’s Markdown the whole time, and checking the vision model’s numbers against it would have caught the mistake.</p><p>I’d also like to rerun this with real retrieval over a larger corpus, with PixelRAG’s own retriever and LlamaParse’s vision-based tiers, and with an open-source parser such as Docling as a third pipeline.</p><h3>Notes</h3><ol><li>Code, per-run results and all disagreements between pipelines: <a href="https://github.com/Sij-Agentic/rag-benchmark">https://github.com/Sij-Agentic/rag-benchmark</a></li><li>PixelRAG: <a href="https://github.com/StarTrail-org/PixelRAG">https://github.com/StarTrail-org/PixelRAG</a></li><li>Gemini 2.5 Flash at $0.30 per 1M input tokens and $2.50 per 1M output tokens; LlamaParse at $1.25 per 1,000 credits. Prices checked on 28 September 2026.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6df0ea88747d" width="1" height="1" alt=""><hr><p><a href="https://aiadvances.org/llamaparse-vs-pixelrag-they-tie-until-the-question-mentions-a-colour-6df0ea88747d">LlamaParse vs PixelRAG: They Tie, Until the Question Mentions a Colour</a> was originally published in <a href="https://aiadvances.org/">AI Advances</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stability, Perception, and Control: A Look at Modern Reinforcement Learning]]></title>
            <link>https://medium.com/matrixntensors/stability-perception-and-control-a-look-at-modern-reinforcement-learning-24c35c01d2eb?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c35c01d2eb</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[autonomous-driving]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Mon, 02 Feb 2026 21:42:03 GMT</pubDate>
            <atom:updated>2026-02-02T21:42:03.199Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A practical journey through actor–critic methods, TD3, and the complexity of learning to act</em></p><h4>The Driving Environment</h4><p>I built a reinforcement learning agent to drive a car along white lane markings in a simulated city map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SyM6xT943acMfSxTiPbklg.png" /><figcaption>A top-down 2D simulation of a 2d city map with white lane markings.</figcaption></figure><p>The task was simple enough: observe the environment, steer left or right, adjust speed, and avoid obstacles while driving towards the target.</p><p>The environment is simple on the surface: a 2D city map, white lane markings, obstacles, and a car that needs to navigate between targets. At each step, the agent observes its surroundings through a small set of sensors and decides how to steer and how fast to move.</p><h4>What the Agent Has to Deal With</h4><p>Before getting into the algorithms, its worth to be explicit about what this environment demands of the driving agent, as many of the RL concepts that follow only make sense in that context.</p><p>At each timestep, the agent must make decisions under many simultaneous constraints.</p><p><strong>Perception Is Local and Imperfect</strong></p><p>The car does not have a global map. It relies instead on a number of distance sensors that detect if the pixel ahead is road or obstacle.</p><p>This means:</p><ul><li>The agent’s view of the world is partial.</li><li>Different situations can look locally similar.</li><li>and long term consequences aren’t directly observable from a single step.</li></ul><p>Designing a state representation that is useful becomes part of the problem.</p><p><strong>Control Is Continuous</strong></p><ul><li>The agent doesn’t just choose from a fixed set of actions. It outputs real-valued steering angles and speeds at every step.</li><li>Small changes matter and tiny errors compound over time. Many classical RL methods and solutions do not cleanly apply here.</li></ul><p>Taken together, these properties make the driving environment a compact example of what modern reinforcement learning is about: Learning under partial observability, acting in continuous spaces, using value estimates to guide behaviour over time, and stabilizing learning when your own actions influence the data you learn from.</p><p>When viewed through this lens, some questions naturally arise: How do you evaluate actions when feedback is delayed? How do you keep learning stable when estimates are noisy? How do you explore without destroying what you’ve already learned?</p><h3>Why Actor–Critic Is Right for This Problem</h3><p>The agent needs to do two things at the same time:</p><ul><li><strong>Decide how to act</strong> in a continuous control setting</li><li><strong>Evaluate whether those actions were good or bad</strong>, even when feedback is delayed.</li></ul><h4>Acting and Evaluating Are Different Problems</h4><p>In this driving task, choosing an action and judging an action are fundamentally different:</p><ul><li><strong>Choosing an action is about control</strong>: given what I see now, how should I steer, and how fast should I go?</li><li><strong>Judging an action is about consequences</strong>: did that decision make it more likely that I’ll stay on the road and reach a target later?</li></ul><p>Actor–critic methods formalize this separation. The <strong>actor</strong> focuses on <em>behavior</em>: mapping observations to actions. The <strong>critic</strong> focuses on <em>evaluation</em>: estimating how good states or state–action pairs are over time. This division makes learning tractable in environments such as this one.</p><h4>Why Do Pure Value-Based Methods Fall Short?</h4><p>One approach would be to learn a single function that assigns a value to every possible action and then always pick the best one. In practice, that idea runs into immediate trouble here because the action space is continuous, there’s no clean way to search over “all actions,” and and small estimation errors can lead to wildly different behavior.</p><h4>Why Pure Policy Learning Isn’t Enough Either</h4><p>At the other extreme, one could try to learn the policy directly, adjusting it based on how well things turned out. That works in principle, but in environments like this one, rewards arrive late, outcomes are noisy, and it’s hard to tell which decision was responsible for what happened. Without some notion of value to anchor learning, updates become unstable and inefficient.</p><h4>Actor–critic methods sit in the middle</h4><p>The actor and critic are solving <strong>different optimization problems</strong>, even though they interact closely. The actor is not trying to predict rewards. The critic is not trying to choose actions. Each does one thing well.</p><p>The critic answers the question: <em>“From here, how good does the future look if I behave the way I currently do?”.</em></p><p>The actor then uses that signal to adjust its behavior: increasing the likelihood of actions that tend to lead to better futures, and decreasing the likelihood of those that don’t.</p><p>This separation also makes it clear where things can go wrong:</p><ul><li>If the critic’s estimates drift or become too optimistic, the actor will learn the wrong behavior.</li><li>If the actor changes too quickly, the critic is forced to chase a moving target.</li><li>Learning collapses, if both are unstable at the same time.</li></ul><p>This dynamic sets the stage for <strong>TD3</strong>(Twin Delayed Deep Deterministic Policy Gradient). It identifies and corrects the systemic instabilities inherent in <strong>DDPG(</strong>Deep Deterministic Policy Gradient)<strong>.</strong></p><h3>TD3</h3><p>TD3 doesn’t fundamentally change actor–critic. Instead, it asks a more pragmatic question:</p><blockquote>What goes wrong when a learned policy depends too heavily on a learned value estimate?</blockquote><blockquote>The answer, it turns out, is quite a lot.</blockquote><p>In continuous control, the critic is asked to assign values to state–action pairs it has only partially explored. Small estimation errors are unavoidable.</p><p>The problem is that the actor is trained to <strong>maximize</strong> those estimates.</p><p>When the critic slightly overestimates the value of some action, the actor is incentivized to exploit that error. The critic then sees more of that biased behavior, reinforcing the mistake. TD3’s first design choice is simply to be more cautious.</p><h3>Twin Critics and Pessimism</h3><p>Instead of learning a single value function, TD3 learns two. When computing targets, it uses the <em>minimum</em> of the two estimates rather than trusting either one outright. This introduces a mild pessimism that significantly reduces overestimation bias.</p><p>It’s not sexy in a theoretical sense, but it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/0*eGrJ_n5H2Ux87BnX" /><figcaption>The TD3 Target Formula</figcaption></figure><p>In this equation:</p><ul><li><strong>r</strong> is the immediate reward received after taking an action.</li><li><strong>γ (Gamma)</strong> is the discount factor, which controls how much future outcomes matter compared to immediate ones.</li><li><strong>Q_1 and Q_2 </strong>are two independently learned critics.</li><li><strong>s’ and a’</strong> refer to the next state and the action proposed by the current policy.</li></ul><h4>Delaying the Actor</h4><p>TD3 also decouples how often the actor and critic are updated. The critic learns more frequently, while the actor updates more slowly. This gives the value function time to stabilize before the policy tries to exploit it. In practice, this simple delay prevents the actor from chasing noise.</p><h4>Smoothing the Target Policy</h4><p>Finally, TD3 adds small noise to the actions used in value targets. This discourages the critic from assigning unrealistically high values to extremely narrow action choices and encourages smoother value landscapes.</p><p>All together, TD3 adds a level of discipline to actor-critic methods. And that discipline turned out to matter a lot in the driving environment.</p><h3>Bringing It Back to the Driving Task</h3><p>The agent’s entire understanding of the world comes from a small state vector derived from visual input: distance sensors, angle to target, and distance remaining.</p><p>Early on, subtle issues in how these signals were computed had major effects on behavior. Sensors that blurred the difference between road and obstacle produced agents that behaved confidently, and incorrectly.</p><pre># PROBLEM: Treats colored areas as partially valid road<br>c = QColor(self.map.pixel(int(sx), int(sy)))<br>brightness = (c.red() + c.green() + c.blue()) / 3<br>sensor_value = brightness / 255  # Returns 0.0 to 1.0</pre><pre># SOLUTION: Only white pixels are road, everything else is obstacle<br>c = QColor(self.map.pixel(int(sx), int(sy)))<br>r, g, b = c.red(), c.green(), c.blue()<br>is_white = (min(r, g, b) &gt; 200) and (max(r, g, b) - min(r, g, b) &lt; 30)<br>sensor_value = 1.0 if is_white else 0.0  # Binary: road or not</pre><h4>Rewards Had to Be Interpretable</h4><p>Overly complex rewards produced fragile policies. Simpler signals led to smoother behavior and more consistent learning. The lesson learned here was simple: the critic needs clarity more than rich instruction.</p><pre># Base time penalty<br>reward = -0.1</pre><pre># Crash: strong penalty<br>if car_on_obstacle:<br>    reward = -100<br>    done = True</pre><pre># Target reached: strong reward<br>elif distance &lt; TARGET_RADIUS:<br>    reward = +100</pre><pre># Normal navigation: 3 simple components<br>else:<br>    reward += num_clear_sensors * 2.5      # Safety<br>    reward += distance_improvement * 15     # Progress<br>    reward += (1.0 - abs(angle_error)) * 3  # Alignment</pre><blockquote>the full codebase is available on GitHub: <a href="https://github.com/cydal/T3D-car-navigation">https://github.com/cydal/T3D-car-navigation</a></blockquote><h4>Exploration Became a Problem</h4><p>TD3 still relied on external exploration noise. Once the policy began to improve, the added noise became harmful, introducing failures late in training that had nothing to do with the agent’s competence. Gradually reducing exploration helped respect what the critic had already learned.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTAtcViTyd7E%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTAtcViTyd7E&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTAtcViTyd7E%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/63f9ab293a9824f57fbb0d27a5b36772/href">https://medium.com/media/63f9ab293a9824f57fbb0d27a5b36772/href</a></iframe><h3>The Evolution from Rules to Intuition</h3><p>TD3 essentially tells the car: Follow this path, but wiggle the wheel randomly so you don’t get stuck. It’s an effective fix, but it’s still a clunky, manual way to discover new strategies, like a student following a rigid instruction manual.</p><p>In the general spirit of machine learning, we aim to replace such rules with systems that learn for themselves. Instead of forcing the car to wiggle the wheel, we want it to <em>want</em> to try new things.</p><h3>Enter SAC(Soft Actor-Critic)</h3><p><strong>SAC </strong>achieves this by internalizing curiosity, making the search for new maneuvers a core part of the car’s goal rather than a rule enforced from the outside.</p><p>Ultimately, it is a perfect hand-off in the evolution of these models: <strong>where TD3 fixed exploitation, SAC fixed exploration.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/0*YxyL--P32p2Xq2eS" /><figcaption>The SAC Objective: Learning to be Curious</figcaption></figure><p>In this objective:</p><ul><li><strong>r_t</strong> is the reward at timestep t.</li><li><strong>$\mathcal{H}(\pi(\cdot \mid s_t))$</strong> is the <strong>entropy</strong> of the policy’s action distribution — a mathematical measure of how uncertain or exploratory the car is being.</li><li>α<strong> (Alpha)</strong> is the “temperature” parameter. It acts as a dial that controls the trade-off between reward maximization (exploitation) and exploration.</li></ul><h3>Why This Isn’t Just About Cars</h3><p>While this project is framed around a driving task, the structure of the problem is not unique to cars, or even to robotics. The problem has the following properties:</p><ul><li>decisions are continuous rather than discrete</li><li>observations are partial and noisy</li><li>rewards reflect long-term outcomes rather than immediate correctness</li><li>actions change the future data the system learns from.</li></ul><p>This shows up in many real-world systems.</p><ul><li>In robotics, it appears in manipulation, locomotion, and navigation.</li><li>In operations and logistics, it appears in resource allocation and scheduling.</li><li>In energy systems, it appears in load balancing and control.</li><li>In finance, it appears in portfolio management and execution strategies.</li></ul><p>In all of these cases, the challenge is not prediction in isolation, but <strong>learning how to act under uncertainty while managing long-term consequences</strong>.</p><p>The driving environment simply makes these dynamics visible.</p><h3>References</h3><p><strong>Shravan, Rohan</strong> TSAI</p><h4>Foundational Concepts</h4><ul><li><strong>Sutton, R. S., &amp; Barto, A. G. (2018).</strong> <em>Reinforcement Learning: An Introduction (2nd Edition).</em> MIT Press. (The definitive “Bible” of RL for understanding the basics of value and policy methods).</li><li><strong>Williams, R. J. (1992).</strong> “Simple statistical gradient-following algorithms for connectionist reinforcement learning.” <em>Machine Learning</em>. (The original paper for the <strong>REINFORCE</strong> algorithm).</li></ul><h4>Value-Based Methods (DQN)</h4><ul><li><strong>Mnih, V., et al. (2015).</strong> “Human-level control through deep reinforcement learning.” <em>Nature</em>. (The landmark paper that introduced <strong>DQN</strong> and combined Reinforcement Learning with Deep Learning).</li></ul><h4>Policy-Based &amp; Actor-Critic (DDPG)</h4><ul><li><strong>Lillicrap, T. P., et al. (2016).</strong> “Continuous control with deep reinforcement learning.” <em>ICLR</em>. (The original <strong>DDPG</strong> paper that brought DQN concepts to continuous action spaces).</li></ul><h4>The “Discipline” of TD3</h4><ul><li><strong>Fujimoto, S., van Hoof, H., &amp; Meger, D. (2018).</strong> “Addressing Function Approximation Error in Actor-Critic Methods.” <em>ICML</em>. (The official <strong>TD3</strong> paper — this is where “Clipped Double-Q Learning” and “Delayed Updates” were introduced).</li></ul><h4>Maximum Entropy &amp; SAC</h4><ul><li><strong>Haarnoja, T., et al. (2018).</strong> “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.” <em>ICML</em>. (The foundational <strong>SAC</strong> paper that introduced the entropy-regularized objective).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c35c01d2eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/matrixntensors/stability-perception-and-control-a-look-at-modern-reinforcement-learning-24c35c01d2eb">Stability, Perception, and Control: A Look at Modern Reinforcement Learning</a> was originally published in <a href="https://medium.com/matrixntensors">MatrixnTensors</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Embedding Memory Into An AI Agent: A Practical Guide with FastAPI, ChromaDB, and Ollama]]></title>
            <link>https://sijpapi.medium.com/embedding-memory-into-an-ai-agent-a-practical-guide-with-fastapi-chromadb-and-ollama-7dd765567ac5?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dd765567ac5</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ollama]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Fri, 23 Jan 2026 22:47:25 GMT</pubDate>
            <atom:updated>2026-01-23T22:47:25.633Z</atom:updated>
            <content:encoded><![CDATA[<p>A modular AI agent project that perceives, decides, acts, and remembers</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*g8YLQSudalQja6tW4TDVjg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@anniespratt?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Annie Spratt</a> on <a href="https://unsplash.com/photos/public-storage-building-with-a-sign-loSMnj4Piq8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong><em>Memory transforms agents from reactive tools into adaptive collaborators.</em></strong><em> It allows them to carry lessons forward, build richer internal models of the world, and ultimately, act with greater relevance and autonomy.</em></p><p><em>In this article, I’ll walk through a simple but powerful project where I built a modular AI agent that doesn’t just perceive and act — it remembers.</em></p><p><em>Using FastAPI, ChromaDB, and Ollama, I created an agent that integrates short-term and long-term memory, enabling it to store, retrieve, and act on past experiences.</em></p><p><em>We’ll explore how memory fits into the agent’s architecture, how semantic embeddings unlock smarter recall, and what challenges and opportunities emerge when you move beyond stateless AI design.</em></p><p><strong><em>Let’s dive into how memory, even in a simple system, can fundamentally change what an agent can do.</em></strong></p><h3>Overview: Building a Memory-Enabled Agent</h3><p>Building an agent that can perceive, decide, act, and <strong>remember</strong> required a modular design and a focused technology stack.</p><p>Rather than aiming for complexity, I set out to create a simple, understandable agent whose parts could evolve independently. This modularity allowed me to focus on memory as a first-class citizen within the system architecture.</p><p>At a high level, the agent is composed of four main components:</p><ul><li><strong>Perception</strong> (perception.py) — Processes incoming requests and interprets the environment.</li><li><strong>Memory</strong> (memory.py) — Stores and retrieves both short-term and long-term contextual information.</li><li><strong>Decision Making</strong> (decision.py) — Determines what action the agent should take based on its perception and memory.</li><li><strong>Action Execution</strong> (action.py) — Carries out the decided action in the environment.</li><li><strong>Schemas</strong> (schemas.py) — Defines and validates the structured data exchanged between components.</li></ul><p>These components are orchestrated by a simple controller (main.py) and served via an API using <strong>FastAPI</strong>.</p><h3>Core Technologies</h3><p>To bring memory into the agent in a practical way, I used the following tools:</p><p><strong>FastAPI — </strong>Lightweight, asynchronous API framework to expose the agent.</p><p><strong>Ollama— </strong>Local embedding generation engine for turning perceptions into semantic vectors.</p><p><strong>ChromaDB — </strong>A vector database to store and retrieve long-term memories based on semantic similarity.</p><pre>.<br>├── main.py          # The Controller/Orchestrator<br>├── perception.py    # Input processing logic<br>├── memory.py        # ChromaDB &amp; Ollama integration<br>├── decision.py      # Logic for choosing actions<br>├── action.py        # Execution of tasks<br>└── schemas.py       # Pydantic data models</pre><h3>The Architecture: Perception → Memory → Decision → Action</h3><p>At the heart of this project is a simple but powerful architecture that mirrors how intelligent systems — and even humans — interact with their environments:</p><p><strong>Perceive → Recall → Decide → Act → Learn.</strong></p><p>The agent’s internal workflow follows this loop every time it receives a new input:</p><h3>Step 1: Perception</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grr8ee1YW6uoadNV_lATpg.png" /></figure><p>Incoming requests are first processed by the <strong>perception module</strong>. This step extracts key information from the input and formats it for internal use. This “understanding” forms the basis of the agent’s next steps.</p><h3>Step 2: Memory Lookup</h3><p>Before deciding what to do, the agent checks its <strong>memory</strong>:</p><ul><li><strong>Short-Term Memory (STM):</strong></li><li>Immediate context from the current session (what just happened, recent perceptions).</li><li><strong>Long-Term Memory (LTM):</strong></li><li>Semantic memories retrieved from <strong>ChromaDB</strong>, based on similarity to the current perception.</li></ul><p>This dual memory system ensures the agent acts <strong>not only based on what it sees now</strong>, but also on <strong>what it remembers</strong>.</p><h3>Step 3: Decision</h3><p>With both perception and retrieved memories in hand, the <strong>decision module</strong> determines the next action.</p><p>The decision could be based purely on the new perception, influenced heavily by past similar experiences, or some combination of both.</p><h3>Step 4: Action</h3><p>Once a decision is made, the <strong>action module</strong> executes it, whether that means replying, triggering a system event, or updating its environment.</p><h3>Step 5: Memory Update</h3><p>After the action is completed, the agent <strong>writes back</strong> into memory:</p><ul><li>Storing the new perception and outcome.</li><li>Updating its context for future use.</li></ul><p>This way, every interaction has the potential to enrich the agent’s understanding over time.</p><pre>async def process_request(request: AgentRequest):<br>    perception = perceive(request)<br>    past_memories = memory_lookup(perception)<br>    decision = decide(perception, past_memories)<br>    action_result = act(decision)<br>    update_memory(perception, decision, action_result)<br>    return action_result</pre><h3>The Architecture in Action</h3><p>To see why this matters, imagine this scenario:</p><ol><li><strong>Perception:</strong> User says, “What’s for dinner?”</li><li><strong>Memory Lookup:</strong> The agent recalls a long-term memory: <em>“User is allergic to shellfish”</em> and a short-term memory: <em>“User said they wanted to eat healthy today.”</em></li><li><strong>Decision:</strong> The agent filters out shrimp recipes and chooses a Salad.</li><li><strong>Action:</strong> The agent suggests a “Zesty Lemon Chicken Salad.”</li><li><strong>Memory Update:</strong> The agent saves the fact that the user accepted this suggestion.</li></ol><h3>Tools: FastAPI, ChromaDB, and Ollama</h3><p>Building a modular, memory-enabled agent is one thing — but choosing the right tools made the project <strong>feasible, efficient, and scalable</strong>.</p><p>Here’s a closer look at the key technologies that powered this system:</p><h3>FastAPI: Lightweight, Async Serving</h3><p>At the core of the agent’s API layer is <strong>FastAPI</strong> — a modern, high-performance framework for building web APIs with Python.</p><p><strong>Why FastAPI?</strong></p><ul><li><strong>Asynchronous support</strong> out of the box, perfect for non-blocking embedding generation and memory queries.</li><li><strong>Automatic data validation</strong> using Pydantic models (schemas.py), ensuring structured communication between modules.</li><li><strong>Easy scalability</strong> for future extensions (authentication, session management, logging, etc.)</li></ul><p>A simple endpoint for interacting with the agent looks like this:</p><pre>from fastapi import FastAPI<br>from schemas import AgentRequest<br>from main import process_request<br><br>app = FastAPI()<br><br>@app.post(&quot;/agent&quot;)<br>async def agent_endpoint(request: AgentRequest):<br>    response = await process_request(request)<br>    return response</pre><p>FastAPI kept the project <strong>fast to develop</strong> and <strong>fast to serve</strong>, crucial for local experimentation.</p><h3>Ollama: Local Embedding Generation</h3><p>Instead of relying on remote APIs like OpenAI embeddings, this project uses <strong>Ollama</strong> to generate embeddings <strong>locally</strong>.</p><p><strong>Why Ollama?</strong></p><ul><li><strong>Local-first</strong> — No dependency on external services, better privacy, faster iteration.</li><li><strong>Model flexibility</strong> — Easy to swap embedding models depending on speed or quality needs.</li><li><strong>Reduced cost</strong> — No per-token fees or external API limitations.</li></ul><p>Generating an embedding is straightforward:</p><pre>import ollama<br><br># Generate a semantic vector using a local model<br>response = ollama.embeddings(model=&#39;nomic-embed-text&#39;, prompt=perception_text)<br>embedding = response[&#39;embedding&#39;]</pre><p>This embedding process converts raw perceptions into <strong>semantic vectors</strong> that can be stored, retrieved, and compared.</p><h3>ChromaDB: Semantic Memory Storage</h3><p>To manage the agent’s growing memory base, I used <strong>ChromaDB</strong>, an open-source vector database optimized for local or scalable cloud deployment.</p><p><strong>Why ChromaDB?</strong></p><ul><li><strong>Designed for embeddings</strong> — Built specifically for vector-based semantic search.</li><li><strong>Fast retrieval</strong> — Even with growing memory, queries remain efficient.</li><li><strong>Flexible storage</strong> — Can run purely in-memory for local setups or persist to disk for long-term storage.</li></ul><p>Inserting and retrieving memory into ChromaDB:</p><pre>import chromadb<br><br># Initialize the database and collection<br>client = chromadb.PersistentClient(path=&quot;./memory_db&quot;)<br>collection = client.get_or_create_collection(name=&quot;agent_memories&quot;)<br><br># Store the memory with its vector<br>collection.add(<br>    ids=[&quot;mem_001&quot;],<br>    embeddings=[embedding],<br>    documents=[perception_text]<br>)<br><br># Search for the top 3 most relevant past experiences<br>results = collection.query(query_embeddings=[embedding], n_results=3)</pre><p><strong>Full Repository:</strong> <a href="https://github.com/Sij-Agentic/agent-heuristics-buf-fix">https://github.com/Sij-Agentic/agent-heuristics-buf-fix</a></p><p>ChromaDB acts as the agent’s <strong>long-term semantic memory</strong>, enabling it to recall similar experiences even if the words used are different.</p><h3>Final Thoughts</h3><p>While the core architecture came together cleanly, weaving memory into the system introduced a range of new engineering decisions, tradeoffs, and future possibilities.</p><h3>Challenges and Tradeoffs</h3><ul><li><strong>Embedding Latency vs. Quality:</strong> Local inference with Ollama is private but takes a second to “think.” I found smaller models like nomic-embed-text keep the agent snappy.</li><li><strong>Retrieval Relevance:</strong> Not every memory is a good one. I had to set a “similarity threshold” so the agent doesn’t get distracted by irrelevant past events.</li><li><strong>Memory Growth:</strong> Over time, databases get crowded. Future versions will need “Memory Decay” — the ability to forget unimportant details.</li></ul><h3>Future Directions and Opportunities</h3><p><strong>Smarter Memory Filtering</strong></p><ul><li>Adding relevance scoring, recency weighting, or contextual filtering could make memory retrieval even sharper.</li></ul><p><strong>Memory Prioritization and Decay</strong></p><ul><li>Not all memories are equal.</li><li>Implementing decay functions (e.g., slowly forgetting old memories unless reinforced) would help keep long-term memory both meaningful and scalable.</li></ul><p><strong>Expanding Perception Modalities</strong></p><ul><li>Currently, the agent perceives text.</li><li>Future expansions could introduce:</li><li>Image perception (storing image embeddings).</li><li>Structured data or event streams.</li></ul><p><strong>Self-Refining Memories</strong></p><ul><li>Allowing the agent to refine, merge, or correct old memories based on new experiences could move the system closer to truly adaptive learning.</li></ul><h3>Final Reflection</h3><p>I saw firsthand how memory transforms an agent from a simple reactivity machine into a system capable of <strong>learning, adapting, and improving</strong>.</p><p>Even in a small, local setup with FastAPI, ChromaDB, and Ollama, embedding memory created a qualitative leap in what the agent could do.</p><h3>References &amp; Further Reading</h3><ul><li><strong>FastAPI Documentation:</strong> <a href="https://fastapi.tiangolo.com/">High-performance, easy to learn, fast to code</a> — <em>The framework used for our agent’s API.</em></li><li><strong>ChromaDB Docs:</strong> <a href="https://docs.trychroma.com/">The AI-native open-source embedding database</a> — <em>Our core long-term semantic memory storage.</em></li><li><strong>Ollama API:</strong> <a href="https://ollama.com/library">Run Llama 3, Mistral, and other LLMs locally</a> — <em>The engine powering our local embeddings and heuristic decisions.</em></li><li><strong>Semantic Search &amp; Embeddings:</strong> <a href="https://www.pinecone.io/learn/vector-database/">Understanding Vector Databases</a> — <em>A deep dive into how semantic similarity works in modern AI.</em></li><li><strong>Agentic Workflows:</strong> <a href="https://www.google.com/search?q=https://www.deeplearning.ai/the-batch/how-ai-agents-can-improve-workflows-with-iterative-design/">Andrew Ng on Agentic Design Patterns</a> — <em>The conceptual inspiration for modular, iterative agent design.</em></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dd765567ac5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turn Your Browser into a Smart Semantic Search Engine]]></title>
            <link>https://sijpapi.medium.com/turn-your-browser-into-a-smart-semantic-search-engine-aa20e37aaa7d?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa20e37aaa7d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[chrome-extension]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[semantic-search]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Sun, 27 Apr 2025 13:32:27 GMT</pubDate>
            <atom:updated>2025-04-27T13:32:27.896Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Build your own private, lightning-fast web knowledge index with Chrome, FastAPI, FAISS, and Ollama</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s84ihUpvAacoXa4sHnIdBA.png" /></figure><p><strong>Introduction</strong></p><p>Have you ever wasted time digging through your browser’s history or scrolling endlessly through bookmarks just to find that one critical insight you once came across? We’ve all been there: desperately recalling keywords, guessing titles, or struggling to retrace steps to a crucial blog post or documentation snippet. What if your browser could remember not just the pages you’ve visited, but precisely what mattered most about them?</p><p>In this guide, I’ll show you how I transformed my Chrome browser into a private, fast semantic search engine. Powered entirely by local technologies — Chrome, FastAPI, Ollama, and FAISS — my setup remembers every meaningful snippet from my browsing history and retrieves it instantly using natural-language questions.</p><p>Imagine never again having to recall exact keywords or manually manage bookmarks. Instead, effortlessly ask your browser questions like, <em>“Where did I read about async data loading in React?”</em> or <em>“Show me that blog post on vector search.”</em> Moments later, you’re staring at precisely highlighted passages — no guesswork, no hassle, just clarity.</p><p>Let’s dive into exactly how to build this powerful, privacy-first browsing tool yourself.</p><h3><strong>Motivation</strong></h3><p>Every day, we sift through mountains of online information — articles, tutorials, blogs, documentation. Yet when we need to revisit an idea or recall an insight, our browsers let us down. Bookmarks quickly become cluttered, neglected, or entirely forgotten. Keyword searches are limited to the exact phrases we remember, not the underlying concepts we care about. Browser history itself offers nothing more than a list of visited URLs, devoid of context or significance.</p><p>The result? wasted time — precious minutes spent retracing digital footsteps rather than solving real-world problems. But what if our browsers could truly understand what we’ve read? What if they remembered the meaning behind each page visited, rather than just raw text?</p><p>By combining Chrome with locally-hosted semantic embedding (via Ollama’s powerful mxbai-embed-large model) and blazing-fast indexing through FAISS, I created a browsing experience that goes far beyond bookmarks or keyword search. Now, when I ask, <em>“where did I read about AI and the future of work?”</em>, I’m instantly provided with the exact snippet I need, highlighted right on the page — transforming hours of tedious searching into mere seconds of effortless retrieval.</p><p>In the sections ahead, I’ll walk you step-by-step through building your own private, local semantic browsing powerhouse — because your time should be spent gaining insights, not chasing them down.</p><h3>Architecture Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KPbqgN-SS8jO9aHX0ZeY0g.png" /></figure><p>To build your browser-based semantic search engine, you’ll bring together three powerful yet simple-to-integrate components: a Chrome extension, a FastAPI backend paired with Ollama embeddings, and a FAISS vector index.</p><p>Here’s how it all seamlessly fits together:</p><h4>Content Extraction (Chrome Extension)</h4><ul><li>A lightweight JavaScript content script (content.js) runs silently on every visited page.</li><li>It intelligently strips away irrelevant content — scripts, navigation elements, hidden or invisible text — to capture a clean, meaningful snapshot of page content.</li><li>Relevant metadata (URL, title, timestamp) is collected alongside this cleaned text.</li></ul><h4>Embedding &amp; Indexing (FastAPI + Ollama + FAISS)</h4><ul><li>The Chrome extension forwards the cleaned text chunks to your local FastAPI backend.</li><li>FastAPI routes these chunks into a locally-hosted Ollama instance running the high-performance mxbai-embed-large embedding model, rapidly transforming text into semantic embeddings.</li><li>Generated embeddings are stored directly into a FAISS index — built for blazing-fast semantic similarity searches.</li><li>Lightweight metadata (URLs, titles, timestamps) is locally cached in Chrome’s storage for easy management and retrieval.</li></ul><h4>Semantic Query &amp; Highlight (Popup + Highlight Script)</h4><ul><li>Enter a natural-language query (e.g., <em>“Show me the article about vector search”</em>).</li><li>FastAPI rapidly queries the FAISS index, returning top semantic matches.</li><li>Clicking on a result opens the original webpage and precisely highlights the relevant text — instantly reconnecting you to the original context.</li></ul><p>This modular architecture keeps every step — data extraction, embedding, indexing, querying, and highlighting — completely local on your device. As a result, you get unmatched speed, uncompromised privacy, and a deeply intuitive browsing experience.</p><p><strong>Key Implementation Details</strong></p><p>Let’s dive into some aspects of turning this powerful concept into reality, complete with actionable code snippets.</p><h4>Capturing the Web Without Noise</h4><p>To ensure accurate semantic search, you first need to reliably extract meaningful content:</p><ul><li>A Chrome content script (content.js) clones the page’s DOM.</li><li>It filters out unnecessary noise — scripts, navigation menus, footers, and hidden elements — using predefined exclusion rules.</li><li>Only visible and meaningful text nodes remain, resulting in clean text data.</li></ul><h4>Perfectly Sized Embeddings (Chunking &amp; Vectorizing)</h4><p>To maintain meaningful context within model constraints:</p><ul><li>Text is segmented into semantic chunks based on sentence boundaries, carefully balancing context and embedding model window limits (approximately 300 tokens per 500 characters).</li></ul><h4>Fast Semantic Recall (FAISS Indexing &amp; Search)</h4><p>FastAPI serves as your semantic search orchestrator:</p><ul><li>Text chunks become embeddings via Ollama’s local mxbai-embed-large model.</li><li>Embeddings are stored efficiently in a FAISS index (IndexFlatL2) for rapid semantic retrieval.</li></ul><pre>@app.post(&quot;/index&quot;)<br>def index_page(payload: IndexPayload):<br>    for chunk in payload.chunks:<br>        vec = embed_model.embed_text(chunk)<br>        index.add(np.array([vec], dtype=&quot;float32&quot;))<br>        metadata.append((payload.url, chunk))<br>    return {&quot;added&quot;: len(payload.chunks)}<br><br>@app.post(&quot;/search&quot;)<br>def search(req: SearchRequest):<br>    qv = embed_model.embed_text(req.query)<br>    D, I = index.search(np.array([qv], dtype=&quot;float32&quot;), req.k)<br>    return {<br>      &quot;results&quot;: [<br>        {&quot;url&quot;: metadata[i][0], &quot;text&quot;: metadata[i][1], &quot;score&quot;: 1 - float(d)}<br>        for d, i in zip(D[0], I[0])<br>      ]<br>    }</pre><p><strong>Index Persistence: </strong>FAISS index state is saved (faiss.write_index) and restored (faiss.read_index) seamlessly to preserve your browsing memory across sessions.</p><h4>Instant Highlights</h4><p>When viewing results:</p><ul><li>Clicking a result injects JavaScript directly onto the target page, precisely highlighting your retrieved snippet.</li><li>It gracefully handles exact and fuzzy text matches, ensuring robust highlights even on dynamic or responsive web pages.</li></ul><h3>Keeping It Local (Why It Matters)</h3><p>Running your entire semantic browsing engine locally might seem unconventional — but it comes with powerful advantages:</p><p><strong>Absolute Privacy: </strong>Your browsing data, content embeddings, and search queries never leave your device. No third-party can track or analyze your insights.</p><p><strong>Instant Performance: </strong>By hosting the Ollama embedding model and FAISS index locally, semantic searches execute at fast speeds.</p><p><strong>Uninterrupted Offline Access: </strong>Once your pages are indexed, semantic queries continue to work seamlessly, even without an internet connection.</p><p><strong>Cost Efficiency &amp; Control: </strong>Forget API rate limits, subscription fees, or surprise charges. Your only investment is your own hardware.</p><p>Running a minimal stack — Chrome, FastAPI, Ollama, and FAISS — right on your own machine ensures your browsing history becomes a personal, powerful, and completely private semantic knowledge base.</p><h3>Lessons Learned</h3><p>Here’s what I discovered while transforming my browser into a semantic powerhouse:</p><p><strong>Surprisingly Easy: Ollama</strong></p><p>Setting up Ollama locally to run mxbai-embed-large embeddings on CPU was astonishingly smooth. Embeddings were generated in milliseconds.</p><p><strong>Surprisingly Challenging: Reliable On-Page Highlights</strong></p><p>Implementing accurate highlighting was unexpectedly tough. Exact text matches required heavy text normalization and regex-escaping, yet dynamic page layouts or content reflows regularly broke highlights. Experiments with tracking chunk positions using character offsets were derailed by responsive designs..</p><h4><strong>What I’d Improve Next Time</strong></h4><p><strong>Smarter Semantic Chunking: </strong>Simple sentence-splitting works but leaves room for improvement. In the next iteration, I plan to explore:</p><ul><li><strong>Topic-driven segmentation</strong>, clustering semantically related sentences.</li><li><strong>Overlapping sliding windows</strong>, preserving better context at chunk boundaries.</li><li><strong>Adaptive chunk sizing</strong>, based on embedding token counts instead of fixed character limits.</li></ul><h3>Final Thoughts</h3><p>Instead of wasting time on endless bookmark searches or vague keyword guessing, I now effortlessly reconnect with key insights in seconds.</p><p>With just a few hundred lines of intuitive code — and leveraging Chrome, FastAPI, Ollama, and FAISS — I’ve built a solution that’s fully local, fast, and privacy-first.</p><p>Check out the full source code <a href="https://github.com/Sij-Agentic/web-rag-search">here on GitHub</a>, give it a spin, and feel free to open an issue or drop a comment — I’d love to hear your feedback, insights, and suggestions for improvements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa20e37aaa7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Machines Are Ready. We Are Not.]]></title>
            <link>https://sijpapi.medium.com/the-machines-are-ready-we-are-not-ce2e32068f61?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce2e32068f61</guid>
            <category><![CDATA[science-fiction]]></category>
            <category><![CDATA[humanity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[future-of-work]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Sun, 27 Apr 2025 13:29:32 GMT</pubDate>
            <atom:updated>2026-07-15T11:36:41.286Z</atom:updated>
            <content:encoded><![CDATA[<p>Technology Has Opened the Door. Human Nature May Keep It Shut.</p><h3>The Vision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Q15R6a2Ue0xLAaQq" /></figure><p>We are building machines that could, in theory, do most of what we currently call work. AI systems managing logistics, diagnosing illness, accelerating research. Robots growing food, assembling goods. While this isn’t perfect yet, the direction is clear.</p><p>If it gets there, labor (the thing most of us trade our waking hours for) becomes optional. Time opens up. People could learn, make things, be with each other, simply because they want to. Not because rent demands it.</p><p>I was watching <a href="https://en.wikipedia.org/wiki/Pantheon_(TV_series)"><em>Pantheon</em></a>, an animated tv show about human minds uploaded into digital systems, free of bodies, free of death. here, we’re building toward something not quite as strange, much simpler, and requires no upload. Just automation, AI, and robotics doing what hunger used to force us to do.</p><blockquote>Energy is abundant.<br>Scarcity is obsolete.<br>Work, in the traditional sense, is a thing of the past.</blockquote><p>The promise is tantalizing. A world where every individual is free to become more fully themselves, and no longer constrained just by survival.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*41wEGF-2oiSLoiJn" /></figure><p>But technology is only half the story…</p><h3><strong>The first problem is economic.</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*brOLgQVNMSx8zriG" /></figure><p>Abundance does not mean access. The same systems that concentrate wealth today will concentrate the benefits of automation tomorrow.</p><blockquote>Capitalism was not designed for distribution, it instead rewards accumulation. And automation, left to that same logic, will concentrate its benefits upward.</blockquote><p>We can see the pattern already. A small number of companies are racing to own the models. When machines replace labor at scale, the wealth they generate flows to whoever owns the infrastructure. That is not a side effect, but the system working exactly as designed.</p><p>The people most displaced by automation are the least likely to own a share of it. The billionaires funding these systems are not building toward collective liberation but towards leverage. More of it, faster, with fewer human dependencies getting in the way.</p><p>The machines may be ready. But who will they work for?</p><h3><strong>The second problem is psychological…</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sS5KQidCTkAdB7Qk" /></figure><p>Work is not just how most people earn money. It is how they explain themselves. What they answer when someone asks what they do. We already have an idea of what happens when that disappears. Retirement, for some people brings disorientation instead of peace.</p><p>Factory towns that lose their industry don’t just lose income, but often times their identity. People who defined themselves by what they built, or made, or did, and suddenly no longer can.</p><p>When people hear this, many conclude that humans simply need work. That we’re wired for it. I’m not sure that’s true.</p><p>I think a lot of us are unable to imagine free time because we’ve never actually had it. Not real free time. Not two weeks of holiday with a return flight already booked and unread emails waiting. I mean waking up with nowhere to be, no output expected, no performance to maintain, and genuinely choosing what to do with your life.</p><p>Most of us have been inside a system since childhood that told us our worth is tied to our productivity. When you spend decades measuring yourself that way, “what would you do if you didn’t have to work?” stops being an exciting question. It becomes an unsettling one. The blank you draw isn’t human nature. It’s conditioning.</p><p>But the question of meaning is still real. Free time alone doesn’t answer it. The goal was never just leisure.</p><p>It’s agency.</p><p>Spend the morning with your children. Learn something new. Build something, go somewhere, be somewhere fully. The tragedy of work-as-identity isn’t that people love what they do. It’s that most of us were never given the space to discover what else we might love.</p><p>A post-labor future asks a question that no machine can answer:</p><blockquote><em>who are you, when you are no longer what you do?</em></blockquote><p>Without a real answer, abundance produces its own kind of crisis.</p><p>That’s a frightening question if you’ve never been allowed to ask it. It’s an exciting one if you have.</p><p>History has examples. According to Nietzsche, remove the structures that organized life for centuries, and something has to fill the void. It might be new ways of living.</p><blockquote>It might be nihilism.</blockquote><h3><strong>The third problem is political.</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9ucKAYAa50a_BwVs" /></figure><p>Governmental institutions are built to manage scarcity, distribute resources, regulate access, and justify their authority by administering need. When survival becomes effortless, when there is nothing left to ration, those institutions face an existential question they were never designed to answer.</p><p>History suggests they will not answer it gracefully. We have seen governments prop up dying industries not because those industries make sense, but because they preserve familiar patterns of dependency. Technologies delayed to protect the existing order and delay the inevitable.</p><p>And some governments aren’t waiting to be disrupted. They’re running toward AI for reasons that have nothing to do with humanity flourishing. China’s social infrastructure already uses AI to monitor behavior. Western governments are investing heavily in predictive surveillance and weapons targeting systems. The technology that could liberate people is also, in the wrong hands, extraordinarily good at controlling them. This is not a conspiracy theory, it is already happening.</p><p>The governing instinct, when threatened, is to hold on to power, and not necessarily to adapt. A population that no longer <em>needs</em> to be managed will still need to be governed somehow. Without a genuine rethinking of what power is for, the machinery of liberation could end up enforcing the status quo, but by different means.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o0_He4PwobWIMdD1" /></figure><p>The obstacles ahead are not engineering problems. The technology continues to advance.</p><p>The harder questions are about who controls what it produces, whether people can find meaning when survival is no longer the thing organizing their days, and whether political systems can change faster than they historically do.</p><p>None of those questions have easy or obvious answers. But they are the right questions. Which is more than most conversations about AI tend to ask.</p><p>The same technology that could entrench inequality could also dismantle it. The same automation that could displace people could free them. The difference lies entirely in choices we haven’t made yet, and whether we’re willing to make them.</p><p>Whenever I speak with people who are skeptical of AI, I usually find that their concerns aren’t actually rooted in the technology itself. Instead, they are reacting to the systems behind it. AI didn’t invent exploitation, inequality, or greed. Capitalism and human nature did.</p><p>The machines are ready.</p><p>We are still figuring out if we are.</p><h3><strong>References</strong></h3><ul><li>Frankl, Viktor E. <em>Man’s Search for Meaning.</em> Beacon Press, 2006.</li><li>Piketty, Thomas. <em>Capital in the Twenty-First Century.</em> Harvard University Press, 2014.</li><li>World Economic Forum. <em>The Future of Jobs Report 2023.</em></li><li>World Health Organization. <em>COVID-19 Vaccine Access and Equity: 2021 Review.</em></li><li>GiveDirectly. Early findings from the world’s largest UBI study 2023</li><li>philanthropy.com. OpenAI Looks to Shift Away From Nonprofit Roots and Convert Itself to For-Profit Company 2024</li><li>Friedrich Nietzsche The Joyful Wisdom, 1882.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce2e32068f61" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Social CSR (Corporate Social Responsibility) Benefits Businesses and Communities Alike]]></title>
            <link>https://medium.com/the-gamesville-foundation/social-csr-corporate-social-responsibility-benefits-businesses-and-communities-alike-bfb1474c7cc9?source=rss-deb787286fa9------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfb1474c7cc9</guid>
            <category><![CDATA[charity]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[philanthropy]]></category>
            <category><![CDATA[csr]]></category>
            <category><![CDATA[charitable-giving]]></category>
            <dc:creator><![CDATA[Sijuade Oguntayo]]></dc:creator>
            <pubDate>Thu, 30 Jan 2025 10:39:38 GMT</pubDate>
            <atom:updated>2025-02-03T21:04:46.196Z</atom:updated>
            <content:encoded><![CDATA[<p>In recent times, businesses are increasingly recognized not just for their economic contributions but also for their role in fostering societal well-being. Companies that embrace Corporate Social Responsibility (CSR) focused on social initiatives are not only enhancing their reputations but also paving the way for a brighter, more equitable future.</p><h3>What is Corporate Social Responsibility (CSR)?</h3><p>Corporate Social Responsibility (CSR) is a self-regulating business model that helps companies be socially accountable — to themselves, their stakeholders, and the public. By practicing CSR, organizations ensure their operations are sustainable and ethical, addressing social, economic, and environmental impacts.</p><h3>Benefits of Social CSR for Organizations</h3><p>Engaging in social CSR initiatives offers numerous advantages for businesses:</p><ul><li><strong>Enhanced Brand Reputation:</strong> Demonstrating a genuine commitment to social issues builds trust and loyalty among consumers. For instance, companies known for their educational support programs are often seen as compassionate and responsible.</li><li><strong>Customer Loyalty and Trust:</strong> Consumers are more likely to support brands that align with their personal values. When a company invests in child education, it resonates deeply with families and communities.</li><li><strong>Attracting and Retaining Talent:</strong> Employees take pride in working for organizations that prioritize social good, leading to higher job satisfaction and lower turnover rates.</li><li><strong>Positive Workplace Culture:</strong> Social CSR fosters a sense of purpose and community within the organization, boosting overall morale and productivity.</li><li><strong>Networking and Partnerships:</strong> Engaging in social initiatives opens doors to collaborations with non-profits and other socially-minded organizations, expanding a company’s network and impact.</li></ul><h3>Why Social CSR Matters Today</h3><p>The importance of social CSR has never been greater. Here’s why:</p><ul><li><strong>Changing Consumer Expectations:</strong> Modern consumers expect businesses to contribute positively to society. Addressing child educational hardship is a tangible way to meet these expectations.</li><li><strong>Social Media Influence:</strong> In an age where information spreads rapidly, positive social initiatives can significantly enhance a company’s image, while neglecting them can lead to public criticism.</li><li><strong>Global Social Challenges:</strong> Issues like educational disparities, especially exacerbated by events like the COVID-19 pandemic, require urgent and sustained attention from the corporate sector.</li><li><strong>Regulatory and Compliance Pressures:</strong> Governments and regulatory bodies are increasingly mandating social responsibility practices, making it essential for businesses to integrate CSR into their core strategies.</li></ul><h3>The Gamesville Foundation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ymt1iq0jnTgcmAKW2gXinA.jpeg" /><figcaption>Echoes of Mercy and Hope Foundation — Ayo Tournament</figcaption></figure><p>Established in 2017 and based in Lagos, Nigeria, the Gamesville Foundation is dedicated to providing recreation and aid to individuals affected by hardship, old age, loneliness, or disability.</p><p><strong>Mission and Objectives:</strong></p><ul><li><strong>Hardship Relief:</strong> Offering support and recreational activities to individuals facing difficult times, fostering a sense of community and empowerment.</li><li><strong>Elderly Support:</strong> Enriching the lives of older adults through activities and social engagement to combat isolation.</li><li><strong>Combating Loneliness:</strong> Creating spaces for individuals to form meaningful connections and alleviate feelings of loneliness.</li><li><strong>Disability Inclusion:</strong> Providing inclusive recreational activities and support for individuals with disabilities, promoting self-expression and personal growth.</li></ul><p><strong>Partnerships:</strong></p><p>The Gamesville Foundation collaborates with several reputable non-profit organizations to extend its reach and effectiveness:</p><ul><li><strong>Echoes of Mercy and Hope Foundation (EMHF):</strong> A Christian residential home offering shelter, rehabilitation, and education for neglected and abused young girls aged 13–21.</li><li><strong>JAKIN NGO:</strong> Provides health, education, and entrepreneurship support to vulnerable groups, notably through the JAKIN Children’s Club, serving around 500 disadvantaged children monthly.</li><li><strong>Street Child Care and Welfare Initiative (SCCWI):</strong> Rescues street children and provides shelter, rehabilitation, education, and empowerment programs.</li><li><strong>Diocesan Special Education Center (DSEC):</strong> Offers quality education and support for children with special needs, including Autism and Down syndrome.</li><li><strong>Society for the Empowerment of Young Persons (SEYP):</strong> Focuses on reducing poverty, violence, discrimination, and abuse among women, youths, and children, emphasizing the implementation of the Child Rights Law.</li></ul><p><strong>Impact and Initiatives:</strong></p><p>The foundation has undertaken several impactful initiatives that align perfectly with social CSR goals:</p><ul><li><strong>Educational Support:</strong> During the COVID-19 pandemic in 2020, the foundation equipped 750 children with school supplies and transistor radios to facilitate remote learning during lockdowns.</li><li><strong>Scholarships:</strong> In 2022, Gamesville raised ₦7.6 million in scholarships from a corporate sponsor to support 18 underprivileged children. The previous year, they secured ₦3.5 million for ten children across three charitable organizations.</li><li><strong>Community Engagement:</strong> Supporting programs like JAKIN NGO’s annual “Dress a Child for School” project, Gamesville provided 1,000 school sandals and other essentials to children, ensuring they are prepared for the school year.</li></ul><h3>How Your Organization Can Make a Difference</h3><p>Partnering with the Gamesville Foundation offers organizations a meaningful way to enhance their social CSR initiatives. By supporting our programs, businesses can directly contribute to alleviating educational hardships faced by children, fostering a more educated and empowered future generation.</p><p><strong>Ready to Make a Difference?</strong></p><p>If your organization is looking to create a tangible impact in the community, consider collaborating with the Gamesville Foundation. Together, we can leverage the power of play and education to support those in need.</p><p><strong>Contact Us:</strong> Contact <a href="http://www.thegamesvillefoundation.com/">the Gamesville Foundation</a> @ thegamesvillefoundation@gmail.com to explore partnership opportunities and learn how your organization can contribute to meaningful social change.</p><h3>Conclusion</h3><p>Embracing social CSR, particularly in addressing child educational hardship, not only enhances your organization’s reputation and fosters a positive workplace culture but also plays a crucial role in building stronger, more resilient communities. By partnering with organizations like the Gamesville Foundation, businesses can ensure their CSR efforts lead to lasting and meaningful impact. Together, we can create a brighter, more inclusive future for all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfb1474c7cc9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-gamesville-foundation/social-csr-corporate-social-responsibility-benefits-businesses-and-communities-alike-bfb1474c7cc9">Social CSR (Corporate Social Responsibility) Benefits Businesses and Communities Alike</a> was originally published in <a href="https://medium.com/the-gamesville-foundation">The Gamesville Foundation</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>