<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dmitry Ziolkovskiy on Medium]]></title>
        <description><![CDATA[Stories by Dmitry Ziolkovskiy on Medium]]></description>
        <link>https://medium.com/@ziodmitry?source=rss-d7d59eefacd9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*hoz4bJStDWyuaN1M.</url>
            <title>Stories by Dmitry Ziolkovskiy on Medium</title>
            <link>https://medium.com/@ziodmitry?source=rss-d7d59eefacd9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 17:34:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ziodmitry/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[OCSR with foundation models]]></title>
            <link>https://medium.com/@ziodmitry/ocsr-with-foundation-models-d66a5e9365c8?source=rss-d7d59eefacd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/d66a5e9365c8</guid>
            <category><![CDATA[chemical-structure-search]]></category>
            <category><![CDATA[generative-ai-in-chemical]]></category>
            <category><![CDATA[chemistry]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Dmitry Ziolkovskiy]]></dc:creator>
            <pubDate>Tue, 06 Jan 2026 13:01:31 GMT</pubDate>
            <atom:updated>2026-01-06T13:01:31.720Z</atom:updated>
            <content:encoded><![CDATA[<p>Just as OCR (optical character recognition) is important in business processes and everyday life, OCSR (optical chemical structure recognition) is important in chemical science.</p><p>As a simple example, consider the large body of chemistry articles — many published in the pre‑internet era — that we would like to make searchable and, once found, to ingest into a working database for downstream processing. Since mid‑2025, it is reasonable to say that the core OCSR task is largely solved: <a href="https://arxiv.org/html/2505.03777v2">LG AI Research described</a> MolMole, a unified framework for molecule detection, reaction diagram parsing, and OCSR, which reports nearly perfect results. Unfortunately, it is a commercial tool available only by inquiry, which severely limits its use. The authors also <a href="https://huggingface.co/papers/2505.03777">published</a> an evaluation dataset of 300 annotated patent pages, making it straightforward to compare other tools to their framework. Another notable work from that time is the open‑source <a href="https://link.springer.com/article/10.1007/s11227-025-07406-4">OCSAug</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rKpoVBbStE6le4UMd5Y6_Q.png" /></figure><p>Reliable tools for OCSR therefore exist and can support conventional, human‑centric workflows. However, that is only one side of the coin. In 2025, AI assistants became powerful and ubiquitous enough that scientists increasingly delegate parts of their work to “chats” or agents: search, idea consultation, data analysis, and more. If we do so, we need to be confident that the underlying models actually “understand” the source data — in chemistry that means reading and interpreting structures, reactions, plots, spectra, and everything else in the articles. More broadly, the ability to “comprehensively read articles” and interpret technical drawings is crucial for further progress toward AGI and singularity. If a foundation model (the generic one behind the “chats” people use) could take a scanned old article, infer a reaction pathway, and attribute a small shift in an IR peak to bond shortening in the product, that would be extremely powerful. Are we far from that magic, or do we already have it?</p><p>Comprehensive assessments of multimodal abilities are tedious and have been attempted by researchers (e.g., <a href="https://arxiv.org/html/2407.03418v1">Liang et al.</a>). In this “weekend research”, the goal is to qualitatively answer just a few questions:</p><ul><li>Are current foundation models suitable for OCSR at all?</li><li>Do models understand structures as molecular graphs (not just as symbols and connections)?</li><li>Can they operate on molecular graphs in their “thinking”?</li><li>Can we automate research and let AI answer these questions itself?</li></ul><blockquote><em>This is an adaptation of the </em><a href="https://github.com/dmzio/ocsr_eval/blob/main/report/report.md"><em>original report</em></a><em> from the GitHub repository. Refer to </em><a href="https://github.com/dmzio/ocsr_eval"><em>https://github.com/dmzio/ocsr_eval</em></a><em> for any additional details and code for reproduction/extension of research.</em></blockquote><h4>Note on molecular representations</h4><p>In chemoinformatics, SMILES is the most common string representation for structural formulas. It is compact and human‑readable, with simple rules. For ML models — and LLMs in particular — SMILES are not trivial: LLM tokenizers are optimized for natural language, and SMILES tokenization is often suboptimal, wasting context. SMILES is also fragile, and a large fraction of generated strings do not correspond to valid molecules.</p><p>To address validity, <a href="https://arxiv.org/abs/1905.13741">SELFIES</a> was proposed: every SELFIES string corresponds to a valid molecule, and SELFIES can represent any molecule. However, it is less widely used and has drawbacks. In OCSR, for example, a single misread character may still yield a valid SELFIES string that encodes the wrong molecule.</p><p>For any string representation, generating it during OCSR requires multiple steps: locating the structure in the image, recognizing atoms and bonds, assigning charges or isotopes, and only then generating a string from an internal graph representation. For foundation models, this is a complex multi‑step process. It may be simpler to request a direct graph representation (atoms/bonds JSON) similar to an RDKit Mol, eliminating the need for graph‑to‑string conversion. The drawback is that graph outputs are large and can be fragile in their own way.</p><h4>Methodology</h4><p>This is intentionally “weekend research”: it is designed to take limited time and effort. Manual work is limited to setting the initial idea for the project, steering a few decisions, and writing this final report. Most of the remaining work — background reading, creating code to execute evaluations, running experiments, and drafting an initial analysis — was delegated to an OpenAI Codex agent powered by the GPT‑5.2 model (with minor contributions from other tools such as ChatGPT).</p><p>To answer the questions above, we defined the following pipeline:</p><ul><li>Collect the MolMole dataset from the official source.</li><li>Query a set of vision‑enabled foundation models with a subset of images from the MolMole dataset. In a simple prompt (created and partially optimized by Codex), we ask the model to find and extract molecules as (i) graph JSON (atoms/bonds), (ii) SMILES, or (iii) SELFIES (in separate runs).</li><li>Convert extracted data from other formats into SMILES where possible (for successfully processed pages).</li><li>Evaluate extraction metrics.</li></ul><p>Metrics represent different levels of molecular‑graph extraction capability: from detecting atoms and bonds in a molecule (via Tanimoto similarity over Morgan fingerprints) to converting a molecular graph into a canonical SMILES string (see <a href="https://github.com/dmzio/ocsr_eval/tree/main/src/molmole_research">code</a> for details).</p><p>We deliberately run extraction “naively”, in one shot. We do not detect molecule bounding boxes and handle those areas separately (which would greatly simplify the task and improve the metrics). The goal is to test whether models can see the “whole picture”.</p><p>The repository includes code to reproduce or extend the experiments. The current implementation calls models via OpenRouter, but any OpenAI‑compatible service is supported via configuration.</p><h3>Experimental Scope and Models</h3><p>Experiments evaluate multimodal models as OCSR engines on a small subset of MolMole_Patent300. Most runs process just 5 pages (as reflected in pages_seen), to obtain qualitative results.</p><p>Model set (by provider/family):</p><ul><li>Google: gemini-2.5-flash-lite, gemini-3-flash-preview, gemini-3-pro-preview, gemma-3-27b-it</li><li>OpenAI: gpt-5-nano, gpt-5.2</li><li>Anthropic: claude-haiku-4.5, claude-sonnet-4.5</li><li>Mistral: ministral-14b-2512, mistral-small-3.2-24b-instruct</li><li>Qwen: qwen3-vl-235b-a22b-thinking; plus qwen2.5-vl-72b-instruct (graph only)</li><li>NVIDIA: nemotron-nano-12b-v2-vl</li><li>ByteDance: seed-1.6</li><li>Z.ai: glm_4.6v (graph only)</li></ul><h3>Evaluation Mechanics</h3><p>Metrics are computed on the subset of pages with valid prediction records (pages_valid). Ground-truth structures and reactions are indexed only for these pages; as a result, structures_total and reactions_total vary across runs depending on which pages were successfully processed.</p><p>Two validity layers determine what is scored:</p><ol><li><strong>Page-level validity</strong>: a page is invalid if the prediction record contains an API error or cannot be parsed into the expected JSON structure. Invalid pages contribute to pages_invalid and are excluded from structure-level evaluation.</li><li><strong>Structure-level evaluation</strong>: within valid pages, each predicted structure may contribute to one or more metrics:</li></ol><ul><li><strong>SMILES / SELFIES metrics</strong>: exact-match accuracy (accuracy_smiles, accuracy_inchi) and similarity (tanimoto_mean) require a canonical SMILES. Predictions that cannot be canonicalized are counted as invalid_predictions; ground-truth structure IDs without a corresponding prediction are counted as missing_predictions.</li><li><strong>Graph metrics</strong>: each predicted structure with an atoms/bonds payload is converted into an RDKit molecule and fingerprinted. Canonical SMILES is derived from the graph only when RDKit sanitization succeeds. Graph similarity is computed even when canonical SMILES derivation fails.</li></ul><p><strong>Graph similarity</strong> (graph_tanimoto_mean) is defined as follows:</p><ul><li>For each fingerprinted prediction on a page, compute its maximum Tanimoto similarity against the set of ground-truth structure fingerprints on the same page.</li><li>Report the mean of these per‑prediction best‑match scores (where graph_predictions_total is the number of predictions that contributed fingerprints).</li></ul><p>This metric captures whether predicted graphs resemble any true structure on the page, but it does not enforce a one‑to‑one assignment between predicted and ground‑truth structure IDs, and it does not directly penalize missed structures (those are reflected in missing_predictions).</p><h3>Results</h3><h4>Graph (Atoms/Bonds JSON) format</h4><p>This format produces large, schema‑sensitive outputs, but many models produce parseable JSON on at least a subset of pages. Page‑level failures are concentrated in a small number of cases (e.g., JSON schema violations, context‑length limits, or empty responses), whereas the dominant limitation for structure‑level correctness is chemical validity (via RDKit sanitization).</p><p>Several models extract graphs whose fingerprints show moderate similarity to ground truth (best mean best‑match Tanimoto ≈0.64), indicating that many predicted graphs have close matches to at least one annotated structure on the same page.</p><p>However, converting predicted graphs to canonical SMILES frequently fails due to RDKit sanitization. The dominant errors are aromaticity (kekulization) failures and explicit valence violations, so syntactically valid extracted graphs are often chemically incorrect.</p><p>Canonical SMILES accuracy derived from graph outputs remains low, reflecting both incomplete coverage and frequent sanitization failures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W-zBa3mohVAWwUqhJ5YPwQ.png" /></figure><p><strong>Examples of graph extraction failure modes</strong></p><ol><li><strong>Context-length / transport errors</strong> (page-level invalidation). Example: qwen_qwen2.5-vl-72b-instruct returns an API error indicating the request exceeded a 128k token context limit.</li><li><strong>Empty responses</strong> (page-level invalidation). Example: z_ai-glm_4.6v records contain empty response content leading to parse errors.</li><li><strong>Missing graph payload components</strong>. Example log: “structure[n] has no SMILES/SELFIES/graph payload”.</li><li><strong>Graph canonicalization failures</strong> (structure-level invalidity for SMILES metrics). Representative errors: “Can’t kekulize mol …”, “Explicit valence … greater than permitted”.</li><li><strong>Reaction extraction</strong> remains weak. A single non-zero result appears in the graph runs: anthropic_claude-sonnet-4.5 reports 1 correct reaction on 5 pages. All other graph runs report 0 correct reactions.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y0om3JAI_dyvw-3LKJPZdA.png" /><figcaption>Extracted graph santization bottleneck (chemical correctness)</figcaption></figure><h4>SELFIES format</h4><p>As expected given its lower popularity and syntactic features, SELFIES is a poor choice for molecular graph extraction.</p><p>SELFIES yields near‑zero exact‑match performance: the best observed accuracy_smiles is 0.03 (1 correct structure across 33 annotated structures) for the Claude model. Most extractions are invalid due to parse errors and malformed outputs; qualitatively, many outputs resemble those of a severely undertrained model.</p><p><strong>Notable Failure Modes</strong></p><ol><li><strong>Invalid ring/branch grammar</strong> (undecodable strings). Example invalid SELFIES: [O][=C][Branch1][C][O][C][=N][NH]…[Ring1][=Branch1][Ring1][=Branch2]. SELFIES is grammar‑constrained; incorrect ring/branch tokens make the entire string undecodable. These errors are common even when the token sequence “looks chemical”.</li><li><strong>Degenerate repetition (“collapse”)</strong> into long trivial chains. Example pattern (raw output): extremely long sequences dominated by [C] tokens. This resembles a failure mode where the model emits a high‑probability token repeatedly rather than a structured molecule. The result is chemically meaningless and often too long or ill‑formed for clean JSON completion.</li><li><strong>Malformed escaping and partial JSON</strong> (parse errors that invalidate the whole page record). Example pattern: broken escaping within SELFIES tokens (backslashes mid-token).</li></ol><p>Only a few models were able to provide even somewhat valid SELFIES outputs.</p><h4>SMILES format</h4><p>SMILES provides the highest exact‑match accuracy among the three formats, with the best observed accuracy_smiles≈0.20 on 5 pages.</p><p>Page-level validity is relatively good due to the popularity of the format and its simplicity. Structure‑level validity remains a major limiter: a large portion of “Invalid SMILES” events are logged across the SMILES evaluations.</p><p>Claude and GPT models often try to assign chemical names to structures, either by reading surrounding text or by analyzing the structure itself. Gemini models mostly do not include names in the notes with the current prompt.</p><p>Reaction extraction is ineffective in this format: the sum of reactions_correct across SMILES runs is 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyKVn-dzPG3vwjVoaj7Isw.png" /><figcaption>Models with non-zero tanimoto_mean, sorted by accuracy_smiles then tanimoto_mean</figcaption></figure><p><strong>Notable Failure Modes</strong></p><ol><li><strong>Invalid SMILES tokens / shorthand</strong> (RDKit rejects informal chemistry abbreviations). Example invalid string: MeOc1cc(F)(c2cc(F)c(F)c2Nc1O)O</li><li><strong>Aromaticity and ring-closure errors</strong> (single-token issues invalidate the whole structure). Example invalid string (log-reported): COC(=O)CC(C3=CC=CC=C3C4=CC=CC=C4S(=O)(=O)C1=CC(=O)C2=CC(CC)C=CC=C2C3=CC=CC=C3)C</li><li><strong>Abstentions</strong> (empty SMILES with explanatory notes), which effectively become missing or invalid predictions. Example pattern: {&quot;smiles&quot;: &quot;&quot;, &quot;notes&quot;: &quot;… unreadable at this resolution&quot;}</li><li><strong>Malformed or truncated JSON</strong> (page-level invalidation even if the beginning of the response looks correct). Example pattern: very long enumerations with repeated &quot;notes&quot;: &quot;&quot; and missing closing brackets.</li><li><strong>Reaction schema errors</strong> (present in outputs but rejected by validator). Examples: “missing reactants/products”, “Reaction conditions must be a list of strings”.</li></ol><h4>Observations by Model Family and Model Size</h4><p>Gemini models are the most consistently useful for SMILES extraction and can also produce graph outputs that resemble annotated structures.</p><p>GPT‑5.2 often yields detailed graphs with high fingerprint similarity to ground truth but suffers from the same sanitization bottleneck and incomplete coverage as Gemini, while GPT‑5‑nano frequently returns empty structure lists despite producing valid JSON.</p><p>Anthropic models are generally stable at producing well‑formed JSON, are the only family to yield any correct SELFIES extractions in this sample, and are the only runs to register a correct reaction edge.</p><p>Mistral models show a distinctive SELFIES failure mode — very long [C]‑dominated repetitive strings — often accompanied by parse errors; their graph outputs are plentiful but often chemically inconsistent.</p><p>ByteDance Seed is comparatively stable in producing structured outputs with broad coverage, but still struggles to produce chemically correct canonical strings.</p><h3>Conclusions</h3><p>Overall, the results support a narrow conclusion: contemporary multimodal foundation models can often localize and transcribe parts of molecular diagrams, but <strong>they do not consistently produce chemically valid molecular representations without additional constraints</strong>.</p><p>Graph outputs provide the clearest evidence of partial structural understanding: fingerprint similarity indicates that many predicted graphs resemble at least one annotated structure on the same page. However, canonicalization to chemically valid molecules often fails, indicating that extracted graphs frequently violate chemical constraints even when the JSON schema is satisfied.</p><p>Top-tier “thinking” models sometimes transform recognized structures into valid SMILES strings and may attempt to assign chemical names, but performance is still too low to be usable as‑is. In the current state, these models are best treated as components in multi‑step OCSR systems and, as this vibe‑coded experiment demonstrates, such systems can be built mostly by AI agents themselves.</p><p>Most importantly, although vision‑enabled models often exceed human‑level performance on everyday tasks, recognition of scientific literature remains far behind. <strong>When using general AI‑enabled search and analysis systems, we still cannot rely on them to understand the chemical world as humans do: they handle text well, but chemical understanding remains a key limitation.</strong></p><p>Can this be improved? Definitely. Recent frontier models already perform much better than their predecessors, and deeper multimodal training in future generations could bring a real “enlightenment” for these systems.</p><p>Use of AI agents for research automation right now? Definitely — with properly crafted tools we can focus on idea generation, research supervision, and final analysis and let the rest be done by machines.</p><h3>Links and notes</h3><ul><li>GitHub repo (code, prompts, original report): <a href="https://github.com/dmzio/ocsr_eval">https://github.com/dmzio/ocsr_eval</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d66a5e9365c8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Few notes on AGI: what, when, how and what to do?]]></title>
            <link>https://medium.com/@ziodmitry/few-notes-on-agi-what-when-how-and-what-to-do-3a0a3d41a687?source=rss-d7d59eefacd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a0a3d41a687</guid>
            <category><![CDATA[agi]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[singularity]]></category>
            <category><![CDATA[social-change]]></category>
            <category><![CDATA[progress]]></category>
            <dc:creator><![CDATA[Dmitry Ziolkovskiy]]></dc:creator>
            <pubDate>Sat, 04 May 2024 16:52:04 GMT</pubDate>
            <atom:updated>2024-05-05T11:38:57.644Z</atom:updated>
            <content:encoded><![CDATA[<p>When generative AI first gained widespread popularity about a year ago, <a href="https://www.linkedin.com/posts/dmzio_planning-for-agi-and-beyond-activity-7035991228915548160-9x_H">I expressed concerns</a> about potential monopolization by corporations and governments, leading to a dystopian future with ultimate control and limited freedom. Since then, we’ve witnessed significant progress in all aspects of AI development. Discussions about safety, restrictions, and development are ongoing, but the first regulations (like the <a href="https://artificialintelligenceact.eu/">EU AI Act</a>) appear reasonable and well-considered. Fortunately, open-source AI is not only alive and well, but continues to evolve thanks to Meta, Mistral, Stability AI, and a vast and diverse community. Every day, we see more capable models and innovative applications emerge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HUDkeYd6X7TRr7gz2XdzA.jpeg" /></figure><p>This rapid progress fosters excitement as well as anxiety. Like many others working with related technologies, I experience a mix of apprehension and FOMO (fear of missing out). On the other side, for some reason, I’m waiting eagerly for the arrival of true technological singularity. This note is the attempt to summarize (not so) random observations / thoughts on what and when we may expect on the road to singularity and what to do with all that stuff.</p><p><strong>What is AGI (once again)?</strong></p><p>Last couple months we hear that AGI is right about to come. So firstly let’s define AGI — Artificial General Intelligence for ourselves. We can <a href="https://en.wikipedia.org/wiki/Artificial_general_intelligence">agree with wikipedia</a> here that AGI is a type of artificial intelligence that can perform as well or better than humans on a wide range of cognitive tasks. Also Wiki provides a set of criteria, which system should meet to be an AGI. I would add ‘<em>ability to limitless selfimprovent</em>’ to that set (<a href="https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion">some</a> refer to such AI as SeedAI, leading to ), and I believe that we (corporations + open source) can create such a system in <strong>up to two years from now</strong> (a bit more optimistic than <a href="https://openai.com/blog/governance-of-superintelligence">OpenAI forecast from year ago</a>).</p><p><strong>What do we need to create an AGI?</strong></p><p>So, what do we need to create AGI (also referred to as superintelligence)? Is there a critical missing component? Often, news headlines sensationalize the “sentience” of chatbots or models as a scary breakthrough. Do current models truly possess sentience? Here again, we must refer to philosophical definitions of consciousness, a topic with no clear consensus. A simplistic view equates it with “awareness” or “self-awareness.” Current LLM chatbots might be considered self-aware because they are trained to answer questions about themselves and their nature. However, this has no bearing on their other capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L6OudfePK3EmxqV7EScQYQ.jpeg" /></figure><p>Reasoning is a crucial emergent ability that LLMs already possess to some degree. It’s more important for successful cognitive processes. Understanding sits atop reasoning, and in this area, <a href="https://arxiv.org/abs/2311.00059">some research</a> suggests that current models are paradoxically far from the basic human level, although they are continuously improving.</p><p>Thinking, imagination, and will are also often mentioned as required components for superintelligence. These may seem challenging to achieve, but <a href="https://arxiv.org/pdf/2309.03409">take a deep breath</a> and think: Do you <em>genuinely</em> possess all of these qualities inherently? Perhaps most of them are simply a result of years of experience, similar to how LLMs learn. Your vast training data (life experiences) and a flexible brain have allowed you to develop your own world model, encompassing your self-awareness, reasoning, understanding, and even imagination and will. Some parts of your brain are clearly “overfitted” to specific tasks, like navigating your home environment or performing frequent work routines — just as fine-tuned LLMs excel at answering specific questions.</p><p>Even the <strong>limitations of current AI systems share similarities with the sometimes overrated human mind.</strong> The “hallucinations” of generative models are even named after human medical symptoms. However, they are more akin to the behavior of people who lack information but want to contribute, often making up facts and stories — evident in gossip, false claims, conspiracy theories, and disinformation, all quite typical of “natural intelligence.”</p><p><strong>Lack of generalization or understanding abilities are also human characteristics.</strong> People with autism provide prime examples of “cognitive paradoxes,” where seemingly intelligent individuals struggle with some socially obvious tasks. We don’t even need to delve into other psychological conditions to find even closer parallels to current AI difficulties. Simply talking to people from different cultural backgrounds can reveal how seemingly “easy” reasoning and causality understanding can become nearly impossible.</p><p><strong>The Path to AGI</strong></p><p>Considering these definitions, benchmarks, and the current state of research, I argue that we already possess all the necessary components to build AGI. <strong>Algorithmically, there’s no need for a drastic breakthrough.</strong> What do we need then? Basically — effectively combine existing components and leverage appropriate hardware. Now it’s clear that AGI wouldn’t be a single “program”, it would be a <a href="https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/">compound system</a>, consisting of many components, which do some specific tasks.</p><p>Most natural way to combine components — use creative <a href="https://en.wikipedia.org/wiki/Collaborative_intelligence">collaborative agents</a>.</p><p>Make each of agent stable and performant — use Mixture of Experts (MoE) models on top of foundational and specialized (like <a href="https://huggingface.co/blog/starcoder">StarCoder</a>, <a href="https://github.com/deepseek-ai/DeepSeek-Math">DeepSeekMath</a>) models, use introspection and posprocessing (like <a href="https://blog.langchain.dev/agentic-rag-with-langgraph/">self-reflective RAG</a> or other approach, new variations appear every week). Improve training schemas for models to achieve reasoning beyond human level (f.e. ‘<a href="https://arxiv.org/pdf/2402.08939.pdf">reshuffled reasoning</a>’ and <a href="https://www.topbots.com/llm-reasoning-research-papers/">other</a>).</p><p><a href="https://research.aimultiple.com/ai-chip-makers/">Recent announcements</a> point towards the imminent release of specialized processors and memory specifically designed for AI applications. This signifies that launching large-scale AI systems will soon be within reach for even mid-sized corporations.</p><p><strong>Will AGI Answer Every Question?</strong></p><p>No, of course not. AGI, like any intelligent being, requires access to source information to formulate answers based on experimentation and facts (unless you desire “hallucinations” similar to those produced by current models). Remember, a well-defined question is crucial for a useful answer, regardless of who or what is providing it. Otherwise, you might end up with a response like “42” from “The Hitchhiker’s Guide to the Galaxy,” which is witty but ultimately meaningless without context.</p><p>However, AGI will be able to ask non-trivial questions itself and this curiosity will drive self development of it.</p><p><strong>The Singularity: limited and full</strong></p><p>Will AGI in this form trigger a technological singularity? Not entirely. The current pace of AI development can be described as a “virtual” or “limited” singularity, where exponential growth is primarily confined to the field of information technology.</p><p>To achieve wider-reaching effects, we need to expand the embodiment of AI systems. This involves developing models capable of controlling real-world robots, along with creating agile and efficient robots themselves that can operate in diverse environments. Fortunately, research in these areas is actively progressing. Google’s <a href="https://research.google/blog/palm-e-an-embodied-multimodal-language-model/">PaLM-E</a> model and recent <a href="https://gizmodo.com/robot-legit-elon-musk-fake-optimus-tesla-demo-astribot-1851455513">announcements</a> of humanoid robots, including those from Boston Dynamics or <a href="https://nvidianews.nvidia.com/news/foundation-model-isaac-robotics-platform">NVIDIA</a>, are prime examples.</p><p>The development of automated research and manufacturing equipment will truly usher in the age of singularity. Systems like <a href="https://www.nature.com/articles/s41586-023-06792-0.pdf">Coscientist</a> demonstrate initial steps towards this goal.</p><p><strong>Bumps on the Road</strong></p><p>Several factors could impede progress on our journey to AGI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0FWmeuq3rJzS74Dchfts6w.jpeg" /></figure><ul><li><strong>Widespread war conflicts:</strong> Especially those impacting major <a href="https://www.topbots.com/semiconductor-supply-chain-overview/">semiconductor manufacturers</a> or disrupting global communication networks.</li><li><strong>Government restrictions:</strong> Particularly if pressured by rising unemployment due to AI automation, leading to significant anti-AI movements.</li></ul><p>hope we can navigate and avoid both of these. Natural disasters, pendemies and quasi-religious anti-AI movements shouldn’t have huge impact on progress.</p><p><strong>Preparing for the Future</strong></p><p>Regardless of potential delays or obstacles, AGI will come, and we need to be prepared. The societal shifts it will bring will likely be even more profound than those following the internet revolution. Many jobs will be replaced by AI. In an ideal scenario, universal basic income could mitigate social tensions and provide people with more leisure time. However, history suggests that humans rarely choose the “nice way,” so the coming transition will likely be challenging.</p><p>Here’s a general approach to navigate the upcoming changes:</p><ul><li><strong>Upskilling:</strong> Continuously develop your skillset to a level that remains difficult for pre-general AI to replicate.</li><li><strong>Learning AI tools:</strong> Master how to leverage AI tools to complement your existing skills and prepare for future collaboration with AGI.</li><li><strong>Embrace lifelong learning.</strong></li></ul><p>This applies to everyone.</p><p>For example, software developers and related specialists should actively adopt tools like Copilot or Devin (despite its unfair over-marketing). Digital creators have already widely embraced generative tools and are experiencing the impact of AI firsthand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-bZuz9fXKiY36F02BE4Bw.jpeg" /></figure><p>All other knowledge workers should follow suit. When AGI arrives, only the most highly qualified individuals will be in demand. Manual laborers should closely monitor automation advancements in their fields and consider alternative employment options. Drivers and inventory workers already feel the threat, and others should be prepared as humanoid robots become more commonplace and affordable.</p><p>Service workers performing complex, non-repetitive tasks, such as plumbers, electricians, HVAC technicians, and furniture/renovation specialists, are more or less safe and likely to be replaced by robots much later in the process.</p><p>Artisans, on the other hand, may be the sole category to truly “survive.” This encompasses everyone from woodcarvers and manicurists to “bio”-farmers and winemakers. As ubiquitous AI and robotics become commonplace, people will likely place a greater value on handcrafted goods and experiences offered by nature. They will be willing to pay a premium for such artisanal products.</p><p>So, the general recipe for the coming era seems to be: maximize your skillset, learn to collaborate with AI, and cultivate an artisanal hobby. This creative pursuit could eventually become your backup career.</p><p><strong>The exciting era of AI is upon us. Let’s prepare ourselves for the challenges and opportunities it brings. Let’s actively participate in shaping the future of AGI, ensuring it comes soon and benefits all of humanity.</strong></p><p><strong>Important notes:</strong></p><p>Images were generated by AI. Text has been written by me. Proofreading done by AI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a0a3d41a687" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Want to use AI? Be ready to verify the work]]></title>
            <link>https://medium.com/blueriders/want-to-use-ai-be-ready-to-verify-the-work-5fbf2fbe497d?source=rss-d7d59eefacd9------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fbf2fbe497d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[quality-control]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[business-problem-solution]]></category>
            <dc:creator><![CDATA[Dmitry Ziolkovskiy]]></dc:creator>
            <pubDate>Tue, 05 Jan 2021 15:57:36 GMT</pubDate>
            <atom:updated>2021-01-05T15:57:36.101Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nd/4.0/</cc:license>
            <content:encoded><![CDATA[<p>So, you’ve decided to employ the power of AI for your business. Your problem is unique enough and requires building a custom solution — great, it may bring lots of benefits! You’ve selected a contractor or have an own team to build the solution. No matter which kind of task you have, it’s necessary to <em>teach (or train) the AI model</em>. In that sense artificial intelligence is no different than natural one — we have to learn how to perform a task before we can do it effectively. Moreover, before any “teaching”, you have to understand how to assess the results of AI’s work — define a <em>test dataset</em> and <em>performance measures</em>.</p><p>In this note, we will not dive into such a huge topic as training data preparation. Instead, we’ll discuss only the basic requirements for test data and how to satisfy them.</p><p>For you, as a business owner, this is especially important to understand what you can achieve at the final and what it will cost you. Lack of this understanding may simply cause that output from even a successfully trained AI model will be disappointing.</p><p>Each type of data has its own specific peculiarities, but the basics are the same — let’s then focus on the easiest to understand and extremely popular type — images.</p><p>As an example, let’s imagine that you have a set of cameras set in the wild and have a flow of photos with animals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*X1KXv_TzaulDXhIB" /><figcaption>Typical safari shot, perfect to explore ML-algorithms¹.</figcaption></figure><p>To check how our future AI will be good at our task, we need to define a few items:</p><ul><li>task itself (not so easy as it may seem, most often consists of few smaller ones);</li><li>measure of quality (for solution as a whole and for the core AI-task);</li><li>test data, on which to calculate these measures;</li></ul><p>All of these items depend on each other, so let’s go in order and define the <em>task</em>. For you, it may sound exactly in terms of your business, for example, “count the number of gnus crossing the river”, or “estimate the remaining volume of the water in the reservoir”. Obviously, there’s no single AI-model, which can look at on image and tell the answer: the solution for your <em>business objective</em> will consist of a few steps, including preprocessing, one or a few <em>core ML-tasks,</em> and postprocessing. All steps are equally important, but the core task is the place where AI works and it defines how successful the full pipeline will be.</p><p>When we work with images, there are many options: from simple image classification to object and action recognition.</p><p>For example, if you need a tool to estimate traffic on a certain route in the savanna, perhaps an <em>object detector</em> will be inside of the solution: train it to detect your kinds of animals and just count detections. The addition of some simple geometrical processing will give you info about movement directions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ms7tQzY8ZUg--m_I" /><figcaption>Example of object detection results: each found animal is placed in a bounding box. Different colors represent different classes of detected animals</figcaption></figure><p>The same object detector can also help you to <em>detect anomalies</em> — like a lion in a herd of gnus. But if you need to count, track specific individuals of cheetah, <em>object recognition</em> needs to be applied.</p><p>If a task will be to assess volumes of the water reserve, definitely <em>semantic segmentation</em> will help you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/0*-_gB794Mbh-YoL4H" /><figcaption>If we’re interested in water reserves monitoring, segmentation of water on the image can be an important step to assess the remaining volume</figcaption></figure><p>When a task is defined and formalized, you can choose a measure of quality for the AI-solution. Of course, you can define a measure for the whole pipeline only: for a problem with lions among gnus it can be a number of remaining gnus next day and for the volume of water in the lake it can be a volume itself, but then, in the case of the errors you will not be able to track the nature of the error (and as a consequence — can’t fix it). Again for the examples above: gnu can disappear with the help of crocodile, not a lion, and for water volume, you can just make mistakes in calibration. So, you need to have metrics at least for a full pipeline to verify the achievement of the final goal and for core ML-task to understand how AI itself is good. <em>Both of them should be as easy for you to understand as possible.</em></p><p>Proper understanding of core ML-task in your problem almost always helps you to choose a standard metric and not reinvent the wheel. Just select a metric, which takes into account what’s important for you.</p><p>Even a short description of the metrics for listed tasks will be too big for this post. We want just to emphasize that you need to define which characteristics of the systems are most important and which can be compromised, then choose the metric(s) with your data science experts and be sure that you understand what resulting numbers mean. Success here will guarantee that you’ll have under control progress of the creation of the solution and stable work of it in the future.</p><p>Having defined the task and measure of success, we need testing data — data on which we can calculate that measure. This dataset determines which values of the metrics you will have and whether the whole solution will have a sense or not. Despite this huge impact on success, many business owners and project managers underestimate the importance of having it properly prepared and this is one of the reasons why ML-projects may fail.</p><p>You can prepare this set yourself, outsource it, or just fetch some existing one, but in any case, you need to decide on what will be included in it and how big it should be. And again this directly depends on a problem.</p><p>There are a few points to pay attention to.</p><p>First important thing — dataset should be <em>representative of a problem</em>, shortly: it should represent all desired types of objects in all conditions, which will be expected in production:</p><ul><li>be sure that the test dataset includes all <em>classes (types of animals) of interest </em>to check how well each of them is detected;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*RPnBKNOMkDEvDN71" /><figcaption>Meerkats can be the object of your interest too, but they’re not so typical for safari photos</figcaption></figure><ul><li>take into account all possible lighting and weather conditions, as well as other external factors, which may change performance drastically;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*p4l03T8TZ3E6hFKd" /><figcaption>Photos at sunset or during the rain can look completely different, check for these cases too</figcaption></figure><ul><li>don’t forget about <em>multimodality</em>: if most of the images will be taken in color during the day, night images will be grayscale with IR illumination or even thermal. Obviously, AI will perform differently on each of these modalities;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*MxAqMUdmI19khrrm" /><figcaption>Leopard at night, illuminated with IR — typical image of wildlife, but not so typical for AI training. Image credit: <a href="https://commons.wikimedia.org/wiki/File:Leopard_at_night_(Kruger_National_Park,_South_Africa,_2001).jpg">Wikimedia</a></figcaption></figure><ul><li>geometric deformations may have a huge impact on results. Even for established algorithms, it’s good to check samples with different sizes of objects and points of view;</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*ONih7WWVVog0maqf" /><figcaption>Will your AI properly recognize the hunting fox here?</figcaption></figure><ul><li>ensure that AI-solution will be <em>transferable</em> to every place of your intended usage. It’s clear that model, which performs well for savanna, not necessary has the same performance for Antarctica (in our examples — both different animals and environment);</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*hSyPHzna9PKJLwJL" /><figcaption>Pinguins in the white Antarctic are animals too, will you track them with your model?</figcaption></figure><ul><li>be ready to check how the model reacts on unexpected objects (related metric is <em>precision</em>): how a truck near the herd or spider with the net on the camera are classified?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/0*YzV7P-NxtLHaGuY8" /><figcaption>Often unexpected but frequent guest on the camera lenses</figcaption></figure><p>After that, it would be good to have a <em>balanced dataset</em>. This means that for any task with many classes (types of objects), it would be good to have similar numbers of data points for all of them, otherwise, it may be difficult or impossible to properly calculate performance metrics and your whole assessment will not make sense.</p><p>For example of our gnu and lions, if we will have 95 samples of gnus and 5 images with lions and will naively calculate the simplest accuracy metric, even when we’ll be wrong and say that all 100 objects are gnus, it’s still been 95% of 100% for this metric²!</p><p>This directly leads us to the estimation of the size of the dataset — just calculate how many samples you need to evaluate defined metric values for all required conditions. For our example above, if you need to obtain 98% detection (recall) for a lion, 100 samples of that class in the dataset will just allow you to calculate the metric without quantization error (so from 100 samples with lions in the dataset, the model will properly detect 98).</p><p>For a few classes, you need to multiply accordingly. To take into account different conditions in evaluations, the number of samples per set of conditions also should be representative.</p><p>Obviously, for a comprehensive set of evaluations, there will be an astronomical number of required samples, which is unfeasible, too long, and too expensive to prepare. So usually one needs to <em>select what is most important</em> and by the Pareto principle, achieve 80% of the result (understand how well the algo performs) with 20% of the efforts (number of samples in the test set). Just be conscious that your team still needs to get a minimal set of data somewhere.</p><p>It may seem that all discussed issues are internal, technical and for you, as a business owner it’s important just to know “how well algorithm counts the animals”. Partially it’s true — most of the discussed questions should be answered by your AI-experts, with your help though. How well you will describe final requirements to the system and all variety of possible data will define the architecture of the solution. Experts will choose proper ML-algorithms and corresponding performance metrics, they can also prepare a test dataset for you, but it’s only your responsibility to be sure that it reflects your real-world situation. <em>Raise the question of the testing dataset and understandable performance metrics as early as possible and this will maximize your chances of success.</em></p><h3>Footnotes</h3><ul><li>¹- if not specified, all photos credit: <a href="https://www.pikist.com/">www.pikist.com</a></li><li>²- Of course, most of the ML-experts are aware of such issues and can properly handle even unbalanced datasets, be sure that they don’t forget about them, especially if some results look too good. As this may sound silly, unfortunately, such mistakes sometimes appear even in peer-reviewed articles.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fbf2fbe497d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/blueriders/want-to-use-ai-be-ready-to-verify-the-work-5fbf2fbe497d">Want to use AI? Be ready to verify the work</a> was originally published in <a href="https://medium.com/blueriders">blueriders</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>