<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yusuf Sarıgöz on Medium]]></title>
        <description><![CDATA[Stories by Yusuf Sarıgöz on Medium]]></description>
        <link>https://medium.com/@yusufsarigoz?source=rss-e2732a62029c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*h-CBD-XqjDFaujRRcVCKXg.jpeg</url>
            <title>Stories by Yusuf Sarıgöz on Medium</title>
            <link>https://medium.com/@yusufsarigoz?source=rss-e2732a62029c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:57:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yusufsarigoz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AfterImage is now open source for infrastructure-level dataset generation]]></title>
            <link>https://medium.com/altai-dev/afterimage-is-now-open-source-for-infrastructure-level-dataset-generation-e729507c3b03?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e729507c3b03</guid>
            <category><![CDATA[dataset-generation]]></category>
            <category><![CDATA[structured-output]]></category>
            <category><![CDATA[conversational-dataset]]></category>
            <category><![CDATA[calling-tool]]></category>
            <category><![CDATA[synthetic-datasets]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 09:17:27 GMT</pubDate>
            <atom:updated>2026-04-13T09:17:27.282Z</atom:updated>
            <content:encoded><![CDATA[<p>A Python library and CLI for synthetic conversational datasets — grounded, diverse, and observable.</p><p>Today we are releasing <a href="https://github.com/altaidevorg/afterimage">AfterImage</a> as open source software under the Apache 2.0 license, with packages on <a href="https://pypi.org/project/afterimage/">PyPI</a> (pip install afterimage / uv add afterimage) and documentation at <a href="https://afterimage.altai.dev/">afterimage.altai.dev</a>.</p><p>If you have ever tried to scale instruction tuning or evaluation data with an LLM, you have probably felt the same tension: raw volume is easy; <em>useful</em> volume is not. AfterImage exists to treat synthetic dataset generation as a systems problem — one where you can steer grounding, diversity, structure, and quality instead of hoping a clever prompt loop will “just work.”</p><h3>Why we built it</h3><p>Modern LLM workflows lean heavily on synthetic data: bootstrapped instructions, evolved difficulty, self-curation, RAG-style corpora, and more. The research and practice around these methods make a consistent point easy to miss in day-to-day engineering: the value of synthetic data is not automatic. Without controls, you get shallow diversity, weak grounding to sources, evaluator leakage, and answers that <em>sound</em> right but are not anchored in anything you trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tI3iO4NHoxjeaOdA6x3A5A.png" /><figcaption>Afterimage architecture</figcaption></figure><p>When we first designed the architecture of AfterImage internally, the core idea was simple to state and hard to operationalize: synthetic data quality has to be measured and steered, not assumed. The system was designed around explicit roles, composable stages, schema constraints where you need them, and monitoring that does not get in the way of throughput.</p><p>That design is still the spine of the project. What changed since that early write-up is everything around it: a YAML-driven CLI for getting JSONL in one command, export to common fine-tuning shapes, preference / DPO-style pair generation, broader provider support (including local OpenAI-compatible servers), a public docs site, and a codebase that has been exercised on real pipelines rather than described only on paper.</p><h3>What AfterImage actually does</h3><p>AfterImage simulates conversations between two modeled roles:</p><ul><li>Correspondent — the side that initiates: questions, tasks, follow-ups. Behavior is driven by instruction generators and can be shaped by personas (tone, expertise, intent) so your dataset is not “one generic user, forever.”</li><li>Respondent — the assistant side, defined by your system prompt and runtime prompt modifiers (for example, injecting retrieved chunks for RAG-style grounding).</li></ul><p>Generation is async-first, so you can run concurrent workers and make better use of API throughput. Persistence (JSONL or SQL) is decoupled from the core loop, so the same generation logic can land in different environments without rewrites.</p><p>There are two “front doors,” intentionally:</p><ol><li>CLI + config — describe a run in YAML, set keys in the environment, run afterimage generate. Dry-run with --dry-run when you want to validate the plan without spending tokens. When you are ready to train, afterimage export converts datasets into formats like ShareGPT-style or Hugging Face messages layouts; afterimage preference helps produce preference pairs for alignment workflows.</li><li>Python API — compose instruction generators, respondent prompt modifiers, stopping criteria, storage, judges, and monitoring the same way the CLI does under the hood. The API is where specialized flows live: document-grounded instruction, structured extraction, tool-calling-oriented setups, custom storage, and tighter integration with your stack.</li></ol><p>If you want a quick way that your coding agents can get up and running with Afterimage, the project publishes <a href="https://afterimage.altai.dev/llms.txt">llms.txt</a> with install steps, CLI entry points, and links to the Markdown guides and examples.</p><h3>Grounding, structure, and quality loops</h3><p>Document providers can feed local files, JSONL, in-memory lists, or Qdrant — so “what the model is allowed to know” can be a first-class input to generation, not an afterthought pasted into a prompt.</p><p>Structured generation uses Pydantic schemas so single-turn outputs are valid JSON-shaped objects: extraction from unstructured text, synthetic rows with typed fields, or evaluation artifacts you can consume downstream without brittle parsing.</p><p>Evaluation is meant to be practical at scale: embedding-based signals for fast filtering (coherence, relevance, grounding-style checks where configured) and LLM-as-judge style rubrics when you need a second pass. The framework also supports quality gates and regeneration paths (for example, auto_improve workflows) when you want the generator to retry or revise under explicit criteria rather than silently accepting bad rows.</p><p>Monitoring tracks latency, token usage, errors, and evaluation signals over time so long runs remain operable — synthetic data generation is as much an engineering workload as a modeling one.</p><h3>Design choices that survived contact with reality</h3><p>Two ideas from the original technical framing stayed true in the shipped library:</p><p>Composition over inheritance. You should not need to fork a monolithic “generator” class for every new corpus or policy. Behavior is injected through callbacks and strategies — instruction logic, prompt modification, storage, evaluation — so the core stays stable while your pipeline evolves.</p><p>Provider portability. AfterImage normalizes chat sessions, structured calls, token accounting, and metadata across Gemini, OpenAI-compatible APIs (including DeepSeek and OpenRouter where applicable), and local OpenAI-compatible servers (for example vLLM, Ollama, llama.cpp). Conversation state is managed independently of any single vendor’s chat API quirks.</p><p>Internally, recent work strengthened the parts that matter most in production: smarter context and persona sampling (including coverage-oriented behavior), clearer separation between orchestration, sampling, and quality gating, and optional capture of reasoning / thinking content from compatible providers when you want that signal preserved in your dataset.</p><h3>What AfterImage is not</h3><p>AfterImage is not a promise that synthetic data is “as good as hand-labeled gold for all sorts of use cases” without validation, and it is not a theory of instruction learning. It is a pragmatic engine for people who need repeatable, configurable, observable generation — from a quick experiment to a large batch job with budgets and filters.</p><p>The same limitations that applied when we wrote the internal report still apply in public form: evaluator bias can leak into curated sets; LLM judges cost time and money; personas need care so diversity does not collapse into caricature. We ship controls and visibility so you can see those trade-offs instead of discovering them only after training.</p><h3>Try it</h3><ul><li>Repository: <a href="https://github.com/altaidevorg/afterimage">github.com/altaidevorg/afterimage</a></li><li>Documentation: <a href="https://afterimage.altai.dev/">afterimage.altai.dev</a></li><li>Install: Python 3.11+, then pip install afterimage or uv add afterimage</li><li>Quick start: afterimage generate -c examples/configs/basic.yaml (see <a href="https://github.com/altaidevorg/afterimage/tree/main/examples/configs">examples/configs/</a> for RAG, local models, stopping budgets, and more)</li></ul><p>Optional extras cover local embeddings (embeddings-local), a small FastAPI server (server), and a training / demo stack (training) for the Gradio demo and training helpers — see pyproject.toml for the exact dependency sets.</p><h3>Closing</h3><p>Open sourcing AfterImage is our invitation to treat synthetic data as infrastructure: something you configure, measure, and iterate on like any other part of the ML stack. If you build something with it — or hit a wall we should address — we would love to hear from you in <a href="https://github.com/altaidevorg/afterimage/issues">issues</a> and <a href="https://github.com/altaidevorg/afterimage/discussions">discussions</a> on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e729507c3b03" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altai-dev/afterimage-is-now-open-source-for-infrastructure-level-dataset-generation-e729507c3b03">AfterImage is now open source for infrastructure-level dataset generation</a> was originally published in <a href="https://medium.com/altai-dev">Altai.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning Is Not Dead: How Synthetic QA Fine-Tuning Makes RAG Smarter]]></title>
            <link>https://medium.com/altai-dev/fine-tuning-is-not-dead-how-synthetic-qa-fine-tuning-makes-rag-smarter-dde835a04c8c?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/dde835a04c8c</guid>
            <category><![CDATA[prague]]></category>
            <category><![CDATA[synthetic-data]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Mon, 17 Nov 2025 10:56:50 GMT</pubDate>
            <atom:updated>2025-11-17T11:09:20.218Z</atom:updated>
            <content:encoded><![CDATA[<p>At Altai, we’re frequently asked one question:</p><blockquote><em>“Why bother fine-tuning when you can just use RAG?”</em></blockquote><p>Retrieval-Augmented Generation (RAG) — the idea of feeding external documents to a frozen large language model (LLM) — quickly became the default pattern for domain-specific enterprise AI. It’s conversational, so it sounds intuitive to even non-technicals. It avoids retraining, so it’s developer-friendly. It’s flexible and composable, so you can build different pipelines with multiple components.</p><p>But something has been missing in that story: it’s simply like giving a complex textbook to someone without the background and expecting expert reasoning — they can find surface-level answers to trivial questions, but they will struggle to synthesize deeper, domain-grounded insights and inevitably fall short on higher-order reasoning.</p><p>A recent academic paper has empirically shown this to be true. The authors of <a href="https://arxiv.org/abs/2510.12668">The Role of Parametric Injection-A Systematic Study of Parametric Retrieval</a> systematically compared <strong>contextual RAG</strong>, i.e., standard RAG with retrieval results injected in the context, <strong>PRAG</strong>, i.e., knowledge injected in the form of LoRA adapters trained on documents, and <strong>PRAG-Combine</strong>, a hybrid method combining both. With this study, they concluded that RAG gets smarter, more robust, and more faithful when combined with domain-specific fine-tuning, which supports the hypothesis we built Altai around.</p><p><strong>Join the discussion</strong></p><p>‘Join the Altai community to reach out to like-minded developers, ’practitioners and researchers, have a discussion around the cutting-edge AI, LLM customization techniques, show off your projects, participate in paper reading sessions and much more:</p><p><a href="https://discord.com/invite/XuyfFW2X">Join the ALTAI Discord Server!</a></p><h3>The core insight: synthetic QA fine-tuning improves RAG itself</h3><p>The authors fine-tuned LoRA adapters (small, efficient fine-tuning modules) on <strong>document–question–answer triples</strong> — many of which were synthetically generated. These adapters captured domain-specific semantics, effectively encoding <em>how to interpret retrieved text</em>, not just what the text says.</p><p>Then they tested three setups:</p><ul><li><strong>RAG</strong>: Retrieve relevant text and prompt the base model. Baseline.</li><li><strong>PRAG</strong>: Inject fine-tuned LoRAs, no retrieval.Worse than RAG — lacks fine-grained factual details.</li><li><strong>PRAG-Combine: </strong>Use both fine-tuned LoRAs <em>and</em> retrieval.Outperformed RAG on all benchmarks. Excels in the case of noisy retrieval results</li></ul><p>The hybrid model — PRAG-Combine — <strong>consistently beat RAG</strong> across factual QA, multi-hop reasoning, and robustness tests with noisy or irrelevant passages.</p><p>That’s the headline:</p><blockquote><em>Fine-tuning with synthetic QA pairs doesn’t compete with RAG — it </em>amplifies<em> and </em>completes<em> it exactly where it fails.</em></blockquote><h3>Why this matters for Altai</h3><p>At Altai, we’ve built a platform where companies can train small LLMs on <strong>synthetic, domain-specific QA datasets</strong> — generated automatically from their own knowledge base.</p><p>The intuition has always been simple:</p><blockquote><em>“You can’t expect a model that’s never seen your domain to interpret your documents optimally — even if you retrieve the right ones.”</em></blockquote><p>This new research provides empirical backing for that intuition. It shows that domain fine-tuning changes how the model <em>uses</em> retrieved context. LoRAs trained on QA pairs help the model integrate text more coherently, reduce hallucination, and stay grounded even when retrieval is messy.</p><p>Altai’s approach — generating synthetic QAs, fine-tuning, and combining that with retrieval — aligns exactly with PRAG-Combine, the configuration that outperformed everything else.</p><h3>What’s actually happening under the hood</h3><p>The paper’s layer-wise analysis revealed the mechanism that explains why PRAG-Combine works better: Fine-tuned adapters increase <strong>parametric knowledge scores</strong> in the <em>later layers</em> of the transformer — the parts responsible for semantic reasoning. In plain terms, fine-tuning on synthetic QA pairs teaches the model “how to think” about domain-specific information.</p><p>That means when the retriever feeds a new document, the model doesn’t just read it literally — it interprets it through the lens of its domain knowledge. It knows what really matters, resolves conflicts, and synthesizes a thoughtful response instead of copy-pasting from the context.</p><p>In the Altai pipeline, this is exactly what happens:</p><ol><li><strong>Synthetic QA generation</strong>: Altai’s Afterimage framework leveraging our custom-made models generates diverse, domain-relevant questions and answers from a company’s documents.</li><li><strong>Fine-tuning adapters</strong>: These QA pairs train a smaller model or LoRA module to internalize the domain’s structure — concepts, relationships, typical instruction forms.</li><li><strong>Hybrid inference (letsearch)</strong>: During serving, we still use retrieval — but now the model interprets the retrieved chunks with domain intuition.</li></ol><p>Result: faster convergence, better grounding, and much more reliable outputs in specialized domains (law, healthcare, finance, etc.).</p><h3>Why synthetic QA data works so well</h3><p>Synthetic QA pairs are not just convenient — they’re structurally ideal training data:</p><ul><li><strong>Question form</strong> teaches <em>information need recognition</em> — what matters in the text.</li><li><strong>Answer form</strong> teaches <em>fact grounding</em> — where and how to extract it.</li><li>Together, they encode “retrieval intent” directly into the model’s parameters.</li></ul><p>The paper shows that even when generated automatically, such QA data <em>improves interpretive alignment</em> between the model and retrieved context.</p><p>In other words, the model learns not just facts, but also how to <em>think in the light of facts</em>.</p><h3>The big picture: fine-tuning and retrieval aren’t rivals</h3><p>The industry treated RAG as the “fine-tuning killer.” However, this paper demonstrates that RAG does not suffice in settings involving domain knowledge. The solution isn’t RAG <em>or</em> fine-tuning — it’s RAG <em>plus</em> domain fine-tuning via synthetic data.</p><p>Altai operationalizes that theoretical finding. Our platform:</p><ul><li><strong>Generates synthetic QA datasets</strong> tailored to your documents thanks to our enterprise-grade synthetic dataset engine (Afterimage).</li><li><strong>Fine-tunes or LoRA-adapts models</strong> to embed domain semantics on the Altai platform.</li><li><strong>Combines that with retrieval</strong> through our RAG layer (letsearch) for deployment.</li></ul><p>The result: leaner models that know your domain and interpret your knowledge base intelligently. Retrieval gets you access to knowledge, but fine-tuning teaches you how to use it.</p><p>Altai’s synthetic QA fine-tuning turns your documents into domain intuition — the missing half of the retrieval story.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dde835a04c8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altai-dev/fine-tuning-is-not-dead-how-synthetic-qa-fine-tuning-makes-rag-smarter-dde835a04c8c">Fine-Tuning Is Not Dead: How Synthetic QA Fine-Tuning Makes RAG Smarter</a> was originally published in <a href="https://medium.com/altai-dev">Altai.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launching Altai: Unlocking Enterprise AI with Customization & Open Source]]></title>
            <link>https://medium.com/altai-dev/launching-altaiunlocking-enterprise-ai-with-customization-open-source-0238d43f27c5?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0238d43f27c5</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[synthetic-datasets]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Thu, 29 May 2025 11:14:27 GMT</pubDate>
            <atom:updated>2025-05-29T16:13:05.699Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FugiHXAUYdjU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DugiHXAUYdjU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FugiHXAUYdjU%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/125129133b0b0e2d847ebd4786ecaeea/href">https://medium.com/media/125129133b0b0e2d847ebd4786ecaeea/href</a></iframe><h3>Enterprises are eager to leverage large language models (LLMs) to drive automation, enhance decision-making, and streamline operations. However, adopting AI at scale comes with significant hurdles:</h3><p>Among other things:</p><ul><li><strong>Customization</strong>: Off-the-shelf models don’t fit specific enterprise needs.</li><li><strong>Data Privacy</strong>: Cloud-hosted solutions pose compliance risks.</li><li><strong>Talent Gap</strong>: Hiring AI experts is costly and competitive.</li></ul><p>At <a href="https://altai.dev/"><strong>Altai</strong></a>, we’re solving these challenges by providing enterprises with a <strong>no-code AI platform</strong> for effortless <strong>LLM customization and deployment</strong>, whether in the cloud or on-premises. Our vision is to democratize AI by combining the power of <strong>open-source models</strong> with high-quality <strong>synthetic dataset generation</strong> and enterprise-grade adaptation.</p><h3>The Open-Source AI Opportunity</h3><p>Open-source LLMs like <strong>Llama</strong> and <strong>DeepSeek</strong> have made cutting-edge AI more accessible. However, <strong>enterprises require tailored solutions</strong> — domain-specific fine-tuning, secure on-prem deployment, and reliable data pipelines. As Y Combinator recently stated:</p><blockquote>“Just as RedHat and YC W15 company GitLab built public companies around open-source projects, we are looking for companies built around open source AI models like DeepSeek and Llama. There is a huge opportunity to build startups that offer support and services to help people use open-source AI.”<em> (</em><a href="https://www.linkedin.com/posts/y-combinator_just-as-redhat-and-yc-w15-company-gitlab-activity-7291198562363740161-Rw_D/?utm_source=share&amp;utm_medium=member_desktop"><em>YC on LinkedIn</em></a><em>)</em></blockquote><p>Altai seizes this opportunity by <strong>bridging the gap between open-source AI and real-world enterprise needs</strong>.</p><h3>How Altai Works</h3><p>Our <strong>end-to-end workflow</strong> enables organizations to customize and deploy AI models without writing code:</p><ol><li><strong>Upload Documents</strong> → Process business-specific data.</li><li><strong>Fine-Tune or Continual Pretrain</strong> → Adapt smaller LLMs to fit unique requirements.</li><li><strong>Generate Synthetic Datasets</strong> → Use purpose-built LLMs sophisticated algorithms to create and validate high-quality training data.</li><li><strong>Instruction Fine-Tuning</strong> → Enhance models for enterprise-specific use cases by fine-tuning them on the validated synthetic dataset.</li><li><strong>Deploy Securely</strong> → Deploy models <strong>on-prem</strong> or in the <strong>cloud</strong>.</li></ol><p>By integrating automation at every step, Altai empowers organizations to build smarter, more relevant AI solutions in days rather than months.</p><h3>Why Now?</h3><p>The demand for <strong>enterprise-ready AI</strong> is exploding. Companies struggle with:</p><ul><li><strong>The need for domain-specific customization</strong> to get real business value from LLMs.</li><li><strong>Vendor lock-in</strong> from closed-source AI providers.</li><li><strong>Skyrocketing compute costs</strong> for training and inference.</li></ul><p>Altai eliminates these barriers, making AI <strong>useful, adaptable, enterprise-ready, and cost-effective.</strong>.</p><h3>A Future Powered by Open, Customizable AI</h3><p>The AI landscape is shifting. Enterprises no longer need to rely on closed, black-box models from a handful of providers. With Altai, we’re giving companies the tools to take control of their AI strategy — leveraging <strong>open-source innovation while ensuring customization, security, and scalability</strong>.</p><p><strong>The future of AI is open</strong>, and Altai is here to make it work for you.</p><p>We’re just getting started. If you’re building with open-source AI and looking for enterprise-ready solutions, <a href="https://altai.dev/contact"><strong>let’s talk</strong></a>.</p><h3>The Road Ahead</h3><p>We’re building <strong>the foundation for scalable enterprise AI</strong> — a <strong>no-code, high-performance AI platform</strong> that empowers companies to own and control their models. If you’re an enterprise looking to deploy AI <strong>without hiring a massive ML team</strong>, <strong>Altai is your answer</strong>.</p><p><a href="https://altai.dev/"><strong>Sign up for early access.</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0238d43f27c5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altai-dev/launching-altaiunlocking-enterprise-ai-with-customization-open-source-0238d43f27c5">Launching Altai: Unlocking Enterprise AI with Customization &amp; Open Source</a> was originally published in <a href="https://medium.com/altai-dev">Altai.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are you tired of getting only garbage instead of Markdown from PDFs? Yeah, same.]]></title>
            <link>https://medium.com/altai-dev/are-you-tired-of-getting-only-garbage-instead-of-markdown-from-pdfs-yeah-same-f7816fe2f87b?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7816fe2f87b</guid>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Mon, 12 May 2025 09:31:31 GMT</pubDate>
            <atom:updated>2025-05-16T08:27:47.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsjmuANKG9ExW5C4DMzVoA.png" /><figcaption>Serving files for hungry LLMs</figcaption></figure><p>Are you tired of getting only garbage instead of Markdown from PDFs? <em>Yeah, same.</em></p><p>That’s why we built <a href="https://github.com/altaidevorg/llm-food"><strong>llm-food</strong></a> — a FastAPI-based service that converts documents and URLs into clean, LLM-friendly Markdown. It supports batch processing, integrates with Google’s Gemini for blazing-fast PDF OCR, and wraps everything in a CLI and Python client that doesn’t make you cry.</p><h3>🥡 tl;dr</h3><ul><li>-Converts PDFs, DOCX, RTF, PPTX, and HTML/web pages into Markdown.</li><li>-Synchronous and async modes via HTTP API, CLI, and Python client.</li><li>-Batch PDFs go through <strong>Gemini Batch Prediction API</strong> for high throughput (<strong>$1 for ~6,000 pages</strong>).</li><li>-Tracks batch jobs in DuckDB.</li><li>-Fully dockerized. Optional auth. One-liner CLI.</li></ul><h3>Why?</h3><p>Extracting clean text from PDFs is still a mess.</p><p>You’ve probably seen dockling, marker, or pymupdf4llm. They’re okay—but they’re either slow, resource-hungry, or AGPL-licensed (ouch!). If you just want clean Markdown to fine-tune or RAG with your LLM, it&#39;s way more effort than it should be.</p><p>Enter <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/batch-prediction-gemini"><strong>Gemini Batch Prediction</strong></a>. It’s fast and insanely cheap — <a href="https://www.sergey.fyi/articles/gemini-flash-2">$1 per 6,000 pages</a>. But it’s not dev-friendly.</p><p>So we wrapped it all up in a neat little microservice: llm-food.</p><h3>What You Get</h3><h3>✅ Features</h3><ul><li>Convert:</li><li>PDFs (Gemini, pymupdf4llm, or pypdf)</li><li>DOC/DOCX (via mammoth)</li><li>RTF (via striprtf)</li><li>PPTX (python-pptx)</li><li>HTML or URLs (trafilatura)</li><li>Run as:</li><li>FastAPI server (sync and async modes)</li><li>CLI (llm-food convert-file my.pdf)</li><li>Python client</li><li>Async batch jobs with task tracking (DuckDB-based).</li><li>Docker-ready, with optional Bearer token auth.</li><li>Extensible and configurable.</li></ul><h3>📦 Batch Mode: Fast, Cheap, Parallel</h3><p>PDFs get chunked and sent to Gemini’s Batch API. You just upload files and give it a GCS path — we handle the queuing, status tracking, and Markdown export.</p><p>Other formats (DOCX, RTF, PPTX) are handled individually as background tasks.</p><h3>How to Use It</h3><h3>🍳 Install</h3><pre>pip install &#39;llm-food[server]&#39;  # Full server<br>pip install &#39;llm-food[server,pymupdf]&#39; # Optional: if you want pymupdf backend</pre><p>Or just the client?</p><pre>uv add llm-food</pre><h3>🛰️ Start the Server</h3><pre>llm-food-serve</pre><p>API lives at <a href="http://localhost:8000/docs.">http://localhost:8000/docs.</a></p><p>Want Docker?</p><pre>docker build -t llm-food .<br>docker run -d -p 8000:8000 --env-file .env llm-food</pre><h3>⚡ CLI Usage</h3><pre>llm-food convert-file ./paper.pdf<br>llm-food convert-url https://example.com<br>llm-food batch-create doc1.pdf doc2.pdf gs://my-bucket/outputs/<br>llm-food batch-status &lt;task_id&gt;<br>llm-food batch-results &lt;task_id&gt;</pre><p>Set server URL/token via env:</p><pre>export LLMFOOD_SERVERURL=http://my-host:8000<br>export LLMFOOD_APITOKEN=secret</pre><h3>⚙️ Auth &amp; Config</h3><ul><li>Add a .env file to configure:</li><li>GCS credentials</li><li>Gemini API location/project</li><li>File size limits</li><li>PDF backend</li><li>Auth token</li><li>Run locally or on a containerized server.</li></ul><h3>🛣️ Roadmap</h3><ul><li>Support for transcription of media files (videos and audios)</li><li>YouTube crawling and batch transcription</li><li>Configurable automatic chunking</li></ul><h3>🍽️ Final Thoughts</h3><p>Building a RAG pipeline, a search agent, or just need clean Markdown from messy enterprise docs? <a href="https://github.com/altaidevorg/llm-food">llm-food</a> gets you there fast.</p><p>But let’s be real: raw Markdown is just raw ingredients. We’re cooking up an enterprise-grade engine that builds custom synthetic datasets and trains LLMs tailored to your docs. <a href="https://altai.dev/">Join the waitlist</a> or <a href="/service/mailto:hi@altai.dev">drop us a line</a> if that’s what you’re hungry for.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7816fe2f87b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/altai-dev/are-you-tired-of-getting-only-garbage-instead-of-markdown-from-pdfs-yeah-same-f7816fe2f87b">Are you tired of getting only garbage instead of Markdown from PDFs? Yeah, same.</a> was originally published in <a href="https://medium.com/altai-dev">Altai.dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open-source implementation of OpenAI’s Tasks feature]]></title>
            <link>https://levelup.gitconnected.com/open-source-implementation-of-openais-tasks-feature-df274c3e5839?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/df274c3e5839</guid>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[opeai-tasks]]></category>
            <category><![CDATA[job-scheduling]]></category>
            <category><![CDATA[gemini]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Tue, 04 Feb 2025 16:10:15 GMT</pubDate>
            <atom:updated>2025-02-04T16:10:15.708Z</atom:updated>
            <content:encoded><![CDATA[<h4>Let LLMs schedule tasks and reminders for you and itself</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHhH5kWrXDayrrNY9EySXQ.jpeg" /><figcaption>Image generated by Google Imagen</figcaption></figure><p>OpenAI introduced a featured called <a href="https://help.openai.com/en/articles/10291617-scheduled-tasks-in-chatgpt">Tasks</a> to schedule tasks directly in ChatGPT.</p><p>At the first glance, it might seem to be a ground-braking feature, but it’s nothing more than function calling, so I just wanted to implement my own with Gemini –in fact it can be ported to any LLM that supports function calling.</p><h3>How it works?</h3><p>It has two separate functions defined to schedule one-time or recurring tasks. The configured LLM is supposed to call one of them when it thinks that it needs to set a reminder for itself.</p><p>In order to set a one-time reminder, the LLM is supposed to pass the firing time in addition to the reminder name and tasks instruction as an argument to the function. Then, it is scheduled with <a href="https://apscheduler.readthedocs.io/en/3.x/">APScheduler</a> by using a <a href="https://apscheduler.readthedocs.io/en/3.x/modules/triggers/date.html">DateTrigger</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d3bee8d1f93c771d1b7a078bec336c45/href">https://medium.com/media/d3bee8d1f93c771d1b7a078bec336c45/href</a></iframe><p>It has access to another function to get the current date and time so that it can call the function above with the correct date and time expression.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c534d8ee33a71b7456f46277aa6feff4/href">https://medium.com/media/c534d8ee33a71b7456f46277aa6feff4/href</a></iframe><p>To set a recurring reminder, it should pass a cron expression instead of a date string, and this time it is schedule with <a href="https://apscheduler.readthedocs.io/en/3.x/modules/triggers/cron.html">CronTrigger</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d46e621bba20421bc5e048418e93f1f/href">https://medium.com/media/5d46e621bba20421bc5e048418e93f1f/href</a></iframe><p>The callback function for the reminders receives the task instructions and metadata, and it makes a new call to the LLM with these information injected in the prompt.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8bf6e84272ec80e3ac56111148f84258/href">https://medium.com/media/8bf6e84272ec80e3ac56111148f84258/href</a></iframe><h3>Implementation Details</h3><p>I designed the agent to act as an “AI chief of staff to the CTO,” and I’m planning to really use it as a part of the team at my upcoming startup after adding more capabilities. It can currently do web or news search with DuckDuckGo –easily replaceable with other search providers-, create documents and send me emails, and schedule tasks to be performed later and/or repeatedly. I didn’t want to introduce dependencies on projects such as Langchain and CrewAI to make it really hackable, and I implemented all the tools and agentic workflow myself.</p><ul><li>You can see and modify the system prompts in <a href="https://github.com/monatis/open-tasks/blob/main/prompts.py">prompts.py</a>.</li><li>See <a href="https://github.com/monatis/open-tasks/blob/main/tools.py">tools.py</a> for the functions used by the LLM.</li><li><a href="https://github.com/monatis/open-tasks/blob/main/agent.py">agent.py</a> hosts the main agent.</li></ul><h3>Future work</h3><p>I’m planning to connect it to the whole knowledge base of my startup, giving more tools to do research, and enable interaction with the team on Slack. I also have some experimental ideas with sub-agents and multimodal interaction, so stay tuned for more info.</p><h3>Usage</h3><p>If you would like to give it a try:</p><ol><li>Clone the repository: git clone <a href="https://github.com/monatis/open-tasks.git">https://github.com/monatis/open-tasks.git</a></li><li>Rename .env.example to .env and fill in your Gemini API key and SMTP credentials for sending emails.</li><li>Install dependencies:</li></ol><ul><li>If you Have uv installed: uv sync</li><li>Otherwise: pip install .</li></ul><ol><li>Start the API: python serve.py</li><li>I haven’t implemented a UI yet, so go to localhost:5000/docs to test it on the Swagger UI.</li><li>Try writing a prompt such as “Remind me to call my mom 3 minutes later from now,” and wait for 3 minutes or so to get the notification.</li></ol><p>To try something more ambitious, you can write “Collect news stories about XXX and send them with summaries and the URLs every morning at 8.”</p><p><strong>Note</strong>: If you have any suggestion or a use case in mind, feel free to reach out to me in <a href="https://github.com/monatis/open-tasks/issues">issues</a>.</p><h3>Fun fact</h3><p>In order to test the the agent, I used the following prompt:</p><pre>I want to publish a post regarding how I developed you, i.e., a Gemini-backed agent serving as the chief of staff to the CTO that can set reminders for the CTO and itself. Create a document that contains a blog post mentioning the technical stack: apscheduler, SMTP, and duckduckgo. I was inspired by OpenAI&#39;s new feature called &#39;Tasks&#39;. You can use web search to gather missing information.</pre><p>Then it performed several web searches, read some of the search results, created <a href="https://github.com/monatis/open-tasks/blob/main/examples/CTO&#39;s%20blog%20post%20about%20his%20AI%20assistant.md">this document in </a><a href="https://github.com/monatis/open-tasks/blob/main/examples/CTO&#39;s%20blog%20post%20about%20his%20AI%20assistant.md">examples</a> and then scheduled a reminder for itself to follow up about it with me.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df274c3e5839" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/open-source-implementation-of-openais-tasks-feature-df274c3e5839">Open-source implementation of OpenAI’s Tasks feature</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Launch a web API for Stable Diffusion under 45 seconds]]></title>
            <link>https://medium.com/data-science/launch-a-web-api-for-stable-diffusion-under-45-seconds-bbd88cfe41d8?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd88cfe41d8</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Sat, 08 Oct 2022 06:09:46 GMT</pubDate>
            <atom:updated>2022-10-08T06:09:46.787Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Run Stable Diffusion in Docker with a Simple Web API and GPU Support</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*o2w4ls_r-vo6UqedOsj65Q.jpeg" /><figcaption>An imaginary black goat generated by Stable Diffusion</figcaption></figure><p><a href="https://github.com/CompVis/stable-diffusion">Stable Diffusion</a> is a latent text-to-image diffusion model, made possible thanks to a collaboration with Stability AI and Runway. It features state-of-the-art text-to-image synthesis capabilities with relatively small memory requirements (10 GB). Stable Diffusion features several improvements over other Diffusion models to achieve this efficiency, but these innovations are beyond the scope of this post — a future post will cover how to train a Diffusion model in TensorFlow and detail its inner workings technically.</p><p>Divam Gupta <a href="https://github.com/divamgupta/stable-diffusion-tensorflow">ported Stable Diffusion to TensorFlow / Keras</a> from original weights, and this post focuses on how to run it in a Docker image with a simple web API and GPU support.</p><p><strong>Fun fact</strong>: The featured image in this post was also generated by Stable Diffusion.</p><h4>How does it work?</h4><p>I decided to run it on a GPU from <a href="https://marketplace.tensordock.com/">TensorDock Marketplace</a>., It should work on other machines with little-to-no change needed, but it is a direct outcome of my experiments with recently-launched TensorDock Marketplace. It is currently in Public Alpha, but I already like their innovative idea to democratize access to high-performance computing. Launched in addition to their <a href="https://www.tensordock.com/">affordable Core Cloud GPU service</a>, the Marketplace edition serves as a marketplace that brings clients and GPU providers together. Hosts, i.e., those who have spare GPUs, can rent them to clients including independent researchers, startups, hobbiests, tinkerers etc. with insanely cheap prices. <a href="https://www.tensordock.com/host">Acording to TensorDock</a>, this also lets hosts earn 2x-to-3x mining profits. And, for a better purpose than mining meaningless cryptos.</p><p>Servers are customizable for required RAM, vCPU and disc allocated, and boot times are too short around ~45 seconds. You may choose to start with a minimal Ubuntu image with NVIDIA drivers and Docker already installed, or you can jump into experiments with fully-fledged images with NVIDIA drivers, Conda, TensorFlow, PyTorch and Jupyter configured. I choose to work with Docker instead of Conda or virtual environments to isolate my AI projects, and I’ll begin with a minimal image with Docker installed on TensorDock Marketplace.</p><p>First, I’ll show how to containerize a web API serving Stable Diffusion in a Docker image. Then, I’ll provide a step-by-step description of how to serve it on a TensorDock GPU. If you want to be up and running in 45 seconds, you can directly jump to the “Ok, show me how to run” section, or you may choose to check out the <a href="https://github.com/monatis/stable-diffusion-tf-docker">GitHub repo</a>.</p><h4>Let’s Dockerize!</h4><p>TensorFlow provides official <a href="https://www.tensorflow.org/install/docker">prebuilt Docker images</a> for each version to start your Dockerfile. If you have <a href="https://developer.nvidia.com/nvidia-container-runtime">NVIDIA Container Runtime</a> configured and prefer a GPU-enabled Docker image by TensorFlow, you can run your TensorFlow code with GPU support instantly without getting trouble with CUDA or CUDNN. Luckily, TensorDock’s minimal Ubuntu images come with NVIDIAContainerRuntime support.</p><p>ForStable Diffusion, we start our Dockerfile with tensorflow/tensorflow:2.10.0-gpu. Then we install Stable Diffusion requirements and FastAPI to serve a web API. Finally, we copy app.py that contains the web API and configure it to run at the container startup:</p><pre>from tensorflow/tensorflow:2.10.0-gpu<br><br>RUN apt update &amp;&amp; \<br>    apt install -y git &amp;&amp; \<br>    pip install --no-cache-dir Pillow==9.2.0 tqdm==4.64.1 \<br>    ftfy==6.1.1 regex==2022.9.13 tensorflow-addons==0.17.1 \<br>    fastapi &quot;uvicorn[standard]&quot; git+https://github.com/divamgupta/stable-diffusion-tensorflow.git<br><br>WORKDIR /app<br><br>COPY ./app.py /app/app.py<br><br>CMD uvicorn --host 0.0.0.0 app:app</pre><h4>Compose for easier configuration and launch</h4><p>Docker is even greater when you have a docker-compose.yml file. Thereby, you can simply run docker compose up to put everything up and running with a single command. It shines especially when working with multiple containers, but it&#39;s also quite useful for managing a single-container.</p><p>The following docker-compose.yml file defines several environment variables and passes them to the container. It also enables GPU access inside the container and configures the Docker network properly as required on TensorDock Marketplace. You may need to remove this section regarding the network on other platforms</p><pre>version: &quot;3.3&quot;<br><br>services:<br>  app:<br>    image: myusufs/stable-diffusion-tf<br>    build:<br>      context: .<br>    environment:<br>      # configure env vars to your liking<br>      - HEIGHT=512<br>      - WIDTH=512<br>      - MIXED_PRECISION=no<br>    ports:<br>      - &quot;${PUBLIC_PORT?Public port not set as an environment variable}:8000&quot;<br>    volumes:<br>      - ./data:/app/data<br><br>    deploy:<br>      resources:<br>        reservations:<br>          devices:<br>            - driver: nvidia<br>              count: 1<br>              capabilities: [ gpu ]<br><br>networks:<br>  default:<br>    driver: bridge<br>    driver_opts:<br>      com.docker.network.driver.mtu: 1442</pre><h4>Time to code</h4><p>Now we are readyfor coding our webAPI. We begin with required imports and then create an image generator as well as FastAPI app with environment variables passed by docker-compose.yml.</p><pre>import os<br>import time<br>import uuid<br><br>from fastapi import FastAPI<br>from fastapi.exceptions import HTTPException<br>from fastapi.responses import FileResponse<br>from PIL import Image<br>from pydantic import BaseModel, Field<br>from stable_diffusion_tf.stable_diffusion import Text2Image<br>from tensorflow import keras<br><br>height = int(os.environ.get(&quot;WIDTH&quot;, 512))<br>width = int(os.environ.get(&quot;WIDTH&quot;, 512))<br>mixed_precision = os.environ.get(&quot;MIXED_PRECISION&quot;, &quot;no&quot;) == &quot;yes&quot;<br><br>if mixed_precision:<br>    keras.mixed_precision.set_global_policy(&quot;mixed_float16&quot;)<br><br>generator = Text2Image(img_height=height, img_width=width, jit_compile=False)<br><br>app = FastAPI(title=&quot;Stable Diffusion API&quot;)</pre><p>Then, we define request and response bodies for our /generate endpoint. Values are self-explanatory, so there&#39;s no need for extra wordshere.</p><pre>class GenerationRequest(BaseModel):<br>    prompt: str = Field(..., title=&quot;Input prompt&quot;, description=&quot;Input prompt to be rendered&quot;)<br>    scale: float = Field(default=7.5, title=&quot;Scale&quot;, description=&quot;Unconditional guidance scale: eps = eps(x, empty) + scale * (eps(x, cond) - eps(x, empty))&quot;)<br>    steps: int = Field(default=50, title=&quot;Steps&quot;, description=&quot;Number of dim sampling steps&quot;)<br>    seed: int = Field(default=None, title=&quot;Seed&quot;, description=&quot;Optionally specify a seed for reproduceable results&quot;)<br><br><br>class GenerationResult(BaseModel):<br>    download_id: str = Field(..., title=&quot;Download ID&quot;, description=&quot;Identifier to download the generated image&quot;)<br>    time: float = Field(..., title=&quot;Time&quot;, description=&quot;Total duration of generating this image&quot;)</pre><p>Finally, it’s time to write our endpoints. The /generate endpoint will accept a textual prompt as well as several configuration values to control the generation, and it will response with the unique ID of the image generated. Then, the image can be downloadable through the /download endpoint. Generated images are saved to a directory configured as a Docker volume in docker-compose.yml.</p><pre>@app.post(&quot;/generate&quot;, response_model=GenerationResult)<br>def generate(req: GenerationRequest):<br>    start = time.time()<br>    id = str(uuid.uuid4())<br>    img = generator.generate(req.prompt, num_steps=req.steps, unconditional_guidance_scale=req.scale, temperature=1, batch_size=1, seed=req.seed)<br>    path = os.path.join(&quot;/app/data&quot;, f&quot;{id}.png&quot;)<br>    Image.fromarray(img[0]).save(path)<br>    alapsed = time.time() - start<br>    <br>    return GenerationResult(download_id=id, time=alapsed)<br><br>@app.get(&quot;/download/{id}&quot;, responses={200: {&quot;description&quot;: &quot;Image with provided ID&quot;, &quot;content&quot;: {&quot;image/png&quot; : {&quot;example&quot;: &quot;No example available.&quot;}}}, 404: {&quot;description&quot;: &quot;Image not found&quot;}})<br>async def download(id: str):<br>    path = os.path.join(&quot;/app/data&quot;, f&quot;{id}.png&quot;)<br>    if os.path.exists(path):<br>        return FileResponse(path, media_type=&quot;image/png&quot;, filename=path.split(os.path.sep)[-1])<br>    else:<br>        raise HTTPException(404, detail=&quot;No such file&quot;)</pre><h4>Ok, show me how to run</h4><p>Don’t let the number of steps scare you — it’s only a detailed step-by-step walkthrough of the whole process from registering to making requests. It should take no longer than ~10 minutes.</p><ol><li><a href="https://marketplace.tensordock.com/register">Registre</a> and sign in to TensorDock Marketplace.</li><li><a href="https://marketplace.tensordock.com/order_list">Go to the order page</a>, and choose a physical machine that offers a GPU with at least 10 GB of memory. I’d suggest one that offers RTX 3090.</li><li>This will open up a model that lets you configure your server. My suggestestions are as follows:</li></ol><ul><li>Select amount of each GPU model: 1 x GeForce RTX 3090 24 GB</li><li>Select amount of RAM (GB): 16</li><li>Select number of vCPUs: 2</li><li>Check checkboxes for up to 15 port forwardings. You will be able to access to your server through these ports.</li><li>Under “Customize your installation”, select “Ubuntu 20.04 LTS”.</li></ul><ol><li>Choose a password for your server, and give it a name, e.g., “stable-diffusion-api”</li><li>Hit “Deploy Server” and voila! Your server will be ready in seconds.</li><li>When you see the success page, click “Next” to see the details.</li><li>Find IPv4 address of your server. This may be an real IP or a subdomain like mass-a.tensordockmarketplace.com.</li><li>Find the external port mapped to internal port 22. You will use this one to SSH into your server. It might be something like 20029, for example.</li><li>Connect to your server using these cridentials, e.g.:</li></ol><ul><li>ssh -p 20029 user@mass-a@tensordockmarketplace.com</li></ul><p>Docker is already configured for GPU access, but we need to configure Docker networking to make external requests.</p><ol><li>Clone this repository and cd into it:</li></ol><ul><li>git clone https://github.com/monatis/stable-diffusion-tf-docker.git &amp;&amp; cd stable-diffusion-tf-docker</li></ul><ol><li>Copy daemon.json over the existing /etc/docker/daemon.json and restart the service. Don&#39;t worry --this only adds a setting for the MTU value.</li></ol><ul><li>sudo cp ./daemon.json /etc/docker/daemon.json</li><li>sudo systemctl restart docker.service</li></ul><ol><li>Set an environment variable for the public port that you want to use, run Docker Compose. Our docker-compose.yml file will pick it up from environment variables, and it should be one of the port forwardings you configured, e.g., 20020.</li></ol><ul><li>export PUBLIC_PORT=20020</li><li>docker compose up -d</li></ul><ol><li>Once it’s up and running, go to http://mass-a.tensordockmarketplace.com:20020/docs for the Swagger UI provided by FastAPI.</li><li>Use the POST /generate endpoint to generate images with Stable Diffusion. It will respond with a download ID.</li><li>Hit the GET /download/&lt;download_id&gt; endpoint to download your image.</li></ol><h4>Conclusion</h4><p>We were able to run Stable Diffusion, one of the state-of-the-art tex-to-image models, on a cloud GPU by TensorDock Marketplace. It’s very affordable and thus suitable for experimentation and side projects. I’m continuing to use it for one of my side projects, and a subsequent post will provide a step-by-step walkthrough of a training job on TensorDock Marketplace.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd88cfe41d8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/launch-a-web-api-for-stable-diffusion-under-45-seconds-bbd88cfe41d8">Launch a web API for Stable Diffusion under 45 seconds</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Metric Learning for Anomaly Detection]]></title>
            <link>https://medium.com/data-science/metric-learning-for-anomaly-detection-7bde550cfa56?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/7bde550cfa56</guid>
            <category><![CDATA[similarity-learning]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[quality-assessment]]></category>
            <category><![CDATA[metric-learning]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Tue, 17 May 2022 15:23:06 GMT</pubDate>
            <atom:updated>2022-05-17T15:23:06.140Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to use metric learning to detect anomalies: quality assessment of coffee beans with just 200 labelled samples</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*5OaUgrDJcda9r3CrW8-yDg.png" /><figcaption>Image by author.</figcaption></figure><p>Anomaly detection is a thirsting yet challenging task that has numerous use cases across various industries. The complexity results mainly from the fact that the task is data-scarce by definition.</p><p>Similarly, anomalies are, again by definition, subject to frequent change, and they may take unexpected forms. For that reason, supervised classification-based approaches are:</p><ul><li>Data-hungry — requiring quite a number of labeled data;</li><li>Expensive — data labeling is an expensive task itself;</li><li>Time-consuming — you would try to obtain what is necessarily scarce;</li><li>Hard to maintain — you would need to re-train the model repeatedly in response to changes in the data distribution.</li></ul><p>These are not desirable features if you want to put your model into production in a rapidly-changing environment. And, despite all the mentioned difficulties, they do not necessarily offer superior performance compared to the alternatives. In this post, we will detail the lessons learned from such a use case.</p><h4>Coffee Beans</h4><p><a href="https://agrivero.ai/">Agrivero.ai</a> — is a company making AI-enabled solution for quality control &amp; traceability of green coffee for producers, traders, and roasters. They have collected and labeled more than <strong>30 thousand</strong> images of coffee beans with various defects — wet, broken, chipped, or bug-infested samples. This data is used to train a classifier that evaluates crop quality and highlights possible problems.</p><figure><img alt="Anomalies in coffee" src="https://cdn-images-1.medium.com/proxy/1*akKOEtWmY62-ZqBaE-qHTw.gif" /><figcaption>Anomalies in coffee. Image by author.</figcaption></figure><p>We should note that anomalies are very diverse, so the enumeration of all possible anomalies is a challenging task on it’s own. In the course of work, new types of defects appear, and shooting conditions change. Thus, a one-time labeled dataset becomes insufficient.</p><p>Let’s find out how metric learning might help to address this challenge.</p><h4>Metric Learning Approach</h4><p>In this approach, we aimed to encode images in an n-dimensional vector space and then use learned similarities to label images during the inference.</p><p>The simplest way to do this is KNN classification. The algorithm retrieves K-nearest neighbors to a given query vector and assigns a label based on the majority vote.</p><p>In production environment kNN classifier could be easily replaced with <a href="https://qdrant.tech/">Qdrant</a> vector search engine.</p><figure><img alt="Production deployment" src="https://cdn-images-1.medium.com/proxy/1*l_8CVIRwKiTKfgglfAuMqw.png" /><figcaption>Production deployment. Image by author.</figcaption></figure><p>This approach has the following advantages:</p><ul><li>We can benefit from unlabeled data, considering labeling is time-consuming and expensive.</li><li>The relevant metric, e.g., precision or recall, can be tuned according to changing requirements during the inference without re-training.</li><li>Queries labeled with a high score can be added to the KNN classifier on the fly as new data points.</li></ul><p>To apply metric learning, we need to have a neural encoder, a model capable of transforming an image into a vector.</p><p>Training such an encoder from scratch may require a significant amount of data we might not have. Therefore, we will divide the training into two steps:</p><ul><li>The first step is to train the autoencoder, with which we will prepare a model capable of representing the target domain.</li><li>The second step is finetuning. Its purpose is to train the model to distinguish the required types of anomalies.</li></ul><figure><img alt="Model training
architecture" src="https://cdn-images-1.medium.com/proxy/1*X3esbgoXcxs6xgG9-5FV-A.png" /><figcaption>Model training architecture. Image by author.</figcaption></figure><h4>Step 1 — Autoencoder for Unlabeled Data</h4><p>First, we pretrained a Resnet18-like model in a vanilla autoencoder architecture by leaving the labels aside. Autoencoder is a model architecture composed of an encoder and a decoder, with the latter trying to recreate the original input from the low-dimensional bottleneck output of the former.</p><p>There is no intuitive evaluation metric to indicate the performance in this setup, but we can evaluate the success by examining the recreated samples visually.</p><figure><img alt="Example of image reconstruction with
AutoEncoder" src="https://cdn-images-1.medium.com/proxy/1*kZiLbZbWqQv4IJSP0kOTjQ.png" /><figcaption>Example of image reconstruction with autoencoder model. Image by author.</figcaption></figure><p>Then we encoded a subset of the data into 128-dimensional vectors by using the encoder, and created a KNN classifier on top of these embeddings and associated labels.</p><p>Although the results are promising, we can do even better by finetuning with metric learning.</p><h4>Step 2 — Finetuning with Metric Learning</h4><p>We started by selecting 200 labeled samples randomly without replacement.</p><p>In this step, The model was composed of the encoder part of the autoencoder with a randomly initialized projection layer stacked on top of it. We applied transfer learning from the frozen encoder and trained only the projection layer with Triplet Loss and an online batch-all triplet mining strategy.</p><p>Unfortunately, the model overfitted quickly in this attempt. In the next experiment, we used an online batch-hard strategy with a trick to prevent vector space from collapsing. We will describe our approach in the further articles.</p><p>This time it converged smoothly, and our evaluation metrics also improved considerably to match the supervised classification approach.</p><figure><img alt="Metrics for the autoencoder model with KNN
clasifier" src="https://cdn-images-1.medium.com/proxy/1*uH-3MhZW3siZB5xS296weg.png" /><figcaption>Metric for the autoencoder model with KNN classifier. Image by author.</figcaption></figure><figure><img alt="Metrics for the finetuned model with KNN
classifier" src="https://cdn-images-1.medium.com/proxy/1*sGWINhuVKueemxzb9dT7XQ.png" /><figcaption>Metric for the fine-tuned model with KNN classifier. Image by author.</figcaption></figure><p>We repeated this experiment with 500 and 2000 samples, but it showed only a slight improvement. Thus we decided to stick to 200 samples — see below for why.</p><h4>Supervised Classification Approach</h4><p>We also wanted to compare our results with the metrics of a traditional supervised classification model. For this purpose, a Resnet50 model was finetuned with ~30k labeled images, made available for training. Surprisingly, the F1 score was around ~0.86.</p><p>Please note that we used only 200 labeled samples in the metric learning approach instead of ~30k in the supervised classification approach. These numbers indicate a huge saving with no considerable compromise in the performance.</p><h4>Conclusion</h4><p>We obtained results comparable to those of the supervised classification method by using <strong>only 0.66%</strong> of the labeled data with metric learning. This approach is time-saving and resource-efficient, and that may be improved further. Possible next steps might be:</p><ul><li>Collect more unlabeled data and pretrain a larger autoencoder.</li><li>Obtain high-quality labels for a small number of images instead of tens of thousands for finetuning.</li><li>Use hyperparameter optimization and possibly gradual unfreezing in the finetuning step.</li><li>Use <a href="https://github.com/qdrant/qdrant">vector search engine</a> to serve Metric Learning in production.</li></ul><p>We are actively looking into these, and we will continue to publish our findings in this challenge and other use cases of metric learning.</p><p>If you want to enjoy such interesting discussion about metric learning, similarity learning and vector search, come join us at Discord:</p><p><a href="https://discord.qdrant.tech/">Join the qdrant Discord Server!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7bde550cfa56" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/metric-learning-for-anomaly-detection-7bde550cfa56">Metric Learning for Anomaly Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Triplet Loss — Advanced Intro]]></title>
            <link>https://medium.com/data-science/triplet-loss-advanced-intro-49a07b7d8905?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/49a07b7d8905</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[metric-learning]]></category>
            <category><![CDATA[similarity-learning]]></category>
            <category><![CDATA[contrastive-loss]]></category>
            <category><![CDATA[triplet-loss]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 19:40:38 GMT</pubDate>
            <atom:updated>2022-03-25T09:29:43.201Z</atom:updated>
            <content:encoded><![CDATA[<h3>Triplet Loss — Advanced Intro</h3><h4>What are the advantages of Triplet Loss over Contrastive loss, and how to efficiently implement it?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*Evpi9IqvPVRqqv9HEVLitg.png" /><figcaption>Paths followed by moving points under Triplet Loss. Image by author.</figcaption></figure><p>Triplet Loss was first introduced in <a href="https://arxiv.org/abs/1503.03832">FaceNet: A Unified Embedding for Face Recognition and Clustering</a> in 2015, and it has been one of the most popular loss functions for supervised similarity or metric learning ever since. In its simplest explanation, Triplet Loss encourages that dissimilar pairs be distant from any similar pairs by at least a certain margin value. Mathematically, the loss value can be calculated as L=max(d(a, p) - d(a, n) + m, 0), where:</p><ul><li>p, i.e., positive, is a sample that has the same label as a, i.e., anchor,</li><li>n, i.e., negative, is another sample that has a label different from a,</li><li>d is a function to measure the distance between these three samples,</li><li>and m is a margin value to keep negative samples far apart.</li></ul><p>The paper uses Euclidean distance, but it is equally valid to use any other distance metric, e.g., cosine distance.</p><p>The function has a learning objective that can be visualized as in the following:</p><figure><img alt="Triplet Loss objective. Image by author." src="https://cdn-images-1.medium.com/max/977/1*ADu4F-SoI-cLqyO0IqqBrQ.png" /><figcaption>Triplet Loss objective. Image by author</figcaption></figure><p>Notice that Triplet Loss does not have a side effect of urging to encode anchor and positive samples into the same point in the vector space as in Contrastive Loss. This lets Triplet Loss tolerate some intra-class variance, unlike Contrastive Loss, as the latter forces the distance between an anchor and any positive essentially to 0. In other terms, Triplet Loss allows to stretch clusters in such a way as to include outliers while still ensuring a margin between samples from different clusters, e.g., negative pairs.</p><p>Additionally, Triplet Loss is less greedy. Unlike Contrastive Loss, it is already satisfied when different samples are easily distinguishable from similar ones. It does not change the distances in a positive cluster if there is no interference from negative examples. This is due to the fact that Triplet Loss tries to ensure a margin between distances of negative pairs and distances of positive pairs. However, Contrastive Loss takes into account the margin value only when comparing dissimilar pairs, and it does not care at all where similar pairs are at that moment. This means that Contrastive Loss may reach a local minimum earlier, while Triplet Loss may continue to organize the vector space in a better state.</p><p>Let’s demonstrate how two loss functions organize the vector space by animations. For simpler visualization, the vectors are represented by points in a 2-dimensional space, and they are selected randomly from a normal distribution.</p><figure><img alt="Animation that shows how Contrastive Loss moves points in the course of training. Image by author." src="https://cdn-images-1.medium.com/max/432/0*9ouGHpVpgmNy1Nrv.gif" /><figcaption>Animation that shows how Contrastive Loss moves points in the course of training. Image by author.</figcaption></figure><figure><img alt="Animation that shows how Triplet Loss moves points in the course of training." src="https://cdn-images-1.medium.com/max/432/0*tn3xh9wT3E9uyonE.gif" /><figcaption>Animation that shows how Triplet Loss moves points in the course of training. Image by author.</figcaption></figure><p>From mathematical interpretations of the two-loss functions, it is clear that Triplet Loss is theoretically stronger, but Triplet Loss has additional tricks that help it work better. Most importantly, Triplet Loss introduce online triplet mining strategies, e.g., automatically forming the most useful triplets.</p><h4>Why triplet mining matters?</h4><p>The formulation of Triplet Loss demonstrates that it works on three objects at a time:</p><ol><li>anchor,</li><li>positive - a sample that has the same label as the anchor,</li><li>and negative - a sample with a different label from the anchor and the positive.</li></ol><p>In a naive implementation, we could form such triplets of samples at the beginning of each epoch and then feed batches of such triplets to the model throughout that epoch. This is called “offline strategy.” However, this would not be so efficient for several reasons:</p><ul><li>It needs to pass 3n samples to get a loss value of n triplets.</li><li>Not all these triplets will be useful for the model to learn anything, e.g., yielding a positive loss value.</li><li>Even if we form “useful” triplets at the beginning of each epoch with one of the methods that I will be implementing in this series, they may become “useless” at some point in the epoch as the model weights will be constantly updated.</li></ul><p>Instead, we can get a batch of n samples and their associated labels, and form triplets on the fly. That is called &quot;online strategy.&quot; Normally, this gives n^3 possible triplets, but only a subset of such possible triplets will be actually valid. Even in this case, we will have a loss value calculated from much more triplets than the offline strategy.</p><p>Given a triplet of (a, p, n), it is valid only if:</p><ol><li>a and p has the same label,</li><li>a and p are distinct samples,</li><li>and n has a different label from a and p.</li></ol><p>These constraints may seem to be requiring expensive computation with nested loops, but it can be efficiently implemented with tricks such as distance matrix, masking, and broadcasting. The rest of this series will focus on the implementation of these tricks.</p><h4>Distance matrix</h4><p>A distance matrix is a matrix of shape (n, n) to hold distance values between all possible pairs made from items in two n-sized collections. This matrix can be used to vectorize calculations that would need inefficient loops otherwise. Its calculation can be optimized as well, and we will implement <a href="https://www.robots.ox.ac.uk/~albanie/notes/Euclidean_distance_trick.pdf">Euclidean Distance Matrix Trick (PDF)</a> explained by Samuel Albanie. You may want to read this three-page document for the full intuition of the trick, but a brief explanation is as follows:</p><ol><li>Calculate the dot product of two collections of vectors, e.g., embeddings in our case.</li><li>Extract the diagonal from this matrix that holds the squared Euclidean norm of each embedding.</li><li>Calculate the squared Euclidean distance matrix based on the following equation: ||a - b||^2 = ||a||^2 - 2 ⟨a, b⟩ + ||b||^2</li><li>Get the square root of this matrix for non-squared distances.</li></ol><p>We will implement it in PyTorch, so let’s start with imports.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7f0afb8fcc0d836e1591d7219235e77/href">https://medium.com/media/d7f0afb8fcc0d836e1591d7219235e77/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e0bdacd7f3235062e06901b92b44c3a7/href">https://medium.com/media/e0bdacd7f3235062e06901b92b44c3a7/href</a></iframe><h4>Invalid triplet masking</h4><p>Now that we can compute a distance matrix for all possible pairs of embeddings in a batch, we can apply broadcasting to enumerate distance differences for all possible triplets and represent them in a tensor of shape (batch_size, batch_size, batch_size). However, only a subset of these n^3 triplets are actually valid as I mentioned earlier, and we need a corresponding mask to compute the loss value correctly. We will implement such a helper function in three steps:</p><ol><li>Compute a mask for distinct indices, e.g., (i != j and j != k).</li><li>Compute a mask for valid anchor-positive-negative triplets, e.g., labels[i] == labels[j] and labels[j] != labels[k].</li><li>Combine two masks.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5369d155bb5426912a25d6f1a941281/href">https://medium.com/media/e5369d155bb5426912a25d6f1a941281/href</a></iframe><h4>Batch-all strategy for online triplet mining</h4><p>Now we are ready for actually implementing Triplet Loss itself. Triplet Loss involves several strategies to form or select triplets, and the simplest one is to use all valid triplets that can be formed from samples in a batch. This can be achieved in four easy steps thanks to utility functions we’ve already implemented:</p><ol><li>Get a distance matrix of all possible pairs that can be formed from embeddings in a batch.</li><li>Apply broadcasting to this matrix to compute loss values for all possible triplets.</li><li>Set loss values of invalid or easy triplets to 0.</li><li>Average the remaining positive values to return a scalar loss.</li></ol><p>I will start by implementing this strategy, and more complex ones will follow as separate posts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d9e450c1577009a7027b50512d54e78/href">https://medium.com/media/6d9e450c1577009a7027b50512d54e78/href</a></iframe><h4>Conclusion</h4><p>I mentioned that Triplet Loss is different from Contrastive Loss not only mathematically but also in its sample selection strategies, and I implemented the batch-all strategy for online triplet mining in this post efficiently by using several tricks. There are other more complicated strategies such as batch-hard and batch-semihard mining, but their implementations, and discussions of the tricks I used for efficiency in this post, are worth separate posts of their own. The future posts will cover such topics and additional discussions on some tricks to avoid vector collapsing and control intra-class and inter-class variance. Meanwhile, you can join Qdrant’s Discord server to discuss, learn more and ask questions on metric learning.</p><p><a href="https://discord.com/invite/tdtYvXjC4h">Join the qdrant Discord Server!</a></p><p>Originally posted at <a href="https://qdrant.tech/articles/triplet-loss/">https://qdrant.tech/articles/triplet-loss/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49a07b7d8905" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/triplet-loss-advanced-intro-49a07b7d8905">Triplet Loss — Advanced Intro</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transforming an occupation of 2500 years with AI]]></title>
            <link>https://medium.com/@yusufsarigoz/transforming-an-occupation-of-2500-years-with-ai-3c4b65c122ab?source=rss-e2732a62029c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c4b65c122ab</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[semantic-search]]></category>
            <category><![CDATA[legaltech]]></category>
            <dc:creator><![CDATA[Yusuf Sarıgöz]]></dc:creator>
            <pubDate>Thu, 02 Dec 2021 14:22:19 GMT</pubDate>
            <atom:updated>2021-12-16T12:04:37.695Z</atom:updated>
            <content:encoded><![CDATA[<h3>How can AI help lawyers with one of the most traditional and regulated occupations in the world?</h3><figure><img alt="A bearded immigrant appearing before a board of inquiry, Ellis Island." src="https://cdn-images-1.medium.com/max/1024/1*KO7h_wej5WbqGEyZbOIUxA.jpeg" /></figure><p>The history of lawyers dates back to ancient Athens, and it has been regulated as an occupation since ancient Rome. In almost all jurisdictions, lawyers enjoy a monopoly right to plead and practice in courts of law. Therefore, it is common that their work may not be well understood by those outside the legal domain. Apart from appearing and pleading in courts, they also need to conduct comprehensive research to locate relevant information when preparing court papers and pleadings. Traditionally, they make use of standard boolean search tools to do so. However, boolean search does not work well with the massive amount of data found in the legal domain. They may need to continue searching for multiple hours or even days. Can AI help them with their repetitive tasks?</p><h4>Deep metric learning to the rescue</h4><p>Deep metric learning aims at designing machine learning algorithms to learn a distance function that can be used to measure how similar, or related, the entities of a particular type are to one another. Once a metric learning model is trained, its output vectors can be employed to compare, classify, cluster, or retrieve similar entities. It has extensive applications in ranking, recommendation systems, cross-modal and/or semantic search.</p><p>Semantic search makes it possible to unearth meaningful relationships between entities as perceived by humans. Unlike boolean search, which essentially operates at the bit level, semantic search can relate more abstract relations such as the meaning expressed in a sentence or paragraph as a whole. I’ve recently consulted with a Turkish startup named De Jure to help them develop a semantic search solution for the legal domain powering an “integrated writing environment” — yes, it’s like IDE for lawyers. As the simplest use case, lawyers will be presented with n previous court decisions semantically most similar to the current paragraph they are editing, and it will be automatically added to the document with proper formatting when they choose one of the suggested decisions. I will try to detail the lessons learned from this experience.</p><p><a href="https://home.e-dejure.com/">De Jure</a></p><h4>Lessons from developing a semantic search solution in the real world</h4><p>The versatility of deep metric learning makes it quite powerful, but things get complicated when you want to deploy them for use in the real world. We need to have special considerations for the following:</p><ul><li>A semantic text encoder model,</li><li>a vector index to store text vectors for queries later on,</li><li>and another storage solution to relate metadata with vectors.</li></ul><h4>Semantic text encoder for the legal domain</h4><p>The journey starts with a specially trained language model to properly encode legal texts into vectors. Unfortunately, Transformer models, which give state-of-the-art results in many natural language understanding generation tasks, cannot be directly used for semantic search. They typically output per-token vectors; however, we need to be able to represent a whole sentence or paragraph of different lengths, so we have to come up with an aggregation function to obtain a single vector per sequence. For this purpose, I finetuned a <a href="https://huggingface.co/mys/bert-base-turkish-cased-nli-mean">BERT model in Turkish</a> with a natural language inference (NLI) task. NLI data typically consists of sentence pairs and their associated labels as entailment, contradiction, and neutral specifying the semantic relationship of the sentences in that particular pair. The pre-trained model&#39;s per-token vectors are aggregated with a mean function respecting attention masks, and it is finetuned with a loss function to minimize the negative log-likelihood of the ground truth label. This finetuning task helps obtain averaged vectors better representing the whole sequence.</p><p>However, we can do even better by directly enforcing our distance metric to the model. For this purpose, I selected the pairs with entailment as the label and used them to further finetune the model with a Multiple Negatives Ranking (MNR) loss function. In MNR finetuning, we calculate cosine similarities of paired sentences in a batch and use them as logits. Then, we generate pseudo-labels as 1 for similarities of two sentences in a pair and 0 for similarities of sentences in different pairs --hence it&#39;s called multiple negatives as we use all the sentences in other pairs in a batch as negative samples. Now it is easy to calculate cross-entropy with these logits and labels.</p><p>The story does not end there: the current model is finetuned with texts in a more casual registry, but legal texts are infamous for their heavy use of some domain-specific jargon. So, the model needs another finetuning with such texts. Unfortunately, it is difficult to gather paired sentences as in NLI or MNR. However, we can use a denoising autoencoder loss function by forcing the model to reconstruct the original one given a corrupted version of passages, and it does the job indeed.</p><h4>Challenges of vector indexing and querying</h4><p>Searching with vector similarity seems to be intuitive in the first place, but it may require a considerable amount of effort to scale it properly. Comparison of a high-dimensional query vector with tens of millions of vectors in the index may add a serious latency to your application if you want to respond to search requests in an online manner. There exist several approximate nearest neighbor (ANN) search algorithms implemented in various libraries such as <a href="https://github.com/nmslib/hnswlib">hnswlib</a>, <a href="https://github.com/nmslib/nmslib">nmslib</a>, <a href="https://github.com/spotify/annoy">Annoy</a>, and <a href="https://github.com/facebookresearch/faiss">Faiss</a>. They all do a great job to reduce the time complexity to sub-linear, but they bring their challenges such as managing index files and moving them around as well as making associations between the indexed vectors and the payload data stored elsewhere — probably in a document database-, and we also need to coordinate query filters between two data stores, and come up with hacky solutions.</p><h4>An efficient solution: Qdrant</h4><p>Luckily, we have recently started to see several vector search databases such as <a href="https://github.com/milvus-io/milvus">Milvus</a> and <a href="https://github.com/qdrant/qdrant">Qdrant</a>. They work as a service and expose an API to interact with, query, and filter vectors and objects. We chose Qdrant in this project for the following reasons among others:</p><ol><li>It’s quite lightweight — only 116 MB.</li><li>It can run as a single container with no dependency on 3rd party services — yes, it stores objects with vectors in itself.</li><li>It can quickly get up and running, and this is important in the development phase.</li></ol><p>Of course, it is still at an early stage and has room for improvement. First, it exposes a JSON API, but JSON is not the best way to send and receive a large array of floating points — a binary protocol would instead do a better job. Second, it makes use of SIMD hardware acceleration, but there is no support for acceleration on other types of hardware. Finally, additional metrics such as Jaccard, superstructure, and substructure supported by Milvus may open door to new opportunities for researchers from a diverse set of domains.</p><h4>Bonus: a fun fact</h4><p>I found the featured image in this post by semantically searching millions of images indexed in Qdrant with a query text expressed in natural language, and the next post will cover how it can be possible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c4b65c122ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>