<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Cyabra - Medium]]></title>
        <description><![CDATA[Cyabra’s AI shields corporations and the public sector by uncovering fake profiles, harmful narratives, and GenAI content. Welcome to the Cyabra Medium page where you can find posts from our engineering team. - Medium]]></description>
        <link>https://medium.com/cyabra?source=rss----c966612e4dbe---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Cyabra - Medium</title>
            <link>https://medium.com/cyabra?source=rss----c966612e4dbe---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:39:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/cyabra" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Entity-Level Synthetic Dataset for Factual Verification]]></title>
            <link>https://medium.com/cyabra/entity-level-synthetic-dataset-for-factual-verification-d82b19cf757a?source=rss----c966612e4dbe---4</link>
            <guid isPermaLink="false">https://medium.com/p/d82b19cf757a</guid>
            <dc:creator><![CDATA[Liza Miller]]></dc:creator>
            <pubDate>Tue, 19 Aug 2025 15:15:11 GMT</pubDate>
            <atom:updated>2025-08-19T15:15:10.876Z</atom:updated>
            <content:encoded><![CDATA[<h4>Designed for evaluation with LLMs as a judge</h4><p><strong><em>LLMs, Entity-Level Evaluation, Factual Verification, Synthetic Data, Fact-Checking</em></strong></p><h3><strong>Abstract</strong></h3><p>Cyabra is building agentic systems that rely on multiple LLMs, often updated or replaced, and because some models are more faithful than others, continuous output validation is essential to maintain credibility. Large language models (LLMs) often produce fluent but factually incorrect content, a problem that can spread misinformation in domains like healthcare, journalism, and education. Despite advances in model design, benchmarks such as the <a href="https://github.com/vectara/hallucination-leaderboard">Vectara Hallucination Leaderboard</a> show hallucinations remain a persistent issue.<br>Factual errors have long challenged natural language generation, from earlier models like BART and T5 to today’s LLMs, making robust verification essential. One promising solution is the LLM-as-a-Judge approach, where models assess factual correctness — but designing effective evaluation prompts is nontrivial.<br>To advance this work, we present a synthetic dataset for testing prompts in entity-rich contexts. It is a full recreation aimed at evaluating LLMs’ ability to distinguish accurate statements from false ones. The dataset is available at <a href="https://huggingface.co/datasets/Cyabra/ag_news_fact_check_with_llm">Cyabra/ag_news_fact_check_with_llm</a></p><h3>1. Motivation: Hallucinations and Misinformation</h3><p>LLMs are powerful but often produce hallucinations — factually incorrect content delivered fluently and confidently. This is dangerous in high-stakes applications. Benchmarks such as the <a href="https://github.com/vectara/hallucination-leaderboard">Vectara Hallucination Leaderboard</a> underscore the urgent need for robust factual verification mechanisms.</p><h3>2. Solution: The LLM-as-a-Judge Paradigm</h3><p>This approach frames LLMs as evaluators of factual accuracy. Applying this method effectively — and exploring limitations through stronger models, alternative prompts, adjusted parameters, or teacher–student training setups — requires a diverse and well-structured dataset. Existing datasets do not align with our specific goals, so we created a synthetic parallel corpus to support researchers in building, evaluating, and refining factuality-focused prompts.</p><h3>3. <strong>Related Work</strong></h3><p><strong>Historical Context: Pre-LLM Era Datasets</strong><br>Before large language models, factual error detection focused on summarization, reading comprehension, and structured-to-text generation. Benchmarks such as <strong>CNN/DailyMail</strong> (Hermann et al., 2015; See et al., 2017), <strong>WikiBio</strong> (Lebret et al., 2016), <strong>MS MARCO</strong> (Nguyen et al., 2016), <strong>WebNLG</strong> (Gardent et al., 2017), <strong>RotoWire</strong> (Wiseman et al., 2017), <strong>XSum</strong> (Narayan et al., 2018), <strong>PersonaChat</strong> (Zhang et al., 2018), and <strong>Wizard of Wikipedia</strong> (Dinan et al., 2019) have been widely used to assess general factual consistency. However, these datasets do not provide explicit mappings between correct facts and their erroneous counterparts, nor do they isolate specific entity-level mistakes. As a result, they capture broad hallucination patterns but cannot support detailed, fine-grained fact<strong> </strong>verification.</p><h3><strong>4. Why Existing Datasets Fall Short</strong></h3><p>More recent hallucination benchmarks (e.g., <strong>HaluEval</strong>, <strong>DefAn</strong>, <strong>HalluDial</strong>, <strong>TRUE</strong>) have shifted toward LLM-generated free-form text and dialogue, but still emphasize overall truthfulness and fluency rather than targeted, fine-grained fact-checking.<br>Our dataset fills this gap by introducing controlled<strong>, </strong>entity-level factual<strong> </strong>perturbations, enabling precise evaluation of how models handle specific incorrect facts. This approach extends beyond summarization into QA, dialogue, and other generation tasks, making it a more direct tool for fact-check research than earlier broad-consistency datasets.</p><h3><strong>5. </strong>Dataset Construction: Challenges, Advantages, and Error Types</h3><p><strong>Challenges in Synthetic Dataset Creation:</strong> Creating high-quality synthetic datasets using only NER-based methods like <a href="https://spacy.io/">spaCy</a> or <a href="https://github.com/amazon-science/ReFinED">ReFinED</a> presents several challenges. While these tools detect entities and perform simple replacements, they often fail to maintain grammaticality, sentence flow, and semantic coherence. NER-only replacements struggle with complex constructions such as coordinated phrases, subordinate clauses, or idiomatic expressions. They cannot ensure domain consistency, frequently swapping entities across unrelated sectors or roles (e.g., replacing a politician with a celebrity or a tech company with a bank). Additionally, maintaining pronoun and coreference agreement, verb tense, and article usage is difficult without contextual understanding.</p><p><strong>Advantages of LLMs</strong>: LLMs, while not error-free, excel at preserving grammar, style, and domain expertise, producing synthetic data that is contextually plausible, fluent, and higher quality than naive NER-based replacements. They ensure that entity swaps are contextually and semantically appropriate and maintain sentence- and discourse-level coherence. A subset of the dataset was manually validated, although some noise still persists. Cases where the augmentation type is very specific — for example, Temporal — may contain replaced entities that are not DATE-related and can be further filtered if needed.</p><p><strong>Factual Error Types and Focus of Synthetic Dataset</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vnehw3zAOsbH2Gk6YTjkgA.png" /></figure><h3>6. Transformation Strategies</h3><p>We used a subset of the <a href="https://huggingface.co/datasets/SetFit/ag_news">SetFit/AG_News</a> dataset as the base. For each type of factual error and paraphrasing, we created a specific LLM prompt to generate the augmentation while preserving grammar, style, and domain coherence. Original texts (text_snippet1) were augmented using these custom prompts and OpenAI model gpt-4o-mini . This approach allows controlled manipulation of entity, number, date, and other information while maintaining sentence-level and discourse-level coherence.</p><h3><strong>7. Data Stats</strong></h3><ul><li><strong>Total number of text snippets used:</strong> 1,878 (initial subset)</li><li><strong>Total Final Dataset:</strong> 2,533 samples</li><li>similar: 182</li><li>not_similar: 2,351</li></ul><p>The dataset is primarily designed for optimizing prompts for LLM-as-a-Judge. More emphasis was placed on covering different types of factual errors rather than balancing the dataset, which can be relatively easily achieved via paraphrasing, aliasing, or adding unchanged sentences to the existing dataset.</p><h4><strong>7.1 Token Count Distribution Stats</strong></h4><p>Text tokens were counted using a simple space-based tokenization approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4YtpAJSywE3n9_3HqUijxg.png" /></figure><h4><strong>7.2 NER Distributions</strong></h4><p>The following table presents NER replacements in the synthetic dataset. To produce aggregated statistics, compare entity distributions across error types, and ensure consistent labeling in synthetic augmentations, we mapped a diverse LLM NER outputs to canonical spaCy-style NERs. For details, see <strong>Appendix A</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yh-h1DRK0SmxGm8n84f3sw.png" /></figure><h4><strong>7.3 Step-wise Augmentations</strong></h4><p>Fist we applied a general prompt that replaced any first entity detected in the text. After initial general replacement, further augmentations were applied with specific focus on entity types and error categories using LLM prompts designed per error type. The counts for these augmentations are as follows:</p><ul><li>MONEY: 81 examples</li><li>CARDINAL: 42 examples</li><li>PERCENT: 23 examples</li><li>DATE: 227 examples</li><li>Negation: 100 examples</li><li>NER alias paraphrased texts: 82 examples (e.g., “Donald Trump” → “Trump”)</li><li>Paraphrased texts (general): 100 examples</li></ul><h4>7.4 Data Structure and Access</h4><p><strong>Data Structure</strong></p><pre>{<br>  &quot;text_snippet1&quot;: &quot;Original text snippet.&quot;,<br>  &quot;text_snippet2&quot;: &quot;Augmented or paraphrased text snippet.&quot;,<br>  &quot;similarity_type&quot;: &quot;Label indicating whether the snippets are &#39;similar&#39; or &#39;not_similar&#39;.&quot;,<br>  &quot;detector_origin&quot;: &quot;Source of entity detection&quot;,<br>  &quot;replacement_tool&quot;: &quot;Tool used for augmentation/replacement: e.g., &#39;LLM&#39;, &#39;WikidataEntityReplacer&#39;, &#39;PythonDateReplacer&#39;, &#39;without changes&#39;.&quot;,<br>  &quot;detected_entities&quot;: [&quot;List of original entities detected in text_snippet1.&quot;],<br>  &quot;replaced_entities&quot;: [&quot;List of corresponding entities replaced in text_snippet2.&quot;],<br>  &quot;detected_entities_types&quot;: [&quot; List of NER types of detected entities in text_snippet1.&quot;],<br>  &quot;detected_replaced_pairs&quot;: [<br>    {&quot;detected&quot;: &quot;Original entity&quot;, &quot;replaced&quot;: &quot;Augmented entity&quot;}<br>  ]<br>,<br>  &quot;type&quot;: &quot;Type of augmentation or factual error (e.g., &#39;Entity&#39;, &#39;Negation&#39;, &#39;Temporal&#39;,&#39;Money&#39;, &#39;Number&#39;, &#39;Percent&#39;, &#39;Paraphrase&#39;, &#39;Alias&#39;).&quot;<br>}</pre><p><strong>Access:</strong></p><pre>from datasets import load_dataset<br><br>dataset = load_dataset(<br>    &quot;Cyabra/ag_news_fact_check_with_llm&quot;<br>)<br>print(dataset)             # Overview of dataset splits<br>print(dataset[&quot;train&quot;][:5])  # First 5 rows for preview</pre><h3><strong>8. Conclusion and Future Directions</strong></h3><p>This dataset supports research in prompt engineering, evaluator robustness, and model alignment within the LLM-as-a-Judge paradigm. Each augmentation technique came with its own challenges: negation required careful preservation of meaning while reversing intent, aliasing demanded nuanced substitutions that remained plausible, and even strict date replacements occasionally triggered unintended changes in other entity types. These challenges required careful result analysis and iterative prompt engineering improvements to ensure the dataset’s integrity and diversity. Future work includes expanding to real-world examples, adding multilingual coverage, and benchmarking across domains and tasks. A follow-up post will demonstrate practical applications of LLM-as-a-Judge.</p><h3><strong>Acknowledgments</strong></h3><p>Special thanks to our Data Analyst ,<a href="http://www.linkedin.com/in/vlada-finkelson-paulauskiene-183192275">Vlada Finkelson Paulauskiene</a>, for her tremendous effort in creating the initial spaCy-based dataset, which provided a better understanding of the challenges and paved the way for the improved LLM-based version.</p><h3><strong>References</strong></h3><p>[1] Vectara. (n.d.). <strong>Hallucination Leaderboard.</strong> GitHub. <a href="https://github.com/vectara/hallucination-leaderboard">https://github.com/vectara/hallucination-leaderboard</a> <br>[2] Chen, C., &amp; Shu, K. (2023). <strong>Combating misinformation in the age of LLMs</strong>. arXiv:2311.05656. <a href="https://arxiv.org/abs/2311.05656">https://arxiv.org/abs/2311.05656 </a><br>[3] Setty, R., &amp; Setty, V. (2023). <strong>QuestGen for Fact-Checking Applications</strong>. arXiv:2407.21441. <a href="https://arxiv.org/abs/2407.21441">https://arxiv.org/abs/2407.21441</a> <br>[4] Xu, R., et al. (2023). HaluEval: Benchmark for Evaluating Hallucination in LLMs. <a href="https://arxiv.org/abs/2305.11747">https://arxiv.org/abs/2305.11747</a> <br>[5] Zhou, W., et al. (2024). DefAn: Definitive Answer Benchmark. <a href="https://arxiv.org/abs/2406.09155">https://arxiv.org/abs/2406.09155</a> <br>[6] Wang, Z., et al. (2024). HalluDial: Dialogue-Level Hallucination Detection. <a href="https://arxiv.org/abs/2406.07070">https://arxiv.org/abs/2406.07070</a> <br>[7] Laban, O., et al. (2021). TRUE: Factual Consistency in Summarization. <a href="https://arxiv.org/abs/2204.04991">https://arxiv.org/abs/2101.10686</a></p><p><strong>Datasets and foundational papers</strong></p><p>[8] Hermann, K. M., et al. (2015). Teaching Machines to Read and Comprehend. Advances in Neural Information Processing Systems. <a href="https://arxiv.org/abs/1506.03340">https://arxiv.org/abs/1506.03340</a><br>[9] See, A., Liu, P. J., &amp; Manning, C. D. (2017). Get To The Point: Summarization with Pointer-Generator Networks. <a href="https://arxiv.org/abs/1704.04368">https://arxiv.org/abs/1704.04368</a><br>[10] Zhang, X., Zhao, J., &amp; LeCun, Y. (2015). Character-level CNNs for Text Classification. <a href="https://arxiv.org/abs/1509.01626">https://arxiv.org/abs/1509.01626</a><br>[11] Lebret, R., Grangier, D., &amp; Auli, M. (2016). Neural Text Generation from Structured Data with Application to the Biography Domain. <a href="https://arxiv.org/abs/1603.07771">https://arxiv.org/abs/1611.09327</a> <br>[12] Nguyen, T., et al. (2016). MS MARCO: A Human Generated Machine Reading Comprehension Dataset. <a href="https://arxiv.org/abs/1611.09268">https://arxiv.org/abs/1611.09268</a> <br>[13] Gardent, C., et al. (2017). The WebNLG Challenge: Generating Text from RDF Data. <a href="https://aclanthology.org/W17-3518.pdf">https://aclanthology.org/E17-1028.pdf</a> <br>[14] Wiseman, S., et al. (2017). Challenges in Data-to-Document Generation. <a href="https://arxiv.org/abs/1707.08052">https://arxiv.org/abs/1707.08052</a> <br>[15] Narayan, S., Cohen, S. B., &amp; Lapata, M. (2018). Don’t Give Me the Details, Just the Summary! <a href="https://arxiv.org/abs/1808.08745">https://arxiv.org/abs/1808.08745</a> <br>[16] Zhang, S., Dinan, E., et al. (2018). Personalizing Dialogue Agents: I have a dog, do you have pets too? <a href="https://arxiv.org/abs/1801.07243">https://arxiv.org/abs/1801.07243</a> <br>[17] Dinan, E., et al. (2019). Wizard of Wikipedia: Knowledge-Powered Conversational Agents. <a href="https://arxiv.org/abs/1811.01241">https://arxiv.org/abs/1811.01241</a></p><p><strong>Tools and libraries</strong></p><p>[18] Zaporojets, K., et al. (2021). Entity Linking via ReFinED. GitHub. <a href="https://github.com/Babelscape/refined">https://github.com/Babelscape/refined</a><br> [19] Explosion AI. spaCy: Industrial-strength NLP in Python. <a href="https://spacy.io/">https://spacy.io/</a><br> [20] OpenAI. GPT-4o-mini API: Large language model for text generation, reasoning, and factual verification. <a href="https://platform.openai.com/docs/models/gpt-4o-mini">https://platform.openai.com/docs/models/gpt-4o-mini</a></p><h3><strong>Appendix A: NER Mapping and Additional Statistics</strong></h3><h4><strong>Mapping of LLM NER Types to Canonical spaCy-Style NERs</strong></h4><p>The following mapping was used in order to standardize a diverse LLM NER outputs to canonical spaCy-style NERs. It is used in the dataset to produce aggregated statistics, compare entity distributions across error types, and ensure consistent labeling in synthetic augmentations.</p><pre>mapping = {<br>&quot;ORG&quot;: &quot;ORG&quot;,<br>&quot;ORGANIZATION&quot;: &quot;ORG&quot;,<br>&quot;GPE&quot;: &quot;GPE&quot;,<br>&quot;LOC&quot;: &quot;LOC&quot;,<br>&quot;LOCATION&quot;: &quot;LOC&quot;,<br>&quot;PERSON&quot;: &quot;PERSON&quot;,<br>&quot;PER&quot;: &quot;PERSON&quot;,<br>&quot;DATE&quot;: &quot;DATE&quot;,<br>&quot;TIME&quot;: &quot;TIME&quot;,<br>&quot;MONEY&quot;: &quot;MONEY&quot;,<br>&quot;CURRENCY&quot;: &quot;MONEY&quot;,<br>&quot;PERCENT&quot;: &quot;PERCENT&quot;,<br>&quot;CARDINAL&quot;: &quot;CARDINAL&quot;,<br>&quot;ORDINAL&quot;: &quot;ORDINAL&quot;,<br>&quot;PRODUCT&quot;: &quot;PRODUCT&quot;,<br>&quot;EVENT&quot;: &quot;EVENT&quot;,<br>&quot;WORK_OF_ART&quot;: &quot;WORK_OF_ART&quot;,<br>&quot;LANGUAGE&quot;: &quot;LANGUAGE&quot;,<br>&quot;NORP&quot;: &quot;NORP&quot;,<br>&quot;FAC&quot;: &quot;FAC&quot;,<br># Everything else -&gt; MISC<br>&quot;DISEASE&quot;: &quot;MISC&quot;,<br>&quot;TECH&quot;: &quot;MISC&quot;,<br>&quot;TECHNOLOGY&quot;: &quot;MISC&quot;,<br>&quot;DATA&quot;: &quot;MISC&quot;,<br>&quot;ARCH&quot;: &quot;MISC&quot;,<br>&quot;AGE&quot;: &quot;MISC&quot;,<br>&quot;IP&quot;: &quot;MISC&quot;,<br>&quot;OCCUPATION&quot;: &quot;MISC&quot;,<br>&quot;RELIGION&quot;: &quot;MISC&quot;,<br>&quot;FIELD&quot;: &quot;MISC&quot;,<br>&quot;TITLE&quot;: &quot;MISC&quot;,<br>&quot;MUSIC_GROUP&quot;: &quot;MISC&quot;,<br>&quot;VARIABLE&quot;: &quot;MISC&quot;,<br>&quot;LICENSE&quot;: &quot;MISC&quot;,<br>&quot;TAXON&quot;: &quot;MISC&quot;,<br>&quot;SPECIES&quot;: &quot;MISC&quot;,<br>&quot;URL&quot;: &quot;MISC&quot;,<br>&quot;NUMBER&quot;: &quot;CARDINAL&quot;,<br>&quot;LENGTH&quot;: &quot;QUANTITY&quot;,<br>&quot;QUANTITY&quot;: &quot;QUANTITY&quot;,<br>&quot;ANIMAL&quot;: &quot;MISC&quot;,<br>&quot;VERSION&quot;: &quot;MISC&quot;<br>}</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d82b19cf757a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyabra/entity-level-synthetic-dataset-for-factual-verification-d82b19cf757a">Entity-Level Synthetic Dataset for Factual Verification</a> was originally published in <a href="https://medium.com/cyabra">Cyabra</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Journey to a Better Feature Implementation Flow]]></title>
            <link>https://medium.com/cyabra/a-journey-to-a-better-feature-implementation-flow-3710d626b0bf?source=rss----c966612e4dbe---4</link>
            <guid isPermaLink="false">https://medium.com/p/3710d626b0bf</guid>
            <category><![CDATA[agile-methodology]]></category>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[process-improvement]]></category>
            <category><![CDATA[improvement]]></category>
            <category><![CDATA[workflow-optimization]]></category>
            <dc:creator><![CDATA[Elad Franklin]]></dc:creator>
            <pubDate>Mon, 14 Jul 2025 12:34:20 GMT</pubDate>
            <atom:updated>2025-07-14T07:29:39.023Z</atom:updated>
            <content:encoded><![CDATA[<p>As a Software as a Service (SaaS) company employing Agile Scrum methodologies, we at Cyabra are committed to continuously enhancing our processes and delivering exceptional features to our customers.</p><p>Recently, we identified several challenges within our workflow that were slowing us down. In this article, I will outline the issues we encountered, the solutions we implemented, and the positive outcomes we have observed as a result of our new approach.</p><h3>The Challenges We Faced</h3><p>We had a few key issues in our workflow:</p><ul><li><strong>No Single Source of Truth</strong>: Feature definitions were placed in several different locations with no synchronization: Confluence PRDs, Epic descriptions, and User Story definitions. When the requirements changed, not all sources were updated, leading to confusion and misinformation.</li><li><strong>Not Enough Feature Grooming</strong>: We didn’t spend enough time grooming features, which led to missing some use cases and edge cases. This often made us stop progress during development and to delay shipping to customers.</li><li><strong>Too Big Epics</strong>: We created large epics with many stories, making it hard to oversee the work and have clear visibility of what was left. As a result, it was very hard to complete these epics, which impacted the completeness and quality of the features.</li><li><strong>Hard to Prioritize Work</strong>: Working on large epics made it tough to prioritize tasks, leading to development that didn’t match end-to-end user flows.</li><li><strong>Confusion About Story Creation</strong>: Both the product managers and developers created stories, which caused misalignment in ownership and unsynchronized definitions.</li><li><strong>What should QA Verify</strong>: The QA team had a hard time figuring out which stories needed verification, especially when some stories were just technical tasks for developers.</li><li><strong>Unclear Completion Criteria</strong>: For features that needed both backend and frontend development, it was unclear when everything was done and when QA could check the user flow.</li><li><strong>Frustration Between Teams</strong>: All these issues caused a lot of frustration as features often stalled or got blocked because of miscommunication and missed use cases.</li></ul><h3>How We Fixed These Issues</h3><p>To tackle these challenges, we made some important changes to our workflow:</p><ul><li><strong>Creating the “Feature Structure”:</strong> Each feature is broken down to Initiatives-&gt;Epics-&gt;Stories.<br><strong>Initiative: </strong>A full or partial implementation of the feature that can be released to customers.<br><strong>Epic: </strong>An end-to-end use case that can be tested and demonstrated for feedback.<br><strong>Story: </strong>A technical breakdown of the epic (for example, one story for the UI and another for the backend work).</li><li><strong>Creating a Single Source of Truth</strong>: We decided that the PRD in Confluence would be our single source of truth for feature definitions. Each epic links to specific sections in the PRD (meaning that the epic description only has a link to a section in Confluence), and stories are created only by developers, also linking back to the PRD.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vnFrvybchrlpzbWM" /></figure><ul><li><strong>Clarifying Responsibilities</strong>: PMs create Initiatives and Epics, while developers create Stories. Developers also verify their stories after deploying them to the TEST environment, while QA focus is on verifying the epics.</li><li><strong>Investing in Refinement and Grooming</strong>: We decided to focus more on the refinement and grooming phases. Now, we only mark an initiative or an epic as “Ready For Development” after all the stakeholders (PM, designer, QA, developer etc.) discuss all the use cases. We set up a structured workflow for each initiative/epic with these phases:</li></ul><ol><li><strong>Initiative</strong>: We break down each feature into initiatives.</li><li><strong>Research</strong>: Every initiative needs careful research and validation by Product and UX/UI to understand the “why” and align on the value.</li><li><strong>UX/UI</strong>: Once we have a clear direction, PMs and UI/UX designers create a concept design to share the vision with everyone.</li><li><strong>Kickoff</strong>: This meeting happens only after the PM and UI/UX designer have the definitions of the feature and have a partial PRD and a UI draft. It’s for presenting the planned work to R&amp;D for early feedback.</li><li><strong>Refinement</strong>: After the kickoff, PMs and UI/UX incorporate feedback, finalize epics and UX flows, and prepare for development by aligning Confluence, JIRA, and all stakeholders on scope and design.</li><li><strong>Grooming Sign-off</strong>: This meeting ensures that the Engineering team fully understands the planned epics, confirms that we have created all stories, set estimations, and developed a test plan.</li><li><strong>Hand Off</strong>: In this phase, we discuss how the feature will be released to production and what post-release actions we need to take</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7o6O8pzOaUG-DcUQ" /></figure><h3>Results of the New Workflow</h3><p>We see great improvement in our velocity and how we ship value faster to our customers. This is the result of:</p><ul><li><strong>Catching Missing Definitions Early</strong>: We are now finding missing definitions and edge cases much earlier, allowing PMs to refine feature definitions before development starts.</li><li><strong>Fewer Questions and Blocks</strong>: Investing in the grooming phase has really cut down on the number of questions and blocks during implementation, which means we can deliver features faster.</li><li><strong>Clarity and Alignment</strong>: Having a single source of truth has minimized confusion, ensuring that R&amp;D works with the most up-to-date definitions. This clarity allows all team members to align their efforts towards common goals, reducing the chances of miscommunication and ensuring that everyone is on the same page throughout the development process. As a result, we can work more efficiently and effectively, leading to a smoother implementation of features.</li></ul><h3>Conclusion</h3><p>In conclusion, our journey to improve our workflow has been a valuable experience that taught us a lot about collaboration and efficiency. Here are some key takeaways:</p><ul><li><strong>Continuous Improvement</strong>: We should always aim to enhance our work processes. If this new workflow doesn’t meet our needs in the long run, we will be open to exploring other ways to improve. Adapting to change is essential in our fast-paced environment at Cyabra.</li><li><strong>Embrace Feedback</strong>: Being open to feedback from everyone — whether it’s from R&amp;D, Product, or QA — is crucial. We encourage all team members to share their thoughts and ideas for improvement. This collaborative approach helps us identify blind spots and fosters a culture of innovation.</li><li><strong>Clear Communication</strong>: Maintaining clear communication across teams is vital. By ensuring everyone is on the same page, we can reduce misunderstandings and streamline our processes.</li><li><strong>Flexibility in Roles</strong>: Clarifying responsibilities has made our workflow smoother, but we also recognize the importance of flexibility. Team members should feel empowered to step in and help each other when needed, promoting a sense of teamwork.</li><li><strong>Celebrate Successes</strong>: As we implement changes, it’s important to celebrate our successes, no matter how small. Recognizing achievements boosts morale and motivates the team to keep pushing for improvement.</li></ul><p>By focusing on these principles, we at Cyabra can continue to refine our processes and deliver better features to our customers. Our commitment to improvement will not only enhance our workflow but also strengthen our team dynamics and overall productivity.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3710d626b0bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyabra/a-journey-to-a-better-feature-implementation-flow-3710d626b0bf">A Journey to a Better Feature Implementation Flow</a> was originally published in <a href="https://medium.com/cyabra">Cyabra</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Automatic Data Labeling: An Advanced LLM-Powered Approach]]></title>
            <link>https://medium.com/cyabra/automatic-data-labeling-an-advanced-llm-powered-approach-e2a6b941c409?source=rss----c966612e4dbe---4</link>
            <guid isPermaLink="false">https://medium.com/p/e2a6b941c409</guid>
            <category><![CDATA[active-learning]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[semi-supervised-learning]]></category>
            <category><![CDATA[data-labeling]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Elad Vromen]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 14:44:32 GMT</pubDate>
            <atom:updated>2024-09-24T14:44:32.782Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*PyFcf3bW6NtZ5npW" /></figure><p>This article was written by Elad Vromen, NLP/LLMs data analyst at Cyabra. For more information visit <a href="http://www.cyabra.com/">www.cyabra.com</a></p><h3>Introduction</h3><p>In the ever-evolving landscape of machine learning, data labeling remains a critical yet often time-consuming process. The advent of Large Language Models (LLMs) has brought new opportunities, offering unprecedented capabilities in processing and generating human-like text. These capabilities allow for the development of a new approach to textual data processing and data labeling in particular.</p><p>The main purpose of this article is to present an LLM-based automatic labeling system. The project under discussion focuses on sentiment analysis, but the approach is generalizable to other NLP tasks.</p><h4>Automating the labeling pipeline</h4><p>Text labeling in general, and sentiment analysis in particular, is a challenging task, even at the manual level. Texts are ambiguous by nature, especially when the labeling task addresses abstract and implicit content such as the sentiment or emotion of the speaker. Therefore, a common practice in labeling schemes is to cross-check several human annotators’ annotations, choosing the majority vote among them as the correct label. This approach aims to mitigate labeling challenges such as ambiguity of interpretation, inconsistency due to tiredness, and other factors to increase the confidence level of the labels.</p><p>However, this process is time-consuming, expensive, and may still suffer from inconsistencies. The manual labeling pipeline is often the bottleneck of machine learning projects.</p><p>For these reasons, we created a special system to incorporate LLMs into the data labeling cycle, transitioning to a ‘human in the loop’ approach that minimizes dependency on human effort. Many research studies have demonstrated the promise of LLMs for data labeling tasks (e.g., ‘ChatGPT Outperforms Crowd-Workers for Text Annotation Tasks,’ Gilardi et al., 2023). In this project, we aimed to develop a system that utilizes this potential to the fullest on a complex task in a real-world setting.</p><h4>Combining semi-supervised learning with LLMs</h4><p>The auto-labeling process combines advanced techniques from active learning and weak supervision with state-of-the-art LLM capabilities and prompt optimization to create a robust and efficient labeling system.</p><p>Active learning is a machine learning approach where the algorithm selects the most informative data points for labeling. To identify these data points, a confidence score is attributed, and the low-confidence data is sent for manual labeling. This method aims to maximize model performance while minimizing the amount of labeled data required, making the labeling process more efficient and cost-effective.</p><p>Weak supervision, on the other hand, is a paradigm that enables the creation of labeled training data using higher-level, potentially noisy, or imprecise sources of information. These sources can include heuristics, knowledge bases, or outputs from other models. Weak supervision allows for the rapid creation of large labeled datasets in a semi-supervised way. The basic assumption of the approach is that while each weak classifier is prone to mistakes on its own, the intersection of weak classifiers may result in high-quality classification.</p><p>By combining these two approaches, the system leverages the efficiency of active learning in selecting the most crucial data points for detailed analysis, while utilizing weak supervision techniques to generate initial labels and refine them through a multi-stage process.</p><p>In addition, it is well known that prompt optimization (prompt engineering) can significantly increase the performance of LLMs. Therefore, different prompts were optimized to be used as weak classifiers in the weak supervision process. The prompts’ optimization was conducted using a high-quality human-labeled benchmark containing around 300 records, with labels determined by the majority vote of three different human annotators.</p><h3>The Auto-Labeling Process: A Deep Dive</h3><p>The system is built on three fundamental components:</p><ol><li>LLM API Wrapper (for prompt engineering): This component allows running and evaluating different LLMs’ use cases with varying configurations, metrics, models, and output formats.</li><li>Agreement Mechanism: this component combines predicted labels from multiple prompts on the same data to calculate the agreement among them.</li><li>Auto-Labeling System: This system labels data based on the agreement mechanism and various heuristics.</li></ol><p>Here’s a detailed look at how the full labeling scheme works:</p><ol><li><strong>Multiple Prompts Labeling:</strong> In the first step, the data is labeled using multiple prompts. Each prompt was carefully selected after extensive prompt engineering. At this stage, OpenAI’s GPT-3.5-turbo was used due to its cost-effectiveness. Additionally, precision was prioritized over recall in the prompt engineering process to maintain high-quality labels. While this approach might initially result in lower coverage, potential gaps are addressed later through heuristics and manual labeling stages.</li><li><strong>Agreement Calculation:</strong> The labeled data is clustered based on agreement levels, originating from the different prompts utilized in the previous stage. These clusters serve as confidence levels for the labels. Importantly, research has shown that agreement is correlated with accuracy and improved performance [​​Automated Annotation with Generative AI Requires Validation, Pangakis et al., 2023]. This correlation underscores the validity of the approach, as higher agreement levels typically indicate more reliable labels.</li></ol><p><em>steps 1 and 2 are presented inthe following diagram:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZmGfIfMF2BEOmACB" /></figure><p>3. <strong>Heuristic Application: </strong>Different heuristics are applied according to the agreement level, allowing for effective handling of various scenarios:</p><ul><li>Full agreement records (3 out of 3 agreement scores) represent the most reliable labels and require no further processing.</li><li>Low agreement records (1/3) are flagged as potentially problematic. This helps identify nuanced cases that require human attention.</li><li>Partial agreement records (2\3) go through a two-step heuristic process to determine if the majority or minority vote (of the initial prompts labeling) is correct. This stage reduces the complexity of the problem from multi-class to binary. This simplification allows for using more informative, class-specific prompts. For example, if the majority vote is neutral, the prompt will ask whether the given text is neutral while describing in depth what “neutral” means for the use case. This targeted approach enhances the accuracy of the labeling process. This stage leveraged GPT-4o’s advanced performance to ensure maximum accuracy while remaining cost-effective, as the heuristic is applied only to specific subsets of the data.</li></ul><p><em>the following diagram depicts the partial agreement heuristics:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AUjkJRu4xxsNPeeN" /></figure><h3><strong>Results</strong></h3><p><strong>Coverage:</strong><br>Through this multi-stage process, approximately 75% of the data can be automatically labeled with high confidence. The remaining 25% is sent for manual labeling, ensuring that the most challenging cases receive human attention.</p><p><strong>Quality:</strong><br>Several quality validation efforts have been performed.</p><p>1. A set of around 300 records was manually validated by a single annotator who was asked, “Is the label correct?” The overall accuracy (answered “yes”) was above 90%.</p><p>2. A transformer model was fine-tuned based on the automatically labeled data and achieved an F1 score of over 70% in a multi-class sentiment task (four classes), suggesting the data is consistent enough for the model’s convergence.</p><p>3. During the evaluation of the fine-tuned model, many edge cases (where the fine-tuned model and the labeled data conflicted) were manually inspected. In the vast majority of cases, the labeled data was found to be accurate after careful consideration.</p><p>4. Errors that did occur were almost always limited to specific cases, such as the identification of commercial content, irony or sarcasm, highly culturally nuanced text, or slang language. It would likely be feasible to handle these cases with specific heuristics (prompts) in the system flow.</p><p><strong>Price:</strong><br>The cost of the complete process is around $35–$40 for 10,000 input records, resulting in approximately 7,500 automatically labeled records.</p><h3><strong>Practical </strong>Discussion</h3><p>A main challenge of the system is that current LLM APIs struggle to efficiently support such an extensive mechanism, particularly when scaling to larger datasets or higher throughput scenarios. Therefore, while testing the system, two different approaches were tested:</p><p>1. A complex version that utilizes a larger number of class-specific prompts (e.g., top 3 for neutral, top 3 for positive, etc.) at the first layer of labeling.<br>2. A simple version that uses the three best prompts on average for all classes.</p><p>Using the complex version increased the number of correctly labeled full-agreement records by almost 30%, and the overall coverage (of the full system process) reached 75%. In contrast, when using a single set of prompts, the overall coverage (high confidence) is reduced to 60%.</p><p>To test the differences in the quality of the results of the two methods, a subset of data that received full agreement from the complex method was examined. The labels were then compared with the results achieved from the simple method on the same records. While the agreement level of 30% of the records dropped from high confidence to partial, the final label (the majority vote, 3/3 or 2/3) was identical in 95% of cases. This may suggest that while there’s more certainty about labels with the complex method, the actual accuracy improvement is modest.</p><p>However, it is important to note that lower confidence may suggest more statistical variance. This aspect should be examined on a larger scale to identify the true variations in performance. The increased confidence in the complex method could lead to more stable and reliable results. While the complex prompt selection method shows promising improvements in confidence and coverage, the relevant version should be chosen based on practical considerations of the use case.</p><p><strong>Conclusion</strong></p><p>This LLM-powered approach to automatic data labeling is promising. By combining active learning, weak supervision, and advanced prompt engineering, we’ve achieved high-confidence automatic labeling for 75% of the data, dramatically reducing the need for manual labeling. This efficiency allows for better resource management in the labeling process, directing human expertise to the most challenging cases.</p><p>This pipeline saves both time and money. More importantly, it has the potential to enhance both the quantity and quality of labeled data. In most cases, LLM-based labeling is consistent, unlike human annotations, which can degrade due to tiredness and boredom. Additionally, the more challenging data segments still receive human attention, and even more of it, as annotators no longer need to focus on the easier cases.</p><p>While API limitations present challenges for scaling, this method opens up exciting possibilities for more efficient and accurate data preparation across various NLP tasks. Future investigations may focus on mitigating potential biases introduced by LLM-based labeling, as well as further improving efficiency and accuracy. Another interesting avenue could be integrating open-source models into the system to remove dependency on API throughput.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2a6b941c409" width="1" height="1" alt=""><hr><p><a href="https://medium.com/cyabra/automatic-data-labeling-an-advanced-llm-powered-approach-e2a6b941c409">Automatic Data Labeling: An Advanced LLM-Powered Approach</a> was originally published in <a href="https://medium.com/cyabra">Cyabra</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>