<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moritz Diener on Medium]]></title>
        <description><![CDATA[Stories by Moritz Diener on Medium]]></description>
        <link>https://medium.com/@diener.moritz?source=rss-47a9d315fd94------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*14vqFlev4B20J6Rv</url>
            <title>Stories by Moritz Diener on Medium</title>
            <link>https://medium.com/@diener.moritz?source=rss-47a9d315fd94------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 22:48:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@diener.moritz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[From Reading to Orchestrating: How LLMs Reshape Knowledge Work]]></title>
            <link>https://medium.com/kit-ksri/from-reading-to-orchestrating-how-llms-reshape-knowledge-work-72c8ffa467a4?source=rss-47a9d315fd94------2</link>
            <guid isPermaLink="false">https://medium.com/p/72c8ffa467a4</guid>
            <dc:creator><![CDATA[Moritz Diener]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 13:35:11 GMT</pubDate>
            <atom:updated>2026-07-13T13:35:11.683Z</atom:updated>
            <content:encoded><![CDATA[<p><em>The shift at the heart of our study: an LLM-based system does not remove complexity from knowledge work; it changes and redistributes it.</em></p><p><strong>Knowledge workers drown in heterogeneous documents, strict compliance rules, and cross-team coordination. We built an LLM-based system for public tender analysis and found that it does not remove this complexity, it moves it. Work shifts from reading and extracting information to orchestrating AI outputs and verifying them, a mode we call orchestrate-and-verify.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCt7R4UkUAAD67NRngmmWg.png" /></figure><h3>The Scenario</h3><p>A sales manager at a mid-sized data center company opens a new public tender on a Monday morning. It is 340 pages across nine PDFs, each with a different layout. Somewhere in there is the submission deadline, the list of mandatory certifications, and a clause that will disqualify the bid on the spot if one document is uploaded late. She has until Friday to decide whether the company should bid at all.</p><p>Before she can decide anything, she has to read. She scrolls, she highlights, she copies fragments into a spreadsheet, she emails the legal team about an ambiguous eligibility requirement and waits a day for the answer. By the time she has an overview, most of the week is gone. Interviewees in our study called this “finding a needle in a haystack” and reported spending dozens of hours on a single screening.</p><p><strong>The problem is not any one document. It is complexity that arrives in three forms at once, and technology usually shifts it rather than removing it.</strong></p><h3>Why This Is Harder Than It Looks</h3><p>Public procurement is a useful stress test for knowledge work. Public tenders account for roughly 15% of global GDP, and in Germany alone €123.5 billion moved through nearly 195,000 procurement procedures in 2023. For a small or medium-sized enterprise (SME) with limited staff, deciding whether to bid is a high-stakes, resource-intensive judgment made under time pressure.</p><p>Existing research treats the underlying difficulty mostly as information overload, the mismatch between how much information arrives and how much a person can process. That framing is real but incomplete. It misses the coordination burden of aligning legal, technical, and sales colleagues, and it misses the compliance burden of formal rules where a single oversight ends the bid. Prior work on generative AI in organizations also tends to stay generic, promising productivity or discussing LLMs in the abstract. What was missing is prescriptive design knowledge: concrete guidance on how to build systems that address these forms of complexity together. That is the gap we set out to close, using design science research in partnership with a German SME that operates data centers and bids regularly on public tenders.</p><h3>The Three Faces of Complexity</h3><p>From a workshop and seven expert interviews, we identified three distinct but interacting dimensions of complexity. Naming them separately matters, because a system that solves only one leaves the worker stuck on the others.</p><p><strong>Document complexity</strong></p><p>The sheer volume and inconsistency of what must be read. Tenders span hundreds of pages in non-standardized formats, with critical details scattered across attachments. Every new tender forces the reviewer to relearn where things are.</p><p><em>Example. The submission deadline sits on page 12 of the main document, but the certification list is in annex 4 and the technical scope in annex 7. There is no table of contents that maps to what actually matters.</em></p><p><strong>Compliance complexity</strong></p><p>Strict, sometimes ambiguous formal rules where minor errors are fatal. A missing certificate or a late upload triggers immediate disqualification, and unclear requirements force teams to file formal clarification requests and wait.</p><p><em>Example. The tender says a bidder “may provide references.” Is that optional or a disguised requirement? Guessing wrong either wastes effort or gets the bid excluded.</em></p><p><strong>Process complexity</strong></p><p>The coordination overhead of pulling in sales, legal, finance, and engineering. Cross-functional work is necessary but slow, and without a repository of past bids, every tender starts from zero. Staff also experience “tender fatigue,” treating the work as burdensome extra effort.</p><p><em>Example. The kickoff meeting slips a week because three departments cannot find a shared slot, and no one can reuse last quarter’s near-identical proposal because it was never archived in a findable way.</em></p><h3>Two Principles for Designing the System</h3><p>We translated four design requirements into two generalizable design principles for LLM-based systems, each grounded in established theory rather than intuition.</p><p><strong>DP1: LLM-based document analysis for complexity reduction</strong></p><p>The system should automatically retrieve, condense, and structure key information into task-aligned overviews, and link every item back to its source passage. This attacks document complexity directly and reflects task-technology fit theory: a tool helps only when its outputs match the actual task of the person using it.</p><p><em>Example. On upload, the system returns a one-screen overview with procuring entity, scope, and deadline, each field clickable straight to the page it came from, so the reviewer trusts it without rereading the PDF.</em></p><p><strong>DP2: Human-based verification and human-AI orchestration</strong></p><p>The system should present outputs as checklists, timelines, and structured overviews that the human coordinates and checks, never a black box that decides alone. This addresses compliance and process complexity and draws on socio-technical systems theory and responsible-AI research: outputs must stay explainable and reviewable so experts can verify, correct, and justify decisions.</p><p><em>Example. The compliance view lists every deadline and mandatory certificate as a checklist. The expert scans it, spots that one certification is inferred rather than stated, clicks through to confirm, and corrects it before the bid decision.</em></p><p>We instantiated these principles in a working artifact with three views. A structured overview for rapid screening, a compliance checklist and timeline, and a source-grounded question-and-answer view for ad hoc queries. Behind the interface, documents are parsed, chunked, stored in a vector database, and served through retrieval-augmented generation, with LLM agents handling summarization, compliance extraction, and Q&amp;A. Critically, every output links back to the passage it came from.</p><h3>What the Evaluation Showed</h3><p>We evaluated the artifact on 34 real tender documents and in think-aloud sessions with five domain experts. On accuracy, the system reached 88.82% content-match on structured text fields such as procuring entity, place of execution, and deadline, and 83.85% on binary compliance checks. It was weakest where wording is heterogeneous, for example the type of service (79.40%) and indirectly phrased reference requirements (76.50%).</p><p>On speed, experts averaged 3 minutes 42 seconds to screen a tender manually. The artifact did it in 1 minute 30 seconds including upload and indexing, roughly 59% faster. Experts consistently described getting the initial overview as the step that takes longest, and this is the step the system compresses.</p><p>The qualitative sessions revealed the deeper finding. Experts did not hand decisions to the system. They used its summary as a navigational aid, drilled into linked passages for anything critical, and passed structured outputs to colleagues as a shared starting point. They treated the tool as a pre-screening aid, not an autonomous compliance assessor, and accepted small inaccuracies as long as the orientation was reliable.</p><h3>What This Changes in Practice</h3><p>The headline is not that AI saves time, though it does. It is that the nature of the work changes. Complexity is not eliminated, it is redistributed. Workers spend less effort reading heterogeneous PDFs and more effort orchestrating agents and validating extractions. Manual deadline tracking becomes reviewing a machine-generated checklist. Fragmented information gathering becomes coordinating pre-analyzed summaries. We call the emergent pattern the orchestrate-and-verify work mode: knowledge workers direct LLM systems to produce structured outputs, then verify and correct those outputs using source links and expert knowledge.</p><p>This reframing has practical consequences. Organizations should position generative AI as a complexity-management and screening aid, not a decision-maker, which aligns expectations, reduces fear of replacement, and keeps human expertise central where it belongs, at the final bid decision and the interpretation of ambiguous rules. It also reshapes training. The valuable skill is no longer reading every document end to end, it is formulating good requests, reading source links critically, knowing the system’s limits, and documenting how AI outputs feed decisions.</p><p>We are also honest about the risks, because they are not hypothetical. If workers read fewer primary documents, tacit expertise may erode over time, a slow deskilling that short-term efficiency gains can hide. Accountability can become diffuse when a system quietly omits a requirement and everyone assumed someone else verified it. And verification introduces its own strain: monitoring an opaque model and reconciling its outputs is a subtler cognitive load than reading, not the absence of one. Efficiency metrics alone will not surface these effects.</p><h3>Conclusion</h3><p>For years the promise of automation in knowledge work has been subtraction: less reading, less coordination, less drudgery. What we observed is closer to a change of shape. The LLM-based system did not make complexity disappear. It converted the work of reading and extracting into the work of orchestrating and verifying, and that new work carries its own demands on skill, trust, and governance.</p><p>Seeing LLMs as complexity redistributors rather than complexity eliminators has a practical payoff. It tells designers to build for verification, with source links and human oversight as first-class features rather than afterthoughts. It tells organizations to invest in orchestration skills and clear lines of responsibility before the efficiency gains quietly hollow out expertise. And it gives researchers a concrete pattern to study as single-model tools give way to multi-agent systems, where the human is coordinating not one output but many interacting ones.</p><p>The shift from read-and-extract to orchestrate-and-verify is, in the end, a shift in what expertise means. The expert of the near future is measured less by how much they can read and more by how well they can direct, question, and correct the systems reading for them.</p><p><em>This article is based on research by Moritz Diener, Simon Kaps, Philipp Spitzer, Robin Hirt, Michael Vössing, and Gerhard Satzger (2026). “Rethinking Knowledge Work: Designing LLM-based Systems for Complexity Management.” European Conference on Information Systems (ECIS 2026). Read the full paper on </em><a href="https://www.researchgate.net/publication/405018781_Rethinking_Knowledge_Work_Designing_LLM-based_Systems_for_Complexity_Management"><em>ResearchGate</em></a><em> or in the </em><a href="https://aisel.aisnet.org/ecis2026/genai/genai/12/"><em>AIS eLibrary</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72c8ffa467a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/kit-ksri/from-reading-to-orchestrating-how-llms-reshape-knowledge-work-72c8ffa467a4">From Reading to Orchestrating: How LLMs Reshape Knowledge Work</a> was originally published in <a href="https://medium.com/kit-ksri">KIT-KSRI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>