Ten principles for research, mentorship, and the people behind scientific discovery.

Science is a human endeavor. Behind every publication, dataset, discovery, and technological advance are people who think, question, create, struggle, collaborate, and grow.

I believe that the pursuit of scientific excellence must go hand in hand with intellectual integrity, human dignity, fairness, and mutual respect. A successful research environment should not be measured solely by what it produces, but also by how it treats the people who make those achievements possible.

The following ten principles express the values I seek to uphold in my group’s research, collaborations, and professional relationships.

1. People Before Metrics

Researchers are human beings before they are authors, employees, students, or measures of institutional productivity. Their worth cannot be reduced to publication counts, citation metrics, grants, or performance indicators. Scientific achievement matters, but never at the expense of human dignity.

2. Scientific Integrity Above All

We pursue knowledge with honesty, rigor, curiosity, and intellectual humility. We value reproducibility, transparency, responsible research practices, and the courage to acknowledge uncertainty or correct mistakes. Scientific integrity must never be compromised for prestige, recognition, or competitive advantage.

3. Mentorship Is a Shared Commitment

Good mentorship requires time, patience, intellectual generosity, honest feedback, and genuine investment in another person’s development. It also requires initiative, responsibility, openness, and mutual respect from those being mentored. Mentorship is neither unquestioning authority nor unconditional accommodation. It is a professional relationship in which everyone has responsibilities, even when those responsibilities differ.

4. Respect Must Extend in Every Direction

Respect is not a privilege attached to seniority, status, or institutional position. Students, doctoral researchers, postdoctoral researchers, supervisors, professors, technical staff, and administrators all deserve dignity and a fair hearing. Differences in authority must never justify intimidation, exploitation, humiliation, discrimination, or retaliation. Equally, professional expertise, judgment, and legitimate responsibilities deserve recognition and respect.

5. Accountability Without Scapegoating

Difficulties, disagreements, and unsuccessful working relationships are part of human collaboration. They should be addressed seriously, fairly, and constructively, with attention to evidence, context, power differences, and the perspectives of everyone involved. Accountability must not become a search for someone to blame, nor should isolated difficulties erase a broader record of meaningful contributions. We believe in learning, proportionality, and fair processes rather than automatic assumptions of guilt or fault.

6. Recognition for Visible and Invisible Work

Scientific success is rarely the achievement of one individual. Behind research outputs are countless hours of mentoring, reviewing, debugging, coordinating, supporting, encouraging, and building the conditions in which others can succeed. We believe that intellectual contributions, emotional labour, technical expertise, and sustained mentorship deserve fair acknowledgment. Recognition should reflect contributions, not merely visibility, hierarchy, or the ability to advocate for oneself.

7. The Freedom to Question, Disagree, and Grow

Science advances through questioning accepted ideas, challenging assumptions, and exploring unfamiliar directions. We value constructive disagreement and intellectual independence. Researchers must be able to express reasoned concerns, challenge scientific arguments, admit mistakes, and pursue new ideas without fear of humiliation or retaliation. We expect rigorous criticism of ideas without attacks on the people expressing them.

8. Sustainable Science Requires Humane Working Conditions

Dedication to science should not require sacrificing physical health, mental well-being, personal relationships, or life outside work. We reject the glorification of overwork, chronic exhaustion, and unnecessary urgency as measures of commitment. We support realistic expectations, transparent responsibilities, reasonable boundaries, and the right to rest. Sustainable working conditions are essential to sustainable scientific excellence.

9. Fairness, Inclusion, and Equal Opportunity Require Action

Talent, ambition, and scientific potential exist across genders, cultures, backgrounds, identities, and career stages. Yet opportunities, recognition, and credibility are not always distributed equally. We commit to recognizing structural inequalities, questioning unconscious biases, and promoting fair access to opportunities and advancement. No one should have to diminish their achievements, repeatedly apologize for exercising professional judgment, or remain silent to be accepted.

10. Science Should Help People Flourish

The ultimate purpose of research is not simply to generate knowledge, secure funding, or advance careers. It is to contribute meaningfully to our understanding of the world and, wherever possible, improve it. We strive to create an environment in which people can become better scientists without losing their humanity, where ambition coexists with kindness, and where individual achievement strengthens collective progress.

Our guiding conviction …

Excellence without integrity is hollow. Achievement without fairness is incomplete. Leadership without respect is unsustainable. And science without humanity loses sight of why it matters.

The internet feels exhausted.

Researchers scrape it, clean it, filter it, deduplicate it, translate it, paraphrase it, ask one model to generate questions for another model, ask another model to judge the answers, then call the resulting pile a new dataset.

Sometimes it works.

Sometimes the model learns something.

Sometimes it learns only the habits of another model.

Synthetic data is cheap, fast, obedient, tireless, available at 3 o’clock in the morning, never asks for a contract, never needs a clarification meeting, never says that the figure is ambiguous, never admits that the axis label is unreadable, never pauses over a spectrum and wonders whether the apparent peak shift is chemically meaningful or merely an artifact of normalization.

That is precisely the problem.

As researchers turn increasingly toward synthetic training data, this post reflects on why expert-annotated scientific data still matters: because its value lies not only in what is collected, but in the meaning patiently recognized, checked and shaped by those who understand the domain from within. In other words, by domain experts.

A figure is not merely an image

A scientific figure in a research paper rarely looks simple. Even from a distance, it can appear dense, compressed, overlaid with curves, symbols, markers, panels, legends, axes, insets and labels that assume the reader already knows the scientific language in which the figure is speaking.

But to a researcher, that density is not merely visual complexity. It is often where the paper’s argument becomes more concrete, where a general claim begins to acquire conditions, comparisons, measurements and limits.

The text may say that performance improved.

The figure says when, by how much, under which conditions, compared with what, with which uncertainty, and sometimes where the trend begins to fail.

This is particularly true in scientific domains such as materials science, where spectra, phase diagrams, band diagrams, reaction schemes, process diagrams and multiaxis plots can encode relationships that are not always restated completely in the surrounding prose.

A line is not always just a line.

It may be a growth-per-cycle curve, a saturation response, a compositional trend, a temperature window, or an indication that a surface reaction has reached completion.

A peak is not always just a peak.

Its position, intensity, width and relationship to other peaks may suggest nucleation, bonding, oxidation state, composition or structural change.

A model can detect the pixels.

A domain expert knows what would make the interpretation scientifically defensible.

What we built

The ALD/E-ImageMiner benchmark was established to create gold-standard annotations for scientific-figure classification, structured data extraction, summarization and visual question answering. The corpus is organized around atomic layer deposition (ALD) and atomic layer etching (ALE) publications and further distinguishes experimental from simulation-based work. The dataset repository uniquely presents structured document representations with domain-expert annotations of scientific images with their scientific provenance.

The resulting benchmark contains 205 scientific publications and 1,951 figures, divided into training, development and test sets across ALD and ALE experimental and simulation studies.

The four expert-annotated signals resemble stages of understanding.

First, what am I looking at (figure classification), then, what information does it contain (structured data extraction), then, what does that information say (figure content summarization), then, what scientific questions does it present (visual question answering)?

The movement from classification to reasoning is the movement from seeing a scientific object to understanding why it matters.

Ten experts, not ten anonymous clicks

Given our annotation budget, we recruited ten annotators with materials-science backgrounds and direct experience related to ALD or ALE.

They included MSc-level researchers, doctoral researchers, postdoctoral researchers and associate professors, recruited from universities and research environments across the United Kingdom, Finland, the Netherlands, Pakistan and India, together with an independent contractor holding an MSc. Their backgrounds included experimental ALD/E, simulation-based research, physics and computational materials science.

They were selected through a prescreening task conducted on a scientific paper, from a pool of strong applicants, and the candidates producing the highest-quality work were hired.

Each annotator received 25 papers, enough to establish familiarity, not enough to turn the work into a factory shift — intentional to avoid, five papers were deliberately shared by all annotators, assignments were balanced across deposition and etching, experiments and simulations, because fatigue creates mistakes, repetition creates blindness, and a dataset remembers every shortcut taken during its construction.

The process combined machine assistance with human responsibility. A vision-language model produced preliminary annotations for classification, data extraction and summarization, but the experts reviewed and refined the material, while the visual-question-answering task was reserved for manual expert annotation. Automated validation scripts were run every two weeks, inconsistencies were returned to annotators, annotations were revised and finally consolidated into a distributable structure.

Expertise changes the questions

Almost anyone can look at a chart and ask:

What color is the line?

A domain expert asks:

How does the Ti–O–Si peak intensity change with the number of ALD cycles, and what does this indicate about film formation?

Almost anyone can ask:

Is one point higher than another?

A domain expert asks:

How does the temperature required for the modification step compare with that required for the etching step, and what does this imply for controlling the process window?

The difference is not simply in the wording of the question. It comes from years of looking at such figures, knowing what matters, and knowing what would make an answer scientifically meaningful. The difference is that the second kind of question tests whether a system can connect visual evidence to a scientific process, a material property, a mechanism, an experimental condition or an application-level consequence.

For the benchmark, experts produced Q&A pairs at the subfigure level, guided by categories including process-oriented reasoning, comparative or trend reasoning, structure–property relationships, and application or performance reasoning. Answers were represented as yes/no, factoid, list or paragraph responses.

The figure above helps make this distinction more concrete. These examples are not only about identifying what is visible in the image, but about interpreting the figure in relation to the underlying scientific context. A polar chart may involve reasoning about why a measurement was taken at a particular angle. A multi-spectra chart may require understanding alignment to a reference peak. A 3D plot may ask how surface roughness changes after annealing, while a contour heatmap may involve interpreting plasma-density patterns under specific process assumptions.

In each case, the question is not only what is shown? but also how should this visual evidence be interpreted? That is where domain-expert annotation becomes valuable: it connects the visible elements of the figure to the scientific context in which they matter.

Without that expertise, the questions might become easier, cleaner, more generic, and more visually obvious. They might also become less useful.

A benchmark can scale and still fail to ask the questions that matter.

Domain experts protect the boundary between plausible and correct

Generative models are excellent producers of plausibility.

They know how an answer should sound.

They know the rhythm of an explanation, the shape of a table, the tone of certainty, the words commonly found near words such as deposition, etching, surface, film and temperature.

But scientific correctness often lives in details that language fluency can conceal: a unit copied incorrectly, a logarithmic axis treated as linear, two curves associated with the wrong legend entries, a peak attributed to the wrong chemical species, an experimental observation presented as a causal mechanism, a simulation result confused with a measured value, a trend summarized correctly in direction but incorrectly in magnitude, a conclusion inferred from the caption although the plotted evidence does not support it.

A fluent non-expert may not notice these errors. A model may explain them beautifully. A domain expert is more likely to stop.

The competition tested the data as much as the models

The ALD/E-ImageMiner benchmark was subsequently released as an ICDAR 2026 competition called Sci-ImageMiner. It attracted 68 active participants and 1,263 public and private submissions across the four tasks.

The results were revealing.

Current multimodal systems performed comparatively well on classification and summarization, but data extraction and visual question answering remained difficult. The winning classification system reached an F1 score of 0.81, while the best weighted VQA score was only 0.31. Even sophisticated systems using fine-tuning, retrieval, context injection, task-specific prompting, multistage pipelines and model alignment continued to struggle with precise extraction and scientific reasoning.

To me, this is one of the more interesting outcomes of the benchmark. The results suggest that scientific figure understanding is not a single capability that improves uniformly once a model becomes stronger. A system may learn to recognize the type of a figure, or produce a plausible summary, while still struggling when it has to recover structure precisely or connect visual evidence to domain-specific reasoning. The benchmark therefore helps separate different layers of understanding that are often treated as one in the guise of scientific figure comprehension.

Synthetic data still has a place

This is not an argument for returning to a world in which every example is created manually.

We will need synthetic data… because expert time is limited, expert annotation can be prohibitively expensive, especially for small research groups and less well-resourced labs, scientific domains are vast, rare figure types are difficult to collect, distributions are uneven, new models consume more examples than any annotation team can reasonably produce.

Synthetic data can expand a taxonomy. It can balance classes. It can create controlled perturbations. It can generate practice examples. It can support curriculum learning, preference optimization and robustness testing.

Indeed, the winning approaches in the competition used synthetic augmentation and other model-assisted strategies effectively, particularly for structured data extraction.

But synthetic data needs an anchor.

It needs examples whose meaning did not originate in the same model family that will later be trained or evaluated on them.

It needs expert-created seeds, expert-defined categories, expert-checked outputs, authentic scientific documents and test sets that remain outside the loop of machine self-confirmation.

Otherwise the model generates the lesson, studies the lesson, writes the examination and awards itself the grade.

An annotated dataset is encoded scientific judgement

We often describe datasets using nouns of quantity: papers, figures, classes, questions, tokens, gigabytes.

Yet the most valuable content is not entirely countable.

It is the decision that two visually similar plots represent different scientific objects. It is the choice to ask about mechanism instead of color. It is the recognition that a caption does not fully explain a curve. It is the correction of an extracted value. It is the insistence that an answer distinguish observation from inference.

It is the knowledge that a figure from an experimental ALD study should not be interpreted as though it came from a simulation, although both may contain the same axes and nearly the same lines.

This is why domain-expert annotation should not be treated as low-level data preparation performed before the interesting AI research begins.

It is knowledge engineering.

It is scientific modeling.

It is the construction of the concepts, distinctions, questions and evidential standards by which the AI system will later be judged.

The dataset is not merely fuel placed into a model.

It is a record of what a community considered meaningful enough to name, difficult enough to test and important enough to preserve.

The internet may be tired, but science is not finished

There will be larger models.

There will be longer context windows, more agents, better retrieval, faster synthetic-data factories, systems able to produce a million plausible scientific examples before breakfast.

Still, someone must decide what is true.

Someone must know why two curves that look almost identical imply different process behavior. Someone must notice that an answer is elegant and wrong. Someone must formulate the difficult question before the model can learn to answer it.

In this age of generative AI, high-quality, domain-expert annotated datasets are not obsolete remnants of an earlier machine-learning era.

They are the ground beneath the new one. The machines may generate the sentences. Experts still give them something worth saying.

✨ In the age of generative AI, perhaps the question is not only how much data we can create, but how much meaning we can preserve.

Resources

🌐Explore the ALD/E-ImageMiner project page for the dataset, competition links, evaluation scripts and related resources:
https://sciknoworg.github.io/ALD-E-ImageMiner/

🏆Read the competition scientific report preprint: ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures | Fahad Ahmed, Sören Auer, Jennifer D’Souza

At some point during the Build Small Hackathon, I found myself asking a question that felt both ridiculous and irresistible:

What would happen if 1960s folk pop accidentally fell through a wormhole and woke up inside a 2000s indie-rock radio station?

It sounded like a joke at first, but it turned out to be a good hackathon challenge: can a small AI app make music generation feel like a tiny radio experience?

The result was Turntable Time Machine, a Gradio app built for the Hugging Face × Gradio Build Small Hackathon. The app lets users choose a source era, a remix era, a fictional DJ persona, a language, a mood, an audio texture, and a vocal mode. Then, with a multilingual text layer supported by CohereLabs/tiny-aya-global, it generates a short original “radio postcard” from a musical timeline that never existed.

The app is not a famous-song remix machine. It does not make covers, imitate artists, clone voices, or reproduce melodies. Instead, it works with broad musical vocabularies: 1960s soul, 1970s disco, 1980s synth-pop, 1990s trip-hop, 2000s indie rock, 2010s festival EDM, 2020s lo-fi, and so on.

The core idea is simple:

Pick two musical eras. Bend one through the other. Let a fictional DJ announce the impossible timeline.

That DJ voice became the feature that made the app feel less like a prompt form and more like a tiny radio object.

What the User Controls

The interface is structured around a few playful choices:

• Source era: the musical vocabulary to start from
• Source genre: for example folk pop, disco, synth-pop, trip-hop, garage, lo-fi
• Remix era: the time period that transforms the source
• Remix genre: the production lens
• Fictional DJ persona: the voice of the broadcast
• Broadcast language: English, German, French, or Spanish
• Prompt language mode: English, selected language, or bilingual
• Vocal mode: instrumental only, wordless vocal texture, or original micro-lyrics
• Audio texture: vinyl crackle, cassette warmth, FM radio, mono radio, club PA, VHS glow, early MP3 compression, or clean digital master
• Mood: nostalgic, dreamy, energetic, futuristic, intimate, and so on

This combination gives the user enough control to feel creative, but not so much control that the app becomes a studio dashboard.

That balance mattered. The Build Small Hackathon encouraged compact, focused, playful apps. So I wanted the app to feel like a toy with real structure underneath.

The Timeline Engine

Under the hood, the app is not just a single prompt box. It runs a lightweight Timeline Engine.

The process looks like this:

User selections
→ source era / genre lookup
→ remix era / genre lookup
→ fictional DJ persona selection
→ language and prompt-mode routing
→ BPM selection from genre ranges
→ optional Tiny Aya micro-lyrics
→ safety-aware music prompt
→ optional spoken DJ narration
→ music generation or fallback synth audio
→ nostalgia texture processing
→ final broadcast and mixtape card

The Core Build

The app works like a little time-bending machine.

The user might choose:

Source: 1960s folk pop
Remix: 2000s indie rock
Mood: nostalgic
Texture: FM radio
DJ: The Midnight Radio Curator
Language: German
Vocal mode: original micro-lyrics

The app then builds a structured prompt for the audio model, optionally generates short original micro-lyrics, creates a DJ intro, and combines everything into a final broadcast.

A simplified version of the prompt-building logic looks like this:

def build_music_prompt(source, remix, mood, texture, bpm, lyrics=None):
    prompt = (
        f"Create an original music clip. "
        f"Start from {source} and transform it into {remix}. "
        f"Mood: {mood}. Texture: {texture}. BPM: {bpm}. "
    )
    if lyrics:
            prompt += f"Use only these original micro-lyrics:\n{lyrics}"
        else:
            prompt += "No vocals. No lyrics."
        prompt += (
            " Do not imitate any artist, song, melody, hook, riff, "
            "voice, sample, or copyrighted lyric."
        )
        return prompt

This was one of the main lessons of the build: small AI apps benefit from structure. The app is not just one giant prompt. It has curated era metadata, genre vocabulary, DJ personas, texture settings, language options, and safety constraints.

The model does not have to invent the whole world. The app gives it a world to work inside.

The DJ Voice Changed Everything

Originally, I thought the app would mainly be about music generation. But the DJ voice became the feature that made the output feel alive.

The fictional DJ might say something like:

Good evening, time travelers. Tonight we are taking 1960s folk pop
and sending it through the glow of 2000s indie rock.
Let the record spin from another timeline.

This turns the output into a small performance. The user is not just waiting for a file. They are listening to a broadcast.

That was my favorite lesson from this build: sound modality is not only instrumental output. In audio-first apps, human voice can be part of the interface. A spoken intro, a pause, and then the music can create expectation and atmosphere.

The final assembly can be as simple as:

broadcast = []

if dj_intro_audio:
    broadcast.append(dj_intro_audio)
    broadcast.append(short_pause())
    broadcast.append(generated_music)
    final_audio = concatenate_audio(broadcast)

The pause matters. It makes the transition feel intentional, like a real radio moment.

Where Tiny Aya Global Fits In

The app uses CohereLabs/tiny-aya-global as the multilingual text layer.

In the current version, its main role is to generate short, original micro-lyrics when the user selects a vocal mode. The prompt is intentionally small and bounded: two lines, original wording, selected language, selected mood, and no references to real songs or artists.

For example:

lyrics_prompt = (
    f"Write 2 short original lyric lines in {language}. "
    f"Mood: {mood}. "
    f"Style: {source_genre} transformed into {remix_genre}. "
    "No famous lyrics, no artist names, no song titles."
)

This is a good role for a small multilingual model. Tiny Aya does not need to control the whole app. It contributes a compact creative layer inside a larger pipeline.

But it could be integrated much more deeply.

In future versions, Tiny Aya could generate multilingual DJ intros, mixtape liner notes, short explanations of the genre transformation, or persona-specific lyrical fragments. It could also adapt the broadcast more naturally across languages instead of simply filling a template.

For example, the app could ask Tiny Aya to write:

A two-sentence DJ intro in French, in the voice of a warm late-night
radio host, introducing 1980s synth-pop transformed into 2020s lo-fi.

That would make the app feel less templated and more like a living multilingual radio station.

What I Learned

This build taught me three things.

First, sound-based apps should think beyond “generate audio.” Voice, texture, timing, and transitions are also design materials.

Second, small models become more useful when they are given precise jobs. Tiny Aya is not asked to write a whole song. It writes tiny original lyric fragments. The audio model is not asked to imitate music history. It receives a constrained, safety-aware transformation prompt.

Third, fallbacks are not failure. For a hackathon app, models may load slowly, hardware may vary, and deployment may behave differently from local testing. A good small app should degrade gracefully: text instead of voice, fallback lyrics instead of model-generated ones, simple synth audio instead of full music generation when needed.

A playful app should still be robust.

Future Extensions

I would love to extend Turntable Time Machine into a fuller fictional radio experience.

Future versions could support three-track mini radio episodes, more languages, stronger DJ voice control, richer musical geographies, and user-provided memories that become safe, original audio postcards.

The concept also transfers beyond music.

In education, a “time machine” interface could explain concepts across historical periods or communication styles. In science communication, papers could become audio briefings, museum-guide explanations, or multilingual summaries. In digital humanities, archives could become narrated listening experiences with careful provenance and ethical boundaries.

The serious idea behind the playful app is this:

Structured transformation plus multimodal narration can make information feel alive.

Turntable Time Machine just applies that idea to music.

Pick an era. Bend the genre. Let the fictional DJ open the portal.

Resources

• 🤗 Live Space: https://huggingface.co/spaces/build-small-hackathon/turntable-time-machine
• 💻 GitHub repo: https://github.com/jd-coderepos/turntable-time-machine
• 🎬 Demo video: https://www.youtube.com/watch?v=EKCwTK88pUE

For the Hugging Face × Gradio Build Small Hackathon, I built Council of Chuckles, a small-model Gradio app that lets users bring a messy everyday question to a council of historical and literary-inspired advisors. The user can type a question, or speak it aloud, choose a group of advisors, and receive a playful response in the form of advisor cards or a short “Campfire Council” dialogue.

The app was built for the Thousand Token Wood track, so the spirit was not to use the largest possible model, but to ask: how much interaction, personality, multilinguality, and structure can we build around small models?

The core idea was simple. Instead of asking a language model for generic advice, I wanted the app to stage a small reflective encounter. The council can include figures such as Confucius, Socrates, Rumi, Brené Brown, Carl Jung, or other thinkers. But the goal is not solemn wisdom. It is perspective-shifting with a wink. 😉

What I Built

Council of Chuckles has three main interaction layers. First, the user asks a question in text or voice. Second, they assemble a council from 100 advisor profiles. Third, the app runs a lightweight “Council Engine” that analyzes the question, selects suitable voices, builds a prompt, generates a response, and optionally produces a final verdict called The Gavel Falls.

The working logic looks roughly like this:

User input
→ safety routing
→ topic / emotion / need analysis
→ advisor matching
→ active speaker selection
→ dialogue planning
→ prompt construction
→ small-model generation or fallback
→ final verdict

What I liked about this architecture is that the model is only one part of the system. Much of the behavior comes from structured profiles, routing, prompt design, safety checks, and interface choices. That was one of my main lessons from the hackathon: small models become much more useful when the application around them gives them a clear stage to perform on.

Main Technologies Used

The app is built with Gradio, Hugging Face Transformers, Tiny Aya Water, Whisper small, and Hugging Face Spaces / ZeroGPU.

Gradio was one of my biggest practical learnings. It is a Python framework that lets you turn Python functions into interactive web apps quickly. You define UI components such as text boxes, audio inputs, dropdowns, buttons, and HTML panels, then connect them to Python functions.

A simplified version of the app structure looks like this:

import gradio as gr


with gr.Blocks(title="Council of Chuckles") as demo:
    audio = gr.Audio(
        label="Record or upload your question",
        sources=["microphone", "upload"],
        type="filepath",
    )
    transcript = gr.Textbox(label="Editable transcript")
    question = gr.Textbox(label="Your question")
    transcribe_btn = gr.Button("Transcribe")
    generate_btn = gr.Button("Generate council response")
    transcribe_btn.click(
        transcribe,
        [audio, spoken_language],
        [transcript, voice_status],
    )
    generate_btn.click(
        generate,
        [...],
        [stage_display, session_state],
    )
demo.launch(css=CSS)

This pattern made the app feel manageable: build the interface, define the state, then bind events to functions.

For multilingual text generation, the app uses CohereLabs/tiny-aya-water, a small multilingual model with 3.35B parameters and broad coverage across 70+ languages. In the app, this supports text interaction in languages such as English, German, French, Spanish, Italian, Portuguese, Polish, Greek, Hindi, Tamil, Vietnamese, Mandarin Chinese, Japanese, Korean, Swahili, Arabic, and more. The model is loaded lazily so that the Gradio interface can appear before the full Hugging Face Transformers model pipeline is ready:

from functools import lru_cache
from transformers import AutoModelForCausalLM, AutoTokenizer


TEXT_MODEL_ID = "CohereLabs/tiny-aya-water"
@lru_cache(maxsize=2)
def load_text_model(model_id=TEXT_MODEL_ID):
    tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
    return tokenizer, model

For voice input, I used openai/whisper-small, a 244M-parameter speech recognition model, through the Hugging Face Transformers automatic speech recognition (ASR) pipeline. This was another important learning point: voice models need more than just “load model and run.” The app has to handle audio duration, silence, language selection, sampling rate, and graceful failure. In the app, voice input is supported through a curated language selector for English, German, Hindi, French, Spanish, Italian, Portuguese, Dutch, Polish, Greek, Arabic, Vietnamese, Mandarin Chinese, Japanese, and Korean.

A simplified version of the voice flow is:

from transformers import pipeline


ASR_MODEL_ID = "openai/whisper-small"
@lru_cache(maxsize=1)
def load_asr_pipeline():
    return pipeline("automatic-speech-recognition", model=ASR_MODEL_ID)
def transcribe_audio(audio, language_code):
    asr = load_asr_pipeline()
    result = asr(
        audio,
        generate_kwargs={
            "language": language_code,
            "task": "transcribe",
        },
    )
    return result["text"]

The final piece is the advisor layer. Each advisor has structured metadata in data/advisors.json: role, core_wisdom, signature_style, mastermind_voice, jokester_voice, catchphrase, best_for, and avoid. This made the app less like a single prompt and more like a tiny orchestrated NLP system.

The NLP Themes Inside the App

Although Council of Chuckles is playful, it touches several core NLP themes:

• Multilingual text generation: Users can type questions in different languages and receive responses in their chosen language.
• Automatic speech recognition: Spoken questions are transcribed before they enter the council engine.
• Text classification and lightweight semantic routing: The app detects themes, emotions, and user needs before choosing suitable advisors.
• Persona-conditioned generation: Each advisor has a distinct reasoning style, tone, and perspective.
• Dialogue planning: In Campfire Council mode, the app stages a short multi-voice exchange rather than producing a single generic answer.
• Prompt engineering and controlled generation: The prompts guide the model toward playful, structured, and advisor-specific responses.
• Safety routing: The app keeps the humor bounded so responses remain playful without becoming cruel, dismissive, or inappropriate.

For me, this was the most interesting part of this hackathon build. The app is light-hearted on the surface, but underneath it is about structured interaction design for small language models. It asks how far we can go with model orchestration, transparent routing, and carefully designed roles before reaching for heavier infrastructure.

What I Learnt

I learnt that building with small models is not mainly about accepting limitations. It is about designing around them. A small model may not always produce perfect long-form reasoning, but it can do surprisingly well when the task is bounded, the style is clear, the output format is constrained, and the application provides scaffolding.

I also learnt that Gradio is a very approachable way to ship an AI idea as an actual app. The distance between “I have a Python function” and “someone can try this in a browser” is very short. For a hackathon, that matters.

Finally, working with voice input made the app feel more alive. Typing a frustration is one thing; saying it aloud and watching it become a question for a tiny council of philosophers is another. It adds a theatrical quality that fits the project.

The idea is simple: when life gets dramatic, call in the great thinkers — but make them funny. Each advisor applies their own style of reasoning, yet the app nudges the tone away from solemn advice and toward playful perspective-shifting.

Extensions

Council of Chuckles is a toy example, but the underlying pattern is more general. One obvious extension is to make the advisor profiles richer. Instead of only historical and literary-inspired figures, the same structure could support domain experts, learning companions, research mentors, writing coaches, or debate partners. The important part is not the name of the persona, but the structured profile behind it: what perspective it brings, what tone it should use, what kinds of problems it is good for, and what boundaries it should respect.

The same idea could also be used beyond playful reflection. A similar council-style interface could support education, brainstorming, writing feedback, research planning, or decision exploration. For example, a student could ask for feedback from a “clarity coach,” a “critical reviewer,” and a “friendly explainer.” A researcher could use a council of methodological perspectives to examine a project idea. A writer could test whether an argument sounds convincing, confusing, or too dramatic.

Another natural extension is sound. The current app supports voice input, but the council could become even more theatrical if each advisor had an optional spoken voice. A Campfire Council mode with audio output would turn the interaction from a written response into a small staged performance. That would also raise new design questions: how should voices differ, how long should the dialogue be, and how can audio remain fun without becoming distracting?

For me, this is where the hackathon idea becomes especially interesting. The project began as a playful app about tiny AI advisors with jokes, but it points toward a broader design pattern: small models, structured personas, transparent routing, and multimodal interaction can be combined to create experiences that feel more intentional than a single chatbot reply.

Agent-based workflows introduce a transformative capability to Semantic Web technologies by embedding autonomy, memory, and modular reasoning into information extraction and grounding tasks. — schema-miner^pro

This is the central idea behind our recent work on schema-miner^pro: Agentic AI for Ontology Grounding Over LLM-Discovered Scientific Schemas in a Human-in-the-Loop Workflow (DOI: 10.1177/22104968261431521).

But let’s step back for a moment.

🌿 Schemas Have Been Around Longer Than We Think

Schemas are not new.

They’ve quietly lived in the background of computing for decades — from database design to knowledge representation, from XML to RDF. At their core, they are simple: they describe what exists and how things relate.

But science has always been different.

Scientific knowledge is rarely born structured. It emerges in prose — carefully written, context-heavy, and often difficult to compare across papers. A temperature here, a rate there, a parameter described differently depending on the experiment. Across a corpus, even basic concepts begin to drift.

And so, while schemas shaped the digital world, scientific schemas have remained elusive.

Large language models changed the starting point.

With schema-miner (our earlier work on LLMs4SchemaDiscovery, DOI: 10.1007/978–3–031–94578–6_14), we showed that LLMs can move beyond entity extraction and begin to induce schemas directly from scientific text.

In practice, the workflow is iterative but intuitive. You begin with a process description, from which the LLM proposes an initial schema — properties, types, relationships. This draft is then refined using a small, high-quality set of papers, and later generalized across a broader corpus. At each stage, domain experts step in — sometimes by editing the schema directly, sometimes by providing structured feedback — which is fed back into the model.

Over time, this loop transforms a rough output into something that is structurally consistent, semantically meaningful, and grounded in real scientific practice.

But even at this stage, a key limitation remains.

🌐 Why Semantic Interoperability Matters

A schema alone is not enough.

If two schemas describe the same concept differently — or use incompatible units — they cannot be meaningfully combined. This is the core challenge of semantic interoperability.

Without grounding in shared ontologies:

• schemas remain local
• comparisons remain fragile
• integration becomes manual

In other words, we extract structure — but we still lack shared meaning.

schema-miner^pro: Introducing the Agent 🤖

schema-miner^pro addresses this by adding an agentic ontology grounding stage.

Instead of relying on a single LLM prompt to align schema elements to an ontology, the system introduces a LangChain-based agent (https://reference.langchain.com/python/langchain/overview) that performs this alignment step-by-step.

This is the key shift.

The problem is no longer treated as a one-shot generation task, but as a reasoning process with tool use.

⚙️ A Practical View: How It Works

Once you have a refined schema, the agent takes over.

It is given:

• the schema
• a short domain description (for context)
• a target ontology (e.g., QUDT https://www.qudt.org/ for quantities and units)

From there, grounding happens property by property.

For straightforward cases like temperature or pressure, the agent uses direct lexical matching against ontology terms.

When no clear match exists, it switches strategy.

The ontology is split into smaller chunks, embedded into vector representations, and indexed using FAISS (Facebook AI Similarity Search, https://github.com/facebookresearch/faiss), a library designed for efficient nearest-neighbor search in high-dimensional vector spaces. This enables semantic retrieval — where the agent finds conceptually similar ontology fragments rather than relying on exact string matches — before passing them to the LLM for final selection.

This hybrid approach — heuristics first, retrieval second — keeps the process both efficient and precise.

Just as importantly, the agent is allowed to return an empty result when a property does not correspond to a valid ontology concept. This avoids forcing incorrect mappings, a common failure mode in prompt-only approaches.

🧩 From Schema to Semantically Grounded Knowledge

Once a match is found, the agent integrates ontology metadata back into the schema using a template.

A simple field like temperature is no longer just a label. It becomes a structured quantity — linked to a unit, associated with a defined type, and connected to a canonical ontology identifier.

The schema, at this point, is no longer just structured — it is semantically grounded. It becomes machine-readable, interoperable, and ready for integration into knowledge graphs or downstream analysis pipelines.

👩‍🔬 The Role of the Human

Despite the automation, the final step remains essential: expert validation.

Domain experts review the grounded schema and correct subtle issues — often around unit choice or contextual interpretation. These refinements are fed back into the workflow, improving future grounding decisions.

This is not a limitation. It is what makes the system reliable in real scientific settings.

✨ Why This Shift Matters

What schema-miner^pro represents is a subtle but important shift.

We are moving from:

→ extracting structure
to
→ engineering meaning

And we are doing so not with static pipelines, but with agentic systems that reason, adapt, and collaborate.

In the broader vision of AI for science, this matters.

Because scientific progress depends not just on data, but on our ability to connect, interpret, and reuse knowledge across domains.

🔗 Resources

📄 Paper: schema-miner^pro (DOI: 10.1177/22104968261431521)
💻 Software: https://github.com/sciknoworg/schema-miner
📦 Pip: https://pypi.org/project/schema-miner
📚 Docs: https://schema-miner.readthedocs.io/en/latest

On February 2nd 2025, OpenAI released its Deep Research feature [1]. Much of it was pretty much a blackbox for a few months. Then came David Zhang’s X/Twitter post about his Deep Research implementation, which gave the community a peek under the hood. That post sparked curiosity and open-source spin-offs, including Python reimplementations (like mine!) designed to make this workflow more transparent, tunable, and useful for scientific research.

Ever felt lost in a jungle of scientific papers, unsure which path will lead to the answers you seek? What if you could send a team of tireless AI agents into that literature jungle to scout for insights while you sip your coffee? Enter DeepResearch — a novel AI system that turns complex scientific Q&A into a recursive treasure hunt through research literature. Developed for ecological research (but useful far beyond), DeepResearch is like having a squad of super-smart robot interns comb through papers day and night. The result? A thorough, multifaceted report on your question, complete with citations and a dash of expert-level analysis. And the best part: you control how deep and how broad the hunt goes, dialing the scope up or down like a research DJ.

In this post, we’ll explore how DeepResearch works (in plain English, with fun analogies), highlight a new Python reimplementation of the system (based on David Zhang’s original TypeScript code), and show why this approach is a game-changer for ecologists and AI researchers tackling big, knotty questions. We’ll keep it casual and witty — imagine we’re chatting over coffee about this cool AI tool that acts like Indiana Jones in the Library of Science. Grab your fedora (or lab coat), and let’s dive in!

At its heart, DeepResearch sends AI “agents” on a literature treasure hunt. Instead of one-pass searching, these agents work recursively — searching, reading, learning, and then asking better follow-up questions. It’s like an iterative brainstorm with the entire internet and scientific databases at their disposal. The original DeepResearchEco paper describes it as an “agentic LLM-based system for automated scientific synthesis” that uses a “recursive, depth- and breadth-controlled exploration” of research questions. In non-academese: the system searches far and wide (breadth) and drills deep (depth), doing multiple rounds of search-and-summarize to gather a high-throughput integration of evidence. This leads to answers that aren’t just surface-level — they’re enriched with nuggets from many sources (one evaluation showed a 21-fold increase in sources integrated into answers when using DeepResearch!). High depth/breadth runs produced “expert-level analytical depth and contextual diversity,” essentially matching domain experts in thoroughness. Not too shabby for an AI agent on autopilot!

How DeepResearch Works (The AI Treasure Hunt Workflow)

So how does DeepResearch actually pull off this feat? Let’s break down the recursive agent workflow in relatable terms. Imagine you’ve got a crack team of AI agents, each with a special role, tackling your question “What are the impacts of climate change on grassland biodiversity?” (one of those juicy, complex ecological questions). Here’s how the adventure unfolds:

1. The Scout (Query Generation Agent): This agent takes your broad research question and comes up with specific search queries — like drawing treasure maps. For our example, it might generate queries like “climate change effects on grassland plant diversity” or “grassland biodiversity temperature rainfall interaction.” It also sets a “research goal” for each query — a hint of what we’re looking for. Essentially, the scout is figuring out where to dig first. (In DeepResearch, this corresponds to the generate SERP queries sub-agent, which turns the question into search-engine-friendly queries.)

2. The Explorer (Search Agent): Now the queries go to the explorer agent, which actually ventures out to find information. DeepResearch can use two different explorers — think of them as two search modes. The first is Firecrawl, a web-scale search API that fetches full web pages (it’s like sending an explorer to scour blogs, articles, even PDFs and bring back the raw text). The second is ORKG Ask (from the Open Research Knowledge Graph), which searches a huge scholarly index (80+ million publications!) and returns structured results — titles, abstracts, and links. Firecrawl is our wide-net, general-purpose explorer (covering everything from news to scientific papers), while ORKG Ask is the specialist, focused on academic literature. You can choose either or even use both in tandem — dual providers mean you won’t miss treasure whether it’s hidden in a journal or on a website. The explorer agent executes each query and brings back the top results (usually the top 10 docs per query). It works in parallel too, so no time wasted.

3. The Scribe (Summarize-and-Learn Agent): Now we have piles of content — what next? The scribe agent steps in to read the retrieved documents and jot down the important bits. It merges all the text (be it full web content from Firecrawl or abstracts from ORKG) and asks the LLM to summarize key “learnings.” Think of this as your AI research assistant reading a stack of papers and highlighting the juicy findings. The scribe produces concise bullet-point insights (usually up to 3 per query) capturing the essence of what was found. But that’s not all — it also brainstorms follow-up questions based on those findings. These follow-ups are like clues on a treasure hunt map: if the initial query found something interesting, the agent asks, “Hmm, they mentioned invasive species — should we explore how invasives factor into climate impacts on grasslands?” This gives fresh leads for the next round. In short, the scribe agent is extracting knowledge gems and deciding where to dig deeper next.

4. The Editor-in-Chief (Report Synthesis Agent): Finally, after several rounds of the above loop (we’ll talk about how many rounds in a second), DeepResearch has a trove of insights. The generate report agent acts as the editor-in-chief, taking all accumulated learnings and composing the final narrative — a structured Markdown report that directly answers the original question. It’s not just a brain dump; it writes a cohesive synthesis, complete with citations for each fact (all those URLs from the search are kept for transparency). The result is something you could honestly mistake for a thorough literature review written by a human (and a diligent one at that). It might start with an overview, then discuss various angles (each potentially coming from different sources), and end with a conclusion — all grounded in the references that were found along the way.

This whole process isn’t a one-shot affair — it iterates recursively. After the scribe agent generates follow-up questions, the scout agent takes those and turns them into new queries for the explorer, which fetches more info, which the scribe reads, and so on. It’s a feedback loop, refining the search as it goes deeper, very much like a detective chasing down leads. The loop continues until it reaches a predefined depth (more on that next). At that point, the editor compiles everything and voila — you have a detailed answer with multiple sources and angles.

To put it in a fun way: DeepResearch is like sending a bunch of AI treasure hunters into the library. One maps out where to look, another grabs the books, another reads and scribbles notes, and the last one compiles the final “treasure report”. The coolest part is that the system makes its reasoning transparent — you can see the intermediate queries it used and the notes (“learnings”) it took. It’s not a black box; it’s more of a guided tour through the literature, where you can see how the answer was built step by step.

Depth and Breadth: Tuning Your AI Expedition

Every good treasure hunt needs a plan. In DeepResearch, the plan is guided by two key parameters: depth and breadth. These are essentially dials you (the user) set to control how far and how wide the AI should explore. They turn a one-size-fits-all search into a customizable adventure. Let’s demystify these with a bit of analogy:

• Breadth (b): Think of breadth as the number of parallel paths the agents will explore at each step. A higher breadth means casting a wide net — pursuing more sub-questions or leads in each round. In our treasure hunt analogy, if breadth = 5, our scout agent draws five different treasure maps at the start, and the explorer splits into five teams to gather info on all of them. You’ll cover a lot of ground (ensuring you don’t miss major areas of the literature). The default in the paper was 3, but you can set it higher for more expansive searching. There’s a twist: as the recursion goes deeper, the number of queries can halve each round (breadth // 2 each level). So the search naturally narrows focus the deeper you go — it’s like a funnel. Early on, you explore broadly (many leads); later, you focus on the most promising leads in detail.
• Depth (d): Depth is how many layers of recursion the system will do — basically how many rounds of the search-summarize loop before it stops and compiles the report. Depth is like how many levels deep into the cave our treasure hunters will venture. Depth = 1 would be just one round (shallow survey and done). Depth = 4 (as used in the paper’s high setting) means the system will do initial search, then 3 rounds of follow-up searching based on the leads (so 4 cycles total counting the first). With each layer, it’s digging deeper into the topic. In ecology terms, depth might mean you start with general climate-biodiversity papers, then drill down into specific mechanisms like “climate x land-use interaction effects on grassland insects vs plants”, then even deeper into “case studies from particular regions”, and so on. A higher depth yields a more nuanced and detailed answer, as the system uncovers finer-grained insights that a shallow search would miss. In the DeepResearchEco experiments, the high-depth (d=4) answers captured subtle details — “cross-regional evidence, mechanistic pathways, system feedbacks,” as the paper notes, compared to a shallow d=1 answer that stayed generic.

The beauty of having depth and breadth controls is you decide the strategy. For a broad exploratory question, you might set a moderate depth but high breadth to capture diverse perspectives. For a very specific question, maybe high depth and lower breadth to focus on deep analysis of a particular aspect. DeepResearch’s creators emphasize balancing “breadth (covering many sources) versus depth (deeply analyzing each source)” to produce high-quality synthesis. And indeed, finding that balance can make the difference between a superficial answer and an exhaustive one.

What’s the impact of cranking these dials to the max? In my tests on 49 ecology questions, the highest setting (d=4, b=4) yielded the most comprehensive answers, integrating far more sources and details than low settings. They measured things like domain specificity, rigor, novelty, and information density of the answers. High depth/breadth scored better across the board — basically approaching “expert-level synthesis” of the topic. One striking stat: information density (how much info packed in per word) went up nearly 15-fold from the lowest to highest setting. So not only do you get more sources, you get more insight per paragraph. Of course, cranking it up also means more computation (the agents will read a ton more content). But if you’ve got a beast of a question, it’s nice to have the option to truly throw the kitchen sink at it in a controlled way.

From TypeScript to Python: A New Flavor of DeepResearch

The original DeepResearch (as described in the paper and initial code release) was implemented in TypeScript by David Zhang and collaborators. TypeScript is great, but let’s face it — many AI researchers and scientists live in the world of Python. That’s where the user (hi, that’s me! 👋) decided to reimplement DeepResearch in Python, staying true to the original algorithm but making it more accessible and extensible for the community.

So what’s new in this Python reimplementation? Firstly, it’s built to be developer-friendly: if you know Python and have a hankering to tweak AI agents, you can dive right in. I’ve kept the core workflow the same (the scout, explorer, scribe, editor agents doing their thing recursively), but also included some extensions and improvements:

• Dual Search Providers Out of the Box: The reimplementation makes it easy to switch between or combine Firecrawl and ORKG Ask for the search step. You can configure which one to use, or even use both in sequence to get both web content and scholarly abstracts for the same query (maximizing coverage). The original design supported both, but now you have a simple interface to toggle them. If you’re running a search on a general topic, you might lean on Firecrawl; if it’s super academic, ORKG Ask might be your go-to — or why not both? The Python version lets you treat it like a choose-your-own-adventure for search.
• Parameter Tweaks and Customization: Depth and breadth are of course adjustable, but in Python it’s trivial to change defaults, set per-query settings, or even try dynamic strategies (like maybe you want to adjust breadth halving behavior or how many learnings to extract per document — go for it!). It’s your playground. Because it’s Python, integrating with other tools (maybe you want to plug in a different LLM or a custom search API) is straightforward.
• Transparency and Logging: When I built this, I made sure you can see what’s happening at each step — the queries generated, which results came back, what learnings were extracted, etc. This transparency was a big selling point in the paper (they highlight the “transparent and traceable knowledge workflow”), and as a user, it’s both educational and reassuring to watch your AI agent think. In code, this means logging or printing out intermediate steps, and a nicely formatted final Markdown output.

Now, I won’t pretend I did this all alone — credit goes to the original creators for the blueprint. My role was like taking a fantastic concept car and rebuilding the engine in a more familiar chassis (Python) so more folks can take it for a spin. And guess what — you can take it for a spin too. The code is open source on GitHub, under the friendly MIT license. In fact, the official paper even links to the GitHub repo (yep, they really want people to use it). We’ll link it below and encourage you to check it out.

Why AI-Powered Deep Research Matters (Especially in Ecology)

By now you might be thinking, “This sounds cool, but does it actually matter for my work?” For anyone doing research that requires synthesizing information from many sources — yes, it matters a ton. And if you’re in ecology or environmental science, buckle up, because the case is particularly strong.

Ecology questions are famously complex. They often span multiple scales (genes to ecosystems, local to global), involve many variables (climate, land use, species interactions, oh my!), and have huge bodies of literature scattered across sub-disciplines. Answering something like “How do invasive species impact grassland biodiversity under climate change?” could mean reading dozens of papers on climate effects, invasive species biology, soil health, etc., then piecing them together. That’s a Herculean task for a human, especially under time constraints.

DeepResearch shines here by doing the heavy lifting of gathering and distilling information from across the spectrum. It doesn’t get tired or bored combing through 50 papers — it will dutifully find and summarize them, and even highlight conflicting findings or knowledge gaps if they appear. This means as a researcher, you get to see a more comprehensive picture. For example, the DeepResearchEco evaluation found that high breadth settings pulled in a lot more content related to conservation and climate policy aspects (things that a narrow focus might miss). In one case, the broad search picked up cross-regional evidence that enriched the answer with context from different continents. Depth, on the other hand, allowed the system to unpack mechanistic details — e.g., why biodiversity drops, what the common effects of fertilization are on plants over long term, etc., which only surfaced after digging through several layers of follow-up. The end result is akin to an expert narrative: covering broad trends and detailed mechanisms, all supported by citations.

For ecologists, this approach can accelerate literature reviews, policy briefs, or research ideation. Instead of spending weeks gathering references, an ecologist could use DeepResearch to draft a baseline synthesis in hours, then refine it with their own expertise. It’s not about replacing the scientist (no robo-ecologist is stealing your job, promise!); it’s about augmenting your ability to see the forest and the trees in a massive forest of publications. In fact, I’ve found that using it often surfaces papers or insights I might have overlooked — a nice antidote to our human confirmation biases or just the sheer overwhelm of information out there.

Moreover, because the system keeps track of sources, you can directly follow the paper trail. If DeepResearch says “study X found that increased rainfall mitigates some negative effects of warming on grasslands,” it’ll cite that study — you can click the link, read the original, and verify or dive deeper. This makes the AI’s output a starting point for deeper exploration, not a final black-box answer. It’s almost like having an assistant professor who writes your related work section, with all references neatly cited.

Try It Yourself — Open Source and Ready to Explore

Enough talk — I bet you’re curious to play with this toy. Good news: DeepResearch is open-source and ready for you to clone and run! The GitHub repo is sciknoworg/deep-research, and it includes instructions to get started. Whether you’re an AI researcher interested in agent architectures or an ecologist wanting to turbocharge your literature reviews, you’re invited to give it a whirl.

Here are some fun things you can try once you have it running:

• Swap Search Engines: Test out both Firecrawl and ORKG Ask. Try a question with the general web vs. the scholarly index. For example, “What are the latest trends in reforestation techniques?” might yield different flavors of info from web blogs vs academic papers. You can compare the outputs or even combine them for maximum info.
• Tweak Depth and Breadth: Start with a modest setting (say d=1, b=2) and see what the answer looks like. Then crank it up (d=3, b=4) and observe the difference. You’ll notice the report gets way richer and longer with more depth/breadth — more sources, more angles. It’s actually pretty fun to watch the agent “think” through each round if you have logging on. Find the combo that best suits your question — there’s no one-size-fits-all, which is the beauty of it.
• Ask a Hard Question: Pose some complex, open-ended question in ecology (or any field you like). Something you genuinely want to know but haven’t had time to dig into. Let DeepResearch do its thing, then read the report. Did it find useful info? Any surprises? Use it as a jumping-off point to guide further research. It’s like having a knowledgeable colleague give you a briefing.
• Contribute or Customize: Since this is a community project, you’re welcome to contribute. Spot an issue or have an idea for improvement? Open an issue or pull request on GitHub. Maybe you want to add a new search provider (ArXiv API, anyone?), or improve the prompt engineering for generating follow-up questions — go for it! The Python code is meant to be hackable. Even changing how the “learnings” are extracted (maybe you want more or fewer, or to include quotes) is relatively easy. I’d love to see folks extend this in directions we haven’t thought of.

For AI research folks, DeepResearch is also an interesting testbed for agentic AI in a real-world task. The idea of AI agents autonomously doing research is exciting, and here’s a concrete implementation you can study and build upon. How does it break, where does it excel, how to make it better — plenty of research questions right there!

And if you run into any questions or just want to chat about recursive research agents, I’m all ears. Seriously, feel free to contact me — drop an issue on GitHub, or connect via email/Twitter (my contact info is on the repo). I’m eager to hear about your experiences, ideas, or use cases. Collaboration or even just swapping war stories about AI literature reviews is very welcome.

Rethinking Scientific Synthesis in the Age of Recursive AI Agents

We’re at an inflection point where AI moves from just answering questions to actually researching questions. DeepResearch is a glimpse into that future — one where we can task an AI with “Tell me everything about X, and make it good,” and it will literally go on a multi-hop knowledge expedition to do so. For fields like ecology, where the challenges are complex and interdisciplinary, this could transform how we gather insights. Instead of being limited by how much one person (or a small team) can read, we leverage these tireless agents to gather and summarize evidence at scale, without sacrificing detail or rigor.

Is it perfect? Of course not. It can only be as good as the sources it finds and the guidance we give (via prompts and parameters). It might miss something, or misinterpret a nuance — that’s where human experts remain crucial. But it augments our capabilities tremendously. It’s like having an army of grad students who never sleep, each an expert speed-reader, coordinated by a thoughtful synthesizer. If that doesn’t get you a bit excited about the future of research, I don’t know what will!

So, to all the AI enthusiasts and ecologists (and anyone else who made it this far): I invite you to rethink scientific synthesis with me. What new questions could we tackle if we aren’t afraid of information overload? How might our approaches to literature reviews, policy analysis, or even education change when AI agents can do the first pass of knowledge gathering? DeepResearch is one early example, and I suspect many more will follow in various domains.

Give the repo a look, try out the system, and imagine the possibilities. And if you’re as intrigued as I am, let’s talk! This is an open journey, and I’d love more companions on the road. Together, we can push the boundaries of how knowledge is collected and synthesized in the 21st century. It’s an exciting time — let’s embrace these AI co-explorers and see where the adventure leads. 🌱🤖🎓

Happy researching, and may your treasure hunts yield rich knowledge! 🚀

And just for fun, I want to close with another cartoonish ChatGPT-generated image on the theme of this post! :) In the meantime, happy hacking scientific AI assistance!

References

1. https://openai.com/index/introducing-deep-research/
2. David Zhang’s TypeScript Implementation https://github.com/dzhng/deep-research/
3. My Python Implementation https://github.com/sciknoworg/deep-research

From Raw LLM Text Outputs to Validated Structured Information

As psychology research expands in complexity and volume, so does the need for structured, machine-readable access to its knowledge. While academic papers remain the main vehicle of scientific communication, they are notoriously hard to search, compare, and integrate — especially across constructs, measures, and methodologies. This post introduces a practical pipeline that transforms unstructured psychology papers into structured knowledge triples using OpenAI’s function calling capability for structured outputs.

The goal: extract psychological constructs and their associated measurement instruments, producing clean, machine-readable outputs that can populate a lightweight knowledge graph.

To make this work, we use Pydantic and OpenAI Function Calling — a streamlined way to define and parse structured outputs directly on the client side. It’s a practical example of how to use OpenAI’s native function calling feature together with Pydantic to generate and validate structured data from research text.

👉 The full code is available on GitHub:
🔗 sciknoworg/psychKG-pilot
🔗 You might also find useful: my related post on generating structured outputs from Hugging Face models (which don’t natively support Pydantic) for data validation. Check it out: “How I Get LLMs on Hugging Face to Speak Structured Data? The Two-Library Magic (Instructor + Pydantic)”

Now, back to this post…

🧩 The Problem: Constructs and Their Measures Are Buried in Text

Psychology research is rich in conceptual structures — constructs like self-esteem, working memory, procrastination, anxiety, rumination, etc. — that are empirically studied via validated measurement scales. However, this construct-measurement relationship is usually embedded in narrative form, making it difficult to reuse for semantic search or meta-analysis.

We want to go from this:

“The paper introduces and utilizes the Extinction Gambling Task (EGT) for studying decision making under extinction risk, providing a structured experimental paradigm to assess…”

To this:

{
  "construct": "decision making under extinction risk",
  "measured_by": "extinction gambling task (egt)",
  "justification": "The paper introduces and utilizes the Extinction Gambling Task (EGT) for studying decision making under extinction risk, providing a structured experimental paradigm to assess participants' choices in scenarios reflecting varying definitions of extinction risks (Keep, Lose, Reset)."
}

🛠️ The Solution: A Minimal Pipeline Using OpenAI + Pydantic

This pipeline has been applied across thousands of full-text psychology papers, enabling the extraction of construct–measurement instrument relationships at scale — unlocking structured insights previously buried in text.

Our approach involves four components:

1. Defining structured output schemas
2. Prompting the LLM to extract structured triples
3. Parsing TEI XML full-text articles as input
4. Validating and structuring model output using Pydantic

Let’s walk through the Python code that brings this pipeline to life.

📄 Step 1: Define the Structured Output Format

We begin by specifying the structure of the extracted knowledge using simple Python data models. These models capture the key elements we want to extract from each paper.

class PsychTriple(BaseModel):
    construct: str
    measured_by: str
    justification: str

class PsychTripleList(BaseModel):
    extractions: List[PsychTriple]

Each PsychTriple represents a semantic unit in the knowledge graph—linking psychological constructs to their measurement instruments, with justifications.
The PsychTripleList wraps these triples into a list, representing all extractions from a single research paper.

🤖 Step 2: Prompting the Model for Structured Extraction

We define a clear, role-based system prompt to guide the model. The prompt tells the LLM it’s acting as an expert in psychology and knowledge representation, and that it should return a list of construct–measured_by pairs with short justifications.

SYSTEM_PROMPT = '''
You are an expert in psychology and computational knowledge representation. Your task is to extract key scientific information from psychology research articles to build a structured knowledge graph. 

The knowledge graph aims to represent the relationships between psychological **topics or constructs** and their associated **measurement instruments or scales**. Specifically, for each article, you will extract information of the form:

1) "Construct"
2) [Property: "measured by"] -> [Value: "Measure"]

Each article may contain multiple such records. These extractions help build a machine-readable semantic network that connects psychological concepts with their empirical operationalizations.

For each extraction, provide a short justification from the text (1-3 sentences) that supports why this construct was measured by the identified measure.

As far as possible, extract meaningful **phrases** (rather than whole sentences or vague descriptions) as **entities** suitable for inclusion in a knowledge graph for both `construct` and `measured_by` fields.

However, not all articles are relevant to this task. If the article does not discuss psychological constructs and how they are measured (e.g., does not mention constructs, measurement techniques, or instruments), return an empty list `[]`.
'''

We then call the OpenAI API using responses.parse with PsychTripleList as the output format. Here I have used the o3 model which you could replace with your OpenAI model of choice.

response = client.responses.parse(
    model="o3",
    input=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Extract the constructs and their measures from the following article:\n{article_text}"},
    ],
    text_format=PsychTripleList
)

This ensures the output is a well-formed, strongly-typed list of triples.

🧼 Step 3: TEI XML Parsing

We assume full-text articles are stored in TEI XML (e.g., output from GROBID). The extract_text_from_tei() function handles text cleanup and formatting:

def extract_text_from_tei(tei_path):
    with open(tei_path, 'r', encoding='utf-8') as file:
        soup = BeautifulSoup(file, 'lxml-xml')
        body = soup.find('text')
        return body.get_text(separator='\n') if body else ''

🧩 Step 4: Structuring and Validating Output with Pydantic

While the OpenAI model generates text, we need to ensure that this text follows a strict, machine-readable structure suitable for downstream applications like knowledge graph population or semantic search. This is where Pydantic comes in.

Pydantic provides the data models (PsychTriple and PsychTripleList) that define the expected structure of the LLM output. When paired with the responses.parse method from the OpenAI Python client, it allows us to automatically validate and convert the raw LLM output into strongly typed Python objects.

Here’s how that looks in code:

from pydantic import BaseModel
from typing import List

class PsychTriple(BaseModel):
    construct: str
    measured_by: str
    justification: str

class PsychTripleList(BaseModel):
    extractions: List[PsychTriple]

Then, during inference:

response = client.responses.parse(
    model="o3",
    input=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Extract the constructs and their measures from the following article:\n{article_text}"},
    ],
    text_format=PsychTripleList  # Pydantic schema used here
)

This ensures the returned data conforms to the specified schema: a list of structured triples where each entry is validated for presence and type. If the model’s output deviates from the expected format, the validation will fail — helping catch inconsistencies early and allowing for robust error handling or retry mechanisms.

By enforcing structured output at the schema level, Pydantic acts as a contract between the LLM and downstream consumers, ensuring that what you get is predictable, reliable, and immediately usable for building knowledge graphs or JSON exports.

✅ Example Output

Here’s what a structured extraction might look like:

[
  {
    "construct": "decision making under extinction risk",
    "measured_by": "extinction gambling task (egt)",
    "justification": "The paper introduces and utilizes the Extinction Gambling Task (EGT) for studying decision making under extinction risk, providing a structured experimental paradigm to assess participants' choices in scenarios reflecting varying definitions of extinction risks (Keep, Lose, Reset)."
  },
  {
    "construct": ["risk aversion and risk seeking"],
    "measured_by": ["choice patterns in keep, lose, and reset conditions"],
    "justification": "Participant choice patterns across the Keep, Lose, and Reset conditions were indicative of risk aversion or risk-seeking behaviors, with patterns reflecting considerations like potential gains, losses, and the nature of the extinction event."
  },
  {
    "construct": ["opportunity cost neglect"],
    "measured_by": ["computational modeling analysis"],
    "justification": "Computational modeling was employed to analyze participants' choice strategies, revealing differences between conditions reflecting opportunity cost neglect, as participants were more aligned with the optimal strategy when their endowment was at stake compared to the opportunity to continue playing."
  }
]

📊 Summary: What You Get

• Input: Full-text TEI XML files from psychology papers
• Output: JSON files with construct-measurement triples + justification
• Applications: Lightweight knowledge graphs, semantic search, instrument discovery, QA systems

🕸️ Publishing the PsychKG Subgraph on the ORKG

Now let’s turn to publishing the extracted and validated JSON data as a subgraph on the Open Research Knowledge Graph (ORKG) platform.

The process involves the following steps.

1️⃣ Define the Template

In ORKG, a template represents a semantic schema — a structured model of properties that can be consistently applied to research papers for describing their contributions in a machine-readable way.

2️⃣ Bulk Data Upload: One PsychTripleList Item + One Paper at a Time

Each extracted PsychTripleList corresponds to one paper’s structured contributions. The upload script reads each item and maps it to the ORKG template, linking the extracted constructs, measurements, and justifications to their respective paper metadata.

To do this programmatically we will use relevant functions from the ORKG Python Client https://orkg.readthedocs.io/en/latest/index.html

The complete workflow is in the script upload-to-orkg.py. Let’s unpack the details.

1. Connect to ORKG and authenticate

To begin, connect to the ORKG API and authenticate using your credentials. You’ll be prompted for the host URL, email, and password:

from orkg import ORKG
from getpass import getpass

host = input("Enter ORKG host URL (e.g., https://www.orkg.org/): ").strip()
email = input("Enter your ORKG email: ").strip()
password = getpass("Enter your ORKG password: ").strip()

print("🔗 Connecting to ORKG...")
orkg = ORKG(host=host, creds=(email, password))
print("✅ Connected.")

📌 Tip: ORKG provides three hosted environments for different purposes:

• 🧪 Sandbox: https://sandbox.orkg.org/ — for testing data upload workflows
• 🚧 Incubating: https://incubating.orkg.org/ — for experimenting with platform features under development
• 🚀 Live: https://orkg.org/ — the production environment where validated data and stable features are published

Use the sandbox for safe trial runs, and the live system only when your data is final and reviewed.

2. Load a Pydantic-validated JSON file (i.e., PsychTripleList)

Each JSON file represents the extracted knowledge for one paper. This is read and parsed:

with open(json_path, 'r', encoding='utf-8') as f:
    data = json.load(f)

for entry in entries:
    c = entry["construct"].strip().lower()
    meas = entry["measured_by"].strip().lower()
    just = entry["justification"].strip()

3. Match to the ORKG Template

Each contribution is created using the Psychology Constructs and Measurements template, identified by its template ID (e.g., R1411854). The script calls the materialize_template() function (https://orkg.readthedocs.io/en/latest/client/templates.html) to fill in this schema programmatically:

template_id = input("Enter the ORKG template ID to use (e.g., R1411854): ").strip()
orkg.templates.materialize_template(template_id)
tp = orkg.templates

4. Create the Contribution for Each PsychTriple

The script loops through each triple and creates resources for the construct and measurement (if they don’t already exist), then fills in the justification field. For example:

for entry in entries:
    c = entry["construct"].strip().lower()
    meas = entry["measured_by"].strip().lower()
    just = entry["justification"].strip()

    for label in [c, meas]:
        if label and label not in resource_cache:
            try:
                response = orkg.resources.find_or_add(label=label)
                resource_id = response.content["id"]
                resource_cache[label] = resource_id
                print(f"✅ Added/Found: '{label}' → {resource_id}")
                save_resource_cache(resource_cache, cache_path)
            except Exception as e:
                print(f"❌ Failed to create/find resource for '{label}': {e}")

    c_id = resource_cache.get(c)
    meas_id = resource_cache.get(meas)

    if not c_id or not meas_id:
        print(f"⚠️ Missing resource for '{c}' or '{meas}'")
        continue

    c = tp.psychology_constructs_and_measurements(
        label=f"contribution {counter}",
        topic_or_construct=OID(toc_id),
        measured_by=OID(meas_id),
        justification=just
    )
    contributions.append(c)

    resource_entries.append({
        "topic_or_construct": toc_id,
        "measured_by": meas_id,
        "justification": just
    })

    print(f"✅ Created contribution {counter}")
    counter += 1

5. Link to the Paper Metadata

The script links each contribution to the corresponding paper resource in ORKG. The user is prompted (or a mapping is provided) for the paper’s title, DOI, or existing ORKG resource ID.

paper = {
    "predicates": [],
    "paper": {
        "title": metadata["title"],
        "doi": metadata["doi"],
        "authors": metadata["authors"],
        "publicationMonth": metadata["publicationMonth"],
        "publicationYear": metadata["publicationYear"],
        "publishedIn": metadata["publishedIn"],
        "researchField": "R343", #resource id for the research field Psychology on the ORKG https://orkg.org/fields/R343/Psychology
        "contributions": [c.template_dict['resource'] for c in contributions]
    }
}

try:
    response = orkg.papers.add(params=paper)
    paper_id = response.content["id"]
    log_uploaded_paper_detailed(paper_log_detailed_path, metadata["title"], paper_id)
    append_structured_log(structured_log_path, paper_id, resource_entries)
    print(f"📄 Paper uploaded: {metadata['title']} → {paper_id}")
except Exception as e:
    print(f"❌ Failed to upload paper: {e}")

6. Repeat for Each Paper

The script processes each JSON file in the directory — each representing one paper — uploading all its extracted triples as a single contribution.

This approach ensures each extraction becomes a searchable, semantically structured entry in the ORKG. Over time, these micro-contributions accumulate into a rich, navigable knowledge graph of psychological constructs and their empirical operationalizations.

Below are some example snapshots of instances of the PsychKG subgraph on the ORKG.

The ORKG offers curation interfaces. So for instance, for all the papers added to the ORKG from our dataset, the extraction_method has been set to automatic. With this the fronted renders a blue box around the contribution with a cautionary note to the reader that the data is automatically extracted. Given this, an expert can via the edit mode verify this information which resets the extraction_method flag to manual and then resets the frontend interface to the usual display without the warning as shown in the paper view below.

Returning back to PsychKG. Another feature of the ORKG platform is comparisons of structured research.

This end-to-end workflow — from full-text parsing to validated triple extraction and knowledge graph publication — not only demonstrates technical feasibility, but also showcases how general-purpose LLMs can be adapted for domain-specific, structured scientific metadata generation at scale. By publishing this data as FAIR, semantically linked contributions in the ORKG, we make psychological research more searchable, comparable, and reusable — paving the way for more systematic reviews, instrument discovery, and large-scale meta-analyses.

🧠 Why It Matters

This workflow lowers the barrier to building domain-specific knowledge graphs. Instead of training a custom NLP pipeline or building a rule-based system, we delegate the task to a general-purpose LLM guided by a schema and expert prompt.

In doing so, we make empirical psychological knowledge — once locked in prose — available for structured reasoning and reuse.

🚀🧵 Next Steps

Here are a few recommendations to build on this work:

• Add support for gold-standard metadata extraction (authors, title, abstract)
• Enrich the schema with additional properties with a tool like schema-miner https://github.com/sciknoworg/schema-miner over a large-scale collection of the publications
• Explore integration with LLM-powered interfaces (e.g., chat-based exploration or semantic QA) to make the knowledge graph more accessible to non-expert users
• Develop autonomous agents that continuously monitor new psychology publications, extract structured triples, and suggest updates to the graph
• Implement agent workflows for validation (e.g., human-in-the-loop curation) and refinement (e.g., schema expansion, relation alignment)
• Investigate multi-agent collaboration setups where one agent extracts, another verifies, and a third links entries to external ontologies like UMLS or MeSH

💡 Want to try it yourself?
Grab the code here 👉 sciknoworg/psychKG-pilot
Curious about structured extraction with Hugging Face models? Check out my related post:
👉 How I Get LLMs on Hugging Face to Speak Structured Data

Let me know how you’re using structured extraction in your own research projects!

This is pilot is part of a collaborative project with Dirk Wulff.

Building MVP Knowledge Graphs from Research Articles

📍 The Problem: LLMs Speak Text. I Want Structure.

I work a lot with research papers. And when I use large language models (LLMs) to extract knowledge from them — whether it’s entities, relationships, or claims — I don’t want free-form text. I want clean, structured data I can plug into a knowledge graph, a database, or an analysis pipeline.

But LLMs, especially open-weights ones on Hugging Face, don’t natively speak JSON. And when they try, it’s often wrong: missing commas, wrong fields, stray quotes, invalid arrays. This makes downstream parsing brittle and painful.

🤔 So what’s the fix?

1. 🧭 Instructor

Instructor transforms any Hugging Face text generation model into a structured output generator. With just one line:

results = llm(prompt, output_type=List[PsychTriple])

Instructor:

• 📜 Formats the prompt to include JSON schema hints
• ✅ Parses and validates the model output
• 🔁 Retries automatically if the output is malformed or incomplete

It turns a generic language model into something that behaves more like an API client for structured responses.

2. 🧱 Pydantic

Pydantic is a Python library for defining and validating structured data with types and constraints. Think of it as JSON Schema, but in Pythonic form.

Here’s the schema I use for extracting psychological triples:

class PsychTriple(BaseModel):
    topic_or_construct: str
    measured_by: str
    justification: str

This gives me:

• ⚖️ Type validation
• ✅ Auto-parsing from model output
• 🔍 Clear error messages when something goes wrong

And because Instructor directly integrates with Pydantic, I can just say:

output_type=List[PsychTriple]

…and the LLM is bound to that schema. No post-processing needed.

3. 🔮 The Magic Duo: Instructor + Pydantic

Together, these libraries let you build a typed interface on top of open-source language models.

You get:

• The openness and flexibility of Hugging Face models
• The structure enforcement of Pydantic
• The automation and retry logic of Instructor

Result: A modular, robust pipeline for structured information extraction from unstructured text.

4. 🔍 Use Case: Extracting Knowledge Triples from Psychology Papers

Let’s walk through the full script I use to extract structured information from .xml psychology articles encoded in the TEI format. The goal is to populate a lightweight knowledge graph with the relationships between psychological constructs and their measurement instruments, along with a justification from the paper.

🔧 Start off with a small set of import statements for the relevant libraries

import os
import json
from pathlib import Path
from typing import List
from bs4 import BeautifulSoup

from pydantic import BaseModel, ValidationError
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
from instructor import Instructor, patch_instructor
from tqdm import tqdm
import torch

🧱 Step 1: Define the schema

class PsychTriple(BaseModel):
    topic_or_construct: str
    measured_by: str
    justification: str

📄 Step 2: Extract text from TEI XML

def extract_text_from_tei_xml(tei_path: str) -> str:
    try:
        with open(tei_path, 'r', encoding='utf-8') as file:
            soup = BeautifulSoup(file, 'lxml-xml')
            body = soup.find('text')
            return body.get_text(separator='\n') if body else ''
    except Exception as e:
        print(f"Error parsing {tei_path}: {e}")
        return ""

✍️ Step 3: Build the prompt with domain-specific instructions

def build_prompt(text: str) -> str:
    return f"""
You are an expert in psychology and computational knowledge representation. Your task is to extract key scientific information from psychology research articles to build a structured knowledge graph.

The knowledge graph aims to represent the relationships between psychological **topics or constructs** and their associated **measurement instruments or scales**. Specifically, for each article, extract information in the form of triples that capture:

1) The psychological topic or construct being studied  
2) The measurement instrument or scale used to assess it  
3) A brief justification (1–3 sentences) from the article text supporting this measurement link

Guidelines:
- Extract meaningful **phrases** (not full sentences or vague descriptions) for both `topic_or_construct` and `measured_by`, suitable for inclusion in a knowledge graph.
- Include a short justification for each extraction that clearly supports the connection.
- If the article does not discuss psychological constructs and how they are measured (e.g., no mention of constructs, instruments, or scales), return an empty list `[]`.

Input Paper:
\"\"\"{text}\"\"\"

Output: Provide your response as a JSON list in the following format:

[
  {{
    "topic_or_construct": "...",
    "measured_by": "...",
    "justification": "..."
  }},
  ...
]
""".strip()

This prompt guides the LLM to extract structured triples — only if the article contains relevant content.

📤 Step 4: Process each file using Instructor

def process_file(file_path: str, prompt_dir: str, output_dir: str, llm: Instructor) -> bool:
    text = extract_text_from_tei_xml(file_path)
    if not text:
        print(f"⚠️ Skipping empty or malformed text in {file_path}")
        return False

    prompt = build_prompt(text[:100000])
    prompt_path = Path(prompt_dir) / (Path(file_path).stem + ".prompt.txt")
    output_path = Path(output_dir) / (Path(file_path).stem + ".json")

    with open(prompt_path, "w", encoding="utf-8") as f:
        f.write(prompt)

    try:
        results = llm(prompt, output_type=List[PsychTriple])
        if not results:
            print(f"✅ {file_path} | Extracted 0 triples")
            return False

        with open(output_path, "w", encoding="utf-8") as f:
            json.dump([r.model_dump() for r in results], f, indent=2)
        print(f"✅ {file_path} | Extracted {len(results)} triples")
        return True
    except ValidationError as ve:
        print(f"❌ Validation failed for {file_path}:\n{ve}")
    except Exception as e:
        print(f"❌ Failed to process {file_path}: {e}")
    return False

Prompts and structured outputs are saved per file, and invalid or irrelevant files are gracefully skipped.

🚀 Step 5: Run the pipeline with an LLM on Hugging Face

def main():
    input_dir = input("Enter input directory with .xml files: ").strip()
    prompt_dir = input("Enter directory to save prompts: ").strip()
    output_dir = input("Enter directory to save JSON outputs: ").strip()
    model_name = input("Enter Hugging Face model name (e.g., Qwen/Qwen2.5-72B-Instruct): ").strip()

    for d in [input_dir, prompt_dir, output_dir]:
        os.makedirs(d, exist_ok=True)

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        device_map="auto",
        torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32
    )

    text_gen_pipeline = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_new_tokens=8000,
        return_full_text=False
    )

    llm = Instructor(text_gen_pipeline)

    files = list(Path(input_dir).glob("*.xml"))
    if not files:
        print(f"⚠️ No .xml files found in {input_dir}")
        return

    total_articles = 0
    satisfied_articles = 0

    for xml_file in tqdm(files, desc="Processing TEI XML files"):
        total_articles += 1
        success = process_file(str(xml_file), prompt_dir, output_dir, llm)
        if success:
            satisfied_articles += 1

    print("\n=== Extraction Summary ===")
    print(f"Total articles processed: {total_articles}")
    print(f"Articles with extracted constructs and measures: {satisfied_articles}")
    print(f"Articles without relevant content: {total_articles - satisfied_articles}")

🪷 But Wait — What Is Instructor Actually Doing?

You might wonder: Couldn’t I do this with just a prompt, a for loop, and some json.loads() calls?

Here’s what Instructor is really doing for you behind the scenes:

1. Prompt Engineering with Schema Injection

It takes your Pydantic model (e.g., List[PsychTriple]) and:

• Converts it into a schema hint
• Embeds this schema into the prompt automatically
• Encourages the LLM to emit structured output that matches

2. Parsing + Validation

Once the LLM responds, Instructor:

• Parses the output into JSON
• Validates every record against your schema using Pydantic
• If it doesn’t match (e.g., wrong types, missing fields), it raises errors

3. Automatic Retries

If the model output is malformed or doesn’t pass validation, Instructor:

• Automatically re-prompts the model
• Optionally adds error hints (if using patch_instructor)
• Retries multiple times until a valid response is received or the retry limit is reached

4. Clean, Typed Output

Rather than raw strings or loose dictionaries, Instructor returns Pythonic structured objects, ready for downstream use:

🏁 Finally… What I Got

After running the pipeline:

• 🗂️ Clean, validated JSON outputs per article
• ⚙️ Consistent, typed data ready for ingestion into a knowledge graph
• 🧾 Full traceability (saved prompts per input file)
• ✅ Zero parsing bugs

🧩 Summary

Instructor + Pydantic + Hugging Face = Structured Generation Made Easy
If you work with messy scientific text and want to turn it into something structured, this trio is a game-changer.

🚀 Bonus: Want to Go Further?

• 🧪 Add unit tests with mockLLM responses
• 📊 Analyze extraction patterns across corpora
• 🕸️ Build a Streamlit app to browse your structured results

🧠 Final Thought

LLMs don’t have to speak in riddles. With the right tools, they can speak structured data fluently.

Instructor and Pydantic bring that into reach — today, and fully open-source.

How do you handle structured data with LLMs?
Do you use parsing tricks, few-shot prompting, function calling, or something else entirely?
I’d love to hear your workflows, hacks, or questions in the comments or replies.

Until then — may your JSONs be valid and your triples meaningful. ✨

🔗 Resources

• 📦 Code repository: https://github.com/sciknoworg/psychKG-pilot/
• ✍️ Related post: How I Use Instructor + Pydantic with OpenAI for Structured Outputs

Atomic Layer Deposition (ALD) [1] is a technique used to coat materials with extremely thin layers, one atomic layer at a time. This process allows for very precise control over the thickness and composition of the coating, making it ideal for applications where such precision is crucial, like in electronics and nanotechnology. Essentially, it’s like painting with atoms, using a method that builds up layers step-by-step to get the desired finish.

In ALD, the “process material” and “process reactants” are pivotal elements. The process material, which is the substrate on which atomic layers are deposited, significantly influences the adherence and quality of the film due to its chemical composition, surface structure, and temperature. Meanwhile, process reactants are the chemicals used to build these layers. ALD involves the sequential exposure of the substrate to these reactants, each interacting with the surface in a self-limiting manner to deposit a single atomic layer per cycle. This precise interaction dictates the film’s final composition. For instance, creating a thin film of alumina (Al₂O₃) might involve using trimethylaluminum and water as reactants. Such meticulous control is crucial for applications in electronics, optics, and other fields, ensuring uniformity and exactness in film thickness and composition.

With this in mind, in this post, I introduce and discuss a novel way to report ALD processes in knowledge graphs (KGs), specifically the Open Research Knowledge Graph (ORKG) [2], such that the resulting data is AI-ready. I also discuss a lightweight, yet effective, approach to leverage LLMs to retrospectively extract information from existing research articles to populate an AI-ready ORKG focused on ALD process information.

Utilizing Semantic Web [3] tools such KGs to create AI-ready ALD databases transforms traditional ALD process reporting, enhancing discoverability, interoperability, and semantic richness of data. These structured databases allow for more dynamic updates and collaborations, enabling automated reasoning to infer new knowledge and make predictions about ALD processes. Knowledge graphs make ALD data more AI-ready by structuring it in a way that is directly usable by machine learning models, which can predict new material properties or optimize process parameters more efficiently than traditional statistical approaches. By making ALD data more accessible and actionable, KGs empower researchers to harness advanced analytical and predictive capabilities, pushing the boundaries of what’s possible in the field.

Knowledge graphs are used to store interconnected descriptions of entities — objects, events, or concepts — with data that is interlinked and semantically understandable both by human operators and machines. They leverage Semantic Web technologies like ontologies, RDF (Resource Description Framework), and SPARQL to link data across domains and enable better data integration, querying, and reasoning. By doing so, KGs play a key role in making data interoperable and accessible for AI and machine learning applications.

The first step to creating a ALD-process-related KG is having a schema, i.e. a set of properties that capture the salient ALD process information. Such a schema is shown in the 41 lines JSON file below.

While this schema presents various opportunities for refinement w.r.t. “Linking temperature and pressure units to standard unit ontologies (QUDT ontology) or materials science ontologies (EMMO, MatOnto)” [4], it offers a good starting point for additional details to document ALD processes in structured way. Now since the end goal of this exercise is to create a AI-ready database in the ORKG, I went ahead and defined this schema as a template in the ORKG. Here is a link to the template https://orkg.org/template/R733029. A UML-like class diagram illustration is offered in Figure 2 below.

Then, one needs to compile a list of ALD research papers to extract structured ALD process descriptions from based on the schema. I harnessed the AtomicLimits ALD database — an esteemed resource meticulously curated by experts in the field. You can explore it here. It addresses the explosive growth in ALD processes, catering to an ever-expanding array of materials, and thereby improving the findability of relevant research. Given the development of a huge number of ALD processes for a very large set — and still growing number — of materials, researchers at Eindhoven University of Technology led by Prof. Erwin Kessels created an online database where the expert community could submit the research materials and reactants linked to research papers. Here is the post that introduced this database to the community. Moreover, the database has been creatively visualized as an ALD periodic table, making it not only a practical tool but also a visually engaging one for users.

After compiling the list of papers from the ALD database as well as carrying out supporting task for knowledge extraction, i.e. getting the full text of the papers to mine the knowledge using the ORKG Ask API, I shifted focus to the LLM and, specifically, to defining the knowledge extraction prompt. The OpenAI GPT models is a relatively easy-to-setup starting point since once can directly use their API to query the models without being concerned about GPU resources to host an LLM in-house. Here I then decided to use the recent GPT-4o LLM and leveraged the Assistants API. I went ahead and defined the system prompt. A system prompt defines the GPT assistant role to align its behavior (“persona”) to the end task. On the other hand, as another class of prompts, user prompts are then used to interact with this role-aligned GPT assistant providing queries or statements or input text from which one wants to extract data expecting it to perform a task successfully. In keeping with the in-context learning paradigm first introduced in the GPT-3 paper by Brown et al., the user prompt can also contain few-shot examples that demonstrate the successful task completion. Anyway, going back to the system prompt definition for the ALD process structured information extraction, Figure 3 below illustrates the final prompt used.

Here’s an exercise you can try: as a quick enhancement to the above workflow, modify the code with Instructor python library that makes it easy to work with structured outputs from LLMs. Another one: replace the OpenAI API with LiteLLM instead that makes it easier to query various other LLMs.

Summing things up, yes, the post so far has covered several aspects of establishing an end-to-end workflow to compile research knowledge in a complex scientific domain using an LLM. It began by introducing the ALD domain and outlining the schema for knowledge extraction that would shape the research knowledge graph. It then explored the main tool in question, the LLM, which, with its vast parameter space equipping it with generative AI capabilities, could be applied as a potent tool for the knowledge extraction task in this complex domain and is expected to handle it with reasonable accuracy. I will leave out the specific topic of evaluating the LLM output and defining knowledge curation workflows for another post.

Equipped with the structured knowledge, the last step was storing it in the ORKG as an AI-ready database. For this, various modules of the ORKG Python Package can employed to programmatically upload the structured data, aligning it with the Comprehensive ALD Profile ORKG semantic template. And, voila, we have compiled a research knowledge graph for a complex scientific domain using an LLM!

Outcomes and Impact

Sample ORKG Paper: View an example of an imported ORKG paper, titled “Atomic Layer Deposition of CsI and CsPbI3”, which includes structured ALD process descriptions for the two materials CsI and CsPbI3, respectively, as detailed, machine-actionable data.

• CsI Contribution: https://orkg.org/paper/R732976/R739356
• CsPbI3 Contribution: https://orkg.org/paper/R732976/R739360

ORKG Comparison as a ALD Process Research Overview: Explore a comparative overview of 167 contributions on ALD processes illustrated as a snapshot in Figure 4 below. This comparison provides a structured summary, facilitating a quick review of research findings and methodologies on ALD processes.

• View Comparison: https://orkg.org/comparison/R739481

The workflow in this post is defined in the context “AI-Aware Pathways to Sustainable Semiconductor Process and Manufacturing Technologies (AWASES)” research project funded by Merck and Intel co-operations. Collaborating together with researchers from Eindhoven University, L3S Research Center, and University of Warwick, one of our aims is to develop AI-ready databases by extracting and integrating knowledge from ALD/E (Atomic Layer Deposition/Etching) processes. We build on existing crowdsourced databases, such as the one launched by TU/e in 2019 (TU/e Atomic Limits ALD Database, DOI: 10.6100/alddatabase). Our goal is to make these databases ready for AI applications, fostering innovations in materials design, autonomous experimentation, and AI-driven process development [5]. As a step toward this, with the structured ALD process data imported in the ORKG, users can navigate through the ORKG platform to access papers or comparisons. Specifically the relevant papers can be found in our AWASES project observatory. This integration supports enhanced discovery and research analytics capabilities, promoting a deeper understanding and further research in ALD technology.

Finally, I’ve made the scripts to reproduce the workflow of this post publicly available on Github https://github.com/jd-coderepos/awases-ald/.

🌟 ~ Until next time, keep exploring and stay curious! Cheers! ~ 🌟

As parting words, I’d like to leave you with some recommendations that I’ve found particularly relevant and effective for handling text data mining tasks, especially when working with scientific articles…

Additional Resources for Text Data Mining — A Small Tool List (cf: I couldn’t recommend enough this excellent resource for additional tools https://matextract.pub/index.html [4])

1. crossrefapi — pythonic library to get metadata of relevant scientific articles from various sources via crossref. Usage example: 100 sources including metadata on the topic ‘buchwald-hartwig coupling’ are extracted and saved into a JSON file https://matextract.pub/content/obtaining_data/crossref_search.html
2. pygetpapers — articles from open-access databases can be collected using this library with support for multiple open access APIs including eupmc, crossref, arXiv, bioRxiv, medRxiv, rxivist-bio, rxivist-med
3. paperscraper — a python package for scraping publication metadata or full PDF files from PubMed or preprint servers such as arXiv, medRxiv, bioRxiv and chemRxiv. Usage example: download full-text articles from ChemRxiv on the topic of ‘buchwald-hartwig coupling’ https://matextract.pub/content/obtaining_data/data_mining.html
4. marker — a python library to convert PDF to markdown quickly with high accuracy; has been compared to nougat.
5. docTR (Document Text Recognition) — a seamless, high-performing & accessible library for OCR-related tasks powered by Deep Learning. Documents can be interpreted from PDF or images.

References

1. Steven M. George (2010). Atomic layer deposition: an overview. Chemical reviews, 110(1), 111–131.
2. Sören Auer, Allard Oelen, Muhammad Haris, Markus Stocker, Jennifer D’Souza, Kheir Eddine Farfar, Lars Vogt, Manuel Prinz, Vitalis Wiens, and Mohamad Yaser Jaradeh (2020). Improving Access to Scientific Literature with Knowledge Graphs. Bibliothek Forschung und Praxis, vol. 44, no. 3, pp. 516–529 https://doi.org/10.1515/bfp-2020-2042
3. Tim Berners-Lee, James Hendler, and Ora Lassila (2023). The Semantic Web: A new form of Web content that is meaningful to computers will unleash a revolution of new possibilities. In Linking the World’s Information: Essays on Tim Berners-Lee’s Invention of the World Wide Web, pp. 91–103 https://doi.org/10.1145/3591366.3591376
4. Mara Schilling-Wilhelmi, Martiño Ríos-García, Sherjeel Shabih, María Victoria Gil, Santiago Miret, Christoph T. Koch, José A. Márquez, and Kevin Maik Jablonka (2024). From Text to Insight: Large Language Models for Materials Science Data Extraction. arXiv preprint arXiv:2407.16867.
5. Adrie Mackus, Bart Macco, Bora Karasulu, Jennifer D’Souza, Sören Auer, & Erwin Kessels. Turning Online ALD and ALE Databases Into AI-Ready Tools for Development of New Sustainable Materials and Fabrication Processes. Poster presented at AVS 24th Int. Conf. on Atomic Layer Deposition (ALD 2024). View Poster

So, you’ve read up on the latest Large Language Models (LLMs) like GPT, Llama, and Mistral, and you’re intrigued by their potential. But how do they really perform when tasked with the heavy lifting of structured science summarization within the Open Research Knowledge Graph (ORKG)? And more importantly, how can their performance be accurately evaluated? You’re not alone in asking these questions.

In this post, I’ll walk you through a comprehensive study that not only tests these LLMs in the realm of “structured science summarization” but also details the multifaceted evaluation approach to assessing their effectiveness. From semantic alignment to expert evaluations, we’ll explore how these models have the potential to transform the landscape of scientific literature summarization. Whether you’re developing your own application or simply curious about the future of AI in academia, understanding these evaluation strategies will provide crucial insights into the capabilities and current limitations of LLMs.

So, what are structured science summaries? One notable initiative aimed at publishing structured representations of scientific papers is the Open Research Knowledge Graph (ORKG, https://orkg.org/ accessed on 9 September 2024) [1]. The ORKG endeavors to describe papers as structured summaries in a KG in terms of salient properties of the research. Furthermore, a distinguishing characteristic of the properties is that they are also generically applicable across various contributions on the same problem, thus making the structured paper descriptions comparable. For instance, properties like ‘model name’, ‘model family’, ‘pretraining architecture’, ‘number of parameters’, and ‘hardware used’ — illustrated in Figure 1 — effectively create structured, machine-actionable summaries of research on ‘transformer models’ within Computer Science.

As another example, papers with the research problem ‘DNA sequencing techniques’ in the domain of Biology can be described as structured summaries based on the following properties: ‘sequencing platform’, ‘read length in base pairs’, ‘reagents cost’, and ‘runtime in days’ (Figure 2).

Such comparisons aggregate structured summaries based on common research themes, simplifying how researchers keep up with advancements through targeted, strategic reading over large volumes of papers. The ORKG aims to revolutionize the summarization and comparison of scientific papers by structuring them into an interconnected knowledge graph. Traditionally, this process has been manual, time-consuming, and prone to inconsistency. The introduction of LLMs promises to automate and standardize this process, potentially transforming the landscape of scientific research documentation. In a recent work [2], we applied 3 LLMs, viz. GPT, Llama, and Mistral, on a multidiscplinary corpus (publicly accessible here https://doi.org/10.25835/6oyn9d1n) of various research themes and their paper collections compiled from the ORKG prompting the LLMs to generate their structured properties. Note human-annotated structured descriptions were also obtained from the ORKG as a reference for evaluating the LLM performance.

Turning then to the central question of the study… “How good are LLMs in performing structured science summarization? Can they replace humans?” This is where I now emphasize the comprehensive and multifaceted evaluation approach of the LLM performance from this study. This is outlined by the acronym SMES.

Semantic alignment measurement of the LLM-generated properties vs. human annotations. In this context, semantic alignment refers to the degree to which the two sets of properties share similar meanings. GPT-3.5 was applied as the evaluator and was prompted to produce an alignment rating on a scale from 1 to 5. Conversely, semantic deviation was also tested.

Mappings counted between the LLM-generated properties vs. human annotations. This approach diverges from the previous semantic alignment and deviation evaluations, which considered the lists as a whole. This approach used GPT-3.5 to count the number of overlapping properties. A snapshot of the prompt is shown in Figure 4.

Embedding-based approach to compute semantic distances between LLM-generated and human-annotated properties. Specifically,
a scientific language model, like SciNCL, can be applied to generate vector embeddings for both the ORKG properties and the LLM-generated research dimensions. The embeddings can then be compared using cosine similarity as a measure of distance.

Survey soliciting opinions from human participants was designed to solicit the impressions of domain experts when shown their original annotated properties versus the research dimensions generated by the LLM. The survey questionnaire is shown in Figure 5.

With SMES, the reader is drawn to the observation that evaluating the performance of LLMs is more often than not — multifaceted!

And there you have it! The journey through the capabilities of LLMs in structured science summarization reveals both their potential and limitations. While LLMs like GPT, Llama, and Mistral have demonstrated impressive strides towards understanding and structuring scientific knowledge, the study underscores the need for more tuning and targeted training. The multifaceted evaluation SMES strategy — ranging from semantic alignment to expert surveys — highlight that while LLMs can significantly aid the research process, they aren’t quite ready to replace human expertise. Instead, they serve as robust tools, augmenting human efforts to manage and navigate the burgeoning landscape of scientific literature.

So, as we continue pushing the boundaries of what these AI models can achieve, let’s remain both critical and optimistic. The future of AI in science is not just about replacing the old with the new but enhancing our capabilities and reaching new heights of innovation and efficiency. Stay tuned, and keep experimenting, because the next breakthrough in AI-powered research could be just around the corner!

References

[1] Sören Auer, Allard Oelen, Muhammad Haris, Markus Stocker, Jennifer D’Souza, Kheir Eddine Farfar, Lars Vogt, Manuel Prinz, Vitalis Wiens, and Mohamad Yaser Jaradeh (2020). Improving Access to Scientific Literature with Knowledge Graphs. Bibliothek Forschung und Praxis, vol. 44, no. 3, pp. 516–529 https://doi.org/10.1515/bfp-2020-2042

[2] Vladyslav Nechakhin, Jennifer D’Souza, and Steffen Eger (2024). Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph. Information 2024, 15, 328. https://doi.org/10.3390/info15060328