Run Stanford MedHELM on Any vLLM Model with One Bash Script

What Is HELM?

In 2022, Stanford’s Center for Research on Foundation Models (CRFM) released HELM — Holistic Evaluation of Language Models. The premise was straightforward: stop evaluating AI models on cherry-picked benchmarks and start evaluating them holistically — across dozens of tasks simultaneously, with multiple metrics, on a standardized reproducible infrastructure.

HELM answered a question the field had been avoiding: how do you compare models fairly when every lab runs different tests? It became the reference leaderboard for foundation model evaluation, maintained at github.com/stanford-crfm/helm, and provides a unified pipeline covering prompt construction, model querying, metric computation, and a web-based results dashboard.

What Is MedHELM?

General language understanding and clinical reasoning are not the same thing. A model that aces the SAT may still hallucinate drug dosages or produce dangerous triage advice.

In early 2025, Stanford HAI announced MedHELM — an extension of HELM built specifically for medicine. It was developed through a collaboration between Stanford’s Center for Biomedical Informatics Research (BMIR), Stanford Health Care’s Technology and Digital Solutions team, and Microsoft Health and Life Sciences, working alongside clinicians from 15 medical specialties.

MedHELM is a structured taxonomy of clinical AI use cases validated by 29 practicing clinicians. The framework covers:

• 5 categories: Clinical Decision Support, Clinical Note Generation, Patient Communication and Education, Medical Research Assistance, and Administration and Workflow
• 22 subcategories spanning the breadth of clinical AI applications
• 121 distinct clinical tasks
• 35 benchmarks across public, gated-access, and private datasets

Tasks range from computing clinical scores from patient notes, to summarizing doctor-patient conversations, to SQL generation over EHR databases, to evaluating race-based medical bias. Each task has a defined prompt, reference output, and a clinically appropriate evaluation metric.

The full leaderboard is live at crfm.stanford.edu/helm/medhelm/latest.

The Problem: Evaluation Is Hard to Run

Running MedHELM yourself means navigating a complex setup: managing Python dependencies, making various custom config files and yaml files defining their tasks and special attributes and format.

So i combined all hustle into one script where just pass you model endpoint and name and it will do all the rest for you.

The Solution: One Script

We built medhelm.sh — a single bash script that handles the complete MedHELM evaluation lifecycle against any local vLLM server. Point it at your model, run it, and come back to a results dashboard.

The script is fully idempotent: if a run is interrupted, re-running it skips completed tasks and resumes from where it left off.

Here is what happens under the hood:

Environment setup — Creates an isolated Python venv, installs HELM directly from source at a pinned release, and resolves all dependency requirements automatically.

Model configuration — Generates the HELM config files (model_deployments.yaml, model_metadata.yaml, credentials.conf) from variables at the top of the script. Changing models is a one-line edit.

Data preparation — Checks that all required benchmark datasets are in place and handles the download requirements of each data source.

Per-task execution — Runs all 22 tasks individually rather than as a single batch. One failing task never blocks the others. A summary at the end shows exactly which tasks passed and which failed.

Get Shaaf Salman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Results dashboard — Runs helm-summarize and helm-server automatically, making results available at http://localhost:8080.

Configuration

At the top of the script, set your model details:

MODEL_NAME="Qwen3.5-27B"
VLLM_BASE_URL="http://localhost:8000/v1"
SUITE="medhelm-qwen35-27b"
MAX_EVAL_INSTANCES=100
MAX_TOKENS=32768
MODEL_ID="local/qwen3.5-27b"
TOKENIZER_ID="qwen/qwen2.5-7b-instruct"

That is the only configuration needed. Everything else is handled automatically.

Prerequisites

• Ubuntu 20.04+ with Python 3.10
• NVIDIA GPU with CUDA 12.x
• vLLM serving your model on localhost:8000
• git, wget, curl in PATH

Then, Download this file

helm/medhelm.sh at main · shaafsalman/helm

and just run with

chmod +x medhelm.sh
bash medhelm.sh

On a single A100 80GB GPU at MAX_EVAL_INSTANCES=100, a full 22-task run completes in approximately 1 hours approx.

What You Get

After the run completes you have:

• Per-task output files including scores, per-instance predictions, and scenario metadata
• A helm-server web dashboard at http://localhost:8080 with full comparisons and drill-down views
• A timestamped log of the complete run

Conclusion

MedHELM is the most rigorous publicly available framework for evaluating LLMs on clinically grounded tasks. Its scope reflects a serious, multi-stakeholder effort to measure what actually matters in healthcare AI — not exam performance, but real clinical utility across 121 tasks and 35 benchmarks.

Running it should not require days of setup. With this script, it requires one command.

References

• MedHELM Leaderboard — crfm.stanford.edu/helm/medhelm/latest
• Stanford HAI Announcement — hai.stanford.edu/news/holistic-evaluation-of-large-language-models-for-medical-applications
• HELM Documentation — crfm-helm.readthedocs.io/en/latest/medhelm
• HELM GitHub — github.com/stanford-crfm/helm
• My Script URL: https://github.com/shaafsalman/helm/blob/main/medhelm.sh