Multi-model AI advisory board (building LLM Councils)
Credit to Andrej Karpathy for this one: https://github.com/karpathy/llm-council
Instead of asking a question to your preferred LLM provider (e.g. OpenAI, Google Gemini, Anthropic Claude), you can group them into your “LLM Council”.
It’s a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, and then asks them to review and rank each other’s work, and finally a Chairman LLM produces the final response.
This was a project 99% vibe coded as a weekend hack because Andrej wanted to explore and evaluate a number of LLMs side by side.
Step 1: 🗣️ The Council Answers
- What happens: You give your question to a group of different AIs (like GPT-5.1, Claude 4.5, Gemini 3 Pro, etc.).
- Goal: Each AI gives its own independent answer to your question.
Step 2: ⚖️ The Peer Review
- What happens: Another group of AIs reads all the answers from Step 1. They don’t know which AI wrote which answer (it’s anonymous).
- Goal: The AIs critique and check each other’s work, pointing out strengths and weaknesses in the reasoning.
Step 3: 🧠 The Final Decision (The Chairman)
- What happens: One designated “Chairman” AI looks at the original answers and all the critiques from Step 2.
- Goal: The Chairman combines the best parts of the discussion to create one final, comprehensive and definitive answer.
Step 4: 📩 Send the Report
- What happens: The final answer is tidied up and sent to you as an easy-to-read email report.
Tip: Setting this up in a tool like n8n (AI workflow automation) is more straightforward than expected, just linking one step to the next.
You can use something like Gemini to help write the prompts for the Peer Review and Chairman steps, examples here:
Prompt Title: Anonymous Peer Analysis and Critique
Get David Ahmed Walby’s stories in your inbox
Join Medium for free to get updates from this writer.
Instructions: You have been provided with [N number] different, competing answers to the same user question. Your task is to critique each answer based on its reasoning, completeness, and factual accuracy. Do not compare them to each other directly; evaluate each one on its own merit.
- User Question: [Insert the original user prompt here]
- Answer 1: [Insert the first LLM’s answer]
- Answer 2: [Insert the second LLM’s answer]
- (… continue for all N answers …)
For each Answer, provide a separate, structured critique under the corresponding heading:
Answer [1, 2, 3, etc.] Critique:
- Strengths: What is the most insightful or helpful part of this response? Is the reasoning sound? (Be specific)
- Weaknesses: Where is the response incomplete, vague or potentially incorrect? What is it missing?
- Suggestion: If this answer were to be the final response, what is one concrete piece of information or clarification it needs to be definitive?
🧠 The Chairman Prompt
This prompt should be given to the final “Chairman” AI, which will synthesise the initial answers and the critiques.
Prompt Title: Definitive Answer Synthesis and Final Report
Instructions: You are the Chairman. Your task is to create the single, most comprehensive, and definitive final answer. You must synthesize the best parts of the initial answers while rigorously addressing all weaknesses identified in the peer critiques.
- Original User Question: [Insert the original user prompt here]
- Initial Council Answers (for context): [Insert all raw answers from Step 1]
- Peer Review Critiques (for guidance): [Insert all structured critiques from Step 2]
Your final answer must meet these criteria:
- Completeness: It must fully address all aspects of the original user question.
- Accuracy: It must filter out any factual errors or inconsistencies noted in the critiques.
- Clarity: It must be structured logically and easy to read.
- Synthesis: It must integrate the strongest arguments and unique insights from the different council members.
Begin your final, definitive answer now.
