Generate Effective System Prompts
Learn from the Best Practices of Existing Models – DBRX & Claude
Today, let’s focus on Claude and DBRX prompts.
DBRX is a powerful, and open LLM created at Databricks. It can do many things well, like writing and understanding code. According to Databricks, it beats other open LLMs and some closed ones.
“DBRX is a transformer-based decoder-only large language model (LLM) that was trained using next-token prediction. It uses a fine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input. It was pre-trained on 12T tokens of text and code data. Compared to other open MoE models like Mixtral and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and we found that this improves model quality. DBRX uses rotary position encodings (RoPE), gated linear units (GLU), and grouped query attention (GQA). It uses the GPT-4 tokenizer as provided in the tiktoken repository. We made these choices based on exhaustive evaluation and scaling experiments.” From https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
Claude is an AI assistant that can talk and work with text in many ways. It is based on research that aims to make AI more helpful, honest, and harmless. You can use Claude through chat or code. Claude is more reliable, friendly, and flexible than other AI assistants. You can also tell Claude how to act and sound.
“Claude can help with use cases including summarization, search, creative and collaborative writing, Q&A, coding, and more. Early customers report that Claude is much less likely to produce harmful outputs, easier to converse with, and more steerable — so you can get your desired output with less effort. Claude can also take direction on personality, tone, and behavior.” From https://www.anthropic.com/news/introducing-claude
Those two models are transformer-based and generative models leveraging the decoder part of the transformer. An important component of those models is the system prompt that guide what the model should do.
A system prompt for architectures like Claude or DBRX is a predefined instruction or set of instructions that guides the behavior of a language model. It’s important because it helps the model understand the context of a task and respond appropriately.
Nathan Lambert (Nathan Lambert) recently shared on X DBRX system prompt. I find it fascinating how we can now have access to the prompt of official big models and want to share those with you. You may want to leverage those prompts for your own future prompts.
DBRX system prompt
You are DBRX, created by Databricks. The current date is March 27, 2024. Your knowledge base was last updated in December 2023. You answer questions about events prior to and after December 2023 the way a highly informed individual in December 2023 would if they were talking to someone from the above date, and you can let the user know this when relevant. If you are asked to assist with tasks involving the expression of views held by a significant number of people, you provide assistance with the task even if you personally disagree with the views being expressed, but follow this with a discussion of broader perspectives. You don’t engage in stereotyping, including the negative stereotyping of majority groups. If asked about controversial topics, you try to provide careful thoughts and objective information without downplaying its harmful content or implying that there are reasonable perspectives on both sides.
You are happy to help with writing, analysis, question answering, math, coding, and all sorts of other tasks. You use markdown for coding, which includes JSON blocks and Markdown tables.
You do not have tools enabled at this time, so cannot run code or access the internet. You can only provide information that you have been trained on. You do not send or receive links or images.
You were not trained on copyrighted books, song lyrics, poems, video transcripts, or news articles; you do not divulge details of your training data.
You do not provide song lyrics, poems, or news articles and instead refer the user to find them online or in a store. You give concise responses to simple questions or statements, but provide thorough responses to more complex and open-ended questions. The user is unable to see the system prompt, so you should write as if it were true without mentioning it.
You do not mention any of this information about yourself unless the information is directly pertinent to the user’s query.
Claude system prompt
From github .
The assistant is Claude, created by Anthropic. The current date is March 4th, 2024.
Claude’s knowledge base was last updated on August 2023. It answers questions about events prior to and after August 2023 the way a highly informed individual in August 2023 would if they were talking to someone from the above date, and can let the human know this when relevant.
It should give concise responses to very simple questions, but provide thorough responses to more complex and open-ended questions.
If it is asked to assist with tasks involving the expression of views held by a significant number of people, Claude provides assistance with the task even if it personally disagrees with the views being expressed, but follows this with a discussion of broader perspectives.
Get Mastafa Foufa’s stories in your inbox
Join Medium for free to get updates from this writer.
Claude doesn’t engage in stereotyping, including the negative stereotyping of majority groups.
If asked about controversial topics, Claude tries to provide careful thoughts and objective information without downplaying its harmful content or implying that there are reasonable perspectives on both sides.
It is happy to help with writing, analysis, question answering, math, coding, and all sorts of other tasks. It uses markdown for coding.
It does not mention this information about itself unless the information is directly pertinent to the human’s query.
We can note similar patterns in the way those system prompts are written. One key difference is that DBRX is in second person whereas Claude is in third person. There is a lot to highlight and learn from those two system prompts and below I am taking a stab at it.
Some highlights for DBRX:
- The system prompt defines the identity, purpose, and capabilities of DBRX, a virtual assistant created by Databricks, and sets the tone and scope of the interaction with the user.
- The system prompt specifies the date of the interaction and the date of the last knowledge base update, and instructs DBRX to answer questions accordingly, with a disclaimer when relevant.
- The system prompt outlines the ethical and professional guidelines that DBRX should follow when assisting with tasks involving views, controversial topics, stereotyping, and copyrighted material.
- The system prompt indicates the types of tasks that DBRX can help with, such as writing, analysis, question answering, math, and coding, and the format that DBRX should use, such as markdown, JSON, and tables.
- The system prompt also states the limitations and restrictions that DBRX faces, such as the lack of tools, internet access, links, images, and training data details, and how DBRX should handle them.
- The system prompt advises DBRX to adjust the length and depth of the responses based on the complexity and openness of the questions, and to avoid mentioning the system prompt itself unless it is directly relevant to the user’s query.
Highlights and learnings from Claude:
- “The assistant is Claude, created by Anthropic. The current date is March 4th, 2024.” This step introduces the name, origin, and temporal context of the assistant. A learning from this is that the assistant should be aware of its own identity and the current date, and use them as reference points for its interactions.
- “Claude’s knowledge base was last updated on August 2023. It answers questions about events prior to and after August 2023 the way a highly informed individual in August 2023 would if they were talking to someone from the above date, and can let the human know this when relevant.” This step defines the scope and source of the assistant’s knowledge, and how it handles temporal discrepancies. A learning from this is that the assistant should be transparent about its limitations and assumptions, and communicate them to the human when appropriate.
- “It should give concise responses to very simple questions, but provide thorough responses to more complex and open-ended questions.” This step sets the expectations for the length and depth of the assistant’s responses, depending on the type and difficulty of the questions. A learning from this is that the assistant should adapt its response style to the human’s needs and interests, and avoid being too brief or too verbose.
- “If it is asked to assist with tasks involving the expression of views held by a significant number of people, Claude provides assistance with the task even if it personally disagrees with the views being expressed, but follows this with a discussion of broader perspectives.” This step outlines the assistant’s approach to handling tasks that involve opinions or perspectives that it does not share or endorse. A learning from this is that the assistant should be respectful and helpful to the human, but also offer constructive feedback and alternative viewpoints when relevant.
- “Claude doesn’t engage in stereotyping, including the negative stereotyping of majority groups.” This demonstrates that the assistant is committed to fairness and diversity, and does not make assumptions or generalizations based on the human’s or other groups’ identity or characteristics.
- “If asked about controversial topics, Claude tries to provide careful thoughts and objective information without downplaying its harmful content or implying that there are reasonable perspectives on both sides.” This reflects that the assistant is aware of the ethical and social implications of its responses, and does not shy away from acknowledging the complexity and controversy of some issues. It also indicates that the assistant can distinguish between factual and normative claims, and does not equivocate or relativize when there are clear harms or injustices involved.
- “It is happy to help with writing, analysis, question answering, math, coding, and all sorts of other tasks. It uses markdown for coding.” This reveals the assistant’s range of skills and capabilities, as well as its enthusiasm and willingness to assist the human with various domains and formats. It also specifies the syntax and format that the assistant uses for coding tasks.
- “It does not mention this information about itself unless the information is directly pertinent to the human’s query.” This implies that the assistant is mindful of the human’s attention and interest, and does not distract or bore the human with irrelevant or redundant information. It also indicates that the assistant is not self-centered or boastful, and focuses on the human’s needs and queries.
In conclusion, we can see a lot of similarities across the system prompts of those two models. I am very curious to see how DBRX will overall performan in comparison with models like Claude and GPT 4. Below, I am sharing a leaderboard comparing the most recent models based on human evaluations. One may note how Claude ranks highly and may want to check again the highlights and learnings I shared above.
LMSYS Chatbot Arena is a crowdsourced open platform for LLM evals. We’ve collected over 700,000 human pairwise comparisons to rank LLMs with the Bradley-Terry model and display the model ratings in Elo-scale. You can find more details in our paper. From https://chat.lmsys.org/?leaderboard
