Running Mixture-of-Agents on Groq with Streamlit on Localhost
The concept of a mixture of agents (MoA) is a groundbreaking approach that maximizes efficiency and precision. This innovative framework leverages the strengths of multiple specialized agents, working together to tackle complex tasks. Today, we’ll dive deep into how MoA integrates with Groq’s ultra-fast hardware, utilizing Streamlit for an interactive and user-friendly interface. I have this project on Github that I would love to take through, I am serving a Mixture of Agents on Groq. It is fast, accurate and damn the part where it compares responses from different agents and then gives the most accurate is just fascinating. The best part is that you run it locally on a local host. This could replace your dependence on ChatGPT.
Mixture-of-Agent
Mixture-of-Agents represents a paradigm shift in AI architecture. Unlike traditional single-agent models, MoA employs a team of specialized agents, each excelling in specific areas. This collaborative effort enhances flexibility, precision, and decision-making capabilities, making MoA ideal for applications requiring diverse skill sets.
Groq, renowned for its blazing-fast speeds, is engineered to handle the most demanding AI workloads. Its Tensor Streaming Processor (TSP) architecture is pivotal, enabling low-latency and high-throughput computations. This architecture is well-suited for MoA, allowing multiple agents to run in parallel without bottlenecks, ensuring swift and efficient data processing.
Key Benefits of Groq:
- Unmatched Speed: Groq’s TSP delivers industry-leading performance, significantly reducing the time needed for computations.
- Scalability: Its architecture easily scales to accommodate growing workloads, making it ideal for dynamic AI applications.
- Energy Efficiency: Groq offers a lower energy footprint, making it a sustainable choice for large-scale deployments.
Streamlit: The Perfect UI Choice
Streamlit is a Python-based web application framework designed to create interactive, real-time data apps. It stands out due to its simplicity and speed of development, allowing developers to build complex UIs with minimal code. For the Mixture-of-Agents project, Streamlit provides an intuitive interface for configuring agents, visualizing data, and interacting with the model outputs.
Why Streamlit?
- Ease of Use: Developers can focus on coding the logic without worrying about frontend intricacies.
- Real-Time Updates: Streamlit’s hot-reloading feature ensures that changes are reflected instantly, enhancing the development workflow.
- Interactive Widgets: With built-in widgets, Streamlit makes it easy to create forms, sliders, and other interactive elements, providing a seamless user experience.
Understanding the MoA Framework
In this setup, MoA employs multiple agents with distinct roles, coordinated through Groq’s efficient architecture. Here’s a look at the core components:
import streamlit as st
import json
from typing import Iterable
from moa.agent import MOAgent
from moa.agent.moa import ResponseChunk
from streamlit_ace import st_ace
import copy# Default configuration
default_config = {
"main_model": "llama3-70b-8192",
"cycles": 3,
"layer_agent_config": {}
}The code above initializes the MoA configuration, specifying the main model and agent layers. Each layer agent is responsible for a specific aspect of the decision-making process, from logical reasoning to expert planning.
Groq’s Role in Enhancing MoA
Groq’s hardware plays a crucial role in optimizing MoA’s performance. The TSP architecture accelerates the processing of tensor operations, which are fundamental to deep learning models. This results in lower latency and higher throughput, enabling real-time decision-making. Groq’s ability to handle parallel computations efficiently means that each agent can operate simultaneously without performance degradation.
Get Emmanuel Mark Ndaliro’s stories in your inbox
Join Medium for free to get updates from this writer.
Code implementation
layer_agent_config_def = {
"layer_agent_1": {
"system_prompt": "Think through your response step by step. {helper_response}",
"model_name": "llama3-8b-8192"
},
"layer_agent_2": {
"system_prompt": "Respond with a thought and then your response to the question. {helper_response}",
"model_name": "gemma-7b-it",
"temperature": 0.7
},
"layer_agent_3": {
"system_prompt": "You are an expert at logic and reasoning. Always take a logical approach to the answer. {helper_response}",
"model_name": "llama3-8b-8192"
},
}
# Recommended Configuration
rec_config = {
"main_model": "llama3-70b-8192",
"cycles": 2,
"layer_agent_config": {}
}
layer_agent_config_rec = {
"layer_agent_1": {
"system_prompt": "Think through your response step by step. {helper_response}",
"model_name": "llama3-8b-8192",
"temperature": 0.1
},
"layer_agent_2": {
"system_prompt": "Respond with a thought and then your response to the question. {helper_response}",
"model_name": "llama3-8b-8192",
"temperature": 0.2
},
"layer_agent_3": {
"system_prompt": "You are an expert at logic and reasoning. Always take a logical approach to the answer. {helper_response}",
"model_name": "llama3-8b-8192",
"temperature": 0.4
},
"layer_agent_4": {
"system_prompt": "You are an expert planner agent. Create a plan for how to answer the human's query. {helper_response}",
"model_name": "mixtral-8x7b-32768",
"temperature": 0.5
},
}
def stream_response(messages: Iterable[ResponseChunk]):
layer_outputs = {}
for message in messages:
if message['response_type'] == 'intermediate':
layer = message['metadata']['layer']
if layer not in layer_outputs:
layer_outputs[layer] = []
layer_outputs[layer].append(message['delta'])
else:
# Display accumulated layer outputs
for layer, outputs in layer_outputs.items():
st.write(f"Layer {layer}")
cols = st.columns(len(outputs))
for i, output in enumerate(outputs):
with cols[i]:
st.expander(label=f"Agent {i+1}", expanded=False).write(output)
# Clear layer outputs for the next iteration
layer_outputs = {}
# Yield the main agent's output
yield message['delta']
def set_moa_agent(
main_model: str = default_config['main_model'],
cycles: int = default_config['cycles'],
layer_agent_config: dict[dict[str, any]] = copy.deepcopy(layer_agent_config_def),
main_model_temperature: float = 0.1,
override: bool = False
):
if override or ("main_model" not in st.session_state):
st.session_state.main_model = main_model
else:
if "main_model" not in st.session_state: st.session_state.main_model = main_model
if override or ("cycles" not in st.session_state):
st.session_state.cycles = cycles
else:
if "cycles" not in st.session_state: st.session_state.cycles = cycles
if override or ("layer_agent_config" not in st.session_state):
st.session_state.layer_agent_config = layer_agent_config
else:
if "layer_agent_config" not in st.session_state: st.session_state.layer_agent_config = layer_agent_config
if override or ("main_temp" not in st.session_state):
st.session_state.main_temp = main_model_temperature
else:
if "main_temp" not in st.session_state: st.session_state.main_temp = main_model_temperature
cls_ly_conf = copy.deepcopy(st.session_state.layer_agent_config)
if override or ("moa_agent" not in st.session_state):
st.session_state.moa_agent = MOAgent.from_config(
main_model=st.session_state.main_model,
cycles=st.session_state.cycles,
layer_agent_config=cls_ly_conf,
temperature=st.session_state.main_temp
)
del cls_ly_conf
del layer_agent_config
st.set_page_config(
page_title="Karios Agents Powered by Groq",
page_icon='static/favicon.ico',
menu_items={
'About': "## Groq Mixture-Of-Agents \n Powered by [Groq](https://groq.com)"
},
layout="wide"
)
valid_model_names = [
'llama3-70b-8192',
'llama3-8b-8192',
'gemma-7b-it',
'gemma2-9b-it',
'mixtral-8x7b-32768'
]
st.markdown("<a href='https://groq.com'><img src='app/static/banner.png' width='500'></a>", unsafe_allow_html=True)
st.write("---")
# Initialize session state
if "messages" not in st.session_state:
st.session_state.messages = []
set_moa_agent()
# Sidebar for configuration
with st.sidebar:
# config_form = st.form("Agent Configuration", border=False)
st.title("MOA Configuration")
with st.form("Agent Configuration", border=False):
if st.form_submit_button("Use Recommended Config"):
try:
set_moa_agent(
main_model=rec_config['main_model'],
cycles=rec_config['cycles'],
layer_agent_config=layer_agent_config_rec,
override=True
)
st.session_state.messages = []
st.success("Configuration updated successfully!")
except json.JSONDecodeError:
st.error("Invalid JSON in Layer Agent Configuration. Please check your input.")
except Exception as e:
st.error(f"Error updating configuration: {str(e)}")
# Main model selection
new_main_model = st.selectbox(
"Select Main Model",
options=valid_model_names,
index=valid_model_names.index(st.session_state.main_model)
)
# Cycles input
new_cycles = st.number_input(
"Number of Layers",
min_value=1,
max_value=10,
value=st.session_state.cycles
)
# Main Model Temperature
main_temperature = st.number_input(
label="Main Model Temperature",
value=0.1,
min_value=0.0,
max_value=1.0,
step=0.1
)
# Layer agent configuration
tooltip = "Agents in the layer agent configuration run in parallel _per cycle_. Each layer agent supports all initialization parameters of [Langchain's ChatGroq](https://api.python.langchain.com/en/latest/chat_models/langchain_groq.chat_models.ChatGroq.html) class as valid dictionary fields."
st.markdown("Layer Agent Config", help=tooltip)
new_layer_agent_config = st_ace(
value=json.dumps(st.session_state.layer_agent_config, indent=2),
language='json',
placeholder="Layer Agent Configuration (JSON)",
show_gutter=False,
wrap=True,
auto_update=True
)
if st.form_submit_button("Update Configuration"):
try:
new_layer_config = json.loads(new_layer_agent_config)
set_moa_agent(
main_model=new_main_model,
cycles=new_cycles,
layer_agent_config=new_layer_config,
main_model_temperature=main_temperature,
override=True
)
st.session_state.messages = []
st.success("Configuration updated successfully!")
except json.JSONDecodeError:
st.error("Invalid JSON in Layer Agent Configuration. Please check your input.")
except Exception as e:
st.error(f"Error updating configuration: {str(e)}")
st.markdown("---")
st.markdown("""
### Credits
- MOA: [Together AI](https://www.together.ai/blog/together-moa)
- LLMs: [Groq](https://groq.com/)
- Paper: [arXiv:2406.04692](https://arxiv.org/abs/2406.04692)
""")
# Main app layout
st.header("Karios Agents", anchor=False)
st.write("This project oversees implementation of Mixture of Agents architecture Powered by Groq LLMs.")
# st.image("./static/moa_groq.svg", caption="Mixture of Agents Workflow", width=1000)
# Display current configuration
with st.expander("Current MOA Configuration", expanded=False):
st.markdown(f"**Main Model**: ``{st.session_state.main_model}``")
st.markdown(f"**Main Model Temperature**: ``{st.session_state.main_temp:.1f}``")
st.markdown(f"**Layers**: ``{st.session_state.cycles}``")
st.markdown(f"**Layer Agents Config**:")
new_layer_agent_config = st_ace(
value=json.dumps(st.session_state.layer_agent_config, indent=2),
language='json',
placeholder="Layer Agent Configuration (JSON)",
show_gutter=False,
wrap=True,
readonly=True,
auto_update=True
)Practical Applications and Use Cases
The combination of MoA and Groq hardware is transformative across various industries:
- Autonomous Systems: The low-latency processing capabilities of Groq are crucial for real-time decision-making in autonomous vehicles and robotics.
- Financial Modeling: MoA can perform complex risk assessments and predictive analytics, providing insights for investment strategies.
- Personalized Recommendations: The system’s ability to analyze vast amounts of data quickly makes it ideal for crafting personalized user experiences in e-commerce and media.
Challenges and Considerations
Implementing MoA on Groq hardware, while highly efficient, presents challenges such as the need for specialized knowledge in configuring and maintaining the system. Scalability and integration with existing infrastructures are also key considerations, particularly for large enterprises.
The future of MoA on specialized hardware like Groq is promising. As AI continues to evolve, we can expect advancements in multi-agent coordination, adaptive learning, and more. Groq’s continued innovation in hardware solutions will likely push the boundaries of what’s possible, offering new opportunities for AI applications.
Conclusion
The Mixture-of-Agents framework, coupled with Groq’s cutting-edge hardware and the intuitive Streamlit UI, represents a significant advancement in AI technology. This setup not only enhances computational efficiency but also provides a flexible and interactive platform for developing sophisticated AI solutions.
For a deeper dive into this innovative project, check out the GitHub repository
