Skip to content

Conversation

@Yuvraj3905
Copy link

Feature: Custom Self-Hosted Model Support

Summary

This PR adds support for integrating custom self-hosted models (e.g., Ollama, vLLM, or private endpoints) into the LLM Council. Users can now define custom API endpoints and keys for specific models, allowing them to participate in the council alongside standard OpenRouter models.

Changes

  • Backend Configuration (backend/config.py):
    • Introduced CUSTOM_MODELS dictionary to map model IDs to their specific api_url and api_key.
    • Added km/maxai as an example/default custom model in the configuration.
  • API Client (backend/openrouter.py):
    • Updated query_model function to check CUSTOM_MODELS before making requests.
    • Implemented dynamic routing: if a model is found in CUSTOM_MODELS, the request is sent to its configured URL with its specific headers; otherwise, it defaults to OpenRouter.
  • Documentation (README.md):
    • Added a new section "Custom Self-Hosted Models (Optional)" explaining how to configure and add local/private models to the council.

How to Test

  1. Configure a Custom Model:
    In backend/config.py, add a custom model definition:
    CUSTOM_MODELS = {
        "my-local-model": {
            "api_url": "http://localhost:11434/v1/chat/completions",
            "api_key": "custom"
        }
    }
  2. Add to Council:
    Add "my-local-model" to the COUNCIL_MODELS list in backend/config.py.
  3. Run Application:
    Start the backend and frontend.
  4. Verify:
    Send a query. The application should successfully send a request to http://localhost:11434/... for the custom model and display its response in the UI alongside other council members.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant