LiteLLM: An Open-Source AI Gateway — Gaurdrails

Photo by Jacob McGowin on Unsplash

Continued from Part 1 (Overview, APIM comparison, macOS setup, Admin UI, and the Azure AI Foundry gpt-5.4-mini connection). This part assumes the gateway is already running with gpt-5.4-mini registered as a model.

Guardrails

Guardrails let you inspect, mask, or block content flowing through the gateway — before it reaches the model (pre_call), after the model responds (post_call), or in parallel with the call (during_call). This walkthrough uses the LiteLLM Content Filter, a built-in guardrail that needs no external service or API key, plus a note on Azure Content Safety as a more production-grade alternative.

Prerequisite: guardrails apply to models already registered on the gateway. If you’re using the Admin UI + database setup, connect gpt-5.4-mini first (see "Connect the Azure AI Foundry model via the UI" below) — a fresh database-backed deployment starts with zero models, separate from anything configured in the original bare docker run setup.

Connect the Azure AI Foundry model via the UI

1. Go to http://localhost:4000/ui → Models + Endpoints → Add Model.
2. Fill in:

3. If local key auth is disabled on the resource (as covered earlier), the UI’s key field won’t work — use the azure_ad_token config snippet from the Foundry section above instead, mounted alongside the Compose stack. Models added via config and via the UI (with store_model_in_db: true) merge together.

Connect to Azure Foundry service

4. Test before moving on — Playground tab, or:

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Hello via Admin UI"}]
  }'

# Expected result
{"id":"chatcmpl-E6EenaOYUs1cIq8DwzNhfioY2lDaf","created":1785155017,"model":"azure_ai/gpt-5.4-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"Hello! How can I help you today?","role":"assistant","provider_specific_fields":{"refusal":null},"annotations":[]},"provider_specific_fields":{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"protected_material_code":{"detected":false,"filtered":false},"protected_material_text":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}}],"usage":{"completion_tokens":13,"prompt_tokens":10,"total_tokens":23,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"latency_checkpoint":{"engine_tbt_ms":5,"engine_ttft_ms":37,"engine_ttlt_ms":97,"pre_inference_ms":88,"service_tbt_ms":5,"service_ttft_ms":179,"service_ttlt_ms":231,"total_duration_ms":153,"user_visible_ttft_ms":90}},"service_tier":"default","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"jailbreak":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

Confirm a real response comes back before adding guardrails, so any later issue is clearly the guardrail, not a broken model connection.

Via the Admin UI (no restart required, since it’s stored in Postgres):

1. Guardrails → Add New Guardrail → Provider: LiteLLM Content Filter.
2. Name it basic-content-filter, mode pre_call.
3. Under Prebuilt Patterns, check Email → action Mask.
4. Add Custom Regex twice:

• azure_storage_key, regex [a-zA-Z0-9+/]{86}==, action Block
• azure_client_secret, regex [a-zA-Z0-9~._-]{34,40}, action Block

5. Under Blocked Words, add project_titan → action Block.

6. Save.

The .yaml route requires docker compose down && docker compose up -d to pick up changes; the UI route applies immediately.

Make sure Gaurdrails are set to ON by default. You may want to enable certain gaurdrails only if specified explicitly. LiteLLM provides you with those choices. Here’s an example of how we can do that

Get Krishnan Sriram’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

If we have the basic_content_filter default_on as No We may have to explicitly invoke it

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Please send the report to john.doe@example.com when ready"}]
  }'

# Expected result
{"id":"chatcmpl-E6Eo6U2sgD1zl7TtXsOgw3XPfuPjL","created":1785155594,"model":"azure_ai/gpt-5.4-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"I can’t send emails directly, but I can help you draft the message for John Doe or prepare the report text to paste into an email.\n\nIf you want, I can format it like this:\n\nSubject: Report Ready\n\nHi John,\n\nThe report is ready. Please see the attached file.\n\nBest,\n[Your Name]\n\nIf you’d like, I can also help you write the report itself or make the email more formal.","role":"assistant","provider_specific_fields":{"refusal":null},"annotations":[]},"provider_specific_fields":{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"protected_material_code":{"detected":false,"filtered":false},"protected_material_text":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}}],"usage":{"completion_tokens":90,"prompt_tokens":18,"total_tokens":108,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"latency_checkpoint":{"engine_tbt_ms":5,"engine_ttft_ms":39,"engine_ttlt_ms":508,"pre_inference_ms":93,"service_tbt_ms":5,"service_ttft_ms":190,"service_ttlt_ms":654,"total_duration_ms":576,"user_visible_ttft_ms":97}},"service_tier":"default","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"jailbreak":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

Let’s enable the same with content filter explicitly passed

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Please send the report to john.doe@example.com when ready"}],
    "guardrails": ["basic-content-filter"]
  }'

# Expected Result
{"id":"chatcmpl-E6EowjHrhDHPXErl18p7ap2VgbXzj","created":1785155646,"model":"azure_ai/gpt-5.4-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"I can’t send emails directly. If you’d like, I can help draft the report email now, and you can send it to **[EMAIL_REDACTED]** once it’s ready.","role":"assistant","provider_specific_fields":{"refusal":null},"annotations":[]},"provider_specific_fields":{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"protected_material_code":{"detected":false,"filtered":false},"protected_material_text":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}}],"usage":{"completion_tokens":45,"prompt_tokens":20,"total_tokens":65,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"latency_checkpoint":{"engine_tbt_ms":5,"engine_ttft_ms":32,"engine_ttlt_ms":279,"pre_inference_ms":93,"service_tbt_ms":6,"service_ttft_ms":182,"service_ttlt_ms":416,"total_duration_ms":331,"user_visible_ttft_ms":88}},"service_tier":"default","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"jailbreak":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

You’ll see email has been redacted — you can send it to **[EMAIL_REDACTED]** once it’s ready

For the rest of the testing we will havebasic_content_filter default_on as Yes

Testing the Guardrail

With default_on: true, no "guardrails": [...] array is needed in requests — it applies automatically.

Test 1 — clean request (should pass through):

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

# Expected result
{"id":"chatcmpl-E6EXKoG0ZJoIGOCvMXOeG5PppoSix","created":1785154554,"model":"azure_ai/gpt-5.4-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"The capital of France is **Paris**.","role":"assistant","provider_specific_fields":{"refusal":null},"annotations":[]},"provider_specific_fields":{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"protected_material_code":{"detected":false,"filtered":false},"protected_material_text":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}}],"usage":{"completion_tokens":13,"prompt_tokens":13,"total_tokens":26,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"latency_checkpoint":{"engine_tbt_ms":4,"engine_ttft_ms":34,"engine_ttlt_ms":84,"pre_inference_ms":110,"service_tbt_ms":4,"service_ttft_ms":203,"service_ttlt_ms":246,"total_duration_ms":149,"user_visible_ttft_ms":93}},"service_tier":"default","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"jailbreak":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

Expected: normal 200 OK.

Test 2 — email (should be masked, not blocked):

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Please send the report to john.doe@example.com when ready"}]
  }'
# Expected result
{"id":"chatcmpl-E6EcBbI0DjitQmRIYN03a5ZZsEcJe","created":1785154855,"model":"azure_ai/gpt-5.4-mini","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"I can help draft the report, but I can’t send emails directly from here.\n\nIf you’d like, I can:\n1. prepare the report content,\n2. format it as an email, and\n3. give you a ready-to-send message for [EMAIL_REDACTED].\n\nJust tell me what the report should include.","role":"assistant","provider_specific_fields":{"refusal":null},"annotations":[]},"provider_specific_fields":{"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"protected_material_code":{"detected":false,"filtered":false},"protected_material_text":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}}],"usage":{"completion_tokens":72,"prompt_tokens":20,"total_tokens":92,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0},"latency_checkpoint":{"engine_tbt_ms":6,"engine_ttft_ms":31,"engine_ttlt_ms":485,"pre_inference_ms":88,"service_tbt_ms":6,"service_ttft_ms":174,"service_ttlt_ms":612,"total_duration_ms":533,"user_visible_ttft_ms":85}},"service_tier":"default","prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"jailbreak":{"detected":false,"filtered":false},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}

Expected: 200 OK; the model's reply reflects [EMAIL] rather than the real address — the model never saw it.

Test 3 — Azure Storage key pattern (should be blocked):

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Here is my storage key: 8xJ3nQpL9vR2mK7wY4tZ6bC1dF5gH0aE3jM8nP2qS7tU9vW1xY4zB6cD8eF0gH2iJ4kL=="}]
  }'

# Expected Result
{"error":{"message":"Content blocked: azure_client_key pattern detected","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"Content blocked: azure_client_key pattern detected","pattern":"azure_client_key","guardrail_name":"basic-content-filter","guardrail_mode":"pre_call"}}}

Expected: 400 — {"error": {"message": "Content blocked: Policy violation", ...}}.

Test 4 — Azure client secret pattern (should be blocked):

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "The client secret is aB3~xY7.zK9_qR2mN5pL8wV1tS4dF6gH0jC~E"}]
  }'

# Expected Result
{"error":{"message":"Content blocked: azure_client_key pattern detected","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"Content blocked: azure_client_key pattern detected","pattern":"azure_client_key","guardrail_name":"basic-content-filter","guardrail_mode":"pre_call"}}

Expected: 400 blocked response.

Test 5 — blocked keyword (should be blocked):

curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{
    "model": "azure_ai/gpt-5.4-mini",
    "messages": [{"role": "user", "content": "Can you give me a status update on project_titan?"}]
  }'

# Expected Result
{"error":{"message":"Content blocked: keyword 'project_titan' detected","type":"None","param":"None","code":"400","provider_specific_fields":{"error":"Content blocked: keyword 'project_titan' detected","keyword":"project_titan","description":null,"guardrail_name":"basic-content-filter","guardrail_mode":"pre_call"}}}

Expected: 400 blocked response.

Verifying in the UI instead of curl: Guardrails → Test Playground → select basic-content-filter → paste each test message → see live pass/mask/block results, including a preview of the masked text the model actually receives.

Confirming logging: Usage → Logs (or the Guardrails tab) should show blocked/masked requests tagged with basic-content-filter, confirming the guardrail fired rather than the model itself refusing.

This takes us through the entire gaurdrail journey. Although we have only touched the basics, we can continue to expand on this work with MCP and agentic implementation soon.

AI Gaurdrails are ultimate enterprise luxury: spending millions to make your AI super-smart, and then spending millions more on three extra middleware checks to ensure it’s only allowed to answer like a second-grader.

Until then stay safe and keep reading!