<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saeedafzal2030 on Medium]]></title>
        <description><![CDATA[Stories by Saeedafzal2030 on Medium]]></description>
        <link>https://medium.com/@saeedafzal2030?source=rss-ac59c13fc279------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3Wj0-tIFDuigWwQi</url>
            <title>Stories by Saeedafzal2030 on Medium</title>
            <link>https://medium.com/@saeedafzal2030?source=rss-ac59c13fc279------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:38:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@saeedafzal2030/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Production LiteLLM on AWS EKS: High Availability with GitOps]]></title>
            <link>https://pub.towardsai.net/production-litellm-on-aws-eks-high-availability-with-gitops-996eda87e14d?source=rss-ac59c13fc279------2</link>
            <guid isPermaLink="false">https://medium.com/p/996eda87e14d</guid>
            <category><![CDATA[litellm]]></category>
            <category><![CDATA[llm-proxy]]></category>
            <category><![CDATA[gitops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[aws-eks]]></category>
            <dc:creator><![CDATA[Saeedafzal2030]]></dc:creator>
            <pubDate>Tue, 16 Jun 2026 13:01:02 GMT</pubDate>
            <atom:updated>2026-06-16T13:01:02.599Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Eliminating multi-provider chaos with a unified LLM gateway: automatic scaling, budget controls, and 99.9% uptime</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oHXK8vBd92NMFAXElOrPyQ.png" /></figure><h4>Introduction</h4><p>In today’s multi-cloud AI landscape, organizations often find themselves juggling multiple LLM providers — OpenAI, Anthropic Claude via Azure AI Foundry, Cohere for embeddings, and more. Each provider has its own API, authentication mechanism, rate limits, and pricing model. Managing access for multiple users while enforcing budget controls and usage restrictions quickly becomes a nightmare.</p><p>Enter <strong>LiteLLM</strong>: a unified proxy that acts as a single gateway to 100+ LLM providers. But deploying LiteLLM in production isn’t just about running a container — it requires careful consideration of high availability, scalability, persistence, observability, and cost optimization.</p><p>In this article, I’ll walk you through our production-grade deployment of LiteLLM on AWS EKS, managed via ArgoCD for GitOps. I’ll explain every architectural decision, configuration parameter, and best practice we followed to build a system that serves 500–1000 requests per second with automatic scaling, persistent logging, and enterprise-grade reliability.</p><h4>The Problem We Were Solving</h4><p>Our data science team Lab Enablement (LENA) group faced several challenges:</p><p><strong>1. Multi-Provider Chaos</strong>: We used Azure AI Foundry for Claude models and Cohere v3 for embeddings. Each required separate authentication, SDKs, and management.</p><p><strong>2. No Central Budget Control</strong>: Azure AI Foundry doesn’t provide granular budget restrictions. Creating separate Azure resources for each user was operationally expensive and unscalable.</p><p><strong>3. User Management Nightmare</strong>: Provisioning API keys, tracking usage, and revoking access required manual intervention.</p><p><strong>4. Cost Visibility Gap</strong>: No unified view of spending across providers made it impossible to optimize costs or forecast budgets.</p><p><strong>LiteLLM solved all of these problems </strong>by providing:</p><ul><li>A <strong>single API endpoint</strong> compatible with OpenAI’s SDK</li><li>Built-in <strong>budget tracking</strong> and <strong>spend limits</strong> per user/team</li><li>Request <strong>caching</strong> to reduce costs</li><li><strong>Load balancing</strong> across multiple model deployments</li><li>Detailed <strong>logging</strong> and usage analytics</li></ul><h4>Architecture Overview</h4><p>Our deployment consists of four core components orchestrated on AWS EKS:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOrdWaf3SMnu7cu2kY05Lw.png" /><figcaption>architectural diagram</figcaption></figure><h4><strong>Why This Architecture?</strong></h4><p><strong>Kubernetes-Native</strong>: Leverages EKS for orchestration, scaling, and self-healing</p><p><strong>GitOps-Driven</strong>: ArgoCD ensures declarative state, audit trails, and rollback capability</p><p><strong>Highly Available</strong>: Pod anti-affinity, HPA, and multiple replicas prevent single points of failure</p><p><strong>Cost-Optimized</strong>: Redis caching and batch database writes reduce infrastructure load</p><p><strong>Secure</strong>: Internal ALB, VPC-only access, TLS termination, no public exposure</p><h4>Component 1: The LiteLLM Proxy Deployment</h4><p>Why `ghcr.io/berriai/litellm-database:v1.86.2`?</p><p>We chose the <strong>database-enabled image</strong> because:</p><ol><li><strong>Prisma ORM Built-In</strong>: Simplifies database migrations and schema management</li><li><strong>Latest Features Over Stability</strong>: v1.86.2 included critical fixes for Azure AI Foundry integration and improved Redis caching</li><li><strong>Active Development</strong>: LiteLLM’s rapid iteration cycle means newer versions often resolve edge cases quickly</li></ol><p>Deployment Configuration</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: litellm<br>  namespace: litellm-ns<br>  labels:<br>    app: litellm<br>spec:<br>  replicas: 1  # HPA will scale this dynamically<br>  revisionHistoryLimit: 10<br>  selector:<br>    matchLabels:<br>      app: litellm<br>  strategy:<br>    type: RollingUpdate<br>    rollingUpdate:<br>      maxSurge: 1<br>      maxUnavailable: 0  # Zero-downtime deployments<br>  template:<br>    metadata:<br>      labels:<br>        app: litellm<br>    spec:<br>      affinity:<br>        podAntiAffinity:<br>          preferredDuringSchedulingIgnoredDuringExecution:<br>            - weight: 100<br>              podAffinityTerm:<br>                labelSelector:<br>                  matchLabels:<br>                    app: litellm<br>                topologyKey: kubernetes.io/hostname<br>      containers:<br>        - name: litellm<br>          image: ghcr.io/berriai/litellm-database:v1.86.2<br>          imagePullPolicy: IfNotPresent<br>          args:<br>            - &quot;--config&quot;<br>            - &quot;/etc/litellm/config.yaml&quot;<br>            - &quot;--port&quot;<br>            - &quot;4000&quot;<br>            - &quot;--num_workers&quot;<br>            - &quot;1&quot;  # Single worker per pod for predictable resource usage - recomended<br>          ports:<br>            - name: http<br>              containerPort: 4000<br>          env:<br>            - name: POSTGRES_USER<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: litellm-pg-config<br>                  key: POSTGRES_USER<br>            - name: POSTGRES_DB<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: litellm-pg-config<br>                  key: POSTGRES_DB<br>            - name: POSTGRES_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-pg-secret<br>                  key: POSTGRES_PASSWORD<br>            - name: DATABASE_URL<br>              value: &quot;postgresql://$(POSTGRES_USER):$(POSTGRES_PASSWORD)@litellm-pg-svc.litellm-ns.svc.cluster.local:5432/$(POSTGRES_DB)&quot;<br>            - name: MASTER_KEY<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-secrets<br>                  key: MASTER_KEY<br>            - name: REDIS_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-redis-secret<br>                  key: REDIS_PASSWORD<br>            - name: LITELLM_MODE<br>              value: &quot;PRODUCTION&quot;<br>            - name: LITELLM_LOG<br>              value: &quot;ERROR&quot;  # Log only errors to reduce noise<br>            - name: USE_PRISMA_MIGRATE<br>              value: &quot;True&quot;  # Auto-apply database migrations on startup<br>            - name: DOCS_URL<br>              value: &quot;/docs&quot;<br>            - name: ROOT_REDIRECT_URL<br>              value: &quot;/ui&quot;  # Redirect root to admin UI<br>          volumeMounts:<br>            - name: config<br>              mountPath: /etc/litellm<br>              readOnly: true<br>          resources:<br>            requests:<br>              cpu: &quot;1500m&quot;<br>              memory: &quot;3Gi&quot;<br>            limits:<br>              cpu: &quot;1500m&quot;<br>              memory: &quot;3Gi&quot;<br>          livenessProbe:<br>            httpGet:<br>              path: /health/liveliness<br>              port: 4000<br>            initialDelaySeconds: 60  # Allow time for Prisma migration<br>            periodSeconds: 30<br>            failureThreshold: 5  # Tolerates 2.5 minutes of failures<br>          readinessProbe:<br>            httpGet:<br>              path: /health/readiness<br>              port: 4000<br>            initialDelaySeconds: 30<br>            periodSeconds: 10<br>            failureThreshold: 5<br>      volumes:<br>        - name: config<br>          configMap:<br>            name: litellm-config<br>            items:<br>              - key: config.yaml<br>                path: config.yaml</pre><p>Key Design Decisions</p><ol><li><strong>Pod Anti-Affinity: Spreading Pods Across Nodes</strong></li></ol><pre>affinity:<br>  podAntiAffinity:<br>    preferredDuringSchedulingIgnoredDuringExecution:<br>      - weight: 100<br>        podAffinityTerm:<br>          labelSelector:<br>            matchLabels:<br>              app: litellm<br>          topologyKey: kubernetes.io/hostname</pre><p><strong>Why?</strong> If one EKS node fails, we want LiteLLM pods on other nodes to keep serving traffic. The `preferred` (soft) constraint means Kubernetes will <em>try</em> to spread pods but won’t block scheduling if impossible.</p><p><strong>Alternative Considered</strong>: ‘requiredDuringSchedulingIgnoredDuringExecution` (hard constraint) was too strict for our small cluster — it would prevent scaling if only one node had capacity.</p><p><strong>2. Resource Limits = Requests (No Burstability)</strong></p><pre>resources:<br>  requests:<br>    cpu: &quot;1500m&quot;<br>    memory: &quot;3Gi&quot;<br>  limits:<br>    cpu: &quot;1500m&quot;<br>    memory: &quot;3Gi&quot;</pre><p><strong>Why?</strong></p><ul><li><strong>HPA works on requests, not limits</strong>: The Horizontal Pod Autoscaler calculates CPU/memory utilization as <strong>`(current usage / requests) * 100`</strong>. Setting requests = limits ensures predictable scaling behavior.</li><li><strong>CPU Throttling Prevention</strong>: If limits &gt; requests, pods can burst but risk throttling when all pods burst simultaneously.</li><li><strong>Memory OOM Prevention</strong>: Memory limits prevent pods from being OOM-killed during traffic spikes.</li><li><strong>Performance Validated</strong>: With 1.5 CPU cores and 3Gi memory, a single pod handles <strong>50–60 requests/second</strong> comfortably (tested with Claude Haiku/Sonnet/Opus 4.5 and Cohere Embed v3).</li></ul><p>3. <strong>Health Check Tuning</strong></p><pre>livenessProbe:<br>  httpGet:<br>    path: /health/liveliness<br>    port: 4000<br>  initialDelaySeconds: 60  # Critical: Allow Prisma migration to complete<br>  periodSeconds: 30<br>  failureThreshold: 5<br><br>readinessProbe:<br>  httpGet:<br>    path: /health/readiness<br>    port: 4000<br>  initialDelaySeconds: 30<br>  periodSeconds: 10<br>  failureThreshold: 5</pre><p><strong>Why 60s initial delay for liveness?</strong></p><ul><li>LiteLLM runs Prisma migrations on startup (`USE_PRISMA_MIGRATE=True`)</li><li>Migrations can take 20–40 seconds on PostgreSQL 17 with 100Gi EBS</li><li>Without sufficient delay, Kubernetes kills the pod mid-migration, causing a restart loop</li></ul><p><strong>Why 5 failure thresholds?</strong></p><ul><li>Liveness: Tolerates up to 2.5 minutes of unresponsiveness before restart (5 failures × 30s period)</li><li>Readiness: Removes pod from load balancer after 50 seconds (5 failures × 10s period)</li></ul><blockquote><strong>Lesson Learned</strong>: Initially, we set “initialDelaySeconds: 30”, which caused 40% of deployments to fail during migrations. Increasing to 60s achieved 100% success rate.</blockquote><h4>Component 2: Horizontal Pod Autoscaler (HPA)</h4><pre>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: litellm-hpa<br>  namespace: litellm-ns<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: litellm<br>  minReplicas: 1<br>  maxReplicas: 5<br>  metrics:<br>    - type: Resource<br>      resource:<br>        name: cpu<br>        target:<br>          type: Utilization<br>          averageUtilization: 60<br>    - type: Resource<br>      resource:<br>        name: memory<br>        target:<br>          type: Utilization<br>          averageUtilization: 80<br>  behavior:<br>    scaleUp:<br>      stabilizationWindowSeconds: 120<br>      policies:<br>        - type: Pods<br>          value: 1<br>          periodSeconds: 120<br>      selectPolicy: Max<br>    scaleDown:<br>      stabilizationWindowSeconds: 120<br>      policies:<br>        - type: Pods<br>          value: 1<br>          periodSeconds: 120<br>      selectPolicy: Max</pre><p>Why These Thresholds?</p><ol><li><strong>CPU: 60% Utilization<br></strong>LiteLLM is <strong>CPU-intensive</strong> due to:</li></ol><ul><li>JSON serialization / deserialization for requests/responses</li><li>Request routing logic across multiple model deployments</li><li>Redis cache key computation</li></ul><p>At 60% CPU, we observed response latencies starting to creep above 200ms. Scaling at 60% keeps p95 latency under 150ms.</p><p><strong>2. Memory: 80% Utilization<br> </strong>Memory usage is more predictable:</p><ul><li>~1.2 Gi baseline for LiteLLM process</li><li>~500 Mi for in-flight requests</li><li>~1.3 Gi buffer to prevent OOM</li></ul><p>Scaling at 80% (2.4 Gi) provides comfortable headroom.</p><p><strong>3. Stabilization Windows: 120 Seconds</strong></p><p><strong>Scale-Up</strong>: Prevents “flapping” during brief traffic spikes (e.g., batch inference jobs). HPA waits 2 minutes before adding a pod, filtering out transient load.</p><p><strong>Scale-Down</strong>: Avoids premature pod removal. If traffic drops temporarily, HPA waits 2 minutes before scaling down, preventing a scale-up/down cycle.</p><p><strong>Real-World Impact</strong>: During a traffic spike from 20 → 80 rps:</p><ul><li>Without stabilization: HPA added 4 pods in 30 seconds, then removed 3 when spike ended after 90 seconds</li><li>With 120s stabilization: HPA added 1 pod, observed for 2 minutes, added 1 more. Traffic normalized, no thrashing.</li></ul><h4>Component 3: LiteLLM Configuration (ConfigMap)</h4><pre>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: litellm-config<br>  namespace: litellm-ns<br>data:<br>  config.yaml: |<br>    general_settings:<br>      master_key: os.environ/MASTER_KEY<br>      database_url: os.environ/DATABASE_URL<br>      store_model_in_db: true<br>      use_redis_transaction_buffer: true<br>      block_robots: true  # Prevent search engine indexing<br>      background_health_checks: true<br>      health_check_interval: 300  # Check model endpoints every 5 minutes<br>      maximum_spend_logs_retention_period: &quot;60d&quot;<br>      maximum_spend_logs_retention_interval: &quot;1d&quot;<br>      maximum_spend_logs_cleanup_cron: &quot;0 4 * * *&quot;  # Daily cleanup at 4 AM<br>      user_api_key_cache_ttl: 600  # Cache API key lookups for 10 minutes<br>      database_connection_pool_limit: 20<br>      database_connection_timeout: 60<br>      proxy_batch_write_at: 30  # Batch database writes every 30 seconds<br><br>    litellm_settings:<br>      json_logs: true  # Structured logging for Grafana/CloudWatch<br>      cache: true<br>      set_verbose: false  # Disable verbose logs (reduces noise)<br>      request_timeout: 600  # 10 minutes for long-running model requests<br>      cache_params:<br>        type: redis<br>        host: redis-service<br>        port: 6379<br>        password: os.environ/REDIS_PASSWORD<br>        ttl: 600  # Cache responses for 10 minutes<br>        namespace: &quot;litellm.caching&quot;<br><br>    router_settings:<br>      routing_strategy: simple-shuffle<br>      redis_host: redis-service<br>      redis_password: os.environ/REDIS_PASSWORD<br>      redis_port: 6379<br>      cooldown_time: 10  # Retry failed models after 10 seconds</pre><h4>Configuration Deep Dive</h4><p>1. <strong>Database Connection Pooling: 20 Connections</strong></p><pre>database_connection_pool_limit: 20</pre><p>PostgreSQL’s default `max_connections` is 100. With 5 HPA replicas max: `5 pods × 20 connections = 100 total`. This prevents connection exhaustion while avoiding the memory overhead of increasing PostgreSQL’s connection limit.</p><p>2. <strong>Batch Writing: 30-Second Intervals</strong></p><pre>proxy_batch_write_at: 30</pre><p>Groups 30 seconds of request logs into a single transaction, reducing PostgreSQL CPU by 60% (from 3,000 transactions/min to 2 transactions/min).</p><blockquote>Trade-off: up to 30 seconds of logs lost if a pod crashes, which is acceptable for analytics workloads.</blockquote><p>3. <strong>Caching Strategy: 10-Minute TTL</strong></p><pre>user_api_key_cache_ttl: 600 # API key validation<br>cache_params.ttl: 600 # Response caching</pre><p><strong>API Key</strong>: Reduces database load from 50 QPS (cold cache) to ~0.008 QPS (warm cache) for 5 unique users.</p><p>4. <strong>Spend Log Retention: 60 Days</strong></p><pre>maximum_spend_logs_retention_period: &quot;60d&quot;<br>maximum_spend_logs_cleanup_cron: &quot;0 4 * * *&quot;</pre><p>Retains 2 months of spend data for financial auditing. Cleanup runs at 4 AM (lowest traffic period) to allow PostgreSQL `VACUUM` to reclaim disk space. At 50 rps, 60 days of logs consume ~129 GB (80% of our 100 Gi EBS volume).</p><p>5. <strong>Routing Strategy: Simple Shuffle</strong></p><pre>routing_strategy: simple-shuffle</pre><p>Randomly distributes requests across model deployments. We chose this over `least-latency` (tracks metrics in Redis) and `weighted` (traffic percentages) because we have a single Azure AI Foundry deployment — simpler is better for debugging. Switch to `least-latency` when adding multi-region redundancy.</p><h4>Component 4: PostgreSQL StatefulSet</h4><pre>apiVersion: apps/v1<br>kind: StatefulSet<br>metadata:<br>  name: litellm-pg-statefulset<br>  namespace: litellm-ns<br>spec:<br>  serviceName: litellm-pg-headless-svc<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: litellm-pg<br>  template:<br>    metadata:<br>      labels:<br>        app: litellm-pg<br>    spec:<br>      containers:<br>        - name: postgres<br>          image: postgres:17<br>          imagePullPolicy: IfNotPresent<br>          ports:<br>            - name: postgres<br>              containerPort: 5432<br>          volumeMounts:<br>            - name: litellm-pg-data<br>              mountPath: /var/lib/postgresql/data<br>          env:<br>            - name: POSTGRES_USER<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: litellm-pg-config<br>                  key: POSTGRES_USER<br>            - name: POSTGRES_DB<br>              valueFrom:<br>                configMapKeyRef:<br>                  name: litellm-pg-config<br>                  key: POSTGRES_DB<br>            - name: POSTGRES_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-pg-secret<br>                  key: POSTGRES_PASSWORD<br>            - name: PGDATA<br>              value: /var/lib/postgresql/data/pgdata<br>          resources:<br>            requests:<br>              cpu: &quot;250m&quot;<br>              memory: &quot;512Mi&quot;<br>            limits:<br>              cpu: &quot;2000m&quot;<br>              memory: &quot;8Gi&quot;<br>          readinessProbe:<br>            exec:<br>              command:<br>                - /bin/sh<br>                - -c<br>                - pg_isready -U &quot;$POSTGRES_USER&quot; -d &quot;$POSTGRES_DB&quot;<br>            initialDelaySeconds: 10<br>            periodSeconds: 5<br>          livenessProbe:<br>            exec:<br>              command:<br>                - /bin/sh<br>                - -c<br>                - pg_isready -U &quot;$POSTGRES_USER&quot; -d &quot;$POSTGRES_DB&quot;<br>            initialDelaySeconds: 30<br>            periodSeconds: 10<br>  volumeClaimTemplates:<br>    - metadata:<br>        name: litellm-pg-data<br>      spec:<br>        accessModes:<br>          - ReadWriteOnce<br>        storageClassName: ebs-sc<br>        resources:<br>          requests:<br>            storage: 100Gi</pre><h4>Why PostgreSQL 17?</h4><p>PostgreSQL 17 delivers <strong>20% faster bulk INSERT’s</strong> (critical for batch writes), better JSONB indexing for metadata storage, and incremental VACUUM that reduces write amplification by 30%.</p><p><strong>RDS vs Self-Managed</strong>: RDS db.t3.medium costs $50/month vs. $12/month self-managed on EKS. We needed custom `max_connections` tuning and avoided VPC peering complexity. Trade-off: manual backups (pg_dump to S3 daily) instead of RDS automated snapshots.</p><h4>Why StatefulSet?</h4><p>StatefulSets provide stable DNS names, ordered deployment (critical for single-replica databases), and persistent volume binding that survives pod restarts. We use a <strong>headless service</strong> (ClusterIP: None) for direct pod connections, eliminating load balancer overhead.</p><p><em>Resource Allocation: Wide Gap Between Requests and Limits</em></p><pre>resources:<br>  requests:<br>    cpu: &quot;250m&quot;<br>    memory: &quot;512Mi&quot;<br>  limits:<br>    cpu: &quot;2000m&quot;<br>    memory: &quot;8Gi&quot;</pre><p>PostgreSQL is <strong>idle 90% of the time</strong> due to <strong>30-second batch writes</strong>, justifying low requests (250m CPU fits on smaller nodes). High limits accommodate burst capacity during batch writes (1.5 CPU, 3 Gi) and daily VACUUM operations (up to 6 Gi memory). Measured performance: P50 CPU at 150m, P95 at 1,200m during writes.</p><p>Setting requests = limits would waste EKS node capacity during idle periods.</p><h4>Component 5: Redis for Caching</h4><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: litellm-redis<br>  namespace: litellm-ns<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: litellm-redis<br>  template:<br>    metadata:<br>      labels:<br>        app: litellm-redis<br>    spec:<br>      containers:<br>        - name: redis<br>          image: redis:8-alpine<br>          imagePullPolicy: IfNotPresent<br>          command: [&quot;sh&quot;, &quot;-c&quot;]<br>          args:<br>            - &#39;exec redis-server --requirepass &quot;$REDIS_PASSWORD&quot; --appendonly yes&#39;<br>          ports:<br>            - name: redis<br>              containerPort: 6379<br>          env:<br>            - name: REDIS_PASSWORD<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-redis-secret<br>                  key: REDIS_PASSWORD<br>            - name: REDISCLI_AUTH<br>              valueFrom:<br>                secretKeyRef:<br>                  name: litellm-redis-secret<br>                  key: REDIS_PASSWORD<br>          volumeMounts:<br>            - name: litellm-redis-pvc<br>              mountPath: /data<br>          resources:<br>            requests:<br>              cpu: &quot;200m&quot;<br>              memory: &quot;512Mi&quot;<br>            limits:<br>              cpu: &quot;1000m&quot;<br>              memory: &quot;2Gi&quot;<br>          readinessProbe:<br>            exec:<br>              command: [&quot;redis-cli&quot;, &quot;ping&quot;]<br>            initialDelaySeconds: 5<br>            periodSeconds: 10<br>            timeoutSeconds: 3<br>          livenessProbe:<br>            exec:<br>              command: [&quot;redis-cli&quot;, &quot;ping&quot;]<br>            initialDelaySeconds: 15<br>            periodSeconds: 20<br>            timeoutSeconds: 3<br>      volumes:<br>        - name: litellm-redis-pvc<br>          persistentVolumeClaim:<br>            claimName: litellm-redis-pvc</pre><h4>Why LiteLLM Needs Redis Caching</h4><p>LiteLLM makes Redis essential for three critical functions:</p><p><strong>1. Response Caching</strong></p><p>Identical prompts (same model, same input) return cached responses instead of hitting the upstream LLM provider. This is crucial for:</p><ul><li><strong>Batch workloads</strong>: Data science teams run the same prompt across thousands of documents</li><li><strong>Repetitive queries</strong>: Users retry failed requests or re-run notebooks</li><li><strong>Development/testing</strong>: Same test prompts executed repeatedly</li></ul><p>Our 10-minute TTL achieved a <strong>40% cache hit rate.</strong></p><p><strong>2. API Key Validation Caching</strong></p><p>Every LiteLLM request validates the user’s API key against PostgreSQL. Without caching, this means 50 database queries per second. Redis caches API key lookups for 10 minutes, reducing database load by <strong>99.98%</strong> (from 50 QPS to ~0.01 QPS).</p><p><strong>3. Router State Management**</strong></p><p>LiteLLM tracks model endpoint health (which deployments are responding, which are failing) in Redis. This enables:</p><ul><li><strong>Fast failover</strong>: Dead endpoints marked instantly, no database writes</li><li><strong>Load balancing</strong>: Distribute requests across healthy endpoints</li><li><strong>Cooldown tracking</strong>: Retry failed models after configurable intervals</li></ul><h4>Component 6: Ingress with AWS ALB</h4><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: litellm-ingress<br>  namespace: litellm-ns<br>  annotations:<br>    alb.ingress.kubernetes.io/scheme: internal<br>    alb.ingress.kubernetes.io/target-type: ip<br>    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-west-2:01234567890:certificate/a1639c-6c2c-4e60-9211-59cd0a<br>    alb.ingress.kubernetes.io/listen-ports: &#39;[{&quot;HTTP&quot;:80}, {&quot;HTTPS&quot;:443}]&#39;<br>    alb.ingress.kubernetes.io/ssl-redirect: &quot;443&quot;<br>    alb.ingress.kubernetes.io/backend-protocol: HTTP<br>    alb.ingress.kubernetes.io/healthcheck-path: /health/readiness<br>    alb.ingress.kubernetes.io/healthcheck-interval-seconds: &quot;30&quot;<br>    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: &quot;10&quot;<br>    alb.ingress.kubernetes.io/healthy-threshold-count: &quot;2&quot;<br>    alb.ingress.kubernetes.io/unhealthy-threshold-count: &quot;3&quot;<br>spec:<br>  ingressClassName: alb<br>  rules:<br>    - host: litellm.data-lab.io<br>      http:<br>        paths:<br>          - path: /<br>            pathType: Prefix<br>            backend:<br>              service:<br>                name: litellm-svc<br>                port:<br>                  number: 4000</pre><p>Creates a VPC-internal ALB (no public IP). LiteLLM contains sensitive API keys and exposes admin UI — public exposure would require WAF, rate limiting, and OAuth2, which wasn’t cost-justified for internal usage.</p><p><strong>Target Type: IP</strong></p><pre>alb.ingress.kubernetes.io/target-type: ip</pre><p>Routes traffic directly to pod IPs, eliminating kube-proxy overhead (~5ms latency reduction) and enabling faster health checks. IP targets are free; instance targets charge per registration.</p><p><strong>Health Check Configuration</strong></p><pre>alb.ingress.kubernetes.io/healthcheck-path: /health/readiness<br>alb.ingress.kubernetes.io/healthcheck-interval-seconds: &quot;30&quot;</pre><p>Uses `/health/readiness` (pod can serve traffic) vs. `/health/liveliness` (pod process running). Pods rejoin ALB after 60 seconds (2 successes × 30s) and are removed after 90 seconds (3 failures × 30s).</p><blockquote><strong>Lesson Learned</strong>: 10-second intervals caused false negatives during brief database blips. 30-second intervals tolerate transient issues.</blockquote><h4>GitOps with ArgoCD</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*beOXWrml96XlVDr4tn0BAg.png" /></figure><pre>apiVersion: argoproj.io/v1alpha1<br>kind: Application<br>metadata:<br>  name: litellm<br>  namespace: argocd<br>spec:<br>  project: litellm<br>  source:<br>    repoURL: https://gitlab.com/devops1952445/argocd-playground.git<br>    targetRevision: main  # Production branch<br>    path: src/apps/litellm<br>  destination:<br>    server: https://kubernetes.default.svc<br>    namespace: litellm-ns<br>  syncPolicy:<br>    automated:<br>      prune: true<br>      selfHeal: true<br>    syncOptions:<br>      - CreateNamespace=true</pre><p>Why ArgoCD for LiteLLM?</p><p>1. <strong>Declarative State</strong>: Git is source of truth, not `kubectl apply` commands</p><p>2. <strong>Audit Trail</strong>: Every change tracked via Git commits (who, what, when, why)</p><p>3. <strong>Rollback</strong>: `git revert` instantly rolls back deployments</p><p>4. <strong>Multi-Environment</strong>: Same manifests for dev/staging/prod with Kustomize overlays</p><p>5. <strong>Self-Healing</strong>: If someone `kubectl delete` a pod, ArgoCD recreates it within 3 minutes</p><p>Auto-Sync with Prune and Self-Heal</p><pre>syncPolicy:<br>automated:<br>prune: true # Delete resources removed from Git<br>selfHeal: true # Revert manual kubectl changes</pre><p><strong>Prune</strong>: If we delete `litellm_redis_deployment.yml` from Git, ArgoCD deletes the Redis deployment from the cluster.</p><p><strong>Self-Heal</strong>: If an engineer runs `kubectl scale deployment/litellm — replicas=10`, ArgoCD reverts to `replicas: 1` within 3 minutes.</p><p><strong>Is This Too Aggressive?</strong></p><p><strong>Concern</strong>: Self-heal might revert emergency hotfixes during incidents.</p><p><strong>Mitigation</strong>:</p><p>1. <strong>Emergency Override</strong>: Pause ArgoCD sync during incidents: `kubectl patch app litellm -n argocd -p ‘{“spec”:{“syncPolicy”:{“automated”:null}}}’`</p><p>2. <strong>Sync Wave</strong>: Critical resources (database, Redis) have `argocd.argoproj.io/sync-wave: “0”`, LiteLLM pods have wave `”1&quot;`. Database won’t self-heal during LiteLLM rollouts.</p><p>3. <strong>Notification</strong>: ArgoCD webhooks to Slack alert on any manual changes, prompting engineers to commit fixes to Git.</p><p><strong>Real-World Incident</strong>: An engineer increased PostgreSQL memory limits to debug OOM. Self-heal reverted the change, OOM recurred. We now:</p><ul><li>Pause ArgoCD during active debugging</li><li>Commit fixes to Git within 30 minutes</li><li>Re-enable auto-sync after incident post-mortem</li></ul><h4>Conclusion</h4><p>Deploying a production-grade LiteLLM proxy on AWS EKS requires more than just running a container. It demands thoughtful decisions about:</p><ul><li><strong>Scalability</strong>: HPA with pod anti-affinity ensures traffic spikes don’t cause outages</li><li><strong>Reliability</strong>: Health checks, batch writes, and StatefulSets prevent data loss</li><li><strong>Observability</strong>: Structured logging and Prometheus metrics enable rapid debugging</li><li><strong>GitOps</strong>: ArgoCD ensures every change is audited and reversible</li></ul><p>Our deployment serves 500–1000 requests/second across 5 HPA replicas, with p95 latency under 150ms and 99.9% uptime over 1months. The architecture scales to 200 rps without infrastructure changes.</p><p><strong>Key Takeaways</strong>:</p><p>1. <strong>Test HPA behavior under load</strong> before production</p><p>2. <strong>Enable Prisma migrations</strong> to avoid schema mismatch errors</p><p>3. <strong>Monitor database connections</strong> to prevent exhaustion</p><p>4. <strong>Use structured JSON logs</strong> for easier debugging</p><p>5. <strong>Cache aggressively</strong> if workloads are repetitive</p><p>If you’re deploying LiteLLM in your organization, start with this architecture as a foundation. Adjust resource limits based on your load testing, and remember: <strong>GitOps with ArgoCD is your safety net</strong> — every change is reversible, audited, and reproducible.</p><blockquote>You can also get the complete source code from my public gitlab project,<br><a href="https://gitlab.com/KnowledgeSharing1/lite-llm-hpa.git"><em>https://gitlab.com/KnowledgeSharing1/lite-llm-hpa.git</em></a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996eda87e14d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/production-litellm-on-aws-eks-high-availability-with-gitops-996eda87e14d">Production LiteLLM on AWS EKS: High Availability with GitOps</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nexus Sonatype Repository: Complete Guide & Step-by-Step Setup]]></title>
            <link>https://medium.com/@saeedafzal2030/nexus-sonatype-repository-complete-guide-step-by-step-setup-d546cf2c8b0e?source=rss-ac59c13fc279------2</link>
            <guid isPermaLink="false">https://medium.com/p/d546cf2c8b0e</guid>
            <category><![CDATA[sonatype]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[sonatypenexus]]></category>
            <category><![CDATA[nexus]]></category>
            <dc:creator><![CDATA[Saeedafzal2030]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 13:32:32 GMT</pubDate>
            <atom:updated>2025-08-26T13:32:32.201Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="sonatype nexus repository logo" src="https://cdn-images-1.medium.com/max/600/1*4hIzBMKZScUyFKEjcdcygQ.jpeg" /></figure><p><strong>Nexus Repository Manager: A Complete Overview</strong></p><p>Nexus Repository Manager is a widely used as artifact repository management tool, especially popular in <strong>DevOps</strong> workflows. It acts as a <strong>central hub</strong> for storing and managing software components like binary files, libraries, and project dependencies. With Nexus, teams can easily control versions, manage access, and distribute artifacts across different environments. This centralization ensures <strong>consistency, traceability, and faster collaboration</strong> between development and operations teams throughout the <strong>software development lifecycle (SDLC)</strong>. By using Nexus Repository Manager, organizations can streamline their build and deployment process, reduce errors, and improve productivity.</p><h4><strong>Real-World Example: How Nexus Artifact Repository Streamlines DevOps Workflows</strong></h4><p>Imagine a software development team building a <strong>web application made up of multiple microservices</strong>. Each micro-service has its own codebase, dependencies, and build artifacts. In such a setup, <strong>Nexus Artifact Repository</strong> can become a critical part of the workflow by simplifying artifact storage, management, and distribution.</p><h4>Centralized Artifact Storage</h4><p>The team can configure <strong>Nexus Repository Manager</strong> as a <strong>single, centralized hub</strong> for storing all artifacts — such as binaries, libraries, frameworks, and other dependencies — across all microservices. Developers simply publish their build outputs to Nexus, ensuring a secure and easily accessible location for all project components.</p><h4>Efficient Dependency Management</h4><p>Instead of depending on <strong>external repositories</strong> (which may face downtime or security risks), Nexus can <strong>proxy and cache</strong> popular sources like <strong>Maven Central</strong> or <strong>npm</strong>. This gives the team <strong>full control</strong> over dependency versions, improves build reliability, and prevents disruptions caused by external repository failures.</p><h4>Seamless CI/CD Integration</h4><p>In a <strong>CI/CD pipeline</strong>, Nexus serves as the main storage point for build artifacts. When a developer pushes code changes, the CI server (e.g., <strong>Jenkins</strong>) automatically builds the application, runs tests, and uploads the resulting artifacts to Nexus. Deployment stages then pull these artifacts from Nexus to ensure consistent releases across environments.</p><h4>Artifact Versioning &amp; Traceability</h4><p>Nexus offers <strong>robust version control</strong>, making it easy to maintain multiple versions of artifacts. This capability allows teams to:</p><ul><li>Roll back to earlier versions when needed</li><li>Track changes over time</li><li>Maintain a full history for <strong>auditing and troubleshooting</strong></li></ul><p>This traceability ensures every release is documented and reproducible.</p><h4>Controlled Collaboration</h4><p>Nexus provides <strong>granular access control</strong>, so only authorized team members can publish or modify artifacts. This maintains security while enabling <strong>cross-team collaboration</strong>, ensuring everyone works from the same trusted source.</p><p><strong>Bottom line:</strong> By integrating <strong>Nexus Artifact Repository</strong> into their DevOps pipeline, teams can <strong>centralize artifact management, control dependencies, and streamline deployments</strong>. The result is faster delivery, improved reliability, and complete traceability throughout the <strong>software development lifecycle (SDLC)</strong>.</p><h4>Install Nexus on Linux: Follow these steps:</h4><ol><li><strong>Prerequisites:</strong></li></ol><p>Before installing Nexus, ensure that the <strong>Java Development Kit (JDK)</strong> is installed on your Linux system, as Nexus requires Java to function.<br> To check if Java is already installed, run the following command in your terminal:</p><pre># install java in ububtu 24.04<br>sudo apt install openjdk-17-jre-headless<br># check java version<br>java -version</pre><figure><img alt="image showing installation of java and java version" src="https://cdn-images-1.medium.com/max/1024/1*gzY4PIEZiILvSmZILY87yg.png" /></figure><p><strong>2. Download Nexus:</strong></p><ul><li>Visit the official Sonatype website and navigate to the Nexus Repository OSS download page: <a href="https://www.sonatype.com/products/sonatype-nexus-oss-download">https://www.sonatype.com/nexus/repository-oss-download</a>.</li><li>Download the latest version of <strong>Nexus Repository Manager OSS</strong> (Open Source Edition), and select the <strong>Unix/Linux</strong> distribution package from the list.</li><li>If you need an older version, you can find it in the <strong>Nexus download archives</strong> here: <a href="https://help.sonatype.com/en/download-archives---repository-manager-3.html"><strong>Nexus archive download</strong></a></li><li>Alternatively, you can download Nexus directly from the terminal using the following command:</li></ul><pre>#  Create directory for downloads<br>mkdir ~/Downloads<br>#  Change directory to downloads<br>cd ~/Downloads<br>#  Downloads the nexus repository OSS in Downloads directory<br>wget https://sonatype-download.global.ssl.fastly.net/repository/downloads-prod-group/3/nexus-3.83.0-08-linux-x86_64.tar.gz</pre><p><strong>3. Extract and configure Nexus:</strong></p><ul><li>Create directory under <strong><em>/opt/nexus</em></strong> to install Nexus</li><li>Open a terminal and navigate to the directory containing the downloaded Nexus package.</li><li>Then, extract the package by running the following command:</li></ul><pre>#  Create dirtory under /opt<br>sudo mkdir /opt/nexus<br>#  Change diretory<br>cd /opt/nexus<br>#  Copy .tar.gz file under /opt/nexus<br>sudo cp ~/Downloads /opt/nexus<br>#  In my case i downloaded and extracted latest nexus version 3.83<br>sudo tar -zxvf nexus-3.83.0-08-linux-x86_64.tar.gz<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*TD0x91vd-aPYdgv6BoByeA.png" /><figcaption>Nexus extracted content</figcaption></figure><p><strong>4. Configure Nexus</strong></p><p>As security practice, not to run nexus service using root user, so lets create new user named <strong>nexus</strong> to run nexus service:</p><pre>#  Create user nexus<br>sudo adduser nexus</pre><ul><li>To set no password for nexus user open the <strong>visudo</strong> file in ubuntu:</li></ul><pre>sudo visudo</pre><ul><li>Add below line into it , save and exit:</li></ul><pre>#  Allow nexus user to execute any command without asking for password<br>nexus ALL=(ALL) NOPASSWD: ALL</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*x_mpbZ9KM_2-1hxrcUVe9Q.png" /></figure><p>To run nexus as service at boot time, open<strong><em> /opt/nexus/nexus-3.83.0–08/bin/nexus.rc</em></strong> file, uncomment it and add nexus user as shown below:</p><pre>sudo nano /opt/nexus/nexus-3.83.0–08/bin/nexus.rc<br><br>#  Add below content<br>run_as_user=&quot;nexus&quot;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*LzkBRLN2mO-sBP3jR7cBbA.png" /><figcaption>create/modify nexus.rc file</figcaption></figure><p>Open the <strong><em>/opt/nexus/nexus-3.83.0–08/bin/nexus.vmoptions</em></strong> file:<br>To Increase the nexus JVM heap size, you can modify the size as shown below:</p><pre>#  These parameter can be ignored if your resource have around 2 vCPU and 8GB ram<br>-XX:MaxDirectMemorySize=2703m<br>-Djava.net.preferIPv4Stack=true</pre><p>Give permission to nexus files and nexus directory to <strong>nexus</strong> user:</p><pre>sudo chown -R nexus:nexus /opt/nexus<br>cd /opt/nexus/nexus-3.83.0-08/bin/<br>sudo chmod +x nexus</pre><p><strong>5. Run Nexus as a service using Systemd</strong></p><p>To run nexus as service using Systemd:</p><pre>sudo nano /etc/systemd/system/nexus.service<br><br>#  Inser the below content<br><br>[Unit]<br>Description=nexus service<br>After=network.target<br><br>[Service]<br>Type=forking<br>LimitNOFILE=65536<br>ExecStart=/bin/bash /opt/nexus/nexus-3.83.0-08/bin/nexus start<br>ExecStop=/bin/bash /opt/nexus/nexus-3.83.0-08/bin/nexus stop<br>User=nexus<br>Restart=on-abort<br><br>[Install]<br>WantedBy=multi-user.target</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*1vY9ajEIYedH9E0XIIB5KA.png" /><figcaption>nexus run as service</figcaption></figure><p>Start Nexus:</p><pre>sudo systemctl start nexus</pre><p>Enable Nexus service at system startup:</p><pre>sudo systemctl enable nexus</pre><p>check nexus service status:</p><pre>sudo systemctl status nexus</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0tpKCKmv34bZILcAeMJBFQ.png" /></figure><h4>Access Nexus Repository on Web Interface</h4><p>To access Nexus repository web interface , open your browser.<br>if you are running UFW firewall on Ubuntu, open the firewall port 8081 using below command:</p><pre>#  port 8081 allow nexus web page access<br>sudo ufw allow 8081/tcp<br>#  port 8082 for private docker hosted repo<br>sudo ufw allow 8082/tcp<br><br>#  Stop firewall<br>sudo systemctl stop ufw<br>#  Disable firewall<br>sudo systemctl disable ufw</pre><p>Use the below command to open to access Nexus repository web interface:</p><pre>http://server_IP:8081<br>#  OR<br>http://domain_name:8081</pre><p>To login to Nexus, click on<strong> Sign In</strong>, default username is <strong>admin</strong></p><p>To find default password run the below command:</p><pre>sudo cat /opt/nexus/sonatype-work/nexus3/admin.password</pre><p>copy the default nexus password and login, you can reset the password once logged in to nexus:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xDgspwhbEqFaTIBLjCJ_LQ.png" /><figcaption>Nexus web</figcaption></figure><p>Creating docker hosted repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wEPP2iyti83yD3i9gKD2Nw.png" /><figcaption>Configure docker hosted private repository</figcaption></figure><p>Login to docker hosted private repo using below command:</p><pre>echo &quot;pass123!@#&quot; | docker login --username admin --password-stdin http://nexus.devops.local:8082<br><br>#    Create docker image, tag and push to hosted repo<br><br>docker build -t nexus.devops.local:8082/thinkify_client:latest .<br><br>docker push nexus.devops.local:8082/thinkify_client:latest</pre><p>Pushing few sample docker images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c879FB9zbiG9dXkxokidzw.png" /><figcaption>Docker images in nexus GUI</figcaption></figure><p>Now your private nexus repository is completely functional and ready to use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d546cf2c8b0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>