LLM Provider Setup¶

Configure cloud and local LLM providers for semantic analysis workflows.

Overview¶

The coding infrastructure uses LLMs for:

• Knowledge extraction (UKB workflows)
• Semantic classification (LSL content routing)
• Continuous learning (real-time session analysis)
• Code analysis (pattern recognition)

You can use subscription providers, cloud APIs, local models, or a combination:

Subscription Providers (Zero Cost)¶

Claude Code (Recommended - Zero Cost)¶

Route requests through your existing Claude max subscription.

Setup:

# 1. Install Claude Code
# Download from: https://claude.ai/downloads

# 2. Verify installation
claude --version

# 3. Authenticate
claude login

# 4. Test
claude --print --silent "Say hello"

No environment variables needed - uses CLI directly.

Supported Models:

• sonnet (Claude Sonnet 4.5) - fast and standard tiers
• opus (Claude Opus 4.6) - premium tier

Features: - ✅ Zero per-token cost - ✅ Automatic quota tracking - ✅ Exponential backoff on exhaustion - ✅ Seamless fallback to API providers

GitHub Copilot (Primary Provider — Parallelism-Optimized)¶

Route requests through your GitHub Copilot subscription via direct HTTP POST to the Copilot API. This is the primary provider for all tiers because it scales beautifully with parallelism — 0.77s effective per call at 10 concurrent (vs 5s sequential). Batch agents already use Promise.all, so copilot as primary unlocks peak throughput.

Setup:

The Copilot provider reads OAuth tokens automatically from ~/.local/share/opencode/auth.json (populated by OpenCode on login). No manual setup required if you use OpenCode.

# Verify auth tokens exist
cat ~/.local/share/opencode/auth.json | jq '.github_token | length'

# If missing, launch OpenCode to trigger OAuth flow
opencode

No environment variables or CLI tools needed — uses direct HTTP to the Copilot API.

Supported Models:

• claude-haiku-4.5 (fast tier — benchmarked fastest at 0.77s @10 parallel)
• claude-sonnet-4.5 (standard tier)
• claude-opus-4.6 (premium tier)

Features: - ✅ Zero per-token cost - ✅ Parallelism-optimized (0.77s effective @10 concurrent) - ✅ Shared quota tracking - ✅ Automatic rotation on exhaustion - ✅ Work and personal subscriptions supported

Worker Pool Tuning (claude-code)¶

When a claude-code request can't use the fast direct-OAuth path and falls back to the CLI, the proxy serves it from a warm worker pool (v7.3) instead of cold-spawning claude -p every time — dropping steady-state fallback latency from ~10-14s to ~2-3s. This applies to claude-code only; copilot uses direct HTTP and is never pooled. See LLM Proxy Bridge → Claude CLI Worker Pool for the architecture.

The pool ships with safe defaults — most operators never need to touch it. Tune these env vars on the proxy host when you want to trade RAM for latency, bound prompt-pool memory, or harden against a crash-storming key:

Tuning notes:

• Quiet hosts: leave LLM_PROXY_WORKER_IDLE_MS at 30 min — idle workers evict themselves and free RAM; the next request transparently respawns one.
• High prompt diversity: raise LLM_PROXY_WORKER_PROMPT_CAP if many distinct (model × prompt) pairings thrash the LRU; lower it to cap memory.
• Recycling: LLM_PROXY_WORKER_MAX_REQUESTS and LLM_PROXY_WORKER_MAX_INPUT_TOKENS proactively retire long-lived workers before they accumulate state; lower them if you observe drift.
• Crash storms: if a key keeps crashing, the pool routes it to cold one-shot overflow after LLM_PROXY_WORKER_CRASH_THRESHOLD crashes within LLM_PROXY_WORKER_CRASH_WINDOW_MS, avoiding a spawn→crash→respawn loop.
• Escape hatch: set LLM_PROXY_DISABLE_WORKER_POOL=1 to disable pooling entirely and force every fallback through the original cold one-shot CLI path — useful for isolating pool behaviour during debugging.

Cloud API Providers¶

Groq (Recommended Fallback)¶

Fastest inference for open models. Recommended for UKB workflows.

# .env
GROQ_API_KEY=gsk_...

Supported Models:

• llama-3.3-70b-versatile (default for semantic analysis)
• llama-3.1-8b-instant (fast, lower quality)
• mixtral-8x7b-32768 (good for long context)

Anthropic Claude¶

High-quality analysis, used as fallback.

# .env
ANTHROPIC_API_KEY=sk-ant-...

Supported Models:

• claude-sonnet-4-5 (standard tier)
• claude-haiku-4-5 (fast tier)
• claude-opus-4-6 (premium tier)

OpenAI¶

GPT models and embeddings.

# .env
OPENAI_API_KEY=sk-...

Supported Models:

• gpt-4.1 (standard tier)
• gpt-4.1-mini (fast tier)
• o4-mini (premium tier - reasoning)
• text-embedding-3-small (embeddings)

Google Gemini¶

Alternative provider.

# .env
GOOGLE_API_KEY=...

Supported Models:

• gemini-2.5-flash (fast and standard tiers)
• gemini-2.5-pro (premium tier)

GitHub Models¶

Free tier access to OpenAI models via GitHub.

# .env
GITHUB_TOKEN=ghp_...

Supported Models:

• gpt-4.1 (standard tier)
• gpt-4.1-mini (fast tier)
• o4-mini (premium tier)

Base URL: https://models.github.ai/inference/v1

Local Models (DMR/llama.cpp)¶

Run models locally for zero cost and complete privacy.

Architecture¶

flowchart LR
    subgraph Host
        A[Coding Services] --> B[DMR Port 12434]
    end

    subgraph Docker
        B --> C[llama.cpp Server]
        C --> D[Local Model]
    end

Docker Model Runner (DMR)¶

DMR provides an OpenAI-compatible API for local models.

Setup:

1. Configure Port (in .env.ports):

DMR_PORT=12434
DMR_HOST=localhost

1. Start DMR (via Docker):

# DMR is included in docker-compose.yml
# Starts automatically with: coding

1. Verify Connection:

curl http://localhost:12434/v1/models

llama.cpp Direct¶

For non-Docker setups or custom configurations.

Build llama.cpp:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j

Start Server:

./llama-server \
  --model /path/to/model.gguf \
  --host 0.0.0.0 \
  --port 12434 \
  --ctx-size 4096

Recommended Local Models¶

Model Download¶

# Using Hugging Face CLI
huggingface-cli download TheBloke/Llama-2-7B-GGUF \
  llama-2-7b.Q4_K_M.gguf \
  --local-dir ./models

# Or via DMR (if supported)
curl -X POST http://localhost:12434/v1/models/pull \
  -d '{"model": "llama-3.2-3b"}'

Unified LLM Layer¶

All LLM requests route through the @rapid/llm-proxy unified layer, which provides:

• 14 providers: 2 subscription (Copilot via direct HTTP, Claude Code via CLI), 5 cloud API, 2 local, 1 mock, plus proxy and OpenAI-compatible
• Copilot-first parallelized routing: Copilot scales with concurrency (0.77s @10 parallel), always tried first
• Tier-based routing: Automatic provider selection based on task complexity
• Quota tracking: Persistent usage tracking with exponential backoff
• Circuit breaker: Prevents cascading failures (threshold: 5 failures, reset: 60s)
• LRU cache: Deduplicates requests (1000 entries, 1-hour TTL)
• Metrics tracking: Cost, performance, and usage stats per provider

Cost Savings: Subscription-first routing pushes most UKB/LSL analysis through Copilot/Claude max subscriptions before any paid API call, eliminating the per-token spend on those flows entirely.

See @rapid/llm-proxy Architecture and LLM Architecture for complete details.

Tier-Based Routing¶

Requests are automatically routed based on task complexity and cost optimization:

Tier Definitions¶

Routing Flow¶

flowchart TB
    A[LLM Request] --> B{Determine Tier}
    B -->|Fast| C[Copilot → Groq → Claude Code → ...]
    B -->|Standard| D[Copilot → Groq → Claude Code → Anthropic → OpenAI → ...]
    B -->|Premium| E[Copilot → Groq → Claude Code → Anthropic → OpenAI → ...]
    C --> F{Quota Available?}
    D --> F
    E --> F
    F -->|No| G[Skip to Next Provider]
    F -->|Yes| H{Circuit Breaker OK?}
    H -->|No| G
    H -->|Yes| I{Check Cache}
    I -->|Hit| J[Return Cached]
    I -->|Miss| K[Make Request]
    K --> L{Success?}
    L -->|Yes| M[Record Usage & Cache]
    M --> N[Return Result]
    L -->|No - Quota| O[Mark Exhausted]
    O --> G
    L -->|No - Other| P{More Providers?}
    P -->|Yes| F
    P -->|No| Q[Try Local: DMR → Ollama]
    Q --> R{Success?}
    R -->|Yes| N
    R -->|No| S[Fail with Error]

Configuration¶

Tier routing is configured in config/llm-providers.yaml:

# Copilot first — scales with parallelism (0.77s effective @10 concurrent)
provider_priority:
  fast: ["copilot", "groq", "claude-code", "anthropic", "openai", "gemini", "github-models"]
  standard: ["copilot", "groq", "claude-code", "anthropic", "openai", "gemini", "github-models"]
  premium: ["copilot", "groq", "claude-code", "anthropic", "openai", "gemini", "github-models"]

task_tiers:
  fast:
    - git_file_extraction
    - commit_message_parsing
    - basic_classification
  standard:
    - git_history_analysis
    - semantic_code_analysis
    - ontology_classification
  premium:
    - insight_generation
    - observation_generation
    - pattern_recognition

# Subscription quota tracking
providers:
  claude-code:
    cliCommand: "claude"
    quotaTracking:
      enabled: true
      softLimitPerHour: 100
  copilot:
    cliCommand: "copilot-cli"
    quotaTracking:
      enabled: true
      softLimitPerHour: 100

Semantic Workload Routing¶

Different workloads are routed to appropriate providers:

Routing Rules¶

Sensitivity-Based Routing¶

The 5-layer sensitivity classifier automatically routes:

• Safe content → Cloud providers (fast, accurate)
• Sensitive content → Local models (private, free)

Sensitive patterns detected:

• API keys and tokens
• Passwords and credentials
• Email addresses
• Corporate identifiers
• Financial data

Workflow-Specific Configuration¶

UKB Workflows¶

# .env
UKB_LLM_PROVIDER=groq
UKB_LLM_MODEL=llama-3.3-70b-versatile
UKB_FALLBACK_PROVIDER=local

Continuous Learning¶

# .env
LEARNING_LLM_PROVIDER=groq
LEARNING_BUDGET_LIMIT=10  # Monthly USD cap (configurable)
LEARNING_FALLBACK_TO_LOCAL=true

LSL Classification¶

# .env
LSL_CLASSIFIER_PROVIDER=local  # Use local for privacy
LSL_CLASSIFIER_MODEL=llama-3.2-3b

Debug Mode¶

For development and testing, use mock or local modes:

Mock Mode¶

Returns simulated responses without LLM calls:

# Via Claude chat
ukb debug  # Enables mock mode

# Via environment
export SEMANTIC_LLM_MODE=mock

Local-Only Mode¶

Force all calls to local models:

export SEMANTIC_LLM_MODE=local

Provider Override¶

Force a specific provider:

export SEMANTIC_LLM_PROVIDER=groq
export SEMANTIC_LLM_MODEL=llama-3.3-70b-versatile

Cost Management¶

Budget Tracking¶

The system tracks LLM costs per provider:

# Check current budget status
curl http://localhost:3848/api/budget

Response:

{
  "monthlyLimit": 10,
  "used": 2.50,
  "remaining": 7.50,
  "percentage": 25.0,
  "providers": {
    "groq": 1.20,
    "anthropic": 0.85,
    "openai": 0.45
  }
}

Provider Costs¶

Cost Savings: When subscription quota is available, those requests cost $0 per token instead of the per-token rates above — the per-call savings compound across the day-to-day volume of UKB/LSL analysis.

Budget Alerts¶

Configure alerts in .env:

BUDGET_ALERT_50=log      # Log at 50%
BUDGET_ALERT_80=warn     # Warning at 80%
BUDGET_ALERT_90=notify   # Notification at 90%
BUDGET_EXCEEDED=local    # Switch to local when exceeded

Troubleshooting¶

DMR Not Responding¶

# Check if DMR is running
curl http://localhost:12434/health

# Check Docker container
docker ps | grep dmr

# View logs
docker logs coding-dmr

Model Not Found¶

# List available models
curl http://localhost:12434/v1/models

# Pull model (if DMR supports)
curl -X POST http://localhost:12434/v1/models/pull \
  -d '{"model": "llama-3.2-3b"}'

Slow Local Inference¶

• Use smaller quantized models (Q4_K_M)
• Increase context size only if needed
• Ensure GPU acceleration is enabled
• Check available RAM

Provider Timeout¶

# Increase timeout (milliseconds)
LLM_TIMEOUT=60000
LLM_RETRY_COUNT=3
LLM_RETRY_DELAY=1000

Related Documentation¶

• Health Dashboard - Workflow debug controls
• Knowledge Workflows - UKB system details
• Configuration - API keys setup