Session 8 - Module B: Enterprise Scaling & Architecture (70 minutes)¶
Prerequisites: Session 8 Core Section Complete
Target Audience: Platform engineers and architects building scalable agent systems
Cognitive Load: 6 advanced concepts
Module Overview¶
This module explores enterprise-scale architecture patterns for Agno agent systems including Kubernetes orchestration, auto-scaling strategies, multi-tenant architectures, service mesh integration, and global deployment patterns. You'll learn to build massively scalable agent platforms that can handle enterprise workloads across multiple regions.
Learning Objectives¶
By the end of this module, you will:
- Design Kubernetes-native agent architectures with sophisticated auto-scaling
- Implement multi-tenant agent platforms with resource isolation and fair scheduling
- Create service mesh architectures for secure inter-agent communication
- Build global deployment strategies with regional failover and data locality
Part 1: Kubernetes-Native Agent Architecture (25 minutes)¶
Step 1: Understanding Enterprise-Scale Challenges¶
Before we dive into the code, let's understand what makes enterprise agent scaling different from simple deployments. We're dealing with thousands of agents, multi-tenancy, cost optimization, and strict SLAs.
🗂️ File: src/session8/k8s_native_architecture.py - Kubernetes-native agent systems
First, let's import our essential libraries for enterprise Kubernetes management:
from typing import Dict, List, Any, Optional
from dataclasses import dataclass, field
from datetime import datetime, timedelta
import asyncio
import yaml
import json
Step 2: Designing the Cluster Configuration Schema¶
Let's start by defining what makes up an enterprise agent cluster. This configuration will drive everything from resource allocation to security policies:
@dataclass
class K8sAgentCluster:
"""Kubernetes-native agent cluster configuration"""
cluster_name: str
namespace: str = "agno-agents"
node_pools: List[Dict[str, Any]] = field(default_factory=list)
scaling_policies: Dict[str, Any] = field(default_factory=dict)
resource_quotas: Dict[str, str] = field(default_factory=dict)
network_policies: List[Dict[str, Any]] = field(default_factory=list)
Design philosophy: We separate concerns into distinct configuration areas - compute (node pools), scaling, resources, and networking. This makes the system much easier to manage at scale.
Step 3: Building the Kubernetes Orchestrator Foundation¶
Next, we'll create the main orchestrator class that manages Kubernetes resources for our agent platform:
class KubernetesAgentOrchestrator:
"""Advanced Kubernetes orchestration for Agno agents"""
def __init__(self, cluster_config: K8sAgentCluster):
self.cluster_config = cluster_config
self.custom_resources = {}
self.operators = {}
Orchestrator design: The orchestrator maintains references to cluster configuration and manages both custom resources and operators for comprehensive agent lifecycle management.
Step 4: Creating Custom Resource Definitions¶
Custom Resource Definitions (CRDs) extend Kubernetes API to support agent-specific objects. Let's start with the basic CRD structure:
def create_agent_custom_resource(self) -> Dict[str, Any]:
"""Create custom Kubernetes resource for Agno agents"""
agent_crd = {
"apiVersion": "apiextensions.k8s.io/v1",
"kind": "CustomResourceDefinition",
"metadata": {
"name": "agnoagents.ai.company.com"
},
CRD foundation: This establishes our custom API group and resource name that Kubernetes will recognize.
Step 5: Defining Agent Specification Schema¶
Now we define what parameters our AgnoAgent custom resource accepts:
"spec": {
"group": "ai.company.com",
"versions": [{
"name": "v1",
"served": True,
"storage": True,
"schema": {
"openAPIV3Schema": {
"type": "object",
"properties": {
"spec": {
"type": "object",
"properties": {
"agentName": {"type": "string"},
"modelName": {"type": "string"},
"replicas": {"type": "integer"},
"tools": {
"type": "array",
"items": {"type": "string"}
},
Schema validation: OpenAPI v3 schema ensures only valid agent configurations are accepted by Kubernetes.
Step 6: Resource Requirements Definition¶
Defining resource limits and requests is crucial for proper scheduling and resource management:
"resources": {
"type": "object",
"properties": {
"requests": {
"type": "object",
"properties": {
"cpu": {"type": "string"},
"memory": {"type": "string"}
}
},
"limits": {
"type": "object",
"properties": {
"cpu": {"type": "string"},
"memory": {"type": "string"}
}
}
}
},
Resource management: Requests guarantee minimum resources while limits prevent resource abuse in multi-tenant environments.
Step 7: Scaling Configuration Schema¶
Autoscaling parameters need to be configurable per agent type:
"scaling": {
"type": "object",
"properties": {
"minReplicas": {"type": "integer"},
"maxReplicas": {"type": "integer"},
"targetCPU": {"type": "integer"},
"targetMemory": {"type": "integer"},
"customMetrics": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {"type": "string"},
"target": {"type": "number"}
}
}
}
}
}
Scaling flexibility: Custom metrics allow scaling based on business metrics like queue depth or response time, not just CPU/memory.
Step 8: Status Tracking Schema¶
Every custom resource needs status tracking for operators and monitoring:
}
},
"status": {
"type": "object",
"properties": {
"phase": {"type": "string"},
"readyReplicas": {"type": "integer"},
Basic status fields: Phase and replica count provide essential operational information.
Step 8b: Detailed Condition Tracking¶
"conditions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"type": {"type": "string"},
"status": {"type": "string"},
"reason": {"type": "string"},
"message": {"type": "string"}
}
}
}
}
}
Status management: Kubernetes operators use status fields to communicate resource state and health.
Step 9: CRD Metadata and Naming¶
Finally, we define how the custom resource appears in Kubernetes:
}
}
}
}],
"scope": "Namespaced",
"names": {
"plural": "agnoagents",
"singular": "agnoagent",
"kind": "AgnoAgent",
"shortNames": ["aa"]
}
}
}
return agent_crd
### Step 10: Advanced Horizontal Pod Autoscaler (HPA) Configuration
Now let's implement intelligent autoscaling that responds to multiple metrics and handles scaling behavior intelligently:
```python
def generate_advanced_hpa_configuration(self, agent_name: str) -> Dict[str, Any]:
"""Generate advanced HPA with custom metrics and behaviors"""
hpa_config = {
"apiVersion": "autoscaling/v2",
"kind": "HorizontalPodAutoscaler",
"metadata": {
"name": f"{agent_name}-advanced-hpa",
"namespace": self.cluster_config.namespace
},
HPA v2 features: We use the latest autoscaling API for advanced metrics and behavioral controls.
Step 11: HPA Target Reference and Boundaries¶
"spec": {
"scaleTargetRef": {
"apiVersion": "apps/v1",
"kind": "Deployment",
"name": agent_name
},
"minReplicas": 2,
"maxReplicas": 100,
Scaling boundaries: Minimum 2 replicas ensures high availability, maximum 100 prevents runaway scaling.
Step 12: Multi-Metric Scaling Configuration¶
Advanced HPA uses multiple metrics for more intelligent scaling decisions:
"metrics": [
{
"type": "Resource",
"resource": {
"name": "cpu",
"target": {
"type": "Utilization",
"averageUtilization": 70
}
}
},
CPU metrics: 70% CPU utilization threshold provides responsive scaling without thrashing.
Step 12b: Memory Utilization Metrics¶
{
"type": "Resource",
"resource": {
"name": "memory",
"target": {
"type": "Utilization",
"averageUtilization": 80
}
}
},
Resource metrics: CPU at 70% and memory at 80% utilization trigger scaling - different thresholds reflect their different performance impacts.
Step 13: Custom Business Metrics¶
Beyond resource metrics, we use business-specific metrics for smarter scaling:
{
"type": "Pods",
"pods": {
"metric": {
"name": "agno_pending_requests"
},
"target": {
"type": "AverageValue",
"averageValue": "30"
}
}
},
Pod-based metrics: Pending requests per pod provide direct insight into agent workload pressure.
Step 13b: External Queue Metrics¶
{
"type": "External",
"external": {
"metric": {
"name": "pubsub.googleapis.com|subscription|num_undelivered_messages",
"selector": {
"matchLabels": {
"resource.labels.subscription_id": f"{agent_name}-queue"
}
}
},
"target": {
"type": "Value",
"value": "30"
}
}
}
],
Business-aware scaling: Pending requests and queue depth provide better scaling signals than resource utilization alone.
Step 14: Intelligent Scaling Behavior¶
Modern HPA supports sophisticated scaling policies to prevent thrashing:
"behavior": {
"scaleUp": {
"stabilizationWindowSeconds": 30,
"policies": [
{
"type": "Percent",
"value": 100,
"periodSeconds": 15
},
{
"type": "Pods",
"value": 4,
"periodSeconds": 15
}
],
"selectPolicy": "Max"
},
Aggressive scale-up: 30-second stabilization with either 100% increase or 4 pods every 15 seconds - whichever is more aggressive.
Step 15: Conservative Scale-Down Behavior¶
"scaleDown": {
"stabilizationWindowSeconds": 300,
"policies": [
{
"type": "Percent",
"value": 10,
"periodSeconds": 60
}
]
}
}
}
}
return hpa_config
Conservative scale-down: 5-minute stabilization with maximum 10% reduction per minute prevents over-scaling during temporary load drops.
Step 16: Cluster Autoscaler Configuration¶
While HPA scales pods, Cluster Autoscaler scales the underlying nodes. Let's configure intelligent node scaling:
def create_cluster_autoscaler_config(self) -> Dict[str, Any]:
"""Configure cluster autoscaler for agent workloads"""
autoscaler_config = {
"scale_down_delay_after_add": "10m",
"scale_down_unneeded_time": "10m",
"scale_down_utilization_threshold": 0.5,
"scale_down_gpu_utilization_threshold": 0.5,
"max_node_provision_time": "15m",
Autoscaler timing: 10-minute delays prevent node thrashing while 50% utilization threshold ensures efficient resource usage.
Step 17: Specialized Node Pools for Different Workloads¶
Different agent types need different compute resources. Let's define specialized node pools:
"node_groups": [
{
"name": "agno-agents-cpu",
"min_size": 2,
"max_size": 100,
"instance_type": "n1-standard-4",
"labels": {
"workload-type": "cpu-agents",
"cost-optimization": "enabled"
},
"taints": []
},
CPU node pool: General-purpose nodes for standard agent workloads with cost optimization enabled.
Step 18: GPU-Enabled Node Pool¶
{
"name": "agno-agents-gpu",
"min_size": 0,
"max_size": 20,
"instance_type": "n1-standard-4-gpu",
"labels": {
"workload-type": "gpu-agents",
"accelerator": "nvidia-tesla-t4"
},
"taints": [
{
"key": "nvidia.com/gpu",
"value": "present",
"effect": "NoSchedule"
}
]
},
GPU specialization: Minimum 0 nodes saves costs when no GPU workloads exist. Taints ensure only GPU-requiring pods are scheduled here.
Step 19: Memory-Intensive Node Pool¶
{
"name": "agno-agents-memory",
"min_size": 1,
"max_size": 50,
"instance_type": "n1-highmem-8",
"labels": {
"workload-type": "memory-intensive-agents"
},
"taints": []
}
]
}
return autoscaler_config
Memory optimization: High-memory instances for large language models or agents with extensive context windows.
Step 20: Multi-Tenant Platform Architecture¶
Enterprise agent platforms must support multiple customers or business units with proper isolation and resource management:
class MultiTenantAgentPlatform:
"""Multi-tenant platform for enterprise agent deployments"""
def __init__(self):
self.tenant_configurations = {}
self.resource_quotas = {}
self.network_policies = {}
self.rbac_policies = {}
Multi-tenancy foundation: Separate configurations for each tenant ensure proper isolation and resource management.
Step 21: Creating Isolated Tenant Namespaces¶
Each tenant gets their own Kubernetes namespace with resource quotas and limits:
def create_tenant_namespace(self, tenant_id: str,
resource_limits: Dict[str, str]) -> Dict[str, Any]:
"""Create isolated namespace for tenant with resource quotas"""
namespace_config = {
"namespace": {
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"name": f"agno-tenant-{tenant_id}",
"labels": {
"tenant-id": tenant_id,
"platform": "agno-agents",
"isolation": "namespace"
},
"annotations": {
"scheduler.alpha.kubernetes.io/node-selector": f"tenant-{tenant_id}=true"
}
}
},
Namespace isolation: Each tenant gets a dedicated namespace with clear labeling for management and optional node affinity.
Step 22: Resource Quota Management¶
Resource quotas prevent any single tenant from consuming excessive cluster resources:
"resource_quota": {
"apiVersion": "v1",
"kind": "ResourceQuota",
"metadata": {
"name": f"tenant-{tenant_id}-quota",
"namespace": f"agno-tenant-{tenant_id}"
},
"spec": {
"hard": {
"requests.cpu": resource_limits.get("requests_cpu", "10"),
"requests.memory": resource_limits.get("requests_memory", "20Gi"),
"limits.cpu": resource_limits.get("limits_cpu", "20"),
"limits.memory": resource_limits.get("limits_memory", "40Gi"),
"persistentvolumeclaims": resource_limits.get("pvc_count", "10"),
"pods": resource_limits.get("pod_count", "50"),
"services": resource_limits.get("service_count", "20"),
"secrets": resource_limits.get("secret_count", "30")
}
}
},
Comprehensive quotas: Limits cover compute resources, storage, and Kubernetes objects to prevent resource abuse.
Step 23: Container-Level Resource Limits¶
LimitRange provides default and maximum resource constraints for individual containers:
"limit_range": {
"apiVersion": "v1",
"kind": "LimitRange",
"metadata": {
"name": f"tenant-{tenant_id}-limits",
"namespace": f"agno-tenant-{tenant_id}"
},
"spec": {
"limits": [
{
"type": "Container",
"default": {
"cpu": "500m",
"memory": "1Gi"
},
"defaultRequest": {
"cpu": "100m",
"memory": "128Mi"
},
"max": {
"cpu": "2",
"memory": "4Gi"
},
"min": {
"cpu": "50m",
"memory": "64Mi"
}
}
]
}
}
}
return namespace_config
### Step 24: Network Isolation Policies
Network policies ensure tenants cannot access each other's resources:
```python
def create_tenant_network_policies(self, tenant_id: str) -> List[Dict[str, Any]]:
"""Create network isolation policies for tenant"""
policies = [
{
"apiVersion": "networking.k8s.io/v1",
"kind": "NetworkPolicy",
"metadata": {
"name": f"tenant-{tenant_id}-default-deny",
"namespace": f"agno-tenant-{tenant_id}"
},
"spec": {
"podSelector": {},
"policyTypes": ["Ingress", "Egress"]
}
},
Default deny policy: By default, all traffic is blocked - security by default principle.
Step 25: Intra-Tenant Communication Policy¶
Tenants need to communicate within their own namespace but not with other tenants:
{
"apiVersion": "networking.k8s.io/v1",
"kind": "NetworkPolicy",
"metadata": {
"name": f"tenant-{tenant_id}-allow-intra-tenant",
"namespace": f"agno-tenant-{tenant_id}"
},
"spec": {
"podSelector": {},
"policyTypes": ["Ingress", "Egress"],
"ingress": [{
"from": [{
"namespaceSelector": {
"matchLabels": {
"tenant-id": tenant_id
}
}
}]
}],
Selective access: Only pods from the same tenant namespace can communicate with each other.
Step 26: Essential External Access¶
Even with strict isolation, agents need DNS resolution and HTTPS access:
"egress": [
{
"to": [{
"namespaceSelector": {
"matchLabels": {
"tenant-id": tenant_id
}
}
}]
},
{
"to": [],
"ports": [
{"protocol": "TCP", "port": 53},
{"protocol": "UDP", "port": 53}
]
},
{
"to": [],
"ports": [{"protocol": "TCP", "port": 443}]
}
]
}
}
]
return policies
**Essential services**: DNS (port 53) and HTTPS (port 443) access ensure agents can resolve names and access external APIs.
### Step 27: Fair Scheduling Policies
In multi-tenant environments, we need fair scheduling to prevent any tenant from monopolizing cluster resources:
```python
def create_fair_scheduling_policies(self, tenant_id: str,
priority_class: str = "normal") -> Dict[str, Any]:
"""Create fair scheduling policies for multi-tenant workloads"""
scheduling_config = {
"priority_class": {
"apiVersion": "scheduling.k8s.io/v1",
"kind": "PriorityClass",
"metadata": {
"name": f"tenant-{tenant_id}-{priority_class}"
},
"value": self._get_priority_value(priority_class),
"globalDefault": False,
"description": f"Priority class for tenant {tenant_id} with {priority_class} priority"
},
Priority classes: Different tenants can have different priorities, ensuring critical workloads get scheduled first.
Step 28: Pod Disruption Budgets¶
Pod Disruption Budgets ensure tenant workloads maintain availability during cluster maintenance:
"pod_disruption_budget": {
"apiVersion": "policy/v1",
"kind": "PodDisruptionBudget",
"metadata": {
"name": f"tenant-{tenant_id}-pdb",
"namespace": f"agno-tenant-{tenant_id}"
},
"spec": {
"minAvailable": "50%",
"selector": {
"matchLabels": {
"tenant-id": tenant_id
}
}
}
}
}
return scheduling_config
Availability guarantee: At least 50% of tenant pods must remain available during voluntary disruptions.
Step 29: Priority Value Mapping¶
def _get_priority_value(self, priority_class: str) -> int:
"""Get numeric priority value for scheduling"""
priority_mapping = {
"critical": 1000,
"high": 800,
"normal": 500,
"low": 200,
"best-effort": 100
}
return priority_mapping.get(priority_class, 500)
Priority hierarchy: Clear numeric priorities help Kubernetes scheduler make consistent decisions during resource contention.
Part 2: Service Mesh and Global Architecture (25 minutes)¶
Istio Service Mesh Integration¶
🗂️ File: src/session8/service_mesh_architecture.py - Service mesh for agent communication
Step 30: Service Mesh Foundation¶
Service mesh provides advanced traffic management, security, and observability for inter-agent communication:
from typing import Dict, List, Any, Optional
import yaml
class IstioServiceMeshConfig:
"""Istio service mesh configuration for agent systems"""
def __init__(self, mesh_name: str = "agno-mesh"):
self.mesh_name = mesh_name
self.gateway_configs = {}
self.virtual_services = {}
self.destination_rules = {}
Service mesh architecture: Centralized configuration for all network policies, traffic routing, and security across the agent platform.
Step 31: Creating Istio Gateway¶
Gateways control traffic entering the service mesh from external sources:
def create_agent_gateway(self, domain: str) -> Dict[str, Any]:
"""Create Istio gateway for agent services"""
gateway_config = {
"apiVersion": "networking.istio.io/v1beta1",
"kind": "Gateway",
"metadata": {
"name": "agno-agents-gateway",
"namespace": "agno-system"
},
"spec": {
"selector": {
"istio": "ingressgateway"
},
Gateway targeting: Selects the Istio ingress gateway for external traffic entry into the mesh.
Step 32: HTTPS Configuration with TLS¶
"servers": [
{
"port": {
"number": 443,
"name": "https",
"protocol": "HTTPS"
},
"tls": {
"mode": "SIMPLE",
"credentialName": f"agno-agents-{domain}-tls"
},
"hosts": [f"agents.{domain}"]
},
TLS termination: Gateway handles SSL/TLS termination using certificates stored as Kubernetes secrets.
Step 33: HTTP to HTTPS Redirection¶
{
"port": {
"number": 80,
"name": "http",
"protocol": "HTTP"
},
"hosts": [f"agents.{domain}"],
"tls": {
"httpsRedirect": True
}
}
]
}
}
return gateway_config
Security by default: All HTTP traffic is automatically redirected to HTTPS for secure communication.
Step 34: Canary Deployment Configuration¶
Canary deployments allow safe rollouts of new agent versions by gradually shifting traffic:
def create_canary_deployment_config(self, service_name: str) -> Dict[str, Any]:
"""Create canary deployment configuration with traffic splitting"""
virtual_service = {
"apiVersion": "networking.istio.io/v1beta1",
"kind": "VirtualService",
"metadata": {
"name": f"{service_name}-canary",
"namespace": "agno-agents"
},
"spec": {
"hosts": [service_name],
"http": [
VirtualService foundation: Routes traffic to different versions based on headers and weights.
Step 35: Header-Based Canary Routing¶
First, we define explicit canary traffic routing for testing:
{
"match": [
{
"headers": {
"canary": {
"exact": "true"
}
}
}
],
"route": [
{
"destination": {
"host": service_name,
"subset": "canary"
}
}
]
},
Explicit canary routing: Requests with canary: true header go directly to the canary version for controlled testing.
Step 36: Weighted Traffic Distribution¶
For general traffic, we use weighted distribution to gradually roll out the canary:
Stable traffic: 90% of traffic goes to the proven stable version for reliability.
Step 36b: Canary Traffic Allocation¶
Gradual rollout: 90% stable, 10% canary traffic split allows safe testing with real user traffic.
Step 37: Destination Rules for Traffic Policies¶
Destination rules define traffic policies and service subsets:
destination_rule = {
"apiVersion": "networking.istio.io/v1beta1",
"kind": "DestinationRule",
"metadata": {
"name": f"{service_name}-destination",
"namespace": "agno-agents"
},
"spec": {
"host": service_name,
"trafficPolicy": {
"loadBalancer": {
"simple": "LEAST_CONN"
},
Load balancing: LEAST_CONN ensures even distribution across healthy instances.
Step 38: Connection Pool and Circuit Breaker¶
"connectionPool": {
"tcp": {
"maxConnections": 100
},
"http": {
"http1MaxPendingRequests": 50,
"maxRequestsPerConnection": 10,
"consecutiveGatewayErrors": 5,
"interval": "30s",
"baseEjectionTime": "30s"
}
},
"circuitBreaker": {
"consecutiveErrors": 5
}
},
Resilience patterns: Connection limits and circuit breakers prevent cascade failures.
Step 39: Service Subsets Definition¶
"subsets": [
{
"name": "stable",
"labels": {
"version": "stable"
}
},
{
"name": "canary",
"labels": {
"version": "canary"
}
}
]
}
}
return {"virtual_service": virtual_service, "destination_rule": destination_rule}
Version targeting: Subsets use labels to distinguish between stable and canary deployments.
Step 40: Global Agent Deployment Strategy¶
For enterprise scale, agents must be deployed across multiple regions for performance and disaster recovery:
class GlobalAgentDeployment:
"""Global deployment strategy for agent systems"""
def __init__(self):
self.regions = {}
self.failover_policies = {}
self.data_locality_rules = {}
Global state management: Separate tracking for regional deployments, failover policies, and data locality requirements.
Step 41: Multi-Region Deployment Configuration¶
Deploying across multiple regions requires careful coordination of resources and traffic:
def create_multi_region_deployment(self, regions: List[str]) -> Dict[str, Any]:
"""Create multi-region deployment configuration"""
global_config = {
"regions": {},
"traffic_routing": {},
"data_replication": {},
"failover_strategy": {}
}
Configuration structure: Organized into regional resources, traffic management, data handling, and failure scenarios.
Step 42: Regional Cluster Configuration¶
Each region needs its own Kubernetes cluster with appropriate sizing:
for region in regions:
global_config["regions"][region] = {
"cluster_config": {
"name": f"agno-agents-{region}",
"zone": f"{region}-a",
"node_pools": [
{
"name": "agent-pool",
"min_size": 2,
"max_size": 50,
"machine_type": "n1-standard-4"
}
]
},
Regional scaling: Each region has independent scaling boundaries appropriate for expected load.
Step 43: Regional Network Configuration¶
"network_config": {
"vpc_name": f"agno-vpc-{region}",
"subnet_ranges": ["10.0.0.0/16"],
"secondary_ranges": ["10.1.0.0/16"]
}
}
Network isolation: Each region gets its own VPC with non-overlapping IP ranges for secure inter-region communication.
Step 44: Global Traffic Routing Strategy¶
# Traffic routing configuration
global_config["traffic_routing"][region] = {
"weight": 100 // len(regions), # Equal distribution
"priority": 1,
"health_checks": {
"path": "/health",
"interval": "30s",
"timeout": "5s"
}
}
return global_config
Equal distribution: Traffic is initially distributed equally across all healthy regions with comprehensive health monitoring.
Step 45: Comprehensive Disaster Recovery Planning¶
Enterprise systems need detailed disaster recovery plans with clear objectives and automation:
def create_disaster_recovery_plan(self) -> Dict[str, Any]:
"""Create comprehensive disaster recovery plan"""
dr_config = {
"rpo_target": "1h", # Recovery Point Objective
"rto_target": "30m", # Recovery Time Objective
Recovery objectives: 1-hour RPO means maximum 1 hour of data loss, 30-minute RTO means service restoration within 30 minutes.
Step 46: Backup Strategy Definition¶
Different data types require different backup frequencies and retention policies:
"backup_strategy": {
"agent_configurations": {
"frequency": "daily",
"retention": "30d",
"storage": "multi-region"
},
"conversation_data": {
"frequency": "hourly",
"retention": "90d",
"encryption": "AES-256"
},
"model_artifacts": {
"frequency": "weekly",
"retention": "1y",
"versioning": True
}
},
Tiered backup strategy: Critical conversation data backed up hourly, configurations daily, and large model artifacts weekly.
Step 47: Automated Failover Configuration¶
"failover_automation": {
"health_check_failures": 3,
"failure_window": "5m",
"automatic_failover": True,
"notification_channels": ["slack", "pagerduty"]
},
Failover triggers: Three consecutive health check failures within 5 minutes trigger automatic failover with immediate notifications.
Step 48: Data Recovery Procedures¶
"recovery_procedures": {
"data_recovery": {
"primary_source": "continuous_replication",
"backup_source": "point_in_time_backup",
"validation_steps": [
"verify_data_integrity",
"test_agent_functionality",
"validate_performance_metrics"
]
},
Recovery validation: Multi-step process ensures recovered data is complete, functional, and performing within acceptable parameters.
Step 49: Service Recovery Strategy¶
"service_recovery": {
"deployment_strategy": "blue_green",
"rollback_triggers": [
"error_rate > 5%",
"latency_p95 > 2s",
"availability < 99%"
]
}
}
}
return dr_config
Automated rollback: Clear metrics-based triggers automatically rollback deployments that degrade service quality.
Disaster recovery summary: This comprehensive plan ensures business continuity with clear objectives, automated responses, and validated recovery procedures.
Service Mesh Summary¶
The service mesh architecture provides:
- Secure communication with mutual TLS and certificate management
- Traffic management through canary deployments and weighted routing
- Resilience patterns with circuit breakers and connection pooling
- Global reach with multi-region deployment and disaster recovery
Part 3: Advanced Scaling Strategies (20 minutes)¶
Predictive and Reactive Scaling¶
🗂️ File: src/session8/advanced_scaling.py - Intelligent scaling strategies
Understanding Predictive Scaling¶
Predictive scaling goes beyond reactive scaling by anticipating demand changes before they happen. This is crucial for agent systems where:
- Startup time matters - Agents need time to initialize models and connections
- Cost optimization - Proactive scaling prevents over-provisioning
- User experience - Prevents response time degradation during traffic spikes
Step 1: Essential Imports for Scaling Intelligence¶
from typing import Dict, List, Any, Optional
from datetime import datetime, timedelta
import asyncio
import json
Step 2: Predictive Scaling Engine Foundation¶
class PredictiveScalingEngine:
"""AI-powered predictive scaling for agent workloads"""
def __init__(self):
self.historical_metrics = []
self.scaling_models = {}
self.prediction_horizon = timedelta(hours=1)
Architecture principle: We maintain historical data to learn patterns and build prediction models with a 1-hour forecasting window.
Step 3: Comprehensive Metrics Collection¶
Effective scaling decisions require data from multiple dimensions. Let's build a comprehensive metrics collection system:
def collect_scaling_metrics(self) -> Dict[str, Any]:
"""Collect comprehensive metrics for scaling decisions"""
metrics = {
"timestamp": datetime.now().isoformat(),
Timestamp importance: Every metric point needs precise timing for pattern analysis and correlation.
Step 4: Resource Utilization Metrics¶
"resource_utilization": {
"cpu_percent": 65.5,
"memory_percent": 78.2,
"gpu_percent": 45.0,
"network_mbps": 125.3
},
Multi-resource monitoring: CPU, memory, GPU, and network - each can be a bottleneck for different agent workloads.
Step 5: Workload Pattern Analysis¶
"workload_patterns": {
"requests_per_second": 450,
"average_response_time": 0.85,
"queue_depth": 25,
"active_agents": 12
},
Performance indicators: These metrics directly correlate with user experience and scaling needs.
Step 6: Business and Cost Metrics¶
"business_metrics": {
"user_sessions": 1250,
"conversation_starts": 85,
"cost_per_hour": 12.50,
"sla_compliance": 99.2
},
Business alignment: Scaling decisions must balance performance with cost and business value.
Step 7: External Factor Integration¶
"external_factors": {
"time_of_day": datetime.now().hour,
"day_of_week": datetime.now().weekday(),
"seasonal_factor": self._get_seasonal_factor(),
"promotion_active": False
}
}
return metrics
Contextual awareness: External factors help predict demand patterns beyond just historical resource usage.
Step 8: Intelligent Prediction Algorithm¶
Now let's build the core prediction logic that analyzes patterns and forecasts scaling needs:
def predict_scaling_needs(self, prediction_window: int = 60) -> Dict[str, Any]:
"""Predict scaling needs for the next prediction window (minutes)"""
# Simplified ML prediction (in production, use proper ML models)
current_metrics = self.collect_scaling_metrics()
Prediction approach: We start with pattern-based predictions and can evolve to ML models as data accumulates.
Step 9: Time-Based Traffic Pattern Analysis¶
# Time-based patterns
hour = datetime.now().hour
if 9 <= hour <= 17: # Business hours
traffic_multiplier = 1.5
elif 18 <= hour <= 22: # Evening
traffic_multiplier = 1.2
else: # Night/early morning
traffic_multiplier = 0.7
Pattern recognition: Most agent workloads follow predictable daily patterns - this knowledge drives proactive scaling.
Step 10: Load Prediction Calculation¶
# Predict future load
predicted_rps = current_metrics["workload_patterns"]["requests_per_second"] * traffic_multiplier
predicted_cpu = current_metrics["resource_utilization"]["cpu_percent"] * traffic_multiplier
Prediction logic: We apply traffic multipliers to current metrics to forecast future demand.
Step 11: Capacity Requirement Calculation¶
# Calculate required capacity
current_agents = current_metrics["workload_patterns"]["active_agents"]
if predicted_cpu > 80:
required_agents = int(current_agents * 1.5)
elif predicted_cpu > 60:
required_agents = int(current_agents * 1.2)
elif predicted_cpu < 30:
required_agents = max(2, int(current_agents * 0.8))
else:
required_agents = current_agents
Scaling thresholds: Conservative scaling with safety margins - 80% CPU triggers aggressive scaling, 30% allows scale-down.
Step 12: Comprehensive Prediction Response¶
scaling_prediction = {
"prediction_timestamp": datetime.now().isoformat(),
"prediction_window_minutes": prediction_window,
"current_state": {
"agent_count": current_agents,
"cpu_utilization": current_metrics["resource_utilization"]["cpu_percent"],
"requests_per_second": current_metrics["workload_patterns"]["requests_per_second"]
},
Step 13: Predicted State and Confidence¶
"predicted_state": {
"agent_count_needed": required_agents,
"predicted_cpu": predicted_cpu,
"predicted_rps": predicted_rps,
"confidence": 0.85
},
Confidence scoring: 85% confidence indicates high reliability in our time-based patterns.
Step 14: Actionable Scaling Recommendations¶
"scaling_recommendation": {
"action": "scale_out" if required_agents > current_agents else "scale_in" if required_agents < current_agents else "maintain",
"target_replicas": required_agents,
"estimated_cost_impact": self._calculate_cost_impact(current_agents, required_agents),
"lead_time_minutes": 5
}
}
return scaling_prediction
Actionable insights: Clear recommendations with cost impact and timing help operations teams make informed decisions.
Step 15: Cost Impact Analysis¶
Every scaling decision has cost implications. Let's build transparent cost analysis:
def _calculate_cost_impact(self, current_agents: int, target_agents: int) -> Dict[str, float]:
"""Calculate estimated cost impact of scaling decision"""
cost_per_agent_per_hour = 0.50 # Simplified cost model
agent_difference = target_agents - current_agents
hourly_cost_change = agent_difference * cost_per_agent_per_hour
return {
"hourly_change": hourly_cost_change,
"daily_change": hourly_cost_change * 24,
"monthly_change": hourly_cost_change * 24 * 30
}
Cost transparency: Operations teams need clear cost impact data to make informed scaling decisions.
Step 16: Seasonal Factor Calculation¶
def _get_seasonal_factor(self) -> float:
"""Calculate seasonal traffic multiplier"""
month = datetime.now().month
if month in [11, 12]: # Holiday season
return 1.4
elif month in [6, 7, 8]: # Summer
return 0.9
elif month in [1, 2]: # Post-holiday
return 0.8
else:
return 1.0
Seasonal awareness: Different months have predictable traffic patterns that should influence scaling decisions.
Step 17: Cost-Optimized Scaling Architecture¶
Cost optimization is crucial for enterprise deployments. Let's build intelligent cost-aware scaling:
class CostOptimizedScaling:
"""Cost-aware scaling strategies"""
def __init__(self):
self.cost_budgets = {}
self.spot_instance_policies = {}
self.preemption_handlers = {}
Cost-first approach: Every scaling decision considers budget constraints and cost optimization opportunities.
Step 18: Dynamic Cost-Aware Scaling Policies¶
def create_cost_aware_scaling_policy(self, max_hourly_cost: float) -> Dict[str, Any]:
"""Create scaling policy that respects cost constraints"""
policy = {
Step 19: Cost Constraint Framework¶
"cost_constraints": {
"max_hourly_cost": max_hourly_cost,
"max_daily_cost": max_hourly_cost * 24,
"cost_alert_threshold": max_hourly_cost * 0.8
},
Budget management: Clear cost limits with early warning alerts prevent budget overruns.
Step 20: Adaptive Scaling Strategies¶
"scaling_strategies": [
{
"name": "cost_optimized_normal",
"condition": "cost_budget_remaining > 50%",
"scaling_factor": 1.0,
"instance_types": ["n1-standard-4", "n1-standard-2"],
"spot_instance_ratio": 0.7
},
Normal operations: Full scaling with premium instances when budget allows, 70% spot instances for cost savings.
Step 21: Cost-Constrained Scaling¶
{
"name": "cost_constrained",
"condition": "cost_budget_remaining <= 50% and cost_budget_remaining > 20%",
"scaling_factor": 0.8,
"instance_types": ["n1-standard-2", "e2-standard-2"],
"spot_instance_ratio": 0.9
},
Budget pressure: Reduced scaling (80%) with smaller instances and higher spot ratio (90%).
Step 22: Emergency Cost Management¶
{
"name": "emergency_cost_saving",
"condition": "cost_budget_remaining <= 20%",
"scaling_factor": 0.5,
"instance_types": ["e2-standard-2"],
"spot_instance_ratio": 1.0,
"aggressive_scale_down": True
}
],
Crisis mode: Severe scaling restrictions (50%), cheapest instances only, 100% spot instances.
Step 23: Spot Instance Management¶
"spot_instance_config": {
"enabled": True,
"max_spot_ratio": 0.9,
"preemption_handling": {
"drain_timeout": "120s",
"graceful_shutdown": True,
"workload_migration": True
}
}
}
return policy
Spot instance strategy: Up to 90% spot instances with graceful handling of preemptions to minimize disruption.
Module Summary¶
You've now mastered enterprise-scale architecture for Agno agent systems:
✅ Kubernetes-Native Architecture: Implemented custom resources, advanced HPA, and cluster autoscaling
✅ Multi-Tenant Platform: Built tenant isolation with resource quotas and fair scheduling
✅ Service Mesh Integration: Created Istio configuration with traffic management and security
✅ Global Deployment: Designed multi-region architecture with disaster recovery
✅ Predictive Scaling: Built AI-powered scaling with cost optimization strategies
Key Enterprise Architecture Patterns¶
Infrastructure as Code: All configurations are declarative YAML/JSON for reproducible deployments
Defense in Depth: Multiple layers of security from network policies to service mesh to RBAC
Observability First: Every component includes monitoring, metrics, and health checks
Cost Optimization: Automatic scaling adapts to budget constraints and uses spot instances
Disaster Recovery: Comprehensive backup, replication, and automated failover strategies
Next Steps¶
- Continue to Module C: Performance Optimization for caching and cost management
- Continue to Module D: Security & Compliance for enterprise security
- Return to Core: Session 8 Main
📝 Multiple Choice Test - Module B¶
Test your understanding of Enterprise Scaling & Architecture:
Question 1: What is the primary advantage of using Custom Resource Definitions (CRDs) in Kubernetes for agent management? A) They reduce CPU usage
B) They extend the Kubernetes API to support agent-specific configurations
C) They automatically scale pods
D) They provide built-in monitoring
Question 2: In the multi-tenant architecture, what is the purpose of ResourceQuota objects? A) To improve network performance
B) To prevent any single tenant from consuming excessive cluster resources
C) To enable automatic scaling
D) To provide load balancing
Question 3: What traffic distribution does the canary deployment configuration implement by default? A) 50% stable, 50% canary
B) 80% stable, 20% canary
C) 90% stable, 10% canary
D) 95% stable, 5% canary
Question 4: In the disaster recovery plan, what are the RPO and RTO targets? A) RPO: 30m, RTO: 1h
B) RPO: 1h, RTO: 30m
C) RPO: 24h, RTO: 4h
D) RPO: 15m, RTO: 1h
Question 5: What happens when the cost budget remaining drops to 20% or below in the cost-aware scaling policy? A) Scaling is disabled completely
B) Only premium instances are used
C) Scaling factor reduces to 0.5 with 100% spot instances
D) The system sends alerts but continues normal scaling
🗂️ Source Files for Module B:
src/session8/k8s_native_architecture.py- Kubernetes-native agent orchestrationsrc/session8/service_mesh_architecture.py- Istio service mesh integrationsrc/session8/advanced_scaling.py- Predictive and cost-aware scaling strategies