Load balancing for ML infrastructure - A practical approach

I lead MLOps at SAP's AI Center of Excellence (AICOE), an internal IT organization focused on enterprise-wide ML solutions.

Our team builds and maintains AI/ML infrastructure that serves 100,000+ SAP employees worldwide. We support various use cases including intelligent ticket routing, OCR systems, document review automation, and knowledge graphs.

These services require reliable and cost-effective load balancing within our corporate network.

A primer on load balancers

Think of load balancers as traffic controllers at a busy intersection. Just as a traffic controller directs vehicles to different lanes to prevent congestion, load balancers distribute incoming network traffic across multiple servers to ensure smooth operations.

Some controllers only know about roads and intersections (Layer 4), while others understand the destination written on each vehicle and can make smarter routing decisions (Layer 7).

At the heart of our solution is Kubernetes Ingress - a powerful abstraction that enables sophisticated HTTP routing based on hosts or URLs.

While Ingress defines what we want to achieve with traffic routing, we need an ingress controller to implement these rules. Think of Ingress as the traffic regulation manual, and the ingress controller as the actual traffic officer following those rules.

The NGINX ingress controller, which we chose for our ML infrastructure, is deployed as a set of pods that continuously watch the Kubernetes API server for changes to Ingress resources.

When we deploy a new ML model version or update routing rules, the controller automatically adjusts its configuration to match our requirements. This dynamic nature is particularly valuable in our ML environment, where routing needs evolve as we deploy new model versions or adjust traffic patterns for different services.

Our ML services demanded specific capabilities from the ingress layer:

• Content-based routing for directing traffic to different model versions
• Fine-grained rate limiting to protect our inference endpoints
• Flexible timeout configurations for varying inference times
• Header-based routing for gradual rollouts of new models

The NGINX ingress controller excels at these requirements while maintaining tight integration with Kubernetes. We deploy it as a Kubernetes Deployment in each namespace, giving us the flexibility to tune configurations specifically for each environment's needs.

One significant benefit we discovered was the ability to use internal NGINX ingress controllers for different namespaces. This approach simplified our network architecture while maintaining strict isolation between environments, all within our corporate network.

The choice of load balancer isn't just about directing traffic—it's about finding the right balance between control, cost, and complexity.

In the Azure ecosystem, we have two native traffic controllers. The Azure Load Balancer is like a basic traffic cop, directing traffic based on simple rules like which road to take.

The Azure Application Gateway is more sophisticated, reading the destination on each vehicle and knowing all the shortcuts, but it comes with a premium price tag.

Then there's the NGINX ingress controller approach – it's like building your own traffic control system inside your private compound.

You have full control over the rules, and since you're already paying for the compound (your Kubernetes cluster), you only need to assign some of your existing staff (compute resources) to handle the traffic management.

How much do they cost?

The economics of load balancing aren't always obvious at first glance. When we started, Azure Application Gateway seemed like a reasonable choice for our Layer 7 routing needs.

However, as our platform grew to serve 20+ ML services, the costs began to multiply. Running separate gateways for our production services, testing environment, and monitoring stack would have required multiple Application Gateway instances.

Azure's basic Load Balancer, while more cost-effective, would have required additional components to handle our application-layer routing needs. These extras, like Azure Front Door for advanced routing capabilities, would have increased the total cost of the solution.

In contrast, our NGINX ingress controller approach leverages resources we already have in our AKS cluster. Each namespace needs just a few small pods – about half a CPU core and 256MB of memory each.

This approach allows us to achieve advanced Layer 7 routing capabilities while utilizing our existing compute resources efficiently.

Our service architecture: A three-layered approach

Our ML platform evolved into a natural hierarchy, each layer with its own distinct needs and characteristics.

At the foundation lies our Infrastructure Layer, housing the essential services that keep everything running smoothly. Here, Prometheus and Grafana work tirelessly to monitor our system's health, while our EFK stack processes all logs coming in from our services. We also leverage Sloth for SLO monitoring and alerting, integrated with Prometheus and Grafana to provide comprehensive SRE capabilities. Our Elasticsearch deployment not only supports logging but also powers search functionality across our services.

These services form the backbone of our operations – they're like the building's maintenance systems, crucial but meant for internal access only.

Moving up to the Platform Layer, we find the tools our ML engineers interact with daily. This layer serves as the engine room of our ML operations, where Airflow orchestrates complex workflows across six components.

At the top sits our ML Services Layer, where our models serve real business value. This layer hosts more than 20 distinct model inference services, each designed for specific use cases. Most of our services are built using FastAPI with Celery for asynchronous task processing, while some specialized use cases leverage Torchserve for model serving. All services are consistently deployed and managed using Helm charts, ensuring standardized deployment patterns across our infrastructure.

Our document processing services tackle the heavy lifting of parsing and analyzing documents. The real-time classification services power instant decision-making systems where every millisecond counts.

Knowledge graph services navigate complex relationships in our data, while batch processing services handle large-scale data transformations. Each service type brings its own requirements for request handling, timeout configurations, and resource allocation.

We also maintain critical services outside the Kubernetes cluster, including Jenkins (CI/CD), internal ML Model Registry, and ML Experimentation Framework on dedicated resources.

Understanding our requirements

Our load balancing needs were specific and non-negotiable:

• Internal traffic only (within corporate network)
• Path-based routing for model versions
• Different rate limits per service
• Namespace isolation for prod/test environments
• Cost-effective for 20+ services

It's worth noting that our approach differs from typical public cloud patterns. In public clouds, organizations typically follow several common patterns:

1. NLB + Ingress Controller: Many organizations use a Network Load Balancer (NLB) in front of their NGINX ingress controllers. This provides stable entry points and fixed IPs, crucial for public internet traffic.

2. Cloud-Native API Gateways: Some teams opt for managed API gateways (like Azure API Management or AWS API Gateway) in front of their Kubernetes services, especially when they need features like API key management or request transformation.

3. Application Load Balancer (ALB) with Ingress: AWS users often leverage ALB Ingress Controllers, which automatically provision and configure Application Load Balancers for their Kubernetes services.

Our setup is simpler than these patterns. Since we operate entirely within SAP's corporate network, we can use internal NGINX ingress controllers directly.

This eliminates an entire layer of infrastructure while still maintaining the isolation and control we need through namespace-specific ingress controllers.

The available options

Let's compare our options across key dimensions that mattered for our ML infrastructure:

While each option had its strengths, Internal NGINX Ingress Controllers aligned best with our specific needs, particularly for cost efficiency at scale and fine-grained traffic control within our corporate network.

Our solution: A practical approach to traffic management

Instead of opting for the obvious choice of Azure's managed services, we took a more pragmatic approach. A key advantage of this setup is that our ingress controller acts as an additional layer of routing and control behind the load balancer, giving us fine-grained control over traffic patterns specific to ML workloads.

# Source: aks-cluster/main.tf
module "namespace-mlh-prod" {
  source              = "./tf-mlh-namespace"
  name                = "mlh-prod"
  loadbalancer_ip     = "<internal-ip>"
  priority_class_name = "prod-priority"
  priority_class_value = 1000
}

Each namespace gets its own ingress controller with specific configurations:

# Source: aks-cluster/tf-mlh-namespace/main.tf
args = [
"--configmap=$(POD_NAMESPACE)/nginx-configuration",
"--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services",
"--udp-services-configmap=$(POD_NAMESPACE)/udp-services",
"--publish-service=$(POD_NAMESPACE)/ingress-nginx",
"--annotations-prefix=nginx.ingress.kubernetes.io",
"--watch-namespace=${var.name}",
"--enable-ssl-chain-completion=false",
"--default-ssl-certificate=$(POD_NAMESPACE)/thor-tls-certs"
]

Performance tuning and security settings:

# Source: aks-cluster/tf-mlh-namespace/main.tf
resource "kubernetes_config_map" "nginx_configuration" {
  metadata {
    name      = "nginx-configuration"
    namespace = kubernetes_namespace.nginx.metadata.0.name
  }
  data = {
    "proxy-body-size"          = "50m"
    "proxy-read-timeout"       = "300"
    "proxy-send-timeout"       = "300"
    "client-max-body-size"     = "50m"
    "use-forwarded-headers"    = "true"
    "compute-full-forwarded-for" = "true"
    "proxy-buffer-size"        = "16k"
  }
}

Beyond the numbers: Real impact

The impact of our decision extends beyond resource optimization. Our implementation provides several key benefits:

• Source IP preservation: Unlike other solutions, we maintain the original source IP addresses, which is crucial for our audit and monitoring requirements.
• Long-lived connections: Our setup supports long-running TCP connections, essential for services like our real-time model inference endpoints.
• Zonal isolation: We can route traffic to specific availability zones when needed, which helps with resource optimization and compliance requirements.
• Flexible backend security: Security is controlled at the pod level through Kubernetes network policies, giving us precise control over traffic flows.

When our document processing service needs to test new model versions, we can easily split traffic based on request headers – something that would have required additional infrastructure with Azure's solutions. Our ticket routing system demands lightning-fast responses, while document processing needs time to handle larger files. With NGINX, we can fine-tune these parameters for each service without adding complexity or resource overhead.

Challenges and trade-offs

While the NGINX ingress controller served our needs well, it wasn't without its challenges. The most significant trade-off was the increased operational responsibility.

Unlike Azure's managed services, we needed to handle upgrades, configuration management, and troubleshooting ourselves.

One particularly memorable incident occurred during a routine SSL certificate update. We missed updating one of our ingress controllers during the staged rollout, which resulted in intermittent SSL errors for our document processing services in that namespace.

While the fix was straightforward (updating the certificate in the Kubernetes secret), the incident highlighted the importance of maintaining consistent configurations across all ingress controllers.

We also learned that scaling the ingress controller requires careful monitoring and proactive adjustment. During a period of high document processing load, when multiple teams were uploading large batches of documents simultaneously, we noticed increased latency in request routing.

Investigation revealed resource contention on the ingress controller pods - they were hitting CPU limits due to the high volume of concurrent connections and SSL termination overhead. We had to adjust our resource allocations and implement better monitoring for ingress controller metrics.

These challenges reinforced the need for:

• Robust automation for configuration updates
• Comprehensive monitoring of the ingress layer
• Clear procedures for certificate management
• Regular capacity planning based on usage patterns

Conclusion

Choosing NGINX ingress controllers over Azure's native load balancing solutions proved to be the right decision for our use case. The solution gave us the control and flexibility we needed while maintaining efficient resource utilization.

Most importantly, it integrated seamlessly with our existing infrastructure and development workflows.

This decision reinforced an important lesson in infrastructure design: understanding your specific requirements and constraints is more valuable than following conventional choices.

For our internal enterprise ML services, NGINX ingress controllers delivered the precise combination of features we needed – fine-grained traffic control, resource efficiency, and operational simplicity.