Skip to content

Session

Run your agents in Kubernetes: Build once, deploy anywhere. But really?

with Michal Salanci

About This Session

You know that - A demo works on your laptop, then you deploy it elswere and it's not. Migrating a Kubernetes workloads from one platform to another is not just flipping the switch. I created the LLM app utilizing a small quantized open model (~4B, CPU-only, serving inside the cluster) running it on a local kind cluster on my M4 MacBook. However, once moved to the AWS EKS, the pod died with 'exec format error'. The fix exposed the next failure, and the next... a liveness probe killing the model server mid-load, ':latest' meaning different images on different clusters, kubelet garbage-collecting the multi-gigabyte model weights, a hardcoded DNS suffix, a NetworkPolicy silently doing nothing. Let's find out what lies between "works on my laptop" and "runs anywhere", because "Build once, deploy anywhere" is a claim you must earn.

Topics

  • AI Models
  • AWS
  • Containers
  • Docker
  • Large Language Models (LLMs)
  • Migration