the problem is that k8s is a half arsed implementation of borg/tupperware.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.
> The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.)
Hmm, I thought you could specify larger node cidr at cluster creation time so you can have >254 ip addresses on each node? The default is /24 but you should be able to specify a larger block, unless you're on GKE (not sure if you can do that on GKE).
The network provides simple and easily controllable ip allocation. You don't need to do any of it, just offload it to AWS/GC/Azure.
Slam another network card/VLAN, give it a seperate ip range from the control network, and everything is handled for you, quickly, securely and efficiently.
Having supported a large team migrate from a custom system based on fleet (god know why they were allowed to waste two years on that) jump to k8s (around 1.2) it struck me how many features it actually lacked (like pod drain, although thats fixed now.)
The worst offender in K8s world is the networking scheme. Its utterly batshit, and clearly designed by people who've never had to support anything out of hours. (statically assign 254 IPs to each host, or fuck with the routing table.) Combined with how extra ordinarily chatty the backplane is, makes it expensive and annoying to run at scale.
Plus you have to continually upgrade, migrate and generally tweak the thing, every 6 months. Unless you have a big ops team, your frankly better off with lambdas.