Skip to main content

cluster-autoscaler


cluster-autoscalerλŠ” λ””ν”Œλ‘œμ΄λ¨ΌνŠΈλ‘œ λ°°ν¬λ©λ‹ˆλ‹€. Leader Election을 μ‚¬μš©ν•©λ‹ˆλ‹€.

AWS

Prerequisites

ASG(Auto Scaling Group)

  • Auto Scaling 그룹에 μ•„λž˜ νƒœκ·Έκ°€ μžˆμ–΄μ•Όν•©λ‹ˆλ‹€.
    • k8s.io/cluster-autoscaler/<cluster-name>: owned
    • k8s.io/cluster-autoscaler/enabled: true
  • μ‚¬μš©μž 데이터λ₯Ό μ‚¬μš©ν•˜μ—¬ Control plane node μ—°κ²°, node label 섀정을 μžλ™ν™” ν•΄μ€λ‹ˆλ‹€.
eks-asg-example.sh
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=xxx
API_SERVER_URL=xxx
K8S_CLUSTER_DNS_IP=xxx
/etc/eks/bootstrap.sh hhk-eks --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,alpha.eksctl.io/cluster-name=hhk-eks,alpha.eksctl.io/nodegroup-name=worker-m5-large-ng,role=worker,eks.amazonaws.com/nodegroup-image=ami-0eb873de16468e55e,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=worker-m5-large-ng,eks.amazonaws.com/sourceLaunchTemplateId=lt-0676b3adb0ce8d3cc' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP

--//--

Installation

mkdir -p aws-cluster-autoscaler/base/patches
wget https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml \
-O aws-cluster-autoscaler/base/cluster-autoscaler-autodiscover.yaml
aws-cluster-autoscaler/base/patches/annotations.yaml
- op: replace
path: /spec/template/metadata/annotations
value:
prometheus.io/scrape: 'true'
prometheus.io/port: '8085'
cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'
  • cluster-autoscaler.kubernetes.io/safe-to-evict: 'false': 제거 λΉ„μš©μ΄ 많이 λ“œλŠ” podλΌλŠ” ν‘œμ‹œλ‘œ, ν•΄λ‹Ή pod이 node에 μžˆλŠ” 경우 nodeλŠ” scalie in λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.
aws-cluster-autoscaler/base/patches/command.yaml
- op: replace
path: /spec/template/spec/containers/0/command
value:
- ./cluster-autoscaler
- --v=4
- --stderrthreshold=info
- --cloud-provider=aws
- --skip-nodes-with-local-storage=false
- --expander=least-waste
- --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<cluster-name>
- --balance-similar-node-groups
- --skip-nodes-with-system-pods=false
aws-cluster-autoscaler/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster-autoscaler-autodiscover.yaml

patches:
- path: patches/annotations.yaml
target:
kind: Deployment
- path: patches/command.yaml
target:
kind: Deployment

mkdir -p aws-cluster-autoscaler/overlays/iam/patches
aws-cluster-autoscaler/overlays/iam/patches/role.yaml
- op: add
path: /metadata/annotations
value:
eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<AmazonEKSClusterAutoscalerRole>
aws-cluster-autoscaler/overlays/iam/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base

patches:
- path: patches/role.yaml
target:
kind: ServiceAccount
kustomize build aws-cluster-autoscaler/overlays/iam | kubectl apply -f -
warning

On-premise ν™˜κ²½μ—μ„œ μ‚¬μš©ν•˜λŠ” 경우 μ•„λž˜μ™€ 같은 인증 방식을 μ‚¬μš©ν•  수 μžˆμŠ΅λ‹ˆλ‹€. (λ³΄μ•ˆ μœ„ν˜‘ μš”μ†Œκ°€ 될 수 있음)

mkdir -p aws-cluster-autoscaler/overlays/credentials/patches
aws-cluster-autoscaler/overlays/credentials/credentials.yaml
apiVersion: v1
kind: Secret
metadata:
name: aws-secret
namespace: kube-system
type: Opaque
data:
aws_access_key_id: <BASE64_OF_YOUR_AWS_ACCESS_KEY_ID>
aws_secret_access_key: <BASE64_OF_YOUR_AWS_SECRET_ACCESS_KEY>
aws-cluster-autoscaler/overlays/credentials/patches/env.yaml
- op: add
path: /spec/template/spec/containers/0/env
value:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-secret
key: aws_access_key_id
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-secret
key: aws_secret_access_key
- name: AWS_REGION
value: <REGION>
aws-cluster-autoscaler/overlays/credentials/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- credentials.yaml
- ../../base

patches:
- path: patches/env.yaml
target:
kind: Deployment
aws-cluster-autoscaler
β”œβ”€β”€ base/
β”‚ β”œβ”€β”€ cluster-autoscaler-autodiscover.yaml
β”‚ β”œβ”€β”€ kustomization.yaml
β”‚ └── patches/
β”‚ β”œβ”€β”€ annotations.yaml
β”‚ └── command.yaml
└── overlays/
β”œβ”€β”€ credentials/
β”‚ β”œβ”€β”€ credentials.yaml
β”‚ β”œβ”€β”€ kustomization.yaml
β”‚ └── patches/
β”‚ └── env.yaml
└── iam/
β”œβ”€β”€ cluster-autoscaler-policy.json
β”œβ”€β”€ kustomization.yaml
└── patches/
└── role.yaml

Troubleshooting

kubectl logs -n kube-system -f deployment.apps/cluster-autoscaler

Reference