5 분 소요

이번 실습에서는 EKS 기반 vLLM 실습에서 생성한 모든 리소스를 정리하는 절차를 단계별로 설명한다. 불필요한 AWS 요금이 발생하지 않도록 다음 단계를 수행한다.

1. 사전 요구 사항

AWS CLI에 접근할 수 있고, 클러스터에 맞게 kubectl이 구성되어 있는지 확인한다.

2. 환경변수 설정

export AWS_REGION=us-west-2
export CLUSTER_NAME=ai-infra-summit-test-cluster
export MONITORING_NAMESPACE=monitoring

# Verify current context
kubectl config current-context

3. 1단계: vLLM 애플리케이션 리소스 삭제

vLLM과 관련된 모든 Kubernetes 리소스를 제거한다.

echo "=== Deleting vLLM application resources ==="

# Delete HPA
kubectl delete hpa vllm-hpa --ignore-not-found=true

# Delete vLLM specific resources (no ingress or network policies created in this workshop)
kubectl delete service vllm-service --ignore-not-found=true
kubectl delete deployment vllm-deployment --ignore-not-found=true
kubectl delete configmap vllm-shared-config --ignore-not-found=true
kubectl delete pvc s3-model-cache-pvc --ignore-not-found=true
kubectl delete pv s3-model-cache-pv --ignore-not-found=true

echo "vLLM application resources deleted!"

4. 2단계: 모니터링 스택 제거

Prometheus, Grafana 및 CloudWatch 리소스를 정리한다.

echo "=== Removing monitoring stack ==="

# Uninstall Helm releases
helm uninstall grafana -n $MONITORING_NAMESPACE --ignore-not-found=true
helm uninstall prometheus -n $MONITORING_NAMESPACE --ignore-not-found=true

# Delete monitoring namespace
kubectl delete namespace $MONITORING_NAMESPACE --ignore-not-found=true

# Remove CloudWatch Container Insights
kubectl delete namespace amazon-cloudwatch --ignore-not-found=true

# Delete CloudWatch dashboard
aws cloudwatch delete-dashboards \
  --dashboard-names "vLLM-EKS-Monitoring" \
  --region $AWS_REGION \
  --no-cli-pager 2>/dev/null || true

# Delete CloudWatch log groups
aws logs delete-log-group \
  --log-group-name "/aws/eks/$CLUSTER_NAME/vllm" \
  --region $AWS_REGION \
  --no-cli-pager 2>/dev/null || true

echo "Monitoring stack removed!"

5. 3단계: AWS Load Balancer Controller 제거

AWS Load Balancer Controller를 제거한다.

echo "=== Removing AWS Load Balancer Controller ==="

# Uninstall the controller
helm uninstall aws-load-balancer-controller -n kube-system --ignore-not-found=true

# Delete the IAM service account
eksctl delete iamserviceaccount \
  --cluster=$CLUSTER_NAME \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --region=$AWS_REGION \
  --ignore-not-found=true

# Delete the IAM policy
aws iam delete-policy \
  --policy-arn "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/AWSLoadBalancerControllerIAMPolicy" \
  --region $AWS_REGION \
  --no-cli-pager 2>/dev/null || true

echo "AWS Load Balancer Controller removed!"

6. 4단계: S3 모델 캐시 버킷 삭제

모델 캐시용으로 생성한 S3 버킷을 제거한다.

echo "=== Deleting S3 model cache bucket ==="

# Get the bucket name
BUCKET_NAME="ai-infra-summit-vllm-models-cache-$(aws sts get-caller-identity --query Account --output text)"

# Delete all objects in the bucket first
aws s3 rm s3://$BUCKET_NAME --recursive --region $AWS_REGION 2>/dev/null || echo "Bucket not found or already empty"

# Delete the bucket
aws s3api delete-bucket --bucket $BUCKET_NAME --region $AWS_REGION 2>/dev/null || echo "Bucket not found or already deleted"

echo "S3 model cache bucket deleted!"

7. 5단계: EKS 클러스터 삭제

전체 EKS 클러스터와 관련 리소스를 제거한다.

echo "=== Deleting EKS cluster ==="

# Delete the cluster (this will take 10-15 minutes)
eksctl delete cluster --name $CLUSTER_NAME --region $AWS_REGION --wait

echo "EKS cluster deleted!"

8. 6단계: 정리 결과 확인

모든 리소스가 제대로 정리되었는지 확인한다.

echo "=== Verifying cleanup ==="

# Check EKS clusters
echo "Remaining EKS clusters:"
aws eks list-clusters --region $AWS_REGION --query 'clusters[]' --output table

# ECR repositories were not created in this workshop, so skipping ECR cleanup

# Check Load Balancers
echo -e "\nRemaining Application Load Balancers:"
aws elbv2 describe-load-balancers \
  --region $AWS_REGION \
  --query 'LoadBalancers[?contains(LoadBalancerName, `vllm`)].LoadBalancerName' \
  --output table 2>/dev/null || echo "No vLLM ALBs found"

# Check CloudWatch log groups
echo -e "\nRemaining CloudWatch log groups:"
aws logs describe-log-groups \
  --log-group-name-prefix "/aws/eks/$CLUSTER_NAME" \
  --region $AWS_REGION \
  --query 'logGroups[*].logGroupName' \
  --output table 2>/dev/null || echo "No log groups found"

# Check IAM policies
echo -e "\nChecking for leftover IAM policies:"
aws iam list-policies \
  --query 'Policies[?contains(PolicyName, `AWSLoadBalancerController`)].PolicyName' \
  --output table 2>/dev/null || echo "No related policies found"

echo -e "\n=== Cleanup verification completed ==="

9. 7단계: 로컬 파일 정리

로컬 구성 파일과 스크립트를 제거한다.

echo "=== Cleaning up local files ==="

# Remove generated YAML files created during workshop
rm -f vllm-configmap.yaml
rm -f vllm-storage.yaml
rm -f vllm-deployment.yaml
rm -f vllm-service.yaml
rm -f vllm-hpa.yaml
rm -f prometheus-values.yaml
rm -f grafana-values.yaml

# Remove Python test scripts
rm -f test-vllm-pod.py

# Remove any backup files
rm -f *.bak

echo "Local files cleaned up!"

10. 8단계: kubectl 컨텍스트 업데이트

삭제한 클러스터를 kubectl 구성에서 제거한다.

echo "=== Updating kubectl context ==="

# Remove cluster context from kubectl config
kubectl config get-contexts
kubectl config delete-context arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "Context not found"

# Remove cluster and user entries
kubectl config unset clusters.arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "Cluster not found in config"
kubectl config unset users.arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "User not found in config"

echo "kubectl context updated!"

11. 비용 최적화 확인

예상하지 않은 리소스가 계속 실행 중이지 않은지 확인한다.

echo "=== Final cost optimization check ==="

# Check for running EC2 instances in the region
echo "EC2 instances (should be empty after cluster deletion):"
aws ec2 describe-instances \
  --region $AWS_REGION \
  --query 'Reservations[*].Instances[?State.Name!=`terminated`].[InstanceId,InstanceType,State.Name,Tags[?Key==`Name`].Value|[0]]' \
  --output table

# Check for unused EBS volumes
echo -e "\nEBS volumes (check for unused volumes):"
aws ec2 describe-volumes \
  --region $AWS_REGION \
  --query 'Volumes[?State==`available`].[VolumeId,Size,VolumeType,CreateTime]' \
  --output table

# Check for unused Elastic IPs
echo -e "\nElastic IPs (should be empty or associated):"
aws ec2 describe-addresses \
  --region $AWS_REGION \
  --query 'Addresses[?!AssociationId].[PublicIp,AllocationId]' \
  --output table

# Check for any remaining NAT Gateways
echo -e "\nNAT Gateways (should be deleted with cluster):"
aws ec2 describe-nat-gateways \
  --region $AWS_REGION \
  --query 'NatGateways[?State==`available`].[NatGatewayId,VpcId,State]' \
  --output table

echo -e "\n=== Cost optimization check completed ==="
echo "Review the above output for any unexpected running resources."

12. 정리 문제 해결

정리 과정에서 문제가 발생하면 다음 명령을 사용한다.

echo "=== Troubleshooting cleanup issues ==="

# Force delete stuck namespaces
kubectl get namespace | grep Terminating
# If namespaces are stuck in Terminating state:
# kubectl patch namespace <namespace-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

# Check for protection on resources
aws ec2 describe-instances \
  --region $AWS_REGION \
  --query 'Reservations[*].Instances[?State.Name!=`terminated`].[InstanceId,DisableApiTermination]' \
  --output table

# Note: CloudFormation stacks for the workshop infrastructure should not be deleted
# These are managed by the workshop environment and contain VPC, EC2, and other base resources

echo "Use the above commands to troubleshoot any cleanup issues."

13. 정리 완료 요약

echo "================================================================"
echo "                    CLEANUP COMPLETED"
echo "================================================================"
echo "Thank you for completing the vLLM on EKS lab!"
echo "================================================================"

14. 정리 후 다음 단계

1) AWS 청구 내역 검토: 앞으로 며칠 동안 AWS 청구서를 확인해 모든 리소스가 제대로 종료되었는지 확인한다. 2) 문서화: 나중에 참고할 수 있도록 실습에서 얻은 성능 지표나 학습 내용을 저장한다. 3) 피드백: 실습 경험에 대한 피드백 제공을 고려한다. 4) 추가 학습: 다른 AWS AI/ML 서비스 또는 고급 Kubernetes 패턴을 살펴본다.

이로써 EKS 기반 vLLM 실습의 해체 절차를 완료했다. 이제 모든 리소스가 정리되어 더 이상 요금이 발생하지 않아야 한다.

댓글남기기