[워크삽] AWS 트레니엄 워크삽(14)-해체 및 정리
이번 실습에서는 EKS 기반 vLLM 실습에서 생성한 모든 리소스를 정리하는 절차를 단계별로 설명한다. 불필요한 AWS 요금이 발생하지 않도록 다음 단계를 수행한다.
1. 사전 요구 사항
AWS CLI에 접근할 수 있고, 클러스터에 맞게 kubectl이 구성되어 있는지 확인한다.
2. 환경변수 설정
export AWS_REGION=us-west-2
export CLUSTER_NAME=ai-infra-summit-test-cluster
export MONITORING_NAMESPACE=monitoring
# Verify current context
kubectl config current-context
3. 1단계: vLLM 애플리케이션 리소스 삭제
vLLM과 관련된 모든 Kubernetes 리소스를 제거한다.
echo "=== Deleting vLLM application resources ==="
# Delete HPA
kubectl delete hpa vllm-hpa --ignore-not-found=true
# Delete vLLM specific resources (no ingress or network policies created in this workshop)
kubectl delete service vllm-service --ignore-not-found=true
kubectl delete deployment vllm-deployment --ignore-not-found=true
kubectl delete configmap vllm-shared-config --ignore-not-found=true
kubectl delete pvc s3-model-cache-pvc --ignore-not-found=true
kubectl delete pv s3-model-cache-pv --ignore-not-found=true
echo "vLLM application resources deleted!"
4. 2단계: 모니터링 스택 제거
Prometheus, Grafana 및 CloudWatch 리소스를 정리한다.
echo "=== Removing monitoring stack ==="
# Uninstall Helm releases
helm uninstall grafana -n $MONITORING_NAMESPACE --ignore-not-found=true
helm uninstall prometheus -n $MONITORING_NAMESPACE --ignore-not-found=true
# Delete monitoring namespace
kubectl delete namespace $MONITORING_NAMESPACE --ignore-not-found=true
# Remove CloudWatch Container Insights
kubectl delete namespace amazon-cloudwatch --ignore-not-found=true
# Delete CloudWatch dashboard
aws cloudwatch delete-dashboards \
--dashboard-names "vLLM-EKS-Monitoring" \
--region $AWS_REGION \
--no-cli-pager 2>/dev/null || true
# Delete CloudWatch log groups
aws logs delete-log-group \
--log-group-name "/aws/eks/$CLUSTER_NAME/vllm" \
--region $AWS_REGION \
--no-cli-pager 2>/dev/null || true
echo "Monitoring stack removed!"
5. 3단계: AWS Load Balancer Controller 제거
AWS Load Balancer Controller를 제거한다.
echo "=== Removing AWS Load Balancer Controller ==="
# Uninstall the controller
helm uninstall aws-load-balancer-controller -n kube-system --ignore-not-found=true
# Delete the IAM service account
eksctl delete iamserviceaccount \
--cluster=$CLUSTER_NAME \
--namespace=kube-system \
--name=aws-load-balancer-controller \
--region=$AWS_REGION \
--ignore-not-found=true
# Delete the IAM policy
aws iam delete-policy \
--policy-arn "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):policy/AWSLoadBalancerControllerIAMPolicy" \
--region $AWS_REGION \
--no-cli-pager 2>/dev/null || true
echo "AWS Load Balancer Controller removed!"
6. 4단계: S3 모델 캐시 버킷 삭제
모델 캐시용으로 생성한 S3 버킷을 제거한다.
echo "=== Deleting S3 model cache bucket ==="
# Get the bucket name
BUCKET_NAME="ai-infra-summit-vllm-models-cache-$(aws sts get-caller-identity --query Account --output text)"
# Delete all objects in the bucket first
aws s3 rm s3://$BUCKET_NAME --recursive --region $AWS_REGION 2>/dev/null || echo "Bucket not found or already empty"
# Delete the bucket
aws s3api delete-bucket --bucket $BUCKET_NAME --region $AWS_REGION 2>/dev/null || echo "Bucket not found or already deleted"
echo "S3 model cache bucket deleted!"
7. 5단계: EKS 클러스터 삭제
전체 EKS 클러스터와 관련 리소스를 제거한다.
echo "=== Deleting EKS cluster ==="
# Delete the cluster (this will take 10-15 minutes)
eksctl delete cluster --name $CLUSTER_NAME --region $AWS_REGION --wait
echo "EKS cluster deleted!"
8. 6단계: 정리 결과 확인
모든 리소스가 제대로 정리되었는지 확인한다.
echo "=== Verifying cleanup ==="
# Check EKS clusters
echo "Remaining EKS clusters:"
aws eks list-clusters --region $AWS_REGION --query 'clusters[]' --output table
# ECR repositories were not created in this workshop, so skipping ECR cleanup
# Check Load Balancers
echo -e "\nRemaining Application Load Balancers:"
aws elbv2 describe-load-balancers \
--region $AWS_REGION \
--query 'LoadBalancers[?contains(LoadBalancerName, `vllm`)].LoadBalancerName' \
--output table 2>/dev/null || echo "No vLLM ALBs found"
# Check CloudWatch log groups
echo -e "\nRemaining CloudWatch log groups:"
aws logs describe-log-groups \
--log-group-name-prefix "/aws/eks/$CLUSTER_NAME" \
--region $AWS_REGION \
--query 'logGroups[*].logGroupName' \
--output table 2>/dev/null || echo "No log groups found"
# Check IAM policies
echo -e "\nChecking for leftover IAM policies:"
aws iam list-policies \
--query 'Policies[?contains(PolicyName, `AWSLoadBalancerController`)].PolicyName' \
--output table 2>/dev/null || echo "No related policies found"
echo -e "\n=== Cleanup verification completed ==="
9. 7단계: 로컬 파일 정리
로컬 구성 파일과 스크립트를 제거한다.
echo "=== Cleaning up local files ==="
# Remove generated YAML files created during workshop
rm -f vllm-configmap.yaml
rm -f vllm-storage.yaml
rm -f vllm-deployment.yaml
rm -f vllm-service.yaml
rm -f vllm-hpa.yaml
rm -f prometheus-values.yaml
rm -f grafana-values.yaml
# Remove Python test scripts
rm -f test-vllm-pod.py
# Remove any backup files
rm -f *.bak
echo "Local files cleaned up!"
10. 8단계: kubectl 컨텍스트 업데이트
삭제한 클러스터를 kubectl 구성에서 제거한다.
echo "=== Updating kubectl context ==="
# Remove cluster context from kubectl config
kubectl config get-contexts
kubectl config delete-context arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "Context not found"
# Remove cluster and user entries
kubectl config unset clusters.arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "Cluster not found in config"
kubectl config unset users.arn:aws:eks:$AWS_REGION:$(aws sts get-caller-identity --query Account --output text):cluster/$CLUSTER_NAME 2>/dev/null || echo "User not found in config"
echo "kubectl context updated!"
11. 비용 최적화 확인
예상하지 않은 리소스가 계속 실행 중이지 않은지 확인한다.
echo "=== Final cost optimization check ==="
# Check for running EC2 instances in the region
echo "EC2 instances (should be empty after cluster deletion):"
aws ec2 describe-instances \
--region $AWS_REGION \
--query 'Reservations[*].Instances[?State.Name!=`terminated`].[InstanceId,InstanceType,State.Name,Tags[?Key==`Name`].Value|[0]]' \
--output table
# Check for unused EBS volumes
echo -e "\nEBS volumes (check for unused volumes):"
aws ec2 describe-volumes \
--region $AWS_REGION \
--query 'Volumes[?State==`available`].[VolumeId,Size,VolumeType,CreateTime]' \
--output table
# Check for unused Elastic IPs
echo -e "\nElastic IPs (should be empty or associated):"
aws ec2 describe-addresses \
--region $AWS_REGION \
--query 'Addresses[?!AssociationId].[PublicIp,AllocationId]' \
--output table
# Check for any remaining NAT Gateways
echo -e "\nNAT Gateways (should be deleted with cluster):"
aws ec2 describe-nat-gateways \
--region $AWS_REGION \
--query 'NatGateways[?State==`available`].[NatGatewayId,VpcId,State]' \
--output table
echo -e "\n=== Cost optimization check completed ==="
echo "Review the above output for any unexpected running resources."
12. 정리 문제 해결
정리 과정에서 문제가 발생하면 다음 명령을 사용한다.
echo "=== Troubleshooting cleanup issues ==="
# Force delete stuck namespaces
kubectl get namespace | grep Terminating
# If namespaces are stuck in Terminating state:
# kubectl patch namespace <namespace-name> -p '{"metadata":{"finalizers":[]}}' --type=merge
# Check for protection on resources
aws ec2 describe-instances \
--region $AWS_REGION \
--query 'Reservations[*].Instances[?State.Name!=`terminated`].[InstanceId,DisableApiTermination]' \
--output table
# Note: CloudFormation stacks for the workshop infrastructure should not be deleted
# These are managed by the workshop environment and contain VPC, EC2, and other base resources
echo "Use the above commands to troubleshoot any cleanup issues."
13. 정리 완료 요약
echo "================================================================"
echo " CLEANUP COMPLETED"
echo "================================================================"
echo "Thank you for completing the vLLM on EKS lab!"
echo "================================================================"
14. 정리 후 다음 단계
1) AWS 청구 내역 검토: 앞으로 며칠 동안 AWS 청구서를 확인해 모든 리소스가 제대로 종료되었는지 확인한다. 2) 문서화: 나중에 참고할 수 있도록 실습에서 얻은 성능 지표나 학습 내용을 저장한다. 3) 피드백: 실습 경험에 대한 피드백 제공을 고려한다. 4) 추가 학습: 다른 AWS AI/ML 서비스 또는 고급 Kubernetes 패턴을 살펴본다.
이로써 EKS 기반 vLLM 실습의 해체 절차를 완료했다. 이제 모든 리소스가 정리되어 더 이상 요금이 발생하지 않아야 한다.
댓글남기기