Use Batch Processing Gateway to automate job administration in multi-cluster Amazon EMR on EKS environments

AWS prospects usually course of petabytes of knowledge utilizing Amazon EMR on EKS. In enterprise environments with various workloads or various operational necessities, prospects steadily select a multi-cluster setup as a result of following benefits:

• Higher resiliency and no single level of failure – If one cluster fails, different clusters can proceed processing vital workloads, sustaining enterprise continuity
• Higher safety and isolation – Elevated isolation between jobs enhances safety and simplifies compliance
• Higher scalability – Distributing workloads throughout clusters allows horizontal scaling to deal with peak calls for
• Efficiency advantages – Minimizing Kubernetes scheduling delays and community bandwidth rivalry improves job runtimes
• Elevated flexibility – You’ll be able to take pleasure in easy experimentation and price optimization via workload segregation to a number of clusters

Nevertheless, one of many disadvantages of a multi-cluster setup is that there isn’t any easy methodology to distribute workloads and help efficient load balancing throughout a number of clusters. This submit proposes an answer to this problem by introducing the Batch Processing Gateway (BPG), a centralized gateway that automates job administration and routing in multi-cluster environments.

Challenges with multi-cluster environments

In a multi-cluster atmosphere, Spark jobs on Amazon EMR on EKS must be submitted to totally different clusters from numerous shoppers. This structure introduces a number of key challenges:

• Endpoint administration – Shoppers should keep and replace connections for every goal cluster
• Operational overhead – Managing a number of consumer connections individually will increase the complexity and operational burden
• Workload distribution – There is no such thing as a built-in mechanism for job routing throughout a number of clusters, which impacts configuration, useful resource allocation, price transparency, and resilience
• Resilience and excessive availability – With out load balancing, the atmosphere lacks fault tolerance and excessive availability

BPG addresses these challenges by offering a single level of submission for Spark jobs. BPG automates job routing to the suitable EMR on EKS clusters, offering efficient load balancing, simplified endpoint administration, and improved resilience. The proposed resolution is especially helpful for purchasers with multi-cluster Amazon EMR on EKS setups utilizing the Spark Kubernetes Operator with or with out Yunikorn scheduler.

Nevertheless, though BPG gives important advantages, it’s presently designed to work solely with Spark Kubernetes Operator. Moreover, BPG has not been examined with the Volcano scheduler, and the answer isn’t relevant in environments utilizing native Amazon EMR on EKS APIs.

Answer overview

Martin Fowler describes a gateway as an object that encapsulates entry to an exterior system or useful resource. On this case, the useful resource is the EMR on EKS clusters working Spark. A gateway acts as a single level to confront this useful resource. Any code or connection interacts with the interface of the gateway solely. The gateway then interprets the incoming API request into the API supplied by the useful resource.

BPG is a gateway particularly designed to supply a seamless interface to Spark on Kubernetes. It’s a REST API service to summary the underlying Spark on EKS clusters particulars from customers. It runs in its personal EKS cluster speaking to Kubernetes API servers of various EKS clusters. Spark customers submit an utility to BPG via shoppers, then BPG routes the appliance to one of many underlying EKS clusters.

The method for submitting Spark jobs utilizing BPG for Amazon EMR on EKS is as follows:

1. The consumer submits a job to BPG utilizing a consumer.
2. BPG parses the request, interprets it right into a customized useful resource definition (CRD), and submits the CRD to an EMR on EKS cluster in accordance with predefined guidelines.
3. The Spark Kubernetes Operator interprets the job specification and initiates the job on the cluster.
4. The Kubernetes scheduler schedules and manages the run of the roles.

The next determine illustrates the high-level particulars of BPG. You’ll be able to learn extra about BPG within the GitHub README.

The proposed resolution entails implementing BPG for a number of underlying EMR on EKS clusters, which successfully resolves the drawbacks mentioned earlier. The next diagram illustrates the main points of the answer.

Supply Code

You will discover the code base within the AWS Samples and Batch Processing Gateway GitHub repository.

Within the following sections, we stroll via the steps to implement the answer.

Stipulations

Earlier than you deploy this resolution, make sure that the next stipulations are in place:

Clone the repositories to your native machine

We assume that each one repositories are cloned into the house listing (~/). All relative paths offered are primarily based on this assumption. If in case you have cloned the repositories to a unique location, regulate the paths accordingly.

1. Clone the BPG on EMR on EKS GitHub repo with the next command:

cd ~/
git clone git@github.com:aws-samples/batch-processing-gateway-on-emr-on-eks.git

The BPG repository is presently beneath lively improvement. To offer a steady deployment expertise in keeping with the offered directions, we’ve pinned the repository to the steady commit hash aa3e5c8be973bee54ac700ada963667e5913c865.

Earlier than cloning the repository, confirm any safety updates and cling to your group’s safety practices.

2. Clone the BPG GitHub repo with the next command:

git clone git@github.com:apple/batch-processing-gateway.git
cd batch-processing-gateway
git checkout aa3e5c8be973bee54ac700ada963667e5913c865

Create two EMR on EKS clusters

The creation of EMR on EKS clusters isn’t the first focus of this submit. For complete directions, seek advice from Operating Spark jobs with the Spark operator. Nevertheless, on your comfort, we’ve included the steps for organising the EMR on EKS digital clusters named spark-cluster-a-v and spark-cluster-b-v within the GitHub repo. Comply with these steps to create the clusters.

After efficiently finishing the steps, it’s best to have two EMR on EKS digital clusters named spark-cluster-a-v and spark-cluster-b-v working on the EKS clusters spark-cluster-a and spark-cluster-b, respectively.

To confirm the profitable creation of the clusters, open the Amazon EMR console and select Digital clusters beneath EMR on EKS within the navigation pane.

Arrange BPG on Amazon EKS

To arrange BPG on Amazon EKS, full the next steps:

1. Change to the suitable listing:

cd ~/batch-processing-gateway-on-emr-on-eks/bpg/

2. Arrange the AWS Area:

export AWS_REGION="<AWS_REGION>"

3. Create a key pair. Be sure you comply with your group’s finest practices for key pair administration.

aws ec2 create-key-pair 
--region "$AWS_REGION" 
--key-name ekskp 
--key-type ed25519 
--key-format pem 
--query "KeyMaterial" 
--output textual content > ekskp.pem
chmod 400 ekskp.pem
ssh-keygen -y -f ekskp.pem > eks_publickey.pem
chmod 400 eks_publickey.pem

Now you’re able to create the EKS cluster.

By default, eksctl creates an EKS cluster in devoted digital non-public clouds (VPCs). To keep away from reaching the default smooth restrict on the variety of VPCs in an account, we use the --vpc-public-subnets parameter to create clusters in an present VPC. For this submit, we use the default VPC for deploying the answer. Modify the next code to deploy the answer within the applicable VPC in accordance together with your group’s finest practices. For official steering, seek advice from Create a VPC.

4. Get the general public subnets on your VPC:

export DEFAULT_FOR_AZ_SUBNET=$(aws ec2 describe-subnets --region "$AWS_REGION" --filters "Identify=default-for-az,Values=true" --query "Subnets[?AvailabilityZone != 'us-east-1e'].SubnetId" | jq -r '. | map(tostring) | be a part of(",")')

5. Create the cluster:

eksctl create cluster 
--name bpg-cluster 
--region "$AWS_REGION" 
--vpc-public-subnets "$DEFAULT_FOR_AZ_SUBNET" 
--with-oidc 
--ssh-access 
--ssh-public-key eks_publickey.pem 
--instance-types=m5.xlarge 
--managed

6. On the Amazon EKS console, select Clusters within the navigation pane and examine for the profitable provisioning of the bpg-cluster

Within the subsequent steps, we make the next adjustments to the prevailing batch-processing-gateway code base:

To your comfort, we’ve offered the up to date recordsdata within the batch-processing-gateway-on-emr-on-eks repository. You’ll be able to copy these recordsdata into the batch-processing-gateway repository.

7. Substitute POM xml file:

cp ~/batch-processing-gateway-on-emr-on-eks/bpg/pom.xml ~/batch-processing-gateway/pom.xml

8. Substitute DAO java file:

cp ~/batch-processing-gateway-on-emr-on-eks/bpg/LogDao.java ~/batch-processing-gateway/src/foremost/java/com/apple/spark/core/LogDao.java

9. Substitute the Dockerfile:

cp ~/batch-processing-gateway-on-emr-on-eks/bpg/Dockerfile ~/batch-processing-gateway/Dockerfile

Now you’re able to construct your Docker picture.

10. Create a personal Amazon Elastic Container Registry (Amazon ECR) repository:

aws ecr create-repository --repository-name bpg --region "$AWS_REGION"

11. Get the AWS account ID:

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output textual content)

12. Authenticate Docker to your ECR registry:

aws ecr get-login-password --region "$AWS_REGION" | docker login --username AWS --password-stdin "$AWS_ACCOUNT_ID".dkr.ecr."$AWS_REGION".amazonaws.com

13. Construct your Docker picture:

cd ~/batch-processing-gateway/
docker construct 
--platform linux/amd64 
--build-arg VERSION="1.0.0" 
--build-arg BUILD_TIME=$(date -u +"%Y-%m-%dTpercentH:%M:%SZ") 
--build-arg GIT_COMMIT=$(git rev-parse HEAD) 
--progress=plain 
--no-cache 
-t bpg:1.0.0 .

14. Tag your picture:

docker tag bpg:1.0.0 "$AWS_ACCOUNT_ID".dkr.ecr."$AWS_REGION".amazonaws.com/bpg:1.0.0

15. Push the picture to your ECR repository:

docker push "$AWS_ACCOUNT_ID".dkr.ecr."$AWS_REGION".amazonaws.com/bpg:1.0.0

The ImagePullPolicy within the batch-processing-gateway GitHub repo is ready to IfNotPresent. Replace the picture tag in case it’s essential to replace the picture.

16. To confirm the profitable creation and add of the Docker picture, open the Amazon ECR console, select Repositories beneath Non-public registry within the navigation pane, and find the bpg repository:

Arrange an Amazon Aurora MySQL database

Full the next steps to arrange an Amazon Aurora MySQL-Suitable Version database:

1. Listing all default subnets for the given Availability Zone in a selected format:

DEFAULT_FOR_AZ_SUBNET_RFMT=$(aws ec2 describe-subnets --region "$AWS_REGION" --filters "Identify=default-for-az,Values=true" --query "Subnets[*].SubnetId" | jq -c '.')

2. Create a subnet group. Discuss with create-db-subnet-group for extra particulars.

aws rds create-db-subnet-group 
--db-subnet-group-name bpg-rds-subnetgroup 
--db-subnet-group-description "BPG Subnet Group for RDS" 
--subnet-ids "$DEFAULT_FOR_AZ_SUBNET_RFMT" 
--region "$AWS_REGION"

3. Listing the default VPC:

export DEFAULT_VPC=$(aws ec2 describe-vpcs --region "$AWS_REGION" --filters "Identify=isDefault,Values=true" --query "Vpcs[0].VpcId" --output textual content)

4. Create a safety group:

aws ec2 create-security-group 
--group-name bpg-rds-securitygroup 
--description "BPG Safety Group for RDS" 
--vpc-id "$DEFAULT_VPC" 
--region "$AWS_REGION"

5. Listing the bpg-rds-securitygroup safety group ID:

export BPG_RDS_SG=$(aws ec2 describe-security-groups --filters "Identify=group-name,Values=bpg-rds-securitygroup" --query "SecurityGroups[*].GroupId" --output textual content)

6. Create the Aurora DB Regional cluster. Discuss with create-db-cluster for extra particulars.

aws rds create-db-cluster 
--database-name bpg 
--db-cluster-identifier bpg 
--engine aurora-mysql 
--engine-version 8.0.mysql_aurora.3.06.1 
--master-username admin 
--manage-master-user-password 
--db-subnet-group-name bpg-rds-subnetgroup 
--vpc-security-group-ids "$BPG_RDS_SG" 
--region "$AWS_REGION"

7. Create a DB Author occasion within the cluster. Discuss with create-db-instance for extra particulars.

aws rds create-db-instance 
--db-instance-identifier bpg 
--db-cluster-identifier bpg 
--db-instance-class db.r5.giant 
--engine aurora-mysql 
--region "$AWS_REGION"

8. To confirm the profitable creation of the RDS Regional cluster and Author occasion, on the Amazon RDS console, select Databases within the navigation pane and examine for the bpg database.

Arrange community connectivity

Safety teams for EKS clusters are usually related to the nodes and the management airplane (if utilizing managed nodes). On this part, we configure the networking to permit the node safety group of the bpg-cluster to speak with spark-cluster-a, spark-cluster-b, and the bpg Aurora RDS cluster.

1. Establish the safety teams of bpg-cluster, spark-cluster-a, spark-cluster-b, and the bpg Aurora RDS cluster:

# Establish Node Safety Group of the bpg-cluster
BPG_CLUSTER_NODEGROUP_SG=$(aws ec2 describe-instances 
--filters Identify=tag:eks:cluster-name,Values=bpg-cluster 
--query "Reservations[*].Situations[*].SecurityGroups[?contains(GroupName, 'eks-cluster-sg-bpg-cluster-')].GroupId" 
--region "$AWS_REGION" 
--output textual content | uniq)

# Establish Cluster safety group of spark-cluster-a and spark-cluster-b
SPARK_A_CLUSTER_SG=$(aws eks describe-cluster --name spark-cluster-a --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output textual content)
SPARK_B_CLUSTER_SG=$(aws eks describe-cluster --name spark-cluster-b --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output textual content)

# Establish Cluster safety group of bpg Aurora RDS cluster Author Occasion
BPG_RDS_WRITER_SG=$(aws ec2 describe-security-groups --filters "Identify=group-name,Values=bpg-rds-securitygroup" --query "SecurityGroups[*].GroupId" --output textual content)

2. Permit the node safety group of the bpg-cluster to speak with spark-cluster-a, spark-cluster-b, and the bpg Aurora RDS cluster:

# spark-cluster-a
aws ec2 authorize-security-group-ingress --group-id "$SPARK_A_CLUSTER_SG" --protocol tcp --port 443 --source-group "$BPG_CLUSTER_NODEGROUP_SG"

# spark-cluster-b
aws ec2 authorize-security-group-ingress --group-id "$SPARK_B_CLUSTER_SG" --protocol tcp --port 443 --source-group "$BPG_CLUSTER_NODEGROUP_SG"

# bpg-rds
aws ec2 authorize-security-group-ingress --group-id "$BPG_RDS_WRITER_SG" --protocol tcp --port 3306 --source-group "$BPG_CLUSTER_NODEGROUP_SG"

Deploy BPG

We deploy BPG for weight-based cluster choice. spark-cluster-a-v and spark-cluster-b-v are configured with a queue named dev and weight=50. We anticipate statistically equal distribution of jobs between the 2 clusters. For extra data, seek advice from Weight Primarily based Cluster Choice.

1. Get the bpg-cluster context:

BPG_CLUSTER_CONTEXT=$(kubectl config view --output=json | jq -r '.contexts[] | choose(.identify | comprises("bpg-cluster")) | .identify')
kubectl config use-context "$BPG_CLUSTER_CONTEXT"

2. Create a Kubernetes namespace for BPG:

kubectl create namespace bpg

The helm chart for BPG requires a values.yaml file. This file consists of numerous key-value pairs for every EMR on EKS clusters, EKS cluster, and Aurora cluster. Manually updating the values.yaml file may be cumbersome. To simplify this course of, we’ve automated the creation of the values.yaml file.

3. Run the next script to generate the values.yaml file:

cd ~/batch-processing-gateway-on-emr-on-eks/bpg
chmod 755 create-bpg-values-yaml.sh
./create-bpg-values-yaml.sh

4. Use the next code to deploy the helm chart. Ensure the tag worth in each values.template.yaml and values.yaml matches the Docker picture tag specified earlier.

cp ~/batch-processing-gateway/helm/batch-processing-gateway/values.yaml ~/batch-processing-gateway/helm/batch-processing-gateway/values.yaml.$(date +'%YpercentmpercentdpercentHpercentMpercentS') 
&& cp ~/batch-processing-gateway-on-emr-on-eks/bpg/values.yaml ~/batch-processing-gateway/helm/batch-processing-gateway/values.yaml 
&& cd ~/batch-processing-gateway/helm/batch-processing-gateway/

kubectl config use-context "$BPG_CLUSTER_CONTEXT"

helm set up batch-processing-gateway . --values values.yaml -n bpg

5. Confirm the deployment by itemizing the pods and viewing the pod logs:

kubectl get pods --namespace bpg
kubectl logs <BPG-PODNAME> --namespace bpg

6. Exec into the BPG pod and confirm the well being examine:

kubectl exec -it <BPG-PODNAME> -n bpg -- bash 
curl -u admin:admin localhost:8080/skatev2/healthcheck/standing

We get the next output:

{"standing":"OK"}

BPG is efficiently deployed on the EKS cluster.

Take a look at the answer

To check the answer, you’ll be able to submit a number of Spark jobs by working the next pattern code a number of instances. The code submits the SparkPi Spark job to the BPG, which in flip submits the roles to the EMR on EKS cluster primarily based on the set parameters.

1. Set the kubectl context to the bpg cluster:

kubectl config get-contexts | awk 'NR==1 || /bpg-cluster/'
kubectl config use-context "<CONTEXT_NAME>"

2. Establish the bpg pod identify:

kubectl get pods --namespace bpg

3. Exec into the bpg pod:

kubectl exec -it "<BPG-PODNAME>" -n bpg -- bash

4. Submit a number of Spark jobs utilizing the curl. Run the under curl command to submit jobs to spark-cluster-a and spark-cluster-b:

curl -u consumer:go localhost:8080/skatev2/spark -i -X POST 
-H 'Content material-Sort: utility/json' 
-d '{
"applicationName": "SparkPiDemo",
"queue": "dev",
"sparkVersion": "3.5.0",
"mainApplicationFile": "native:///usr/lib/spark/examples/jars/spark-examples.jar",
"mainClass":"org.apache.spark.examples.SparkPi",
"driver": {
"cores": 1,
"reminiscence": "2g",
"serviceAccount": "emr-containers-sa-spark",
"labels":{
"model": "3.5.0"
}
},
"executor": {
"situations": 1,
"cores": 1,
"reminiscence": "2g",
"labels":{
"model": "3.5.0"
}
}
}'

After every submission, BPG will inform you of the cluster to which the job was submitted. For instance:

HTTP/1.1 200 OK
Date: Sat, 10 Aug 2024 16:17:15 GMT
Content material-Sort: utility/json
Content material-Size: 67
{"submissionId":"spark-cluster-a-f72a7ddcfde14f4390194d4027c1e1d6"}
{"submissionId":"spark-cluster-a-d1b359190c7646fa9d704122fbf8c580"}
{"submissionId":"spark-cluster-b-7b61d5d512bb4adeb1dd8a9977d605df"}

5. Confirm that the roles are working within the EMR cluster spark-cluster-a and spark-cluster-b:

kubectl config get-contexts | awk 'NR==1 || /spark-cluster-(a|b)/'
kubectl get pods -n spark-operator --context "<CONTEXT_NAME>"

You’ll be able to view the Spark Driver logs to seek out the worth of Pi as proven under:

kubectl logs <SPARK-DRIVER-POD-NAME> --namespace spark-operator --context "<CONTEXT_NAME>"

After profitable completion of the job, it’s best to be capable of see the under message within the logs:

Pi is roughly 3.1452757263786317

We’ve efficiently examined the weight-based routing of Spark jobs throughout a number of clusters.

Clear up

To scrub up your assets, full the next steps:

1. Delete the EMR on EKS digital cluster:

VIRTUAL_CLUSTER_ID=$(aws emr-containers list-virtual-clusters --region="$AWS_REGION" --query "virtualClusters[?name=='spark-cluster-a-v' && state=='RUNNING'].id" --output textual content)
aws emr-containers delete-virtual-cluster --region="$AWS_REGION" --id "$VIRTUAL_CLUSTER_ID"
VIRTUAL_CLUSTER_ID=$(aws emr-containers list-virtual-clusters --region="$AWS_REGION" --query "virtualClusters[?name=='spark-cluster-b-v' && state=='RUNNING'].id" --output textual content)
aws emr-containers delete-virtual-cluster --region="$AWS_REGION" --id "$VIRTUAL_CLUSTER_ID"

2. Delete the AWS Id and Entry Administration (IAM) position:

aws iam delete-role-policy --role-name sparkjobrole --policy-name EMR-Spark-Job-Execution
aws iam delete-role --role-name sparkjobrole

3. Delete the RDS DB occasion and DB cluster:

aws rds delete-db-instance 
--db-instance-identifier bpg 
--skip-final-snapshot

aws rds delete-db-cluster 
--db-cluster-identifier bpg 
--skip-final-snapshot

4. Delete the bpg-rds-securitygroup safety group and bpg-rds-subnetgroup subnet group:

BPG_SG=$(aws ec2 describe-security-groups --filters "Identify=group-name,Values=bpg-rds-securitygroup" --query "SecurityGroups[*].GroupId" --output textual content)
aws ec2 delete-security-group --group-id "$BPG_SG"
aws rds delete-db-subnet-group --db-subnet-group-name bpg-rds-subnetgroup

5. Delete the EKS clusters:

eksctl delete cluster --region="$AWS_REGION" --name=bpg-cluster
eksctl delete cluster --region="$AWS_REGION" --name=spark-cluster-a
eksctl delete cluster --region="$AWS_REGION" --name=spark-cluster-b

6. Delete bpg ECR repository:

aws ecr delete-repository --repository-name bpg --region="$AWS_REGION" --force

7. Delete the important thing pairs:

aws ec2 delete-key-pair --key-name ekskp
aws ec2 delete-key-pair --key-name emrkp

Conclusion

On this submit, we explored the challenges related to managing workloads on EMR on EKS cluster and demonstrated the benefits of adopting a multi-cluster deployment sample. We launched Batch Processing Gateway (BPG) as an answer to those challenges, showcasing the way it simplifies job administration, enhances resilience, and improves horizontal scalability in multi-cluster environments. By implementing BPG, we illustrated the sensible utility of the gateway structure sample for submitting Spark jobs on Amazon EMR on EKS. This submit offers a complete understanding of the issue, the advantages of the gateway structure, and the steps to implement BPG successfully.

We encourage you to guage your present Spark on Amazon EMR on EKS implementation and contemplate adopting this resolution. It permits customers to submit, look at, and delete Spark purposes on Kubernetes with intuitive API calls, without having to fret in regards to the underlying complexities.

For this submit, we centered on the implementation particulars of the BPG. As a subsequent step, you’ll be able to discover integrating BPG with shoppers resembling Apache Airflow, Amazon Managed Workflows for Apache Airflow (Amazon MWAA), or Jupyter notebooks. BPG works properly with the Apache Yunikorn scheduler. You may as well discover integrating BPG to make use of Yunikorn queues for job submission.

In regards to the Authors

Umair Nawaz is a Senior DevOps Architect at Amazon Internet Companies. He works on constructing safe architectures and advises enterprises on agile software program supply. He’s motivated to resolve issues strategically by using fashionable applied sciences.

Ravikiran Rao is a Information Architect at Amazon Internet Companies and is enthusiastic about fixing advanced information challenges for numerous prospects. Outdoors of labor, he’s a theater fanatic and novice tennis participant.

Sri Potluri is a Cloud Infrastructure Architect at Amazon Internet Companies. He’s enthusiastic about fixing advanced issues and delivering well-structured options for various prospects. His experience spans throughout a variety of cloud applied sciences, making certain scalable and dependable infrastructure tailor-made to every mission’s distinctive challenges.

Suvojit Dasgupta is a Principal Information Architect at Amazon Internet Companies. He leads a crew of expert engineers in designing and constructing scalable information options for AWS prospects. He makes a speciality of growing and implementing modern information architectures to deal with advanced enterprise challenges.