Amazon Elastic Kubernetes Service (EKS) is a managed Kubernetes service that makes it easy to run Kubernetes on AWS without needing to install, operate, and maintain your own Kubernetes control plane or worker nodes. One popular use case for Amazon EKS is deploying Apache Kafka, a highly popular open-source distributed streaming platform.
To install a Kafka cluster on an Amazon EKS cluster, we can use the AWS Cloud Development Kit (CDK). The AWS CDK is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation.
This article provides a step-by-step guide on how to use AWS CDK to install a Kafka cluster on an EKS cluster.
## Prerequisites
Before you get started, you will need:
- AWS account with necessary permissions to create EKS clusters and other necessary resources
- 2. AWS CLI and CDK installed and configured
- 3. A basic understanding of AWS CDK, Kubernetes, and Apache Kafka
## Step 1: Initialize a New AWS CDK App
To start with, let's initialize a new AWS CDK app:
```bash
cdk init app — language=typescript
```
This command will create a new AWS CDK app in TypeScript, which is a statically typed superset of JavaScript that adds optional types.
## Step 2: Install AWS CDK Kubernetes (cdk8s) Construct Library
We'll be using the AWS CDK Kubernetes (cdk8s) construct library to define our EKS cluster and Kafka deployment:
```bash
npm install @aws-cdk/aws-eks @aws-cdk/aws-ec2 @aws-cdk/aws-iam @aws-cdk/core
```
## Step 3: Define the AWS EKS Cluster
Now, we're ready to define our EKS cluster and Kafka deployment. Replace the contents of `lib/<stack-name>.ts` with the following code:
```typescript
import * as cdk from '@aws-cdk/core'
import * as eks from '@aws-cdk/aws-eks'
import * as ec2 from '@aws-cdk/aws-ec2'
import * as iam from '@aws-cdk/aws-iam'
export class KafkaEksStack extends cdk.Stack {
. constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
. super(scope, id, props);
. // Create a new VPC for our EKS Cluster
. const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 3 });
. // Define the IAM role that will be used by EKS
. const clusterAdmin = new iam.Role(this, 'AdminRole', {
. assumedBy: new iam.AccountRootPrincipal(),
. });
. // Create the EKS Cluster
. const cluster = new eks.Cluster(this, 'Cluster', {
. vpc: vpc,
. mastersRole: clusterAdmin,
. defaultCapacity: 2, // number of worker nodes
. version: eks.KubernetesVersion.V1_21, // kubernetes version
. });
. // Output the EKS Cluster Name
. new cdk.CfnOutput(this, 'ClusterName', {
. value: cluster.clusterName,
. });
. // Output the EKS Cluster Endpoint
. new cdk.CfnOutput(this, 'ClusterEndpoint', {
. value: cluster.clusterEndpoint,
. });
. }
}
```
This code creates a VPC with three availability zones, an IAM role that can be assumed by the root account to administer the EKS cluster, and an EKS cluster with two worker nodes running Kubernetes v1.21.
## Step 4: Define the Kafka Cluster
Defining the Kafka cluster requires writing Kubernetes manifest files (YAML), which can be a bit cumbersome and error-prone. A better approach is to use Helm charts or operators like Strimzi, which greatly simplify the process of managing Kafka on Kubernetes. Here's an example of how to use the Strimzi operator to manage your Kafka cluster:
```typescript
import * as cdk8s from 'cdk8s'
// In the KafkaEksStack constructor, after defining the EKS cluster:
// Deploy Strimzi Operator
const strimziOperator = new eks.HelmChart(this, 'StrimziOperator', {
. cluster: cluster,
. chart: 'strimzi-kafka-operator',
. repository: 'https://strimzi.io/charts',
. namespace: 'kafka',
});
// Define Kafka cluster via cdk8s
const kafkaCluster = new cdk8s.Chart(this, 'KafkaCluster');
new eks.KubernetesManifest(this, 'KafkaCluster', {
. cluster: cluster,
. manifest: [kafkaCluster.toJson()],
});
```
Please note, for this to work, the Strimzi operator Helm chart must be available in the specified repository.
## Step 5: Deploy the Stack
Finally, to deploy the stack, compile the TypeScript code and deploy the CDK stack:
```bash
npm run build
cdk deploy
```
This will create an EKS cluster with a running Kafka cluster managed by the Strimzi operator.
## Conclusion
In this tutorial, we've explored how to use AWS CDK to define and provision an Apache Kafka cluster running on Amazon EKS. AWS CDK is a powerful tool for defining cloud infrastructure in a flexible, reusable, and safe manner. With the help of Helm charts and operators, managing complex systems like Kafka becomes much more manageable.
Do note that this is a basic example and actual production setup might need more fine-tuning and configurations like security settings, networking, storage classes, resource requests and limits, etc.
Enjoy your new Kafka cluster and happy streaming!