Kpow is the all-in-one toolkit to manage, monitor, and learn about your Kafka resources.
This Helm chart is designed for the Kpow Annual listing on AWS Marketplace. It uses a custom AWS Marketplace container that integrates with AWS to verify subscriptions and manage entitlements. The container is automatically licensed to the subscribing AWS account, which is billed directly for the subscription. There is no need to arrange a separate license with us if you subscribe to a Kpow product through the AWS Marketplace.
This Helm chart is for the Kpow Annual offering on AWS Marketplace.
The minimum information Flex requires to operate is:
See the Kpow Documentation for a full list of configuration options.
eksctl create iamserviceaccount \
--name kpow \
--namespace factorhouse \
--cluster <ENTER_YOUR_CLUSTER_NAME_HERE> \
--attach-policy-arn arn:aws:iam::aws:policy/service-role/AWSLicenseManagerConsumptionPolicy \
--approve \
--override-existing-serviceaccounts
You can now deploy Kpow to EKS using this Service Account, which includes an IAM Role with the AWSLicenseManagerConsumptionPolicy policy attached.
You need to connect to a Kubernetes environment before you can install Kpow.
The following examples demonstrate installing Kpow in Amazon EKS.
aws eks --region <your-aws-region> update-kubeconfig --name <your-eks-cluster-name>
Updated context arn:aws:eks:<your-aws-region>:123123123:cluster/<your-eks-cluster-name> in /your/.kube/config
kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-...-21.ec2.internal Ready <none> 2m15s v1.32.9-eks-113cf36
...
Download and extract the Helm chart from the Marketplace listing repository.
export HELM_EXPERIMENTAL_OCI=1
aws ecr get-login-password \
--region us-east-1 | helm registry login \
--username AWS \
--password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com
mkdir awsmp-chart && cd awsmp-chart
helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/factor-house/kpow-aws-annual \
--version <VERSION_NUMBER>
tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete
When using helm install, you can pass configuration with the --set env.XYZ flag. This requires careful formatting for certain values.
Some fields, particularly integers and strings containing quotation marks, require quoting. You may also need to escape special characters (like commas or nested quotes) with a backslash (\). For more details, see Helm’s documentation on The Format and Limitations of --set.
The following example shows how to install Kpow from the command line, highlighting how to handle escaped commas and quotes:
helm install kpow ./kpow-aws-annual/ \
--set serviceAccount.create=false \
--set serviceAccount.name=kpow \
--set env.BOOTSTRAP="b-1.<cluster-name>.<cluster-identifier>.c8.kafka.us-east-1.amazonaws.com:9096" \
--set env.SECURITY_PROTOCOL="SASL_PLAINTEXT" \
--set env.SASL_MECHANISM="PLAIN" \
--set env.SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"user\" password=\"secret\";" \ # <-- note the escaped quotes
--create-namespace --namespace factorhouse
NAME: kpow
LAST DEPLOYED: Mon May 31 17:22:21 2021
NAMESPACE: factorhouse
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace factorhouse -l "app.kubernetes.io/name=kpow,app.kubernetes.io/instance=kpow" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to use your application"
kubectl --namespace factorhouse port-forward $POD_NAME 3000:3000
You can configure Kpow with a ConfigMap of environment variables as follows:
helm install kpow ./kpow-aws-annual/ \
--set envFromConfigMap=kpow-config \
--create-namespace --namespace factorhouse
This approach requires a ConfigMap named kpow-config to already exist in the factorhouse namespace. To configure Kpow with a local ConfigMap template, see Configuring with an Existing ConfigMap. You can also refer to kpow-config.yaml.example for a sample ConfigMap manifest.
For general guidance, see the Kubernetes documentation on configuring all key-value pairs in a ConfigMap as environment variables.
Follow the notes instructions to set the $POD_NAME variable and configure port forwarding to the Kpow UI.
export POD_NAME=$(kubectl get pods --namespace factorhouse -l "app.kubernetes.io/name=kpow,app.kubernetes.io/instance=kpow" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:3000 to use your application"
kubectl --namespace factorhouse port-forward $POD_NAME 3000:3000
Kpow is now available on http://127.0.0.1:3000.
kubectl describe pods --namespace factorhouse
Name: kpow-9988df6b6-vvf8z
Namespace: factorhouse
Priority: 0
Node: ip-172-31-33-42.ap-southeast-2.compute.internal/172.31.33.42
Start Time: Mon, 31 May 2021 17:22:22 +1000
Labels: app.kubernetes.io/instance=kpow
app.kubernetes.io/name=kpow
pod-template-hash=9988df6b6
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
kubectl logs $POD_NAME --namespace factorhouse
11:36:49.111 INFO [main] kpow.system ? start Kpow
...
helm delete kpow --namespace factorhouse
You can run Kpow with local edits to chart files to provide custom configuration.
Make any edits required to kpow-aws-annual/Chart.yaml or kpow-aws-annual/values.yaml (adding volume mounts, etc).
This is the recommended method for managing configuration separately from the Helm chart.
1. Prepare Your ConfigMap Manifest
Copy the example file (kpow-config.yaml.example), then edit it to set your desired metadata.name (e.g., kpow-config) and fill in your configuration under the data section.
cp ./kpow-aws-annual/kpow-config.yaml.example kpow-config.yaml
# now edit kpow-config.yaml
2. Create the ConfigMap in Kubernetes
Before installing, use kubectl to create the ConfigMap object in your cluster from the file you just prepared.
kubectl apply -f kpow-config.yaml --namespace factorhouse
3. Install the Chart
Install the Helm chart, using --set to reference the name of the ConfigMap you just created. The --create-namespace flag will ensure the target namespace exists.
helm install kpow ./kpow-aws-annual \
--set envFromConfigMap=kpow-config \
--create-namespace --namespace factorhouse
The Kpow pod will now start using the environment variables from your externally managed ConfigMap.
See kpow-config.yaml.example for an example ConfigMap file.
See the Kubernetes documentation on configuring all key-value pairs in a config map as container environment variables for more information.
This helm chart accepts the name of a secret containing sensitive parameters, e.g.
apiVersion: v1
kind: Secret
metadata:
name: kpow-secrets
data:
SASL_JAAS_CONFIG: a3JnLmFwYWNoXS5rYWZrYS5jb21tb24uc2VjdXJpdHkucGxhaW4uUGxhaW5Mb2dpbk2vZHVsZSByZXF1aXJiZCB1c2VybmFtZT0iTFQ1V0ZaV1BRWUpHNzRJQyIgcGFzc3dvcmQ9IjlYUFVYS3BLYUQxYzVJdXVNRjRPKzZ2NxJ0a1E4aS9yWUp6YlppdlgvZnNiTG51eGY4SnlFT1dUeXMvTnJ1bTAiBwo=
CONFLUENT_API_SECRET: NFJSejlReFNTTXlTcGhXdjNLMHNYY1F6UGNURmdadlNYT0ZXSXViWFJySmx2N3A2WStSenROQnVpYThvNG1NSRo=
kubectl apply -f ./kpow-secrets.yaml --namespace factorhouse
Then run the helm chart (this can be used in conjunction with envFromConfigMap)
See the Kubernetes documentation on configuring all key value pairs in a secret as environment variables for more information.
helm install kpow ./kpow-aws-annual/ \
--set envFromSecret=kpow-secrets \
--set envFromConfigMap=kpow-config \
--create-namespace --namespace factorhouse
There are occasions where you must provide files to the Kpow Pod in order for Kpow to run correctly, such files include:
How you provide these files is down to user preference, we are not able to provide any support or instruction in this regard.
You may find the Kubernetes documentation on injecting data into applications useful.
The chart runs Kpow with Guaranteed QoS, having resource request and limit set to these values by default:
resources:
limits:
cpu: 2
memory: 8Gi
requests:
cpu: 2
memory: 8Gi
These default resource settings are conservative, suited to a deployment of Kpow that manages multiple Kafka clusters and associated resources.
When running Kpow with a single Kafka cluster you can experiment with reducing those resources as far as our suggested minimum:
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
Adjust these values from the command line like so:
helm install kpow ./kpow-aws-annual/ \
--set resources.limits.cpu=1 \
--set resources.limits.memory=2Gi \
--set resources.requests.cpu=1 \
--set resources.requests.memory=2Gi \
--create-namespace --namespace factorhouse
We recommend always having limits and requests set to the same value, as this set Kpow in Guaranteed QoS and provides a much more reliable operation.
We preset an attribute for Snappy compression in read-only filesystems. It is disabled by default and can be enabled - modify the volume configuration if necessary.
ephemeralTmp:
enabled: true
volume:
emptyDir:
medium: Memory # Optional: for better performance
sizeLimit: "100Mi" # Configurable size
This Helm chart includes extra resources required for the token-based IAM authentication used by EKS Anywhere. It can be configured as follows.
1. Create the namespace and a dedicated service account
kubectl create namespace factorhouse
kubectl create serviceaccount kpow --namespace factorhouse
2. Create the license secret with the values from Step 1
# IMPORTANT: Replace the placeholder values below with your actual token and role ARN.
AWSMP_TOKEN="<YOUR_LICENSE_TOKEN_HERE>"
AWSMP_ROLE_ARN="<YOUR_IAM_ROLE_ARN_HERE>"
kubectl create secret generic awsmp-license-token-secret \
--from-literal=license_token=$AWSMP_TOKEN \
--from-literal=iam_role=$AWSMP_ROLE_ARN \
--namespace factorhouse
3. Create an ECR image pull secret using the license token
AWSMP_ACCESS_TOKEN=$(aws license-manager get-access-token \
--output text --query '*' --token $AWSMP_TOKEN --region us-east-1)
AWSMP_ROLE_CREDENTIALS=$(aws sts assume-role-with-web-identity \
--region 'us-east-1' \
--role-arn $AWSMP_ROLE_ARN \
--role-session-name 'AWSMP-guided-deployment-session' \
--web-identity-token $AWSMP_ACCESS_TOKEN \
--query 'Credentials' \
--output text)
export AWS_ACCESS_KEY_ID=$(echo $AWSMP_ROLE_CREDENTIALS | awk '{print $1}' | xargs)
export AWS_SECRET_ACCESS_KEY=$(echo $AWSMP_ROLE_CREDENTIALS | awk '{print $3}' | xargs)
export AWS_SESSION_TOKEN=$(echo $AWSMP_ROLE_CREDENTIALS | awk '{print $4}' | xargs)
kubectl create secret docker-registry awsmp-image-pull-secret \
--docker-server=709825985650.dkr.ecr.us-east-1.amazonaws.com \
--docker-username=AWS \
--docker-password=$(aws ecr get-login-password --region us-east-1) \
--namespace factorhouse
4. Link the image pull secret to the service account
kubectl patch serviceaccount kpow \
--namespace factorhouse \
-p '{"imagePullSecrets": [{"name": "awsmp-image-pull-secret"}]}'
Download and extract the Helm chart from the Marketplace listing repository.
export HELM_EXPERIMENTAL_OCI=1
aws ecr get-login-password \
--region us-east-1 | helm registry login \
--username AWS \
--password-stdin 709825985650.dkr.ecr.us-east-1.amazonaws.com
mkdir awsmp-chart && cd awsmp-chart
helm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/factor-house/kpow-aws-annual \
--version <VERSION_NUMBER>
tar xf $(pwd)/* && find $(pwd) -maxdepth 1 -type f -delete
Install Kpow, referencing the Kubernetes resources you created above.
helm install kpow ./kpow-aws-annual/ \
--set serviceAccount.create=false \
--set serviceAccount.name=kpow \
--set aws.licenseConfigSecretName=awsmp-license-token-secret \
--create-namespace --namespace factorhouse
If you have any issues or errors, please contact [email protected].
This repository is Apache 2.0 licensed, you are welcome to clone and modify as required.