Setting up AWS Lambda with Confluent Cloud Kafka
A technical walk-through
Serverless is a whole new computing paradigm, Lambda is a great tool, but until recently, it was quite tightly coupled to AWS services, you want to shoot up a lambda from SQS? Easy, API gateway? It Goes without saying. S3 Event? Sure. And A plethora of other options available. But for me, at least fell short on triggers out of AWS services — specifically on missing Kafka integration.
Kafka is my go-to solution when I need partitioned message streams with high guarantees for ordering and persistence. And Confluent is my go-to Kafka provider (Especially the “serveless” confluent cloud offer). But what happens when I need to connect a serverless Lambda to A confluent cloud Kafka topic? Until recently there was no “AWS first” solution for that (Confluent for example has a built-in “sink” connector for lambda as part of its managed Kafka connect offering).
And behold: Recently AWS announced that Lambda now supports Kafka. But reading the docs it’s mostly about supporting the built-in AWS Kafka offering: MSK and lacking documentation on how to work with confluent Kafka for example. Serverless, a popular framework abstracting lambda for common use cases, also now has docs on using it with Kafka. Unfortunately, both docs are somewhat cryptic, especially around authentication. What is the meaning of different auth options? How should we construct the referenced secret manager secrets? I couldn’t figure it out. And couldn’t find an example specifically for confluent Kafka.
So I ventured into it and managed to set it up in a private side project. A few weeks later, we bumped into this at work, and while I was glad I had a working example, I decided to document the (simple but undocumented) process.
Let’s walk through the process of setting this up. I’d focus on setting up a Kafka lambda trigger as the rest is quite straight forward and the docs provide a good working example of the expected data structure.
Setting up a Kafka trigger:
First, let’s specify the bootstrap servers ( as copy pasted from the confluent UI or CLI) and enter the topic name.
If you need specific control on the consumer group id, then choose it, mostly it won’t be the case and you can skip
Get Alon Nisser’s stories in your inbox
Join Medium for free to get updates from this writer.
Now for the tricky part. Authentication, if you are using the serverless framework (or other abstraction above Cloudformation) it’s under accessConfiguration. or in the UI under Authentication
You should choose BASIC_AUTH from the UI, or (saslPlainAuth for the code described path) and as a value, you should provide the ARN of an AWS secrets manager “secret”. Note that if you’re doing this in code, you’ll also need to provide relevant IAM policy for the specific lambda role, allowing it to read this secret.
And now for the tricky part, which isn’t documented very well. In the Amazon Secrets manager (that should exist before setting up the lambda) you should create a secret in this format:
{“username”:”confluent_username”,”password:”confluent key”}
Where you replace “confluent_username” and “confluent_key” with those provided when creating a service account and key for accessing a cluster in the confluent UI or CLI.
And that’s all, now you have an AWS serverless lambda wired with Confluent stand-alone Kafka setup.
Resources:
