Skip to main content

Kafka integration

Connect your existing Kafka instance to Squid Cloud

Set up a Kafka integration

When using your own Kafka integration rather than the built-in queue, first add the integration to Squid.

  • In the Squid Cloud Console, navigate to the Integrations page and select the Kafka integration.

  • Provide the following details:

    • Integration ID - Choose an ID that is brief and helps identify the integration.
    • Bootstrap servers - A comma-separated list of servers and ports.
    • Key - Your Kafka API Key.
    • Secret - Your Kafka API secret. This is securely stored in Squid Secrets.
    • Avro schema registry - In the case that you would like to consume or produce Avro messages, provide the URL of your Avro schema registry. You can find this value in your cluster settings.
  • Click Test connection to verify the integration information. Once verified, click Add integration.

To access the queue from the client, create a reference to a QueueManager using queue and passing the topic name and integration ID:

Client code
const queue = squid.queue('topic-name', 'kafka-integration-id');

To read messages from a queue, use the consume method. This returns an Observable that updates whenever a new message is posted to the queue. Messages are of string type.

Client code
const topicMessagesObs = queue.consume();

topicMessagesObs.subscribe((message: string) => {
console.log(message);
});
Note

When you subscribe to a topic, the Observable you receive is configured to only include messages produced after the subscription is established. This behavior is due to the subscription process involving a server call, which introduces an approximate delay of ~100ms. Therefore, you can expect a short wait before the Observable starts delivering new messages.

To prevent memory leaks, complete your Observable when no longer in use:

Client code
topicMessagesObs.unsubscribe();

To add messages to a queue, use the produce method. This method takes an array of string messages that are then consumed by clients subscribed to the queue.

Client code
queue.produce(['hello', 'world']);

Securing your Apache Kafka topics

Accessing your CApache Kafka topics on the client requires security functions.

To secure a Squid queue topic, use the @secureTopic decorator, passing the topic name and the type of action. The following code allows read and write access to a queue for the 'topic-name' topic:

Backend code
import { SquidService, secureTopic } from '@squidcloud/backend';

export class ExampleService extends SquidService {
@secureTopic('topic-name', 'all', 'kafka-integration-id')
allowTopicAccess(): boolean {
return true;
}
}

To learn more about security functions, see the Queue security docs.