Kafka integration

Connect your existing Kafka instance to Squid

Set up a Kafka integration

When using your own Kafka integration rather than the built-in queue, first add the integration to Squid.

• In the Squid Console, navigate to the Integrations page and select the Kafka integration.

• Provide the following details:

  • Integration ID - Choose an ID that is brief and helps identify the integration.
  • Bootstrap servers - A comma-separated list of servers and ports.
  • Key - Your Kafka API Key.
  • Secret - Your Kafka API secret. This is securely stored in Squid Secrets.
  • Avro schema registry - In the case that you would like to consume or produce Avro messages, provide the URL of your Avro schema registry. You can find this value in your cluster settings.

• Click Test connection to verify the integration information. Once verified, click Add integration.

To access the queue from the client, create a reference to a QueueManager using queue and passing the topic name and integration ID:

Client code

const queue = squid.queue('topic-name', 'kafka-integration-id');

To read messages from a queue, use the consume method. This returns an Observable that updates whenever a new message is posted to the queue. Messages are of string type.

Client code

const topicMessagesObs = queue.consume();

topicMessagesObs.subscribe((message: string) => {
  console.log(message);
});

Note

When you subscribe to a topic, the Observable you receive is configured to only include messages produced after the subscription is established. This behavior is due to the subscription process involving a server call, which introduces an approximate delay of ~100ms. Therefore, you can expect a short wait before the Observable starts delivering new messages.

To prevent memory leaks, complete your Observable when no longer in use:

Client code

topicMessagesObs.unsubscribe();

To add messages to a queue, use the produce method. This method takes an array of string messages that are then consumed by clients subscribed to the queue.

Client code

queue.produce(['hello', 'world']);

Securing your Apache Kafka topics

Accessing your CApache Kafka topics on the client requires security functions.

To secure a Squid queue topic, use the @secureTopic decorator, passing the topic name and the type of action. The following code allows read and write access to a queue for the 'topic-name' topic:

Backend code

import { SquidService, secureTopic } from '@squidcloud/backend';

export class ExampleService extends SquidService {
  @secureTopic('topic-name', 'all', 'kafka-integration-id')
  allowTopicAccess(): boolean {
    return true;
  }
}

To learn more about security functions, see the Queue security docs.