Create Topic
In Kafka, a topic is a category or stream of messages. Producers publish messages on a specific subject, and consumers subscribe to one or more topics to receive the messages. The topic helps in better parallel processing with partitioned processing. Kafka topics can contain any kind of message in any format, and the sequence of all these messages is called a data stream.
Unlike database tables, Kafka topics are not directly queryable. Instead, we use Kafka producers to send data to a topic and Kafka consumers to read the data from the topic sequentially.
Kafka Topics Example
A transportation company wants to track its fleet of trucks. Each truck is equipped with a GPS locator that reports its position to Apache Kafka. We will create a Kafka topic named trucks_gps where the trucks will publish their positions. Each truck will send a message to Kafka every 20 seconds, with each message containing the truck ID and its position (latitude and longitude). The trucks_gps topic can be divided into a suitable number of partitions, for example, 10. Multiple consumers can read from this topic. For instance, one application might display truck locations on a dashboard, while another might send notifications if specific events of interest occur.
- To create a topic, click on the + in the Topics tab.
In Kafka, partition and replication factor are key concepts related to how data is distributed and stored within a Kafka cluster. These two are very important to set correctly as they impact the performance and durability in the system. To choose replication factor and partition count refer to this guide.
After choosing replication factor and partition count, click on Create Topic.
Update Topic
To update a topic, click on the edit icon.
After creating a topic, you can change the partition count, retention hours, and retention bytes. However, increasing the partition count in a Kafka topic is a dangerous operation if your applications rely on key-based ordering.