Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming apps.
It is a publish-subscribe messaging system that provides a scalable and fault-tolerant way of handling high volumes of data.
SetUp Kafka:
- Download and install Apache Kafka: You can download the latest version of Apache Kafka from the official website (https://kafka.apache.org/downloads).
- Start a ZooKeeper server: ZooKeeper is a distributed coordination service that is used by Kafka to coordinate the activities of the brokers in a cluster. You can start a standalone ZooKeeper server using the command
bin/zookeeper-server-start.sh config/zookeeper.properties. - Start a Kafka broker: To start a Kafka broker, use the command
bin/kafka-server-start.sh config/server.properties. By default, the broker will listen on port 9092. - Create a topic: Topics are the core abstraction in Kafka. To create a topic, use the command
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic-name. - Start a producer: Producers are clients that write data to a topic. You can start a producer using the command
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-name. - Start a consumer: Consumers are clients that read data from a topic. You can start a consumer using the command
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-name --from-beginning.
Kafka Components
The components of Kafka are:
- Topics: A stream of records, logically grouped by the topic. Each topic is split into partitions.
- Partitions: A partition is an ordered, immutable sequence of records that is split across multiple servers for scalability.
- Producers: Producers are clients that write data to a topic.
- Consumers: Consumers are clients that read data from a topic.
- Brokers: Brokers are the nodes in a Kafka cluster that receive data from producers and serve data to consumers.
example of how to produce and consume records in Java:
- Producing the records
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("acks", "all"); props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); Producer<String, String> producer = new KafkaProducer<>(props); for (int i = 0; i < 100; i++) { producer.send(new ProducerRecord<>("topic-name", Integer.toString(i), Integer.toString(i))); } producer.close(); |
2. Consuming the Record:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("group.id", "test"); props.put("enable.auto.commit", "true"); props.put("auto.commit.interval.ms", "1000"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props); consumer.subscribe(Arrays.asList("topic-name")); while (true) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100)); for (ConsumerRecord<String, String> record : records) { System.out.println("offset: "+ record.offset()+ "key: " +record.key()+ "value: " +record.value()); } } |
go to next tutorial for complete example of java-maven-kafka implementation
What’s up fгiends, its enormous piece ⲟf writing regarding tutoringand ϲompletely defined,
keep it Õ½p all the time.