The goal of this GitHub repository is to demonstrate with concrete code examples of how Apache Pulsar can be used as a powerful and unified platform for common messaging and streaming processing use cases.
This repository is composed of a series of demo scenarios (or simply demos). Each demo is a self-contained unit that covers a complete, end-to-end messaging/streaming processing use case using a specific message processing protocol that is supported by Pulsar.
The majority of the demos are organized under the subfolder of scenarios, and they're then further grouped into following categories based on the underlying API and/or messaging protocols being used.
- Native Pulsar API and protocol (
native-pulsar) - JMS specification with Starlight for JMS (S4J) API (
jms-s4j) - Kafka protocol with Starlight for Kafka (S4J) protocol handler (
kafka-s4k) - AMQP protocol with Starlight for RabbitMQ (S4R) protocol handler (
rabbitmq-s4r)
There are also a subset of the demos that are organized under another subfolder of spring-scenarios. These demos are written using the Java Spring Boot framework for Pulsar. Because this framework requires JDK17 as the minimum version requirement. This is different from the rest of the demos that run at JDK11. (Please see Build. Programs for more details)
Running the scenario require the following software to be installed on your local computer:
JDK 11(forscenarios) andJDK17(forspring-scenarios)curlutility
Each demo has its own set of programs to showcase how message publishing/producing/sending or subscribing/consuming/receiving works in that particular demo. In order to run these demos, we first need to build their programs Please refer to the document of Build Scenario Programs for more details.
Each of the demos included in this repository has 3 standard bash scripts that can be found under the _bash sub-folder of each demo.
deploy.shis used to deploy the Pulsar resources (tenant, namespace, topic, function, etc.) as needed in the demo.runProducer.shis used to publish/send/produce messages to one Pulsar topicrunConsumer.shis used to subscribe/receive/consume messages from one Pulsar topic
We first need to run deploy.sh in order to deploy the Pulsar resources (tenant, namespace, topic, function, etc.) that are needed for a particular demo. The deploy.sh has the standard input command line input parameters; but they will deploy different sets of Pulsar resources that are unique to each demo. For more details, please see the document of Deploy Demos.
After that, we can either run runProducer.sh to publish/produce/send messages or run runConsumer.sh to subscribe/consume/receive messages. These 2 scripts take some common command line input parameters but may be slightly different from demo to demo. For the general description of running a demo, please see the document of Run Demos. The demo specific execution steps will be explained in details under each demo.
| API | Demo Name | Description |
|---|---|---|
| Native Pulsar | native-pulsar/message-enrichment | Demonstrate how an IoT sensor reading raw data can be published and enriched in Pulsar before consumed by a consumer. |
| Native Pulsar | native-pulsar/message-enrichment-avro | Similar to the message-enrichment scenario. The only difference is this scenario demonstrates the message processing with AVRO/JSON schema instead of as pure text. |
| Native Pulsar | native-pulsar/message-redelivery | Demonstrate how message redelivery is handled in Pulsar. |
| JMS + S4J | jms-s4j/p2p-basic | Demonstrate basic JMS message sending and receiving with a JMS Queue using S4J API and Pulsar |
| JMS + S4J | jms-s4j/pubsub-basic | Demonstrate basic JMS message publishing and subscribing with a JMS Topic using S4J API for Pulsar |
| Kafka + S4K | kafka-s4k/pubsub-basic | Demonstrate basic Kafka message producing and consuming with a Kafka Topic with Pulsar's S4K protocol handler |
| RabbitMQ + S4R | rabbitmq-s4r/pubsub-queue | Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Queue using S4R API and Pulsar |
| RabbitMQ + S4R | rabbitmq-s4r/fanout-exchange | Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Fanout Exchange using S4R API and Pulsar |
| RabbitMQ + S4R | rabbitmq-s4r/topic-exchange | Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Topic Exchange and Routing Key using S4R API and Pulsar |
| RabbitMQ + S4R | rabbitmq-s4r/direct-exchange | Demonstrate RabbitMQ message producing and consuming with a RabbitMQ Direct Exchange and Routing Key using S4R API and Pulsar |
| Spring Boot Demo Name | Description |
|---|---|
| spb-pulsar-pubsub-basic | Demonstrate basic message publishing and consuming using Spring boot for Pulsar framework |