> Markdown version of [/videos/538-event-messaging-and-streaming-with-apache-pulsar?t=494](https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar?t=494). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Event Messaging and Streaming with Apache Pulsar

Does your streaming architecture suffer from scaling bottlenecks? Learn how Apache Pulsar's separation of compute and storage enables robust, low-latency event messaging at global scale.

- **Speakers:** [Mary Grygleski](https://www.wearedevelopers.com/@mary-grygleski)
- **Event:** WeAreDevelopers LIVE
- **Published:** April 18, 2023
- **Duration:** 57:19
- **URL:** https://www.wearedevelopers.com/videos/538-event-messaging-and-streaming-with-apache-pulsar

## Summary

As organizations require immediate reactions to shifting data states, modern architectures are moving away from traditional batch processing and ETL toward real-time event messaging and streaming. Event computing fundamentally treats data as immutable occurrences across space and time, enabling asynchronous, high-throughput workflows. Understanding the distinction between event-driven publication and message-driven queuing is crucial for designing responsive, concurrent systems that imitate real-world data interactions.

Apache Pulsar emerges as a highly scalable, cloud-native solution for next-generation enterprise messaging. Originally developed by Yahoo, Pulsar's defining differentiator is its architectural separation of compute and storage. Stateless Pulsar brokers manage computational routing while Apache BookKeeper handles underlying log storage, allowing teams to scale computing and log capacity independently. This decoupled design guarantees persistent message delivery, significantly lowers cluster operational complexity, and facilitates robust horizontal scaling without introducing performance bottlenecks.

Out of the box, Pulsar provides a suite of advanced developer features, including built-in geo-replication, multi-tenancy for isolated data management, and tiered offloads to seamlessly push older topics into cold storage. Through Pulsar Functions, developers can build lightweight, serverless data pipelines to process, enrich, and mediate streams without deploying external computing frameworks. By supporting customizable subscription modes—exclusive, shared, and failover—alongside native schema evolution tracking, teams can reliably route high-frequency metrics to diverse data sinks, empowering low-latency complex event processing at a global scale.

**Keywords:** apache pulsar event streaming, real-time data pipelines, cloud-native message broker, pub-sub event-driven architecture, message queuing methodologies, apache bookkeeper log storage, decoupled compute and storage, tiered cluster storage offloads, lightweight serverless transformations, pulsar functions stream enrichment, geo-replication deployment, stateless message routing, structured schema management, multi-tenant streaming infrastructure, complex event ingestion

## Chapters

1. **Introducing data management and the shift to streaming** (00:05) — Reviewing background experience with scalable databases and defining the agenda for distributed event processing.
1. **Understanding data in motion and foundational messaging concepts** (04:35) — Recognizing how continuous event streaming represents changing data states sequentially across space and time.
1. **Clarifying terminology around different event computing software forms** (08:14) — Distinguishing between complex architectures like event sourcing, event-driven design, and localized reactive systems.
1. **Defining complex event processing and actionable continuous streams** (12:38) — Analyzing data series to identify real-time patterns for targeted operations like fraud detection.
1. **Comparing pub-sub architectures with message queuing point delivery** (16:37) — Analyzing differences between topic-based message broadcasting and queue-based direct consumption for system services.
1. **Recognizing architectural drivers pushing event streaming system adoption** (21:21) — Highlighting the need for real-time memory processing and immediate data ingestion over traditional batch loads.
1. **Evaluating cloud-native fundamentals within the pulsar open ecosystem** (25:36) — Unpacking the decoupled compute and storage framework supporting massive scale and automated multi-tenant workloads.
1. **Diving into tiered storage operations and horizontal scaling** (31:59) — Leveraging distinct stateless brokers and localized log managers to spread network loads evenly.
1. **Highlighting core enterprise differentiators and adaptable subscription parameters** (35:37) — Supporting active geo-replication constraints and configuring explicit consumption models tailored to multi-tenant environments.
1. **Transforming data pipelines natively through lightweight serverless functions** (39:26) — Enriching raw data dynamically using integrated schema registries and versatile input-output platform connectors.
1. **Managing cloud clusters and validating provisioned streaming endpoints** (42:57) — Creating designated database environments and verifying active pipeline connections through a centralized graphical framework.
1. **Securing topic access and prioritizing inclusive community involvement** (46:49) — Discussing localized token authentication practices and outlining the advantages of contributing actively to software communities.

## Related Moments

- [Comparing cloud-based message broker and event stream solutions](https://www.wearedevelopers.com/videos/395-what-is-a-message-queue-and-when-and-why-would-i-use-it) (from "What is a Message Queue and when and why would I use it?")
- [Moving from traditional databases to decoupled event streaming](https://www.wearedevelopers.com/videos/91-from-event-streaming-to-event-sourcing-101) (from "From event streaming to event sourcing 101")
- [Core concepts of Apache Kafka and topic topologies](https://www.wearedevelopers.com/videos/168-kafka-streams-microservices) (from "Kafka Streams Microservices")
- [Processing asynchronous event streams via standard messaging patterns](https://www.wearedevelopers.com/videos/1111-rust-beyond-systems-revolutionizing-web-development) (from "Rust Beyond Systems: Revolutionizing Web Development")
- [Modernizing legacy applications for real-time streaming data consumption](https://www.wearedevelopers.com/videos/1367-unlocking-value-from-data-the-key-to-smarter-business-decisions) (from "Unlocking Value from Data: The Key to Smarter Business Decisions-")
- [Overview of the Kusto big data analytics platform](https://www.wearedevelopers.com/videos/1543-from-tables-to-graphs-in-minutes-supercharging-kusto-graph-analytics-with-ai-powered-development) (from "From Tables to Graphs in Minutes: Supercharging Kusto Graph Analytics with AI-Powered Development")

## Related Articles

- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [MLops – Deploying, Maintaining And Evolving Machine Learning Models in Production](https://www.wearedevelopers.com/magazine/115-mlops-deploying-maintaining-and-evolving-machine-learning-models-in-production)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Software Engineer](https://www.wearedevelopers.com/jobs/ext/1304791-software-engineer) at **Bitpanda**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/209570-software-engineer-l2-segment-team) at **Twilio**
- [Software Engineer (L2) Segment Team](https://www.wearedevelopers.com/jobs/ext/238794-software-engineer-l2-segment-team) at **Twilio**
- [Artificial Intelligence (AI)](https://www.wearedevelopers.com/jobs/ext/1952055-artificial-intelligence-ai) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/486993-software-engineer-l3-data-substrate) at **Twilio**
- [Software Engineer (L3) Data Substrate](https://www.wearedevelopers.com/jobs/ext/487210-software-engineer-l3-data-substrate) at **Twilio**