Skip to main content

Your submission was sent successfully! Close

Thank you for signing up for our newsletter!
In these regular emails you will find the latest updates from Canonical and upcoming events where you can meet our team.Close

Thank you for contacting us. A member of our team will be in touch shortly. Close

What is Kafka?

Apache Kafka is a free, open source event streaming platform that enables you to create queues for data buffering.

Kafka is typically used to help build streaming data pipelines for distributed applications that continuously process data. It offers functionality for storing, processing and exposing event data streams.

Download the Kafka reference architecture guide Learn more more about Apache Kafka ›

The benefits of Kafka

Kafka provides critical functionality within modern
distributed systems engineering:

Produce and consume streams of event data

Hold event data in the order in which it was produced

Process continuous event data streams

Why choose Kafka?

  • Clustered for resilience

  • Scale out for capacity

  • Broad ecosystem of tools

Why do companies use Kafka?

Proven solution

Apache Kafka has been battle tested in real world deployments around the globe.

Massive scalability

Kafka is designed to run at web scale – with horizontal scaling capabilities built in.

Large user community

Kafka is a mature, actively developed project and has a vast user community.

How do companies use Kafka?

Streaming data

You can use Kafka to buffer and distribute continuous data streams, for example to distribute web clickstream to downstream applications like databases, storage archives and reporting applications.

Microservices

Kafka is a popular choice for enabling microservices architectures, acting as a reliable channel for distributed, inter-process communications.

Analytics

With Kafka, you can integrate analytics processing directly into your data streams, enabling live insights and reporting.

How does Kafka work?

Kafka enables you to group events with a common theme into a logical structure called a topic. Topics are ordered sequences of event data records. Topics are usually replicated across Kafka broker servers in order to ensure event data durability and availability.

Producers publish events to a given topic. Consumers subscribe to a topic and read the events that have been published to the topic. Many producers can publish events to a given topic.

Likewise, many consumers can consume from a given topic. Kafka will track their read position in the sequence. Multiple consumer processes can be deployed to operate in a coordinated, parallel manner in order to scale reading from a topic by using Kafka's consumer groups feature.

In addition to the core functionality, Kafka includes the Kafka Streams API, which enables users to develop data processing and analytics applications that run directly on the Kafka brokers. The Kafka ecosystem also includes Kafka Connect, an integration system for connecting all kinds of technologies to Kafka, from relational databases and other messaging technologies, to object storage systems.

Feature breakdown

Horizontally scalable

Kafka clusters can be scaled to increase processing capacity by adding additional servers, a huge improvement over the vertical scalability limitations of other message brokers.

Distributed processing

Kafka workloads are distributed across many servers, which can massively improve performance by spreading concurrent client requests across the cluster.

Durable

Kafka is designed to replicate copies of your event data across the cluster, so that data loss is minimised when a server goes offline.

Installing Kafka

Kafka is a distributed system developed in Java, designed for computers running Ubuntu and other Linux distributions.

You can use Canonical's Charmed Kafka solution to deploy a fully supported Kafka service on cloud servers, Kubernetes, or even on your laptop.

Try Charmed Kafka

Kafka security

Apache Kafka includes mature information security controls like TLS wire encryption, authentication and authorisation and access logging.

Canonical's Charmed Kafka

Charmed Kafka delivers up to 10 years of support and security maintenance for Apache Kafka as an integrated, turnkey solution with advanced management features.

Charmed Kafka

Included in Ubuntu Pro + Support

When you purchase an Ubuntu Pro + Support plan, you also get support for the full Charmed Kafka solution.

  • Up to 10 years of Kafka support per release track
  • 24/7 or weekday phone and ticket support
  • Up to 10 years of security maintenance for Kafka covering critical and high severity CVEs

Charmed Kafka allows you to automate deployment and operation of Kafka at web scale in the environment of your choice – on the cloud or in your data centre. Supports deployment to most popular clouds or to CNCF conformant Kubernetes.

Learn more about Apache Kafka ›

Kafka Rock container image

Included in Ubuntu Pro + Support

Also included in Ubuntu Pro + Support, you get support for Canonical's container image for Kafka in Docker Hub and Amazon ECR public – the slimmest, fastest, niftiest Kafka container image on the market, based on Ubuntu LTS. So solid and secure, we call it a Rock.

  • Up to 10 years of support per release track
  • Same 24/7 or weekday phone and ticket support commitment
  • Same 10 years of security maintenance covering critical and high severity CVEs in the image
Try out the Rock ›

Kafka consultancy and support

Advanced professional services for Kafka, when you need them

Get help designing, planning and building and even operating a hyper automated production Kafka service that perfectly fits your needs, with Canonical's expert services.

  • Help with design and build of both production and non-production Kafka environments with Charmed Kafka
  • Managed services for Kafka in your cloud tenancy or data centre, backed by an SLA
  • Firefighting support with a Kafka operations expert, who works alongside your team when crisis hits
Contact us ›

Learn more about Kafka

Apache Kafka is more than just a buzzword in the tech community; it's a critical tool for data processing and management. Learn more in our webinar

Access our webinar ›

Kafka resources