Python with Kafka and AWS MSK: A Practical Guide

Abstract

This comprehensive guide explores the powerful combination of Python and Apache Kafka for building robust event streaming applications, with a special focus on deploying and managing Kafka on AWS using Amazon MSK. Starting with foundational concepts and progressing to advanced techniques, this book equips readers with the knowledge and skills to design, implement, and operate high-performance, scalable, and resilient event-driven systems using modern Kafka features including KRaft mode.

Audience

This book is designed for:

• Python developers seeking to build event-driven applications
• Data engineers working with streaming data pipelines
• Cloud architects designing AWS-based event streaming solutions
• DevOps engineers responsible for Kafka cluster operations
• Software engineers interested in distributed systems and event-based architectures

Book Structure

Important Note: All code examples throughout this book will be presented in both confluent-kafka and kafka-python libraries to provide comprehensive coverage and practical implementation options.

Topics Covered

Part I: Foundations

1: Understanding Event Streaming with Kafka

• Event-driven architecture principles
• Kafka's role in modern data pipelines
• Key Kafka concepts: topics, partitions, consumer groups
• Advantages of event streaming with Python