Kafka, as a cornerstone of real-time data processing, is foundational in many modern application architectures. But, like any system, it's not without its challenges. This presentation explores common pitfalls developers encounter with Kafka. We will not only shine a light on these problematic areas but also introduce proven troubleshooting techniques. A highlight of our discussion will be a live demonstration of connecting, consuming, and debugging Kafka using a Python Flask app. We'll also briefly discuss how OpenTelemetry can provide additional visibility into Kafka-based workflows. By integrating the demonstration with a Python Flask app, we'll showcase tangible ways in which developers can interact with and troubleshoot Kafka, ensuring more reliable deployments and a better understanding of this powerful data-streaming platform.

Domain Driven Design teaches us to separate the repository (our data storage inferface) from the rest of the model in order to prevent our storage interface from contaminating the rest of our domain model.So, with the data layer successfully separated from the rest of the domain model into its own repository, now comes the time to test it - a task that may seem rather daunting. In this talk I will use Test Driven Development to show how to use mocking frameworks to mock datastores at both the API-, and server-level, and show techniques to test against a real database as well as how we can make the tests more portable by leveraging containers when appropriate.

Although machine learning is a broad subject with many different applications, there are key aspects that should be considered for any ML-algorithm.

For example, before applying ML the domain and data must be analyzed, and additional processing may be required.

In this talk, I'll present some common machine learning paradigms using scikit-learn. Data preparation, decision trees, random forests, neural networks, training and evaluation of supervised learning will all be covered.

This presentation will primarily serve as an introduction to machine learning in general and to the respective Python frameworks.

In a few hours we'll instrument a demo application and see what OpenTelmetry can do. We'll also cover basic data filtering and compression.

### OpenTelemetry

* History
* What's in the project
* Use cases
* Status of development - Logs support is still growing, but you can send logs through the collector

### Collector Design

* Most basic - direct from SDK to datastore, don't do this
* Single collector
* Collector -> collector
* Daemonset in Kubernetes (I promise you don't need to know Kubernetes to follow this talk!)

### Sample Application

* Basic data emitter from the CLI
* Auto-instrumentation with the language-appropriate SDK
* Configuring data sends
* A basic dashboard

### Configuring the Collector

* YAML config
* restarting the collector

### OpenTelemetry Processors - config

* batching - reducing sends
* parsing - finding data early
* crushing cardinality - getting the data you need
* filtering - removing sensitive data
* transforming with OTTL