Watch the webinar: Ingesting varied and distributed configuration metadata

More details

Step 1

Install Quix Streams

python3 -m pip install quixstreams

Step 2

Connect to Kafka and process streaming data with DataFrames

from quixstreams import Application
import uuid

# Connect to the public Quix hosted broker to consume data
app = Application(
    broker_address="publickafka.quix.io:9092",  # Kafka broker address
    consumer_group=str(uuid.uuid4()),  # Kafka consumer group
    auto_offset_reset='latest',  # Read topic from the end
    producer_extra_config={'enable.idempotence': False}
)

input_topic = app.topic("demo-onboarding-prod-chat", value_deserializer='json')

sdf = app.dataframe(input_topic)

sdf["tokens_count"] = sdf["message"].apply(lambda message: len(message.split(" ")))  
sdf = sdf[["role", "tokens_count"]]

sdf = sdf.update(lambda row: print(row))

app.run(sdf)

Step 3

Build and test streaming pipelines locally with Quix CLI

Quix CLI is a free terminal tool that enables you to create, debug, and run data pipelines on your local machine. Start building with Quix Streams, Docker and Git on your preferred IDE.

Learn more about Quix Streams

Dive into the Docs

Learn more about the full feature set of Quix Streams in our detailed developer documentation.

Open source Python client library

Support the project by starring the repo, or submit a PR to become a contributor.

Build with our community

Join our Slack to get help with Quix Streams and the Quix platform from our friendly developer community.