diff --git a/README.md b/README.md index a7df5f0a..34ec44ed 100644 --- a/README.md +++ b/README.md @@ -12,10 +12,6 @@ This repository houses example projects and code for products in the [Buf][buf] [This directory](./bsr) contains quickstart code that demonstrates key functions and usage of the [BSR][bsr], and how to create [Buf plugins][plugins] for custom lint and breaking change rules that you can upload to the BSR. -## Bufstream - -A [quickstart guide](bufstream/iceberg-quickstart) for [Bufstream][bufstream] demonstrates how to archive Kafka messages to Iceberg tables and perform analytical queries with Spark. - ## Protovalidate [Quickstart guides and examples](./protovalidate) for [Protovalidate][protovalidate] show how standard annotations and custom CEL expressions can be used to validate Protobuf messages. @@ -27,7 +23,6 @@ A [quickstart guide](bufstream/iceberg-quickstart) for [Bufstream][bufstream] de [bsr-docs]: https://buf.build/docs/bsr/ [bsr]: https://buf.build [buf]: https://github.com/bufbuild -[bufstream]: https://buf.build/docs/bufstream/ [cli-docs]: https://buf.build/docs/cli/ [docs]: https://buf.build/docs [plugins]: https://buf.build/docs/cli/buf-plugins/overview/ diff --git a/bufstream/docker-compose/.gitignore b/bufstream/docker-compose/.gitignore deleted file mode 100644 index 177afaf9..00000000 --- a/bufstream/docker-compose/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -minio -postgres diff --git a/bufstream/docker-compose/README.md b/bufstream/docker-compose/README.md deleted file mode 100644 index 9bfc3f12..00000000 --- a/bufstream/docker-compose/README.md +++ /dev/null @@ -1,58 +0,0 @@ -![The Buf logo](https://raw.githubusercontent.com/bufbuild/buf-examples/main/.github/buf-logo.svg) - -# Docker Compose for Bufstream - -This directory contains an example `docker-compose.yml` file that sets up all required infrastructure for a [Bufstream][bufstream] environment suitable for local testing and development. - -## Prerequisites - -* [Docker][docker] must be installed. -* If you're using Docker Desktop, [host mode networking][docker-host-mode-networking] must be enabled. - -## Starting the project - -Just run: - -``` -$ docker compose up -``` - -## Services - -This Compose project will start the following services: - -- MinIO (https://min.io/) for S3 compatible storage. - * Its API endpoint is available at http://localhost:9000. - * Its admin endpoint is available at http://localhost:9001. -- An AWS CLI sidecar that bootstraps an initial bucket. -- PostgreSQL (https://www.postgresql.org/) for metadata storage. - * It is available on the standard Postgres port (5432). -- A Bufstream broker. - * Its Kafka endpoint is available at localhost:9092. - * Its admin endpoint listens on its default port (9089), allowing you to run [`bufstream admin`](https://buf.build/docs/bufstream/reference/cli/admin/) commands against the broker. - -Note that the MinIO image is provided by [lithus](https://github.com/golithus/minio-builds). [MinIO ceased publishing public Docker images](https://github.com/minio/minio/issues/21647) in October 2025. - -## Memory - -The Compose project uses the `BUFSTREAM_AVAILABLE_MEMORY` environment variable to limit Bufstream to 1 GiB of memory. -This should be suitable for simple local development and testing, but may be insufficient for high-throughput workloads or performance testing. - -If you encounter out-of-memory issues, increase the value in the `docker-compose.yml` file. For reference, production Kubernetes deployments default to 8 GiB (8589934592 bytes). - -## Volumes - -When this Compose project starts, it will create two directories: - -1. `minio`: A volume used by the `minio` service for object storage. -2. `postgres`: A volume used by the `postgres` service for its database storage. - -Before committing this `docker-compose.yml` file to your own repository, add these to your `.gitignore`. - -## Networks - -This example uses host networking. You may encounter issues if there are port conflicts. If you need to use port mappings, see [Docker's documentation](https://docs.docker.com/compose/how-tos/networking/). - -[bufstream]: https://buf.build/product/bufstream -[docker]: https://docs.docker.com/engine/install/ -[docker-host-mode-networking]: https://docs.docker.com/engine/network/drivers/host/#docker-desktop diff --git a/bufstream/docker-compose/bufstream.yaml b/bufstream/docker-compose/bufstream.yaml deleted file mode 100644 index 58c97290..00000000 --- a/bufstream/docker-compose/bufstream.yaml +++ /dev/null @@ -1,15 +0,0 @@ -version: v1beta1 -data: - s3: - # URI of the storage bucket and prefix, e.g. s3://my-bucket/my-prefix/ - uri: s3://bufstream/ - region: example-region - endpoint: http://localhost:9000 - force_path_style: true - access_key_id: - string: minioadmin - secret_access_key: - string: minioadmin -metadata: - postgres: - string: postgresql://root:password@localhost:5432/bufstream diff --git a/bufstream/docker-compose/docker-compose.yml b/bufstream/docker-compose/docker-compose.yml deleted file mode 100644 index 83c57c90..00000000 --- a/bufstream/docker-compose/docker-compose.yml +++ /dev/null @@ -1,49 +0,0 @@ -services: - # Bufstream uses PostgreSQL 14+ (https://www.postgresql.org/) for storing cluster metadata. - postgres: - image: postgres:14 - environment: ["POSTGRES_USER=root", "POSTGRES_PASSWORD=password", "POSTGRES_DB=bufstream"] - network_mode: host - volumes: ["./postgres:/var/lib/postgresql/data"] - healthcheck: - test: ["CMD", "pg_isready", "-d", "bufstream"] - interval: 2s - retries: 30 - - # MinIO provides local S3-compatible object storage. - minio: - image: ghcr.io/golithus/minio:latest - network_mode: host - volumes: ["./minio:/data"] - command: ["server", "/data", "--console-address", ":9001"] - - # create-buckets uses the AWS CLI to create a bucket. - create-buckets: - image: amazon/aws-cli - network_mode: host - environment: - AWS_ACCESS_KEY_ID: minioadmin - AWS_SECRET_ACCESS_KEY: minioadmin - AWS_EC2_METADATA_DISABLED: "true" # Skip delays from metadata lookups. - entrypoint: /bin/sh - command: ["-c", "until curl -sf http://localhost:9000/minio/health/live > /dev/null; do sleep 2; done && aws --endpoint-url http://localhost:9000 s3 mb s3://bufstream || true"] - depends_on: - - minio - - # The Bufstream broker, available at localhost:9092. - bufstream: - image: bufbuild/bufstream:0.4.7 - depends_on: - postgres: { "condition": "service_healthy" } - create-buckets: { "condition": "service_completed_successfully" } - network_mode: host - hostname: bufstream - volumes: ["./bufstream.yaml:/bufstream.yaml"] - # If you encounter out-of-memory issues, increase this value to suit your - # workload. 1 GiB is reasonable for simple local development and testing. - # Production Kubernetes deployments default to 8 GiB (8589934592). - environment: - - BUFSTREAM_AVAILABLE_MEMORY=1073741824 - healthcheck: - test: ["CMD", "/usr/local/bin/bufstream", "admin", "status", "--exit-code", "--url", "http://127.0.0.1:9089"] - command: ["serve", "--config", "/bufstream.yaml"] diff --git a/bufstream/iceberg-quickstart/README.md b/bufstream/iceberg-quickstart/README.md deleted file mode 100644 index 32ded8d5..00000000 --- a/bufstream/iceberg-quickstart/README.md +++ /dev/null @@ -1,9 +0,0 @@ -![The Buf logo](https://raw.githubusercontent.com/bufbuild/buf-examples/main/.github/buf-logo.svg) - -# Bufstream and Iceberg quickstart - -This example has been consolidated into the [Bufstream quickstart][quickstart]. -Its code is in the [bufstream-demo][repo] repository. - -[quickstart]: https://buf.build/docs/bufstream/quickstart/ -[repo]: https://github.com/bufbuild/bufstream-demo diff --git a/protovalidate/README.md b/protovalidate/README.md index 27500c7e..8c766e74 100644 --- a/protovalidate/README.md +++ b/protovalidate/README.md @@ -4,7 +4,7 @@ [Protovalidate][protovalidate] provides standard annotations to validate common rules on messages and fields, as well as the ability to use [CEL][cel] to write custom rules. It's the next generation of [protoc-gen-validate][protoc-gen-validate], the only widely used validation library for Protobuf. -Supplementing its [documentation][protovalidate], this repository provides quickstart code exercises and reference examples for its use. +Supplementing its [documentation][protovalidate], this repository provides quickstart code exercises and reference examples for its use. ## Protovalidate quickstart @@ -42,10 +42,6 @@ Quickstart code showing how to use Protovalidate in unary RPCs is available in t * [Quickstart code](grpc-java/README.md) for [Protovalidate in gRPC and Java][grpc-java] * [Quickstart code](grpc-python/README.md) for [Protovalidate in gRPC and Python][grpc-python] -## Using Protovalidate in Kafka - -[Quickstart code](bufstream/README.md) for [Using Protovalidate in Kafka with Bufstream][bufstream] shows how to enforce schemas within the Bufstream Kafka broker. The only code changed during the tutorial is in Protobuf and YAML files, so only a Go example is provided. It contains a `start` directory with a starting state and `finish` with a working solution. - [protovalidate]: https://buf.build/docs/protovalidate/ [quickstart]: https://buf.build/docs/protovalidate/quickstart/ [adding-protovalidate]: https://buf.build/docs/protovalidate/schemas/adding-protovalidate/ @@ -56,7 +52,6 @@ Quickstart code showing how to use Protovalidate in unary RPCs is available in t [grpc-go]: https://buf.build/docs/protovalidate/quickstart/grpc-go [grpc-java]: https://buf.build/docs/protovalidate/quickstart/grpc-java [grpc-python]: https://buf.build/docs/protovalidate/quickstart/grpc-python -[bufstream]: https://buf.build/docs/protovalidate/quickstart/bufstream [protoc-gen-validate]: https://github.com/bufbuild/protoc-gen-validate [yup]: https://github.com/jquense/yup