diff --git a/README.md b/README.md index b404b592..c5278495 100644 --- a/README.md +++ b/README.md @@ -82,7 +82,7 @@ defmodule MyBroadway do end ``` -Once your Broadway module is defined, you just need to add it as a child of your application supervision tree as `{MyBroadway, []}`. +Once your Broadway module is defined, add it as a child of your application supervision tree as `{MyBroadway, []}`. ## Comparison to Flow diff --git a/guides/examples/amazon-sqs.md b/guides/examples/amazon-sqs.md index 67479f6d..96894552 100644 --- a/guides/examples/amazon-sqs.md +++ b/guides/examples/amazon-sqs.md @@ -20,7 +20,7 @@ Broadway can work seamlessly with both, Standard and FIFO queues. ## Getting Started -In order to use Broadway with SQS, we need to: +To use Broadway with SQS, we need to: 1. Create a SQS queue (or use an existing one) 1. Configure our Elixir project to use Broadway @@ -43,7 +43,7 @@ which is a Broadway SQS Connector provided by [Dashbit](https://dashbit.co/). ### Starting a new project -If you plan to start a new project, just run: +If you plan to start a new project, run: $ mix new my_app --sup @@ -71,7 +71,7 @@ pipeline, we need to define three functions: `start_link/1`, `handle_message/3` and `handle_batch/4`. We will cover `start_link/1` in this section and the `handle_` callbacks in the next one. -Similar to other process-based behaviour, `start_link/1` simply +Similar to other process-based behaviour, `start_link/1` delegates to `Broadway.start_link/2`, which should define the producers, processors, and batchers in the Broadway pipeline. Assuming we want to consume messages from a queue called @@ -135,9 +135,9 @@ module docs as well as `Broadway.start_link/2`. ## Implement Broadway callbacks -In order to process incoming messages, we need to implement the +To process incoming messages, we need to implement the required callbacks. For the sake of simplicity, we're considering that -all messages received from the queue are just numbers: +all messages received from the queue are numbers: defmodule MyBroadway do use Broadway @@ -169,7 +169,7 @@ For more information, see `c:Broadway.handle_message/3` and ## Run the Broadway pipeline -To run your `Broadway` pipeline, you just need to add as a child in +To run your `Broadway` pipeline, add it as a child in a supervision tree. Most applications have a supervision tree defined at `lib/my_app/application.ex`. You can add Broadway as a child to a supervisor as follows: @@ -233,7 +233,7 @@ your needs. ...callbacks... end -In order to get a good set of configurations for your pipeline, it's +To get a good set of configurations for your pipeline, it's important to respect the limitations of the servers you're running, as well as the limitations of the services you're providing/consuming data to/from. Broadway comes with telemetry, so you can measure your diff --git a/guides/examples/apache-kafka.md b/guides/examples/apache-kafka.md index 97f9477b..084ef6f6 100644 --- a/guides/examples/apache-kafka.md +++ b/guides/examples/apache-kafka.md @@ -8,7 +8,7 @@ Kafka is a distributed streaming platform that has three key capabilities: ## Getting Started -In order to use Broadway with Kafka, we need to: +To use Broadway with Kafka, we need to: 1. Create a stream of records (or use an existing one) 1. Configure your Elixir project to use Broadway @@ -69,7 +69,7 @@ we need to define three functions: `start_link/1`, `handle_message/3` and optionally `handle_batch/4`. We will cover `start_link/1` in this section and the `handle_` callbacks in the next one. -Similar to other process-based behaviours, `start_link/1` simply +Similar to other process-based behaviours, `start_link/1` delegates to `Broadway.start_link/2`, which should define the producers, processors, and batchers in the Broadway pipeline. Assuming we want to consume messages from a topic called @@ -125,9 +125,9 @@ module docs as well as `Broadway.start_link/2`. ## Implement Broadway callbacks -In order to process incoming messages, we need to implement the +To process incoming messages, we need to implement the required callbacks. For the sake of simplicity, we're considering that -all messages received from the topic are just numbers: +all messages received from the topic are numbers: defmodule MyBroadway do use Broadway @@ -163,7 +163,7 @@ For more information, see `c:Broadway.handle_message/3` and ## Run the Broadway pipeline -To run your `Broadway` pipeline, you just need to add as a child in +To run your `Broadway` pipeline, add it as a child in a supervision tree. Most applications have a supervision tree defined at `lib/my_app/application.ex`. You can add Broadway as a child to a supervisor as follows: @@ -242,7 +242,7 @@ for details. By setting the `concurrency` option, you define the number of concurrent processes that will be started by Broadway, allowing you to have full control over the concurrency level in each layer of the pipeline. Keep in mind that since the -concurrency model provided by **Kafka** is based on **partitioning**, in order to take +concurrency model provided by **Kafka** is based on **partitioning**, to take full advantage of this model, you need to set the `concurrency` option for your processors and batchers accordingly. Having less concurrency than topic/partitions assigned will result in individual processors handling more than one partition, diff --git a/guides/examples/google-cloud-pubsub.md b/guides/examples/google-cloud-pubsub.md index 2dbbc594..44f395ae 100644 --- a/guides/examples/google-cloud-pubsub.md +++ b/guides/examples/google-cloud-pubsub.md @@ -4,7 +4,7 @@ Cloud Pub/Sub is a fully-managed real-time messaging service provided by Google. ## Getting Started -In order to use Broadway with Cloud Pub/Sub you need to: +To use Broadway with Cloud Pub/Sub you need to: 1. Setup a Cloud Pub/Sub project 1. Configure your Elixir project to use Broadway @@ -13,7 +13,7 @@ In order to use Broadway with Cloud Pub/Sub you need to: 1. Run the Broadway pipeline 1. Tune the configuration (Optional) -If you are just getting familiar with Google Pub/Sub, refer to [the documentation](https://cloud.google.com/pubsub/docs/) +If you are getting familiar with Google Pub/Sub, refer to [the documentation](https://cloud.google.com/pubsub/docs/) to get started. Instead of testing against a live environment, you may also consider using the [emulator](https://cloud.google.com/pubsub/docs/emulator) to simulate integrating with Cloud Pub/Sub. @@ -46,7 +46,7 @@ A new topic: $ gcloud pubsub topics create test-topic --project test-pubsub Created topic [projects/test-pubsub/topics/test-topic]. -> Note: If you run this command immediately after creating a new Google Cloud project, you may receive an error indicating that your project's organization policy is still being provisioned. Just wait a couple minutes and try again. +> Note: If you run this command immediately after creating a new Google Cloud project, you may receive an error indicating that your project's organization policy is still being provisioned. Wait a couple minutes and try again. And a new subscription: @@ -54,7 +54,7 @@ And a new subscription: Created subscription [projects/test-pubsub/subscriptions/test-subscription]. We also need a [service account](https://cloud.google.com/iam/docs/service-accounts), an IAM -policy, as well as API credentials in order to programmatically work with the service. First, let's +policy, as well as API credentials to programmatically work with the service. First, let's create the service account: $ gcloud iam service-accounts create test-account --project test-pubsub @@ -91,7 +91,7 @@ which is a Broadway Cloud Pub/Sub Connector provided by [Dashbit](https://dashbi ### Starting a new project -If you plan to start a new project, just run: +If you plan to start a new project, run: $ mix new my_app --sup @@ -118,7 +118,7 @@ Broadway is a process-based behaviour and to define a Broadway pipeline, we need functions: `start_link/1`, `handle_message/3` and `handle_batch/4`. We will cover `start_link/1` in this section and the `handle_` callbacks in the next one. -Similar to other process-based behaviour, `start_link/1` simply delegates to +Similar to other process-based behaviour, `start_link/1` delegates to `Broadway.start_link/2`, which should define the producers, processors, and batchers in the Broadway pipeline. Assuming we want to consume messages from the `test-subscription`, the minimal configuration would be: @@ -164,7 +164,7 @@ For general information about setting up Broadway, see `Broadway` module docs as ## Implement Broadway callbacks -In order to process incoming messages, we need to implement the required callbacks. For the sake +To process incoming messages, we need to implement the required callbacks. For the sake of simplicity, we're considering that all messages received from the queue are strings and our processor calls `String.upcase/1` on them: @@ -285,7 +285,7 @@ your needs. ...callbacks... end -In order to get a good set of configurations for your pipeline, it's +To get a good set of configurations for your pipeline, it's important to respect the limitations of the servers you're running, as well as the limitations of the services you're providing/consuming data to/from. Broadway comes with telemetry, so you can measure your diff --git a/guides/examples/rabbitmq.md b/guides/examples/rabbitmq.md index f9ee8656..0a723180 100644 --- a/guides/examples/rabbitmq.md +++ b/guides/examples/rabbitmq.md @@ -6,7 +6,7 @@ Queuing Protocol (AMQP). ## Getting Started -In order to use Broadway with RabbitMQ, we need to: +To use Broadway with RabbitMQ, we need to: 1. [Create a queue](#create-a-queue) (or use an existing one) 1. [Configure our Elixir project to use Broadway](#configure-the-project) @@ -35,7 +35,7 @@ following command: $ rabbitmqadmin declare queue name=my_queue durable=true -You can list all declared queues to see our the one we've just created: +You can list all declared queues to see the one we created: $ rabbitmqctl list_queues Timeout: 60.0 seconds ... @@ -76,7 +76,7 @@ we need to define three functions: `start_link/1`, `handle_message/3` and optionally `handle_batch/4`. We will cover `start_link/1` in this section and the `handle_` callbacks in the next one. -Similar to other process-based behaviours, `start_link/1` simply +Similar to other process-based behaviours, `start_link/1` delegates to `Broadway.start_link/2`, which should define the producers, processors, and batchers in the Broadway pipeline. Assuming we want to consume messages from a queue called @@ -141,9 +141,9 @@ module docs as well as `Broadway.start_link/2`. ## Implement Broadway callbacks -In order to process incoming messages, we need to implement the +To process incoming messages, we need to implement the required callbacks. For the sake of simplicity, we're considering that -all messages received from the queue are just numbers: +all messages received from the queue are numbers: defmodule MyBroadway do use Broadway @@ -181,7 +181,7 @@ For more information, see `c:Broadway.handle_message/3` and ## Run the Broadway pipeline -To run your `Broadway` pipeline, you just need to add as a child in +To run your `Broadway` pipeline, add it as a child in a supervision tree. Most applications have a supervision tree defined at `lib/my_app/application.ex`. You can add Broadway as a child to a supervisor as follows: @@ -260,7 +260,7 @@ The `:prefetch_count` setting provides back-pressure by instructing RabbitMQ to See the ["Back-pressure and :prefetch_count"](https://hexdocs.pm/broadway_rabbitmq/BroadwayRabbitMQ.Producer.html#module-back-pressure-and-prefetch_count) section of the `BroadwayRabbitMQ` documentation for details. -In order to get a good set of configurations for your pipeline, it's +To get a good set of configurations for your pipeline, it's important to respect the limitations of the servers you're running, as well as the limitations of the services you're providing/consuming data to/from. Broadway comes with telemetry, so you can measure your diff --git a/guides/internals/architecture.md b/guides/internals/architecture.md index e1073e11..2b71ea55 100644 --- a/guides/internals/architecture.md +++ b/guides/internals/architecture.md @@ -32,7 +32,7 @@ handle_batch/4 runs here -> [batch processor][batch processor] calculations, convert data into a custom json format etc. Here is where the code from `handle_message/3` runs. * `Broadway.Batcher` - Creates batches of messages based on the - batcher's key. One batcher for each key will be created. + batcher's key. Broadway creates one batcher for each key. * `Broadway.BatchProcessor` - This is where the code from `handle_batch/4` runs. ## The supervision tree diff --git a/lib/broadway.ex b/lib/broadway.ex index 4b903146..865149d6 100644 --- a/lib/broadway.ex +++ b/lib/broadway.ex @@ -63,7 +63,7 @@ defmodule Broadway do ## The Broadway behaviour - In order to use Broadway, you need to: + To use Broadway, you need to: 1. Define your pipeline configuration 2. Define a module implementing the Broadway behaviour @@ -530,7 +530,7 @@ defmodule Broadway do This can be done with the `:partition_by` option, which enforces that messages with a given property are always forwarded to the same stage. - In order to provide partitioning throughout the whole pipeline, just + To provide partitioning throughout the whole pipeline, set `:partition_by` at the root of your configuration: defmodule MyBroadway do @@ -929,7 +929,7 @@ defmodule Broadway do `min_demand` value. Producers which are push-based, rather than pull-based, such as `BroadwayRabbitMQ.Producer`, are more likely to send messages as they arrive (which may skip batching altogether and always be single element lists). - In other words, this callback is simply a convenience for preparing messages, + In other words, this callback is a convenience for preparing messages, it does not guarantee the messages will be accumulated to a certain length. For effective batch processing, see `c:handle_batch/4`. @@ -959,10 +959,10 @@ defmodule Broadway do This is the place to do any kind of processing with the incoming message, e.g., transform the data into another data structure, call specific business - logic to do calculations. Basically, any CPU bounded task that runs against + logic to do calculations. Any CPU bounded task that runs against a single message should be processed here. - In order to update the data after processing, use the + To update the data after processing, use the `Broadway.Message.update_data/2` function. This way the new message can be properly forwarded and handled by the batcher: @@ -1127,7 +1127,7 @@ defmodule Broadway do ## Options - In order to set up how the pipeline created by Broadway should work, + To set up how the pipeline created by Broadway should work, you need to specify the blueprint of the pipeline. You can do this by passing a set of options to `start_link/2`. Each component of the pipeline has its own set of options. diff --git a/lib/broadway/acknowledger.ex b/lib/broadway/acknowledger.ex index cf697549..daff6fae 100644 --- a/lib/broadway/acknowledger.ex +++ b/lib/broadway/acknowledger.ex @@ -7,7 +7,7 @@ defmodule Broadway.Acknowledger do implement this behaviour and consider how the technology you're working with handles message acknowledgement. - The `c:ack/3` callback must be implemented in order to notify + The `c:ack/3` callback must be implemented to notify the origin of the data that a message can be safely removed after been successfully processed and published. In case of failed messages or messages without acknowledgement, depending