|
| 1 | +# clue/mq-react [](https://travis-ci.org/clue/php-mq-react) |
| 2 | + |
| 3 | +Mini Queue, the lightweight in-memory message queue to concurrently do many (but not too many) things at once, |
| 4 | +built on top of [ReactPHP](https://reactphp.org/). |
| 5 | + |
| 6 | +Let's say you crawl a page and find that you need to send 100 HTTP requests to |
| 7 | +following pages which each takes `0.2s`. You can either send them all |
| 8 | +sequentially (taking around `20s`) or you can use |
| 9 | +[ReactPHP](https://reactphp.org) to concurrently request all your pages at the |
| 10 | +same time. This works perfectly fine for a small number of operations, but |
| 11 | +sending an excessive number of requests can either take up all resources on your |
| 12 | +side or may get you banned by the remote side as it sees an unreasonable number |
| 13 | +of requests from your side. |
| 14 | +Instead, you can use this library to effectively rate limit your operations and |
| 15 | +queue excessives ones so that not too many operations are processed at once. |
| 16 | +This library provides a simple API that is easy to use in order to manage any |
| 17 | +kind of async operation without having to mess with most of the low-level details. |
| 18 | +You can use this to throttle multiple HTTP requests, database queries or pretty |
| 19 | +much any API that already uses Promises. |
| 20 | + |
| 21 | +* **Async execution of operations** - |
| 22 | + Process any number of async operations choose how many should be handled |
| 23 | + concurrently and how many operations can be queued in-memory. Process their |
| 24 | + results as soon as responses come in. |
| 25 | + The Promise-based design provides a *sane* interface to working with out of bound results. |
| 26 | +* **Lightweight, SOLID design** - |
| 27 | + Provides a thin abstraction that is [*just good enough*](http://en.wikipedia.org/wiki/Principle_of_good_enough) |
| 28 | + and does not get in your way. |
| 29 | + Builds on top of well-tested components and well-established concepts instead of reinventing the wheel. |
| 30 | +* **Good test coverage** - |
| 31 | + Comes with an automated tests suite and is regularly tested in the *real world* |
| 32 | + |
| 33 | +**Table of contents** |
| 34 | + |
| 35 | +* [Quickstart example](#quickstart-example) |
| 36 | +* [Usage](#usage) |
| 37 | + * [Queue](#queue) |
| 38 | + * [Promises](#promises) |
| 39 | + * [Cancellation](#cancellation) |
| 40 | +* [Install](#install) |
| 41 | +* [Tests](#tests) |
| 42 | +* [License](#license) |
| 43 | + |
| 44 | +> Note: This project is in an early alpha stage! Feel free to report any issues you encounter. |
| 45 | +
|
| 46 | +## Quickstart example |
| 47 | + |
| 48 | +Once [installed](#install), you can use the following code to access an |
| 49 | +HTTP webserver and send a large number of HTTP GET requests: |
| 50 | + |
| 51 | +```php |
| 52 | +$loop = React\EventLoop\Factory::create(); |
| 53 | +$client = new Clue\React\Buzz\Browser($loop); |
| 54 | + |
| 55 | +// load a huge array of URLs to fetch |
| 56 | +$urls = file('urls.txt'); |
| 57 | + |
| 58 | +// each job should use the browser to GET a certain URL |
| 59 | +// limit number of concurrent jobs here |
| 60 | +$q = new Queue(3, null, function ($url) use ($browser) { |
| 61 | + return $browser->get($url); |
| 62 | +}); |
| 63 | + |
| 64 | +foreach ($urls as $url) { |
| 65 | + $q($url)->then(function (ResponseInterface $response) use ($url) { |
| 66 | + echo $url . ': ' . $response->getBody()->getSize() . ' bytes' . PHP_EOL; |
| 67 | + }); |
| 68 | +} |
| 69 | + |
| 70 | +$loop->run(); |
| 71 | +``` |
| 72 | + |
| 73 | +See also the [examples](examples). |
| 74 | + |
| 75 | +## Usage |
| 76 | + |
| 77 | +### Queue |
| 78 | + |
| 79 | +The `Queue` is responsible for managing your operations and ensuring not too |
| 80 | +many operations are executed at once. It's a very simple and lightweight |
| 81 | +in-memory implementation of the |
| 82 | +[leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket#As_a_queue) algorithm. |
| 83 | + |
| 84 | +This means that you control how many operations can be executed concurrently. |
| 85 | +If you add a job to the queue and it still below the limit, it will be executed |
| 86 | +immediately. If you keep adding new jobs to the queue and its concurrency limit |
| 87 | +is reached, it will not start a new operation and instead queue this for future |
| 88 | +execution. Once one of the pending operations complete, it will pick the next |
| 89 | +job from the qeueue and execute this operation. |
| 90 | + |
| 91 | +The `new Queue(int $concurrency, ?int $limit, callable $handler)` call |
| 92 | +can be used to create a new queue instance. |
| 93 | +You can create any number of queues, for example when you want to apply |
| 94 | +different limits to different kind of operations. |
| 95 | + |
| 96 | +The `$concurrency` parameter sets a new soft limit for the maximum number |
| 97 | +of jobs to handle concurrently. Finding a good concurrency limit depends |
| 98 | +on your particular use case. It's common to limit concurrency to a rather |
| 99 | +small value, as doing more than a dozen of things at once may easily |
| 100 | +overwhelm the receiving side. |
| 101 | + |
| 102 | +The `$limit` parameter sets a new hard limit on how many jobs may be |
| 103 | +outstanding (kept in memory) at once. Depending on your particular use |
| 104 | +case, it's usually safe to keep a few hundreds or thousands of jobs in |
| 105 | +memory. If you do not want to apply an upper limit, you can pass a `null` |
| 106 | +value which is semantically more meaningful than passing a big number. |
| 107 | + |
| 108 | +```php |
| 109 | +// handle up to 10 jobs concurrently, but keep no more than 1000 in memory |
| 110 | +$q = new Queue(10, 1000, $handler); |
| 111 | +``` |
| 112 | + |
| 113 | +```php |
| 114 | +// handle up to 10 jobs concurrently, do not limit queue size |
| 115 | +$q = new Queue(10, null, $handler); |
| 116 | +``` |
| 117 | + |
| 118 | +```php |
| 119 | +// handle up to 10 jobs concurrently, reject all further jobs |
| 120 | +$q = new Queue(10, 10, $handler); |
| 121 | +``` |
| 122 | + |
| 123 | +The `$handler` parameter must be a valid callable that accepts your job |
| 124 | +parameters, invokes the appropriate operation and returns a Promise as a |
| 125 | +placeholder for its future result. |
| 126 | + |
| 127 | +```php |
| 128 | +// using a Closure as handler is usually recommended |
| 129 | +$q = new Queue(10, null, function ($url) use ($browser) { |
| 130 | + return $browser->get($url); |
| 131 | +}); |
| 132 | +``` |
| 133 | + |
| 134 | +```php |
| 135 | +// accepts any callable, so PHP's array notation is also supported |
| 136 | +$q = new Queue(10, null, array($browser, 'get')); |
| 137 | +``` |
| 138 | + |
| 139 | +#### Promises |
| 140 | + |
| 141 | +This library works under the assumption that you want to concurrently handle |
| 142 | +async operations that use a [Promise](https://github.com/reactphp/promise)-based API. |
| 143 | + |
| 144 | +The demonstration purposes, the examples in this documentation use the async |
| 145 | +HTTP client [clue/buzz-react](https://github.com/clue/php-buzz-react), but you |
| 146 | +may use any Promise-based API with this project. Its API can be used like this: |
| 147 | + |
| 148 | +```php |
| 149 | +$loop = React\EventLoop\Factory::create(); |
| 150 | +$browser = new Clue\React\Buzz\Browser($loop); |
| 151 | + |
| 152 | +$promise = $browser->get($url); |
| 153 | +``` |
| 154 | + |
| 155 | +If you wrap this in a `Queue` instance as given above, this code will look |
| 156 | +like this: |
| 157 | + |
| 158 | +```php |
| 159 | +$loop = React\EventLoop\Factory::create(); |
| 160 | +$browser = new Clue\React\Buzz\Browser($loop); |
| 161 | + |
| 162 | +$q = new Queue(10, null, function ($url) use ($browser) { |
| 163 | + return $browser->get($url); |
| 164 | +}); |
| 165 | + |
| 166 | +$promise = $q($url); |
| 167 | +``` |
| 168 | + |
| 169 | +The `$q` instance is invokable, so that invoking `$q(...$args)` will |
| 170 | +actually be forwarded as `$browser->get(...$args)` as given in the |
| 171 | +`$handler` argument when concurrency is still below limits. |
| 172 | + |
| 173 | +Each operation is expected to be async (non-blocking), so you may actually |
| 174 | +invoke multiple operations concurrently (send multiple requests in parallel). |
| 175 | +The `$handler` is responsible for responding to each request with a resolution |
| 176 | +value, the order is not guaranteed. |
| 177 | +These operations use a [Promise](https://github.com/reactphp/promise)-based |
| 178 | +interface that makes it easy to react to when an operation is completed (i.e. |
| 179 | +either successfully fulfilled or rejected with an error): |
| 180 | + |
| 181 | +```php |
| 182 | +$promise->then( |
| 183 | + function ($result) { |
| 184 | + var_dump('Result received', $result); |
| 185 | + }, |
| 186 | + function (Exception $error) { |
| 187 | + var_dump('There was an error', $error->getMessage()); |
| 188 | + } |
| 189 | +); |
| 190 | +``` |
| 191 | + |
| 192 | +Each operation may take some time to complete, but due to its async nature you |
| 193 | +can actually start any number of (queued) operations. Once the concurrency limit |
| 194 | +is reached, this invocation will simply be queued and this will return a pending |
| 195 | +promise which will start the actual operation once another operation is |
| 196 | +completed. This means that this is handled entirely transparently and you do not |
| 197 | +need to worry about this concurrency limit yourself. |
| 198 | + |
| 199 | +#### Cancellation |
| 200 | + |
| 201 | +The returned Promise is implemented in such a way that it can be cancelled |
| 202 | +when it is still pending. |
| 203 | +Cancelling a pending operation will invoke its cancellation handler which is |
| 204 | +responsible for rejecting its value with an Exception and cleaning up any |
| 205 | +underlying resources. |
| 206 | + |
| 207 | +```php |
| 208 | +$promise = $q($url); |
| 209 | + |
| 210 | +$loop->addTimer(2.0, function () use ($promise) { |
| 211 | + $promise->cancel(); |
| 212 | +}); |
| 213 | +``` |
| 214 | + |
| 215 | +Similarly, cancelling an operation that is queued and has not yet been started |
| 216 | +will be rejected without ever starting the operation. |
| 217 | + |
| 218 | +## Install |
| 219 | + |
| 220 | +The recommended way to install this library is [through Composer](https://getcomposer.org). |
| 221 | +[New to Composer?](https://getcomposer.org/doc/00-intro.md) |
| 222 | + |
| 223 | +This will install the latest supported version: |
| 224 | + |
| 225 | +```bash |
| 226 | +$ composer require clue/mq-react:dev-master |
| 227 | +``` |
| 228 | + |
| 229 | +This project aims to run on any platform and thus does not require any PHP |
| 230 | +extensions and supports running on legacy PHP 5.3 through current PHP 7+ and |
| 231 | +HHVM. |
| 232 | +It's *highly recommended to use PHP 7+* for this project. |
| 233 | + |
| 234 | +## Tests |
| 235 | + |
| 236 | +To run the test suite, you first need to clone this repo and then install all |
| 237 | +dependencies [through Composer](https://getcomposer.org): |
| 238 | + |
| 239 | +```bash |
| 240 | +$ composer install |
| 241 | +``` |
| 242 | + |
| 243 | +To run the test suite, go to the project root and run: |
| 244 | + |
| 245 | +```bash |
| 246 | +$ php vendor/bin/phpunit |
| 247 | +``` |
| 248 | + |
| 249 | +## License |
| 250 | + |
| 251 | +MIT |
0 commit comments