consuming-messages-from-external-systems.md

Consuming messages from external systems

This guide explains how to publish messages to a queue backend (RabbitMQ, Kafka, SQS, etc.) from external producers (including non-PHP services) so that yiisoft/queue consumers can correctly deserialize and process these messages.

The key idea is simple:

• The queue adapter reads a raw payload (usually a string) from the broker.
• The adapter passes that payload to a Yiisoft\Queue\Message\Serializer\MessageSerializerInterface implementation (by default MessageSerializer).
• MessageSerializer deserializes the payload into an array. It delegates decoding of payload format to Yiisoft\Queue\Message\Serializer\MessageEncoderInterface implementation.
• By default, yiisoft/queue config binds MessageEncoderInterface to Yiisoft\Queue\Message\Serializer\JsonMessageEncoder.

JsonMessageEncoder is only the default implementation. You can replace it with your own encoder by rebinding Yiisoft\Queue\Message\Serializer\MessageEncoderInterface in your DI configuration.

External systems should produce the same payload format that MessageSerializer expects. The payload shape (type, data, meta keys) is defined by MessageSerializer regardless of the encoder; the encoder only converts between the raw string and an associative array (JSON ↔ array by default).

1. Message type contract (most important part)

yiisoft/queue resolves a handler by message type (MessageInterface::getType()).

For external producers, you should not rely on PHP FQCN handler names as messages types. Prefer a stable short type and map that type to a handler callable in the consumer application configuration (see Message handler: advanced setup).

Example mapping:

return [
    'yiisoft/queue' => [
        'handlers' => [
            'file-download' => \App\Queue\RemoteFileHandler::class,
        ],
    ],
];

External producer then always publishes "type": "file-download".

2. JSON payload format (JsonMessageEncoder)

Yiisoft\Queue\Message\Serializer\MessageSerializer expects the decoded payload to be an object with these keys (with the default JsonMessageEncoder, the message is a JSON string):

• type (string, required)
• payload (any JSON value, optional; defaults to null)
• meta (object, optional; defaults to {})

Minimal example:

{
  "type": "file-download",
  "payload": {
    "url": "https://example.com/file.pdf",
    "destinationFile": "/tmp/file.pdf"
  }
}

Full example:

{
  "type": "file-download",
  "payload": {
    "url": "https://example.com/file.pdf",
    "destinationFile": "/tmp/file.pdf"
  },
  "meta": {
    "trace-id": "1f2c0e10b7b44c67",
    "tenant-id": "acme"
  }
}

Notes about meta

The meta key is a general-purpose metadata container (for example, tracing, correlation, tenant information). The consumer-side application or middleware may read, add, or override keys as needed. Populating meta from external producers is possible but not recommended: the accepted keys and their semantics are entirely defined by the consumer application, so the mapping of field names and their expected values must be maintained manually.

3. Data encoding rules

• The payload must be UTF-8 JSON.
• payload and meta must contain only JSON-encodable values:
  • strings, numbers, booleans, null
  • arrays
  • objects (maps)

If your broker stores bytes, publish the UTF-8 bytes of the JSON string.

4. Publishing to a broker: what exactly to send

yiisoft/queue itself does not define a network protocol. The exact destination or transport for this JSON depends on the adapter:

• Some adapters put this JSON into the broker message body.
• Some adapters may additionally use broker headers/attributes.

For external producers you should:

• Use the adapter documentation of your chosen backend (AMQP / Kafka / SQS / etc.) to know which queue/topic and routing settings to use.
• Ensure the message body is exactly the JSON described above (unless the adapter docs explicitly say otherwise).

5. Examples (non-PHP)

These examples show how to produce the JSON body. You still need to publish it with your broker-specific client.

Python (constructing JSON body)

import json

payload = {
    "type": "file-download",
    "payload": {"url": "https://example.com/file.pdf"}
}

body = json.dumps(payload, ensure_ascii=False).encode("utf-8")

Node.js (constructing JSON body)

const payload = {
  type: 'file-download',
  payload: { url: 'https://example.com/file.pdf' },
};

const body = Buffer.from(JSON.stringify(payload), 'utf8');

curl (for HTTP-based brokers / gateways)

curl -X POST \
  -H 'Content-Type: application/json' \
  --data '{"type":"file-download","payload":{"url":"https://example.com/file.pdf"}}' \
  https://your-broker-gateway.example.com/publish