External brokers

Bitween can read from and publish to a broker the customer runs, through a data source. Two brokers are supported.

There is no Kafka, Azure Service Bus or MQTT provider.

Receiving through a bus gateway#

A bus gateway reads either from Bitween's internal bus, as before, or from a broker data source.

1. Create a broker data source and test its connection.
2. Open the bus gateway and choose its source: the internal bus, or a broker data source plus an endpoint. The endpoint is the queue name for RabbitMQ, or the full queue URL for SQS.
3. Add routes as usual.

Two gateways on one data source cannot read the same endpoint. A gateway can be moved between the internal bus and a broker later, but its information type stays fixed.

What happens to a message#

sequenceDiagram
  participant B as Customer broker
  participant A as Resident adapter
  participant X as Bitween
  B->>A: Deliver message
  A->>X: Hand over body, endpoint and key
  X->>X: Find the gateway for the endpoint
  X->>X: Record the dedupe key and store the exchange, in one transaction
  X-->>A: Accepted
  A->>B: Acknowledge

• The message body becomes the input of a document of the gateway's information type. From there it runs through the filter and routes like any bus message. See Entry points.
• The adapter acknowledges the broker only after Bitween has stored the exchange. If Bitween refuses or is down, the message is requeued rather than lost.
• A message on an endpoint no gateway claims is acknowledged and discarded, with a warning in the log.

Deduplication#

A crash between storing and acknowledging makes the broker redeliver, so Bitween remembers a key per message.

The key is stored with the exchange in one transaction. A second delivery of the same key on the same data source is acknowledged without creating another exchange. Keys are kept for the data source's deduplication window, 30 days by default, and a nightly job deletes older ones. A window of 0 turns deduplication off.

Delivering to a broker#

A subscription can publish its output to a broker.

1. Set the subscription's delivery adapter to bitween.bus.rabbitmq or bitween.bus.sqs.
2. Bind the subscription to the broker data source.
3. Fill in where to publish.

• Publishing works from any enabled node. A node that does not hold the consuming connection opens a send-only one.
• The response file is the broker's receipt: the message id and size for RabbitMQ, or the message id and sequence number for SQS.
• A failed publish fails the exchange, so retry policies apply.
• RabbitMQ publishes without publisher confirms, so a broker that drops a message after accepting it is not detected.
• The message id is the exchange id, and a retry is a new exchange, so a retried publish carries a different id.
• A subscription binds one data source, so it cannot read from a database and publish to a broker in one step. Chain two subscriptions instead.

RabbitMQ settings#

The connection test checks the connection, authentication and each gateway queue. Discover lists only the configured queues with their depth.

SQS settings#

A refused message is made visible again straight away. The connection test calls ListQueues, so the credentials need sqs:ListQueues, and it warns when a queue's visibility timeout is under 30 seconds. Discover lists up to 100 queues.

Limits#

• Per-endpoint settings on a gateway are stored and passed to the adapter, but neither provider reads them.
• Test, Discover and the live panel run on the node that serves the request.
• Settings changes reach the adapter within 30 seconds, and restart it.
• Data source limits apply as well.