Operations

Health endpoint#

GET /health returns 200 while the process is up. It does not check the database, RabbitMQ or storage.

Finding an exchange#

The Exchanges page filters by status, subscription, partner, information type, exchange ids, correlation id, a promoted property value and a date range. Pick a promoted property to search inside its value only; any part of the value matches, ignoring case. Filters live in the URL, so a filtered view can be shared as a link. The list reloads itself every 15 seconds by default (adjustable, or paused); Refresh, beside the pause button, reloads the list and its counts right away.

• Search by promoted property to find a business document, such as an order number.
• Search by correlation id to see every exchange in one flow, including exchanges created from responses.
• Searching by id also finds retries of that exchange and the aggregation that collected it.
• Turn on Latest attempt only to hide attempts that were already retried, leaving only the failures that still need action.

Expanding an exchange shows each stage's file, the exception, its retry chain, the aggregation family, and buttons to retry or to run a waiting retry now. An exchange that was already retried links to its later attempt instead of offering Retry.

Exporting exchange files#

Tick exchanges on the Exchanges page, or select everything a filter matches, and choose Export files to download their files as one zip. It needs only exchanges.view.

• Each exchange gets a folder named like its archive: the main promoted value, then the id (Amman_3f2a…). It holds input, mapped and handled, whichever the exchange has, with the extension of the file's name or content type.
• When some selected exchanges are older than storage keeps files, Export says how many first and asks before downloading the rest. When none have files left, it says so and downloads nothing.
• A file missing for another reason, such as one deleted by hand, is listed in missing.txt with the reason. The rest of the zip still comes.
• One export takes at most 500 exchanges. Narrow the filter for a larger set.

Reading an exchange's state#

Queue health#

The Queue health page reads the RabbitMQ management API every 5 seconds. It needs monitoring.view and the three RabbitMQ management keys.

• Tiles show lanes, queue depth, retry backlog, dead letters, and incoming and acknowledged rates.
• Lanes list each consumer with its nodes, in-flight, queued, retrying and dead messages, prefetch, rates and health. They are grouped as front doors, work, notifications, legacy and control.
• Alerts come from the bus library, such as backpressure and dead letters.
• Nobody is reading these lists queues with Bitween's prefix that no consumer reads. They are usually left behind by a deleted or renamed work group or information type, and RabbitMQ does not remove them.

The bus library retries a message whose consumer throws through the .retry queue, and parks it in the .bad queue when those retries run out. This is transport-level failure, separate from retry policies. An exchange that fails in its adapters still gets a result and does not go to .bad.

The same data is available from GET /api/ops/summary, /api/ops/consumers, /api/ops/queues, /api/ops/retries, /api/ops/deadletters, /api/ops/alerts and /api/ops/unattendedqueues.

Data sources#

• The Data sources page lists each connection with its state, last error and the node that holds it.
• Test connection runs the adapter's own checks and shows each stage. For databases it also checks every statement.
• The live panel refreshes every 3 seconds, but only when the page is served by the node running the adapter.
• A data source nobody runs keeps showing its last recorded state.

See Data sources.

Scheduled work#

• The Scheduled jobs and Aggregations pages show each subscription's last run, reliability, next run and schedule health.
• A subscription's overview lists its receive attempts and the exchanges each created.
• Stuck means a run was killed and left its running flag set, so later runs are skipped. The subscription's page says so and offers Clear the running mark, for anyone with subscriptions.operate; otherwise it clears itself after Bitween:StaleRunAfterMinutes.
• Not scheduled means a trigger is missing. Saving the subscription again recreates it.

See Scheduling.

Dashboard#

The dashboard, opened from the logo, shows today's exchanges, the 7-day success rate, today's failures, waiting retries, queue alerts, a 14-day chart, the latest failures, unhealthy subscriptions and the busiest subscriptions. The browser computes it from up to 1,000 recent exchanges, so figures on a busy instance are approximate.

Two parts come from the server instead. Failures to act on counts failed exchanges that are the newest attempt of their chain, of any age, and links to that search. Chains that keep failing lists the five retry chains with the most attempts, with the success rate of retries over the last 7 days.

Audit trail#

The Audit trail page shows who changed which configuration entity, when, and the values before and after. Filter by entity type, member, entity id, date range, or everything changed in one save. See Security.

Metrics and traces#

Bitween exports metrics and traces over OpenTelemetry (OTLP) when a collector is set, so it shows up in Grafana, Datadog, Azure Monitor, Honeycomb or anything else that takes OTLP. Nothing is exported, and nothing changes, while neither address is set.

What is exported:

An outgoing call made while an exchange is processed is a child of its exchange.process span, so a trace shows which partner call took the time. The dashboard in the admin UI reads Bitween's own database and needs none of this.

With the Helm chart, set the variables under environmentVariables:

environmentVariables:
  OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.monitoring:4317"
  OTEL_SERVICE_NAME: "bitween-acme-prod"

Logs#

Bitween logs to the console. SQL statements are logged without their parameter values, which hold passwords, partner keys and payloads, unless Bitween:LogSqlParameterValues is on; a developer's launchSettings.json turns it on locally, and no environment does by itself. When SwLogger:ElasticsearchUrl is set and the environment is listed in SwLogger:ElasticsearchEnvironments, logs also go to Elasticsearch, enriched with request context.

Messages worth watching for:

• Bitween can't start until these are fixed:, followed by what to change. See When Bitween won't start
• Database migration failed on startup against {Host}:{Port}/{Database}
• A startup warning that the Microsoft sign-in redirect URI is not the sign-in landing page
• Auto-retry evaluation failed for xchange …
• Retry budget of subscription … could not be cleared after a success
• Bus gateway route references subscription …, which is not active; skipping.

Data retention#

The retention job runs on the Retention schedule setting, 04:00 daily by default. It removes exchanges that started more than Keep exchanges (days) ago, oldest first and in small batches. An exchange waiting for a scheduled retry is kept, and so is one a newer retry still points at, until that retry is removed too.

With Archive before deleting on, the default, each exchange is first written to archive/{prefix}/{subscription id}/yyyy/MM/dd/{main value}_{exchange id}.json. {prefix} is the document prefix without its tempN/ part, and the main value is the exchange's value for the first promoted property of its information type; without one the file is named by the id alone. The file holds the exchange's details, its promoted properties, result and notifications, and the contents of its files while storage still has them. Handler and mapper properties are left out, because they can hold resolved secrets. If an exchange can't be archived, the run stops and keeps it for the next run.

Queue health ends with Bitween's own work: each system job (automatic retries, exchange retention, receive attempt clean-up, broker deduplication clean-up) with when it last ran and runs next, flagged when it isn't scheduled, and the outbox - messages saved to be sent to the broker, how many are waiting and since when, and the last error of one that failed.

When a partner says a gateway refuses them, the gateway's page lists Calls turned away: how many calls were refused for each reason (key not recognised, partner not attached, gateway switched off, nothing behind it, too many calls) and the most recent with the caller's address. Each node counts the calls it refused itself, in memory, since it started. A partner's page shows when each API key was last used, so a key nobody uses any more can be found and revoked.

Administration › Nodes lists every node of the cluster. Each node writes itself into ClusterNodes every 30 seconds, with its host, version, whether it runs data sources (Bitween:BusProvidersEnabled) and which adapter runtimes it has; a node not seen for 90 seconds is shown as gone, and one not seen for a week is removed. The page shows the leases and exclusive data sources each node holds, and warns when the online nodes run different versions or none of them runs data sources.

Settings › About this instance shows what this Bitween is and how it is set up, as the node that answered sees it: its version, the node's name and whether it runs data sources, the database and RabbitMQ checks /health/ready runs (run when the page opens), storage and queue prefix, which adapter runtimes (.NET, Python, Node.js) and pip/npm are on the node, the adapter editor's limits, rate limits, and the network and housekeeping settings that live in configuration. In a cluster, open it on each node to compare.

The Documents & storage section of the Settings page shows the bucket's rules and what the settings do: how long files and exchanges are kept, when the job next runs, how many exchanges are listed without their files, and warnings such as exchanges that outlive their files, archives that come too late to hold them, aggregations that run less often than exchanges are kept, or a bucket that lets anyone read Bitween's files without credentials. Saving a change to these settings shows the same list and asks first.

Exchange rows can outlive their files. Opening such a file says it was deleted by the retention policy, and so does retrying an exchange whose input has gone. A waiting retry for one is dropped with a reason.

Common problems#