> ## Documentation Index
> Fetch the complete documentation index at: https://botmesh.afaya.com.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Tracing (OpenTelemetry)

A bot does a lot of work for each message: access lists, persistence, the chat's queue, hooks, routing, a conversation step, an AI call with tools, the provider's send. Tracing shows that work as a tree of **spans** per message, so you can see where the time goes and where things fail, in Jaeger, Grafana Tempo or any OpenTelemetry backend.

Tracing is **opt-in**. Without `BotConfig.telemetry` the bot runs exactly the code it ran before: no spans, no extra awaits.

* `@botmesh/core` has the tracing hooks: a minimal `BotTracer` interface, a no-op default and `bot.tracing`. No new dependency.
* `@botmesh/otel` adapts them to OpenTelemetry and also turns `bot.metrics()` into OpenTelemetry metrics.
* `@botmesh/ai` adds model, tool and retrieval spans to the same trace.

> [!TIP]
>
> **No message content**
>
> Spans carry **kinds, stages, outcomes and ids**, never message bodies, captions, prompts, completions, tool arguments or results, `c.state` or error messages. Chat, user and message ids are hashed by default. See [Personal data](#personal-data).

## Quick start

```bash theme={null}
pnpm add @botmesh/otel @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http @opentelemetry/sdk-metrics
```

```ts theme={null}
// telemetry.ts — load it before the bot (e.g. `node --import ./telemetry.js bot.js`)
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'

export const sdk = new NodeSDK({
  serviceName: 'sales-bot',
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  metricReaders: [new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: 'http://localhost:4318/v1/metrics' }),
  })],
})
sdk.start()
```

```ts theme={null}
// bot.ts
import { Bot } from '@botmesh/core'
import { aiPlugin } from '@botmesh/ai'
import { createOtelTracer, otelMetricsPlugin } from '@botmesh/otel'
import { sdk } from './telemetry.js'

const bot = new Bot({
  provider,
  telemetry: {
    tracer: await createOtelTracer(),
    // Same key on every replica so hashed ids correlate across processes and restarts.
    idKey: process.env.TRACE_ID_KEY,
  },
})
bot.plug(otelMetricsPlugin())   // bot.metrics() as OpenTelemetry metrics
bot.plug(aiPlugin({ /* … */ })) // AI spans join the same trace automatically

process.on('SIGTERM', async () => {
  await bot.stop()
  await sdk.shutdown()          // flushes the spans and metrics still buffered
})
```

`createOtelTracer()` loads `@opentelemetry/api` lazily and rejects with an install hint when it is missing. When the bot must be built synchronously, use `lazyOtelTracer()`: its spans are no-ops until the API is loaded, and a load failure is reported through its `ready` promise (never thrown into the bot).

## Local backends

### Jaeger

```bash theme={null}
docker run --rm --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:latest
```

Point the exporters at `http://localhost:4318` and open `http://localhost:16686`. Jaeger stores traces only: send the metrics to a Prometheus or an OpenTelemetry Collector, or drop the metric reader while you try it.

### Grafana Tempo + Grafana

`docker-compose.yml`:

```yaml theme={null}
services:
  tempo:
    image: grafana/tempo:latest
    command: ['-config.file=/etc/tempo.yaml']
    volumes: ['./tempo.yaml:/etc/tempo.yaml']
    ports: ['4318:4318', '3200:3200']
  grafana:
    image: grafana/grafana:latest
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: 'true'
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    volumes: ['./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml']
    ports: ['3000:3000']
```

`tempo.yaml`:

```yaml theme={null}
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
storage:
  trace:
    backend: local
    local: { path: /var/tempo/traces }
    wal: { path: /var/tempo/wal }
```

`grafana-datasources.yaml`:

```yaml theme={null}
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    isDefault: true
```

`docker compose up`, then open Grafana at `http://localhost:3000` → Explore → Tempo and search by service name (`sales-bot`) or by span name (`bot.send`, `chat claude-…`). Tempo only takes traces: for the metrics, add a Prometheus with its OTLP receiver (`--web.enable-otlp-receiver`, OTLP path `/api/v1/otlp/v1/metrics`) or an OpenTelemetry Collector in front of both. Pin the image versions in production.

## What a message looks like

```
bot.inbound                      consumer · bot.event.kind=message · bot.inbound.outcome=dispatched
├─ bot.persist                   chat / contact / message writes
└─ bot.dispatch                  bot.queue.wait_ms · bot.dispatch.stage=matcher
   └─ bot.handler                bot.handler.stage=matcher · bot.matcher.kind=text · bot.flow.id=order
      └─ bot.conversation.turn   bot.conversation.reason=start · parked=true
         ├─ chat claude-…        gen_ai.* (model, tokens, finish reason)
         │  └─ (http span of the SDK call, from the SDK's own instrumentation)
         ├─ execute_tool lookup  bot.ai.tool.outcome=ok
         └─ bot.pace             delay + typing
            └─ bot.send          client · bot.send.outcome=ok
               └─ (http span of the provider call)
```

The user's answer arrives as another message: its own `bot.inbound` → `bot.dispatch` (stage `conversation`) with the **next** turn of the same conversation (`bot.conversation.id` is the same) as a child.

### Span catalogue

| Span | Kind | Where |
| - | - | - |
| `bot.inbound` | consumer | A message, interaction or call event from the provider (connection, presence and status events have no span). Parent: the active OpenTelemetry context, e.g. the HTTP span of a webhook. |
| `bot.persist` | internal | Chat, contact and message writes of an inbound message (`spans.persist`). |
| `bot.dispatch` | internal | The message's job in the chat's FIFO queue, once its turn came. |
| `bot.handler` | internal | Each handler the dispatcher runs: command, matcher, catch-all, event (call) trigger. |
| `bot.conversation.turn` | internal | A stretch of a conversation between two waits (see below). |
| `bot.trigger` / `bot.broadcast` | internal | `bot.trigger()` / `bot.broadcast()`. |
| `bot.pace` | internal | A paced reply: delay + typing indicator + its send (`spans.pace`). |
| `bot.send` | client | Every send and edit through the outbound pipeline. The provider call runs inside it. |
| `bot.stream.update` | client | A `ctx.stream()` draft/edit — only with `spans.streamUpdates`; otherwise an event on the active span. |
| `chat {model}` | client | A model call from `@botmesh/ai` (OpenTelemetry GenAI conventions). |
| `execute_tool {name}` | internal | A tool call of the AI tool loop. |
| `bot.ai.retrieve` | internal | RAG retrieval (`bot.ai.rag.chunks`). |
| `bot.task` | internal | One run of a scheduled task (`bot.schedule`/`every`/`delay`, plain or durable). Always a **root** span; the task's sends nest under it. See [Scheduled tasks](#scheduled-tasks). |
| `bot.lifecycle` | internal | One `start`/`stop`/`rollback` of a [lifecycle step](https://botmesh.afaya.com.co/guide/architecture.md#lifecycle): `storage`, `coordination`, `restore`, `durability`, `transport`, `jobs`. Root spans taken during `bot.start()`/`bot.stop()`; the `transport` one measures the provider handshake (Baileys connect + QR, Telegram polling, IMAP login), which is where a slow boot shows up. |

### Attributes

| Attribute | On | Values |
| - | - | - |
| `bot.provider.id`, `bot.id` | inbound, send | Provider id, bot id of the store namespace. |
| `bot.event.kind`, `bot.message.type`, `bot.chat.kind` | inbound, dispatch | `message`/`interaction`/`call`; `text`, `image`…; `private`/`group`. |
| `bot.chat.id`, `bot.sender.id`, `bot.message.id` | inbound, send | Through the [ids policy](#personal-data). |
| `bot.inbound.outcome` | inbound | `dispatched`, `blocked`, `duplicate`, `group_filtered`, `stopped`, `from_me`, `echo` (with `echoFilter`), `handoff` (calls). |
| `bot.queue.wait_ms` | dispatch | Time waiting for the chat's turn (includes the cross-replica turn with coordination). |
| `bot.dispatch.stage` | dispatch | `aborted_by_hook`, `handoff`, `command`, `command_unknown`, `conversation`, `dormant`, `matcher`, `catch_all`, `unhandled`, `event`. |
| `bot.handler.stage`, `bot.matcher.kind` | handler | `command`/`matcher`/`catch_all`/`event`; `welcome`/`text`/`function`/`type`. |
| `bot.command.name` | handler | Only for a **registered** command (unknown command text is never recorded). |
| `bot.flow.id` | handler | The flow/conversation a matcher enters. |
| `bot.conversation.id`, `.name`, `.reason`, `.parked`, `.outcome`, `.goto` | turn | See below. |
| `bot.send.outcome`, `bot.content.kind`, `bot.send.origin`, `bot.send.edit` | send | `ok`, `failed`, `skipped`, `deduplicated` (durable outbox), `deferred`; `text`, `image`…; `ctx`/`api`. |
| `bot.task.id`, `.kind`, `.durable`, `.cron`, `.interval_ms`, `.time_zone` | task | Task id (through the [ids policy](#personal-data)), `cron`/`interval`/`once`, durable or not, its spec. |
| `bot.task.scheduled_at`, `.missed`, `.fencing_token`, `bot.scheduler.role`, `bot.scheduler.owner` | task (durable) | The slot (ISO 8601), whether it runs late, the claim's fencing token, `leader`/`follower`/`single`, the replica's scheduler owner. |
| `bot.link.reason` | span link | `resume`: the link of a conversation resumed from a checkpoint to the turn that wrote it. |
| `bot.lifecycle.step`, `bot.lifecycle.phase` | lifecycle | The step's name and what it was doing: `start`, `stop` (after the drain) or `rollback` (a failed `start()` undoing what it started). |
| `error.type` | any failed span | The error class name (`RateLimitedError`, `timeout`…). |

Every name is exported as `TRACE_ATTR` / `SPAN_NAMES` from `@botmesh/core`.

### Conversation turns

A conversation can last days, so it is not one span. Each stretch of work between two waits is a `bot.conversation.turn`:

* the first turn starts under the handler (or `bot.trigger`) that entered it — `reason: start` or `dormant` (a persistent flow resumed from its checkpoint); a `c.goto()` target starts under the turn that ran the goto — `reason: goto`;
* parking on a wait ends the turn with `bot.conversation.parked = true`;
* a reply opens the next turn under the dispatch span of the message that answered — `reason: reply`;
* a wait timeout opens a **root** turn linked to the first one — `reason: timeout`, since no message caused it; a `c.idle()` callback that fires while the conversation is parked runs in a root turn too (`reason: idle`), which parks again when the callback only replied, or carries the outcome when it ended or redirected the conversation;
* an error thrown by the conversation (or by an idle callback) is recorded on the turn it happened in;
* a conversation resumed from a checkpoint written by **another replica or an earlier process** (a persistent flow after a restart, a flow forwarded by `coordination.handover`, a hand-over stub restarted with `fallback: 'restart'`) starts under the dispatch of the waking message and is **linked** to the turn that wrote the checkpoint — see [Across replicas and restarts](#across-replicas-and-restarts);
* the end of the conversation sets `bot.conversation.outcome` (`completed`, `ended`, `guarded`, `cancelled`, `timedOut`, `retriesExceeded`, `error`) on the open turn; a conversation aborted while parked (`/reset`, handoff, `stop()`) gets a short closing turn (`reason: end`).

Flow steps are `flow.step` events on the turn (`bot.flow.step.index`, `bot.flow.step.kind`).

### Across replicas and restarts

With telemetry on, every checkpoint a conversation writes (persistent flows, ephemeral checkpoints of `handover.flows: 'forward'`, hand-over stubs) also stores the **W3C trace context** of the turn that wrote it, in `FlowCheckpoint.trace` (`{ traceparent, tracestate? }`; stores keep it opaquely, like the rest of the checkpoint). The replica — or the restarted process — that resumes the conversation gives its first turn (`reason: dormant`, or `start` for a stub restarted by `fallback: 'restart'`) a **span link** to that context, marked `bot.link.reason = resume` and `isRemote`, with the `tracestate` restored.

**Link, not parent** (deliberate): the resumed turn's parent stays the `bot.dispatch` of the message that woke it, so that message's trace (inbound → dispatch → turn → sends) is whole; making the previous replica's span its parent would cut the waking message's trace in two, and a conversation parked for hours would stretch one trace across processes and exports that were flushed long ago. Your backend shows the link as "follows from" and lets you jump to the earlier trace.

Notes:

* Only W3C-shaped ids are carried (32/16 lowercase hex, which OpenTelemetry produces). A tracer with other ids stores nothing and the turn simply has no link. A malformed stored value is ignored.
* The stored context is always the **latest** turn that wrote or parked: a hand-over stub is rewritten when a later turn parks (only with telemetry on; otherwise it is refreshed only before it expires), and the parked marker of a forwarded step refreshes it too. A turn whose context cannot be carried keeps the one already stored instead of erasing it.
* A custom `handover.fallback` function receives the context in `info.trace`, to link spans of its own.
* `formatTraceparent(spanContext)` and `parseTraceparent(traceparent, tracestate?)` are exported from `@botmesh/core/spi` for your own propagation.

### Scheduled tasks

Each run of a task registered with `bot.schedule()`, `bot.every()` or `bot.delay()` is a `bot.task` span: plain in-memory tasks per run, durable tasks per occurrence (one span per slot of a catch-up). The span is always a **root** span — a task has no causing message, even when it was registered inside a handler — and the handler's sends and AI calls nest under it. A throwing handler sets status error with `error.type`, as anywhere else.

Durable occurrences add the slot (`bot.task.scheduled_at`), `bot.task.missed`, the claim's `bot.task.fencing_token`, and leadership: `bot.scheduler.role` is `leader` (this replica holds the scheduler lease), `follower` (it stepped in for an occurrence the leader left overdue) or `single` (no lease configured), plus `bot.scheduler.owner` (`scheduler.owner`, a random id per process by default). Task ids go through the [ids policy](#personal-data) (hashed by default), since durable ids often embed a user or chat id. The bot's internal timers (per-chat `onIdle` watchers) are not task spans.

### AI spans

`aiPlugin` passes `bot.tracing` to its `AIClient`, so with telemetry on every model call is a `chat {model}` span (kind client) with `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.request.max_tokens`, `gen_ai.response.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.response.finish_reasons` and `bot.ai.purpose` (`reply`, `ask`, `generate`, `memory`, `tool_step`). A timeout is status error with `error.type = timeout`; a refusal is `finish_reasons = [refusal]` with the status unset. Streams are one span from the first chunk to the end (`bot.ai.streamed`, `bot.ai.cancelled` when stopped early); Anthropic and Gemini report streamed token usage, OpenAI-compatible and Ollama do not.

Tool calls are `execute_tool {name}` spans with `gen_ai.tool.name`, `gen_ai.tool.call.id` and `bot.ai.tool.outcome` (`ok`, `error`, `timeout`, `invalid_args`, `unknown`, `duplicate`, `confirm.denied`, `confirm.replied`, `confirm.cancelled`). A tool waiting for the user's confirmation keeps its span open (with a `confirm.requested` event) until it is answered.

A standalone `AIClient` traces only when you pass `tracing` (e.g. `bot.tracing`) in its options.

## Your own spans

`bot.tracing` is the bot's tracing facade (a disabled no-op without telemetry). Spans you start inside a handler nest under the bot's:

```ts theme={null}
bot.when('order', async (ctx) => {
  const order = await bot.tracing.trace('erp.lookup', { kind: 'client' }, () => erp.find(ctx.body))
  await ctx.reply(`Order ${order.id}`)
})
```

With `@botmesh/otel`, bot spans are also the active OpenTelemetry context, so `tracer.startActiveSpan()` and the SDK's auto-instrumentations (http, undici, pg…) nest under them too. Plugins use `ctx.bot.tracing` the same way; `bot.tracing.idAttribute(id)` applies the ids policy to an id of your own.

## Metrics

`otelMetricsPlugin()` (or `await registerOtelMetrics(bot)`) reads `bot.metrics()` on every collection:

| Metric | Type | Attributes |
| - | - | - |
| `botmesh.messages.inbound` | counter | |
| `botmesh.messages.outbound` | counter | `result` = `ok`/`failed` |
| `botmesh.dropped` | counter | `reason` = `duplicate`/`blocked`/`handoff`, plus `echo` with `echoFilter` |
| `botmesh.presence.skipped` | counter | `kind` = `typing`/`mark_read` |
| `botmesh.conversations.started` / `.ended` | counter | `outcome` (ended) |
| `botmesh.flow.conversations.started` / `.ended` | counter | `flow`, `outcome` (`perFlow`) |
| `botmesh.triggered`, `botmesh.errors` | counter | |
| `botmesh.conversations.active`, `botmesh.uptime` (s) | gauge | |
| `botmesh.conversation.duration` (s) | histogram | `flow`, `outcome` |

Every point carries `bot.id` plus your `attributes`. `bot.resetMetrics()` makes the cumulative counters drop, which OpenTelemetry and Prometheus treat as a counter reset.

The tracer also derives histograms from finished spans (`spanMetrics`, on by default), recorded even for unsampled traces: `botmesh.inbound.duration{bot.event.kind, bot.inbound.outcome, bot.provider.id}`, `botmesh.dispatch.duration{bot.dispatch.stage}`, `botmesh.queue.wait`, `botmesh.send.duration{bot.provider.id, bot.send.outcome}`, `botmesh.lifecycle.duration{bot.lifecycle.step, bot.lifecycle.phase}`, `gen_ai.client.operation.duration` and `gen_ai.client.token.usage{gen_ai.token.type}`. No metric ever carries a chat, user or message id.

The histograms come with bucket boundaries that fit their unit (sent as instrument advice, so a `View` in your SDK still wins): 5 ms to 10 s for the bot latencies, the GenAI semantic-convention buckets for model calls and tokens, and 1 s to 1 day for `botmesh.conversation.duration`. A `prefix` must start with a letter and contain only letters, digits, `.`, `_`, `-` or `/`; anything else throws when the tracer or the bridge is built.

## Personal data

* **Never recorded**: message bodies, captions, prompts, completions, tool arguments and results, `c.state`, ask answers, unknown command text, raw provider payloads.
* **Error messages** may contain user text: only the error class and code are recorded unless you set `telemetry.errorMessages: true`.
* **Ids** follow `telemetry.ids`:
  * `'hash'` (default): HMAC-SHA256, base64url, 16 characters. The key is `telemetry.idKey`; without it a random key per process is used, so hashes do not correlate across replicas or restarts. Set the same secret everywhere to search a chat's traces.
  * `'raw'`: the id as the provider gives it — on WhatsApp that is a phone number, personal data.
  * `'omit'`: no ids at all.

## Configuration

```ts theme={null}
new Bot({
  provider,
  telemetry: {
    tracer,                          // BotTracer (required)
    ids: 'hash',                     // 'hash' | 'raw' | 'omit'
    idKey: process.env.TRACE_ID_KEY, // string | Uint8Array
    errorMessages: false,
    spans: { persist: true, pace: true, streamUpdates: false },
  },
})
```

| `@botmesh/otel` option | Default | |
| - | - | - |
| `api` | lazy import | The `@opentelemetry/api` module; injected, `otelTracer()` / `bridgeMetrics()` work synchronously. |
| `tracerProvider` | global (`trace.getTracerProvider()`) | |
| `tracerName` | `'@botmesh/otel'` | Instrumentation scope. |
| `spanMetrics` | `true` | `false`, or `{ meterProvider, prefix }`. |
| `otelMetricsPlugin({ meterProvider, meterName, prefix, attributes, perFlow, conversationDuration })` | global, `'@botmesh/otel'`, `'botmesh'` (`'botwa'` keeps the alpha names), `{}`, `true`, `true` | |

### Sampling

Sample at the root with a parent-based sampler (`new ParentBasedSampler({ root: new TraceIdRatioBasedSampler(0.1) })` in the `NodeSDK` options): a message's spans share its decision. When a span is not recorded, the core skips its id hashing. The span-derived histograms are recorded anyway, so latency dashboards stay complete at any sampling rate.

## Other backends and tests

`BotTracer` is deliberately small: `startSpan(name, options)` returning a span with `setAttribute(s)`, `addEvent`, `recordException`, `setStatus`, `end`, `isRecording` and `spanContext`, plus an optional `withSpan(span, fn)` to make a span the backend's active context. Any tracing library can implement it. A tracer that throws never breaks the bot: the call falls back to a no-op span and a warning is logged at most once a minute.

In tests, `createRecordingTracer()` from `@botmesh/core/testing` records every span in memory:

```ts theme={null}
import { createRecordingTracer, createTestBot } from '@botmesh/core/testing'

const rec = createRecordingTracer()
const { bot, chat } = await createTestBot({ telemetry: { tracer: rec.tracer, ids: 'raw' } })
bot.when('hi', (ctx) => ctx.reply('hello'))
await chat.send('hi')
expect(rec.parentOf(rec.find('bot.send')!)?.name).toBe('bot.pace')
```

## Limitations

* **Durable replays**: an inbound record replayed by the durability engine (after a crash or a retry) has no `bot.inbound` span; its `bot.dispatch` is a root span.
* **Multi-replica**: a conversation resumed on another replica (or after a restart) starts a new trace **linked** to the previous one, not a continuation of it (see [Across replicas and restarts](#across-replicas-and-restarts)). A conversation handed over before telemetry was enabled, or by a replica without it, has no link.
* **Third-party OpenTelemetry spans inside a resumed conversation** (code after `await c.ask()`) parent to OpenTelemetry's own context, which is still the first turn's; the bot's own spans parent correctly.
* Flushing is the SDK's job: call `sdk.shutdown()` after `bot.stop()`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.