Tracing (OpenTelemetry)

• @botmesh/core has the tracing hooks: a minimal BotTracer interface, a no-op default and bot.tracing. No new dependency.
• @botmesh/otel adapts them to OpenTelemetry and also turns bot.metrics() into OpenTelemetry metrics.
• @botmesh/ai adds model, tool and retrieval spans to the same trace.

​

Quick start

pnpm add @botmesh/otel @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http @opentelemetry/sdk-metrics

// telemetry.ts — load it before the bot (e.g. `node --import ./telemetry.js bot.js`)
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'

export const sdk = new NodeSDK({
  serviceName: 'sales-bot',
  traceExporter: new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' }),
  metricReaders: [new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: 'http://localhost:4318/v1/metrics' }),
  })],
})
sdk.start()

// bot.ts
import { Bot } from '@botmesh/core'
import { aiPlugin } from '@botmesh/ai'
import { createOtelTracer, otelMetricsPlugin } from '@botmesh/otel'
import { sdk } from './telemetry.js'

const bot = new Bot({
  provider,
  telemetry: {
    tracer: await createOtelTracer(),
    // Same key on every replica so hashed ids correlate across processes and restarts.
    idKey: process.env.TRACE_ID_KEY,
  },
})
bot.plug(otelMetricsPlugin())   // bot.metrics() as OpenTelemetry metrics
bot.plug(aiPlugin({ /* … */ })) // AI spans join the same trace automatically

process.on('SIGTERM', async () => {
  await bot.stop()
  await sdk.shutdown()          // flushes the spans and metrics still buffered
})

​

Local backends

​

Jaeger

docker run --rm --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/jaeger:latest

​

Grafana Tempo + Grafana

services:
  tempo:
    image: grafana/tempo:latest
    command: ['-config.file=/etc/tempo.yaml']
    volumes: ['./tempo.yaml:/etc/tempo.yaml']
    ports: ['4318:4318', '3200:3200']
  grafana:
    image: grafana/grafana:latest
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: 'true'
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
    volumes: ['./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml']
    ports: ['3000:3000']

server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        http:
          endpoint: 0.0.0.0:4318
storage:
  trace:
    backend: local
    local: { path: /var/tempo/traces }
    wal: { path: /var/tempo/wal }

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    isDefault: true

​

What a message looks like

bot.inbound                      consumer · bot.event.kind=message · bot.inbound.outcome=dispatched
├─ bot.persist                   chat / contact / message writes
└─ bot.dispatch                  bot.queue.wait_ms · bot.dispatch.stage=matcher
   └─ bot.handler                bot.handler.stage=matcher · bot.matcher.kind=text · bot.flow.id=order
      └─ bot.conversation.turn   bot.conversation.reason=start · parked=true
         ├─ chat claude-…        gen_ai.* (model, tokens, finish reason)
         │  └─ (http span of the SDK call, from the SDK's own instrumentation)
         ├─ execute_tool lookup  bot.ai.tool.outcome=ok
         └─ bot.pace             delay + typing
            └─ bot.send          client · bot.send.outcome=ok
               └─ (http span of the provider call)

​

Span catalogue

​

Attributes

​

Conversation turns

• the first turn starts under the handler (or bot.trigger) that entered it — reason: start or dormant (a persistent flow resumed from its checkpoint); a c.goto() target starts under the turn that ran the goto — reason: goto;
• parking on a wait ends the turn with bot.conversation.parked = true;
• a reply opens the next turn under the dispatch span of the message that answered — reason: reply;
• a wait timeout opens a root turn linked to the first one — reason: timeout, since no message caused it; a c.idle() callback that fires while the conversation is parked runs in a root turn too (reason: idle), which parks again when the callback only replied, or carries the outcome when it ended or redirected the conversation;
• an error thrown by the conversation (or by an idle callback) is recorded on the turn it happened in;
• a conversation resumed from a checkpoint written by another replica or an earlier process (a persistent flow after a restart, a flow forwarded by coordination.handover, a hand-over stub restarted with fallback: 'restart') starts under the dispatch of the waking message and is linked to the turn that wrote the checkpoint — see Across replicas and restarts;
• the end of the conversation sets bot.conversation.outcome (completed, ended, guarded, cancelled, timedOut, retriesExceeded, error) on the open turn; a conversation aborted while parked (/reset, handoff, stop()) gets a short closing turn (reason: end).

​

Across replicas and restarts

• Only W3C-shaped ids are carried (32/16 lowercase hex, which OpenTelemetry produces). A tracer with other ids stores nothing and the turn simply has no link. A malformed stored value is ignored.
• The stored context is always the latest turn that wrote or parked: a hand-over stub is rewritten when a later turn parks (only with telemetry on; otherwise it is refreshed only before it expires), and the parked marker of a forwarded step refreshes it too. A turn whose context cannot be carried keeps the one already stored instead of erasing it.
• A custom handover.fallback function receives the context in info.trace, to link spans of its own.
• formatTraceparent(spanContext) and parseTraceparent(traceparent, tracestate?) are exported from @botmesh/core/spi for your own propagation.

​

Scheduled tasks

​

AI spans

​

Your own spans

bot.when('order', async (ctx) => {
  const order = await bot.tracing.trace('erp.lookup', { kind: 'client' }, () => erp.find(ctx.body))
  await ctx.reply(`Order ${order.id}`)
})

​

Metrics

​

Personal data

• Never recorded: message bodies, captions, prompts, completions, tool arguments and results, c.state, ask answers, unknown command text, raw provider payloads.
• Error messages may contain user text: only the error class and code are recorded unless you set telemetry.errorMessages: true.
• Ids follow telemetry.ids:
  • 'hash' (default): HMAC-SHA256, base64url, 16 characters. The key is telemetry.idKey; without it a random key per process is used, so hashes do not correlate across replicas or restarts. Set the same secret everywhere to search a chat’s traces.
  • 'raw': the id as the provider gives it — on WhatsApp that is a phone number, personal data.
  • 'omit': no ids at all.

​

Configuration

new Bot({
  provider,
  telemetry: {
    tracer,                          // BotTracer (required)
    ids: 'hash',                     // 'hash' | 'raw' | 'omit'
    idKey: process.env.TRACE_ID_KEY, // string | Uint8Array
    errorMessages: false,
    spans: { persist: true, pace: true, streamUpdates: false },
  },
})

​

Sampling

​

Other backends and tests

import { createRecordingTracer, createTestBot } from '@botmesh/core/testing'

const rec = createRecordingTracer()
const { bot, chat } = await createTestBot({ telemetry: { tracer: rec.tracer, ids: 'raw' } })
bot.when('hi', (ctx) => ctx.reply('hello'))
await chat.send('hi')
expect(rec.parentOf(rec.find('bot.send')!)?.name).toBe('bot.pace')

​

Limitations

• Durable replays: an inbound record replayed by the durability engine (after a crash or a retry) has no bot.inbound span; its bot.dispatch is a root span.
• Multi-replica: a conversation resumed on another replica (or after a restart) starts a new trace linked to the previous one, not a continuation of it (see Across replicas and restarts). A conversation handed over before telemetry was enabled, or by a replica without it, has no link.
• Third-party OpenTelemetry spans inside a resumed conversation (code after await c.ask()) parent to OpenTelemetry’s own context, which is still the first turn’s; the bot’s own spans parent correctly.
• Flushing is the SDK’s job: call sdk.shutdown() after bot.stop().