> ## Documentation Index
> Fetch the complete documentation index at: https://botmesh.afaya.com.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Redis Cluster

> @botmesh/redis-store runs on a Redis Cluster through an opt-in key layout. A bot that does not opt in keeps the exact key names, scripts and commands it used before.

## Why a separate layout

A Redis Cluster splits the keyspace into 16384 hash slots spread over several
masters. Two rules break the historical (`'legacy'`) key layout:

* a script, a `MULTI` or a multi-key command must only touch keys of **one
  slot**, or the server answers `CROSSSLOT`;
* `KEYS` and `SCAN` only see **one node**, so a listing built on them is
  silently incomplete.

The `'cluster'` layout fixes both. Every group of keys that one script touches
shares a **hash tag** (`{…}`), and listings read **index sorted sets** that live
in the same slot as their records, so they are exact even while slots move.

## Setup

Pass a node-redis cluster client, or let the component create one:

```ts theme={null}
import { createCluster } from 'redis'
import { Bot } from '@botmesh/core'
import { RedisStore, RedisDurability } from '@botmesh/redis-store'

const cluster = createCluster({ rootNodes: [{ url: 'redis://redis-0:6379' }], useReplicas: false })
cluster.on('error', (err) => logger.error({ err }, 'redis cluster'))
await cluster.connect()

const store = new RedisStore({ client: cluster, prefix: 'mybot' })   // layout 'cluster' is picked automatically
const bot = new Bot({
  provider,
  store,
  coordination: { coordinator: store.coordinator() },              // same client, same layout
  scheduler: { store: store.schedules(), lease: store.leases() },
  durability: { backend: new RedisDurability({ client: cluster, prefix: 'mybot' }), instanceId: process.env.HOSTNAME },
})
```

Or with an own client, created and closed by the store:

```ts theme={null}
const store = new RedisStore({
  cluster: {
    rootNodes: ['redis://redis-0:6379', 'redis://redis-1:6379'],
    // When the cluster announces addresses your process cannot reach (Docker, NAT):
    nodeAddressMap: (address) => ({ host: 'localhost', port: Number(address.split(':').pop()) }),
    defaults: { password: process.env.REDIS_PASSWORD },
  },
  prefix: 'mybot',
})
```

`RedisCoordinator` and `RedisDurability` accept the same `cluster` option.
`cluster` cannot be combined with `url` or `client`. A client a component creates
(from `url` or `cluster`) always gets an `'error'` listener, so a dropped node
connection or a failover is retried instead of crashing the process; pass
`logger` to see those errors. A client you pass in is yours: give it an
`'error'` listener, as above.

| Option | Where | Default |
| - | - | - |
| `layout: 'legacy' \| 'cluster'` | `RedisStore`, `RedisCoordinator`, `RedisScheduleStore`, `RedisLeaseProvider` | `'cluster'` with a cluster client or `cluster` options, else `'legacy'` |
| `cluster: { rootNodes, nodeAddressMap?, defaults?, maxCommandRedirections? }` | `RedisStore`, `RedisCoordinator`, `RedisDurability` | none |
| `allowLegacyData: true` | `RedisStore` | `false` |

Rules:

* `layout: 'legacy'` with a cluster client throws a `RangeError`.
* In the cluster layout the prefix cannot contain `{` or `}` (it would break the
  hash tags).
* `layout: 'cluster'` also works on a **single node**. That is how an existing
  keyspace is migrated in place, and how you can test the layout locally.
* `store.coordinator()`, `store.schedules()` and `store.leases()` inherit the
  store's layout. Use the same layout and prefix on every replica.
* **Switch every replica at once.** Replicas on different layouts of one prefix
  see different keyspaces, and their turns and locks stop excluding each other.
  A legacy-layout `RedisStore` whose prefix carries the layout marker logs a
  warning at `start()` (`logger.warn`, else `logger.error`); it still starts.

## Key layout

`p` is the prefix; variable segments are percent-encoded (no `:`, `{`, `}` or
glob character survives), so a chat id can never break a hash tag.

| Data | Keys | One slot per |
| - | - | - |
| Messages | `p:{m:<chat>}:z` (log) and `p:{m:<chat>}:i:<member>` | chat |
| Chats, contacts, checkpoints, schedules | `p:{ch:<b>}:r:<id>` and the index `p:{ch:<b>}:idx` (`ct`, `cp`, `sc`) | bucket `b` (32 per kind) |
| User data (`ctx.user`) | `p:{ud:<user>}:e:<ns>:<key>` and `p:{ud:<user>}:idx` | user |
| Locks | `p:{lk:<key>}:l` and the fence `p:{lk:<key>}:f` | lock |
| Coordinator turns | `p:co:{t:<chat key>}:seq\|tix\|serving\|owner\|last` | chat |
| Leaderships | `p:co:{ld:<name>}` and `…:term` | name |
| Unchanged | `p:conv:*`, `p:cursor:*`, `p:lease:*`, `p:co:rep:*`, `p:{durable:<table>}:*` | (one key, or already tagged) |
| Layout marker | `p:{meta}:layout` = `cluster:v1` | |

The bucket of a record is FNV-1a of its id modulo 32 (part of the `cluster:v1`
contract). Bucket tags carry no prefix, so the bots that share a cluster spread
each kind over the same 32 slots. That is about 10 per master on a 3-master
cluster, and keeps listings to 32 commands.

### What changes inside

* Every script declares all its keys (the store checks they share a slot before
  the call). The cluster layout uses scripts instead of `MULTI`.
* `deleteChat` drops the messages in chunks of 1000, each chunk atomic: an item
  only goes together with its log entry. The chat record goes last. A message
  saved during the delete survives intact. It is no longer one single script,
  because the log and the record live in different slots.
* Checkpoint and user-data indexes carry the server-side expiry as their score
  and are pruned on read, so expired entries disappear from listings like
  expired keys.
* A coordinator waiting for its turn cannot read another replica's heartbeat
  from its script (another slot). The turn script reports the earlier holder,
  and the coordinator probes that holder's heartbeat with a separate one-key
  call before skipping its ticket. Mutual exclusion still rests on the atomic
  owner-key check in the chat's slot.
* `TRYAGAIN` and `CLUSTERDOWN` replies, and a `MOVED`/`ASK` redirect node-redis
  did not follow, are retried with backoff (10 attempts, about 2.5 s; tune with
  `withClusterRetry(client, { attempts, baseMs })` and pass the wrapped client).
  These replies come before anything runs. A lost reply is never retried.

### node-redis 4 notes

The package works around two node-redis 4.x cluster quirks:

* `cluster.eval(script, { keys })` is routed by the wrong argument, so it can
  land on a random master and never follow `MOVED`. The components send `EVAL`
  routed by its first key.
* After a topology change a `MULTI` can fail with `EXECABORT` without node-redis
  re-reading the topology. The cluster layout does not use `MULTI`.

## Migrating an existing keyspace

There is no dual reading of both layouts: it would break the atomicity of the
compare-and-set writes, fences and indexes. Migrate offline instead.

1. **Stop every bot of the prefix.** The helper refuses while a coordinator
   heartbeat or a pending chat ticket exists on the source, or on a separate
   target, unless `allowLive: true`.

2. Copy the keyspace:

   ```ts theme={null}
   import { createClient } from 'redis'
   import { migrateToClusterLayout, inspectKeyspace } from '@botmesh/redis-store'

   const source = createClient({ url: 'redis://old:6379' })
   await source.connect()
   console.log(await inspectKeyspace({ client: source, prefix: 'mybot' }))   // counts per layout
   const report = await migrateToClusterLayout({ source, prefix: 'mybot', dryRun: true })
   await migrateToClusterLayout({ source, prefix: 'mybot' })                 // in place, or:
   await migrateToClusterLayout({ source, target: cluster, prefix: 'mybot' }) // straight into a cluster
   ```

3. Start the bots with the cluster layout (`layout: 'cluster'` on the single
   node, or a cluster client). After an in-place migration you can move the data
   into the cluster with your usual tools.

4. Once verified, delete the source keys with
   `migrateToClusterLayout({ source, prefix, deleteSource: true })`.

What the helper does:

* **Messages** are read from each chat's log (never by parsing item keys). Items
  that do not belong to their chat are skipped, and items no log references are
  reported in `report.orphans`.
* **Checkpoints and user data** keep their remaining TTL.
* **Lock fences and leadership terms** are raised with "set if greater", so
  fences never go backwards. Turn counters are short-lived and are not copied.
* **Schedules**, including deleted markers, keep their fencing tokens, which
  never go backwards: a source record with a higher token replaces the target's
  even without `overwrite`, and an overwritten record keeps the higher token.
* Conversations, cursors, leases and the durability journal keep their names.
  They are copied only into another server or prefix.
* **Idempotent and resumable**: an existing target key wins unless
  `overwrite: true`. The layout marker is written last, and only by a
  complete run: a run limited by `include` leaves it unset
  (`report.layoutMarker` says what happened), so the `start()` guard below
  stays armed until every namespace is copied.

Starting a cluster-layout store on a single node that still holds legacy keys
under its prefix throws ("run migrateToClusterLayout() first"). This check scans
at most 10,000 keys. Pass `allowLegacyData: true` to start anyway.

### Rollback

`direction: 'cluster-to-legacy'` copies the cluster layout back into the legacy
one. The target is a single node; the source is a single node or a **Redis
Cluster client** (scanned on every master), so a keyspace that already lives on
a cluster can be rolled back too.

The rollback **mirrors** by default (`mirror: true`): the legacy target is made
to match the cluster-layout source. Records overwrite, and legacy records the
source no longer holds are deleted (`report.pruned`). Without that, the legacy
keys a forward pass left behind (it keeps them unless `deleteSource`) would win
over newer writes, and records deleted since (`deleteChat`, `clearCheckpoint`,
`ctx.user` deletes) would come back. A schedule only in the legacy layout
becomes a deleted marker that keeps its fencing token; lock fences and
leadership terms are only ever raised. Run it with `dryRun: true` first to see
the counts. Pass `mirror: false` for a copy-only rollback (existing legacy keys
win).

A complete rollback also removes the layout marker, so a cluster-layout store
started by mistake refuses the restored legacy data. Before migrating forward
again, delete the cluster-layout keys (roll back with `deleteSource: true`),
otherwise the stale cluster-layout records win the next forward pass.

## Limits

* **Asynchronous replication.** As with Sentinel, a failover can lose the last
  acknowledged writes: a lease granted twice, a compare-and-set update lost, a
  fence or term handed out again. This is not Redlock. Set
  `min-replicas-to-write 1` and rely on the durable outbox's idempotency for
  sends.
* Each durability table sits in one slot (`{durable:<table>}`), as before.
* Slot moves: listings stay exact. Multi-key writes on a slot being moved get
  `TRYAGAIN` and are retried until the move finishes.
* Read from masters only (`useReplicas: false`): compare-and-set loops and
  checkpoints must see their own writes.

## Testing locally

```bash theme={null}
docker run -d --name botmesh-it-redis-cluster -e IP=0.0.0.0 -e INITIAL_PORT=47100 \
  -e MASTERS=3 -e SLAVES_PER_MASTER=1 -p 127.0.0.1:47100-47105:47100-47105 grokzen/redis-cluster:7.0.10
BOTWA_TEST_REDIS_CLUSTER_URL=redis://127.0.0.1:47100 pnpm --filter @botmesh/redis-store test
# optional: failover and slot migration under load
BOTWA_TEST_REDIS_CLUSTER_CHAOS=1 BOTWA_TEST_REDIS_CLUSTER_URL=redis://127.0.0.1:47100 pnpm --filter @botmesh/redis-store test
docker rm -f botmesh-it-redis-cluster
```

These are the package's own suites (run from a checkout of the repository); the
chaos tests change the cluster's topology, so use a disposable cluster.
`CONTRIBUTING.md` has the details.

See also [Running N replicas](https://botmesh.afaya.com.co/guide/replicas.md) and [Durable message processing](https://botmesh.afaya.com.co/guide/durability.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.