How we tracked down (and fixed) an encryption bug that made conversations permanently unreadable

Context

Bluvy is an end-to-end encrypted messaging app built on Bluesky (AT Protocol). Encryption relies on MLS (Messaging Layer Security), an IETF standard designed to secure group conversations — including one-to-one conversations, which are treated as a minimal two-member group.

One day, users started seeing conversations become unreadable. Not just the occasional lost message: entire conversations became stuck, with GroupNotReady errors looping endlessly and a “repair” button that repaired nothing.

What follows is the full story of the investigation: how the bug was found, why it was so difficult to fix, and the ten other, more subtle bugs that surfaced while testing the fix under real-world conditions.

MLS in Five Minutes, for the Rest of This Article

You do not need to be a cryptographer to follow along, but a few concepts will come up repeatedly:

• An MLS group represents all devices participating in a conversation. A user with a phone and a laptop therefore has two devices in the same group.

• Every change to the group — adding a device, removing one, rotating key material — is called a commit. Each commit advances a counter: the epoch. The group moves from epoch 0 to epoch 1, then 2, and so on, one commit at a time.

• A device must apply commits one by one, strictly in order. It cannot skip one, and it cannot replay one that has already been processed — the underlying cryptography, namely the group key tree, simply does not allow it.

• For a new device to join an existing group, it receives a Welcome: an invitation message containing everything it needs to know the state of the group at the moment it joins.

The crucial point — and the one that explains why this investigation was so difficult — is that there is no magical catch-up mechanism. If a device gets its epoch out of sync, or if two devices in the same group end up with different commit histories — what we call a fork, much like in Git — there is no cryptographic way to merge those two realities back together. The only option is to rebuild a fresh group.

Method: Never Guess, Always Verify

Before fixing anything, the process was:

1. 1.

   Map the existing architecture: the complete lifecycle of a commit, an epoch, and a conversation, on both the client and server sides.

2. 2.

   Empirically prove every hypothesis by running real code against the real cryptographic library (ts-mls), using real simulated devices — never making assumptions about how a third-party dependency behaves.

3. 3.

   Verify against the real database that the detection mechanisms actually found real broken conversations matching the exact signature predicted by the theoretical analysis.

4. 4.

   Systematic TDD: write the test that proves the bug, watch it fail, apply the fix, then watch it pass.

That rigor paid off: several of the bugs below would never have been discovered through code review alone.

A note about the SQL snippets in this article. Some snippets below look like SQL built by concatenating values (${userDid}, ${conversationId}, etc.). That is never the case in the real code: these snippets are simplified for readability. In reality, every query goes through the ORM query builder (Drizzle) or its sql tag, which automatically turns every interpolated value into a bound parameter (?) — never text injected into the query. A value such as ${conversations.id} references a schema column, and is therefore static and safe by construction; a value such as ${userDid} becomes a parameter bound by the SQLite driver. There is no string concatenation anywhere in the actual code — this was explicitly verified for this article.

Part 1 — The Four Root Causes

Bug #1: The Guard That Rejected the Next Legitimate Commit (Client)

Every time a commit arrives, the client has to answer one question: “Have I already applied this commit?” The comparison used to answer that question was off by one:

// BEFORE
if (currentEpoch >= epoch) {
  return null; // "already applied, ignore"
}

The problem: if currentEpoch is 4 and the received commit is precisely the one that moves the group to epoch 4 (epoch === 4), the condition evaluates to true — so the commit is ignored even though it has never been applied. The device believes it is up to date, while in reality it has fallen one step behind the rest of the group.

// AFTER
if (currentEpoch > epoch) {
  console.log('[MLS:observability] applyCommit', { result: 'skipped_already_applied' });
  return null;
}

A simple > instead of >=. But in a protocol where every epoch strictly depends on the previous one, that single line was enough to derail an entire conversation, message after message.

How we proved it: instead of trusting our interpretation of the code, we wrote a standalone Node.js script that imports the real ts-mls library, creates a group with three devices, advances the epochs, and observes exactly when the legitimate commit gets rejected. Proof by execution, not deduction.

Bug #2: The Catch-Up Query That Skipped the Exact Commit We Needed (Server)

When a device reconnects after being offline, it asks the server: “Give me every commit published after my current epoch.” The query mistakenly excluded the very first commit the device needed:

// BEFORE
export function getPendingMlsCommits(conversationId: string, afterEpoch: number) {
  return db.select().from(mlsCommits).where(
    and(
      eq(mlsCommits.conversationId, conversationId),
      gt(mlsCommits.epoch, afterEpoch), // strictly greater
    )
  ).all();
}

// AFTER
gte(mlsCommits.epoch, afterEpoch) // greater than or equal

gt (greater than) excludes the commit whose epoch is exactly equal to afterEpoch — which is precisely the commit the device, already one step behind because of Bug #1, was waiting for in order to catch up.

Together, these two bugs created an unforgiving domino effect: a device falls one step behind because of Bug #1, tries to catch up after reconnecting, but the catch-up mechanism itself is missing a step because of Bug #2. The device never fully catches up, and the gap can grow indefinitely.

Bug #3: The “Brute-Force” Reset When Local State Was Lost (Server)

This one is the most interesting to understand, because its failure mode is not immediately obvious.

Imagine a device loses its local MLS state — after reinstalling the app, clearing the cache, and so on — even though it had already joined a conversation. What should the server do when that device comes back and says, “I no longer know where I am”?

The old code concluded that the entire group should be recreated from scratch, by that device itself:

// BEFORE (simplified logic)
export function claimInitiatorSlot(conversationId: string, deviceId: string) {
  const device = getDeviceLostState(conversationId, deviceId);
  if (device) {
    // Full reset: delete the entire shared commit history...
    db.delete(mlsCommits).where(eq(mlsCommits.conversationId, conversationId)).run();
    db.update(conversations)
      .set({ mlsInitializedAt: null })
      .where(eq(conversations.id, conversationId)).run();
    return 'initiator'; // ...and THIS device rebuilds the cryptographic tree on its own
  }
}

The problem is brutally simple: the other devices in the conversation have not lost anything. They continue happily using the old cryptographic tree, while the device that just “lost its memory” builds a completely new one in parallel, without anyone realizing it. The group splits into two incompatible MLS realities — a fork — and, as explained earlier, there is no cryptographic way to merge two trees once they have diverged.

In production logs, this bug manifested as a device triggering a reset, re-inviting only some of the other devices belonging to the same account — those that happened to be connected at that exact moment — and leaving the others, still perfectly real and legitimate, stranded forever on a dead tree.

The fix completely changes the philosophy: losing the local state of one device proves nothing about the validity of the shared history. That device should simply wait to be invited again normally, exactly like any new device joining a conversation for the first time:

// AFTER
if (deviceLostState) {
  logger.warn('[MLS:observability] claimInitiatorSlot lost-state re-provision nudge', {
    conversationId, deviceId, reason: 'device_lost_state_no_pending_welcome',
    historyRemoved: false, // we NEVER touch the shared history anymore
  });

  // Simply notify the other devices belonging to the same account, if connected,
  // so they can start a normal invitation — the same path used for adding a new device.
  emitToUser(device.userDid, 'device:new', {
    deviceId: device.id, deviceName: device.name, platform: device.platform,
  });

  return 'joiner'; // wait for an invitation, just like a brand-new device
}

No new cryptographic code was required: the solution was simply to reuse the existing, proven device-addition mechanism instead of inventing a destructive parallel path.

Bug #4: Duplicate Private Conversations (Server)

A more conventional issue: two quick clicks on “start a conversation” — or a duplicated browser tab — could create two separate conversations for the same pair of users, each with its own MLS group.

The fix introduces a calculated key that remains stable regardless of the order of the two participants, plus a database uniqueness constraint:

// The two DIDs (Bluesky identifiers) are always sorted in the same order,
// regardless of who started the conversation.
function dmPairKey(didA: string, didB: string): string {
  return didA < didB ? `${didA}|${didB}` : `${didB}|${didA}`;
}

CREATE UNIQUE INDEX conversations_dm_pair_key_unique
  ON conversations (dm_pair_key)
  WHERE dm_pair_key IS NOT NULL;

On the application side, conversation creation is now race-tolerant instead of failing:

// BEFORE: check, then create (two separate queries) — race condition possible
const existing = findExistingDm(a, b);
if (existing) return existing;
return createConversation(a, b);

// AFTER: attempt creation, and if the database rejects a duplicate,
// simply return the winning conversation instead of failing.
try {
  return db.transaction(tx => {
    tx.insert(conversations).values({ id, type: 'dm', dmPairKey: pairKey }).run();
    // ...
  });
} catch (err) {
  if (isUniqueConstraintViolation(err, 'dm_pair_key')) {
    return db.select().from(conversations).where(eq(conversations.dmPairKey, pairKey)).get();
  }
  throw err;
}

Part 2 — Repairing What Is Already Broken

The four fixes above prevent the problem from happening again. But they cannot repair conversations that were already forked by Bug #3. And as established earlier: it is cryptographically impossible to merge two MLS trees once they have diverged.

The only clean option is to create an entirely new conversation — and therefore a new, healthy MLS group — between the same participants, while archiving the old one without ever deleting anything: not its messages, not its metadata.

The Schema: A Pointer, Never a Deletion

// New column: points to the conversation that replaces this one.
supersededByConversationId: text('superseded_by_conversation_id'),

export function recreateConversation(requestingUserDid: string, oldConversationId: string) {
  // ... membership checks, idempotency checks ...

  return db.transaction((tx) => {
    // Crucial and somewhat counterintuitive step: release the old "pair key"
    // BEFORE inserting the new row. SQLite checks UNIQUE constraints statement
    // by statement, not at the end of the transaction, so inserting the new
    // conversation first would conflict with the old one, which still owns
    // the same pair key.
    const updated = tx.update(conversations)
      .set({ dmPairKey: null, supersededByConversationId: newConversationId })
      .where(and(
        eq(conversations.id, oldConversationId),
        sql`superseded_by_conversation_id IS NULL`, // protects against concurrent double execution
      ))
      .run();

    if (updated.changes === 0) {
      throw new AppError(409, 'Conversation was already recreated', 'ALREADY_RECREATED');
    }

    tx.insert(conversations).values({ id: newConversationId, type: 'dm', dmPairKey: pairKey }).run();
    // ... add members to the new conversation ...
  });
}

That operation ordering was itself discovered thanks to the first test attempt: the very first real call unexpectedly returned “already recreated” — a clue that instruction ordering inside the transaction was the actual culprit, not the business logic.

Copy the History, Never Move It

On the client side, the old conversation history must be copied into the new one so the user sees a continuous thread, without ever emptying the original conversation.

// FIRST ATTEMPT — buggy, never shipped to production
async spliceHistory(fromConversationId: string, toConversationId: string) {
  const messages = await this.getAllMessages(fromConversationId);
  for (const m of messages) {
    await this.store({ ...m, conversationId: toConversationId }); // same ID!
  }
}

The problem: the local cache database (IndexedDB) indexes messages by their raw identifier, without including the conversation ID. By reusing the same identifier, store() did not copy the message — it silently moved it by overwriting the existing entry. The old conversation would have been emptied at the exact moment we were trying to preserve it.

// FIXED — caught and blocked by tests before ever reaching production
async spliceHistory(fromConversationId: string, toConversationId: string) {
  const splicedId = (sourceId: string) => `spliced:${toConversationId}:${sourceId}`;
  const messages = await this.getAllMessages(fromConversationId);

  for (const m of messages) {
    await this.store({
      ...m,
      id: splicedId(m.id),
      conversationId: toConversationId,
    });
  }
}

A synthetic, deterministic identifier solves the issue: the copy can never collide with the original, and the operation becomes safely repeatable without side effects — in other words, idempotent.

The “Repair” Button That Appeared on Perfectly Healthy Conversations

A small UI issue discovered by testing the real application rather than simply reading the code: the “Restore encryption” button was answering the wrong question.

// BEFORE — "ready" means "explicitly confirmed as ready"
this.mlsGroupReady = this.coordinator.isConversationReady(this.conversationId);

A brand-new conversation, or one that has just been recreated through the mechanism above, has never been explicitly marked as ready. It simply has not been checked yet, which is very different from being “broken.”

With this code, every new conversation displayed the repair button, encouraging users — and testers — to click it repeatedly and trigger unnecessary recreation cascades.

// AFTER — show the button only when the state is EXPLICITLY failed
this.mlsGroupReady = !this.coordinator.isConversationFailed(this.conversationId);

A new or freshly recreated conversation will establish encryption transparently when the first message is sent. No intervention is required.

Part 3 — What Real-World Testing Revealed

Once the root causes were fixed, real-world testing — a real browser and a real database — uncovered six additional problems completely independent of the original MLS bug.

A 500 Error While Saving Encrypted Message Backups

After the initial fixes, a completely new side effect appeared: encrypted backup synchronization started systematically failing with a generic server error.

Digging directly into the server logs revealed the real SQL error, which the API had hidden behind a generic INTERNAL_SERVER_ERROR:

SqliteError: FOREIGN KEY constraint failed
    at storeMessages (backup.service.ts:180:6)

The affected table had inherited integrity constraints from an old migration that had later been modified without ever being reapplied to the existing database — creating a silent mismatch between the migration file and the actual table state.

-- Fixed by rebuilding the table so it exactly matches the schema
-- actually used by the application, without losing any data.
CREATE TABLE __new_user_message_backups (
  id text PRIMARY KEY NOT NULL,
  user_did text NOT NULL,
  conversation_id text NOT NULL,
  message_id text NOT NULL,
  -- ... unchanged columns ...
  FOREIGN KEY (user_did) REFERENCES users(did) ON DELETE CASCADE
  -- obsolete constraints on conversation_id / message_id are gone
);

INSERT INTO __new_user_message_backups SELECT * FROM user_message_backups;
DROP TABLE user_message_backups;
ALTER TABLE __new_user_message_backups RENAME TO user_message_backups;

The Most Devious Bug in the Entire Investigation: A Function That Deleted Its Own Work

This was by far the most serious of the secondary bugs — arguably even more insidious than the four original root causes, because it affected every newly created conversation, rather than one particular edge case.

When an MLS group is created for the first time, two steps happen consecutively within the same transaction: save the founding commit — the one that creates the group at epoch 0 — and then officially mark the conversation as “initialized.”

// storeMlsCommit() — excerpt
tx.insert(mlsCommits).values({ conversationId, senderDeviceId, commit, epoch }).run();
// ... store the Welcome for the invited device ...
markGroupInitialized(conversationId); // <-- called immediately after inserting the commit above

And here was markGroupInitialized, whose purpose was to clean up any orphaned commits left behind by a previous aborted attempt:

// BEFORE
export function markGroupInitialized(conversationId: string) {
  const conv = getConversation(conversationId);

  if (!conv?.mlsInitializedAt) {
    // Delete every commit... but we cannot distinguish "orphaned" commits
    // from the one that was literally just created by the caller!
    db.delete(mlsCommits)
      .where(eq(mlsCommits.conversationId, conversationId))
      .run();
  }

  db.update(conversations)
    .set({ mlsInitializedAt: Date.now() })
    // ...
}

At the exact moment storeMlsCommit() calls this function, mlsInitializedAt is still null — because setting it is precisely the purpose of this function. The !conv?.mlsInitializedAt condition therefore evaluates to true, and the cleanup runs... deleting the founding commit that was inserted moments earlier, inside the same transaction.

The result: a brand-new conversation marked as “initialized,” with an invitation successfully sent to the other device — but zero commits stored in the database. A broken and unrecoverable state from the moment the conversation is created, with no error reported anywhere.

This bug was discovered by querying real conversations in the database: several conversations created before the fix had never had their epoch 0 stored — the exact signature predicted by the bug.

// AFTER — an explicit parameter distinguishes the two call contexts.
export function markGroupInitialized(
  conversationId: string,
  opts: { purgeStaleCommits?: boolean } = {},
) {
  const { purgeStaleCommits = true } = opts;
  const conv = getConversation(conversationId);

  if (purgeStaleCommits && !conv?.mlsInitializedAt) {
    db.delete(mlsCommits)
      .where(eq(mlsCommits.conversationId, conversationId))
      .run();
  }

  db.update(conversations)
    .set({ mlsInitializedAt: Date.now() })
    // ...
}

// The only caller that has just inserted ITS OWN valid commit inside
// the same transaction explicitly disables the cleanup:
markGroupInitialized(conversationId, { purgeStaleCommits: false });

The historical call from the POST /welcome route — which does not store any commit itself — keeps the default behavior unchanged. Only the new call context required different handling.

Missing History for a Device That Never Saw the Event

The history-copying mechanism described in Part 2 was originally triggered in only two situations: the device was connected in real time when the switch to the new conversation happened, or it explicitly navigated to the old conversation URL.

A device that opened the new conversation directly from the conversation list — without ever encountering either trigger — silently missed the history copy.

// The server now exposes a reverse pointer:
// "Does this conversation have a previous version?"
const predecessorIdSubq = sql<string | null>`(
  SELECT id FROM conversations AS pred
  WHERE pred.superseded_by_conversation_id = ${conversations.id}
  LIMIT 1
)`;

// The client systematically checks this pointer whenever a conversation
// is opened. Since the operation is idempotent, repeating it every time
// is safer than relying on a one-time event.
const predecessorId = this.conversation.predecessorConversationId;

if (predecessorId) {
  await this.messageCacheSvc.spliceHistory(
    predecessorId,
    this.conversationId,
  );
}

The Authentication Token That Was Never Refreshed Proactively

The authentication token expires after 15 minutes. It was only refreshed reactively, after the first failure — visible in the browser console as occasional 401 Unauthorized responses followed by a silent retry.

// BEFORE — nothing proactive: wait for failure, then react.
async request(method, path) {
  try {
    return await this.send(method, path);
  } catch (err) {
    if (!this.is401(err)) throw err;

    await this.refreshToken();
    return this.send(method, path); // retry once
  }
}

// AFTER — schedule a proactive refresh based on the actual expiration
// timestamp stored in the JWT itself.
private scheduleProactiveRefresh(accessToken: string) {
  const exp = this.decodeJwtExp(accessToken);
  const marginMs = 60_000; // one-minute safety margin
  const delay = Math.max(
    1_000,
    exp * 1000 - Date.now() - marginMs,
  );

  this.refreshTimer = setTimeout(
    () => this.refreshTokens(),
    delay,
  );
}

The reactive mechanism remains in place as a safety net — for example, if the application was in the background and the operating system suspended the timer — but it is now needed far less often.

Secondary Accounts That Silently Stopped Working

The application allows multiple accounts to be linked on the same device.

The mechanism that quietly checks unread messages for inactive accounts every 15 seconds intentionally used a different authentication path from the active account — with a token attached manually rather than managed automatically.

// BEFORE
private async fetchHasUnread(did: string) {
  const token = await this.tokenRepo.getAccessToken(did);

  try {
    return await this.api.get('/v1/conversations?limit=20', {
      skipAuth: true, // intentionally bypasses active-account automatic refresh
      headers: {
        Authorization: `Bearer ${token}`,
      },
    });
  } catch {
    return false; // and nothing ever refreshes THIS account's token
  }
}

skipAuth: true was necessary: the HTTP client's automatic refresh mechanism only knows about the active account, and using it here would have refreshed the wrong account.

But the side effect was that nothing at all refreshed the secondary account's token.

Once it expired, that account silently stopped receiving unread-message notifications forever — until the user manually switched to it.

// AFTER — dedicated refresh explicitly scoped to THIS specific account.
} catch (err) {
  if (
    !(err instanceof HttpErrorResponse) ||
    err.status !== 401
  ) {
    return false;
  }

  const refreshedToken =
    await this.authSvc.refreshTokensForDid(did);

  if (!refreshedToken) return false;

  return await this.fetchHasUnreadWithToken(refreshedToken); // retry once
}

The same flaw — and the same fix — applied to push-notification token registration for secondary accounts.

Switching Accounts Broke Decryption — The Last and Most Subtle Bug

This one appeared last, and it is probably the most instructive bug of the entire investigation.

For each conversation, the application keeps a small amount of internal state in memory: “ready,” “failed,” “number of consecutive decryption failures,” and so on.

That state lives in the browser as structures indexed only by conversation ID:

private readonly states =
  new Map<string, ConversationMlsState>();

private readonly failedRecovery =
  new Map<string, { attempts: number; timerId: any }>();

// ...and several similar structures

On the development device used to test the application, multiple accounts are linked and talk to each other — meaning the same conversation ID is literally shared by two different accounts.

When switching accounts without fully reloading the page — a normal feature of the application — this in-memory state was never reset.

In practice: Account A opens a conversation, and its state becomes “ready” in this shared memory. The user then switches to Account B, which shares the same conversation with Account A.

A shortcut in the code, intended to avoid unnecessary work, relies on that memory:

async ensureGroupReady(convId: string, ...) {
  if (this.isConversationReady(convId)) {
    return; // "already ready, nothing to do"
  }

  // ...otherwise, actually establish the group for THIS account
}

The shortcut returns “already ready” — but that information belongs to Account A, not the currently active Account B.

Account B's MLS group has never actually been established in this session.

Even worse, an automatic recovery timer scheduled for Account A before the switch could fire afterward and operate on Account A's MLS group while Account B is now the account visible on screen — a JavaScript closure capturing the wrong account at the wrong time.

A full page reload clears all this memory at once and starts from a clean state — exactly matching the observed symptom: “You have to clear the browser cache to make it work again.”

// AFTER — detect a real account change and fully reset session-scoped state.
override async initializeForSession(
  user: UserProfile,
  device: DeviceInfo,
) {
  if (
    this.currentUserProfile &&
    (
      this.currentUserProfile.did !== user.did ||
      this.currentSessionDevice?.id !== device.id
    )
  ) {
    // Properly cancel pending timers before discarding them,
    // otherwise they may fire later against the wrong account.
    for (const entry of this.failedRecovery.values()) {
      if (entry.timerId !== undefined) {
        clearTimeout(entry.timerId);
      }
    }

    this.failedRecovery.clear();
    this.states.clear();
    this.pendingDerivations.clear();
    this.inFlightDecrypts.clear();
    this.commitFailureCounts.clear();
    this.decryptionFailures.clear();
    this.readyTimestamps.clear();
  }

  this.currentUserProfile = user;
  this.currentSessionDevice = device;

  // ...continue initialization with clean session-scoped state
}

The lesson from this final bug is that in-memory state can work perfectly throughout the entire normal lifetime of a session and still become incorrect as soon as a new dimension is introduced that the original design never anticipated — in this case, multiple accounts sharing the same identifier space.

Nothing in conventional unit tests — which test one session at a time — could reveal this. Only a real-world test involving an actual account switch exposed it.

Conclusion

Eleven distinct bugs in total: four cryptographic root causes and seven discovered while testing the fixes under real-world conditions — each with its own dedicated regression test, written before the fix to prove that it actually captured the problem.

Two lessons stand out from this investigation:

In a system where after-the-fact repair is cryptographically impossible, the only reliable strategy is to strictly prevent divergence — and, when divergence happens anyway, rebuild safely without sacrificing the history that has already been accumulated, rather than attempting an illusory repair.

None of these bugs would have been uncovered by code review alone. The four root causes required proof through actual execution against the real cryptographic library. The seven secondary bugs were all discovered by testing the application under real-world conditions — browser open, real database, actual account switching — never by simply reading the code one more time.

The update fixing all of these issues will be deployed to production within the next few hours.