Post-mortem: How a Database Migration Deleted Conversations

In summary

On August 10, 2026, a server update for Bluvy Messenger, which notably introduced support for group conversations, caused a major incident: the content of 21 existing conversations was deleted.

The conversations themselves remained present in the database, but their messages, members, encryption state, and read receipts were deleted.

No private keys, plaintext messages, or encrypted backups were compromised. These data are either never stored on the server or are stored in database tables that were completely independent from the one affected by the incident.

The root cause has since been identified with certainty and reproduced locally. The issue has been fixed across all affected execution paths.

More importantly, we used this incident as an opportunity to completely review our backup strategy. From now on, no database migration can be executed unless a complete and verified backup has been successfully created beforehand. The server also performs an automatic backup every two hours.

This post-mortem explains what happened, why it happened, how we confirmed the root cause, and what we have changed to prevent a similar incident from causing permanent data loss again.

What happened?

The server update deployed on August 10 introduced support for group conversations.

This required changes to the conversations table, including new information such as:

the group name;

its avatar;

its owner;

the conversation type (dm or group).

A database constraint also had to be added to ensure that a conversation type could only contain an allowed value.

The problem is that SQLite does not allow certain types of constraints to be modified directly on an existing table.

The migration therefore had to use SQLite's standard approach:

Create a new table with the new structure.

Copy the data from the old table.

Delete the old table.

Rename the new table so that it takes the original table's name.

This is a normal and well-established technique, and it had already been used successfully elsewhere in the project.

However, the conversations table is referenced by several other tables:

conversation members;

messages;

MLS encryption state;

read receipts.

These relationships use cascading deletes: when a conversation is deleted, all data associated with it is automatically deleted as well.

This behavior is intentional and necessary during normal operation.

The problem occurred precisely when the migration had to delete the old conversations table and replace it with the newly created one.

The detail that caused the incident

The migration actually accounted for the cascading-delete problem.

Before deleting the old table, it attempted to temporarily disable foreign-key enforcement and cascading deletes, then re-enable them after the migration.

The logic was correct in principle.

But there was one critical detail.

Bluvy's migrations are executed inside a single database transaction.

SQLite does not allow this configuration to be changed while a transaction is active.

More importantly, the request to change the configuration did not fail in the way we would normally expect. SQLite simply ignored it.

The cascading-delete protection therefore remained active throughout the migration.

When the old conversations table was deleted, SQLite correctly applied the cascading-delete rules.

All dependent data was consequently deleted:

messages;

conversation members;

encryption state;

read receipts.

The new conversations table itself survived because it had already been created and populated with the original conversation records.

The result was therefore particularly deceptive: the conversations still existed in the database, but everything associated with them was gone.

The operation completed without an obvious error and without a specific warning appearing in the server logs.

This made the incident particularly difficult to detect immediately.

The migration contained the correct intention and the correct instruction, but a SQLite behavior caused that instruction to be silently ignored in the context in which it was executed.

What data was affected?

The incident affected 21 existing conversations at the time of the migration.

The deleted data included:

messages;

conversation members;

MLS encryption state;

read receipts.

However, several categories of data were not affected.

Private keys

Private keys are never stored on the server.

They were therefore neither exposed nor deleted.

Plaintext messages

The Bluvy server never receives plaintext messages.

Messages are encrypted end-to-end before being transmitted to the server.

Encrypted backups

Backups used, among other things, to restore message history on a new device are stored in a separate database table.

They were not affected by the migration.

User accounts and devices

User accounts, Bluesky identities, and registered devices were not affected.

How did we identify the cause?

The initial symptom was straightforward: conversations were no longer displayed correctly in the application, while the backup system indicated that some messages were still recoverable.

That contradiction was important because it prevented us from stopping at a superficial explanation.

The investigation was performed directly against the production database in read-only mode to determine exactly which data was still present.

We then systematically tested and eliminated several possible explanations.

We verified:

that the server was using the correct database;

that no test had accidentally been executed against production;

that the data had not simply been hidden by an application-level issue;

that the affected database tables were genuinely empty.

Most importantly, we reproduced the incident.

The exact migration file used in production and the actual migration mechanism used by the server were executed locally against a test database containing known data.

The result was identical: the same dependent data was deleted during the migration.

The root cause was therefore no longer a hypothesis.

The incident was reproducible, and its cause was proven.

An attempt to recover the data using SQLite's WAL

Before considering the data permanently lost, we also investigated SQLite-level recovery options.

SQLite uses a WAL (Write-Ahead Log) to record database changes before they are committed to the main database file.

At the time of the investigation, this journal still existed on the server.

We therefore developed a tool specifically to analyze its contents and reconstruct previous versions of database pages.

This analysis allowed us to recover significant traces of the previous data.

We were able to confirm the existence of all 21 affected conversations, as well as all of their members.

A significant portion of the messages and encryption state could also be identified.

A partial restoration was therefore technically possible.

However, we decided not to reinsert this partial data into production.

The reason is straightforward: restoring only part of a conversation's state can result in an inconsistent state, especially when dealing with MLS cryptographic state.

We therefore chose a clean and consistent reconstruction rather than a partial restoration that could introduce additional problems.

The affected conversations were cleaned up, and the affected accounts must start new conversations.

What we changed

This incident resulted in three major changes to Bluvy's infrastructure.

1. The root cause has been fixed

The mechanism used to temporarily disable cascading deletes has been moved outside the transaction.

It is now executed before the transaction begins, in a context where SQLite actually allows the configuration change.

The fix was tested by reproducing the incident before and after the change.

We also discovered that the same mechanism existed in two different places:

the manual migration script;

the server's automatic startup process, which can also execute pending migrations.

Both paths have now been fixed so that they use the corrected mechanism.

2. A mandatory backup before every migration

This is probably the most important change.

Before this incident, Bluvy did not automatically create a complete backup before running a database migration.

That is no longer the case.

Before any migration is executed, the server now creates a complete backup of the SQLite database using SQLite's native backup mechanism.

This produces a consistent database copy even while the database is actively being used by the server.

Most importantly:

if the backup fails, the migration does not run.

The rule is now simple:

No valid backup → no migration.

A migration error should never again be able to turn a software bug into permanent data loss.

3. Automatic backups every two hours

Backups are no longer limited to migrations.

The server now automatically creates a complete database backup every two hours.

The 15 most recent backups are retained automatically.

Older backups are removed to prevent disk usage from growing indefinitely.

This means that a recent database snapshot is continuously available.

If a completely unrelated future incident were to corrupt or delete data, we would have a recent backup available for restoration.

The potential amount of data loss is therefore now significantly limited compared with the situation that existed before this incident.

What we learned from this incident

Losing real user data is obviously not an acceptable outcome.

But this incident also highlighted something important: fixing the immediate bug alone would not have been enough.

We could have simply modified the migration, tested it, and considered the problem resolved.

Instead, we chose to go further.

The specific technical issue has been fixed, but the system has also been changed so that a similar error cannot have the same impact again.

Today:

every migration must be preceded by a successful backup;

automatic backups are performed every two hours;

multiple recent backups are retained;

both migration execution paths have been fixed;

migrations are tested with representative data before deployment.

The goal is therefore not simply to say "this bug has been fixed."

The goal is to make sure that the next unexpected failure cannot cause the same level of damage.

And now?

Bluvy remains in beta.

As the application continues to evolve and more real-world scenarios are tested, additional bugs will likely be discovered.

That is precisely what a beta phase is for: identifying these problems early enough to strengthen the system before it reaches a much larger scale.

This incident has also reinforced an important principle for us:

A reliable system is not simply one that has no bugs.

It is a system that can limit the consequences of a bug when one inevitably occurs.

We identified the cause of this incident, reproduced it, fixed it, and added the protections that were missing.

Future database migrations will therefore no longer run without a successful backup being created first.

And if another problem ever causes unexpected database corruption or deletion, multiple recent backups will now be available to enable recovery.

We are sorry for the data that was lost during this incident. However, we believe it is important to document exactly what happened rather than hide the problem. This transparency is part of how we want to develop Bluvy.