A common solution

A common solution to idempotent database inserts looks as follows, using PostgreSQL as the database of choice:

CREATE TABLE idempotent_table (
	id SERIAL PRIMARY KEY,
	data VARCHAR NOT NULL,
	idempotency_key VARCHAR UNIQUE NOT NULL,
);

-- Insert a row
INSERT INTO idempotent_table (data, idempotency_key)
    VALUES ('foo', 'action1')
	ON CONFLICT (idempotency_key) DO NOTHING;

-- Retry of the operation above
INSERT INTO idempotent_table (data, idempotency_key)
    VALUES ('foo', 'action1')
	ON CONFLICT (idempotency_key) DO NOTHING;

We create a table and attach a unique non-nullable column idempotency_key, when writing the record to the table we check if the operation is colliding on the unique constraint of the idempotency_key and in that case do nothing.

With this implementation, we increased the resilience of the system by allowing it to retry a given action. But let's look a bit deeper into edge cases and the behavior of the proposed solution.

The flaw

Let’s imagine the following. At first, a record is inserted in the database with idempotency key action1:

INSERT INTO idempotent_table (data, idempotency_key)
    VALUES ('foo', 'action1')
    ON CONFLICT (idempotency_key) DO NOTHING;

And then shortly after a new record should be inserted, but due to an error the same idempotency_key value is used.

INSERT INTO idempotent_table (data, idempotency_key)
  VALUES ('bar', 'action1')
  ON CONFLICT (idempotency_key) DO NOTHING;

The outcome of the two statements would be: Only the first insert statement would be saved in the database and the second insert will execute successfully but will not write a second row.

The flaw is, that we assume that the presence of an idempotency key in the database guarantees the correctness of the corresponding data. This oversimplification can lead to systems that quietly accumulate stale, incomplete, or even incorrect records under the radar. And as everyone knows: assuming makes an ass out of you and me.

How to fix it? - The improved solution

The solution to the problem is rather simple, just don't assume that the record you want to insert and the record existing in the database are the same, but rather validate that they are. This can be achieved by building a utility function. The example will be using Python and SQLAlchemy, but this can be adapted to your tech stack.

async def idempotent_insert(con: AsyncConnection, insert_statement: Insert, idempotency_key_column: Column) -> Row:
	# Perform insert
    cursor = await con.execute(
	    insert_statement.on_conflict_do_nothing(index_elements=[idempotency_key_column])
    )

	# If a record has been inserted, it is a new record and stop here
	if cursor.rowcount:
        return

    values_to_insert: dict = {k: v.value for k, v in insert_statement._values.items()}

    # Retrieve the existing idempotency key record from the table
    idempotency_key_value = values_to_insert.get(idempotency_key_column.name)
    cursor = await con.execute(
        select(insert_statement.table).where(idempotency_key_column == idempotency_key_value)
    )

    existing_record: dict = cursor.fetchone()._mapping

    # Compare the record in the database with the record we wanted to insert,
	# if column values are differentiating, they will populate the variable not_matching_columns with a set of tuples (colum, value)
    if not_matching_columns := set(values_to_insert.items()) - set(
        existing_record.items()
      ):
      raise IdempotencyConflictError(
        f"To be inserted record does not match with existing record on the same idempotency "
        f"'{idempotency_key_value}'. Different columns: {', '.join(map(lambda x: x[0], not_matching_columns))}"
      )

With this implementation, a second usage of the same idempotency key leads to a full validation of the record to be inserted is equal to the record existing in the database table:

async with db_connection_pool.begin() as con:
	await idempotent_insert(
		con,
		Insert(idempotent_table).values(idempotency_key='action', data='foo'),
		idempotent_table.c.idempotency_key
	)

	# will raise IdempotencyConflictError pointing out that the entries differ in colum data
	await idempotent_insert(
		con,
		Insert(idempotent_table).values(idempotency_key='action', data='bar'),
		idempotent_table.c.idempotency_key
	)

Caveats

This solution is not a silver bullet and comes with a few caveats, that I need to point out.

Alternative solution: The issue can also be resolved by implementing a reconciliation, that reconciles elements in a table e.g. for a given process A a record for A in the right shape is existing in the database. However, I found that implementing an utility that checks the consistency right at runtime, is simpler and points out the problem faster than implementing a reconciliation.

The proposed solution also works best in insert-only tables. As soon as modifications and updates come into the picture, updated columns need to be excluded when comparing the inserting record to the existing record in the table.

While offloading this work to background jobs via message queues seems straightforward, the moment you introduce them alongside a relational database, you’re walking into a classic trap: the dual write problem.

In this article we explore the failure modes of naïve implementations, demonstrate the elegance of the Transactional Outbox pattern, and discuss how to scale it with modern tooling.

For the purpose of this article lets consider a basic e-commerce setup: where after the user places an order via an API call, the system then continues processing that by sending notifications and starting the order fulfilment.

Doing It Wrong: A Dual Write Dilemma

A common, but incorrect approach looks like the following:

@router.post('/orders')
async def create_order(order):
	order_created = await save_to_database(order)
	order_created_event = ...
	await publish_event(order_created_event)
	return order_created

The API endpoint accepts an incoming order, saves it in the database and publishes an event to Kafka or to a Redis queue and returns the response. A background worker picks up the event and continues with the email notification and the order fulfilment processing.

Seems innocent enough, but what happens if the database write succeeds and the event publishing fails? You are left with an inconsistent state where your order is accepted but not processed.

A common suggestion to fix situation is: “Just commit to the database after you published the event”. Congratulations, you now simply reversed the failure case. If the database write fails, you have emitted an event for an order not existing in your system. Your workers are processing ghost orders!

Using The Transactional Outbox Pattern

The core problem of the proposed solution is the dual write. Writing into two systems as an atomic transaction is not possible in a non durable execution environment like an API call.

The solution is simple - avoid writing into two system in the API call altogether.

The transactional outbox pattern is a simple but effective way to guarantee that messages are only published if the database transaction commits. We achieve this by not publishing the message directly but rather writing it to a event table as part of your transaction. From there a message relay system - a background worker - is picking up the entry and publishing it to your messaging platform.

The API endpoint implementation looks like this:

@router.post('/orders')
async def create_order(order):
	async with get_database_tranaction() as tx:
		order_created = await save_to_database(order)
		order_created_event = ...
		await publish_event_to_outbox(order_created_event)

	return order_created

which effectively translates to the following SQL statements:

BEGIN;

INSERT INTO orders (...) VALUES (...);
INSERT INTO event_outbox (event_type, payload) VALUES ('order_created', '{...}');

COMMIT;

Implementing the message relay

The obvious next question is how do you implement the message relay. There are multiple options to chose from, each with their own advantages, so choose wisely for your current situation.

Simple: cronjob polling

A simple scheduled job looks up the table for outstanding events in a fixed interval. It is advised to limit (e.g. 50) how many events at maximum each iteration can be processed.

After publishing the messages, you can mark the database entries as processed. If you fail to publish a message, a future iteration will pick up the same records and retry it for you. But be aware and monitor how many records are picked up from the database and how many are successfully published. If you are always picking up more events than you are successfully publishing, you build a backlog or worst case have a fault event that blocks all events after it from being published.

This options shines through its simplicity to implement, and it most suitable to get started. It lacks in terms of throughput and lag as its throughput is constant depending on the interval and the number of limit of records, and the lag introduced depending on the polling interval. Example: It polls every 30 seconds for up to 100 records, means your max throughput is 200 records per minute and your worst case lag is 30 seconds.

Since events might be retried it must also be pointed out that your message processing must be able to handle idempotency as messages would be delivered at least once.

Advanced: Leveraging Change Data Capture (CDC)

With CDC, we can hook into the database’s transaction log, e.g. PostgreSQLs Write Ahead Log (WAL) or MySQLs Binary Log, and stream changes in real time.

In PostgreSQL you can create a logical replication slot, by using the pg_create_logical_replication_slot command, leveraging the wal2json plugin.

SELECT pg_create_logical_replication_slot('outbox_replication', 'wal2json');

With the select commands

SELECT pg_logical_slot_peek_changes('outbox_replication', NULL, NULL, 'include-tables', 'event_outbox', 'actions', 'insert') // peek into changes
SELECT pg_logical_slot_get_changes('outbox_replication', NULL, NULL, 'include-tables', 'event_outbox', 'actions', 'insert') // consume changes

we can peek and check for new entries and then consume them with pg_logical_slot_get_changes.

With this we can replace our scheduled poller into a more dynamic solution that has an improved throughput as shown in this pseudo code:

async def run():
	while not_interrupted:
		has_events = await peek_for_outbox_events()

		if not has_events:
			asyncio.sleep(1)

		events = await get_outbox_events(max_events=10)

		for event in events:
			try:
				await publish(event)
			except:
				await dlq(event)

This way we optimised our scheduled job, for a better throughput an less lag. But when changes are consumed from the replication slot, they are no longer available in the next run, so any failures have to be put into a dead-letter-queue to be retried at a later stage.

This option comes with an increased operationally complexity and requires the understanding of replications slots. To reduce complexity managed CDC solutions like AWS DMS, Google DataStream, Artie or the open source variant Debezium can be used.

I personally only tried out AWS DMS and can recommend it, but it is advised to keep an eye on the DMS monitoring of how many records are processed, to ensure everything works as expected. An even better approach would be to monitor inserted records and CDC processed records as metrics as diverging values will directly make you aware of some issues in the system.

Conclusion

The Transaction outbox pattern elegantly sidesteps the dual write problem by ensuring atomicity between state changes and event emission. While it's not a silver bullet, it's a reliable and well-understood solution to one of the fundamental problems in distributed systems.

If you're building systems that need to be both responsive and reliable, this pattern should be a foundational part of your architecture.

Happy building and remember, build your system for failure!

Only afterward did I ask myself: did I just upload an image to a server? I went back to check what happens on the network tab in the developer console when converting an image. And to my surprise (and also not), the image was indeed uploaded.

Almost all results on the first pages of search results uploaded files to their servers for processing. This raised privacy concerns for me, especially when dealing with personal or sensitive images.

The WebAssembly Revelation

I wondered with the recent WebAssembly advancements if it must be possible to do almost all file conversions locally in the browser. I was almost certain that file manipulation tools have already been compiled or ported to WebAssembly.

A quick search later, I found that both libvips (powering Sharp, which powers Next.js image optimization) and ImageMagick already have WebAssembly versions. This was exciting news!

With this discovery, I came to the conclusion it would make for a great small weekend build project and got working. The result you can see at convertlocal.app for yourself.

Building a Local-First Conversion Tool

The core principle behind convertlocal.app is simple: all processing happens directly in your browser. When you upload an image, the file is read into memory in your browser, a Web Worker is started to perform the file conversion using the WebAssembly module and the result is saved back to your device.

Current Features

The current version of convertlocal.app supports common image conversion formats:

• Convert to WebP, PNG, JPEG, and other common formats
• Adjust quality settings for lossy formats
• Preserve metadata options
• Batch processing of multiple files

The tool is completely free to use and with no ads. It's a simple utility that does one thing well - convert your images locally.

Long-Term Plans

While the current version focuses on common image conversions, I have several plans for the future:

• Support for HEIC (High Efficiency Image Format) commonly used by iPhones
• Expand beyond images to support other file types like documents and audio
• Build a Raycast plugin that utilizes the same WebAssembly modules for file conversion directly from your desktop

Why This Matters

In an era where data privacy is increasingly important, tools that process data locally rather than sending it to unknown servers provide a valuable alternative. While there are legitimate use cases for server-side processing (like when you need more computing power than the client device can provide), many common tasks can now be handled locally thanks to technologies like WebAssembly.

This small weekend project demonstrates how modern web technologies can be leveraged to create useful tools that respect user privacy while still providing powerful functionality.

If you need to convert images, give convertlocal.app a try, and feel free to share feedback or feature requests!