The WorkQueue

Here’s the third piece in our series about how to write our controller framework.

If there is one component that defines the correctness of a controller, it is not the watcher, the cache, or even the handler. It is the WorkQueue.

This article explains why controllers are queue-driven systems and why deduplication and backoff are the real foundations of reliability.

The WorkQueue Is the System Boundary

The WorkQueue represents the moment when observation becomes intent to act.

• everything before the queue is allowed to be: noisy, unreliable and redundant
• everything after the queue must be: deterministic, idempotent and retryable

Deduplication: One Key, One Intent

Deduplication allows semantic compression.

The queue does not encode history. It encodes what must be reconciled next.

This is why controllers:

• do not replay events
• do not store event logs
• do not care about intermediate transitions

WorkQueue in our framework

In our framework, the queue is split into two parts:

Backoff: Why Immediate Retries Are Harmful

Consider a failing external dependency.

Backoff: Per-Key, Not Global

A critical property.

This is why backoff is tracked per object, not per worker or per controller.

Retry Is a First-Class Control Flow

Shutdown Semantics Matter

A production-grade WorkQueue must guarantee Shutdown to:

• unblock all workers
• prevent new adds
• cancel delayed retries

Without this goroutines leak, and state becomes inconsistent.

This is often overlooked — and a common source of bugs.

Design Rule of Thumb

If you can delete your watcher and the system still converges, your controller is correct.

The WorkQueue makes that possible.

How (this) WorkQueue Enables Event Loss Tolerance

Because:

• List enqueues everything periodically
• Watch enqueues optimistically
• Queue deduplicates aggressively

The system converges even if:

• watch fails completely
• controller restart
• events duplicated or reordered

The queue turns best-effort observation into deterministic execution.

Closing Reflection

The queue is where your framework becomes a controller framework, not just a worker pool.

It encodes:

• safety
• liveness
• fairness
• failure isolation

Without it, reconciliation collapses into event handling. With it, correctness survives chaos.

Stay tuned for next article: "List, Watch, and Resync: Designing for Event Loss".