I Knew Enough to Be Dangerous: A Deep Dive into .NET Concurrency

I've been writing async/await in C# for years. And for most of that time, I knew enough to make things work. Task.Run here, await there, a CancellationToken when I remembered. It was fine — until it wasn't.

The wake-up call was a high-throughput scenario: thousands of events per second, observers firing into consumers, and an event that just... disappeared. No exception. No log. The system looked perfectly healthy, but data was silently lost.

That moment shifted something in me. I realized I had been building on assumptions, not understanding. So I decided to stop patching symptoms and start understanding the machinery underneath.

Here's what I mean by 'knowing enough to be dangerous.' I could write an async Task method without thinking twice. But could I explain what the compiler actually generates? Could I tell you where an exception goes when you forget to await a Task? Could I explain why TaskCompletionSource without RunContinuationsAsynchronously is a deadlock waiting to happen?

The honest answer was no. And in a system processing thousands of events per second, that gap between syntax knowledge and mechanical understanding is exactly where production bugs hide.

I structured my deep dive around three core primitives that, when combined, form the backbone of any high-throughput .NET system.

The biggest revelation was understanding the async state machine. When you write async Task, the C# compiler rewrites your method into a struct that implements IAsyncStateMachine. Your local variables become fields. Each await is a potential yield point where the thread returns to the pool. Exceptions don't throw on the spot — they get captured into the Task object.

This has real consequences. An unawaited Task that throws? The exception is silently swallowed. Since .NET 4.5, it doesn't even crash the process. Your event just vanishes. This was exactly the bug that started my journey.

I also dove into ValueTask<T> and its strict rules (await it exactly once, never concurrently), ConfigureAwait(false) for library code, and why sync-over-async (.Result, .Wait()) on ThreadPool threads causes cascading starvation.

System.Threading.Channels is .NET's replacement for BlockingCollection<T>, and it's a massive upgrade. Fully async, high-performance, and with built-in backpressure. The key insight: always use bounded channels. An unbounded channel is just an OutOfMemoryException with extra optimism.

With BoundedChannelFullMode.Wait, the producer's WriteAsync suspends (yields the thread, no blocking!) until the consumer catches up. The SingleReader and SingleWriter flags enable lock-free fast paths — but they're trust-based. .NET doesn't enforce them. If you lie about single-reader access, you get silent data corruption, not exceptions.

// Bounded, single-reader, single-writer. When the consumer falls behind the
// producer suspends instead of blocking — backpressure that costs no thread.
var channel = Channel.CreateBounded<Reading>(new BoundedChannelOptions(capacity: 1_000)
{
    FullMode = BoundedChannelFullMode.Wait,
    SingleReader = true,
    SingleWriter = true,
});

// Producer
await channel.Writer.WriteAsync(reading, cancellationToken);

// Consumer
await foreach (var item in channel.Reader.ReadAllAsync(cancellationToken))
{
    Process(item);
}

I built multi-stage pipelines where each stage reads from one channel and writes to the next, with completion and error propagation flowing downstream through Complete(exception).

Here's a surprise: IObservable<T> and IObserver<T> are built into the BCL. No System.Reactive NuGet needed. The catch is that nobody reads the contract, and breaking it leads to undefined behavior with zero runtime enforcement.

The grammar is strict: zero or more OnNext calls, followed by at most one OnError or OnCompleted. After a terminal event, no more calls. Ever. And OnNext calls must be serialized — no concurrent calls from different threads.

Without the Rx library, you build operators yourself: Where, Select, Buffer, Throttle. It's more work, but it's perfect for air-gapped environments where every NuGet dependency is a deployment headache. And frankly, building them from scratch taught me more than any library ever could.

The real power emerges when you combine all three. The pattern that changed everything for me:

Observable (produces events at any rate) → Bounded Channel (absorbs spikes with backpressure) → Task Consumers (process reliably with exception isolation)

The bridge between Observable and Channel is the critical piece. Since OnNext is synchronous, you can't call await WriteAsync inside it. Instead, you use TryWrite — if the channel is full, you log the drop and move on. For high-frequency sources (thousands/sec), I added a BatchingBridge that accumulates items by count or time window before writing a batch to the channel. This dramatically reduces async overhead.

The most important lesson wasn't about any single API. It was a mindset shift. In a reliable system, every event has exactly one of three outcomes:

1. Successfully processed.
2. Sent to a dead letter queue for retry or investigation.
3. Explicitly dropped with a log entry explaining why.

There is no fourth option. No event silently disappears. To enforce this, every consumer loop gets an 'exception wall' — a try-catch around each unit of work that never lets the loop itself die. Individual items can fail; the consumer keeps consuming.

I also built an Envelope<T> pattern with TaskCompletionSource-based acknowledgments, so the producer knows for certain that its message was handled. And a lightweight circuit breaker for external dependencies, because retrying a dead service thousands of times per second is just a DDoS attack on yourself.

Since this deep dive, my concurrent code looks fundamentally different. Every Task is either awaited or routed through a SafeFireAndForget extension that catches and logs exceptions. Every channel is bounded with an explicit FullMode. Every observable subscription is tracked and disposed on shutdown. Graceful shutdown means: stop accepting → drain queues → complete in-flight work → exit.

// A Task nobody awaits is an exception nobody sees. Fire-and-forget calls only
// leave my code through here.
public static void SafeFireAndForget(
    this Task task,
    ILogger logger,
    [CallerMemberName] string caller = "")
{
    _ = task.ContinueWith(
        faulted => logger.LogError(faulted.Exception, "Unobserved exception from {Caller}", caller),
        CancellationToken.None,
        TaskContinuationOptions.OnlyOnFaulted,
        TaskScheduler.Default);
}

It's more code up front, but the systems are predictable. When something fails, I know about it. When I push an event into the pipeline, I know it will be handled. That confidence is worth every extra line.

If you're a .NET developer writing concurrent code and you've never looked at what happens inside your async methods, I'd encourage you to. The gap between 'it compiles and runs' and 'it's reliable under load' is exactly where the interesting engineering lives. Sometimes the longest way around really is the shortest way home.

Link to a interactive guide: https://dotnetconcurrencyguide.pages.dev/