Corrupted state and silent memory leaks in dynamic generator-driven Event Pipeline (Node.js/V8)

Hey everyone,

I’m hitting a wall with a deeply nested asynchronous pipeline architecture. We are using an abstraction layer built on top of async generators (async function*) to stream, batch, and transform high-throughput telemetry data before writing it to a time-series DB.

Under normal load, everything passes our integration tests. However, under high concurrency spikes, the entire pipeline goes sideways. The downstream consumers start receiving overlapping, stale chunks from completely different user sessions, and our Heap Snapshot shows an exponential accumulation of uncollected Closure and Promise contexts, eventually triggering an OOM (Out of Memory) crash.

The strangest part? No errors are thrown. The stream just silently corrupts the data state and leaks memory.

Here is a isolated reduction of the pipeline core that reproduces the issue under high load:

class TelemetryPipeline {
  constructor(transformer) {
    this.transformer = transformer;
    this.buffer = [];
    this.activeWorkers = 0;
  }

  async *processStream(inputStream) {
    for await (const chunk of inputStream) {
      this.buffer.push(chunk);
      
      if (this.buffer.length >= 10 && this.activeWorkers < 5) {
        this.activeWorkers++;
        // Spin off a processing batch asynchronously without waiting
        yield* this.flushBatch(); 
      }
    }
  }

  async *flushBatch() {
    const batchToProcess = this.buffer.splice(0, 10);
    
    try {
      // The transformer is an external async map function mapping schemas
      const transformed = await this.transformer(batchToProcess); 
      
      for (const item of transformed) {
        yield item;
      }
    } finally {
      this.activeWorkers--;
    }
  }
}

// Example Usage simulating concurrent high-velocity network streams:
const pipeline = new TelemetryPipeline(async (batch) => {
  await new Promise(resolve => setTimeout(resolve, Math.random() * 50)); // Simulates network I/O jitter
  return batch.map(b => ({ ...b, processedAt: Date.now() }));
});```

Hi @ThisSlopIsInsane !

The first thing that stands out in your code is that yield* this.flushBatch() doesn’t actually spin off a background task. It delegates iteration to the async generator and waits for it to finish, so the implementation isn’t achieving the concurrency its comments suggest.

There are a few other issues worth investigating:

Shared mutable state: buffer belongs to the pipeline instance, not an individual stream. If multiple user sessions share the same instance, their chunks can end up in the same batch.

Ineffective concurrency limit: Because flushBatch() is delegated to synchronously within the generator’s iteration flow, activeWorkers doesn’t provide a meaningful five-worker concurrency limit.

Unbounded buffering: When the condition prevents a flush, incoming chunks continue accumulating without backpressure or a maximum buffer size.

Unflushed remainder: Any chunks left in the buffer when the input stream ends are never processed.

I’d start by isolating the buffer and worker accounting per stream, then implement explicit bounded concurrency if parallel batch processing is actually required.

One problem: while these issues explain the potential data corruption and unbounded memory growth, the snippet alone doesn’t establish the cause of the reported Closure and Promise retention. I’d inspect the heap snapshot’s retaining paths and the transformer’s implementation before attributing the OOM to a specific leak.

I hope you get this fixed soon.
Regards, AC1DST4R
Self Employed Developer / Unofficial MDN Support

1 Like