Hey everyone,
I’m hitting a wall with a deeply nested asynchronous pipeline architecture. We are using an abstraction layer built on top of async generators (async function*) to stream, batch, and transform high-throughput telemetry data before writing it to a time-series DB.
Under normal load, everything passes our integration tests. However, under high concurrency spikes, the entire pipeline goes sideways. The downstream consumers start receiving overlapping, stale chunks from completely different user sessions, and our Heap Snapshot shows an exponential accumulation of uncollected Closure and Promise contexts, eventually triggering an OOM (Out of Memory) crash.
The strangest part? No errors are thrown. The stream just silently corrupts the data state and leaks memory.
Here is a isolated reduction of the pipeline core that reproduces the issue under high load:
class TelemetryPipeline {
constructor(transformer) {
this.transformer = transformer;
this.buffer = [];
this.activeWorkers = 0;
}
async *processStream(inputStream) {
for await (const chunk of inputStream) {
this.buffer.push(chunk);
if (this.buffer.length >= 10 && this.activeWorkers < 5) {
this.activeWorkers++;
// Spin off a processing batch asynchronously without waiting
yield* this.flushBatch();
}
}
}
async *flushBatch() {
const batchToProcess = this.buffer.splice(0, 10);
try {
// The transformer is an external async map function mapping schemas
const transformed = await this.transformer(batchToProcess);
for (const item of transformed) {
yield item;
}
} finally {
this.activeWorkers--;
}
}
}
// Example Usage simulating concurrent high-velocity network streams:
const pipeline = new TelemetryPipeline(async (batch) => {
await new Promise(resolve => setTimeout(resolve, Math.random() * 50)); // Simulates network I/O jitter
return batch.map(b => ({ ...b, processedAt: Date.now() }));
});```