batch request stream

Make batched and rate-limited requests for data flowing through a node.js Stream; useful when you need to make slow, asynchronous I/O calls (which are optimally batched) for items coming down a fast read stream, and don't want to throttle the responding service.

api

The package exports a single function which will create and return a Writable stream.

createBatchRequestStream(options)

• options is a configuration object which must contain the following mandatory keys and may contain the optional ones:
• request(batch, requestCompleted): The function to execute on each buffered batch of data. Must accept two arguments: batch, an array of the objects written to the stream, and requestCompleted, a function that must be called by whatever callback is sent to the asynchronous I/O function called inside request(); see the example for clarification. This is used to keep track of the number of live requests and, thus, for rate-limiting.
• batchSize (optional, default value: 100): The number of items to buffer before calling request(). If the stream has ended and a non-empty batch remains (because the total number of items written isn't evenly divisible by batchSize), it'll still be sent to request().
• maxLiveRequests (optional, default value: 100): The maximum number of concurrent requests to keep open.
• streamOptions (optional): Options to pass to new stream.Writable().

example

var createBatchRequestStream = require("batch-request-stream");
 
function request(batch, requestCompleted){
    asyncIORequest(batch, function callback(err){
        requestCompleted(); // MUST be called
    });
}
 
var batchRequestStream = createBatchRequestStream({
    request: request,
    batchSize: 4,
    maxLiveRequests: 3,
    streamOptions: {objectMode: true}
});
 
fastReadStream.pipe(batchRequestStream);