Streaming Data Between Services Using REST API

In this article, I would like to focus on how to retrieve streaming data with axios npm package and transmit it to another service.

This technique would be very helpful for projects built on microservice architecture.

First of all, Why do we need to stream data instead of getting all the data at once and giving it to another service? The main reason is that when a lot of data takes up a large amount of RAM space, it can slow down or even break the app. This might be even more likely with Node.js due to its memory limits. Kafka or RabbitMQ could be a better choice for this type of data. If for some reason you need to use Rest API to transfer large amount of data between services without running into memory issues, using streams might be the best approach.

Let’s set up the environment. We need install two npm packages called axios and express. The API set up is very straight except for a few small configuration changes. First, we set up the header to allow a response that can be streamed. Then, we adjust the axios get request settings to make it work well with getting streamed data.

First, configure your reasponse header to send chunk of data.

res.setHeader('Transfer-Encoding', 'chunked')

Second, configure axios response type.

responseType: 'stream'

After making these little changes, you can use the axios ‘on’ method to start listening to the incoming stream.

response.data.on(‘data’, (chunk) => {
 res.write(chunk);
 });

You need to listen for the ‘data’ event to get chunks of data and then send these chunks to another service. For sending data as a stream, use the write() method. It’s important to note that using write() won’t finish the request, even if there’s no more data. To properly end the request, listen for the ‘end’ event. This event triggers as soon as all data is received. Only after this, use the end() method to complete the request. To handle errors, also listen for the ‘error’ event using the ‘on’ method.

Everything is good to go! You can now obtain your data as a stream and send it to the next service. I hope you find this helpful. Thank you!