Contact Us 1-800-596-4880
  1. Homepage
  2. Mule Runtime (3.8)
  3. Reference

  4. Streaming

Streaming

Mule runtime engine version 3.8 reached its End of Life on November 16, 2021. For more information, contact your Customer Success Manager to determine how to migrate to the latest Mule version.

Streaming supports efficient processing of large data objects such as files, documents, and records by streaming the data through Mule rather than reading the whole thing into memory. Streaming provides the following advantages:

  • Allows flows to consume very large messages in an efficient way

  • Message payloads are not read into memory

  • Simple routing rules based on message metadata are still possible

  • You can combine streaming and non-streaming endpoints

Data Streams Consumption

To understand how data streams are consumed, review the following points:

  • Data streams cannot be consumed more than once

    In the following example, the flow shows the HTTP Listener source that receives a POST method with a body payload to write to the files. The flow writes the first file correctly, while the second file is created with empty content because each component that consumes a stream expects to receive a new stream. After the first File Write operation consumes the stream, the second File Write operation receives an empty stream. Thereby, the second operation has no content to write to a file.

    mruntime streaming about 1
    Figure 1. Mule 3 Streaming: Writing a File

    In the following example, something similar happens when you try to log the payload after a DataWeave transformation. The HTTP Listener operation receives the payload stream, and then when the stream gets to the Transform Message component, it is available in memory, so the component consumes the stream. After the Transform Message component consumes the content, the second Logger receives an empty stream.

    mruntime streaming about 2
    Figure 2. Mule 3 Streaming: Logging a Payload

  • Data streams cannot be consumed at the same time

    In the following example, the flow uses a Scatter-Gather router to split a data stream and simultaneously log and write the payload to a file. The application get some parts of the stream in the file and the rest on the log because different processor chains can not process the data stream content simultaneously.

    mruntime streaming about 3
    Figure 3. Mule 3 Streaming: Consuming Data Streams

Streaming Transports Connectors and Modules

The following transports, connectors and modules support streaming:

Streaming Transformers and Filters

Many transformers and filters can read input streams, process the contents, and send them on. However, most of these do not process the stream in real time; instead, they read the stream, load it into memory, process it, and then send it on. Therefore, transformers and filters can become a bottleneck in your application if you regularly stream large files.

The following transformers and filters do support true streaming and process the data as streams without loading them into memory first: