Foreach Scope

Mule Runtime Engine versions 3.5, 3.6, and 3.7 reached End of Life on or before January 25, 2020. For more information, contact your Customer Success Manager to determine how you can migrate to the latest Mule version.

The Foreach scope splits a collection into elements and processes them iteratively through the processors embedded in the scope, then returns the original message to the flow.

Using Foreach versus Splitter and Aggregator Pairs

Typically, to create a flow that processes messages containing a collection of elements you must use one flow control to split the collection into individual elements, which the flow processes iteratively, then you must use another flow control to re-aggregate the elements into a new collection so they can be passed out of the flow. Several of Mule ESB’s flow controls support such splitting-and-re-aggregation, but they cannot match the convenience of the Foreach scope.

This use of splitter and aggregator pairs also presents several potential limitations:

Under some circumstances, splitting a message collection into pieces can cause certain vital bits of XML — metadata in the header or footer, for example — to be dropped from the re-aggregated XML.
When you split certain collection types — Java, for example — into many pieces for processing, the collection may be re-aggregated into a different collection type — MuleMessageCollection, for example. (As a result, you may need to add extra flow steps to transform the processed message collection back into its original collection type.)
When you split, process, or aggregate a message collection, you must choose among several splitter and aggregator types. Sometimes, it proves difficult to determine which splitter/aggregator combination best suits your message processing needs.

Foreach provides several advantages over splitter-aggregator pairs:

Foreach splits collections into elements, then processes them iteratively without losing any of the message payload.
After Foreach splits a message collection and processes the individual elements, it doesn’t re-aggregate those individual elements into a MuleMessageCollection; rather, it returns the original message. (This results in "Java in, Java out" rather than "Java in, MuleMessageCollection out.")
The Foreach scope is versatile; it can iteratively process elements from any type of collection, including maps, lists, arrays, and MuleMessageCollections.
The Foreach scope can split and process collections of elements that are not part of the message payload. For example, Foreach can process message property collections (metadata) from the message header.

If the elements are mutable objects (for example, a bean with different fields in it) and any of the message processors embedded in your Foreach scope modifies a value of the original object (for example, one of the fields), then the changes to the fields will persist once the foreach returns the original message to the flow.

In other words, Foreach does not make a deep copy of the data when processing a message nor when returning the original message to the flow.

Configuring Foreach

Studio Visual Editor

Add a Foreach scope in your flow at the point where you want to initiate the Foreach scope processing.

Insert one or more message processors inside the Foreach scope area to define how Mule should process each element within the message collection. The Foreach scope can contain any number of message processors as well as references to child flows.

The only type of message processor you cannot drag into a Foreach scope is an inbound connector.

If you drag a two-way connector into a Foreach scope, Mule automatically converts it to an outbound-only connector.

Field Default Value Description XML

Field	Default Value	Description	XML
Display Name	For Each	Customize to display a unique name for the scope in your application.	`doc:name="For Each"`
Collection		(Optional) Enter an expression that tells Foreach where to find the data it must split and process. For example, enter an expression that instructs Foreach to split and process a collection from the header section – rather than the payload. Unless this field specifies otherwise, Foreach assumes that the message payload is the collection.	`collection="#[payload.topic]"`
Counter Variable Name	counter	(Optional) Enter a name in this field to label the variable that Foreach uses to record the number of the elements it has processed. If your collection already uses the label counter for another variable, this field will be blank and you will need to enter a different label for the Counter Variable Name, such as `index`.	`counterVariableName="counter"`
Batch Size	1	(Optional) Enter an integer to indicate the number of elements in each batch that Foreach processes. Potentially, these batches promote quicker processing. If greater than one, each batch is treated as a separate Mule message. For example, if a collection has 200 elements and you set the batch size to 50, Foreach will iteratively process 4 batches of 50 elements, each as a separate Mule message.	`batchSize="50"`
Root Message Variable Name	rootMessage	(Optional) Enter a name in this field to label the variable that Foreach uses to reference the complete, unsplit message collection. If your collection already uses the label rootMessage for another variable, this field will be blank and you will need to enter a different label for the Root Message Variable Name.	`rootMessageVariableName="rootMessage"`

Display Name

For Each

Customize to display a unique name for the scope in your application.

doc:name="For Each"

Collection

(Optional) Enter an expression that tells Foreach where to find the data it must split and process. For example, enter an expression that instructs Foreach to split and process a collection from the header section – rather than the payload. Unless this field specifies otherwise, Foreach assumes that the message payload is the collection.

collection="#[payload.topic]"

Counter Variable Name

counter

(Optional) Enter a name in this field to label the variable that Foreach uses to record the number of the elements it has processed. If your collection already uses the label counter for another variable, this field will be blank and you will need to enter a different label for the Counter Variable Name, such as index.

counterVariableName="counter"

Batch Size

(Optional) Enter an integer to indicate the number of elements in each batch that Foreach processes. Potentially, these batches promote quicker processing. If greater than one, each batch is treated as a separate Mule message. For example, if a collection has 200 elements and you set the batch size to 50, Foreach will iteratively process 4 batches of 50 elements, each as a separate Mule message.

batchSize="50"

Root Message Variable Name

rootMessage

(Optional) Enter a name in this field to label the variable that Foreach uses to reference the complete, unsplit message collection. If your collection already uses the label rootMessage for another variable, this field will be blank and you will need to enter a different label for the Root Message Variable Name.

rootMessageVariableName="rootMessage"

XML Editor or Standalone

Add a foreach element to your flow at the point where you want to initiate a foreach processing block. Refer to the code sample below.

Element Description

foreach

Use to create a block of message processors that iteratively process elements within a collection.

Element	Description
foreach	Use to create a block of message processors that iteratively process elements within a collection.

Configure the scope according to the table below

Element Attribute Default Value Description

Element Attribute	Default Value	Description
doc:name	For Each	Customize to display a unique name for the async scope in your application. Note: Attribute not required in Mule Standalone configuration.
collection	Payload	(Optional) Enter an expression that tells Foreach where to find the data it must split and process. For example, enter an expression that instructs Foreach to split and process a collection from the header section – rather than the payload. Unless this attribute specifies otherwise, Foreach assumes that the message payload is the collection.
counterVariableName	counter	(Optional) Specify to label the variable that Foreach uses to record the number of the elements it has processed. If your collection already uses the label `counter` for another variable, you will need to select a unique name.
batchSize	1	(Optional) Specify an integer to indicate the number of elements in each batch that Foreach processes. Potentially, these batches promote quicker processing. For example, if a collection has 200 elements and you set the batch size to 50, Foreach will iteratively process 4 batches of 50 elements.
rootMessageVariableName	rootMessage	(Optional) Specify to label the variable that Foreach uses to reference the complete, unsplit message collection. If your collection already uses the label `rootMessage` for another variable, you will need to select a unique name.

doc:name

For Each

Customize to display a unique name for the async scope in your application.

Note: Attribute not required in Mule Standalone configuration.

collection

Payload

(Optional) Enter an expression that tells Foreach where to find the data it must split and process. For example, enter an expression that instructs Foreach to split and process a collection from the header section – rather than the payload. Unless this attribute specifies otherwise, Foreach assumes that the message payload is the collection.

counterVariableName

counter

(Optional) Specify to label the variable that Foreach uses to record the number of the elements it has processed. If your collection already uses the label counter for another variable, you will need to select a unique name.

batchSize

(Optional) Specify an integer to indicate the number of elements in each batch that Foreach processes. Potentially, these batches promote quicker processing. For example, if a collection has 200 elements and you set the batch size to 50, Foreach will iteratively process 4 batches of 50 elements.

rootMessageVariableName

rootMessage

(Optional) Specify to label the variable that Foreach uses to reference the complete, unsplit message collection. If your collection already uses the label rootMessage for another variable, you will need to select a unique name.

Add nested elements beneath your foreach element to define how Mule should process each element within the message collection. The Foreach scope can contain any number of message processors as well as references to child flows.

<foreach collection="#[payload.name]" doc:name="For Each" counterVariableName="counter" rootMessageVariableName="rootMessage" batchSize="5">
    <some-nested-element/>
    <some-other-nested-element/>
</foreach>

Foreach Error Handling

The exception strategy defined for your flow handles all the exceptions thrown within the Foreach scope. (If you have not explicitly defined an exception strategy for your flow, Mule implicitly applies the default exception strategy to handle exceptions.) If a message in a collection throws an exception, Foreach stops processing that collection and invokes the exception strategy.

For example, Foreach throws an IllegalArgumentException whenever two conditions hold true:

it receives a message payload that is not a collection
you have not identified a message collection outside the message payload (defined by entering an expression in the Collection field in the Studio Visual Editor or including the collection attribute in XML configuration.)

Considerations when Persisting Data

In case the message inside the foreach scope is persisted, not only the item in the collection will be serialized but also all the variables associated with the current message. The rootMessage variable, associated with the message, contains a reference of the complete, unsplit message collection that could potentially be holding thousands of items. Therefore, serialization/deserialization of the rootMessage variable could impact memory consumption considerably when this collection is large enough.

To avoid this issue you must first remove the rootMessage variable from the message before persisting it. For this you can use the <remove-variable> element like so:

<remove-variable variableName="rootMessage" doc:name="Variable"/>

In Studio, you can drag a Variable message processor inside your scope and set it to "Remove Variable".

Example

The following example illustrates a flow that uses Foreach to add information to each message in a collection.

The HTTP connector receives a request from a client, then queries a JDBC database, where a table indicates the model names and the model years of various cars. Foreach breaks the collection (the table) apart into a list of elements (rows), each of which contains information such as about individual elements (maps) model:'ford sierra', model_year=1982}}. Foreach sends each element through the message processors in its scope.

The flow adds a new entry to each element’s map; if the model year is less than 2001, Mule adds type='20th century car', then sends the element to the JMS connector; otherwise, Mule adds type='21st century car' and sends the element to the File connector. Foreach returns a collection at the end of the flow and sends it to the transformer.

This particular example replaces the main flow’s default exception strategy with a custom Catch Exception Strategy that leverages the Set Payload and HTTP Response Builder building blocks.

Complete Example Code

<?xml version="1.0" encoding="UTF-8"?>
<mule xmlns="http://www.mulesoft.org/schema/mule/core"
      xmlns:http="http://www.mulesoft.org/schema/mule/http"
      xmlns:file="http://www.mulesoft.org/schema/mule/file"
      xmlns:jdbc="http://www.mulesoft.org/schema/mule/jdbc"
      xmlns:jms="http://www.mulesoft.org/schema/mule/jms"
      xmlns:scripting="http://www.mulesoft.org/schema/mule/scripting"
      xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
      xmlns:core="http://www.mulesoft.org/schema/mule/core"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/file http://www.mulesoft.org/schema/mule/file/current/mule-file.xsd
http://www.mulesoft.org/schema/mule/jdbc http://www.mulesoft.org/schema/mule/jdbc/current/mule-jdbc.xsd
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/current/mule-scripting.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">

    <jdbc:derby-data-source name="Derby_Data_Source" url="jdbc:derby:${app.home}/muleEmbeddedDB;create=true" transactionIsolation="UNSPECIFIED" doc:name="Derby Data Source"/>
    <jdbc:connector name="JDBCConnector" dataSource-ref="Derby_Data_Source" validateConnections="true" queryTimeout="-1" pollingFrequency="0" doc:name="JDBCConnector">
        <jdbc:query key="allcars" value="SELECT * FROM cars"/>
    </jdbc:connector>
    <jms:activemq-connector name="JMSConnector" doc:name="Active MQ"/>
    <flow name="process" doc:name="process">
        <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="9091" path="process" doc:name="HTTP connector"/>
        <jdbc:outbound-endpoint exchange-pattern="request-response" queryKey="allcars" responseTimeout="10000" mimeType="text/plain" queryTimeout="-1" connector-ref="JDBCConnector" doc:name="Database (JDBC)"/>
        <foreach doc:name="Foreach">
            <choice doc:name="Choice">
                <when expression="payload['MODEL_YEAR'] &#38;lt; 2001">
                    <processor-chain>
                        <expression-component doc:name="Set payload type">payload['TYPE']='20th century car'</expression-component>
                        <jms:outbound-endpoint queue="in" doc:name="JMS"/>
                    </processor-chain>
                </when>
                <otherwise>
                    <processor-chain>
                        <expression-component doc:name="Set payload type">payload['TYPE']='21st century car'</expression-component>
                        <file:outbound-endpoint path="/tmp" responseTimeout="10000" doc:name="File"/>
                    </processor-chain>
                </otherwise>
            </choice>
        </foreach>
        <set-payload value="#[payload.size()] cars where processed: #[payload]" doc:name="Set response"/>
        <http:response-builder contentType="text/html" doc:name="HTTP Response Builder">
            <parse-template location="foreach_info.html" doc:name="Parse Template"/>
        </http:response-builder>
        <catch-exception-strategy doc:name="Catch Exception Strategy">
            <set-payload value="You need to populate the Database first" doc:name="DB is not populated"/>
            <http:response-builder status="500" contentType="text/html" doc:name="HTTP Response Builder">
                <parse-template location="foreach_error.html" doc:name="Parse Template"/>
            </http:response-builder>
        </catch-exception-strategy>
    </flow>
    <flow name="populate" doc:name="populate">
        <http:inbound-endpoint exchange-pattern="request-response" host="localhost" port="9091" path="populate" doc:name="HTTP connector"/>
        <scripting:component doc:name="Script to populate DB">
            <scripting:script engine="Groovy">
                <scripting:text><![CDATA[jdbcConnector = muleContext.getRegistry().lookupConnector("JDBCConnector");
qr = jdbcConnector.getQueryRunner();
conn = jdbcConnector.getConnection();
qr.update(conn, "CREATE TABLE cars (model varchar(256), model_year integer)");
qr.update(conn, "INSERT INTO cars values('Ford Sierra', 1982)");
qr.update(conn, "INSERT INTO cars values('Opel Astra', 2001)");]]></scripting:text>
            </scripting:script>
        </scripting:component>
        <set-payload value="Successfully populated the database" doc:name="Set Payload"/>
        <http:response-builder contentType="text/html" doc:name="HTTP Response Builder">
            <parse-template location="foreach_info.html" doc:name="Parse Template"/>
        </http:response-builder>
        <catch-exception-strategy doc:name="Catch Exception Strategy">
            <set-payload value="DB already populated" doc:name="Database Already populated"/>
            <http:response-builder status="500" contentType="text/html" doc:name="HTTP Response Builder">
                <parse-template location="foreach_error.html" doc:name="Parse Template"/>
            </http:response-builder>
        </catch-exception-strategy>
    </flow>
</mule>

Gartner names MuleSoft a Leader

Create connected experiences with AI

Future of connected AI agents

The world of AI is a world of APIs