Contact Us 1-800-596-4880
  1. Homepage
  2. Mule Runtime (4.1)
  3. Core Components

  4. Batch Processing

Batch Processing

Standard Support for Mule 4.1 ended on November 2, 2020, and this version of Mule reached its End of Life on November 2, 2022, when Extended Support ended.

Deployments of new applications to CloudHub that use this version of Mule are no longer allowed. Only in-place updates to applications are permitted.

MuleSoft recommends that you upgrade to the latest version of Mule 4 that is in Standard Support so that your applications run with the latest fixes and security enhancements.
Batch processing is exclusive to Mule Enterprise runtimes.

Mule allows you to process messages in batches.

Within an application, you can initiate a batch job scope, which is a block of code that splits messages into individual records, performs actions upon each record, then reports on the results and potentially pushes the processed output to other systems or queues.

For example, you can use batch processing when:

  • Synchronizing data sets between business applications, such as syncing contacts between NetSuite and Salesforce.

  • Extracting, transforming and loading (ETL) information into a target system, such as uploading data from a flat file (CSV) to Hadoop.

  • Handling large quantities of incoming data from an API into a legacy system.

If you are already familiar with batch processing in Mule 3.x, you can get an overview of the differences with batch in Mule 4.x in the Migrating the Batch Module article.

Overview

Within a Mule application, batch processing provides a construct for asynchronously processing larger-than-memory data sets that are split into individual records. Batch jobs allow for the description of a reliable process that automatically splits up source data and stores it into persistent queues, which makes it possible to process large data sets while providing reliability. In the event that the application is redeployed or Mule crashes, the job execution is able to resume at the point it stopped.

batch main3

The job is then expressed in terms of processing individual records, providing semantics for record level variables, aggregation, and error handling.

Basic Anatomy

The heart of Mule’s batch processing lies within the batch job. A batch job is a scope that splits large messages into records that Mule processes asynchronously. In the same way flows process messages, batch jobs process records.

batch processing concept d1bdd

The Batch XML structure was modified on Mule 4.0. The example below shows abbreviated details to highlight batch elements.

<flow name="flowOne">
	<batch:job jobName="batchJob">
		<batch:process-records>

			<batch:step name="batchStep1">
				<event processor/>
				<event processor/>
			</batch:step>

			<batch:step name="batchStep2">
				<event processor/>
				<event processor/>
			</batch:step>
		</batch:process-records>
	</batch:job>
</flow>

A batch job contains one or more batch steps that act upon records as they move through the batch job.

Each batch step in a batch job contains processors that act upon a record to transform, route, enrich, or otherwise process data contained within it. By leveraging the functionality of existing Mule processors, the batch step offers a lot of flexibility regarding how a batch job processes records. See Refining Batch Steps Processing for more information.

A batch job executes when the flow reaches the process-records section of the batch job. When triggered, Mule creates a new batch job instance.

When the batch job starts executing, Mule splits the incoming message into records, stores them in a persistent queue, and queries and schedules those records in blocks of records to process. By default, the runtime stores 100 records in each batch step. You can customize this size according to the performance you require. See Refining Batch Steps Processing for more information.

After all the records have passed through all batch steps, the runtime ends the batch job instance and reports the batch job result indicating which records succeeded and which failed during processing.

Error Handling

Batch jobs can handle any record-level failure that might occur in processing to prevent the failure of a complete batch job. Further, you can set or remove variables on individual records so that during batch processing, Mule can route or otherwise act upon records in a batch job according to a record variable. See Handling Errors During Batch Job for more information.

Batch Job vs. Batch Job Instance

Though defined in context the above, it’s worth elaborating upon the terms batch job and batch job instance as they relate to each other.

  • A batch job is the scope element in an application in which Mule processes a message payload as a batch of records. The term batch job is inclusive of all three phases of processing: Load and Dispatch, Process, and On Complete.

  • A batch job instance is an occurrence in a Mule application whenever a Mule flow executes a batch job. Mule creates the batch job instance in the Load and Dispatch phase. Every batch job instance is identified internally using a unique String known as batch job instance id.

This identifier is useful if you want, for example, to pass the local job instance ID to an external system for referencing and managing data, improve the job’s custom logging, or even send an email or SMS notifications for meaningful events around that specific batch job instance. See Batch Job Instance ID to learn more about this identifier and how to customize it.

Batch Job Processing Phases

Each batch job contains three different phases:

  1. Load and Dispatch.

  2. Process.

  3. On Complete.

Load and Dispatch

This first phase is implicit. During this phase, the runtime performs all the "behind the scenes" work to create a batch job instance. Essentially, this is the phase during which Mule turns a serialized message payload into a collection of records for processing as a batch. You don’t need to configure anything for this activity to occur, though it is useful to understand the tasks Mule completes during this phase.

  1. Mule splits the message using Dataweave. This first step creates a new batch job instance.
    Mule exposes the batch job instance ID through the batchJobInstanceId variable. This variable is available in every step and the on-complete phase.

  2. Mule creates a persistent queue and associates it with the new batch job instance.

  3. For each item generated by the splitter, Mule creates a record and stores it in the queue. This activity is "all or nothing" – Mule either successfully generates and queues a record for every item, or the whole message fails during this phase.

  4. Mule presents the batch job instance, with all its queued-up records, to the first batch step for processing.

After this phase completes, the flow will continue.

The next phase, "Process," is asynchronous, meaning that the flow will not wait for the batch job to finish processing all records.

Process

During the "Process" phase, the runtime begins processing the records in the batch asynchronously. Each record moves through the processors in the first batch step, then is sent back to the original queue while it waits to be processed by the second batch step and so on until every record has passed through every batch step. Only one queue exists, and records are picked out of it for each batch step, processed, and then sent back to it; each record keeps track of what stages it has been processed through while it sits on this queue. Note that a batch job instance does not wait for all its queued records to finish processing in one batch step before pushing any of them to the next batch step. Queues are persistent.

Mule persists a list of all records as they succeed or fail to process through each batch step. If a record should fail to be processed by an event processor in a batch step, the runtime continues processing the batch, skipping over the failed record in each subsequent batch step.

At the end of this phase, the batch job instance completes and, therefore, ceases to exist.

batch+diagram

Beyond simple processing of records, there are several things you can do with records within a batch step.

  • You can apply filters by adding acceptExpressions within each batch step to prevent the step from processing certain records. For example, you can set a filter to prevent a step from processing any records which failed to process in the preceding step.

  • You can use a batch aggregator processor to aggregate records in groups, sending them as bulk upserts to external sources or services. For example, rather than upserting each contact (that is, a record) in a batch to Google Contacts, you can configure a batch aggregator to accumulate, say, 100 records, then upsert all of them to Google Contacts in one chunk.

See Refining Batch Steps Processing for more information.

On Complete

During this phase, you can optionally configure the runtime to create a report or summary of the records it processed for the particular batch job instance. This phase exists to give system administrators and developers some insight into which records failed to address any issues that might exist with the input data.

<batch:job name="Batch3">
        <batch:input>
            <poll doc:name="Poll">
                <sfdc:authorize/>
            </poll>
            <set-variable/>
        </batch:input>
        <batch:process-records>
            <batch:step name="Step1">
                <batch:record-variable-transformer/>
                <data-mapper:transform/>
            </batch:step>
            <batch:step name="Step2">
                <logger/>
                <http:request/>
            </batch:step>
        </batch:process-records>
        <batch:on-complete>
            <logger/>
        </batch:on-complete>
</batch:job>

After Mule executes the entire batch job, the output becomes a batch job result object (BatchJobResult). Because Mule processes a batch job as an asynchronous, one-way flow, the results of batch processing do not feed back into the flow which may have triggered it, nor do the results return as a response to a caller. Any event source that feeds data into a batch job must be one-way, not request-response.

You have two options for working with the output:

  • Create a report in the On Complete phase, using DataWeave using information such as the number of failed records and successfully processed records, and in which step any errors might have occurred.

  • Reference the batch job result object elsewhere in the Mule application to capture and use batch metadata, such as the number of records which failed to process in a particular batch job instance.

If you leave the On Complete phase empty and do not reference the batch job result object elsewhere in your application, the batch job simply completes, whether failed or successful.

As a good practice, you should configure some mechanism for reporting on failed or successful records to facilitate further action where required.

See Also