%dw 1.0
%type user = :object { class: "com.anypoint.df.pojo.User"}
%output application/xml
---
{
name : "Mariano",
age : 31
} as :user
Formats
DataWeave 1.1 is compatible with Mule Runtime Engine 3.8. MuleSoft recommends against deployments to Mule 3.8. Standard Support for this version ended on November 16, 2018, and Mule 3.8 will reach its End of Life on November 16, 2021, when Extended Support ends. |
DataWeave supports different types of data formats. Each format has an associated reader and a writer, and in some cases custom types are allowed. Each reader and writer defines configuration properties for customization.
Java
Canonical Model
The mapping between Java objects to DataWeave types is quite simple:
Java Type | DataWeave Type |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Metadata property class
Java developers use the 'class' metadata key as hint for what class needs to be created and sent as an input. If this is not explicitly defined, DataWeave tries to infer from the context or it assigns it the default values:
-
java.util.HashMap for objects
-
java.util.ArrayList for lists
The above code defines the type of the required input as an instance of 'com.anypoint.df.pojo.User'.
:enum
In order to put an enum value in a java.util.Map, the DataWeave java module defines a custom type called :enum. It allows you to specify that a given string should be handled as the name of a specified enum type. It should always be used with the class property with the java class name of the enum.
CSV
Canonical Model
CSV content is modeled in DataWeave as a list of objects, where every record is an object and every field in it is a property. For example:
Name,Last Name
Mariano, De achaval
Leandro, Shokida
[
{
Name: "Mariano",
"Last Name": " De achaval"
},
{
Name: "Leandro",
"Last Name": " Shokida"
}
]
Reader Properties
In CSV you can assign any special character as the indicator for separating fields, toggling quotes, or escaping quotes. Make sure you know what special characters are being used in your input, so that DataWeave can interpret it correctly.
When defining an input of type CSV, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
separator |
char |
, |
Character that separates one field from another |
quote |
char |
" |
Character that delimits the field values |
escape |
char |
\ |
Character used to escape occurrences of the separator or quote character within field values |
bodyStartLineNumber |
number |
0 |
The line number where the body starts. |
ignoreEmptyLine |
bool |
true |
defines if empty lines are ignored |
header |
bool |
true |
Indicates if the first line of the output shall contain field names |
headerLineNumber |
number |
0 |
the line number where the header is located |
When When |
These properties can be either set via the XML of your Mule project:
<dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
doc:name="Transform Message">
<dw:input-payload
<!-- Boolean that defines if the first line in the data contains headers -->
<dw:reader-property name="header" value="false" />
<!-- Character that separates fields, `','` by default -->
<dw:reader-property name="separator" value="," />
<!-- Character that defines quoted text, `" "` by default -->
<dw:reader-property name="quote" value=""" />
<!-- Character that escapes quotes, `\` by default -->
<dw:reader-property name="escape" value="\" />
</dw:input-payload>
<dw:set-payload>
<![CDATA[
%dw 1.0
%output application/java
---
// Your transformation script goes here
]]>
</dw:set-payload>
</dw:transform-message>
Or via the UI of the Transform Message component:
Writer Properties
When defining an output of type CSV, there are a few optional parameters you can add to the output directive to customize how the data is parsed:
Parameter | Type | Default | Description |
---|---|---|---|
separator |
char |
, |
Character that separates one field from another |
encoding |
string |
The character set to be used for the output |
|
quote |
char |
" |
Character that delimits the field values |
escape |
char |
\ |
Character used to escape occurrences of the separator or quote character within field values |
lineSeparator |
string |
system line ending default |
line separator to be used. Example: "\r\n" |
header |
bool |
true |
Indicates if the first line of the output shall contain field names |
quoteHeader |
bool |
false |
Indicates header values should be quoted |
quoteValues |
bool |
false |
Indicates if every value should be quoted whether or not it contains special characters within |
All of these parameters are optional. A CSV output directive might for example look like this:
%output text/csv separator=";", header=false, quoteValues=true
Excel
Canonical Model
An excel workbook is a sequence of sheets, in DataWeave this is mapped to an object where each sheet is a key. Only one table is allowed per excel sheet. A table is expressed as an array of rows. A row is an object where its keys are the columns and the values the cell content.
For example:
%output application/xlsx header=true
---
{
Sheet1: [
{
Id: 123,
Name: George
},
{
Id: 456,
Name: Lucas
}
]
}
Reader Properties
When defining an input of type excel, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
header |
bool |
true |
defines if the excel tables contain headers. When set to false, column names are used. (A, B, C, …) |
ignoreEmptyLine |
bool |
true |
defines if empty lines are ignored |
tableOffset |
string |
A1 |
The position of the first cell of the tables |
These properties can be either set via the XML of your Mule project:
<dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
doc:name="Transform Message">
<dw:input-payload
<!-- Boolean that defines if the first line in the data contains headers -->
<dw:reader-property name="header" value="true" />
<!-- Boolean that defines if empty lines are ignored -->
<dw:reader-property name="ignoreEmptyLine" value="false" />
<!-- Defines that defines what cell to start reading from. In this case Column A is ignored, and all rows above 9 -->
<dw:reader-property name="tableOffset" value="B9" />
</dw:input-payload>
<dw:set-payload>
<![CDATA[
%dw 1.0
%output application/java
---
// Your transformation script goes here
]]>
</dw:set-payload>
</dw:transform-message>
Or via the UI of the Transform Message component:
Writer Properties
When defining an output of type excel, there are a few optional parameters you can add to the output directive to customize how the data is parsed:
Parameter | Type | Default | Description |
---|---|---|---|
header |
bool |
true |
defines if the excel tables contain headers. When there are no headers, column names are used. (A, B, C, …) |
ignoreEmptyLine |
bool |
true |
defines if empty lines are ignored |
tableOffset |
string |
A1 |
The position of the first cell of the tables |
All of these parameters are optional. An excel output directive might for example look like this:
%output application/xlsx header=true
XML
Canonical Model
The XML data-structure is mapped to DataWeave objects that may contain other objects as values to their keys. Repeated keys are supported. For example:
<users>
<company>MuleSoft</company>
<user name="Leandro" lastName="Shokida"/>
<user name="Mariano" lastName="Achaval"/>
</users>
{
users: {
company: "MuleSoft",
user @(name: "Leandro",lastName: "Shokida"): "",
user @(name: "Mariano",lastName: "Achaval"): ""
}
}
Reader Properties
When defining an input of type XML, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
speed |
specifies the strategy to be used by the reader. Posible values = memory/speed |
|
string |
'empty' |
If a tag with empty or blank text should be read as null. |
|
boolean |
true |
Picks which reader modality to use. The indexed reader is faster but uses up a greater amount of memory, whilst the unindexed reader is slower but uses up less memory |
|
integer |
1 |
Limits the number of times that an entity can be referenced within the XML code. This is included to guard against denial of service attacks. |
|
boolean |
false |
Defines if references to entities that are defined in a file outside the XML are accepted as valid. It’s recommended to avoid these for security reasons as well. |
These properties can be either set via the XML of your Mule project:
<dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
doc:name="Transform Message">
<dw:input-payload>
<!-- specifies the strategy to be used by the reader -->
<dw:reader-property name="optimizeFor" value="speed" />
<!-- If a tag with empty or blank text should be read as null. -->
<dw:reader-property name="nullValueOn" value="empty" />
</dw:input-payload>
<dw:set-payload>
<![CDATA[
%dw 1.0
output application/xml
---
// Your transformation script goes here
]]>
</dw:set-payload>
</dw:transform-message>
Or via the UI of the Transform Message component:
Writer Properties
When defining an output of type XML, there are a few optional parameters you can add to the output directive to customize how the data is parsed:
Parameter | Type | Default | Description |
---|---|---|---|
|
boolean |
true |
Defines if the XML code will be indented for better readability, or if it will be compressed into a single line |
|
string |
|
Defines whether an empty XML child element appears as single self-closing tag or with an opening and closing tag. The value |
|
string |
UTF-8 |
The character set to be used for the output |
|
number |
153600 |
The size of the buffer writer |
|
string |
When the writer should use inline close tag. Possible values = |
|
|
string |
Possible values = |
|
|
boolean |
true |
Defines if the XML declaration will be included in the first line (available since Mule 3.8.4) |
%output application/xml indent=false, skipNullOn="attributes"
The inlineCloseOn
parameter defines if the output is structured like this (by default):
<someXml>
<parentElement>
<emptyElement1></emptyElement1>
<emptyElement2></emptyElement2>
<emptyElement3></emptyElement3>
</parentElement>
</someXml>
or like this (set with a value of "empty"):
<payload>
<someXml>
<parentElement>
<emptyElement1/>
<emptyElement2/>
<emptyElement3/>
</parentElement>
</someXml>
</payload>
Skip Null On
You can specify whether your transform generates an outbound message that contains fields with "null" values, or if these fields are ignored entirely. This can be set through an attribute in the output directive named skipNullOn, which can be set to three different values: elements, attributes, or everywhere.
When set to: * elements: A key:value pair with a null value is ignored. * attributes: An XML attribute with a null value is skipped. * everywhere: Apply this rule to both elements and attributes.
Defining a Metadata Type
In the Transform Message component, you can define a XML type through the following methods:
-
By Providing a sample file
-
By pointing to a schema file
Custom Types
:cdata
XML defines a custom type named :cdata, it extends from string and is used to identify a CDATA XML block.
It can be used to tell the writer to wrap the content inside CDATA or to check if the input string arrives inside a CDATA block. :cdata
inherits from the type :string
.
%dw 1.0
%output application/xml
---
{
users:
{
user : "Mariano" as :cdata,
age : 31 as :cdata
}
}
<?xml version="1.0" encoding="UTF-8"?>
<users>
<user><![CDATA[Mariano]]></user>
<age><![CDATA[31]]></age>
</users>
JSON
Canonical Model
JSON data-structures are mapped to DataWeave data-structures in a straight forward way as they share a lot of similarities.
Writer Properties
When defining an output of type JSON, there are a few optional parameters you can add to the output directive to customize how the data is parsed:
Parameter | Type | Default | Description |
---|---|---|---|
|
boolean |
true |
Defines if the JSON code will be indented for better readability, or if it will be compressed into a single line |
|
string |
UTF-8 |
The character set to be used for the output |
|
number |
153600 |
The size of the buffer writer |
|
string |
When the writer should use inline close tag. Possible values = empty/none |
|
|
string |
Possible values = |
|
|
boolean |
false |
JSON language doesn’t allow duplicate keys with one same parent, this usually raises an exception. If set to true, the output contains a single key that points to an array containing all the values assigned to it. |
%output application/json indent=false, skipNullOn="arrays"
Skip Null On
You can specify whether this generates an outbound message that contains fields with "null" values, or if these fields are ignored entirely. This can be set through an attribute in the output directive named skipNullOn, which can be set to three different values: elements, attributes, or everywhere.
When set to: * elements: A key:value pair with a null value is ignored. * attributes: An XML attribute with a null value is skipped. * everywhere: Apply this rule to both elements and attributes.
Flat File
Reader Properties
When defining an input of type Flat File, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Location in your local disk of the schema file used to parse your input. The Schema must have an .ESL extension. |
|
|
string |
The schema file might define multiple different structures, this field selects which to use. In case the schema only defines one, you also need to explicitly select that one through this field. |
|
|
string |
spaces |
(since 3.8.1) How missing values are represented in the input data:
|
|
string |
strict |
expected separation between lines/records:
|
These properties can be either set via the XML of your Mule project:
<dw:input-payload mimeType="text/plain" >
<dw:reader-property name="schemaPath" value="myschema.esl"/>
<dw:reader-property name="structureIdent" value="structure1"/>
</dw:input-payload>
Or via the UI of the Transform Message component
Schemas with type Binary or Packed don’t allow for line breaks to be detected, so setting recordParsing to "lenient"` will only allow long records to be handled, but not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).
|
Writer Properties
When defining an output of type flat file there are a few optional parameters you can add to the output directive to customize how the data is written:
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Path where the schema file to be used is located |
|
|
string |
In case the schema file defines multiple formats, indicates which of them to use |
|
|
string |
UTF-8 |
Output character encoding |
|
string |
spaces |
How to represent optional values missing from the supplied map:
|
|
string |
standard Java line termination for the system |
Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: |
|
boolean |
|
Trim string values longer than field length by truncating trailing characters |
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", structureIdent=“QBResponse"
---
payload
See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file. |
Fixed Width
Fixed width types are technically considered a type of Flat File format, but when selecting this option the Transform Message component offers you settings that are better tailored to the needs of this format.
Reader Properties
When defining an input of type Fixed Width, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Location in your local disk of the schema file used to parse your input. The Schema must have an .FFD extension. |
|
|
string |
spaces |
How missing values are represented in the input data:
|
|
string |
strict |
expected separation between lines/records:
|
These properties can be either set via the XML of your Mule project:
<dw:input-payload mimeType="text/plain" >
<dw:reader-property name="schemaPath" value="myschema.ffd"/>
<dw:reader-property name="structureIdent" value="structure1"/>
</dw:input-payload>
Or via the UI of the Transform Message component
Writer Properties
When defining an output of type fixed width there are a few optional parameters you can add to the output directive to customize how the data is written:
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Path where the schema file to be used is located |
|
|
string |
UTF-8 |
Output character encoding |
|
string |
spaces |
How to represent optional values missing from the supplied map:
|
|
string |
standard Java line termination for the system |
Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: |
|
boolean |
|
Trim string values longer than field length by truncating trailing characters |
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", encoding="UTF-8"
---
payload
See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file. |
Defining a Metadata Type
In the Transform Message component, you can define a Fixed Width type through the following methods:
-
By Providing a sample file
-
By pointing to a [flat file] schema file
-
Via a graphical editor that allows you to set up each field manually
Cobol Copybook
Copybook types are technically considered a type of Flat File format, but when selecting this option the Transform Message component offers you settings that are better tailored to the needs of this format.
Importing a Copybook Definition
To import a copybook definition:
-
Click Define Metadata on the input Payload in the Transform component, and select Set Metadata to open the Set Metadata Type dialog.
You need to create a metadata type to import a copybook definition.
-
Provide a name for your copybook metadata, such as
copybook
. -
Select the Copybook type.
-
Import your copybook definition file.
-
Click Select.
For example, assume that you have a copybook definition file (mailing-record.cpy
) that looks like this:
01 MAILING-RECORD. 05 COMPANY-NAME PIC X(30). 05 CONTACTS. 10 PRESIDENT. 15 LAST-NAME PIC X(15). 15 FIRST-NAME PIC X(8). 10 VP-MARKETING. 15 LAST-NAME PIC X(15). 15 FIRST-NAME PIC X(8). 10 ALTERNATE-CONTACT. 15 TITLE PIC X(10). 15 LAST-NAME PIC X(15). 15 FIRST-NAME PIC X(8). 05 ADDRESS PIC X(15). 05 CITY PIC X(15). 05 STATE PIC XX. 05 ZIP PIC 9(5).
Note: Copybook definitions must always begin with a 01
entry. A separate record type is generated for each 01
definition in your copybook (there must be at least one 01 definition for the copybook to be usable, so add one using an arbitrary name at the start of the copybook if none is present). If there are multiple 01
definitions in the copybook file, you can select which definition to use in the transform from the dropdown list.
When you import the schema, the Transform component converts the copybook file to a flat file schema that it stores in the src/main/resources
folder of your Mule project. In flat file format, the copybook definition above looks like this:
form: COPYBOOK id: 'MAILING-RECORD' values: - { name: 'COMPANY-NAME', type: String, length: 30 } - name: 'CONTACTS' values: - name: 'PRESIDENT' values: - { name: 'LAST-NAME', type: String, length: 15 } - { name: 'FIRST-NAME', type: String, length: 8 } - name: 'VP-MARKETING' values: - { name: 'LAST-NAME', type: String, length: 15 } - { name: 'FIRST-NAME', type: String, length: 8 } - name: 'ALTERNATE-CONTACT' values: - { name: 'TITLE', type: String, length: 10 } - { name: 'LAST-NAME', type: String, length: 15 } - { name: 'FIRST-NAME', type: String, length: 8 } - { name: 'ADDRESS', type: String, length: 15 } - { name: 'CITY', type: String, length: 15 } - { name: 'STATE', type: String, length: 2 } - { name: 'ZIP', type: Integer, length: 5, format: { justify: ZEROES, sign: UNSIGNED } }
After importing the schema, you can use the schemaPath
property to reference the flat file through the output
directive. For example: output application/flatfile schemaPath="src/main/resources/mailing-record.ffd"
Reader Properties
When defining an input of type Copybook, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Location in your local disk of the schema file used to parse your input |
|
|
string |
In case the schema file defines multiple different structures, this field selects which to use |
|
|
string |
nulls |
How missing values are represented in the input data: * none: treat all data as actual values * spaces: interpret a field consisting of only spaces as a missing value * zeroes: interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values * nulls: interpret a field consisting only of 0 bytes as a missing value |
|
string |
strict |
expected separation between lines/records:
|
These properties can be either set via the XML of your Mule project:
<dw:input-payload mimeType="text/plain" >
<dw:reader-property name="schemaPath" value="myschema.ffs"/>
<dw:reader-property name="segmentIdent" value="structure1"/>
</dw:input-payload>
Or via the UI of the Transform Message component
Schemas with type Binary or Packed don’t allow for line breaks to be detected, so setting recordParsing to "lenient"` will only allow long records to be handled, but not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).
|
Writer Properties
When defining an output of type copybook there are a few optional parameters you can add to the output directive to customize how the data is written:
Parameter | Type | Default | Description |
---|---|---|---|
|
string |
Path where the schema file to be used is located |
|
|
string |
In case the schema file defines multiple formats, indicates which of them to use |
|
|
string |
UTF-8 |
Output character encoding |
|
string |
nulls |
How to represent optional values missing from the supplied map: * spaces: fill the field with spaces * nulls: use 0 bytes |
|
string |
standard Java line termination for the system |
Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: |
|
boolean |
|
Trim string values longer than field length by truncating trailing characters |
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", structureIdent=“QBResponse"
---
payload
See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file. |
Defining a Metadata Type
In the Transform Message component, you can define a Cobol Copybook type through the following methods:
-
By pointing to a [flat file] schema file
See Flat File Schemas for more detailed instructions on how to write the required schema. |