salesforce Dreamforce On-Demand
Contact Us 1-800-596-4880
  1. Homepage
  2. DataWeave (1.1)

  3. Formats

Formats

DataWeave 1.1 is compatible with Mule Runtime Engine 3.8. MuleSoft recommends against deployments to Mule 3.8. Standard Support for this version ended on November 16, 2018, and Mule 3.8 will reach its End of Life on November 16, 2021, when Extended Support ends.

DataWeave supports different types of data formats. Each format has an associated reader and a writer, and in some cases custom types are allowed. Each reader and writer defines configuration properties for customization.

Java

Canonical Model

The mapping between Java objects to DataWeave types is quite simple:

Java Type DataWeave Type

Collections/Array/Iterator/Iterable

:array

String/CharSequence/Char/Enum/Class

:string

int/Short/Long/BigInteger/Flat/Double/BigDecimal

:number

Calendar/XmlGregorainCalendar

:datetime

TimeZone

:timezone

sql.Date/util.Date

:date

Bean/Map

:object

InputStream/Array[Byte]

:binary

java.lang.Boolean

:boolean

Custom Types

Metadata property class

Java developers use the 'class' metadata key as hint for what class needs to be created and sent as an input. If this is not explicitly defined, DataWeave tries to infer from the context or it assigns it the default values:

  • java.util.HashMap for objects

  • java.util.ArrayList for lists

Transform
%dw 1.0
%type user = :object { class: "com.anypoint.df.pojo.User"}
%output application/xml
---
{
  name : "Mariano",
  age : 31
} as :user

The above code defines the type of the required input as an instance of 'com.anypoint.df.pojo.User'.

:enum

In order to put an enum value in a java.util.Map, the DataWeave java module defines a custom type called :enum. It allows you to specify that a given string should be handled as the name of a specified enum type. It should always be used with the class property with the java class name of the enum.

Defining a Metadata Type

In the Transform Message component, you can define a Java type through the following methods:

  • By Providing a sample object

See To Define Input and Output Structure of a Transformation.

CSV

Canonical Model

CSV content is modeled in DataWeave as a list of objects, where every record is an object and every field in it is a property. For example:

Input
Name,Last Name
Mariano, De achaval
Leandro, Shokida
DataWeave representation
[
  {
    Name: "Mariano",
    "Last Name": " De achaval"
  },
  {
    Name: "Leandro",
    "Last Name": " Shokida"
  }
]

Reader Properties

In CSV you can assign any special character as the indicator for separating fields, toggling quotes, or escaping quotes. Make sure you know what special characters are being used in your input, so that DataWeave can interpret it correctly.

When defining an input of type CSV, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

separator

char

,

Character that separates one field from another

quote

char

"

Character that delimits the field values

escape

char

\

Character used to escape occurrences of the separator or quote character within field values

bodyStartLineNumber

number

0

The line number where the body starts.

ignoreEmptyLine

bool

true

defines if empty lines are ignored

header

bool

true

Indicates if the first line of the output shall contain field names

headerLineNumber

number

0

the line number where the header is located

When header=true you can then access the fields within the input anywhere by name. Ex: payload.userName.

When header=false you must access the fields by index, referencing first the entry and then the field, Ex: payload[107][2]

These properties can be either set via the XML of your Mule project:

	 <dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
	 			doc:name="Transform Message">
 			<dw:input-payload
 			    <!-- Boolean that defines if the first line in the data contains headers -->
 				<dw:reader-property name="header" value="false" />
 				<!-- Character that separates fields, `','` by default -->
 				<dw:reader-property name="separator" value="," />
 				<!-- Character that defines quoted text, `" "` by default -->
 				<dw:reader-property name="quote" value="&quot;" />
 				<!-- Character that escapes quotes, `\` by default -->
 				<dw:reader-property name="escape" value="\" />
 			</dw:input-payload>
 			<dw:set-payload>
                <![CDATA[
                    %dw 1.0
                    %output application/java
                    ---
                    // Your transformation script goes here
                ]]>
            </dw:set-payload>
     </dw:transform-message>

Or via the UI of the Transform Message component:

dataweave formats 580be

See To Define Input and Output Structure of a Transformation.

Writer Properties

When defining an output of type CSV, there are a few optional parameters you can add to the output directive to customize how the data is parsed:

Parameter Type Default Description

separator

char

,

Character that separates one field from another

encoding

string

The character set to be used for the output

quote

char

"

Character that delimits the field values

escape

char

\

Character used to escape occurrences of the separator or quote character within field values

lineSeparator

string

system line ending default

line separator to be used. Example: "\r\n"

header

bool

true

Indicates if the first line of the output shall contain field names

quoteHeader

bool

false

Indicates header values should be quoted

quoteValues

bool

false

Indicates if every value should be quoted whether or not it contains special characters within

All of these parameters are optional. A CSV output directive might for example look like this:

%output text/csv separator=";", header=false, quoteValues=true

Defining a Metadata Type

In the Transform Message component, you can define a CSV type through the following methods:

  • By Providing a sample file

  • Via a graphical editor that allows you to set up each field manually

    dataweave formats 4a556
See To Define Input and Output Structure of a Transformation.

Excel

Canonical Model

An excel workbook is a sequence of sheets, in DataWeave this is mapped to an object where each sheet is a key. Only one table is allowed per excel sheet. A table is expressed as an array of rows. A row is an object where its keys are the columns and the values the cell content.

For example:

dataweave formats exceltable
Figure 1. Input
DataWeave representation
%output application/xlsx header=true
---
{
  Sheet1: [
    {
      Id: 123,
      Name: George
    },
    {
      Id: 456,
      Name: Lucas
    }
  ]
}

Reader Properties

When defining an input of type excel, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

header

bool

true

defines if the excel tables contain headers. When set to false, column names are used. (A, B, C, …​)

ignoreEmptyLine

bool

true

defines if empty lines are ignored

tableOffset

string

A1

The position of the first cell of the tables

These properties can be either set via the XML of your Mule project:

	 <dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
	 			doc:name="Transform Message">
 			<dw:input-payload
 			    <!-- Boolean that defines if the first line in the data contains headers -->
 				<dw:reader-property name="header" value="true" />
 				<!-- Boolean that defines if empty lines are ignored -->
 				<dw:reader-property name="ignoreEmptyLine" value="false" />
 				<!-- Defines that defines what cell to start reading from. In this case Column A is ignored, and all rows above 9 -->
 				<dw:reader-property name="tableOffset" value="B9" />
 			</dw:input-payload>
 			<dw:set-payload>
                <![CDATA[
                    %dw 1.0
                    %output application/java
                    ---
                    // Your transformation script goes here
                ]]>
            </dw:set-payload>
     </dw:transform-message>

Or via the UI of the Transform Message component:

dataweave formats excell reader

See To Define Input and Output Structure of a Transformation.

Writer Properties

When defining an output of type excel, there are a few optional parameters you can add to the output directive to customize how the data is parsed:

Parameter Type Default Description

header

bool

true

defines if the excel tables contain headers. When there are no headers, column names are used. (A, B, C, …​)

ignoreEmptyLine

bool

true

defines if empty lines are ignored

tableOffset

string

A1

The position of the first cell of the tables

All of these parameters are optional. An excel output directive might for example look like this:

%output application/xlsx header=true

Defining a Metadata Type

In the Transform Message component, you can define a excel type through the following methods:

  • Via a graphical editor that allows you to set up each field manually

    dataweave formats excel metadata
See To Define Input and Output Structure of a Transformation.

XML

Canonical Model

The XML data-structure is mapped to DataWeave objects that may contain other objects as values to their keys. Repeated keys are supported. For example:

Input
<users>
  <company>MuleSoft</company>
  <user name="Leandro" lastName="Shokida"/>
  <user name="Mariano" lastName="Achaval"/>
</users>
DataWeave representation
{
  users: {
    company: "MuleSoft",
    user @(name: "Leandro",lastName: "Shokida"): "",
    user @(name: "Mariano",lastName: "Achaval"): ""
  }
}

Reader Properties

When defining an input of type XML, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

optimizeFor

string

speed

specifies the strategy to be used by the reader. Posible values = memory/speed

nullValueOn

string

'empty'

If a tag with empty or blank text should be read as null.

indexedReader

boolean

true

Picks which reader modality to use. The indexed reader is faster but uses up a greater amount of memory, whilst the unindexed reader is slower but uses up less memory

maxEntityCount

integer

1

Limits the number of times that an entity can be referenced within the XML code. This is included to guard against denial of service attacks.

externalEntities

boolean

false

Defines if references to entities that are defined in a file outside the XML are accepted as valid. It’s recommended to avoid these for security reasons as well.

These properties can be either set via the XML of your Mule project:

<dw:transform-message metadata:id="33a08359-5085-47d3-aa5f-c7dd98bb9c61"
  doc:name="Transform Message">
    <dw:input-payload>
      <!-- specifies the strategy to be used by the reader -->
      <dw:reader-property name="optimizeFor" value="speed" />
      <!-- If a tag with empty or blank text should be read as null. -->
      <dw:reader-property name="nullValueOn" value="empty" />
    </dw:input-payload>
    <dw:set-payload>
      <![CDATA[
          %dw 1.0
          output application/xml
          ---
          // Your transformation script goes here
      ]]>
    </dw:set-payload>
</dw:transform-message>

Or via the UI of the Transform Message component:

dataweave formats 6e5e4

See To Define Input and Output Structure of a Transformation.

Writer Properties

When defining an output of type XML, there are a few optional parameters you can add to the output directive to customize how the data is parsed:

Parameter Type Default Description

indent

boolean

true

Defines if the XML code will be indented for better readability, or if it will be compressed into a single line

inlineCloseOn

string

never

Defines whether an empty XML child element appears as single self-closing tag or with an opening and closing tag. The value empty sets it to output self-closing tags.

encoding

string

UTF-8

The character set to be used for the output

bufferSize

number

153600

The size of the buffer writer

inlineCloseOn

string

When the writer should use inline close tag. Possible values = empty/none

skipNullOn

string

Possible values = elements/attributes/everywhere. See Skip Null On

writeDeclaration

boolean

true

Defines if the XML declaration will be included in the first line (available since Mule 3.8.4)

 
%output application/xml indent=false, skipNullOn="attributes"

The inlineCloseOn parameter defines if the output is structured like this (by default):

<someXml>
  <parentElement>
    <emptyElement1></emptyElement1>
    <emptyElement2></emptyElement2>
    <emptyElement3></emptyElement3>
  </parentElement>
</someXml>

or like this (set with a value of "empty"):

<payload>
  <someXml>
    <parentElement>
      <emptyElement1/>
      <emptyElement2/>
      <emptyElement3/>
    </parentElement>
  </someXml>
</payload>

Skip Null On

You can specify whether your transform generates an outbound message that contains fields with "null" values, or if these fields are ignored entirely. This can be set through an attribute in the output directive named skipNullOn, which can be set to three different values: elements, attributes, or everywhere.

When set to: * elements: A key:value pair with a null value is ignored. * attributes: An XML attribute with a null value is skipped. * everywhere: Apply this rule to both elements and attributes.

Defining a Metadata Type

In the Transform Message component, you can define a XML type through the following methods:

  • By Providing a sample file

  • By pointing to a schema file

See To Define Input and Output Structure of a Transformation.

Custom Types

:cdata

XML defines a custom type named :cdata, it extends from string and is used to identify a CDATA XML block. It can be used to tell the writer to wrap the content inside CDATA or to check if the input string arrives inside a CDATA block. :cdata inherits from the type :string.

Transform
%dw 1.0
%output application/xml
---
{
  users:
  {
    user : "Mariano" as :cdata,
    age : 31 as :cdata
  }
}
Output
<?xml version="1.0" encoding="UTF-8"?>
<users>
  <user><![CDATA[Mariano]]></user>
  <age><![CDATA[31]]></age>
</users>

JSON

Canonical Model

JSON data-structures are mapped to DataWeave data-structures in a straight forward way as they share a lot of similarities.

Writer Properties

When defining an output of type JSON, there are a few optional parameters you can add to the output directive to customize how the data is parsed:

Parameter Type Default Description

indent

boolean

true

Defines if the JSON code will be indented for better readability, or if it will be compressed into a single line

encoding

string

UTF-8

The character set to be used for the output

bufferSize

number

153600

The size of the buffer writer

inlineCloseOn

string

When the writer should use inline close tag. Possible values = empty/none

skipNullOn

string

Possible values = elements/attributes/everywhere. See Skip Null On

duplicateKeyAsArray

boolean

false

JSON language doesn’t allow duplicate keys with one same parent, this usually raises an exception. If set to true, the output contains a single key that points to an array containing all the values assigned to it.

 
%output application/json indent=false, skipNullOn="arrays"

Skip Null On

You can specify whether this generates an outbound message that contains fields with "null" values, or if these fields are ignored entirely. This can be set through an attribute in the output directive named skipNullOn, which can be set to three different values: elements, attributes, or everywhere.

When set to: * elements: A key:value pair with a null value is ignored. * attributes: An XML attribute with a null value is skipped. * everywhere: Apply this rule to both elements and attributes.

Defining a Metadata Type

In the Transform Message component, you can define a JSON type through the following methods:

  • By Providing a sample file

  • By pointing to a schema file

See To Define Input and Output Structure of a Transformation.

Flat File

Reader Properties

When defining an input of type Flat File, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

schemaPath

string

Location in your local disk of the schema file used to parse your input. The Schema must have an .ESL extension.

structureIdent

string

The schema file might define multiple different structures, this field selects which to use. In case the schema only defines one, you also need to explicitly select that one through this field.

missingValues

string

spaces

(since 3.8.1) How missing values are represented in the input data:

  • none: treat all data as actual values

  • spaces: interpret a field consisting of only spaces as a missing value

  • zeroes: interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values

  • nulls: interpret a field consisting only of 0 bytes as a missing value

recordParsing

string

strict

expected separation between lines/records:

  • strict: line break expected at exact end of each record

  • lenient: line break used but records may be shorter or longer than schema specifies

  • noTerminator: means records follow one another with no separation

These properties can be either set via the XML of your Mule project:

<dw:input-payload mimeType="text/plain" >
    <dw:reader-property name="schemaPath" value="myschema.esl"/>
    <dw:reader-property name="structureIdent" value="structure1"/>
</dw:input-payload>

Or via the UI of the Transform Message component

See To Define Input and Output Structure of a Transformation.

Schemas with type Binary or Packed don’t allow for line breaks to be detected, so setting recordParsing to "lenient"` will only allow long records to be handled, but not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

Writer Properties

When defining an output of type flat file there are a few optional parameters you can add to the output directive to customize how the data is written:

Parameter Type Default Description

schemaPath

string

Path where the schema file to be used is located

structureIdent

string

In case the schema file defines multiple formats, indicates which of them to use

encoding

string

UTF-8

Output character encoding

missingValues

string

spaces

How to represent optional values missing from the supplied map:

  • spaces: fill the field with spaces

  • nulls: use 0 bytes

recordTerminator

string

standard Java line termination for the system

Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: lf, cr, crlf, none. Values translate directly to character codes (none leaves no termination on each record).

trimValues

boolean

false

Trim string values longer than field length by truncating trailing characters

 
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", structureIdent=“QBResponse"
---
payload

See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file.

Defining a Metadata Type

In the Transform Message component, you can define a Flat File type through the following methods:

  • By pointing to a schema file

See To Define Input and Output Structure of a Transformation.

Fixed Width

Fixed width types are technically considered a type of Flat File format, but when selecting this option the Transform Message component offers you settings that are better tailored to the needs of this format.

Reader Properties

When defining an input of type Fixed Width, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

schemaPath

string

Location in your local disk of the schema file used to parse your input. The Schema must have an .FFD extension.

missingValues

string

spaces

How missing values are represented in the input data:

  • none: treat all data as actual values

  • spaces: interpret a field consisting of only spaces as a missing value

  • zeroes: interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values

  • nulls: interpret a field consisting only of 0 bytes as a missing value

recordParsing

string

strict

expected separation between lines/records:

  • strict: line break expected at exact end of each record

  • lenient: line break used but records may be shorter or longer than schema specifies

  • noTerminator: means records follow one another with no separation

These properties can be either set via the XML of your Mule project:

<dw:input-payload mimeType="text/plain" >
    <dw:reader-property name="schemaPath" value="myschema.ffd"/>
    <dw:reader-property name="structureIdent" value="structure1"/>
</dw:input-payload>

Or via the UI of the Transform Message component

See To Define Input and Output Structure of a Transformation.

Writer Properties

When defining an output of type fixed width there are a few optional parameters you can add to the output directive to customize how the data is written:

Parameter Type Default Description

schemaPath

string

Path where the schema file to be used is located

encoding

string

UTF-8

Output character encoding

missingValues

string

spaces

How to represent optional values missing from the supplied map:

  • spaces: fill the field with spaces

  • nulls: use 0 bytes

recordTerminator

string

standard Java line termination for the system

Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: lf, cr, crlf, none. Values translate directly to character codes (none leaves no termination on each record).

trimValues

boolean

false

Trim string values longer than field length by truncating trailing characters

 
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", encoding="UTF-8"
---
payload

See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file.

Defining a Metadata Type

In the Transform Message component, you can define a Fixed Width type through the following methods:

  • By Providing a sample file

  • By pointing to a [flat file] schema file

  • Via a graphical editor that allows you to set up each field manually

    dataweave formats 27b3c
See To Define Input and Output Structure of a Transformation.

Cobol Copybook

Copybook types are technically considered a type of Flat File format, but when selecting this option the Transform Message component offers you settings that are better tailored to the needs of this format.

Importing a Copybook Definition

To import a copybook definition:

  1. Click Define Metadata on the input Payload in the Transform component, and select Set Metadata to open the Set Metadata Type dialog.

    You need to create a metadata type to import a copybook definition.

  2. Provide a name for your copybook metadata, such as copybook.

  3. Select the Copybook type.

  4. Import your copybook definition file.

  5. Click Select.

For example, assume that you have a copybook definition file (mailing-record.cpy) that looks like this:

Example: Copybook Definition
01  MAILING-RECORD.
    05  COMPANY-NAME            PIC X(30).
    05  CONTACTS.
        10  PRESIDENT.
            15  LAST-NAME       PIC X(15).
            15  FIRST-NAME      PIC X(8).
        10  VP-MARKETING.
            15  LAST-NAME       PIC X(15).
            15  FIRST-NAME      PIC X(8).
        10  ALTERNATE-CONTACT.
            15  TITLE           PIC X(10).
            15  LAST-NAME       PIC X(15).
            15  FIRST-NAME      PIC X(8).
    05  ADDRESS                 PIC X(15).
    05  CITY                    PIC X(15).
    05  STATE                   PIC XX.
    05  ZIP                     PIC 9(5).

Note: Copybook definitions must always begin with a 01 entry. A separate record type is generated for each 01 definition in your copybook (there must be at least one 01 definition for the copybook to be usable, so add one using an arbitrary name at the start of the copybook if none is present). If there are multiple 01 definitions in the copybook file, you can select which definition to use in the transform from the dropdown list.

When you import the schema, the Transform component converts the copybook file to a flat file schema that it stores in the src/main/resources folder of your Mule project. In flat file format, the copybook definition above looks like this:

Example: Flat File Schema
form: COPYBOOK
id: 'MAILING-RECORD'
values:
- { name: 'COMPANY-NAME', type: String, length: 30 }
- name: 'CONTACTS'
  values:
  - name: 'PRESIDENT'
    values:
    - { name: 'LAST-NAME', type: String, length: 15 }
    - { name: 'FIRST-NAME', type: String, length: 8 }
  - name: 'VP-MARKETING'
    values:
    - { name: 'LAST-NAME', type: String, length: 15 }
    - { name: 'FIRST-NAME', type: String, length: 8 }
  - name: 'ALTERNATE-CONTACT'
    values:
    - { name: 'TITLE', type: String, length: 10 }
    - { name: 'LAST-NAME', type: String, length: 15 }
    - { name: 'FIRST-NAME', type: String, length: 8 }
- { name: 'ADDRESS', type: String, length: 15 }
- { name: 'CITY', type: String, length: 15 }
- { name: 'STATE', type: String, length: 2 }
- { name: 'ZIP', type: Integer, length: 5, format: { justify: ZEROES, sign: UNSIGNED } }

After importing the schema, you can use the schemaPath property to reference the flat file through the output directive. For example: output application/flatfile schemaPath="src/main/resources/mailing-record.ffd"

Reader Properties

When defining an input of type Copybook, there are a few optional parameters you can add in the XML definition of your Mule project to customize how the data is parsed.

Parameter Type Default Description

schemaPath

string

Location in your local disk of the schema file used to parse your input

segmentIdent

string

In case the schema file defines multiple different structures, this field selects which to use

missingValues

string

nulls

How missing values are represented in the input data:

* none: treat all data as actual values * spaces: interpret a field consisting of only spaces as a missing value * zeroes: interpret numeric fields consisting of only '0' characters and character fields consisting of only spaces as missing values * nulls: interpret a field consisting only of 0 bytes as a missing value

recordParsing

string

strict

expected separation between lines/records:

  • strict: line break expected at exact end of each record

  • lenient: line break used but records may be shorter or longer than schema specifies

  • noTerminator: means records follow one another with no separation

These properties can be either set via the XML of your Mule project:

<dw:input-payload mimeType="text/plain" >
    <dw:reader-property name="schemaPath" value="myschema.ffs"/>
    <dw:reader-property name="segmentIdent" value="structure1"/>
</dw:input-payload>

Or via the UI of the Transform Message component

See To Define Input and Output Structure of a Transformation.

Schemas with type Binary or Packed don’t allow for line breaks to be detected, so setting recordParsing to "lenient"` will only allow long records to be handled, but not short ones. These schemas also currently only work with certain single-byte character encodings (so not with UTF-8 or any multibyte format).

Writer Properties

When defining an output of type copybook there are a few optional parameters you can add to the output directive to customize how the data is written:

Parameter Type Default Description

schemaPath

string

Path where the schema file to be used is located

segmentIdent

string

In case the schema file defines multiple formats, indicates which of them to use

encoding

string

UTF-8

Output character encoding

missingValues

string

nulls

How to represent optional values missing from the supplied map:

* spaces: fill the field with spaces * nulls: use 0 bytes

recordTerminator

string

standard Java line termination for the system

Termination for every line/record. In Mule runtime versions 3.8.4 and older, this is only used as a separator when there are multiple records. Possible values: lf, cr, crlf, none. Values translate directly to character codes (none leaves no termination on each record).

trimValues

boolean

false

Trim string values longer than field length by truncating trailing characters

 
%dw 1.0
%output text/plain schemaPath="src/main/resources/test-data/QBReqRsp.esl", structureIdent=“QBResponse"
---
payload

See DataWeave Flat File or EDI Schemas for instructions and examples on how to create the required schema file.

Defining a Metadata Type

In the Transform Message component, you can define a Cobol Copybook type through the following methods:

See Flat File Schemas for more detailed instructions on how to write the required schema.

See Also