Adding DataSense

DevKit is compatible only with Studio 6 and Mule 3. To build Mule 4 connectors, see the Mule SDK documentation.

Mule DataSense is a feature that displays the metadata of entities included in the service in a friendly manner. Although this feature is optional, Mule strongly recommends that you DataSense-enable your connector to make its implementation must easier for users.

Assumptions

This document assumes that you are familiar with the connector architecture presented in Anypoint Connector DevKit and that you are familiar with DataSense from an end-user’s perspective.

Connectors with DataSense

Integration developers often spend great amounts of time simply trying to determine the parameters and types of data that have to be passed or are returned by a Web service. Having to resort to each API’s documentation to find out what these parameters are is an inefficient, fallible, and often frustrating experience. Thanks to DataSense, this information can be readily available with Anypoint™ Studio at design-time.

DataSense provides entities and their internal structure to Studio, which, in turn, presents this information to users employing the connector. DataSense also works with other Anypoint Studio features to:

Access a connector’s metadata so that when the context is right, Studio intelligently makes suggestions for expected values in fields returned by connector.
Use DataMapper’s ability to automatically infer the input or output data within a mapping (when used in conjunction with a connector that is DataSense-enabled).

Learn more about DataSense

Basic Anatomy

There are two key steps to implementing DataSense:

Configure metadata retrieval - Obtains the metadata from the service and provides an implementation on the connector to supply this information.
Configure metadata awareness - Defines how the operations are annotated in order for Anypoint Studio to be aware of the DataSense implementation and provide information about it to end users of the connector.

Metadata Retrieval

The fundamental advantage of DataSene is the ability to extract metadata from the data model exposed by an application through the connector. When the connector has static data model (also known as "strongly typed" data model) - that is, when it exposes its entities as a POJO - this is straightforward: the POJO entity class definitions can be referenced and they can provide all the metadata needed in Java, available via introspection.

When the connector has a dynamic data model (also known as "weekly typed") - that is, when it exposes its entities to Mule as key-value Maps - the metadata is not immediately available. To support DataSense functionality for connectors with dynamic data models, you must implement additional functionality to create the metadata based on data provided from the application.

In either case, metadata retrieval requires that you include two annotated metadata-related methods on your connector which generate DataSense metadata for a dynamic schema.

@MetadataKeyRetriever retrieves a list of all the entity Type names from the connected service
@MetadataRetriever uses the list of metadata keys (retrieved by @MetadataKeyRetriever) to retrieve the entity composition of each entity Type

@MetaDataKeyRetriever
public List<MetaDataKey> getMetadataKeys() {  }

@MetaDataRetriever
public MetaData getMetadata(MetaDataKey key) {  }

Metadata Awareness

This step involves making the captured metadata accessible to the message processor. When implemented, a drop-down in the connector’s properties editor in Studio displays all of the entities returned by @MetaDataKeyRetriever; each of these is coupled with the properties returned by @MetaDataRetriever.

For this to happen, the message processor must include a method that receives the entity type as a parameter annotated with @MetaDataKeyParam. This method must also receive the entity data (that was returned by @MetadataRetriever) on a parameter that is annotated as @Default ("#[payload]").

When building connectors for Mule runtimes prior to 3.5.0, all @Default parameters must also be marked with @Optional. In newer versions, this second annotation is considered redundant, but it is required on Mule runtime 3.4.2 and older.

The types of data expected and generated by this method vary depending on if your metadata is static or dynamic.

If your metadata static, then the entity data is an Object:

public Object create(@MetaDataKeyParam String entityType, @Default("#[payload]") Object entityData) {
    }

If your metadata is dynamic, the entity data is a Map<String,Object>:

public Map<String,Object> create(@MetaDataKeyParam String entityType, @Default("#[payload]") Map<String,Object> entityData) {
    }

If your metadata is a list of dynamic objects, the entity data is a List<Map<String,Object>>:

public List<Map<String,Object>> getList(@MetaDataKeyParam String entityType, @Default("#[payload]") List<Map<String,Object>> entityData) {
    }

Preparing a @Connector Class to Add Metadata Support

To implement DataSense in your connector, you must add the following imports to the @Connector class:

import org.mule.common.metadata.*;
import org.mule.common.metadata.builder.DefaultMetaDataBuilder;
import org.mule.common.metadata.builder.DynamicObjectBuilder;
import org.mule.common.metadata.builder.EnumMetaDataBuilder;
import org.mule.common.metadata.datatype.DataType;

import org.mule.api.annotations.MetaDataKeyRetriever;
import org.mule.api.annotations.MetaDataRetriever;

The org.mule.common.metadata.* classes include the Mule classes for representing and managing metadata.
The org.mule.common.metadata.builder classes are used to construct metadata representations (sets of objects which can be quite complex).
The org.mule.common.metadata.datatype.DataType class represents different object field datatypes and their properties.
If you generate your DevKit connector project in Anypoint™ Studio May 2014 with the 3.5.0 runtime, DevKit automatically generates these imports.

Example with Static Data Model

The following section demonstrates how to build a connector that draws data from a Web service with a static data model.

Download a full working example of this static-metadata connector from GitHub.

In this example, the Web service to which the connector connects a library Web service. The Web service contains two types of elements: books and authors.

The book element contains the following fields:

title
synopsis
author

The author element contains the following fields:

firstName
lastName

Metadata Retrieval

As already stated, the first step is to obtain the metadata from the connector. As the metadata is static, you don’t need to make a call the Web service to obtain it, you can just hard-code the information into the connector. Inside your connector class, add a new method annotated with @MetaDataKeyRetriever
```
@MetaDataKeyRetriever
    public List<MetaDataKey> getEntities() throws Exception {
        List<MetaDataKey> entities = new ArrayList<MetaDataKey>();
        entities.add(new DefaultMetaDataKey("Book_id","Book"));
        entities.add(new DefaultMetaDataKey("Author_id","Author"));
        entities.add(new DefaultMetaDataKey("BookList_id","BookList"));
        return entities;
    }
```
This method returns a list of the entity’s names. In this case, it retrieves a list with three keys: Book, BookList, and Author.

The next step is to implement @MetaDataRetriever method, which obtains a description of each of the entities returned by the previous method. The return type of this java method MUST be MetaData and it MUST receive one MetaDataKey parameter. In this example, assume that the entity classes of the service exist locally .Book.class and Author.class can then be directly referenced in your code when describing them. You can call the interface DefaultMetadataBuilder, provided by DevKit, to easily build a POJO.

@MetaDataRetriever
    public MetaData describeEntity(MetaDataKey entityKey) throws Exception {
        //Here we describe the entity depending on the entity key
        if ("Author_id".equals(entityKey.getId())) {
            MetaDataModel authorModel =  new DefaultMetaDataBuilder().createPojo(Author.class).build();
            return new DefaultMetaData(authorModel);
        }
        if ("Book_id".equals(entityKey.getId())) {
            MetaDataModel bookModel =  new DefaultMetaDataBuilder().createPojo(Book.class).build();
            return new DefaultMetaData(bookModel);
        }
        if ("BookList_id".equals(entityKey.getId())) {
            MetaDataModel bookListModel =  new DefaultMetaDataBuilder().createList().ofPojo(Book.class).build();
            return new DefaultMetaData(bookListModel);
        }
        throw new RuntimeException(String.format("This entity %s is not supported",entityKey.getId()));
   }

This method automatically describe Book, BookList, and Author with all the public fields exposed by them.

The reason why two different operations are used - one to obtain the entities, and another obtain their description - is that describing all the entities through one single method could result in a excessive number of API calls (you probably need one API call per entity). This arrangement is ideal for retrieving metadata from an external service.

Metadata Awareness

So far you have implemented the describing mechanism for all of the entities in the service you aim to connect. Now you must make this information accessible to the message processors.

The method must receive the operation’s type as a parameter annotated with @MetaDataKeyParam. The method must also receive the entity data (that was returned by @MetaDataRetriever) on a parameter annotated with as @Default ("#[payload]").

@Processor
    public Object create(@MetaDataKeyParam String entityType, @Default("#[payload]") Object entityData) {
        if (entityData instanceof Book) {
            return createBook((Book) entityData);
        }
        if (entityData instanceof Author) {
            return createAuthor((Author) entityData);
        }
        throw new RuntimeException("Entity not recognized");
    }

    private Object createAuthor(Author entityData) {
        //CODE FOR CREATING NEW AUTHOR GOES HERE
        return null;
    }

    private Object createBook(Book entityData) {
        //CODE FOR CREATING A NEW BOOK GOES HERE
        return null;
    }

The output metadata changes according to the entity type selected in Studio. This is especially useful when used in conjunction with a DataMapper transformer. Thanks to this method, all the entities returned by @MetaDataRetriever will then will be displayed in a drop-down in Studio.

Also, the metadata about the entity can then be passed on to other Mule elements such as DataMapper.

Example with Dynamic Data Model

The following section demonstrates how to build a connector that draws data from a Web service with a dynamic data model. The most practical way to implement metadata is always dynamically. Doing things this way, if the entity’s attributes in the service you connect to vary over time, your connector effortlessly adapts to the changes.

Download a full working example of this dynamic-metadata connector from GitHub.

In this example, as in the one above, the Web service to which the connector connects a book database. It contains two types of elements: books and authors, both contain the same fields as in the previous example.

Metadata Retrieval

In tis example, because you don’t have access to a POJO with the type structure, you must obtain this structure from the Web service itself. Use Map<String,Object> to represent the dynamics entities.

If you obtain the metadata dynamically through an API call, the @Connect method executes before the @MetaDataKeyRetriever method. This implies that end-users must first resolve any connection issues before gaining access to the metadata.

Inside your connector class, add a new method annotated with @MetaDataKeyRetriever. (This method is no different from the one implemented with static metadata).

@MetaDataKeyRetriever
    public List<MetaDataKey> getEntities() throws Exception {
        List<MetaDataKey> entities = new ArrayList<MetaDataKey>();
        entities.add(new DefaultMetaDataKey("Book_id","Book"));
        entities.add(new DefaultMetaDataKey("Author_id","Author"));
        entities.add(new DefaultMetaDataKey("BookList_id","BookList"));
        return entities;
    }

The next step is to implement @MetaDataRetriever method. This obtains a description of each of the entities returned by the previous method. As in the previous example, this method uses the interface DefaultMetaDataBuilder, but this time it will be called to build dynamic objects instead of POJOs.

@MetaDataRetriever
    public MetaData describeEntity(MetaDataKey entityKey) throws Exception {
        //Here we describe the entity depending on the entity key
        if ("Author_id".equals(entityKey.getId())) {
            MetaDataModel authorModel =  new DefaultMetaDataBuilder().createDynamicObject("Author")
                    .addSimpleField("firstName", DataType.STRING)
                    .addSimpleField("lastName", DataType.STRING)
                    .build();
            return new DefaultMetaData(authorModel);
        }
        if ("Book_id".equals(entityKey.getId())) {
            MetaDataModel bookModel =  new   DefaultMetaDataBuilder().createDynamicObject("Book")
                    .addSimpleField("title",DataType.STRING)
                    .addSimpleField("synopsis",DataType.STRING)
                    .addDynamicObjectField("author")
                    .addSimpleField("firstName",DataType.STRING)
                    .addSimpleField("lastName",DataType.STRING)
                    .endDynamicObject()
                    .build();
            return new DefaultMetaData(bookModel);
        }
        if ("BookList_id".equals(entityKey.getId())) {
            MetaDataModel bookListModel =  new DefaultMetaDataBuilder().createList().ofDynamicObject("book").build();
            return new DefaultMetaData(bookListModel);
        }
        throw new RuntimeException(String.format("This entity %s is not supported",entityKey.getId()));
    }

Metadata Awareness

Thus far, you have implemented the describing mechanism for all of the entities in the service you aim to connect. Now you must make this information accessible to the message processor.

The message processor must receive the operation’s type as a parameter annotated with @MetaDataKeyParam (Studio displays the operations in a drop-down with all the entities returned by @MetaDataRetriever). The message processor must also receive the entity data (returned by @MetadataRetriever) as a Map<String,Object> parameter, annotated as @Default ("#[payload]").

@Processor
    public Object create(@MetaDataKeyParam String entityType, @Default("#[payload]") Object entityData) {
        if (entityData instanceof Book) {
            return createBook((Book) entityData);
        }
        if (entityData instanceof Author) {
            return createAuthor((Author) entityData);
        }
        throw new RuntimeException("Entity not recognized");
    }

    private Map<String, Object> createAuthor(Map<String, Object> entityData) {
        //CODE TO CREATE BOOK GOES HERE
        return entityData;
    }
    private Map<String, Object> createBook(Map<String, Object> entityData) {
        //CODE TO CREATE AUTHOR GOES HERE
        return entityData;
    }

Thanks to this method, Studio displays all the entities returned by @MetaDataRetriever as items in a drop-down field.

Also, the metadata about the entity can then be passed on to other Mule elements such as DataMapper

Grouping Types

To group types, DevKit uses the annotation @MetaDataCategory that you can apply to a new .java class. Within this java class, you define the whole metadata retrieving mechanism, that is the methods annotated with @MetaDataKeyRetriever and @MetaDataRetriever.

For example, suppose you wish to offer a regular message processor and, additionally, a special message processor that provides access to a secret field on the Author entity named “books” that represents the Author’s written books. You can use group types to bundle several distinct message processors into one connector, and display different groups of entities in each.

The example below displays a @MetaDataCategory class that contains both methods, and resides in a separate java file. In this default class, both @MetaDataKeyRetriever and @MetaDataRetriever methods are the same as in the previous example. You must establish a link between this class and your connector module; the most common way of doing this is to @Inject your connector class as shown below.

@MetaDataCategory
public class DefaultCategory {

    @Inject
    private MyConnector myconnector;

    @MetaDataKeyRetriever
    public List<MetaDataKey> getEntities() throws Exception {
        List<MetaDataKey> entities = new ArrayList<MetaDataKey>();
        entities.add(new DefaultMetaDataKey("Book_id","Book"));
        entities.add(new DefaultMetaDataKey("Author_id","Author"));
        entities.add(new DefaultMetaDataKey("BookList_id","BookList"));
        return entities;
    }

    @MetaDataRetriever
        public MetaData describeEntity(MetaDataKey entityKey) throws Exception {
        //Here we describe the entity depending on the entity key
        if ("Author_id".equals(entityKey.getId())) {
            MetaDataModel authorModel =  new DefaultMetaDataBuilder().createDynamicObject("Author")
                    .addSimpleField("firstName", DataType.STRING)
                    .addSimpleField("lastName", DataType.STRING)
                    .build();
            return new DefaultMetaData(authorModel);
        }
        if ("Book_id".equals(entityKey.getId())) {
            MetaDataModel bookModel =  new   DefaultMetaDataBuilder().createDynamicObject("Book")
                    .addSimpleField("title",DataType.STRING)
                    .addSimpleField("synopsis",DataType.STRING)
                    .addDynamicObjectField("author")
                    .addSimpleField("firstName",DataType.STRING)
                    .addSimpleField("lastName",DataType.STRING)
                    .endDynamicObject()
                    .build();
            return new DefaultMetaData(bookModel);
        }
        if ("BookList_id".equals(entityKey.getId())) {
            MetaDataModel bookListModel =  new DefaultMetaDataBuilder().createList().ofDynamicObject("book").build();
            return new DefaultMetaData(bookListModel);
        }
        throw new RuntimeException(String.format("This entity %s is not supported",entityKey.getId()));
}

Then, define a second group for a different message processor that can access different metadata as shown below.

@MetaDataCategory
public class AdvancedCategory {

    @Inject
    private MyConnector myconnector;

    @MetaDataKeyRetriever
    public List<MetaDataKey> getEntities() throws Exception {
        List<MetaDataKey> entities = new ArrayList<MetaDataKey>();
        entities.add(new DefaultMetaDataKey("Book_id","Book"));
        entities.add(new DefaultMetaDataKey("Author_id","Author"));
        entities.add(new DefaultMetaDataKey("BookList_id","BookList"));
        return entities;
    }
    @MetaDataRetriever
    public MetaData describeEntity(MetaDataKey entityKey) throws Exception {

        //Here we describe the entity depending on the entity key
        if ("Author_id".equals(entityKey.getId())) {
            MetaDataModel authorModel =  new DefaultMetaDataBuilder().createDynamicObject("Author")
                    .addSimpleField("firstName", DataType.STRING)
                    .addSimpleField("lastName", DataType.STRING)
                    .build();
            return new DefaultMetaData(authorModel);
        }
        if ("Book_id".equals(entityKey.getId())) {
            MetaDataModel bookModel =  new   DefaultMetaDataBuilder().createDynamicObject("Book")
                    .addSimpleField("title",DataType.STRING)
                    .addSimpleField("synopsis",DataType.STRING)
                    .addDynamicObjectField("author")
                    .addSimpleField("firstName",DataType.STRING)
                    .addSimpleField("lastName",DataType.STRING)
                    .endDynamicObject()
                    .build();
            return new DefaultMetaData(bookModel);
        }
        if ("BookList_id".equals(entityKey.getId())) {
            MetaDataModel bookListModel =  new DefaultMetaDataBuilder().createList().ofDynamicObject("book").build();
            return new DefaultMetaData(bookListModel);
        }
        throw new RuntimeException(String.format("This entity %s is not supported",entityKey.getId()));
    }
}

Finally, like these categories within your connector’s message processor as shown below.

/**
 * Connector for integration tests
 *
 * @author MuleSoft, inc.
 */
@MetaDataScope(DefaultCategory.class)
@Connector(name = "my-connector", minMuleVersion = "3.5")
public class MyConnector {
..

  @MetaDataScope(AdvancedCategory.class)
    @Processor
    public Map<String,Object> advancedOperation(@MetaDataKeyParam String entityType, @Default("#[payload]") Map<String,Object> entityData) {
       //here you can use the books field in authors//

    }
}

Gartner names MuleSoft a Leader

Unleash the power of Salesforce Customer 360 through integration

Anypoint Platform Fundamentals

MuleSoft at World Tour

Gartner names MuleSoft a Leader

Unleash the power of Salesforce Customer 360 through integration

Anypoint Platform Fundamentals

MuleSoft at World Tour

Adding DataSense

Assumptions

Connectors with DataSense

Basic Anatomy

Metadata Retrieval

Metadata Awareness

Preparing a @Connector Class to Add Metadata Support

Example with Static Data Model

Metadata Retrieval

Metadata Awareness

Example with Dynamic Data Model

Metadata Retrieval

Metadata Awareness

Grouping Types

See Also