Contact Us 1-800-596-4880
  1. Homepage
  2. Anypoint Connector DevKit (3.4)
  3. DevKit Advanced Topics

  4. Supporting DataSense with Dynamic Data Models

Supporting DataSense with Dynamic Data Models

The fundamental capability underlying DataSense is the ability to extract metadata from the data model exposed by an application through the connector. When the connector exposes its entities as POJOs – that is, when it has a static data model – this is straightforward: the POJO entity class definitions provide all the metadata needed in Java, available via introspection.

When the connector has a dynamic data model – that is, when it exposes its entities to Mule as key-value Maps – the metadata is not immediately available.To support DataSense functionality for connectors with dynamic data models, implement additional functionality to create the metadata based on data provided from the application.

If the connectors with dynamic data models rely on OAuth, DataSense metadata is not supported. DataSense is a design-time feature, and Mule Studio does not currently have the ability to perform the "OAuth dance" and manage tokens during the design process to get access to metadata on the target application.

The most straightforward way to understand the construction of a metadata-enabled connector is to look at an example of a connector that supports complex objects with lots of metadata. Accordingly, this discussion will draw its examples from the Salesforce.com connector.

Assumptions

Representing DataSense Metadata

The package org.mule.common.metadata defines classes that manage and represent DataSense metadata. The most important DevKit classes for building metadata include:

  • Interface MetadataModel

  • Interface MetadataKey and class DefaultMetadataKey. These represent a key used to reference metadata in the model.

The next section introduces proper use of these classes to build up a metadata representation.

DevKit requires two metadata-related methods on a connector to generate DataSense metadata for a dynamic schema:

  • A @MetadataKeyRetriever method retrieves a list of all the entity Type names from the application.

  • A @MetadataRetriever method uses the list of metadata keys to retrieve metadata about each entity Type.

Operations on a DataSense-enabled connector with a dynamic schema must also include a parameter annotated @MetadataKeyParam that identifies the entity Type of the passed Map.

Preparing a @Connector Class to Add Metadata Support

Add the following imports to the @Connector class:

import org.mule.common.metadata.*;
import org.mule.common.metadata.builder.DefaultMetaDataBuilder;
import org.mule.common.metadata.builder.DynamicObjectBuilder;
import org.mule.common.metadata.builder.EnumMetaDataBuilder;
import org.mule.common.metadata.datatype.DataType;

import org.mule.api.annotations.MetaDataKeyRetriever;
import org.mule.api.annotations.MetaDataRetriever;

Note:

  • The org.mule.common.metadata.* classes include the Mule classes for representing and managing metadata.

  • The org.mule.common.metadata.builder classes are used to construct metadata representations (sets of objects which can be quite complex).

  • The org.mule.common.metadata.datatype.DataType class represents different object field datatypes and their properties.

The Salesforce.com connector code uses these classes in context, as reviewed below.

Implementing a @MetadataKeyRetriever Method

The @MetadataKeyRetriever method returns a List<MetaDataKey> that represents the list of object types defined in the application. The specifics of retrieving the object type names will depend upon the particular target application; in the case of Salesforce.com, the class com.sforce.soap.partner.DescribeGlobal retrieves the metadata from Salesforce, and the `@MetadataKeyRetrieve`r method builds the list of types based on the results.

@MetaDataKeyRetriever
public List<MetaDataKey> getMetaDataKeys() throws Exception {
    List<MetaDataKey> keys = new ArrayList<MetaDataKey>();
    DescribeGlobalResult describeGlobal = describeGlobal();

    if (describeGlobal != null) {
        DescribeGlobalSObjectResult[] sobjects = describeGlobal.getSobjects();
        for (DescribeGlobalSObjectResult sobject : sobjects) {
            keys.add(new DefaultMetaDataKey(sobject.getName(), sobject.getLabel(), sobject.isQueryable()));
        }
    }
 return keys;
}

List<MetaDataKey> keys store DefaultMetadataKey instances. DefaultMetadataKey’s constructor identifies metadata keys by a String id:

public DefaultMetaDataKey(String id, String displayName, boolean isFromCapable) {
        this.id = id;
        this.displayName = displayName;
        this.isFromCapable = isFromCapable;
    }

Note:

  • The DefaultMetadataKey property id is set to the name of the sObject from Salesforce.

  • The isFromCapable attribute indicates whether a given Type can be used in a FROM clause in a DSQL query. Some types of objects supported by an application may not be queryable.

Implement similar functionality to populate a list of metadata keys based on metadata available from your application.

Implementing a @MetadataRetriever Method

This section describes how Mule represents the actual metadata for an entity type named in the output of the @MetadataKeyRetriever.

The SalesForce connector @MetadataRetriever method is shown below, along with helper method addField() – the getDataType() method called by addField follows.

getMetadata() and addField()

@MetaDataRetriever
    public MetaData getMetaData(MetaDataKey key) throws Exception {
        DescribeSObjectResult describeSObject = describeSObject(key.getId());


        MetaData metaData = null;
        if (describeSObject != null) {
            Field[] fields = describeSObject.getFields();
            DynamicObjectBuilder dynamicObject = new DefaultMetaDataBuilder().createDynamicObject(key.getId());
            for (Field f : fields) {
               addField(f, dynamicObject);
            }
            MetaDataModel model = dynamicObject.build();
            metaData = new DefaultMetaData(model);
        }
        return metaData;
    }

    private void addField(Field f, DynamicObjectBuilder dynamicObject) {
        DataType dataType = getDataType(f.getType());
        switch (dataType){
            case POJO:
                dynamicObject.addPojoField(f.getName(), Object.class);
                break;
            case ENUM:
                EnumMetaDataBuilder enumMetaDataBuilder = dynamicObject.addEnumField(f.getName());
                if (f.getPicklistValues().length != 0){
                    String[] values = new String[f.getPicklistValues().length];
                    int i =0;
                    for (PicklistEntry picklistEntry : f.getPicklistValues()){
                        values[i] = (picklistEntry.getValue());
                        i++;
                    }
                    enumMetaDataBuilder.setValues(values)
                            .isWhereCapable(f.isFilterable())
                            .isOrderByCapable(f.isSortable());
                }
                break;
            default:
                dynamicObject.addSimpleField(f.getName(), dataType)
                        .isWhereCapable(f.isFilterable())
                        .isOrderByCapable(f.isSortable());
        }
    }

SalesforceConnector.getDataType()

private DataType getDataType(FieldType fieldType) {
        DataType dt;
        switch (fieldType) {
            case _boolean:
                dt = DataType.BOOLEAN;
                break;
            case _double:
                dt = DataType.DOUBLE;
                break;
            case _int:
                dt = DataType.INTEGER;
                break;
            case anyType:
                dt = DataType.POJO;
                break;
            case base64:
                dt = DataType.STRING;
                break;
            case combobox:
                dt = DataType.ENUM;
                break;
            case currency:
                dt = DataType.STRING;
                break;
            case datacategorygroupreference:
                dt = DataType.STRING;
                break;
            case date:
                dt = DataType.DATE_TIME;
                break;
            case datetime:
                dt = DataType.DATE_TIME;
                break;
            case email:
                dt = DataType.STRING;
                break;
            case encryptedstring:
                dt = DataType.STRING;
                break;
            case id:
                dt = DataType.STRING;
                break;
            case multipicklist:
                dt = DataType.ENUM;
                break;
            case percent:
                dt = DataType.STRING;
                break;
            case phone:
                dt = DataType.STRING;
                break;
            case picklist:
                dt = DataType.ENUM;
                break;
            case reference:
                dt = DataType.STRING;
                break;
            case string:
                dt = DataType.STRING;
                break;
            case textarea:
                dt = DataType.STRING;
                break;
            case time:
                dt = DataType.DATE_TIME;
                break;
            case url:
                dt = DataType.STRING;
                break;
            default:
                dt = DataType.STRING;
        }
        return dt;
    }

Note:

  • For the MetaDataKey passed, the @MetadataRetriever getMetadata() method calls Salesforce to extract field names and field types for the object. The addField() method is used to populate the dynamicObject’s representation of the field metadata, including attributes such as whether a field can be used for sorting (`isOrderCapable) and filtering (isWhereCapable) in queries.
    +

  • The different values in the DataType enumerated type in getDataType() represent the different data types Mule can represent in its metadata. For many applications, the data types are richer and more domain-specific than Mule’s data types. For example, many Salesforce field types like string, url, textarea, reference, email, and encryptedString are all represented as DataType.STRING. `However `date, time and date_time are all represented as Mule’s DataType.DATE_TIME.

Implement comparable logic in your connector’s @MetadataRetriever method and any supporting methods.

Implementing Metadata-Enabled Operations

For a connector with operations that use Map<String, Object> as parameters or return values:

  • There is an assumption that there will only be one Map<String, Object> (or List<Map<String,Object>>) parameter to such an operation.

  • One parameter, annotated with @MetaDataKeyParam, must provide the entity type name (same as the DefaultMetadataKey.id field for each type) from the metadata model for the connector. The named type is used as the entity type represented by the Map<String, Object>.

Consider the implementation of one such operation, BaseSalesforceConnector.create():

public List<SaveResult> create(@MetaDataKeyParam String type,
                                   @Optional @Default("#[payload]") List<Map<String, Object>> objects) throws Exception {
        try {
                return Arrays.asList(
                   getConnection().create(toSObjectList(type, objects)));
        } catch (Exception e) {
                throw handleProcessorException(e);
        }
    }

The @MetaDataKeyParam parameter type identifies the Salesforce entity type being added, and is passed to the toSObjectList(type, objects) method that converts the list of incoming maps to a list of SObjects. For another application, the implementation is the same: take the application’s type name, and use the application’s means of converting the passed object to that type.

In this instance, a List<Map<String, Object>> is passed in, and likewise a List<SaveResult> is used to collect the success or failure of each operation.

In some scenarios, it is valid and reasonable for a @MetaDataKeyParam operation that takes a Map or List<Map> as arguments to return a POJO or List of POJOs. For predictability for the user however, well-designed connectors should be consistent in using POJOs as both input and output for either all operations or for none.

See Also