Contact Us 1-800-596-4880
  1. Homepage
  2. Mule Runtime (3.2)
  3. Essentials of Using Mule ESB 3
  4. Mule Query Language

  5. MQL Merge Datasets

MQL Merging Datasets

If you’ve ever had two datasets which you need to merge into a common data format, you know that it can be a pain. (Ever used XSLT?) In this example, we take a look at what it takes to get two activity streams, Twitter and Yammer, into a common format.

Here are some abbreviated snippets of each individual format:

Twitter Yammer 
{    "coordinates": null,    "favorited": false,    "created_at": "Thu Jul 15 23:26:04 +0000 2010",    "truncated": false,    "text": "My Twitter message.",    "contributors": null,    "id": 18639444000,    "geo": null,    "in_reply_to_user_id": null,    "place": null,    "in_reply_to_screen_name": null,    "user": { ... }   }
 
 {      "client_url": "https://www.yammer.com/",      "created_at": "2011/03/28 20:39:12 +0000",      "system_message": false,      "body": {        "parsed": "Myyammer message.",        "plain": "My Yammer message."      },      "sender_type": "user",      "network_id": 104604,      "thread_id": 84402777,      "web_url": "https://www.yammer.com/yammerdeveloperstestcommunity/messages/84402777",      "direct_message": false,      "id": 84402777,      "url": "https://www.yammer.com/api/v1/messages/84402777",      "client_type": "Web",      "message_type": "update",      "sender_id": 4022984,      "replied_to_id": null,      "attachments": ...,      "liked_by": ...,      "privacy": "public"    }

Here is what we’d like to see instead:

[  { "text" : "My Twitter Message.", "source" : "twitter" },  { "text" : "My Yammer Message.", "source" : "yammer" }]

Given that there are Cloud Connectors for both of these, we can write two simple queries to get a timeline from each service and put them in the desired data format:

from yammer.messages select new { text = body.plain, source = 'yammer' }from twitter.publicTimeline as tweet select new { text = tweet.text, source = 'twitter' }

"yammer" and "twitter" can become variables that we can refer to by declaring them in our Mule application XML:

<twitter:config name="twitter"                 format="JSON"                 consumerKey="${twitter.consumer.key}"                 consumerSecret="${twitter.consumer.secret}"/>                          <yammer:config name="yammer"                consumerKey="${yammer.consumer.key}"                consumerSecret="${yammer.consumer.secret}"  />

Now, we can put this in a Mule which aggregates both data sets:

<flow name="get-activity">    <inbound-endpoint address="http://localhost:9002/activity" exchange-pattern="request-response"/>    <all>        <mql:transform query="from yammer.messages select new { text = body.plain }" />        <mql:transform query="from twitter.publicTimeline as tweet select new { text = tweet.text }" />    </all>    <combine-collections-transformer/>    <response>        <json:object-to-json-transformer/>    </response></flow>

Step by step, here is what this flow does: . Listens for requests on http://localhost:9002/activity . When a request comes in, do two queries - one to pull yammer messages and one to pull messages from the twitter public timeline. Each query transforms the result into the desired format. . Combines the Lists of messages from Yammer and Twitter into a single list . Returns the response as JSON objects

Running the example

To run this example, you need to first register the application with Twitter and Yammer so you can use OAuth.

Next, we’ll configure and start the examples application:

  • Download Mule and extract the distribution

  • Download MQL and extract the distribution

  • Copy the examples/mql-examples-0.9.zip file from the MQL distribution to MULE_HOME/apps

  • Start Mule with your OAuth tokens:

 $ cd mule-standalone-3.1.2/bin$ ./mule -M-Dyammer.consumer.key=YAMMER_CONSUMER_KEY -M-Dyammer.consumer.secret=YAMMER_CONSUMER_SECRET \   -M-Dtwitter.consumer.key=TWITTER_CONSUMER_KEY -M-Dtwitter.consumer.secret=TWITTER_CONSUMER_SECRET

Now, we need to perform OAuth authentication for Yammer:

Finally, retrieve the combined Yammer and Twitter activity streams:

 $ curl -v http://localhost:9002/activity