All Priorities
View and Configure Logging in Runtime Fabric
Runtime Fabric generates log files for the following entities:
-
Deployed Mule applications
-
Deployed API proxies
-
Runtime Fabric services
-
Kubernetes services
Log Levels
Runtime Fabric enables you to specify the severity level of messages that are written to the log file.
Value | Description | Command |
---|---|---|
List all messages. |
N/A |
|
ERROR |
List only error messages, such when an exception occurs. |
priority:ERROR |
FATAL |
List only fatal messages when an application fails. |
priority:FATAL |
INFO |
List informative messages. |
priority:INFO |
SYSTEM |
List messages about application and worker startup. |
priority:SYSTEM |
CONSOLE |
List messages about console events such as setting the object store. |
priority:CONSOLE |
WARN |
List warning messages. |
priority:WARN |
DEBUG |
List debugging messages. |
priority:DEBUG |
Log levels are specified per Mule application or API proxy during deployment.
You can use Anypoint Runtime Manager to change the application log level by configuring the following property:
logging.level.<PACKAGE>=LEVEL
For example, if you use Runtime Manager to add the
logging.level.org.mule.service.http.impl.service.HttpMessageLogger=DEBUG
property, DEBUG
level log information is emitted from HttpMessageLogger
.
View Logs from an Application
Ops Center shows a stream of logs generated by applications and services running on Runtime Fabric. This is useful when log forwarding is not configured.
-
From Ops Center, navigate to Logging.
-
Select Kubernetes.
-
Select the Pods tab.
-
In the drop-down list adjacent to the search input area, select the ID of the environment where the application is deployed.
-
Locate the pod name that starts with the name of your application.
-
Select that pod name and then select Logs.
The Logs tab is displayed with a filter applied to your application.
To view the latest logs, select Refresh. |
Forward Logs to External Services
Runtime Fabric supports log forwarding to:
-
Anypoint Monitoring
-
Third-party logging solutions using log forwarding output plugins:
-
Azure Log Analytics
-
Elasticsearch
-
Graylog Extended Log Format (GELF)
-
Splunk
-
|
Enable Log Forwarding
-
Install and configure the applicable third-party logging solution. Verify that you have the required resources allocated. Refer to the documentation for your third-party logging solution for details.
-
From Anypoint Platform, select Runtime Manager.
-
Select Runtime Fabrics in the sidebar navigation.
-
Select the applicable Runtime Fabric based on the name used during installation.
-
Select the Log Forwarding tab.
-
Select the Forward to Anypoint Monitoring (Requires Titanium) checkbox if you have a Titanium subscription and you want to forward logs to Anypoint Monitoring.
-
Select the Forward application logs to an external service checkbox to forward logs using a log forwarding output plugin.
-
In the Connects to drop-down list, select the applicable third-party log forwarding solution.
-
Enter configuration information for your third-party logging solution, as shown in the following sections:
-
Select Apply changes.
-
To verify successful log forwarding, manually check the logs using the external log forwarding service.
-
Azure Log Analytics Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Customer ID |
Specifies the customer ID or workspace ID string. |
Yes |
|
Shared Key |
Specifies the primary or secondary connected sources client authentication key. |
Yes |
|
Log Type |
Specifies the event type name. |
No |
|
Example output plugin configuration:
[OUTPUT]
Name azure
Customer_ID id
Shared_Key key
Elasticsearch Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Host |
Specifies the IP address or hostname of the target Elasticsearch instance. |
Yes |
127.0.0.1 |
Port |
Specifies the TCP port of the target Elasticsearch instance. |
Yes |
9200 |
Index |
Specifies the index name. |
Yes |
runtime_fabric |
Path |
Elasticsearch accepts new data on HTTP query path |
No |
Empty string |
Buffer Size |
Specifies the buffer size used to read the response from the Elasticsearch HTTP service. This option is useful for debugging purposes where reading the full response is needed. Note that the size of the response grows depending on the number of records inserted. To specify an unlimited amount of memory, set this value to |
No |
4KB |
HTTP User |
Specifies an optional username credential for Elastic X-Pack access. |
No |
|
HTTP Passwd |
Specifies a password for the user defined in |
No |
|
Type |
Specifies the type name. |
No |
runtime_fabric |
Logstash Format |
Enables Logstash format compatibility. This option takes a Boolean value: |
No |
|
Logstash Prefix |
When |
No |
logstash |
Logstash DateFormat |
Specifies the time format (based on |
No |
%Y.%m.%d |
Time Key |
When |
No |
@timestamp |
Time Key Format |
When |
No |
%Y-%m-%dT%H:%M:%S |
Include Tag Key |
When enabled, the tag name is appended to the record. |
No |
Off |
Tag Key |
When |
No |
|
Generate ID |
When enabled, generates the |
No |
Off |
Replace Dots |
When enabled, replaces field name dots ('.') with an underscore ('_'), which is required by Elasticsearch 2.0 through 2.3. |
No |
Off |
Trace Output |
When enabled, prints the Elasticsearch API calls to stdout (for diagnosis). |
No |
Off |
Trace Error |
When enabled, prints the Elasticsearch API calls to stdout when Elasticsearch returns an error. |
No |
Off |
Current Time Index |
Specifies using the current time for index generation instead of message record information. |
No |
Off |
Logstash Prefix Key |
Prefixes keys with this string. |
No |
|
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name es
Host 192.168.2.3
Port 9200
Index my_index
Type my_type
GELF Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Match |
Specifies the pattern to match for log tags to be output by this plugin. |
Yes |
|
Host |
Specifies the IP address or hostname of the target Graylog server. |
Yes |
127.0.0.1 |
Port |
Specifies the port on which your Graylog GELF input is listening. |
Yes |
12201 |
Mode |
Specifies the protocol to use ( |
Yes |
udp |
Gelf Short_Message Key |
Specifies a short descriptive message (This must be set in GELF). |
Yes |
log |
Gelf Timestamp Key |
Specifies your log timestamp (This should be set in GELF). |
No |
timestamp |
Gelf Host Key |
Specifies the key for the value that is used as the name of the host, source, or application that sent the message. (This must be set in GELF). |
No |
host |
Gelf Full Message Key |
Specifies the key to use as the long message, which can contain a backtrace. (This is optional in GELF.) |
No |
full_message |
Gelf Level Key |
Specifies the key to be used as the log level. Its value must be a standard syslog level (between 0 and 7). |
No |
level |
Packet Size |
Specifies the size of packets to be sent if the transport protocol is set to |
No |
1420 |
Compress |
Specifies compression of your UDP packets if the transport protocol is set to |
No |
true |
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name gelf
Match kube.*
Host <your-graylog-server>
Port 12201
Mode tcp
Gelf_Short_Message_Key log
Splunk Configuration Parameters
Key | Description | Required | Default Value |
---|---|---|---|
Host |
Specifies the IP address or hostname of the target Splunk service. |
Yes |
127.0.0.1 |
Port |
Specifies the TCP port of the target Splunk service. |
Yes |
8088 |
Splunk Token |
Specifies the Authentication Token for the HTTP Event Collector interface. |
Yes |
|
HTTP User |
Specifies an optional username for basic authentication on the HTTP Event Collector. |
No |
|
HTTP Passwd |
Specifies a password for the user defined in |
No |
|
TLS |
Enables or disables TLS support. |
No |
Off |
CA Path Certificate File |
Specifies the CA certificate file to be uploaded. This option is available only when TLS is enabled. |
No |
Example output plugin configuration:
[OUTPUT]
Name splunk
Host 127.0.0.1
Port 8088
Splunk_Token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
Tls On
Configure Log Forwarding
The following configuration options are available:
-
Mule Applications: Forward logs from Mule applications.
-
Internal load balancer and components: Forward logs from Runtime Fabric components and the internal load balancer.
-
Runtime Fabric appliance: Forward logs from monitoring and appliance services.
|
Kubernetes Metadata
When forwarding logs to external services, additional metadata might be available in some providers. When performing searches, the user can leverage these metadata labels to help narrow down search results and focus on the logs only from containers that matters.
Key | Description | Sample Value |
---|---|---|
container_name |
Specifies the name of the container. |
app |
container_image |
Specifies the full name of the container image. |
000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0:v1.0.0 |
container_hash |
Specifies the name of the container image and includes the immutable identifier (SHA256 value). |
000.dkr.ecr.us-east-1.amazonaws.com/mulesoft/poseidon-runtime-4.3.0@sha256:866daf14a55da1db21c562ddc09487bbb8ffae1dd4cc80b4dc5c64e0ac9c124b |
docker_id |
Specifies the ID of the container in Docker. |
a54a0bf80400b6cd44cae53b9705f5b6330ce4e8efb85c92b3f9c1709829c6fa |
host |
Specifies the IP of the node on which the pod is running. |
10.0.178.87 |
namespace_name |
Specifies the Kubernetes namespace of the pod. |
rtf |
pod_id |
Specifies the ID of the pod. |
118cff63-a5ac-11ea-8414-0e281fcb2861 |
pod_name |
Specifies the name of the pod. |
testapp-5789f9774-6lxv7 |
labels_app |
Specifies the app label for the application the pod is supporting. |
testapp |
labels_environment |
Specifies the Anypoint environment ID of the environment in which the application is deployed. |
24166311-4e5e-4091-b417-42307747267e |
labels_organization |
Specifies the Anypoint organization ID of the organization in which the application is deployed. |
24166311-4e5e-4091-b417-42307747267e |
labels_rtf_mulesoft_com_id |
Specifies the ID of the application deployment on Anypoint Platform. |
194a32bf-bf7e-4c91-aa4e-44132e5aa76d |
labels_rtf_mulesoft_com_version |
Specifies the spec version ID of the application deployment on Anypoint Platform. |
194a32bf-bf7e-4c91-aa4e-44132e5aa76d |
labels_type |
Specifies the type of deployment that the pod is running. |
MuleApplication |
Send a Test Message
You can send a test message to verify connectivity between Runtime Fabric and the third-party log forwarding solution:
-
In Runtime Manager, select Runtime Fabrics in the sidebar navigation.
-
Select the applicable Runtime Fabric based on the name used during installation.
-
Select the Log Forwarding tab.
-
Select Forward application logs to an external service.
-
Select a service and configure the fields as needed for your log forwarding solution.
-
Save the changes and apply the configuration to the Runtime Fabric cluster.
-
To test the configuration and reachability of the log forwarding solution:
-
Trigger a test message for the selected service.
-
Verify the text that is forwarded to log forwarding solution.
-
Disable Anypoint Monitoring or External Log Forwarding per Application
-
Navigate to Runtime Manager and select Applications.
-
Select the application for which you want to disable log forwarding.
-
Select the Logs tab.
-
Deselect the applicable log forwarding options:
-
Forward logs to Anypoint Monitoring
-
Forward logs to <third-party service>
-
-
Select Deploy to redeploy the application.
Forward Logs to Other External Services
Anypoint Runtime Fabric also enables you to forward application and cluster logs to external logging solutions other than the third-party solutions supported using log forwarding output plugins. The log forwarder (rsyslog
client service) built into Runtime Fabric enables you to send log data to an rsyslog
server over TCP or UDP for viewing, retention, and receiving alerts in a centralized destination. Logging solutions such as Logstash provide methods to receive log data from rsyslog
clients.
Anypoint Runtime Fabric provides dashboards and alerts for critical metrics when performance or availability is compromised. An SMTP server is required to receive alerts. To view and configure Anypoint Runtime Fabric dashboards and alerts using Ops Center:
-
Using a terminal, open a shell/SSH connection to a controller VM.
-
Create a file named
log-forwarder.yaml
. -
Add the following content to this file after customizing based on the table below:
kind: logforwarder version: v2 metadata: name: log-forwarder spec: address: 192.168.100.1:514 protocol: udp
Use the following values as applicable to your environment:
Table 7. Log Forwarding Configuration Parameters Key Description name
Specifies the name of the log forwarding rule.
address
Specifies the endpoint and port to forward the log data.
protocol
Specifies the protocol to send the data to. Supported protocols are TCP or UDP.
-
Run the following command on the controller VM, referencing the file created earlier.
gravity resource create log-forwarder.yaml
Your logs are forwarded to your external logging solution.
Troubleshoot Log Forwarding Issues
If you experience issues with log forwarding:
-
Verify that the log forwarding configuration specifies the correct host and port information.
-
Verify that the server side configuration, such as networking, is correct and that the server is accessible.
-
Verify log forwarding status:
-
Navigate to Runtime Manager and select Runtime Fabrics.
-
Select the name of your Runtime Fabric to open the management page.
-
Select the Health Details tab.
-
Verify the following information as applicable:
-
The status of Forwarding Logs To External Provider is
Healthy
. -
The status Forwarding Logs to Anypoint Monitoring is
Healthy
.
-
-
-
Run the following commands inside the RTF cluster:
-
kubectl -nrtf get pods -lapp=external-log-forwarder
for a list of all external log forwarding pods. Verify that these pods are inRunning
state. -
kubectl -nrtf logs -l app=external-log-forwarder
to view the logs of external log forwarding pods.
-