Collect text logs with Azure Monitor Agent

Article
01/05/2023
10 minutes to read

Many applications log information to text files instead of standard logging services such as Windows Event log or Syslog. This article explains how to collect text logs from monitored machines using Azure Monitor Agent by creating a data collection rule (DCR).

Prerequisites

To complete this procedure, you need:

Log Analytics workspace where you have at least contributor rights.
Data collection endpoint.
Permissions to create Data Collection Rule objects in the workspace.
A VM, Virtual Machine Scale Set, or Arc-enabled on-premises server that writes logs to a text file.
- The log file must be stored on the local drive of the machine on which Azure Monitor Agent is running.
- Each entry in the log file must be delineated with an end of line.
- The log file must not allow circular logging, log rotation where the file is overwritten with new entries, or renaming where a file is moved and a new file with the same name is opened.

Create a custom table

This step will create a new custom table, which is any table name that ends in _CL. Currently a direct REST call to the table management endpoint is used to create a table. The script at the end of this section is the input to the REST call.

The table created in the script has two columns TimeGenerated: datetime and RawData: string, which is the default schema for a custom text log. If you know your final schema, then you can add columns in the script before creating the table. If you do not, columns can always be added in the log analytics table UI.

The easiest way to make the REST call is from an Azure Cloud PowerShell command line (CLI). To open the shell, go to the Azure Portal, press the Cloud Shell button, and select PowerShell. If this is your first-time using Azure Cloud PowerShell, you will need to walk through the one-time configuration wizard.

Copy and paste the following script in to PowerShell to create the table in your workspace. Make sure to replace the {subscription}, {resource group}, {workspace name}, and {table name} in the script. Make sure that there are no extra blanks at the beginning or end of the parameters

$tableParams = @'
{
   "properties": {
       "schema": {
              "name": "{TableName}_CL",
              "columns": [
       {
                               "name": "TimeGenerated",
                               "type": "DateTime"
                       }, 
                      {
                               "name": "RawData",
                               "type": "String"
                      }
             ]
       }
   }
}
'@

Invoke-AzRestMethod -Path "/subscriptions/{subscription}/resourcegroups/{resourcegroup}/providers/microsoft.operationalinsights/workspaces/{WorkspaceName}/tables/{TableName}_CL?api-version=2021-12-01-preview" -Method PUT -payload $tableParams

Press return to execute the code. You should see a 200 response, and details about the table you just created will show up. To validate that the table was created go to your workspace and select Tables on the left blade. You should see your table in the list.

Create data collection rule to collect text logs

The data collection rule defines:

Which source log files Azure Monitor Agent scans for new events.
How Azure Monitor transforms events during ingestion.
The destination Log Analytics workspace and table to which Azure Monitor sends the data.

You can define a data collection rule to send data from multiple machines to multiple Log Analytics workspaces, including workspaces in a different region or tenant. Create the data collection rule in the same region as your Log Analytics workspace.

Note

To send data across tenants, you must first enable Azure Lighthouse.

To create the data collection rule in the Azure portal:

On the Monitor menu, select Data Collection Rules.
Select Create to create a new data collection rule and associations.
Enter a Rule name and specify a Subscription, Resource Group, Region, Platform Type, and Data Collection Endpoint:
- Region specifies where the DCR will be created. The virtual machines and their associations can be in any subscription or resource group in the tenant.
- Platform Type specifies the type of resources this rule can apply to. The Custom option allows for both Windows and Linux types.
- Data Collection Endpoint is required to collect custom logs.
On the Resources tab:
1. Select + Add resources and associate resources to the data collection rule. Resources can be virtual machines, Virtual Machine Scale Sets, and Azure Arc for servers. The Azure portal installs Azure Monitor Agent on resources that don't already have it installed.
  
  Important
  
  The portal enables system-assigned managed identity on the target resources, along with existing user-assigned identities, if there are any. For existing applications, unless you specify the user-assigned identity in the request, the machine defaults to using system-assigned identity instead.
  
  If you need network isolation using private links, select existing endpoints from the same region for the respective resources or create a new endpoint.
2. Select Enable Data Collection Endpoints.
3. Select a data collection endpoint for each of the resources associate to the data collection rule.
On the Collect and deliver tab, select Add data source to add a data source and set a destination.
Select Custom Text Logs.
Specify the following information:
- File Pattern - Identifies where the log files are located on the local disk. You can enter multiple file patterns separated by commas.
  
  Examples of valid inputs:
  - 20220122-MyLog.txt
  - ProcessA_MyLog.txt
  - ErrorsOnly_MyLog.txt, WarningOnly_MyLog.txt
  Note
  
  Multiple log files of the same type commonly exist in the same directory. For example, a machine might create a new file every day to prevent the log file from growing too large. To collect log data in this scenario, you can use a file wildcard. Use the format C:\directoryA\directoryB\*MyLog.txt for Windows and /var/*.log for Linux. There is no support for directory wildcards.
- Table name - The name of the destination table you created in your Log Analytics Workspace. For more information, see Prerequisites.
- Record delimiter - Will be used in the future to allow delimiters other than the currently supported end of line (/r/n).
- Transform - Add an ingestion-time transformation or leave as source if you don't need to transform the collected data.
On the Destination tab, add one or more destinations for the data source. You can select multiple destinations of the same or different types. For instance, you can select multiple Log Analytics workspaces, which is also known as multihoming.
Select Review + create to review the details of the data collection rule and association with the set of virtual machines.
Select Create to create the data collection rule.

The data collection rule requires the resource ID of your workspace. Navigate to your workspace in the Log Analytics workspaces menu in the Azure portal. From the Properties page, copy the Resource ID and save it for later use.
In the Azure portal's search box, type in template and then select Deploy a custom template.
Select Build your own template in the editor.

Paste this Resource Manager template into the editor:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "type": "string",
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        },
        "workspaceName": {
            "type": "string",
            "metadata": {
                "description": "Name of the Log Analytics workspace to use."
            }
        },
        "workspaceResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Log Analytics workspace to use."
            }
        },
        "endpointResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Data Collection Endpoint to use."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-09-01-preview",
            "properties": {
                "dataCollectionEndpointId": "[parameters('endpointResourceId')]",
                "streamDeclarations": {
                    "Custom-MyLogFileFormat": {
                        "columns": [
                            {
                                "name": "TimeGenerated",
                                "type": "datetime"
                            },
                            {
                                "name": "RawData",
                                "type": "string"
                            }
                        ]
                    }
                },
                "dataSources": {
                    "logFiles": [
                        {
                            "streams": [
                                "Custom-MyLogFileFormat"
                            ],
                            "filePatterns": [
                                "C:\\JavaLogs\\*.log"
                            ],
                            "format": "text",
                            "settings": {
                                "text": {
                                    "recordStartTimestampFormat": "ISO 8601"
                                }
                            },
                            "name": "myLogFileFormat-Windows"
                        },
                        {
                            "streams": [
                                "Custom-MyLogFileFormat" 
                            ],
                            "filePatterns": [
                                "//var//*.log"
                            ],
                            "format": "text",
                            "settings": {
                                "text": {
                                    "recordStartTimestampFormat": "ISO 8601"
                                }
                            },
                            "name": "myLogFileFormat-Linux"
                        }
                    ]
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaceResourceId')]",
                            "name": "[parameters('workspaceName')]"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-MyLogFileFormat"
                        ],
                        "destinations": [
                            "[parameters('workspaceName')]"
                        ],
                        "transformKql": "source",
                        "outputStream": "Custom-MyTable_CL"
                    }
                ]
            }
        }
    ],
    "outputs": {
        "dataCollectionRuleId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))]"
        }
    }
}

Update the following values in the Resource Manager template:
- streamDeclarations: Defines the columns of the incoming data. This must match the structure of the log file.
- filePatterns: Specifies the location and file pattern of the log files to collect. This defines a separate pattern for Windows and Linux agents.
- transformKql: Specifies a transformation to apply to the incoming data before it's sent to the workspace.
See Structure of a data collection rule in Azure Monitor (preview) if you want to modify the text log DCR.

Important

Custom data collection rules have a suffix of Custom-; for example, Custom-rulename. The Custom-rulename in the stream declaration must match the Custom-rulename name in the Log Analytics workspace.
Select Save.
On the Custom deployment screen, specify a Subscription and Resource group to store the data collection rule and then provide values defined in the template. This includes a Name for the data collection rule and the Workspace Resource ID and Endpoint Resource ID. The Location should be the same location as the workspace. The Region will already be populated and is used for the location of the data collection rule.
Select Review + create and then Create when you review the details.
When the deployment is complete, expand the Deployment details box and select your data collection rule to view its details. Select JSON View.
Change the API version to 2021-09-01-preview.
Copy the Resource ID for the data collection rule. You'll use this in the next step.
Create a data collection association that associates the data collection rule to the agents with the log file to be collected. You can associate the same data collection rule with multiple agents:
1. From the Monitor menu in the Azure portal, select Data Collection Rules and select the rule that you created.
2. Select Resources and then select Add to view the available resources.
3. Select either individual agents to associate the data collection rule, or select a resource group to create an association for all agents in that resource group. Select Apply.

Note

It can take up to 5 minutes for data to be sent to the destinations after you create the data collection rule.

Troubleshoot

Use the following steps to troubleshoot collection of text logs.

Check if any custom logs have been received

Start by checking if any records have been collected for your custom log table by running the following query in Log Analytics. If records aren't returned, check the other sections for possible causes. This query looks for entires in the last two days, but you can modify for another time range. It can take 5-7 minutes for new data from your tables to be uploaded. Only new data will be uploaded any log file last written to prior to the DCR rules being created won't be uploaded.

<YourCustomLog>_CL
| where TimeGenerated > ago(48h)
| order by TimeGenerated desc

Verify that you created a custom table

You must create a custom log table in your Log Analytics workspace before you can send data to it.

Verify that the agent is sending heartbeats successfully

Verify that Azure Monitor agent is communicating properly by running the following query in Log Analytics to check if there are any records in the Heartbeat table.

Heartbeat
| where TimeGenerated > ago(24h)
| where Computer has "<computer name>"
| project TimeGenerated, Category, Version
| order by TimeGenerated desc

Verify that you specified the correct log location in the data collection rule

The data collection rule will have a section similar to the following. The filePatterns element specifies the path to the log file to collect from the agent computer. Check the agent computer to verify that this is correct.

"dataSources": [{
            "configuration": {
                "filePatterns": ["C:\\JavaLogs\\*.log"],
                "format": "text",
                "settings": {
                    "text": {
                        "recordStartTimestampFormat": "yyyy-MM-ddTHH:mm:ssK"
                    }
                }
            },
            "id": "myTabularLogDataSource",
            "kind": "logFile",
            "streams": [{
                    "stream": "Custom-TabularData-ABC"
                }
            ],
            "sendToChannels": ["gigl-dce-00000000000000000000000000000000"]
        }
    ]

This file pattern should correspond to the logs on the agent machine.

Verify that the text logs are being populated

The agent will only collect new content written to the log file being collected. If you're experimenting with the text logs collection feature, you can use the following script to generate sample logs.

# This script writes a new log entry at the specified interval indefinitely.
# Usage:
# .\GenerateCustomLogs.ps1 [interval to sleep]
#
# Press Ctrl+C to terminate script.
#
# Example:
# .\ GenerateCustomLogs.ps1 5

param (
    [Parameter(Mandatory=$true)][int]$sleepSeconds
)

$logFolder = "c:\\JavaLogs"
if (!(Test-Path -Path $logFolder))
{
    mkdir $logFolder
}

$logFileName = "TestLog-$(Get-Date -format yyyyMMddhhmm).log"
do
{
    $count++
    $randomContent = New-Guid
    $logRecord = "$(Get-Date -format s)Z Record number $count with random content $randomContent"
    $logRecord | Out-File "$logFolder\\$logFileName" -Encoding utf8 -Append
    Start-Sleep $sleepSeconds
}
while ($true)

If everything is configured properly, but you're still not collecting log data, use the following procedure to collect diagnostics logs for Azure Monitor agent to share with the Azure Monitor group.

Open an elevated PowerShell window.
Change to directory C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\[version]\.
Execute the script: .\CollectAMALogs.ps1.
Share the AMAFiles.zip file generated on the desktop.

Next steps

Learn more about:

Feedback

Submit and view feedback for

This product This page

View all page feedback

Collect text logs with Azure Monitor Agent

Prerequisites

Create a custom table

Create data collection rule to collect text logs

Troubleshoot

Check if any custom logs have been received

Verify that you created a custom table

Verify that the agent is sending heartbeats successfully

Verify that you specified the correct log location in the data collection rule

Verify that the text logs are being populated

Next steps

Feedback

Additional resources

Additional resources

Collect text logs with Azure Monitor Agent

Prerequisites

Create a custom table

Create data collection rule to collect text logs

Troubleshoot

Check if any custom logs have been received

Verify that you created a custom table

Verify that the agent is sending heartbeats successfully

Verify that you specified the correct log location in the data collection rule

Verify that the text logs are being populated

Share logs with Microsoft

Next steps

Feedback

Additional resources

Additional resources