Timestamp recognition of dates with two-digit years fails beginning January 1, 2020

Problem

Beginning on January 1, 2020, un-patched Splunk platform instances will be unable to recognize timestamps from events where the date contains a two-digit year. This means data that meets this criteria will be indexed with incorrect timestamps.

Beginning on September 13, 2020 at 12:26:39 PM Coordinated Universal Time (UTC), un-patched Splunk platform instances will be unable to recognize timestamps from events with dates that are based on Unix time, due to incorrect parsing of timestamp data.

Impact

This issue affects all un-patched Splunk platform instance types, on any operating system:

• Splunk Cloud
• Splunk Light
• Splunk Enterprise

  • Indexers, clustered or not
  • Heavy forwarders
  • Search heads, clustered or not
  • Search head deployers
  • Deployment servers
  • Cluster masters
  • License masters

• Splunk universal forwarders, under the following known conditions:

  • When they have been configured to process structured data, such as CSV, XML, and JSON files, using the INDEXED_EXTRACTIONS setting in props.conf
  • When they have been configured to process data locally, using the force_local_processing setting in props.conf
  • When they have been configured with a monitor input, and that input subsequently encounters an unknown file type

The issue appears when you have configured the input source to automatically determine timestamps, and can result in one or more of the following problems:

• Incorrect timestamping of incoming data
• Incorrect rollover of data buckets due to the incorrect timestamping
• Incorrect retention of data overall
• Incorrect search results due to data ingested with incorrect timestamps

There is no method to correct the timestamps after the Splunk platform has ingested the data when the problem starts. If you ingest data with an un-patched Splunk platform instance beginning on January 1, 2020, you must patch the instance and re-ingest that data for its timestamps to be correct.

Cause

The Splunk platform input processor uses a file called datetime.xml to help the processor correctly determine timestamps based on incoming data. The file uses regular expressions to extract many different types of dates and timestamps from incoming data.

On un-patched Splunk platform instances, the file supports the extraction of two-digit years of "19", that is, up to December 31, 2019. Beginning on January 1, 2020, these un-patched instances will mistakenly treat incoming data as having an invalid timestamp year, and could either add timestamps using the current year, or misinterpret the date incorrectly and add a timestamp with the misinterpreted date.

Solutions

Splunk Cloud customers will receive the fix automatically. If you are a Splunk Cloud customer, your support representative will advise you when the upgrade will take place. As this is a critical update, there is no option to defer it.

If you have forwarders that send data to your Splunk Cloud instance, perform one of these solutions on those on-premises instances only. Contact your support representative if you have any questions.

There are four solutions to this problem for on-premises customers. You only need to perform one of the solutions to fix the problem:

• Download and deploy an app to temporarily replace the defective datetime.xml with the fixed file
• Download an updated version of datetime.xml and apply it to each of your Splunk platform instances
• Upgrade Splunk platform instances to a version with an updated version of datetime.xml
• Make modifications to the existing datetime.xml file on your Splunk platform instances

Download and deploy an app that temporarily replaces the defective datetime.xml file with the fixed file

Splunk has released a Splunk app that temporarily replaces the defective datetime.xml file with the fixed file. This option is the preferred path for customers who have large deployments including many universal forwarders and indexer and search head clusters.

You do not need to stop the Splunk platform before you deploy the apps, but you must restart each instance that receives the apps.

To ensure that you have no problems with universal forwarders and timestamp recognition, deploy this fix on all universal forwarders, even those that do not specifically suffer from this problem.

This solution is temporary only. Do not run it long-term. Carefully read the instructions that come with the app download and make sure to deploy the correct app to the correct Splunk platform instance type. Do not deploy both apps to a single instance. The preferred method to address this problem is to upgrade to a version that has the fixed datetime.xml file. After you complete upgrades, remove the apps immediately.

1. Download the datetime.xml fix app archive (MD5 hash: 991ac7780b30e448c74a385930db8079) from splunk.com.
2. Unarchive the .tgz file to a location that is accessible from all of your Splunk platform instances.
3. Open the README file and follow the instructions to deploy one of the apps to each Splunk platform instance.

Download an updated version of datetime.xml and apply it to each of your Splunk platform instances

Splunk is providing an updated version of the datetime.xml file for download. This option is the preferred path for customers who cannot upgrade right away to a version of the Splunk platform with the fixed file, or who run an unsupported version that is lower than 6.6.x.

After you download the file, you must apply it directly over the existing datetime.xml file using the following procedure. You must apply the updated file to all affected on-premises Splunk platform instances prior to January 1, 2020, or you will experience timestamp recognition problems from that point forward.

1. Download the datetime.zip timestamp recognition ZIP file from splunk.com.
2. Unarchive the ZIP file to a location that is accessible from all of your Splunk platform instances.
3. On each Splunk platform instance, do the following:

   1. Using your operating system file management utilities, copy the updated datetime.xml from the location where you downloaded it to the $SPLUNK_HOME/etc directory on the Splunk platform instance. Ensure that the updated file overwrites the existing file.
   2. Confirm that the new datetime.xml has been written to the $SPLUNK_HOME/etc directory.
   3. Restart the Splunk platform. Your Splunk platform instance is now patched.

Upgrade Splunk platform instances to a version with an updated version of datetime.xml

Splunk is releasing updated versions of the Splunk platform that contain an updated datetime.xml.

To fix the problem, download and install a version of Splunk Enterprise or Splunk Light that contains the fixed datetime.xml. The following table lists the minor versions that include the fixed file and provide links on how to upgrade Splunk Enterprise and Splunk Light. Apply the fixed version directly over the existing Splunk platform instance to patch it.

Make modifications to the existing datetime.xml file on your Splunk platform instances

If you feel comfortable with making changes to the datetime.xml file directly, you can do so without upgrading Splunk Enterprise or Splunk Light. If you are a Splunk Cloud customer that uses forwarders to send data to your Splunk Cloud instance, you can use this procedure to update those instances.

Use caution when making changes to the $SPLUNK_HOME/etc/datetime.xml file. Outside of fixing this specific issue, never make changes to this file. Typos and incompatible characters in the file can cause direct, lasting negative effects on timestamp recognition and data ingestion. If you are not comfortable with making these changes, consider one of the previous options for fixing this problem, or contact Splunk Support for assistance.

You must complete these changes on all affected Splunk platform instances by January 1, 2020, or you will experience timestamp recognition problems from that point forward.

1. Using either a shell prompt or your operating system file management utilities, go to the $SPLUNK_HOME/etc directory in your Splunk Enterprise installation.
2. Open datetime.xml for editing with a text editor.
3. Search for and replace the following strings, according to this table:

   Search for this string	Replace with this string
   <text><![CDATA[(20\d\d|19\d\d|[901]\d(?!\d))]]></text>	<text><![CDATA[(20\d\d|19\d\d|[9012]\d(?!\d))]]></text>
   <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([901]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>	<text><![CDATA[(?:^|source::).*?(?<!\d|\d\.|-)(?:20)?([9012]\d)(0\d|1[012])([012]\d|3[01])(?!\d|-| {2,})]]></text>
   <text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([901]\d)(?!\d| {2,})]]></text>	<text><![CDATA[(?:^|source::).*?(?<!\d|\d\.)(0\d|1[012])([012]\d|3[01])(?:20)?([9012]\d)(?!\d| {2,})]]></text>
   <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>	<text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[0123456]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>

4. Save the file and close it. If the editor asks if you want to overwrite the file, answer yes.
5. Confirm that the datetime.xml file has been updated.
6. Restart the Splunk platform. Your Splunk platform instance is now patched.

Solutions for customers with large installations

If you have a large on-premises deployment of many Splunk platform instances, contact Professional Services for assistance. If you feel comfortable with deploying Splunk apps, you can also use the app deployment solution, as described in the "Solutions" section.

Monitoring Console errors after an update to the file

If you perform either of the file update or the direct file modification solutions, upon restart, the Monitoring Console will display a message like the following:

File Integrity checks found 1 files that did not match the system-provided manifest. Review the list of problems reported by the InstalledFileHashChecker in splunkd.log File Integrity Check View ; potentially restore files from installation media, change practices to avoid changing files, or work with support to identify the problem.

You can safely disregard this message in these two scenarios only. After you upgrade to a version that has the fixed datetime.xml, this message should no longer appear.

More information

• For additional assistance with this issue, visit the Splunk Support portal.
• If you have a large on-premises deployment of many Splunk platform instances, contact Professional Services for assistance.
• For information on the datetime.xml file, see Configure advanced timestamp recognition with datetime.xml in the Getting Data In Manual.
• For information on how to upgrade Splunk Enterprise, see How to upgrade Splunk Enterprise in the Installation Manual.