If you are running a version listed in the Versions known to be vulnerable column, you can eliminate this vulnerability by installing a version listed in the Fixes introduced in column. If the table lists only an older version than what you are currently running, or does not list a non-vulnerable version, then no update candidate currently exists.
If you are using public cloud marketplaces (AWS, Azure, GCP, and Alibaba) to deploy BIG-IP Virtual Edition (VE), F5 recommends that you install the latest releases of BIG-IP versions listed in the Fixes introduced in column, subject to their availability on those marketplaces. See K84554955: Overview of BIG-IP systems software upgrades.
Mitigation
Important: F5 recommends that you install a fixed software version to fix this vulnerability.
If it is not possible to update quickly, you can use the following sections as temporary configuration mitigations until updating is complete:
- Restrict Access:
- Self IPs: addresses unauthenticated and authenticated attackers on self IPs, by blocking all access
- Management interface: addresses unauthenticated attackers on management interface, by restricting access
- TMUI httpd: addresses unauthenticated attackers on all interfaces
Important: F5 strongly recommends installing fixed versions of the software to address the underlying vulnerability. The risk may be mitigated by restricting access to all TMUI interfaces using the following mitigation steps provided for self IPs and the management interface.
Restrict Access
Self IPs
You can block all access to the Configuration utility of your BIG-IP system using self IPs. To do so, you can change the Port Lockdown setting to Allow None for each self IP in the system. If you must open any ports, you should use the Allow Custom option, taking care to disallow access to the Configuration utility. By default, the Configuration utility listens on TCP port 443; however, beginning in BIG-IP 13.0.0, Single-NIC BIG-IP VE deployments use TCP port 8443. Alternatively, you can configure a custom port.
Note: Performing this action prevents all access to the Configuration utility using the self IP. These changes may also impact other services, including breaking HA configurations.
Before you make changes to the configuration of your self IPs, F5 strongly recommends that you refer to the following articles:
Management interface
To mitigate this vulnerability for affected F5 products, you should permit management access to F5 products only over a secure network. For more information about securing access to BIG-IP systems, refer to K13309: Restricting access to the Configuration utility by source IP address (11.x - 16.x) and K13092: Overview of securing access to the BIG-IP system.
Note: Until a fixed release is installed, authenticated users accessing the Configuration utility will always be able to exploit this vulnerability.
TMUI httpd
Important: This section was last updated on July 8, 2020 at 17:00 Pacific time.
The mitigation provided below is based on the information available to F5 at this time. It may not be a complete mitigation. However, F5 feels this information is useful to our customers as it mitigates the unauthenticated exploits currently known to us. As F5 becomes aware of any future variants, we will continue to update this article.
To prevent unauthenticated attackers from exploiting this vulnerability, add a LocationMatch configuration element to httpd. To do so, perform the following procedure:
Note: Authenticated users will still be able to exploit the vulnerability, independent of their privilege level.
Impact of workaround: The following mitigation adds an include statement to the httpd properties. If your httpd properties contain an existing include statement (the include statement is something other than the default of none), you need to prepend/append your existing included configuration to the changes, or it will be overwritten. For example, if your existing include statement is:
include "FileETag MTime Size"
Then you can append the LocationMatch statement in the mitigation to the existing configuration as follows:
include 'FileETag MTime Size
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
'
You can perform the mitigation locally using the command line or remotely using the iControl REST interface.
Command line
- Log in to the TMOS Shell (tmsh) by entering the following command:
tmsh
- Edit the httpd properties by entering the following command:
edit /sys httpd all-properties
Note: Running this command puts you into the vi editor.
- Locate the line that starts with include none and replace it with the following:
include '
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
'
- Write and save the changes to the configuration file by entering the following vi commands:
Esc
:wq!
- When further prompted to Save Changes (y/n/e), enter y.
- Save the configuration by entering the following tmsh command:
save /sys config
- To activate the mitigation, restart the httpd service by entering the following command:
restart sys service httpd
- To exit tmsh, enter the following command:
quit
- Ensure that the workaround is inserted in the configuration by comparing the output of the following command to the configured LocationMatch fragment that you inserted in step 3. To do so, enter the following command:
grep -C1 'Redirect 404' /etc/httpd/conf/httpd.conf
The output should match the following:
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
Note: You may disregard any leading white spaces.
- If you have a high availability (HA) configuration, you may now perform a ConfigSync operation as documented in K14856: Performing a ConfigSync using tmsh. No restart of httpd on the peer system should be required after syncing. Confirm the mitigation is working using the following instructions.
iControl REST
# Patch the httpd configuration
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/httpd -H content-type:application/json -X PATCH -d '{"include":"\n <LocationMatch \\\";\\\">\n Redirect 404 /\n </LocationMatch>\n <LocationMatch \\\"hsqldb\\\">\n Redirect 404 /\n </LocationMatch>\n "}' | jq .
# Save the system config
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/config -H "Content-Type: application/json" -X POST -d '{"command":"save"}' | jq .
# Verify the configuration
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/httpd -H "Content-Type: application/json" -X GET | jq .
If you have an HA configuration, you may also synchronize these changes with the peers:
curl -ku admin:[password] https://[IP Address]/mgmt/tm/cm -H "Content-Type: application/json" -X POST -d '{"command":"run","utilCmdArgs":"config-sync to-group [Device Group Name]"}' | jq .
No restart of httpd on the peer should be required after syncing. Confirm the mitigation is working using the following instructions.
Note: Running REST commands in rapid succession may cause issues. F5 advises that you allow time for completion between commands.
Note: This mitigation previously included a step to restart httpd. It has been determined that the restart is not required when using REST. Furthermore, the restart command may cause issues with httpd. See K13292945: httpd failing to start after restarting the service using the iControl REST API.
Note: F5 is aware of customers attempting to mitigate using tmsh modify sys httpd allow instead of the recommended mitigation above. This has not been an effective mitigation and devices using this have been compromised.
Verifying the mitigation
You can verify that the mitigation is working by using the following URL syntax:
- https://[IP ADDRESS]/tmui/login.jsp/..;/login.jsp
- https://[IP ADDRESS]/hsqldb%0a
Before applying the mitigation, the pages load. After the mitigation, you receive 404 responses.
Indicators of compromise
Important: This section was last updated on July 20, 2020 at 20:00 Pacific time.
This information is based on the evidence F5 has seen on compromised devices, which we feel are reliable indicators. It is important to note that not all exploited systems may show the same indicators, and, indeed, a skilled attacker may be able to remove traces of their work. It is not possible to prove a device has not been compromised; when there is any uncertainty, you should consider the device compromised.
All versions
F5 iHealth is updated with heuristics which flag indicators of compromise in uploaded QKView diagnostic files. Refer to K27404821: Using F5 iHealth to diagnose vulnerabilities for more information on using F5 iHealth. Through the DevCentral GitHub, F5 has also released the CVE-2020-5902 IoC Detection Tool. This is a Python script designed to run on the command line to locally identify indicators of compromise.
If you are unable to use the CVE-2020-5902 IoC Detection Tool, you may also perform some of the following checks manually:
- Look for the creation of aliases for the Advanced Shell (bash); the presence of an alias is a strong indicator of a potential compromise. To do so, run the following command:
awk '/^cli.alias/,/^}/' /config/bigip_*.conf
You may observe results similar to the following:
cli alias private list {
command bash
user root
}
If you see results similar to this, it is a possible indicator of compromise. You should determine if the result is legitimate for your configuration.
- Check for user ‘systems’ in /config/bigip_user.conf and /etc/passwd; several exploits have created this non-standard user. To do so, run the following commands:
awk '/systems/' /config/bigip_user.conf
grep -i 'systems' /etc/passwd
- Examine /var/log/audit for common patterns seen with exploits. To do so, run the following command:
zgrep -e "create cli alias" -e "run /util bash /tmp" -e "list auth user admin" -e "_alias" -e "create auth user" -e "load user credentials for user" /var/log/audit*
You may see a result similar to the following:
Jul 14 12:59:57 [REDACTED] notice tmsh[13316]: 01420002:5: AUDIT - pid=13316 user=root folder=/Common module=(tmos)# status=[Command OK] cmd_data=create cli alias private list command bash
If this command returns a similar result, it may be a possible indicator of compromise. You need to examine the results to determine if you can account for them due to legitimate activity.
- Check for files created in /usr/local/www/ since the CVE announcement. It is common for exploit scripts to create files in this directory. If you find any files, determine if they can be accounted for from legitimate activity. If not, this is a strong indicator of compromise. To check files, run the following command:
touch -t 202006290100 /tmp/kbtime; find /usr/local/www/ -type f -newer /tmp/kbtime -ls;
Additionally, refer to the following articles:
Versions prior to BIG-IP 14.1.0
In versions earlier than BIG-IP 14.1.0, with the default configuration, you can examine /var/log/audit and /var/log/ltm as follows. There is no supported mechanism to expose additional log entries.
To examine the logs, use the following command:
zgrep '%tmui' /var/log/audit* /var/log/ltm*
Log entries similar to the following are indicators of possible compromise:
audit.1:Jul 6 15:33:38 [REDACTED] notice tmsh[27903]: 01420002:5: AUDIT - Cannot load user credentials for user "%tmui" Current session has been terminated.
ltm.1:Jul 6 15:33:38 [REDACTED] notice tmsh[27903]: 01420003:5: Cannot load user credentials for user "%tmui" Current session has been terminated.
BIG-IP 14.1.0 and later
In BIG-IP 14.1.0 and later, you can examine the output of journalctl for evidence of attempts to exploit this vulnerability by entering the following command in bash:
journalctl /bin/logger | grep -F ';'
The command output may appear similar to the following example on a device where compromise was attempted; note that some elements are redacted (normally the complete URL is visible, along with the IP address that sent the request):
Jul 06 12:59:01 hostname logger[29929]: [ssl_acc] nnn.nnn.nnn.nnn - - [06/Jul/2020:12:59:01 +0000] "/[REDACTED]/..;/[REDACTED]" 200 252
If any log entries are shown, this may be an indicator of an attempt to compromise the BIG-IP system. Take specific note of the second to last value in the line, in this case 200; the HTTP Response Code. A 200 indicates that the request was successful, which is a strong indicator of a successful exploit. A 404 response code means the requested item was not found. This may be a sign of an attempted compromise or a scanner being run against the device. You may also see 404 requests logged on devices using mitigation or which are running fixed software. The requests are still being made, but they are unsuccessful.
Note: Journal log entries are rotating and limited to ~20 MB and therefore may contain limited historical information.
Note: These log entries are only created by unauthenticated attacks. Authenticated attackers do not leave this record behind.
Other indicators of compromise may include unexpected modifications to any files, configurations, or running processes. F5 has iHealth heuristics designed to detect unknown processes running (Heuristic H511618) and also heuristics designed to detect when the Configuration utility has been exposed to the Internet through the management interface (H444724) or when a self IP address has Port Lockdown set to Allow All (H458565).
Note: Lack of log entries or heuristic reports do not categorically indicate that a unit has not been compromised. A skilled attacker can remove evidence of compromise, including log files, following successful exploitation.
