If you are running a version listed in the Versions known to be vulnerable column, you can eliminate this vulnerability by installing a version listed in the Fixes introduced in column. If the table lists only an older version than what you are currently running, or does not list a non-vulnerable version, then no update candidate currently exists.
If you are using public cloud marketplaces (AWS, Azure, GCP, and Alibaba) to deploy BIG-IP Virtual Edition (VE), F5 recommends that you install the latest releases of BIG-IP versions listed in the Fixes introduced in column, subject to their availability on those marketplaces. See K84554955: Overview of BIG-IP systems software upgrades.
Mitigation
Important: F5 recommends that you install a fixed software version to fix this vulnerability.
If it is not possible to update quickly, you can use the following sections as temporary configuration mitigations until updating is complete:
- Restrict Access:
- Self IPs: addresses unauthenticated and authenticated attackers on self IPs, by blocking all access
- Management interface: addresses unauthenticated attackers on management interface, by restricting access
- TMUI httpd: addresses unauthenticated attackers on all interfaces
Important: F5 strongly recommends installing fixed versions of the software to address the underlying vulnerability. The risk may be mitigated by restricting access to all TMUI interfaces using the mitigation steps provided below for self IPs and the management interface.
Restrict Access
Self IPs
You can block all access to the Configuration utility of your BIG-IP system using self IPs. To do so, you can change the Port Lockdown setting to Allow None for each self IP in the system. If you must open any ports, you should use the Allow Custom option, taking care to disallow access to the Configuration utility. By default, the Configuration utility listens on TCP port 443; however, beginning in BIG-IP 13.0.0, Single-NIC BIG-IP VE deployments use TCP port 8443. Alternatively, you can configure a custom port.
Note: Performing this action prevents all access to the Configuration utility using the self IP. These changes may also impact other services, including breaking HA configurations.
Before you make changes to the configuration of your self IPs, F5 strongly recommends that you refer to the following articles:
Management interface
To mitigate this vulnerability for affected F5 products, you should permit management access to F5 products only over a secure network. For more information about securing access to BIG-IP systems, refer to K13309: Restricting access to the Configuration utility by source IP address (11.x - 16.x) and K13092: Overview of securing access to the BIG-IP system.
Note: Authenticated users accessing the Configuration utility will always be able to exploit this vulnerability until a fixed release is installed.
TMUI httpd
Important: This section was last updated on July 8, 2020 at 17:00 Pacific time.
The mitigation provided below is based on the information available to F5 at this time. It may not be a complete mitigation. However, F5 feels this information is useful to our customers as it mitigates the unauthenticated exploits currently known to us. As F5 becomes aware of any future variants, we will continue to update this article.
To prevent unauthenticated attackers from exploiting this vulnerability, add a LocationMatch configuration element to httpd. To do so, perform the following procedure:
Note: Authenticated users will still be able to exploit the vulnerability, independent of their privilege level.
Impact of workaround: The following mitigation adds an include statement to the httpd properties. If your httpd properties contain an existing include statement (the include statement is something other than the default of none), you need to prepend/append your existing included configuration to the changes, or it will be overwritten. For example, if your existing include statement is:
include "FileETag MTime Size"
Then you can append the LocationMatch statement in the mitigation to the existing configuration as follows:
include 'FileETag MTime Size
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
'
The mitigation may be performed locally via the command line, or remotely via the iControl REST interface.
Command line
- Log in to the TMOS Shell (tmsh) by entering the following command:
tmsh
- Edit the httpd properties by entering the following command:
edit /sys httpd all-properties
Note: Running this command puts you into the vi editor.
- Locate the line that starts with include none and replace it with the following:
include '
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
'
- Write and save the changes to the configuration file by entering the following vi commands:
Esc
:wq!
- When further prompted to Save Changes (y/n/e), enter y.
- Save the configuration by entering the following tmsh command:
save /sys config
- To activate the mitigation, restart the httpd service by entering the following command:
restart sys service httpd
- To exit tmsh, enter the following command:
quit
- Ensure that the workaround is inserted in the configuration by comparing the output of the following command to the configured LocationMatch fragment that you inserted in step 3. To do so, enter the following command:
grep -C1 'Redirect 404' /etc/httpd/conf/httpd.conf
The output should match the following:
<LocationMatch ";">
Redirect 404 /
</LocationMatch>
<LocationMatch "hsqldb">
Redirect 404 /
</LocationMatch>
Note: You may disregard any leading white spaces.
- If you have a high availability (HA) configuration, you may now perform a ConfigSync operation as documented in K14856: Performing a ConfigSync using tmsh. No restart of httpd on the peer system should be required after syncing. Confirm the mitigation is working using the following instructions.
iControl REST
# Patch the httpd configuration
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/httpd -H content-type:application/json -X PATCH -d '{"include":"\n <LocationMatch \\\";\\\">\n Redirect 404 /\n </LocationMatch>\n <LocationMatch \\\"hsqldb\\\">\n Redirect 404 /\n </LocationMatch>\n "}' | jq .
# Save the system config
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/config -H "Content-Type: application/json" -X POST -d '{"command":"save"}' | jq .
# Verify the configuration
curl -ku admin:[password] https://[IP Address]/mgmt/tm/sys/httpd -H "Content-Type: application/json" -X GET | jq .
If you have an HA configuration, you may also synchronize these changes with the peers:
curl -ku admin:[password] https://[IP Address]/mgmt/tm/cm -H "Content-Type: application/json" -X POST -d '{"command":"run","utilCmdArgs":"config-sync to-group [Device Group Name]"}' | jq .
No restart of httpd on the peer should be required after syncing. Confirm the mitigation is working using the following instructions.
Note: Running REST commands in rapid succession may cause issues. F5 advises that you allow time for completion between commands.
Note: This mitigation previously included a step to restart httpd. It has been determined that the restart is not required when using REST. Furthermore, the restart command may cause issues with httpd. See K13292945: httpd failing to start after restarting the service using the iControl REST API.
Note: F5 is aware of customers attempting to mitigate using tmsh modify sys httpd allow instead of the recommended mitigation above. This has not been an effective mitigation and devices using this have been compromised.
Verifying the mitigation
You can verify that the mitigation is working by visiting the following URLs:
- https://[IP ADDRESS]/tmui/login.jsp/..;/login.jsp
- https://[IP ADDRESS]/hsqldb%0a
Before applying the mitigation, the pages load. After the mitigation, you receive 404 responses.
Indications of compromise
This information is based on the evidence F5 has seen on compromised devices, which we feel are reliable indicators. It is important to note that not all exploited systems may show the same indicators, and indeed a skilled attacker may be able to remove traces of their work. It is not possible to prove a device has not been compromised; when there is any uncertainty it should be considered compromised.
All versions
- Look for the creation of aliases for the Advanced Shell (bash); the presence of an alias is a strong indication of a potential compromise. To do so, run the following command:
awk '/^cli.alias/,/^}/' /config/bigip_*.conf
- Check for user ‘systems’ in /config/bigip_user.conf and /etc/passwd; several exploits have created this non-standard user. To do so, run the following commands:
awk '/systems/' /config/bigip_user.conf
grep -i 'systems' /etc/passwd
- Examine /var/log/audit for common patterns seen with exploits. To do so, run the following command:
zgrep -e "create cli alias" -e "run /util bash /tmp" -e "list auth user admin" -e "_alias" -e "create auth user" -e "load user credentials for user" /var/log/audit*
If this command returns any results, it may be an indication of an attempted compromise. You will need to examine the results to determine if you can account for them due to legitimate activity.
- Check for files created in /usr/local/www/ since the CVE was announced. It is common for exploit scripts to create files in this directory. If any files are found, determine if they can be accounted for from legitimate activity. If not, it is a strong indication of compromise. To do so, run the following command:
touch -t 202006290100 /tmp/kbtime; find /usr/local/www/ -type f -newer /tmp/kbtime -ls;
Additionally, please refer to the following articles:
Versions prior to 14.1.0
In versions earlier than 14.1.0, with the default configuration, you can examine /var/log/audit and /var/log/ltm as follows. There is no supported mechanism to expose additional log entries.
To examine the logs, use the following command:
zgrep '%tmui' /var/log/audit* /var/log/ltm*
Log entries similar to the following are an indication of possible compromise:
audit.1:Jul 6 15:33:38 [REDACTED] notice tmsh[27903]: 01420002:5: AUDIT - Cannot load user credentials for user "%tmui" Current session has been terminated.
ltm.1:Jul 6 15:33:38 [REDACTED] notice tmsh[27903]: 01420003:5: Cannot load user credentials for user "%tmui" Current session has been terminated.
Versions 14.1.0 and later
In version 14.1.0 and later, you can examine the output of journalctl for evidence of attempts to exploit this vulnerability by entering the following command in bash:
journalctl /bin/logger | grep -F ';'
The command output may appear similar to the following example on a device where compromise has been attempted; note that some elements have been redacted (normally the complete URL will be visible along with the IP address that sent the request):
Jul 06 12:59:01 hostname logger[29929]: [ssl_acc] nnn.nnn.nnn.nnn - - [06/Jul/2020:12:59:01 +0000] "/[REDACTED]/..;/[REDACTED]" 200 252
If any log entries are shown, this may be an indication of an attempt to compromise the BIG-IP system. Take specific note of the second to last value in the line, in this case 200; the HTTP Response Code. A 200 indicates that the request was successful, which is a strong indication of a successful exploit. A 404 response code means the requested item was not found. This may be a sign of an attempted compromise, or a scanner being run against the device. You may also see 404 requests logged on devices using mitigation or which are running fixed software. The requests are still being made, but they are unsuccessful.
Note: Journal log entries are rotating and limited to ~20 MB and, therefore, may contain limited historical information.
Note: These log entries will only be created by unauthenticated attacks. Authenticated attackers will not leave this record behind.
Other indicators of compromise may include unexpected modifications to any files, configurations, or running processes. F5 has iHealth heuristics designed to detect unknown processes running (Heuristic H511618) and also heuristics designed to detect when the Configuration utility has been exposed to the Internet through the management interface (H444724), or when a self IP address has Port Lockdown set to Allow All (H458565).
Note: Lack of log entries or heuristic reports do not categorically indicate that a unit has not been compromised. A skilled attacker could remove evidence of compromise, including log files, following successful exploitation.
