After a period of monitoring, the Salesforce Technology team confirmed that Core services were restored as of 15:22 UTC. We will perform a full investigation of the incident, confirming the technical trigger, underlying cause, and preventive action to avoid a repeat in the future. We apologize for how you and your business may have been affected by this incident.
Posted 2:03 am JST, Nov 16 2024 · Last updated 2:03 am JST, Nov 16 2024
Incident Resolved
12:22 am JST, Nov 16
Service DisruptionEnded
12:22 am JST, Nov 16
Duration: 9 hours, 27 minutes
Posted 2:03 am JST, Nov 16 2024
Update
The team has rolled back all production instances and is conducting final checks. The team is now focusing on the sandbox environment in groups, which they expect to take considerable time.
We will provide another update in 60 minutes or sooner if new information becomes available.
Posted 11:33 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
The rollback is confirmed to successfully resolve the issue and the team will continue to implement the rollback across the remaining instances. The team has developed a script to perform this rollback on the impacted instances in groups to speed up the resolution.
We will provide another update in 60 minutes or sooner if new information becomes available.
Posted 10:38 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
The team continues to see positive results following the completion of the rollback on a number of instances. The rollback is being done in a staggered manner due to the scale of the issue. As rollbacks complete, affected customers should begin to see improved performance. The rollback efforts are expected to take several hours.
We will provide another update in 90 minutes or sooner if new information becomes available.
Posted 9:07 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
The team successfully rolled back the change on a sandbox instance, where they observed improved database connectivity performance. The team is in the process of rolling back the change on a small number of production instances and will analyze the results.
We’ll provide an update in 30 minutes or sooner if additional information becomes available.
Posted 7:52 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
The team is performing a rollback on a test instance to verify whether the application can establish a stable connection to the database.
We’ll provide an update in 30 minutes or sooner if additional information becomes available.
Posted 7:09 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
The database team has analyzed affected databases and identified some inconsistencies. The databases are now being restored from the most recent stable backup. The Network team has identified and applied a rate limit to regulate and optimize connections. Further investigations have highlighted a recent change as a potential trigger, and as of 09:00 UTC, teams are working to validate a rollback of this in a canary instance in parallel.
We’ll provide an update in 30 minutes or sooner if additional information becomes available.
Posted 6:29 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Update
At 05:55 UTC on November 15, 2024, the Salesforce Technology team became aware of a service disruption impacting instances across multiple Data Centers. At approximately 08:15 UTC, teams began taking steps to control excessive traffic, network utilization, and database stability to mitigate the impact quickly. We don’t yet understand when customers may begin experiencing improvement as a result of these actions.
Initially, this was posted as an Informational Message due to the widespread impact, but as instances were identified, we have moved to post updates here against impacted instances. Previous posts related to this issue can be found at https://status.salesforce.com/generalmessages/1476
We’ll provide an update in 30 minutes or sooner if additional information becomes available.
Posted 5:53 pm JST, Nov 15 2024 · Last updated 2:03 am JST, Nov 16 2024
Service DisruptionBegan
2:55 pm JST, Nov 15
There’s a Salesforce service disruption. During a service disruption, end users can’t access the service. We’re investigating the issue and an update will be provided in 30 minutes or sooner if we have more information.
Posted 5:53 pm JST, Nov 15 2024
We use cookies to make your interactions with our website more meaningful. They help us better understand how our websites are used, so we can tailor content for you. For more information about the different cookies we are using, read the Privacy Statement. To change your cookie settings and preferences, see the Cookie Consent Manager.
Cookie Consent Manager
General Information
General Information
We use three kinds of cookies on our websites: required, functional, and advertising. You can choose whether functional and advertising cookies apply. Click on the different cookie categories to find out more about each category and to change the default settings.
Privacy Statement
Required Cookies
Required Cookies
Always Active
Required cookies are necessary for basic website functionality. Some examples include: session cookies needed to transmit the website, authentication cookies, and security cookies.
Functional Cookies
Functional Cookies
Functional cookies enhance functions, performance, and services on the website. Some examples include: cookies used to analyze site traffic, cookies used for market research, and cookies used to display advertising that is not directed to a particular individual.
Advertising Cookies
Advertising Cookies
Advertising cookies track activity across websites in order to understand a viewer’s interests, and direct them specific marketing. Some examples include: cookies used for remarketing, or interest-based advertising.
