Some customers may have also had trouble with loading channels, adding users to group direct messages, API hyperlink rendering, and missing Saved items.Ī mass de-provisioning event inadvertently increased load across our infrastructure, causing issues with Threads for many users.Īs an immediate mitigation strategy, we temporarily deactivated the Threads view for all users. PDT, a number of customers had trouble loading and receiving thread notifications.

If you'd like to receive a full Root Cause Analysis (RCA) report, please reach out to to request one.On Octofrom 3:00 a.m. We continued to closely monitor the situation until we had full confidence the issue was resolved and no further impact would reoccur. By 9:14 AM PST, everyone should have been able to connect to Slack again after reloading their app or browser tab. This allowed the system to recover while steadily restoring full access to all users. With traffic distributed across the additional databases, we gradually lifted the rate limits to allow more people to connect to Slack. We reinstated the rate limits and redirected some traffic to the database replicas to relieve the demand on our primary databases.

However, we moved too quickly and the increased activity affected the system again. Once the system had stabilized, we began lifting these rate limits to enable more connections to Slack. This meant that some people could not access Slack at all, but also that Slack would continue working for those who were already connected. We introduced tighter rate limits on connection requests to reduce the load on the system. Due to this increased activity, the affected databases failed to serve incoming requests to connect to Slack. On Februfrom 6:00 AM PST to 9:14 AM PST, customers may not have been able to access Slack.Ī configuration change inadvertently lead to a sudden increase in activity on our database infrastructure.

Slack status update