Navigation Data Service Outage

Incident Report for Nacelle

Resolved

Duration: 3/10 from ~9:00 am - 10:00 am PT (~1 hour)

Issue: Starting at ~9:00 am PT on 3/10, some merchant builds failed due to 500 responses when fetching navigation data from the Storefront API. The navigation data service had encountered scaling issues that resulted in a lack of enough replicas being available for the volume of requests.

Solution: By ~10:00 am PT, existing self-healing services had allowed navigation data service to stabilize and handle all incoming traffic. As a long-term solution, the number of minimum replicas enabled for the navigation data service has been doubled, and additional monitoring alerts have been implemented for this service.

Posted Mar 10, 2024 - 16:00 UTC