Resolved -
All services have been fully restored at this time. Thank you for your patience and support while we worked to resolve the affected services.
Feb 7, 13:36 MST
Update -
We are continuing to monitor Order Services while they recover and we will update again as all services are fully restored and backlogged events have processed.
Feb 7, 12:57 MST
Update -
Order services are recovering and clients should be able to save, load, and update orders without issue. We are still seeing some impact to orders coming in from integration partners and we are monitoring until the issue is fully resolved. We will continue to update until all services are fully restored.
Feb 7, 11:44 MST
Monitoring -
The Cova Team has worked around the Redis Database issue, and the services that were slow to respond should be back up and running: including catalogs, reporting, pricing, and cash management. Orders on line and in store are still currently impacted and our teams are working on a resolution while Azure is still down. We will continue to update as progress is made to return the Orders service to a healthy state.
Feb 7, 10:59 MST
Identified -
We are continuing to see a service disruption across multiple Cova POS and HUB services as a result of impacted Redis databases hosted in Microsoft Azure. We are attempting to mitigate the disruption by moving traffic to a different data center after standing up a new Redis database. We will provide updates as our teams work to bypass the impacted Azure services.
Feb 7, 10:01 MST
Update -
We are continuing to monitor this issue with both Azure and Redis teams are working to resolve the outage and we will continue to provide updates as those platforms become healthy again.
Feb 7, 09:26 MST
Update -
We are currently seeing a service disruption across multiple Cova POS and HUB services as a result of impacted Redis databases hosted in Microsoft Azure. Azure is experiencing a wider outage, which is the root cause of the Redis database issue and subsequent Cova service disruptions. Both the Azure and Redis teams are working to resolve the outage and we will continue to provide updates as those platforms become healthy again.
Feb 7, 09:16 MST
Investigating -
We are currently aware of an issue delaying orders arriving in the app or populating in hub reporting. This is also affecting Cash Management, and some pricing changes being correctly reflected within the App. We are actively working to mitigate the issue and will provide more information as it becomes available.
Feb 7, 09:02 MST