SaaS instances experiencing intermittent 400/500 errors

Incident Report for Vasion

Postmortem

Wednesday, June 25, 2025

SaaS instances experiencing intermittent 400/500 errors

Issue Summary:

On 25 June 2025 between 9:38 PM UTC and 10:33 PM UTC some users with instances hosted in the US region experienced intermittent issues accessing the platform.

Root Cause: 

A database node restarted unexpectedly and experienced high CPU and connection utilization once it came back online.

Resolution: 

Data teams were able to identify the source of the high utilization and took action to resolve.

Mitigation:

To address high system utilization following unexpected database restarts, the Data team will promptly investigate and identify root causes, taking immediate corrective action to restore normal operations. Additionally, the team will validate all relevant system data and confirm the health of the database to ensure data integrity and system stability.

Conclusion:

We acknowledge this had an impact on some customers, we thank you for your patience as we resolved this service disruption.

Posted Jun 27, 2025 - 19:28 UTC

Resolved

This incident has been resolved.
Posted Jun 25, 2025 - 23:47 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 25, 2025 - 22:53 UTC

Investigating

We are currently investigating this issue.
Posted Jun 25, 2025 - 22:17 UTC
This incident affected: Scheduled Release - US Region (Administrative Console).