Issue Summary:
On May 21, 2025, a critical issue was identified where IdP groups and users were not syncing in Quota Management (QM). When selecting an IdP group in QM, the expected user list was returned empty, displaying no users to the end-user. Initially considered an instance-level database issue, it was immediately established as a Critical customer bug. After determining the affected regions, it was elevated to Blocking status on May 21, 2025, at 2:43 PM MT.
Root Cause:
The issue originated from a change implemented in Production on May 2, 2025, which affected the QM endpoint. The endpoint received an incorrect parameter, causing the failure to retrieve and display user lists.
Resolution:
Vasion’s Engineering teams resolved the issue by adjusting the calling procedures to pass the correct UUIDs instead of INTs. The fix was validated on May 22, 2025. It was then deployed to Production and validated on May 23, 2025.
Mitigation:
To prevent similar issues, QM testing will be included in future changes with Identity, given QM’s dependency on Identity. Each dependent service owner, including QM, will create and provide API tests for the Identity team to run whenever changes are made to Identity endpoint inputs or outputs.
Conclusion:
The issue was successfully resolved by correcting the parameter type passed to the endpoint, with the fix validated in Production. To enhance system reliability, integrating QM-specific API tests into the Vasion’s Engineering teams change process will help detect potential issues early, ensuring smoother operation of dependent services.