Unable to login to customer instances from the MSP portal

Incident Report for Vasion

Postmortem

MSP portal not showing login option for instances

Issue Summary:

On 06 Mar 2025 05:02UTC, a change to MSP Portal to improve error handling was tested and pushed into Production.  Users were still able to log into the MSP Portal and Print Admin portals, but, as a result of this change, the functionality in MSP Portal that allows direct access to the Print Admin console was inadvertently disabled.  This degradation required MSP Admins to directly log into Print Admin consoles instead of using the direct access button in MSP Portal.  The Admin console direct access capability is heavily used by MSP Admins of Vasion products and led to many users being unable to access Print Admin consoles in the usual way.

A fix to restore the direct access button for the Print Admin console in MSP Portal was pushed to Production at 2025-03-06 18:02UTC.

Root Cause:

The initial change to correct the undesired error condition had the side effect of hiding the user interface elements that allowed direct access to Print Admin console.  Manual testing failed to discover the missing UI elements and led to the regression in capability.

Resolution:

Update the MSP Portal code to ensure the direct access capability was available for customers to use.  In addition, more automated tests were added to catch future regression of this capability before these issues reach Production.

Conclusion:

To prevent further degradations of MSP Portal functionality, more automated tests were added to testing suites to confirm that the direct access button is present and working properly before allowing new changes in MSP Portal to reach Production.

Posted May 14, 2025 - 21:09 UTC

Resolved

Unable to login to customer instances from the MSP portal
Posted Mar 06, 2025 - 16:00 UTC