Processing issues with 20220525
Affected
Partial outage from 5:29 PM to 6:59 PM
- Update · UTC
**Impact:** 5 - Very high **Description & Scope of incident** All merchants running gateway software version 20220525 were not able to process payments from 17:53 to 18:58 after applying aversion patch **Analysis of Incident** When applying the version patch for 20220525 it was at first reported as successful by the upgrade tool even though the patching was incomplete. Monitoring showed a processing issue and remedial action was initiated. **Remedial Action** Automatic rollback was attempted but failed Manual correction was initiated, and patching was completed at 18:58 where all services were restored. Mitigation plan: * Implement additional checks in the deployment automation tool to validate changes are persisted according to expectations * Ensure integrity of auxiliary resources when performing maintenance tasks
- Resolved · UTC
A patch has been applied at 18:59, and the issue has been solved. An incident report will be deliver during the next day.
- Monitoring · UTC
A patch has been applied, and the issue should have been resolved. We are monitoring the situation.
- Identified · UTC
There was an issue with the deployment process of a revision for 20220525. A patch is being applied.
- Investigating · UTC
Installations running 20220525 are not reachable at the moment. We are investigating the situation.
