Altapay - Processing issues with 20220525 – Incident details

Processing issues with 20220525

Resolved
Partial outage
Started over 2 years agoLasted about 2 hours

Affected

Processing

Operational from 5:29 PM to 5:29 PM, Partial outage from 5:29 PM to 6:59 PM

Updates
  • Update
    Update

    **Impact:** 5 - Very high ‌ **Description & Scope of incident** All merchants running gateway software version 20220525 were not able to process payments from 17:53 to 18:58 after applying aversion patch ‌ **Analysis of Incident** When applying the version patch for 20220525 it was at first reported as successful by the upgrade tool even though the patching was incomplete. Monitoring showed a processing issue and remedial action was initiated. ‌ **Remedial Action** Automatic rollback was attempted but failed Manual correction was initiated, and patching was completed at 18:58 where all services were restored. Mitigation plan: * Implement additional checks in the deployment automation tool to validate changes are persisted according to expectations * Ensure integrity of auxiliary resources when performing maintenance tasks

  • Resolved
    Resolved

    A patch has been applied at 18:59, and the issue has been solved. An incident report will be deliver during the next day.

  • Monitoring
    Monitoring

    A patch has been applied, and the issue should have been resolved. We are monitoring the situation.

  • Identified
    Identified

    There was an issue with the deployment process of a revision for 20220525. A patch is being applied.

  • Investigating
    Investigating

    Installations running 20220525 are not reachable at the moment. We are investigating the situation.