Altapay - Notice history

**Impact:** High - 4 One of 4 nodes partially dropped 3DSecure requests ‌ **Description & Scope of incident:** A change put in place to correct a DNS routing issue affected the connectivity between 1 of the 4 nodes and the connector service related to 3DSecure validation was returning HTTP 403 error. ‌ **Analysis of Incident:** The change put in place to correct the DNS routing issue was silently failing when the required ip configuration of apache2 was blocking it from the new source. In the morning the monitoring showed a high number of declined transactions around 3DSecure for multiple merchants. This was confirmed by Support and Merchants calling in. The changes implemented 27-05-2022 was reverted as part of the rollback procedure and the affected services were restarted.

After extended monitoring we have registered no further issues with increased 3DS declines since the fix was implemented this morning.

After further investigation an issue processing Mastercard between 09:49 - 09:56 has been identified and solved. We are awaiting further details from the acquirer who is investigating the possible cause with Mastercard.

This incident has been resolved.

A fix has been implemented by Nets, and the seems to have been mitigated at 16:00 CET. We are monitoring the situation and waiting for an Incident Report from Nets.

Nets has confirmed the issue and they are working on a solution. At the moment there is no estimation on when this will be fixed. The issue has started at 14:30. We will keep you updated.

**Scope of Incident:** The automatic monitoring system reported lots of failed transactions which indicated a possible payment processing issue to be investigated ‌ **Analysis of incident:** Upon investigation it turned out that a few 3DS pages were sending hundreds of events per second which resulted in degraded performance on the reporting database. In general payments were processed as expected apart from a very limited number of transactions experiencing the issue with 3DS. ‌ Remedial Action: Several corrections were deployed to prevent this from happening again. * Limit number of failed events per transaction * Server side we only write one event to logging per transaction Further actions to improve queue processing logic will be implemented so timeout items will be placed in a separate queue to prevent possible blocking of regular payment processing.

A fix has been implemented and we are monitoring the systems

After further investigation we can see that payments are being processed as expected. The status of the payments are not being updated on the payment gateway so we will continue to investigate to find the underlying issue. Once again we see that payments are being processed as expected.