Post Mortem

Production issue 3 February 2023

Summary

On 3rd February 2023, we encountered a slowness issue on the platform.

That incident leads to delays during the annotation and review process

Incident Timing (UTC+1)

03/02/23  10:39 am =>  06/02 11:21 Am

Incident Timeline (UTC+1)

03/02/2023

• First alert or first ticket

  • Internally at 10:39 am

• First investigations on long transaction query

• First Announcement

  • Status page incident creation at 9:36 pm

• Status page update time

  • 9:36 pm  Investigation in progress:  We are currently investigating this issue.

04/02/2023

• Status page update time

  • 04/02 03:10 pm  Situation under control: The issue has been identified and a fix is being implemented

06/02/2023

• Status page update time

  • 06/02 11:21 am  Incident resolved: This incident has been resolved.

Actions

4/02/2023

Container crash identified. Finding the root cause.

End-User Impact

Global slowness on the platform could lead to delays during the annotation and review process.

What caused the incident?

A memory leak triggered to slow the container. That’s triggering kills by the orchestrator that triggers to none closed SQL transactions.

Corrective elements put in place to ensure that this does not happen again

We improved the monitoring and alerting

We improved the feature rollout

We decrease the memory consumption for big SQL queries

We start more containers