We are very sorry for the slowness that you experienced in production this morning.

This was due to a sudden increase in users activity.

Please find below the actions taken to prevent such events in the future.

Explanation

The button “Start labeling“ was not available because the underlying SQL query preparing the labeling queue was too slow.

This was due to a bug in the management of the user session.

Remediation plan

Short term

We also increased the memory available for the backend to ease horizontal scalability.

We solved it by cleaning this table, dividing by 40 the response time.

Long term

Further actions are taken in the coming weeks:

• Improve the asset servicing (new service + signed urls)
• Increase the size of the DB
• Decommission the internal scheduler service