Prevent duplicates

[rimi-itk]

Link to ticket

https://leantime.itkdev.dk/_#/tickets/showTicket/7131

Description

When importing systems and reports, we currently duplicate inactive items. This is due to EntityActiveFilter filtering out all inactive entities and thus the importer does not detect if an inactive items already exists (and creates a duplicate).

We resolve this by

• Disabling the doctrine filter in data importers

As added bonusses we also clean up a bit by

• Nullifying empty dates; they're currently set to “now”
• Cleaning up

---

We need to clean up the current data, and one way to do this (and probably also the best and most correct) is to delete all but the first row of any group of duplicated rows (sharing a sys_internal_id). This can be done by observing that the first row in a group is the one with the lowest ID (AUTO_INCREMENT actually increments values!), and then using the following SQL incantations to clean up:

DELETE t1
FROM report t1 INNER JOIN report t2
WHERE t1.sys_internal_id = t2.sys_internal_id AND t1.id > t2.id;

DELETE t1
FROM system t1 INNER JOIN system t2
WHERE t1.sys_internal_id = t2.sys_internal_id AND t1.id > t2.id;

Shout out to https://stackoverflow.com/a/79695983 for coming up with this!