Gitlab Community Edition Instance

Skip to content

Handle records updates in DACE properly

Continues ingest should work fine in Marketplace. So even if the data will be harvested by DACE twice it should not create duplicates in the MP (thanks to MP's internal mechanisms). However, it does create duplicate records in DACE's database.

In this issue SSHOC related record harvesters should be changed not to create duplicate records (if possible). Duplicates will not be created if DACE will be updating records, and it happens when we upload a record with internalId that already exists for this source in DACE. If possible harvesters should know the internalIds of records and when the record is harvested for the second time it should get the same internalId as for the previous harvest. So probably the internalId should be based on an identifier exposed by the data source.