Form Improvement: Duplicate Check

Data hygiene is becoming more and more of a priority within my company’s Fibery workspace.

Specifying a specific field for a duplication check within that form’s database would be extremely helpful.

For example: add new contact form with the email as the duplication check. If that email exists within the database already, the form sends an error when trying to submit

@Chr1sG’s Deduplication template is awesome, but in situations like this is a workaround rather than stopping the duplication before it happens.

Hey @ccollins I ran into this exact problem the other day and was able to write an automation using info from @Chr1sG 's post and this post from @Matt_Blais .

Automation:

  1. The automation triggers for new entities created from a form. I put a hidden checkbox on all of my forms to indicate this.
  2. The script checks a single text field (‘SKU’ in my case) to see if there is more than one copy in the DB.
  3. If there is, it deletes the new entry and throws an error.

There are a few downsides to this approach.

  • Creating → Checking → Deleting is probably not the most efficient way to stop duplicates. I would be better to stop entity creation altogether
  • The only way to indicate to the user that a duplicate has been created is to throw an error. Unfortunately if an automation errors more than 3 times it is disabled and has to be manually restarted from the automation page. This feature could help the situation.

Here is a screenshot and the code snippet:

const fibery = context.getService('fibery')
const entity_type = args.currentEntities[0].type
// Replace 'Data' with the appropriate space name
const SPACE_NAME = 'Data'

for (const entity of args.currentEntities) {
    // Replace 'findMaterials' with the appropriate GraphQL command for the DB
    // Replace 'sku:' and '.sku" with the appropriate text field name 
    const query = `{findMaterials(sku:{contains: "${entity.sku}"}){id}}`
    const result = await fibery.graphql(SPACE_NAME, query)
    const ids = result.data.findMaterials.length
    if (ids > 1) {
        await fibery.deleteEntity(entity.type, entity.id);
        throw new Error("This material already exist, please check database")
    }
}
2 Likes

I would also love some suggestions on this approach if people have better ideas. TBH I’m still learning how to use Fibery scripts.

I do not believe that there is any way to perform a duplication check that does not involve comparing a created entity with entities already in the database, and then making a decision.
From a technical perspective, Fibery can’t currently know what the values of the entity’s fields are before it has been created.

Theoretically, a form view feature could be developed to perform a field-level search at the time the user is entering data, and then object to a duplicate being entered (in the same way as it can object to empty field values) but I suspect this may be quite technically challenging.
Furthermore, any duplication check would effectively serve as a backdoor method for someone to brute-force query the database - not all users would want people who are filling forms to know whether an entity with a specific value existed :thinking:

Overall, I do not expect this problem will be addressed in the short- to medium-term to be honest.

With that said, I suppose the real question is why is duplication-detection-and-deletion objectionable? Presumably because there is currently no nice way to provide immediate feedback to the user that the entity they believe they are creating is actually going to evaporate a fraction of a second later.

If nothing else, you could always add to the deletion automation a notification to the entity creator that their entity was rejected.
(but won’t work for anonymous submissions)

2 Likes

I hadn’t thought about how the pre-creation check would have to be implemented, I definitely see how there could be problems. I will look into adding creator notifications, thanks @ChrisG !

This is incredibly helpful! Thanks Tommy!

1 Like

Is there a way that this automation could be executed on a formula field created after form submission? My example: have “first name” and “last name” text fields in a database and upon entity creation, the formula “name” is filled in with a concatenation of first name and last name. I’d like to check for duplicates on “name” formula field.

You can trigger an automation when the formula field is updated (irrespective of the source of the data) which checks for duplication.
We have a deduplication template that you can refer to for inspiration.