I’m currently working on a Fibery integration via the Integrations API with DatoCMS. I am pretty close to dynamically supporting most of the fields and relationships, rather than hard-coding a schema mapping.
Questions:
Protecting Data:
We will be syncing in types of content from the CMS, then will be adding additional data, mostly to help with prioritizing content production tasks. While working on the integration, I get the feeling that it could be easy to lose data. For example, if we have a bug in the integration, i’ve seen cases where the syncing process errors out and all records disappear. I assume that something like this would lead to potentially losing any extra fields added to the model?
It feels like if I was manually building this syncing behavior in something like n8n, I’d feel a little more comfortable knowing exactly the cases where a record could potentially be deleted. Is there any insight into this behavior?
Webhook/Data Sync:
When the data from a webhook is sent to the /transform endpoint, is there any way to know which fields have actually been configured on the model to sync? Sometimes I’m having to fetch related objects to the updated record and it seems like it could be wasteful without knowing which fields need to be returned
It seems like ideally we’d be provided with the list of fields that need to be synced as input to the webhook endpoints so the integration could be stateless and independent of a particular configuration
File Upload:
I looked at the notion example, which references files, but haven’t been able to get anything to work.
The schema I return for 1 or more images associated with a record is image: { type: "array[text]", subType: "files", name: "Image" },, then when the data is synced, I provide an array of urls pointing to the images. It seems like the files aspect of the integration api isn’t documented enough to really understand how to use it. It would be really great if we could have a field that is a gallery of 1 or more images that are referenced externally by url, rather than having to be uploaded as files.
When I do return the subType of “files” the list of fields for the model that are suggested uses a Text field for the images field I’m mapping to it. This seems a little odd, given there is a fibery field type called “Files”, that I thought would be selected. Either way, it doesn’t work for me. As a text field, it ends up just being my list of image urls as basic text. As a Files field type, the field is always empty.
i’ve seen cases where the syncing process errors out
Hm. this should not be a case. If you respond with 4xx,5xx then we won’t start sync and won’t delete previous data. But the problem can indeed arise if you respond with 200, it’s a full sync mode and “Do not delete Fibery entities…” is turned off
. If you respond with 500 and data gets deleted then it’s a bug on our side
the cases where a record could potentially be deleted. Is there any insight into this behavior
As I mentioned above entity should be deleted only when “Do not delete Fibery entities…” is turned off (which is default) and either full sync is happening OR the returned entity has __syncAction: 'REMOVE' field.
When the data from a webhook is sent to the /transform endpoint, is there any way to know which fields have actually been configured on the model to sync?
Well, I believe it can be done on Connector’s side. /transform endpoint receives account and filter, so you can easily re-construct schema for it and see what fields it have. Actually we are doing something similar in Intercom/Hubspot connectors. Probably you’ll want to add a cache for schema construction to not hit rate limits of 3rd party system
File Upload
Just to make sure: As I can see it should be subType: "file". Singular form is used. Does it resolve your issue?
Thanks for the response about deleting data, I’ll do some more testing.
I am supporting a sync between a flexible type system, much like fibery, and one where there are numerous fields that could be synced, but we might not sync all. So, while I can rebuild the schema in the transform endpoint for the entire model I could sync (which is what I’m doing), I didn’t see where I could know what the schema is that the user actually chose to sync in fibery.
Not the worst case scenario, but just thought it would be nice to have in that transform endpoint.
I’ll try, but don’t think that matches the fibery docs. Looking again at the notion example, it does use the singular form. Generally the issue I had here was not being sure what was expected to be passed. Is it we have to upload the binary file, or pass in the url, etc.
Then, what happens on the sync. If we return more files (assuming by url) do you replace what exists, or only replace what exists if not matched by url, etc?
what the schema is that the user actually chose to sync in fibery
Ah, I see. Yeah, that makes sense. Same can be applied for synchronizer/data endpoint as well. But for now passing excessive data shouldn’t make any evil.
what a shame . Will fix it, thanks for spotting it )
what happens on the sync.
Basically we do a diff between old and new file urls. URLs that are present in both sets are kept untouched, new ones get downloaded, old ones get deleted.
This did indeed solve my issue. Changing to “file” correctly applied the field type in fibery when setting up the sync. Appreciate the help.
If you do update the docs, it would be helpful to see what the valid mapping is between the type and subtype, if that doesn’t exist somewhere else. Without that notion example I’m not sure i would have been able to figure out what the type should be. Right now they are outlined as two independent fields.
I know, integrations API has its quirks. Unfortunately custom integrations aren’t our top priority right now that’s why docs are underloved (
Can’t give you any estimates when the situation will be changed. Meanwhile feel free to ask any question and I’ll be glad to help you with your integration
. Will fix it, thanks for spotting it )