Limitations of Custom Integration Filters?

Get Help
1

Question: In my Custom Integration, if I make a CI filter to restrict the entities that my connector app sends to Fibery for sync, will newly synced Fibery entities get correctly linked to pre-existing entities?

Explanation:
What I need to accomplish is to sync all entities not in one operation, but gradually in a number of discrete “waves” over some time, due to API limits imposed by the data source (Zoho).

I have discovered that if I limit the Fibery sync to certain DBs only, then the synced entities will not get linked to any pre-existing entities that were previously created (apparently if you exclude some Types/DBs from the sync configuration, relations fields to those DBs are ignored) :cry:

So I cannot sync in my entities one DB at a time by changing the sync configuration – all DBs must be part of every sync.

My plan is to implement a CI filter for a “Date Created” range, so that I can first sync (all) the oldest entities in one “wave”, then sync in the next-most-recently-created wave, etc. But this will only work if the entities will get correctly linked to previously created/synced entities.

Does this sound plausible?

2

Hi, @Matt_Blais

apparently if you exclude some Types/DBs from the sync configuration, relations fields to those DBs are ignored

That’s true. Could you please clarify if you are using data paging in your connector? Almost all limitations can be resolved with a paging. Also it sounds natural solution since data source Zoho has limitations and data can be retrieved from source using paging and retry technic.

In case if paging is not possible to use for some reasons the way you described may be considered.

  1. Create some filter for source, it can be Date Created for your case
  2. Configure for filter for first portion of data
  3. Turn on checkbox don’t delete data
    image
    image1222×184 15.4 KB
  4. Sync
  5. Change filter for next wave
  6. Sync
    …
    Repeat 5-6 until all data synced without switching off any sync databases to have linking

Thanks,
Oleg

2 Likes
3

@Oleg, thanks for verifying – I will try that.

Yes I have implemented paging in my connector on both Zoho side and Fibery side.

Also I do have the capability to store the full response body of every packet returned to Fibery, in case we need that for future debugging.

1 Like