🧮 Feature Suggestion: Query from external database and create types and entities

Had an interesting thought.

How possible would it be to query records from an external database and create records from it, live?

Example:

GET https://api.someexampleapp.com/data

Then… Create records from GET
Also… Update every X minutes or on manual sync

I know, we can technically use Fibery API to push updates from our own database, but it would be awesome to be able to do from Fibery and allow the returned data to be transformed into types, entities, etc.

We use tools like “Retool” to make internal tools that need real-time data syncing/pulling… but honestly, prefer Fibery for this. Just the lack of ability to pull data natively makes it more cumbersome.

Is this something that Fibery or other users would find useful?

Cheers

Is what you’re after not already possible? I.e. using the Fibery API to make GET (and POST) calls to http services…
https://api.fibery.io/#script

As said above:

I know, we can technically use Fibery API to push updates from our own database, but it would be awesome to be able to do from Fibery and allow the returned data to be transformed into types, entities, etc.

As you described, using “GET” calls only work when using action buttons. It also does not incorporate seamless record deletion or creation (like, if a record from the external database is deleted, we then have to set up a deletion flow separately to remove it from Fibery… rather than Fibery simply pulling “only” what is available in the external database to begin with).

The idea would be to GET data natively in the form of a Custom Integration (external REST API) and transform the response (JSON) into types and/or entities, automatically, in real-time as changes are made in the databaes or on page Fibery page refresh, etc. (like you can with tools like Retool).

Example:

Here is how Retool creates a “table” from an external database.

6cf3ce2-add_data (1)

As you can see here in there docs, you can query data and automatically transform the response into “collections” (tables/entities).

Same with similar tools like ForestAdmin.


Again, the idea is to use a “REST” integration addon/plugin where data can be pulled in every x or on page table refresh and only show available returned data.

Hope this clarifies what I mean a bit more.

To be fair, it can also be used in any automation rules, so that hopefully covers a few more use cases :slight_smile:

Automations can be triggered by many things (including once a day if needed, and hopefully more frequently soon).

Otherwise, could your use cases be covered either with Integromat or Zapier?

Might be interesting to look at some concrete examples…

1 Like

Indeed, while I like the elegance of a persistent, generic data ingest system with options in the setup flow to create new Types, assign fields, etc. (most of this functionality is already in Fibery’s various integration and import functions), I tend to agree that most of the active connection stuff would be handled by Integromat. The initial Type setup, no. But you should be able to just do an Import with a CSV exported from the Database once, setup your types and fields that way, and then use Integromat to just map the SQL connection (MySQL, MS SQL both supported) to the existing fields in Fibery and put it on whatever regular schedule… So basically it sounds like the original idea here would add a bit of convenience, but I’m not sure how much real value. Unless I’m missing something?

1 Like

Well…if you were inclined, you could use scripting (triggered by Integromat or whatever) to actually create the Fibery schema to match the data source, but I don’t know how often the work needed to produce the script would be less than the work to just set up the types as and when needed…

1 Like

Yeah, I find the CSV ingest functionality that already exists to be pretty great actually. It’s hard to imagine setting up a Type to map to external data being any better from some other method, assuming it’s basically a 1-time thing (for each unique data source).

Totally appreciate both of your inputs.

Was trying to say this a few times already. I am aware that you can use Fibery API to push/pull data. But it requires what I’d call, workarounds and extra setup and maintenance. Whether you are using Fibery js, Integromat, Zapier, etc – it requires more “workarounds” to push and pull data. My idea came from using Fibery’s awesome API and feeling like this would be a much better and useful feature for database pulling.

Example with current Fibery scripting/rules/integromat/unofficialjs/etc:

Tigger push via Intgegromat or uojs… Check if database record entity exists in Fibery… update if so… if not… create new record…

Additionally… check if Fibery entities against database entities and determine if they are no longer in database… if they are not present in database any longer, delete entity…

Example with custom rest API pull:

GET Database and return as collection… populate as entities. Done.

Another issue is if we add new columns, we then need to incorporate updates into whichever method we are using to import data, instead of simply being able to pull the columns fields and data as entities live from the database as it is shown.

The idea is not for it to be a 1-time thing. The idea is to pull the data as it is shown in the database. So if columns are added or deleted, it would conform automatically to Fibery “Types”. It’s just like the other tools I mentioned do. Pulls the data as it is and displays it in table form. Then, you can manipulate it as is.

Also, with the current setups… what happens if there was an error with our flow and a record wasn’t deleted properly from Fibery but was deleted in the database (now, it’s still showing in Fibery)? These are little things that become tedious.

I know there are options and workarounds, but I am basing the suggestion (not an official request) on what is available with other apps that work to create internal tools e.g. Retool (raised $75M), ForestAdmin (raised $11M), Internal (raised $5M), Airplane, etc.

I think a native Custom REST Pull would be extremely useful. Then, we can focus on using action buttons to take “action” on the live data.

1 Like

Mmm, you have some good points, certainly. This is not an area of need for me at the moment, so I can’t fully empathize with your position and goals, but it seems like a reasonable feature request at any rate.

1 Like

Thanks for the elaboration @Illusory, I can definitely hear the pain points more clearly now.
Just so I understand, it sounds like your need is focussed on ‘pulling’ into Fibery (like a one-way sync/mirror) including Fibery schema updates to reflect source schema changes.
Did you imagine needing to push data (and schema changes) from Fibery as well? Or is it mostly about having ‘external’ data viewable and connectable from within Fibery?

Did you at all follow the discussion here:

Is your need conceptually related to this sort of thing?
As Michael indicated in that discussion, opening up an ‘integration SDK’ is not impossible, but might not be widely used.

BTW, do you have specific external data sources that you consider v important? Perhaps an integration you desperately need is on the horizon… :slight_smile:

1 Like