Questions about the integration API webhook endpoints

[oooooreona]

I am currently working on a personal development of an integration between Todoist and Fibery.
I was able to implement synchronizing Todoist information to Fibery by referring to the document and sample repository.

However, I have not been able to find a specific example using the integration API webhook endpoint shown below, nor have I been able to find similar implementations in other projects.

https://api.fibery.io/apps.html#post-api-v1-synchronizer-webhooks
POST /api/v1/synchronizer/webhooks
DELETE /api/v1/synchronizer/webhooks
POST /api/v1/synchronizer/webhooks/verify
POST /api/v1/synchronizer/webhooks/transform

Question 1
First of all, is it correct that these endpoints are used to update the corresponding entities on the Fibery side in response to webhooks coming in from external systems?

Question 2
If I understand the above correctly, I can set a specific webhook callback URL on the Todoist side, but which endpoint URL should I set for this?
Is there a dedicated endpoint for receiving webhooks, similar to https://oauth-svc.fibery.io/callback in oauth?

Question 3
Instead of answering these questions, it would be totally fine if you could just publish a sample that actually uses the above endpoints.

Thank you very much for providing the integrations API!

[misha_yermolayev]

Hello, @oooooreona

I will try to answer your questions:

Question 1:

Yes, this is correct

Question 2:
When you create new integration type the new url for incoming messages is created as well, so each integration has its own unique url.

It works well for systems like GitHub where you have API to register webhooks on custom urls but it’s hard to use it with systems like Todoist which expect you to have only one Public url for all notifications.

But there is a trick you can try to use:
You will need to implement the public endpoint by yourself in your connector and proxy payload to registered webhooks.
We do the same for our Intercom integration so I’ll try to describe the idea briefly here:

POST /api/v1/synchronizer/webhooks

This endpoint is required to register webhook for specific Fibery types. As one of parameters it also passes url (docs miss it for some reasons). We save all the info in a database (e.g. MongoDB) and return our webhook object

POST /api/v1/synchronizer/webhooks/verify

here you check that event is valid and indeed came from todoist. Should return object with webhook’s id if it’s valid and can be trusted

POST /api/v1/synchronizer/webhooks/transform
map event’s payload to data that needs to be sent to Fibery

DELETE /api/v1/synchronizer/webhooks

is required to delete webhook for specific Fibery types. Here we remove the webhook we created from our MongoDB

And here comes the trick:
We implement one more endpoint:

POST /api/v1/synchronizer/webhooks/income

We make it publicly available and use it as endpoint for all incoming messages from external system.
When we receive a message we look in our MongoDB for webhooks that should be triggered for this message and send the payload to the urls of those webhooks…

I know it’s hard without examples but I hope I was able to explain the main idea…

Question 3
I’d be happy to give you an example but unfortunately we don’t have them open-sourced yet… But I will be glad to help you with your Todoist connector if you are still going to implement it )

[oooooreona]

Thank you so much for the details!
I will try to develop it more based on what you have taught me.

I would like to share my project with you once I have developed it a bit more and it has taken shape.

Again, thank you very much.

[oooooreona]

It has been quite some time since my last post, and although I finished all the development work by the end of February, I started working for a new company, so I could not make time to write this article.

I have implemented what I am currently able to do in the development of the Todoist integration application, and I would like to ask you to check the code and ask some questions.

I know you are busy, so I will wait patiently and would appreciate it if you could check it when you have some time.

I would be very happy if this app I developed could be of use to someone in the future!

Project link

GitHub - un4v5s/fibery-todoist

Questions about features I want to implement

About Bi-directional sync

The bi-directional sync of JIRA released in the beta version was great.

It would be great if you could release the actual implementation code as OSS in the near future!

This is not a question, just a request.

About Entity update with incoming webhook

I tried to implement this based on your previous answer, but I was not able to implement it to the best of my ability.

I would like to wait for actual samples from Intercom app and others to be released.

Also, I actually tried Intercom’s integration and it was still a great experience.

As I’m sure you know, I have seen requests on GitHub for near realtime sync based on Webhook to be implemented, so it would be great to see a detailed real-world example published.

I think many people would be happy to see this feature enabled in the new JIRA integration as well.

This is also not a question, just a request.

Q. About the partial update of Entity.

This is a question and a request for partial entity override in the integration API.

The Todoist API handles completed tasks differently than normal tasks, and Tasks and Completed Tasks are retrieved from different API endpoints.

In addition, Tasks retrieved from the Completed Task endpoint omit some properties.

As shown in the image below, for example, “task creation date” and “multiline text description” are properties that are omitted in the Completed Task endpoint.

On the Fibery side, if you separate the Task and Completed Task DBs, the task will be moved to a different DB when it is completed, so if it is mentions in a text somewhere, it will no longer exist.

Therefore, I implemented to display both Tasks and Completed Tasks on one Task DB.

However, if the Fibery data is updated with the information obtained from the new Completed Task, the missing properties from the already existing task will disappear.

To avoid this, if there is a synchronization method called “Partial Overwrite”, I can update only the properties that already exist, such as “Completion Date” and “Status”, etc. Is it possible to implement this at this time?

By the way, I found a description of delta synchronization in the document of the Integration api, and thought this might be the feature I was looking for, so I tried it. However, it seems that this one does not “partially overwrite” but “completely replace”.

The documentation didn’t give any details about delta synchronization, so I would also like to ask you to check if my understanding is correct.

Specification of delta synchronization:

• Only one of “SET” or “REMOVE” can be specified.
• If SET is specified, the target Entity will be overwritten with all newly passed information. (Missing properties for the Schema definition are overwritten with null.)
• If REMOVE is specified, the target Entity is deleted.
• Entities that are not included in the items array will remain in the Fibery side.

If the above is correct, I would like to have another option like “UPDATE”, “UPSERT”.

I think it would be possible to implement this at this time by starting up a database on my application side and keeping a copy of the data, but I don’t want to do that for maintenance and security reasons.

I would be very happy if you could publish an actual example here as well!

That’s all I have to ask.

Requests for Integration API

Creating views from the Integration app

This is just a suggestion, but I thought it would be nice to be able to generate views that the developer recommends while developing the integration.

Here is a concrete example from the integration I created this time.

In Todoist, not only for tasks as mentioned above, but also for projects, it is necessary to hit different end points for active projects and archived projects.

However, on the other hand, when retrieving a Task, it is normally retrieved with active status, even though the Task is in the Archived project.

In the integration I created this time, I omitted the getting the Archived Project.

All Todoist tasks always belong to a parent project, but since Archived Projects are not retrieved, they are displayed in Fibery as uncompleted tasks with no parent.

Therefore, in order to match the Todoist display, the “Project is Empty” filter must be applied after the view is created.

I think it would be troublesome for users of the integration feature to go through this kind of detailed setup each time they use it, so I thought it would be a good idea to provide a function to generate a sample view.

In particular, since the integration feature is likely to be used by users of the original app, it would be a memorable experience if a view similar to the original app is generated on Fibery at the same time they start using the integration feature.

I have come up with several ideas for this matter that can be implemented at this time.

1. Have the user provide the Fibery api key and domain name when setting up the integration, and generate the view via the Fibery API.
   
   Receiving Fibery’s API key doesn’t seem like a good security idea.

2. Provide JavaScript that can Create View by copying and pasting it into Action Button’s script and executing it.
   
   I don’t think it’s a good idea to have no-code users to do the black box work, because if this becomes the norm, there is a risk that someone with malicious intent could exploit it.

So, I know it’s not a priority, but it would be nice if there was a way to do create view safely with integration api.

About Formula field

I have implemted a db for date and week within this app.

I thought it would be nice to be able to add pictograms to “today” and “this week” to make them stand out in the view that is shown to the user.

I can provide that field on the integration app side, but it will not be reflected until synchronization occurs, so it would be nice if the integration app could be given the ability to generate Fibery’s formula field.

Sorry again for the very long post.

I would be very happy if you could answer briefly when you have time.

Now that GPT-4 has been released, it may be easier to develop integration features.

[misha_yermolayev]

oh, that’s super cool

I will try to take a look at it by the end of the week. Great job

[misha_yermolayev]

Hi, I still haven’t given your code a proper look but here are some answers to your question:

Q: About Entity update with incoming webhook

We are now discussing how to refactor our webhook flow to support better systems like todoist. So I think we will publish detailed examples after we finalise the API for it. Unfortunately, can’t give you any estimates right now(

@oooooreona

Q. About the partial update of Entity.

Delta synchronization should do the trick. This statement “(Missing properties for the Schema definition are overwritten with null.)” is not correct so it will overwrite only passed properties. If property is missing in object then it shouldn’t do anything with the previous value. We use same mechanism for applying updates from webhooks

Q: Creating views from the Integration app
Q: About Formula field

I doubt this will happen in any time. The reason for this is that integrations know nothing about Fibery. They are just sources of data.
But this is something you can setup using Fibery’s templates. E.g. you create a space with integration where you configure it and create default views and formulas. Later you share this space as a template and that’s it

[oooooreona]

Thank you for taking the time to get back to me!
Also, thank you for taking the time to answer my questions.

Q: About Entity update with incoming webhook

We are now discussing how to refactor our webhook flow to support better systems like todoist. So I think we will publish detailed examples after we finalise the API for it. Unfortunately, can’t give you any estimates right now

This is very good news that you are considering this feature.
I’m looking forward to hearing something new about this in the future.

Q. About the partial update of Entity.

Delta synchronization should do the trick. This statement “(Missing properties for the Schema definition are overwritten with null.)” is not correct so it will overwrite only passed properties. If property is missing in object then it shouldn’t do anything with the previous value. We use same mechanism for applying updates from webhooks

I see that this was something that could be accomplished with delta synchronization. I am glad I was wrong.
That means there must be a mistake somewhere in my code.
I will check and fix it.

Q: Creating views from the Integration app
Q: About Formula field

I doubt this will happen in any time. The reason for this is that integrations know nothing about Fibery. They are just sources of data.
But this is something you can setup using Fibery’s templates. E.g. you create a space with integration where you configure it and create default views and formulas. Later you share this space as a template and that’s it

I understand that what I requested is beyond the scope of what the integration should do.

I also understand that a template can be used to achieve what I want.

Actually, I tried the template during development, but it didn’t work, probably because I was testing the same integration functionality in multiple spaces.
So I gave up on it for a while, but I am glad to know that templates can be used with regard to integration as well.

I don’t know when I will be able to show you the next results, but I will try to improve it a bit more based on this advice.

Thank you very much!

[Illusory]

Hello there!

Thanks for making this awesome plugin. Installation is not clear, however! How is authentication properly applied?

Best,
Josiah

[oooooreona]

Thanks for finding and commenting on this topic!
I am yet a hobby developer and this is just something I created for myself, so I think you would need to deploy this code to any hosting service in order to use this app.
In my case, I have created an app for $5/month with DigitalOcean to host this.
I then connected the GitHub project to the DigitalOcean App and it worked.
I think with ngrok you could host it on your computer and test its use.

Besides that, you need to create an app on the Todoist developer page.

As you know, as mentioned above, this project has some issues that need to be resolved (partial update feature, bi-directional sync), but unfortunately, I am busy with my day job and haven’t made time to fix them.
And the busy period is likely to last a little longer…

I don’t know if I’ve answered your question, but I would love to help anyone, so please feel free to ask additional questions if I’ve missed the point.
If you could be more specific about the situation you are having, I may be able to help you as I have read quite a bit of Fibery’s integration feature development documentation (it’s been a little while since then).

Best. Reona

[Illusory]

Hey there! Yes, we’re hosting the app as well as created a Todoist app. Where are authentication params supposed to be set to authenticate properly? When attempting to authorize the app, we get a redirect error.

[ESGdataBoi]

I know nothing about app development/coding, but I would love to make use of this as well. The more info that anyone could provide to help me host this app somewhere and use it, I’d be forever grateful!

[oooooreona]

Sorry for the very late reply.

I see that you are getting a Redirect error.
I had already closed the app for myself, but I connected the GitHub source code and deployed in Digital Ocean again.

In the process, I got an Invalid Client Error when I tried to add the Integration App to Fibery without setting the Client ID and Client Secret to environment variables in the Digital Ocean configuration screen.
(In the video below, I also set ENV_TEST_TOKEN, but the app worked without it)

Is this different from the redirect error you are facing?

Fibery - Todoist Integration Setup Demo - Watch Video

The Todoist Client ID and Client Secret were obtained by creating an app in Todoist (see image below).

I just showed you that the app works properly with my settings.

However, considering making this available to others, I may have to do it on the Todoist side as well, as there is a page to Submit the app. But any more than this is technically out of my hands, sorry.

If I have not been able to solve any of the problems you are facing, I hope other engineers can help.

As mentioned even recently in a topic here, the development of integration features does not seem to be a high priority…
Of course, I understand that it is inevitable since their resources are limited!

Bi-directional integrations/sync

I think it is unlikely that there will be two-way-sync for these platforms any time soon. Fibery is trying to be a replacement for them, so there is less incentive to develop a two-way-sync that would imply that users keep Fibery and Trello (or F and N, F and CU etc.) I think it’s more likely that the product team would focus on adding features to Fibery that would make it less and less likely that a user needs to also use T/N/CU.

But it would be really nice if a diverse set of integrations existed

If you could host the Todoist app, that would be really great!