Performance decreasing as complexity/size increases

Get Help
1

Hi! I’ve been putting together an app to track progress on projects, generate weekly status updates and track contacts, products, companies, meeting notes, etc. It’s all coming together really nicely, and I’m frankly delighted with the result so far, and the power and (deceiving) simplicity of the fibery platform. I’ve been developing this app and using it as I go: ie. I’ve been adding records, and also adding new databases, relations, lookups, etc. as I go. Unfortunately, I’ve noticed that as time passes the app is becoming slower and slower to respond: to the point where now there’s a quite noticeable delay in populating any new view, pop-up card, etc.: I can only assume that this is due to the increasing complexity of the data model OR (or possibly AND) the increasing number of records I’m creating. Since I’ve created a few 100 records so far, I’m guessing that perhaps it’s the complexity that’s biting me. Is the feasible? Can anyone speak to performance expectations? ie. should I expect performance to fall off (quite drastically) as my complexity (or possibly complexity x size) increases? If not, then are then any optimizations I can consider to get the performance back up again?

Note: I’m still using a free (personal) account, but I’m hoping to persuade my company to adopt this app, in which case it would obviously move to a paid plan: is there any performance implication in free vs paid accounts in general?

Many thanks!

1 Like
2

I cannot speak to complexity as my use of Fibery is generally fairly simple (few formulae, a pretty “normal” amount of relations, etc.). However I have been doing some large data set testing recently as part of a project to hopefully migrate a company from Airtable + ClickUp (and other tools) into Fibery in the long-term.

What I can tell you is that from my experience a few hundred records should not, in itself, be causing performance issues. I am, however, seeing long load times on 35,000 Entities imported from Airtable (~20 fields in this Database). A single Table view showing all records takes about 20-25 seconds to load. Once loaded it is fairly fast to scroll to anywhere in the list, though. Opening entities is almost instant, and returning to the list from an open entity likewise.

There are some other slow operations in Views as well, such as Adding a Field, or enabling visibility of one in e.g. a List view (truncated to 3000 record max in view). Interestingly this stuff causes some pretty big spikes in client-side CPU usage (but not partciularly in memory). And performance in List view is a bit slower than Table View, although not too bad.

Searches, meanwhile, remain extremely fast. That includes in-line entity linking search as well as “quick search” (ctrl-k). Even Vizydrop graphs are remarkably fast, as long as I’m not trying to create a graph that shows a whole ton of items in a single view (e.g. a unique point for each Entity).

Overall I have honestly been fairly impressed with the performance. Airtable is faster and a little more responsive in Table view, but not by a lot.

3

Thanks for that context! You’re right: search is blisteringly fast: it’s only switching views that seems to take (a lot) longer now. I do have quite a lot of relations and lookups via relations at this point: I wonder if that is a predictor for poor performance?

1 Like
4

Could be a mix of things. There’s your connection sending request to server (when loading the page), the server to find the content, your connection for receiving the data, the CPU and memory to store it and lay it out, calculate the HTML for all the elements. Last few of these scale when you have more fields and more entities to load.

So far we have 6 655 entities, with relations and a few formulas here and there, some that are complex.

image
image764×474 37.2 KB

I’ve tested against other Workspace I have where the total of entities in all spaces are less than 100.
I only did quick tested creating entities in a table view, but what I found was this: the more fields you have showing, the longer it takes. If you hide them, it becomes more snappy.

Furthermore, I also tested having a filter showing entities created today (which was 0) against no filter where it shows some 1 600 entities, but only one field showing. I didn’t notice it being slower, but this may vary depending on the view used.

Formulas, lookups, and relationships might take a bit to load, could be various factors here too, like some formulas require you to iterate over all entities or an index, so that can also scale more or less with count of entities. Thankfully in my case I am not dependant on these being snappy and readily available.

Honestly, I think the biggest reason for sluggishness in most cases is because of HTML render cycle and client-side JavaScript, and you can likely remedy this by hiding fields.

That said we have no idea how long it takes for you or how sluggish it is, for all we know it might be the case for you that it takes an abnormal amount of time :thinking:

1 Like
5

Let me provide some info/hints here.

  1. Fibery can handle large data volumes, we have accounts with 300K records. It works relatively fast for up to 10K records in a single table, and it works “bearably” for 30K+ records in a table. Larger datasets will take time to load, so initial table load may be slow.
  2. It all depends on data complexity. If you have many relations and want to show them all in a table view, it becomes slower.
  3. Formulas and Lookups do not affect performance, since they are calculated async and results are stored in a database. So Views performance is not affected.
  4. Most of the slowness is UI. We recommend to use Filters to show less data and do not add unnecessary fields into Views. What kind of Workstation you have? Fibery is quite CPU-demanding for some Views.
  5. Reports works OK for up to 30K tables, then you may wait 2-5 mins to load the data.

It would be very helpful to see the video of your usage patterns and see what you perceive as “slow”, since it might be normal or not. If not, we would like to dig into details and find out what is going on. If the data is private, you may contact us via Intercom and share the video privately.

3 Likes
6

Thanks! I’d be happy to put together a video if no obvious solution appears! The rendering and JS execution sounds like a likely culprit: the slowing down definitely correlates with increasing numbers of fields: particularly lookups. You mentioned hiding fields: I’d be delighted to do that: most of the fields I’ve added are only required in limited views (as a ‘field’ here, for a filter there, etc.), and yet the ‘node’ views have become quite cumbersome (and take the longest to load): but I haven’t figured out how to hide fields: would you mind pointing me to the relevant docs? (I do know how to hide columns in the Board view, although I don’t know how to unhide them again!)

1 Like
7

Hmm, by default when you create a View you see just Name. Can you post a screenshot of a view where you see all the fields? Maybe you are using Database view mainly, default in the Space?

1 Like
8

The views that are the slowest are the views I get when I click on “Open” to the left of one row of a Database. I would love to be able to tune those views to hide (most of) the fields, and I imagine that would likely fix my performance issue. I think I need to use intercom to send you the screenshot: how do I initiate that? (Or I can email or slack or discord it to you).

9

Just click on the Chat button and paste a screen shot :slight_smile:
image

10

This is embarrassing but I don’t see the chat button (and I have all my ad/beacon blockers disabled). Is it only present on certain pages? Update: found it. Sent screenshot.

11

I got it, this is exactly what we are working right now :slight_smile: We call it Entity View. I hope in a months or two it will be possible to show/hide fields there.

2 Likes
12

Sounds great: so presumably I can expect that at that point I’ll be able to configure my entity views to only show the fields I care about, and that will make it render much more quickly?

2 Likes
13

Yes, but we will improve this View performance as well.

1 Like
14

I ran into performance issues when leveraging the zendesk integration for some relevant discussion.

The issue I ran into was that it is difficult to apply the filtering before you are hit with the browser trying to load all entities. At one point the browser was crashing before I could apply enough filtering so that it wouldn’t crash. It does appear to be much better now that only the name field is included by default, but I also don’t have near as many items in the database now.

Ultimately, it seems there must be paging of some kind added at some point. There are just too many HTML elements on the page, which is what drives up the CPU usage. That can happen from very wide, complicated tables and/or many rows and would be made worse the more complicated the columns are. This is also why apps that must have large tables (e.g., google sheets) have tables that use canvas, rather than HTML. However, fibery has many view types, so leveraging canvas for one view I imagine is not a big priority.

Even if all the data is still delivered to the client, not creating all the HTML elements for all the items would improve the situation quite a bit. As the number of HTML elements increases on a page, everything will slow down.

2 Likes
15

We addressed this issue already here → Limit max amount of entities in Board/Calendar/List Views

16

Great, yeah maybe that explains why I’m not seeing the same performance issues anymore.

17

Great discussion. Throwing in some findings from my end. Fibery is heavy on Javascript which requires lots of CPU. Screen-sharing within Slack etc is also heavy on the CPU. My MacBook Pro from 2017 struggles a lot with this setup. We are turning to Google Meets which is easier on the CPU when doing screen-sharing. This frees up enough CPU to run Fibery pretty good. I’ve heard the new Apple M1 processors are brilliant at Javascript so perhaps that’s an upgrade to aim for. In the meantime I am confident that Fibery will optimize their code as much as they can.

2 Likes
18

@mdubakov: I have a test case that I ran into multiple issues with in regards to performance I can share.

We publish content and we have our own simplified category tree. We often post deals from amazon and I wanted to map the amazon category tree to our own. In something like airtable, it is difficult to represent this hierarchy, so I decided to try the hierarchical list in fibery, which is especially well suited to this.

I cleaned up the category data, which is a list of each category (not nested) with about ~60k rows, and each category contains a list of its children (not super useful in this case). However, there is also a field with the parent category’s id.

The issues I ran into in regard to performance:

csv import

  • the csv importer really really struggles with this number of rows and makes fibery unresponsive across all my tabs. You don’t know if it has hung or whether it is still making progress
  • I split the csv into multiple chunks of 5k rows and I was able to get that working
  • at some point, one of the csv imports stopped halfway through, so i wasn’t sure how to keep importing them and not ending up with duplicates. There is no way to sort by the public-id in fibery, so it was hard to tell what was the last item actually imported from the csv to know where to start again
  • while importing via csv or via the API with a tab open, it seems like there is no initial limit on the number of rows shown in the views until you reload the tab. So, during the import having a view of the data being imported open seems to cause the browser to hang
  • I got partially through this with about 20k categories loaded and the hierarchical list was actually working great after the relations all were processed

deleting

  • So, I decided to scrap the idea of csv importing because I was in an unknown state, so I decided to delete all the rows and start over via the api
  • I couldn’t find a way to efficiently delete all the rows without deleting the entire database
  • I could select all in a table view, but then the page would hang
  • I had to create filters for arbitrary chunks of items that were ~1k items to get the delete to work without hanging my browser

auto-relations

  • The ux of the auto-relations to the same table is quite confusing in this case. I thought i had it right, then ended up confused if I was seeing the auto-relations in the middle of reprocessing or not

image
image1020×1128 57.6 KB

questions

I know this use case is on the extreme side, but it is not massive in terms of the amount of data. It is two numeric columns, then a handful of basic text columns, then an auto-relation to itself. I’m curious to the best practice for loading data like this. Should I be setting the relationship via the API? If so, how would I do that? Load all the data first without relationships, then go back through for a second pass setting the relationships? (otherwise many times the categories won’t exist yet that a particular category would be related to)

You can download the csv file of the amazon category tree here.

1 Like
19

It’s a tiny thing to mention, but giving the database a name in the singular sometimes makes it easier to understand how a relationship is working (when one-to-many).

20

Thank you for reporting this. I have been running into similar problems, particularly with CSV import performance (which seems entirely unnecessary - no need to load the whole thing into memory just to set up the field mappings!), but hadn’t yet made the time to post the problem. I almost think it should be a separate post, but the important thing is that it’s documented. I’ll add details of my situation at some point if it seems relevant, but the problems are basically the same.

1 Like