Calculations within a Database with Formulas (Comparing Entities from the same Collection/Database)

Following my free trial I’ve recently subscribed to Fibery (which I’m finding fascinating!) but I’ve got to a hard wall when trying to perform simple calculations with Formulas within a single Database. Here it is an example:

My Database, called “Tracker”, with no Relations, has a series of entities (entries or rows) with Fields like “Cost”. So an entity Named “Door”, for example, may have a cost of EUR 500, while another entity called “Window” may have a cost of EUR 600, etc. I wanted to add a Formula to calculate the percentage of the entity Cost from the total cost of all entities within the Database, I’m having several issues to do so:

  1. I tried to use Tracker.Sum(Cost) -which to me shall sum all Costs fields within Tracker, but I get an error “Reference to undefined field Tracker”. This is weird because even the Formula’s suggestions indicate to provide this structure (even when using the AI assistant!). However, it seems this is meant for calculating things from Related databases/entities.
  2. I cannot see a way to indicate “Just sum all the entities values for a specific Field”.
  3. It’s even weirder when you can actually see Total the quantities in each Field using the bottom row feature (which I think has been release recently). So, how come Fibery backlog can run this simple calculation but I cannot make it work with Formulas?
  4. Following point 3) above, even though Fibery can sum, average, etc, all values of a specific Field within a Database, I don’t see a way to “call” this figure from a formula.

Another thing I cannot see to do with Formulas (or with anything really) is comparing/calculating values between entities in the same Database. Using the same examples above, I would like to add a formula that tells me the cost difference between Doors and Windows. For example, if every entity has a Field “Cost”, then I want to make a formula that calculate the difference between that entity Cost and the previous entity, or a specific entity. I cannot manage to make any calculations between the database entities, I only can use Formulas to make calculations between the Fields of the specific entity (row).

I hope I’m making sense, I’m sure I must be missing something because this seems really a basic operation.

Thank you in advance for any suggestion on resolving this issue.

Best,
Miguel

You’re not missing anything

Dear @Chr1sG,

Thank you for your prompt reply and video link.

I can see that Formulas can only manage items within the same entity or related entities. But surely there must be a work around for this issue? It seems like a very basic task to me and I don’t find a clear path to solve it.

There are many applications for which comparing items within the same database may be necessary, for example:

  1. Let’s say I’m tracking Prices for a Product (or even a Stock for example), each month I input the Price of such Product/Stock into the Database as a new entity. And it would be great if I can create a Formula that calculate the Price differential with the previous Price from last month, or the differential of the Price with the average Price of the whole collection.

This does not seem too complex, and in addition this kind of calculation may have several other applications, I’m surprise I couldn’t find anyone else having a similar issue. Do you have a work around this? Have anyone faced the same issue?

There is a new feature introduced by Fibery, where you can actually sum, average, etc., all the values of a Field within a Database, this can be done using the bottom row at the end of a Database. To me this means that this sort of calculation can be done in the back-end of Fibery, but some how cannot be directly done by the user using Formulas (I wonder why?). Going further, I cannot find a way to “call” (fetch) the value that Fibery calculates using the bottom row and use it for the purposes I’m mentioning above (that could have been a work around this, as I could compare a specific Field within an entity with the results Fibery calculates in the bottom row (sum, avg, etc.).

As I’m just starting with Fibery I would be really grateful if you could guide me through this, as I’m sure your answer (and potentially your work around this issue) will not only help me to fix this problem but further understand “How Fibery thinks”.

Best,
Miguel

Based on what you’re describing, it sounds like something that would be well-served by using reports (probably a table visualisation).

Actually, the aggregation calculations in table view are ‘front-end’ only.

The main reason why Fibery formulas only support calculations based on the the entity’s values (and related entities) is that the formula service which is responsible for updates could get easily overwhelmed as the number of entities in the database grows.
Each calculation (e.g. average value) would need to be remade any time any entity in the database was changed, or a new entity added. To allow databases to scale in size (we have plenty of customers who have 100’s of thousands of entities) we have to limit the capabilities of the formula service.

1 Like

Hi @Chr1sG , thank you again for your answer. It sounds like a solid point as I can see how calculations within each entity may need to be remade every time another entity changes or a new entity is added to the collection, potentially overloading the system when large databases are involved. I get it.

Ok, let me check the report feature and see where I can get with that then. I will also try to get around this with creating some sort of Relation. I’ll update this post if I find a novel approach to the issue.

Best,
Miguel

Hi @Chr1sG ,

After playing a bit with Databases I found two workaround this issue. The first one, using the Reports as you suggested (Thank you!) and a second one that I find very interesting as it allowed the use of recursive formulas within a Database (which apparently was not possible -but it seems it is!). I will elaborate on the two options as it could be helpful for other new users, and also there are some questions arising from each one where I would love to have your thoughts:

Using Reports:

As an Example, I create a Database called “Weight Tracker” to monitor weight gain/loss during a diet. The input entities have Fields such “Weight” and “Date” to record the weight each time is measured. The aim is to calculate the Cumulative Weight (Total Weight Loss/Gain) and the Variation against the last measure (the difference between current measure and the last recorded one).

Here it is a screenshot of the Database

I created a Table Report for the Database adding the following Fields: Weight and Date. Then I created two additional fields using calculations within the Report’s configuration menu, “Weekly Weight Loss” (->Variation) and “Total Weight Loss” (-> Cumulative). Please find below the formulas used for each:

Weekly Weight Loss Formula

IFNone(
	DIFFERENCE(
		SUM([Weight])
	),
	0
)

Total Weight Formula

IFNone(
	RUNNING_SUM(
		DIFFERENCE(
			SUM([Weight])
		)
	),
	0
)

As result, the following Report was Generated:

Questions/Comments for @Chr1sG consideration:

  1. I loved the report, it’s clean and easy to use, also allow some cool extra details such adding colour and bars, etc. Fantastic.
  2. The only “but” is that to generate the report using the “ad-hoc” calculations it is mandatory to include the necessary Fields that will be used to make those calculations (all fine with this); however when the report is generated I cannot “hide” any of these columns. For example, if I only want to see the last two columns (for which I created the calculations) I’m unable to do so. This may be an inconvenient when you have calculations requiring several Fields that will not be relevant to display once the calculations are done.

Now, the second approach (which I find much more interesting to discuss), generating the formulas directly on the Database.

Using Relations

Context: as per the initial trend message, the issue was that Formulas within a Database cannot call, use or aggregate information from other entities in the collection. The formulas can only call Fields within the same entity or Relations of the same entity. Therefore, tasks such Running Sums or calculating the difference between a new entity (Field) and the previous one, were not possible. @Chr1sG commented that one of the reasons to avoid Formulas to be dependant of other entities within the Database was to prevent overloading when using large database - which make total sense.

The workaround I found to solve this directly from the Database (without using Reports) was as Follows:

Step 1: Generate a Relation (many-to-many) of the Database with itself. For this example I created a new Database called “Rsum” simulating a “Sales Tracker”: it records total sales of the month, and similar to the previous example, the aim is to calculate The Running Sum of total sales (-> Cumulative) and the Sales Diferencial from the previous entry (->Variation). Therefore the Database has a Field called “Sales” (numeric Entry) and two Formula Fields named “Cumulative” and “Variation”. And again, I Related the Database with itself. See screenshots below:


Step 2: I create an Automated Rule using a Script that will automatically Link (Related) any new entry to the Database with all existing entries of the Database (which has been related to itself as per the previous Step). So, the Scrip runs every time an entity is created. Here it is the script (which was generated using your AI assistant - which was great for me as I don’t now how to code):

const fibery = context.getService('fibery');

// Get the IDs of the current entities
const currentIds = args.currentEntities.map(entity => entity.id);

// Retrieve all RSum entities
const allRSums = await fibery.executeSingleCommand({
    "command": "fibery.entity/query",
    "args": {
        "query": {
            "q/from": "Main Menu/RSum",
            "q/select": ["fibery/id"],
            "q/limit": "q/no-limit"
        }
    }
});

// Extract IDs of all RSum entities
const allRSumIds = allRSums.map(entity => entity["fibery/id"]);

// Prepare the batch operation to link the current RSum entities to all RSum entities
const linkOperations = [];
for (const currentId of currentIds) {
    for (const rsumId of allRSumIds) {
        linkOperations.push({
            id: currentId,
            itemId: rsumId
        });
    }
}

// Perform the batch linking operation
if (linkOperations.length > 0) {
    await fibery.addCollectionItemBatch("Main Menu/RSum", "RSums", linkOperations);
}

return `${currentIds.length} RSum(s) linked to all RSums`;

Step 3: Generate the Formulas for “Cumulative” and “Variation” Fields in the Database.

Cumulative Formula

RSums.Filter(Period < [This RSum].Period).Sum(Sale) + Sale

Variation Formula

If(
  Month(RSums.Sort(Period).First().Period) >= Month(Period) and
    Year(RSums.Sort(Period).First().Period) = Year(Period),
  0,
  Sale -
    RSums.Filter(
      Month(Period) = Month([This RSum].Period) - 1 and
        Year(Period) = Year([This RSum].Period)
    ).Sum(Sale)
)

Step 4: The Database will now automatically calculate the Running Sum of the sales as well as the difference between the entity Sales and the previous recorded value. Without using Reports!

Step 5: Generate a custom View of the Database. See screenshot below:

Questions/Comments for @Chr1sG consideration:

  1. Although this method is a bit more complicated to setup I think it is a bit more customizable than the Report, and then you can create a custom View using only the specific fields you want to show, and hidden the rest of unwanted fields from the Database (which apparently cannot be done when using Reports).
  2. I guess this is a viable workaround when a situation requires to make Formulas or Calculations between entities of a single Database.

P.S.:

  1. I don’t know how to code and I’m just starting with Fibery, so perhaps the formulas used here could be more “elegant”.
  2. @Chr1sG can you think of another Title for this Trend? Perhaps it could be useful for other users (specially newbies like me). And perhaps I’m not using the best Title to address the issues presented in the trend. Let me know.

Have a wonderful weekend and I’m looking foward to having your thoughts about this.

Best,
Miguel

I am not sure why you are saying this. As far as I can tell from your description, the report should work perfectly well if you only included the Week column and the two weight loss columns. I see no reason why the Date and Weight columns would need to be shown.

(by the way, how is the Week column calculated?)


With respect to the second method, you might find it easier to use an alternative method to create a self-relation which includes all entities in the database:

Create a workflow field with only one state option (e.g. ‘Dummy’). Since the field cannot be empty, every entity in the database will have the same state. Now you can use formulas to calculate things based on the entities that share the same state (which is all of them!) e.g.
Weight - State.Objects.Filter(DateField < [This Weight Tracker].DateField).Sort(DateField).Last().Weight
which will find the difference between this entity’s weight and the weight of the latest entity with a datefield value earlier than this one (i.e. the delta value)


Or you can use formulas to create some useful (read-only) collection fields:

Create a formula whose value is a date range for the month that encompasses the entities datefield value, e.g.

DateRange(
  Date(Year(DateField), Month(DateField), 1),
  Date(
    If(Month(DateField) = 12, Year(DateField) + 1, Year(DateField)),
    If(Month(DateField) = 12, 1, Month(DateField) + 1),
    1
  ) - Days(1)
)

This is the result:

Now you can use auto-linking to connect entities that are in one month to all the entities in the previous month, e.g.

Note that it works to match the start date of one month with the end date of a previous month because of the way date ranges are stored in Fibery.

Now you will have two fields: ‘Previous month records’ + the opposite end of the relation, which logically you will rename to ‘Next month records’.

With this you can do some interesting things with formulas, e.g. to get the last recorded weight at the end of the previous month, you can do this:
[Previous month records].Sort(DateField).Last().Weight

And if you want, you can do other clever stuff like creating a lookup from the previous month’s records to get this month’s records as a collection:

So you now have three fields: Previous month records, This month records, Next month records.

And all sorts of formula possibilities open up :slight_smile:

Hi @Chr1sG ,

Thank you once again for your prompt reply and feedback. I was already loving Fibery, but this community support makes it even better.

  1. To Create the “Week” Column in the Report (First Example) I used the following formula/field within the Report Configuration:
RUNNING_COUNT(
	COUNT([Id])
) - 1

And then added the Prefix “Week”. Do you know any better way? How would you do this on a Database? I’ve been looking for away to number the entities in a Database with a sequence (1,2,3… or similar) but I haven’t managed to find a simple way to do it.

  1. Reg the columns in the Report, it may be my lack of expertise, but I needed to include the Date column together with the “Weight” column in order to make the calculations in the configuration section of the report. But then, there was no way for me to hide the Date column and just keep the Week Column together with the two calculated columns when the report is generated. Even though for this example there may be a way to fix it, the bottom line is that for sure there may be cases when you would desire hide some columns in the final generated report and as it is now, it cannot be done (In the Databases or Views you can easily hide Fields/Columns effortless, so I wonder why the Report Table does not give you that option).

  2. Thank you for the suggestion for creating a “Dummy” state and use that as a way to make calculations on the entities using formulas. This is definitely a simpler approach to the Scrip I used.

  3. Reg the additional method to auto link entities -wow, I’ll need to digest that a bit better, It is amazing all the things that can be made with Relations in Fibery. Thank you for this, will try it and try to understand it better.

Thank you!

Mike

1 Like

Oh, OK. Not what I expected :wink:

Why not just use the date field, and choose grouping by week?

I guess my point was not to get/calculate the “real” week number but rather to have a way to generate a continues and custom sequence for all the entities, it could have well being “Shot 1, Shot 2, Shot 3” or something like that.

Why have any column at all then?
Just show the columns you need (make sure you’re not using any aggregation function) and if necessary, sort by date