After converting keywords to References, search for keywords is much less useful

Hi! I’ve nearly got this now (converting References to Reference names): the following code works: it traverses the rich text JSON structure and locates any entities (which I assume is sufficient for finding all References) but now I need to get the actual Reference objects: I’m trying to call getEntityById() but my script is erroring out at that point: possibly because I’m not specifying the entity type correctly? Or not specifying the fields: see the comment “ERRORS OUT” below for the offending line. Would you be able to suggest a call (to getEntityById()?) that will return an Reference object including its name? Once I have that I’ll be able to replace the existing entity reference with its name (text) and then write out the (altered) JSON into the new rich text field, and I’ll be done. Note: this script works fine (copies the JSON intact) with the “ERRORS OUT” line commented out.

Thanks! (hopefully!)…

// Developer reference is at api.fibery.io/#action-buttons

function traverse(fibery, obj) {
    let is_entity = false
    for (let k in obj) {
        const v = obj[k]
        if (k == "type" && v == "entity") {
            is_entity = true
        }
        if (typeof v === "object") {
            if (is_entity && k == "attrs") {
                const id = v['id']
                const type_id = v['typeId']
                console.log("FOUND ENTITY: " + id + type_id)
                // This is an entity (Reference?); now need to fetch the actual Reference and parse out its name to replace this entity with.
                //const reference = await fibery.getEntityById("entity", id, [])  //  ERRORS OUT
                //console.log(reference)
                // TODO: obj[k] = reference["Name"] // or whatever
            } else {
                traverse(fibery, v)
            }
        } else {
            // base case, stop recurring
            console.log(k)
            console.log(v)
        }
    }
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

getEntitybyId is the right function to use to get info on an entity, but the first argument needs to be the name of the database of the referred-to entity. Is “entity” really the right name in your case?

Good point: I had tried ‘entity’ and ‘Reference’ but neither worked. So: if this entity is (say) a Reference to a Tag, should I pass in ‘Tag’? I actually use References to several different Databases, so I won’t know a priori which Database name to pass in: the only information I have at that point is the id and the typeId. I assume typeId tells me what kind of Database it is: is there a lookup from typeId to Database that I can use?

OK I’m nearly there: I had a stupid JS error (traverse() needed to be async). Fixed that, and if hardcode type as “Tag” then it works. (I still need to replace the Reference with reference.Name, but I’ll figure that out next). However, I have one problem: I’m using References to multiple Databases, so I can’t hardcode the type to “Tag”: so I still need to know how to infer the type (eg. “Tag”, “Task”, “Person”) from the typeId. Is there a hidden API function that would do that for me?

// Developer reference is at api.fibery.io/#action-buttons

async function traverse(fibery, obj) {
    let is_entity = false
    for (let k in obj) {
        const v = obj[k]
        if (k == "type" && v == "entity") {
            is_entity = true
        }
        if (typeof v === "object") {
            if (is_entity && k == "attrs") {
                const id = v['id']
                const type_id = v['typeId']
                console.log("FOUND ENTITY: " + id + type_id)
                const reference = await fibery.getEntityById("Tag", id, ["Name"]) // Hard-coded as Tag
                console.log(reference)
                // TODO: obj[k] = reference["Name"] // or whatever
            } else {
                await traverse(fibery, v)
            }
        } else {
            // base case, stop recurring
            console.log(k)
            console.log(v)
        }
    }
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    await traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

If there is, it’s hidden from me :wink:

That’s the thing with references, they could be pointing to any entity from any database, so if you want to get info on the entity, you need to know which database to look in.

So if you know that all entities will be Tags, then just pass the name of the Tag database. Otherwise, you’re a bit stuck, unless you want to exhaustively search all databases(!)
Or you could determine all the typeIDs in advance, create a lookup table, and use this to get the correct database name :woozy_face:

This is why I wrote

:wink:

Sure: a lookup table will work: can you confirm that typeId is a constant? ie. if I get typeId=ABCDEF for a Tag today, will I get typeId=ABCDEF for all Tags, any time? I assume so, because a lookup table wouldn’t work otherwise, but just wanted to check :slight_smile:

OK this is working (uses a lookup to fetch the correct type of Reference, and successfully replaces each Reference with its name):

// Lookup table for mapping typeId to Name.  Note: this table must be
// extended whenever a new Database is added(if it might be Referenced),
// or modified whenever a Database's Name is updated.
const lookup = {
    '<snip>': 'Tag',
    '<snip>': 'Person',
    // etc...
}

async function traverse(fibery, parent) {
    let is_entity = false
    for (let child_key in parent) {
        const child = parent[child_key]
        if (child_key == "type" && child == "entity") is_entity = true
        if (typeof child === "object") {
            if (is_entity && child_key == "attrs") {
                const id = child['id']
                const type_id = child['typeId']
                const inferred_type = lookup[type_id]
                const reference = await fibery.getEntityById(inferred_type, id, ['Name'])
                return reference['Name']
            } else {
                const name = await traverse(fibery, child)
                if (name !== null) parent[child_key] = { 'type': 'text', 'text': name }
            }
        }
    }
    return null
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    await traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

Now I just need to trigger it efficiently: ideally I’d like to trigger it only when I add or modify or remove a Reference from the Details (rich text) field, but I suspect your suggestion of triggering when the count(References) has changed is probably the best I can do. I guess I need a formula that counts the references, and then use that formula field as the trigger?

I’m a bit confused: I added a formula defined as Count(References), but it’s always zero, even for Tasks that have one or more References in their rich text fields. Am I doing it wrong?

Sorry if I have unwittingly misled you, the References.Count() function returns the number of references to the entity (i e. the things that are listed in the References field.
There is currently no way to determine when references/mentions have been added in a rich text field, sorry.
I guess a time-based trigger is the best you can do at the moment.

OK, no problem: I put the script in an action button for now, so I’ll just remember to run it whenever I’ve updated a Task.Details field for now. Other than that (automation) this is a perfect solution for now, although of course I’m looking forward (hopefully) to Reference names being searchable by default!

BTW I noticed that all the Reference (entity) objects have an empty ‘text’ field (in addition to id and typeId); my script could have been simpler if the ‘text’ field could have been pre-populated by the name of the entity being referenced (rather than empty). Alternatively, it would have been simpler if the entity object included the name of the entity being referenced (rather than having to maintain a manual lookup table). Any point in adding those two observations as feature requests?

Afterthought: I now have two identical copies of my script: one as a Rule, and one as an Action Button; it would be nice if I could maintain a library of scripts, and connect both Rule and Action Button to the same script via the library.

This script is working perfectly (no errors, applied to all 200 existing Tasks): thanks for the help!

I’d like to extend it so that in addition to rewriting the rich text with all References replaced by the name of the Referenced entity, it also overwrites a Collection field ([Referenced Tags]) with the list of all Tag References encountered. I can generate the list of Tags easily: at the same point where I replace the Reference with its name in the rich text string I can simply append it to a list: what I don’t know is: how do I pass this list of (Tag) References to the API as a new (set of) values for the [Referenced Tags] Collection field? I think I need to know:

  1. what metadata should I extract from the rich text JSON (and the looked-up Tag JSON) and store for each Referenced Tag? (just the name (string)? the Id? etc.)
  2. what API call I use to replace whatever Tags are currently in [Referenced Tags] with my new list of Referenced Tags?

@ChrisG I think you may given me some of this information already, but I can’t easily find it if so: if you could give me a push in the right direction now I’d appreciate it! Thanks!

PS. Concrete example: let’s say that [Referenced Tags] = (Tag1, Tag2, Tag5) and I apply my script to the Details (rich text) field, which currently contains: "blah blah [T] Tag1 with [T] Tag4 and [T] Tag6" (where the [T]'s represent the embedded Tag References): the result should be that now [Referenced Tags] = (Tag1, Tag4, Tag6).

Update: following example code in Make "creation context" available to scripts - #2 by Chr1sG I tried this:

    const entityWithExtraFields = await fibery.getEntityById(entity.type, entity.id, ['References']);
    const references = entityWithExtraFields['References']
    console.log(references)

but references is empty (didn’t pick up any Tags Referenced in my rich text field). What sort of References should I expect from entityWithExtraFields? (Update 2: I see your comment Make "creation context" available to scripts - #5 by Chr1sG to the effect that References are not available immediately after creation: any idea how long it should take for them to populate? I’m still getting an empty list after about 10 minutes…)

Oh, I figured it out: same problem as before (you did explain this to me already!): that entityWithExtraFields[‘References’] array is for listing references FROM other entities, not references TO other entities.

Is there no possible way to elicit a list of outgoing references? (Other than be crawling the rich text fields and building it up via a script?)

If not then I’m back to my original question: what values should I be storing as I crawl the rich text, and what function can I use to insert those values into my [Referenced Tags] Collection field?

OK all figured out: please ignore my last two requests: here’s the code, for reference: it parses the rich text field and stores all the referenced entities, then replaces them in-place with their (text) names: it then writes that (replaced) text into a second rich text field (to make it searchable) and then it populates a set of (Database-specific) collections with lists of all the referenced entities of each Database type. Code is currently tied to an action button because there seems no way to detect change of a rich text field:

// Developer reference is at api.fibery.io/#action-buttons

const logging = false
const create_searchable_details = true
const update_collection_fields = true
const create_outcome = false

// Lookup table for mapping typeId to entity Name and corresponding
// Referenced Entity field.  Note: this table must be extended
// whenever a new Database is added (if it might be Referenced), or
// modified whenever a Database's Name is updated.

const lookup = {
    'e5f3e990-815a-11ec-adbf-35e3df6ae56f': {
        'name': 'Tag',
        'field': 'Referenced Tags'
    },
    '53c77c70-7cb5-11ec-bdf4-9d8359eb2d89': {
        'name': 'Person',
        'field': 'Referenced People'
    },
    'bdefbc60-7cb6-11ec-bdf4-9d8359eb2d89': {
        'name': 'Product',
        'field': 'Referenced Products'
    },
    'ee89f9a0-7cb4-11ec-bdf4-9d8359eb2d89': {
        'name': 'Organization',
        'field': 'Referenced Organizations'
    },
    '87c49720-7cdc-11ec-bdf4-9d8359eb2d89': {
        'name': 'Trial',
        'field': 'Referenced Trials'
    },
    '54a09420-5dac-11eb-836f-5117371ac1ee': {
        'name': 'Task',
        'field': 'Referenced Tasks'
    },
    '8f95e150-70c7-11ec-b97d-972ab98146c5': {
        'name': 'Epic',
        'field': 'Referenced Epics'
    },
    '848451a0-78e9-11ec-9cbf-ed2f2cff2041': {
        'name': 'Project',
        'field': 'Referenced Projects'
    },
    'ad167630-7014-11ec-9df5-c99d2f84b20b': {
        'name': 'Milestone',
        'field': 'Referenced Milestones'
    },
    'efbd97f0-5dac-11eb-836f-5117371ac1ee': {
        'name': 'Goal',
        'field': 'Referenced Goals'
    },
    '1ad6b940-7cb4-11ec-bdf4-9d8359eb2d89': {
        'name': 'Note',
        'field': 'Referenced Notes'
    }
}

// Create and initialize a field_names list with the names of the
// collection fields listing the corresponding referenced entities.
const field_names = []
for (var key in lookup) {
    field_names.push(lookup[key]['field'])
}

// Recursively traverse the (rich text) document object, storing all
// entity references and then replacing them in-place with the names
// of the referenced entities.
async function traverse1(fibery, parent) {
    let is_entity = false
    for (let child_key in parent) {
        const child = parent[child_key]
        if (child_key == "type" && child == "entity") is_entity = true
        if (typeof child === "object") {
            if (is_entity && child_key == "attrs") {
                const id = child['id']
                const type_id = child['typeId']
                if (logging) console.log("FOUND ENTITY: id: " + id + " type_id: " + type_id)
                const inferred_type = lookup[type_id]['name']
                const reference = await fibery.getEntityById(inferred_type, id, ['Name'])
                found_refs[inferred_type].push(reference)
                return reference['Name']
            } else {
                const name = await traverse1(fibery, child)
                if (name !== null) parent[child_key] = { 'type': 'text', 'text': name }
            }
        }
    }
    return null
}

async function traverse(fibery, found_refs, parent) {
    let is_entity = false
    for (let child_key in parent) {
        const child = parent[child_key]
        if (child_key == "type" && child == "entity") is_entity = true
        if (typeof child === "object") {
            if (is_entity && child_key == "attrs") {
                const id = child['id']
                const type_id = child['typeId']
                if (logging) console.log("FOUND ENTITY: id: " + id + " type_id: " + type_id)
                const inferred_type = lookup[type_id]['name']
                const reference = await fibery.getEntityById(inferred_type, id, ['Name'])
                found_refs[inferred_type].push(reference)
                return reference['Name']
            } else {
                const name = await traverse(fibery, found_refs, child)
                if (name !== null) parent[child_key] = { 'type': 'text', 'text': name }
            }
        }
    }
    return null
}

// Remove all entity references in list from designated collection.
async function emptyReferenceList(fibery, entity, ref_type, refs) {
    for (const ref of refs) {
        if (logging) console.log("REMOVE " + ref_type + " REF: " + ref.Id)
        await fibery.removeCollectionItem(entity.type, entity.id, ref_type, ref.Id)
    }
}

// Add all entity references in list to designated collection.
async function setReferenceList(fibery, entity, ref_type, refs) {
    for (const ref of refs) {
        if (logging) console.log("ADD " + ref_type + " REF: " + ref.Id)
        await fibery.addCollectionItem(entity.type, entity.id, ref_type, ref.Id)
    }
}

// Fibery API is used to retrieve and update entities
const fibery = context.getService('fibery')

// affected entities are stored in args.currentEntities;
// to support batch actions they always come in an array
for (const entity of args.currentEntities) {
    // an entity contains all fields apart from collections;
    // to access a field refer to it by its UI name

    // Note: we could also get a restricted copy of the entity contain only
    // the Details field (+ Id) like this (but not necessary):
    //   const x = await fibery.getEntityById(entity.type, entity.id, ['Details'])
    // That is also the ONLY way to get collection fields, eg:
    //   const entity2 = await fibery.getEntityById(entity.type, entity.id, ['Tags', 'Epic Tags'])
    //   const tags = entity2['Tags']
    //   const epic_tags = entity2['Epic Tags']
    // We could also use that method to get the *incoming* references (entities
    // which have referenced this entity):
    //   const entity2 = await fibery.getEntityById(entity.type, entity.id, ['References']);
    //   const references = entity2['References']
    // although unfortunately there's no way to get a list of entities that this
    // entity references: for that we have to crawl through the rich text fields, as below.

    const details = entity['Details']
    const searchable_details = entity['Searchable Details']

    if (logging) {
        console.log("Entity:")
        console.log(entity)
        console.log("Details:")
        console.log(details)
    }

    const document = await fibery.getDocumentContent(details['secret'], "json")
    const entity2 = await fibery.getEntityById(entity.type, entity.id, field_names)

    if (logging) {
        console.log("Document:")
        console.log(document)
        console.log("Entity + collection fields:")
        console.log(entity2)
    }

    if (update_collection_fields) {
        // Remove existing references (in any case, since if there's
        // no rich text then all collection fields should be empty).
        for (var key in lookup) {
            const field_name = lookup[key]['field']
            await emptyReferenceList(fibery, entity, field_name, entity2[field_name])
        }
    }

    if (document != null) {
        // Create and initialize a found_refs dictionary with the names of
        // the referenced entities.
        const found_refs = {}
        for (var key in lookup) {
            found_refs[lookup[key]['name']] = []
        }

        // Store all entity references found in document, replacing
        // them in-place with the names of the referenced entities.
        await traverse(fibery, found_refs, document)

        if (logging) {
            console.log("Stripped document:")
            console.log(document)
        }

        if (create_searchable_details) {
            // Update the search details field with the stripped text.
            await fibery.setDocumentContent(searchable_details['secret'], document, "json")
        }

        if (create_outcome) {
            // Update the (non-rich text) outcome field with the
            // stripped text (this will strip all remaining structure,
            // but the reference names will survive because they've
            // already been replaced with text).
            await fibery.updateEntity(entity.type, entity.id, {'Outcome': document})
        }

        if (update_collection_fields) {
            // Update all the collection fields with the stored lists of
            // referenced entities.
            for (var key in lookup) {
                const name = lookup[key]['name']
                const field_name = lookup[key]['field']
                await setReferenceList(fibery, entity, field_name, found_refs[name])
            }
        }
    } else if (create_searchable_details) {
        // There's no rich text, so the search text field must be empty.
        await fibery.setDocumentContent(searchable_details['secret'], "", "md")
    }
}

One remaining thing: it would be nice if I could make the collections that I’m auto-populating read-only (they should only ideally only change when entities are added to or removed from the rich text field, and the user certainly should not be able to modify them, because they would no longer by in sync with the rich text field). I see some references to marking fields as read-only in the API: does that apply to collections (relations)? Is it something that I can do as a one-off (programmatically)? Or even better via the UI?

If the field was read-only, you wouldn’t be able to update it via the API (or the GUI) so your script would fail.

That’s surprising! I can only think of 4 ways in which a field can be changed:

  1. direct user interaction (via UI)
  2. formula (cannot be scripted)
  3. rule (could be scripted)
  4. action (could be scripted)

When I say read-only I really just meaning preventing #1. I’ve seen that fields that are set by formulas are automatically greyed out, meaning they can be changed by #2 but not by #1 (haven’t tried #3 or #4 but presumably also blocked?). I was hoping that there was some way to grey-out a field even if it’s not being set by a formula: I guess not?

1 Like

This would be useful. I have lots of fields that i update with automations that i really don’t want users to mess with.

Actions in buttons/rules can basically do the same things a user can do with the ui. It is technically quite challenging to allow them to have permissions that differ from those of a user/admin.
So #3 and #4 (which I guess means buttons) can do only what #1 can do.

I was distinguishing rules from action buttons only because rules are triggered automatically (by definition) so the user doesn’t initiate them (and probably doesn’t even know they’re happening), whereas action buttons are deliberately initiated by the user; but honestly all I’m really interested in is: is this parameter (from Fibery API):

meta. fibery/readonly? If users are able to change value from UI true

something I can use to make collection fields readonly after the fact? either via a script or via the UI? or is it something I could use to create a collection field that is readonly from the start? Or am I misinterpreting its definition?

You can certainly make a field read-only via the API by changing this meta parameter, but then any attempt to write to that field using automations (rules or buttons) or via UI becomes impossible :frowning:

FYI the reason I asked if ‘action’ = button was because we tend to use the word action to mean the things that happen when an automation runs, irrespective of what caused it (triggered by a rule or button press)

Got it: what would be really useful I guess is if a Formula could execute a script, as an alternative to the (rather simple) query language it currently supports: then I could create a readonly field that’s the output of a Formula, but with all the flexibility of the scripting API to power it: is that a possible roadmap item? (I realise that’s partly possible now by creating a non-Formula field and then scripting a Rule to modify it, but then the field can’t be readonly.)