After converting keywords to References, search for keywords is much less useful

[masp]

I really like the search functionality: I could search for a keyword and it would instantly pull up a long list of results: all the places where the keyword was found, with surrounding context: very useful. I’ve just spent a fair amount of time converting a lot of the important keywords in my Rich Text fields into References (to entities in other Databases), because I really the back-reference feature (being able to pull up all the entities in which a given entity is referenced). However, to my surprise when I now try to search for a keyword that I’ve replaced with an entity reference I only get a single result: the name of the referenced entity: all the per-entity results (with their useful context snippets) have gone: if I want to see all the per-entity results I now can apparently ONLY see them if I click on the Referenced entity and check the back-references: this is less satisfying in 2 ways:

1. it takes an extra click;
2. the back-references show ALL the text of the given Rich Text field: no more contextual snippets.

Is there any way to get the best of both worlds? I’d like to be able to search for keyword “foo” and see in the result list all the instances of both “foo” and #foo (the References), with their included contextual snippets. (But of course retaining the back-reference functionality for its own purposes.)

Example: searching for “kibana”. Before converting to References (“kibana” instances are still plain text) the result list looks like:

  [T] foo bar baz **kibana** blah blah
  [T] bar baz bong **kibana** bing bang
  [T] boo boz **kibana** bunga bing

After converting to References (“kibana” instances converted to References to a Kibana entity) the result list contains just a single line:

  [P] Kibana

and to see the actual 3 results I have to click on that “Kibana” result and then I see the list of 3 entities, but now with ALL their text and no bolding of the word “kibana”.

Thanks!

[Chr1sG]

I guess the thing is, before conversion, you had three examples of kibana, in different places (which may or may not be/mean the same ‘thing’) whereas after conversion, you have a single kibana, which happens to be referenced in 3 places.

If the search worked as you would like it to, I could imagine it being frustrating for some users, where it is not desirable to return dozens of results, of which one is the ‘true’ object, and the rest are merely references to the object.

If I search for ‘verb’ in a dictionary, I may want to get the definition of the word ‘verb’ but not all the places where the word ‘verb’ is used.

[masp]

I take your point, but in that case I may be using references wrong (and have spent about 8 hours over the weekend destroying information in my Rich Text fields!). I’ve been using references as “magic” replacements for existing entities: so for example a Task’s Details field might have contained:

… explore using Kibana as a front-end for OpenSearch using SAML authentication…

and if I searched for “Kibana” then that Task would show up, and I’d see that contextual snippet with highlighting:

… explore using Kibana as a front-end for OpenSearch using SAML authentication…

which was perfect. But now I’ve rewritten that Task’s Details field so it contains:

… explore using [T] Kibana as a front-end for [T] OpenSearch using [T] SAML authentication…

ie. the entites are still present (“Kibana”, “OpenSearch”, “SAML”) but now they’re References to actual fibery Entities (Tags in this case) rather than plain text. This is still perfectly readable, but it’s also great as far as the References go (I can easily pull up a list of all Tasks that Reference Kibana for instance), but now when I search for “Kibana” (or “OpenSearch” or “SAML”) I no longer see that Task showing up in my search results: it’s effectively invisible to searching. (I can still find it, but only by going to the Referenced “Kibana” (Tag) Entity view and scrolling through the list of back-references: and at that point I no longer see the contextual snippets or keyword highlight, so it’s a lot more opaque than the search process used to be.

If I had to choose between plain text (“Kibana”, “OpenSearch”, “SAML”) and References ([T] Kibana , [T] OpenSearch,[T] SAML) then at this point I would choose plain text, because as useful as the References are, search is even more useful. But that would mean spending another 8 hours rewriting 190 Tasks’ Details to replace all the References with plain text: I dread doing that! What would be much better would be (maybe an option?) so that search can be told to include the names of References in its index. That would give me the best of both words: I could still search my Tasks for these entities, but I’d still keep the awesomeness of having them encoded as References. I hope that might be possible!

I guess a workaround would be to expose each entity in plain text and as a Tag Reference; something like this:

… explore using Kibana ([T] Kibana) as a front-end for OpenSearch ([T] OpenSearch) using SAML ([T] SAML) authentication…

but that would be very ugly and hard to read (and time-consuming and error prone) so if I can’t configure fibery to treat these References as searchable replacements for the plain text words then I’ll have to convert them all back to plain text again, which would be a shame.

[masp]

Thanks very much! I’m going to assume for now that it’s going to be approved, and so I’ll continue incorporating Tags (References) in my Tasks’ Details. Is there a visible process for approving feature requests, so that I can watch for this feature getting into the roadmap?

[masp]

I do have an immediate problem: having converted all my Tasks’ Details (Rich Text) fields by replacing plain text entities (“Kibana”) with References ([T] Kibana) I’m finding it much more difficult to keep track of progress: I was relying on search very heavily, and now I’m having to remember which Epic of which Project any given Task might belong to and browse through a fairly complex and well-populated hierarchy of Project → Epic → Task, and/or use my Tags’ back-references to find said Tasks. I guess if References are going to be become searchable it likely won’t be for a few months at minimum, so I need to find a workaround in the meantime: I really don’t want to take the time to manually convert all my References back to plain text again, especially if they’re going to end up being searchable as References, so I hoped that maybe I could create a formula that would automatically copy the contents of each (Rich Text) Details field to a Text field, with all formatting stripped (seems to happen properly already) but also with any References converted to their names (ie. [T] Kibana would be converted to “Kibana”). I could even do this manually, if a formula won’t work (I only have 190 Tasks so far so it would take some time, but nothing like the time it will take to manually replace every reference with its plain-text name!), but when I tried a manual copy from Rich Text to Text just now, the References were converted into what looks like MD5 (or some other) hash string.
Can you recommend possible solutions for:

1. converting References to plain-text names (rather than hash strings) on copy to Text fields?
2. automating copy (ideally syncing) of Rich Text content to Text?

Thanks!

[masp]

(Replying to myself!) After seeing your very useful Reference scripting example in (How) can I display References in my Board and Feed views? I’m wondering: could I use a script to process the Rich Text contents of my Task’s Details field (whenever it changes) in order to reproduce it with all References converted to their names? And then write the result out into a Text field? So eg. if Details contained the (rich) text:

Here is a [T] Kibana reference

then my script would convert it to:

Here is a Kibana reference

ie. is the Rich Text a dict that can be walked (via JS) to find all the References and replace them with their names?

If so then I could automatically maintain a read-only up-to-date non-Reference copy of the Details field, which would be fully searchable and the search results would show up with all the Reference names suitably highlighted: it would achieve my goal perfectly.

[Chr1sG]

Extracting useful info from Rich Text fields is harder than extracting info from References I’m afraid.
The contents can be accessed via API as markdown, html or json, but I’m not aware of an easy way to parse it I’m afraid. Maybe some smart community member will figure something out

[masp]

If it’s accessible as a JSON string then I can trivially convert into a hierarchy of dicts and arrays in javascript if I have access to JSON.parse(). Given that, then, would the JSON give me access to the name of the Referenced entity?

[masp]

BTW I also discovered that I can link a word (eg. “Kibana”) to an Entity in a different way (via the popup toolbar in the Rich Text field): that highlights the word in a colored box and when clicked on it generates a floating popup containing the actual Reference: and that (highlighted, linked) word does show up in the search results. So if I was to re-write all my Rich Text to replace the explicit References with this other type of Entity reference, then I would also solve my problem (although that would take an awful lot of time, and also the UI experience is not so nice (you can’t see the type of entity or the entity name until you float over the linked word, and you have to click twice to follow the link to the referenced Entity). Is this a possible direction for me to go in? Might it be possible to combine the handling of the two kinds of references, so that they both leave their names searchable? (NB it would be much more painful incorporating this other type of reference as you type, because you can’t just type #keyword and autocomplete: you’d have to stop and click and type each time.)

[masp]

I’m afraid I’m not doing very well getting my test script to work (just trying to verbatim copy from (Rich Text field) Details to (Rich Text field) Details2 for now):

const fibery = context.getService('fibery');

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']

    console.log(details)
    console.log(details['secret'])
    console.log(details2)
    console.log(details2['secret'])

    const document = await fibery.getDocumentContent(details['secret'], "json")
    console.log(document)
    const content = document['doc']['content']
    console.log(content)

    await fibery.setDocumentContent(details2['secret'], content, "json")

    //await fibery.updateEntity(entity.type, entity.id, {
    //    'Details2': content
    //});
}

This gets as far as printing the content, then errors out (500). Am I missing something?

[Chr1sG]

This code will get the contents of the Description (rich text) field and put it in the simpler text field as JSON:

const fibery = context.getService('fibery');
for (const entity of args.currentEntities) {
    const x = await fibery.getEntityById(entity.type, entity.id, ['Description']);
    const content = await fibery.getDocumentContent(x["Description"].secret,'json');
    await fibery.updateEntity(entity.type, entity.id, { 'Simple text': content });
}

In your case, I don’t think you needed the line const content = document['doc'][['content']
I think you can just use document directly

[masp]

Thanks! That works: unfortunately I’ve discovered that search does not seem to index simple text fields, so I now need to copy the contents of Details to a new rich text field (Details2); I found that calling updateEntity() doesn’t seem to work (for a rich text field) but the following does work (the trick was to pass the document itself, as you pointed out):

    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    await fibery.setDocumentContent(details2['secret'], document, "json")

So now I need to write a function to fetch the Reference entities and then replace them with their names: hopefully not too difficult

Thanks again!

Mike

[masp]

Hi! I’ve nearly got this now (converting References to Reference names): the following code works: it traverses the rich text JSON structure and locates any entities (which I assume is sufficient for finding all References) but now I need to get the actual Reference objects: I’m trying to call getEntityById() but my script is erroring out at that point: possibly because I’m not specifying the entity type correctly? Or not specifying the fields: see the comment “ERRORS OUT” below for the offending line. Would you be able to suggest a call (to getEntityById()?) that will return an Reference object including its name? Once I have that I’ll be able to replace the existing entity reference with its name (text) and then write out the (altered) JSON into the new rich text field, and I’ll be done. Note: this script works fine (copies the JSON intact) with the “ERRORS OUT” line commented out.

Thanks! (hopefully!)…

// Developer reference is at api.fibery.io/#action-buttons

function traverse(fibery, obj) {
    let is_entity = false
    for (let k in obj) {
        const v = obj[k]
        if (k == "type" && v == "entity") {
            is_entity = true
        }
        if (typeof v === "object") {
            if (is_entity && k == "attrs") {
                const id = v['id']
                const type_id = v['typeId']
                console.log("FOUND ENTITY: " + id + type_id)
                // This is an entity (Reference?); now need to fetch the actual Reference and parse out its name to replace this entity with.
                //const reference = await fibery.getEntityById("entity", id, [])  //  ERRORS OUT
                //console.log(reference)
                // TODO: obj[k] = reference["Name"] // or whatever
            } else {
                traverse(fibery, v)
            }
        } else {
            // base case, stop recurring
            console.log(k)
            console.log(v)
        }
    }
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

[Chr1sG]

getEntitybyId is the right function to use to get info on an entity, but the first argument needs to be the name of the database of the referred-to entity. Is “entity” really the right name in your case?

[masp]

Good point: I had tried ‘entity’ and ‘Reference’ but neither worked. So: if this entity is (say) a Reference to a Tag, should I pass in ‘Tag’? I actually use References to several different Databases, so I won’t know a priori which Database name to pass in: the only information I have at that point is the id and the typeId. I assume typeId tells me what kind of Database it is: is there a lookup from typeId to Database that I can use?

[masp]

OK I’m nearly there: I had a stupid JS error (traverse() needed to be async). Fixed that, and if hardcode type as “Tag” then it works. (I still need to replace the Reference with reference.Name, but I’ll figure that out next). However, I have one problem: I’m using References to multiple Databases, so I can’t hardcode the type to “Tag”: so I still need to know how to infer the type (eg. “Tag”, “Task”, “Person”) from the typeId. Is there a hidden API function that would do that for me?

// Developer reference is at api.fibery.io/#action-buttons

async function traverse(fibery, obj) {
    let is_entity = false
    for (let k in obj) {
        const v = obj[k]
        if (k == "type" && v == "entity") {
            is_entity = true
        }
        if (typeof v === "object") {
            if (is_entity && k == "attrs") {
                const id = v['id']
                const type_id = v['typeId']
                console.log("FOUND ENTITY: " + id + type_id)
                const reference = await fibery.getEntityById("Tag", id, ["Name"]) // Hard-coded as Tag
                console.log(reference)
                // TODO: obj[k] = reference["Name"] // or whatever
            } else {
                await traverse(fibery, v)
            }
        } else {
            // base case, stop recurring
            console.log(k)
            console.log(v)
        }
    }
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    await traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

[Chr1sG]

@masp

Is there a hidden API function that would do that for me?

If there is, it’s hidden from me

That’s the thing with references, they could be pointing to any entity from any database, so if you want to get info on the entity, you need to know which database to look in.

So if you know that all entities will be Tags, then just pass the name of the Tag database. Otherwise, you’re a bit stuck, unless you want to exhaustively search all databases(!)
Or you could determine all the typeIDs in advance, create a lookup table, and use this to get the correct database name

This is why I wrote

@Chr1sG

Extracting useful info from Rich Text fields is harder than extracting info from References

[masp]

Sure: a lookup table will work: can you confirm that typeId is a constant? ie. if I get typeId=ABCDEF for a Tag today, will I get typeId=ABCDEF for all Tags, any time? I assume so, because a lookup table wouldn’t work otherwise, but just wanted to check

[masp]

OK this is working (uses a lookup to fetch the correct type of Reference, and successfully replaces each Reference with its name):

// Lookup table for mapping typeId to Name.  Note: this table must be
// extended whenever a new Database is added(if it might be Referenced),
// or modified whenever a Database's Name is updated.
const lookup = {
    '<snip>': 'Tag',
    '<snip>': 'Person',
    // etc...
}

async function traverse(fibery, parent) {
    let is_entity = false
    for (let child_key in parent) {
        const child = parent[child_key]
        if (child_key == "type" && child == "entity") is_entity = true
        if (typeof child === "object") {
            if (is_entity && child_key == "attrs") {
                const id = child['id']
                const type_id = child['typeId']
                const inferred_type = lookup[type_id]
                const reference = await fibery.getEntityById(inferred_type, id, ['Name'])
                return reference['Name']
            } else {
                const name = await traverse(fibery, child)
                if (name !== null) parent[child_key] = { 'type': 'text', 'text': name }
            }
        }
    }
    return null
}

const fibery = context.getService('fibery')

for (const entity of args.currentEntities) {
    const details = entity['Details']
    const details2 = entity['Details2']
    const document = await fibery.getDocumentContent(details['secret'], "json")
    await traverse(fibery, document)
    await fibery.setDocumentContent(details2['secret'], document, "json")
}

Now I just need to trigger it efficiently: ideally I’d like to trigger it only when I add or modify or remove a Reference from the Details (rich text) field, but I suspect your suggestion of triggering when the count(References) has changed is probably the best I can do. I guess I need a formula that counts the references, and then use that formula field as the trigger?