References reloaded

[kalman]

Thanks for joining the discussion! Here’s a quick response to your points:

Saving Highlighted Text in a Reference Database in Markdown Format
We are planning to save highlighted text in a separate reference database, and it will be in Markdown format. This means the highlight won’t update automatically if the original text changes. For instance, if you quote something like “the customer loves the red color,” and later, the original text changes to “the customer hates the red color,” the meaning of your quote would change too. We think it’s important to notify the person who created the reference if the original text changes so it can be reviewed and updated it as needed.

We plan to save the quoted text without links (since we cannot follow them) or images, replacing them with placeholder text.

Organizing References Chronologically
We are also working on organizing reference entries in a database. This will allow you to view them in a list and create formulas, automation, and reports. I believe this will address most of your concerns.

Feel free to ask if you have more questions.

[kalman]

After a long radio silence, we have begun developing the Reference Creation and Settings feature. While there are still some moving parts, the core concept is finalized.

Please watch the video that explains how the Reference creation will work.

This is the right time to provide feedback on the functionality and flow of the feature.

• Reference creations
• Reference settings

[Minh_Tri_Do_Hang]

Seriously I’m really waiting for this, day by day, week by week to looking this release. I dont know how Thursday evening become my routine to check the release weekly.
I just watched the demo, that’s really make a fantastic move when It can solve both pkm and user feedback
Really looking for this week release. Anyway thank you for your team

[Chr1sG]

You might want to dampen down your enthusiasm I don’t expect there will be a release of this functionality for a couple of months yet, at least.

[Yuri_BC]

@kalman

This is the right time to provide feedback on the functionality and flow of the feature.

My first question is: Does References 2.0 extend to the standard Relations between database items, do they also get fields, or are Relations still a completely separate thing technically?

[kalman]

@Yuri_BC

My first question is: Does References 2.0 extend to the standard Relations between database items, do they also get fields, or are Relations still a completely separate thing technically?

According to the current plan, this will be a separate solution with a possibility of migration: please see here.

please let me know if you have further questions.

[Yuri_BC]

Thank you, I will study that first. In the meantime, please update the important but dead link Reference entity in your initial post of this topic? Thanks

[kalman]

fixed

Reference entity — Roadmap Item | Fibery

[cannibalflea]

Although I have been very interested in this work, I have to admit I haven’t fully reviewed and understood the proposal, so apologies if this is clearly laid out in the roadmap or in the threads.

But I was hoping for a quick clarification of my understanding:

The Reference Entity feature (discribed here) is intended to extend unstructured text with links to structured date (i.e. fibery entities). This was already possible before in some form with the System Reference Database (from any database to any other). However, with the new User Reference Database, it is possible to add more structured context to the reference itself (with the only limitation being limits on number of sources & targets, at least for now). As such it finds its best use in identifying insights & ideas, indicating related entities (concepts, people, projects, metrics, etc.) and providing some data on things like degree of relevance, … and providing ways of measuring & reporting on these references.

The Polymorphic Relations feature is intended to extend the relationship model between structured entities/data to be more flexible than fibery’s existing entity relation model. My understanding of this work is that whereas relations limit the source and target of relations to one type as well as very rigidly & specifically define the context/type of relationship between entities, polymorphic relations would allow not only mixing of different sources and targets but also (hopefully) richer description of the relations without a need to create a relation field for each type of relation (e.g. separate many-to-many fields for “relatives” and “co-workers” and “friends” …) or creating separate join table to support the relation model. I can see two types of uses:

• Being able to select different types as targets would allow curation of specialized entities with similar relations together (e.g. being able to capture a person’s attendance in different types of meetings, events, etc. in a consolidated relation)
• An equally useful feature would come from the properties of the relations through representing semantic relationships between entities (on the edges), for example between people (e.g. John is Jimmy’s brother, John is Tina’s partner, John and Amy worked together) or a persons employment history which links people and their roles to organizations/companies along with a temporal context (all useful things to have in a CRM).

Is this more or less aligned with how these features are viewed by the fibery team?

If the above is true, I think that there is one type of relation which is not covered by this: relation one between unstructured/text data (i.e. text block in one entity/document and another). This is currently not possible in fibery (apart from hyperlinks to titles). However, it is covered under block references or transclusion blocks in tools like Roam or Obsidian. Does this make sense?

[B_Sp]

@kalman thanks for your response to me!

Could you let me know, in particular, if these two requests are on your radar as far as improvements to references:

Show in Reference Highlights Referenced Entity more clearly

and

Ability to quote text across Fibery like in Discourse

This second one is such a natural extension of your existing iteration in Fibery that it would really be too bad if something like this doesn’t come into existence. With the way references are structured already, my team extensively uses #-tagging to point out other entities when writing and commenting around Fibery. We are missing however the ability to simply quote some text somewhere, and then reference that in a continued dialogue. Instead, we have to either screen shot the quoted text and past it, or use the primitive existing quotation capability, which has one huge flaw in that if you are trying to write text with a “soft” return - using “shift” + “return” in order to maintain the reference itself, the quotes won’t work because you’ll quote the entire set of text.

Here’s an example: I am commenting in an entity and I want to write a few references. I have to write like this, using “shift” + “Return” in the breaks here, or I lose the reference (because if I do a “hard” return hitting just “Return” the reference doesn’t pick up the text above the hard return):

Let’s imagine I’m commenting in an entity that is a project:

During the meeting that took place of FIRST REFERENCED ENTITY, we said:

(now I have to use a soft return or I lose the reference above to the meeting)

THIS PROJECT (the entity I’m commenting in) is going to be late 2 weeks. This is due to the issue raised in NOW THIS IS ANOTHER ENTITY, LET’S CALL IT AN “INCIDENT” ENTITY by user x…AND NOW HERE I’D LIKE TO QUOTE THE COMMENT IN THAT OTHER ENTITY I JUST MENTIONED

if I want to actually put the quote from the other entity here with Fibery quotes, I’ll wind up quoting everything above, unless I do a hard return. But if I do the hard return, the above references are lost

I hope I’ve explained this right and I’m eager to get your response, thanks again!

[kalman]

Thanks for diving deep into the features we’ve been working on. You’ve got a solid grasp on it

Just to highlight a bit more: when we talk about the polymorphic relation and fields that describe the reference relation, right now, they’re only available through something we call “References 2.0.” But we’re planning to expand this in the future.

You’re also spot on about the relation between unstructured data. We aware this gap, and I can tell that addressing this is among the different features in our backlog.

If you have any more questions or if there’s something you think we might’ve missed, please let me know.

[kalman]

You’ve touched upon an interesting point regarding the potential benefits of placing References in a separate database. We believe that this approach could indeed address many of the challenges you highlighted.

By displaying References as a target entity through a database call, we can achieve a level of flexibility that surpasses our current capabilities. While this part of the functionality is still in the works, it’s definitely on our roadmap. The vision is to offer a configurable database view that provides more flexibility than what we have now.

Furthermore, the ability to display quoted text is also in our plans. This would allow specific parts of Reference entities, such as the referenced text, to be showcased within the main text. The display could be similar to our current entity display or even presented in a callout-like format, enhancing readability and context.

I hope I’ve captured the essence of your query and provided some meaningful answer. Please let me know if you further questions and/or idea for this feature.

[B_Sp]

Ok thank you @kalman glad what I was trying to explain appears to have been clear to you!

I will eagerly await the development of references 2.0. Wouldn’t be here in Fibery if it wasn’t for your great references feature in the first place, so I trust you guys will keep making it even better, can’t wait to see what you come up with!

[Yuri_BC]

I made an overview of the New References, to check with you if I fully understand this (pretty big) feature development.
I think to receive useful user feedback, fibery may want to create a method/system that gives users better insight in the whole project as well as all the feature requests and issues related to it. Discourse currently scatters that information, and for the users there is no clear insight in how the architect/developers gather, assess and prioritize the related needs requests or impacts.

Anyway, here is my overview, please comment if this is complete enough, and what is prioritized, what is not considered and what may be a future development? And users may add related features that are not mentioned yet.

Overview of the ‘New References’ Feature in Fibery

Introduction: The ‘References’ feature in Fibery is a pivotal tool in knowledge management, acting as breadcrumbs that trace the origins of ideas and knowledge. This case study delves into the intricacies of this feature, its functionalities, and the challenges it addresses.

Purpose: References eliminate the need to start from scratch, building upon previous discoveries and insights. They provide a structured way to connect rich text highlights to other entities and define these connections with attributes.

Approach:

• References are treated as a separate database entity in Fibery.
• The goal is to create horizontal features that vertical use cases can utilize.
• The User Reference Database (URD) and the polymorphic lookup field are integral to Fibery’s functioning.

Problem Statement: There’s a need to connect one rich text highlight to another entity and define this connection with attributes, without the hassle of reselection or redundancy.

Functionalities:

1. Highlighting Text: Users can highlight a text passage and add user-defined attributes to the highlighted information.
2. Target Reference: Add a target reference to the highlight.
3. Organization: Organize references by attributes, source attributes, and target attributes.
4. Formulas & Automation: Use formulas, automation, and reports across the source-attributes-target relationship line.

Solution:

• Ideal Scenario: Any highlighted rich text field in a database entity could reference any other in the Fibery Workspace.
• Practical Approach: Introduce the User Reference Database (URD) to store references between entities in different databases. The current Link to Entity records will remain in the System Reference Database (SRD).

User Reference Database (URD) Details:

• Source, Source Field, Highlight, Target, Created by, Creation date, Modification date, User-created fields, and System Relations.

Functionality Highlights:

• Each reference in the URD has one source and one target.
• Multiple URDs can be created with different settings.
• URDs are directly linked to the workspace, requiring admin rights for certain functionalities.
• Users can connect any rich text highlight to any database entry.
• Highlights should be visible in any Fibery view.

Creating References:

• Creating a reference is akin to the Link to Entity feature, with some distinctions.
• Users can create an open-ended reference and connect it later.

URD Management:

• URDs require separate management in the Left menu, managed by Admin.
• Multiple URDs can be created with different settings.

Views, Filters, Search, and Sort:

• Entity View, Views with Highlighted text, and Reference view on the Target.
• Filters should cover the Reference Database fields.
• A Reference appears in search results based on certain criteria.

Formulas, Automation, and Reports:

• Formulas involve using a reference for various calculations.
• Reference is involved in Automations.
• Reports based on References.

Access Management:

• The System and User Reference Databases have access permissions.

Whiteboard Integration:

• To add a reference to the Whiteboard, the same process as adding other entities is followed.

Comments and Notifications:

• Comments are allowed in the reference and can be accessed from the Entity view.
• Notifications should be handled in conjunction with the Inbox feature.

URD vs SRD

• User Reference Database (URD):
  • Purpose: Store references between entities across different databases.
  • Features: Captures highlighted text, user-defined attributes, and offers richer context for references.
• System Reference Database (SRD):
  • Role: Houses traditional “Link to Entity” records, allowing links within and across databases.

Migration from SRD to URD:

• Process: On creating a URD, entries from SRD can be optionally transferred. Later, links can be migrated by matching source-target pairs.

---

Related potential features/requests/forumtopics

Which are currently considered for implementation?

User Experience & Interface:

• Alternative Pane Solutions: Introduce new panes for features like backlinks, outgoing links, tags, and file searches.
• Tooltip Popup for Inline Entity Reference: Introduce tooltips for inline entity references.
• Aliases for References: Allow references to have multiple names or labels, enhancing discoverability and flexibility.
• Content Folding in Rich Text Fields: Introduce the ability to collapse and expand reference sections within rich text fields, especially when they contain multiple references.

Functionality & Integration:

• Whiteboard Integration: Integrate references with a whiteboard for organizing references visually.
• Transclusion: essentially a form of reference where the content of one entity is displayed within another.
• Block-Level Linking: linking specific blocks or sections within a text, rather than entire entities.
• Daily Notes Integration: If Fibery introduced a daily note-like functionality, integrating it with the References feature could provide users with a chronological context, aiding in understanding the evolution of interactions and insights over time.
• GitHub Sync Enhancements: How could the references feature enhance the github sync functionality?

Data & Relationship Management:

• Polymorphic Relations: Introduce more flexible relationships between entities.
• Editing Transcluded Content: Ensure references are edited in their original context.
• Field Tokens & New References: Explore the role of field tokens in the new references.
• Bulk Editing and Management of References: Implement features that allow users to edit, delete, or modify multiple references at once.
• Reference Aliases: Implement a system where a reference can be recognized under different terms or labels.
• Transclusion Enhancements: Dive deeper into the potential of transclusion in relation to the References feature, especially in displaying content from one entity within another.

Visualization & Analysis:

• Enhanced Relationship Visualization for References: Integrate visual cues, such as color codes, to represent different types of references and their relationships.
• Analytics and Insights on References: Provide analytics on the usage of references.
• Graph View for References: Consider the growing demand for a graph view as references become more intricate and numerous.
• References in Relation Graphs: When implementing relation graphs, explore how references play a role in that.

Development & Feedback:

• Extending references features to relations: Which features of references are interesting to implement for the standard relations? Such as fields.
• Plugin and Extension Support for References: Support the Fibery community to develop plugins or extensions that enhance or add functionalities to the References feature.

Data Storage & Organization:

• Saving Highlighted Text in Reference Database: Implement a feature to save highlighted text in a reference database in markdown format.
• Organizing References Chronologically: Work on organizing reference entries in a database.
• Queries and Dynamic Display: Explore the potential of queries to dynamically display or list references inline based on certain criteria.

Collaboration & Access Management:

• Collaborative Editing and Notifications: Enable multiple users to collaboratively edit a reference and receive notifications about changes made by others.
• Customizable Access Management for References: Provide advanced access management capabilities.

Enhancements & Improvements:

• relation/reference confusion: Address the disconnect between inline references and relationships.
• Reference Templates: Allow users to create templates for references.
• Mobile and Tablet Support for References: Ensure that the References feature is optimized and user-friendly on mobile and tablet devices.

[kalman]

Thank you for the effort to pull this together. this was a great exercise for me. I created four categories:

– clarification is needed, I cannot answer
– yes it will come
– it will come later
– not part of the feature

User Experience & Interface:

• Alternative Pane Solutions: Introduce new panes for features like backlinks, outgoing links, tags, and file searches. – please clarify this.
• Tooltip Popup for Inline Entity Reference: Introduce tooltips for inline entity references. – this is a tooltip that appears when you hover your mouse on a highlighted text of a URD Reference.
• Aliases for References: Allow references to have multiple names or labels, enhancing discoverability and flexibility. – these are the database fields of the URD entities
• Content Folding in Rich Text Fields: Introduce the ability to collapse and expand reference sections within rich text fields, especially when they contain multiple references. – but there is nothing about multiple references.

Functionality & Integration:

• Whiteboard Integration: Integrate references with a whiteboard for organizing references visually. – Because URD is a database, they will be accessible on the whiteboard, the new feature is that we have to show the highlighted text, this functionality does not exist today (show rich text field in whiteboard).
• Transclusion: essentially a form of reference where the content of one entity is displayed within another. – this is a different feature and not tightly related, although showing a referenced text from the URD is in the backlog.
• Block-Level Linking: linking specific blocks or sections within a text rather than entire entities. – this is a different feature.
• Daily Notes Integration: Please help me understand why you cannot create today’s daily note functionality. please clarify this feature.
• GitHub Sync Enhancements: How could the references feature enhance the github sync functionality? – what are you looking for here? which functionality? please help me to understand it.

Data & Relationship Management:

• Polymorphic Relations: Introduce more flexible relationships between entities. – this is a risky feature. we are testing this nowadays, and we have to solve it somehow in the database core or on the UI.
• Editing Transcluded Content: Ensure references are edited in their original context. Yes, you cannot edit referenced text, but still unclear how we will manage this sync.
• Field Tokens & New References: Explore the role of field tokens in the new references. – please give me an example. do you mean that one of the URD entity field can be inserted to the text?
• Bulk Editing and Management of References: Implement features that allow users to edit, delete, or modify multiple references at once. – yes, but this is not separate feature, since we already have functionality for this: automation.
• Reference Aliases: Implement a system where a reference can be recognized under different terms or labels. – please explain this.
• Transclusion Enhancements: Dive deeper into the potential of transclusion in relation to the References feature, especially in displaying content from one entity within another. – yes, if you mean to include a highlighted text of a URD entity

Visualization & Analysis:

• Enhanced Relationship Visualization for References: Integrate visual cues, such as color codes, to represent different types of references and their relationships. – the visual representation of the highlighted text is connected to the target database color, not sure what else you think of.
• Analytics and Insights on References: Provide analytics on the usage of references. - yes, but this is not a separate feature. This is the visual reports of any database.
• Graph View for References: Consider the growing demand for a graph view as references become more intricate and numerous. – please explain the differences between this and the above.
• References in Relation Graphs: When implementing relation graphs, explore how references play a role in that. – please explain this

Development & Feedback:

• Extending references features to relations: Which features of references are interesting to implement for the standard relations? Such as fields. – we have a plan for that but this is in the backlog, not in this year.
• Plugin and Extension Support for References: Support the Fibery community to develop plugins or extensions that enhance or add functionalities to the References feature. – we provide API but what else you need for a plugin?

Data Storage & Organization:

• Saving Highlighted Text in Reference Database: Implement a feature to save highlighted text in a reference database in markdown format. – we will save the highlighted reference into two fields: the rich text and a shadow field in markdown to show in different views.
• Organizing References Chronologically: Work on organizing reference entries in a database. – this is not a new functionality, this is possible because we create a database from references
• Queries and Dynamic Display: Explore the potential of queries to dynamically display or list references inline based on certain criteria. – this is not a new functionality, this is possible because we create a database from references

Collaboration & Access Management:

• Collaborative Editing and Notifications: Enable multiple users to collaboratively edit a reference and receive notifications about changes made by others. – not sure how it will work, this is an uncharted area
• Customizable Access Management for References: Provide advanced access management capabilities. – we have to wait entity permission to figure this out.

Enhancements & Improvements:

• relation/reference confusion: Address the disconnect between inline references and relationships. – please explain this
• Reference Templates: Allow users to create templates for references. – it will come
• Mobile and Tablet Support for References: Ensure that the References feature is optimized and user-friendly on mobile and tablet devices. – it will work, but not optimized for mobile

I hope it helps.

[YvetteLans]

Hi @kalman,

We have the following set-up for our second brain in the workspace.

Database for

• Note
• Tag
• Theme
• Person / source

A note can have multiple tags, themes, persons etc.

Currently we use tags instead of references.

• Example: the tag ‘tools’ → so a note with tag tool
• Example: the tag 'quotes → so a note with tag quotes

So the basis of everything in this example is a note.

Really like the new reference set-up, but I’m also curious about the following use case.

• Let’s say I have a note and in that note there is great quote → I want to highlight the text and add reference ‘quote’
  

  

• But sometimes I create the quote as an entity (so Name = the quote) → then I also want to link that note to ‘quote’
  

So sometimes the reference is a highlighted text, but I think there are multiple use cases where you want to link an entity to the same ‘quote’ entity (but then without highlighting a text).

Whereby the end goal is "I want to see everything that is related to “quote” (either notes or highlights within a note).

Would that be also possible in the future or would you recommend to keep it seperate?

A second brain is difficult to migrate so if we can anticipate now, we’d like to do so before we put it into full use.

[kalman]

Hi @YvetteLans, thanks for bringing up this question.

We’ve talked about this a lot before, and basically, the answer is YES, we need to support that use case somehow.

However, we don’t have a concrete plan for how to do this yet because of the following open questions:

1. How to tell the difference between a reference and a relation between two entities. This is tricky, and I’m worried many users will have trouble with it.
2. Whether the reference should be to the entire database entity or just its rich text field. This is interesting because an entity can have multiple rich text fields.
3. What to do if there is no referenced text. Maybe we could use an AI-generated summary in that case?

In your use case, what would be the best solution/answer for these questions?

[YvetteLans]

Great questions

Was writing down answers but bumped into a new insight that I first want to share.

What if…

• You use the current ‘Link to an entity’ functionality to create the reference. So user selects text in a rich text field and press button or Ctrl + L.
• User then selects the entity that they want to link to (or create a new entity like you currently can)
• In my example I have a tag database with ‘Quote’ so user chooses to link to the existing entity ‘Quote’.
  

The entity ‘Quote’ already has a relation to ‘Note’. So if a user has a note entity that needs to be linked to ‘Quote’ directly without a reference (because there is no rich text field for example), then they can set the relation like it is now.

So this means that you’ll leave everything as it is now, with the explanation that a user has full flexibility to link a highlighted text to whatever entity they want. If they want to create a ‘Label’ database with labels like ‘Insight’ / ‘Dopamine’ etc., they can do that They then simply link the text to the applicable label.

We currently have two types of reference

Let’s say we store the first one as ‘highlight’ → so the text itself is stored.

In the current situation, it looks like this when you’ve linked a text to an entity.

Would be great if highlight can be an entity. And if it all text can be shown also mentioned here.

Maybe there should be a visual difference since it’s not a complete normal database. Since the highlight + source should be read only I assume. But further I hope it can act more or less the same → create views, use it in automations etc.

Full ‘Quote’ page
Since the tag ‘Quote’ also has a relation with note (and maybe other stuff), the full screen of the user looks like this

Note: ideally you maybe want to combine all references visually; either a highlighted text or a Link to an entity.

@kalman

How to tell the difference between a reference and a relation between two entities. This is tricky, and I’m worried many users will have trouble with it.

I understand your worries. In our workspace we make everything ‘dummy proof’ → each entity has an internal Wiki to avoid questions and explain what they need to do. When they click on it, it will show on the right.

Keeping that in mind → maybe Fibery can also help the user while linking an entity. There is enough space on the screen to show an example / visual / image.

So explanation can be that:

• A reference is a text that links to a database entity. It will not create a relation with that entity.
  

• If you want to link one database entity to another database entity, you’ll need to fill in the relation instead.
  

@kalman

Whether the reference should be to the entire database entity or just its rich text field. This is interesting because an entity can have multiple rich text fields.

Because of my new insight, I think that a reference can only be created in a rich text field by using the link to entity functionality (Ctrl L)

@kalman

What to do if there is no referenced text. Maybe we could use an AI-generated summary in that case?

In that case it’s more helpful to link the database entity, since there is no rich text.

Disclaimer
Maybe I think way too simple about this I’m sure that there are many use cases where my solution isn’t sufficient. It looks really awesome what you want to achieve; that’s way more than I mention in this reply.

But if you do like this more simple approach and have more questions, please let me know. Happy to answer them in text or in a call

[kalman]

thanks for the insightful reply

@YvetteLans

Maybe there should be a visual difference since it’s not a complete normal database. Since the highlight + source should be read only I assume. But further I hope it can act more or less the same → create views, use it in automations etc.

It will be a full-featured database with some limitations like source and targets, but you can create views, and use formulas and automation.

@YvetteLans

Note: ideally you maybe want to combine all references visually; either a highlighted text or a Link to an entity.

I think you nailed the problem. If they are references as well, they should be in the same reference database. Otherwise, it will be difficult to harvest them from different places.

One idea: On the entity (or on the rich text field), there should be a “Reference to Entity” button that opens the same UI that we will use for the User Reference Entity. This can have two extra options:

• Leave the highlighted text field empty or
• Create a summary with AI from the full text (and show it this is not a proper highlight).

@YvetteLans

In that case it’s more helpful to link the database entity, since there is no rich text.

I think using Relation for Reference could cause confusion, but I do believe that having a database for relations is also a very important use case. But this is a different feature.

@YvetteLans

But if you do like this more simple approach and have more questions, please let me know. Happy to answer them in text or in a call

I am grateful for this conversation. I appreciate your and everyone else’s perspectives, which generate new ideas. I am happy to continue the discussion here or on a call because these conversations help us to create better products for all of you and for all of us.

[YvetteLans]

@kalman

I think using Relation for Reference could cause confusion

Yes I think you are right!

I was hoping that we could somehow use database entity for

• when you want to create a text reference (highlight text in a rich text field)
  

• as well as when you want to link an entity to that reference database entity → so then user can also link a Readwise entity directly to Quote (without highlighting a text at all)

@kalman

One idea: On the entity (or on the rich text field), there should be a “Reference to Entity” button that opens the same UI that we will use for the User Reference Entity. This can have two extra options:

• Leave the highlighted text field empty or
• Create a summary with AI from the full text (and show it this is not a proper highlight).

But I think this is a better approach!