Entity Migration Script: Hierarchical Data Transfer Between Databases

Purpose

This script automates the migration of entities between two Fibery databases, ensuring the preservation of key attributes (name, description), hierarchical relationships (parent-child), and specific relationships (tags, parent projects). It supports structured data transfer with depth control to maintain organizational integrity.

Use cases:

  • Project Management Reorganization: Migrates projects, tasks, and sub-tasks to new workspace, preserving hierarchy and details.
  • Content Management Update: Moves articles, drafts, and edits to an optimized database, maintaining editorial workflow and organization.
  • Team Reassignment: Reorganizes team projects and tasks, keeping project associations and categorizations intact.
  • CRM Update: Transfers customer records and interactions to a new CRM structure, preserving customer hierarchies and segmentation.

How to adapt the script to your needs

  1. Identify Source and Target Databases: Determine the databases you’re migrating from and to.
  2. Update Database Constants: In the script’s initial constants section, replace SOURCE_DB with your source database name and TARGET_DB with your target database name.
  3. Match Field Names: Adjust the field name constants (SOURCE_NAME_FIELD, TARGET_NAME_FIELD, etc.) to correspond with the field names in your source and target databases.
  4. Parent Project and Tags Handling: Note that the script assumes ‘Parent Project’ and ‘Tags’ already exist in the target database. It will link the migrated entities to these existing items without creating new ones.

The script

// Configuration constants for source and target databases
const SOURCE_DB = 'Space A/Database X';
const TARGET_DB = 'Space B/Database Y';

// Configuration constants for field names
const SOURCE_NAME_FIELD = 'Name';
const TARGET_NAME_FIELD = 'Name';
const SOURCE_DESCRIPTION_FIELD = 'Description';
const TARGET_DESCRIPTION_FIELD = 'Description';
const SOURCE_SUB_PAGES_FIELD = 'Sub Pages';
const TARGET_SUB_PAGES_FIELD = 'Sub Pages';
const SOURCE_PARENT_PAGE_FIELD = 'Parent Page';
const TARGET_PARENT_PAGE_FIELD = 'Parent Page';
const SOURCE_PARENT_PROJECT_FIELD = 'Parent Project'; 
const TARGET_PARENT_PROJECT_FIELD = 'Parent Project';
const SOURCE_TAGS_FIELD = 'Tags'; 
const TARGET_TAGS_FIELD = 'Tags';

const MAX_DEPTH = 5; // Maximum depth for conversion

const fibery = context.getService('fibery');

// Function to fetch document content
async function fetchDocumentContent(documentSecret) {
    if (!documentSecret) {
        return null;
    }
    return await fibery.getDocumentContent(documentSecret, 'md');
}

// Function to convert entities from the source database to the target database, with depth tracking
async function convertEntity(entityId, parentPageId = null, currentDepth = 1) {
    if (currentDepth > MAX_DEPTH) {
        return; // Stop recursion if maximum depth is reached
    }
    try {
        const entity = await fibery.getEntityById(SOURCE_DB, entityId, [
            SOURCE_NAME_FIELD, SOURCE_DESCRIPTION_FIELD, SOURCE_SUB_PAGES_FIELD, SOURCE_PARENT_PROJECT_FIELD, SOURCE_TAGS_FIELD
        ]);
        let descriptionContent = null;
        if (entity[SOURCE_DESCRIPTION_FIELD] && entity[SOURCE_DESCRIPTION_FIELD].Secret) {
            descriptionContent = await fetchDocumentContent(entity[SOURCE_DESCRIPTION_FIELD].Secret);
        }

        const newEntityData = { 
            [TARGET_NAME_FIELD]: entity[SOURCE_NAME_FIELD],
            [TARGET_PARENT_PROJECT_FIELD]: entity[SOURCE_PARENT_PROJECT_FIELD] ? entity[SOURCE_PARENT_PROJECT_FIELD].Id : null,
        };
        const newEntity = await fibery.createEntity(TARGET_DB, newEntityData);

        // Set the 'Description' content for the new entity
        if (descriptionContent) {
            await fibery.setDocumentContent(newEntity[TARGET_DESCRIPTION_FIELD].Secret, descriptionContent, 'md');
        }

        // Link the new entity to its parent, if applicable
        if (parentPageId) {
            await fibery.updateEntity(TARGET_DB, newEntity.id, { [TARGET_PARENT_PAGE_FIELD]: parentPageId });
        }

        // Handling Tags through addCollectionItem for each tag
        if (entity[SOURCE_TAGS_FIELD] && entity[SOURCE_TAGS_FIELD].length > 0) {
            for (const tag of entity[SOURCE_TAGS_FIELD]) {
                await fibery.addCollectionItem(TARGET_DB, newEntity.id, TARGET_TAGS_FIELD, tag.Id);
            }
        }

        // Recursively convert and link sub-entities, increasing depth
        if (entity[SOURCE_SUB_PAGES_FIELD] && entity[SOURCE_SUB_PAGES_FIELD].length > 0) {
            for (const subEntity of entity[SOURCE_SUB_PAGES_FIELD]) {
                await convertEntity(subEntity.Id, newEntity.id, currentDepth + 1);
            }
        }
    } catch (error) {
        console.error('Error converting entity:', error);
    }
}

// Main function to initiate the conversion process based on context entities
async function convertEntities() {
    try {
        for (const currentEntity of args.currentEntities) {
            await convertEntity(currentEntity.id); // Start with depth 1
        }
    } catch (error) {
        console.error('Error in converting entities:', error);
    }
}

// Execute the conversion process
await convertEntities();

Explanation

  1. Define Configuration Constants:
  • Specifies source and target databases, field names, and maximum conversion depth, allowing easy adaptation of the script for different migrations.
  1. Fetch Document Content (fetchDocumentContent):
  • Uses fibery.getDocumentContent(documentSecret, 'md') to retrieve the content of document fields based on their secret. This operation is crucial for copying the ‘Description’ field content from source to target entities.
  1. Convert Entity (convertEntity):
  • Fetch Entity Data: Retrieves the current entity’s data from the source database using fibery.getEntityById(SOURCE_DB, entityId, fields), including name, description, sub-entity relationships, and any specified relationship fields like ‘Parent Project’ and ‘Tags’.
  • Create New Entity: Creates an entity in the target database with the same ‘Name’ (and potentially other simple fields) using fibery.createEntity(TARGET_DB, newEntityData).
  • Set Description Content: Updates the ‘Description’ of the newly created entity with the content fetched from the source entity using fibery.setDocumentContent(newEntity[TARGET_DESCRIPTION_FIELD].Secret, descriptionContent, 'md').
  • Handle Parent-Child Relationships: For hierarchical structures, recursively calls itself to convert and link sub-entities, managing depth to avoid exceeding MAX_DEPTH.
  • Update Relationship Fields: Specifically for ‘Parent Project’, directly sets the relationship by ID. For ‘Tags’ (a many-to-many relationship), iterates over each tag and adds it to the new entity using fibery.addCollectionItem(TARGET_DB, newEntity.id, TARGET_TAGS_FIELD, tag.Id).
  1. Initiate Conversion Process (convertEntities):
  • Iterates over entities provided in args.currentEntities and starts their conversion by calling convertEntity for each, marking the entry point for the script’s execution.
  1. Execute Conversion Process:
  • The final step where await convertEntities() is called to begin the migration based on the entities and relationships defined by the configuration constants.

Summary of Methods Used:

  • fibery.getEntityById: Fetches detailed information about an entity, including specified fields.
  • fibery.createEntity: Creates a new entity in the target database with specified field values.
  • fibery.setDocumentContent: Sets the content of a document-type field for a newly created entity.
  • fibery.addCollectionItem: Adds items to a collection-type field, used for managing many-to-many relationships like ‘Tags’.
  • fibery.updateEntity: Updates simple fields of an entity, used for setting single-select relationships not covered by addCollectionItem.

If you want to remove fields from this example script

For example, if you want to remove the ‘Tags’ field conversion form the script because your databases don’t have that field:

  1. Remove Constants:
  • Delete the SOURCE_TAGS_FIELD and TARGET_TAGS_FIELD constants from the initial constants section of the script.
  • Example:

javascriptCopy code

// Remove these lines
const SOURCE_TAGS_FIELD = 'Tags';
const TARGET_TAGS_FIELD = 'Tags';
  1. Remove Handling Code:
  • Within the convertEntity function, delete the code block that handles the ‘Tags’ field, specifically the loop that adds tags to the target entity using fibery.addCollectionItem.
  • Example:

javascriptCopy code

// Delete this code block
if (entity[SOURCE_TAGS_FIELD] && entity[SOURCE_TAGS_FIELD].length > 0) {
    for (const tag of entity[SOURCE_TAGS_FIELD]) {
        await fibery.addCollectionItem(TARGET_DB, newEntity.id, TARGET_TAGS_FIELD, tag.Id);
    }
}
3 Likes

It’s just for recursive 1:n self-relations, right?

FYI, you can achieve thing manually by duplicating the entities using copy+paste and then duplicating the relationships using copy+paste.

firefox_JwoEm8hj3u

3 Likes

Yes, it’s for recursive one-to-many self-relations.

Yes you are right about copy paste in a table, thank you for mentioning that.
I think the script is still useful to automate this process for larger datasets or more complex entity structures.

@Chr1sG The copy pasting using table views does not work for rich text fields, right?

Correct. The rich text snippets are read-only