Editing registry data model table structure in the web code editor

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. General description

Development of the registry regulations includes the development of a registry data model. The administrative portal provides functionality for viewing the registry data model. It is necessary to provide the ability to edit the data model of the registry regulations.

2. Problem description

The admin portal can view the status of the registry regulations data model. The existing approach involves committing changes to Gerrit to the corresponding MR candidate version while developing the registry regulations data model.

3. Actors

  • Registry regulations developer

4. Glossary

  • LS: language server

5. Functional scenarios

  • Changing the data structure in the registry regulations data model.

  • Saving the changes made to the candidate version of the registry regulations.

  • Viewing error notifications in the edit window of the registry regulations data model table structure.

  • Auto-prompts and auto-updates when editing liquibase changelog xml.

  • Checking liquibase configuration according to liquibase and DDM xsd.

6. Existing solution design

6.1. Reviewing and editing the data model

data model edit current design

6.2. Interacting with Language Server

ls current design

6.3. General principles

  • Changes to the data model of the registry regulations are made by making changes directly to Gerrit.

  • The source file for data model deployment is data-model/main-liquibase.xml.

  • The structure of files on the file system is not clearly regulated. Liquibase changeset’s with instructions for creating data model tables can technically be found in any file in the file system.

  • ddm-language-server only supports the Groovy language.

7. Technical solution design

edit data model target design

7.1. General principles

  • The data structure of the data model is only edited in the createTables.xml file.

  • registry-regulation-management receives and modifies createTables.xml using corresponding RestAPI methods.

  • Language Server supports Groovy and XML.

  • In the list of statuses of the candidate version components, the change in the database structure is displayed as a single item without a list of detailed changes for each data model table.

7.2. Changing the registry regulations data model description contract

To ensure that the structure of the data model tables of the registry regulations is edited, it is necessary to change the approach to organizing the structure of liquibase configuration files as follows:

  • All operations on creation or changing the structure of database tables must be stored in the data-model/createTables.xml file.

  • data-model/createTables.xml must be explicitly included in the list of files for deployment in the data-model/main-liquibase.xml configuration.

7.2.1. Migrating existing registries

To ensure the compatibility of the registry regulations file structure, it is necessary to verify the above rules, and change the file structure (without changing the liquibase changeset content), if necessary.

If createTables.xml is not available, the admin portal does not provide the ability to edit the liquibase configuration using the Web UI. The performance of the existing configuration of the registry regulations does not change.

7.3. Distributing perspectives for editing and viewing the data model

To edit the registry regulations data model, you have to create a new editing perspective. All operations associated with viewing the state of the data model (present and future) are allocated in the view perspective.

7.4. Sequence diagram

edit-data-model edit-data-model-tables-sequence

7.5. Deployment of XML LS

  • To deploy XML Language Server, use the lemminx language server.

    This LS is published in the maven central repository (there is no need to place the code of this component in your gerrit).

  • It is necessary to connect the above artifact and use the org.eclipse.lemminx.XMLLanguageServer LS implementation in the ddm-language-server module.

  • In accordance with the LS specification, the URL for creating websocket must be /xml.

7.6. Working with DDM XSD

To provide hints in full scope when editing XML documents (displaying field descriptions from xsd schemas as hoover messages, auto-completion, etc.), it is necessary to provide access to the ddm-language-server to load the corresponding xsd schemas using the specified URL in the XSD document.

All required XSD for editing XML Liquibase documents must be placed in platform nexus.

XSD required to edit liquibase scripts:

To ensure compatibility of the liquibase configurations with ddm-language-server, it is necessary to migrate the URI for the http://www.liquibase.org/xml/ns/dbchangelog namespace. The URI must point to the corresponding xsd published in platform nexus.

7.7. RestAPI for liquibase configurations

It is necessary to extend the existing registry-regulation-management RestAPI by adding the methods for:

  • Getting the liquibase configuration of the data model table structure for the candidate version.

  • Getting the liquibase configuration of the data model table structure for master versions.

  • Changing the liquibase configuration of the data model table structure for the candidate version.

7.7.1. Getting the liquibase configuration of the data model tables

Request:

GET /versions/candidates/{versionCandidateId}/data-model/tables
GET /versions/master/data-model/tables

Response structure:

Content-type=text/xml 
<?xml version="1.1" encoding="UTF-8" standalone="no"?>

<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.5.xsd
        http://www.liquibase.org/xml/ns/dbchangelog-ext https://nexus-public-mdtu-ddm-edp-cicd.apps.cicd2.mdtu-ddm.projects.epam.com/repository/extensions/com/epam/digital/data/platform/liquibase-ext-schema/latest/liquibase-ext-schema-latest.xsd">

    ...
</databaseChangeLog>

7.7.2. Changing the liquibase configuration of the data model tables

Request:

PUT /versions/candidates/{versionCandidateId}/data-model/tables
Content-type=text/xml

Body:

<?xml version="1.1" encoding="UTF-8" standalone="no"?>

<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.5.xsd
        http://www.liquibase.org/xml/ns/dbchangelog-ext https://nexus-public-mdtu-ddm-edp-cicd.apps.cicd2.mdtu-ddm.projects.epam.com/repository/extensions/com/epam/digital/data/platform/liquibase-ext-schema/latest/liquibase-ext-schema-latest.xsd">
    ...
</databaseChangeLog>

7.8. Receiving and displaying changes in the status view of the registry regulations components

The work with the registry database structure is performed by editing only one resource in terms of RestAPI /versions/candidates/{versionCandidateId}/data-model/tables.

Therefore, it is necessary to operate with one single item called "Database table structure" in the list of the registry regulations component states.

This item can have the only Modified state if any changes were made to the data-model/createTables.xml file through the admin portal or directly by adding of a patchset in Gerrit to the corresponding MR to the candidate version.

8. High-level development plan

8.1. Required expertise

  • DevOps

  • BE

  • FE

  • QA/AQA

  • One TeamLead or One QALead

8.2. DevOps

  • Configure Kong for working with XML LS.

8.3. Backend

  • Extend RestAPI for getting and saving liquibase configurations (for the candidate versions and master versions). Add RestAPI Optimistic locking headers to the /versions/candidates/{versionCandidateId}/data-model/tables endpoints.

  • Connect XML lemminx LS. Extend ddm-language-server to work with the /xml websocket endpoint.

  • Add the information about changes in the database structure of the data model to the /versions/candidates/{versionCandidateId}/changes RestAPI endpoint.

  • Publish dbchangelog-4.5.xsd on platform nexus.

8.4. Frontend

  • Create a perspective view of the data model state and a perspective view for editing data model tables.

  • Integrate monaco editor into the perspective for editing data model tables. Configure interaction with XML LS.

  • Provide for displaying and storage of liquibase configuration of data model tables (integration with the /versions/candidates/{versionCandidateId}/data-model/tables endpoints). Provide for working with the RestAPI Optimistic Locking mechanism.

  • Add the display of information about changes in the data model table structure to the candidate version overview.

8.5. One TeamLead or One QALead

  • Develop migration instructions for existing registries. Publish instructions, organize knowledge transfer with the DevOps command.