View the list of registry data model tables in the read mode for candidate versions

🌐 This document is available in both English and Ukrainian. Use the language toggle in the top right corner to switch between versions.

1. General description

The development of registry regulations involves the creation of a registry data model. The administrative portal provides functionality for viewing the list of tables of the registry data model and their structure.

2. Problem description

The admin portal allows you to view the list of data model tables only for the master version of registry regulations. This functionality allows you to get a list of tables of regulations, which is deployed in the production environment, but does not allow viewing the registry regulations data model tables of a candidate version.

  • When developing a data model of the registry regulations, it is necessary to use the list of data model tables described in the liquibase candidate version configuration.

  • When registry developers edit a data model by pushing changes to gerrit, it is necessary to display up-to-date information with a list of data model tables in the administrative portal.

  • Undefined state of changes in a data model until the registry regulations are published.

3. Actors

  • Registry regulations developer

4. Glossary

  • rrm: registry-regulation-management backend service

  • DataModelSnapshot - a file for storing the state of the registry data model.

  • DataModelSnapshot files: A set of files (one file per table) showing the state of the registry regulations data model for each of the tables.

5. Functional scenarios

  • Review the current state of the registry regulations data model (a list of tables) being developed (within the framework of a candidate version).

  • Review the value of the "subjectivity" attribute in the list of tables.

  • Get the result of checking whether the data model can be successfully deployed.

6. Design of the existing solution

current design

  • A database is deployed in 2 stages:

    • * OKD run-db-scripts job that runs during Citus deployment. It deploys preconditions to deploy the actual detailed behavior out of the scope data model.

    • * Jenkins pipeline deployment of the registry regulations, namely the <registry-name>-data-model job. It deploys the registry regulations model.

  • registry-regulations-management reads and stores the database structure in the xref: arch: architecture/registry/administrative/regulation-management/admin-portal/data-model/admin-portal-data-model-json-schema -description.adoc [DataModelSnapshot] file separately for each table. The process of creating files occurs when starting the registry-regulations-management service once.

  • Rest API through DataModelSnapshot access services reads the list of tables by reading the list of DataModelSnapshot table file files.

7. General principles

  • Changes to the data model of the registry regulation are made by amending the corresponding liquibase files.

  • The structure of liquibase registry rules files in the file system (and in Git) does not change

  • Each candidate version uses its dedicated database to deploy the data model.

  • The reference database is used to create temporary databases for candidate versions of the registry regulations. Reference database does not contain the registry data.

  • The registry regulations deployment subsystem (Jenkins) creates a database structure by deploying liquibase of the registry regulation configurations.

  • registry-regulation-management reads the deployment state of the registry regulations (liquibase deployment) from Gerrit of the corresponding MR. The execution state of the corresponding Jenkins job is displayed in Gerrit MR as a candidate version using the specific label (Verified + 1).

8. Solution technical design

target design

  • OKD run-db-script job creates a reference database when creating and deploying a database according to the settings in the citus repository. The reference database must contain only the database structure without any data from the registry.

  • The registry regulations deployment subsystem deploys the registry regulations for candidate versions after making any changes to the corresponding Gerrit MR.

  • The registry regulations deployment subsystem creates temporary databases for each of the candidate versions in work.

  • Registry regulations deployment subsystem deploys the registry regulations data model by using liquibase configurations from the registry regulations.

  • rrm reads the data structure from the temporary database and uses the DataModelSnapshot json data schema for communication (at the RestAPI level).

  • The registry database is used for reading the database structure for the master version. Temporary files are not used in the file system in the DataModelSnapshot format.

The reference and temporary databases must be created only in DEV environment.

8.1. Sequence diagrams

Getting a list of tables of a candidate version
Figure 1. Getting a list of tables of a candidate version

8.2. Extension of the registry regulation deployment subsystem:

It is necessary to extend the routine jenkins with a new deployment mechanism:

  • When creating a new MR for the candidate version, create and deploy a temporary database for the corresponding candidate version.

  • When saving changes to the candidate version (new patchset in gerrit), rebuild the temporary database for the corresponding candidate version and deploy liquibase changelog from the corresponding master version, and then from the candidate version.

  • The process of deploying the data model of the registry regulations uses a mechanism to check for changes in the registry regulations publishing pipeline (namely the get-changes step). Temporary databases are deployed only if there are new changes in the data model and when creating a candidate version.

A single jenkins codereview pipeline is responsible for the above actions.
Only one temporary database deployment process can be run at a time within one candidate version.

8.2.1. Description of work with temporary databases

Logic of work:

  • Create a temporary database for the candidate version using the reference database.

CREATE DATABASE [registry-dev-<vcid>] WITH TEMPLATE registry-template OWNER [our owner user?];
registry-template is the name of the reference database received as a result of OKD run-db-script-job. registry-dev-<vcid> is the template of the temporary database name for the candidate version.

  • Deployment of the liquibase structure from the corresponding registry regulations version (the state of the master version from which the candidate version was created, or to which the rebase operation was last performed).

  • Deploy the current liquibase structure from the candidate version.

For the access of registry-regulation-management to temporary databases, you must create a separate registry_dev_role user. The user must have the following rights:

  • READ temporary databases

  • READ the reference database

8.2.2. Periodic reconciliation process

The reconciliation process is required for deleting out-of-date temporary databases for candidate versions (versions of candidates that were integrated into the master version or those that were deleted without integration). You must create a separate jenkins pipeline for the reconciliation process.

8.2.3. Reconciliation process logics

  • Get a list of candidate versions in gerrit.

  • Get a list of temporary databases for candidate versions in the database.

  • Delete temporary databases of candidate versions having no open MRs in gerrit.

8.2.4. Schedule settings

  • Calling the reconciliation process can be selected individually and configured at the level of the pipeline’s jenkins trigger.

8.3. Using AbstractRoutingDatasource to access temporary databases

8.3.1. Description of the problem

To access the temporary databases, you must use the DataSource bean for each temporary database. The existing code that provides work with the master version database uses Spring bean of the DataSource type. To reuse the mechanism for reading the database structure in the DataModelSnapshot file, it is suggested to use the AbstractRoutingDatasource mechanism, which allows you to create a DataSource dynamically, depending on the candidate version with which you are currently working.

8.4. RestAPI

The current RestAPI is extended by the ability to obtain a list of tables of the registry regulations data model. The request structure is the same as for the corresponding queries for the master version ('GET/versions/master/tables').

8.4.1. Example of requests and responses to receive a list of data model tables of the candidate version

Request:

GET /versions/candidates/{versionCandidateId}/tables/

Response structure:

[
    {"name":"Table 1","description":"Table 1 description","objectReference":true},
    {"name":"Table 2","description":"Table 2 description","objectReference":true},
    ...
    {"name":"Table n","description":"Table 2 description","objectReference":true},
]
You have to use the same data structures and error codes for the requests and responses used to get the list of data model tables in the master version.

9. Extension of the candidate version status display unit with information on the status of the temporary database deployment process

During the deployment of temporary databases, the health of the existing liquibase changelog configuration of the registry regulations is also checked. If there are any problems during this process, you must pass this information to the client application to be displayed to the registry developer.

To get the deployment status information, it is necessary to read the Verified +1 label from the corresponding MR in Gerrit.

It is necessary to extend the VersionInfoDetailed (Data structure in RestAPI) with the information about the status of the registry regulations deployment to the temporary environment by adding a new type of validation to the Validation bean.

Change ResultValues to:

  • UNKNOWN: the deployment and validation process is running/did not run (no Verified label).

  • SUCCESS: the deployment and validation process is successful (Verified + 1).

  • FAILED: the deployment and validation process has failed (Verified -1).

Add to resultDetails the information (link) to the corresponding gerrit MR (Optional).

TODO: work out UI design

10. Extension of the env cleanup process

It is necessary to extend the existing environment cleanup mechanism by adding changes to the delete-registry stage. It is required to delete all temporary databases for candidate versions. Changes involve making appropriate instructions to remove temporary databases in CleanupRegistry.sql or creating a separate corresponding sql file with instructions. Deletion is possible using the database name mask registry-dev-*. It is also necessary to delete the reference database named registry-template.

11. High level development plan

11.1. Required expertise

  • DevOps

  • BE

  • FE

  • QA/AQA

  • DBA

11.2. DBA

  • Extend the database deployment mechanism in the citus repository to create a reference database.

11.3. DevOps

  • Create a user in the database for the registry-regulation-management service. Permissions:

    • READ the temporary databases.

  • Extend the cleanup env by removing temporary databases and reference databases.

  • Extend the registry regulations deployment subsystem with the deployment mechanism of the registry regulations liquibase configurations to temporary databases.

  • Add the reconciliation and cleanup mechanism between the list of versions of candidates and temporary databases.

  • Add the mechanism for publishing the DB temporary deployment status to corresponding MR in Gerrit.

11.4. Backend

  • Create a database reading mechanism for a given DataSource.

  • Create DataSource spring bean for each candidate version and master version separately.

  • Extend RestAPI to handle data model requests for candidate versions.

  • Delete the mechanism of using the file system to store the DataModelSnapshot files for the master version. Reading the structure of the master version must occur every time the request is processed at the RestAPI levels.

  • Add reading of the data model deployment state to the temporary database in GET /versions/candidates/{versionCandidateId}

11.5. Frontend

  • Add the display of the temporary database deployment status to the block with the results of the candidate version checking.

  • Create a page with a list of tables for candidate versions.