Skip to content
Fluid Attacks Development Portal

Database Design

Database schema definition

Since the design for the database is among the interest of several components of universe, we have decided to place the sources in the common component. This way, the schema can be shared among all components that need to interact with the database.

Following the NoSQL design principles, we have facets which represent different entities in the database. Each facet is a separate cue file that contains the schema for a specific entity, even though these items are in the same table. Some tools like NoSQL Workbench for DynamoDB can help you visualize the schema in a more friendly way.

The database schema definition itself can be found in common/schemas/database_schema/database_schema.cue. This contains the most basic constraints for the schema.

Tables

Integrates requires two dynamodb tables: integrates_vms and integrates_vms_historic. You may find its definitions in common/schemas/tables and facets contents in common/schemas/tables/<table name>/facets.

As best practice, please keep a separate cue package for every facet.

Format your changes with cue fmt

When applying changes to cue files, please apply formatter:

Terminal window
m . /common/schemas/fmt

Export changes to database-design

For updating changes made to the schema or tables contents to common/schemas/database-design.json simply execute:

Terminal window
m . /common/schemas/export

This job will fail if the produced output is not up to date with the previous state of the database-design and will be successfull if there are no changes. This behaviour is intentional so that CI can catch differences between the current state of the cue schemas and the final content.

Types generation

Since cue is now our source of truth for the database schema, we can generate types for the database schema for avoiding double maintenance. By now, the produced types are python TypedDict which should model how the dynamodb item looks like right before being inserted or right after being retrieved. Once generated, types as available in the fluidattacks-core external package.

For producing the types, run:

Terminal window
m . /common/schemas/gen_types

This job will fail if types are not up to date. The output will be placed according to the table name in common/utils/pypi/fluidattacks-core/fluidattacks_core/types/dynamo/items.py.

Run all

For convenience, you can run all the previous commands fmt, export and gen_types in one go:

Terminal window
m . /common/schemas

Validate current schema without changing database-design

If you want to run schema validations against your running database instance, run:

Terminal window
m . /common/schemas/val $stage $table_name

Valid stages are dev and prod. For example:

Terminal window
m . /common/schemas/val dev integrates_vms

This will help you find items which are non compliant with the desired schema.

Learn more about the cue language in https://cuelang.org/.

If vscode is your IDE, this extension is recommended: https://marketplace.visualstudio.com/items?itemName=asdine.cue

A vim plugin for syntax highlighting is also available at: https://github.com/jjo/vim-cue