Skip to content
Fluid Attacks Development Portal

Database Design

Database schema definition

Following the NoSQL design principles, we have facets which represent different entities in the database. Each facet is a separate cue file that contains the schema for a specific entity, even though these items are in the same table. Some tools like NoSQL Workbench for DynamoDB can help you visualize the schema in a more friendly way.

The database schema definition itself can be found in integrates/schemas/database_schema/database_schema.cue. This contains the most basic constraints for the schema.

Tables

Integrates requires two dynamodb tables: integrates_vms and integrates_vms_historic. You may find its definitions in integrates/schemas/tables and facets contents in integrates/schemas/tables/<table name>/facets.

As best practice, please keep a separate cue package for every facet.

Format your changes with cue fmt

When applying changes to cue files, please apply formatter:

Terminal window
m . /integrates/schemas/fmt

Export changes to database-design

For updating changes made to the schema or tables contents to integrates/schemas/database-design.json simply execute:

Terminal window
m . /integrates/schemas/export

This job will fail if the produced output is not up to date with the previous state of the database-design and will be successfull if there are no changes. This behaviour is intentional so that CI can catch differences between the current state of the cue schemas and the final content.

Types generation

Since cue is now our source of truth for the database schema, we can generate types for the database schema for avoiding double maintenance. By now, the produced types are python TypedDict which should model how the dynamodb item looks like right before being inserted or right after being retrieved. Once generated, types as available in the fluidattacks-core external package.

For producing the types, run:

Terminal window
m . /integrates/schemas/gen_types

This job will fail if types are not up to date. The output will be placed according to the table name in integrates/utils/pypi/fluidattacks-core/fluidattacks_core/types/dynamo/items.py.

Run all

For convenience, you can run all the previous commands fmt, export and gen_types in one go:

Terminal window
m . /integrates/schemas

Validate current schema without changing database-design

If you want to run schema validations against your running database instance, run:

Terminal window
m . /integrates/schemas/val $stage $table_name

Valid stages are dev and prod. For example:

Terminal window
m . /integrates/schemas/val dev integrates_vms

This will help you find items which are non compliant with the desired schema.

Utilities

NoSQL Workbench

NoSQL Workbench for Amazon DynamoDB is a cross-platform client-side GUI application that you can use for modern database development and operations.

for FluidAttacks users it can be installed via Self Service on your work machine.

To connect to the database you must follow the steps below:

  1. Run the db using:

    Terminal window
    m . /integrates

    or

    Terminal window
    m . /integrates/db

  2. Open NoSQL Workbench

  3. Follow this steps in the GUI:

    1. Operation Builder (in the left pane)
    2. Add connection
    3. DynamoDB local
    4. Type in Hostname: localhost and in Port: 8022
    5. Connect
    6. Click Open button in the new connection.

Now you will be able to visualize the tables and information.

Learn more about the cue language in https://cuelang.org/.

If vscode is your IDE, this extension is recommended: https://marketplace.visualstudio.com/items?itemName=asdine.cue

A vim plugin for syntax highlighting is also available at: https://github.com/jjo/vim-cue