Skip to content
Fluid Attacks Development Portal

Unit tests

Philosophy

Unit tests focus on verifying the functionality of individual units or components of our software. A unit would be the smallest testable part of a software, such as a function, method or class. Unit tests in Fluid Attacks must be:

  • Repeatable: Regardless of where they are executed, the result must be the same.
  • Fast: Unit tests should take little time to execute because, being the first level of testing, where you have isolated functions/methods and classes, the answers should be immediate. A unit test should take at most two (2) seconds.
  • Independent: The functions or classes to be tested should be isolated, no secondary effect behaviors should be validated, and, if possible, we should avoid calls to external resources such as databases; for this, we use mocks.
  • Descriptive: For any developer, it should be evident what is being tested in the unit test, what the result should be, and in case of an error, what is the source of the error.

Architecture

  • Location: To be discover by the testing framework, test files must be located next to the file to be tested with the _test suffix and every test method must start with the test_ prefix. Take a look at add group tests for reference.
  • Utilities: Some utilities are added to simplify tasks like to populate database, mock comfortably and include test files. It allows developers to focus on actually testing the code.
  • Coverage: Coverage is a module-scoped integer between 0 and 100. Current coverage for a given module con be found at <module-path>/coverage. For example, api/coverage.

Writing tests

See the following examples to understand how to write tests:

from integrates.dataloaders import Dataloaders, get_new_context
from integrates.db_model.organizations.types import Organization
from integrates.organizations.utils import get_organization
from integrates.testing.aws import IntegratesAws, IntegratesDynamodb
from integrates.testing.fakers import OrganizationFaker
from integrates.testing.mocks import mocks


@mocks(
    aws=IntegratesAws(
        dynamodb=IntegratesDynamodb(
            organizations=[
                OrganizationFaker(id="test-org-1"),
            ],
        ),
    ),
)
async def test_get_organization() -> None:
    # Arrange
    organization_id = "test-org-1"


    # Act
    loaders: Dataloaders = get_new_context()
    organization: Organization = await get_organization(
        loaders,
        organization_id,
    )


    # Assert
    assert organization is not None

@mocks decorator allows to populate the database with test data using aws parameter. A clean database will be created and populated for each parameter provided via @utils.parametrize. We got deeper into this decorators and helper methods in the next sections.

DynamoDB

Integrates database is populated using IntegratesAws.dynamodb in the @mocks decorator. This parameter is an instance of IntegratesDynamodb, a helper class to populate the main tables with valid data:

@mocks(
    aws=IntegratesAws(
        dynamodb=IntegratesDynamodb(
            organizations=[OrganizationFaker(id=ORG_ID)],
            stakeholders=[
                StakeholderFaker(email=USER_MANAGER_EMAIL),
                StakeholderFaker(email=ADMIN_EMAIL),
            ],
            organization_access=[
                OrganizationAccessFaker(
                    organization_id=ORG_ID,
                    email=USER_MANAGER_EMAIL,
                    state=OrganizationAccessStateFaker(has_access=True, role="user_manager"),
                ),
                OrganizationAccessFaker(
                    organization_id=ORG_ID,
                    email=ADMIN_EMAIL,
                    state=OrganizationAccessStateFaker(has_access=True, role="admin"),
                ),
            ],
        ),
    ),
    others=[
        Mock(logs_utils, "cloudwatch_log", "sync", None),
    ],
)

In the example above, we are populating the database with one organization, two stakeholders, and giving access to both stakeholders to the Organization with a different role.

Every faker is a fake data generator for one element. Parameters are optional to modify your data for your tests (e.g., assigned role in OrganizationAccessStateFaker or the email in StakeholderFaker). Faker name gives a hint about where it should be used in the IntegratesDynamodb parameters.

The others parameter is a way to list all the startup mocks that you require in your test. In the example above, we are mocking the cloudwatch_log function from logs_utils module to avoid to call CloudWatch directly and return always a None value. Mock is a helper class that creates a mock based on module, function or variable name, mode (sync or async), and a return value.

This declarative approach ensures isolation. Each test will have its own data and will not conflict with other tests.

S3

Integrates buckets are created for testing at the same time when @mocks is called and no more actions are required. You can use the buckets in your tests and, also, load files to buckets automatically before every test run.

To load files to the buckets automatically, you must use:

@mocks(aws=IntegratesAws(s3=IntegratesS3(autoload=True)))

Use the following file structure as reference:

  • main.py (logic here)
  • main_test.py (tests here)
  • Directorytest_data/
    • Directorytest_name_1/
      • file_1.txt (It won’t be loaded)
      • file_2.txt (It won’t be loaded)
    • Directorytest_name_2/
      • Directoryintegrates.dev/
        • README.md (Loaded to integrates.dev bucket)
    • Directorytest_name_3/
      • Directoryintegrates/
        • README.md (Loaded to integrates bucket)

<test_name> directory is searched to load files into the corresponding buckets. For example, a README.md file will be loaded into integrates.dev for test_name_2, and a different README.md file will be loaded into integrates for test_name_3. This approach ensures both isolation and simplicity in the tests.

Utils

For easy testing, some utilities and decorators are provided.

Use @parametrize to include several test cases:

from integrates.testing.utils import parametrize


@parametrize(
    args=["arg", "expected"],
    cases=[
        ["a", "A"],
        ["b", "B"],
        ["c", "C"],
    ],
)
def test_capitalize(arg: str, expected: str) -> None:
    ...

Use raises to handle errors during tests:

from integrates.testing.utils import raises


def test_fail() -> None:
    with raises(ValueError):
       ...

Use get_file_abs_path to get the file’s absolute path in the test_data/<test_name> directory:

from integrates.testing.utils import get_file_abs_path




def test_name_1() -> None:
  abs_path = get_file_abs_path("file_1.txt")


  assert "/test_data/test_name_1/file_1.txt" in abs_path   # True

Use @freeze_time when you want to set the execution time (time-based features).

from integrates.testing.utils import freeze_time


@freeze_time("2024-01-01")
def test_accepted_until() -> None:
    ...

Running tests

You can run tests for specific modules with the following command:

Terminal window
m . /integrates/back/test <module> [test-1] [test-2] [test-n]...

where:

  1. <module> is required and can be any Integrates module.
  2. [test-n] is optional and can be any test within that module.

If no specific tests are provided, this command will:

  1. Run all tests for the given module.
    1. Fail if any of the tests fail.
  2. Generate a coverage report.
    1. Fail if the new coverage is below the current one for the given module (Developer must add tests to at least keep the same coverage).
    2. Fail if the new coverage is above the current one for the given module (Developer must add new coverage to their commit).
    3. Pass if new and current coverage are the same.

If specific tests are provided, this command will:

  1. Only run the given tests.
    1. Fail if any of the provided tests fail.
  2. Skip the coverage report generation and evaluation.

Old unit tests

You can run tests using the following command:

Terminal window
m . /integrates/back/test/unit not_changes_db [module]

To run the ones that modify the mock database:

Terminal window
m . /integrates/back/test/unit changes_db [module]

Currently, every time our unit tests run, we launch a mock stack that is populated with the necessary data required for our tests to execute. We utilize mocking to prevent race conditions and dependencies within the tests.

When writing unit tests, you can follow these steps to ensure that the test is repeatable, fast, independent, and descriptive:

  • Test file: We store our tests using the same structure as our repository. Inside universe/integrates/back/test/unit/src you can find our unit tests. Look for the test_module_to_test.py file or add it if missing.
  • Write the test: Once the file is ready, you can start writing the test. Consider the purpose of the function, method, or class that you want to test. Think about its behavior when different inputs are provided. Also, identify extreme scenarios to test within the test. These will form our test cases and are important for writing our assertions. We use the parametrize decorator if possible to declare different test cases.
  • Mocks: What do you mock? A general guideline is to look for the await statement inside the function, method or class that you want to test. In most cases, await indicates that the awaited function requires an external resource, such as a database. To learn more about mocks, you can refer to the official documentation.
  • Mock data: When using mocks, you need to provide the data required for your unit test to run. We accomplish this by using pytest fixtures, which allow us to have mock data available from conftest.py files.
  • Assertions: Test the expected behavior. We use assertions to validate results, the number of function or mock calls, and the arguments used in mocks.