Unit tests

Philosophy

Unit tests focus on verifying the functionality of individual units or components of our software. A unit would be the smallest testable part of a software, such as a function, method or class. Unit tests in Fluid Attacks must be:

• Repeatable: Regardless of where they are executed, the result must be the same.
• Fast: Unit tests should take little time to execute because, being the first level of testing, where you have isolated functions/methods and classes, the answers should be immediate. A unit test should take at most two (2) seconds.
• Independent: The functions or classes to be tested should be isolated, no secondary effect behaviors should be validated, and, if possible, we should avoid calls to external resources such as databases; for this, we use mocks.
• Descriptive: For any developer, it should be evident what is being tested in the unit test, what the result should be, and in case of an error, what is the source of the error.

Architecture

• Location: Unit tests for a given file are located right next to the file to be tested. For example, tests for mailmap/create.py are located at mailmap/create_test.py.
• Utilities: It provides vital utilities for populating DynamoDB, comfortably mocking, and local reproducibility. This with the purpose of allowing developers to focus on actually testing the code.
• Coverage: Current coverage for a given module con be found at <module-path>/coverage. For example, malmap/coverage.

Writing tests

Here’s an example test:

integrates/back/integrates/organizations/utils_test.py

from dataloaders import (
    get_new_context,
)
from integrates.db_model.organizations.types import (
    Organization,
)
from organizations.utils import (
    get_organization,
)
from testing import (
    mock,
    tag,
)
from testing.aws import (
    aws_populate,
    IntegratesDomain,
)
from testing.fakers import (
    OrganizationFaker,
)


@tag.parametrize(
    ("organization_id"),
    [
        ("test-org-1"),
        ("test-org-2"),
    ],
)
# Arrange
@aws_populate(
    IntegratesDomain(
        organizations=[
            OrganizationFaker(id="test-org-1"),
            OrganizationFaker(id="test-org-2"),
        ],
    ),
)
async def test_get_organization(organization_id: str) -> None:
    # Act
    loaders: Dataloaders = get_new_context()
    organization: Organization = await get_organization(
      loaders,
      organization_id,
    )

    # Assert
    assert organization.id == organization_id


async def test_get_organization_mock(): -> None:
  # Arrange
  organization_id = "test-org-3"
  mock(
    organizations.utils,
    "get_organization",
    "async",
    OrganizationFaker(id=organization_id),
  )

  # Act
  loaders: Dataloaders = get_new_context()
  organization: Organization = await get_organization(
    loaders,
    organization_id,
  )

  # Assert
  assert organization.id == organization_id

Key aspects of the provided tests are:

1. As we’re testing integrates/back/integrates/organizations/utils.py::get_organization, its corresponding test function is integrates/back/integrates/organizations/utils_test.py::get_organization_test. This naming convention should be followed for all tests.
2. These tests will increase coverage for the integrates/back/integrates/organizations module.
3. aws_populate decorator allows to populate the database with the necessary data for the test to execute. A clean database will be created and populated for each parameter provided via @tag.parametrize.
4. Make sure you provide each test with its own data to avoid conflicts.
5. A test_get_organization_mock test is provided to demonstrate how to mock a function. This is useful when you want to test a function that connects to an external service that you don’t want to connect to in your tests.

Note

Make sure you write tests for the specific module you’re working on, as coverage is only calculated for files within that specific module.

Caution

Prioritize using aws_populate over mock, as it provides a more realistic scenario where a real DB exists. mock should only be used for third party services.

Tip

Make sure you use existing tests as a reference for creating your own.

Running tests

Running tests for specific modules

You can run tests for specific modules with the following command:

m . /integrates/back/test [module1] [module2]... # Pass as many modules as you want

where [moduleN] can be any Integrates module.

For each provided module, this command will:

1. Run all tests for the given module.
   1. Fail if any of the tests fail.
2. Generate a coverage report.
   1. Fail if the new coverage is below the current one for the given module (Developer must add tests to at least keep the same coverage).
   2. Fail if the new coverage is above the current one for the given module (Developer must add new coverage to their commit).
   3. Pass if new and current coverage are the same.

Tip

Try running tests for mailmap, roots and sessions with:

m . /integrates/back/test mailmap roots sessions

Running tests for all modules

You can run tests for all modules with the following command:

m . /integrates/back/test

Old unit tests

Caution

These tests are currently being deprecated due to their poor architecture.

You can run tests using the following command:

m . /integrates/back/test/unit not_changes_db [module]

To run the ones that modify the mock database:

m . /integrates/back/test/unit changes_db [module]

Note

[module] is an optional argument. If you do not specify it, all the unit tests will be executed.

Currently, every time our unit tests run, we launch a mock stack that is populated with the necessary data required for our tests to execute. We utilize mocking to prevent race conditions and dependencies within the tests.

When writing unit tests, you can follow these steps to ensure that the test is repeatable, fast, independent, and descriptive:

• Test file: We store our tests using the same structure as our repository. Inside universe/integrates/back/test/unit/src you can find our unit tests. Look for the test_module_to_test.py file or add it if missing.
• Write the test: Once the file is ready, you can start writing the test. Consider the purpose of the function, method, or class that you want to test. Think about its behavior when different inputs are provided. Also, identify extreme scenarios to test within the test. These will form our test cases and are important for writing our assertions. We use the parametrize decorator if possible to declare different test cases.
• Mocks: What do you mock? A general guideline is to look for the await statement inside the function, method or class that you want to test. In most cases, await indicates that the awaited function requires an external resource, such as a database. To learn more about mocks, you can refer to the official documentation.
• Mock data: When using mocks, you need to provide the data required for your unit test to run. We accomplish this by using pytest fixtures, which allow us to have mock data available from conftest.py files.
• Assertions: Test the expected behavior. We use assertions to validate results, the number of function or mock calls, and the arguments used in mocks.