Testing

Welcome to our section describing how to test Flags.

The tests in Flags are organized in the flags/test/lib_cspm directory and follow these main components:

1. Test Groups: Tests are organized into groups based on finding ranges (e.g., 001-099, 100-199, etc.).
2. Mock Data: Test data for each cloud provider (AWS, Azure, GCP) is stored in flags/test/lib_cspm/data/.
3. Test Configurations: Template configurations are stored in flags/test/lib_cspm/test_configs/.
4. Test Results: Expected results are stored in flags/test/lib_cspm/results/.

Running Tests

To run tests locally, use the following command:

m . /flags/test group_name

Where group_name is one of the available test groups, for example:

• cspm_findings_001_099
• cspm_findings_100_199
• etc.

Note

Tests use the test_clean_cspm_cache fixture to disable caching during test execution, ensuring each test runs with a clean state.

Writing Tests

When writing new tests for Flags, follow these steps:

1. Add Mock Data: Create mock responses for cloud provider APIs in the appropriate directory:
   • AWS: Uses moto library for service simulation and mocked responses in data/aws/
   • Azure: data/azure/
   • GCP: data/gcp/
2. Define Expected Results:
   • Add expected finding results in results/ directory
   • Use the format FXXX.csv where XXX is the finding number
3. Add Test Case:
   • Add your finding to the appropriate test group in test_findings_cspm.py
   • Use the existing test infrastructure which handles:
     • Configuration generation
     • Mock data injection
     • Result verification

Cloud Provider Mocking

AWS Mocking with Moto

Flags uses the moto library to simulate AWS services in tests. Moto provides a mock AWS environment that allows testing AWS interactions without making real API calls.

Example of using moto in tests:

from test.lib_cspm.data.aws.moto_patch import mock_aio_aws

# Using moto context manager for AWS service simulation
with mock_aio_aws():
    run_finding("F101")  # The finding will use moto's mock AWS environment

Other Cloud Providers

For other cloud providers, we use custom mock responses:

• Azure: Uses custom mock responses defined in data/azure/
• GCP: Uses custom mock responses defined in data/gcp/

Example Test

Here’s a simplified example of how findings are tested:

@pytest.mark.usefixtures("test_clean_cspm_cache")
@pytest.mark.flags_test_group("cspm_findings_100_199")
def test_cspm_findings_100_199() -> None:
    findings = [
        "F101",
        "F148",
        "F157",
        # Add your new finding here
    ]
    run_multiple_findings_test(findings)

The test infrastructure will:

1. Create a temporary configuration
2. Set up mock environments (including moto for AWS)
3. Run the finding check
4. Verify the results against expected output