CI/CD Pipeline

Relevant source files

• .github/workflows/us-gaap-store-integration-test.yml
• python/narrative_stack/Dockerfile.simd-r-drive-ci-server
• python/narrative_stack/us_gaap_store_integration_test.sh

Purpose and Scope

This document explains the continuous integration and continuous deployment (CI/CD) infrastructure for the rust-sec-fetcher repository. It covers the GitHub Actions workflow configuration, integration test automation, Docker container orchestration for testing, and the complete test execution flow.

The CI/CD pipeline focuses specifically on the Python narrative_stack system's integration testing. For general testing strategies including Rust unit tests and Python test fixtures, see Testing Strategy. For production Docker deployment of the simd-r-drive WebSocket server, see Docker Deployment.

GitHub Actions Workflow Overview

The repository implements a single GitHub Actions workflow named US GAAP Store Integration Test that validates the Python machine learning pipeline's integration with external dependencies.

Workflow Trigger Conditions

The workflow is configured to run automatically under specific conditions defined in .github/workflows/us-gaap-store-integration-test.yml:3-11:

Sources: .github/workflows/us-gaap-store-integration-test.yml:3-11

The workflow only executes when changes affect:

• Any file under python/narrative_stack/ directory
• The workflow definition file itself (.github/workflows/us_gaap_store_integration_test.yml)

This targeted triggering prevents unnecessary CI runs when changes are made to the Rust sec-fetcher components or unrelated Python modules.

Integration Test Job Structure

The workflow defines a single job named integration-test that runs on ubuntu-latest runners. The job executes a sequence of setup and validation steps.

Sources: .github/workflows/us-gaap-store-integration-test.yml:12-51

graph TB
    Start["Job: integration-test\nruns-on: ubuntu-latest"]
Checkout["Step 1: Checkout repo\nactions/checkout@v4\nwith lfs: true"]
SetupPython["Step 2: Set up Python\nactions/setup-python@v5\npython-version: 3.12"]
InstallUV["Step 3: Install uv\ncurl astral.sh/uv/install.sh\nAdd to PATH"]
InstallDeps["Step 4: Install Python dependencies\nuv venv --python=3.12\nuv pip install -e . --group dev"]
Ruff["Step 5: Check code style with Ruff\nruff check ."]
Chmod["Step 6: Make test script executable\nchmod +x us_gaap_store_integration_test.sh"]
RunTest["Step 7: Run integration test\n./us_gaap_store_integration_test.sh"]
Start --> Checkout
 
   Checkout --> SetupPython
 
   SetupPython --> InstallUV
 
   InstallUV --> InstallDeps
 
   InstallDeps --> Ruff
 
   Ruff --> Chmod
 
   Chmod --> RunTest
    
    style Start fill:#f9f9f9
    style RunTest fill:#f9f9f9

Key Workflow Steps

Sources: .github/workflows/us-gaap-store-integration-test.yml:17-50

Integration Test Architecture

The integration test orchestrates multiple Docker containers to create an isolated test environment that validates the narrative_stack system's ability to ingest, process, and store US GAAP financial data.

graph TB
    subgraph "Docker Compose Project: us_gaap_it"
        MySQL["Container: us_gaap_test_db\nImage: mysql\nPort: 3306\nUser: root\nPassword: onlylocal\nDatabase: us_gaap_test"]
SimdRDrive["Container: simd_r_drive_ws_server_test\nBuilt from: Dockerfile.simd-r-drive-ci-server\nPort: 8080\nBinary: simd-r-drive-ws-server@0.10.0-alpha"]
TestRunner["Test Runner\npytest process\ntests/integration/test_us_gaap_store.py"]
end
    
    Schema["SQL Schema\ntests/integration/assets/us_gaap_schema_2025.sql"]
TestRunner -->|SQL queries| MySQL
 
   TestRunner -->|WebSocket connection| SimdRDrive
    
 
   Schema -->|Loaded via mysql CLI| MySQL
    
    style MySQL fill:#f9f9f9
    style SimdRDrive fill:#f9f9f9
    style TestRunner fill:#f9f9f9

Container Architecture

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:1-39 python/narrative_stack/Dockerfile.simd-r-drive-ci-server:1-34

The test environment consists of:

1. MySQL Database Container (us_gaap_test_db): Provides persistent storage for US GAAP financial data with pre-loaded schema
2. simd-r-drive WebSocket Server (simd_r_drive_ws_server_test): Handles embedding matrix storage and data synchronization
3. pytest Test Runner : Executes integration tests against both containers

Test Execution Flow

The integration test script python/narrative_stack/us_gaap_store_integration_test.sh:1-39 orchestrates the complete test lifecycle from container startup through teardown.

graph TB
    Start["Start: us_gaap_store_integration_test.sh"]
SetVars["Set variables\nPROJECT_NAME=us_gaap_it\nCOMPOSE=docker compose -p us_gaap_it"]
ActivateVenv["Activate virtual environment\nsource .venv/bin/activate"]
InstallDeps["Install dependencies\nuv pip install -e . --group dev"]
RegisterTrap["Register cleanup trap\ntrap 'cleanup' EXIT"]
DockerUp["Start Docker containers\ndocker compose up -d\nProfile: test"]
WaitMySQL["Wait for MySQL ready\nmysqladmin ping loop"]
CreateDB["Create database\nCREATE DATABASE us_gaap_test"]
LoadSchema["Load schema\nmysql < us_gaap_schema_2025.sql"]
SetPythonPath["Set PYTHONPATH=src"]
RunPytest["Execute pytest\npytest -s -v tests/integration/test_us_gaap_store.py"]
Cleanup["Cleanup function\ndocker compose down --volumes"]
Start --> SetVars
 
   SetVars --> ActivateVenv
 
   ActivateVenv --> InstallDeps
 
   InstallDeps --> RegisterTrap
 
   RegisterTrap --> DockerUp
 
   DockerUp --> WaitMySQL
 
   WaitMySQL --> CreateDB
 
   CreateDB --> LoadSchema
 
   LoadSchema --> SetPythonPath
 
   SetPythonPath --> RunPytest
 
   RunPytest --> Cleanup
    
    style Start fill:#f9f9f9
    style RunPytest fill:#f9f9f9
    style Cleanup fill:#f9f9f9

Test Script Flow Diagram

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Cleanup Mechanism

The script implements a robust cleanup mechanism using bash trap handlers python/narrative_stack/us_gaap_store_integration_test.sh:14-19:

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:14-19

The cleanup function ensures that all Docker resources are removed regardless of test outcome, preventing resource leaks and port conflicts in subsequent test runs.

Docker Container Configuration

simd-r-drive-ws-server Container Build

The Dockerfile python/narrative_stack/Dockerfile.simd-r-drive-ci-server:1-34 creates a single-stage image optimized for CI environments.

Build Configuration:

Sources: python/narrative_stack/Dockerfile.simd-r-drive-ci-server:9-33

graph LR
    BuildTime["Build Time\n--build-arg SERVER_ARGS=..."]
BakeArgs["Bake into ENV variable"]
Runtime["Run Time\n-e SERVER_ARGS=... OR\ndocker run ... extra flags"]
Entrypoint["ENTRYPOINT interpolates\n$SERVER_ARGS + $@"]
ServerExec["Execute:\nsimd-r-drive-ws-server\nwith combined args"]
BuildTime --> BakeArgs
 
   BakeArgs --> Runtime
 
   Runtime --> Entrypoint
 
   Entrypoint --> ServerExec
    
    style Entrypoint fill:#f9f9f9

The Dockerfile uses build arguments to support configuration at both build-time and run-time:

Sources: python/narrative_stack/Dockerfile.simd-r-drive-ci-server:18-33

MySQL Container Configuration

The MySQL container is configured through Docker Compose with the following specifications python/narrative_stack/us_gaap_store_integration_test.sh:23-35:

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:25-35

Environment Configuration and Dependencies

graph TB
    UVInstall["Install uv package manager\ncurl -LsSf astral.sh/uv/install.sh"]
CreateVenv["Create virtual environment\nuv venv --python=3.12"]
ActivateVenv["Activate environment\nsource .venv/bin/activate"]
InstallPackage["Install narrative_stack\nuv pip install -e . --group dev"]
DevGroup["Dev dependency group includes:\n- pytest\n- ruff\n- integration test utilities"]
UVInstall --> CreateVenv
 
   CreateVenv --> ActivateVenv
 
   ActivateVenv --> InstallPackage
 
   InstallPackage --> DevGroup
    
    style InstallPackage fill:#f9f9f9

Python Environment Setup

The CI pipeline uses uv as the package manager to create isolated Python environments and install dependencies:

Sources: .github/workflows/us-gaap-store-integration-test.yml:27-37 python/narrative_stack/us_gaap_store_integration_test.sh:11-12

Code Quality Validation

Before running integration tests, the workflow executes Ruff code quality checks .github/workflows/us-gaap-store-integration-test.yml:39-43:

The Ruff check acts as a fast pre-test gate that fails the CI pipeline if code quality issues are detected, preventing broken code from reaching the integration test phase.

Docker Compose Project Isolation

The integration test uses Docker Compose project isolation to prevent conflicts with other running containers python/narrative_stack/us_gaap_store_integration_test.sh:7-9:

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:7-9

The project name us_gaap_it creates an isolated namespace for all Docker resources (containers, networks, volumes) associated with the integration test, allowing multiple test runs or development environments to coexist without interference.

Test Execution and Reporting

pytest Integration Test Execution

The final step executes the integration test suite using pytest python/narrative_stack/us_gaap_store_integration_test.sh:37-38:

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:37-38

The pytest command runs with stdout capture disabled (-s flag), allowing real-time visibility into test progress and enabling debugging output from the integration tests.

graph TB
    GitPush["Git Push/PR\nto python/narrative_stack/"]
GHAStart["GitHub Actions Trigger\nus-gaap-store-integration-test.yml"]
EnvSetup["Environment Setup:\n- Python 3.12\n- uv package manager\n- Dev dependencies"]
CodeQuality["Code Quality Check:\nruff check ."]
QualityFail["❌ Fail CI"]
QualityPass["✓ Pass"]
ScriptExec["Execute Test Script:\nus_gaap_store_integration_test.sh"]
DockerSetup["Docker Compose Setup:\n- MySQL (us_gaap_test_db)\n- simd-r-drive-ws-server"]
SchemaLoad["Load SQL Schema:\nus_gaap_schema_2025.sql"]
PytestRun["Run Integration Tests:\npytest tests/integration/"]
TestFail["❌ Fail CI"]
TestPass["✓ Pass"]
Cleanup["Cleanup:\ndocker compose down --volumes"]
CIComplete["✅ CI Complete"]
GitPush --> GHAStart
 
   GHAStart --> EnvSetup
 
   EnvSetup --> CodeQuality
    
 
   CodeQuality -->|Issues detected| QualityFail
 
   CodeQuality -->|Clean| QualityPass
    
 
   QualityPass --> ScriptExec
 
   ScriptExec --> DockerSetup
 
   DockerSetup --> SchemaLoad
 
   SchemaLoad --> PytestRun
    
 
   PytestRun -->|Tests fail| TestFail
 
   PytestRun -->|Tests pass| TestPass
    
 
   TestFail --> Cleanup
 
   TestPass --> Cleanup
    
 
   Cleanup --> CIComplete
    
    style GHAStart fill:#f9f9f9
    style CodeQuality fill:#f9f9f9
    style PytestRun fill:#f9f9f9
    style CIComplete fill:#f9f9f9

CI/CD Pipeline Flow Summary

The complete CI/CD pipeline integrates all components into a continuous validation workflow:

Sources: .github/workflows/us-gaap-store-integration-test.yml:1-51 python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Key Design Decisions

Git LFS Requirement

The workflow explicitly enables Git LFS .github/workflows/us-gaap-store-integration-test.yml:19-20 to support large test data files. The test script includes a comment python/narrative_stack/us_gaap_store_integration_test.sh:1-2 noting this requirement for local execution.

Single-Stage Docker Build

The Dockerfile uses a single-stage build python/narrative_stack/Dockerfile.simd-r-drive-ci-server:9-15 rather than multi-stage, prioritizing simplicity for CI environments where image size is less critical than build reliability and transparency.

Profile-Based Container Selection

The Docker Compose command uses --profile test python/narrative_stack/us_gaap_store_integration_test.sh9 to selectively start only test-related services, preventing unnecessary containers from consuming CI resources.

Working Directory Isolation

All workflow steps and script commands execute from the python/narrative_stack working directory .github/workflows/us-gaap-store-integration-test.yml33 ensuring consistent path resolution and dependency isolation from other repository components.