CI/CD Pipeline

Loading…

CI/CD Pipeline

Relevant source files

• .github/dependabot.yml
• .github/workflows/build-docs.yml
• README.md
• src/enums/url_enum.rs

Purpose and Scope

This document explains the continuous integration and continuous deployment (CI/CD) infrastructure for the rust-sec-fetcher repository. It covers the GitHub Actions workflow configuration, integration test automation, documentation deployment, and dependency management.

The CI/CD architecture is split between Rust-specific validation (linting, testing) and the Python narrative_stack system’s integration testing. For general testing strategies including Rust unit tests and Python test fixtures, see [Testing Strategy](https://github.com/jzombie/rust-sec-fetcher/blob/345ac64c/Testing Strategy)

GitHub Actions Workflows

The repository implements several GitHub Actions workflows to ensure code quality and system reliability across the dual Rust/Python architecture.

1. US GAAP Store Integration Test

This workflow validates the Python machine learning pipeline’s integration with external dependencies. It is triggered by changes to the python/narrative_stack/ directory [.github/workflows/us-gaap-store-integration-test.yml:3-11].

Sources: [.github/workflows/us-gaap-store-integration-test.yml:3-11]

graph TB
    PushTrigger["Push Event"]
PRTrigger["Pull Request Event"]
PathCheck{"Changed paths include:\npython/narrative_stack/**\nor workflow file itself?"}
WorkflowRun["Execute us-gaap-store-integration-test.yml"]
Skip["Skip workflow execution"]
PushTrigger --> PathCheck
 
   PRTrigger --> PathCheck
    
 
   PathCheck -->|Yes| WorkflowRun
 
   PathCheck -->|No| Skip

2. Build and Deploy Documentation

This workflow automates the generation of the project’s documentation using deepwiki-to-mdbook. It runs weekly or on manual dispatch [.github/workflows/build-docs.yml:4-7].

Sources: [.github/workflows/build-docs.yml:1-81]

Integration Test Job Structure

The us-gaap-store-integration-test workflow defines a single job named integration-test that executes on ubuntu-latest [.github/workflows/us-gaap-store-integration-test.yml:12-15].

Sources: [.github/workflows/us-gaap-store-integration-test.yml:17-50]

graph TB
    Start["Job: integration-test"]
Checkout["Step 1: Checkout repo\nactions/checkout@v4\nwith lfs: true"]
SetupPython["Step 2: Set up Python\nactions/setup-python@v5\npython-version: 3.12"]
InstallUV["Step 3: Install uv\ncurl astral.sh/uv/install.sh"]
InstallDeps["Step 4: Install Python dependencies\nuv pip install -e . --group dev"]
Ruff["Step 5: Check style with Ruff\nruff check ."]
RunTest["Step 6: Run integration test\n./us_gaap_store_integration_test.sh"]
Start --> Checkout
 
   Checkout --> SetupPython
 
   SetupPython --> InstallUV
 
   InstallUV --> InstallDeps
 
   InstallDeps --> Ruff
 
   Ruff --> RunTest

Integration Test Architecture

The integration test orchestrates multiple Docker containers to create an isolated environment for validating the narrative_stack data flow.

Container & Entity Mapping

This diagram maps the CI orchestration to specific code entities and external services.

Sources: [python/narrative_stack/us_gaap_store_integration_test.sh:1-39], [python/narrative_stack/Dockerfile.simd-r-drive-ci-server:1-34]

graph TB
    subgraph "Docker Compose Project: us_gaap_it"
        MySQL["Container: us_gaap_test_db\n(MySQL)"]
SimdRDrive["Container: simd_r_drive_ws_server_test\n(WebSocket Server)"]
TestRunner["Test Runner\npytest process"]
end
    
    Schema["SQL Schema\ntests/integration/assets/us_gaap_schema_2025.sql"]
PyTestFile["tests/integration/test_us_gaap_store.py"]
TestRunner -->|Executes| PyTestFile
 
   PyTestFile -->|SQL queries| MySQL
 
   PyTestFile -->|WS connection| SimdRDrive
    
 
   Schema -->|Loaded via mysql CLI| MySQL

Test Execution Flow

The integration test script [python/narrative_stack/us_gaap_store_integration_test.sh:1-39] manages the container lifecycle.

Sources: [python/narrative_stack/us_gaap_store_integration_test.sh:1-39]

graph TB
    Start["Start: us_gaap_store_integration_test.sh"]
SetVars["Set variables\nPROJECT_NAME=us_gaap_it"]
RegisterTrap["Register cleanup trap\ntrap 'cleanup' EXIT"]
DockerUp["Start Docker containers\ndocker compose up -d --profile test"]
WaitMySQL["Wait for MySQL ready\nmysqladmin ping loop"]
LoadSchema["Load schema\nmysql < us_gaap_schema_2025.sql"]
RunPytest["Execute pytest\npytest tests/integration/test_us_gaap_store.py"]
Cleanup["Cleanup function\ndocker compose down --volumes"]
Start --> SetVars
 
   SetVars --> RegisterTrap
 
   RegisterTrap --> DockerUp
 
   DockerUp --> WaitMySQL
 
   WaitMySQL --> LoadSchema
 
   LoadSchema --> RunPytest
 
   RunPytest --> Cleanup

Docker Container Configuration

simd-r-drive-ws-server Container Build

The Dockerfile [python/narrative_stack/Dockerfile.simd-r-drive-ci-server:1-34] creates a single-stage image for the CI server. It installs the simd-r-drive-ws-server crate version 0.10.0-alpha [python/narrative_stack/Dockerfile.simd-r-drive-ci-server:12].

Sources: [python/narrative_stack/Dockerfile.simd-r-drive-ci-server:18-33]

graph LR
    BuildTime["Build Time\n--build-arg SERVER_ARGS"]
BakeArgs["ENV SERVER_ARGS"]
Entrypoint["ENTRYPOINT interpolates\n$SERVER_ARGS + $@"]
ServerExec["Execute:\nsimd-r-drive-ws-server"]
BuildTime --> BakeArgs
 
   BakeArgs --> Entrypoint
 
   Entrypoint --> ServerExec

Dependency Management

The project uses Dependabot to maintain up-to-date dependencies for the Rust components.

Sources: [.github/dependabot.yml:1-10]

Environment Configuration

Python Environment

The CI pipeline uses uv for fast, reproducible environment setup [.github/workflows/us-gaap-store-integration-test.yml:27-37].

• Python Version : 3.12
• Installation : uv pip install -e . --group dev

Project Isolation

To prevent resource conflicts, the integration test uses a specific Docker Compose project name: us_gaap_it [python/narrative_stack/us_gaap_store_integration_test.sh:7]. This ensures that networks and volumes are isolated from other development or CI tasks.

Sources: [python/narrative_stack/us_gaap_store_integration_test.sh:7-9]

Dismiss

Refresh this wiki

Enter email to refresh