Development Guide

Loading…

Development Guide

Relevant source files

• .github/dependabot.yml
• .github/workflows/build-docs.yml
• README.md
• src/enums/url_enum.rs

Purpose and Scope

This guide provides an overview of development practices, code organization, and workflows for contributing to the rust-sec-fetcher project. It covers environment setup, code organization principles, development workflows, and common development tasks.

For detailed information about specific development topics, see:

• Testing strategies and test fixtures: Testing Strategy
• Continuous integration and automated testing: CI/CD Pipeline
• Docker container configuration: Docker Deployment

Development Environment Setup

Prerequisites

The project requires the following tools installed:

Rust Development Setup

1. Clone the repository and navigate to the root directory.

2. Build the Rust application:

3. Run tests to verify setup:

The Rust workspace is configured with necessary dependencies. Key development dependencies include:

• mockito for HTTP mocking in tests.
• tempfile for temporary file/directory creation in tests.
• tokio test macros for async test support.

Python Development Setup

1. Create a virtual environment:

2. Install dependencies using uv:

3. Verify installation by running integration tests (requires Docker):

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Configuration Setup

The application requires a configuration file at ~/.config/sec-fetcher/config.toml or a custom path. Minimum configuration:

For non-interactive testing, use AppConfig directly in test code as shown in tests/config_manager_tests.rs:36-57

Sources: tests/config_manager_tests.rs:36-57 tests/sec_client_tests.rs:8-20

Code Organization and Architecture

Repository Structure

Sources: src/network/sec_client.rs:1-181 tests/config_manager_tests.rs:1-95 tests/sec_client_tests.rs:1-159 python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Module Dependency Flow

The dependency flow follows a layered architecture:

1. Configuration Layer : ConfigManager loads settings from TOML files and credentials.
2. Network Layer : SecClient wraps HTTP client with caching and throttling middleware.
3. Data Fetching Layer : Network module functions fetch raw data from SEC APIs via Url variants src/enums/url_enum.rs:5-116
4. Transformation Layer : Transformers normalize raw data into standardized concepts.
5. Model Layer : Data structures represent domain entities.

Sources: src/network/sec_client.rs:1-181 tests/config_manager_tests.rs:1-95 src/enums/url_enum.rs:5-116

Development Workflow

Standard Development Cycle

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Running Tests Locally

Rust Unit Tests

Run all Rust tests with cargo:

Run specific test modules:

Test Structure Mapping:

Sources: tests/config_manager_tests.rs:1-95 tests/sec_client_tests.rs:1-159

Python Integration Tests

Integration tests require Docker services. Run via the provided shell script:

This script performs the following steps as defined in python/narrative_stack/us_gaap_store_integration_test.sh:1-39:

1. Activates Python virtual environment.
2. Installs dependencies with uv.
3. Starts Docker Compose services.
4. Waits for MySQL availability.
5. Creates us_gaap_test database and loads schema.
6. Runs pytest integration tests.
7. Tears down containers on exit.

Sources: python/narrative_stack/us_gaap_store_integration_test.sh:1-39

Writing Tests

Unit Test Pattern (Rust)

The codebase uses mockito for HTTP mocking:

Sources: tests/sec_client_tests.rs:35-62

Test Fixture Pattern

The codebase uses temporary directories for file-based tests via tempfile::tempdir() as shown in tests/config_manager_tests.rs:8-17

Sources: tests/config_manager_tests.rs:8-17

Common Development Tasks

Adding a New SEC Data Endpoint

To add support for a new SEC data endpoint:

1. Add URL enum variant in src/enums/url_enum.rs src/enums/url_enum.rs:5-116
2. UpdateUrl::value() to return the formatted URL string src/enums/url_enum.rs:121-165
3. Create fetch function in src/network/ using the new Url variant.
4. Define data models in src/models/ for the response structure.

Sources: src/enums/url_enum.rs:5-165

Modifying HTTP Client Behavior

The SecClient is configured in src/network/sec_client.rs:21-89 Key configuration points:

Sources: src/network/sec_client.rs:21-108

Code Quality Standards

CI/CD and Maintenance

The project uses GitHub Actions for automated quality checks:

• Linting : rust-lint.yml runs clippy and rustfmt.
• Testing : rust-tests.yml runs the test suite.
• Documentation : build-docs.yml generates documentation weekly build-docs.yml:1-81
• Dependency Updates : Dependabot is configured for weekly Cargo updates dependabot.yml:1-10

Sources: .github/workflows/build-docs.yml:1-81 .github/dependabot.yml:1-10

Dismiss

Refresh this wiki

Enter email to refresh