Network Layer & SecClient

Loading…

Network Layer & SecClient

Relevant source files

• src/network.rs
• src/network/fetch_cik_submissions.rs
• src/network/sec_client.rs
• src/utils.rs
• tests/sec_client_tests.rs

Purpose and Scope

This page documents the network layer of the rust-sec-fetcher application, specifically the SecClient HTTP client and its associated infrastructure. The SecClient provides the foundational HTTP communication layer for all SEC EDGAR API interactions, implementing throttling, caching, and retry logic to ensure reliable and compliant data fetching.

This page covers:

• The SecClient structure and initialization src/network/sec_client.rs:14-22
• Throttling and rate limiting policies src/network/sec_client.rs:65-88
• HTTP caching and preprocessor cache integration src/network/sec_client.rs:58-63 src/network/sec_client.rs:90-91
• Request/response handling including cache bypass src/network/sec_client.rs:159-184 src/network/sec_client.rs:186-214
• User-Agent management and email validation src/network/sec_client.rs:111-123

For information about the specific network fetching functions that use SecClient, see Data Fetching Functions. For details on the caching system architecture, see Caching & Storage System.

SecClient Architecture Overview

Component Diagram

Sources: src/network/sec_client.rs:14-108 src/network/sec_client.rs:159-214

SecClient Structure and Initialization

SecClient Fields

The SecClient struct maintains core components for networking and caching:

Sources: src/network/sec_client.rs:14-22

Construction from ConfigManager

The from_config_manager() constructor performs the following initialization sequence:

1. Extract Configuration : Reads email, app_name, app_version, max_concurrent, min_delay_ms, and max_retries from AppConfig src/network/sec_client.rs:31-56
2. Create CachePolicy : Configures cache with 1-week TTL (hardcoded) and disables header respect src/network/sec_client.rs:58-63
3. Create ThrottlePolicy : Configures rate limiting based on max_concurrent and min_delay_ms with a 500ms adaptive jitter src/network/sec_client.rs:82-88
4. Initialize Caches : Retrieves both HTTP and preprocessor cache stores from the ConfigManager src/network/sec_client.rs:90-91
5. Build Middleware Stack : Combines the DataStore with policies using reqwest_drive helpers src/network/sec_client.rs:93-98

Sources: src/network/sec_client.rs:26-109

Throttle Policy & Compliance

The SEC’s public guidance for EDGAR states a maximum request rate of 10 requests/second. The SecClient chooses a conservative default of max_concurrent=1 and min_delay_ms=500 (~2 req/s) src/network/sec_client.rs:67-81

Sources: src/network/sec_client.rs:65-88

Request Flow and Caching

raw_request vs raw_request_bypass_cache

The client provides two primary ways to interact with the network:

1. raw_request : The standard path. It uses the http_client which includes the CacheMiddleware. Requests are served from the on-disk cache if available and within TTL src/network/sec_client.rs:159-184
2. raw_request_bypass_cache : Uses the CacheBypass extension to force a network fetch. This skips both reading from and writing to the cache while still respecting the ThrottlePolicy src/network/sec_client.rs:186-214

Request Pipeline

Sources: src/network/sec_client.rs:159-224 src/network/sec_client.rs:93-98

User-Agent Management

The get_user_agent() method generates a compliant User-Agent string as required by the SEC. It validates the email format every time it is called src/network/sec_client.rs:111-123

Format: {app_name}/{app_version} (+{email})
Example: sec-fetcher/0.1.0 (+user@example.com)

If the email provided in the configuration is invalid according to EmailAddress::is_valid, the client will panic src/network/sec_client.rs:115-118

Sources: src/network/sec_client.rs:111-123 tests/sec_client_tests.rs:7-23

Fetch and Render Pipeline

The fetch_and_render function (exported in src/network.rs) provides a high-level orchestration for retrieving filing documents and converting them to specific views (e.g., Markdown).

1. Fetch : Retrieves the raw document from EDGAR using SecClient.
2. Parse : Normalizes the document structure.
3. Render : Applies a View transformation (like MarkdownView).

Sources: src/network.rs:45-46 src/network/fetch_and_render.rs:1-10 (Note: file content for fetch_and_render.rs was not provided in detail, but its existence is noted in the module structure).

Testing Infrastructure

SecClient logic is verified in tests/sec_client_tests.rs using mockito to simulate SEC API responses.

Sources: tests/sec_client_tests.rs:1-233

Dismiss

Refresh this wiki

Enter email to refresh