Getting Started

Loading…

Getting Started

Relevant source files

• .github/workflows/build-docs.yml
• Cargo.toml
• README.md
• src/enums/url_enum.rs

This page guides you through installing, configuring, and running the rust-sec-fetcher application. It covers building the Rust binary, setting up required credentials, and executing your first data fetch. For detailed configuration options, see Configuration System. For comprehensive examples, see Running Examples.

The rust-sec-fetcher is the Rust component of a dual-language system. It fetches and transforms SEC financial data into structured formats. The companion Python system (Python narrative_stack System) processes these files for machine learning applications.

Prerequisites

Before installation, ensure you have:

Optional Components:

• Python 3.8+ for ML pipeline (Python narrative_stack System)
• Docker for simd-r-drive WebSocket server (Docker Deployment)

Sources: Cargo.toml:1-82 README.md:1-205

Installation

Clone Repository

Build from Source

The project includes several specialized binaries for data maintenance and bulk pulling Cargo.toml:15-35

Verify Installation

Run the configuration display example to ensure the environment is ready:

Installation and Setup Flow

Sources: Cargo.toml:15-35 README.md:51-53 src/enums/url_enum.rs:30-52

Basic Configuration

The application uses ConfigManager to coordinate settings from files, environment variables, and the system keychain.

Configuration Entry Points

Every program using the library initializes through one of two patterns README.md:17-49:

1. ConfigManager::load() : The standard approach. It reads from local config files or environment variables and handles interactive email prompts if missing.
2. ConfigManager::from_app_config(&AppConfig { ... }): Used for hard-coding values directly in specialized tools.

Required Email Identification

The SEC mandates a contact email in the User-Agent header README.md:13-14 sec-fetcher provides multiple ways to supply this:

• Interactive : Prompted on first run and stored via keyring (if feature enabled) Cargo.toml42
• Environment : Setting SEC_FETCHER_EMAIL.
• Code : Passing it directly to AppConfig.

Configuration Loading Logic

Sources: README.md:15-49 Cargo.toml:36-43 src/enums/url_enum.rs:121-163

Running Your First Data Fetch

Example: Ticker to CIK Lookup

The library maps human-readable tickers to SEC Central Index Keys (CIKs) using fetch_cik_by_ticker_symbol README.md:63-71

Example: Fetch and Render Filings

You can retrieve specific forms (like 10-K or 8-K) and render them into Markdown for LLM processing or human reading README.md:112-140

Example: Fund Holdings (13F/N-PORT)

The library can parse complex XML filings into structured data README.md:195-202

Data Retrieval Flow

Sources: README.md:112-140 src/enums/url_enum.rs:26-29 src/enums/url_enum.rs:146-150

Data Output & Caching

sec-fetcher is designed to be a “good citizen” on the SEC servers:

• Throttling : Automatically limits requests to 10 per second per SEC policy README.md:5-8
• Caching : Uses simd-r-drive to cache raw responses locally, reducing redundant network traffic Cargo.toml:74-75
• Structured Storage : Data is typically organized by CIK or Ticker in the data/ directory.

Specialized Binaries

For bulk operations, use the provided binaries Cargo.toml:15-35:

• pull-us-gaap-bulk: Downloads large-scale XBRL datasets for ML training.
• pull-fund-holdings: Aggregates holdings from multiple investment companies.
• refresh-test-fixtures: Updates local mock data for the test suite.

For details on the transformation of these datasets, see US GAAP Concept Transformation.

Sources: Cargo.toml:15-35 README.md:204-205 src/enums/url_enum.rs:54-55

Dismiss

Refresh this wiki

Enter email to refresh