System

A config-driven pipeline of independent, adapter-based components communicating through versioned dict contracts. Each stage is a separate Python package with its own repository, CI, and release cycle — wired together by an orchestrator at runtime.

Pipeline stages: Measure, Evaluate, Allocate, Scale

Design Decisions

Protocol-based interfaces

Components satisfy a typing.Protocol — structural subtyping, not inheritance. Each package defines PipelineComponent locally. No shared base import, no circular dependencies, no coupling beyond the method signature.

Dict boundaries

execute() takes and returns a plain dict. No cross-package type imports. Components are coupled only by dict key names — making each package independently deployable and testable.

Schema versioning

Every output dict includes schema_version: "X.Y". Consumers validate the major version and fail explicitly on mismatch. Minor bumps (additive fields) are always safe. Breaking changes require coordinated major bumps across producer and consumer.

Configuration-driven execution

A YAML file selects components, parameters, and data sources. A static registry maps component names to classes; a factory instantiates from config at runtime. No code changes to run a different scenario — change the config, re-run. 

# Each package defines its own Protocol (no shared import)
class PipelineComponent(Protocol):
    def execute(self, event: dict) -> dict: ...

# Registry maps config names → classes
COMPONENT_REGISTRY = {
    "Measure": Measure,
    "Evaluate": Evaluate,
    "MinimaxRegretAllocate": MinimaxRegretAllocate,
}

Pipeline Architecture

Pipeline flow showing parallel fan-out per initiative, fan-in at Allocate, and fan-out for selected initiatives

Measure

Causal effect estimation. Parallel — one call per initiative.

Evaluate

Evidence quality scoring. Parallel — one call per initiative.

Allocate

Portfolio optimization. Fan-in — single decision over all initiatives.

Scale

Run at production scale. Fan-out — selected initiatives only.

Each stage is an independent component exposing a single method: execute(event) → result. The orchestrator wires them together using a ThreadPoolExecutor for parallel fan-out, synchronizes at the fan-in boundary before Allocate, and fans back out for the selected initiatives.

Component Architecture

Each component follows the same internal pattern. The Measure package illustrates it: an internal pipeline of Load, Transform, Measure, and Store — each backed by a pluggable adapter.

Measure internal pipeline: Load → Transform → Measure → Store, each backed by an adapter

Measure component: Load → Transform → Measure → Store

Three-tier adapter pattern. External libraries (SARIMAX, statsmodels, PuLP) sit behind thin adapter wrappers that implement a shared interface — connect, validate, fit. A manager layer coordinates adapters, handles dependency injection, and manages storage. The adapter is the only code that knows about the library; everything above works against the interface.

Self-registering adapters. Each adapter registers itself via a decorator: @MODEL_REGISTRY.register_decorator("interrupted_time_series"). Six model adapters are available today. Adding a new one means writing one adapter class — zero changes to the pipeline, registry, or orchestrator.

Same pattern across stages. Allocate follows the identical structure: an AllocationSolver Protocol with pluggable solvers (minimax regret, Bayesian), injected into the component via constructor. Evaluate uses deterministic scoring functions keyed by model type.

Agentic Support

A shared support library (utils-agentic-support) structures AI-assisted development across all packages. Installed as a git submodule, it provides Claude Code with custom skills, specialized subagents, and ecosystem-aware templates.

Skills

General-purpose slash commands (feature workflows, code review, tech debt scanning) and ecosystem-specific ones (package scaffolding, cross-repo config sync, convention auditing).

Subagents

Specialized AI agents for code review, architecture evaluation, cross-boundary consistency checks, documentation generation, and test writing — each with scoped tool permissions.

Templates

Feature-type scaffolding for new pipeline components, adapters, and measurement models. Hierarchical CLAUDE.md files propagate conventions from workspace to component level.

Back to Home