gnss-product-management: Layers & Abstractions

Architecture blueprint for packages/gnss-product-management/src/gnss_product_management/

Application Boundaries

Boundary	Direction	Modules	Operations
Network (FTP/FTPS)	Outbound	`ConnectionPoolFactory`	List directories, download files
Network (HTTP/HTTPS)	Outbound	`ConnectionPoolFactory`	List directories (HTML parsing), download files
Filesystem (bundled configs)	Read	`gnss-management-specs` package	Load YAML specs at startup
Filesystem / Cloud (user workspace)	Read/Write	`WorkSpace`, `LockfileManager`	Search/store product files (local paths or `s3://` URIs)
User input	Inbound	`GNSSClient`, `ProductQuery`	Date, product name, constraints

Layer Definitions

┌──────────────────────────────────────────────────────────────┐
│  Layer 4: INTERFACE                                          │
│  GNSSClient, ProductQuery — user-facing entry point          │
│  ProductRegistry, WorkSpace — environment setup              │
│  (client/, environments/)                                    │
├──────────────────────────────────────────────────────────────┤
│  Layer 3: ORCHESTRATION                                      │
│  SearchPlanner, WormHole, ConnectionPoolFactory,             │
│  LockfileManager                                             │
│  Pipelines: DownloadPipeline, LockfileWriter,                │
│             ResolvePipeline                                  │
│  (factories/)                                                │
├──────────────────────────────────────────────────────────────┤
│  Layer 2: CATALOG (Resolution + Registry)                    │
│  FormatCatalog, ProductCatalog, ResourceCatalog,             │
│  SourcePlanner (Protocol)                                    │
│  (specifications/format/, specifications/products/,          │
│   specifications/remote/, factories/source_planner.py)       │
├──────────────────────────────────────────────────────────────┤
│  Layer 1: SPECIFICATION (Data Models + YAML Loading)         │
│  ParameterCatalog, FormatSpec, ProductSpec, SearchTarget,    │
│  LocalResourceSpec, DependencySpec, LockProduct              │
│  (specifications/, lockfile/models.py)                       │
├──────────────────────────────────────────────────────────────┤
│  Layer 0: CONFIGURATION (Static Data + Utilities)            │
│  Bundled YAML files,                                         │
│  as_path(), hash_file(), decompress_gzip()                   │
│  (utilities/, gnss-management-specs)                         │
└──────────────────────────────────────────────────────────────┘

Dependency Rule

Each layer may depend only on layers below it. No upward or lateral dependencies.

Layer 4 → Layer 3 → Layer 2 → Layer 1 → Layer 0

Layer 0: Configuration (Static Data + I/O Adapters)

Responsibility: Provide static bundled data, path constants, protocol-level I/O, and shared utility functions. No business logic.

Modules

Module	Concern	Boundary
`gnss-management-specs`	Path constants (META_SPEC_YAML, etc.) and all bundled YAML files	Filesystem (package resources)
`utilities/helpers.py`	`hash_file`, `decompress_gzip`, `_PassthroughDict`, `_listify`, `expand_dict_combinations`, `_ensure_datetime`	—
`utilities/metadata_funcs.py`	Computed field registration (DDD, GPSWEEK, etc.)	—
`utilities/paths.py`	`as_path(uri) -> Path \| CloudPath`, `AnyPath` type alias	—

Abstractions

as_path(uri) (utilities/paths.py): Single dispatch point for path construction. Returns cloudpathlib.CloudPath for s3://, gs://, az:// URIs and pathlib.Path for everything else. All filesystem operations throughout the package flow through this helper.
AnyPath: Union[Path, CloudPath] type alias used in signatures throughout layers 2–4.

Key Rule

Layer 0 must not import from any other layer. Utility functions operate on primitive types (strings, paths), not domain models.

Layer 1: Specification (Data Models + YAML Loading)

Responsibility: Define the domain vocabulary as Pydantic models. Load specifications from YAML. No resolution, no query logic, no I/O beyond YAML parsing.

Modules & Key Abstractions

Abstraction	Module	Description
`Parameter`	`specifications/parameters/parameter.py`	Single metadata field (name, value, pattern, derivation)
`ParameterCatalog`	`specifications/parameters/parameter.py`	Registry: parameter name → Parameter definition
`FormatFieldDef`	`specifications/format/spec.py`	Field definition within a format version
`FormatVersionSpec`	`specifications/format/spec.py`	A format version with metadata fields + file templates
`FormatSpec`	`specifications/format/spec.py`	Top-level format with versions
`FormatSpecCatalog`	`specifications/format/format_spec.py`	Loaded format specs from YAML
`FormatRegistry`	`specifications/format/spec.py`	Read-only lookup of raw format specs
`Product`	`specifications/products/product.py`	Concrete product: name, parameters, directory/filename templates
`PathTemplate`	`specifications/products/product.py`	Template string with `{PARAM}` placeholders + resolved value
`VariantCatalog[T]`	`specifications/products/product.py`	Generic: variant name → T
`VersionCatalog[T]`	`specifications/products/product.py`	Generic: version name → VariantCatalog[T]
`ProductSpec`	`specifications/products/catalog.py`	Abstract binding: product name + format ref + parameter overrides
`ProductSpecCatalog`	`specifications/products/catalog.py`	Loaded product specs from YAML
`Server`	`specifications/remote/resource.py`	Server endpoint (hostname, protocol, auth)
`ResourceSpec`	`specifications/remote/resource.py`	Root center spec: servers + product offerings
`SearchTarget`	`specifications/remote/resource.py`	Concrete query target: product + server + directory + filename
`ResourceCatalog`	`specifications/remote/resource_catalog.py`	Resolved SearchTargets per center
`LocalCollection`	`specifications/local/local.py`	Group of product specs sharing a directory template
`LocalResourceSpec`	`specifications/local/local.py`	Root local storage spec
`Dependency`	`specifications/dependencies/dependencies.py`	Single product dependency (spec name, required, constraints)
`SearchPreference`	`specifications/dependencies/dependencies.py`	Sort preference for a dependency resolution pass
`DependencySpec`	`specifications/dependencies/dependencies.py`	Full dependency declaration for a processing task
`ResolvedDependency`	`specifications/dependencies/dependencies.py`	Resolution result (status, URI string, remote URL)
`DependencyResolution`	`specifications/dependencies/dependencies.py`	Aggregated results for all dependencies in a spec
`LockProduct`	`lockfile/models.py`	Per-file lock entry (hash, size, source URL, sink URI)
`DependencyLockFile`	`lockfile/models.py`	Aggregate lockfile for one processing day

Interfaces

Each spec type exposes a from_yaml(path) -> Self classmethod for loading. All models are Pydantic BaseModel subclasses.

ResolvedDependency.local_path is a str URI — pass it through as_path() to get a filesystem object for I/O.

Key Rule

Layer 1 models are declarative data. They define what exists, not how to build or query it.

Layer 2: Catalog (Resolution + Registry)

Responsibility: Transform abstract specifications into concrete, queryable objects. Maintain registries for lookup. This is where specs become usable products.

Modules & Key Abstractions

Abstraction	Module	Input → Output
`Catalog` (ABC)	`specifications/catalog.py`	Base class enforcing `@classmethod build()` on all catalogs
`SourcePlanner` (Protocol)	`factories/source_planner.py`	Shared interface: `resource_ids`, `source_product()`, `sink_product()`, `register()`
`FormatCatalog`	`specifications/format/format_spec.py`	FormatSpecCatalog + ParameterCatalog → resolved Products per format/version/variant
`ProductCatalog`	`specifications/products/catalog.py`	ProductSpecCatalog + FormatCatalog → resolved Products per product/version/variant
`ResourceCatalog`	`specifications/remote/resource_catalog.py`	ResourceSpec + ProductCatalog → expanded SearchTarget list

Resolution Chain

ParameterCatalog ──┐
                   ├──► FormatCatalog ──► ProductCatalog ──► ResourceCatalog
FormatSpecCatalog ─┘    ProductSpecCatalog ─┘                   ↑ ResourceSpec[]

ProductRegistry (Layer 4 setup, Layer 2 usage) holds the built ResourceCatalog objects and implements the SourcePlanner Protocol for remote resource lookup. WorkSpace implements SourcePlanner for local resource lookup. SearchPlanner (Layer 3) delegates to both.

Interfaces

FormatCatalog.build(format_spec_catalog, parameter_catalog) -> FormatCatalog
ProductCatalog.build(product_spec_catalog, format_catalog) -> ProductCatalog
ResourceCatalog.build(resource_spec, product_catalog) -> ResourceCatalog

ProductRegistry and WorkSpace satisfy the SourcePlanner Protocol used by SearchPlanner (Layer 3):

resource_ids -> List[str]
source_product(product, resource_id) -> List[SearchTarget]

Key Rule

Catalogs are immutable after construction. Resolution happens once; the result is cached as data. No network I/O, no filesystem writes.

Layer 3: Orchestration (Query Building + Transport + Resolution)

Responsibility: Combine catalogs with user constraints to build, execute, and resolve queries. Touches external boundaries (network, filesystem, cloud storage).

Modules & Key Abstractions

Abstraction	Module	Responsibility
`SearchPlanner`	`factories/search_planner.py`	Date + product + constraints → `List[SearchTarget]`
`WormHole`	`factories/remote_transport.py`	Directory listing + filename matching + file download
`ConnectionPoolFactory`	`factories/connection_pool.py`	fsspec-backed connection pools per host (FTP, FTPS, HTTP, HTTPS, file)
`LockfileManager`	`lockfile/manager.py`	Lockfile lifecycle: check, load, save
`DownloadPipeline`	`factories/pipelines/download.py`	Pipeline: `FoundResource` → downloaded local/cloud path
`LockfileWriter`	`factories/pipelines/lockfile_writer.py`	Pipeline: write sidecar + aggregate lockfiles
`ResolvePipeline`	`factories/pipelines/resolve.py`	Pipeline: `DependencySpec` → `DependencyResolution` + lockfile

Data Flow

User constraints (date, product, parameters...)
        │
        ▼
   SearchPlanner
   ├── Resolve product templates (ProductCatalog)
   ├── Compute date fields (ParameterCatalog)
   ├── Narrow parameters (user constraints)
   ├── Expand combinations (cartesian product)
   ├── Local targets (WorkSpace.source_product)
   └── Remote targets (ProductRegistry.source_product)
        │
        ▼
   List[SearchTarget]  (directory pattern + filename pattern)
        │
        ▼
   WormHole
   ├── Group by (hostname, directory)
   ├── List directories in parallel (ConnectionPoolFactory)
   ├── Match filename patterns (regex)
   └── Optionally download (ConnectionPoolFactory.download_file)
        │
        ▼
   List[SearchTarget]  (filename.value populated)

Cloud / Local Filesystem Transparency

WorkSpace and LockfileManager accept base directories as URI strings, dispatched through as_path():

Local path: /data/gnss → pathlib.Path
S3 URI: s3://bucket/gnss → cloudpathlib.S3Path

All path operations (.exists(), .iterdir(), .read_text(), .write_text(), .mkdir(), / operator) are identical across backends. The LockfileManager stores aggregate lockfiles at the resource’s base_dir / "dependency_lockfiles", enabling distributed workers to coordinate via shared cloud storage.

Interfaces

SearchPlanner.get(date, product, parameters, ...) -> List[SearchTarget]
WormHole.search(List[SearchTarget]) -> List[SearchTarget] (one per matched file)
WormHole.download_one(query, local_resource_id, local_factory, date) -> AnyPath | None
ResolvePipeline.run(spec, date, sink_id, ...) -> Tuple[DependencyResolution, AnyPath | None]
LockfileManager(lockfile_dir: str | Path | CloudPath) — storage-agnostic

Key Rule

Orchestration modules coordinate between catalogs and I/O. They do not define domain models — they consume them. Network/filesystem operations are delegated to ConnectionPoolFactory (via fsspec) or cloudpathlib.

Layer 4: Interface (Entry Point)

Responsibility: Provide user-facing APIs that wire all layers together. Hide internal complexity behind clean, fluent interfaces.

Modules & Key Abstractions

Abstraction	Module	Responsibility
`GNSSClient`	`client/gnss_client.py`	Primary entry point: search, download, resolve dependencies
`ProductQuery`	`client/product_query.py`	Fluent query builder (`.for_product()`, `.on()`, `.where()`, `.sources()`, `.prefer()`, `.on_range()`)
`FoundResource`	`factories/models.py`	User-facing search result (hostname, filename, parameters, local_path)
`ProductRegistry`	`environments/environment.py`	Loads YAML specs, builds catalog chain, holds registered resource catalogs
`WorkSpace`	`environments/workspace.py`	Registers local/cloud storage directories against `LocalResourceSpec` layouts
`RegisteredLocalResource`	`environments/workspace.py`	Bound spec + `base_dir` URI (local path or cloud URI) + `base_path` property

Interfaces

# Construct from bundled defaults
client = GNSSClient.from_defaults(base_dir="/data/gnss")          # local
client = GNSSClient.from_defaults(base_dir="s3://bucket/gnss")   # S3

# Fluent query builder
results = (
    client.query()
          .for_product("ORBIT")
          .on(date)
          .where(TTT="FIN")
          .sources("COD", "WUM")
          .prefer(TTT=["FIN", "RAP"])
          .search()
)

# Date-range query (searches each day in parallel)
results = (
    client.query()
          .for_product("ORBIT")
          .on_range(start_date, end_date)
          .where(TTT="FIN")
          .search()
)

# Download
paths = client.download(results, sink_id="local")

# Full dependency resolution
resolution, lockfile_path = client.resolve_dependencies(dep_spec, date, sink_id="local")

For advanced use, construct manually:

registry = ProductRegistry()
registry.add_parameter_spec(META_SPEC_YAML)
registry.add_format_spec(FORMAT_SPEC_YAML)
registry.add_product_spec(PRODUCT_SPEC_YAML)
registry.add_resource_spec(center_yaml)
registry.build()

workspace = WorkSpace()
workspace.add_resource_spec(local_spec_yaml)
workspace.register_spec(base_dir="s3://my-bucket/gnss", spec_ids=["local_config"])

client = GNSSClient(env=registry, workspace=workspace)

Key Rule

All standard user code should interact through GNSSClient or ProductQuery. ProductRegistry and WorkSpace are setup objects; the pipelines and planners are internal implementation details.

Abstraction Inventory by Layer

Layer 0 (Configuration)         Layer 1 (Specification)
─────────────────────           ───────────────────────
as_path() / AnyPath             Parameter
hash_file()                     ParameterCatalog
decompress_gzip()               FormatFieldDef
_ensure_datetime()              FormatVersionSpec
register_computed_fields()      FormatSpec
gnss-management-specs YAMLs     FormatSpecCatalog
                                FormatRegistry
                                Product
                                PathTemplate
                                VariantCatalog[T]
                                VersionCatalog[T]
                                ProductSpec
                                ProductSpecCatalog
                                Server
                                ResourceSpec
                                SearchTarget
                                ResourceCatalog
                                LocalCollection
                                LocalResourceSpec
                                Dependency
                                SearchPreference
                                DependencySpec
                                ResolvedDependency
                                DependencyResolution
                                LockProduct
                                DependencyLockFile

Layer 2 (Catalog)               Layer 3 (Orchestration)
─────────────────               ───────────────────────
Catalog (ABC base)              SearchPlanner
SourcePlanner (Protocol)        WormHole
FormatCatalog                   ConnectionPoolFactory
ProductCatalog                  LockfileManager
ResourceCatalog                 DownloadPipeline
                                LockfileWriter
                                ResolvePipeline

Layer 4 (Interface)
───────────────────
GNSSClient
ProductQuery
FoundResource
ProductRegistry
WorkSpace
RegisteredLocalResource

Summary

Layer	Purpose	Depends On	Boundary
0 — Configuration	Static data, path utilities, bundled YAMLs	Nothing	Filesystem
1 — Specification	Domain models, YAML loading	Layer 0	Filesystem (YAML reads)
2 — Catalog	Resolve specs → concrete objects, registries	Layer 1	None (pure computation)
3 — Orchestration	Build queries, execute fetches, resolve deps, manage lockfiles	Layers 0, 1, 2	Network, Filesystem, Cloud
4 — Interface	User-facing API, environment wiring	Layers 1, 2, 3	User input