Skip to content

architecture.md

Latest commit

 

History

History
229 lines (196 loc) · 9.16 KB

architecture.md

File metadata and controls

229 lines (196 loc) · 9.16 KB
Raw

Architecture

This document describes the high-level architecture of the ERE Prototype Template, including internal component structure and deployment topology.

For runtime behaviour, see sequence-diagrams.md. For type hierarchies, see class-diagrams.md. For setup instructions, see development-guide.md.

1. Component Architecture (ERE Service Internals)

The ERE Service is composed of several loosely-coupled components, each with a single responsibility. The service follows the Composition Root pattern where all dependencies are wired together in main.py.

graph TB
    subgraph ERE Service
        Main[main.py<br/>Entry Point & Composition Root]
        Config[config.py<br/>YAML Config + Pydantic Validation]
        Service[service.py<br/>Request Orchestrator]
        Engine[engine.py<br/>Resolution & Clustering Logic]
        PropertyPipeline[property_extraction.py<br/>RDF Property Extraction Pipeline]
        RDFParser[rdf_parser.py<br/>TTL Parsing & Property Queries]
        Preprocessing[preprocessing.py<br/>Text Preprocessing Filters]
        Messaging[messaging.py<br/>Transport Abstraction]
        Storage[storage.py<br/>Persistence Interface]
        Embedder[embedder.py<br/>Text → Vector Embeddings]
        Traffic[traffic.py<br/>Structured Traffic Logger]
        Models[models.py<br/>Pydantic Message Models]
    end

    subgraph Storage Backends
        InMemory[InMemoryStorage<br/>dict-based, no persistence]
        MongoDB[MongoDBStorage<br/>MongoDB 8.x + Atlas Vector Search]
        Postgres[PostgresStorage<br/>PostgreSQL 15 + pgvector HNSW]
    end

    subgraph Messaging Backends
        InMemBroker[InMemoryBroker<br/>deque-based, testing only]
        RedisBroker[RedisBroker<br/>Redis LIST FIFO queues]
    end

    Main --> Config
    Main --> Service
    Service --> Engine
    Service --> Messaging
    Service --> Traffic
    Engine --> Storage
    Engine --> Embedder
    Engine --> PropertyPipeline
    PropertyPipeline --> RDFParser
    PropertyPipeline --> Preprocessing
    PropertyPipeline --> Embedder
    Storage --> InMemory
    Storage --> MongoDB
    Storage --> Postgres
    Messaging --> InMemBroker
    Messaging --> RedisBroker
Loading

Component Responsibilities

Component File Responsibility
Config config.py Load YAML, validate via Pydantic, apply env var overrides
Service service.py Validate requests, delegate to engine, publish responses
Engine engine.py Similarity computation, clustering decisions, canonical IDs
Property Pipeline property_extraction.py Orchestrate RDF parsing → extraction → preprocessing → embedding → weighted combination
RDF Parser rdf_parser.py Parse TTL content into RDF graphs, extract property values via XPath selectors
Preprocessing preprocessing.py Text normalization filters (uppercase, lowercase, trim, remove_special_characters)
Storage storage.py Abstract persistence interface + InMemory implementation
Messaging messaging.py Abstract transport interface + InMemory/Redis implementations
Embedder embedder.py Sentence-transformer embeddings with file cache
Traffic traffic.py Coloured console logging of message payloads
Models models.py Pydantic models for the ERS–ERE message contract
Main main.py CLI args, component wiring, consumer loop, signal handling
Blocking blocking.py Deterministic blocking key computation for search space reduction
Health health.py HTTP liveness/readiness probe server (daemon thread)

Dependency Flow

main.py
  ├── load_config() → EREConfig
  ├── create_storage(config.storage) → StorageBackend
  ├── create_broker(config.messaging) → MessageBroker
  ├── create_embedder(config.embedding) → HuggingFaceEmbedder (optional)
  ├── create_property_pipelines(config.property_extraction, embedder)
  │     └── For each entity type with property config:
  │           PropertyExtractionPipeline(properties, embedder, RDFParser())
  ├── ResolutionEngine(config.clustering, storage, embedder, pipelines)
  ├── EREService(config, engine, broker, storage)
  ├── start_health_server() → HTTP on port 8081
  └── consumer loop: broker.consume_request() → service.process_request()
        └── workers > 1: dispatch to ThreadPoolExecutor

2. Deployment Architecture (Docker Compose)

The project uses Docker Compose with profiles to enable selective infrastructure deployment. Only the ERE service runs by default; infrastructure services are activated by specifying profiles.

graph TB
    subgraph "Docker Compose Stack"
        subgraph "Core (always active)"
            ERE[ERE Service<br/>Python 3.12<br/>Port: internal only]
        end

        subgraph "Profile: redis"
            Redis[Redis 7<br/>Port 6379<br/>Persistence: appendonly]
        end

        subgraph "Profile: redis-insight"
            RedisInsight[Redis Insight<br/>Port 5540<br/>Web UI for Redis]
        end

        subgraph "Profile: mongodb"
            MongoDB[MongoDB 8.x<br/>Port 27017<br/>SSPL License]
            MongoExpress[Mongo Express<br/>Port 8081<br/>Web Admin UI]
        end

        subgraph "Profile: postgres"
            PostgreSQL[PostgreSQL 15 + pgvector<br/>Port 5432<br/>HNSW indexes]
            pgAdmin[pgAdmin 4<br/>Port 5050<br/>Web Admin UI]
        end
    end

    ERE -->|"BLPOP/RPUSH<br/>ere:requests/responses"| Redis
    ERE -->|"pymongo<br/>mongodb://localhost:27017"| MongoDB
    ERE -->|"psycopg<br/>postgresql://localhost:5432"| PostgreSQL
    RedisInsight -->|"inspect"| Redis
    MongoExpress -->|"admin UI"| MongoDB
    pgAdmin -->|"admin UI"| PostgreSQL
Loading

Profile Combinations

Use Case Command Services Started
Development (InMemory) docker compose up ERE only
Redis messaging docker compose --profile redis up ERE + Redis
MongoDB storage docker compose --profile redis --profile mongodb up ERE + Redis + MongoDB + Mongo Express
PostgreSQL storage docker compose --profile redis --profile postgres up ERE + Redis + PostgreSQL + pgAdmin
Full stack docker compose --profile redis --profile mongodb --profile postgres --profile redis-insight up All services
Monitoring only docker compose --profile redis --profile redis-insight up ERE + Redis + Redis Insight

Network Topology

  • All services communicate on the default Docker bridge network
  • Redis is the primary inter-service communication channel (ERE ↔ Client)
  • Storage backends (MongoDB, PostgreSQL) are accessed directly by the ERE service
  • Admin UIs (Mongo Express, pgAdmin, Redis Insight) are accessed by developers via host port mapping

Port Mappings

Service Container Port Host Port Purpose
Redis 6379 6379 Messaging transport
Redis Insight 5540 5540 Redis monitoring UI
MongoDB 27017 27017 Document storage
Mongo Express 8081 8081 MongoDB admin UI
PostgreSQL 5432 5432 Relational + vector storage
pgAdmin 5050 5050 PostgreSQL admin UI

Volume Management

  • Redis: redis-data volume for AOF persistence
  • MongoDB: mongo-data volume for WiredTiger storage
  • PostgreSQL: postgres-data volume for tablespace data
  • Admin UIs are stateless (no persistent volumes)

3. Data Flow Summary

ERS Client (ere-client CLI)
    │
    │ RPUSH ere:requests (JSON)
    ▼
Redis (FIFO queue)
    │
    │ BLPOP ere:requests
    ▼
EREService
    │
    ├── validate(entity_type, content_size)
    │
    ▼
ResolutionEngine
    │
    ├── IF content_type == "text/turtle" AND property config exists:
    │   │
    │   ▼ PropertyExtractionPipeline
    │   ├── RDFParser.parse(ttl_content) → Graph
    │   ├── For each configured property:
    │   │     extract_property(graph, xpath) → values[]
    │   │     sort + concatenate → raw_value
    │   │     PreprocessingFilter.apply(raw_value, filter) → clean_value
    │   │     Embedder.embed(clean_value) → vector[384]
    │   ├── combine_weighted_embeddings(vectors, weights) → Composite_Vector
    │   └── normalize(Composite_Vector) → unit vector
    │
    ├── ELSE (non-TTL or no property config):
    │   └── embed(full_content) → vector[384]
    │
    ├── get_all_clusters(entity_type) → clusters[]
    ├── compute_similarity(embedding, representative) per cluster
    │
    ├── IF similarity >= threshold → assign to cluster
    │   ├── store_entity()
    │   └── add_entity_to_cluster()
    │
    └── ELSE → create singleton
        ├── compute_canonical_id(triad)
        ├── store_entity()
        └── create_cluster()
    │
    ▼
EREService
    │
    │ RPUSH ere:responses (JSON)
    ▼
Redis (FIFO queue)
    │
    │ BLPOP ere:responses
    ▼
ERS Client (display response)