ERE Prototype Template

A modular, configurable Entity Resolution Engine (ERE) prototype that implements the ERS–ERE contract defined in entity-resolution-spec.

This template provides a working skeleton for building conformant ERE implementations, complete with BDD test cases, pluggable storage backends (MongoDB, PostgreSQL+pgvector), YAML-based configuration, Docker deployment, and structured traffic logging for use as an ERS testing platform.

To build your own ERE, you can start by cloning this repository, choosing the storage engine (InMemory, Postgres or MongoDN) and selecting the components you need. Then implement the "clustering" code that adjusts to your particular use case.

Note: Currently the configuration files enable MongoDB for storage.

Key Features

Pluggable storage backends — InMemory, MongoDB 8.2 with mongot vector search, PostgreSQL + pgvector
Pluggable messaging — InMemory (for testing) or Redis with connection pool
Concurrent workers — Configurable thread pool (1–32 workers) for parallel request processing
Property extraction pipeline — Parse RDF/TTL content, extract structured properties, apply preprocessing filters, batch-embed, and combine with configurable weights
Blocking keys — Deterministic key partitioning to reduce similarity comparison space at scale
Per-entity-type singleton lock — Prevents duplicate cluster creation under concurrent load
Representative update — Replaces cluster representative when a near-identical entity has more extracted properties, with automatic member re-scoring
Vector search — Native $vectorSearch via MongoDB mongot sidecar or pgvector HNSW indexes
Vectorized similarity — NumPy matrix multiplication for batch cosine computation in the fallback loop
Two-tier embedding cache — In-memory LRU + filesystem cache with automatic eviction
Structured traffic logging — Colour-coded console output for all incoming/outgoing messages

Architecture

The ERE Service is composed of loosely-coupled components, each with a single responsibility:

graph TB
    subgraph ERE Service
        Main[main.py<br/>Entry Point]
        Config[config.py<br/>YAML Config]
        Service[service.py<br/>Orchestrator]
        Engine[engine.py<br/>Resolution Logic]
        PropertyPipeline[property_extraction.py<br/>RDF Extraction Pipeline]
        Blocking[blocking.py<br/>Blocking Keys]
        Health[health.py<br/>HTTP Health Probes]
        Messaging[messaging.py<br/>Transport Layer]
        Storage[storage.py<br/>Persistence Interface]
        Embedder[embedder.py<br/>Text Embeddings]
        Traffic[traffic.py<br/>Traffic Logger]
        Models[models.py<br/>Pydantic Models]
    end

    subgraph Storage Backends
        InMemory[InMemoryStorage]
        MongoDB[MongoDBStorage<br/>MongoDB 8.2 + mongot]
        Postgres[PostgresStorage + pgvector]
    end

    subgraph Messaging Backends
        InMemBroker[InMemoryBroker]
        RedisBroker[RedisBroker<br/>ConnectionPool]
    end

    Main --> Config
    Main --> Service
    Main --> Health
    Service --> Engine
    Service --> Messaging
    Service --> Traffic
    Engine --> Storage
    Engine --> Embedder
    Engine --> PropertyPipeline
    Engine --> Blocking
    Storage --> InMemory
    Storage --> MongoDB
    Storage --> Postgres
    Messaging --> InMemBroker
    Messaging --> RedisBroker

Resolution Request Flow

sequenceDiagram
    participant Client as ERE Client (CLI)
    participant Redis as Redis Broker
    participant Service as EREService
    participant Engine as ResolutionEngine
    participant Pipeline as PropertyExtractionPipeline
    participant Storage as StorageBackend
    participant Embedder as HuggingFaceEmbedder

    Client->>Redis: RPUSH ere_requests (JSON)
    Redis-->>Service: BLPOP ere_requests
    Service->>Service: _validate_request()
    Service->>Engine: resolve(entity_mention, proposed, excluded)
    alt TTL content + pipeline configured
        Engine->>Pipeline: extract_and_embed(content)
        Pipeline->>Pipeline: parse → extract → preprocess
        Pipeline->>Embedder: embed_many(property_values)
        Embedder-->>Pipeline: vectors[]
        Pipeline->>Pipeline: combine_weighted_embeddings()
        Pipeline-->>Engine: ExtractionResult(embedding, parsed_repr)
    else Full-content embedding
        Engine->>Embedder: embed(content)
        Embedder-->>Engine: vector[384]
    end
    Engine->>Storage: find_similar_clusters(embedding, type)
    Storage-->>Engine: [(cluster_id, score)]
    alt similarity >= threshold
        Engine->>Storage: store_entity + add_entity_to_cluster
        Engine->>Engine: _maybe_update_representative (Q7)
    else new singleton (with lock)
        Engine->>Engine: _acquire_singleton_lock(entity_type)
        Engine->>Storage: store_entity + create_cluster
        Engine->>Engine: _release_singleton_lock(entity_type)
    end
    Engine-->>Service: ResolutionResult(candidates)
    Service->>Redis: RPUSH ere_responses (JSON)
    Redis-->>Client: BLPOP ere_responses

For the full set of diagrams (error handling, CLI interactions, storage operations, messaging layer), see docs/sequence-diagrams.md.

Documentation

Detailed documentation is available in the docs/ folder:

Document	Contents
Architecture	Component diagram, deployment topology, data flow
Sequence Diagrams	All runtime message flow diagrams (6 diagrams)
Class Diagrams	UML class hierarchies for both packages
Development Guide	Setup, testing, storage config, Docker patterns, troubleshooting

Modules

Module	Purpose
`ere/config.py`	YAML configuration loading and validation via Pydantic
`ere/models.py`	Pydantic data models implementing the ERS–ERE message contract
`ere/engine.py`	Core resolution/clustering logic with pluggable similarity, singleton locks, and quality enhancements
`ere/messaging.py`	Transport layer (InMemory for testing, Redis with connection pool for production)
`ere/storage.py`	Storage interface, in-memory backend, and factory
`ere/storage_mongodb.py`	MongoDB persistent storage backend with `$vectorSearch` and blocking key queries
`ere/storage_postgres.py`	PostgreSQL + pgvector persistent storage backend
`ere/property_extraction.py`	Weighted property extraction pipeline with batch embedding
`ere/preprocessing.py`	Text normalization filters (uppercase, strip suffixes, strip_telephone_chars, etc.)
`ere/rdf_parser.py`	RDF/Turtle parser with graph caching and multi-hop property path extraction
`ere/embedder.py`	HuggingFace sentence-transformers with two-tier cache (memory + filesystem)
`ere/blocking.py`	Blocking key computation for search space reduction (Q4)
`ere/similarity_utils.py`	Shared NumPy-based cosine similarity
`ere/health.py`	HTTP liveness/readiness probe server for container orchestration
`ere/service.py`	Top-level orchestration: validation, resolution, response publishing
`ere/traffic.py`	Structured, colour-coded traffic logging for all incoming/outgoing messages
`ere/main.py`	Entry point: consumer loop with concurrent workers and graceful shutdown

Requirements

Python 3.12+
Poetry for dependency management
Make (for convenience targets)
Docker & Docker Compose (for containerized deployment)
Redis (optional — for messaging backend)
MongoDB 8.2 with mongot (optional — for MongoDB storage backend with vector search)
PostgreSQL 15+ with pgvector (optional — for vector similarity storage)

Quick Start

Local Development (no Docker)

# Install core dependencies
make install

# Install with MongoDB storage support
poetry install --extras mongodb

# Install with PostgreSQL storage support
poetry install --extras postgres

# Install all storage backends
poetry install --extras all-storage

# Run all tests (unit + BDD)
make test

# Start the service with in-memory backends (no Redis needed)
make run

Docker Deployment

# Build the Docker image
make docker-build

# Start ERE only (in-memory backends)
make docker-up

# Start ERE + Redis
make docker-up-redis

# Start ERE + MongoDB + Vector Search + Mongo Express
make docker-up-mongodb

# Start ERE + PostgreSQL (pgvector)
make docker-up-postgres

# Start all services (full stack)
make docker-up-full

# View logs
make docker-logs

# Stop everything
make docker-down

MongoDB with Vector Search (first-time setup)

The MongoDB profile uses MongoDB Community Server 8.2 with a mongot sidecar for native $vectorSearch support. Before first use:

# Generate keyfile and mongot password (one-time)
bash config/mongodb/generate-secrets.sh

# Start the MongoDB profile
docker compose --profile mongodb up -d

The stack will:

Start MongoDB 8.2 as a single-node replica set
Run an init container to create the replica set + users (admin, mongotUser, ere)
Start the mongot sidecar (connects to mongod, syncs data for vector search)
Start the ERE service and Mongo Express GUI

Concurrency

The ERE supports configurable concurrent request processing via a thread pool:

limits:
  workers: 4  # 1 = sequential, 2-32 = thread pool

When workers > 1, the service dispatches incoming requests to a ThreadPoolExecutor. Concurrency-safe operations include:

Per-entity-type singleton lock — Serializes new cluster creation within an entity type to prevent duplicate singletons when multiple workers process similar entities simultaneously.
Dedup check before lock — Brute-force similarity scan catches clusters created by other workers not yet in the vector search index.
Post-lock verification — After acquiring the singleton lock, re-checks for matching clusters created by other workers during the wait.
Redis connection pool — The RedisBroker uses a ConnectionPool (20 connections) so worker threads can publish responses concurrently without contention.
Thread-safe storage — InMemoryStorage uses threading.Lock for all mutations.

Quality Enhancements

The engine implements several quality improvements beyond basic cosine similarity:

Enhancement	Config Parameter	Description
Q1: Identifier Boost	`identifier_exact_match_boost`	When incoming entity's `legal_identifier` exactly matches a cluster representative's, the similarity score is boosted (default: +0.15)
Q4: Blocking Keys	`blocking_enabled`	Pre-filter clusters by blocking keys (country + name prefix, identifier prefix) to reduce comparison space. Disable for small datasets.
Q6: Mismatch Penalty	`identifier_mismatch_penalty`	When both entities have `legal_identifier` but they differ, similarity score is penalized (default: -0.20)
Q7: Representative Update	`representative_update_threshold`	When a new entity matches a cluster at ≥ threshold AND has more extracted properties, it replaces the representative. All existing member scores are re-computed against the new representative embedding.

Health Check

The service exposes HTTP health check endpoints on port 8081:

Endpoint	Description
`GET /health/live`	Returns 200 if the process is alive
`GET /health/ready`	Returns 200 if all components are initialized and the consumer loop is running

These are suitable for Kubernetes livenessProbe and readinessProbe configuration.

Docker Deployment

Profiles

The docker-compose.yml uses Docker Compose profiles to make Redis and Redis Insight optional:

Profile	Services Started	Use Case
(none)	ERE only	In-memory testing, no external deps
`redis`	ERE + Redis	Production-like messaging
`redis-insight`	ERE + Redis + Redis Insight	Development with Redis GUI
`mongodb`	ERE + MongoDB + mongot + init + Mongo Express	MongoDB storage with vector search
`postgres`	ERE + PostgreSQL + pgAdmin	pgvector storage with GUI
`full`	All services	Full development/testing stack

Usage Examples

# ERE with in-memory backends (default)
docker compose up -d ere

# ERE + Redis (messaging via Redis queues)
docker compose --profile redis up -d

# Full stack with Redis Insight GUI
docker compose --profile full up -d

# Using the override file (activates all services without profiles)
docker compose -f docker-compose.yml -f docker-compose.redis.yml up -d

Connecting to an External Redis

If your ERS already runs a Redis instance (e.g., ersys-redis on the ersys-local network), you don't need the containerized one. Simply:

Edit config/ere_config_docker.yaml:

messaging:
  backend: "redis"
  redis:
    host: "ersys-redis"
    port: 6379
    db: 0
    request_channel: "ere_requests"
    response_channel: "ere_responses"

Start only the ERE container:
```
docker compose up -d ere
```

Or override via environment variable:

ERE_REDIS_HOST=ersys-redis docker compose up -d ere

Ports

Service	Default Port	Environment Variable
Redis	6379	`REDIS_PORT`
Redis Insight	5540	`REDISINSIGHT_PORT`
MongoDB	27017	`MONGODB_PORT`
mongot health	28080	`MONGOT_HEALTH_PORT`
mongot metrics	29946	`MONGOT_METRICS_PORT`
Mongo Express	8081	`MONGO_EXPRESS_PORT`
PostgreSQL	5432	`POSTGRES_PORT`
pgAdmin	5050	`PGADMIN_PORT`
ERE Health	8081	—

Volumes

Volume	Purpose
`redis-data`	Redis persistence (AOF)
`redisinsight-data`	Redis Insight configuration
`mongodb-data`	MongoDB document storage
`mongot-data`	mongot vector search index data
`postgres-data`	PostgreSQL data directory
`pgadmin-data`	pgAdmin configuration and saved servers

To reset all data:

make docker-down-volumes

Storage Backends

The ERE supports pluggable persistent storage. Select the backend via storage.backend in the config file.

In-Memory (default)

No external dependencies. Fast but all data is lost on restart. Ideal for testing and development.

storage:
  backend: "in_memory"

MongoDB

Document-based storage using MongoDB Community Server 8.2+ with native $vectorSearch via the mongot sidecar process. Stores entities and clusters as JSON documents with vector search indexes for efficient approximate nearest-neighbor similarity queries at scale.

storage:
  backend: "mongodb"
  mongodb:
    uri: "mongodb://ere:ere_dev_password@mongodb:27017/ere_prototype?authSource=admin&directConnection=true"
    database: "ere_prototype"

Install the driver: poetry install --extras mongodb

Docker: docker compose --profile mongodb up -d

Architecture: The MongoDB profile runs 4 containers:

ere-mongodb — MongoDB 8.2 as a single-node replica set with keyfile auth
ere-mongodb-init — One-shot container that initializes the replica set and creates users
ere-mongot — The vector search sidecar (syncs data via change streams, serves $vectorSearch)
ere-mongo-express — Web GUI for inspecting documents

Vector Search: When mongot is available, the storage layer creates vector search indexes on the entities and clusters collections automatically. The engine uses $vectorSearch for similarity queries instead of loading all embeddings into memory. Includes an automatic fallback to in-memory cosine scan when $vectorSearch is unavailable.

Blocking Keys: The MongoDB backend has a dedicated blocking_keys index and uses $in queries for efficient blocking key pre-filtering (Q4).

Licensing note: MongoDB 8.x is licensed under the Server Side Public License (SSPL), which is NOT an OSI-approved open-source license. See the FerretDB alternative below if SSPL compliance is a concern.

PostgreSQL + pgvector

Relational storage with vector similarity search via the pgvector extension. Enables embedding-based ANN (approximate nearest neighbour) queries directly in the database.

storage:
  backend: "postgres"
  postgres:
    host: "localhost"
    port: 5432
    database: "ere_prototype"
    user: "ere"
    password: "ere_dev_password"
    vector_dimension: 384

Install the driver: poetry install --extras postgres

Docker: docker compose --profile postgres up -d

Roadmap: FerretDB

Due to MongoDB's SSPL licensing, the project roadmap includes support for FerretDB as a drop-in replacement. FerretDB is:

Licensed under Apache 2.0 (OSI-approved)
Wire-protocol compatible with MongoDB (uses the same pymongo driver)
Backed by PostgreSQL or SQLite for actual storage

When FerretDB support is validated, it will be usable by simply pointing the mongodb.uri to a FerretDB instance — no code changes required.

Configuration

All behaviour is controlled via config/ere_config.yaml. Key sections:

service:
  supported_entity_types: [...]   # Entity types this ERE handles

clustering:
  similarity_threshold: 0.75      # Min similarity to join existing cluster
  confidence_threshold: 0.50      # Min confidence to include in candidates
  max_candidates: 5               # Max candidates per response
  similarity_algorithm: "cosine"  # cosine | jaccard | levenshtein
  # Quality enhancements
  identifier_exact_match_boost: 0.30   # Q1: boost on identifier match
  identifier_mismatch_penalty: 0.02    # Q6: penalty on identifier mismatch
  representative_update_threshold: 0.90 # Q7: threshold for representative swap
  blocking_enabled: false               # Q4: blocking key pre-filter

messaging:
  backend: "redis"                # in_memory | redis
  redis:
    host: "localhost"
    port: 6379
    password: "changeme"
    request_channel: "ere_requests"
    response_channel: "ere_responses"
  poll_timeout: 5

storage:
  backend: "mongodb"              # in_memory | mongodb | postgres
  mongodb:
    uri: "mongodb://localhost:27017"
    database: "ere_prototype"

embedding:
  enabled: true
  model_name: "sentence-transformers/all-MiniLM-L6-v2"
  batch_size: 32

limits:
  max_content_size: 1048576       # 1 MB
  processing_timeout: 25          # seconds
  workers: 4                      # concurrent worker threads

property_extraction:              # Per-entity-type property extraction
  ORGANIZATION:
    - name: "legal_name"
      xpath: "org:Organization/epo:hasLegalName"
      weight: 4.0
      filter: ["trim", "uppercase", "strip_company_suffixes", "strip_dots"]
    - name: "legal_identifier"
      xpath: "org:Organization/epo:hasLegalIdentifier/skos:notation"
      weight: 2.0
      filter: ["uppercase", "alphanumeric"]
    # ... additional properties (email, telephone, address, etc.)

logging:
  level: "INFO"
  traffic_log_payloads: true
  traffic_log_max_content_length: 500

Two config files are provided:

config/ere_config.yaml — Local development (Redis on localhost)
config/ere_config_docker.yaml — Docker deployment (Redis via service name)

Traffic Logging

When used as a testing platform for the ERS, the ERE provides structured, colour-coded console output for all message traffic:

Incoming Requests

────────────────────────────────────────────────────────────────────────
▶ INCOMING REQUEST  [324fs3r345vx:01]
────────────────────────────────────────────────────────────────────────
  type:           EntityMentionResolutionRequest
  ere_request_id: 324fs3r345vx:01
  entity_triad:   TEDSWS / 324fs3r345vx / ORGANIZATION
  content_type:   text/turtle
  content_size:   142 chars
  content:
    epd:ent005 a org:Organization; cccev:telephone "+44 1924306780" .
────────────────────────────────────────────────────────────────────────

Outgoing Responses

────────────────────────────────────────────────────────────────────────
◀ OUTGOING RESPONSE  [324fs3r345vx:01]  (12.3ms)
────────────────────────────────────────────────────────────────────────
  type:           EntityMentionResolutionResponse
  ere_request_id: 324fs3r345vx:01
  entity_triad:   TEDSWS / 324fs3r345vx / ORGANIZATION
  candidates:     2 cluster(s)
    ★ 324fs3r345vx-aa32wa                  conf=0.910  sim=0.880
    · 324fs3r345vx-bb45we                  conf=0.650  sim=0.620
────────────────────────────────────────────────────────────────────────

Testing

Tests are organized into:

Unit tests (tests/test_*.py) — Test individual modules in isolation
BDD feature tests (tests/features/*.feature + tests/step_defs/) — Test contract compliance

Running Tests

make test          # All tests
make test-bdd      # BDD scenarios only
make test-unit     # Unit tests only
make coverage      # Tests with coverage report

Contract Compliance

This template implements the following contract guarantees:

Type discrimination — type field enables polymorphic deserialization
Correlation — ere_request_id links responses to requests
Positional selection — Best candidate is always candidates[0]
Deterministic IDs — Singleton cluster IDs derived from SHA-256 of entity triad
Advisory constraints — proposed_cluster_ids and excluded_cluster_ids are hints, not mandates
Re-resolution — Re-submitted entities are removed from old clusters before re-resolving
Idempotency — Same request → same response (stable ordering)
Error model — RFC-9457 structured errors via EREErrorResponse

Make Targets

Target	Description
`install`	Install dependencies via Poetry
`lint`	Run ruff linter
`format`	Auto-format code
`test`	Run all tests
`test-bdd`	Run BDD tests only
`test-unit`	Run unit tests only
`coverage`	Tests with coverage report
`run`	Start ERE locally (in-memory)
`run-redis`	Start ERE locally (Redis backend)
`docker-build`	Build Docker image
`docker-up`	Start ERE container only
`docker-up-redis`	Start ERE + Redis
`docker-up-mongodb`	Start ERE + MongoDB + Mongo Express
`docker-up-postgres`	Start ERE + PostgreSQL + pgAdmin
`docker-up-full`	Start all services + GUIs
`docker-up-override`	Start all via override compose file
`docker-down`	Stop all containers
`docker-down-volumes`	Stop and remove volumes
`docker-logs`	Tail all container logs
`docker-logs-ere`	Tail ERE logs only
`docker-ps`	Show container status
`docker-restart`	Restart full stack
`clean`	Remove Python caches
`clean-docker`	Remove Docker images/volumes
`clean-all`	Full cleanup

Extending

To build a production ERE from this template:

Tune property extraction weights — Adjust weights in the property_extraction config section based on your domain. Properties like legal_name should have high weight (most discriminative), while structured identifiers should have lower weight.
Tune similarity thresholds — Adjust similarity_threshold based on evaluation results. Higher values reduce false positives but increase singletons.
Enable blocking — Set blocking_enabled: true for datasets with 10K+ clusters to reduce comparison space.
Choose a storage backend — MongoDB with mongot for vector search at scale, or PostgreSQL + pgvector as an alternative.
Add entity type handlers — Extend the property extraction configuration to handle different entity types (e.g., PERSON, PROCEDURE) with specialized XPath selectors and filter chains.
Scale workers — Increase limits.workers for higher throughput. Use 2–4 for CPU-bound embedding workloads, up to 8 for I/O-bound scenarios.
Consider a better embedding model — The default all-MiniLM-L6-v2 (384 dimensions) is lightweight but limited for very short texts. Larger models like all-mpnet-base-v2 (768d) may improve quality at the cost of speed.
Customize Docker — Adjust resource limits, configure the health check endpoint for your orchestrator, or integrate with Kubernetes.

License

This project is licensed under the European Union Public Licence (EUPL) v1.2.

The EUPL is a copyleft license approved by the OSI (Open Source Initiative) and compatible with GPL v2/v3, AGPL v3, LGPL, MPL 2.0, and other major open-source licenses. It was created by the European Commission and is available in all EU official languages.

Third-Party Licensing Notes

Component	License	Notes
This project	EUPL v1.2	OSI-approved, GPL-compatible
MongoDB 8.x	SSPL	NOT OSI-approved. See FerretDB alternative.
PostgreSQL	PostgreSQL License	Permissive, BSD-like
pgvector	PostgreSQL License	Same as PostgreSQL
Redis	BSD-3-Clause (server), MIT (client)	Permissive
FerretDB (roadmap)	Apache 2.0	OSI-approved, EUPL-compatible
Python dependencies	Various (MIT, BSD, Apache)	All OSI-approved

If your deployment requires fully OSI-compliant dependencies, use the PostgreSQL backend or wait for validated FerretDB support as a MongoDB alternative.

Name		Name	Last commit message	Last commit date
Latest commit History 49 Commits
config		config
docs		docs
ere-client		ere-client
ere		ere
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.redis.yml		docker-compose.redis.yml
docker-compose.yml		docker-compose.yml
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

ERE Prototype Template

Key Features

Architecture

Resolution Request Flow

Documentation

Modules

Requirements

Quick Start

Local Development (no Docker)

Docker Deployment

MongoDB with Vector Search (first-time setup)

Concurrency

Quality Enhancements

Health Check

Docker Deployment

Profiles

Usage Examples

Connecting to an External Redis

Ports

Volumes

Storage Backends

In-Memory (default)

MongoDB

PostgreSQL + pgvector

Roadmap: FerretDB

Configuration

Traffic Logging

Incoming Requests

Outgoing Responses

Testing

Running Tests

Contract Compliance

Make Targets

Extending

License

Third-Party Licensing Notes

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages