mod-scheduler

Copyright (C) 2023-2024 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

Table of contents

• Introduction
• Compiling
• Running It
• Docker
• Environment variables
  • Kafka environment variables
    • Kafka tenant filtering
  • Retry environment variables
  • Secure storage environment variables
    • AWS-SSM
    • VAULT
    • Folio Secure Store Proxy (FSSP)
  • Quartz configuration properties
  • Cron format for timers

Introduction

mod-scheduler provides following functionality:

• schedule recurring jobs and/or events to happen at predefined times/intervals.

Compiling

mvn clean install

To compile project without tests

mvn clean install -DskipTests=true

See that it says "BUILD SUCCESS" near the end.

Running It

Run locally with proper environment variables set (see Environment variables below) on listening port 8081 (default listening port):

java \
  -Dserver.port=8081 \
  -DDB_HOST=localhost \
  -DDB_PORT=5432 \
  -DDB_DATABASE=postgres \
  -DDB_USERNAME=postgres \
  -DDB_PASSWORD=mysecretpassword \
  -Dokapi.url=http://localhost:9130 \
  -Dokapi.token=${okapiToken} \
  -jar target/mod-scheduler-*.jar

Docker

This method will require PostgreSQL database running as docker container.

docker run \
  --name postgres \
  -e PGUSER=postgres \
  -e POSTGRES_USERNAME=postgres \
  -e POSTGRES_PASSWORD=mysecretpassword \
  -p 5432:5432 \
  -d postgres:16-alpine

Build the docker container with:

docker build -t mod-scheduler .

Test that it runs with:

docker run \
  --name mod-scheduler \
  --link postgres:postgres \
  -e DB_HOST=postgres \
  -e DB_PORT=5432 \
  -e DB_DATABASE=postgres \
  -e DB_USERNAME=postgres \
  -e DB_PASSWORD=mysecretpassword \
  -e okapi.url=http://okapi:9130 \
  -e okapi.token=${okapiToken} \
  -p 8081:8081 \
  -d mod-scheduler

Environment variables

Name	Default value	Description
DB_HOST	localhost	Postgres hostname
DB_PORT	5432	Postgres port
DB_USERNAME	postgres	Postgres username
DB_PASSWORD	postgres	Postgres username password
DB_DATABASE	okapi_modules	Postgres database name
QUARTZ_STARTUP_DELAY	0s	Quartz initialization delay
QUARTZ_SCHEDULER_IDLE_WAIT_TIME	10000	Is the amount of time in milliseconds that the scheduler will wait before re-queries for available triggers
QUARTZ_MISFIRE_THRESHOLD	250	The number of milliseconds the scheduler will 'tolerate' a trigger to pass its next-fire-time
QUARTZ_CLUSTER_CHECKIN_INTERVAL	500	Set the frequency (in milliseconds) at which this instance "checks-in"* with the other instances of the cluster. Affects the quickness of detecting failed instances.
QUARTZ_POOL_THREAD_COUNT	5	The number of threads that are available for concurrent execution of jobs.
okapi.url	-	Okapi URL used to perform HTTP requests for recurring jobs, required.
OKAPI_URL	-	Alias for okapi.url.
SECRET_STORE_TYPE	VAULT	Secure storage type. Supported values: EPHEMERAL, AWS_SSM, VAULT, FSSP, required.
KC_INTEGRATION_ENABLED	true	Defines if Keycloak integration is enabled or disabled. If it set to false - it will exclude all keycloak-related beans from spring context.
KC_URL	http://keycloak:8080	Keycloak URL used to perform HTTP requests.
KC_IMPERSONATION_CLIENT	impersonation-client	Defined client in Keycloak, that has permissions to impersonate users.
KC_ADMIN_CLIENT_ID	be-admin-client	Keycloak admin client id.
SYSTEM_USER_USERNAME_TEMPLATE	{tenantId}-system-user	System user username template, used to generate system user username
KC_CLIENT_TLS_ENABLED	false	Enables TLS for keycloak clients.
KC_CLIENT_TLS_TRUSTSTORE_PATH	-	Truststore file path for keycloak clients.
KC_CLIENT_TLS_TRUSTSTORE_PASSWORD	-	Truststore password for keycloak clients.
KC_CLIENT_TLS_TRUSTSTORE_TYPE	-	Truststore file type for keycloak clients.
CLIENT_SECRET_KEY_CACHE_MAX_SIZE	200	Property sets the maximum number of client secret keys that can be stored in the cache
CLIENT_SECRET_KEY_CACHE_TTL	6000s	Property specifies the time-to-live for each cache entry
TOKEN_CACHE_INITIAL_CAPACITY	10	Token cache initial capacity.
TOKEN_CACHE_MAX_CAPACITY	50	Token cache max capacity.
TOKEN_CACHE_REFRESH_PRIOR_EXPIRATION	25	Specifies the amount of seconds for a cache entry invalidation prior to the token expiration.
SCHEDULER_API_ALLOW_SYSTEM_TIMER_MUTATION	false	Allow REST APIs to create, update, and delete SYSTEM timers.
SCHEDULER_API_ALLOW_USER_ID_UPDATE	false	Allow a USER timer's userId to be refreshed to the updating user on update. When false, userId is set once on creation and preserved across updates.

Kafka environment variables

Name	Default value	Description
KAFKA_HOST	kafka	Kafka broker hostname
KAFKA_PORT	9092	Kafka broker port
KAFKA_SECURITY_PROTOCOL	PLAINTEXT	Kafka security protocol used to communicate with brokers (SSL or PLAINTEXT)
KAFKA_SSL_KEYSTORE_LOCATION	-	The location of the Kafka key store file. This is optional for client and can be used for two-way authentication for client.
KAFKA_SSL_KEYSTORE_PASSWORD	-	The store password for the Kafka key store file. This is optional for client and only needed if 'ssl.keystore.location' is configured.
KAFKA_SSL_TRUSTSTORE_LOCATION	-	The location of the Kafka trust store file.
KAFKA_SSL_TRUSTSTORE_PASSWORD	-	The password for the Kafka trust store file. If a password is not set, trust store file configured will still be used, but integrity checking is disabled.
KAFKA_JOB_CONSUMER_PATTERN	(${folio.environment}.)(.*.)mgr-tenant-entitlements.scheduled-job	Custom subscription pattern for the scheduled-job Kafka consumer.
KAFKA_JOB_CONCURRENCY	1	Number of concurrent threads for the scheduled-job Kafka consumer.
KAFKA_ENTITLEMENT_CONSUMER_PATTERN	(${folio.environment}.)(.*.)entitlement	Custom subscription pattern for the entitlement-events Kafka consumer.
KAFKA_ENTITLEMENT_CONCURRENCY	1	Number of concurrent threads for the entitlement-events Kafka consumer.
KAFKA_CONSUMER_MAX_POLL_RECORDS	200	Maximum number of records returned in a single call to poll().
KAFKA_TENANT_FILTER_ENABLED	false	Enables tenant-entitlement filtering for Kafka messages. When true, events for tenants not entitled to this module version are filtered.
KAFKA_TENANT_FILTER_TENANT_DISABLED_STRATEGY	skip	Strategy when a message's tenant is not in the entitled set. skip silently discards the record; fail throws an exception and retries with backoff.
KAFKA_TENANT_FILTER_ALL_TENANTS_DISABLED_STRATEGY	fail	Strategy when no tenants at all are entitled (e.g. during startup). skip discards the record; fail retries with backoff until tenants become entitled.
MODULE_VERSION	-	Module version used for tenant-entitlement filter queries (e.g. 1.0.0). Typically injected by the deployment pipeline.

Kafka tenant filtering

When KAFKA_TENANT_FILTER_ENABLED=true, the module queries the tenant-entitlement service (GET /entitlements/modules/{moduleId}) before processing each scheduled-job Kafka event. Only events belonging to tenants currently entitled to this module version are delivered to the listener; others are handled according to the configured strategy.

Two independent strategies control the behaviour:

• KAFKA_TENANT_FILTER_TENANT_DISABLED_STRATEGY — applied when the entitled-tenant set exists but does not contain the event's tenant. Default: skip.
• KAFKA_TENANT_FILTER_ALL_TENANTS_DISABLED_STRATEGY — applied when no tenants are entitled at all (e.g. during startup or a full rollout). Default: fail, which causes the event to be retried with backoff until at least one tenant becomes entitled.

Retry environment variables

Name	Default value	Description
SYSTEM_USER_RETRY_DELAY	1s	Retry delay between attempts to retrieve system user
SYSTEM_USER_MAX_DELAY	1m	Maximum delay between attempts to retrieve system user
SYSTEM_USER_RETRY_ATTEMPTS	2147483647	Number of retry attempts to retrieve system user (default value is Long.MAX_VALUE ~= infinite amount of retries)
SYSTEM_USER_RETRY_MULTIPLIER	1.5	Retry attempts delay multiplier to retrieve system user
USER_IMPERSONATION_RETRY_DELAY	1s	Retry delay between attempts to obtain a user impersonation token for scheduled job execution
USER_IMPERSONATION_MAX_DELAY	30s	Maximum delay between attempts to obtain a user impersonation token
USER_IMPERSONATION_RETRY_ATTEMPTS	3	Number of retry attempts to obtain a user impersonation token
USER_IMPERSONATION_RETRY_MULTIPLIER	1.5	Retry attempts delay multiplier to obtain a user impersonation token
SCHEDULED_TIMER_EVENT_RETRY_DELAY	1s	Retry delay between attempts to process event from scheduled-job Kafka topic
SCHEDULED_TIMER_EVENT_ATTEMPTS	2147483647	Number of attempts to process event from scheduled-job Kafka topic (default value is Integer.MAX_VALUE ~= infinite amount of retries)
TIMER_EXECUTION_RETRY_DELAY	3s	Initial backoff before the first retry of a timer HTTP call
TIMER_EXECUTION_RETRY_MAX_DELAY	10s	Ceiling for the backoff between retries
TIMER_EXECUTION_RETRY_ATTEMPTS	4	Number of attempts for a timer HTTP call, including the initial one; 1 disables retries
TIMER_EXECUTION_RETRY_MULTIPLIER	2	Exponential multiplier applied to the backoff between retries

Timer execution uses a Keycloak user impersonation token as X-Okapi-Token. If token exchange returns a blank, missing, or literal null token, the response is not cached and the impersonation request is retried according to the USER_IMPERSONATION_* settings. If a valid token still cannot be obtained, the timer request is not sent.

Timer-execution retry behaviour

When a scheduled timer's HTTP call to a module fails with a transient error, mod-scheduler retries it with exponential backoff before giving up. The set of failures treated as transient is an allowlist — everything not listed is treated as permanent and not retried:

• any 5xx with sidecar error code authorization_error;
• connection refused, connect timeout, or connection-pool timeout before the request is sent.

Notably not retried: generic 5xx, any 4xx including 400, 401, 403, 404, 408 and 429, read timeouts, other I/O failures, DNS failures, and TLS failures. A read timeout is excluded because the target may still be processing the first request.

The allowlist deliberately excludes failures that may occur after the target module started processing the request. Each retry emits a structured timer.execution.retry log with its retry number and reason; response bodies and exception messages are not logged.

A single timer never overlaps itself: @DisallowConcurrentExecution holds the next fire until the running execution completes, per timer and cluster-wide. This applies to every timer, not only ones that retry.

Secure storage environment variables

Name	Default value	Description
SECURE_STORE_ENV	folio	First segment of the secure store key, for example prod or test. Defaults to folio. In Ramsons and Sunflower defaults to ENV with fall-back folio.

AWS-SSM

Required when SECRET_STORE_TYPE=AWS_SSM

Name	Default value	Description
SECRET_STORE_AWS_SSM_REGION	-	The AWS region to pass to the AWS SSM Client Builder. If not set, the AWS Default Region Provider Chain is used to determine which region to use.
SECRET_STORE_AWS_SSM_USE_IAM	true	If true, will rely on the current IAM role for authorization instead of explicitly providing AWS credentials (access_key/secret_key)
SECRET_STORE_AWS_SSM_ECS_CREDENTIALS_ENDPOINT	-	The HTTP endpoint to use for retrieving AWS credentials. This is ignored if useIAM is true
SECRET_STORE_AWS_SSM_ECS_CREDENTIALS_PATH	-	The path component of the credentials endpoint URI. This value is appended to the credentials endpoint to form the URI from which credentials can be obtained.

VAULT

Required when SECRET_STORE_TYPE=VAULT

Name	Default value	Description
SECRET_STORE_VAULT_TOKEN	-	token for accessing vault, may be a root token
SECRET_STORE_VAULT_ADDRESS	-	the address of your vault
SECRET_STORE_VAULT_ENABLE_SSL	false	whether or not to use SSL
SECRET_STORE_VAULT_PEM_FILE_PATH	-	the path to an X.509 certificate in unencrypted PEM format, using UTF-8 encoding
SECRET_STORE_VAULT_KEYSTORE_PASSWORD	-	the password used to access the JKS keystore (optional)
SECRET_STORE_VAULT_KEYSTORE_FILE_PATH	-	the path to a JKS keystore file containing a client cert and private key
SECRET_STORE_VAULT_TRUSTSTORE_FILE_PATH	-	the path to a JKS truststore file containing Vault server certs that can be trusted

Folio Secure Store Proxy (FSSP)

Required when SECRET_STORE_TYPE=FSSP

Name	Default value	Description
SECRET_STORE_FSSP_ADDRESS	-	The address (URL) of the FSSP service.
SECRET_STORE_FSSP_SECRET_PATH	secure-store/entries	The path in FSSP where secrets are stored/retrieved.
SECRET_STORE_FSSP_ENABLE_SSL	false	Whether to use SSL when connecting to FSSP.
SECRET_STORE_FSSP_TRUSTSTORE_PATH	-	Path to the truststore file for SSL connections.
SECRET_STORE_FSSP_TRUSTSTORE_FILE_TYPE	-	The type of the truststore file (e.g., JKS, PKCS12).
SECRET_STORE_FSSP_TRUSTSTORE_PASSWORD	-	The password for the truststore file.

Quartz configuration properties

mod-scheduler uses spring-boot-starter-quartz in cluster mode. Required configuration properties defined in application.yml under spring.quartz section.

In addition, Quartz can be tuned using Quart configuration properties

Each timer's Quartz job and trigger are placed in the group <tenant>#<moduleName> (rather than the default group), so scheduled jobs are isolated per tenant and module in the shared cluster.

Cron format for timers

mod-scheduler supports both Unix and Quartz cron formats for timers. The Unix format is automatically converted to Quartz, so you can use either format for cron-based timers. The formats are as follows:

Unix cron format:

<minute> <hour> <day-of-month> <month> <day-of-week>

Quartz cron format:

<second> <minute> <hour> <day-of-month> <month> <day-of-week> [year]

AI Documentation