Frame retry default/max should be configurable without recompiling Cuebot

[DiegoTavares]

Summary

There is no way to configure the default (or maximum) frame retry count for submitted jobs without editing Java source and recompiling Cuebot. The values are compile-time constants in JobSpec.java, and the current defaults silently clamp every job to 1 retry regardless of what the client requests. This should be configurable via opencue.properties (and/or the CONFIG DB table) so operators can set a facility-wide default without a custom build.

Current behavior

Retries at submission time are governed by two compile-time constants in cuebot/src/main/java/com/imageworks/spcue/service/JobSpec.java:

public static final int FRAME_RETRIES_DEFAULT = 1;  // applied only when the spec omits <maxretries>
public static final int FRAME_RETRIES_MAX = 1;      // hard cap — clamps whatever the client sends
public static final int FRAME_RETRIES_MIN = 0;

Parse path (JobSpec.java ~L301, L317-324):

job.maxRetries = FRAME_RETRIES_DEFAULT;
...
if (jobTag.getChildTextTrim("maxretries") != null) {
    job.maxRetries = Integer.valueOf(jobTag.getChildTextTrim("maxretries"));
    if (job.maxRetries > FRAME_RETRIES_MAX) {
        job.maxRetries = FRAME_RETRIES_MAX;   // clamps DOWN to 1
    } else if (job.maxRetries < FRAME_RETRIES_MIN) {
        job.maxRetries = FRAME_RETRIES_MIN;
    }
}

The clamp is the footgun: pyoutline always emits a <maxretries> tag (default 2 from pyoutline/outline/outline.cfg), so every normal job hits the > FRAME_RETRIES_MAX branch and is clamped to 1. This has been the case since the initial 2018 import (FRAME_RETRIES_MAX = 1 unchanged).

At runtime, FrameCompleteHandler marks a frame DEAD when frame.retries >= job.maxRetries, so the clamped value directly limits how many times a frame reruns before dying.

Things that look like config but aren't

• job.int_max_retries SMALLINT DEFAULT 3 (V1__Initial_schema.sql) — never applies; the job INSERT always supplies job.maxRetries explicitly (JobDaoJdbc), overriding the column default.
• CONFIG.MAX_FRAME_RETRIES = 16 (seed_data.sql) — only consulted by JobDaoJdbc.updateMaxFrameRetries(), i.e. the post-submission SetMaxRetries gRPC/API ceiling. It is not read during job submission.
• No opencue.properties key exists for default/max frame retries. (dispatcher.frame_kill_retry_limit is unrelated — it governs kill-RPC retries.)

Why this is a problem

To make all jobs default to, say, 3 retries today, an operator must:

1. Edit FRAME_RETRIES_DEFAULT = 3 and FRAME_RETRIES_MAX = 3 in JobSpec.java (editing only DEFAULT does nothing, because the client's <maxretries> tag gets clamped by MAX),
2. Edit pyoutline/outline/outline.cfg maxretries = 3,
3. Rebuild and redeploy a custom Cuebot image.

That's a source fork just to change a policy number, which is a poor operator experience for a Dockerized deployment.

Proposed solution

Make the default and max frame retries runtime-configurable, with the existing constants as fallback defaults. Preferred: opencue.properties (consistent with other dispatcher tunables), e.g.:

# Default frame retries applied when a job spec omits <maxretries>
dispatcher.frame_retries_default=1
# Upper bound clamp applied to job-spec <maxretries> at submission
dispatcher.frame_retries_max=16

Wire these into JobSpec (via injected config) so FRAME_RETRIES_DEFAULT / FRAME_RETRIES_MAX read the property with the current constant as the fallback default. Alternatively (or additionally), source the submission-time max from the existing CONFIG.MAX_FRAME_RETRIES row so there's a single DB-backed source of truth shared with the SetMaxRetries API.

Acceptance criteria

• Operators can set a facility-wide default retry count via configuration only (no recompile).
• The submission-time clamp is configurable and defaults to a sane value (>= the client default so pyoutline's maxretries isn't silently reduced).
• Backward compatible: absent config, behavior matches today's constants.
• Document the new keys in opencue.properties (relates to Documentation: many properties in cuebot/opencue.properties lack inline docs #2393).

Additional notes

• Consider raising the stock FRAME_RETRIES_MAX default — clamping to 1 while pyoutline ships a default of 2 means the two defaults contradict each other out of the box.
• There's also a hardcoded job.maxRetries = 2 in the PostJobs path (JobSpec.java ~L945) that should honor the same config.