feat: packagist worker

[epipav]

No description provided.

[cursor]

PR Summary

Medium Risk
Large new external crawl (~500M HTTP calls at scale) and significant versions/package_dependencies growth, but it mirrors proven npm/pypi patterns with bounded concurrency, watermark drains, and transactional writes.

Overview
Introduces end-to-end Packagist (PHP/Composer) ingestion because deps.dev has no Packagist coverage—the registry crawl is the universe source. A new packagist-worker Temporal service runs four workflows: weekly seed from list.json, a metadata drain (dynamic stats + p2 manifests with If-Modified-Since), monthly 30d download capture for all packages, and daily downloads for the critical slice only.

Persistence & design: Adds packagist_package_state (per-lane watermarks and run results), writes versions and direct/dev dependency edges with declared constraints (depends_on_version_id NULL per ADR-0009), repo links, and audit trails. Shared DAL gains reconcileVersionDependencies for manifest reconciliation. Documented in ADR-0009 and worker README.

Ops & wiring: Docker service YAML, build scripts, CROWD_PACKAGES_PACKAGIST_MAILTO, manual trigger-packagist CLI, vitest project for packagist tests. Security-contacts registry fetcher key changes from composer to packagist.

Smaller fixes: canonicalizeRepoUrl now handles GitLab nested subgroups and legacy deep-link routes (e.g. raw, blob); minor ADR-0005 decider name typo fix.

^{Reviewed by Cursor Bugbot for commit 5c0e3e2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Adds a Packagist worker that discovers Composer packages and ingests metadata, dependencies, maintainers, repositories, and download statistics.

Changes:

• Adds Packagist Temporal workflows, schedules, persistence, and deployment wiring.
• Adds DAL queries and ingestion-state migration.
• Adds documentation and unit tests.

Reviewed changes

Copilot reviewed 41 out of 43 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/packages/versions.ts	Adds Packagist version upserts.
services/libs/data-access-layer/src/packages/packagistPackageState.ts	Adds ingestion watermark queries.
services/libs/data-access-layer/src/packages/packages.ts	Adds Packagist package DAL operations.
services/libs/data-access-layer/src/packages/index.ts	Exports new DAL modules.
services/libs/data-access-layer/src/packages/dependencies.ts	Adds dependency-edge upserts.
services/apps/packages_worker/vitest.config.ts	Isolates Packagist tests.
services/apps/packages_worker/src/workflows/index.ts	Exports Packagist workflows.
services/apps/packages_worker/src/security-contacts/extractors/registry/index.ts	Registers Packagist contact extraction.
services/apps/packages_worker/src/scripts/triggerPackagistSeed.ts	Adds manual workflow trigger.
services/apps/packages_worker/src/packagist/workflows.ts	Defines ingestion workflows.
services/apps/packages_worker/src/packagist/upsertPackageInfo.ts	Persists package information.
services/apps/packages_worker/src/packagist/upsertMetadata.ts	Persists versions and dependencies.
services/apps/packages_worker/src/packagist/types.ts	Defines Packagist types.
services/apps/packages_worker/src/packagist/schedule.ts	Registers Temporal schedules.
services/apps/packages_worker/src/packagist/retryPolicy.ts	Defines retry limit.
services/apps/packages_worker/src/packagist/README.md	Documents worker behavior.
services/apps/packages_worker/src/packagist/normalize.ts	Normalizes registry metadata.
services/apps/packages_worker/src/packagist/listPackages.ts	Fetches and parses package catalog.
services/apps/packages_worker/src/packagist/fetchPackage.ts	Fetches package endpoints.
services/apps/packages_worker/src/packagist/expandMetadata.ts	Expands minified metadata.
services/apps/packages_worker/src/packagist/downloads.ts	Persists 30-day downloads.
services/apps/packages_worker/src/packagist/activities.ts	Implements Temporal activities.
services/apps/packages_worker/src/packagist/__tests__/wiring.test.ts	Tests worker wiring.
services/apps/packages_worker/src/packagist/__tests__/persistMetadata.test.ts	Tests metadata persistence.
services/apps/packages_worker/src/packagist/__tests__/normalize.test.ts	Tests normalization.
services/apps/packages_worker/src/packagist/__tests__/listPackages.test.ts	Tests catalog handling.
services/apps/packages_worker/src/packagist/__tests__/ingest.test.ts	Tests ingestion behavior.
services/apps/packages_worker/src/packagist/__tests__/fetchPackage.test.ts	Tests HTTP handling.
services/apps/packages_worker/src/packagist/__tests__/expandMetadata.test.ts	Tests metadata expansion.
services/apps/packages_worker/src/packagist/__tests__/dueSelection.test.ts	Tests due-selection queries.
services/apps/packages_worker/src/packagist/__tests__/downloads.test.ts	Tests download windows.
services/apps/packages_worker/src/bin/packagist-worker.ts	Adds worker entrypoint.
services/apps/packages_worker/src/activities.ts	Exports Packagist activities.
services/apps/packages_worker/package.json	Adds worker scripts.
scripts/services/packagist-worker.yaml	Adds local service configuration.
scripts/builders/packages.env	Adds worker build target.
docs/adr/README.md	Indexes Packagist ADR.
docs/adr/0006-packagist-worker-design-decisions.md	Records design decisions.
docs/adr/0005-pypi-downloads-bigquery-merge-scoping.md	Expands decider name.
backend/src/osspckgs/migrations/V1783382400__packagist_worker.sql	Creates ingestion-state table.
backend/.env.dist.local	Adds local crawler contact setting.
backend/.env.dist.composed	Adds composed crawler contact setting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 836b50e. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 47 out of 48 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

• This ADR does not follow the repository's mandatory ADR structure: .claude/rules/adr-format.md:16-28 requires ## Decision, ## Alternatives Considered, and a top-level ## Consequences with Positive/Negative/Risks subsections. The plural consolidated section here replaces those required sections, so the document needs restructuring before merge.

## Decisions

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

• This new ADR does not follow the mandatory repository ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences in that order (with Positive/Negative/Risks subsections), but this document uses a living ## Decisions layout and omits the alternatives section. Please restructure it to the enforced template or split the consolidated decisions into compliant ADRs.

## Decisions

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

docs/adr/0009-packagist-worker-design-decisions.md:33

• This new ADR does not follow .claude/rules/adr-format.md: the mandatory heading is ## Decision (singular), followed by ## Alternatives Considered and ## Consequences with ### Positive, ### Negative, and ### Risks. Those required sections are absent or use a different structure, so please reformat the ADR before merging.

## Decisions

[Copilot]

Pull request overview

Copilot reviewed 48 out of 49 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

services/apps/packages_worker/src/packagist/activities.ts:225

• A successful response without a numeric downloads.monthly currently becomes a no-op and still advances downloads_30d_last_run_at. Because this lane has no backfill and the month is never revisited, a partial/malformed response permanently loses that month's observation. Treat the missing count as an error (so Temporal retries and eventually records an error) and only mark success after a numeric value is persisted.

  await persistPackagist30dWindow(qx, purl, info.value.package.downloads?.monthly ?? null, runDate)
  await markPackagist30dProcessed(qx, purl, { status: 'success', attempts: info.attempts })

services/libs/data-access-layer/src/packages/versions.ts:165

• This cleanup can change versions.is_latest, but its row count is discarded, so changedFields may omit the only mutation (for example, when the incoming latest was already true while another stale row was also true). persistPackagistMetadata then writes no audit token for the cleanup. Capture the affected-row count and add versions.is_latest when any stale flag was cleared.
  docs/adr/0009-packagist-worker-design-decisions.md:33
• This ADR does not follow the repository's mandatory ADR structure. .claude/rules/adr-format.md:18-28 requires ## Decision, ## Alternatives Considered, and ## Consequences with ### Positive, ### Negative, and ### Risks; this file uses ## Decisions, has no alternatives section, and embeds differently named consequence blocks per decision. Restructure it to the required template or update the governing rule before accepting the ADR.

## Decisions

services/apps/packages_worker/src/packagist/activities.ts:270

• When downloads.daily is absent or non-numeric, this records the run as successful even though no row was inserted. The next schedule uses a new run date, so the missing day is never retried or backfilled. Treat the missing count as an error and only advance the daily watermark after a numeric count has been persisted.

  await markPackagistDailyProcessed(qx, candidate.purl, {
    status: 'success',
    attempts: info.attempts,
  })