Merge pull request #216 from cortexapps/jeff/entity-relationship-reprocess

fix: two-pass re-import for catalog entities with x-cortex-relationships

Author: jeff-schnitter

.claude/skills/cli-release/SKILL.md

@@ -0,0 +1,174 @@
+---
+name: cli-release
+description: Use before creating any commit intended for release — determines correct version bump, ensures commit prefix is right, and walks through the release flow. Triggers on "release", "bump version", "merge to staging", "merge to main", "create a release", "what version will this be", or any PR from a feature branch to staging or main.
+---
+
+# CLI Release Skill
+
+Use this skill **before writing the commit message** for any change headed to production.
+Getting the prefix wrong means the wrong version ships — and fixing it requires an amended force-push.
+
+---
+
+## Step 1: Determine the correct version bump
+
+1. Check the current published tag:
+   ```bash
+   git fetch origin --tags
+   git describe --tags --abbrev=0
+   ```
+
+2. Examine commits since that tag on your branch:
+   ```bash
+   git log <last-tag>..HEAD --oneline
+   ```
+
+3. Apply the bump rules (highest wins):
+   | Commit prefix | Bump |
+   |---|---|
+   | `feat:` | **minor** — X.(Y+1).0 |
+   | `fix:` | **patch** — X.Y.(Z+1) |
+   | `chore:`, `docs:`, other | no bump (don't use for deliverable changes) |
+
+4. **State the expected version out loud** before any commit:
+   > "This will produce **1.20.1** because all commits use `fix:` (no `feat:`)."
+
+---
+
+## Step 2: Craft the commit message
+
+- Use `fix:` for bug fixes, `feat:` for new features.
+- If the prefix and bump disagree, add an explicit keyword override:
+  - `#patch` — force patch regardless of prefix
+  - `#minor` — force minor regardless of prefix
+  - `#major` — force major (rare)
+- **Only use `feat:`/`fix:` for commits that touch deliverable paths**: `cortexapps_cli/`, `pyproject.toml`, `poetry.lock`, `tests/`, `docker/`. Use `chore:` for anything else.
+
+Example:
+```
+fix: two-pass re-import for catalog entities with x-cortex-relationships #patch
+```
+
+---
+
+## Step 3: Choose the release path
+
+**Two valid paths — pick based on whether you're batching fixes:**
+
+### Option A: Feature → main directly (single fix, ship now)
+Use this for most fixes. CI runs, tests pass, PR auto-merges, publish fires.
+
+```bash
+gh pr create --base main --head <feature-branch> --title "fix: <description>"
+```
+
+- `test-pr.yml` runs tests and auto-merges when they pass.
+- `publish.yml` triggers on the resulting push to `main`.
+- The workflow auto-syncs HISTORY.md back to `staging` after publish.
+
+### Option B: Feature → staging → main (batching multiple fixes)
+Use this when you want to collect several fixes before cutting a release.
+
+```bash
+# Step 1: land feature on staging (tests run + automerge)
+gh pr create --base staging --head <feature-branch> --title "fix: <description>"
+# Step 2: when ready to release, merge staging → main (triggers publish)
+gh pr create --base main --head staging --title "Release X.Y.Z: <description>"
+```
+
+---
+
+## Human review (when needed)
+
+By default, PRs auto-merge when tests pass. When you need a human to review first:
+
+1. Open the PR as a **draft** — tests run for fast feedback, but automerge is skipped.
+2. Share for review. Reviewer approves while still in draft.
+3. Convert to ready (`gh pr ready <PR_NUMBER>`) — tests re-run, automerge fires.
+
+The reviewer approves *before* you convert, so you control the merge timing.
+
+```bash
+# Open as draft
+gh pr create --draft --base main --head <feature-branch> --title "fix: <description>"
+
+# After reviewer approves, convert to ready
+gh pr ready <PR_NUMBER>
+```
+
+---
+
+## Step 4: Monitor PR CI and auto-merge (no human in the loop)
+
+CI takes ~5 minutes. Use this polling loop — it waits 5 minutes before the first check, then polls every 60 seconds. This keeps API calls to a minimum.
+
+```bash
+echo "Waiting 5 minutes for CI..."; sleep 300
+while true; do
+  STATUS=$(gh pr checks <PR_NUMBER> 2>/dev/null | awk '{print $2}' | sort -u)
+  echo "$(date '+%H:%M:%S') checks: $STATUS"
+  if echo "$STATUS" | grep -q "fail"; then
+    echo "CI failed — inspect with: gh pr checks <PR_NUMBER>"; break
+  fi
+  if ! echo "$STATUS" | grep -qE "pending|in_progress|queued"; then
+    echo "All checks passed — merging"
+    gh pr merge <PR_NUMBER> --merge
+    break
+  fi
+  sleep 60
+done
+```
+
+**Critically: do NOT add any commits during or after this loop.** The publish workflow auto-commits `chore: update HISTORY.md for main` after merging. That commit is excluded from the `paths:` trigger filter (`HISTORY.md` is not in `cortexapps_cli/**`, `docker/**`, `pyproject.toml`, or `poetry.lock`), so it will **not** trigger a second publish run. Any commit you push that *does* touch those paths will trigger a new, unwanted build.
+
+---
+
+## Step 5: Monitor the publish workflow after merge
+
+The publish workflow runs 5 parallel jobs under the same `GH_TOKEN` PAT:
+`pypi` → `pypi-deploy-event`, `docker`, `docker-deploy-event`, `homebrew`
+
+Wait 2 minutes for the workflow to start, then poll:
+
+```bash
+echo "Waiting 2 minutes for publish workflow to start..."; sleep 120
+while true; do
+  gh run list --limit 5 --branch main --json status,conclusion,name,databaseId \
+    --jq '.[] | "\(.name) \(.status) \(.conclusion // "running") \(.databaseId)"'
+  echo "---"
+  DONE=$(gh run list --limit 1 --branch main --json status --jq '.[0].status')
+  [ "$DONE" = "completed" ] && break
+  sleep 60
+done
+# Check final result
+gh run list --limit 1 --branch main --json conclusion --jq '.[0].conclusion'
+```
+
+**If a job fails with a rate limit error — do NOT push a new commit.** Re-run only the failed jobs:
+```bash
+gh run rerun --failed <run-id>
+```
+
+**Confirm the tag was cut after a successful run:**
+```bash
+git fetch origin --tags && git tag --sort=-version:refname | head -3
+```
+
+---
+
+## Step 5: Homebrew dependency caveat
+
+`mislav/bump-homebrew-formula-action` **cannot** update `resource` blocks for Python dependencies.
+If `pyproject.toml` or `poetry.lock` changed dependency versions, manually update `cortexapps/homebrew-tap/Formula/cortexapps-cli.rb` resource blocks after release.
+
+---
+
+## Quick-reference: what lives where
+
+| Thing | Location |
+|---|---|
+| Versioning rules (full reference) | `CLAUDE.md` lines 165–227 |
+| Changelog prefixes (`add:`, `change:`, `remove:`) | `CLAUDE.md` lines 219–224 |
+| GitHub Actions release workflow | `.github/workflows/publish.yml` |
+| Homebrew formula (local copy) | `homebrew/cortexapps-cli.rb` |
+| Re-run failed jobs | `gh run rerun --failed <run-id>` |

.github/workflows/test-pr.yml

@@ -16,6 +16,7 @@ on:
   pull_request:
     branches:
       - staging
+      - main
     paths:
       - 'cortexapps_cli/**'
       - 'tests/**'
@@ -68,3 +69,18 @@ jobs:
     - name: Test with pytest
       run: |
         just test-all
+
+  automerge:
+    needs: test
+    # Only auto-merge on non-draft PRs. For PRs requiring human review, keep as
+    # draft until approved, then convert to ready — that re-triggers this workflow.
+    if: github.event_name == 'pull_request' && github.event.pull_request.draft == false
+    runs-on: ubuntu-latest
+    permissions:
+      pull-requests: write
+      contents: write
+    steps:
+    - name: Auto-merge PR
+      env:
+        GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      run: gh pr merge ${{ github.event.pull_request.number }} --merge --repo ${{ github.repository }}

cortexapps_cli/commands/backup.py

@@ -1,4 +1,5 @@
 from datetime import datetime
+import time
 from typing import Optional
 from typing import List
 from typing_extensions import Annotated
@@ -7,7 +8,6 @@
 import os
 import tempfile
 import sys
-from io import StringIO
 from contextlib import redirect_stdout, redirect_stderr
 from rich import print, print_json
 from rich.console import Console
@@ -471,6 +471,18 @@ def import_relationships_file(file_info):
 
     return ("entity-relationships", len(results) - failed_count, [(fp, et, em) for rt, fp, et, em in results if et])
 
+def _has_relationships(file_path):
+    """Check if a catalog YAML file contains x-cortex-relationships."""
+    try:
+        with open(file_path) as f:
+            content = yaml.safe_load(f)
+        info = content.get('info', {})
+        relationships = info.get('x-cortex-relationships')
+        return relationships is not None and len(relationships) > 0
+    except Exception:
+        return False
+
+
 def _import_catalog(ctx, directory):
     results = []
     failed_count = 0
@@ -493,7 +505,7 @@ def import_catalog_file(file_info):
             except Exception as e:
                 return (filename, file_path, type(e).__name__, str(e))
 
-        # Import all files in parallel
+        # Pass 1: Import all files
         with ThreadPoolExecutor(max_workers=30) as executor:
             futures = {executor.submit(import_catalog_file, file_info): file_info[0] for file_info in files}
             results = []
@@ -506,6 +518,113 @@ def import_catalog_file(file_info):
         if failed_count > 0:
             print(f"\n   Total catalog import failures: {failed_count}")
 
+        # Pass 2: Delete and re-create entities with x-cortex-relationships.
+        # On first creation (pass 1), the relationship processor runs synchronously but
+        # the entity hasn't been committed to the DB yet, so it caches a failed state.
+        # Subsequent updates don't clear this cache. Deleting the entity clears the cache,
+        # so the fresh re-create triggers proper relationship processing.
+        #
+        # Two-wave approach handles multi-tier hierarchies: Wave 1 re-creates all
+        # relationship entities (e.g., clusters → services, where services are stable from
+        # pass 1). Wave 2 re-creates entities whose destinations were also recreated in
+        # wave 1 (e.g., accounts → clusters), ensuring destinations exist before the
+        # relationship processor runs for those dependent sources.
+        relationship_files = [(fn, fp) for fn, fp in files if _has_relationships(fp)]
+        if relationship_files:
+            print(f"\n   Re-importing {len(relationship_files)} entities with relationships...")
+
+            # Collect all tags being recreated in pass 2
+            pass2_tags = set()
+            for fn, fp in relationship_files:
+                try:
+                    with open(fp) as f:
+                        content = yaml.safe_load(f)
+                    tag = content.get('info', {}).get('x-cortex-tag')
+                    if tag:
+                        pass2_tags.add(tag)
+                except Exception:
+                    pass
+
+            def delete_entity(file_info):
+                filename, file_path = file_info
+                try:
+                    with open(file_path) as f:
+                        content = yaml.safe_load(f)
+                    tag = content.get('info', {}).get('x-cortex-tag')
+                    if tag:
+                        catalog.delete(ctx, tag=tag)
+                except Exception:
+                    pass  # Entity may not exist; that's fine
+
+            def create_entity(file_info):
+                filename, file_path = file_info
+                print(f"   Re-importing: {filename}")
+                try:
+                    with open(file_path) as f:
+                        catalog.create(ctx, file_input=f, _print=False)
+                    return (filename, file_path, None, None)
+                except typer.Exit as e:
+                    return (filename, file_path, "HTTP", "Validation or HTTP error")
+                except Exception as e:
+                    return (filename, file_path, type(e).__name__, str(e))
+
+            def has_pass2_destination(file_path):
+                """Return True if any of this entity's relationship destinations are
+                also being recreated in pass 2 (i.e., they need to exist first)."""
+                try:
+                    with open(file_path) as f:
+                        content = yaml.safe_load(f)
+                    info = content.get('info', {})
+                    for rel in info.get('x-cortex-relationships', []):
+                        for dest in rel.get('destinations', []):
+                            if dest.get('tag') in pass2_tags:
+                                return True
+                except Exception:
+                    pass
+                return False
+
+            reprocess_results = []
+
+            # Wave 1: Delete all, then create all. This establishes every entity
+            # in the DB and resolves relationships whose destinations are stable
+            # (i.e., not being recreated in this pass).
+            with ThreadPoolExecutor(max_workers=30) as executor:
+                futures = {executor.submit(delete_entity, fi): fi[0] for fi in relationship_files}
+                for future in as_completed(futures):
+                    future.result()
+
+            time.sleep(2)  # Allow deletes to commit before re-creating
+
+            with ThreadPoolExecutor(max_workers=30) as executor:
+                futures = {executor.submit(create_entity, fi): fi[0] for fi in relationship_files}
+                for future in as_completed(futures):
+                    reprocess_results.append(future.result())
+
+            # Wave 2: For entities whose destinations were also recreated in wave 1,
+            # do another delete+create now that those destinations exist in the DB.
+            dependent_files = [(fn, fp) for fn, fp in relationship_files if has_pass2_destination(fp)]
+            if dependent_files:
+                time.sleep(5)  # Let wave 1 entities settle in the DB
+
+                with ThreadPoolExecutor(max_workers=30) as executor:
+                    futures = {executor.submit(delete_entity, fi): fi[0] for fi in dependent_files}
+                    for future in as_completed(futures):
+                        future.result()
+
+                time.sleep(2)
+
+                with ThreadPoolExecutor(max_workers=30) as executor:
+                    futures = {executor.submit(create_entity, fi): fi[0] for fi in dependent_files}
+                    for future in as_completed(futures):
+                        reprocess_results.append(future.result())
+
+            reprocess_failed = sum(1 for fn, fp, et, em in reprocess_results if et)
+            if reprocess_failed > 0:
+                print(f"\n   Total catalog re-import failures: {reprocess_failed}")
+                failed_count += reprocess_failed
+            else:
+                print(f"   Note: relationships are processed asynchronously and will be visible within 30 seconds.")
+
     return ("catalog", len(results) - failed_count, [(fp, et, em) for fn, fp, et, em in results if et])
 
 def _import_plugins(ctx, directory):