🦄 Unicorn.Land ▸ docs/plans/2026-06-14-oannes-claim-identity.md
updated 2026-06-14

Oannes Claim Identity & Link Repair Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Give claims a stable identity so episode→claim links stop dangling, and repair the ~1,768 existing dangling links / 407 orphan claims across the vault.

Architecture: Two phases. Phase 1 (deterministic, high-impact): route every claim filename and every claim link through one shared canonical-name function, then run a one-time migration that rewrites existing episode links to match the real claim files. This fixes the mismatches caused by inconsistent normalization/truncation. Phase 2 (semantic, optional): add a stable id to each claim’s frontmatter and a Claude-assisted dedup pass to merge claims that are the same assertion phrased differently (the residue Phase 1 can’t reach).

Tech Stack: Python 3, pytest, the existing Oannes pipeline (generate.py, shared/io.py, enrich_links.py), Obsidian vault markdown with YAML frontmatter, Claude CLI (claude --print) for the semantic pass.

Background: the two failure mechanisms

  1. Inconsistent normalization (deterministic — Phase 1 target). - File creation: generate_claim_file builds safe_filename("Claim - {summary}") then truncates to 117 chars + “…” (generate.py:406-411). - Link creation: _resolve_claim_name returns f"Claim - {extracted_summary}" raw — no safe_filename, no truncation (generate.py:729-730). - Result: any summary with unsafe characters or >120 chars yields a link that cannot match its own file. This is mechanical and fully repairable in code.

  2. Wording drift (semantic — Phase 2 target). - The same assertion is summarized with different words by the extraction step vs. the resolution step vs. across episodes (e.g. “…hovered over a Minuteman II silo at Ellsworth AFB…” vs. “…was hovering directly over the November 5 silo blast door…”). - No string-normalization can reconcile these; only semantic matching can.

File Structure

Run all commands from oannes/Oannes/_scripts/. Run tests with python3 -m pytest.

Task 1: Single canonical claim-name function

Files: - Modify: oannes/Oannes/_scripts/shared/io.py - Test: oannes/Oannes/_scripts/tests/test_claim_identity.py

# tests/test_claim_identity.py
from shared.io import canonical_claim_name, safe_filename

def test_canonical_is_safe_and_truncated():
    long = "x" * 300
    name = canonical_claim_name(long)
    assert name.startswith("Claim - ")
    assert len(name) <= 120
    # must equal what a filesystem-safe filename would be
    assert name == safe_filename(name)

def test_canonical_strips_unsafe_chars():
    name = canonical_claim_name('UFO over "Area 51": part 1/2?')
    for ch in '/\\:|<>"?*':
        assert ch not in name

def test_canonical_is_idempotent():
    once = canonical_claim_name("A spherical UFO hovered over a silo")
    twice = canonical_claim_name(once.removeprefix("Claim - "))
    assert once == twice

Run: python3 -m pytest tests/test_claim_identity.py -v Expected: FAIL with ImportError: cannot import name 'canonical_claim_name'

# shared/io.py — add below safe_filename
CLAIM_NAME_MAX = 120

def canonical_claim_name(summary: str) -> str:
    """Canonical filename stem AND link text for a claim.

    The single source of truth: file creation and link generation must both
    call this so a claim's file and the links pointing at it always agree.
    """
    stem = f"Claim - {summary.strip()}"
    stem = safe_filename(stem)
    if len(stem) > CLAIM_NAME_MAX:
        stem = stem[: CLAIM_NAME_MAX - 3] + "..."
    return stem

Run: python3 -m pytest tests/test_claim_identity.py -v Expected: PASS (3 passed)

git add oannes/Oannes/_scripts/shared/io.py oannes/Oannes/_scripts/tests/test_claim_identity.py
git commit -m "oannes: add canonical_claim_name as single source of truth for claim naming"

Task 2: Route generation and linking through the canonical function

Files: - Modify: oannes/Oannes/_scripts/generate.py:406-411 (file creation) - Modify: oannes/Oannes/_scripts/generate.py:722-730 (_resolve_claim_name) - Test: oannes/Oannes/_scripts/tests/test_claim_identity.py

# append to tests/test_claim_identity.py
from generate import _resolve_claim_name
from shared.io import canonical_claim_name

def test_resolve_claim_name_matches_canonical_for_long_summary():
    summary = "A spherical UFO the size of a Walmart building hovered over a Minuteman II missile silo at Ellsworth AFB on November 5 1977"
    # resolution has no entry -> falls back to canonical, NOT raw "Claim - {summary}"
    resolved = _resolve_claim_name(summary, {"claims": []})
    assert resolved == canonical_claim_name(summary)
    assert len(resolved) <= 120

Run: python3 -m pytest tests/test_claim_identity.py::test_resolve_claim_name_matches_canonical_for_long_summary -v Expected: FAIL — current code returns raw f"Claim - {summary}" (length 130, not truncated)

In generate.py, ensure the import near the other shared.io imports includes it:

from shared.io import safe_filename, atomic_write, canonical_claim_name

Replace the filename block in generate_claim_file (currently generate.py:405-411):

    # Canonical claim name = single source of truth (file + links agree)
    claim_name = canonical_claim_name(summary)

Replace _resolve_claim_name (currently generate.py:722-730):

def _resolve_claim_name(extracted_summary: str, resolution: dict) -> str | None:
    """Look up a claim's vault name from the resolution data."""
    for entry in resolution.get("claims", []):
        if entry.get("extracted_summary") == extracted_summary:
            action = entry.get("action", "skip")
            if action == "skip":
                return None
            vault_name = entry.get("vault_name")
            if vault_name:
                # canonicalize the resolution's name the same way files are named
                return canonical_claim_name(vault_name.removeprefix("Claim - "))
            return canonical_claim_name(extracted_summary)
    return canonical_claim_name(extracted_summary)

Run: python3 -m pytest tests/test_claim_identity.py -v Expected: PASS (4 passed)

Run: python3 -c "import generate; print('ok')" Expected: ok

git add oannes/Oannes/_scripts/generate.py oannes/Oannes/_scripts/tests/test_claim_identity.py
git commit -m "oannes: generate claim files and links through canonical_claim_name"

Files: - Create: oannes/Oannes/_scripts/migrate_claim_links.py - Test: oannes/Oannes/_scripts/tests/test_claim_identity.py

The migration builds a lookup of real claim files, then for every [[Claim - …]] link in every episode rewrites it to the matching real file. Matching tiers: (1) exact, (2) canonicalized form of the link, (3) canonicalized truncation-prefix match against real files. Anything still unmatched is logged, never guessed.

# append to tests/test_claim_identity.py
from migrate_claim_links import best_match

def test_best_match_exact():
    files = {"Claim - A hovered over B"}
    assert best_match("Claim - A hovered over B", files) == "Claim - A hovered over B"

def test_best_match_via_truncation_prefix():
    real = "Claim - " + "y" * 109 + "..."   # 120 chars, truncated file
    files = {real}
    link = "Claim - " + "y" * 200            # raw long link
    assert best_match(link, files) == real

def test_best_match_returns_none_when_no_match():
    assert best_match("Claim - totally unrelated", {"Claim - something else"}) is None

Run: python3 -m pytest tests/test_claim_identity.py -k best_match -v Expected: FAIL with ModuleNotFoundError: No module named 'migrate_claim_links'

# migrate_claim_links.py
"""One-time repair of dangling [[Claim - ...]] links in episode files.

Rewrites each episode's claim links to the canonical filename of the real
claim file they were meant to point at. Logs links it cannot resolve; never
invents a target. Use --dry-run to preview.
"""
import os, re, glob, sys

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, SCRIPT_DIR)
from shared.io import canonical_claim_name

VAULT = os.path.dirname(SCRIPT_DIR)
EPISODES = os.path.join(VAULT, "Episodes")
CLAIMS = os.path.join(VAULT, "Claims")
LINK_RE = re.compile(r"\[\[(Claim - [^]|#]+?)(\|[^]]+)?\]\]")


def real_claim_files():
    return {os.path.basename(p)[:-3] for p in glob.glob(os.path.join(CLAIMS, "*.md"))}


def best_match(link_name, files):
    """Resolve a link to a real claim filename, or None."""
    if link_name in files:
        return link_name
    canon = canonical_claim_name(link_name.removeprefix("Claim - "))
    if canon in files:
        return canon
    # truncation-prefix: a real file is canon's truncated form
    stripped = canon[:-3] if canon.endswith("...") else canon
    prefix = stripped[: max(0, len(stripped) - 1)]
    cands = [f for f in files if f.startswith(prefix[:60]) and f.endswith("...")]
    return cands[0] if len(cands) == 1 else None


def migrate(dry_run=True):
    files = real_claim_files()
    rewritten = unresolved = eps_changed = 0
    for ep in glob.glob(os.path.join(EPISODES, "*.md")):
        text = open(ep, encoding="utf-8").read()
        changed = False

        def repl(m):
            nonlocal rewritten, unresolved, changed
            name, alias = m.group(1).strip(), (m.group(2) or "")
            target = best_match(name, files)
            if target is None:
                unresolved += 1
                return m.group(0)
            if target != name:
                rewritten += 1
                changed = True
                return f"[[{target}{alias}]]"
            return m.group(0)

        new_text = LINK_RE.sub(repl, text)
        if changed and not dry_run:
            open(ep, "w", encoding="utf-8").write(new_text)
        if changed:
            eps_changed += 1
    print(f"{'DRY-RUN ' if dry_run else ''}rewritten={rewritten} "
          f"unresolved={unresolved} episodes_changed={eps_changed}")


if __name__ == "__main__":
    migrate(dry_run="--apply" not in sys.argv)

Run: python3 -m pytest tests/test_claim_identity.py -k best_match -v Expected: PASS (3 passed)

Run: python3 migrate_claim_links.py Expected: a line like DRY-RUN rewritten=NNNN unresolved=MMM episodes_changed=KK. Sanity check: rewritten should be in the high hundreds/low thousands (it targets the ~1,768 dangling links); unresolved is the Phase-2 residue.

Run: python3 migrate_claim_links.py --apply Then verify dangling links dropped:

python3 - <<'PY'
import os, re, glob
cf={os.path.basename(p)[:-3] for p in glob.glob("../Claims/*.md")}
d=sum(1 for ep in glob.glob("../Episodes/*.md")
      for l in re.findall(r"\[\[(Claim - [^]|#]+)", open(ep,encoding="utf-8").read())
      if l.strip() not in cf)
print("remaining dangling claim links:", d)
PY

Expected: a number far below the starting 1,768 (the remainder is wording-drift for Phase 2).

git add oannes/Oannes/_scripts/migrate_claim_links.py oannes/Oannes/_scripts/tests/test_claim_identity.py
git commit -m "oannes: add migrate_claim_links to repair dangling claim links"

Files: none (verification only)

Run: python3 pipeline.py link (re-inserts backrefs now that links resolve) Then: python3 enrich_links.py status Expected: claims orphan_pct well below the starting 19% (407 orphans); record the new number.

git add oannes/Oannes/_scripts/manifest.json
git commit -m "oannes: re-link after claim-link repair"

Phase 2 — Stable IDs + semantic dedup (optional, heavier)

Only undertake if Phase 1’s unresolved count (wording-drift residue) is large enough to matter. This adds a permanent stable identity and merges duplicates.

Task 5: Add a stable id to claim frontmatter

Files: - Modify: oannes/Oannes/_scripts/generate.py (generate_claim_file frontmatter block, generate.py:426-433) - Test: oannes/Oannes/_scripts/tests/test_claim_identity.py

# append to tests/test_claim_identity.py
from shared.io import claim_id

def test_claim_id_stable_and_slug():
    a = claim_id("A spherical UFO hovered over a silo")
    b = claim_id("a spherical ufo hovered over a silo")   # case/space-insensitive
    assert a == b
    assert re.fullmatch(r"clm-[0-9a-f]{10}", a)

(Add import re at the top of the test file if not already present.)

Run: python3 -m pytest tests/test_claim_identity.py -k claim_id -v Expected: FAIL with ImportError: cannot import name 'claim_id'

# shared/io.py
import hashlib, re as _re

def claim_id(summary: str) -> str:
    """Stable content hash id for a claim, robust to case/whitespace noise."""
    norm = _re.sub(r"\s+", " ", summary.strip().lower())
    norm = _re.sub(r"[^\w ]", "", norm)
    digest = hashlib.sha1(norm.encode("utf-8")).hexdigest()[:10]
    return f"clm-{digest}"

Run: python3 -m pytest tests/test_claim_identity.py -k claim_id -v Expected: PASS

In generate_claim_file, add to the frontmatter lines list (after type: claim):

        f'id: "{claim_id(summary)}"',

# run once from _scripts/
python3 - <<'PY'
import glob, re, os
from shared.io import claim_id
for p in glob.glob("../Claims/*.md"):
    t = open(p, encoding="utf-8").read()
    if re.search(r'^id:', t, re.M):
        continue
    summary = os.path.basename(p)[:-3].removeprefix("Claim - ")
    t = t.replace("type: claim", f'type: claim\nid: "{claim_id(summary)}"', 1)
    open(p, "w", encoding="utf-8").write(t)
print("backfilled ids")
PY

git add oannes/Oannes/_scripts/shared/io.py oannes/Oannes/_scripts/generate.py oannes/Oannes/_scripts/tests/test_claim_identity.py
git commit -m "oannes: add stable claim_id and backfill existing claims"

Task 6: Semantic dedup of wording-drift duplicates

Files: - Create: oannes/Oannes/_scripts/dedup_claims.py - Test: oannes/Oannes/_scripts/tests/test_dedup_claims.py

Approach: group claims by shared topics slug (cheap candidate generation), send each candidate cluster to the Claude CLI asking which claims are the same assertion, then for each merge-group keep one file and rewrite all links/sources of the others to it. Conservative: only merge when Claude marks them identical in meaning, and always log merges.

# tests/test_dedup_claims.py
from dedup_claims import apply_merges

def test_apply_merges_rewrites_links(tmp_path):
    eps = tmp_path / "Episodes"; eps.mkdir()
    (eps / "ep.md").write_text("see [[Claim - old wording]] here", encoding="utf-8")
    merges = {"Claim - old wording": "Claim - canonical wording"}
    changed = apply_merges(str(eps), merges, dry_run=False)
    assert changed == 1
    assert "[[Claim - canonical wording]]" in (eps / "ep.md").read_text(encoding="utf-8")

Run: python3 -m pytest tests/test_dedup_claims.py -v Expected: FAIL with ModuleNotFoundError: No module named 'dedup_claims'

# dedup_claims.py
"""Semantic dedup of claims that are the same assertion phrased differently.

Candidate clusters come from shared topic slugs; Claude judges sameness; this
module applies the resulting merge map by rewriting links across episodes.
The Claude-driven clustering lives in build_merge_map() (not unit-tested — it
calls the CLI); apply_merges() is pure and tested.
"""
import os, re, glob

def apply_merges(episodes_dir, merges, dry_run=True):
    """merges: {old_claim_name: keep_claim_name}. Returns episodes changed."""
    changed = 0
    for ep in glob.glob(os.path.join(episodes_dir, "*.md")):
        text = open(ep, encoding="utf-8").read()
        new = text
        for old, keep in merges.items():
            new = new.replace(f"[[{old}]]", f"[[{keep}]]")
        if new != text:
            changed += 1
            if not dry_run:
                open(ep, "w", encoding="utf-8").write(new)
    return changed

Run: python3 -m pytest tests/test_dedup_claims.py -v Expected: PASS

# dedup_claims.py — append
import json, subprocess

def build_merge_map(claims_dir, claude_cli="claude"):
    """Cluster claims by shared topic, ask Claude which are duplicates."""
    by_topic = {}
    for p in glob.glob(os.path.join(claims_dir, "*.md")):
        t = open(p, encoding="utf-8").read()
        name = os.path.basename(p)[:-3]
        for slug in re.findall(r"^\s*-\s*([\w-]+)\s*$", t, re.M):
            by_topic.setdefault(slug, []).append(name)
    merges = {}
    env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"}
    for slug, names in by_topic.items():
        if len(names) < 2:
            continue
        listing = "\n".join(f"{i+1}. {n}" for i, n in enumerate(names))
        prompt = (
            "These claims share a topic. Group ones that state the SAME assertion "
            "(differently worded). For each group of 2+, pick the clearest as KEEP. "
            'Respond ONLY as JSON: [{"keep":"<name>","duplicates":["<name>",...]}].\n\n'
            f"{listing}"
        )
        r = subprocess.run([claude_cli, "--print"], input=prompt,
                           capture_output=True, text=True, timeout=600, env=env)
        out = r.stdout.strip()
        s, e = out.find("["), out.rfind("]")
        if s == -1 or e == -1:
            continue
        try:
            for g in json.loads(out[s:e+1]):
                for dup in g.get("duplicates", []):
                    if dup != g["keep"]:
                        merges[dup] = g["keep"]
        except json.JSONDecodeError:
            continue
    return merges

python3 - <<'PY'
from dedup_claims import build_merge_map, apply_merges
m = build_merge_map("../Claims")
print("merge groups:", len(m))
print("episodes that would change:", apply_merges("../Episodes", m, dry_run=True))
PY

Review the merge map for false positives before applying. To apply: change dry_run=True to dry_run=False, and additionally delete/redirect the merged claim files (move duplicate files to an _archive/ folder rather than deleting).

git add oannes/Oannes/_scripts/dedup_claims.py oannes/Oannes/_scripts/tests/test_dedup_claims.py
git commit -m "oannes: add semantic claim dedup (topic-clustered, Claude-judged)"

Self-review notes