pam ▸ docs/generated/ralph_loop_2026-03-16_recent_source_journal_coverage.md
updated 2026-03-28

Ralph Loop 2026-03-16: Recent Source Journal Coverage

Change

Added a bounded PRD 001 / PRD 004 diagnostic for current raw-journal quality.

Files changed:

  1. research/prepare_replay.py
  2. research/continuous_loop.py
  3. research/status.py
  4. research/discord_status.py
  5. research/README.md
  6. tests/test_prepare_replay.py
  7. tests/test_continuous_loop.py
  8. tests/test_research_status.py

Why

Replay diagnostics already showed that paired eventful metadata coverage was almost absent, but they did not answer a critical operator question:

Is the replay file still bad because of old legacy rows, or is the current journal still writing metadata-thin payloads right now?

The new source-journal profile makes that explicit by measuring eventful metadata coverage directly on raw opportunity_scan_results.payload_json rows, with a recent-run slice alongside the all-history totals.

What changed

  1. research.prepare_replay.summarize_source_rows() now reports: - all-source metadata coverage - all-source positive eventful-score coverage - recent-run metadata coverage - recent-run positive eventful-score coverage
  2. research/continuous_loop.py now carries that source profile through replay prep events and into research/latest_state.json as replay_source_profile.
  3. research/status.py and research/discord_status.py now show the current raw-journal coverage directly from the DB, even when the stored loop snapshot is stale.
  4. Status notes now add recent_source_eventful_metadata_missing when recent raw journal rows are present but none carry eventful metadata.

Validation

Commands run:

PYTHONPATH=. pytest -q tests/test_prepare_replay.py tests/test_continuous_loop.py tests/test_research_status.py
python3 research/prepare_replay.py --db data/unicorn.db --out /tmp/pam_replay_rows.jsonl
python3 main.py research-status
python3 main.py research-discord-status

Observed:

  1. Focused tests passed: 14 passed
  2. Replay prep now emits a raw source profile alongside the replay-pair profile.
  3. Current status output shows:
current source journal coverage: all_metadata=68/1742 all_positive=60 recent_metadata=0/46 recent_positive=0 recent_runs=6

Conclusion

The current bottleneck is now explicit:

  1. some older rows do carry eventful metadata
  2. the recent raw journal does not
  3. replay quality is therefore still being degraded by an active payload-persistence gap, not just a legacy replay mix

The next bounded change should inspect why the latest scanner/journal path is still writing truncated payload_json rows without eventful_score, eventful_reasons, and related eventfulness fields.