The Spielraum — ingest and scoring

How the engine gets from “all the news” to one worked case a day. Written to be read on its own; the wider project design is in ../BRIEF.md.

What needs deciding is in §6. Everything before it is the reasoning behind those questions.

1. The shape of the problem

The engine reads a lot and publishes one thing. Roughly:

The whole design exists to protect the last two rows. You cannot put a model on five thousand articles a day and stay solvent, so everything above the scoring line has to be crude, fast and willing to throw away things it shouldn’t. That is fine — the engine only needs one good collision a day, and a good collision surfaces in fifty outlets, not one.

2. Ingest — breadth without picking a worldview

Sources stay pluggable. Adding a feed is a config line, never code, because the source list is an editorial decision that will change and should not require a deploy.

Two layers, doing different jobs:

Breadth: GDELT. Free, global, machine-readable, indexes news in a hundred-plus languages, and updates every fifteen minutes. It is the only realistic answer to “world news, lots of it, no budget.” Crucially it is source-agnostic — it does not curate, so using it does not smuggle in a worldview through the outlet list.

Depth: RSS from named outlets. A curated set for the places you actually want read closely — German nationals, a few European, the trade press for care and mental health. This layer is opinionated by construction, so it stays small and explicit rather than pretending to be neutral.

The tension worth naming: choosing outlets chooses what the engine can see. GDELT’s breadth is the mitigation, not a solution. The honest posture is to keep the RSS list short, write down why each outlet is on it, and treat the list as a thing that gets argued about rather than settled.

3. Clustering — the same story is not many stories

A significant story arrives as forty near-identical articles. Cluster before anything expensive touches it: title and lede similarity plus a time window is enough, and GDELT already supplies partial clustering.

Two things fall out of clustering for free, and both matter more than they look. Cluster size is a salience signal — a story carried by sixty outlets is one people are actually arguing about. And cluster spread across outlet type is a contention signal — a story that appears in outlets that rarely agree is, empirically, one containing a real collision.

That second one is the cheapest useful proxy the engine has, and it costs nothing to compute.

4. Filtering — cheap rejection before expensive judgment

Two passes, both crude:

Domain. Does the story touch anything POSITION.md can speak to? Keyword and embedding match against the thirteen dimensions and the launch domain. Everything else is dropped — sport, celebrity, weather, crime blotter.

Shape. Does it look like a decision rather than an event? A decision has an actor, a choice, and someone who carries the cost: a court ruling, a budget line, a regulation, a policy reversal, a report with a recommendation. A plane crash is an event; the safety rule written afterwards is a decision. The engine has nothing to say about events.

This pass will wrongly reject things. That is acceptable and worth stating plainly: the engine needs one good case a day, not complete coverage, and a false negative costs nothing while a false positive costs a model call.

5. Scoring — the four questions

Only the survivors get a model. Each is asked, in this order, because each is cheaper than the next:

1. Are two goods genuinely in conflict? Not “is this contested” — contested means two sides want different things. A collision means two things we value pull against each other. No collision, no case. This rejects most of what survives filtering.
2. Is the ruling non-obvious? If POSITION.md resolves it in one step with no tension, there is nothing to show. Skip.
3. Does it cost us something? Weighted upward. A ruling that goes against one of our own principles is the most valuable output the engine can produce and the thing no other outlet runs.
4. Can it be explained in under a minute to someone who has not followed the story?

Then the open-question gate: if the collision lands on anything in POSITION.md §5, the engine does not rule. It escalates the story back as an elicitation case. Your answer becomes a rule and the engine can rule on that class of story thereafter. This is how the case bank grows out of real events rather than authored hypotheticals — which wave one already showed produces better cases.

Everything scored gets banked, including the rejects, with its scores. Storage is free and the reject pile is the only way to ever answer “is the scorer any good?” Without it there is no way to tell a well-tuned engine from a broken one.

6. What needs deciding

6.1 — Domain scope. The launch domain is mental health and care, but you have asked for world news. Is a story about, say, Japanese elder care in scope on day one? My read: yes, and breadth is the point — the doctrine is not German, only the launch audience is. But that widens ingest substantially and should be a choice rather than a drift.

6.2 — Language. World news means non-English, non-German sources. Translate everything at ingest (costly, uniform) or filter in the original and translate only survivors (cheap, and risks the filter being worse in languages it handles badly)? I lean to the second.

6.3 — Daily budget for scoring. This is the only real running cost and it sets how deep the funnel can go. A rough figure per day is enough to size stages 4 and 5.

6.4 — Does the engine produce one case a day, or several? Dark-phase output is not publication, so there is no editorial reason to cap it at one. Three a day is three hundred extra cases in the twelve months and a much better test of whether POSITION.md holds. It also triples the scoring bill.

6.5 — What arrives in front of you, and how often? The engine escalates open questions. Is that a Discord message per case, a weekly digest, or a page on the hub you visit when you feel like it? This decides whether the dark year is a background process or a standing obligation.

6.6 — Where does the bank live? Flat files in the repo (simple, greppable, versioned, gets unwieldy past a few thousand) or Postgres (Oannes already runs one, queryable, more moving parts). Files are probably right for a year of cases.

7. What this reuses, and what is new

Reusable from Oannes: the launchd scheduling pattern, config and secret handling, the failure alerting wrapper, the Discord notification path, the state-file discipline.

New: everything above the ruling. Oannes reads a fixed vault of claims that were curated in advance. This reads a live firehose and has to decide what is worth thinking about, which is a harder problem and the actual substance of the build.