Evaluation dossier · rendered from WHAT_THIS_IS.md
Rendered 2026-07-08 11:52

What This Is, Marvel MTG Dashboard Suite (evaluation dossier, rev 3)

One line: a set-review and speculation system for the unreleased Magic: The Gathering x Marvel sets (Marvel Super Heroes msh Standard and Marvel Super Heroes Commander msc), built from real Scryfall card data joined to LLM-generated editorial grades and dated market signals, with an independent aggregate audit and a transparent three-axis spec ranking on top.

This document is written to be evaluated. Every number traces to a named file or a reproducible command. Rev 2 folds in an external review. It clarifies how grades are produced, separates the kinds of market signal, adds a weight-sensitivity check, adds a reproducibility bundle, and scopes the audit precisely. Rev 3 (2026-06-20) re-pulls the Scryfall card pool and the market snapshot six days before release, and re-keys grades by card name after a reprint reclassification left the old index-based join misaligned.

Scope of validation (read first)

This project validates internal aggregation, source joins, and deployment parity. It does not validate card-evaluation accuracy, market outcome, or future price movement. The grades are an LLM set-review opinion, the prices are a dated snapshot, and the spec ranking is a transparent model over those inputs, not a forecast.


1. The four live artifacts

All are static pages on the permanent drwu-htmls Vercel project. public/ is /Users/drwu/Projects/drwu-htmls/public/.

Page URL Local source
Standard set review (msh) https://drwu-htmls.vercel.app/marvel-standard-feedback/ public/marvel-standard-feedback/index.html
Commander set review (msc) https://drwu-htmls.vercel.app/marvel-commander-feedback/ public/marvel-commander-feedback/index.html
Aggregate spot-check report https://drwu-htmls.vercel.app/marvel-dashboard-spotcheck/ public/marvel-dashboard-spotcheck/index.html
Spec target sheet https://drwu-htmls.vercel.app/marvel-spec-targets/ public/marvel-spec-targets/index.html
This dossier (rendered) https://drwu-htmls.vercel.app/marvel-what-this-is/ public/marvel-what-this-is/index.html

A fifth page, the Sealed Deck Builder (https://drwu-htmls.vercel.app/marvel-limited-deckbuilder/), sits on top of the same data for the Limited format: paste a sealed pool, get recommended colors and a deck cut to 40 with a rationale for every add and cut. It draws Limited grades from Draftsim and CoolStuffInc, not from the msc/msh Constructed grades used here.


2. Data provenance

All build inputs live in /Users/drwu/Projects/_outputs/mtg-marvel-feedback/.

File Size What it is Records
data/msh_standard.json 241 KB Scryfall card objects, Standard set 276 cards
data/msc_commander_new.json 333 KB Scryfall card objects, Commander set (brand-new, non-reprint) 385 cards
data/signals.json 82 KB Per-card market signals (EDHREC rank, market USD, foil USD) 893 names, 652 with USD, 888 with EDHREC rank
evals_merged.json 270 KB LLM-generated grades keyed by card name 276 std + 385 cmd

The card facts (name, mana cost, type, oracle text, power/toughness, rarity, image, Scryfall link) are real Scryfall fields, pulled by fetch_cards.py. signals.json carries the market layer, pulled 2026-06-20 (file mtime). The card pool was re-pulled from Scryfall on 2026-06-20; Toxic Deluge dropped out of the Commander pool after Scryfall reclassified it as a reprint, taking the Commander count from 386 to 385.

Two kinds of market signal, kept distinct

These are not interchangeable. The spec model (Section 7) scores on market-price signals and editorial grades only. EDHREC rank is shown on the set-review pages but is deliberately excluded from the spec score, because pre-release ranks are unreliable.


3. How the grades are produced

The grades are LLM-generated, not human-written and not manually reviewed after generation.

So a grade is a schema-validated, rubric-guided LLM set-review opinion over each card's printed rules text. It is reproducible in method, not in exact wording, since LLM output varies run to run. No human editorial pass was applied on top. As of rev 3, evals_merged.json is keyed by card name rather than position, so a card entering or leaving the Scryfall pool no longer shifts every later grade.


4. Pipeline scripts

Script Role
fetch_cards.py Pulls and compacts Scryfall card objects. Handles double-faced cards via oracle_of() / img_of().
fetch_signals.py Refreshes the price/EDHREC snapshot, merging over the prior signals.json.
generate_workflow.py / eval_workflow.js Produce the LLM grades that become evals_merged.json.
build_dashboards.py Joins cards + evals + signals into the two set-review pages by card name. Aggregates at build time.
verify_aggregates.py Independent audit. Recomputes every headline stat from source and diffs it against the deployed HTML.
spec_rank.py Three-axis spec model, applies the rarity floor, writes spec_targets.json.
build_spec_dashboard.py Renders the sortable spec sheet from spec_targets.json.
build_dossier_html.py Renders this dossier to HTML.
build_limited_deckbuilder.py Renders the Limited sealed deck builder from data/limited_cards.json.

5. Set-review headline numbers

Computed by build_dashboards.py, independently re-derived by verify_aggregates.py.

Standard (msh), 276 graded: A 12, B 98, C/D/F 166, 12 cards at a $5+ price.

Commander (msc), 385 graded: A 25, B 186, C/D/F 174, 165 build-around commanders, 87 cards at a $5+ price.

Total: 661 graded cards.

The $5+ counts moved sharply against the 2026-06-14 snapshot: Standard fell from 59 to 12 as asking prices softened, while Commander rose from 11 to 87 as far more of the pool picked up a live price (243 priced names to 652).

Data caveat, two empty rules-text cards

Two of the 385 Commander cards carry an empty oracle_text from the Scryfall pull, Black Widow, Natasha Romanoff (collector #846) and Happy Hogan, Dauntless Driver (collector #545). Both are unique printings in the high-collector-number variant range, and both still carry grade, score, verdict, image, power/toughness and Scryfall link. From a reader's view, the single effect is that the rules-text drawer renders empty on those two cards. A re-run of fetch_cards.py closer to release repopulates them.


6. Aggregate audit result

verify_aggregates.py enforces one rule: for every headline stat, three independent reads must agree.

Literal verifier output after the 2026-06-20 refresh:

VERDICT: ALL CHECKS PASS — dashboards are real aggregates of their sources

All 16 headline metrics across both formats matched, plus card-count parity and the distribution bar on each page. The name-keyed join leaves zero ungraded cards in either set (276/276 and 385/385). Live Vercel parity is confirmed by SHA-256, the live bytes hash-match the local files exactly (Standard 290,315 bytes, Commander 395,980 bytes, both HTTP 200).

What this audit does and does not prove

The audit proves internal consistency (the page renders a faithful aggregate of its own source files) and deployment parity (the live page equals the local source). It does not prove the sources are externally correct. It cannot confirm a grade is good or a price will hold.

External corroboration of the one externally checkable claim: the release date. Wizards of the Coast confirms Marvel Super Heroes releases 2026-06-26, and listed the full card-image gallery as scheduled for 2026-06-12. - https://magic.wizards.com/en/news/announcements/where-to-find-marvel-super-heroes-previews - https://magic.wizards.com/en/products/marvel/marvel-super-heroes/card-image-gallery


7. The spec model

Each priced card is weighed on three axes, scored 0 to 100.

SPEC = 0.30 * COST + 0.45 * UPSIDE + 0.25 * BUZZ

Supply filter: commons and uncommons under $5 are excluded as capped-supply floors, rares and mythics stay at any price, and COST is re-percentiled over the cards that remain.

Current pool (2026-06-20): 661 graded, 652 priced, 252 after the filter (201 rare, 46 mythic, 5 uncommon survivors). Median price $3.83, range $0.15 to $79.12.

Are the weights arbitrary? A sensitivity check

The headline thesis is the cheap-staple-and-commander core, and it holds. Re-ranking the top 10 under ten-point weight shifts:

Weight mix Base top-10 retained New entrants
Base 0.30 / 0.45 / 0.25 reference reference
Cost-tilt 0.40 / 0.40 / 0.20 7 of 10 Taskmaster, Mercenary Mimic, Captain Marvel, Earth's Protector, The Astonishing Ant-Man
Upside-tilt 0.25 / 0.55 / 0.20 7 of 10 The Thing, Hawkeye, Trick Shot, Typhoid Mary, Fractured
Buzz-tilt 0.25 / 0.40 / 0.35 8 of 10 Namor, Scourge of the Seas, Bruce Banner // The Incredible Hulk

Reading: at least 7 of the base top 10 survive every ten-point reweighting, so the headline list is not an artifact of the exact weights. The cards that swap in under each tilt are the expected ones (cheaper staples under cost-tilt, higher-graded cards under upside-tilt, foil-premium chase cards under buzz-tilt), which is the model behaving, not breaking.

Current top five by spec score

  1. Kang the Conqueror, STD, $2.66, spec 68.4
  2. Doctor Strange, Surgeon, CMD, $1.90, spec 68.2
  3. The Mighty Thor, Jane Foster, STD, $0.81, spec 67.6
  4. T'Challa, the Black Panther, CMD, $4.91, spec 66.1
  5. Winter Soldier, Icy Assassin, STD, $0.32, spec 64.8

8. Honesty notes and limitations


9. Reproduce and verify

From /Users/drwu/Projects/_outputs/mtg-marvel-feedback/:

python3 fetch_cards.py              # re-pull the Scryfall card pool
python3 fetch_signals.py            # refresh the price/EDHREC snapshot
python3 verify_aggregates.py        # re-audits both set-review pages (3-read agreement)
python3 spec_rank.py                # re-derives spec_targets.json from source
python3 build_spec_dashboard.py     # re-renders the spec sheet

Live parity for any page:

curl -s https://drwu-htmls.vercel.app/marvel-spec-targets/ | shasum -a 256
shasum -a 256 ~/Projects/drwu-htmls/public/marvel-spec-targets/index.html
# the two hashes match when the live deploy is current

Repro bundle


10. Claim-to-evidence ledger

Claim Where it is proven
661 cards graded evals_merged.json, re-counted by verify_aggregates.py
Grades are LLM-generated, schema-constrained, oracle-grounded generate_workflow.py:67-110 (schema, prompts, rubric)
Grades join cards by name, robust to set-membership changes build_dashboards.py, spec_rank.py, verify_aggregates.py (all join on c.name)
Headline stats are real aggregates verify_aggregates.py prints ALL CHECKS PASS, 16/16
Live pages match local source SHA-256 of live URL equals SHA-256 of local file
Card facts are genuine Scryfall realness() in verify_aggregates.py, image + link parity
Spec pool is 252 after the filter spec_rank.py over spec_targets.json
Top-10 robust to weight shifts sensitivity table, Section 7, recomputable from spec_targets.json
Release date 2026-06-26 Wizards official pages (Section 6)

Appendix A, local scanner note

A repo-wide keyword scanner (session_fraud_check.py) returns a global flag during this work because it scans whole directory trees and the session transcript, not this project alone. The Marvel pages built here contribute no flags of their own.


Marvel Super Heroes (msh) and Marvel Super Heroes Commander (msc) release 2026-06-26. Card data from Scryfall, grades LLM-generated and editorial in style, market signals from Scryfall and EDHREC. Built and audited in this session; refreshed 2026-06-20.

🛍️ Get the printable