Community Mirror Index — v3 (with TVTropes layer)
v3 inherits everything in v2 and adds a TVTropes harvest layer. TVTropes runs behind Cloudflare with a managed challenge, so the puller pulls slowly, in random order, with a clean egress IP (Cloudflare WARP active) and a full Chrome header set. Plain curl gets 403; the Python session in this puller gets 200.
What v3 adds
A self-discovering puller that starts at Series/Community, harvests in-show namespaces (Recap / Characters / Heartwarming / Awesome / Funny / Tearjerker / NightmareFuel / YMMV / WMG / Trivia / Fridge / Headscratchers / Quotes / Analysis / FanWorks), randomizes the queue, and sleeps 25-75 seconds between fetches with an occasional 5-15 minute long pause to avoid signaling.
- Puller:
tvtropes_pull.py - Output root:
Projects/tvtropes-pull/ - Raw HTML:
Projects/tvtropes-pull/raw/ - Extracted text:
Projects/tvtropes-pull/text/ - Resume state:
state.json - Per-fetch log:
pull.log.jsonl - Stdout:
pull.stdout.log - Running PID:
puller.pid
Operating model
One process serves both shows. The queue interleaves Community and Gilmore URLs by random shuffle, so the pull pattern looks like a person browsing across two interests rather than scraping a series in order.
- Cloudflare bypass = WARP egress + realistic Chrome headers. No JS solver needed at this pace.
- On 403 / 429 / 503 the puller backs off 60s, 180s, 600s before giving up on a URL.
- Files are atomic-saved; resume after a kill picks the queue back up.
Stop / resume / inspect
- Inspect progress:
tail -f ~/Projects/tvtropes-pull/pull.stdout.log - Stop:
kill $(cat ~/Projects/tvtropes-pull/puller.pid)(state is preserved) - Resume: re-run the launch command (
python3 tvtropes_pull.py --out . --shows Community=... GilmoreGirls=... --min-delay 25 --max-delay 75) - Re-pull a URL: delete the raw/
.html and remove the URL from state.json['done']
Integration with v2
Once the harvest finishes (or hits a stopping point), the TVTropes text becomes a third source layer in the mirror-extraction pipeline. The DeepSeek extractor that already runs against ChatGPT conversation pulls can be re-pointed at Projects/tvtropes-pull/text/Recap__Community*.txt for episode-level tropes per recap, and at text/Characters__Community*.txt for character-arc cross-references against the v6 mirror corpus.
- Live mirror corpus (target for merge):
community-mirrors-v6.json
Inherited layers from v2
- v1 seed:
Community v1 - v2 integrated index:
Community v2 - v2 covers: FULL conversation exports, S4E10 deep breakdown, live mirror bundle, Community-S4-Genius HTML pack, gap-analysis file.
Operational notes
- If raw files start showing "Just a moment" in their first 600 bytes, WARP rotated to a flagged egress IP or TVTropes tightened the challenge. Fall back to playwright-stealth (not installed; would need
pip install playwright && playwright install chromium). - Pacing: at 25-75s per page with occasional long pauses, ~50 pages per hour. A full series (Series + Recap + Characters + 8 trope subpages, roughly 80-120 pages) takes 2-3 hours.
- Disk: each raw HTML averages ~250-500 KB; extracted text ~50-100 KB. Total per-show under 100 MB.
Provenance
- Puller built: 2026-06-02
- Launched as PID in puller.pid
- Egress: Cloudflare WARP, confirmed connected
- First successful fetch: GilmoreGirls 480 KB at 19:38, 21 sub-pages discovered