Methodology → Accuracy Metrics
Eval Suite: 25 Golden-Set Cases
This page documents 25 curated internal fixtures covering different property types, inputs, geographies, and condition levels. They are regression checks for expected product behavior, not completed-project accuracy measurements.
Published Fixture Inventory
Curated internal fixtures
25Synthetic and hand-authored cases used to catch scope, flag, confidence, and range regressions
Represented states
4Texas, Arizona, Colorado, Illinois
Named markets / localities
7Houston, Plano (Dallas-Fort Worth), Austin, Conroe, Scottsdale, Aurora (Denver metro), Chicago
Range sanity rule
≤ 2.5×low < mid < high, with high no more than 2.5× low
Geographic Coverage
| State | Markets covered | Cases |
|---|---|---|
| Texas | Houston, Austin, Plano (Dallas-Fort Worth), Conroe | 22 |
| Arizona | Scottsdale | 1 |
| Colorado | Aurora (Denver metro) | 1 |
| Illinois | Chicago | 1 |
Evidence is deepest for Houston (RSMeans CCI 0.97). National coverage uses RSMeans city cost indexes. Local contractor pricing may vary ±20–30% — always verify with a local contractor quote.
Full Golden-Set Case List
| # | Case | Confidence |
|---|---|---|
| 01 | Old Houston house — foundation + pre-war systems | Medium |
| 02 | HVAC end-of-life — single dominant system | High |
| 03 | Roof leak + water staining | Medium-High |
| 04 | Outdated electrical panel — code upgrade | High |
| 05 | Plumbing and sewer issue | Medium |
| 06 | Cosmetic-only rehab — no major systems | Very High |
| 07 | Vague description — limited input quality | Low |
| 08 | Lender-grade request — full detail | Medium-High |
| 09 | No ARV supplied — internal analysis | Medium-High |
| 10 | Uploaded inspection PDF findings | Medium |
| 11 | DFW suburban ranch — HVAC + roof | Medium-High |
| 12 | Austin TX urban infill — full gut | Low-Medium |
| 13 | Severe mold + moisture intrusion | Low |
| 14 | Full gut rehab — all systems + finishes | Low |
| 15 | Condo — HOA owns exterior + roof | High |
| 16 | Phoenix AZ — HVAC-critical desert climate | Medium-High |
| 17 | Duplex — shared systems, two units | Medium |
| 18 | Partial fire damage — kitchen to attic | Low |
| 19 | Rural property — well and septic | Medium |
| 20 | Denver CO — freeze-thaw foundation + hail | Medium |
| 21 | Turnkey ready — fresh flip, minimal scope | Very High |
| 22 | Chicago IL pre-war two-flat — knob-and-tube | Low-Medium |
| 23 | Renegotiation — option period active | Medium |
| 24 | Lender-grade detail — draw schedule format | Medium-High |
| 25 | Minimal vague input — address + year only | Very Low |
What This Validates — and What It Doesn't
What these evals validate
- Scope recall: known finding types produce the expected scope categories
- Flag accuracy: risk flags appear when triggered, don't appear when not triggered
- Confidence calibration: stated confidence matches actual accuracy band
- Cost range sanity: ranges are internally consistent and proportionate to scope
- Geographic routing: different markets use the correct cost index
- Edge case handling: vague inputs, no ARV, condos, rural, fire damage all produce reasonable output
What these evals don't cover
- Real contractor bid comparison — no live deal data is included in the eval suite
- Post-renovation actual cost variance — actuals data is collected through the in-product actuals tracking feature
- Properties outside the training data geography — accuracy may degrade in markets with no RSMeans data
- Hidden defects not mentioned in the input — Scopebase can only scope what it can see in the description or PDF
- Hyperlocal micro-market pricing — CCI indexes are metro-level, not neighborhood-level
Help improve calibration
Scopebase is a pre-inspection screening tool, not a final contractor bid. If you complete a renovation, you can submit final invoices through Actuals tracking. Those comparisons help identify where future calibration work is needed.
Submit actuals from your estimate detail page, or contact support@scopebase.org with your estimate ID and final contractor invoices.