eval::benchmark_config::ConfigError¶

Inherits from Exception

Public Attributes¶

	Name
tuple	BENCHMARK_REQUIRED_FIELDS

Protected Attributes¶

	Name
tuple	_THRESHOLD_FIELDS

Detailed Description¶

class eval::benchmark_config::ConfigError;

Raised when a benchmark config or specialist mapping fails validation.

The error message names the file and the missing/invalid field so the
operator can pinpoint the bad YAML without a stack dive.

Public Attributes Documentation¶

variable BENCHMARK_REQUIRED_FIELDS¶

static tuple BENCHMARK_REQUIRED_FIELDS =  (
    ("name", str),
    ("task_name", str),
    ("num_fewshot", int),
    ("output_type", str),
    ("blocking", bool),
    ("hard_floor", float),
    ("regression_max_pct", float),
    ("deviation_max_pct", float),
);

Protected Attributes Documentation¶

variable _THRESHOLD_FIELDS¶

static tuple _THRESHOLD_FIELDS =  (
    "hard_floor",
    "regression_max_pct",
    "deviation_max_pct",
);

Updated on 2026-07-25 at 22:56:57 +0000