eval::benchmark_config¶

Classes¶

	Name
class	eval::benchmark_config::ConfigError

Functions¶

	Name
Dict[str, Dict[str, Any]]	validate_benchmarks_config(Path
Dict[str, Dict[str, List[str]]]	load_specialist_mapping(Path
Tuple[List[str], List[str]]	get_benchmarks_for_specialist(str specialist, Dict]] mapping[str, Dict[str, List[str])
None	check(str name, bool condition, str detail ="")

Attributes¶

	Name
Path	BENCHMARKS_CONFIG_DIR
str	SPECIALIST_MAPPING_FILENAME
int	passed
int	failed
Dict[str, Dict[str, Any]]	validated
dict	expected_benchmarks
Dict[str, Dict[str, List[str]]]	mapping
dict	expected_specialists
	block
	diag

Detailed Description¶

ConfigLoader extension for per-benchmark YAML configs and specialist mapping.

Per Phase 04-02 Task 2: validates the schema of every per-benchmark config YAML
in ``config/benchmarks/`` (required fields: name, task_name, num_fewshot,
output_type, blocking, hard_floor, regression_max_pct, deviation_max_pct per
D-04/D-08) and loads the specialist-to-benchmark mapping per D-05
(``specialist_mapping.yaml``).

Threat mitigations:
- T-04-06: ``yaml.safe_load`` exclusively — never ``yaml.load`` or full_load.
- T-04-08: ``load_specialist_mapping`` cross-validates referenced benchmark
  names against the validated per-benchmark config set.

Functions Documentation¶

function validate_benchmarks_config¶

Dict[str, Dict[str, Any]] validate_benchmarks_config(
    Path|None config_dir =None
)

Read and validate every per-benchmark YAML in ``config_dir``.

Each YAML must define all fields in ``BENCHMARK_REQUIRED_FIELDS`` with the
correct type. Threshold fields (hard_floor, regression_max_pct,
deviation_max_pct) must be strictly positive floats. ``num_fewshot`` must
be a non-negative int. ``blocking`` must be a Python ``bool`` (string
"true"/"false" from a misconfigured YAML is rejected).

Args:
    config_dir: Directory containing per-benchmark YAML files. Defaults to
        ``<project_root>/config/benchmarks/``.

Returns:
    Dict mapping ``name`` field -> validated config dict.

Raises:
    ConfigError: If any YAML is missing a required field, has an invalid
        type, or fails a value-range check. The error message names the
        file and the offending field.
    FileNotFoundError: If ``config_dir`` does not exist.

function load_specialist_mapping¶

Dict[str, Dict[str, List[str]]] load_specialist_mapping(
    Path|None config_dir =None
)

Load and validate ``specialist_mapping.yaml`` per D-05.

Validates that:
  - the file exists and parses as a YAML mapping,
  - the top-level key is ``specialists``,
  - each specialist entry has both ``blocking_benchmarks`` and
    ``diagnostic_benchmarks`` lists,
  - every referenced benchmark exists in the per-benchmark config set
    (T-04-08 mitigation).

Args:
    config_dir: Directory containing ``specialist_mapping.yaml``. Defaults
        to ``<project_root>/config/benchmarks/``.

Returns:
    Dict mapping specialist name -> {
        "blocking_benchmarks": [...],
        "diagnostic_benchmarks": [...],
    }.

Raises:
    ConfigError: On any schema violation, including a referenced benchmark
        that does not have a per-benchmark config YAML.
    FileNotFoundError: If the file or directory does not exist.

function get_benchmarks_for_specialist¶

Tuple[List[str], List[str]] get_benchmarks_for_specialist(
    str specialist,
    Dict]] mapping[str, Dict[str, List[str]
)

Return ``(blocking_benchmarks, diagnostic_benchmarks)`` for a specialist.

Args:
    specialist: Specialist name (e.g. "medical", "code").
    mapping: Loaded specialist mapping (output of ``load_specialist_mapping``).

Returns:
    Tuple of (blocking_benchmarks, diagnostic_benchmarks) lists.

Raises:
    KeyError: If *specialist* is not in *mapping*.

function check¶

None check(
    str name,
    bool condition,
    str detail =""
)

Attributes Documentation¶

variable BENCHMARKS_CONFIG_DIR¶

Path BENCHMARKS_CONFIG_DIR =  _PROJECT_ROOT / "config" / "benchmarks";

variable SPECIALIST_MAPPING_FILENAME¶

str SPECIALIST_MAPPING_FILENAME =  "specialist_mapping.yaml";

variable passed¶

int passed =  0;

variable failed¶

int failed =  0;

variable validated¶

Dict[str, Dict[str, Any]] validated =  validate_benchmarks_config();

variable expected_benchmarks¶

dict expected_benchmarks =  {"mmlu", "humaneval", "medmcqa", "gpqa", "pubmedqa", "bigpatent"};

variable mapping¶

Dict[str, Dict[str, List[str]]] mapping =  load_specialist_mapping();

variable expected_specialists¶

dict expected_specialists =  {"code", "medical", "qa_technical", "encyclopedic", "patents"};

variable block¶

block;

variable diag¶

diag;

Updated on 2026-07-25 at 22:56:57 +0000