scripts::prepare_datasets¶

Functions¶

	Name
	load_niche_config()
	extract_niche_samples(niche_name niche_name, niche_config niche_config, target_niches_config target_niches_config)
	create_splits(samples samples, niche_name niche_name)
	format_for_training(samples samples, niche_name niche_name)
	save_datasets(niche_name niche_name, splits splits)
	main()

Attributes¶

	Name
	PROJECT_ROOT
dict	NCHE_TOKENIZER_MODELS
list	SELECTED_NICHES
float	VAL_SPLIT
float	TEST_SPLIT
int	RANDOM_SEED
int	MAX_SAMPLES_PER_NICHE
	OUTPUT_DIR
	exist_ok

Detailed Description¶

Prepare training datasets for GNUS.ai specialists
Creates clean train/val splits from source-based niches

Functions Documentation¶

function load_niche_config¶

load_niche_config()

Load the source-based niche analysis```


### function extract_niche_samples

```python
extract_niche_samples(
    niche_name niche_name,
    niche_config niche_config,
    target_niches_config target_niches_config
)

Extract all samples for a specific niche from Common Pile

function create_splits¶

create_splits(
    samples samples,
    niche_name niche_name
)

Create train/val/test splits

function format_for_training¶

format_for_training(
    samples samples,
    niche_name niche_name
)

Format samples for Qwen3 instruction tuning using tokenizer.apply_chat_template().

Per FOUND-01: Uses the actual tokenizer's native chat template (Qwen3 format)
instead of hand-rolled <|im_start|> strings that cause Qwen2.5/Qwen3 mismatch.

function save_datasets¶

save_datasets(
    niche_name niche_name,
    splits splits
)

Save as Hugging Face datasets for easy loading

function main¶

main()

Attributes Documentation¶

variable PROJECT_ROOT¶

PROJECT_ROOT =  Path(__file__).resolve().parent.parent.parent;

variable NCHE_TOKENIZER_MODELS¶

dict NCHE_TOKENIZER_MODELS =  {
    "medical":      "mlx-community/Qwen3-30B-A3B-Instruct-2507-bf16",
    "qa_technical": "mlx-community/Qwen3-30B-A3B-Instruct-2507-bf16",
    "code":         "mlx-community/Qwen3-Coder-30B-A3B-Instruct-bf16",
    "encyclopedic": "mlx-community/Qwen3-30B-A3B-Instruct-2507-bf16",
    "patents":      "mlx-community/Qwen3-30B-A3B-Instruct-2507-bf16",
};

variable SELECTED_NICHES¶

list SELECTED_NICHES =  ['medical', 'qa_technical', 'code', 'encyclopedic', 'patents'];

variable VAL_SPLIT¶

float VAL_SPLIT =  0.1;

variable TEST_SPLIT¶

float TEST_SPLIT =  0.05;

variable RANDOM_SEED¶

int RANDOM_SEED =  42;

variable MAX_SAMPLES_PER_NICHE¶

int MAX_SAMPLES_PER_NICHE =  10000;

variable OUTPUT_DIR¶

OUTPUT_DIR =  str(PROJECT_ROOT / 'data' / 'specialists');

variable exist_ok¶

exist_ok;

Updated on 2026-07-25 at 22:56:57 +0000