training::dedup¶

Functions¶

	Name
dict	compute_overlap_matrix(dict samples_by_niche, int num_perm =128)
list	deduplicate_within_niche(list samples, float jaccard_threshold =0.8, int num_perm =128)

Attributes¶

	Name
	parser
	required
	True
	help
	args
	project_root
str	input_path
list	samples
	line
list	deduped
	removed
str	out_dir
	parents
	exist_ok

Detailed Description¶

Cross-niche deduplication using MinHash LSH.```


## Functions Documentation

### function compute_overlap_matrix

```python
dict compute_overlap_matrix(
    dict samples_by_niche,
    int num_perm =128
)

function deduplicate_within_niche¶

list deduplicate_within_niche(
    list samples,
    float jaccard_threshold =0.8,
    int num_perm =128
)

Attributes Documentation¶

variable parser¶

parser =  argparse.ArgumentParser(description="Deduplicate synthetic data for a specialist niche");

variable required¶

required;

variable True¶

True;

variable help¶

help;

variable args¶

args =  parser.parse_args();

variable project_root¶

project_root =  Path(__file__).resolve().parent.parent;

variable input_path¶

str input_path =  project_root / "artifacts" / "synthetic" / f"{args.niche}.jsonl";

variable samples¶

list samples =  [];

variable line¶

line =  line.strip();

variable deduped¶

list deduped =  deduplicate_within_niche(samples);

variable removed¶

removed =  len(samples) - len(deduped);

variable out_dir¶

str out_dir =  project_root / "artifacts" / "dedup";

variable parents¶

parents;

variable exist_ok¶

exist_ok;

Updated on 2026-07-25 at 22:56:57 +0000