scripts::analyze_common_pile¶

Functions¶

	Name
str	clean_text(str text)
List[str]	extract_keywords(str text, int top_n =10)
Tuple[List[str], List[Dict]]	load_and_sample_common_pile(int sample_size =SAMPLE_SIZE)
Tuple[np.ndarray, TfidfVectorizer, MiniBatchKMeans]	cluster_documents(List texts[str], int n_clusters =N_CLUSTERS)
List[Dict]	analyze_clusters(List texts[str], np.ndarray labels, List metadata[Dict], TfidfVectorizer vectorizer)
List[Dict]	suggest_niche_names(List niches[Dict])
	save_analysis(List niches[Dict], List texts[str], np.ndarray labels)
	print_recommendations(List niches[Dict])
	main()

Attributes¶

	Name
int	SAMPLE_SIZE
int	N_CLUSTERS
int	MIN_NICHE_SIZE
int	MAX_FEATURES
int	RANDOM_SEED
	PROJECT_ROOT
	OUTPUT_DIR
	exist_ok

Detailed Description¶

Common Pile Niche Discovery Script
Analyzes Common Pile dataset to identify viable niches for GNUS.ai specialists

This script:
1. Streams Common Pile to avoid memory issues
2. Extracts topics using TF-IDF + clustering
3. Identifies niches with sufficient data (>10k samples recommended)
4. Outputs niche recommendations with sample texts

Functions Documentation¶

function clean_text¶

str clean_text(
    str text
)

Clean and normalize text for analysis```


### function extract_keywords

```python
List[str] extract_keywords(
    str text,
    int top_n =10
)

Extract potential domain keywords from text```


### function load_and_sample_common_pile

```python
Tuple[List[str], List[Dict]] load_and_sample_common_pile(
    int sample_size =SAMPLE_SIZE
)

Load Common Pile and extract representative sample
Returns: (texts, metadata)

function cluster_documents¶

Tuple[np.ndarray, TfidfVectorizer, MiniBatchKMeans] cluster_documents(
    List texts[str],
    int n_clusters =N_CLUSTERS
)

Cluster documents using TF-IDF + MiniBatchKMeans
Returns: (cluster_labels, vectorizer, clustering_model)

function analyze_clusters¶

List[Dict] analyze_clusters(
    List texts[str],
    np.ndarray labels,
    List metadata[Dict],
    TfidfVectorizer vectorizer
)

Analyze each cluster to identify niche characteristics
Returns: List of niche descriptions

function suggest_niche_names¶

List[Dict] suggest_niche_names(
    List niches[Dict]
)

Suggest human-readable names for niches based on top terms

function save_analysis¶

save_analysis(
    List niches[Dict],
    List texts[str],
    np.ndarray labels
)

Save analysis results for later use```


### function print_recommendations

```python
print_recommendations(
    List niches[Dict]
)

Print top niche recommendations```


### function main

```python
main()

Main execution```



## Attributes Documentation

### variable SAMPLE_SIZE

```python
int SAMPLE_SIZE =  50000;

variable N_CLUSTERS¶

int N_CLUSTERS =  20;

variable MIN_NICHE_SIZE¶

int MIN_NICHE_SIZE =  5000;

variable MAX_FEATURES¶

int MAX_FEATURES =  5000;

variable RANDOM_SEED¶

int RANDOM_SEED =  42;

variable PROJECT_ROOT¶

PROJECT_ROOT =  Path(__file__).resolve().parent.parent.parent;

variable OUTPUT_DIR¶

OUTPUT_DIR =  str(PROJECT_ROOT / "data" / "analysis");

variable exist_ok¶

exist_ok;

Updated on 2026-07-25 at 22:56:57 +0000