Model Performance Optimization & Error Analysis (False Positives/Negatives)

[MorganRff]

Description

The current Machine Learning filter (Random Forest/SVM) is used to classify leads as "Nautical Professional" or "Irrelevant." While the global accuracy is satisfying, a deeper analysis of the confusion matrix reveals specific error patterns that impact the pipeline's efficiency. In a B2B prospecting context, the cost of a False Positive (mailing a bakery by mistake) and a False Negative (missing a high-value shipyard) are not equal.

Objective

Refine the classification threshold and feature engineering to minimize critical errors. The goal is to reach an optimal balance between Precision (avoiding "noise" in the final database) and Recall (ensuring no valid nautical lead is discarded).
Technical Challenges

The "False Positive" Trap (Precision Issue):

• Problem: Some businesses use nautical keywords in a non-professional context (e.g., a "Seaside Restaurant" or a "Captain's Bar").
• Impact: High risk of brand damage if automated marketing is sent to non-professionals.
• Solution: Implementing a "Negative Keyword" penalty system and enhancing the current weight of specific banwords

The "False Negative" Loss (Recall Issue):

• Problem: High-value professionals with poor SEO are often filtered out because the model lacks enough "signal."
• Impact: Loss of potential business opportunities.
• Solution: Developing a "Second-Pass" heuristic. If a lead is rejected by the ML but has a high-authority domain name or is located in a high-density port area, it is flagged for manual review instead of immediate deletion.

Threshold Calibration:

I am currently evaluating the Probability Threshold for the predict command. Instead of a binary 0.5 cutoff, I am considering a "Gray Zone" approach:

• => 0.6: Automated Sync.
• ]0.4 ; 0.6[: Tagged as "To Verify."
• <= 0.4: Discarded.