+
+
+
1) Classification Metrics
+
+
Accuracy:
+ The proportion of correct predictions over all predictions:
+ (TP + TN) / (TP + TN + FP + FN).
+ Use when classes are balanced and you want a single easy‐to‐interpret number.
+
+
Precision:
+ The fraction of positive predictions that are actually positive:
+ TP / (TP + FP).
+ Use when false positives are costly (e.g. spam filter—better to miss some spam than flag good mail).
-
2) Precision, Recall & Specificity
-
Classification
-
Precision: The proportion of positive predictions that are correct, calculated as TP / (TP + FP). High Precision is crucial when false positives are costly, such as in spam email detection, where misclassifying legitimate emails as spam disrupts user experience.
-
Recall (Sensitivity): The proportion of actual positives correctly predicted, calculated as TP / (TP + FN). High Recall is vital when missing positives is risky, such as in disease diagnosis, where failing to identify a sick patient could have severe consequences.
-
Specificity: The true negative rate, calculated as TN / (TN + FP). It measures how well a model identifies negatives, making it valuable in medical testing to minimize false alarms (e.g., incorrectly diagnosing healthy patients as sick).
+
Recall (Sensitivity):
+ The fraction of actual positives correctly identified:
+ TP / (TP + FN)
+ Use when false negatives are costly (e.g. disease screening—don’t miss sick patients).
+
+
F1 Score:
+ The harmonic mean of Precision and Recall:
+ 2·(Precision·Recall)/(Precision+Recall)
+ Use when you need a balance between Precision & Recall on an imbalanced dataset.
+
+
ROC-AUC (Area Under ROC Curve):
+ Measures ability to distinguish classes across all thresholds.
+ Ranges from 0.5 (random) to 1 (perfect).
+ Use when you care about ranking positives above negatives.
+
+
PR-AUC (Area Under Precision-Recall Curve):
+ Summarizes Precision vs. Recall trade-off.
+ More informative than ROC-AUC when positives are rare.
+ Use when dealing with highly imbalanced data.
-
3) Macro, Micro, and Weighted Averages
-
Classification
-
Macro Precision / Recall / F1: Computes the metric for each class independently and averages them, treating all classes equally. This is ideal for balanced datasets or when all classes are equally important, such as in multiclass image classification with similar class frequencies.
-
Micro Precision / Recall / F1: Aggregates true positives (TP), false positives (FP), and false negatives (FN) across all classes before computing the metric. It provides a global perspective and is suitable for imbalanced datasets or multilabel problems, as it accounts for class frequency.
-
Weighted Precision / Recall / F1: Averages the metric across classes, weighted by the number of true instances per class. This balances the importance of classes based on their frequency, making it useful for imbalanced datasets where larger classes should have more influence but smaller classes are still considered.
+
Log Loss:
+ Penalizes confident wrong predictions via negative log-likelihood.
+ Lower is better.
+ Use when you need well-calibrated probability estimates.
+
+
Cohen’s Kappa:
+ Measures agreement between predictions and true labels accounting for chance.
+ 1 is perfect, 0 is random.
+ Use when you want to factor out chance agreement.
-
4) Average Precision (PR-AUC Variants)
-
Classification
-
Average Precision: The Area Under the Precision-Recall Curve (PR-AUC) summarizes the trade-off between Precision and Recall. It is particularly useful for imbalanced datasets, where ROC-AUC may overestimate performance. Average Precision is computed by averaging Precision values at different Recall thresholds, providing a robust measure for ranking tasks or rare class detection.
+
+
+
+
2) Regression Metrics
+
+
R² (Coefficient of Determination):
+ Proportion of variance in the target explained by features:
+ 1 is perfect, 0 means no better than predicting the mean, negative is worse than mean.
+ Use when you want a normalized measure of fit.
-
5) ROC-AUC Variants
-
Classification
-
ROC-AUC: The Area Under the Receiver Operating Characteristic Curve plots the true positive rate (Recall) against the false positive rate (1 - Specificity) at various thresholds. It quantifies the model’s ability to separate classes, with higher values indicating better performance.
-
Macro ROC-AUC: Averages the ROC-AUC scores across all classes, treating each class equally. This is suitable for balanced multiclass problems where all classes are of equal importance.
-
Micro ROC-AUC: Computes a single ROC-AUC by aggregating predictions and true labels across all classes. It is effective for multiclass or multilabel problems with class imbalance, as it accounts for the overall prediction distribution.
+
MAE (Mean Absolute Error):
+ Average absolute difference between predictions and actual values:
+ mean(|y_pred − y_true|)
+ Use when you need an interpretable “average” error and want to downweight outliers.
+
+
RMSE (Root Mean Squared Error):
+ Square root of the average squared errors:
+ √mean((y_pred − y_true)²).
+ Penalizes large errors more heavily.
+ Use when large deviations are especially undesirable.
-
6) Confusion Matrix Stats (Per Class)
-
Classification
-
True Positives (TP): The number of correct positive predictions for a given class.
-
True Negatives (TN): The number of correct negative predictions for a given class.
-
False Positives (FP): The number of incorrect positive predictions for a given class (false alarms).
-
False Negatives (FN): The number of incorrect negative predictions for a given class (missed detections). These stats are visualized in PyCaret’s confusion matrix plots, aiding class-wise performance analysis.
+
MSE (Mean Squared Error):
+ The average squared error:
+ mean((y_pred − y_true)²).
+ Similar to RMSE but in squared units; often used in optimization.
-
7) Other Useful Metrics
-
Classification
-
Cohen’s Kappa: Measures the agreement between predicted and actual labels, adjusted for chance. It ranges from -1 to 1, where 1 indicates perfect agreement, 0 indicates chance-level agreement, and negative values suggest worse-than-chance performance. Kappa is useful for multiclass problems with imbalanced labels.
-
Matthews Correlation Coefficient (MCC): A balanced measure that considers TP, TN, FP, and FN, calculated as (TP * TN - FP * FN) / sqrt((TP + FP)(TP + FN)(TN + FP)(TN + FN)). It ranges from -1 to 1, with 1 being perfect prediction. MCC is particularly effective for imbalanced datasets due to its symmetry across classes.
-
Regression
-
MSE (Mean Squared Error): The average of squared differences between predicted and actual values. It amplifies larger errors, making it sensitive to outliers. Lower MSE indicates better performance.
-
MAPE (Mean Absolute Percentage Error): The average of absolute percentage differences between predicted and actual values, calculated as (1/n) * Σ(|actual - predicted| / |actual|) * 100. It is useful when relative errors are important but can be unstable if actual values are near zero.
+
RMSLE (Root Mean Squared Log Error):
+ √mean((log(1+y_pred) − log(1+y_true))²).
+ Less sensitive to large differences when both true and predicted are large.
+ Use when target spans several orders of magnitude.
+
+
MAPE (Mean Absolute Percentage Error):
+ mean(|(y_true − y_pred)/y_true|)·100.
+ Expresses error as a percentage.
+ Use when relative error matters—but avoid if y_true≈0.
+
'
- f"
'
- f"
'
+ "