Merge pull request #182 from codeharborhub/dev-1

Update Docs: ml-docs

Author: ajay-dhangar

docs/machine-learning/advanced-ml-topics/explainable-ai/lime-shap.mdx

@@ -0,0 +1,115 @@
+---
+title: "LIME & SHAP: Interpreting the Black Box"
+sidebar_label: LIME & SHAP
+description: "A deep dive into local explanations using LIME and game-theory-based global/local explanations with SHAP."
+tags: [machine-learning, xai, lime, shap, interpretability]
+---
+
+When using complex models like XGBoost, Random Forests, or Neural Networks, we often need to know: *"Why did the model deny this specific loan?"* or *"What features are driving the predictions for all users?"* **LIME** and **SHAP** are the industry standards for answering these questions.
+
+## 1. LIME (Local Interpretable Model-Agnostic Explanations)
+
+LIME works on the principle that while a model may be incredibly complex globally, it can be approximated by a simple, linear model **locally** around a specific data point.
+
+### How LIME Works:
+1.  **Select a point:** Choose the specific instance you want to explain.
+2.  **Perturb the data:** Create new "fake" samples by slightly varying the features of that point.
+3.  **Get predictions:** Run these fake samples through the "black box" model.
+4.  **Weight the samples:** Give more weight to samples that are closer to the original point.
+5.  **Train a surrogate:** Train a simple Linear Regression model on this weighted, perturbed dataset.
+6.  **Explain:** The coefficients of the linear model act as the explanation.
+
+## 2. SHAP (SHapley Additive exPlanations)
+
+SHAP is based on **Shapley Values** from cooperative game theory. It treats every feature of a model as a "player" in a game where the "payout" is the model's prediction.
+
+### The Core Concept:
+SHAP calculates the contribution of each feature by comparing what the model predicts **with** the feature versus **without** it, across all possible combinations of features.
+
+**The result is "Additive":**
+
+$$
+f(x) = \text{base\_value} + \sum \text{shap\_values}
+$$
+
+* **Base Value:** The average prediction of the model across the training set.
+* **SHAP Value:** The amount a specific feature pushed the prediction higher or lower than the average.
+
+## 3. LIME vs. SHAP: The Comparison
+
+| Feature | LIME | SHAP |
+| :--- | :--- | :--- |
+| **Foundation** | Local Linear Surrogates | Game Theory (Shapley Values) |
+| **Consistency** | Can be unstable (results vary on perturbation) | Mathematically consistent and fair |
+| **Speed** | Very Fast | Can be slow (computationally expensive) |
+| **Scope** | Strictly Local | Both Local and Global |
+
+## 4. Visualization Logic
+
+The following diagram illustrates the flow of SHAP from the model output down to the feature-level contribution.
+
+```mermaid
+graph TD
+    Output[Model Prediction: 0.85] --> Base[Base Value / Mean: 0.50]
+    
+    subgraph Feature_Contributions [Shapley Attribution]
+    F1[Income: +0.20]
+    F2[Credit Score: +0.10]
+    F3[Age: +0.10]
+    F4[Debt: -0.05]
+    end
+    
+    Base --> F1
+    F1 --> F2
+    F2 --> F3
+    F3 --> F4
+    F4 --> Final[Final Prediction: 0.85]
+    
+    style F1 fill:#e8f5e9,stroke:#2e7d32,color:#333
+    style F4 fill:#ffebee,stroke:#c62828,color:#333
+    style Final fill:#e1f5fe,stroke:#01579b,color:#333
+
+```
+
+## 5. Global Interpretation with SHAP
+
+One of SHAP's greatest strengths is the **Summary Plot**. By aggregating the SHAP values of all points in a dataset, we can see:
+
+1. Which features are the most important.
+2. How the *value* of a feature (high vs. low) impacts the output.
+
+## 6. Implementation Sketch (Python)
+
+Both LIME and SHAP have robust Python libraries.
+
+```python
+import shap
+import lime
+import lime.lime_tabular
+
+# --- SHAP Example ---
+explainer = shap.TreeExplainer(my_model)
+shap_values = explainer.shap_values(X_test)
+
+# Visualize the first prediction's explanation
+shap.initjs()
+shap.force_plot(explainer.expected_value, shap_values[0,:], X_test.iloc[0,:])
+
+# --- LIME Example ---
+explainer_lime = lime.lime_tabular.LimeTabularExplainer(
+    X_train.values, feature_names=X_train.columns, class_names=['Negative', 'Positive']
+)
+exp = explainer_lime.explain_instance(X_test.values[0], my_model.predict_proba)
+exp.show_in_notebook()
+
+```
+
+## References
+
+* **SHAP GitHub:** [Official Repository and Documentation](https://github.com/slundberg/shap)
+* **LIME Paper:** ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://arxiv.org/abs/1602.04938)
+* **Distill.pub:** [Interpretable Machine Learning Guide](https://christophm.github.io/interpretable-ml-book/shap.html)
+
+---
+
+**LIME and SHAP help us understand "Tabular" and "Text" data. But what if we are using CNNs for images? How do we know which pixels the model is looking at?**

docs/machine-learning/advanced-ml-topics/explainable-ai/xai-basics.mdx

@@ -0,0 +1,91 @@
+---
+title: "XAI Basics: Beyond the Black Box"
+sidebar_label: XAI Basics
+description: "An introduction to Explainable AI, its importance in ethics and regulation, and the trade-off between performance and interpretability."
+tags: [machine-learning, xai, ethics, transparency, interpretability]
+---
+
+As Machine Learning models become more complex (like Deep Neural Networks and Transformers), they often become **"Black Boxes."** We can see the input and the output, but we don't truly understand *why* the model made a specific decision.
+
+**Explainable AI (XAI)** is a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms.
+
+## 1. Why do we need XAI?
+
+In many industries, a simple "prediction" isn't enough. We need justification for the following reasons:
+
+* **Trust and Accountability:** If a medical AI diagnoses a patient, the doctor needs to know which features (symptoms) led to that conclusion.
+* **Bias Detection:** XAI helps uncover if a model is making decisions based on protected attributes like race, gender, or age.
+* **Regulatory Compliance:** Laws like the **GDPR** include a "right to explanation," meaning users can demand to know how an automated decision was made about them.
+* **Model Debugging:** Understanding why a model failed is the first step toward fixing it.
+
+## 2. The Interpretability vs. Accuracy Trade-off
+
+There is generally an inverse relationship between how well a model performs and how easy it is to explain.
+
+| Model Type | Interpretability | Accuracy (Complex Data) |
+| :--- | :--- | :--- |
+| **Linear Regression** | High (Coefficients) | Low |
+| **Decision Trees** | High (Visual paths) | Medium |
+| **Random Forests** | Medium | High |
+| **Deep Learning** | Low (Black Box) | Very High |
+
+## 3. Key Concepts in XAI
+
+To navigate the world of explainability, we must distinguish between different scopes and methods:
+
+### A. Intrinsic vs. Post-hoc
+* **Intrinsic (Ante-hoc):** Models that are simple enough to be self-explanatory (e.g., a small Decision Tree).
+* **Post-hoc:** Methods applied *after* a complex model is trained to extract explanations (e.g., SHAP, LIME).
+
+### B. Global vs. Local Explanations
+* **Global Explainability:** Understanding the *entire* logic of the model. "What features are most important for all predictions?"
+* **Local Explainability:** Understanding a *single* specific prediction. "Why was *this* specific loan application rejected?"
+
+## 4. Logical Framework of XAI (Mermaid)
+
+The following diagram illustrates how XAI bridges the gap between the Machine Learning model and the Human User.
+
+```mermaid
+graph LR
+    Data[Training Data] --> ML_Model[Complex ML Model]
+    ML_Model --> Prediction[Prediction/Result]
+    
+    subgraph XAI_Layer [Explainability Layer]
+    ML_Model --> Explain_Method[XAI Method: SHAP/LIME/Grad-CAM]
+    Explain_Method --> Explanation[Human-Interpretable Explanation]
+    end
+    
+    Explanation --> User((Human User))
+    Prediction --> User
+    
+    style XAI_Layer fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#333
+    style ML_Model fill:#f5f5f5,stroke:#333,color:#333
+    style Explanation fill:#fff3e0,stroke:#ef6c00,color:#333
+
+```
+
+## 5. Standard XAI Techniques
+
+We will cover these in detail in the following chapters:
+
+1. **Feature Importance:** Ranking which variables had the biggest impact on the model.
+2. **Partial Dependence Plots (PDP):** Showing how a feature affects the outcome while holding others constant.
+3. **LIME:** Approximating a complex model locally with a simpler, interpretable one.
+4. **SHAP:** Using game theory to fairly attribute the "payout" (prediction) to each "player" (feature).
+
+## 6. Evaluation Criteria for Explanations
+
+What makes an explanation "good"?
+
+* **Fidelity:** How accurately does the explanation represent what the model actually did?
+* **Understandability:** Is the explanation simple enough for a non-technical user?
+* **Robustness:** Does the explanation stay consistent for similar inputs?
+
+## References
+
+* **DARPA:** [Explainable Artificial Intelligence (XAI) Program](https://www.darpa.mil/program/explainable-artificial-intelligence)
+* **Book:** [Interpretable Machine Learning by Christoph Molnar](https://christophm.github.io/interpretable-ml-book/)
+
+---
+
+**Now that we understand the "why," let's look at one of the most popular methods for explaining individual predictions.**