Merge pull request #27 from Hypercart-Dev-Tools/rules/add-logic-clone-detection-fuzzy

Rules/add logic clone detection fuzzy to Development

Author: noelsaw1

CHANGELOG.md

@@ -5,6 +5,51 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.79] - 2026-01-02
+
+### Fixed
+- **HTML Report Generation** - Fixed `url_encode: command not found` error
+  - Changed `url_encode()` to `url_encode_path()` on line 859
+  - Function was removed in v1.0.77 but one call site was missed
+  - Now uses correct function from `common-helpers.sh`
+  - **Impact:** HTML reports now generate without errors
+
+## [1.0.78] - 2026-01-02
+
+### Added
+- **Function Clone Detector (Tier 1)** - Hash-based detection of duplicate function definitions across files
+  - New pattern: `dist/patterns/duplicate-functions.json` - Detects exact function clones (Type 1)
+  - New function: `process_clone_detection()` - Extracts functions, normalizes code, computes MD5 hashes
+  - Thresholds: min 5 lines, min 2 files, min 2 occurrences
+  - Normalization: Strips comments and whitespace before hashing
+  - **Impact:** Catches copy-paste violations where identical functions exist in multiple files
+  - **Coverage:** 60-70% of all clones (Type 1 exact copies only)
+  - **False Positive Rate:** < 5% (proven hash-based approach)
+
+- **Test Fixtures for Clone Detection**
+  - `dist/tests/fixtures/dry/duplicate-functions.php` - Single-file fixture with documented test cases
+  - `dist/tests/fixtures/dry/file-a.php` - Multi-file test (includes/user-validation.php)
+  - `dist/tests/fixtures/dry/file-b.php` - Multi-file test (admin/settings.php)
+  - `dist/tests/fixtures/dry/file-c.php` - Multi-file test (ajax/handlers.php)
+  - Expected violations: `validate_user_email` (3 files), `sanitize_api_key` (2 files)
+
+### Changed
+- **HTML Report Template** - Updated "Magic Strings" section to "DRY Violations"
+  - Now includes both magic strings and duplicate functions
+  - Added subtitle: "Includes magic strings and duplicate functions"
+  - Stat card label changed from "Magic Strings" to "DRY Violations"
+
+- **Scanner Output** - Added new section "FUNCTION CLONE DETECTOR"
+  - Displays after "MAGIC STRING DETECTOR" section
+  - Shows count of duplicate functions found
+  - Uses same violation reporting format as magic strings
+
+### Improved
+- **File Path Handling** - Enhanced `process_clone_detection()` to handle both files and directories
+  - Detects if `$PATHS` is a single file or directory
+  - Uses `safe_file_iterator()` for paths with spaces
+  - Excludes vendor/, node_modules/ directories
+
 ## [1.0.77] - 2026-01-02
 
 ### Added

PROJECT/1-INBOX/PROJECT-LOGIC-DUPLICATION.md

@@ -349,6 +349,7 @@ detect_exact_function_clones() {
 **Method:** Replace identifiers with placeholders, hash
 **Threshold:** Minimum 10 lines, 70%+ similarity
 **Expected False Positive Rate:** 10-20%
+**Status**: Deferred as of Jan. 2, 2026 - wait for feedback
 
 ```bash
 detect_normalized_clones() {

PROJECT/1-INBOX/TRIAGE-2026-01-02.md

@@ -0,0 +1,286 @@
+# PROJECT Inbox Triage – 2026-01-02
+
+**Created:** 2026-01-02  
+**Status:** In Progress  
+**Purpose:** Summarize recommended actions for each `PROJECT/1-INBOX` document so the maintainer can review and decide, one by one.
+
+---
+
+## 1. CONSOLIDATION-COMPLETE.md
+
+**File:** `PROJECT/1-INBOX/CONSOLIDATION-COMPLETE.md`  
+**Type:** Update log for logic-duplication spec
+
+### Observation
+- States that `LOGIC-DUPLICATION-UPDATES.md` has been merged into `PROJECT-LOGIC-DUPLICATION.md`.
+- Status inside file already says consolidation is **complete**.
+- `LOGIC-DUPLICATION-UPDATES.md` currently lives in `PROJECT/3-COMPLETED/` (summary of changes).
+
+### Recommended Changes
+- Treat as a **completed documentation update**, not an active task.
+- Add/confirm frontmatter:
+  - `Status: Completed`
+  - `Completed: 2026-01-01` (or actual date)
+- **Move:** `CONSOLIDATION-COMPLETE.md` → `PROJECT/3-COMPLETED/`.
+
+### Questions for You
+1. Do you want to keep this as a separate “update log” file, or fold its contents into the bottom of `PROJECT-LOGIC-DUPLICATION.md` under a "Document History" section and then archive/delete it?
+
+---
+
+## 2. DRY-POC-SUMMARY.md
+
+**File:** `PROJECT/1-INBOX/DRY-POC-SUMMARY.md`  
+**Type:** Proof-of-concept summary for Magic String Detector (DRY)
+
+### Observation
+- Status line: `Planning Complete – Ready for Implementation`.
+- Describes:
+  - Planning doc: `NEXT-FIND-DRY.md`.
+  - 3 DRY JSON patterns under `dist/patterns/dry/`.
+  - `dist/patterns/dry/README.md` for documentation.
+- Lists remaining work: fixtures + aggregation wiring.
+- Codebase already contains DRY patterns/docs; there are also fixture-related reports in `PROJECT/3-COMPLETED/`.
+
+### Recommended Changes
+- Treat this as a **completed POC planning+design summary**, not an active task.
+- Update status block to:
+  - `Status: Completed`
+  - Add a small **“Implementation Outcome”** subsection linking to:
+    - `dist/patterns/dry/` JSONs
+    - `dist/patterns/dry/README.md`
+    - Any relevant `TEST_FIXTURES_*` docs if they cover DRY
+- **Move:** `DRY-POC-SUMMARY.md` → `PROJECT/3-COMPLETED/`.
+
+### Questions for You
+1. Is Phase 1 DRY (3 patterns + fixtures + aggregation) effectively shipped, or are fixtures/aggregation still partially TODO?
+2. Should we add a short "Shipped In" version tag here (e.g., `Shipped In: v1.0.xx`)?
+
+---
+
+## 3. FIND-DRY.md
+
+**File:** `PROJECT/1-INBOX/FIND-DRY.md`  
+**Type:** Generic DRY + architecture checker specification (repo-agnostic)
+
+### Observation
+- Large, general spec for a grep-first DRY/architecture checker.
+- Not specific to wp-code-check; more of a background design document.
+- You now have wp-code-check–specific DRY plan in `NEXT-FIND-DRY.md` and POC summary in `DRY-POC-SUMMARY.md`.
+
+### Recommended Changes
+- Reclassify as **reference/spec**, not an INBOX task.
+- Add a short header note:
+  - `Status: Reference`
+  - `Note: Background spec informing NEXT-FIND-DRY.md and DRY implementation.`
+- **Move:** `FIND-DRY.md` → `PROJECT/FIND-DRY.md` (root-level reference).
+
+### Questions for You
+1. Do you still actively iterate on this generic spec, or is it stable reference now?
+2. Do you want this re-titled to something like `PROJECT-DRY-ARCH-SPEC.md`, or keep the existing filename for now?
+
+---
+
+## 4. GITHUB-ADD-ISSUE.md
+
+**File:** `PROJECT/1-INBOX/GITHUB-ADD-ISSUE.md`  
+**Type:** Raw research note / web-scraped style answer
+
+### Observation
+- Describes third-party browser extensions for quickly creating GitHub issues.
+- No task framing (no status, acceptance criteria, or link to wp-code-check roadmap).
+- Looks more like an ad-hoc research answer than a concrete project task.
+
+### Recommended Changes
+Two options depending on whether you care about this item:
+
+**Option A – Archive as Cancelled**
+- Add frontmatter:
+  - `Status: Cancelled`
+  - `Cancelled Reason: Out of scope for wp-code-check core roadmap (developer tooling only).`
+- **Move:** `GITHUB-ADD-ISSUE.md` → `PROJECT/3-COMPLETED/` as an archived research note.
+
+**Option B – Keep as Deferred Idea**
+- Add frontmatter:
+  - `Status: Deferred`
+  - `Deferred Until: After v1.x baseline + DRY work`
+  - `Deferred Reason: Nice-to-have productivity tool, not core functionality.`
+- Keep in `1-INBOX/` but clearly marked as deferred.
+
+### Questions for You
+1. Is "quick GitHub issue creation" something you realistically want to pursue as part of this repo, or should we mark it Cancelled and archive it?
+
+---
+
+## 5. NEW-PATTERN-OPPORTUNITIES.md
+
+**File:** `PROJECT/1-INBOX/NEW-PATTERN-OPPORTUNITIES.md`  
+**Type:** Pattern ideas from Woo All Products for Subscriptions audit
+
+### Observation
+- Lists 8+ pattern ideas with details (severity, rule IDs, proposed grep expressions).
+- Some entries already marked as COMPLETE or ENHANCED (e.g., unsanitized `$_GET`, admin capability checks).
+- Mix of:
+  - Implemented patterns
+  - Pending ideas
+  - Future possibilities
+
+### Recommended Changes
+- Convert into a **pattern backlog / idea bank**, not an INBOX task.
+- Add a top-level **Status Summary** section grouping patterns into:
+  - Implemented (with version numbers)
+  - Pending / To Evaluate
+  - Rejected / Low ROI
+- Add frontmatter:
+  - `Status: Reference`
+  - `Purpose: Backlog of potential detection rules derived from IRL audits.`
+- **Move:** `NEW-PATTERN-OPPORTUNITIES.md` → `PROJECT/NEW-PATTERN-OPPORTUNITIES.md` (root-level reference).
+
+### Questions for You
+1. Do you want to keep treating this as a living idea bank, updated as patterns ship, or should we snapshot it and spin off a smaller "NEXT-PATTERNS" task doc for near-term work?
+
+---
+
+## 6. NEXT-BASELINE-ISSUE.md
+
+**File:** `PROJECT/1-INBOX/NEXT-BASELINE-ISSUE.md`  
+**Type:** Support-style explanation of baseline behavior for a specific plugin
+
+### Observation
+- Reads like a conversation transcript explaining baseline behavior (69 baselined findings, 0 new).
+- Clarifies that behavior is correct but UX could be clearer.
+- Not structured as a spec (no acceptance criteria or implementation plan).
+
+### Recommended Changes
+If you want to preserve it:
+- Add frontmatter:
+  - `Status: Completed`
+  - `Type: Support Explanation / UX Note`
+- Add short summary section:
+  - "Key UX Observations & Potential Improvements" (2–3 bullets).
+- **Move:** `NEXT-BASELINE-ISSUE.md` → `PROJECT/3-COMPLETED/`.
+
+If you don't need it:
+- Mark `Status: Cancelled` and archive in `3-COMPLETED/` (or delete).
+
+### Questions for You
+1. Would you like this turned into a more formal "Baseline UX / FAQ" entry under `PROJECT/` (e.g., `BASELINE-BEHAVIOR-FAQ.md`), or is the existing explanation enough and can be archived as-is?
+
+---
+
+## 7. NEXT-CALIBRATION.md
+
+**File:** `PROJECT/1-INBOX/NEXT-CALIBRATION.md`  
+**Type:** Calibration plan (false-positive reduction roadmap)
+
+### Observation
+- Status: `In Progress` (already in the file).
+- Phase 1 priorities 1–3.5 marked as COMPLETED.
+- Phase 2 / 3 tasks still open.
+- Multiple COMPLETED docs in `3-COMPLETED/` directly relate (fixtures eval, severity overrides, test fixtures reports).
+
+### Recommended Changes
+- This is an **active working document**.
+- Keep as the primary calibration tracker.
+- Update/confirm metadata:
+  - `Status: In Progress`
+  - Possibly `Assigned Version:` if you want a target.
+- **Move:** `NEXT-CALIBRATION.md` → `PROJECT/2-WORKING/NEXT-CALIBRATION.md`.
+
+### Questions for You
+1. Do you want to split out finished Phase 1 into a separate `CALIBRATION-PHASE-1-COMPLETED.md` summary, leaving this file only for future work (Phase 2/3)?
+
+---
+
+## 8. NEXT-FIND-DRY.md
+
+**File:** `PROJECT/1-INBOX/NEXT-FIND-DRY.md`  
+**Type:** WP Code Check–specific DRY implementation plan
+
+### Observation
+- Status: `Planning`.
+- Concrete 3-phase plan reusing existing pattern infrastructure.
+- `DRY-POC-SUMMARY.md` shows Phase 1 planning and pattern creation are done.
+
+### Recommended Changes
+- Treat this as a **completed planning spec** backing the DRY POC.
+- Add metadata:
+  - `Status: Completed (Phase 1 Planning)`
+  - Optional: `Linked Summary: DRY-POC-SUMMARY.md`
+- **Move:** `NEXT-FIND-DRY.md` → `PROJECT/NEXT-FIND-DRY.md` (root-level reference for DRY work).
+
+### Questions for You
+1. Do you want to keep Phases 2–3 in this same file, or spin them into a new `NEXT-DRY-EXPANSION.md` so this doc reflects "what we planned for Phase 1" only?
+
+---
+
+## 9. PROJECT-LOGIC-DUPLICATION.md
+
+**File:** `PROJECT/1-INBOX/PROJECT-LOGIC-DUPLICATION.md`  
+**Type:** Canonical spec for logic clone detection (grep+aggregation)
+
+### Observation
+- `LOGIC-DUPLICATION-UPDATES.md` (already in `3-COMPLETED/`) claims this doc has been updated with:
+  - Real-world context
+  - Adjusted timelines
+  - Go/No-Go criteria
+  - JSON schema example
+  - Lessons learned from v1.0.73
+  - Integration checklist & WP-specific patterns
+- Implementation itself (duplicate-functions pattern, fixtures, wiring) is **not yet obviously present** from the high-level repo listing.
+
+### Recommended Changes
+Two paths, depending on your intent:
+
+**Option A – Treat as Ready-to-Start Implementation**
+- Add frontmatter:
+  - `Status: Not Started`
+  - `Priority: HIGH`
+- Keep in `1-INBOX/` for now as "next candidate big feature".
+- Add a small "Next Actions" list pointing to the concrete files from the integration checklist.
+
+**Option B – Move to Active Work**
+- If you intend to start implementation soon:
+  - Set `Status: In Progress`.
+  - **Move:** `PROJECT-LOGIC-DUPLICATION.md` → `PROJECT/2-WORKING/`.
+
+### Questions for You
+1. Is logic-clone detection something you want to start in the current milestone (then we move it to `2-WORKING`), or should it remain a high-priority backlog item in `1-INBOX`?
+
+---
+
+## 10. LOGIC-DUPLICATION-UPDATES.md (Context Check)
+
+**File:** `PROJECT/3-COMPLETED/LOGIC-DUPLICATION-UPDATES.md`  
+**Type:** Summary of updates applied to `PROJECT-LOGIC-DUPLICATION.md`
+
+### Observation
+- Already in `3-COMPLETED/`.
+- Describes how the feasibility study was updated to align with proven v1.0.73 architecture.
+
+### Recommended Changes
+- No folder move needed (already COMPLETED).
+- Optional: Add at top:
+  - `Status: Completed`
+  - `Linked Spec: PROJECT-LOGIC-DUPLICATION.md`.
+
+### Questions for You
+1. Do you want `CONSOLIDATION-COMPLETE.md` and `LOGIC-DUPLICATION-UPDATES.md` eventually merged into a single "Document History" section inside `PROJECT-LOGIC-DUPLICATION.md`?
+
+---
+
+## Next Steps Checklist (For You)
+
+You can now go through each item and decide:
+
+- [ ] **CONSOLIDATION-COMPLETE.md** – Archive to COMPLETED? Merge into spec?
+- [ ] **DRY-POC-SUMMARY.md** – Mark Completed + move to COMPLETED?
+- [ ] **FIND-DRY.md** – Move to PROJECT root as reference?
+- [ ] **GITHUB-ADD-ISSUE.md** – Cancel & archive, or defer?
+- [ ] **NEW-PATTERN-OPPORTUNITIES.md** – Convert to pattern backlog and move to PROJECT root?
+- [ ] **NEXT-BASELINE-ISSUE.md** – Archive as support note or convert to FAQ?
+- [ ] **NEXT-CALIBRATION.md** – Move to 2-WORKING as active calibration driver?
+- [ ] **NEXT-FIND-DRY.md** – Mark Phase 1 planning completed and move to PROJECT root?
+- [ ] **PROJECT-LOGIC-DUPLICATION.md** – Keep as backlog or promote to 2-WORKING?
+
+If you confirm your decisions on each bullet, I can apply the moves, add the frontmatter changes, and keep everything consistent with the `AGENTS.md` workflow rules.