Phase K: Add search analytics and admin ranking controls

Introduce search analytics and admin-tunable ranking weights. Added RankingSettings, RankingWeight and multiple Search analytics DTOs under app/Search; repositories RankingSettingsRepository and SearchAnalyticsRepository under app/Repository; migration database/migrations/005_create_search_analytics.sql and updated doogle-tables-no-data.sql.

Public changes: new admin page public/ranking.php, record searches and load ranking overrides in public/search.php, minor crawl UI link and CSS for admin UI. Repositories (Site/Image/Video) now read weights from RankingSettings and apply full-text caps from settings.

Tests and docs updated (ARCHITECTURE2.md, README-refactored.md, new integration/unit tests) and analytics store only hashed IP/user-agent to protect raw identifiers. Searches continue to work if DB migration is not applied; failures fall back to defaults.

Author: safesploit

ARCHITECTURE2.md

@@ -108,11 +108,19 @@ The next target layout is:
 │   ├── Repository/
 │   │   ├── ImageRepository.php
 │   │   ├── ImageSearchRepository.php
+│   │   ├── RankingSettingsRepository.php
+│   │   ├── SearchAnalyticsRepository.php
 │   │   ├── SiteRepository.php
 │   │   ├── SiteSearchRepository.php
 │   │   ├── VideoRepository.php
 │   │   └── VideoSearchRepository.php
 │   ├── Search/
+│   │   ├── RankingExpression.php
+│   │   ├── RankingSettings.php
+│   │   ├── RankingWeight.php
+│   │   ├── SearchAnalyticsEvent.php
+│   │   ├── SearchAnalyticsTerm.php
+│   │   └── SearchAnalyticsTypeSummary.php
 │   └── Security/
 │       ├── CrawlerSecurityPolicy.php
 │       ├── CsrfToken.php
@@ -140,6 +148,7 @@ The next target layout is:
 │   ├── login.php
 │   ├── logout.php
 │   ├── crawl.php
+│   ├── ranking.php
 │   ├── ajax/
 │   │   ├── setBroken.php
 │   │   ├── updateImageCount.php
@@ -189,6 +198,7 @@ public/
 ├── login.php
 ├── logout.php
 ├── crawl.php
+├── ranking.php
 ├── ajax/
 │   ├── setBroken.php
 │   ├── updateImageCount.php
@@ -1229,11 +1239,6 @@ Ranking by result type:
 - Images prioritise exact/partial title, then alt text, image URL, HTTPS, and available title/alt text.
 - Videos prioritise exact/partial title, then description, video URL, site URL, HTTPS, thumbnail availability, and metadata completeness.
 
-Future optional work:
-
-- Search analytics beyond click telemetry.
-- Authenticated admin ranking settings for viewing/tuning ranking weights.
-
 Acceptance criteria:
 
 - all search verticals use the same ranking model shape
@@ -1242,6 +1247,30 @@ Acceptance criteria:
 - ranking order is deterministic
 - repository tests cover ranking behaviour
 
+### Phase K - Search Analytics And Admin Ranking Controls
+
+Goal: make search quality observable and allow authenticated admins to tune the
+existing ranking weights without changing the default ranking behaviour.
+
+Tasks:
+
+- add `search_queries` table and migration
+- record public site/image/video searches with term, vertical, page, result count, and hashed request identifiers
+- add `ranking_settings` table and migration
+- load ranking weight overrides from the database with safe defaults when the migration is absent
+- add admin-only `/ranking.php`
+- show search analytics by vertical, top terms, zero-result terms, and recent searches
+- allow admins to update per-vertical ranking weights with CSRF protection
+- log ranking update/security events without secrets
+
+Acceptance criteria:
+
+- public search continues working if analytics/ranking tables are not migrated yet
+- default ranking weights match Phase J
+- saved ranking weights affect subsequent search results
+- only authenticated admins can view or update ranking controls
+- tests cover analytics persistence, ranking settings persistence, DTOs, and ranking behaviour
+
 ---
 
 ## 20. Security Requirements
@@ -1258,7 +1287,7 @@ Acceptance criteria:
 - Keep crawler security policy enabled by default.
 - Apply baseline security headers at Nginx.
 - Rate limit login and crawl POST requests.
-- Log auth, crawl, CSRF, and rate-limit events without secrets.
+- Log auth, crawl, ranking, CSRF, and rate-limit events without secrets.
 
 ### 20.2 Session Cookie Settings
 

README-refactored.md

@@ -24,10 +24,7 @@ Completed from `ARCHITECTURE2.md`:
 - Phase H: Production Hardening
 - Phase I: Videos Search Vertical
 - Phase J: Search Ranking
-
-Remaining:
-
-- Phase K: Search analytics and admin ranking controls are future optional work.
+- Phase K: Search Analytics And Admin Ranking Controls
 
 Key changes now in place:
 
@@ -37,6 +34,8 @@ Key changes now in place:
 - Legacy root `crawl.php`, root `crawl-manual.php`, and `classes/` have been removed.
 - Search uses repositories, services, DTOs, full-text indexes, relevance ranking, and bounded click boost for sites, images, and videos.
 - Ranking uses a shared deterministic formula with capped click/full-text boosts and per-vertical quality signals.
+- Public searches are recorded in `search_queries` with hashed request identifiers for admin analytics.
+- Admins can view analytics and tune per-vertical ranking weights at `/ranking.php`.
 - Browser crawling requires an admin login and CSRF token.
 - CLI crawling does not require a browser session, but still enforces crawler safety policy.
 - Web and CLI crawl jobs are stored in `crawl_jobs` and shown on the authenticated crawl page.
@@ -70,6 +69,7 @@ Open:
 
 - Doogle search: http://localhost:8000
 - Admin crawl page: http://localhost:8000/crawl.php
+- Admin ranking page: http://localhost:8000/ranking.php
 - phpMyAdmin: http://localhost:8081
 
 Default local database details without a `.env` override:
@@ -138,6 +138,26 @@ Site search returns 20 results per page. Image search returns 30 results per
 page. Video search returns 24 results per page in a responsive thumbnail-card
 grid.
 
+Search analytics are recorded when `database/migrations/005_create_search_analytics.sql`
+has been applied. If the analytics table is not present, public search still
+renders normally.
+
+## Admin Ranking And Analytics
+
+The admin ranking page is available after login:
+
+```text
+http://localhost:8000/ranking.php
+```
+
+It shows search analytics by vertical, top terms, zero-result terms, and recent
+searches. Request IP and user-agent values are stored as SHA-256 hashes, not as
+raw values.
+
+The same page lets admins update per-vertical ranking weights for Sites, Images,
+and Videos. Default weights match Phase J; saved overrides are loaded by public
+search on the next request. Click boost and full-text boost caps remain fixed.
+
 ## Crawling: Web UI
 
 Use the web UI when you want an authenticated browser workflow and crawl
@@ -266,13 +286,15 @@ database/migrations/001_add_search_fulltext_indexes.sql
 database/migrations/002_update_users_auth_schema.sql
 database/migrations/003_create_crawl_jobs.sql
 database/migrations/004_add_video_search.sql
+database/migrations/005_create_search_analytics.sql
 ```
 
 Apply a migration to the Docker database:
 
 ```sh
 docker compose -f docker/compose.yml exec -T mysql_db mysql -uroot -proot doogle < database/migrations/003_create_crawl_jobs.sql
 docker compose -f docker/compose.yml exec -T mysql_db mysql -uroot -proot doogle < database/migrations/004_add_video_search.sql
+docker compose -f docker/compose.yml exec -T mysql_db mysql -uroot -proot doogle < database/migrations/005_create_search_analytics.sql
 ```
 
 For non-Docker MySQL, apply the same SQL files with your normal MySQL client.
@@ -375,13 +397,14 @@ If you already have an older Docker volume or database:
 
 Resetting the Docker database deletes local indexed sites, images, videos,
 users, and crawl history.
+It also deletes search analytics and ranking setting overrides.
 
 ## Security Defaults
 
 - Browser crawl requires an authenticated admin.
 - Browser crawl POST requires CSRF validation.
 - Login and crawl POST requests are rate limited.
-- Auth and crawl security events are logged.
+- Auth, crawl, and ranking security events are logged.
 - CLI crawl is trusted local/container execution, not public HTTP access.
 - Private/reserved network crawling is blocked by default.
 - Passwords are hashed with `password_hash()`.

app/Repository/ImageRepository.php

@@ -5,15 +5,18 @@
 namespace Doogle\Repository;
 
 use Doogle\Search\RankingExpression;
+use Doogle\Search\RankingSettings;
 use PDO;
 use PDOStatement;
 
 final class ImageRepository implements ImageSearchRepository
 {
     private const IMAGE_FULL_TEXT_COLUMNS = 'title, alt, imageUrl';
+    private readonly RankingSettings $rankingSettings;
 
-    public function __construct(private readonly PDO $pdo)
+    public function __construct(private readonly PDO $pdo, ?RankingSettings $rankingSettings = null)
     {
+        $this->rankingSettings = $rankingSettings ?? new RankingSettings();
     }
 
     public function countBySearchTerm(string $term): int
@@ -92,15 +95,16 @@ private function imageWhereSql(): string
 
     private function imageRankingSql(): string
     {
+        $weights = $this->rankingSettings->weightsFor('images');
         $scores = [
-            RankingExpression::weightedEquals('title', ':rankTitleExactTerm', 220),
-            RankingExpression::weightedEquals('alt', ':rankAltExactTerm', 180),
-            RankingExpression::weightedLike('title', ':rankTitleTerm', 100),
-            RankingExpression::weightedLike('alt', ':rankAltTerm', 75),
-            RankingExpression::weightedLike('imageUrl', ':rankImageUrlTerm', 25),
-            RankingExpression::httpsUrl('imageUrl', 3),
-            RankingExpression::nonEmpty('alt', 5),
-            RankingExpression::nonEmpty('title', 3),
+            RankingExpression::weightedEquals('title', ':rankTitleExactTerm', $weights['title_exact']),
+            RankingExpression::weightedEquals('alt', ':rankAltExactTerm', $weights['alt_exact']),
+            RankingExpression::weightedLike('title', ':rankTitleTerm', $weights['title_partial']),
+            RankingExpression::weightedLike('alt', ':rankAltTerm', $weights['alt_partial']),
+            RankingExpression::weightedLike('imageUrl', ':rankImageUrlTerm', $weights['image_url_partial']),
+            RankingExpression::httpsUrl('imageUrl', $weights['https_image_url']),
+            RankingExpression::nonEmpty('alt', $weights['alt_present']),
+            RankingExpression::nonEmpty('title', $weights['title_present']),
             RankingExpression::boundedClickBoost(),
         ];
 
@@ -110,8 +114,8 @@ private function imageRankingSql(): string
                 RankingExpression::boundedMysqlFullTextBoost(
                     self::IMAGE_FULL_TEXT_COLUMNS,
                     ':rankFullTextTerm',
-                    50,
-                    120
+                    $this->rankingSettings->fullTextWeight(),
+                    $this->rankingSettings->fullTextCap()
                 )
             );
         }

app/Repository/RankingSettingsRepository.php

@@ -0,0 +1,83 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Doogle\Repository;
+
+use Doogle\Search\RankingSettings;
+use PDO;
+
+final class RankingSettingsRepository
+{
+    public function __construct(private readonly PDO $pdo)
+    {
+    }
+
+    public function load(): RankingSettings
+    {
+        $statement = $this->pdo->query(
+            'SELECT setting_key, setting_value
+             FROM ranking_settings'
+        );
+
+        if ($statement === false) {
+            return new RankingSettings();
+        }
+
+        $overrides = [];
+
+        foreach ($statement->fetchAll(PDO::FETCH_ASSOC) as $row) {
+            $key = (string) ($row['setting_key'] ?? '');
+
+            if ($key === '') {
+                continue;
+            }
+
+            $overrides[$key] = (int) ($row['setting_value'] ?? 0);
+        }
+
+        return new RankingSettings($overrides);
+    }
+
+    /**
+     * @param array<string, int> $weights
+     */
+    public function saveWeights(string $type, array $weights): void
+    {
+        $settings = new RankingSettings();
+        $type = in_array($type, $settings->supportedTypes(), true) ? $type : 'sites';
+        $sanitizedWeights = $settings->sanitizeWeights($type, $weights);
+
+        foreach ($sanitizedWeights as $key => $value) {
+            $this->upsertSetting(RankingSettings::settingKey($type, $key), (string) $value);
+        }
+    }
+
+    private function upsertSetting(string $key, string $value): void
+    {
+        if ($this->isMysql()) {
+            $statement = $this->pdo->prepare(
+                'INSERT INTO ranking_settings (setting_key, setting_value)
+                 VALUES (:setting_key, :setting_value)
+                 ON DUPLICATE KEY UPDATE
+                    setting_value = VALUES(setting_value),
+                    updated_at = CURRENT_TIMESTAMP'
+            );
+        } else {
+            $statement = $this->pdo->prepare(
+                'INSERT OR REPLACE INTO ranking_settings (setting_key, setting_value, updated_at)
+                 VALUES (:setting_key, :setting_value, CURRENT_TIMESTAMP)'
+            );
+        }
+
+        $statement->execute([
+            ':setting_key' => $key,
+            ':setting_value' => $value,
+        ]);
+    }
+
+    private function isMysql(): bool
+    {
+        return $this->pdo->getAttribute(PDO::ATTR_DRIVER_NAME) === 'mysql';
+    }
+}