refactor: enhance frontend structure and backend dataset insights

[cristofima]

This PR enhances the frontend user experience and adds dataset insights while removing obsolete manual tooling. The changes improve navigation consistency through a centralized Header component, add smart problem type detection with column statistics, and implement optional time budget configuration with auto-calculation. The backend now provides detailed column statistics (unique values, missing data) to enable more informed target column selection.

Key changes:

• Frontend Structure: Consolidated header markup into reusable Header component with page-specific layouts for SEO metadata
• Dataset Insights: Added ColumnStats (unique values, missing data) calculated during dataset confirmation and displayed in the configuration UI
• Smart Defaults: Time budget now auto-calculates based on dataset size (120s-1200s) when not provided, with frontend validation for manual entries

[Copilot]

Pull request overview

This PR enhances the frontend user experience and adds dataset insights while removing obsolete manual tooling. The changes improve navigation consistency through a centralized Header component, add smart problem type detection with column statistics, and implement optional time budget configuration with auto-calculation. The backend now provides detailed column statistics (unique values, missing data) to enable more informed target column selection.

Key changes:

• Frontend Structure: Consolidated header markup into reusable Header component with page-specific layouts for SEO metadata
• Dataset Insights: Added ColumnStats (unique values, missing data) calculated during dataset confirmation and displayed in the configuration UI
• Smart Defaults: Time budget now auto-calculates based on dataset size (120s-1200s) when not provided, with frontend validation for manual entries

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/*	Removed obsolete manual verification and setup scripts
docs/README.md, docs/PROJECT_REFERENCE.md	Removed references to deleted tools directory
.github/SETUP_CICD.md	Removed setup-backend.ps1 references
infrastructure/terraform/TERRAFORM_BEST_PRACTICES.md	Updated Next.js version reference from 14 to 16
docs/LESSONS_LEARNED.md, docs/FRONTEND_DEPLOYMENT_ANALYSIS.md	Updated Next.js version documentation consistently
amplify.yml, .github/git-commit-messages-instructions.md	Updated Next.js version references in build config and commit guidelines
.github/copilot-instructions.md	Condensed coding guidelines for clarity, maintained critical patterns
frontend/components/Header.tsx	New centralized header component with conditional navigation links
frontend/app/layout.tsx	Added metadata template pattern for page titles
frontend/app/**/layout.tsx	Added page-specific layouts for training, results, history, configure pages
frontend/app/globals.css	Added global cursor-pointer styles for better UX consistency
frontend/app/page.tsx	Integrated Header component, updated cost estimate to $10-25/month
frontend/app/configure/[datasetId]/page.tsx	Enhanced with smart problem type detection, optional time budget with validation, column stats display
frontend/app/training/[jobId]/page.tsx	Integrated Header component, added cursor-pointer classes
frontend/app/results/[jobId]/page.tsx	Integrated Header, added copy-to-clipboard functionality with visual feedback, removed unused Link import
frontend/app/history/page.tsx	Integrated Header, enhanced delete modal with job details preview
frontend/components/FileUpload.tsx	Added cursor-pointer classes for consistency
frontend/lib/api.ts	Added ColumnStats TypeScript interface matching backend schema
backend/api/models/schemas.py	Added ColumnStats model, made time_budget optional
backend/api/routers/datasets.py	Calculate column statistics (unique, missing, missing_pct) during CSV analysis
backend/api/routers/training.py	Implemented auto time budget calculation based on dataset size
backend/api/services/dynamo_service.py	Added float-to-Decimal conversion helper for DynamoDB compatibility
backend/README.md, frontend/README.md	Documented new smart features (auto time budget, problem type detection)
README.md	Updated feature descriptions, added smart features table, updated status to MVP complete

Comments suppressed due to low confidence (1)

frontend/app/configure/[datasetId]/page.tsx:304

• [nitpick] The button applies cursor-pointer in line 298 but also conditionally applies cursor-not-allowed when disabled in line 300. This is redundant since disabled buttons should only have cursor-not-allowed.

Consider simplifying to:

className={`
  flex-1 py-3 px-6 rounded-lg font-medium text-white transition-colors
  ${!selectedTarget || isStarting || !isTimeBudgetValid
    ? 'bg-gray-400 cursor-not-allowed'
    : 'bg-indigo-600 hover:bg-indigo-700 cursor-pointer'
  }
`}

              className={`
                flex-1 py-3 px-6 rounded-lg font-medium text-white cursor-pointer
                ${!selectedTarget || isStarting || !isTimeBudgetValid
                  ? 'bg-gray-400 cursor-not-allowed'
                  : 'bg-indigo-600 hover:bg-indigo-700'
                }
                transition-colors
              `}

[Copilot]

[nitpick] Multiple navigation props (showViewHistory, showBackToUpload, showViewAllJobs) can be set to true simultaneously, but the current structure only displays one link at a time. While this may be intentional for the current use cases, consider handling multiple navigation links if needed in the future.

For example, if both showBackToUpload and showViewAllJobs are true, only showViewAllJobs would display. If multiple links should be supported, wrap them in a flex container:

<nav className="flex space-x-4">
  {showViewHistory && <Link ...>View History →</Link>}
  {showBackToUpload && <Link ...>← Back to Upload</Link>}
  {showViewAllJobs && <Link ...>View All Jobs →</Link>}
</nav>

[Copilot]

[nitpick] The time_budget field should not allow None to pass validation when a value is provided. The current schema allows None to be explicitly set, which could bypass validation. Consider using a validator or changing the logic.

However, the Field validation ge=60, le=3600 will still apply when a non-None value is provided, so this works correctly for the intended use case. The frontend validation (lines 25-26 in configure/[datasetId]/page.tsx) ensures invalid values aren't sent.