diff --git a/README.md b/README.md index ded4f60..507d88f 100644 --- a/README.md +++ b/README.md @@ -9,25 +9,24 @@ A comprehensive, modern dashboard for monitoring and managing Task Execution Ser ### ๐Ÿ” **Service Monitoring** - **Real-time Service Status**: Monitor health of TES gateway and all federated nodes -- **Network Topology Visualization**: Interactive map showing TES instances across ELIXIR federation +- **Network Topology Visualization**: Interactive map showing TES instances across the ELIXIR federation - **Performance Metrics**: Response times, connectivity status, and service availability -- **Automated Health Checks**: Continuous monitoring with 3-second refresh intervals +- **Automated Health Checks**: Continuous monitoring with configurable refresh intervals ### ๐Ÿ“Š **Task & Workflow Management** - **Task Submission**: Submit and track computational tasks across the federation - **Workflow Orchestration**: Support for CWL, Nextflow, and Snakemake workflows -- **Batch Processing**: Manage large-scale batch job submissions +- **Batch Processing**: Submit CWL, Nextflow, and Snakemake batch jobs - **Real-time Task Tracking**: Monitor task progress, logs, and execution status ### ๐Ÿ›  **Administrative Tools** -- **Node Management**: Add, remove, and configure TES nodes in the federation -- **Service Administration**: Comprehensive control panel for system administrators -- **Authentication System**: Secure admin access with role-based permissions -- **Configuration Management**: Dynamic service configuration and monitoring +- **Node Management**: Add, remove, update, and test TES nodes in the federation +- **Instance Management**: View and manage TES instances and their locations +- **Middleware Manager**: Configure and monitor the middleware pipeline +- **Authentication System**: Secure admin access with session-based permissions ### ๐ŸŽจ **Modern User Experience** - **Responsive Design**: Works seamlessly on desktop, tablet, and mobile devices -- **Dark Theme Support**: Modern authentication interface with clean aesthetics - **Interactive Components**: Rich data visualizations and intuitive navigation - **Real-time Updates**: Live data refresh without page reloads @@ -35,44 +34,38 @@ A comprehensive, modern dashboard for monitoring and managing Task Execution Ser ### Prerequisites -- **Python 3.8+** (recommended: 3.9 or higher) -- **Node.js 16+** (recommended: 18 LTS or higher) -- **npm** or **yarn** package manager +- **Docker** and **Docker Compose** (for containerised deployment) +- **Python 3.9+** (for local backend development) +- **Node.js 18+** and **npm** (for local frontend development) - **Git** for cloning the repository -### ๐Ÿƒโ€โ™‚๏ธ One-Command Setup +### ๐Ÿณ Docker Deployment (Recommended) -1. Clone the repository: - ```bash - git clone https://github.com/elixir-cloud-aai/elixir-cloud-demos.git - cd elixir-cloud-demos - ``` +Run both services with a single command: -2. Start the complete dashboard: - ```bash - ./run.sh - ``` +```bash +# Build and start frontend + backend +docker compose up --build -d - The startup script will: - - โœ… Create and activate Python virtual environment - - โœ… Install all backend dependencies - - โœ… Install all frontend dependencies - - โœ… Start Flask backend server on `http://localhost:8000` - - โœ… Start React development server on `http://localhost:3000` +# Follow logs +docker compose logs -f -3. **Open your browser** and navigate to: - - ๐ŸŒ **Frontend Dashboard**: http://localhost:3000 - - ๐Ÿ”ง **Backend API**: http://localhost:8000 +# Stop services +docker compose down +``` -### ๐Ÿ”ง Manual Setup +This starts: +- ๐ŸŒ **Frontend Dashboard**: http://localhost:3000 +- ๐Ÿ”ง **Backend API**: http://localhost:8000 -If you prefer to run the services separately: +### ๐Ÿ”ง Local Development Setup #### Backend (Flask API) ```bash cd backend +cp .env.example .env # configure environment variables python3 -m venv venv -source venv/bin/activate # On Windows: venv\Scripts\activate +source venv/bin/activate pip install -r requirements.txt python app.py ``` @@ -86,154 +79,154 @@ npm start ``` The frontend will be available at `http://localhost:3000` -### ๐Ÿณ Docker Deployment - -Build and run with Docker: - -```bash -# Build images -./build-and-push-images.sh - -# Deploy with Docker Compose (if available) -docker-compose up -d -``` - ## ๐Ÿ“ Project Structure ``` tes-dashboard/ -โ”œโ”€โ”€ ๐Ÿ“„ README.md # This file -โ”œโ”€โ”€ ๐Ÿ“ backend/ # Flask Backend API (Modular Architecture) -โ”‚ โ”œโ”€โ”€ app.py # Main Flask application entry point -โ”‚ โ”œโ”€โ”€ config.py # Configuration management -โ”‚ โ”œโ”€โ”€ requirements.txt # Python dependencies -โ”‚ โ”œโ”€โ”€ tes_instance_locations.json # TES federation configuration -โ”‚ โ”œโ”€โ”€ Dockerfile # Backend container image -โ”‚ โ”œโ”€โ”€ ๐Ÿ“ routes/ # API route handlers -โ”‚ โ”‚ โ”œโ”€โ”€ dashboard.py # Dashboard endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ tasks.py # Task management endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ workflows.py # Workflow endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ batch.py # Batch processing endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ nodes.py # Node management endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ instances.py # TES instance endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ network.py # Network topology endpoints -โ”‚ โ”‚ โ”œโ”€โ”€ health.py # Health check endpoints -โ”‚ โ”‚ โ””โ”€โ”€ logs.py # Log endpoints -โ”‚ โ”œโ”€โ”€ ๐Ÿ“ services/ # Business logic layer -โ”‚ โ”‚ โ”œโ”€โ”€ tes_service.py # TES API interactions -โ”‚ โ”‚ โ”œโ”€โ”€ task_service.py # Task operations -โ”‚ โ”‚ โ”œโ”€โ”€ workflow_service.py # Workflow orchestration -โ”‚ โ”‚ โ””โ”€โ”€ batch_service.py # Batch processing -โ”‚ โ”œโ”€โ”€ ๐Ÿ“ middleware/ # Custom middleware -โ”‚ โ”‚ โ””โ”€โ”€ middleware_api.py # Middleware API handlers -โ”‚ โ”œโ”€โ”€ ๐Ÿ“ utils/ # Helper utilities -โ”‚ โ”‚ โ”œโ”€โ”€ tes_utils.py # TES utility functions -โ”‚ โ”‚ โ”œโ”€โ”€ file_utils.py # File handling -โ”‚ โ”‚ โ””โ”€โ”€ auth_utils.py # Authentication helpers -โ”‚ โ”œโ”€โ”€ ๐Ÿ“ logs/ # Application logs (gitignored) -โ”‚ โ””โ”€โ”€ ๐Ÿ“ uploads/ # File upload directory (gitignored) -โ”œโ”€โ”€ ๐Ÿ“ frontend/ # React Frontend Application -โ”‚ โ”œโ”€โ”€ package.json # Node.js dependencies -โ”‚ โ”œโ”€โ”€ Dockerfile # Frontend container image -โ”‚ โ”œโ”€โ”€ src/ -โ”‚ โ”‚ โ”œโ”€โ”€ components/ # Reusable UI components -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ auth/ # Authentication components -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ charts/ # Data visualization -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ common/ # Shared components -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ forms/ # Input forms -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ logs/ # Log viewing -โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ tables/ # Data tables -โ”‚ โ”‚ โ”œโ”€โ”€ pages/ # Application pages -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Dashboard.js # Main dashboard -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Utilities.js # Service status monitoring -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ NodeManagement.js # Admin node management -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Tasks.js # Task management -โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ Workflows.js # Workflow management -โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ NetworkTopology.js # Network visualization -โ”‚ โ”‚ โ”œโ”€โ”€ contexts/ # React contexts -โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ AuthContext.js # Authentication state -โ”‚ โ”‚ โ”œโ”€โ”€ services/ # API communication -โ”‚ โ”‚ โ”œโ”€โ”€ hooks/ # Custom React hooks -โ”‚ โ”‚ โ”œโ”€โ”€ utils/ # Utility functions -โ”‚ โ”‚ โ””โ”€โ”€ styles/ # CSS stylesheets -โ”‚ โ””โ”€โ”€ public/ # Static assets +โ”œโ”€โ”€ README.md +โ”œโ”€โ”€ docker-compose.yml # Single-command deployment +โ”œโ”€โ”€ backend/ # Flask Backend API +โ”‚ โ”œโ”€โ”€ app.py # Application entry point & blueprint registration +โ”‚ โ”œโ”€โ”€ config.py # Configuration (env vars, CORS, paths) +โ”‚ โ”œโ”€โ”€ requirements.txt # Python dependencies +โ”‚ โ”œโ”€โ”€ .env.example # Environment variable template +โ”‚ โ”œโ”€โ”€ tes_instance_locations.json # TES federation node geo-data +โ”‚ โ”œโ”€โ”€ .tes_instances # Persisted TES instance list +โ”‚ โ”œโ”€โ”€ Dockerfile # Development container image +โ”‚ โ”œโ”€โ”€ Dockerfile.production # Hardened production image +โ”‚ โ”œโ”€โ”€ middleware_manager.py # Middleware chain orchestration +โ”‚ โ”œโ”€โ”€ middleware_config.py # Middleware configuration & wiring +โ”‚ โ”œโ”€โ”€ middleware_implementations.py # Middleware implementations +โ”‚ โ”œโ”€โ”€ middleware/ +โ”‚ โ”‚ โ””โ”€โ”€ middleware_api.py # Middleware REST API blueprint +โ”‚ โ”œโ”€โ”€ routes/ # Flask blueprints (one per domain) +โ”‚ โ”‚ โ”œโ”€โ”€ health.py # GET /api/health +โ”‚ โ”‚ โ”œโ”€โ”€ dashboard.py # GET /api/dashboard_data +โ”‚ โ”‚ โ”œโ”€โ”€ instances.py # /api/instances, /api/tes_locations, /api/service_info +โ”‚ โ”‚ โ”œโ”€โ”€ nodes.py # /api/nodes CRUD, /api/service_status +โ”‚ โ”‚ โ”œโ”€โ”€ tasks.py # /api/tasks, /api/submit_task +โ”‚ โ”‚ โ”œโ”€โ”€ workflows.py # /api/workflows, /api/submit_workflow +โ”‚ โ”‚ โ”œโ”€โ”€ batch.py # /api/batch_runs, /api/batch_cwl|nextflow|snakemake +โ”‚ โ”‚ โ”œโ”€โ”€ logs.py # /api/task_log, /api/workflow_log, /api/batch_log +โ”‚ โ”‚ โ””โ”€โ”€ network.py # /api/network_topology, /api/network_metrics +โ”‚ โ”œโ”€โ”€ services/ # Business logic layer +โ”‚ โ”‚ โ”œโ”€โ”€ tes_service.py # TES API interactions +โ”‚ โ”‚ โ”œโ”€โ”€ task_service.py # Task operations & auto-updater +โ”‚ โ”‚ โ”œโ”€โ”€ workflow_service.py # Workflow orchestration +โ”‚ โ”‚ โ””โ”€โ”€ batch_service.py # Batch job processing +โ”‚ โ””โ”€โ”€ utils/ +โ”‚ โ”œโ”€โ”€ tes_utils.py # TES utility functions +โ”‚ โ”œโ”€โ”€ file_utils.py # File handling helpers +โ”‚ โ””โ”€โ”€ auth_utils.py # Authentication helpers +โ”œโ”€โ”€ frontend/ # React Frontend Application +โ”‚ โ”œโ”€โ”€ package.json # Node.js dependencies +โ”‚ โ”œโ”€โ”€ nginx.conf # Nginx config (proxies /api โ†’ backend) +โ”‚ โ”œโ”€โ”€ server.js # Express server for k8s deployments +โ”‚ โ”œโ”€โ”€ Dockerfile # Development container image +โ”‚ โ”œโ”€โ”€ Dockerfile.production # Hardened production image +โ”‚ โ”œโ”€โ”€ public/ # Static assets (index.html, icons, manifest) +โ”‚ โ””โ”€โ”€ src/ +โ”‚ โ”œโ”€โ”€ App.js # Router & top-level layout +โ”‚ โ”œโ”€โ”€ components/ +โ”‚ โ”‚ โ”œโ”€โ”€ MiddlewareManager.js # Middleware manager component +โ”‚ โ”‚ โ”œโ”€โ”€ auth/ +โ”‚ โ”‚ โ”‚ โ”œโ”€โ”€ AdminLogin.js # Admin login form +โ”‚ โ”‚ โ”‚ โ””โ”€โ”€ ProtectedRoute.js # Route guard for admin pages +โ”‚ โ”‚ โ””โ”€โ”€ common/ +โ”‚ โ”‚ โ”œโ”€โ”€ Header.js # Top navigation bar +โ”‚ โ”‚ โ”œโ”€โ”€ Sidebar.js # Side navigation +โ”‚ โ”‚ โ”œโ”€โ”€ ServiceStatus.js # Service status widget +โ”‚ โ”‚ โ”œโ”€โ”€ ServiceStatusIndicator.js +โ”‚ โ”‚ โ”œโ”€โ”€ LoadingSpinner.js +โ”‚ โ”‚ โ””โ”€โ”€ ErrorMessage.js +โ”‚ โ”œโ”€โ”€ contexts/ +โ”‚ โ”‚ โ””โ”€โ”€ AuthContext.js # Global auth state +โ”‚ โ”œโ”€โ”€ hooks/ +โ”‚ โ”‚ โ”œโ”€โ”€ useInstances.js # TES instance data hook +โ”‚ โ”‚ โ””โ”€โ”€ usePolling.js # Generic polling hook +โ”‚ โ”œโ”€โ”€ pages/ +โ”‚ โ”‚ โ”œโ”€โ”€ Dashboard.js # Main overview dashboard +โ”‚ โ”‚ โ”œโ”€โ”€ Tasks.js # Task list & management +โ”‚ โ”‚ โ”œโ”€โ”€ TaskDetails.js # Individual task detail view +โ”‚ โ”‚ โ”œโ”€โ”€ SubmitTask.js # Task submission form +โ”‚ โ”‚ โ”œโ”€โ”€ Workflows.js # Workflow list & submission +โ”‚ โ”‚ โ”œโ”€โ”€ BatchProcessing.js # Batch job submission +โ”‚ โ”‚ โ”œโ”€โ”€ BatchRuns.js # Batch run history +โ”‚ โ”‚ โ”œโ”€โ”€ Logs.js # Task / workflow / batch log viewer +โ”‚ โ”‚ โ”œโ”€โ”€ NetworkTopology.js # Network topology visualisation +โ”‚ โ”‚ โ”œโ”€โ”€ NetworkTopologyPage.tsx # Network topology page wrapper +โ”‚ โ”‚ โ”œโ”€โ”€ RealTimeNetworkTopology.tsx # Live network topology +โ”‚ โ”‚ โ”œโ”€โ”€ NodeManagement.js # Admin: manage TES nodes +โ”‚ โ”‚ โ”œโ”€โ”€ InstanceManagement.js # TES instance management +โ”‚ โ”‚ โ”œโ”€โ”€ MiddlewareManager.js # Middleware configuration page +โ”‚ โ”‚ โ”œโ”€โ”€ ServiceInfo.js # TES service info viewer +โ”‚ โ”‚ โ”œโ”€โ”€ SystemStatus.js # System-wide status overview +โ”‚ โ”‚ โ”œโ”€โ”€ Utilities.js # Service health & utilities +โ”‚ โ”‚ โ”œโ”€โ”€ Settings.js # Application settings +โ”‚ โ”‚ โ””โ”€โ”€ ApiTest.js # API connectivity tester +โ”‚ โ”œโ”€โ”€ services/ # API client layer +โ”‚ โ”‚ โ”œโ”€โ”€ api.js # Axios instance & base URL config +โ”‚ โ”‚ โ”œโ”€โ”€ taskService.js # Task API calls +โ”‚ โ”‚ โ”œโ”€โ”€ workflowService.js # Workflow API calls +โ”‚ โ”‚ โ”œโ”€โ”€ batchService.js # Batch API calls +โ”‚ โ”‚ โ”œโ”€โ”€ instanceService.js # Instance API calls +โ”‚ โ”‚ โ”œโ”€โ”€ logService.js # Log API calls +โ”‚ โ”‚ โ”œโ”€โ”€ serviceInfoService.js # TES service info calls +โ”‚ โ”‚ โ”œโ”€โ”€ serviceService.js # General service calls +โ”‚ โ”‚ โ”œโ”€โ”€ serviceStatusService.js # Service status polling +โ”‚ โ”‚ โ”œโ”€โ”€ statusService.js # System status calls +โ”‚ โ”‚ โ””โ”€โ”€ mapService.js # Map/geo data service +โ”‚ โ”œโ”€โ”€ styles/ +โ”‚ โ”‚ โ””โ”€โ”€ MiddlewareManager.css +โ”‚ โ””โ”€โ”€ utils/ +โ”‚ โ”œโ”€โ”€ constants.js # App-wide constants +โ”‚ โ”œโ”€โ”€ formatters.js # Data formatting helpers +โ”‚ โ”œโ”€โ”€ helpers.js # General helper functions +โ”‚ โ””โ”€โ”€ validators.js # Input validation +โ””โ”€โ”€ test-data/ # Sample payloads for manual testing + โ”œโ”€โ”€ config.json + โ”œโ”€โ”€ sample-data.csv + โ”œโ”€โ”€ sample-input.txt + โ”œโ”€โ”€ tasks/ # Example TES task JSON payloads + โ”‚ โ”œโ”€โ”€ hello-world-task.json + โ”‚ โ”œโ”€โ”€ data-processing-task.json + โ”‚ โ”œโ”€โ”€ bioinformatics-analysis.json + โ”‚ โ””โ”€โ”€ gpu-ml-training.json + โ”œโ”€โ”€ batch-processing/ # Example batch submission payloads + โ”‚ โ”œโ”€โ”€ hello-world-batch.json + โ”‚ โ”œโ”€โ”€ data-analysis-batch.json + โ”‚ โ”œโ”€โ”€ genomics-pipeline-batch.json + โ”‚ โ””โ”€โ”€ ml-hyperparameter-batch.json + โ””โ”€โ”€ workflows/ # Example workflow definitions + โ”œโ”€โ”€ hello-world.cwl + โ”œโ”€โ”€ data-processing.cwl + โ”œโ”€โ”€ hello-world.nf + โ”œโ”€โ”€ rna-seq-analysis.nf + โ”œโ”€โ”€ Snakefile + โ”œโ”€โ”€ bioinformatics-pipeline.smk + โ””โ”€โ”€ config.yaml ``` -## ๐ŸŽฏ Key Components - -### Backend API (`/backend`) -- **Modular Flask Architecture** with separate routes, services, and middleware layers -- **Service Layer Pattern** for clean separation of business logic -- **Middleware System** for request/response processing and cross-cutting concerns -- **Flask REST API** with comprehensive TES management endpoints -- **Service Health Monitoring** with real-time status checks -- **Node Management API** for federated TES network administration -- **Batch Processing** support for large-scale job submissions -- **CORS-enabled** for seamless frontend integration - -### Frontend Dashboard (`/frontend`) -- **React 18** with modern hooks and context API -- **Styled Components** for component-scoped styling -- **Axios** for HTTP client communication -- **Lucide React** for consistent iconography -- **Responsive Design** with mobile-first approach - -### Authentication System -- **Admin Authentication** with session management -- **Protected Routes** for administrative functions -- **Role-based Access** for sensitive operations -- **24-hour Sessions** with automatic logout - -## ๐Ÿ“– Usage Guide - -### ๐Ÿ” **Service Status Monitoring** - -Navigate to **Utilities โ†’ Service Status** to monitor the health of your TES federation: - -- **Gateway Status**: Monitor the main TES gateway service -- **Node Health**: Real-time status of all federated TES nodes -- **Response Times**: Performance metrics for each service -- **Auto-refresh**: Status updates every 3 seconds while viewing - -### ๐Ÿ‘ฅ **Node Management** (Admin Only) - -Access **Administration โ†’ Node Management** to manage your TES federation: - -1. **Login**: Use admin credentials (`tesadmin` / `admin@dashboard`) -2. **Add Nodes**: Configure new TES instances in the federation -3. **Test Connectivity**: Verify node health and connectivity -4. **Remove Nodes**: Safely remove outdated or offline nodes - -### ๐Ÿ“Š **Task & Workflow Management** - -- **Submit Tasks**: Use the task submission interface for individual jobs -- **Batch Processing**: Submit multiple jobs with batch processing -- **Monitor Progress**: Track task execution in real-time -- **View Logs**: Access detailed execution logs and error messages - -### ๐ŸŒ **Network Topology** - -Visualize your TES federation: -- **Geographic View**: See TES nodes distributed across regions -- **Connection Status**: Monitor network connectivity between nodes -- **Performance Metrics**: View latency and throughput statistics - ## โš™๏ธ Configuration -### Backend Configuration +### Backend -The backend can be configured through environment variables: +Copy `.env.example` to `.env` in the `backend/` directory and fill in your values: -```bash -# Backend configuration -export TES_GATEWAY_URL="http://localhost:8000" -export FLASK_ENV="development" # or "production" -export CORS_ORIGINS="http://localhost:3000" +```env +SECRET_KEY=your-secret-key-here +FLASK_ENV=development + +# TES gateway credentials (optional) +FUNNEL_SERVER_USER= +FUNNEL_SERVER_PASSWORD= +TES_TOKEN= +TES_GATEWAY= ``` -### TES Federation Setup +### TES Federation Nodes -Configure your TES instances in `backend/tes_instance_locations.json`: +Add or edit nodes in `backend/tes_instance_locations.json`: ```json { @@ -248,190 +241,94 @@ Configure your TES instances in `backend/tes_instance_locations.json`: } ``` -### Frontend Configuration +## ๐Ÿ”Œ API Reference -Frontend settings in `frontend/.env`: - -```env -REACT_APP_API_URL=http://localhost:8000 -REACT_APP_REFRESH_INTERVAL=30000 +### Health +```http +GET /api/health # Backend health check ``` -## ๐Ÿ”Œ API Documentation - -### Service Status Endpoints - +### Dashboard & Instances ```http -GET /api/service-status # Get all service statuses -GET /api/service-health/{id} # Get specific service health -GET /api/nodes # List all TES nodes +GET /api/dashboard_data # Aggregated dashboard data +GET /api/instances # All TES instances +GET /api/tes_locations # Instance geo-locations +GET /api/healthy-instances # Healthy instances only +GET /api/service_info?tes_url= # Service info for a specific instance ``` -### Node Management Endpoints - +### Nodes ```http -POST /api/nodes # Add new TES node -PUT /api/nodes/{id} # Update TES node -DELETE /api/nodes/{id} # Remove TES node -GET /api/nodes/{id}/test # Test node connectivity +GET /api/nodes # List nodes +POST /api/nodes # Add a node +GET /api/nodes/ # Get node details +PUT /api/nodes/ # Update a node +DELETE /api/nodes/ # Remove a node +GET /api/nodes//health # Node health check +GET /api/service_status # All nodes service status +GET /api/test_connection # Test backend connectivity ``` -### Task Management Endpoints - +### Tasks ```http -GET /api/tasks # List tasks -POST /api/tasks # Submit new task -GET /api/tasks/{id} # Get task details -GET /api/tasks/{id}/logs # Get task logs +GET /api/tasks # List tasks +POST /api/submit_task # Submit a new task +GET /api/task_details?task_id=&tes_url= # Task details +GET /api/task_log/ # Task logs ``` -## ๐Ÿงช Testing - -### Backend Tests -```bash -cd backend -python -m pytest tests/ +### Workflows +```http +GET /api/workflows # List workflows +POST /api/submit_workflow # Submit a workflow (CWL/NF/Snakemake) +GET /api/latest_workflow_status # Latest workflow run status +GET /api/workflow_log/ # Workflow logs ``` -### Frontend Tests -```bash -cd frontend -npm test +### Batch +```http +GET /api/batch_runs # Batch run history +POST /api/batch_cwl # Submit CWL batch +POST /api/batch_nextflow # Submit Nextflow batch +POST /api/batch_snakemake # Submit Snakemake batch +GET /api/batch_log/ # Batch run logs ``` -### Integration Tests -```bash -./test-integration.sh +### Network +```http +GET /api/network_topology # Full network topology +GET /api/network_status # Network connectivity status +GET /api/network_metrics # Aggregated network metrics +GET /api/instance_metrics/ # Per-instance metrics +GET /api/data_transfers # Active data transfers +GET /api/storage_locations # Storage endpoint list ``` -## ๐Ÿ›  Development - -### Adding New Pages - -1. Create page component in `frontend/src/pages/` -2. Add route in `frontend/src/App.js` -3. Update navigation in `frontend/src/components/common/Sidebar.js` - -### Adding New API Endpoints - -1. Add endpoint in `backend/app.py` -2. Create service function in `frontend/src/services/` -3. Use service in React components - -### Styling Guidelines - -- Use **Styled Components** for component styling -- Follow **mobile-first** responsive design -- Use **Lucide React** icons for consistency -- Maintain **dark theme** support - ## ๐Ÿ› Troubleshooting -### Common Issues - -**Backend won't start:** -- Check Python version (3.8+ required) -- Verify virtual environment activation -- Install dependencies: `pip install -r requirements.txt` - -**Frontend won't start:** -- Check Node.js version (16+ required) -- Clear node_modules: `rm -rf node_modules && npm install` -- Check port 3000 availability - -**Service status shows all offline:** -- Verify backend is running on port 8000 -- Check CORS configuration -- Validate TES node URLs in configuration - -**Admin login not working:** -- Use credentials: `tesladmin` / `admin@dashboard` -- Check browser local storage -- Clear browser cache if needed - -## ๐Ÿ“ˆ Performance Optimization - -### Backend Optimization +**TES instances not loading (Network Error)** +- When running via Docker Compose the frontend proxies `/api` requests through nginx to the backend container. Ensure `REACT_APP_API_URL` is left empty (the default) so the proxy is used. +- Verify both containers are running: `docker compose ps` +- Check backend logs: `docker compose logs tes-dashboard-backend-service` -- Use **connection pooling** for TES node communications -- Implement **caching** for service status responses -- Add **rate limiting** for API endpoints -- Use **async/await** for concurrent node health checks +**Backend container not healthy** +- The health check calls `GET /api/health`. Confirm the backend started without import errors: `docker compose logs tes-dashboard-backend-service` -### Frontend Optimization +**Admin login not working** +- Default credentials: `tesadmin` / `admin@dashboard` +- Check browser console for CORS errors โ€” the backend must be reachable from the origin used by the browser. -- Implement **React.memo** for expensive components -- Use **lazy loading** for pages and components -- Add **service worker** for offline functionality -- Optimize **bundle size** with code splitting +**Frontend won't start locally** +- Requires Node.js 18+. Clear cached modules with `rm -rf node_modules && npm install`. -## ๐Ÿ”’ Security Considerations - -### Authentication - -- Admin sessions expire after 24 hours -- Passwords are validated client-side -- Consider implementing JWT tokens for production - -### Network Security - -- All TES communications use HTTPS in production -- CORS is properly configured for frontend-backend communication -- API rate limiting prevents abuse - -### Data Protection - -- No sensitive data stored in local storage -- All user inputs are sanitized -- TES node credentials handled securely - -## ๐Ÿ“š Additional Resources - -### TES Specification -- [GA4GH TES API](https://github.com/ga4gh/task-execution-schemas) -- [TES Implementation Guide](https://ga4gh.github.io/task-execution-schemas/) - -### Elixir Cloud Documentation -- [Elixir Cloud Portal](https://elixir-cloud.dcc.sib.swiss/) -- [Federation Architecture](https://github.com/elixir-cloud-aai) - -### Technology Stack Documentation -- [React Documentation](https://react.dev/) -- [Flask Documentation](https://flask.palletsprojects.com/) -- [Styled Components](https://styled-components.com/) +**Backend won't start locally** +- Requires Python 3.9+. Activate the virtual environment before running `python app.py`. ## ๐Ÿค Contributing -We welcome contributions to the TES Dashboard! Here's how to get started: - -### **How to Contribute** - 1. **Fork** the repository -2. **Clone** your fork locally -3. **Create** a feature branch: `git checkout -b feature/amazing-feature` -4. **Make** your changes and test thoroughly -5. **Commit** with descriptive messages: `git commit -m 'Add amazing feature'` -6. **Push** to your branch: `git push origin feature/amazing-feature` -7. **Submit** a pull request - -### **Contribution Guidelines** - -- **Code Style**: Follow existing patterns and linting rules -- **Testing**: Add tests for new functionality -- **Documentation**: Update README and inline comments -- **Backwards Compatibility**: Ensure existing features still work - -### **Reporting Bugs** - -1. Check existing issues first -2. Provide detailed reproduction steps -3. Include system information (OS, browser, versions) -4. Add screenshots or logs if helpful - -### **Feature Requests** - -1. Describe the use case clearly -2. Explain the expected behavior -3. Consider implementation approaches -4. Discuss with maintainers first for large changes +2. **Create** a feature branch: `git checkout -b feature/amazing-feature` +3. **Commit** your changes: `git commit -m 'Add amazing feature'` +4. **Push** to your branch: `git push origin feature/amazing-feature` +5. **Open** a pull request diff --git a/backend/.tes_instances b/backend/.tes_instances index 3396567..cb5f160 100644 --- a/backend/.tes_instances +++ b/backend/.tes_instances @@ -3,13 +3,12 @@ # Lines starting with # are comments # ELIXIR Cloud TES Instances -Funnel/OpenPBS @ ELIXIR-CZ,https://funnel.cloud.e-infra.cz/ -Funnel/Slurm @ ELIXIR-FI,https://vm4816.kaj.pouta.csc.fi/ -TESK/Kubernetes @ ELIXIR-CZ (Prod),https://tesk-prod.cloud.e-infra.cz/ -TESK/Kubernetes @ ELIXIR-CZ (NA),https://tesk-na.cloud.e-infra.cz/ -TESK/Kubernetes @ ELIXIR-DE,https://tesk.elixir-cloud.bi.denbi.de/ -TESK/Kubernetes @ ELIXIR-GR,https://tesk-eu.hypatia-comp.athenarc.gr/ -TESK/OpenShift @ ELIXIR-FI,https://csc-tesk-noauth.rahtiapp.fi/ +Funnel/OpenPBS @ ELIXIR-CZ,https://funnel.cloud.e-infra.cz +Poiesis @ ELIXIR-CZ,https://poiesis.dyn.cloud.e-infra.cz/ +TESK @ ELIXIR-FI,https://csc-tesk-noauth.rahtiapp.fi +Funnel/Slurm @ ELIXIR-FI,https://fip-86-50-228-254.kaj.poutavm.fi/ +Poiesis @ ELIXIR-FI,https://poiesis.rahtiapp.fi/ +Test Poiesis @ ELIXIR-FI,https://test-poiesis.rahtiapp.fi/ +Funnel/Slurm @ BIH,https://spe4hd.tes.bihealth.org +TESK @ ELIXIR-GR,https://tesk-eu.hypatia-comp.athenarc.gr -# Local Development -Local TES,http://localhost:8080 diff --git a/backend/Dockerfile.production b/backend/Dockerfile.production index 68512e9..70f90b2 100644 --- a/backend/Dockerfile.production +++ b/backend/Dockerfile.production @@ -6,7 +6,7 @@ ENV PYTHONDONTWRITEBYTECODE=1 \ FLASK_ENV=production RUN groupadd --gid 1000 appuser && \ - useradd --uid 1000 --gid appuser --shell / + useradd --uid 1000 --gid appuser --no-create-home --shell /usr/sbin/nologin appuser WORKDIR /app RUN apt-get update && apt-get install -y \ curl \ @@ -18,8 +18,6 @@ RUN pip install --no-cache-dir --upgrade pip && \ COPY . . RUN mkdir -p uploads logs && \ chown -R appuser:appuser /app -COPY tes_instance_locations.json . -COPY .tes_instances . USER appuser VOLUME ["/app/uploads"] EXPOSE 8000 diff --git a/backend/services/tes_service.py b/backend/services/tes_service.py index 7a66f87..c50aed6 100644 --- a/backend/services/tes_service.py +++ b/backend/services/tes_service.py @@ -135,13 +135,29 @@ def fetch_tes_status(instance): def get_service_info(tes_url): """Get service info from a TES instance with multiple endpoint attempts""" - try: + try: + tes_url = (tes_url or "").rstrip("/") + credentials = get_instance_credentials("unknown", tes_url) + headers = { + 'Accept': 'application/json', + 'User-Agent': 'TES-Dashboard/1.0' + } + auth = None + + if credentials.get('token'): + headers['Authorization'] = f"Bearer {credentials['token']}" + elif credentials.get('user') and credentials.get('password'): + auth = (credentials['user'], credentials['password']) + endpoints_to_try = [ f"{tes_url}/ga4gh/tes/v1/service-info", - f"{tes_url}/v1/tasks", + f"{tes_url}/v1/service-info", + f"{tes_url}/tasks/v1/service-info", f"{tes_url}/service-info", f"{tes_url}/api/service-info", f"{tes_url}/api/v1/service-info", + f"{tes_url}/v1/tasks", + f"{tes_url}/tasks/v1/tasks", ] last_error = None @@ -153,10 +169,8 @@ def get_service_info(tes_url): response = requests.get( endpoint, timeout=10, - headers={ - 'Accept': 'application/json', - 'User-Agent': 'TES-Dashboard/1.0' - }, + headers=headers, + auth=auth, verify=True ) @@ -164,19 +178,42 @@ def get_service_info(tes_url): if response.status_code == 200: try: - service_info = response.json() + payload = response.json() + if endpoint.endswith('/v1/tasks'): + print(f"โœ… TES API reachable via {endpoint}; building synthetic service info") + return { + 'name': f"TES Service @ {tes_url}", + 'id': tes_url, + 'organization': { + 'name': 'Unknown', + 'url': tes_url + }, + 'description': 'TES instance reachable, service-info endpoint not exposed at common paths.', + 'type': { + 'group': 'ga4gh', + 'artifact': 'tes', + 'version': 'Unknown' + }, + 'contactUrl': 'Unknown', + 'documentationUrl': 'Unknown', + 'storage': ['Unknown'], + 'version': payload.get('version', 'Unknown') if isinstance(payload, dict) else 'Unknown', + 'message': 'TES endpoint is reachable; service metadata is limited.', + 'timestamp': datetime.now(timezone.utc).isoformat() + } + print(f"โœ… Successfully got service info from {endpoint}") - return service_info + return payload except ValueError as json_error: print(f"โš ๏ธ Invalid JSON response: {json_error}") last_error = f"Invalid JSON: {json_error}" continue - elif response.status_code == 403: + elif response.status_code in (401, 403): print(f"๐Ÿ”’ Endpoint {endpoint} requires authentication") auth_required = True - last_error = "Authentication required" - break + last_error = "Authentication required or credentials rejected" + continue else: print(f"โš ๏ธ Status {response.status_code} from {endpoint}") diff --git a/backend/tes_instance_locations.json b/backend/tes_instance_locations.json index c92b168..d6a13e6 100644 --- a/backend/tes_instance_locations.json +++ b/backend/tes_instance_locations.json @@ -1,62 +1,106 @@ [ { - "id": "elixir-cz", - "name": "TESK Production", - "url": "https://tesk-prod.cloud.e-infra.cz", - "country": "Czech Republic", - "lat": 49.75, - "lng": 15.5, - "lon": 15.5, - "status": "healthy", - "description": "Primary TESK production instance", + "id": "elixir:cz:1", + "name": "Funnel/OpenPBS @ ELIXIR-CZ", + "url": "https://funnel.cloud.e-infra.cz", + "country": "Czechia", + "lat": 50.0755, + "lng": 14.4378, + "lon": 14.4378, + "status": "online", + "description": "Funnel with OpenPBS backend", + "version": "1.1", + "region": "EU-Central" + }, + { + "id": "elixir:cz:2", + "name": "Poiesis @ ELIXIR-CZ", + "url": "https://poiesis.dyn.cloud.e-infra.cz/", + "country": "Czechia", + "lat": 50.0755, + "lng": 14.4378, + "lon": 14.4378, + "status": "online", + "description": "Poiesis TES instance", + "version": "1.1", "region": "EU-Central" }, { - "id": "elixir-fi", - "name": "TESK/OpenShift @ ELIXIR-FI", - "url": "https://csc-tesk-noauth.rahtiapp.fi/ga4gh/tes", + "id": "elixir:fi:1", + "name": "TESK @ ELIXIR-FI", + "url": "https://csc-tesk-noauth.rahtiapp.fi", "country": "Finland", "lat": 60.1699, "lng": 24.9384, "lon": 24.9384, - "status": "healthy", - "description": "OpenShift-based TESK instance", + "status": "online", + "description": "TESK instance at CSC", + "version": "1.0", "region": "EU-North" }, { - "id": "elixir-gr", - "name": "TESK/Kubernetes @ ELIXIR-GR", - "url": "https://tesk.c3g.calculquebec.ca", - "country": "Greece", - "lat": 37.9838, - "lng": 23.7275, - "lon": 23.7275, - "status": "healthy", - "description": "Kubernetes-based TESK instance", - "region": "EU-South" + "id": "elixir:fi:2", + "name": "Funnel/Slurm @ ELIXIR-FI", + "url": "https://fip-86-50-228-254.kaj.poutavm.fi/", + "country": "Finland", + "lat": 60.1699, + "lng": 24.9384, + "lon": 24.9384, + "status": "online", + "description": "Funnel with Slurm backend", + "version": "unknown", + "region": "EU-North" }, { - "id": "funnel-cz", - "name": "Funnel/OpenPBS @ ELIXIR-CZ", - "url": "https://funnel.cloud.e-infra.cz", - "country": "Czech Republic", - "lat": 50.0755, - "lng": 14.4378, - "lon": 14.4378, - "status": "healthy", - "description": "Funnel with OpenPBS backend", + "id": "elixir:fi:3", + "name": "Poiesis @ ELIXIR-FI", + "url": "https://poiesis.rahtiapp.fi/", + "country": "Finland", + "lat": 60.1699, + "lng": 24.9384, + "lon": 24.9384, + "status": "online", + "description": "Poiesis TES instance", + "version": "1.1", + "region": "EU-North" + }, + { + "id": "elixir:fi:4", + "name": "Test Poiesis @ ELIXIR-FI", + "url": "https://test-poiesis.rahtiapp.fi/", + "country": "Finland", + "lat": 60.1699, + "lng": 24.9384, + "lon": 24.9384, + "status": "online", + "description": "Test Poiesis instance", + "version": "1.1", + "region": "EU-North" + }, + { + "id": "elixir:de:1", + "name": "Funnel/Slurm @ BIH", + "url": "https://spe4hd.tes.bihealth.org", + "country": "Germany", + "lat": 52.52, + "lng": 13.40, + "lon": 13.40, + "status": "online", + "description": "Funnel with Slurm backend", + "version": "1.0", "region": "EU-Central" }, { - "id": "local-tes", - "name": "Local TES", - "url": "http://localhost:8080", - "country": "Local", - "lat": 0, - "lng": 0, - "lon": 0, - "status": "unknown", - "description": "Local development TES instance", - "region": "Local" + "id": "elixir:gr:1", + "name": "TESK @ ELIXIR-GR", + "url": "https://tesk-eu.hypatia-comp.athenarc.gr", + "country": "Greece", + "lat": 37.9838, + "lng": 23.7275, + "lon": 23.7275, + "status": "online", + "description": "TESK instance", + "version": "1.0", + "region": "EU-South" } ] \ No newline at end of file diff --git a/backend/utils/auth_utils.py b/backend/utils/auth_utils.py index 5259bfe..8948a10 100644 --- a/backend/utils/auth_utils.py +++ b/backend/utils/auth_utils.py @@ -2,21 +2,37 @@ from config import FUNNEL_SERVER_USER, FUNNEL_SERVER_PASSWORD, TES_TOKEN def get_instance_credentials(instance_name, instance_url): + instance_url = (instance_url or '').rstrip('/') default_user = FUNNEL_SERVER_USER default_pass = FUNNEL_SERVER_PASSWORD default_token = TES_TOKEN - if 'tesk-prod.cloud.e-infra.cz' in instance_url: + if 'funnel.cloud.e-infra.cz' in instance_url: return { - 'user': os.getenv('TESK_PROD_USER', default_user), - 'password': os.getenv('TESK_PROD_PASSWORD', default_pass), - 'token': os.getenv('TESK_PROD_TOKEN', default_token) + 'user': os.getenv('FUNNEL_CZ_USER', default_user), + 'password': os.getenv('FUNNEL_CZ_PASSWORD', default_pass), + 'token': None } - elif 'tesk-na.cloud.e-infra.cz' in instance_url: + + elif 'fip-86-50-228-254.kaj.poutavm.fi' in instance_url: + return { + 'user': os.getenv('FUNNEL_FI_USER', default_user), + 'password': os.getenv('FUNNEL_FI_PASSWORD', default_pass), + 'token': None + } + + elif 'tesk-eu.hypatia-comp.athenarc.gr' in instance_url: + return { + 'user': os.getenv('TESK_GR_USER', default_user), + 'password': os.getenv('TESK_GR_PASSWORD', default_pass), + 'token': None + } + + elif 'spe4hd.tes.bihealth.org' in instance_url: return { - 'user': os.getenv('TESK_NA_USER', default_user), - 'password': os.getenv('TESK_NA_PASSWORD', default_pass), - 'token': os.getenv('TESK_NA_TOKEN', default_token) + 'user': None, + 'password': None, + 'token': os.getenv('BIHEALTH_TOKEN', default_token) } else: return { diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..eedf6e0 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,39 @@ +services: + tes-dashboard-backend-service: + build: + context: ./backend + dockerfile: Dockerfile.production + container_name: tes-dashboard-backend + environment: + PORT: 8000 + FLASK_ENV: production + FLASK_DEBUG: "false" + ports: + - "8000:8000" + volumes: + - ./backend/uploads:/app/uploads + - ./backend/tes_instance_locations.json:/app/tes_instance_locations.json + - ./backend/.tes_instances:/app/.tes_instances + extra_hosts: + - "spe4hd.tes.bihealth.org:194.94.4.48" + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/api/health', timeout=5)"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 40s + restart: unless-stopped + + tes-dashboard-frontend-service: + build: + context: ./frontend + dockerfile: Dockerfile + args: + REACT_APP_API_URL: "" + container_name: tes-dashboard-frontend + depends_on: + tes-dashboard-backend-service: + condition: service_healthy + ports: + - "3000:80" + restart: unless-stopped diff --git a/frontend/Dockerfile b/frontend/Dockerfile index d087525..4cabc6d 100644 --- a/frontend/Dockerfile +++ b/frontend/Dockerfile @@ -13,6 +13,10 @@ RUN npm ci --only=production # Copy source code COPY . . +# Accept API URL build argument (empty = use nginx proxy) +ARG REACT_APP_API_URL="" +ENV REACT_APP_API_URL=$REACT_APP_API_URL + # Build the application RUN npm run build diff --git a/frontend/src/services/api.js b/frontend/src/services/api.js index c852128..e4ca753 100644 --- a/frontend/src/services/api.js +++ b/frontend/src/services/api.js @@ -1,7 +1,7 @@ import axios from 'axios'; const getApiBaseUrl = () => { - if (process.env.REACT_APP_API_URL) { + if (process.env.REACT_APP_API_URL !== undefined && process.env.REACT_APP_API_URL !== null) { return process.env.REACT_APP_API_URL; } diff --git a/frontend/src/utils/constants.js b/frontend/src/utils/constants.js index ae9352c..80f52d1 100644 --- a/frontend/src/utils/constants.js +++ b/frontend/src/utils/constants.js @@ -88,13 +88,12 @@ export const THEME = { }; export const TES_INSTANCES = [ - { name: 'Funnel/OpenPBS @ ELIXIR-CZ', url: 'https://funnel.cloud.e-infra.cz', id: 'funnel-cz' }, - { name: 'Funnel/Slurm @ ELIXIR-FI', url: 'https://vm4816.kaj.pouta.csc.fi', id: 'funnel-fi' }, - { name: 'TESK/Kubernetes @ ELIXIR-CZ (Prod)', url: 'https://tesk-prod.cloud.e-infra.cz', id: 'elixir-cz-prod' }, - { name: 'TESK/Kubernetes @ ELIXIR-CZ (NA)', url: 'https://tesk-na.cloud.e-infra.cz', id: 'elixir-cz-na' }, - { name: 'TESK/Kubernetes @ ELIXIR-DE', url: 'https://tesk.elixir-cloud.bi.denbi.de', id: 'elixir-de' }, - { name: 'TESK/Kubernetes @ ELIXIR-GR', url: 'https://tesk-eu.hypatia-comp.athenarc.gr', id: 'elixir-gr' }, - { name: 'TESK/OpenShift @ ELIXIR-FI', url: 'https://csc-tesk-noauth.rahtiapp.fi', id: 'elixir-fi' }, - { name: 'TESK North America', url: 'https://tesk-na.cloud.e-infra.cz', id: 'tesk-na' }, - { name: 'Local TES', url: 'http://localhost:8080', id: 'local-tes' } + { name: 'Funnel/OpenPBS @ ELIXIR-CZ', url: 'https://funnel.cloud.e-infra.cz', id: 'elixir:cz:1' }, + { name: 'Poiesis @ ELIXIR-CZ', url: 'https://poiesis.dyn.cloud.e-infra.cz/', id: 'elixir:cz:2' }, + { name: 'TESK @ ELIXIR-FI', url: 'https://csc-tesk-noauth.rahtiapp.fi', id: 'elixir:fi:1' }, + { name: 'Funnel/Slurm @ ELIXIR-FI', url: 'https://fip-86-50-228-254.kaj.poutavm.fi/', id: 'elixir:fi:2' }, + { name: 'Poiesis @ ELIXIR-FI', url: 'https://poiesis.rahtiapp.fi/', id: 'elixir:fi:3' }, + { name: 'Test Poiesis @ ELIXIR-FI', url: 'https://test-poiesis.rahtiapp.fi/', id: 'elixir:fi:4' }, + { name: 'Funnel/Slurm @ BIH', url: 'https://spe4hd.tes.bihealth.org', id: 'elixir:de:1' }, + { name: 'TESK @ ELIXIR-GR', url: 'https://tesk-eu.hypatia-comp.athenarc.gr', id: 'elixir:gr:1' } ];