Skip to content

Latest commit

 

History

History
789 lines (594 loc) · 16.4 KB

File metadata and controls

789 lines (594 loc) · 16.4 KB

OpenTextShield v2.7 Deployment Checklist

Last Updated: October 23, 2025 Status: ✅ Production Ready


Pre-Deployment Checklist

☐ System Requirements Verification

  • Docker Installed

    docker --version  # Should be 20.10+ or later
    docker compose version  # Should be 2.0+
  • Available Resources

    # Single Container
    - Memory: 4GB minimum, 8GB recommended
    - Disk: 10GB minimum (includes model cache)
    - CPU: 2+ cores (4+ cores recommended)
    
    # 10-Server Deployment
    - Memory: 32GB minimum, 64GB recommended
    - Disk: 100GB minimum
    - CPU: 8+ cores (16+ recommended)
  • Network Connectivity

    # Test Docker Hub access
    curl -I https://hub.docker.com
    
    # Test pulling small image
    docker pull hello-world
  • Ports Available

    # Check ports not in use
    Single Container:
    - 8002 (API)
    - 8080 (Frontend)
    
    10-Server:
    - 8002 (Load balancer)
    - 9001-9010 (Individual servers)
    
    # Verify on macOS
    lsof -i :8002
    
    # Verify on Linux
    netstat -tulpn | grep 8002

☐ Architecture Selection

  • Identify Your Architecture

    # macOS
    arch  # Output: "arm64" or "i386"
    uname -m  # Output: "aarch64" or "x86_64"
    
    # Linux
    uname -m  # Output: "aarch64" (ARM64) or "x86_64" (AMD64)
  • Choose Image

    • ☐ ARM64 (Apple Silicon, Graviton, Pi): telecomsxchange/opentextshield:v2.7
    • ☐ AMD64 (EC2, GCP, Azure, x86): telecomsxchange/opentextshield:v2.7-amd64

☐ Docker Hub Authentication

  • Login (if needed)

    docker login
    # Enter Docker Hub username and password
    
    # Verify
    cat ~/.docker/config.json | grep "auths"
  • Optional: Create Access Token

    # More secure than password
    # Go to hub.docker.com → Account Settings → Security → Access Tokens
    docker login --username=USERNAME --password-stdin
    # Paste token when prompted

Single Container Deployment

☐ Pull Image

# ☐ For Apple Silicon/ARM64
docker pull telecomsxchange/opentextshield:v2.7

# ☐ For x86-64 servers
docker pull telecomsxchange/opentextshield:v2.7-amd64

# ☐ Verify pull successful
docker images | grep opentextshield

☐ Run Container

# ☐ Start container (substitute correct image)
docker run -d \
  --name opentextshield \
  -p 8002:8002 \
  -p 8080:8080 \
  telecomsxchange/opentextshield:v2.7

# ☐ Verify container started
docker ps | grep opentextshield

☐ Wait for Startup

# ☐ Monitor logs (~2-3 minutes)
docker logs -f opentextshield

# ☐ Look for these lines:
#    "🚀 OpenTextShield API - Starting..."
#    "✅ Virtual environment created"
#    "INFO: Uvicorn running on http://0.0.0.0:8002"

☐ Health Check

# ☐ Test health endpoint
curl http://localhost:8002/health

# ☐ Expected response:
# {
#   "status": "healthy",
#   "version": "2.6.0",
#   "models_loaded": {
#     "mbert_multilingual": true
#   }
# }

☐ API Test

# ☐ Test prediction endpoint
cat > /tmp/test.json << 'EOF'
{
  "text": "Click here to claim your free prize",
  "model": "ots-mbert"
}
EOF

curl -X POST http://localhost:8002/predict/ \
  -H "Content-Type: application/json" \
  -d @/tmp/test.json

# ☐ Expected response includes:
#    "label": "spam" (or "ham", "phishing")
#    "probability": 0.9996...
#    "processing_time": ~70ms (after warmup)

☐ Frontend Test

# ☐ Open browser
open http://localhost:8080
# or
firefox http://localhost:8080

# ☐ Verify you see the OpenTextShield UI

☐ Swagger Documentation

# ☐ Open browser
open http://localhost:8002/docs
# or
firefox http://localhost:8002/docs

# ☐ Verify Swagger UI loads with API endpoints visible

Docker Compose Deployment (10 Servers)

☐ Prepare Files

# ☐ Check for required files
ls -la docker-compose.10x.yml
ls -la nginx.conf

# ☐ If files don't exist, create them or download:
curl -O https://raw.githubusercontent.com/TelecomsXChangeAPi/OpenTextShield/main/docker-compose.10x.yml
curl -O https://raw.githubusercontent.com/TelecomsXChangeAPi/OpenTextShield/main/nginx.conf

☐ Configure Images

# ☐ Edit docker-compose.10x.yml
# ☐ Change all "opentextshield:v2.6" to "opentextshield:v2.7"
# ☐ For x86-64: use "opentextshield:v2.7-amd64" instead

# ☐ Verify configuration
grep "image:" docker-compose.10x.yml | head -3
# Should show: opentextshield:v2.7 or opentextshield:v2.7-amd64

☐ Start Deployment

# ☐ Start all services
docker compose -f docker-compose.10x.yml up -d

# ☐ Verify all containers started
docker compose -f docker-compose.10x.yml ps

# ☐ Check status (should show "Up")
docker compose -f docker-compose.10x.yml ps | grep "Up"

☐ Monitor Startup

# ☐ Watch logs (~3-5 minutes)
docker compose -f docker-compose.10x.yml logs -f

# ☐ Exit with Ctrl+C once you see all servers running

☐ Verify Individual Servers

# ☐ Check health of each server
for i in {1..10}; do
  echo -n "Server $i: "
  curl -s http://localhost:900$i/health | jq .status
done

# ☐ Expected output:
# Server 1: "healthy"
# Server 2: "healthy"
# ... etc for all 10

☐ Test Load Balancer

# ☐ Test through load balancer (port 8002)
curl -X POST http://localhost:8002/predict/ \
  -H "Content-Type: application/json" \
  -d '{"text":"test","model":"ots-mbert"}'

# ☐ Expected: classification result with ~70ms latency

☐ Load Test

# ☐ Run basic load test
for i in {1..5}; do
  time curl -X POST http://localhost:8002/predict/ \
    -H "Content-Type: application/json" \
    -d '{"text":"test","model":"ots-mbert"}' \
    -s | jq .processing_time
done

# ☐ Expected: latencies ~70-90ms

☐ Monitor Resources

# ☐ Check container stats
docker compose -f docker-compose.10x.yml stats

# ☐ Expected:
#    - CPU: 60-70% utilization
#    - Memory: 70% of available
#    - All containers using similar resources

Kubernetes Deployment

☐ Cluster Preparation

  • Kubernetes Cluster Running

    kubectl version
    kubectl cluster-info
  • Sufficient Resources

    kubectl describe nodes | grep -A 5 "Allocated resources"
    # Expected: Free CPU and memory available
  • Image Pull Secrets (if private registry)

    kubectl create secret docker-registry regcred \
      --docker-server=docker.io \
      --docker-username=USERNAME \
      --docker-password=PASSWORD \
      -n opentextshield

☐ Create Namespace

# ☐ Create dedicated namespace
kubectl create namespace opentextshield

# ☐ Verify
kubectl get namespace opentextshield

☐ Deploy Application

# ☐ Apply deployment manifest
kubectl apply -f - << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: opentextshield
  namespace: opentextshield
spec:
  replicas: 5
  selector:
    matchLabels:
      app: opentextshield
  template:
    metadata:
      labels:
        app: opentextshield
    spec:
      containers:
      - name: api
        image: telecomsxchange/opentextshield:v2.7  # or :v2.7-amd64
        ports:
        - containerPort: 8002
        livenessProbe:
          httpGet:
            path: /health
            port: 8002
          initialDelaySeconds: 120
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8002
          initialDelaySeconds: 60
          periodSeconds: 5
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
EOF

# ☐ Verify deployment created
kubectl get deployment -n opentextshield

☐ Monitor Rollout

# ☐ Watch pods starting (~2-3 minutes)
kubectl rollout status deployment/opentextshield -n opentextshield

# ☐ Expected: "deployment "opentextshield" successfully rolled out"

☐ Verify Pods

# ☐ Check all pods running
kubectl get pods -n opentextshield

# ☐ Expected: All pods "Running" and "Ready"
# ☐ Check pod details
kubectl describe pod <pod-name> -n opentextshield

☐ Test Service

# ☐ Create service
kubectl expose deployment opentextshield \
  --port=80 \
  --target-port=8002 \
  --type=LoadBalancer \
  -n opentextshield

# ☐ Get service endpoint
kubectl get service -n opentextshield

# ☐ Test (may take a minute to get external IP)
EXTERNAL_IP=$(kubectl get svc -n opentextshield -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
curl http://$EXTERNAL_IP/health

AWS EC2 Deployment

☐ Launch Instance

# ☐ Instance type: t3.xlarge (minimum)
# ☐ AMI: Ubuntu 24.04 LTS
# ☐ Storage: 100GB gp3 (for 10-server setup)
# ☐ Security group: Allow inbound 8002 (API), 8080 (Frontend)

☐ Connect & Install Docker

# ☐ SSH into instance
ssh -i key.pem ubuntu@instance-ip

# ☐ Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# ☐ Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# ☐ Verify
docker --version

☐ Deploy Application

# ☐ Clone repository (or download compose file)
git clone https://github.com/TelecomsXChangeAPi/OpenTextShield.git
cd OpenTextShield

# ☐ Edit docker-compose.10x.yml
# ☐ Update image to v2.7-amd64
sed -i 's/v2.6/v2.7-amd64/g' docker-compose.10x.yml

# ☐ Start deployment
docker compose -f docker-compose.10x.yml up -d

# ☐ Monitor
docker compose logs -f

☐ Configure Load Balancer

# ☐ AWS Application Load Balancer (ALB) setup:
# ☐ Create ALB
# ☐ Add target group pointing to EC2 instance:8002
# ☐ Add health check: /health
# ☐ Add listener: HTTP:80 → target group

☐ Test Deployment

# ☐ Test via EC2 instance directly
curl http://localhost:8002/health

# ☐ Test via ALB
curl http://<ALB-DNS-name>/health

# ☐ Monitor CloudWatch metrics
# ☐ Check target group health: all "Healthy"

Post-Deployment Verification

☐ Core Functionality

  • Health Check

    curl http://localhost:8002/health
    # Status should be "healthy"
  • API Prediction

    curl -X POST http://localhost:8002/predict/ \
      -H "Content-Type: application/json" \
      -d '{"text":"test message","model":"ots-mbert"}'
    # Should return classification result
  • Model Loaded

    curl http://localhost:8002/health | jq .models_loaded
    # Should show "mbert_multilingual": true

☐ API Documentation

  • Swagger UI

    curl http://localhost:8002/docs
    # Should return HTML with Swagger interface
  • OpenAPI JSON

    curl http://localhost:8002/openapi.json | jq .
    # Should return valid OpenAPI spec

☐ Frontend

  • Frontend Accessible
    curl http://localhost:8080
    # Should return HTML for OpenTextShield UI

☐ Performance

  • Latency Test

    time curl -X POST http://localhost:8002/predict/ \
      -H "Content-Type: application/json" \
      -d '{"text":"test","model":"ots-mbert"}' \
      -s > /dev/null
    # Should complete in <100ms (after warmup)
  • Concurrent Requests

    for i in {1..10}; do
      curl -X POST http://localhost:8002/predict/ \
        -H "Content-Type: application/json" \
        -d '{"text":"test","model":"ots-mbert"}' \
        -s | jq .processing_time &
    done
    wait
    # All should complete successfully

☐ Logging

  • Check Logs for Errors

    docker logs opentextshield | grep -i error
    # Should return no errors (or only expected warnings)
  • Check Startup Logs

    docker logs opentextshield | head -20
    # Should show successful startup sequence

Monitoring Setup

☐ Health Check Monitoring

# ☐ Set up periodic health checks
watch -n 10 'curl -s http://localhost:8002/health | jq .'

# ☐ Or with cron (every 5 minutes)
# */5 * * * * curl -f http://localhost:8002/health > /dev/null || alert

☐ Resource Monitoring

# ☐ Monitor container stats
docker stats --no-stream opentextshield

# ☐ Set up alerts for:
#    - CPU > 80%
#    - Memory > 90%
#    - Container not running

☐ Log Monitoring

# ☐ Aggregate logs
docker logs --follow opentextshield

# ☐ For production, use ELK/Splunk/CloudWatch

Troubleshooting Checklist

☐ Container Won't Start

  • ☐ Check if port already in use: lsof -i :8002
  • ☐ Check logs: docker logs opentextshield
  • ☐ Verify image exists: docker images | grep opentextshield
  • ☐ Check disk space: df -h
  • ☐ Check memory: free -h or vm_stat on macOS

☐ Health Check Fails

  • ☐ Wait longer (startup is 2-3 minutes)
  • ☐ Check logs: docker logs opentextshield | tail -20
  • ☐ Verify port accessible: curl -v http://localhost:8002/health
  • ☐ Check if container still running: docker ps | grep opentextshield

☐ API Returns 500 Error

  • ☐ Check model loaded: curl http://localhost:8002/health
  • ☐ Check logs for exceptions: docker logs opentextshield | tail -50
  • ☐ Verify JSON payload is valid
  • ☐ Check memory not exhausted: docker stats opentextshield

☐ Slow Performance

  • ☐ First request takes 400ms (normal), subsequent ~70ms
  • ☐ Check if CPU maxed out: docker stats opentextshield
  • ☐ Check if memory maxed out: docker stats opentextshield
  • ☐ Check disk I/O: iostat -x 1

☐ Load Balancer Returns 502

  • ☐ Check if backend servers running: docker compose ps
  • ☐ Check individual server health: curl http://localhost:9001/health
  • ☐ Wait longer for startup
  • ☐ Check nginx logs: docker compose logs load-balancer

Security Checklist

☐ Container Security

  • Running as Non-Root

    docker exec opentextshield id
    # Should show uid=1000 (not uid=0)
  • No Default Passwords

    docker inspect opentextshield | grep -i password
    # Should return nothing
  • No Secrets in Image

    docker history opentextshield:v2.7 | grep -i secret
    # Should return nothing

☐ Network Security

  • Firewall Rules

    • Only expose port 8002 (API) and 8080 (Frontend)
    • Restrict to trusted networks if possible
  • HTTPS/TLS

    • Use reverse proxy (nginx, ALB) for SSL termination
    • Don't expose HTTP directly to internet
  • API Key/Authentication (if needed)

    • Implement at reverse proxy level
    • Use bearer tokens or API keys

☐ Data Security

  • No Persistent Data Stored

    • Each request is stateless
    • No database required
  • Input Validation

    • API validates all input
    • No SQL injection risk (no database)

Upgrade Checklist (If Currently on v2.6)

☐ Backup

  • ☐ Note current configuration
  • ☐ Save any custom docker-compose files
  • ☐ Document any environment variables

☐ Plan Downtime

  • ☐ Determine maintenance window
  • ☐ Notify users if applicable
  • ☐ Prepare rollback plan

☐ Perform Upgrade

# ☐ Stop current version
docker compose down
# or
docker stop opentextshield

# ☐ Pull new version
docker pull telecomsxchange/opentextshield:v2.7

# ☐ Update docker-compose.yml
# Change: image: telecomsxchange/opentextshield:v2.6
# To:     image: telecomsxchange/opentextshield:v2.7

# ☐ Start new version
docker compose up -d
# or
docker run -d ... telecomsxchange/opentextshield:v2.7

# ☐ Verify health
curl http://localhost:8002/health

☐ Post-Upgrade Verification

  • ☐ Health check passes
  • ☐ API responds correctly
  • ☐ No errors in logs
  • ☐ Performance acceptable

☐ Rollback Plan (If Issues)

# ☐ Switch back to v2.6
docker pull telecomsxchange/opentextshield:v2.6
docker stop opentextshield
docker rm opentextshield
docker run -d ... telecomsxchange/opentextshield:v2.6

Documentation References

  • ☐ Read: v2.7_RELEASE_NOTES.md
  • ☐ Read: v2.7_RELEASE_DEPLOYMENT.md
  • ☐ Read: QUICK_REFERENCE.md
  • ☐ Review: REGISTRY_SUMMARY.md

Sign-Off

Once all items are checked:

☐ Pre-deployment verification complete
☐ Deployment method chosen and tested
☐ Post-deployment verification passed
☐ Monitoring configured
☐ Security requirements met
☐ Team notified and trained

✅ Ready for Production Deployment

Need help? Check troubleshooting section or see documentation links above.

Deployment successful? Great! Monitor with health checks and logs. 🎉