Merge pull request #22 from Monadical-SAS/afreydev/improvements-0.1.1

Improvements version 0.1.1

Author: afreydev

.gitignore

@@ -43,4 +43,10 @@ next-env.d.ts
 **/.env
 
 apps/homer/etc/config.yaml
+apps/homer/config/config.yml
 etc/basedomain.txt
+etc/default_admin_email.txt
+etc/environment.txt
+etc/config.yaml
+
+etc/supervisor/conf.d/*

Dockerfile.manager

@@ -20,6 +20,9 @@ RUN apt-get update -y && apt-get install -y \
     git \
     apache2-utils
 
+RUN curl -L https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -o /usr/bin/yq && \
+    chmod +x /usr/bin/yq
+
 RUN python3 -m pip install virtualenv && \
     virtualenv "/opt/$VENV_NAME"
 

README.md

@@ -63,18 +63,19 @@ Get up and running in minutes with our automated installer that configures every
 
 3. Configure Let's Encrypt for SSL certificates:
    ```bash
-   docker compose run --rm manager shtool setup_letsencrypt <domain> <email> [traefik_username] [traefik_password]
+   docker compose run --rm manager shtool setup_letsencrypt <domain> <email> [traefik_username] [traefik_password] [environment]
    ```
 
    Parameters:
    - `<domain>`: Your domain name (required)
    - `<email>`: Email for Let's Encrypt notifications (required)
    - `[username]`: Admin username (optional, defaults to "admin")
    - `[password]`: Admin password (optional, auto-generated if not specified)
+   - `[environment]`: Environment type - "prod" or "dev" (optional, defaults to "prod")
 
    Example:
    ```bash
-   docker compose run --rm manager shtool setup_letsencrypt example.com admin@example.com myadmin mysecurepass
+   docker compose run --rm manager shtool setup_letsencrypt example.com admin@example.com myadmin mysecurepass prod
    ```
    Note: If password is omitted, a secure random password will be generated.
 
@@ -99,6 +100,24 @@ Get up and running in minutes with our automated installer that configures every
    docker compose up -d
    ```
 
+## Environment Configuration
+
+SelfHostYour.Tech supports two deployment environments:
+
+### Production Environment (prod)
+- **HTTPS enabled**: All services use SSL/TLS certificates from Let's Encrypt
+- **Secure redirects**: HTTP traffic is automatically redirected to HTTPS
+- **Full SSL validation**: Complete certificate chain validation
+- **Recommended for**: Live deployments, production use
+
+### Development Environment (dev)
+- **HTTP only**: Services run without SSL certificates
+- **No redirects**: Direct HTTP access without HTTPS enforcement
+- **Local development**: Suitable for localhost testing
+- **Recommended for**: Testing, development, local setups
+
+The environment is configured during the `setup_letsencrypt` step and affects all services automatically.
+
 ## Starting/Stopping/Status Services
 
 ```bash
@@ -107,6 +126,7 @@ docker compose exec -ti manager shtool stop
 docker compose exec -ti manager shtool start <service>
 docker compose exec -ti manager shtool stop <service>
 docker compose exec -ti manager shtool status
+docker compose exec -ti manager shtool reload
 ```
 
 ## Managing Services by Docker Compose wrapper
@@ -123,6 +143,64 @@ Examples:
 - Start a service: `docker compose run --rm manager shtool manage nextcloud up -d`
 - View logs: `docker compose run --rm manager shtool manage ghost logs -f`
 
+## Deployment Management
+
+### Deploy Individual Services
+
+Deploy or redeploy a specific service:
+```bash
+docker compose run --rm manager shtool deploy <service_name> [force_rebuild]
+docker compose run --rm manager shtool redeploy <service_name> [force_rebuild]
+```
+
+Examples:
+```bash
+# Deploy Nextcloud
+docker compose run --rm manager shtool deploy nextcloud
+
+# Redeploy with forced rebuild
+docker compose run --rm manager shtool redeploy nextcloud true
+```
+
+### Deploy All Services
+
+Redeploy all enabled services:
+```bash
+docker compose run --rm manager shtool redeploy_all [force_rebuild]
+```
+
+Example:
+```bash
+# Redeploy all services with forced rebuild
+docker compose run --rm manager shtool redeploy_all true
+```
+
+## Data Management and Cleanup
+
+### Clean All Data
+**⚠️ WARNING: This will permanently delete ALL data**
+
+```bash
+docker compose run --rm manager shtool clean
+```
+
+This command will:
+- Stop all services
+- Remove all Docker volumes
+- Require confirmation with a randomly generated code
+
+### Clean Individual Service Data
+**⚠️ WARNING: This will permanently delete data for the specified service**
+
+```bash
+docker compose run --rm manager shtool clean_app <service_name>
+```
+
+Example:
+```bash
+docker compose run --rm manager shtool clean_app nextcloud
+```
+
 ### Application Configuration
 
 Configure all applications at once:
@@ -227,6 +305,19 @@ The following services are in development or planned for future releases:
 - **SSL certificate problems**: Run `docker compose run --rm manager shtool setup_letsencrypt` again
 - **Incorrect credentials**: Reconfigure application settings with `docker compose run --rm manager shtool configure_app_settings`
 - **Configuration issues**: Check application-specific configuration files in the service directory
+- **Environment mismatch**: Ensure your environment (prod/dev) is correctly configured
+
+### Environment-Specific Issues
+
+**Production Environment:**
+- Verify domain DNS is pointing to your server
+- Ensure ports 80 and 443 are open and accessible
+- Check Let's Encrypt certificate generation in Traefik logs
+
+**Development Environment:**
+- Services will be accessible via HTTP only
+- Use `localhost` or your local IP address
+- No SSL certificates required
 
 ## Best Practices
 
@@ -235,6 +326,8 @@ The following services are in development or planned for future releases:
 3. Monitor system resources to ensure adequate capacity
 4. Implement proper security measures and network isolation
 5. Set up monitoring for critical services
+6. Use production environment for live deployments
+7. Test changes in development environment first
 
 ## Support and Community
 

apps/authentik/bin/app_hook.sh

@@ -13,5 +13,5 @@ update_env_record "PG_PASS" "$(openssl rand -hex 16)" "$ENV_FILE"
 update_env_record "AUTHENTIK_LISTEN__HTTP" "0.0.0.0:9000" "$ENV_FILE"
 update_env_record "AUTHENTIK_LISTEN__HTTPS" "0.0.0.0:9443" "$ENV_FILE"
 
-BASE_DOMAIN_VALUE=$(cat "$ROOT/etc/basedomain.txt")
+BASE_DOMAIN_VALUE=$(yq '.base_domain' "$SELFHOSTYOURTECH_ROOT/etc/config.yaml")
 add_key_in_env "BASE_DOMAIN" "$BASE_DOMAIN_VALUE" "$ENV_FILE"

apps/authentik/docker-compose.yml

@@ -52,28 +52,28 @@ services:
       - traefik-public
       - authentik-internal
     labels:
-      - traefik.enable=true
+      - traefik.enable=${ENABLE_TRAEFIK:-false}
       - traefik.docker.network=traefik-public
       # Define the service explicitly
       - traefik.http.services.authentik.loadbalancer.server.port=9000
       # HTTP router
       - traefik.http.routers.authentik.rule=Host(`authentik.${BASE_DOMAIN}`)
       - traefik.http.routers.authentik.entrypoints=web
-      - traefik.http.routers.authentik.middlewares=redirect-to-https
+      - "${HTTPS_MIDDLEWARE:+traefik.http.routers.authentik.middlewares=${HTTPS_MIDDLEWARE}}"
       # HTTPS router
       - traefik.http.routers.authentik-secure.rule=Host(`authentik.${BASE_DOMAIN}`)
-      - traefik.http.routers.authentik-secure.entrypoints=websecure
-      - traefik.http.routers.authentik-secure.tls=true
-      - traefik.http.routers.authentik-secure.tls.certresolver=letsencrypt
+      - traefik.http.routers.authentik-secure.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.authentik-secure.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.authentik-secure.tls.certresolver=${TLS_RESOLVER:-}
       - traefik.http.routers.authentik-secure.service=authentik
       # Outpost endpoints
       - traefik.http.routers.authentik-outpost.rule=HostRegexp(`{subdomain:[a-zA-Z0-9-]+}.${BASE_DOMAIN}`) && PathPrefix(`/outpost.goauthentik.io/`)
-      - traefik.http.routers.authentik-outpost.entrypoints=websecure
-      - traefik.http.routers.authentik-outpost.tls=true
-      - traefik.http.routers.authentik-outpost.tls.certresolver=letsencrypt
+      - traefik.http.routers.authentik-outpost.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.authentik-outpost.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.authentik-outpost.tls.certresolver=${TLS_RESOLVER:-}
       # HTTPS redirect middleware
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true
+      - "${REDIRECT_SCHEME:+traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=${REDIRECT_SCHEME}}"
+      - "${REDIRECT_PERMANENT:+traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=${REDIRECT_PERMANENT}}"
       # Forward auth configuration
       - traefik.http.middlewares.authentik.forwardauth.address=http://authentik-proxy:9000/outpost.goauthentik.io/auth/traefik
       - traefik.http.middlewares.authentik.forwardauth.trustForwardHeader=true
@@ -103,7 +103,7 @@ services:
       - authentik-internal
 
 networks:
-  traefik-public:
-    external: true
   authentik-internal:
-    internal: true
+  traefik-public:
+    external: ${ENABLE_TRAEFIK:-false}
+    name: ${TRAEFIK_NETWORK:-traefik-public}

apps/baserow/docker-compose.yml

@@ -2,7 +2,7 @@ services:
   baserow:
     image: baserow/baserow:1.30.1
     environment:
-      BASEROW_PUBLIC_URL: 'https://baserow.${BASE_DOMAIN}'
+      BASEROW_PUBLIC_URL: '${HTTP_SCHEME}://baserow.${BASE_DOMAIN}'
     env_file:
       - ./.env
     networks:
@@ -11,20 +11,21 @@ services:
     volumes:
       - ./data/baserow:/baserow/data
     labels:
-      - traefik.enable=true
+      - traefik.enable=${ENABLE_TRAEFIK:-false}
       - traefik.docker.network=traefik-public
       - traefik.http.routers.baserow.rule=Host(`baserow.${BASE_DOMAIN}`)
       - traefik.http.routers.baserow.entrypoints=web
-      - traefik.http.routers.baserow.middlewares=redirect-to-https
+      - "${HTTPS_MIDDLEWARE:+traefik.http.routers.baserow.middlewares=${HTTPS_MIDDLEWARE}}"
       - traefik.http.routers.baserow-secure.rule=Host(`baserow.${BASE_DOMAIN}`)
-      - traefik.http.routers.baserow-secure.entrypoints=websecure
-      - traefik.http.routers.baserow-secure.tls=true
-      - traefik.http.routers.baserow-secure.tls.certresolver=letsencrypt
+      - traefik.http.routers.baserow-secure.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.baserow-secure.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.baserow-secure.tls.certresolver=${TLS_RESOLVER:-}
       - traefik.http.services.baserow.loadbalancer.server.port=80
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true
+      - "${REDIRECT_SCHEME:+traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=${REDIRECT_SCHEME}}"
+      - "${REDIRECT_PERMANENT:+traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=${REDIRECT_PERMANENT}}"
 networks:
   internal:
     driver: bridge
   traefik-public:
-    external: true
+    external: ${ENABLE_TRAEFIK:-false}
+    name: ${TRAEFIK_NETWORK:-traefik-public}

apps/docs/docker-compose.yml

@@ -11,28 +11,29 @@ services:
     environment:
       - ALLOWED_ORIGINS=${BASE_DOMAIN},www.${BASE_DOMAIN}
     labels:
-      - traefik.enable=true
+      - traefik.enable=${ENABLE_TRAEFIK:-false}
       - traefik.docker.network=traefik-public
       - traefik.constraint-label=traefik-public
       # Main domain route
       - traefik.http.routers.docs-base.rule=Host(`${BASE_DOMAIN}`)
       - traefik.http.routers.docs-base.entrypoints=web
-      - traefik.http.routers.docs-base.middlewares=redirect-to-https
+      - "${HTTPS_MIDDLEWARE:+traefik.http.routers.docs-base.middlewares=${HTTPS_MIDDLEWARE}}"
       - traefik.http.routers.docs-base-secure.rule=Host(`${BASE_DOMAIN}`)
-      - traefik.http.routers.docs-base-secure.entrypoints=websecure
-      - traefik.http.routers.docs-base-secure.tls=true
-      - traefik.http.routers.docs-base-secure.tls.certresolver=letsencrypt
+      - traefik.http.routers.docs-base-secure.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.docs-base-secure.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.docs-base-secure.tls.certresolver=${TLS_RESOLVER:-}
       - traefik.http.routers.docs.rule=Host(`www.${BASE_DOMAIN}`)
       - traefik.http.routers.docs.entrypoints=web
-      - traefik.http.routers.docs.middlewares=redirect-to-https
+      - "${HTTPS_MIDDLEWARE:+traefik.http.routers.docs.middlewares=${HTTPS_MIDDLEWARE}}"
       - traefik.http.routers.docs-secure.rule=Host(`www.${BASE_DOMAIN}`)
-      - traefik.http.routers.docs-secure.entrypoints=websecure
-      - traefik.http.routers.docs-secure.tls=true
-      - traefik.http.routers.docs-secure.tls.certresolver=letsencrypt
+      - traefik.http.routers.docs-secure.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.docs-secure.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.docs-secure.tls.certresolver=${TLS_RESOLVER:-}
       - traefik.http.services.docs.loadbalancer.server.port=3000
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true
+      - "${REDIRECT_SCHEME:+traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=${REDIRECT_SCHEME}}"
+      - "${REDIRECT_PERMANENT:+traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=${REDIRECT_PERMANENT}}"
 
 networks:
   traefik-public:
-    external: true
+    external: ${ENABLE_TRAEFIK:-false}
+    name: ${TRAEFIK_NETWORK:-traefik-public}

apps/ghost/bin/app_hook.sh

@@ -11,5 +11,5 @@ update_env_record "MYSQL_USER" "ghost" "$ENV_FILE"
 update_env_record "MYSQL_ROOT_PASSWORD" "$(openssl rand -hex 32)" "$ENV_FILE"
 update_env_record "MYSQL_PASSWORD" "$(openssl rand -hex 32)" "$ENV_FILE"
 
-BASE_DOMAIN_VALUE=$(cat "$SELFHOSTYOURTECH_ROOT/etc/basedomain.txt")
+BASE_DOMAIN_VALUE=$(yq '.base_domain' "$SELFHOSTYOURTECH_ROOT/etc/config.yaml")
 add_key_in_env "BASE_DOMAIN" "$BASE_DOMAIN_VALUE" "$ENV_FILE"

apps/ghost/docker-compose.yml

@@ -4,8 +4,8 @@ services:
     depends_on:
       - mysql
     networks:
-      - ghost-internal  # Add internal network for DB communication
       - traefik-public  # Keep traefik network for external access
+      - internal
     environment:
       # see https://ghost.org/docs/config/#configuration-options
       database__client: mysql
@@ -14,36 +14,36 @@ services:
       database__connection__password: "${MYSQL_PASSWORD}"
       database__connection__database: "${MYSQL_DATABASE}"
       # this url value is just an example, and is likely wrong for your environment!
-      url: "https://ghost.${BASE_DOMAIN}"  # Update to use HTTPS
+      url: "${HTTP_SCHEME}://ghost.${BASE_DOMAIN}"  # Update to use HTTPS
       # contrary to the default mentioned in the linked documentation, this image defaults to NODE_ENV=production (so development mode needs to be explicitly specified if desired)
       NODE_ENV: production
       privacy__useUpdateCheck: false
       # This tells Ghost to trust the X-Forwarded-Proto header from Traefik
-      url_base: "https://ghost.${BASE_DOMAIN}"
+      url_base: "${HTTP_SCHEME}://ghost.${BASE_DOMAIN}"
       server__host: "0.0.0.0"
       server__port: 2368
       trust_proxy: true
     volumes:
       - ./data/ghost:/var/lib/ghost/content
     labels:
-      - traefik.enable=true
+      - traefik.enable=${ENABLE_TRAEFIK:-false}
       - traefik.docker.network=traefik-public
       - traefik.http.services.ghost-service.loadbalancer.server.port=2368
       - traefik.http.routers.ghost.rule=Host(`ghost.${BASE_DOMAIN}`)
       - traefik.http.routers.ghost.entrypoints=web
-      - traefik.http.routers.ghost.middlewares=redirect-to-https
+      - "${HTTPS_MIDDLEWARE:+traefik.http.routers.ghost.middlewares=${HTTPS_MIDDLEWARE}}"
       - traefik.http.routers.ghost-secure.rule=Host(`ghost.${BASE_DOMAIN}`)
-      - traefik.http.routers.ghost-secure.entrypoints=websecure
-      - traefik.http.routers.ghost-secure.tls=true
-      - traefik.http.routers.ghost-secure.tls.certresolver=letsencrypt
+      - traefik.http.routers.ghost-secure.entrypoints=${HTTPS_ENTRYPOINT:-}
+      - traefik.http.routers.ghost-secure.tls=${ENABLE_TLS:-}"
+      - traefik.http.routers.ghost-secure.tls.certresolver=${TLS_RESOLVER:-}
       - traefik.http.routers.ghost-secure.service=ghost-service
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
-      - traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=true
+      - "${REDIRECT_SCHEME:+traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=${REDIRECT_SCHEME}}"
+      - "${REDIRECT_PERMANENT:+traefik.http.middlewares.redirect-to-https.redirectscheme.permanent=${REDIRECT_PERMANENT}}"
 
   mysql:
     image: mysql:8.0
     networks:
-      - ghost-internal  # Only internal network needed for DB
+      - internal  # Only internal network needed for DB
     environment:
       - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
       - MYSQL_DATABASE=${MYSQL_DATABASE}
@@ -53,7 +53,7 @@ services:
       - ./data/mysql:/var/lib/mysql
 
 networks:
-  ghost-internal:
-    internal: true  # This makes it only accessible within the stack
+  internal:
   traefik-public:
-    external: true
+    external: ${ENABLE_TRAEFIK:-false}
+    name: ${TRAEFIK_NETWORK:-traefik-public}

apps/hedgedoc/bin/app_hook.sh

@@ -11,5 +11,5 @@ update_env_record "POSTGRES_DB" "hedgedoc" "$ENV_FILE"
 update_env_record "POSTGRES_PASSWORD" "$(openssl rand -hex 16)" "$ENV_FILE"
 update_env_record "HD_SESSION_SECRET" "$(openssl rand -hex 32)" "$ENV_FILE"
 
-BASE_DOMAIN_VALUE=$(cat "$ROOT/etc/basedomain.txt")
+BASE_DOMAIN_VALUE=$(yq '.base_domain' "$SELFHOSTYOURTECH_ROOT/etc/config.yaml")
 add_key_in_env "BASE_DOMAIN" "$BASE_DOMAIN_VALUE" "$ENV_FILE"