Merge pull request #58 from KnowWhereGraph/deployment-fixes

Deployment fixes

Author: ThomasThelen

.gitignore

@@ -1,12 +1,13 @@
+nginx/logs/
 .DS_STORE
-nginx/logs
 graphdb/license/
-nginx/local-certs/
+nginx/letsencrypt/local-certs/*
 nginx/sites/faceted-search/
 nginx/sites/kw-panels/
 nginx/sites/node-browser/
+kwg-api/
 services/
 graphdb/nginx/*
 grafana/persistent_config
 *drawio.bkp
-*.png.bkp
+*.png.bkp

architecture/architecture.drawio

@@ -2,13 +2,11 @@
 
 Text search & ranking for KnowWhereGraph
 
-## Deploying
-
-The docker-compose file is used for deploying the service. Before deploying, set the password in the docker-compose file.
-
 ## Credentials
 
-```
+Credentials need to be set for Elasticsearch before the stack is deployed. Set this in the docker-compose.yaml file.
+
+```text
 user: elastic
 pass: <refer to docker-compose.yaml>
 ```

elasticsearch/README.md

@@ -37,3 +37,15 @@ The dashboards are taken from pre-designed packages, which are available on the
 - [Node Exporter](https://grafana.com/grafana/dashboards/1860-node-exporter-full/)
 - [NGINX](https://grafana.com/grafana/dashboards/14900-nginx/)
 - KWG-API: Custom
+
+## Data Sources
+
+Grafana makes use of two data sources
+
+### Prometheus
+
+Prometheus stores the metrics data and requires the prometheus user/pass to connect. This is set in the datasource yaml file, and the credentials should match the prometheus docker-compose credentials.
+
+### Loki
+
+Loki is the log data source. The communication with this service is unauthenticated.

grafana/README.md

@@ -11,16 +11,10 @@ There are *six* docker-compose files here. The two main flavors are
 1. Preloading: These compose files are used to the first upload of data. There are three (local/stage/prod)
 2. Running: These compose files are used when running GraphDB to serve content. There are three (local/stage/prod)
 
-
-
 ## Data Persistence
 
 Data is persisted on the host machine, _not_ the container. This is achieved by a volume mount between the host and GraphDB's repository data directory which is set in the docker-compose file. Graph DB stores its repository, configuration, and logging data under `/opt/graphdb/home`. This path can be mounted to the local system, persisting the data. When a new container is launched, it will reference the persisted data and load it.
 
-## Deploying
-
-GraphDB deployments should be managed by the repositories root Makefile. Run `make help` for a description of commands and follow the documentation below to learn more about loading data & deploying.
-
 ### Initial Data Load
 
 GraphDB's initial database is constructed using the `importrdf` tool from Ontotext. This runs with GraphDB offline and offers much faster data loading than other options. In this process, GraphDB creates a new repository and inserts data into it. To account for this, separate docker-compose files are needed to manage the offline instances.

graphdb/README.md

@@ -2,7 +2,7 @@ version: "3"
 
 services:
   loki:
-    image: grafana/loki:3.2.2
+    image: grafana/loki:2.9.11
     container_name: loki
     ports:
       - "3100:3100"

loki/docker-compose.yaml

@@ -34,6 +34,7 @@ repository-setup: # Fetches the GitHub repos needed & configures them
 	git clone https://github.com/KnowWhereGraph/kwg-faceted-search.git nginx/sites/kwg-faceted-search
 	git clone https://github.com/KnowWhereGraph/kwg-api.git kwg-api/
 	git clone https://github.com/KnowWhereGraph/kwg-ontologies.git nginx/sites/onto/
-	cp nginx/robots.txt nginx/sites/robots.txt
+	mkdir nginx/sites/robots/
+	cp nginx/robots.txt nginx/sites/robots/robots.txt
 validate-deployment: # Checks the local deployment files
-	sh validate.sh
+	sh scripts/validate.sh

makefile

@@ -8,17 +8,19 @@ KnowWhereGraph has a handful of services that require networking capabilities. T
 
 The NGINX configuration files are templated with environment variables, which can be customized in the docker-compose file. For more information on how the configuration files are generated from the templates refer to [this](https://github.com/docker-library/docs/tree/master/nginx#using-environment-variables-in-nginx-configuration-new-in-119) documentation page.
 
-## Deploying
+## Certificates
 
-Deploying nginx without using the make command is *not* recommended and most likely not necessary. If you know what you're doing and need to, you can with
+Certificates are mounted into the nginx container from the local filesystem. Non-local deployments work the same way: the `/etc/letsencrypt` files are mounted into the container.
 
-`docker-compose up`
+### Staging
 
-To bring the service down, run
+New certs can be generated with
 
-`docker-compose down`
+`sudo certbot certonly --webroot --webroot-path nginx/data/certbot/www/  -d staging.knowwheregraph.org`
 
-## Certificates
+ A common problem is that the certs may have different permissions for mounting into the nginx container.
+
+### Local
 
 Certificates for local development need to be manually generated and added to the `local-certs` directory.
 
@@ -39,7 +41,7 @@ https://javorszky.co.uk/2019/11/06/get-firefox-to-trust-your-self-signed-certifi
 
 ## Logging
 
-The NGINX logs are found in the container's /var/logs/nginx, which is mounted locally at `./nginx/logs`. For more verbose logging, refer to the NGINX Docker image documentation and modify the deployment script to include any additional flags.
+The NGINX logs should be checked through grafana or by inspecting the container's logs.
 
 ## Metrics
 

nginx/README.md

@@ -9,7 +9,6 @@ services:
       - 'variables.env'
     volumes:
     - ./nginx/templates:/etc/nginx/templates # NGINX config file
-    - ./nginx/logs:/var/log/nginx # NGINX logs
     - ./nginx/sites:/usr/share/nginx/html # Static sites
     - ./nginx/local-certs:/etc/letsencrypt/live/localhost # Local certificates
     depends_on: # These services need to exist for the nginx config to validate