You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Product guide updates including added section outlining volume mount … (#16)
* Product guide updates including added section outlining volume mount usage as a debug tool, updates related to podman transition, and removal of the nginx container. Revision history list has also been updated.
* Fixed cleanup command
* Addressed comments and a few other identified tweaks.
@@ -155,7 +155,7 @@ Although some of the subsystems have a preferred start-up order (to ensure initi
155
155
156
156
=== ANMS Components
157
157
158
-
The subsystems of the ANMS (all containers) are illustrated as gray blocks within the "ANMS Instance" group in the diagram of <<fig-anms-components-protocol>>.
158
+
The subsystems (containers) of the ANMS are illustrated as gray blocks within the "ANMS Instance" group in the diagram of <<fig-anms-components-protocol>>.
159
159
The entire ANMS instance is made to be run on a single host, with future plans to allow installing in a more distributed environment.
160
160
Currently the ANMS provides security at the boundary of the instance but not between comtainers (see <<sec-security>> for details), which would be required for a distributed installation.
161
161
@@ -197,6 +197,10 @@ Specific procedures for performing an installation using a local Puppet apply ar
197
197
Conditions for installing the ANMS are a host with packages identified in <<target-host-packages>>, at least 7{nbsp}GiB of filesystem space for container image storage, and additional space for long-term data warehouse storage.
198
198
The total amount of storage needed depends on the mission use of reports, specifically the average size and rate of reported data.
199
199
200
+
NOTE: Usage of podman is recommended. Docker should continue to function as a drop-in replacement, however only podman deployments will be directly supported by the ANMS team.
201
+
202
+
NOTE: It is recommended to use docker-compose (which is fully compatible with podman). docker-compose is a distinct package from Docker, while podman-compose is a fully compatible alternative. Older versions of compose were called directly (ie: `docker-compose`), while current versions are invoked as a subcommand (ie: `docker compose` or `podman compose`).
203
+
200
204
[#target-host-packages]
201
205
.Target host packages
202
206
[%header,width=75%,cols=2*]
@@ -205,10 +209,10 @@ The total amount of storage needed depends on the mission use of reports, specif
205
209
|Version Minimum
206
210
207
211
|podman
208
-
|5.2
212
+
|5.2+
209
213
210
-
|podman-compose
211
-
|1.0
214
+
|docker-compose (preferred) OR podman-compose.
215
+
|2.29+ OR 5.3+
212
216
213
217
|systemd
214
218
|252
@@ -228,8 +232,9 @@ In any case, deployment and configuration of CAM itself is outside the scope of
228
232
[#sec-deployment-camemu]
229
233
==== Using a CAM Gateway Emulator
230
234
231
-
To allow the ANMS to be tested in environments where a CAM Server is unavailable or too burdensome to set up, the ANMS can be built with an emulator of the CAM Gateway which uses static accounts, credentials, and access controls.
232
-
The environment `AUTHNZ_EMU=demo` during a build (see <<sec-proc-build>>) enables the CAM Gateway emulator behavior.
235
+
To allow the ANMS to be tested in environments where a CAM Server is unavailable or too burdensome to set up, the ANMS can be built with an emulator of the CAM Gateway which uses static accounts, credentials, and access controls. This is now known as the "demo" configuration.
236
+
237
+
Auth mode is determined by the environment variable `AUTHNZ_EMU` during build (see <<sec-proc-build>>). Set `AUTHNZ_EMU=cam-gateway` to utilize the CAM Server, or `AUTHNZ_EMU=demo` to utilize the CAM Gateway emulation (a basic authentication demo configuration).
233
238
234
239
CAUTION: The CAM Gateway emulator is for demonstration only and must not be present in a production installation.
235
240
@@ -254,6 +259,7 @@ The CAM Gateway reverse proxy for authentication, authorization, and auditing \(
254
259
This container uses the external `ammos-tls` volume for TLS configuration (see <<sec-proc-install-tls>>).
255
260
This provides HTTP routing access is to `anms-ui`, `anms-core`, and `grafana` containers.
256
261
Exposes TCP port 443 for HTTPS and 80 for HTTP, both mapped to the same host port numbers.
262
+
This container can be remapped to use the cam-gateway or demo (basic http auth) configuration.
257
263
`anms-core`::
258
264
The ANMS backend REST services.
259
265
Exposes TCP port 5555 for HTTP.
@@ -295,7 +301,7 @@ An intermediate service to bookkeep transcoding requests from the ANMS to the AR
295
301
[#sec-host-files]
296
302
=== Filesystem
297
303
298
-
Because the ANMS is deployed as a Podman Compose configuration, the only primary files present on the host are to configure a compose project, its use as a system service, and the system firewall.
304
+
Because the ANMS is deployed as a Compose configuration, the only primary files present on the host are to configure the containerized project, its use as a system service, and the system firewall.
299
305
300
306
The principal directories and files used by ANMS are:
301
307
@@ -458,6 +464,8 @@ This chapter includes specific procedures related to managing an ANMS production
458
464
459
465
The ANMS source is composed of a top-level repository `anms` and a number of submodule repositories; all of them are required for building the ANMS.
460
466
467
+
Before beginning, ensure that either Docker or Podman (preferred) is installed and functional on your system along with docker-compose. A hello-world image can be run to verify functionality, ie: `docker run --rm hello-world` or `podman run --rm hello-world`. Compose can be verified with `podman compose ps` - If compose is not properly installed it will report 'unrecognized command'
. The container image building is then executed with:
480
+
. If running rootful containers (not recommended. Podman defaults to rootless), it may be necessary to add the local user to the `docker` access group with the following. :
473
481
+
474
482
----
475
-
export DOCKER_IMAGE_PREFIX=<REPOURL>
476
-
export DOCKER_IMAGE_TAG=latest
477
-
./build.sh buildonly
483
+
sudo usermod -a -G docker ${USER}
478
484
----
485
+
. Update `.env` file as needed. Fields that may need to be updated include `DOCKER_IMAGE_PREFIX` and for rootless podman the port mappings for AUTHNZ may need to be uncommented to avoid permissions issues
486
+
. Build images using either: (Note: The podman-build-args flag is required for healthchecks to function under podman).
which by default uses the current top-level branch name as the tag for all container images.
481
494
482
495
[NOTE]
483
496
====
484
-
To build an ANMS that uses an emulator for the CAM Gateway (which means that the ANMS will not require a CAM server), have the following environment set in the build step above:
497
+
To build an ANMS that uses an emulator for the CAM Gateway (which means that the ANMS will not require a CAM server), have the following environment set in the build step above. (Note: The default value that uses the CAM Gateway is `cam-gateay`):
485
498
----
486
499
export AUTHNZ_EMU=demo
487
500
----
@@ -546,12 +559,13 @@ The PEM-encoded CA bundle containing at least the CA chain used to sign the serv
546
559
547
560
=== Upgrading
548
561
549
-
Because the ANMS is deployed as a Docker Compose configuration with associated environment variables and container images, an upgrade involves updating the compose configuration and restarting affected containers.
562
+
Because the ANMS is deployed as a series of containers managed by compose with associated environment variables and configuration, an upgrade involves rebuilding and restarting affected containers.
550
563
551
564
An upgrade can be performed using the same procedure as <<sec-proc-install>>, where Puppet will make any required changes for the upgrade and restart services and containers as necessary.
552
565
Individual ANMS releases may identify pre-upgrade or post-upgrade steps in their specific Release Description Document (RDD).
553
566
554
-
=== Resetting Docker State
567
+
[#sec-reset-system]
568
+
=== Resetting Container State
555
569
556
570
[WARNING]
557
571
The following will reset all database state, including user profiles, ADM configuration, and all historical report data.
@@ -560,9 +574,21 @@ This should only be used for test hosts or after performing a full Postgres DB b
560
574
To force containers and volumes (containing long-term database files) to be cleared, a maintainer can run the following from the host.
561
575
562
576
```
563
-
docker stop $(docker ps -q); docker rm $(docker ps -a -q); docker volume rm $(docker volume ls -q)
577
+
podman stop --all
578
+
podman system prune --all --volumes
564
579
```
565
580
581
+
To verify artifacts have been removed you may use the following. The prune command (or object-specific prune command) may need to be reissued if not all relevant artifacts have been cleared.
582
+
583
+
```
584
+
podman image ls
585
+
podman container ls
586
+
podman volume ls
587
+
podman network ls
588
+
```
589
+
590
+
TIP: In some cases, issues may arise due to system cache files on the host system not being cleared (a potential issue with select podman versions). It is recommended to restart the host system after clearing objects to ensure a clean start.
591
+
566
592
After clearing containers and volumes, the normal `apply_local` step of <<sec-proc-install>> should be performed to re-install and start the containers.
After changing these permissions, run `docker-compose up grafana` again, and the Grafana
836
+
After changing these permissions, run `podman compose up grafana` again, and the Grafana
810
837
container should start sucessfully.
811
838
812
839
===== Agent Registration Issues on ANMS Startup
813
840
814
841
If an Agent is not present in the `Agents` tab on start up, it is likely due to an error in one of the ION containers and their connection to the underlying database.
815
842
816
-
To resolve the issue, restart the ION containers using `docker-compose restart n1 n2`.
843
+
To resolve the issue, restart the ION containers using `podman compose restart n1 n2`.
817
844
818
845
===== New Agent Registration Issues
819
846
@@ -846,8 +873,7 @@ With ANMS running, go to `localhost:8080` and log in to the database with:
846
873
===== ANMS-UI is not visible at hostname:9030
847
874
848
875
This error may indicate that the anms-ui docker is experiencing issues receiving HTTP requests.
849
-
This is most likely related to the `host` or `bind address` specified in `anms-ui/server/shared/config.py`,
850
-
or an environment variable that overrides this.
876
+
This is most likely related to the `host` or `bind address` specified in `anms-ui/server/shared/config.py`, an environment variable that overrides this, or a firewall issue.
851
877
852
878
===== ANMS-UI is not visible at hostname
853
879
@@ -858,6 +884,74 @@ Check the status of the `authnz` container in the compose services list.
858
884
It may be necessary to restart the container using:
859
885
`sudo podman compose -p anms restart authnz`.
860
886
887
+
Port numbers can be overridden through environment variables (see .env file). Check that ports are mapped to the expected ports and are not being blocked by your system's firewall (if applicable).
888
+
889
+
When running rootless podman the container may fail to start if the user does not have permission to bind on the configured port(s). Users typically cannot bind to low numbered ports, including 80 (http) and 443 (https). If this is the issue, try to set AUTHNZ_PORT and AUTHNZ_HTTPS_PORT to higher values and test at the specified port (ie: In .env set AUTHNZ_PORT=9080 and test at http://hostname:9080).
890
+
891
+
Check that the authnz container is running with `podman compose ps`.
892
+
893
+
Logs can be viewed with `podman compose logs authnz` to identify potential issues.
894
+
895
+
896
+
===== Other container startup issues
897
+
898
+
Following an upgrade or failed installation step, it is possible for the system to be in an inconsistent state resulting in containers failing to start, initialize properly, or using outdated caches.
899
+
900
+
As a first debug step, it is recmmended to fully restart the host system and all containers. If that fails to resolve the issue, backup any existing data and proceed with a full reset of your installation files. See <<sec-reset-system>> for details and ensure you restart the host system prior to rebuilding.
901
+
902
+
==== Advanced
903
+
904
+
[#sec-volume-mount-tip]
905
+
===== Override Configuration Files with Volume Mounts
906
+
907
+
It is sometimes necessary to override configuration files or scripts built into the container environment. Volume mounts can be used override select files or directories without rebuilding containers.
908
+
909
+
For example, to override the ION NM Manager configuration file add the following to the `volumes section of `ion-manager` in `docker-compose.yml`. Adjust the source file to any relative or absolute path desired. For this change to take effect, you must restart all containers with `podman compose down` and then restart with `podman compose up`. Simply restarting the affected contaner may not refresh changes made to the compose file.
910
+
911
+
```
912
+
- ./ion/configs/simple-2-node/mgr.rc:/etc/ion.rc
913
+
```
914
+
915
+
This approach can also be used to override sample agent configurations (see agent-compose.yml), extend authentication settings in the selected auth container, or for developers to quickly test changes.
916
+
917
+
===== General Networking Issues
918
+
919
+
If services cannot be accessed from remote machines, verify that it is not being blocked by your system's firewall.
920
+
921
+
In rare cases, it may be useful to utilize Wireshark or Tshark to verify network traffic within the containers, particularly if debugging NM agents within the container network. The following is a quick guide to installing and running these tools within the container if you do not have the ability to view activity from the host system. This example is for the ion-manager container, but can be adapted to any.
922
+
923
+
To temporarily install in the ion-manager container run `podman compose exec ion-manager yum install tshark`. Alternatively, modify the ion/Dockerfile to permanently add it to your installation and rebuild. It can then be run with `podman compose exec ion-manager tshark -i any`, with tshark arguments modified as needed.
924
+
925
+
To use the GUI wireshark, install with `podman compose exec ion-manager yum install wireshark`. To easily run, create a file `novnc-compose.yml` with the definition below. Start it with `podman compose -f novnc-compose.yml up`. From the shell run `podman compose exec ion-manager bash` followed by `DISPLAY=novnc:0.0 wireshark` to launch wireshark in the VNC session acessible from http://hostname:9081
926
+
927
+
```
928
+
# noVNC provides browser access to network to aide debugging.
929
+
# This container is NOT intended for production usage.
930
+
931
+
networks:
932
+
# This network is created by docker-compose.yml
933
+
default:
934
+
name: ${DOCKER_CTR_PREFIX}anms
935
+
external: true
936
+
937
+
services:
938
+
novnc:
939
+
image: chrome-novnc
940
+
build:
941
+
dockerfile: novnc.Dockerfile
942
+
environment:
943
+
# Adjust to your screen size
944
+
- DISPLAY_WIDTH=1920
945
+
- DISPLAY_HEIGHT=1080
946
+
- RUN_XTERM=yes
947
+
ports:
948
+
- "9081:8080"
949
+
```
950
+
951
+
See <<sec-volume-mount-tip>> for an example of volume mounting a local directory for easier saving of captures for later analysis. In this case, you may wish to mount a directory instead of a single file, such as `- ./logs:/logs` and append to the tshark command `-w /logs/log.pcap`
0 commit comments