Skip to content

Commit b942241

Browse files
BrunoCoimbramambelli
BrunoCoimbra
authored and
mambelli
committed
Add installation instructions for CentOS 8
1 parent 4f6fc13 commit b942241

2 files changed

Lines changed: 263 additions & 0 deletions

File tree

‎doc/source/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ Decision Engine using the distributed RPM packages.
3131
:maxdepth: 1
3232

3333
install
34+
install_el8
3435

3536

3637
Developer Documentation

‎doc/source/install_el8.rst‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
.. SPDX-FileCopyrightText: 2017 Fermi Research Alliance, LLC
2+
.. SPDX-License-Identifier: Apache-2.0
3+
4+
Installing and running HEPCloud's Decision Engine on EL8
5+
========================================================
6+
7+
Decision engine uses a PostgreSQL database back-end and Redis as message broker and cache.
8+
9+
You need to install first PostgreSQL, Redis, and then the Decision engine framework (decisionengine) and install and add the standard channels (decisionengine_modules).
10+
11+
The following instructions assume a system installation, performed as ``root``.
12+
decisionengine will run as the decisionengine user.
13+
14+
15+
Install PostgreSQL
16+
------------------
17+
18+
The default postgresql installed on RH8 is 9.2 which is outdated. Suggest to remove it and install 12 instead :
19+
20+
1. Disable the built-in PostgreSQL module ::
21+
22+
sudo dnf -qy module disable postgresql
23+
24+
2. Install postgresql 12 ::
25+
26+
dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
27+
dnf install -y postgresql12 postgresql12-server
28+
# optional, also: postgresql12-devel
29+
30+
3. Enable postgresql ::
31+
32+
systemctl enable postgresql-12
33+
34+
4. Init the database ::
35+
36+
/usr/pgsql-12/bin/postgresql-12-setup initdb
37+
38+
5. edit ``/var/lib/pgsql/12/data/pg_hba.conf`` like the following::
39+
40+
[root@fermicloud371 ~]# diff /var/lib/pgsql/12/data/pg_hba.conf~ /var/lib/pgsql/12/data/pg_hba.conf
41+
80c80
42+
< local all all peer
43+
---
44+
> local all all trust
45+
82c82
46+
< host all all 127.0.0.1/32 ident
47+
---
48+
> host all all 127.0.0.1/32 trust
49+
84c84
50+
< host all all ::1/128 ident
51+
---
52+
> host all all ::1/128 trust
53+
54+
55+
This is setting the authentication method to `trust`
56+
57+
6. start the database ::
58+
59+
systemctl start postgresql-12
60+
61+
7. create decisionengine ::
62+
63+
createdb -U postgres decisionengine
64+
65+
The schema and the connection will be created and configured during the Decision engine framework installation.
66+
67+
To use the database you have to add it to the environment::
68+
69+
export PG_VERSION=12
70+
export PATH="/usr/pgsql-${PG_VERSION}/bin:~/.local/bin:$PATH"
71+
# you may also add these lines to ~/.bashrc
72+
73+
74+
Install Redis
75+
-------------
76+
77+
Install and start the message broker (Redis) container on your system. You can find more details on the :doc:`redis document <redis>`
78+
79+
1. Install Padman ::
80+
81+
dnf install -y podman
82+
83+
2. Run the Redis container ::
84+
85+
podman run --name decisionengine-redis -p 127.0.0.1:6379:6379 -d redis:6 --loglevel warning
86+
# When prompted to select an image, pick "docker.io/library/redis:6".
87+
88+
89+
Install Decision Engine and the standard modules
90+
------------------------------------------------
91+
92+
1. Prerequisites setup. Make sure that the required packages (python39, gcc, ...) are installed and up to date. ::
93+
94+
# gcc, swig and make are needed for dependencies (jsonnet)
95+
dnf install python39 python39-pip python39-setuptools python39-wheel \
96+
gcc gcc-c++ make \
97+
python39-devel swig openssl-devel git rpm-build
98+
python3.9 -m pip install --upgrade --user pip
99+
python3.9 -m pip install --upgrade --user setuptools wheel setuptools-scm[toml]
100+
101+
# To install the modules you will also need GlideinWMS Frontend, which is in the OSG repository.
102+
# Assuming the use of OSG 3.6, here is a brief summary of the setup:
103+
dnf install -y https://repo.opensciencegrid.org/osg/3.6/osg-3.6-el8-release-latest.rpm
104+
# HTCondor 8.9.x or 9.x, required by GlideinWMS, is in the osg-upcoming repository. It should be enabled to find the dependency
105+
# GlideinWMS 3.9.x is in osg-contrib. The repository should be enabled to find the dependency
106+
# In both the following files set: enabled=1
107+
vi /etc/yum.repos.d/osg-upcoming.repo
108+
vi /etc/yum.repos.d/osg-contrib.repo
109+
# Change the Epel repository priority to make sure that comes after the OSG repositories, which are 98. Make sure that epel has:
110+
priority=99
111+
vi /etc/yum.repos.d/epel.repo
112+
113+
The complete version of the GlideinWMS installation instructions is available `here<https://opensciencegrid.org/docs/other/install-gwms-frontend/>`.
114+
For a minimal installation, you can use the following command:
115+
116+
dnf install glideinwms-vofrontend-libs glideinwms-vofrontend-glidein glideinwms-userschedd glideinwms-usercollector
117+
118+
2. Setup the decision engine user and git repositories ::
119+
120+
useradd decisionengine
121+
sudo -u decisionengine git clone https://github.com/HEPCloud/decisionengine.git ~decisionengine/decisionengine
122+
sudo -u decisionengine git clone https://github.com/HEPCloud/decisionengine_modules.git ~decisionengine/decisionengine_modules
123+
124+
3. Install the decision engine from the git repositories ::
125+
126+
# Install the decisionengine framework and modules using setuptools
127+
su - decisionengine
128+
pushd decisionengine
129+
python3.9 setup.py develop --user
130+
popd
131+
pushd decisionengine_modules
132+
python3.9 setup.py develop --user
133+
popd
134+
exit
135+
136+
# Create the required system files and directories (as root)
137+
mkdir /etc/decisionengine
138+
mkdir /var/log/decisionengine/
139+
cp ~decisionengine/decisionengine/config/decision_engine.jsonnet /etc/decisionengine
140+
cp -r ~decisionengine/decisionengine/src/decisionengine/framework/tests/etc/decisionengine/config.d /etc/decisionengine
141+
chown -R decisionengine:decisionengine /etc/decisionengine
142+
chown -R decisionengine:decisionengine /var/log/decisionengine
143+
144+
Now you can type ``decisionengine --help`` while logged in as decisionengine to print the help message.
145+
To do more you need first to configure Decision Engine.
146+
147+
148+
Configure Decision Engine
149+
-------------------------
150+
151+
The default configuration file lives in ``/etc/decisionengine/decision_engine.jsonnet``.
152+
153+
A number of defaults are set for you.
154+
155+
Selecting your datasource
156+
~~~~~~~~~~~~~~~~~~~~~~~~~
157+
158+
You need a datasource to store in the database the channel's data (datablocks).
159+
Each datasource has its own unique schema and cannot be used with a different datasource.
160+
161+
**The SQLAlchemy Data Source**
162+
163+
SQLAlchemy is the default Data Source after v1.7 and is setup with a configuration like::
164+
165+
"datasource": {
166+
"module": "decisionengine.framework.dataspace.datasources.sqlalchemy_ds",
167+
"name": "SQLAlchemyDS",
168+
"config": {
169+
"url": "postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_dbname}",
170+
}
171+
}
172+
173+
Any extra keywords you can pass to the ``sqlalchemy.engine.Engine`` constructor may be set under ``config``.
174+
175+
SQLAlchemy will create any tablespace objects it requires automatically.
176+
177+
178+
**The PostgreSQL Data Source**
179+
180+
The postgresql Data Source is the only one supported pre v1.7 and is setup with a config like::
181+
182+
"datasource": {
183+
"module": "decisionengine.framework.dataspace.datasources.postgresql",
184+
"name": "Postgresql",
185+
"config": {
186+
"user": "postgres",
187+
"blocking": true,
188+
"host": "localhost",
189+
"port": 5432,
190+
"database": "decisionengine",
191+
"maxconnections": 100,
192+
"maxcached": 10
193+
}
194+
}
195+
196+
If you use this datasource you must also load the database schema by hand.
197+
To load the database schema run::
198+
199+
psql -U postgres decisionengine -f /usr/share/doc/decisionengine/datasources/postgresql.sql
200+
201+
202+
Start decision engine
203+
---------------------
204+
205+
Start the service ::
206+
207+
# As decisionengine user
208+
decisionengine --no-webserver &
209+
210+
211+
Add channels to decision engine
212+
-------------------------------
213+
214+
Decision engine decision cycles happen in channels.
215+
You can add channels by adding configuration files in ``/etc/decisionengine/config.d/``
216+
and restarting the decision engine.
217+
218+
Here is a simple test channel configuration.
219+
This test channel is using some NOP classes currently defined in the unit tests and not distributed.
220+
221+
The following configuration has been added as an example to ``/etc/decisionengine/config.d/test_channel.jsonnet`` during the installation process::
222+
223+
{
224+
sources: {
225+
source1: {
226+
module: "decisionengine.framework.tests.SourceNOP",
227+
parameters: {},
228+
schedule: 1,
229+
}
230+
},
231+
transforms: {
232+
transform1: {
233+
module: "decisionengine.framework.tests.TransformNOP",
234+
parameters: {},
235+
schedule: 1
236+
}
237+
},
238+
logicengines: {
239+
le1: {
240+
module: "decisionengine.framework.logicengine.LogicEngine",
241+
parameters: {
242+
facts: {
243+
pass_all: "True"
244+
},
245+
rules: {
246+
r1: {
247+
expression: 'pass_all',
248+
actions: ['publisher1']
249+
}
250+
}
251+
}
252+
}
253+
},
254+
publishers: {
255+
publisher1: {
256+
module: "decisionengine.framework.tests.PublisherNOP",
257+
parameters: {}
258+
}
259+
}
260+
}
261+
262+
Once the decisionengine is running, ``de-client --status`` should show the active test channel.

0 commit comments

Comments
 (0)