Skip to content

Commit 01a426e

Browse files
pradeebanpradeeban
authored
Merge branch 'dev' into main
2 parents 5d5ae09 + bf8a556 commit 01a426e

9 files changed

Lines changed: 646 additions & 135 deletions

File tree

‎fri/DOCKER-README.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# Building FRI Container
2+
3+
Now, we elaborate on building FRI as a container, together with the Kong API Gateway.
4+
5+
Connect to the Server VM, assuming x.x.x.x to be the IP address of your server.
6+
````
7+
$ ssh -i "controlcore.pem" ubuntu@x.x.x.x
8+
````
9+
Perform Git clone if this is the first time you are configuring the Server
10+
````
11+
$ git clone git@github.com/ControlCore-Project/concore.git
12+
````
13+
14+
First build the Docker Container of the FRI.
15+
````
16+
$ git pull
17+
18+
$ sudo docker build -t fri .
19+
````
20+
21+
# Running Control-Core FRI with Kong as containers
22+
23+
If you are already running FRI, make sure to stop and clear existing FRI container as it is likely conflict with the port. If there is Kong gateway running in default ports, stop and clear it too.
24+
````
25+
$ docker stop fri
26+
$ docker rm fri
27+
$ docker stop kong
28+
$ docker rm kong
29+
````
30+
31+
Start and configure Cassandra container for Kong API.
32+
````
33+
$ docker run -d --name kong-database \
34+
-p 9042:9042 \
35+
cassandra:3
36+
37+
38+
$ docker run --rm \
39+
--link kong-database:kong-database \
40+
-e "KONG_DATABASE=cassandra" \
41+
-e "KONG_PG_HOST=kong-database" \
42+
-e "KONG_PG_USER=kong" \
43+
-e "KONG_PG_PASSWORD=kong" \
44+
-e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" \
45+
kong kong migrations bootstrap
46+
````
47+
48+
Start Kong
49+
````
50+
$ docker run -d --name kong \
51+
--link kong-database:kong-database \
52+
-e "KONG_DATABASE=cassandra" \
53+
-e "KONG_PG_HOST=kong-database" \
54+
-e "KONG_PG_PASSWORD=kong" \
55+
-e "KONG_CASSANDRA_CONTACT_POINTS=kong-database" \
56+
-e "KONG_PROXY_ACCESS_LOG=/dev/stdout" \
57+
-e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" \
58+
-e "KONG_PROXY_ERROR_LOG=/dev/stderr" \
59+
-e "KONG_ADMIN_ERROR_LOG=/dev/stderr" \
60+
-e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" \
61+
-p 80:8000 \
62+
-p 8443:8443 \
63+
-p 8001:8001 \
64+
-p 8444:8444 \
65+
kong
66+
````
67+
68+
Start FRI container
69+
````
70+
$ nohup sudo docker run --name fri -p 8090:8081 fri > controlcore.out &
71+
````
72+
73+
Delete if there is a previously configured Kong service. If not, skip this step. First you need to find the ID-VALUE for the route with a GET command before deleting the route and service.
74+
````
75+
$ curl -X GET "http://localhost:8001/services/fri/routes"
76+
````
77+
Use the ID output from above to issue the delete command as below (issue this only if you have a previous conflicting service definiton in kong. Otherwise, skip this step):
78+
````
79+
$ curl -X DELETE "http://localhost:8001/services/fri/routes/ID-VALUE"
80+
81+
$ curl -X DELETE "http://localhost:8001/services/fri/"
82+
````
83+
84+
Define Kong Service and Route.
85+
86+
First Configure a Kong service, replacing the variable "private-ip" with the private IP address of your server below.
87+
````
88+
$ curl -i -X POST --url http://localhost:8001/services/ --data 'name=fri' --data 'url=http://private-ip:8090'
89+
````
90+
Then configure route to the service
91+
````
92+
$ curl -i -X POST --url http://localhost:8001/services/fri/routes --data 'paths=/'
93+
````
94+
95+
Now, controlcore.org is routed through the Kong APIs.
96+
97+
98+
# Troubleshooting the FRI
99+
100+
Connect to the Server VM
101+
````
102+
$ ssh -i "controlcore.pem" ubuntu@x.x.x.x
103+
````
104+
Check the Server logs.
105+
````
106+
$ tail -f controlcore.out
107+
````
108+
or
109+
````
110+
$ sudo docker logs fri -f
111+
````
112+
Find the FRI docker container
113+
````
114+
$ sudo docker ps
115+
````
116+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
117+
dfdd3b3d3308 fri "python main.py" 38 minutes ago Up 38 minutes 0.0.0.0:80->80/tcp fri
118+
119+
Access the container
120+
````
121+
$ sudo docker exec -it dfdd /bin/bash
122+
````
123+

‎fri/Dockerfile‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
FROM centos/python-36-centos7
2+
COPY requirements.txt /tmp/requirements.txt
3+
RUN pip install -r /tmp/requirements.txt
4+
WORKDIR /fri
5+
COPY server /fri
6+
USER root
7+
RUN useradd fri && chown -R fri /fri
8+
USER fri
9+
CMD ["gunicorn", "--timeout=180", "--workers=4", "--bind=0.0.0.0:8080", "--access-logfile=-", "main:app"]

‎fri/README.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# The Control-Core FRI for Closed-Loop Neuromodulation Control Systems
2+
3+
The Control-Core File Receiving Interface (FRI) is built with is Python-3.10. It is the core component that makes the distributed executions a reality in the Control-Core framework.
4+
5+
# Install Dependencies
6+
7+
Install Jupyter lab
8+
````
9+
$ pip install jupyterlab
10+
````
11+
12+
# Running the FRI and a quick test.
13+
14+
To run the FRI as a server:
15+
````
16+
$ cd conore
17+
18+
$ git checkout dev
19+
20+
$ cd server
21+
22+
$ python3 main.py
23+
````
24+
25+
To test:
26+
````
27+
$ cd ..
28+
29+
$ python3 test.py
30+
````
31+

‎fri/example.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
print("Concore/fri")

‎fri/requirements.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Flask
2+
gunicorn==20.1.0

‎fri/server/main.py‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
from flask import Flask, request, jsonify, send_file, send_from_directory
2+
from werkzeug.utils import secure_filename
3+
import os
4+
from subprocess import call
5+
from pathlib import Path
6+
import json
7+
import subprocess
8+
9+
cur_path = os.path.dirname(os.path.abspath(__file__))
10+
concore_path = os.path.abspath(os.path.join(cur_path, '../../'))
11+
12+
13+
app = Flask(__name__)
14+
app.secret_key = "secret key"
15+
16+
# To upload multiple file. For example, /upload/test?apikey=xyz
17+
@app.route('/upload/<dir>', methods=['POST'])
18+
def upload(dir):
19+
apikey = request.args.get('apikey')
20+
dirname = secure_filename(dir) + "_" + apikey
21+
22+
if 'files[]' not in request.files:
23+
resp = jsonify({'message': 'No file in the request'})
24+
resp.status_code = 400
25+
return resp
26+
27+
files = request.files.getlist('files[]')
28+
29+
errors = {}
30+
success = False
31+
32+
directory_name = os.path.abspath(os.path.join(concore_path, secure_filename(dirname)))
33+
34+
if not os.path.isdir(directory_name):
35+
os.mkdir(directory_name)
36+
37+
38+
for file in files:
39+
if file:
40+
filename = secure_filename(file.filename)
41+
file.save(directory_name+"/"+filename)
42+
success = True
43+
44+
if success and errors:
45+
errors['message'] = 'File(s) successfully uploaded'
46+
resp = jsonify(errors)
47+
resp.status_code = 500
48+
return resp
49+
if success:
50+
resp = jsonify({'message': 'Files successfully uploaded'})
51+
resp.status_code = 201
52+
return resp
53+
else:
54+
resp = jsonify(errors)
55+
resp.status_code = 500
56+
return resp
57+
58+
59+
60+
# to download /build/<dir>?fetch=<graphml>. For example, /build/test?fetch=sample1&apikey=xyz
61+
@app.route('/build/<dir>', methods=['POST'])
62+
def build(dir):
63+
graphml_file = request.args.get('fetch')
64+
apikey = request.args.get('apikey')
65+
dirname = secure_filename(dir) + "_" + apikey
66+
makestudy_dir = dirname + "/" + graphml_file #for makestudy
67+
dir_path = os.path.abspath(os.path.join(concore_path, graphml_file)) #path for ./build
68+
if not os.path.exists(dir_path):
69+
proc = call(["./makestudy", makestudy_dir], cwd=concore_path)
70+
if(proc == 0):
71+
resp = jsonify({'message': 'Directory successfully created'})
72+
resp.status_code = 201
73+
else:
74+
resp = jsonify({'message': 'There is an Error'})
75+
resp.status_code = 500
76+
call(["./build"], cwd=dir_path)
77+
return resp
78+
79+
80+
@app.route('/debug/<dir>', methods=['POST'])
81+
def debug(dir):
82+
dir = secure_filename(dir)
83+
dir_path = os.path.abspath(os.path.join(concore_path, dir))
84+
proc = call(["./debug"], cwd=dir_path)
85+
if(proc == 0):
86+
resp = jsonify({'message': 'Close the pop window after obtaining result'})
87+
resp.status_code = 201
88+
return resp
89+
else:
90+
resp = jsonify({'message': 'There is an Error'})
91+
resp.status_code = 500
92+
return resp
93+
94+
95+
@app.route('/run/<dir>', methods=['POST'])
96+
def run(dir):
97+
dir = secure_filename(dir)
98+
dir_path = os.path.abspath(os.path.join(concore_path, dir))
99+
proc = call(["./run"], cwd=dir_path)
100+
if(proc == 0):
101+
resp = jsonify({'message': 'result prepared'})
102+
resp.status_code = 201
103+
return resp
104+
else:
105+
resp = jsonify({'message': 'There is an Error'})
106+
resp.status_code = 500
107+
return resp
108+
109+
@app.route('/stop/<dir>', methods=['POST'])
110+
def stop(dir):
111+
dir = secure_filename(dir)
112+
dir_path = os.path.abspath(os.path.join(concore_path, dir))
113+
proc = call(["./stop"], cwd=dir_path)
114+
if(proc == 0):
115+
resp = jsonify({'message': 'resources cleaned'})
116+
resp.status_code = 201
117+
return resp
118+
else:
119+
resp = jsonify({'message': 'There is an Error'})
120+
resp.status_code = 500
121+
return resp
122+
123+
124+
@app.route('/clear/<dir>', methods=['POST'])
125+
def clear(dir):
126+
dir = secure_filename(dir)
127+
dir_path = os.path.abspath(os.path.join(concore_path, dir))
128+
proc = call(["./clear"], cwd=dir_path)
129+
if(proc == 0):
130+
resp = jsonify({'message': 'result deleted'})
131+
resp.status_code = 201
132+
return resp
133+
else:
134+
resp = jsonify({'message': 'There is an Error'})
135+
resp.status_code = 500
136+
return resp
137+
138+
# to download /download/<dir>?fetch=<downloadfile>. For example, /download/test?fetchDir=xyz&fetch=u
139+
@app.route('/download/<dir>', methods=['POST', 'GET'])
140+
def download(dir):
141+
download_file = request.args.get('fetch')
142+
sub_folder = request.args.get('fetchDir')
143+
dirname = secure_filename(dir) + "/" + secure_filename(sub_folder)
144+
directory_name = os.path.abspath(os.path.join(concore_path, dirname))
145+
if not os.path.exists(directory_name):
146+
resp = jsonify({'message': 'Directory not found'})
147+
resp.status_code = 400
148+
return resp
149+
try:
150+
return send_from_directory(directory_name, download_file, as_attachment=True)
151+
except:
152+
resp = jsonify({'message': 'file not found'})
153+
resp.status_code = 400
154+
return resp
155+
156+
157+
@app.route('/destroy/<dir>', methods=['DELETE'])
158+
def destroy(dir):
159+
dir = secure_filename(dir)
160+
proc = call(["./destroy", dir], cwd=concore_path)
161+
if(proc == 0):
162+
resp = jsonify({'message': 'Successfuly deleted Dirctory'})
163+
resp.status_code = 201
164+
return resp
165+
else:
166+
resp = jsonify({'message': 'There is an Error'})
167+
resp.status_code = 500
168+
return resp
169+
170+
@app.route('/getFilesList/<dir>', methods=['POST'])
171+
def getFilesList(dir):
172+
sub_dir = request.args.get('fetch')
173+
dirname = secure_filename(dir) + "/" + secure_filename(sub_dir)
174+
dir_path = os.path.abspath(os.path.join(concore_path, dirname))
175+
res = []
176+
res = os.listdir(dir_path)
177+
res = json.dumps(res)
178+
return res
179+
180+
181+
@app.route('/openJupyter/', methods=['POST'])
182+
def openJupyter():
183+
proc = subprocess.Popen(['jupyter', 'lab'], shell=False, stdout=subprocess.PIPE, cwd=concore_path)
184+
if proc.poll() is None:
185+
resp = jsonify({'message': 'Successfuly opened Jupyter'})
186+
resp.status_code = 308
187+
return resp
188+
else:
189+
resp = jsonify({'message': 'There is an Error'})
190+
resp.status_code = 500
191+
return resp
192+
193+
194+
195+
if __name__ == "__main__":
196+
app.run(host="0.0.0.0")

‎fri/service_config.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
SERVICES=('upload' 'execute' 'download')
4+
for service in "${SERVICES[@]}"
5+
do
6+
name="name="$service
7+
path="paths=/"$service
8+
service_path="url=http://private-ip:8090/"$service
9+
route_path="http://localhost:8001/services/"$service"/routes"
10+
plugin_path="http://localhost:8001/services/"$service"/plugins"
11+
echo "Name is, $name"
12+
echo "Path is, $path"
13+
echo "Service path is, $service_path"
14+
echo "Route path is, $route_path"
15+
echo "Plugin path is, $plugin_path"
16+
echo "Creating Service, $service..."
17+
curl -i -X POST --url http://localhost:8001/services/ --data $name --data $service_path
18+
echo "Creating route for $service..."
19+
curl -i -X POST --url $route_path --data $path
20+
echo "Creating apikey-based authentication for $service..."
21+
curl -X POST $plugin_path --data "name=key-auth" --data "config.key_names=apikey"
22+
done

0 commit comments

Comments
 (0)