Skip to content
This repository was archived by the owner on Jan 8, 2020. It is now read-only.

Commit faf98a5

Browse files
author
Aurélien Duboc
committed
Add advanced configuration for production
1 parent c301b41 commit faf98a5

3 files changed

Lines changed: 152 additions & 6 deletions

File tree

‎README.md‎

Lines changed: 99 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# RootMe API
1+
# Root-Me API
22

33
URL: [https://root-me-api.hackademint.org](https://root-me-api.hackademint.org) \
44
BOT Discord: [link](https://discordapp.com/api/oauth2/authorize?client_id=523372231561314304&permissions=0&scope=bot)
@@ -11,9 +11,14 @@ It also provides dozens of virtual environments, accessible with a few clicks, t
1111
The purpose of this project is to provide an API to use
1212
[Root-Me](https://www.root-me.org/?lang=en) data, provide a discord bot fetching events and data from this API and to build a distributed system thanks to workers using [redis streams and consumer groups](https://redis.io/topics/streams-intro).
1313

14+
Using different worker instances is necessary to increase performance because
15+
Root-Me website is blocking the requests if it detects too much connections
16+
incoming from a specific public IP (You get an HTTP error code: "429 Too Many Requests"). \
17+
(See advanced configuration).
18+
1419
## How does it works ?
1520

16-
When a client is making a request on the API, the server is checking if there is an existing data in an associated [redis key](https://redis.io/commands/getset). If not, the server is sending a task to workers through the appropriate [redis stream](https://redis.io/topics/streams-intro). If there is existing data, the API is checking if the last update is not too old (depends on `UPDATE_TIMEOUT` in [constants.py](https://github.com/zteeed/Root-Me-API/blob/master/api/api/constants.py)) and ask the workers for updates if necessary.
21+
When a client is making a request on the API, the server is checking if there is existing data in an associated [redis key](https://redis.io/commands/getset). If not, the server is sending a task to workers through the appropriate [redis stream](https://redis.io/topics/streams-intro). If there is existing data, the API is checking if the last update is not too old (depends on `UPDATE_TIMEOUT` in [constants.py](https://github.com/zteeed/Root-Me-API/blob/master/api/api/constants.py)) and ask the workers for updates if necessary.
1722

1823
The form of the data is:
1924

@@ -45,15 +50,104 @@ You need to create a discord bot here: [https://discordapp.com/developers/applic
4550

4651
## Advanced configuration (1+ workers)
4752

53+
- You can expose your redis by adding to redis service `docker-compose.yml`:
54+
```
55+
ports:
56+
- 6379:6379
57+
```
58+
- If you want to deploy several workers you might need to modify [worker/main.py](https://github.com/zteeed/Root-Me-API/blob/master/worker/main.py), using the public ip where you deploy the redis container.
59+
60+
```python
61+
app.redis = await aioredis.create_redis_pool(('PUBLIC_IP', 6379))
62+
```
63+
64+
Then you can deploy a new worker separately using [worker/Dockerfile](https://github.com/zteeed/Root-Me-API/blob/master/worker/Dockerfile)
65+
66+
- If you have a physical interface which can be bind with different public IPv4 address:
67+
68+
1. Use the `docker-compose.yml` file from `advanced_configuration` folder
69+
70+
```bash
71+
mv advanced_configuration/docker-compose.yml ./docker-compose.yml
72+
```
73+
You might need the attributes from the [docker macvlan network](https://docs.docker.com/network/macvlan/).
74+
75+
76+
2. Details
77+
78+
This configuration avoids interfaces from backend network to get an access to internet, but we
79+
want containers from this network to communicate together (api/redis/workers).
80+
81+
Every worker is bridged with a specific physical interface so that Root-Me detects those workers with different public IP. \ (We choose not to [rotate with proxies](https://www.scrapehero.com/how-to-rotate-proxies-and-ip-addresses-using-python-3/) for data privacy issues).
82+
83+
*__Example__* (extract from [advanced_configuration/docker-compose.yml](https://github.com/zteeed/Root-Me-API/blob/master/advanced_configuration/docker-compose.yml):
84+
```
85+
bridge_worker:
86+
driver: macvlan
87+
driver_opts:
88+
parent: eth1
89+
macvlan_mode: bridge
90+
ipam:
91+
config:
92+
- subnet: 157.159.191.0/24
93+
gateway: 157.159.191.2
94+
ip_range: 157.159.191.56/29
95+
```
96+
97+
If you deploy 6 workers (see "Install" subsection), the workers will take
98+
`157.159.191.56-61` as IPv4 address thanks to `ip_range` attribute (those are
99+
taken in ascending order in the subnet specified).
100+
101+
> Communication with the Docker host over macvlan.
102+
> When using macvlan, you cannot ping or communicate with the default namespace IP address. For example, if you create a container and try to ping the Docker host’s eth0, it will not work. That traffic is explicitly filtered by the kernel modules themselves to offer additional provider isolation and security.
103+
104+
[source](https://docs.docker.com/v17.09/engine/userguide/networking/get-started-macvlan/)
105+
106+
107+
3. Additional information
108+
109+
- [docker networking in compose](https://docs.docker.com/compose/networking/)
110+
- [docker macvlan networks](https://docs.docker.com/network/macvlan/)
111+
- [remove docker networks](https://docs.docker.com/engine/reference/commandline/network_rm/)
112+
113+
48114
### Install
49115

116+
For 6 workers:
50117
```bash
51-
docker-compose up -d
118+
docker-compose up -d --scale worker=6
119+
```
120+
121+
Alternatively, you can use `docker-compose --compatibility up` with:
122+
123+
```
124+
deploy:
125+
mode: replicated
126+
replicas: 6
127+
```
128+
129+
It makes docker accept deploy section without using swarm.
130+
131+
### Result
132+
133+
```
134+
docker ps
135+
136+
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
137+
aa46c63b6d8f root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 5 seconds root-me-api_worker_3
138+
a68c6583cbc4 root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 2 seconds root-me-api_worker_5
139+
2b35a79fe8bc root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 2 seconds root-me-api_worker_1
140+
3acc7e2fce96 root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 7 seconds root-me-api_worker_6
141+
d63206e68591 root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 7 seconds root-me-api_worker_4
142+
a14e3e6918b3 root-me-api_worker "/bin/sh -c 'python3…" 18 seconds ago Up 3 seconds root-me-api_worker_2
143+
0361d34552be root-me-api_discord "/bin/sh -c 'python3…" 25 seconds ago Up 17 seconds root-me-api_discord_1
144+
6eab3694fbce root-me-api_api "/bin/sh -c 'python3…" 25 seconds ago Up 17 seconds 0.0.0.0:3000->3000/tcp root-me-api_api_1
145+
f21c0ef802aa redis "docker-entrypoint.s…" 25 seconds ago Up 17 seconds root-me-api_redis_1
52146
```
53147

54148
## [API]
55149

56-
Some endpoints need a valid RootMe username you can extract from the URL of your profile. \
150+
Some endpoints need a valid Root-Me username you can extract from the URL of your profile. \
57151
Here is an example with https://www.root-me.org/zTeeed-115405 --> zTeeed-115405
58152

59153

@@ -88,7 +182,7 @@ Display available commands.
88182

89183
Add a user to team / Remove a user from team.
90184

91-
Extract the username from the RootMe profile link. \
185+
Extract the username from the Root-Me profile link. \
92186
Example with [https://www.root-me.org/zTeeed-115405?inc=info](https://www.root-me.org/zTeeed-115405?inc=info)
93187

94188
![](./images/discord_add_remove_user.png)

‎advanced_configuration/docker-compose.yml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
version: '2'
2+
services:
3+
redis:
4+
image: redis
5+
restart: always
6+
networks:
7+
- backend
8+
api:
9+
restart: always
10+
build:
11+
context: ./api
12+
ports:
13+
- "3000:3000"
14+
networks:
15+
- frontend
16+
- backend
17+
worker:
18+
restart: always
19+
build:
20+
context: ./worker
21+
depends_on:
22+
- api
23+
networks:
24+
- backend
25+
- bridge_worker
26+
discord:
27+
restart: always
28+
build:
29+
context: ./bot_discord
30+
networks:
31+
- frontend
32+
33+
networks:
34+
frontend:
35+
driver: bridge
36+
backend:
37+
# Disable access to internet
38+
driver: bridge
39+
internal: true
40+
bridge_worker:
41+
driver: macvlan
42+
driver_opts:
43+
parent: eth1
44+
macvlan_mode: bridge
45+
ipam:
46+
config:
47+
- subnet: 157.159.191.0/24
48+
gateway: 157.159.191.2
49+
ip_range: 157.159.191.56/29

‎worker/worker/constants.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
1+
import random
2+
import string
3+
14
URL = 'https://www.root-me.org/'
25
REDIS_STREAM_USERS = 'update_users'
36
REDIS_STREAM_CHALLENGES = 'update_challenges'
47
CG_NAME = 'rootme'
5-
CONSUMER_NAME = 'worker1'
8+
CONSUMER_NAME = ''.join(random.choice(string.ascii_lowercase) for _ in range(25))

0 commit comments

Comments
 (0)