Speedup pt 2 (#469)

* Added uids and fids as indicies in the database to speed up queries

* speedup member management

* use user dict instead of ldap where possible

* fix linting :)

* fix eboard group names
SIGH

* moved migrations out of dockerfile and updated documentation

Author: BigSpaceships

README.md

@@ -8,72 +8,103 @@ A comprehensive membership evaluations solution for Computer Science House.
 Development
 -----------
 
-To run the application, you must have the latest version of [Python 3](https://www.python.org/downloads/) and [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) installed. Once you have those installed, create a new virtualenv and install the Python dependencies:
+### Config
+
+You must create `config.py` in the top-level directory with the appropriate credentials for the application to run. See `config.env.py` for an example.
 
+#### Add OIDC Config
+Reach out to an RTP to get OIDC credentials that will allow you to develop locally behind OIDC auth
+```py
+# OIDC Config
+OIDC_ISSUER = "https://sso.csh.rit.edu/auth/realms/csh"
+OIDC_CLIENT_CONFIG = {
+    'client_id': '',
+    'client_secret': '',
+    'post_logout_redirect_uris': ['http://0.0.0.0:6969/logout']
+}
 ```
+
+#### Database 
+You can either develop using the dev database, or use the local database provided in the docker compose file
+
+Using the local database is detailed below, but both options will require the dev database password, so you will have to ask an RTP for this too
+
+### Run (Without Docker)
+
+To run the application without using containers, you must have the latest version of [Python 3](https://www.python.org/downloads/) and [virtualenv](https://virtualenv.pypa.io/en/stable/installation/) installed. Once you have those installed, create a new virtualenv and install the Python dependencies:
+
+```sh
 virtualenv .conditionalenv -p `which python3`
 source .conditionalenv/bin/activate
 pip install -r requirements.txt
-export FLASK_APP=app.py
 ```
 
-In addition, you must have Node, NPM, and Gulp CLI installed to properly execute the asset pipeline. If you don't have Node installed, we recommending installing with [NVM](https://github.com/creationix/nvm):
+In addition, you must have Node, NPM, and Weback CLI installed to properly execute the asset pipeline. If you don't have Node installed, we recommending installing with [NVM](https://github.com/creationix/nvm):
 
-```
+```sh
 nvm install
 nvm use
-npm install -g gulp
+npm install -g webpack
 ```
 
-Then, install the pipeline and frontend dependencies:
+Then, install the pipeline and frontend dependencies: (do this in the `frontend` directory)
 
-```
+```sh
 npm install
 ```
 
-### Config
+Once you have all of the dependencies installed, run
 
-You must create `config.py` in the top-level directory with the appropriate credentials for the application to run. See `config.sample.py` for an example.
+```sh
+npm webpack
+```
 
-#### Add OIDC Config
-Reach out to an RTP to get OIDC credentials that will allow you to develop locally behind OIDC auth
+This will build the frontend assets and put them in the correct place for use with flask
+
+Finally, start the flask app with `gunicorn`
+
+```sh
+gunicorn
 ```
-# OIDC Config
-OIDC_ISSUER = "https://sso.csh.rit.edu/auth/realms/csh"
-OIDC_CLIENT_CONFIG = {
-    'client_id': '',
-    'client_secret': '',
-    'post_logout_redirect_uris': ['http://0.0.0.0:6969/logout']
-}
+
+or 
+
+```sh
+python -m gunicorn
 ```
 
-### Run
+### Run (containerized)
 
-Once you have all of the dependencies installed, simply run:
+It is likely easier to use containers like `podman` or `docker` or the corresponding compose file
 
-```
-npm start
+With podman, I have been using 
+
+```sh
+podman compose up --force-recreate --build
 ```
 
-This will run the asset pipeline, start the Python server, and start BrowserSync. Your default web browser will open automatically. If it doesn't, navigate to `http://127.0.0.1:3000`. Any changes made to the frontend files in `frontend` or the Jinja templates in `conditional/templates` will cause the browser to reload automatically.
+Which can be restarted every time changes are made
 
 ### Dependencies
 
 To add new dependencies, add them to `requirements.in` and then run `pip-compile requirements.in` to produce a new locked `requirements.txt`. Do not edit `requirements.txt` directly as it will be overwritten by future PRs.
 
 ### Local database
 
-You can run the database locally using the docker compose, make sure to upgrade it as explained below
+You can run the database locally using the docker compose
 
 To populate it with dev data for example, you can use the command
 
-```
-PGPASSWORD='[DB PASSWORD]' pg_dump -h postgres.csh.rit.edu -p 5432 -U conditional-dev conditional-dev |  PGPASSWORD='fancypantspassword' psql -h localhost -p 5432 -U conditional conditional
+```sh
+PGPASSWORD='[DB PASSWORD]' pg_dump -h postgres.csh.rit.edu -p 5432 -U conditionaldev conditionaldev |  PGPASSWORD='fancypantspassword' psql -h localhost -p 5432 -U conditional conditional
 ```
 
 This can be helpful for changing the database schema
 
-NOTE: to use flask db commands with a database running in the compose file, you will have to update your url to point to localhost, not conditional-postgres
+To run migration commands in the local database, you can run the commands inside the docker container. Any migrations created will also be in the local repository since migrations are mounted in the docker compose
+```sh
+podman exec conditional flask db upgrade
+```
 
 ### Database Migrations
 

conditional/__init__.py

@@ -1,6 +1,7 @@
 import os
 from datetime import datetime
 
+from pstats import SortKey
 import structlog
 from csh_ldap import CSHLDAP
 from flask import Flask, redirect, render_template, request, g
@@ -33,7 +34,8 @@
 if app.config['PROFILING']:
     app.wsgi_app = ProfilerMiddleware(
         app.wsgi_app,
-        restrictions=[30]
+        sort_by=('cumulative',),
+        restrictions=[80]
     )
 
 # Sentry setup
@@ -195,7 +197,7 @@ def route_errors(error, user_dict=None):
 
     # Handle the case where the header isn't present
     if user_dict['username'] is not None:
-        data['username'] = user_dict['account'].uid
+        data['username'] = user_dict['username']
         data['name'] = user_dict['account'].cn
     else:
         data['username'] = "unknown"

conditional/blueprints/attendance.py

@@ -21,7 +21,7 @@
 from conditional.util.ldap import ldap_get_current_students
 from conditional.util.ldap import ldap_get_member
 from conditional.util.ldap import ldap_is_eboard
-from conditional.util.ldap import ldap_is_eval_director
+from conditional.util.user_dict import user_dict_is_eboard, user_dict_is_eval_director
 
 logger = structlog.get_logger()
 
@@ -160,7 +160,7 @@ def display_attendance_hm(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info('Display House Meeting Attendance Page')
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return redirect("/dashboard")
 
     return render_template('attendance_hm.html',
@@ -175,7 +175,7 @@ def display_attendance_hm(user_dict=None):
 def submit_committee_attendance(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    approved = ldap_is_eboard(user_dict['account'])
+    approved = user_dict_is_eval_director(user_dict)
     post_data = request.get_json()
 
     committee = post_data['committee']
@@ -211,7 +211,7 @@ def submit_seminar_attendance(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info('Submit Technical Seminar Attendance')
 
-    approved = ldap_is_eboard(user_dict['account'])
+    approved = user_dict_is_eboard(user_dict)
 
     post_data = request.get_json()
 
@@ -248,7 +248,7 @@ def submit_house_attendance(user_dict=None):
 
     # status: Attended | Excused | Absent
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be evals", 403
 
     post_data = request.get_json()
@@ -289,7 +289,7 @@ def submit_house_attendance(user_dict=None):
 def alter_house_attendance(uid, hid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be evals", 403
 
     if not uid.isdigit():
@@ -319,7 +319,7 @@ def alter_house_attendance(uid, hid, user_dict=None):
 def alter_house_excuse(uid, hid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be eval director", 403
 
     post_data = request.get_json()
@@ -381,7 +381,7 @@ def get_seminar_attendees(meeting_id):
 
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
 
@@ -444,7 +444,7 @@ def alter_committee_attendance(cid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info('Edit Committee Meeting Attendance')
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
     post_data = request.get_json()
@@ -476,7 +476,7 @@ def alter_seminar_attendance(sid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info('Edit Technical Seminar Attendance')
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
     post_data = request.get_json()
@@ -559,7 +559,7 @@ def get_ts_attendees(cid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info(f'Delete Committee Meeting {cid}')
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
     FreshmanCommitteeAttendance.query.filter(
@@ -582,7 +582,7 @@ def approve_cm(cid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info(f'Approve Committee Meeting {cid} Attendance')
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
     CommitteeMeeting.query.filter(
@@ -600,7 +600,7 @@ def approve_ts(sid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info(f'Approve Technical Seminar {sid} Attendance')
 
-    if not ldap_is_eboard(user_dict['account']):
+    if not user_dict_is_eboard(user_dict):
         return jsonify({"success": False, "error": "Not EBoard"}), 403
 
     TechnicalSeminar.query.filter(

conditional/blueprints/cache_management.py

@@ -13,8 +13,7 @@
 from conditional.util.ldap import ldap_get_intro_members
 from conditional.util.ldap import ldap_get_member
 from conditional.util.ldap import ldap_get_onfloor_members
-from conditional.util.ldap import ldap_is_eval_director
-from conditional.util.ldap import ldap_is_rtp
+from conditional.util.user_dict import user_dict_is_eval_director, user_dict_is_rtp
 
 logger = structlog.get_logger()
 cache_bp = Blueprint('cache_bp', __name__)
@@ -24,7 +23,7 @@
 @auth.oidc_auth("default")
 @get_user
 def restart_app(user_dict=None):
-    if not ldap_is_rtp(user_dict['account']):
+    if not user_dict_is_rtp(user_dict):
         return redirect("/dashboard")
 
     log = logger.new(request=request, auth_dict=user_dict)
@@ -37,7 +36,7 @@ def restart_app(user_dict=None):
 @auth.oidc_auth("default")
 @get_user
 def clear_cache(user_dict=None):
-    if not ldap_is_eval_director(user_dict['account']) and not ldap_is_rtp(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict) and not user_dict_is_rtp(user_dict):
         return redirect("/dashboard")
 
     log = logger.new(request=request, auth_dict=user_dict)

conditional/blueprints/co_op.py

@@ -6,10 +6,9 @@
 from conditional.util.member import req_cm
 from conditional.util.auth import get_user
 from conditional.util.flask import render_template
-from conditional.util.ldap import ldap_is_eval_director, ldap_is_current_student
 from conditional.util.ldap import _ldap_add_member_to_group as ldap_add_member_to_group
 from conditional.util.ldap import _ldap_remove_member_from_group as ldap_remove_member_from_group
-from conditional.util.ldap import _ldap_is_member_of_group as ldap_is_member_of_group
+from conditional.util.user_dict import user_dict_is_current_student, user_dict_is_eval_director, user_dict_is_in_group
 
 co_op_bp = Blueprint('co_op_bp', __name__)
 
@@ -43,7 +42,7 @@ def submit_co_op_form(user_dict=None):
     semester = post_data['semester']
     if post_data['semester'] not in valid_semesters:
         return "Invalid semester submitted", 400
-    if not ldap_is_current_student(user_dict['account']):
+    if not user_dict_is_current_student(user_dict):
         return "Must be current student", 403
 
     log.info(f'Submit {semester} Co-Op')
@@ -70,15 +69,15 @@ def submit_co_op_form(user_dict=None):
 def delete_co_op(uid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be eval director", 403
 
     log.info(f'Delete {uid}\'s Co-Op')
 
     # Remove from corresponding co-op ldap group
-    if ldap_is_member_of_group(user_dict['account'], 'fall_coop'):
+    if user_dict_is_in_group(user_dict, 'fall_coop'):
         ldap_remove_member_from_group(user_dict['account'], 'fall_coop')
-    if ldap_is_member_of_group(user_dict['account'], 'spring_coop'):
+    if user_dict_is_in_group(user_dict, 'spring_coop'):
         ldap_remove_member_from_group(user_dict['account'], 'spring_coop')
 
     CurrentCoops.query.filter(CurrentCoops.uid == uid, CurrentCoops.date_created > start_of_year()).delete()
@@ -97,7 +96,7 @@ def display_co_op_management(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info('Display Co-Op Management')
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be eval director", 403
 
     co_op_list = [(member.semester, member.uid)

conditional/blueprints/conditional.py

@@ -7,7 +7,7 @@
 from conditional.models.models import Conditional, SpringEval, FreshmanEvalData
 from conditional.util.auth import get_user
 from conditional.util.flask import render_template
-from conditional.util.ldap import ldap_is_eval_director
+from conditional.util.user_dict import user_dict_is_eval_director
 
 conditionals_bp = Blueprint('conditionals_bp', __name__)
 
@@ -44,7 +44,7 @@ def display_conditionals(user_dict=None):
 def create_conditional(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return "must be eval director", 403
 
     post_data = request.get_json()
@@ -81,7 +81,7 @@ def create_conditional(user_dict=None):
 def conditional_review(user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
 
-    if not ldap_is_eval_director(user_dict['account']):
+    if not user_dict_is_eval_director(user_dict):
         return redirect("/dashboard", code=302)
 
     post_data = request.get_json()
@@ -119,7 +119,7 @@ def conditional_delete(cid, user_dict=None):
     log = logger.new(request=request, auth_dict=user_dict)
     log.info(f'Delete conditional-{cid}')
 
-    if ldap_is_eval_director(user_dict['account']):
+    if user_dict_is_eval_director(user_dict):
         Conditional.query.filter(
             Conditional.id == cid
         ).delete()