v2.2.0

Researchers

• You can now provide Project Start and End Dates on the Project Details tab when editing your DMP. #2409
• You can now provide information on your Project's collaborators / contributors on the new 'Contributors' tab when editing your project. For example, you can identify more than one PI, project administrators and data curators. #2349 #2408
• The new 'Contributors' tab provides a typeahead box for organizational affiliation. This box searches the Research Organization Registry (ROR) as you type. #2339
• Fixed a bug that was sometimes confusing the funder and template names on PDF exports #2419

Organizational Admins

• You can now make some checkboxes, dropdown lists or radio buttons on your templates trigger conditional logic. For example if a user selects a particular value from a dropdown then logic can be triggered to send an email or hide other questions related to their response. #1772 #2476 #2405 #2451
• The Usage Statistics page now allows you to filter out test plans and the filters now apply to the same graphs instead of the static plan and user counts (which have been removed) #2158 #2343
• You can now click on one of the bars in the bar charts on the Usage Dashboard page which will drill into the results.
• You can now filter the results of the Plans and Users pages by month and year #2450
• You can now provide us with a Google Analytics code that you can then use to monitor your user's activity within the system. #2344
• Updated the API to allow you to update your User's departmental affiliations. See the wiki documentation for more details #2235
• Fixed a bug that prevented you from viewing historical versions of your templates #2509
• Fixed a bug that was giving new templates the incorrect visibility selection by default. This was preventing published templates from being used. #2518
• Fixed a bug that was prevent users from creating more than 5 templates #2458
• Fixed a bug that was preventing you from deleting questions from a published template #2304
• ORCID and Shibboleth ids now appear on the Users page #2353

Super Admins

• You can now deactivate a notification #2382
• You now have the ability to export all plans #2346
• Added filter by permission level to the Users page #2352

Developers
This release contains many structural DB changes and data transformations to help your system move towards the RDA Common Metadata Standard for DMPs. These changes will make future integrations with external systems easier. Please read the following instructions carefully before proceeding with your upgrade.

Please read through these notes BEFORE deploying this release!

If you are running on a machine that has 2 or fewer CPUs then you will need to modify the upgrade Rake tasks. See Issue 2723

• All of the Org selection boxes have been updated. The ones on the 'Create account', 'Edit profile', 'New Org', 'Funder selection Project Details' and the new 'Contributors' tab provide a typeahead that queries the list of Orgs in your DB as well as the [Research Organization Registry (ROR) API] (https://ror.org). The results prioritize results from your DB. It also allows the user to enter their own value for scenarios where an Organization does not appear in the ROR API. If the user selects (or enters their own value) in the box, a new Org record is created and the user is associated with the org.
• The old user_identifiers and org_identifiers tables have been consolidated into the new polymorphic identifiers table. The upgrade script will migrate your existing records into this new table. These legacy tables will be dropped in the next release.
• Two new Identifier Schemes have been added: Fundref and ROR. When new Orgs are created via the org selection box described above, the ROR and Fundref (if applicable) identifiers are added to the new identifiers table. An upgrade script is provided that will retrieve the ROR and Fundref identifiers for all of your existing Org records.
• Introduced a 'managed' flag to the Orgs table. This will eventually replace the old 'is_other' flag. A 'managed' org has administrators. New Orgs created in the future by a user's selection of a ROR Org (or entering free text) will be 'unmanaged' by default
• An upgrade script is supplied that will attempt to match any User's associated with the default is_other Org with a result from the ROR API (using the user's email domain and the org's name stored on the user record). If an appropriate match is found, the org record will be created and the user will be associated with the new org.
• Introduced the 1st half of the RDA metadata standard for DMP compliant API api/v1/. The API currently only allows for the following: templates#index, plans#index (no filters yet), plans#show and plans#create #2390 (Note that the 2nd half of the API work is scheduled for late summer / early fall .. the draft of the new spec can be found here)
• Added a new 'Api Clients' section to allow you register clients for the new v1 API. The v1 API uses JSON Web Tokens for security and allows either an Api Client or User to authenticate. Api Clients are meant for communication with external systems. See the API spec linked above for further details

Updates to the Gemfile dependencies

• We added Httparty to handle communications with external APIs
• We added JWT to handle auth for the new v1 API
• We added Capybara-Webmock to ensure that calls are not made to external apis when running tests.

Most gems were upgraded to their latest patch release. The following gems had minor or major version updates:

Dependency	from	to
Annotate gem	3.0	3.1
Better Errors	2.5	2.7
Brakeman	4.7	4.8
Bullet	6.0	6.1
Bundler Audit	0.6	0.7
Byebug	11.0	11.1
Capybara	3.29	3.32
Database Cleaner	1.7	1.8
Excon	0.71	0.75
FactoryBot & FactoryBot Rails	5.1	5.2
Faraday	0.17	1.0
FFI	1.11	1.13
FOG AWS	3.5	3.6
FOG Core	2.1	2.2
Hashie	3.6	4.1
Launchy	2.4	2.5
Libv8	6.9	7.3
Loofah	2.5	2.6
Lumberjack	1.0	1.2
Method Source	0.9	1.0
Parser	2.6	2.7
Pry	0.12	0.13
MiniProfiler	1.1	2.0
Recaptcha	5.2	5.5
Regex Parser	1.6	1.7
Rollbar	2.22	2.25
Rspec	3.9	4.0
Rubocop	0.77	0.85
Sassc	2.2	2.4
Shoulda	3.6	4.0
Simplecov	0.17	0.18
Thor	0.20	1.0
Unicode Display Width	1.6	1.7
Webmock	3.7	3.8
----------	-----	-----

Updates to the Yarn JS dependencies

• Added bootstrap-select to support conditional questions feature
• Most JS files were upgraded to their latest minor or patch release.

Database Changes

• New table conditions to support conditional questions
• New table api_clients
• New contributors table
• New table identifiers (replaces the user_identifiers and org_identifiers)
• New table trackers to store org Google Analytics codes
• Add a orgs.managed flag
• Added plans.funder_id and plans.org_id (replaces old funder string) relationships
• Added plans.start_date and plans.end_date
• Added plans.api_client_id to identify the provenance of plans created via api v1
• Added a notifications.enabled flag
• Added a users.last_api_access field
• Added a identifier_schemes.context bit flag
• Added question_options.versionable_id
• Added filtered to the stats tables

Upgrade process
This upgrade involves changes to the database structure (see above). It also requires scripts to be run that will transform your existing data to fit into these new structures. We recommend that you run through the upgrade process in a development or staging environment prior to production. Any cleanup that may need to be done can then be scripted to help speed up the upgrade process in production.

1. Back up your database!

2. Update your branding.yml file as follows:

• Add use_recaptcha: false to the application section. Set the value to 'true' to add a reCAPTCHA to the create account form.
• Review the config/initializers/external_apis/ files. The DOI one is a placeholder, this functionality will appear in a future release. You can set the open_aire.rb file's active attribute to enable/disable the API typeahead for the 'Grant ID' field on the Project Details page. You can set the ror.rb file's active attribute to enable/disable calls to the ROR API (this will force the new Org selection boxes to only look at your local database ... users may still enter free text into the field to create a new Org record)

3. Run bundle exec rake db:migrate RAILS_ENV=[env] to make the necessary changes to the DB.

4. We highly recommend that you run the following scripts in the latest version of v2.2.x! For example, v2.2.1 includes several bug fixes to the v2_0_0_part2 script.

5. Run the 1st data transformation script that will populate the new question_options.versionable_id field. This step can take over 5 minutes to run depending on the size of your table: bundle exec rake upgrade:v2_1_6 RAILS_ENV=[env]

6. Run the upgrade:v2_2_0_part1 upgrade script that will add 'fundref' and 'ror' to the 'identifier_schemes' table, populate the 'context' field on the 'identifier_schemes' table, migrate your 'user_identifiers' and 'org_identifiers' into the new 'identifiers' table and search the ROR api and add the 'ror' and 'fundref' ids for your existing orgs into the 'identifiers' table: bundle exec rake upgrade:v2_2_0_part1 RAILS_ENV=[env]

7. Take a moment to review the output of the upgrade:v2_2_0_part1 script and then open the tmp/ror_fundref_ids.csv file. This file shows a side by side comparison of the Org's name in your DB and the Org's name that was found by the ROR API. In some circumstances you may need to correct the match that was found. For example if you had 'University of Example' in your DB, ROR may have selected 'University of Example Medical School'. Follow the suggested instructions in the upgrade:v2_2_0_part1 script's output to resolve any items that need correction. Some example SQL scripts that may be useful are located at the end of the release notes.

8. You should also at this point see if there were any Orgs that did not get a ROR identifier. In some circumstances an organization may not appear within the registry. If this is not the case, you can manually add the entry to the table.

9. Run the upgrade:v2_2_0_part2 upgrade script which will attempt to migrate as many users associated with the is_other Org to new Org records (by searching the ROR API for the email address' domain and the text stored in the 'users.other_organisation' field), convert the old 'data_contact' and 'principal_investigator' fields on the 'plans' table into 'contributors' records, set the 'plans.org_id' field to the plan owner's org_id, lookup the 'plans.funder_name' in the 'orgs' table or ROR api (creating the Org if needed) and setting the 'plans.funder_id', and convert the old 'plans.grant_id' field into an 'identifiers' record. Note that this script can take in excess of 45 minutes to complete depending on the size of your DB.

10. Review the output from the upgrade:v2_2_0_part2 script and then spot check any new Org records that were created for duplicates. You should prefer the Org with the lowest id value (the original record) when merging since the new Org record will only have 'users' and a 'guidance_group' associated with it.

11. Note that the old values were left in the 'user_identifiers', 'org_identifiers', 'plans' 'data_contact', 'principal_investigator', 'funder_name' and 'grant_id' fields for reference. These DB entities will be removed in a future release

The following are some useful queries that can be used to manually correct Org changes or add new ones not created during the upgrade process (note: ROR and FundRef identifiers are expected to be URLs so do not forget the URL prefix when making changes manually!):

# Add a new Org and its corresponding ROR and FundRef IDs found while searching https://ror.org
# ------------------------------------------------------------------------------------------------------
INSERT INTO orgs (name, managed, is_other, org_type, created_at, updated_at) 
VALUES ('Institution Name (website.edu)', 1, false, false, CURDATE(), CURDATE());
# ROR:
INSERT INTO identifiers (identifier_scheme_id, identifiable_id, identifiable_type, value, attrs, created_at, updated_at) 
  (SELECT [identifier_scheme_id for "ror"], id, 'Org', 'https://ror.org/[code]', '{}', CURDATE(), CURDATE() FROM orgs WHERE name = 'Institution Name (website.edu)');
# FUNDREF:
INSERT INTO identifiers (identifier_scheme_id, identifiable_id, identifiable_type, value, attrs, created_at, updated_at) 
  (SELECT [identifier_scheme_id for "fund ref"], id, 'Org', 'https://api.crossref.org/funders/[code]', '{}', CURDATE(), CURDATE() FROM orgs WHERE name = 'Institution Name (website.edu)');

# Attach users with a specific email domain to an Org
# ------------------------------------------------------------------------------------------------------
UPDATE users SET org_id = (SELECT id FROM orgs WHERE name = 'Institution Name (website.edu)') 
WHERE id IN (SELECT id FROM users WHERE email LIKE '%@website.edu');

# Remove a duplicate Org created by the upgrade scripts
# ------------------------------------------------------------------------------------------------------
UPDATE users SET org_id = [corret org] WHERE org_id = (SELECT id FROM orgs WHERE name = ‘Institution Name (website.edu)’);
UPDATE plans SET org_id = [correct org] WHERE org_id = (SELECT id FROM orgs WHERE name = ‘Institution Name (website.edu)’);
DELETE FROM guidance_groups WHERE id = (SELECT id FROM orgs WHERE name = ‘Institution Name (website.edu)’);
DELETE FROM orgs WHERE name = ‘Institution Name (website.edu)’;