Skip to content

6.11-release-notes.md

Latest commit

 

History

History
289 lines (163 loc) · 24.6 KB

6.11-release-notes.md

File metadata and controls

289 lines (163 loc) · 24.6 KB
Raw

Dataverse 6.11

Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.11 rather than the list of releases, which will cut them off.

This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project!

Release Highlights

Highlights for Dataverse 6.11 include:

  • Locally FAIR data
  • New expanded geospatial metadata block
  • Submit for Review improvements
  • Show historical file access requests
  • Local reviews
  • New and improved APIs
  • Bug fixes

Locally FAIR Data

This release includes experimental support for "Locally FAIR" data. When enabled, this feature allows publication of content that will be visible only to authorized users or groups within a Dataverse installation. Users without authorization will not see the Locally FAIR collections, datasets, or files in search results and cannot visit their pages or access them via the Dataverse API.

For more information, see Locally FAIR Data in the guides and #12319.

New Expanded Geospatial Metadata Block

This release introduces a major enhancement to geospatial data support with the addition of an expanded Geospatial Metadata Block, designed to improve how researchers describe, share, and discover geospatial datasets.

This new expanded metadata block aligns with the international ISO 19115 standard for describing geographic data.

  • Includes 22 new metadata fields to provide more detailed and standardized descriptions of geospatial data (e.g. vector, raster, mixed or multi-format geospatial collections)
  • Incorporates and extends current metadata elements (Geographic Coverage and Geographic Bounding Box)
  • Ensures backward compatibility while significantly improving metadata description capabilities

See the guides, #10398, and #11507.

Submit for Review Improvements

  • A disclaimer message and custom text can be added to the "Submit for Review" dialog, similar to Publish disclaimer messages. See :SubmitForReviewDatasetDisclaimerText, :DatasetSubmitForReviewPopupCustomText, #12325 and #12373.
  • When requireFilesToPublishDataset (added in #10994 in Dataverse 6.6) is set on a collection, a dataset must contain files for the Submit for Review button to appear. As before, publishing is also prevented if no files are present. See #12258 and #12266.

Show Historical File Access Requests

A new checkbox called "Show Historical Requests" has been added to the permissions page for a data file. Checking the box will show the number of times each user has been granted or denied access. See #8013 and #12012.

Local Reviews

Datasets can have local reviews, listable via API. A local review is a review dataset ("review" for short) that points at the URL form of a persistent ID of a dataset (e.g. itemReviewedUrl:https://doi.org/10.5072/FK2/ABCDEF) that is in the same Dataverse installation. Local reviews of a dataset can be listed via API (and we plan to build a UI for it some day).

A new metadata block called "Trusted Data Dimensions and Intensities" has been added for testing. This is described in the setup instructions for review datasets.

If you set dataverse.feature.croissant-with-local-reviews to true, local reviews will appear in the croissant and croissantSlim metadata export formats for any dataset that has local reviews. This feature is experimental, which is why it is hidden behind a feature flag.

See the guides for the new list reviews API endpoint, #12313, #12314, and #12425.

Features Added

These are features that weren't already mentioned under "highlights" above.

Original Tabular File Format Shown (Rather Than .tab)

In previous releases of Dataverse, the file, dataset, and collection pages showed the .tab version of tabular files (a plain text preservation-friendly copy). This has been changed to show the original format instead (e.g. Excel, Stata, etc.). The .tab version is still available from the download menu. See #7956 and #12145.

Croissant 1.1 (Summary Statistics)

The Croissant metadata export format has been updated from version 1.0 to 1.1.

Summary statistics (mean, min, max, etc.) are now included for tabular files that were successfully ingested.

You can download an example Croissant file from the Supported Metadata Export Formats section of the guides.

Minor backward-incompatible changes were made, which are noted below. See #12014 and #12214.

Local Contexts Improvements

For instances that have enabled the Local Contexts integration, Dataverse will now add rights information related to the Notices and Labels from a Local Contexts Project associated with a dataset to the metadata sent to DataCite (when using DataCite DOIs) and available in metadata exports (DataCite, OAI-ORE, and JSON).

In addition, it is now possible to use non-string values in the retrieval-filtering context entries for external vocabulary scripts. This can be used to allow filtered JSON that is not valid JSON-LD to be included in the OAI_ORE JSON-LD metadata export in a way that JSON-LD parsers will accept (and not ignore/drop). The OAI_ORE export version has been updated to 1.0.3 with this change. See the guides and #12224.

Improved Support for Handles as Persistent IDs

The following issues were addressed in #12366:

  • When PID registration of persistent IDs for files is enabled, Dataverse will create the handle as soon as the file is created (similar to other persistent ID providers) (issue #12174);
  • When a new handle is created for a dataset or file that is still a draft, it will be reserved and registered, but not visible publicly. The handle will become visible and the redirects will start working once it is published. This is also in line with how DOI providers work (issue #8881).

Bug Fixes

  • We fixed a defect that caused upload of files larger than 1 GB to fail silently for S3 storage, unless direct upload was used. See #12358.
  • When using the Access API to download files from a dataset that has a guestbook, and when using the dataset's :persistentId instead of its database id, the returned signedUrl immediately failed with a 401 error. This bug was fixed so that the signedUrl correctly returns the files. Using the dataset's and datafile's database ids, instead of its :persistentId, was unaffected by this bug and continues to work. See #12245 and #12363.
  • When the "modifyRegistrationMetadata" API was used, some datasets with UTF-8 characters were being unnecessarily updated in DataCite. This has been resolved by switching from the DataCite MDS API to its JSON API. See the upgrade instructions below, #12070, and #12270.
  • Fixed a problem with S3 direct upload to datasets that use lower- or mixed-case PID authority/identifier in the database and to datasets using an alternative identifier for file storage. See #12268.
  • The bug "Search API doesn't return image_url after newly created dataset is published" was fixed in the following manner. The dataset thumbnail will be created automatically when a dataset is published under the following conditions: the dataset has no existing thumbnail; the dataset has image files that can be converted to a thumbnail the Feature Flag "disable-dataset-thumbnail-autoselect" is not enabled. See #11588 and #12403.
  • A bug in the BagPack generator, which caused the export to fail for datasets with multiple Contact Points, of which some had no name while others did, has been fixed. See #12246.
  • Two guestbook API bugs were fixed. First, missing "gbrids" in the signed URL query parameter list will no longer include "&gbrids=" without a value. Second, when a user attempting to download files with a guestbook response has no API token but is authenticated by bearer token, a temporary api token will be generated with an expiration of 1 minute which is used for signing and decoding the signed URL. See #12267, #12277, and #12279
  • We fixed a timing issue fixed where user only had a few seconds instead of a minute to call the file download API after POSTing the guestbook response. See #12340 and #12341.

Other Changes

  • A list of Dataverse features has been added to the guides, thanks to the Documentation Working Group over many meetings (recorded, if you're interested!). Historically, the list of features has been maintained at dataverse.org/software-features but by moving the list into the guides, we hope to keep it up-to-date with help from the community. To make improvements, please feel free to open issues, make pull requests, discuss in Zulip, and join a Documentation Working Group meeting! See #11998 and #12000.
  • The Big Data Support page has been moved from the Developer Guide to the Installation Guide. See #12151.
  • The deaccession reason "Not a valid dataset." has been changed to "Not valid. This dataset does not comply with repository policies." See the Bundle.properties key file.deaccessionDialog.reason.selectItem.notValid and #11985.

API Updates

  • New APIs have been added called Update the Metadata and Instructions of a Template, Update the License or Terms Of Use of a Template, and Update the Terms Of Access of a Template. See #11912 and #12159.
  • A new API endpoint for getting the default contributor role of a collection was added. See the guides and #12112.
  • A new API has been added to get citations in various formats such as EndNote, RIS, BibTeX, and CSL. Previously this functionality was only available via the UI. See the guides, #11733 and #12238.
  • An API endpoint was added that allows the user to make edits to an existing Guestbook, including adding and removing custom guestbook questions: curl -X PUT -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/guestbooks/{ID}" -d "$JSON". Another API was added that allows the user to retrieve guestbook responses for a specific guestbook within a collection. Optional pagination parameters can be added to limit the number of results, as this can get very large: curl -H "X-Dataverse-key:$API_TOKEN" "$SERVER_URL/api/guestbooks/$ID/responses?limit10&offset=0". See #12386 and #12395.
  • The API endpoint /api/guestbooks/{dataverseAlias}/list can now include "usageCount":# and "responseCount":# in the response by adding the query param "includeStats=true". See the guides, #12260, and #12269.
  • A new query parameter (ignoreSettingExcludeEmailFromExport) has been added to the "View a Dataverse Collection" API (/api/dataverses/{collectionId}). This query parameter prevents the contact emails from being excluded when the setting :ExcludeEmailFromExport is set to true and the user has EditDataverse permissions. See #12194 and #12195.
  • All API endpoints that return information about role assignments (such as /api/dataverses/$ID/assignments) now include additional fields in their JSON responses: assigneeName, roleDescription, definitionPointName, definitionPointType, and definitionPointGlobalId (if available). See #11920.
  • The /datafile/{id}/listRequests API has been extended to show request history. See the guides, #8013 and #12012.
  • "includeInherited" was added as a query parameter of /api/guestbooks/{ID}/list?includeInherited=true to return the collection's guestbooks as well as the guestbooks of the collections' hierarchical owners. Also the "EditDataverse" permission requirement to get the list was removed. See #12323 and #12326.
  • Guestbook response parsing now allows a textarea value to be a string along with an array. See #12446 and #12447.
  • The classes TestIngest (#12415) and Meta (#12417) were removed as they were no longer being used.

Security Updates

This release contains important security updates. If you are not receiving security advisories, please sign up by following the steps in the guides.

As we announced on the mailing list, we are experimenting with sending security advisories via Mailchimp. The sign up steps are the same.

We would like to thank Pablo Picurelli Ortiz, Josh Dow (Klogix) and Steven Adair (Volexity) for notifying us about vulnerabilities that were fixed in this release.

Backward Incompatible Changes

Generally speaking, see the API Changelog for a list of backward-incompatible API changes.

  • The GET /api/access/datafile/{id}/userPermissions endpoint now requires authentication.
  • The GET /api/mydata/retrieve endpoint, if the search returns no data, now includes the "data" block with 0 results. The message that was returned in "error_message" will be returned in "message" and the "success" will be true. All other errors will continue to reply with "success":false and the error message in "error_message". See #11447 and #12256.
  • The Croissant metadata export format has been updated from version 1.0 to 1.1, which is reflected in the "conformsTo" property. The unused "wd" property has been dropped.
  • All endpoints related to storage drivers have been moved out of the Admin API. See #12141 and #12182.
    • The GET, PUT, and DELETE endpoints for /api/admin/dataverse/{alias}/storageDriver have been moved to /api/dataverses/{alias}/storageDriver. Write operations continue to be accessible only to superusers, while GET methods are public.
    • The endpoint /api/admin/dataverse/storageDrivers has been made public, moved, and renamed to /api/dataverses/{alias}/allowedStorageDrivers. Regarding the name change, this endpoint will in the future only display the storage drivers that are allowed on the specified collection. For now, it will display the entire list of available drivers on the installation.

End-Of-Life (EOL) Announcements

PostgreSQL 13 Reached EOL on 13 November 2025

We mentioned this in the Dataverse 6.6, 6.8, 6.9, and 6.10 release notes, but as a reminder, according to https://www.postgresql.org/support/versioning/ PostgreSQL 13 reached EOL on 13 November 2025. As stated in the Installation Guide, we recommend running PostgreSQL 16 since it is the version we test with in our continuous integration (since February 2025). The Dataverse 5.4 release notes explained the upgrade process from 9 to 13 (e.g. pg_dumpall, etc.) and the steps will be similar. If you have any problems, please feel free to reach out (see "getting help" in these release notes).

Notes for Dataverse Installation Administrators

CORS Filter Fix and Docs

We fixed an inconsistency where the CorsFilter was not always being invoked when accessing /api/... endpoints, preventing these endpoints from being used from webapps even when CORS was properly configured. In addition, the documentation related to how certain Dataverse features depend on proper CORS configuration has been extended and improved. See #12151 and #12161.

New Settings

New JVM Options (MicroProfile Config Settings)

  • dataverse.feature.allow-locally-fair-data
  • dataverse.feature.croissant-with-local-reviews

New Database Settings

  • :DatasetSubmitForReviewPopupCustomText
  • :SubmitForReviewDatasetDisclaimerText

Updates for Documentation Writers

The dependencies "sphinx-reredirects" and "sphinx-design" have been added. Please re-run the pip install -r requirements.txt setup step to update your environment. Otherwise you will see a "could not import extension" error.

Updates for Developers

Developers making pull requests against the main "dataverse" project on GitHub can now see for themselves which API tests are failing, if any. Look for a GitHub Action called "Container Integration Tests Workflow". See #9916 and #12368.

New "fast redeploy" scripts have been added for the backend. See the guides, #10156, and Zulip. Please note that this solution is being further improved in #12467.

Complete List of Changes

For the complete list of code changes in this release, see the 6.11 milestone in GitHub.

Getting Help

For help with upgrading, installing, or general questions please see getting help in the Installation Guide.

Installation

If this is a new installation, please follow our Installation Guide. Please don't be shy about asking for help if you need it!

Once you are in production, we would be delighted to update our map of Dataverse installations around the world to include yours! Please create an issue or email us at support@dataverse.org to join the club!

You are also very welcome to join the Global Dataverse Community Consortium (GDCC).

Upgrade Instructions

Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc.

Note: These instructions assume that you are upgrading from the immediate previous version. That is to say, you've already upgraded through all the 6.x releases and are now running Dataverse 6.10.1. See tags on GitHub for a list of versions. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version.

If you are running Payara as a non-root user (and you should be!), remember not to execute the commands below as root. By default, Payara runs as the dataverse user. In the commands below, we use sudo to run the commands as a non-root user.

Also, we assume that Payara is installed in /usr/local/payara7. If not, adjust as needed.

  1. List deployed applications.

    /usr/local/payara7/bin/asadmin list-applications

  2. Undeploy the previous version (should match "list-applications" above).

    /usr/local/payara7/bin/asadmin undeploy dataverse-6.10.1

  3. Deploy the Dataverse 6.11 war file.

    wget https://github.com/IQSS/dataverse/releases/download/v6.11/dataverse-6.11.war

    sudo -u dataverse /usr/local/payara7/bin/asadmin deploy dataverse-6.11.war

  4. Check that you get a version number from Dataverse.

    This is just a sanity check that Dataverse has been deployed properly.

    curl http://localhost:8080/api/info/version

  5. Update the geospatial metadata block.

    wget https://raw.githubusercontent.com/IQSS/dataverse/refs/tags/v6.11/scripts/api/data/metadatablocks/geospatial.tsv

    curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file geospatial.tsv

  6. Update Solr schema.

    Due to changes in the Solr schema (the addition of geospatial fields), updating the Solr schema and reindexing is required.

    First, back up your existing schema.xml file.

    cp /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml.orig

    (Note that Docker-based installations use this directory: solr/data/data/collection1/conf/schema.xml.)

    If you do not have any custom metadata blocks, you can simply download an updated schema.xml file and put it into place.

    wget https://raw.githubusercontent.com/IQSS/dataverse/v6.11/conf/solr/schema.xml

    cp schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf

    Reload the Solr core.

    curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"

    If you do have custom metadata blocks, run the update-fields.sh script that we supply. The example below shows the default path for a non-Docker installation, but adjust the path as necessary.

    wget https://raw.githubusercontent.com/IQSS/dataverse/v6.11/conf/solr/update-fields.sh

    chmod +x update-fields.sh

    curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml

    Reload the Solr core

    curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"

  7. Reindex Solr.

    Solr needs to be reindexed because of the change to deaccessionDialog.reason.selectItem.notValid in Bundle.properties (#11985).

    curl http://localhost:8080/api/admin/index

  8. Re-export metadata export formats.

    We re-export because the Croissant format was updated.

    curl http://localhost:8080/api/admin/metadata/reExportAll

  9. For installations with internationalization or text customizations:

    Please remember to update translations via Dataverse language packs.

    If you have text customizations you can get the latest English files from https://github.com/IQSS/dataverse/tree/v6.11/src/main/java/propertyFiles.

    Note that deaccessionDialog.reason.selectItem.notValid was updated. See #11985.

  10. For installations using DataCite, ensure the DataCite REST API URL (now required) is configured:

    Incremental improvements have been made to the process of registering dataset metadata with DataCite. If your instance is using DataCite, please make sure you have a valid DataCite REST API URL configured, since it is now required.

    The JVM options in question are dataverse.pid.*.datacite.rest-api-url if the recommended, new-style PID configuration is used, or doi.dataciterestapiurlstring if the legacy settings are in place. In the latter case however, this is a good occasion to switch to the new configuration setup.

    For installations using registered DataCite authorities in production the URL should be:

    <jvm-options>-Ddataverse.pid.<providername>.datacite.rest-api-url=https://api.datacite.org</jvm-options>

    Or, for test and development instances:

    <jvm-options>-Ddataverse.pid.<providername>.datacite.rest-api-url=https://api.test.datacite.org</jvm-options>

    See the guides, #12070 and #12270.