Merge pull request IQSS#7729 from IQSS/7400-opendp-download

Auxiliary file download

Author: kcondon

conf/docker-aio/run-test-suite.sh

@@ -8,4 +8,4 @@ fi
 
 # Please note the "dataverse.test.baseurl" is set to run for "all-in-one" Docker environment.
 # TODO: Rather than hard-coding the list of "IT" classes here, add a profile to pom.xml.
-source maven/maven.sh && mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT -Ddataverse.test.baseurl=$dvurl
+source maven/maven.sh && mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT,DuplicateFilesIT,DownloadFilesIT,LinkIT,DeleteUsersIT,DeactivateUsersIT,AuxiliaryFilesIT -Ddataverse.test.baseurl=$dvurl

doc/release-notes/7400-opendp-download.md

@@ -0,0 +1,12 @@
+Auxiliary Files can now be downloaded from the web interface.
+
+- Aux files uploaded as type=DP appear under "Differentially Private Statistics" under file level download. The rest appear under "Other Auxiliary Files".
+
+In addition, related changes were made, including the following:
+
+- New tooltip over the lock indicating if you have been granted access to a restricted file or not.
+- When downloading individual files, you will see "Restricted with Access Granted" or just "Restricted" (followed by "Users may not request access to files.") as appropriate.
+- When downloading individual files, instead of "Download" you should expect to see the file type such as "JPEG Image" or "Original File Format" if the type is unknown.
+- Downloaded aux files now have a file extension if it can be determined.
+
+Please note that the auxiliary files feature is experimental and if you don't need it, its API endpoints can be blocked.

doc/sphinx-guides/source/developers/aux-file-support.rst

@@ -1,11 +1,11 @@
 Auxiliary File Support
 ======================
 
-Auxiliary file support is experimental. Auxiliary files in the Dataverse Software are being added to support depositing and downloading differentially private metadata, as part of the OpenDP project (OpenDP.io). In future versions, this approach may become more broadly used and supported. 
+Auxiliary file support is experimental and as such, related APIs may be added, changed or removed without standard backward compatibility. Auxiliary files in the Dataverse Software are being added to support depositing and downloading differentially private metadata, as part of the OpenDP project (opendp.org). In future versions, this approach will likely become more broadly used and supported.
 
 Adding an Auxiliary File to a Datafile
 --------------------------------------
-To add an auxiliary file, specify the primary key of the datafile (FILE_ID), and the formatTag and formatVersion (if applicable) associated with the auxiliary file. There are two form parameters. "Origin" specifies the application/entity that created the auxiliary file, an "isPublic" controls access to downloading the file. If "isPublic" is true, any user can download the file, else, access authorization is based on the access rules as defined for the DataFile itself.
+To add an auxiliary file, specify the primary key of the datafile (FILE_ID), and the formatTag and formatVersion (if applicable) associated with the auxiliary file. There are multiple form parameters. "Origin" specifies the application/entity that created the auxiliary file, and "isPublic" controls access to downloading the file. If "isPublic" is true, any user can download the file if the dataset has been published, else, access authorization is based on the access rules as defined for the DataFile itself. The "type" parameter is used to group similar auxiliary files in the UI. Currently, auxiliary files with type "DP" appear under "Differentially Private Statistics", while all other auxiliary files appear under "Other Auxiliary Files".
 
 .. code-block:: bash
 
@@ -14,9 +14,10 @@ To add an auxiliary file, specify the primary key of the datafile (FILE_ID), and
   export FILE_ID='12345'
   export FORMAT_TAG='dpJson'
   export FORMAT_VERSION='v1'
+  export TYPE='DP'
   export SERVER_URL=https://demo.dataverse.org
  
-  curl -H X-Dataverse-key:$API_TOKEN -X POST -F "file=@$FILENAME" -F 'origin=myApp' -F 'isPublic=true' "$SERVER_URL/api/access/datafile/$FILE_ID/metadata/$FORMAT_TAG/$FORMAT_VERSION"
+  curl -H X-Dataverse-key:$API_TOKEN -X POST -F "file=@$FILENAME" -F 'origin=myApp' -F 'isPublic=true' -F "type=$TYPE" "$SERVER_URL/api/access/datafile/$FILE_ID/metadata/$FORMAT_TAG/$FORMAT_VERSION"
 
 You should expect a 200 ("OK") response and JSON with information about your newly uploaded auxiliary file.
 
@@ -33,4 +34,4 @@ formatTag and formatVersion (if applicable) associated with the auxiliary file:
   export FORMAT_TAG='dpJson'
   export FORMAT_VERSION='v1'
 
-  curl "$SERVER_URL/api/access/datafile/$FILE_ID/$FORMAT_TAG/$FORMAT_VERSION"
+  curl "$SERVER_URL/api/access/datafile/$FILE_ID/metadata/$FORMAT_TAG/$FORMAT_VERSION"

doc/sphinx-guides/source/user/dataset-management.rst

@@ -179,6 +179,8 @@ Additional download options available for tabular data (found in the same drop-d
 - Data File Citation (currently in either RIS, EndNote XML, or BibTeX format); 
 - All of the above, as a zipped bundle. 
 
+Differentially Private (DP) Metadata can also be accessed for restricted tabular files if the data depositor has created a DP Metadata Release. See :ref:`dp-release-create` for more information.
+
 Astronomy (FITS)
 ----------------
 
@@ -210,6 +212,8 @@ Restricted Files
 
 When you restrict a file it cannot be downloaded unless permission has been granted.
 
+Differentially Private (DP) Metadata can be accessed for restricted tabular files if the data depositor has created a DP Metadata Release. See :ref:`dp-release-create` for more information.
+
 See also :ref:`terms-of-access` and :ref:`permissions`.
 
 Edit Files
@@ -302,6 +306,23 @@ If you restrict any files in your dataset, you will be prompted by a pop-up to e
 
 See also :ref:`restricted-files`.
 
+.. _dp-release-create:
+
+Creating and Depositing Differentially Private Metadata (Experimental)
+----------------------------------------------------------------------
+
+Through an integration with tools from the OpenDP Project (opendp.org), the Dataverse Software offers an experimental workflow that allows a data depositor to create and deposit Differentially Private (DP) Metadata files, which can then be used for exploratory data analysis. This workflow allows researchers to view the DP metadata for a tabular file, determine whether or not the file contains useful information, and then make an informed decision about whether or not to request access to the original file.
+
+If this integration has been enabled in your Dataverse installation, you can follow these steps to create a DP Metadata Release and make it available to researchers, while still keeping the files themselves restricted and able to be accessed after a successful access request.
+
+- Deposit a tabular file and let the ingest process complete
+- Restrict the File
+- In the kebab next to the file on the dataset page, or from the "Edit Files" dropdown on the file page, click "OpenDP Tool"
+- Go through the process to create a DP Metadata Release in the OpenDP tool, and at the end of the process deposit the DP Metadata Release back to the Dataverse installation
+- Publish the Dataset
+
+Once the dataset is published, users will be able to request access using the normal process, but will also have the option to download DP Statistics in order to get more information about the file. 
+
 Guestbook
 ---------
 

doc/sphinx-guides/source/user/find-use-data.rst

@@ -153,6 +153,19 @@ Explore Data
 
 Some file types and datasets offer data exploration options if external tools have been installed. The tools are described in the :doc:`/admin/external-tools` section of the Admin Guide.
 
+Exploratory Data Analysis Using Differentially Private Metadata (Experimental)
+------------------------------------------------------------------------------
+
+Through an integration with tools from the OpenDP Project (opendp.org), the Dataverse Software offers an experimental workflow that allows a data depositor to create and deposit Differentially Private (DP) Metadata files, which can then be used for exploratory data analysis. This workflow allows researchers to view the DP metadata for a tabular file, determine whether or not the file contains useful information, and then make an informed decision about whether or not to request access to the original file.
+
+If the data depositor has made available DP metadata for one or more files in their dataset, these access options will appear on the access dropdown on both the Dataset Page and the File Page. These access options will be available even if a file is restricted. Three types of DP metadata will be available:
+
+- .PDF
+- .XML
+- .JSON
+  
+For more information about how data depositors can enable access using the OpenDP tool, visit the :doc:`/user/dataset-management` section of the User Guide.
+
 .. |image-file-tree-view| image:: ./img/file-tree-view.png
    :class: img-responsive
 .. |image-file-search-facets| image:: ./img/file-search-facets.png

src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java

@@ -1,20 +1,39 @@
 
 package edu.harvard.iq.dataverse;
 
+import edu.harvard.iq.dataverse.util.BundleUtil;
 import java.io.Serializable;
+import java.util.MissingResourceException;
 import javax.persistence.Entity;
 import javax.persistence.GeneratedValue;
 import javax.persistence.GenerationType;
 import javax.persistence.Id;
 import javax.persistence.JoinColumn;
 import javax.persistence.ManyToOne;
+import javax.persistence.NamedNativeQueries;
+import javax.persistence.NamedNativeQuery;
+import javax.persistence.NamedQueries;
+import javax.persistence.NamedQuery;
 
 /**
  *
  * @author ekraffmiller 
  * Represents a generic file that is associated with a dataFile.
  * This is a data representation of a physical file in StorageIO
  */
+@NamedQueries({
+    @NamedQuery(name = "AuxiliaryFile.lookupAuxiliaryFile",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId and o.formatTag = :formatTag and o.formatVersion = :formatVersion"),
+    @NamedQuery(name = "AuxiliaryFile.findAuxiliaryFiles",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId"),
+    @NamedQuery(name = "AuxiliaryFile.findAuxiliaryFilesByType",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId and o.type = :type"),
+    @NamedQuery(name = "AuxiliaryFile.findAuxiliaryFilesWithoutType",
+            query = "select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId and o.type is null"),})
+@NamedNativeQueries({
+    @NamedNativeQuery(name = "AuxiliaryFile.findAuxiliaryFileTypes",
+            query = "select distinct type from auxiliaryfile where datafile_id = ?1")
+})
 @Entity
 public class AuxiliaryFile implements Serializable {
 
@@ -44,6 +63,12 @@ public class AuxiliaryFile implements Serializable {
 
     private String checksum;
 
+    /**
+     * A way of grouping similar auxiliary files together. The type could be
+     * "DP" for "Differentially Private Statistics", for example.
+     */
+    private String type;
+
     public Long getId() {
         return id;
     }
@@ -115,6 +140,21 @@ public String getChecksum() {
     public void setChecksum(String checksum) {
         this.checksum = checksum;
     }
-    
-    
+
+    public String getType() {
+        return type;
+    }
+
+    public void setType(String type) {
+        this.type = type;
+    }
+
+    public String getTypeFriendly() {
+        try {
+            return BundleUtil.getStringFromPropertyFile("file.auxfiles.types." + type, "Bundle");
+        } catch (MissingResourceException ex) {
+            return null;
+        }
+    }
+
 }

src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java

@@ -8,13 +8,16 @@
 import java.io.InputStream;
 import java.security.DigestInputStream;
 import java.security.MessageDigest;
+import java.util.ArrayList;
+import java.util.List;
 import java.util.logging.Logger;
 import javax.ejb.EJB;
 import javax.ejb.Stateless;
 import javax.inject.Named;
 import javax.persistence.EntityManager;
 import javax.persistence.PersistenceContext;
 import javax.persistence.Query;
+import javax.persistence.TypedQuery;
 import org.apache.tika.Tika;
 
 /**
@@ -28,7 +31,7 @@ public class AuxiliaryFileServiceBean implements java.io.Serializable {
    private static final Logger logger = Logger.getLogger(AuxiliaryFileServiceBean.class.getCanonicalName());
 
     @PersistenceContext(unitName = "VDCNet-ejbPU")
-    private EntityManager em;
+    protected EntityManager em;
 
     @EJB
     private SystemConfig systemConfig;
@@ -54,9 +57,11 @@ public AuxiliaryFile save(AuxiliaryFile auxiliaryFile) {
      * @param formatVersion - to distinguish between multiple versions of a file
      * @param origin - name of the tool/system that created the file
      * @param isPublic boolean - is this file available to any user?
+     * @param type how to group the files such as "DP" for "Differentially
+     * Private Statistics".
      * @return success boolean - returns whether the save was successful
      */
-    public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile dataFile, String formatTag, String formatVersion, String origin, boolean isPublic) {
+    public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile dataFile, String formatTag, String formatVersion, String origin, boolean isPublic, String type) {
 
         StorageIO<DataFile> storageIO =null;
         AuxiliaryFile auxFile = new AuxiliaryFile();
@@ -81,6 +86,7 @@ public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile
             auxFile.setFormatVersion(formatVersion);
             auxFile.setOrigin(origin);
             auxFile.setIsPublic(isPublic);
+            auxFile.setType(type);
             auxFile.setDataFile(dataFile);         
             auxFile.setFileSize(storageIO.getAuxObjectSize(auxExtension));
             auxFile = save(auxFile);
@@ -101,7 +107,7 @@ public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile
 
     public AuxiliaryFile lookupAuxiliaryFile(DataFile dataFile, String formatTag, String formatVersion) {
 
-        Query query = em.createQuery("select object(o) from AuxiliaryFile as o where o.dataFile.id = :dataFileId and o.formatTag = :formatTag and o.formatVersion = :formatVersion");
+        Query query = em.createNamedQuery("AuxiliaryFile.lookupAuxiliaryFile");
 
         query.setParameter("dataFileId", dataFile.getId());
         query.setParameter("formatTag", formatTag);
@@ -114,4 +120,73 @@ public AuxiliaryFile lookupAuxiliaryFile(DataFile dataFile, String formatTag, St
         }
     }
 
+    public List<AuxiliaryFile> findAuxiliaryFiles(DataFile dataFile) {
+        TypedQuery query = em.createNamedQuery("AuxiliaryFile.findAuxiliaryFiles", AuxiliaryFile.class);
+        query.setParameter("dataFileId", dataFile.getId());
+        return query.getResultList();
+    }
+
+    /**
+     * @param inBundle If true, only return types that are in the bundle. If
+     * false, only return types that are not in the bundle.
+     */
+    public List<String> findAuxiliaryFileTypes(DataFile dataFile, boolean inBundle) {
+        List<String> allTypes = findAuxiliaryFileTypes(dataFile);
+        List<String> typesInBundle = new ArrayList<>();
+        List<String> typeNotInBundle = new ArrayList<>();
+        for (String type : allTypes) {
+            // Check if type is in the bundle.
+            String friendlyType = getFriendlyNameForType(type);
+            if (friendlyType != null) {
+                typesInBundle.add(type);
+            } else {
+                typeNotInBundle.add(type);
+            }
+        }
+        if (inBundle) {
+            return typesInBundle;
+        } else {
+            return typeNotInBundle;
+        }
+    }
+
+    public List<String> findAuxiliaryFileTypes(DataFile dataFile) {
+        Query query = em.createNamedQuery("AuxiliaryFile.findAuxiliaryFileTypes");
+        query.setParameter(1, dataFile.getId());
+        return query.getResultList();
+    }
+
+    public List<AuxiliaryFile> findAuxiliaryFilesByType(DataFile dataFile, String typeString) {
+        TypedQuery query = em.createNamedQuery("AuxiliaryFile.findAuxiliaryFilesByType", AuxiliaryFile.class);
+        query.setParameter("dataFileId", dataFile.getId());
+        query.setParameter("type", typeString);
+        return query.getResultList();
+    }
+
+    public List<AuxiliaryFile> findOtherAuxiliaryFiles(DataFile dataFile) {
+        List<AuxiliaryFile> otherAuxFiles = new ArrayList<>();
+        List<String> otherTypes = findAuxiliaryFileTypes(dataFile, false);
+        for (String typeString : otherTypes) {
+            TypedQuery query = em.createNamedQuery("AuxiliaryFile.findAuxiliaryFilesByType", AuxiliaryFile.class);
+            query.setParameter("dataFileId", dataFile.getId());
+            query.setParameter("type", typeString);
+            List<AuxiliaryFile> auxFiles = query.getResultList();
+            otherAuxFiles.addAll(auxFiles);
+        }
+        otherAuxFiles.addAll(findAuxiliaryFilesWithoutType(dataFile));
+        return otherAuxFiles;
+    }
+
+    public List<AuxiliaryFile> findAuxiliaryFilesWithoutType(DataFile dataFile) {
+        Query query = em.createNamedQuery("AuxiliaryFile.findAuxiliaryFilesWithoutType", AuxiliaryFile.class);
+        query.setParameter("dataFileId", dataFile.getId());
+        return query.getResultList();
+    }
+
+    public String getFriendlyNameForType(String type) {
+        AuxiliaryFile auxFile = new AuxiliaryFile();
+        auxFile.setType(type);
+        return auxFile.getTypeFriendly();
+    }
+
 }

src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java

@@ -280,6 +280,14 @@ private void redirectToBatchDownloadAPI(String multiFileString, Boolean download
         redirectToBatchDownloadAPI(multiFileString, true, downloadOriginal);
     }
 
+    public void redirectToAuxFileDownloadAPI(Long fileId, String formatTag, String formatVersion) {
+        String fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata/" + formatTag + "/" + formatVersion;
+        try {
+            FacesContext.getCurrentInstance().getExternalContext().redirect(fileDownloadUrl);
+        } catch (IOException ex) {
+            logger.info("Failed to issue a redirect to aux file download url (" + fileDownloadUrl + "): " + ex);
+        }
+    }
 
     /**
      * Launch an "explore" tool which is a type of ExternalTool such as