diff --git a/docs/assets/images/userguides/storage/depot_smb_snapshots.png b/docs/assets/images/userguides/storage/depot_smb_snapshots.png new file mode 100644 index 00000000..61f85dcb Binary files /dev/null and b/docs/assets/images/userguides/storage/depot_smb_snapshots.png differ diff --git a/docs/assets/images/userguides/storage/depot_snapshots_ls.png b/docs/assets/images/userguides/storage/depot_snapshots_ls.png new file mode 100644 index 00000000..59848019 Binary files /dev/null and b/docs/assets/images/userguides/storage/depot_snapshots_ls.png differ diff --git a/docs/assets/images/userguides/storage/hipaa-dm.png b/docs/assets/images/userguides/storage/hipaa-dm.png new file mode 100644 index 00000000..3bacc79d Binary files /dev/null and b/docs/assets/images/userguides/storage/hipaa-dm.png differ diff --git a/docs/assets/pdfs/Research_Data_Depot_white_text.pdf b/docs/assets/pdfs/Research_Data_Depot_white_text.pdf new file mode 100644 index 00000000..30f3e4fc Binary files /dev/null and b/docs/assets/pdfs/Research_Data_Depot_white_text.pdf differ diff --git a/docs/assets/pdfs/hsi_9.3_reference_manual.pdf b/docs/assets/pdfs/hsi_9.3_reference_manual.pdf new file mode 100644 index 00000000..a7f39fa7 Binary files /dev/null and b/docs/assets/pdfs/hsi_9.3_reference_manual.pdf differ diff --git a/docs/snippets/compress.md b/docs/snippets/compress.md new file mode 100644 index 00000000..3caee335 --- /dev/null +++ b/docs/snippets/compress.md @@ -0,0 +1,74 @@ +# Archive and Compression + +There are several options for archiving and compressing groups of files or directories. The most commonly used options are: + +## tar + +See the [official documentation for tar](http://www.gnu.org/software/tar/tar.html) for more information. + +Saves many files together into a single archive file, and restores individual files from the archive. Includes automatic archive compression/decompression options and special features for incremental and full backups. + +Examples: + +``` + (list contents of archive somefile.tar) +$ tar tvf somefile.tar + + (extract contents of somefile.tar) +$ tar xvf somefile.tar + + (extract contents of gzipped archive somefile.tar.gz) +$ tar xzvf somefile.tar.gz + + (extract contents of bzip2 archive somefile.tar.bz2) +$ tar xjvf somefile.tar.bz2 + + (archive all ".c" files in current directory into one archive file) +$ tar cvf somefile.tar *.c + + (archive and gzip-compress all files in a directory into one archive file) +$ tar czvf somefile.tar.gz somedirectory/ + + (archive and bzip2-compress all files in a directory into one archive file) +$ tar cjvf somefile.tar.bz2 somedirectory/ +``` + +Other arguments for `tar` can be explored by using the `man tar` command. + +## gzip + +See the [official documentation for gzip](http://www.gnu.org/software/gzip/gzip.html) for more information. + +The standard compression system for all GNU software. + +Examples: + +``` + (compress file somefile - also removes uncompressed file) +$ gzip somefile + + (uncompress file somefile.gz - also removes compressed file) +$ gunzip somefile.gz +``` + +## bzip2 + +See the [official documentation for bzip](http://www.bzip.org/) for more information. + +Strong, lossless data compressor based on the Burrows-Wheeler transform. Stronger compression than gzip. + +Examples: + +``` + (compress file somefile - also removes uncompressed file) +$ bzip2 somefile + + (uncompress file somefile.bz2 - also removes compressed file) +$ bunzip2 somefile.bz2 +``` + +There are several other, less commonly used, options available as well: + +- zip +- 7zip +- xz diff --git a/docs/snippets/sharing_globus.md b/docs/snippets/sharing_globus.md new file mode 100644 index 00000000..ce22c304 --- /dev/null +++ b/docs/snippets/sharing_globus.md @@ -0,0 +1,7 @@ +Data on any RCAC resource can be shared with other users within Purdue or with collaborators at other institutions. Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. + +To share files, login to [https://transfer.rcac.purdue.edu](https://transfer.rcac.purdue.edu/), navigate to the endpoint (collection) of your choice, and follow instructions as described in Globus documentation on how to share data: + +- + +See also [RCAC Globus presentation](https://www.rcac.purdue.edu/training/globus). diff --git a/docs/userguides/boxfolder/accounts.md b/docs/userguides/boxfolder/accounts.md new file mode 100644 index 00000000..6e091b60 --- /dev/null +++ b/docs/userguides/boxfolder/accounts.md @@ -0,0 +1,37 @@ +--- +tags: + - Box Research Lab Folder +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Accounts on Box Research Lab Folders + +## Access to Box Research Lab Folder + +### Obtaining an Account + +All Purdue faculty, staff, and students have a Box.com account for personal files. + +Research projects requiring regulated storage are assigned group data storage space within Box Research Lab Folder. Project PIs may authorize additional users to collaborate within the folder. + +Box research lab Folders **are not** intended to store data protected by Federal privacy and security laws. REED Folders are designed to align with the NIST SP 800-171 standard, and are approved for storing L3 projects requiring HIPAA-aligned storage. Compliance with additional regulations such as CUI, ITAR, EAR, or FERPA are under review. + +Neither Purdue or Box set any limits on the total amount or number of files that you may store within your Box Research Lab Folder. However, there are several restrictions on the nature of files you may store: + +- Any one individual file may be no larger than 15 GB. + +### Outside Collaborators + +Within the bounds of your data use agreements, IRB protocols, or technology control plans, Box Research Lab Folders data may be shared with Purdue or non-Purdue collaborators. + +### Login to Box to Access your Research Lab Folder + +Access to your Box Research Lab Folder is done using your Purdue Career Account by visiting [purdue.box.com](https://purdue.box.com). + +### Transferring Files into Box + +Files may be up/downloaded from Box via [FTPS](https://support.box.com/hc/en-us/articles/360043697414-Using-Box-with-FTP-or-FTPS). diff --git a/docs/userguides/boxfolder/faqs.md b/docs/userguides/boxfolder/faqs.md new file mode 100644 index 00000000..f7a1c62f --- /dev/null +++ b/docs/userguides/boxfolder/faqs.md @@ -0,0 +1,86 @@ +--- +tags: + - Box Research Lab Folder +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Frequently Asked Questions + +Frequently asked questions about Box Research Lab Folder. + +## About Box Research Lab Folder + +### What is a Box Managed Account and REED Folder? + +A Box Managed account is your Purdue managed Box account. A REED folder is a project folder created in secure folder structured within Box. + +### Does Purdue have a Business Associate Agreement with Box? + +Box supports the HIPAA and HITECH regulations, as well as the ability to sign HIPAA Business Associate Agreements (BAAs) with customers. Customers who are required by law to comply with HIPAA, such as HIPAA Covered Entities and HIPAA Business Associates, must have an Enterprise or Elite account with Box and sign a HIPAA Business Associate Agreement (BAA). + +- Purdue University signed a Business Associate Agreement with Box, and Box accounts are Enterprise. +- Box provides the ability to use O365 online office application to open and work with content. Purdue University has signed a Business Associate Agreement with Microsoft. +- All other third-party service providers with our instance of Box Enterprise are not covered by a Business Associate Agreement (BAA). + +Do not use third-party applications to process data considered Restricted via your Purdue managed Box account. Purdue University does not maintain a contractual agreement with the vendor, and the required security controls are not in place. + +## Data + +### Can I integrate 3rd-Party Apps with my Box folder? + +While official Box add-on applications are approved for use with Purdue Box.com folders, third-party apps have not been reviewed for impact to security and are not approved for use until a security review has been completed by IT Security and Policy. If the official Box apps cannot provide the functionality you need, you may request a security review of the third-party app you are considering by submitting an email including the app name and the functionality you are addressing with the app to itpolicyreq@purdue.edu. + +### Can I use Unmanaged Box accounts, and other Cloud Storage Options? + +Unmanaged (free and commercial) Box accounts and all other Cloud Storage options are not approved for storing or sharing sensitive or restricted data. Purdue University does not maintain a contractual agreement with the vendor, and the required security controls are not in place. Personal use of cloud storage can continue, however sensitive and restricted data must not be stored in the account. The table below will help you determine where to store your data. + +| Classification | Personal Box Folder | REED Folder | +| --- | --- | --- | +| Individually Identifiable Health Information | No | Determined by Review | +| Limited Data Set | No | Yes | +| De-Identified Data | No | Yes | +| De-Identified Data with Contractual Requirements | No | Yes | + +### Should I use Box Drive if I work with Restricted Data? + +Currently, Box Drive is not recommended for use to work with restricted data. However, if you have a business need for Box Drive, contact IT Security and Policy so we can help you understand the risk and ensure you have the appropriate controls in place to protect the data. + +### Will Box Drive or Box Edit cache files on the local machine? + +Box Drive does locally cache files that you have opened. Box Drive's cache size limit is based on your free disk space (50% of available space) and has a maximum limit of 25 GB. If you reach this limit, Box Drive begins removing files, starting with those files that have gone the longest without your accessing them. Also, if a cached file has a new version created on Box, Box Drive discards the locally cached version. Do not open data in Box Drive containing restricted data. More details: [Technical Information for Box Drive Administrators](https://community.box.com/t5/Deploying-Managing-Box-Drive/Technical-Information-for-Box-Drive-Administrators/ta-p/37454) + +Box Edit does download the file to a cache when you are modifying a file. The cache can be cleared. See: [Clear Your Box Edit Cache](https://support.box.com/hc/en-us/articles/360043695814-Clear-Your-Box-Edit-Cache) + +### Can I use FTP to access Box? + +No, FTP is disabled. + +### Can I delete my files permanently from Box? + +Individuals can delete a file or folder, and the item will be placed in the Trash. Individuals will not be able to permanently delete the items. An automated process is in place to remove data older than 90 days. Once the item is purged, it can't be recovered using the built in Box tools. + +### What are the recommended permission levels? + +| Role | Permission Level | +| --- | --- | +| Project Team member who needs full control | Editor | +| Project Team members who just needs to work within Box | Viewer Uploader | +| Project Team member who needs Read/Download access | Viewer | +| Partner who needs Read access | Previewer | +| Partner who need to upload data | Uploader | + +### How do I host sensitive or restricted data in Box? + +Storage of restricted data in Box is subject to review from Purdue's IT Security and Policy group (ITSP). Access may also require approval from campus offices and committees responsible for contractual compliance and research regulatory affairs. + +Purdue's managed Box environment can be used to store, de-identified data and limited data sets in scope of HIPAA for research. Fully identified data is subject to a security review and approval process. + +All restricted data must be stored in a REED Folder. Do not use Box for an actual medical practice. + +See the data map below for more details: + +![HIPAA data map](/assets/images/userguides/storage/hipaa-dm.png) \ No newline at end of file diff --git a/docs/userguides/boxfolder/index.md b/docs/userguides/boxfolder/index.md new file mode 100644 index 00000000..a1c9d97b --- /dev/null +++ b/docs/userguides/boxfolder/index.md @@ -0,0 +1,16 @@ +--- +tags: + - Box Research Lab Folder +authors: + - jin456 +search: + boost: 2 +--- + +# Box Research Lab Folder User Guide + +A Box Research Lab Folder is a managed storage solution built on top of the Box.com cloud platform, for research labs to share and collaborate within the lab and with outside collaborators. + +- [**Box Research Lab Folder Overview**](overview.md) +- [**Accounts**](accounts.md) +- [**Frequently Asked Questions**](faqs.md) diff --git a/docs/userguides/boxfolder/overview.md b/docs/userguides/boxfolder/overview.md new file mode 100644 index 00000000..985f7e8e --- /dev/null +++ b/docs/userguides/boxfolder/overview.md @@ -0,0 +1,53 @@ +--- +tags: + - Box Research Lab Folder +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Box Research Lab Folder Overview + +Research labs requiring a lab collaboration space can request Box Research Lab Folders by emailing [rcac-help@purdue.edu](mailto:rcac-help@purdue.edu). + +## Box Research Lab Folder Features + +A Box Research Lab Folder offers research groups in need of centralized data storage for research collaboration many unique features and benefits: + +- **Available** + + To any Purdue research group requiring data storage space for research collaboration. + +- **Accessible** + + Easily accessible through your web browser, and facilitates easy sharing with collaborators within Purdue and without. + +- **Capable** + + A Box Research Lab folder facilitates joint work on shared files across your research group, avoiding the need for numerous copies of datasets across individuals' private storage. It is an ideal place to store a project's data and documents. A Lab Folder can store unlimited data, with a maximum size of a single file up to 15 GB. + +- **Controllable Access** + + Access management is under your direct control, within the bounds of the appropriate data use agreements, IRB protocols, or technology control plans. + +- **Data Retention** + + All data kept in the Box Research Lab Folder remains owned by the research group's lead faculty. Files kept in the project's lab folder remain with the research group, unaffected by turnover, and could head off potentially difficult disputes. Data is also retained and protected in alignment with the appropriate data security standards governing your research projects. + +- **Never Purged** + + A Box Research Lab Folder is never subject to purging. + +- **Reliable** + + Box Research Lab Folders are built on Purdue's centrally-managed Box.com service, which is a highly-available, secure cloud storage platform. + +- **Restricted Data** + + Box Research Lab Folders are not designed to align with the NIST SP 800-171 standard, and are not approved for storing L3 projects requiring HIPAA-aligned storage. Please request a REED Folder for sensitive or restricted research data. + +## Box Research Lab Folder Hardware Details + +Box Research Lab Folders are built on Box.com, an enterprise-grade cloud platform for file storage and collaboration. diff --git a/docs/userguides/depot/faqs.md b/docs/userguides/depot/faqs.md new file mode 100644 index 00000000..203db268 --- /dev/null +++ b/docs/userguides/depot/faqs.md @@ -0,0 +1,75 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Frequently Asked Questions + +Frequently asked questions about Data Depot. + +## About Data Depot + +### Can you remove me from the Data Depot mailing list? + +Your subscription in the Data Depot mailing list is tied to your account on Data Depot. If you are no longer using your account on Data Depot, your account can be deleted from the [My Accounts](https://www.rcac.purdue.edu/account/myinfo) page. Hover over the resource you wish to remove yourself from and click the red 'X' button. Your account and mailing list subscription will be removed overnight. Be sure to make a copy of any data you wish to keep first. + +### What sort of performance should I expect to and from the Data Depot? + +The Data Depot is designed to be a high-capacity, fast, reliable and secure data storage system for research data. During acceptance testing, a number of performance baselines were measured: + +| Access type | Large file, reading | Large file, writing | Many small files, reading | Many small files, writing | +|---|---|---|---|---| +| CIFS access, single client (GigE) | 102.1 MB/sec | 71.64 MB/sec | 12.43 MB/sec | 11.57 MB/sec | + +### Is the Data Depot just a file server? + +The Data Depot is a suite of file service tools, specifically targeted at the needs of an academic research lab. More than just the file service infrastructure and hardware, the Data Depot also encompasses self-service access management, permissions control, and file sharing with [RCAC Globus Server](http://transfer.rcac.purdue.edu). + +### Do I need to do anything to my firewall to access Data Depot? + +No firewall changes are needed to access Data Depot. However, to access data through [Network Drives / CIFS](storage/transfer.md#windows-network-drive--smb) (i.e., CIFS, "Z: Drive"), you must be on a Purdue campus network or connected through [VPN](https://it.purdue.edu/services/vpn.php). + +### What is the best way to mount Data Depot in my lab? + +You can mount your Data Depot space via [Network Drives / CIFS](storage/transfer.md#windows-network-drive--smb) using your Purdue Career Account credentials. + +See the [Windows Network Drive / SMB](storage/transfer.md#windows-network-drive--smb) documentation for step-by-step instructions for Windows, Mac OS X, and Linux. + +### How do Data Depot, Fortress, and PURR relate to each other? + +The Data Depot, Fortress, and PURR, are complementary parts of Purdue's infrastructure for working with research data. The Data Depot is designed for large, actively-used, persistent research data; Fortress is intended for long-term, archival storage of data and results; and PURR is for management, curation, and long-term preservation of research data. + + +## Data + +### Can I store Export-controlled data on Data Depot? + +The Data Depot is not approved for storing data requiring Export control including ITAR, FISMA, DFAR-7012, NIST 800-171. + +Please contact the [Export Control Office](http://www.purdue.edu/research/research-compliance/export-control/) to discuss technology control plans and data storage appropriate for export controlled projects. + +### Can I store HIPAA data on Data Depot? + +The Data Depot is not approved for storing data covered by HIPAA. Please contact the [HIPAA Compliance Office](http://www.purdue.edu/hipaa/) to discuss HIPAA-compliant data storage. + +### What do I need to do in order to store non-HIPAA human subjects data in the Data Depot? + +Use the following IRB-approved text in your IRB documentation when describing your data safeguards, substituting the PI's name for "PROFESSORNAME": + +!!!info + Only individuals specifically approved by PROFESSORNAME may access project data in the Research Data Depot. All membership in the PROFESSORNAME group is authorized by the project PI(s) and/or designees. Purdue University has network firewalls and other security devices to protect the Research Data Depot infrastructure from outside the campus. + + Purdue Career accounts have password security policies that enforce age and quality requirements. + + Auditing is enabled on Research Data Depot fileservers to track login attempts, maintain logs, and generate reports of access attempts. + +### Can I share data with outside collaborators? + +Yes! Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data: + +- diff --git a/docs/userguides/depot/index.md b/docs/userguides/depot/index.md new file mode 100644 index 00000000..24282c2a --- /dev/null +++ b/docs/userguides/depot/index.md @@ -0,0 +1,18 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +--- + +# Data Depot User Guide + +The Data Depot is a high-capacity, fast, reliable and secure data storage service designed, configured and operated for the needs of Purdue researchers in any field and shareable with both on-campus and off-campus collaborators. + +- [**Data Depot Overview**](overview.md) +- [**File Storage and Transfer**](storage/index.md) +- [**Lost File Recovery**](recover/index.md) +- [**Access Permissions and Directories**](permissions/index.md) +- [**Frequently Asked Questions**](faqs.md) diff --git a/docs/userguides/depot/overview.md b/docs/userguides/depot/overview.md new file mode 100644 index 00000000..dfe2aaff --- /dev/null +++ b/docs/userguides/depot/overview.md @@ -0,0 +1,60 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Data Depot Overview + +As with the community clusters, research labs will be able to easily purchase capacity in the Data Depot through the [Data Depot Purchase](https://www.rcac.purdue.edu/purchase) page on this site. For more information, please [contact us](/contact). + +## Data Depot Features + +The Data Depot offers research groups in need of centralized data storage unique features and benefits. + +- **Available** + + To any Purdue research group as a purchase in increments of 1 TB at a competitive annual price or you may [request a 100 GB trial space](https://www.rcac.purdue.edu/orders/products?category=3) free of charge. **Participation in the Community Cluster program is not required.** + +- **Accessible** + + - As a [Windows or Mac OS X network drive](storage/transfer.md#windows-network-drive--smb) on personal and lab computers on campus. + - Directly on Community Cluster nodes. + - From other universities or labs [through Globus](./storage/transfer.md#globus). + +- **Capable** + + The Data Depot facilitates joint work on shared files across your research group, avoiding the need for numerous copies of datasets across individuals' home or scratch directories. It is an ideal place to store group applications, tools, scripts, and documents. + +- **Controllable Access** + + Access management is under your direct control. Unix groups can be created for your group and staff can assist you in setting appropriate permissions to allow exactly the access you want and prevent any you do not. Easily manage who has access through [a simple web application](https://www.rcac.purdue.edu/account/myinfo) — the same application used to manage access to Community Cluster queues. + +- **Data Retention** + + All data kept in the Data Depot remains owned by the research group's lead faculty. When researchers or students leave your group, any files left in their home directories may become difficult to recover. Files kept in Data Depot remain with the research group, unaffected by turnover, and could head off potentially difficult disputes. + +- **Never Purged** + + The Data Depot is never subject to purging. + +- **Reliable** + + The Data Depot is redundant and protected against hardware failures and accidental deletion. All data is mirrored at two different sites on campus to provide for greater reliability and to protect against physical disasters. + +- **Restricted Data** + + The Data Depot **is** suitable for non-HIPAA human subjects data. See the Data Depot [FAQ](faqs.md#what-do-i-need-to-do-in-order-to-store-non-hipaa-human-subjects-data-in-the-data-depot) for a data security statement for your IRB documentation. The Data Depot is **not** approved for regulated data, including HIPAA, ePHI, FISMA, or ITAR data. + +[Download Data Depot Features Guide](/assets/pdfs/Research_Data_Depot_white_text.pdf) + +## Data Depot Hardware Details + +The Data Depot uses an enterprise-class GPFS storage solution with an initial total capacity of over 2 PB. This storage is redundant and reliable, features regular snapshots, and is globally available on all RCAC systems. The Data Depot is non-purged space suitable for tasks such as sharing data, editing files, developing and building software, and many other uses. Built on Data Direct Networks' SFA12k storage platform, the Data Depot has redundant storage arrays in multiple campus datacenters for maximum availability. + +!!! note + **While the Data Depot will scale well for most uses, it is recommended to continue using each cluster's parallel scratch filesystem for use as high-performance working space (scratch) for running jobs.** diff --git a/docs/userguides/depot/permissions/default.md b/docs/userguides/depot/permissions/default.md new file mode 100644 index 00000000..79dc425d --- /dev/null +++ b/docs/userguides/depot/permissions/default.md @@ -0,0 +1,65 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Default Configuration + +This is what a default configuration looks like for a research group called "mylab": + +``` +/depot/mylab/ + +--apps/ + | + +--data/ + | + +--etc/ + | +--bashrc + | +--cshrc + | + (other subdirectories) +``` + +The `/depot/mylab/` directory is the main top-level directory for all your research group storage. All files are to be kept within one of the subdirectories of this, based on your specific access requirements. These subdirectories will be created after consulting with you as to exactly what you need. + +By default, the following subdirectories, with the following access and use models, will be created. All of these details can be changed to suit the particular needs of your research group. + +- **`data/`** + + Intended for read and write use by a limited set of people chosen by the research group's managers. Restricted to not be readable or writable by anyone else. + + !!!info + This is frequently used as **an open space for storage of shared research data**. + +- `apps/` + + Intended for write use by a limited set of people chosen by the research group's managers. + + Restricted to not be writable by anyone else. + + Allows read and execute by anyone who has access to any cluster queues owned by the research group and anyone who has other file permissions granted by the research group (such as "data" access above). + + !!!info + This is frequently used as **a space for central management of shared research applications**. + +- `etc/` + + Intended for write use by a limited set of people chosen by the research group's managers (by default, the same as for "apps" above). + + Restricted to not be writable by anyone else. + + Allows read and execute by anyone who has access to any cluster queues owned by the research group and anyone who has other file permissions granted by the research group (such as "data" access above). + + !!!info + This is frequently used as **a space for central management of shared startup/login scripts, environment settings, aliases, etc**. + +- `etc/bashrc` and `etc/cshrc` + + Examples of group-wide shell startup files. Group members can source these from their own `$HOME/.bashrc` or `$HOME/.cshrc` files so they would automatically pick up changes to their environment needed to work with applications and data for the research group. + +- Additional subdirectories can be created as needed in the top and/or any of the lower levels. Just [contact support](mailto:rcac-help@purdue.edu) and we will be happy to figure out what will work best for your needs. diff --git a/docs/userguides/depot/permissions/index.md b/docs/userguides/depot/permissions/index.md new file mode 100644 index 00000000..8a2db6aa --- /dev/null +++ b/docs/userguides/depot/permissions/index.md @@ -0,0 +1,19 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Access Permissions and Directories + +Data Depot is very flexible in the access permission models and directory structures it can support. New spaces on Data Depot are given a default access model and directory structure at setup time. This can be modified as needed to support your workflows. + +Information follows about the default structure and common access models. + +- [**Default Configuration**](default.md) +- [**Common Access Permission Scenarios**](scenarios.md) +- [**Storage Access Unix Groups**](unixgroups.md) diff --git a/docs/userguides/depot/permissions/scenarios.md b/docs/userguides/depot/permissions/scenarios.md new file mode 100644 index 00000000..22a9f739 --- /dev/null +++ b/docs/userguides/depot/permissions/scenarios.md @@ -0,0 +1,38 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Common Access Permission Scenarios + +Depending on your research group's specific needs and preferred way of sharing, there are various permission models your Data Depot can be designed to reflect. Here are some common scenarios for access: + +- **"We have privately shared data within the group and some software for use only by us and a few collaborators."** + + *Suggested implementation:* + + Keep data in the `data/` subdirectory and limit read and write access to select approved researchers. + Keep applications (if any) in the `apps/` subdirectory and limit write access to your developers and/or application stewards. + Allow read/execute to `apps/` by anyone in the larger research group with cluster queue access and approved collaborators. + +- **"We have privately shared data within the group and some software which is needed by all cluster users (not just our group or known collaborators)."** + + *Suggested implementation:* + + Keep data in the `data/` subdirectory and limit read and write access to select approved researchers. + Keep applications (if any) in the `apps/` subdirectory and limit write access to your developers and/or application stewards. + Allow read/execute to `apps/` by anyone at all by opening read/execute permissions on your base Data Depot directory. + +- **"We have a few different projects and only the PI and respective project members should have any access to files for each project."** + + *Suggested implementation:* + + Create distinct subdirectories within your Data Depot base directory for each project and corresponding Unix groups for read/write access to each. + Approve specific researchers for read and write access to only the projects they are working on. + +Many variants and combinations of the above are also possible covering the range from "very restrictive" to "mostly open" in terms of both read and write access to each subdirectory within your Data Depot space. Your lab can sit down with our staff and explain your specific needs in human terms, and then we can help you implement those requirements in actual permissions and groups. Once the initial configuration is done, you will then be able to easily add or remove access for your people. If your needs change, just let us know and we can accommodate your new requirements as well. diff --git a/docs/userguides/depot/permissions/unixgroups.md b/docs/userguides/depot/permissions/unixgroups.md new file mode 100644 index 00000000..04f3c536 --- /dev/null +++ b/docs/userguides/depot/permissions/unixgroups.md @@ -0,0 +1,34 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Storage Access Unix Groups + +To enable a wide variety of access permissions, users are assigned one or more auxiliary Unix groups. It is the combination of this Unix group membership and the r/w/x permission bits on subdirectories that allow fine-tuning who can and cannot do what within specific areas of your Data Depot. These Unix groups will generally closely match the name of your Data Depot root directory and the name of the subdirectory to which write access is being given. For example, write access to `/depot/mylab/data/` is controlled by membership in the `mylab-data` Unix group. + +There is also one Unix group which has the name of the base directory of your Data Depot, `mylab`. This group serves to limit read/execute access to your base `/depot/mylab/` directory and also helps to define the read/execute permissions of some of the subdirectories within. This Unix group is composed of the union of the following: + +- all members of your more specific Unix groups +- all users authorized to access any of your research group's cluster queues +- any other specific individuals you may have approved + +Research group faculty and their designees may directly manage membership in these Unix groups, and by extension, the storage access permissions they grant, through the online web application. + +- [User Management](https://www.rcac.purdue.edu/account/groups) + +## Checking Your Group Membership + +As a user you can check which groups you are a member of by issuing the `groups` command while logged into any RCAC resource. You can also look on the website at [https://www.rcac.purdue.edu/account/groups](https://www.rcac.purdue.edu/account/groups). + +``` +$ groups +mylab mylab-apps mylab-data +``` + +If you have recently been added to a group you need to log out and then back in again before the permissions changes take effect. diff --git a/docs/userguides/depot/recover/flost.md b/docs/userguides/depot/recover/flost.md new file mode 100644 index 00000000..abf84fcf --- /dev/null +++ b/docs/userguides/depot/recover/flost.md @@ -0,0 +1,25 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# flost + +If you know when you lost the file, the easiest way is to use the `flost` command. This tool is available from any RCAC resource. If you do not have access to a compute cluster, any Data Depot user may use an SSH client to connect to `datadepot.rcac.purdue.edu` and run this command. + +To run the tool you will need to specify the location where the lost file was with the `-w` argument: + +``` +$ flost -w /depot/mylab +``` + +Replace `mylab` with the name of your lab's Data Depot directory. If you know more specifically where the lost file was you may provide the full path to that directory. + +This tool will prompt you for the date on which you lost the file or would like to recover the file from. If the tool finds an appropriate snapshot it will provide instructions on how to search for and recover the file. + +If you are not sure what date you lost the file you may try entering different dates into `flost` to try to find the file, or you may also [manually browse](manual.md) the snapshots as described in the Manual Browsing section. diff --git a/docs/userguides/depot/recover/index.md b/docs/userguides/depot/recover/index.md new file mode 100644 index 00000000..a881baee --- /dev/null +++ b/docs/userguides/depot/recover/index.md @@ -0,0 +1,29 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Lost File Recovery + +Data Depot is protected against accidental file deletion through a series of snapshots taken every night just after midnight. Each snapshot provides the state of your files at the time the snapshot was taken. It does so by storing only the files which have changed between snapshots. A file that has not changed between snapshots is only stored once but will appear in every snapshot. This is an efficient method of providing snapshots because the snapshot system does not have to store multiple copies of every file. + +These snapshots are kept for a limited time at various intervals. **RCAC keeps nightly snapshots for 7 days, weekly snapshots for 4 weeks, and monthly snapshots for 3 months. This means you will find snapshots from the last 7 nights, the last 4 Sundays, and the last 3 first of the months.** Files are available going back between two and three months, depending on how long ago the last first of the month was. Snapshots beyond this are not kept. + +**Only files which have been saved during an overnight snapshot are recoverable.** If you lose a file the same day you created it, the file is **NOT** recoverable because the snapshot system has not had a chance to save the file. + +**Snapshots are not a substitute for regular backups.** It is the responsibility of the researchers to back up any important data to the [Fortress Archive](https://www.rcac.purdue.edu/storage/fortress). Data Depot **does** protect against hardware failures or physical disasters through other means, however these other means are also **NOT** substitutes for backups. + +!!!note + Files in scratch directories are not recoverable. Files in scratch directories are not backed up. If you accidentally delete a file, a disk crashes, or old files are purged, they cannot be restored. + +Data Depot offers several ways for researchers to access snapshots of their files. + +- [**flost Tool**](flost.md) +- [**Manual Browsing**](manual.md) +- [**Windows**](windows.md) +- [**Mac OS X**](mac.md) \ No newline at end of file diff --git a/docs/userguides/depot/recover/mac.md b/docs/userguides/depot/recover/mac.md new file mode 100644 index 00000000..a078e8ff --- /dev/null +++ b/docs/userguides/depot/recover/mac.md @@ -0,0 +1,30 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Mac OS X + +Mac OS X does not provide any way to access the Data Depot snapshots directly. To access the snapshots there are two options: browse the snapshots by hand through a network drive mount or use an automated command-line based tool. + +To browse the snapshots by hand, follow the directions outlined in the [Manual Browsing](manual.md) section. + +To use the automated command-line tool, log into a compute cluster or into the host `datadepot.rcac.purdue.edu` (which is available to all Data Depot users) and use the [flost](flost.md) tool. On Mac OS X you can use the built-in SSH terminal application to connect. + +- Open the Applications folder from Finder. +- Navigate to the Utilities folder. +- Double click the Terminal application to open it. +- Type the following command when the terminal opens. + + ``` + $ ssh username@datadepot.rcac.purdue.edu + ``` + + Replace `username` with your Purdue career account username and provide your password when prompted. + +Once logged in use the [flost](flost.md) tool as described above. The tool will guide you through the process and show you the commands necessary to retrieve your lost file. diff --git a/docs/userguides/depot/recover/manual.md b/docs/userguides/depot/recover/manual.md new file mode 100644 index 00000000..3ee90b6b --- /dev/null +++ b/docs/userguides/depot/recover/manual.md @@ -0,0 +1,29 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Manual Browsing + +You may also search through the snapshots by hand on the Data Depot filesystem if you are not sure what date you lost the file or would like to browse by hand. Snapshots can be browsed from any RCAC resource. If you do not have access to a compute cluster, any Data Depot user may use an SSH client to connect to `datadepot.rcac.purdue.edu` and browse from there. The snapshots are located at `/depot/.snapshots` on these resources. + +You can also mount the snapshot directory over Samba (or SMB, CIFS) on Windows or Mac OS X. Mount (or map) the snapshot directory in the [same way](../storage/transfer.md#windows-network-drive--smb) as you did for your main Data Depot space substituting the server name and path for `\\datadepot.rcac.purdue.edu\depot\.winsnaps` (Windows) or `smb://datadepot.rcac.purdue.edu/depot/.winsnaps` (Mac OS X). + +Once connected to the snapshot directory through SSH or Samba, you will see something similar to this. Snapshots folders may look slightly differently when accessed via SSH on `datadepot.rcac.purdue.edu` or via Samba on `datadepot.rcac.purdue.edu`. Here are examples of both. + +| SSH to `datadepot.rcac.purdue.edu` | Samba mount on `datadepot.rcac.purdue.edu` | +|---|---| +| ![Listing Depot Snapshots](/assets/images/userguides/storage/depot_snapshots_ls.png) | ![Depot Snapshots in SMB](/assets/images/userguides/storage/depot_smb_snapshots.png) | + +Each of these directories is a snapshot of the entire Data Depot filesystem at the timestamp encoded into the directory name. The format for this timestamp is year, two digits for month, two digits for day, followed by the time of the day. + +You may `cd` into any of these directories where you will find the entire Data Depot filesystem. Use `cd` to continue into your lab's Data Depot space and then you may browse the snapshot as normal. + +If you are browsing these directories over a Samba network drive you can simply drag and drop the files over into your live Data Depot folder. + +Once you find the file you are looking for, use `cp` to copy the file back into your lab's live Data Depot space. **Do not attempt to modify files directly in the snapshot directories.** diff --git a/docs/userguides/depot/recover/windows.md b/docs/userguides/depot/recover/windows.md new file mode 100644 index 00000000..db792a5f --- /dev/null +++ b/docs/userguides/depot/recover/windows.md @@ -0,0 +1,25 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Windows + +Windows provides a built-in interface for recovering previous versions of files stored on network drives, including Data Depot. This method allows you to browse and restore snapshots directly through File Explorer without any additional tools. + +## Recovery Steps + +1. Open the folder that contained the lost file in File Explorer. +2. Right-click on the folder and select "Properties". +3. Navigate to the "Previous Versions" tab. +4. A list of available snapshots will be displayed. Each entry shows two columns: the timestamp when the snapshot was taken, and the modification date of files within that snapshot. +5. Select the appropriate snapshot that contains the version of the file you want to restore. +6. A new window will open showing the contents of the folder at that point in time. +7. Locate your desired file and drag and drop it to restore it to your current folder. + +This recovery method allows Windows users to directly restore files through the operating system interface without requiring additional tools or command-line access. diff --git a/docs/userguides/depot/storage/compress.md b/docs/userguides/depot/storage/compress.md new file mode 100644 index 00000000..a50a2881 --- /dev/null +++ b/docs/userguides/depot/storage/compress.md @@ -0,0 +1,11 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +--8<-- "docs/snippets/compress.md" diff --git a/docs/userguides/depot/storage/index.md b/docs/userguides/depot/storage/index.md new file mode 100644 index 00000000..bf23c057 --- /dev/null +++ b/docs/userguides/depot/storage/index.md @@ -0,0 +1,17 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# File Storage and Transfer + +Data Depot supports several methods for file storage and transfer. + +- [**Archive and Compression**](compress.md) +- [**Sharing Files**](sharing.md) +- [**File Transfer**](transfer.md) diff --git a/docs/userguides/depot/storage/sharing.md b/docs/userguides/depot/storage/sharing.md new file mode 100644 index 00000000..845d8358 --- /dev/null +++ b/docs/userguides/depot/storage/sharing.md @@ -0,0 +1,29 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Sharing + +Data Depot supports several methods for file sharing. + +## Globus + +--8<-- "docs/snippets/sharing_globus.md" + +## WWW + +Your research group can easily share static files (images, data, HTML) from your depot space via the WWW. + +- [Contact support](mailto:rcac-help@purdue.edu) to set up a "www" folder in your Data Depot space. +- Copy any files that you wish to share via the WWW into your Data Depot space's "www" folder. + - For example, `cp /path/to/image.jpg /depot/mylab/www/`, where `mylab` is your research group name. + - Or upload to `smb://datadepot.rcac.purdue.edu/depot/mylab/www`, where `mylab` is your research group name. +- Your file is now accessible via your web browser at the URL `https://www.datadepot.rcac.purdue.edu/mylab/image.jpg`. + +Note that it is not possible to run web sites, dynamic content, interpreters (PHP, Perl, Python), or CGI scripts from this web site. diff --git a/docs/userguides/depot/storage/transfer.md b/docs/userguides/depot/storage/transfer.md new file mode 100644 index 00000000..ea043fda --- /dev/null +++ b/docs/userguides/depot/storage/transfer.md @@ -0,0 +1,101 @@ +--- +tags: + - Data Depot +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# File Transfer + +Data Depot supports several methods for file transfer. + +## Globus + +*Globus*, previously known as Globus Online, is a powerful and easy to use file transfer service for transferring files virtually anywhere. It works within RCAC's various research storage systems; it connects between RCAC and remote research sites running Globus; and it connects research systems to personal systems. You may use Globus to connect to your home, scratch, and Fortress storage directories. Since Globus is web-based, it works on any operating system that is connected to the internet. The Globus Personal client is available on Windows, Linux, and Mac OS X. It is primarily used as a graphical means of transfer but it can also be used over the command line. + +### Globus Web + +- Navigate to +- Click "Proceed" to log in with your Purdue Career Account. +- On your first login it will ask to make a connection to a Globus account. Accept the conditions. +- Now you are at the main screen. Click "File Transfer" which will bring you to a two-panel interface (if you only see one panel, you can use the selector in the top-right corner to switch the view). +- You will need to select one collection and file path on one side as the source, and the second collection on the other as the destination. This can be one of several Purdue endpoints, or another University, or even your personal computer (see [Personal Client section](#globus-personal-client-setup) below). + +The RCAC collections are as follows. A search for "Purdue" will give you several suggested results you can choose from, or you can give a more specific search. + +- Home Directory storage: *"Purdue Research Computing - Home Directories"* +- **Research Data Depot:** *"Purdue Research Computing - Data Depot"* — a search for "Depot" should provide appropriate matches to choose from. +- **Fortress:** *"Purdue Fortress HPSS Archive"* — a search for "Fortress" should provide appropriate matches to choose from. + +From here, select a file or folder in either side of the two-pane window, and then use the arrows in the top-middle of the interface to instruct Globus to move files from one side to the other. You can transfer files in either direction. You will receive an email once the transfer is completed. + +### Globus Personal Client Setup + +Globus Connect Personal is a small software tool you can install to make your own computer a Globus endpoint on its own. It is useful if you need to transfer files via Globus to and from your computer directly. + +- On the "Collections" page from earlier, click *"Get Globus Connect Personal"* or download a version for your operating system from here: [Globus Connect Personal](https://www.globus.org/globus-connect-personal) +- Name this particular personal system and follow the setup prompts to create your Globus Connect Personal endpoint. +- Your personal system is now available as a collection within the Globus transfer interface. + +### Globus Command Line + +Globus supports a command line interface, allowing advanced automation of your transfers. + +To use the recommended standalone Globus CLI application (the `globus` command): + +- First time use: issue the `globus login` command and follow instructions for initial login. +- Commands for interfacing with the CLI can be found via [Using the Command Line Interface](https://docs.globus.org/cli/), as well as the [Globus CLI Examples](https://docs.globus.org/cli/examples/) pages. + +### Sharing Data with Outside Collaborators + +Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data: + +- + +For links to more information, please see [Globus Support](https://support.globus.org/home) page and [RCAC Globus presentation](https://www.rcac.purdue.edu/training/globus). + +## Windows Network Drive / SMB + +*SMB* (Server Message Block), also known as CIFS, is an easy-to-use file transfer protocol that is useful for transferring files between RCAC systems and a desktop or laptop. You may use SMB to connect to your home, scratch, and Fortress storage directories. The SMB protocol is available on Windows, Linux, and Mac OS X. It is primarily used as a graphical means of transfer, but it can also be used over the command line. + +Note: to access Data Depot through SMB file sharing, you must be on a Purdue campus network or connected through [VPN](http://www.itap.purdue.edu/connections/vpn/). + +### Windows + +- Windows 7: Click Windows menu > Computer, then click Map Network Drive in the top bar +- Windows 8 & 10: Tap the Windows key, type `computer`, select This PC, click Computer > Map Network Drive in the top bar +- Windows 11: Tap the Windows key, type `File Explorer`, select This PC, click Computer > Map Network Drive in the top bar +- In the folder location, enter the following information and click Finish: + - To access your Data Depot directory, enter `\\datadepot.rcac.purdue.edu\depot\mylab` where `mylab` is your research group name. + - Use your career account login name and password when prompted. (You will **not** need to add "`,push`" nor use your Purdue Duo client.) + +- Your Data Depot directory should now be mounted as a drive in the Computer window. +- If you have issues mapping the drive, check Credential Manager: + - In the Search bar, type "Credential Manager" + - Inside Credential Manager, click "Windows Credentials" and "Add New Credential" + - Servername: `\\datadepot.rcac.purdue.edu` + - Username: `boilerad\PurdueLogin` + - Then save it and try mapping again. If there is already an entry for the server you are trying to map, delete it first. + +### Mac OS X + +- In the Finder, click Go > Connect to Server +- In the Server Address enter the following information and click Connect: + - To access your Data Depot directory, enter `smb://datadepot.rcac.purdue.edu/depot/mylab` where `mylab` is your research group name. + - Use your career account login name and password when prompted. (You will **not** need to add "`,push`" nor use your Purdue Duo client.) +- Your Data Depot directory should now be mounted as a drive in the Computer window. + +### Linux + +- There are several graphical methods to connect in Linux depending on your desktop environment. Once you find out how to connect to a network server on your desktop environment, choose the Samba/SMB protocol and adapt the information from the Mac OS X section to connect. +- If you would like access via samba on the command line you may install `smbclient` which will give you FTP-like access and can be used as shown below. + + ``` + smbclient //datadepot.rcac.purdue.edu/depot/ -U boilerad/username + cd mylab + ``` + +- Note: Use your career account login name and password when prompted. (You will **not** need to add "`,push`" nor use your Purdue Duo client.) diff --git a/docs/userguides/fortress/accounts.md b/docs/userguides/fortress/accounts.md new file mode 100644 index 00000000..f00c4ac4 --- /dev/null +++ b/docs/userguides/fortress/accounts.md @@ -0,0 +1,49 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# Accounts on Fortress + +## Obtaining an Account + +All Purdue faculty, staff, and students participating in the Community Cluster program have access to Fortress along with their cluster nodes and scratch space. Your home directory on Fortress is the default directory in which your archive files are stored and will appear as `/home/username`, but this is not the same directory as your home directory on any other Purdue IT systems. Your home directory on Fortress is your long-term storage directory for all Purdue IT systems. + +Research groups are assigned a group data storage space within Fortress with each Data Depot group space. Faculty should [request a Data Depot trial](https://www.rcac.purdue.edu/purchase#datadepot) to create a shared Fortress space for their research group. The actual location for the group space on Fortress is `/group/mylab/` (replace `mylab` with your group name). + +!!! info + RCAC computing resources are not intended to store data protected by Federal privacy and security laws (e.g., HIPAA, ITAR, classified, etc.). It is the responsibility of the faculty partner to ensure that no protected data is stored on the systems. + +- Particularly in the case of group storage, please keep in mind that such spaces are, by design, accessible by others and should not be used to store private information such as grades, login credentials, or personal data. + +Fortress sets no limits on the amount or number of files that you may store. However, there are several restrictions on the nature of files you may store: + +- Many small files: Fortress is a tape archive and works best with a few, large files. Large sets of small files should be compressed into archives with utilities such as [htar](storage/transfer/htar.md). Other technical limitations are [detailed on the Fortress FAQs](faqs.md#what-limitations-does-fortress-have). +- Backing up individual or departmental computers. Fortress is intended to be a research data store and not a personal or enterprise backup solution. + +Additionally, while Fortress access is included with RCAC services, storing more than 1 PB of data may incur a cost recovery charge. + +## Outside Collaborators + +Your Departmental Business Office can submit a [Request for Privileges (R4P)](https://service.purdue.edu/TDClient/32/Purdue/KB/Article/33/What-is-a-Request-for-Privileges-R4P) to provide access to collaborators outside Purdue, including recent graduates. + +## Login & Keytabs + +It is not possible to login directly to Fortress via SSH or SCP. You may access your files there efficiently using HSI, HTAR, or SFTP. + +A *keytab* file is required to log into Fortress via HSI or HTAR. However, **all RCAC systems may access Fortress without any keytab preparation.** If for some reason you lose your keytab, you may easily regenerate one on any RCAC system by running the command `fortresskey`. + +However, to access Fortress from a **personal or departmental computer**, you will need to first copy your keytab file to the computer you wish to use. This keytab can be found in your research home directory, within the hidden subdirectory named ".private" as the file "hpss.unix.keytab" (`~/.private/hpss.unix.keytab`). This keytab will allow you to access HPSS services without needing to type a password and will remain valid for 90 days. Your keytab on RCAC systems will automatically be regenerated after this time, and you will need to re-copy the new keytab file to any other computers you use to directly access Fortress then. + +If you do not have an account on any RCAC systems other than Fortress, you will need to generate a keytab file using the web interface: + +- [Web Form to Generate New Fortress Keytab](https://www.rcac.purdue.edu/fortresskey) + +!!! warning "Warning" + Using this invalidates your existing keytab on RCAC systems in the `~/.private/hpss.unix.keytab` file. diff --git a/docs/userguides/fortress/faqs.md b/docs/userguides/fortress/faqs.md new file mode 100644 index 00000000..26b1cd08 --- /dev/null +++ b/docs/userguides/fortress/faqs.md @@ -0,0 +1,140 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# Frequently Asked Questions + +Frequently asked questions about Fortress. + +## About Fortress + +### Can you remove me from the Fortress mailing list? + +Your subscription in the Fortress mailing list is tied to your account on Fortress. If you are no longer using your account on Fortress, your account can be deleted from the [My Accounts](https://www.rcac.purdue.edu/account/myinfo) page. Hover over the resource you wish to remove yourself from and click the red 'X' button. Your account and mailing list subscription will be removed overnight. Be sure to make a copy of any data you wish to keep first. + +### Do I need to do anything to my firewall to access Fortress? + +Yes, any machines using HSI or HTAR must have all firewalls (local and departmental) configured to allow open access from the following IP addresses: + +``` +128.211.138.154 +128.211.138.155 +128.211.138.156 +128.211.138.157 +128.211.138.158 +128.211.138.159 +128.211.138.160 +128.211.138.161 +128.211.138.162 +128.211.138.163 +``` + +Firewall issues may manifest with error messages like "[put: Error -50 on transfer](#hsihtar-put-error--5-on-transfer)." If you are unsure of how to modify your firewall settings, please consult with your department's IT support or the documentation for your operating system. Access to Fortress is restricted to on-campus networks. If you need to directly access Fortress from off-campus, please use the Purdue VPN service before connecting. + +!!!info + The list of IP addresses changes occasionally as machines are added or retired. The list above is current, so if you have other IP addresses in your firewall, they can be safely removed. In particular, any IPs in the range 128.211.138.40-128.211.138.48 can be removed. + +### Can I download HSI or HTAR binaries for my OS platform? + +Yes, visit the [Downloads](https://www.rcac.purdue.edu/downloads/archive#hsi) page to download HSI or HTAR packages for your operating system. + +!!!info + **Note:** If your username on your desktop does not match your career account username, HSI and HTAR require configuration to connect using your career account username: + +- For HSI, use the `-l careeraccount` option on the `hsi` command line. +- For HTAR, set the `HPSS_PRINCIPAL` environment variable to your career account username: + + bash: `export HPSS_PRINCIPAL=careeraccount` + + csh/tcsh: `setenv HPSS_PRINCIPAL=careeraccount` + +### What limitations does Fortress have? + +Fortress has a few limitations that you should keep in mind: + +- Fortress does not support direct FTP or SCP transfers. SFTP connections are supported. +- Fortress does not support Unicode filenames. All filenames must contain only ASCII characters. +- Fortress does not support sparse files. +- Fortress is a tape archive. While it can handle use case of *"multitude of small files"*, performance may be severely decreased (compared to a much preferred case of *"fewer files of much larger size"*). If you need to store a large number of small files, we strongly recommend that you bundle them up first (with `zip`, `tar`, `htar`, etc) before placing resulting archive into Fortress. Note: a *"small file"* on Fortress scale is typically considered something under 30-50MB per file. +- HTAR has an individual file size limit of 64GB. If any files you are trying to archive with HTAR are greater than 64GB, then HTAR will immediately fail. This does not limit the number of files in the archive or the total overall size of the archive. To get around this limitation, try using the `htar_large` command. It is slower than using HTAR but it will work around the 64GB file size limit. + +## Data + +### What is the best way to access my data? + +[HSI](storage/transfer/hsi.md) and [HTAR](storage/transfer/htar.md): HSI provides a FTP-style interface taking advantage of the power of HPSS without requiring any special user knowledge. HTAR is a utility to aggregate a set of files into a single tar archive directly into Fortress, without requiring space to first create an archive. + +### Can I set up a shared space for my research group to share data? + +Research groups are assigned a group data storage space within Fortress with each Data Depot group space. Faculty should [request a Data Depot trial](https://www.rcac.purdue.edu/purchase#datadepot) to create a shared Fortress space for their research group. + +!!!note + RCAC resources are not intended to store data protected by Federal privacy and security laws (e.g., HIPAA, ITAR, classified, etc.). It is the responsibility of the faculty partner to ensure that no protected data is stored on the systems. + +Please keep in mind that such spaces are, by design, accessible by others and should not be used to store private information such as grades, login credentials, or personal data. [Contact us](mailto:rcac-help@purdue.edu) to create a group space for your group. + +### How can I verify the contents of my archives while using HTAR? + +You can ask HTAR to verify the contents of an archive during or after creation using the `-Hverify` switch. Please see the [HTAR](storage/transfer/htar.md#htar-archive-verification) documentation for details. + +### HSI/HTAR: put: Error -5 on transfer + +First, check your firewall settings, and ensure that there are no firewall rules interfering with connecting to Fortress. For firewall configuration, please see "[Do I need to do anything to my firewall to access Fortress?](#do-i-need-to-do-anything-to-my-firewall-to-access-fortress)" **If firewalls are not responsible:** + +Open the file named `/etc/hosts` on your workstation, especially if you run a Debian or Ubuntu Linux distribution. Look for a line like: + +``` +127.0.1.1 hostname.dept.purdue.edu hostname +``` + +Replace the IP address `127.0.1.1` with the real IP address for your system. If you don't know your IP address, you can find it with the command: + +``` +host `hostname --fqdn` +``` + +### Can I share data with outside collaborators? + +Yes! Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data: + +- + +### HSI/HTAR: Unable to authenticate user with remote gateway (error 2 or 9) + +There could be a variety of such errors, with wordings along the lines of: + +``` +Could not initialize keytab on remote server. +result = -2, errno = 2rver connection +*** hpssex_OpenConnection: Unable to authenticate user with remote gateway at 128.211.138.40.1217result = -2, errno = 9 +Unable to setup communication to HPSS... +ERROR (main) unable to open remote gateway server connection +HTAR: HTAR FAILED +``` + +and + +``` +*** hpssex_OpenConnection: Unable to authenticate user with remote gateway at 128.211.138.40.1217result = -11000, errno = 9 +Unable to setup communication to HPSS... +*** HSI: error opening logging +Error - authentication/initialization failed +``` + +The root cause for these errors is an expired or non-existent keytab file (a special authentication token stored in your home directory). These keytabs are valid for 90 days and on most RCAC resources they are usually automatically checked and regenerated when you execute `hsi` or `htar` commands. However, if the keytab is invalid, or fails to generate, Fortress may be unable to authenticate you and you would see the above errors. This is especially common on those RCAC clusters that have their own dedicated home directories (such as Bell), or on standalone installations (such as if you downloaded and installed HSI and HTAR on your non-RCAC computer). + +*This is a temporary problem and a permanent system-wide solution is being developed.* In the interim, the recommended workaround is to generate a new valid keytab file in your main research computing home directory, and then copy it to your home directory on Fortress. The `fortresskey` command is used to generate the keytab and can be executed on another cluster or a dedicated data management host `data.rcac.purdue.edu`: + +``` +$ ssh username@data.rcac.purdue.edu fortresskey +$ scp -pr username@data.rcac.purdue.edu:~/.private $HOME +``` + +With a valid keytab in place, you should then be able to use `hsi` and `htar` commands to access Fortress. Note that only one keytab can be valid at any given time (i.e. if you regenerated it, you may have to copy the new keytab to all systems that you intend to use `hsi` or `htar` from if they do not share the main research computing home directory). diff --git a/docs/userguides/fortress/index.md b/docs/userguides/fortress/index.md new file mode 100644 index 00000000..4ab12d79 --- /dev/null +++ b/docs/userguides/fortress/index.md @@ -0,0 +1,18 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +--- + +# Fortress User Guide + +The Fortress system is a large, long-term, multi-tiered file caching and storage system utilizing both online disk and robotic tape drives. + +- [**Fortress Overview**](overview.md) +- [**Accounts on Fortress**](accounts.md) +- [**File Storage and Transfer**](storage/index.md) +- [**Frequently Asked Questions**](faqs.md) diff --git a/docs/userguides/fortress/overview.md b/docs/userguides/fortress/overview.md new file mode 100644 index 00000000..ec2a612c --- /dev/null +++ b/docs/userguides/fortress/overview.md @@ -0,0 +1,43 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# Fortress Overview + +Fortress uses a [Spectra Logic TFinity robotic tape library](https://youtu.be/I3quB7uQPRM) with a capacity of over 200 PB. + +## Hardware Overview + +| Storage Subsystem | Current Capacity | Hardware | +|---|---|---| +| Disk Cache | Over 800 TB | 2 Seagate 5U84 Storage arrays | +| Long-Term Storage | Over 200 PB | LTO-9 Robotic Tape Library | + +All files stored on Fortress appear in at least two separate storage devices: + +- One copy is permanently on tape. +- Recently used or files smaller than 100MB have their primary copy stored on a conventional spinning disk storage array (disk cache). Disk cache provides a rapid restore time. + +Both primary and secondary copies of larger files reside on separate tape cartridges in the robotic tape library. After a period of inactivity, HPSS will migrate files from disk cache to tape. + +Fortress writes two copies of every file either to two tapes, or to disk and a tape, to protect against medium errors. Unfortunately, Fortress does not automatically switch to the alternate copy when it has trouble accessing the primary. If it seems to be taking an extraordinary amount of time to retrieve a file (hours), please [contact support](mailto:rcac-help@purdue.edu). We can then investigate why it is taking so long. If it is an error on the primary copy, we will instruct Fortress to switch to the alternate copy as the primary and recreate a new alternate copy. + +## Fortress Storage Quota + +There is currently no quota on Fortress disk use. Fortress users will receive a monthly email report showing your current Fortress usage. + +Files belonging to deleted accounts will also be retained, but inaccessible except by special request after the accounts have been terminated. The files will be kept for no more than ten years or the usability of the media on which they are stored, whichever comes first. + +## Fortress File Recovery + +Data on Fortress is not backed up elsewhere in a traditional sense. New and modified files in the disk cache are migrated to tape within 30 minutes, and Fortress maintains two copies of every file on different media to protect against media failures, but there is no backup protecting against accidental deletions. + +!!! warning "Warning" + If you remove or overwrite a file on Fortress, it is gone. You cannot request to have it retrieved. diff --git a/docs/userguides/fortress/storage/index.md b/docs/userguides/fortress/storage/index.md new file mode 100644 index 00000000..fdccc0f6 --- /dev/null +++ b/docs/userguides/fortress/storage/index.md @@ -0,0 +1,20 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +--- + +# File Storage and Transfer + +Your home directory on Fortress is the default directory in which your archive files are stored. + +On Fortress, your home directory will appear as `/home/username`, but this is not the same directory as your home directory on any other Purdue IT systems. Your home directory on Fortress is your long-term storage directory for all Purdue IT systems. + +The following link will take you to more information about transferring files in and out of Fortress. + +- [**Sharing Files**](sharing.md) +- [**File Transfer**](transfer/index.md) diff --git a/docs/userguides/fortress/storage/sharing.md b/docs/userguides/fortress/storage/sharing.md new file mode 100644 index 00000000..92702659 --- /dev/null +++ b/docs/userguides/fortress/storage/sharing.md @@ -0,0 +1,23 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +--- + +# Sharing Files + +## Sharing Fortress Data with Globus + +Fortress supports Globus for file sharing. + +Data on any RCAC resource can be shared with other users within Purdue or with collaborators at other institutions. Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. + +To share files, login to [https://transfer.rcac.purdue.edu](https://transfer.rcac.purdue.edu/), navigate to the endpoint (collection) of your choice, and follow instructions as described in Globus documentation on how to share data: + +- + +See also [RCAC Globus presentation](https://www.rcac.purdue.edu/training/globus). \ No newline at end of file diff --git a/docs/userguides/fortress/storage/transfer/globus.md b/docs/userguides/fortress/storage/transfer/globus.md new file mode 100644 index 00000000..6f006fb7 --- /dev/null +++ b/docs/userguides/fortress/storage/transfer/globus.md @@ -0,0 +1,55 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# Globus + +*Globus*, previously known as Globus Online, is a powerful and easy to use file transfer service for transferring files virtually anywhere. It works within RCAC's various research storage systems; it connects between RCAC and remote research sites running Globus; and it connects research systems to personal systems. You may use Globus to connect to your home, scratch, and Fortress storage directories. Since Globus is web-based, it works on any operating system that is connected to the internet. The Globus Personal client is available on Windows, Linux, and Mac OS X. It is primarily used as a graphical means of transfer but it can also be used over the command line. + +## Globus Web + +- Navigate to +- Click "Proceed" to log in with your Purdue Career Account. +- On your first login it will ask to make a connection to a Globus account. Accept the conditions. +- Now you are at the main screen. Click "File Transfer" which will bring you to a two-panel interface (if you only see one panel, you can use the selector in the top-right corner to switch the view). +- You will need to select one collection and file path on one side as the source, and the second collection on the other as the destination. This can be one of several Purdue endpoints, or another University, or even your personal computer (see [Personal Client section](#globus-personal-client-setup) below). + +The RCAC collections are as follows. A search for "Purdue" will give you several suggested results you can choose from, or you can give a more specific search. + +- Home Directory storage: *"Purdue Research Computing - Home Directories"* +- **Research Data Depot:** *"Purdue Research Computing - Data Depot"* — a search for "Depot" should provide appropriate matches to choose from. +- **Fortress:** *"Purdue Fortress HPSS Archive"* — a search for "Fortress" should provide appropriate matches to choose from. + +From here, select a file or folder in either side of the two-pane window, and then use the arrows in the top-middle of the interface to instruct Globus to move files from one side to the other. You can transfer files in either direction. You will receive an email once the transfer is completed. + +## Globus Personal Client Setup + +Globus Connect Personal is a small software tool you can install to make your own computer a Globus endpoint on its own. It is useful if you need to transfer files via Globus to and from your computer directly. + +- On the "Collections" page from earlier, click *"Get Globus Connect Personal"* or download a version for your operating system from here: [Globus Connect Personal](https://www.globus.org/globus-connect-personal) +- Name this particular personal system and follow the setup prompts to create your Globus Connect Personal endpoint. +- Your personal system is now available as a collection within the Globus transfer interface. + +## Globus Command Line + +Globus supports a command line interface, allowing advanced automation of your transfers. + +To use the recommended standalone Globus CLI application (the `globus` command): + +- First time use: issue the `globus login` command and follow instructions for initial login. +- Commands for interfacing with the CLI can be found via [Using the Command Line Interface](https://docs.globus.org/cli/), as well as the [Globus CLI Examples](https://docs.globus.org/cli/examples/) pages. + +## Sharing Data with Outside Collaborators + +Globus allows convenient sharing of data with outside collaborators. Data can be shared with collaborators' personal computers or directly with many other computing resources at other institutions. See the Globus documentation on how to share data: + +- + +For links to more information, please see [Globus Support](https://support.globus.org/home) page and [RCAC Globus presentation](https://www.rcac.purdue.edu/training/globus). diff --git a/docs/userguides/fortress/storage/transfer/hsi.md b/docs/userguides/fortress/storage/transfer/hsi.md new file mode 100644 index 00000000..3f616947 --- /dev/null +++ b/docs/userguides/fortress/storage/transfer/hsi.md @@ -0,0 +1,92 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# HSI + +*HSI*, the Hierarchical Storage Interface, is the preferred method of transferring files to and from Fortress. HSI is designed to be a friendly interface for users of the High Performance Storage System (HPSS). It provides a familiar Unix-style environment for working within HPSS while automatically taking advantage of high-speed, parallel file transfers without requiring any special user knowledge. + +HSI is provided on all research systems as the command `hsi`. HSI is also available for [download](https://www.rcac.purdue.edu/downloads/archive#hsi) for many operating systems. + +Interactive usage: + +``` +$ hsi + +************************************************************************* +* Purdue University +* High Performance Storage System (HPSS) +************************************************************************* +* This is the Purdue Data Archive, Fortress. For further information +* see http://www.rcac.purdue.edu/storage/fortress/ +* +* If you are having problems with HPSS, please call IT/Operational +* Services at 49-44000 or send E-mail to rcac-help@purdue.edu. +* +************************************************************************* +Username: username UID: 12345 Acct: 12345(12345) Copies: 1 Firewall: off [hsi.3.5.8 Wed Sep 21 17:31:14 EDT 2011] + +[Fortress HSI]/home/username->put data1.fits +put 'test' : '/home/username/test' ( 1024000000 bytes, 250138.1 KBS (cos=11)) + +[Fortress HSI]/home/username->lcd /tmp + +[Fortress HSI]/home/username->get data1.fits +get '/tmp/data1.fits' : '/home/username/data1.fits' (2011/10/04 16:28:50 1024000000 bytes, 325844.9 KBS ) + +[Fortress HSI]/home/username->quit +``` + +Batch transfer file: + +``` +put data1.fits +put data2.fits +put data3.fits +put data4.fits +put data5.fits +put data6.fits +put data7.fits +put data8.fits +put data9.fits +``` + +Batch usage: + +``` +$ hsi < my_batch_transfer_file +************************************************************************* +* Purdue University +* High Performance Storage System (HPSS) +************************************************************************* +* This is the Purdue Data Archive, Fortress. For further information +* see http://www.rcac.purdue.edu/storage/fortress/ +* +* If you are having problems with HPSS, please call IT/Operational +* Services at 49-44000 or send E-mail to rcac-help@purdue.edu. +* +************************************************************************* +Username: username UID: 12345 Acct: 12345(12345) Copies: 1 Firewall: off [hsi.3.5.8 Wed Sep 21 17:31:14 EDT 2011] +put 'data1.fits' : '/home/username/data1.fits' ( 1024000000 bytes, 250200.7 KBS (cos=11)) +put 'data2.fits' : '/home/username/data2.fits' ( 1024000000 bytes, 258893.4 KBS (cos=11)) +put 'data3.fits' : '/home/username/data3.fits' ( 1024000000 bytes, 222819.7 KBS (cos=11)) +put 'data4.fits' : '/home/username/data4.fits' ( 1024000000 bytes, 224311.9 KBS (cos=11)) +put 'data5.fits' : '/home/username/data5.fits' ( 1024000000 bytes, 323707.3 KBS (cos=11)) +put 'data6.fits' : '/home/username/data6.fits' ( 1024000000 bytes, 320322.9 KBS (cos=11)) +put 'data7.fits' : '/home/username/data7.fits' ( 1024000000 bytes, 253192.6 KBS (cos=11)) +put 'data8.fits' : '/home/username/data8.fits' ( 1024000000 bytes, 253056.2 KBS (cos=11)) +put 'data9.fits' : '/home/username/data9.fits' ( 1024000000 bytes, 323218.9 KBS (cos=11)) +EOF detected on TTY - ending HSI session +``` + +For more information about HSI: + +- [HSI Reference Manual](/assets/pdfs/hsi_9.3_reference_manual.pdf) +- [Lawrence Livermore National Lab HSI User Guide](https://hpc.llnl.gov/documentation/user-guides/using-hsi-hierarchical-storage-interface) diff --git a/docs/userguides/fortress/storage/transfer/htar.md b/docs/userguides/fortress/storage/transfer/htar.md new file mode 100644 index 00000000..94c8dd35 --- /dev/null +++ b/docs/userguides/fortress/storage/transfer/htar.md @@ -0,0 +1,117 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# HTAR + +*HTAR* (short for "HPSS TAR") is a utility program that writes TAR-compatible archive files directly onto Fortress, without having to first create a local file. Its command line was originally based on `tar`, with a number of extensions added to provide extra features. + +HTAR is provided on all research systems as the command `htar`. HTAR is also available for [download](https://www.rcac.purdue.edu/downloads/archive#hsi) for many operating systems. + +## Usage + +Create a tar archive on Fortress named data.tar including all files with the extension ".fits": + +``` +$ htar -cvf data.tar *.fits +HTAR: a data1.fits +HTAR: a data2.fits +HTAR: a data3.fits +HTAR: a data4.fits +HTAR: a data5.fits +HTAR: a /tmp/HTAR_CF_CHK_17953_1317760775 +HTAR Create complete for data.tar. 5,120,006,144 bytes written for 5 member files, max threads: 3 Transfer time: 16.457 seconds (311.121 MB/s) +HTAR: HTAR SUCCESSFUL +``` + +Unpack a tar archive on Fortress named data.tar into a scratch directory for use in a batch job: + +``` +$ cd $RCAC_SCRATCH/job_dir +$ htar -xvf data.tar +HTAR: x data1.fits, 1024000000 bytes, 2000001 media blocks +HTAR: x data2.fits, 1024000000 bytes, 2000001 media blocks +HTAR: x data3.fits, 1024000000 bytes, 2000001 media blocks +HTAR: x data4.fits, 1024000000 bytes, 2000001 media blocks +HTAR: x data5.fits, 1024000000 bytes, 2000001 media blocks +HTAR: Extract complete for data.tar, 5 files. total bytes read: 5,120,004,608 in 18.841 seconds (271.749 MB/s ) +HTAR: HTAR SUCCESSFUL +``` + +Look at the contents of the data.tar HTAR archive on Fortress: + +``` +$ htar -tvf data.tar +HTAR: -rw-r--r-- username/pucc 1024000000 2011-10-04 16:30 data1.fits +HTAR: -rw-r--r-- username/pucc 1024000000 2011-10-04 16:35 data2.fits +HTAR: -rw-r--r-- username/pucc 1024000000 2011-10-04 16:35 data3.fits +HTAR: -rw-r--r-- username/pucc 1024000000 2011-10-04 16:35 data4.fits +HTAR: -rw-r--r-- username/pucc 1024000000 2011-10-04 16:35 data5.fits +HTAR: -rw------- username/pucc 256 2011-10-04 16:39 /tmp/HTAR_CF_CHK_17953_1317760775 +HTAR: Listing complete for data.tar, 6 files 6 total objects +HTAR: HTAR SUCCESSFUL +``` + +Unpack a single file, "data5.fits", from the tar archive on Fortress named data.tar into a scratch directory: + +``` +$ htar -xvf data.tar data5.fits +HTAR: x data5.fits, 1024000000 bytes, 2000001 media blocks +HTAR: Extract complete for data.tar, 1 files. total bytes read: 1,024,000,512 in 3.642 seconds (281.166 MB/s ) +HTAR: HTAR SUCCESSFUL +``` + +## HTAR Archive Verification + +HTAR allows different types of content verification while creating archives. Users can ask HTAR to verify the contents of an archive during (or after) creation using the `-Hverify` switch. The syntax of this option is: + +``` +$ htar -Hverify=option[,option...] ... other arguments ... +``` + +where `option` can be any of the following: + +| Option | Explanation | +|---|---| +| `info` | Compares tar header info with the corresponding values in the index. | +| `crc` | Enables CRC checking of archive files for which a CRC was generated when the file is added to the archive. | +| `compare` | Enables a byte-by-byte comparison of archive member files and their local file counterparts. | +| `nocrc` | Disables CRC checking of archive files. | +| `nocompare` | Disables a byte-by-byte comparison of archive member files and their local file counterparts. | + +Users can use a comma-separated list of `options` shown above, or a numeric value, or the wildcard `all` to specify the degree of verification. The numeric values for `Hverify` can be interpreted as follows: + +``` +0: Enables "info" verification. +1: Enables level 0 + "crc" verification. +2: Enables level 1 + "compare" verification. +all: Enables all comparison options. +``` + +An example to verify an archive during creation using checksums (crc): + +``` +htar -Hverify=1 -cvf abc.tar ./abc +``` + +An example to verify a previously created archive using checksums (crc): + +``` +htar -Hverify=1 -Kvf abc.tar +``` + +Please note that the time for verifying an archive increases as you increase the verification level. Carefully choose the option that suits your dataset best. + +For more information about HTAR: + +- [Lawrence Livermore National Labs HTAR User Guide](https://hpc.llnl.gov/documentation/user-guides/using-htar-hpss-tape-archiver) + +!!!info + **HTAR has an individual file size limit of 64GB**. If any files you are trying to archive with HTAR are greater than 64GB, then HTAR will immediately fail. This does not limit the number of files in the archive or the total overall size of the archive. To get around this limitation, try using the `htar_large` command. It is slower than using `htar` but it will work around the 64GB file size limit. The usage of `htar_large` is almost the same as `htar` except that `htar_large` will not generate the tar index file. Thus, the `-Hverify=1` option cannot be used since it's based on index file. diff --git a/docs/userguides/fortress/storage/transfer/index.md b/docs/userguides/fortress/storage/transfer/index.md new file mode 100644 index 00000000..8e389490 --- /dev/null +++ b/docs/userguides/fortress/storage/transfer/index.md @@ -0,0 +1,21 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +--- + +# File Transfer + +Fortress supports several methods for file transfer. Fortress does not support SCP or SSH. + +!!!note + Fortress does not support SCP or SSH. + +- [**FTP / SFTP**](sftp.md) +- [**Globus**](globus.md) +- [**HSI**](hsi.md) +- [**HTAR**](htar.md) diff --git a/docs/userguides/fortress/storage/transfer/sftp.md b/docs/userguides/fortress/storage/transfer/sftp.md new file mode 100644 index 00000000..fed4cdac --- /dev/null +++ b/docs/userguides/fortress/storage/transfer/sftp.md @@ -0,0 +1,53 @@ +--- +tags: + - Fortress +authors: + - jin456 + - ramonw +search: + boost: 2 +draft: false +--- + +# FTP / SFTP + +!!!info + FTP is not supported on Fortress because it does not allow for secure transmission of data. Use SFTP instead, as described below. + +*SFTP* (Secure File Transfer Protocol) is a reliable way of transferring files between two machines. SFTP is available as a protocol choice in some graphical file transfer programs and also as a command-line program on most Linux, Unix, and Mac OS X systems. SFTP has more features than SCP and allows for other operations on remote files, remote directory listing, and resuming interrupted transfers. Command-line SFTP cannot recursively copy directory contents; to do so, try using a graphical SFTP client. + +!!!note + **Note:** to access Fortress through SFTP, you must be on a Purdue campus network or connected through [VPN](http://www.itap.purdue.edu/connections/vpn/). + +To transfer files to or from Fortress, your client should connect to the server name `sftp.fortress.rcac.purdue.edu`. + +Command-line usage: + +``` +$ sftp username@sftp.fortress.rcac.purdue.edu + + (to the Fortress system from a local computer) +sftp> put sourcefile somedir/destinationfile +sftp> put -P sourcefile somedir/ + + (from the Fortress system to a local computer) +sftp> get sourcefile somedir/destinationfile +sftp> get -P sourcefile somedir/ + +sftp> exit +``` + +The `-P` flag is optional. When used, it will cause the transfer to preserve file attributes and permissions. + +Linux / Solaris / AIX / HP-UX / Unix: + +- The `sftp` command-line program should already be installed. + +Microsoft Windows: + +- [MobaXterm](https://mobaxterm.mobatek.net/download.html) — Free, full-featured, graphical Windows SSH, SCP, and SFTP client. + +Mac OS X: + +- The `sftp` command-line program should already be installed. You may start a local terminal window from "Applications->Utilities". +- [Cyberduck](https://cyberduck.io/) is a full-featured and free graphical SFTP and SCP client. diff --git a/docs/userguides/reedfolder/faqs.md b/docs/userguides/reedfolder/faqs.md new file mode 100644 index 00000000..5865486c --- /dev/null +++ b/docs/userguides/reedfolder/faqs.md @@ -0,0 +1,159 @@ +--- +tags: + - REED Folder +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# Frequently Asked Questions + +Frequently asked questions about REED Folder. + +## About REED Folder + +### What is a Box Managed Account and REED Folder? + +A Box Managed account is your Purdue managed Box account. A REED folder is a project folder created in secure folder structured within Box. + +### What do I need to do before I access my REED Folder? + +- Anyone with access to data in scope of HIPAA must complete the required [HIPAA training](https://www.purdue.edu/legalcounsel/HIPAA/Training.html) annually. A PI is responsible to ensure the training is completed annually. +- Ensure you are using Purdue Login via the Duo client. +- For limited data sets, a Data Security Plan and Data Use Agreement will be required. This process should be started by [Sponsored Program Services](https://www.purdue.edu/business/sps/). +- Ensure you understand the following [policies and procedures](https://www.purdue.edu/legalcounsel/HIPAA/FormsProcedures.html) specific to HIPAA. +- If you are working with the [IRB](https://www.irb.purdue.edu/), ensure the process is completed prior to requesting access to REED Folder. + +### Does Purdue have a Business Associate Agreement with Box? + +Box supports the HIPAA and HITECH regulations, as well as the ability to sign HIPAA Business Associate Agreements (BAAs) with customers. Customers who are required by law to comply with HIPAA, such as HIPAA Covered Entities and HIPAA Business Associates, must have an Enterprise or Elite account with Box and sign a HIPAA Business Associate Agreement (BAA). + +- Purdue University signed a Business Associate Agreement with Box, and Box accounts are Enterprise. +- Box provides the ability to use O365 online office application to open and work with content. Purdue University has signed a Business Associate Agreement with Microsoft. +- All other third-party service providers with our instance of Box Enterprise are not covered by a Business Associate Agreement (BAA). + +Do not use third-party applications to process data considered Restricted via your Purdue managed Box account. Purdue University does not maintain a contractual agreement with the vendor, and the required security controls are not in place. + +A Box Managed account is your Purdue managed Box account. A REED folder is a project folder created in secure folder structure within Box. + +### Who should I contact for an IT incident? + +- The incident should be reported as soon as it is discovered. Submit an incident request to abuse@purdue.edu +- If you lost a device, your account can be logged out using a Box admin feature. Submit an incident request to abuse@purdue.edu + +## Data + +### How do I host sensitive or restricted data in Box? + +Storage of restricted data in Box is subject to review from Purdue's IT Security and Policy group (ITSP). Access may also require approval from campus offices and committees responsible for contractual compliance and research regulatory affairs. + +Purdue's managed Box environment can be used to store, de-identified data and limited data sets in scope of HIPAA for research. Fully identified data is subject to a security review and approval process. + +All restricted data must be stored in a REED Folder. Do not use Box for an actual medical practice. + +See the data map below for more details: + +![HIPAA data map](/assets/images/userguides/storage/hipaa-dm.png) + +### Can I integrate 3rd-Party Apps with my Box folder? + +While official Box add-on applications are approved for use with Purdue Box.com folders, third-party apps have not been reviewed for impact to security and are not approved for use until a security review has been completed by IT Security and Policy. If the official Box apps cannot provide the functionality you need, you may request a security review of the third-party app you are considering by submitting an email including the app name and the functionality you are addressing with the app to itpolicyreq@purdue.edu. + +### How should I monitor access to my REED Folder? + +The PI or Project Owner's should follow the recommendations below: + +- Review Sharing tab monthly, to ensure only authorized people have access to the data +- Review the Access Stats on files deemed restricted periodically to ensure the actions taken are appropriate. +- Maintain a log of your actions, the log can be a spreadsheet or word document. The log should be stored outside of Box. + +### Can I use Unmanaged Box accounts, and other Cloud Storage Options? + +Unmanaged (free and commercial) Box accounts and all other Cloud Storage options are not approved for storing or sharing sensitive or restricted data. Purdue University does not maintain a contractual agreement with the vendor, and the required security controls are not in place. Personal use of cloud storage can continue, however sensitive and restricted data must not be stored in the account. The table below will help you determine where to store your data. + +| Classification | Personal Box Folder | REED Folder | +| --- | --- | --- | +| Individually Identifiable Health Information | No | Determined by Review | +| Limited Data Set | No | Yes | +| De-Identified Data | No | Yes | +| De-Identified Data with Contractual Requirements | No | Yes | + +### What is De-identified Data? + +De-identified data is created by removing all 18 elements that could be used to identify the individual or the individual's relatives, employers, or household members; these elements are enumerated in the Privacy Rule. The covered entity also must have no actual knowledge that the remaining information could be used alone or in combination with other information to identify the individual who is the subject of the information. De-identified health information, as described in the Privacy Rule, is not PHI, and thus is not protected by the Privacy Rule however it could be subject to contractual requirements. Requests to store additional data types within scope of HIPAA should be sent to ITSP for review prior to uploading to Box Enterprise. + +### Should I use Box Drive if I work with Restricted Data? + +Currently, Box Drive is not recommended for use to work with restricted data. However, if you have a business need for Box Drive, contact IT Security and Policy so we can help you understand the risk and ensure you have the appropriate controls in place to protect the data. + +### Will Box Drive or Box Edit cache files on the local machine? + +Box Drive does locally cache files that you have opened. Box Drive's cache size limit is based on your free disk space (50% of available space) and has a maximum limit of 25 GB. If you reach this limit, Box Drive begins removing files, starting with those files that have gone the longest without your accessing them. Also, if a cached file has a new version created on Box, Box Drive discards the locally cached version. Do not open data in Box Drive containing restricted data. More details: [Technical Information for Box Drive Administrators](https://community.box.com/t5/Deploying-Managing-Box-Drive/Technical-Information-for-Box-Drive-Administrators/ta-p/37454) + +Box Edit does download the file to a cache when you are modifying a file. The cache can be cleared. See: [Clear Your Box Edit Cache](https://support.box.com/hc/en-us/articles/360043695814-Clear-Your-Box-Edit-Cache) + +### Can I use File Level Encryption with a REED Folder? + +The data stored in REED folder is encrypted at rest, however file level encryption is a responsibility of the individual. Files can be encrypted prior to upload for added security measures. Do not use Windows EFS, the data will be uploaded without the encryption. Currently, we recommend using VeraCrypt. + +### Can I use FTP to access Box? + +No, FTP is disabled. + +### What are the requirements for a laptop or desktop if I am working with Restricted data? + +- Encrypt hard drive using Whole Disk Encryption for all drives in the system +- Must be a Purdue managed laptop or desktop +- Must have software security updates applied every 30 days +- Install anti-virus, anti-spyware software and ensure definitions are up to date and run regular scans; utilize an endpoint protection solution. +- Require re-authentication after 15 minutes of inactivity +- Enable host based firewall +- Ensure only authorized software on systems is accessing, transmitting, storing sensitive or restricted information +- Require Authentication from remote devices +- Disable bluetooth on machines unless otherwise required. +- Avoid the use of removable media unless required, in which case removable media must be encrypted. +- Require Administrator privileges to install applications. + +### What is a Limited Data Set? + +A limited data set is a separate legal concept under the privacy rule and is considered identifiable data. Specifically, a limited data set refers to PHI that has had the following 16 unique identifiers removed. A data use agreement is how covered entities obtain satisfactory assurances that the recipient of the limited data set will use or disclose the PHI in the data set only for specified purposes. A Data Use Agreement will continue to be required when working with limited data sets in Box. + +### Should I move restricted data to other folders outside of a REED Folder? + +When storing data within scope of HIPAA, if must be stored in a REED Folder: + +- Do not move and or store the data in a folder hosted directly in your personal Box account. +- Do not store the data outside of REED Folder unless you moving the data to a HIPAA aligned system and you have received approval from Purdue's IT Security and Policy group (ITSP). +- Moving data may also require approval from campus offices and committees responsible for contractual compliance and research regulatory affairs. + +### What are the folder naming conventions and folder description requirements for a REED Folder? + +All folders and subfolders must follow a naming convention. The text must be added manually. The folder name must begin with [L3 HIPAA]-Folder Name. The naming convention should be used on any folder hosted in REED Folder. This is important, to allow for auditing of the service. All folders containing restricted data, must add a banner to the folder by using the Box folder description. + +"The folder contains restricted data in scope of HIPAA. Exercise caution when sharing restricted or sensitive data, individuals must be authorized to access the data. Never sync restricted or sensitive data to an unauthorized system." + +It best that you project team work from root folder of the project. If you create a collaboration in a sub folder, be certain you add the banner text and naming convention to ensure the project team knows the folder contains restricted data. + +As a best practice, you can add [P1 Public] to folder names that don't contain restricted or sensitive data, this will help you managed the various data types you generate. + +### Can I delete my files permanently from Box? + +Individuals can delete a file or folder, and the item will be placed in the Trash. Individuals will not be able to permanently delete the items. An automated process is in place to remove data older than 90 days. Once the item is purged, it can't be recovered using the built in Box tools. + +### What are the recommended permission levels? + +| Role | Permission Level | +| --- | --- | +| Project Team member who needs full control | Editor | +| Project Team members who just needs to work within Box | Viewer Uploader | +| Project Team member who needs Read/Download access | Viewer | +| Partner who needs Read access | Previewer | +| Partner who need to upload data | Uploader | + +## Login + +### Can I manage my Login Activity in Box? + +In Box under your account settings, click the "Security" tab. You can review and remove sessions. diff --git a/docs/userguides/reedfolder/index.md b/docs/userguides/reedfolder/index.md new file mode 100644 index 00000000..b407e6ab --- /dev/null +++ b/docs/userguides/reedfolder/index.md @@ -0,0 +1,15 @@ +--- +tags: + - REED Folder +authors: + - jin456 +search: + boost: 2 +--- + +# REED Folder User Guide + +A REED Folder is a managed storage solution built on top of the Box.com cloud platform, for research projects requiring compliance with regulations or heightened security. + +- [**REED Folder Overview**](overview.md) +- [**Frequently Asked Questions**](faqs.md) diff --git a/docs/userguides/reedfolder/overview.md b/docs/userguides/reedfolder/overview.md new file mode 100644 index 00000000..40298dff --- /dev/null +++ b/docs/userguides/reedfolder/overview.md @@ -0,0 +1,53 @@ +--- +tags: + - REED Folder +authors: + - jin456 +search: + boost: 2 +draft: false +--- + +# REED Folder Overview + +Research labs requiring data storage for regulated research can request REED Folders by emailing [rcac-help@purdue.edu](mailto:rcac-help@purdue.edu). + +## REED Folder Features + +A REED Folder offers research groups in need of centralized data storage for regulated research unique features and benefits: + +- **Available** + + To any Purdue research group requiring data storage space for regulated research. + +- **Accessible** + + Easily accessible through your web browser, and facilitates easy sharing with collaborators within Purdue and without. + +- **Capable** + + A REED folder facilitates joint work on shared files across your research group, avoiding the need for numerous copies of datasets across individuals' private storage. It is an ideal place to store a project's data and documents. A REED Folder can store unlimited data, with a maximum size of a single file up to 15 GB. + +- **Controllable Access** + + Access management is under your direct control, within the bounds of the appropriate data use agreements, IRB protocols, or technology control plans. + +- **Data Retention** + + All data kept in the REED Folder remains owned by the research group's lead faculty. Files kept in the project's REED folder remain with the research group, unaffected by turnover, and could head off potentially difficult disputes. Data is also retained and protected in alignment with the appropriate data security standards governing your research projects. + +- **Never Purged** + + A REED Folder is never subject to purging. + +- **Reliable** + + REED Folders are built on Purdue's centrally-managed [Box.com](https://box.com) service, which is a highly-available, secure cloud storage platform. + +- **Restricted Data** + + REED Folders are designed to align with the NIST SP 800-171 standard, and are approved for storing L3 projects requiring HIPAA-aligned storage. Compliance with additional regulations such as CUI, ITAR, EAR, or FERPA are under review. + +## REED Folder Hardware Details + +REED Folders are built on [Box.com](https://box.com), an enterprise-grade cloud platform for file storage and collaboration. diff --git a/mkdocs.yml b/mkdocs.yml index 76106d4f..fc5cd50f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -183,12 +183,44 @@ nav: - Rossmann: https://www.rcac.purdue.edu/knowledge/rossmann - Weber: https://www.rcac.purdue.edu/knowledge/weber - Storage User Guides: - - Data Depot: https://www.rcac.purdue.edu/knowledge/depot - - Fortress: https://www.rcac.purdue.edu/knowledge/fortress - - Scratch: https://www.rcac.purdue.edu/knowledge/scratch - - Home Directory: https://www.rcac.purdue.edu/knowledge/home - - REED Folder: https://www.rcac.purdue.edu/knowledge/reedfolder - - Box Research Lab Folder: https://www.rcac.purdue.edu/knowledge/boxfolder + - Data Depot: + - userguides/depot/index.md + - Overview: userguides/depot/overview.md + - File Storage and Transfer: + - userguides/depot/storage/index.md + - Archive and Compression: userguides/depot/storage/compress.md + - File Transfer: userguides/depot/storage/transfer.md + - Sharing Files: userguides/depot/storage/sharing.md + - Lost File Recovery: + - userguides/depot/recover/index.md + - flost Tool: userguides/depot/recover/flost.md + - Manual Browsing: userguides/depot/recover/manual.md + - Windows: userguides/depot/recover/windows.md + - Mac OS X: userguides/depot/recover/mac.md + - Access Permissions and Directories: + - userguides/depot/permissions/index.md + - Default Configuration: userguides/depot/permissions/default.md + - Common Scenarios: userguides/depot/permissions/scenarios.md + - Unix Groups: userguides/depot/permissions/unixgroups.md + - Frequently Asked Questions: userguides/depot/faqs.md + - Fortress: + - userguides/fortress/index.md + - Overview: userguides/fortress/overview.md + - Accounts: userguides/fortress/accounts.md + - File Storage and Transfer: + - userguides/fortress/storage/index.md + - Sharing Files: userguides/fortress/storage/sharing.md + - File Transfer: userguides/fortress/storage/transfer/index.md + - Frequently Asked Questions: userguides/fortress/faqs.md + - Box Research Lab Folder: + - userguides/boxfolder/index.md + - Overview: userguides/boxfolder/overview.md + - Accounts: userguides/boxfolder/accounts.md + - Frequently Asked Questions: userguides/boxfolder/faqs.md + - REED Folder: + - userguides/reedfolder/index.md + - Overview: userguides/reedfolder/overview.md + - Frequently Asked Questions: userguides/reedfolder/faqs.md - RCAC Resources: - RCAC Blogs: - blog/index.md diff --git a/tools/generate_breadcrumbs.py b/tools/generate_breadcrumbs.py index 2a557608..6932dcea 100644 --- a/tools/generate_breadcrumbs.py +++ b/tools/generate_breadcrumbs.py @@ -46,6 +46,12 @@ def handle_env_tag(loader, node): "Geddes": "Geddes User Guide", "Hammer": "Hammer User Guide", "Rossmann": "Rossmann User Guide", + "Data Depot": "Data Depot User Guide", + "Fortress": "Fortress User Guide", + "Scratch": "Scratch User Guide", + "Home Directory": "Home Directory User Guide", + "REED Folder": "REED Folder User Guide", + "Box Research Lab Folder": "Box Research Lab Folder User Guide", } # ----------------------------