Data Integrity

Back End

The current backend storage is an OpenStack Swift instance. Each files is replicated a number of times based on configuration.

Uploaded Data

Uploaded files are md5 check summed two ways.

First we checksum the entire file. This is saved with the file on the DukeDS server. The DukeDS team is planning a service to re-check these checksums.

Secondly the file is split up into chunks based on the chunk size. Each of the chunks is check summed and the checksum is sent along with the chunk. The swift backend object store receives the data, recalculates the checksum and raises an error if it doesn’t match. If this happened the upload would terminate.