Data Organization

We elected to base our organization on the traditional Argus organization. Thus, data in this example are stored in the directory /ftp/pub (where 'ftp' refers to the CIL-ftp location at OSU: cil-ftp.oce.orst.edu). Subdirectories under /ftp/pub are defined by the Argus station name. In the case of the UAV Toolbox, the station name is taken from the UAV name. For illustration, we will use the case of Aerielle, a DJI Phantom 3 quadcopter. Folders under /ftp/pub/Aerielle contain lens calibration images and results as well as stationInfo, in this case the history of the UAV and usage and any other relevant details. In practice, data organization can be updated to reflect the user's preference.

Keeping with Argus conventions, data are stored next under year/camera/day. So for example, /ftp/pub/Aerielle/2015/cx/280_Oct.07 contains derived image products (cx) for flights from October 7, 2015 including rectified image products like time exposures and time stacks if these have been created. Folder names can be personalized in the input file. Similarly, the function argusFilename.m takes care of constructing file names using the Argus file naming convention and is included in the toolbox in the folder ‘neededCILRoutines’. Add the folder neededCILRoutines to your Matlab path to access argusFilename.m while using the UAV Toolbox.

Typically, the data collection for a cBathy time series will consist of a snapshot followed by a movie of sufficient length to perform cBathy analysis (hopefully 17.1 minute). The snapshot is taken as a simple record of the scene and also because it will be geotagged, so can be used as a source of estimated lat-long of the parked UAV. For a phantom 3 these geotags appear to be quite useful (~5-10 m accuracy) but for the Phantom 2 they are truncated at integer seconds so are only accurate to ~30 m. The movie is often too long to be stored as a single MP4 file so might be split into two consecutive files automatically by the Phantom. Note that DJI files are stored with simple names like DJI_0001 so it is the responsibility of the user to manually convert these into more useful names and store them in an appropriate location.

Because MP4 videos are usually multi-gigabyte files, they would rapidly overwhelm OSU's online disk storage capabilities (in the same sense that we don’t save 30 Hz full-frame video from Argus stations). Thus our choice is that these are stored offline on a USB disk under some obvious structure like /media/Lacie/Aerielle/bathyDuckDataCollects/100615/exampleFolder (where exampleFolder is the specific folder name for one data collection, usually including at least one snap and one or more mp4 movies). To access the individual 2 Hz frames from the 30 Hz movie to do Argus-like analysis, we use the script loadAndPartitionMovies.m which partitions the movie into 2Hz frames using the command “every 15 soureFn destinationFn” where sourceFn and destinationFn are your source and destination filenames (“every” is a local Perl script that used the mplayer video player routines. If you don't have access to Perl or prefer using matlab to extract frames from video, an example script called video2images.m is provided). For the CIL, we store frames as a series of png’s in a scratch area, e.g. /scratch/temp/Holman/Aerielle/2015/localFolderName. This component of operations will vary to suit the needs and tools of each research group. However, we have found that using the Argus conventions for derived products greatly simplifies our logistics. Alternatively, you can down sample movies using the script loadAndParitionMoviesMatlab.m, which uses the Matlab VideoReader object rather than Perl and mplayer.

Example data that have already been extracted at the 2 Hz frame rate are available here (725MB).