UCIHAR Codebook.Rmd

---
title: "UCI HAR Data Set"
author: "Robert Hadow"
date: "5 October 2015"
output: 
  html_document:
    css: "UCIHAR.CSS"
    fig_caption: yes
    keep_md: yes
    number_sections: yes
---
========================================

# Study Design
The UCI HAR dataset described herein was generated in a series of observations of human physical activity using a smartphone's inbuilt accelerometer and gyroscope.  These observations of thirty volunteers in six activities were collected in Genoa in 2012.

Two detectors of this type can directly measure the following at a rate of fifty observations per second:

- linear acceleration in three dimensions
- orientation in three dimensions

Using time-series of these observations, an analyst may calculate other measures, including:

- axial acceleration in three dimensions
- jerk in three dimensions (time derivative of acceleration)
- angular jerk in three dimensions (time derivative of angular acceleration)
- decomposition of these measures in the frequency domain (Fourier Transform)  

The data is further analyzed for measures of central tendency and error.  In total, 561 measures are generated. 

This data was delivered to us after the first round of calculation and analysis. It includes 18 million data points in 26 data files.  There are two additional files of human-readable description.  The following data files were used and reorganized here:

- features    -   descriptions of summary observations data
- X_test      -   summary observation data 
- X_train     -   summary observation data 
- subject_test  - subject identification
- subject_train - subject identification
- y_test      -   exercise type
- y_train     -   exercise type

# Data Approach

We used the seven data files to develop, conceptually, a tidy cube of data along three dimensions:

- 6 exercises
- 30 subjects
- 77 summary measures

Principles of tidy data organization limited us to presentations in two dimensions: columns, describing attributes of the data, and rows, describing particular observations. A tidy format should allow use of the data with the minimum amount of manipulation,based on the purposes for which the data will be analyzed. We designed our data set to allow its use with the following operations, listed here in decreasing order of priority:

1. viewing as is (no manipulation)
2. summary measures, including sum, mean, median,etc
3. grouping and subsetting (logical rearrangement, row-wise)
4. reordering (physical rearrangement, row-wise)
5. numerical manipulation (modifying or adding columns)
6. disassembly and reassembly of the data block

## Ease of Viewing
To accommodate viewing (goal 1) we put exercise and subject leftmost in the tableau.  
Insofar as we anticipated the user would rearrange the row order to meet his or her needs, we left the order as it was delivered to us, first test data then training data (because that was alphabetical if for no other reason).

## Accomodate Summary Measures
All values of a particular type, i.e. "tBodyAcc-mean" are in the same column (a "wide" or "un-stacked" layout), accommodating summary measures.  We stripped parentheses from the column names.  Because the parentheses were common, they added no information, but they did add significantly to the file size.

## Accomodate Grouping
Each observation appears in a row, all of the attributes in columns.

## Reordering
Observations may be reordered row-wise without other manipulation.

## Numerical Manipulation
We chose to leave activity as a numerical attribute in the main data table.  This keeps the entire table numeric data. Having one character column in a 79 column table, we thought, is unnecessary complexity and adds no information. We provided the activity codes as a separate table.

## Wholesale Changes to Data Structure
A two-table structure is far easier to manipulate than the 26 we started with.

# Details of Data Conversion

We downloaded the raw data and reference materials according to the script below:  

```{r download_unpack}

suppressPackageStartupMessages(library(plyr))
suppressPackageStartupMessages(library(dplyr))
options(width = 120)

targetDir <- dictDir <- "./codebook"
if(!file.exists(targetDir)) dir.create(targetDir)
fileUrl <- "http://archive.ics.uci.edu/ml/machine-learning-databases/00240/UCI HAR Dataset.names"
zipFile = paste(targetDir, "herus.html", sep = "/")

targetDir <- dataDir <- "./data"
if(!file.exists(targetDir)) dir.create(targetDir)
fileUrl <-
        "https://d396qusza40orc.cloudfront.net/getdata%2Fprojectfiles%2FUCI%20HAR%20Dataset.zip"
zipFile = paste(dataDir, "UCIHAR.zip", sep = "/")
download.file(fileUrl, zipFile, mode="wb")
dateDownloaded = date()

unzip(zipFile, exdir = targetDir, junkpaths = TRUE)

targetDir <- dictDir <- "./codebook"
dictFiles <- c("README.txt", "features_info.txt")
from <- sapply(dataDir, paste, dictFiles, sep ="/")
to   <- sapply(targetDir, paste, dictFiles, sep ="/")
success <- file.rename(from, to)

csvFiles <- list.files(dataDir, pattern = "*.txt", recursive = TRUE,
        full.names = TRUE)

```

## Reading Data Files into R

We read all of the data files into R.  We used this opportunity to check the parameters for this conversion and to see which files were relevant.  


```{r, read_data}

csvFiles <- list.files(dataDir, pattern = "*.txt", recursive = TRUE,
                full.names = TRUE)
newObjects <- gsub(".txt", "", csvFiles)
newObjects <- gsub("./data/", "", newObjects)

for (i in 1:length(csvFiles)) {
        assign(newObjects[i], inherits = TRUE,
               read.csv(csvFiles[i],
                header = FALSE,
                stringsAsFactors = FALSE,
                na.strings = "NA",
                sep = ""))
}

dataPoints = 0
for (i in 1:length(newObjects)) {
        dataPoints = dataPoints + (nrow(get(newObjects[i])) * ncol(get(newObjects[i])))
}

```

## Inspection and House Cleaning

We inspected the files, made adjustments to the scripts, and dumped unnecessary files and objects.  

```{r, housecleaning}

````

## Tidy Data

We created a tidy data set using the following steps:

* organize and apply column names to the test data set
* organize subject data, append it to the observations
* organize exercise data, append it to the observations
* redact the data,eliminating unwanted columns
* combine the test and train datasets
  

```{r, tidy_data}

# organize and apply column names to the test data set
features[, 2]  <- gsub("\\()","", features[, 2])
features[, 2]  <- gsub("\\()","", features[, 2])

colnames(X_test)  <- features[, 2]
colnames(X_train) <- features[, 2]

# organize subject data, append it to the observations
colnames(subject_test)  <- "subject"
colnames(subject_train) <- "subject"
X_test  <- cbind(subject_test,  X_test)
X_train <- cbind(subject_train, X_train)

# organize exercise data, append it to the observations
colnames(activity_labels) <- c("activity_code", "activity")
colnames(y_test)  <- "activity_code"
colnames(y_train) <- "activity_code"
X_test  <- cbind(y_test,  X_test)
X_train <- cbind(y_train, X_train)

# now redact the data
selectedColumns <- features[ grep("mean|std", features$V2) , ]
X_test  <- X_test[,  selectedColumns$V1]
X_train <- X_train[, selectedColumns$V1]

# combine the test and train datasets
ucihar <- rbind(X_test, X_train)

# write csv file for activity explanation
write.table(activity_labels, file = "activities.csv", sep = ",",
            eol = "\n", na = "NA", dec = ".", row.names = FALSE,
            col.names = TRUE, qmethod = c("escape", "double"),
            fileEncoding = "")

dataPoints = nrow(ucihar) * (ncol(ucihar) - 2)

````

## Create Codebook

We were careful not to make any assumptions about the data that was not in the raw data package.  

```{r, create_codebook}

uciharTable <- rep("ucihar", ncol(ucihar))
uciharAttribute <- colnames(ucihar)
uciharDescription <- colnames(ucihar)
uciharDescription[substr(uciharDescription,1,1) == "f"] <- "frequency domain data"
uciharDescription[substr(uciharDescription,1,1) == "t"] <- "time domain summary data"
ucihar.dict <- data.frame(uciharTable, uciharAttribute, uciharDescription)

uciharTable <- "subject"
uciharAttribute <- "subject between 18 and 48"
uciharDescription <- "Subjects by number only"
temp <- data.frame(uciharTable, uciharAttribute, uciharDescription)
ucihar.dict = rbind(ucihar.dict, temp)

uciharTable <- "activity_labels"
uciharAttribute <- "activity_code"
uciharDescription <- as.character(activity_labels[,1])
temp <- data.frame(uciharTable, uciharAttribute, uciharDescription)
ucihar.dict = rbind(ucihar.dict, temp)

uciharTable <- "activity_labels"
uciharAttribute <- "activity"
uciharDescription <- activity_labels[,2]
temp <- data.frame(uciharTable, uciharAttribute, uciharDescription)
ucihar.dict = rbind(ucihar.dict, temp)

colnames(ucihar.dict) <- c("Table", "Attribute", "Description")


````
# Codebook

The documentation that came with the raw data contains descriptive attribute names from which a reader may surmise the origin and use of the associated data elements.  For more detailed explanations, the user is referred to the Non Linear Complex Systems Laboratory, Genoa, activity recognition@smartlab.ws.

The data dictionary that follows was generated from the UCIHAR data tables.  Similarly named attributes may be used as primary keys.

Trailing digits refer to raw data not included in this set.

All data in the set is normalized -1.00 - 1.00.

```{r, print_codebook}

print(ucihar.dict, include.rownames = FALSE, max.levels = null, width = 1200)

````

## Summary Dataset - UCIHAR_summary

UCIHAR contains `r I(dataPoints)` measures.  It has `r I(nrow(ucihar))` rows of data.

UCIHAR_summary is a summary of UCIHAR.  It provides the means of the data in UCIHAR. 

```{r, summary_dataset}

ucihar %>% group_by(activity_code, subject) %>% summarise_each(funs(mean)) -> ucihar_summary


# write csv file for summary dataset
write.table(ucihar_summary, file = "UCIHAR_summary.txt", sep = ",",
            eol = "\n", na = "NA", dec = ".", row.names = FALSE,
            col.names = TRUE, qmethod = c("escape", "double"),
            fileEncoding = "")

````