add_data.sh related Python scripts - flexible data loading

[drotheram]

Dynamically load raw model datasets which may have changes to the types of fields by reading and loading those fields. Implement tests on inputs with some reasonable constraints on what the stack will load

See also Issue #48

Major Priorities

PSRA:

• PSRA_copyTables.py (Preview WIP flexible-csv-headers branch)
  • Shrink the list of expected_headers (though these should probably be removed altogether eventually)
  • pylint, flake8
  • Skip OpenQuake comment header (and deal with U+FEFF BOM). (Originally taken care off by sed `add_data.sh
  • Read CSV header from PostgreSQL dynamically, something like psql opendrr -c 'COPY (SELECT * FROM psra_BC.psra_BC_hcurves_pga WHERE FALSE ) TO STDOUT WITH CSV HEADER;'
  • Quote SQL identifiers when necessary for column names with upper-case letters, ., (), etc.
  • Restore reading config.ini and add error checking if file not found (POSTGRES_* variables not defined)
  • Write some kind of unit test?

DSRA:

• DSRA_outputs2postgres_lfs.py (consider refactoring)
• DSRA_ruptures2postgres.py

Minor Priorities

Exposure:

• copyAncillaryTables.py

[jvanulde]

@anthonyfok is this done? If not please move to Sprint 33 Milestone.

[anthonyfok]

@jvanulde Thanks for the reminder! Done, and am in the process of moving other outstanding tasks to Sprint 33 too.