Merge develop

Author: chinyeungli

.gitignore

@@ -47,3 +47,5 @@ docs/_build
 /.cache/
 /.settings/
 /tcl/
+/.python-version
+/.tox/

.travis.yml

@@ -8,7 +8,7 @@ install:
     - ./configure etc/conf/dev
 
 script:
-    - bin/py.test -vvs tests
+    - bin/py.test -vvs
 
 notifications:
     irc:

README.rst

@@ -42,7 +42,7 @@ You will need to install a Python interpreter if you do not have one already
 installed.
 
 On Linux and Mac, Python is typically pre-installed. To verify which
-version may be pre-installed, open a terminal and type::
+version may be pre-installed, open a terminal and type:
 
     python --version
 
@@ -58,25 +58,26 @@ INSTALLATION
 Checkout or download and extract the AboutCode Toolkit from:
     https://github.com/nexB/aboutcode-toolkit/
 
-To install all the needed dependencies in a virtualenv, run (on posix)::
+To install all the needed dependencies in a virtualenv, run (on posix):
     source configure
-or on windows::
+or on windows:
     configure
 
 
-USAGE
+REFERENCE
 -----
-See https://github.com/nexB/aboutcode-toolkit/blob/master/USAGE.rst for USAGE.
+See https://github.com/nexB/aboutcode-toolkit/blob/master/REFERENCE.rst for reference
+on aboutcode-toolkit usage.
 
 
 TESTS and DEVELOPMENT
 ---------------------
-To install all the needed development dependencies, run (on posix)::
+To install all the needed development dependencies, run (on posix):
     source configure etc/conf/dev
-or on windows::
+or on windows:
     configure etc/conf/dev
 
-To verify that everything works fine you can run the test suite with::
+To verify that everything works fine you can run the test suite with:
     py.test
 
 
@@ -121,4 +122,4 @@ See (of course) the about.ABOUT file for details.
     :alt: Windows Master branch tests status
 .. |devel-win| image:: https://ci.appveyor.com/api/projects/status/uwj2gh8i9ga1mqwn/branch/develop?png=true
     :target: https://ci.appveyor.com/project/nexB/aboutcode-toolkit
-    :alt: Windows Develop branch tests status
+    :alt: Windows Develop branch tests status

REFERENCE.rst

@@ -11,8 +11,8 @@ about
 
 ::
 
-    --version    Show the version and exit.
-    --help       Show this message and exit.
+    --version        Show the version and exit.
+    -h, --help       Show this message and exit.
 
 **Commands:**
 
@@ -22,6 +22,7 @@ about
   check      LOCATION: directory
   gen        LOCATION: input file, OUTPUT: directory
   inventory  LOCATION: directory, OUTPUT: csv file
+  transform  LOCATION: csv file, OUTPUT: csv file
 
 
 attrib
@@ -352,3 +353,59 @@ The multiple licenses support format for ABOUT files are by "grouping" with the
         -   key: mit
             name: mit.LICENSE
 
+
+transform
+=========
+
+**Syntax**
+
+::
+
+    about transform [OPTIONS] LOCATION OUTPUT
+
+    LOCATION: Path to a CSV file.
+    OUTPUT: Path to CSV inventory file to create.
+
+**Options:**
+
+::
+
+  -c, --configuration FILE  Path to an optional YAML configuration file. See
+                            --help-format for format help.
+  --help-format             Show configuration file format help and exit.
+  -q, --quiet               Do not print error or warning messages.
+  --verbose                 Show all error and warning messages.
+  -h, --help                Show this message and exit.
+
+Purpose
+-------
+Transform the CSV file at LOCATION by applying renamings, filters and checks and write a new CSV to OUTPUT.
+
+Options
+-------
+
+::
+
+    -c, --configuration
+
+        Path to an optional YAML configuration file. See--help-format for format help.
+
+    $ about transform -c 'path to the YAML configuration file' LOCATION OUTPUT
+
+    --help-format
+
+        Show configuration file format help and exit.
+        This option will print out examples of the the YAML configuration file.
+        
+        Keys configuration are: `column_renamings`, `required_columns` and `column_filters`
+
+    $ about transform --help-format
+
+    --verbose
+
+        This option tells the tool to show all errors found.
+        The default behavior will only show 'CRITICAL', 'ERROR', and 'WARNING'
+
+Special Notes
+=============
+When using the `column_filters` configuration, all the standard required columns (`about_resource` and `name`) and the user definied `required_columns` need to be included.

SPECIFICATION.rst

@@ -1,4 +1,4 @@
-ABOUT File Specification v3.1.1
+ABOUT File Specification v3.1.4
 
 
 Purpose
@@ -22,7 +22,7 @@ versions of a software component.
 Getting Started
 ~~~~~~~~~~~~~~~
 
-A simple and valid ABOUT file named httpd.ABOUT may look like this::
+A simple and valid ABOUT file named httpd-2.4.3.tar.gz.ABOUT may look like this::
 
         about_resource: httpd-2.4.3.tar.gz
         name: Apache HTTP Server
@@ -32,14 +32,15 @@ A simple and valid ABOUT file named httpd.ABOUT may look like this::
         license_expression: apache-2.0
         licenses:
             -   key: apache-2.0
-            -   file: apache-2.0.LICENSE
+                name: Apache License 2.0
+                file: apache-2.0.LICENSE
         notice_file: httpd.NOTICE
         copyright: Copyright (c) 2012 The Apache Software Foundation.
 
 The meaning of this ABOUT file is:
 
 - The file "httpd-2.4.3.tar.gz" is stored in the same directory and side-by-side
-  with the ABOUT file "httpd.ABOUT" that documents it.
+  with the ABOUT file "httpd-2.4.3.tar.gz.ABOUT" that documents it.
 
 - The name of this component is "Apache HTTP Server" with version "2.4.3".
 
@@ -60,6 +61,7 @@ Specification
 An ABOUT file is an ASCII YAML formatted text file.
 Note that while Unicode characters are not supported in
 an ABOUT file proper, external files can contain UTF-8 Unicode.
+The key for the licenses field and the license_expression are dejacode license key.
 
 
 ABOUT file name
@@ -72,7 +74,7 @@ A file name can contain only these US-ASCII characters:
 
 - digits from 0 to 9
 - uppercase and lowercase letters from A to Z
-- the "_" underscore, "-" dash, "." period and "+" period signs.
+- the following symbols: "_", "-", "+", ".", "(", ")", "~", "[", "]", "{", "}"
 
 - The case of a file name is not significant. On case-sensitive file systems
   (such as on Linux), a tool must report an error if two ABOUT files stored in
@@ -104,9 +106,9 @@ A field name can contain only these US-ASCII characters:
 - Field names are not case sensitive. For example, "HOMEPAGE_URL" and "HomePage_url"
   represent the same field name.
 
-- A field name must start at the beginning of a new line. It can be followed by
-  one or more spaces that must be ignored. These spaces are commonly used to
-  improve the readability of an ABOUT file.
+- A field name must start at the beginning of a new line. No spaces is allowed in the 
+  field name. It can be followed by one or more spaces that must be ignored. 
+  These spaces are commonly used to improve the readability of an ABOUT file.
 
 
 Field value
@@ -140,22 +142,13 @@ For instance::
         This text contains multiple lines.
 
 
-Fields are mandatory or optional
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Fields are mandatory, optional or custom extension
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-As defined in this specification, a field can be mandatory or optional. Tools
+A field can be mandatory, optional or custom extension. Tools
 must report an error for missing mandatory fields.
 
 
-Extension and ignored fields
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-An ignored field is a field with a name that is not defined in this
-specification. Custom extension fields are also supported and must be processed
-by tools as ignored fields unless a certain tool can process a certain extension
-field.
-
-
 Fields validation
 ~~~~~~~~~~~~~~~~~
 
@@ -165,9 +158,9 @@ name syntax or invalid content. Tools should report additional validation error
 details. The validation process should check that each field name is
 syntactically correct and that fields contain correct values according to its
 concise, common sense definition in this specification. For certain fields,
-additional and specific validations are relevant such as checksum verification,
-URL validation, path resolution and verification, and so forth. Tools should
-report a warning for ignored fields.
+additional and specific validations are relevant such as URL validation, 
+path resolution and verification, and so forth. Tools should
+report a warning for present fields that do not have any value.
 
 
 Fields order and multiple occurrences
@@ -177,7 +170,7 @@ The field order does not matter. Multiple occurrences of a field name is not
 supported.
 
 The tool processing an ABOUT file or CSV/JSON input will issue an error when a
-field name occurs more than once in the input file (as for any other ignored field).
+field name occurs more than once in the input file.
 
 
 Field referencing a file
@@ -204,6 +197,15 @@ above the ABOUT file directory, using a relative POSIX path::
     licenses:
         -   file: ../docs/ruby.README
 
+In addition, there may be cases that a license can have 2 or more referenced license files.
+If this is the case, a comma ',' is used to identify multiple files
+For instance::
+
+    license_expression: gpl-2.0-plus
+    licenses:
+        -   key: gpl-2.0-plus
+            file: COPYING, COPYING.LESSER
+
 Field referencing a URL
 ~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -219,7 +221,7 @@ download URL is referenced this way::
 Flag fields
 ~~~~~~~~~~~
 
-Flag fields have a "true" or "false" value. True, T, Yes or Y , x must be
+Flag fields have a "true" or "false" value. True, T, Yes, Y or x must be
 interpreted as "true" in any case combination. False, F, No or N must be
 interpreted as "false" in any case combination.
 
@@ -316,14 +318,14 @@ Optional Licensing fields
 
 - license_url: URL to the license text for the component.
 
-- license_expression: The license expression that apply to the component. You
+- license_expression: The DejaCode license expression that apply to the component. You
   can separate each identifier using " or " and " and " to document the
   relationship between multiple license identifiers, such as a choice among
   multiple licenses.
 
-- license_name: The license short name for the license.
+- license_name: The DejaCode license short name for the license.
 
-- license_key: The license key(s) for the component.
+- license_key: The DejaCode license key(s) for the component.
 
 
 Optional Boolean flag fields
@@ -348,9 +350,7 @@ Optional Extension fields
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
 You can create extension fields by prefixing them with a short prefix to
-distinguish these from the standard fields. You should provide documentation for
-these extensions and create or extend existing tools to support these
-extensions. Other tools must ignore these extensions.
+distinguish these from the standard fields (but this is not necessary).
 
 
 Optional Extension fields to reference files stored in a version control system (VCS)
@@ -408,6 +408,9 @@ to verify the integrity of a file documented by an ABOUT file.
 - checksum_sha1: SHA1 for the file documented by this ABOUT file in the
   "about_resource" field.
 
+- checksum_sha256: SHA256 for the file documented by this ABOUT file in the
+  "about_resource" field.
+  
 Some examples::
 
       checksum_md5: f30b9c173b1f19cf42ffa44f78e4b96c

about.ABOUT

@@ -1,26 +1,21 @@
 about_resource: .
 name: AboutCode-toolkit
-about_resource_path: .
-
-version: 3.3.3
-
-description: |
+version: 4.0.0
+author: Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez
+copyright: Copyright (c) 2013-2019 nexB Inc.
+description: |-
     AboutCode Toolkit is a tool to process ABOUT files. An ABOUT file
     provides a simple way to document the provenance (origin and license)
     'about' a software component. This is a small text file stored in the
     codebase side-by-side with the documented software component.
-
 homepage_url: http://www.nexb.com/community.html
-
-license: apache-2.0
-license_name: Apache 2.0
-license_file: apache-2.0.LICENSE
-copyright: Copyright (c) 2013-2017 nexB Inc.
-
+license_expression: apache-2.0
+licenses:
+    -   file: apache-2.0.LICENSE
+        key: apache-2.0
+        name: Apache License 2.0
+        url: https://enterprise.dejacode.com/urn/?urn=urn:dje:license:apache-2.0
 notice_file: NOTICE
-
 owner: nexB Inc.
-author: Jillian Daguil, Chin Yeung Li, Philippe Ombredanne, Thomas Druez
-
-vcs_tool: git
 vcs_repository: https://github.com/nexB/aboutcode-toolkit.git
+vcs_tool: git