Refactor support for 'unknown' with location map

This introduces a new Parser-class variable,
`DEFAULT_UNKNOWN_BY_LOCATION`, which maps location names to
`unknown` values. It populates that and `DEFAULT_UNKNOWN` to pass
`EXCLUDE` in the general case and `RAISE` for request bodies.

The new behavior is layered and defined with lower precedence than
`Parser.unknown` or the `unknown` parameter on parse calls. In
order to implement this and allow users to opt *out* of the behavior
(i.e. in order to use schema-defined values), the way these values are
set is subtly changed. Instead of having a default value of `None`,
parse calls and parser __init__ have a default of
`unknown="_default"`. When that value is detected, the various layers
of defaults (DEFAULT_UNKNOWN and DEFAULT_UNKNOWN_BY_LOCATION) are
applied. However, if a user passes `None`, that will take effect with
the meaning "do not pass a value for `unknown`".

For parsers which define additional locations, they extend the base
DEFAULT_UNKNOWN_BY_LOCATION as appropriate.

The changelog has been updated significantly in order to handle this
and a new section of the advanced usage docs covers the behavior.

Author: sirosen

CHANGELOG.rst

@@ -27,10 +27,11 @@ Refactoring:
 
 Features:
 
-* Add a new ``unknown`` parameter to ``Parser.parse``, ``Parser.use_args``, and
-  ``Parser.use_kwargs``. When set, it will be passed to the ``Schema.load``
-  call. If set to ``None`` (the default), no value is passed, so the schema's
-  ``unknown`` behavior is used.
+* Add ``unknown`` as a parameter to ``Parser.parse``, ``Parser.use_args``, and
+  ``Parser.use_kwargs``, or to parser instantiation. When set, it will be passed
+  to ``Schema.load``. When not set, the value passed will depend on the parser's
+  settings. If set to ``None``, the schema's default behavior will be used (i.e.
+  no value is passed to ``Schema.load``) and parser settings will be ignored.
 
 This allows usages like
 
@@ -45,10 +46,10 @@ This allows usages like
     def foo(q1, q2):
         ...
 
-* Add the ability to set defaults for ``unknown`` on either a Parser instance
-  or Parser class. Set ``Parser.DEFAULT_UNKNOWN`` on a parser class to apply a value
-  to any new parser instances created from that class, or set ``unknown`` during
-  ``Parser`` initialization.
+* Defaults for ``unknown`` may be customized on parser classes via
+  ``Parser.DEFAULT_UNKNOWN_BY_LOCATION``, which maps location names to values
+  to use, and ``Parser.DEFAULT_UNKNOWN``, which is used when a location is not
+  found in ``DEFAULT_UNKNOWN_BY_LOCATION``.
 
 Usages are varied, but include
 
@@ -61,11 +62,25 @@ Usages are varied, but include
 
     # as well as...
     class MyParser(FlaskParser):
-        DEFAULT_UNKNOWN = ma.INCLUDE
+        DEFAULT_UNKNOWN_BY_LOCATION = {"query": ma.INCLUDE}
 
 
     parser = MyParser()
 
+Setting the ``unknown`` value for a Parser instance has higher precedence. So
+
+.. code-block:: python
+
+    parser = MyParser(unknown=ma.RAISE)
+
+will always pass ``RAISE``, even when the location is ``query``.
+
+* By default, webargs will pass ``unknown=EXCLUDE`` for all locations except
+  for request bodies (``json``, ``form``, and ``json_or_form``) and path
+  parameters. Request bodies and path parameters will pass ``unknown=RAISE``.
+  This behavior is defined by the default values for ``DEFAULT_UNKNOWN`` and
+  ``DEFAULT_UNKNOWN_BY_LOCATION``.
+
 Changes:
 
 * Registered `error_handler` callbacks are required to raise an exception.

docs/advanced.rst

@@ -128,6 +128,113 @@ When you need more flexibility in defining input schemas, you can pass a marshma
         # ...
 
 
+Setting `unknown`
+-----------------
+
+webargs supports several ways of setting and passing the `unknown` parameter
+for `handling unknown fields <https://marshmallow.readthedocs.io/en/stable/quickstart.html#handling-unknown-fields>`_.
+
+You can pass `unknown=...` as a parameter to any of
+`Parser.parse <webargs.core.Parser.parse>`,
+`Parser.use_args <webargs.core.Parser.use_args>`, and
+`Parser.use_kwargs <webargs.core.Parser.use_kwargs>`.
+
+
+.. note::
+
+    The `unknown` value is passed to the schema's `load()` call. It therefore
+    only applies to the top layer when nesting is used. To control `unknown` at
+    multiple layers of a nested schema, you must use other mechanisms, like
+    the `unknown` argument to `fields.Nested`.
+
+Default `unknown`
++++++++++++++++++
+
+By default, webargs will pass `unknown=marshmallow.EXCLUDE` except when the
+location is `json`, `form`, or `json_or_form`. In those cases, it uses
+`unknown=marshmallow.RAISE` instead.
+
+You can change these defaults by overriding `DEFAULT_UNKNOWN_BY_LOCATION` and
+`DEFAULT_UNKNOWN`. The first is a mapping of locations to values to pass, and
+the second is the fallback value used if the location is not included in the
+map.
+
+You can also define a default at parser instantiation, which will take
+precedence over these defaults.
+
+For example,
+
+.. code-block:: python
+
+    from flask import Flask
+    from marshmallow import EXCLUDE, INCLUDE, fields
+    from webargs.flaskparser import FlaskParser
+
+    app = Flask(__name__)
+
+
+    class Parser(FlaskParser):
+        DEFAULT_UNKNOWN = INCLUDE
+        DEFAULT_UNKNOWN_BY_LOCATION = {"query": EXCLUDE}
+
+
+    parser = Parser()
+
+
+    # location is "query", which is listed in DEFAULT_UNKNOWN_BY_LOCATION,
+    # so EXCLUDE will be used
+    @app.route("/", methods=["GET"])
+    @parser.use_args({"foo": fields.Int()}, location="query")
+    def get(self, args):
+        return f"foo x 2 = {args['foo'] * 2}"
+
+
+    # location is "json", which is not in DEFAULT_UNKNOWN_BY_LOCATION,
+    # so the parser's default value, `INCLUDE`, will be used
+    @app.route("/", methods=["POST"])
+    @parser.use_args({"foo": fields.Int(), "bar": fields.Int()}, location="json")
+    def post(self, args):
+        unexpected_args = [k for k in args.keys() if k not in ("foo", "bar")]
+        return f"foo x bar = {args['foo'] * args['bar']}; unexpected args={unexpected_args}"
+
+
+Using Schema-Specfied `unknown`
++++++++++++++++++++++++++++++++
+
+If you wish to use the value of `unknown` specified by a schema, simply pass
+``unknown=None``. This will disable webargs' automatic passing of values for
+``unknown``. For example,
+
+.. code-block:: python
+
+    from flask import Flask
+    from marshmallow import Schema, fields, EXCLUDE, missing
+    from webargs.flaskparser import use_args
+
+
+    class RectangleSchema(Schema):
+        length = fields.Float()
+        width = fields.Float()
+
+        class Meta:
+            unknown = EXCLUDE
+
+
+    app = Flask(__name__)
+
+    # because unknown=None was passed, no value is passed during schema loading
+    # as a result, the schema's behavior (EXCLUDE) is used
+    @app.route("/", methods=["POST"])
+    @use_args(RectangleSchema(), location="json", unknown=None)
+    def get(self, args):
+        return f"area = {args['length'] * args['width']}"
+
+
+You can also set ``unknown=None`` when instantiating a parser, or set
+``DEFAULT_UNKNOWN = None`` to make this behavior the default for a parser
+class.
+
+
 When to avoid `use_kwargs`
 --------------------------
 

src/webargs/aiohttpparser.py

@@ -27,7 +27,7 @@ def index(request, args):
 from aiohttp import web
 from aiohttp.web import Request
 from aiohttp import web_exceptions
-from marshmallow import Schema, ValidationError
+from marshmallow import Schema, ValidationError, RAISE
 
 from webargs import core
 from webargs.core import json
@@ -72,6 +72,11 @@ def _find_exceptions() -> None:
 class AIOHTTPParser(AsyncParser):
     """aiohttp request argument parser."""
 
+    DEFAULT_UNKNOWN_BY_LOCATION = {
+        "match_info": RAISE,
+        "path": RAISE,
+        **core.Parser.DEFAULT_UNKNOWN_BY_LOCATION,
+    }
     __location_map__ = dict(
         match_info="load_match_info",
         path="load_match_info",

src/webargs/asyncparser.py

@@ -28,7 +28,7 @@ async def parse(
         req: Request = None,
         *,
         location: str = None,
-        unknown: str = None,
+        unknown: str = core._UNKNOWN_DEFAULT_PARAM,
         validate: Validate = None,
         error_status_code: typing.Union[int, None] = None,
         error_headers: typing.Union[typing.Mapping[str, str], None] = None
@@ -39,7 +39,17 @@ async def parse(
         """
         req = req if req is not None else self.get_default_request()
         location = location or self.location
-        unknown = unknown or self.unknown
+        unknown = (
+            unknown
+            if unknown != core._UNKNOWN_DEFAULT_PARAM
+            else (
+                self.unknown
+                if self.unknown != core._UNKNOWN_DEFAULT_PARAM
+                else self.DEFAULT_UNKNOWN_BY_LOCATION.get(
+                    location, self.DEFAULT_UNKNOWN
+                )
+            )
+        )
         load_kwargs = {"unknown": unknown}
         if req is None:
             raise ValueError("Must pass req object")
@@ -113,7 +123,7 @@ def use_args(
         req: typing.Optional[Request] = None,
         *,
         location: str = None,
-        unknown=None,
+        unknown=core._UNKNOWN_DEFAULT_PARAM,
         as_kwargs: bool = False,
         validate: Validate = None,
         error_status_code: typing.Optional[int] = None,

src/webargs/core.py

@@ -23,6 +23,10 @@
 ]
 
 
+# a value used as the default for arguments, so that when `None` is passed, it
+# can be distinguished from the default value
+_UNKNOWN_DEFAULT_PARAM = "_default"
+
 DEFAULT_VALIDATION_STATUS = 422  # type: int
 
 
@@ -97,15 +101,23 @@ class Parser:
     etc.
 
     :param str location: Default location to use for data
-    :param str unknown: Default value for ``unknown`` in ``parse``,
-        ``use_args``, and ``use_kwargs``
+    :param str unknown: A default value to pass for ``unknown`` when calling the
+        schema's ``load`` method. Defaults to EXCLUDE for non-body
+        locations and RAISE for request bodies. Pass ``None`` to use the
+        schema's setting instead.
     :param callable error_handler: Custom error handler function.
     """
 
     #: Default location to check for data
     DEFAULT_LOCATION = "json"
     #: Default value to use for 'unknown' on schema load
-    DEFAULT_UNKNOWN = None
+    DEFAULT_UNKNOWN = ma.EXCLUDE
+    #: per-location default for 'unknown'
+    DEFAULT_UNKNOWN_BY_LOCATION = {
+        "json": ma.RAISE,
+        "form": ma.RAISE,
+        "json_or_form": ma.RAISE,
+    }
     #: The marshmallow Schema class to use when creating new schemas
     DEFAULT_SCHEMA_CLASS = ma.Schema
     #: Default status code to return for validation errors
@@ -126,12 +138,17 @@ class Parser:
     }
 
     def __init__(
-        self, location=None, *, unknown=None, error_handler=None, schema_class=None
+        self,
+        location=None,
+        *,
+        unknown=_UNKNOWN_DEFAULT_PARAM,
+        error_handler=None,
+        schema_class=None
     ):
         self.location = location or self.DEFAULT_LOCATION
         self.error_callback = _callable_or_raise(error_handler)
         self.schema_class = schema_class or self.DEFAULT_SCHEMA_CLASS
-        self.unknown = unknown or self.DEFAULT_UNKNOWN
+        self.unknown = unknown
 
     def _get_loader(self, location):
         """Get the loader function for the given location.
@@ -219,7 +236,7 @@ def parse(
         req=None,
         *,
         location=None,
-        unknown=None,
+        unknown=_UNKNOWN_DEFAULT_PARAM,
         validate=None,
         error_status_code=None,
         error_headers=None
@@ -235,7 +252,9 @@ def parse(
             default, that means one of ``('json', 'query', 'querystring',
             'form', 'headers', 'cookies', 'files', 'json_or_form')``.
         :param str unknown: A value to pass for ``unknown`` when calling the
-            schema's ``load`` method.
+            schema's ``load`` method. Defaults to EXCLUDE for non-body
+            locations and RAISE for request bodies. Pass ``None`` to use the
+            schema's setting instead.
         :param callable validate: Validation function or list of validation functions
             that receives the dictionary of parsed arguments. Validator either returns a
             boolean or raises a :exc:`ValidationError`.
@@ -248,8 +267,19 @@ def parse(
         """
         req = req if req is not None else self.get_default_request()
         location = location or self.location
-        unknown = unknown or self.unknown
-        load_kwargs = {"unknown": unknown}
+        # precedence order: explicit, instance setting, default per location, default
+        unknown = (
+            unknown
+            if unknown != _UNKNOWN_DEFAULT_PARAM
+            else (
+                self.unknown
+                if self.unknown != _UNKNOWN_DEFAULT_PARAM
+                else self.DEFAULT_UNKNOWN_BY_LOCATION.get(
+                    location, self.DEFAULT_UNKNOWN
+                )
+            )
+        )
+        load_kwargs = {"unknown": unknown} if unknown else {}
         if req is None:
             raise ValueError("Must pass req object")
         data = None
@@ -311,7 +341,7 @@ def use_args(
         req=None,
         *,
         location=None,
-        unknown=None,
+        unknown=_UNKNOWN_DEFAULT_PARAM,
         as_kwargs=False,
         validate=None,
         error_status_code=None,

src/webargs/flaskparser.py

@@ -23,6 +23,8 @@ def user_detail(args, uid):
 import flask
 from werkzeug.exceptions import HTTPException
 
+import marshmallow as ma
+
 from webargs import core
 from webargs.multidictproxy import MultiDictProxy
 
@@ -48,6 +50,11 @@ def is_json_request(req):
 class FlaskParser(core.Parser):
     """Flask request argument parser."""
 
+    DEFAULT_UNKNOWN_BY_LOCATION = {
+        "view_args": ma.RAISE,
+        "path": ma.RAISE,
+        **core.Parser.DEFAULT_UNKNOWN_BY_LOCATION,
+    }
     __location_map__ = dict(
         view_args="load_view_args",
         path="load_view_args",

src/webargs/pyramidparser.py

@@ -30,6 +30,8 @@ def hello_world(request, args):
 from webob.multidict import MultiDict
 from pyramid.httpexceptions import exception_response
 
+import marshmallow as ma
+
 from webargs import core
 from webargs.core import json
 from webargs.multidictproxy import MultiDictProxy
@@ -42,6 +44,11 @@ def is_json_request(req):
 class PyramidParser(core.Parser):
     """Pyramid request argument parser."""
 
+    DEFAULT_UNKNOWN_BY_LOCATION = {
+        "matchdict": ma.RAISE,
+        "path": ma.RAISE,
+        **core.Parser.DEFAULT_UNKNOWN_BY_LOCATION,
+    }
     __location_map__ = dict(
         matchdict="load_matchdict",
         path="load_matchdict",