compatibility hints - refactoring efforts

In 1.5.0, I moved the compability matrix from the tests directory and into the project itself - now I'm doing a major overhaul of it.  This change is not much visible for end users yet - but already now it's possible to configure "compatibility hints" when setting up the davclient, and the idea is that different kind of workarounds may be applied depending on the compatibility-matrix.  Search without comp-type is wonky on many servers, now the `search`-method will automatically deliver a union of a search of the three different comp-types if a comp-type is not set in the parameters *and* it's declared that the compatibility matrix does not work.  In parallel I'm developing a stand-alone tool caldav-server-tester to check the compatibility of a caldav server.

This is "work in progress", the rewriting of old compatibility hints is incomplete.  It's a lot of work, so it will take some time to fully complete it.

Author: tobixen

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## IMPORTANT - "flapping" changeset!
 
-In 2.0.0 I dropped the dependency on the requests library as the project is stagnant and adopted niquests, a fork of the requests library.  It's a small change, and three github issues could be closed just by doing this switch.  In addition niquests supports HTTP/2 and is one possible way forward for implementing async support.  However, the change has proven controversial, shortly after releasing 2.0 I had to revert back to requests and release 2.0.1.  Right after releasing 2.0.1, I reverted again so that the master branch is using niquests.
+The requests library is stagnant, so in 2.0.0 I replaced it with a fork niquests.  It's a very tiny changeset, and three github issues could be closed just by doing this switch.  In addition niquests supports HTTP/2 and is one possible way forward for implementing async support.  However, the change has proven controversial, shortly after releasing 2.0 I had to revert back to requests and release 2.0.1.  Right after releasing 2.0.1, I reverted again so that the master branch is using niquests.
 
 My plan now is to keep doing dual releases while maintaining the 2.x-series - one with niquests and one with requests.  You are encouraged to make an informed decision on weather you are most comfortable with the stable but stagnant requests, or the niquests fork and choose your version accordingly.  When I'm starting to work on 3.0 (which will support async requests), I will think deeply about this and either choose niquests, httpx, or (it's always possible to hope!) requests 3.0.  **Your opinion is valuable for me**.  Feel free to comment on https://github.com/python-caldav/caldav/issues/457,  https://github.com/python-caldav/caldav/issues/530 or https://github.com/jawah/niquests/issues/267 if you have a github account, and if not you can reach out at python-http@plann.no
 
@@ -16,6 +16,10 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0
 
 ## [Unreleased]
 
+### Changed
+
+* In 1.5.0, I moved the compability matrix from the tests directory and into the project itself - now I'm doing a major overhaul of it.  This change is not much visible for end users yet - but already now it's possible to configure "compatibility hints" when setting up the davclient, and the idea is that different kind of workarounds may be applied depending on the compatibility-matrix.  Search without comptype is wonky on many servers, now the `search`-method will automatically deliver a union of a search of the three different comptypes if a comptype is not set in the parameters *and* it's declared that the compatibility matrix does not work.  In parallell I'm developing a stand-alone tool caldav-server-tester to check the compatibility of a caldav server.
+
 ### Fixes
 
 * A search without filtering on comp-type on a calendar containing a mix of events, journals and tasks should return a mix of such.  (All the examples in the RFC includes the comp-type filter, so many servers does not support this).  There were a bug in the auto-detection of comp-type, so tasks would typically be wrapped as events or vice-versa.  https://github.com/python-caldav/caldav/pull/540

README.md

@@ -9,12 +9,9 @@ Features:
  * search events by dates
  * etc.
 
-See the file [examples/basic_usage_examples.py](examples/basic_usage_examples.py) to get started.
+The documentation was freshed up a bit as of version 2.0, and is available at https://caldav.readthedocs.io/
 
-Links:
-
- * [Pypi](https://pypi.org/project/caldav)
- * [Documentation](docs/source/index.rst) - should be automatically mirrored on https://caldav.readthedocs.io/en/latest/
+The package is published at [Pypi](https://pypi.org/project/caldav)
 
 Licences:
 

caldav/collection.py

@@ -763,7 +763,7 @@ def search(
         xml=None,
         comp_class: Optional[_CC] = None,
         todo: Optional[bool] = None,
-        include_completed: bool = False,
+        include_completed: bool = None,
         sort_keys: Sequence[str] = (),
         sort_reverse: bool = False,
         expand: bool = False,
@@ -860,21 +860,18 @@ def search(
         if todo and not include_completed:
             matches1 = self.search(
                 todo=True,
-                comp_class=comp_class,
                 ignore_completed1=True,
                 include_completed=True,
                 **kwargs,
             )
             matches2 = self.search(
                 todo=True,
-                comp_class=comp_class,
                 ignore_completed2=True,
                 include_completed=True,
                 **kwargs,
             )
             matches3 = self.search(
                 todo=True,
-                comp_class=comp_class,
                 ignore_completed3=True,
                 include_completed=True,
                 **kwargs,
@@ -903,30 +900,48 @@ def search(
                 raise error.ConsistencyError(
                     "Inconsistent usage parameters: xml together with other search options"
                 )
+
+            ## For some of the workarounds below, we will do a recursive search, with all
+            ## those arguments:
+            kwargs2 = {
+                "include_completed": include_completed,
+                "sort_reverse": sort_reverse,
+                "expand": expand,
+                "server_expand": server_expand,
+                "split_expanded": split_expanded,
+                "props": props,
+            }
+
+            if not comp_class and not self.client.features.check_support(
+                "search.comp-type-optional"
+            ):
+                if kwargs2["include_completed"] is None:
+                    kwargs2["include_completed"] = True
+                objects = (
+                    self.search(event=True, **kwargs2, **kwargs)
+                    + self.search(todo=True, **kwargs2, **kwargs)
+                    + self.search(journal=True, **kwargs2, **kwargs)
+                )
+                self.sort_objects(objects, sort_keys, sort_reverse)
+                return objects
+
             try:
                 (response, objects) = self._request_report_build_resultlist(
                     xml, comp_class, props=props
                 )
+
             except error.ReportError as err:
-                ## Hack for some calendar servers
-                ## yielding 400 if the search does not include compclass.
-                ## Partial fix for https://github.com/python-caldav/caldav/issues/401
-                ## This assumes the client actually wants events and not tasks
-                ## The calendar server in question did not support tasks
-                ## However the most correct would probably be to join
-                ## events, tasks and journals.
-                ## TODO: we need server compatibility hints!
-                ## https://github.com/python-caldav/caldav/issues/402
-                if not comp_class and not "400" in err.reason:
+                ## This is only for backward compatibility.  The logic is even flawed.
+                ## But it does partially fix https://github.com/python-caldav/caldav/issues/401
+                if (
+                    self.client.features.backward_compatibility_mode
+                    and not comp_class
+                    and not "400" in err.reason
+                ):
                     return self.search(
-                        event=True,
-                        include_completed=include_completed,
                         sort_keys=sort_keys,
                         sort_reverse=sort_reverse,
-                        expand=expand,
-                        server_expand=server_expand,
-                        split_expanded=split_expanded,
-                        props=props,
+                        *kwargs2,
                         **kwargs,
                     )
                 raise
@@ -976,6 +991,17 @@ def search(
             for o in objects_:
                 objects.extend(o.split_expanded())
 
+        ## partial workaround for https://github.com/python-caldav/caldav/issues/201
+        for obj in objects:
+            try:
+                obj.load(only_if_unloaded=True)
+            except:
+                pass
+
+        self.sort_objects(objects, sort_keys, sort_reverse)
+        return objects
+
+    def sort_objects(self, objects, sort_keys, sort_reverse):
         def sort_key_func(x):
             ret = []
             comp = x.icalendar_component
@@ -1003,7 +1029,6 @@ def sort_key_func(x):
                     > datetime.now().strftime("%F%H%M%S")
                 ),
             }
-            ## ref https://github.com/python-caldav/caldav/issues/448 - allow strings instead of a sequence here
             for sort_key in sort_keys:
                 val = comp.get(sort_key, None)
                 if val is None:
@@ -1020,19 +1045,11 @@ def sort_key_func(x):
             return ret
 
         if sort_keys:
+            ## ref https://github.com/python-caldav/caldav/issues/448 - allow strings instead of a sequence here
             if isinstance(sort_keys, str):
                 sort_keys = (sort_keys,)
             objects.sort(key=sort_key_func, reverse=sort_reverse)
 
-        ## partial workaround for https://github.com/python-caldav/caldav/issues/201
-        for obj in objects:
-            try:
-                obj.load(only_if_unloaded=True)
-            except:
-                pass
-
-        return objects
-
     def build_search_xml_query(
         self,
         comp_class=None,