New davclient.get_davclient-method

This method yields a davclient object based on ...

* environment variables
* config file
* test config

For tests, rigging up and tearing down test servers is now done through davclient context.  This should make it easy to make doctests.

Author: tobixen

CHANGELOG.md

@@ -41,12 +41,14 @@ This will be the last minor release before 2.0.  The scheduling support has been
 
 ### Added
 
+* `auto_conn` is more complete now - it could already read from test config, now it can read from environment (including environment variable for reading from test config and for locating the config file).  While the `auto_conn` itself is tested in the functional tests, the code for reading the config file (and all the corner cases) is not tested.  It's allowable with a yaml config file, but the yaml module is not included in the dependencies yet ... so late imports as for now.
 * New option `event.save(all_recurrences=True)` to edit the whole series when saving a modified recurrence.  Part of https://github.com/python-caldav/caldav/pull/500
-* New methods `Event.set_dtend` and `CalendarObjectResource.set_end`. by @tobixen in https://github.com/python-caldav/caldav/pull/499
+* New methods `Event.set_dtend` and `CalendarObjectResource.set_end`. https://github.com/python-caldav/caldav/pull/499
 
-### Refactoring
+### Refactoring and tests
 
 * Partially tossed out all internal usage of vobject, https://github.com/python-caldav/caldav/issues/476.  Refactoring and removing unuseful code.  Parts of this work was accidentally committed directly to master, 2f61dc7adbe044eaf43d0d2c78ba96df09201542, the rest was piggybaced in through  https://github.com/python-caldav/caldav/pull/500.
+* Server-specific setup- and teardown-methods (used for spinning up test servers in the tests) is now executed through the DAVClient context manager.  This will allow doctests to run easily.
 
 ### Time spent and roadmap
 
@@ -80,7 +82,7 @@ Python 3.7 is no longer tested (dependency problems) - but it should work.  Plea
 
 * Minor code cleanups by github user @ArtemIsmagilov in https://github.com/python-caldav/caldav/pull/456
 * The very much overgrown `objects.py`-file has been split into three - https://github.com/python-caldav/caldav/pull/483
-* Refactor compatibility issues by @tobixen in https://github.com/python-caldav/caldav/pull/484
+* Refactor compatibility issues https://github.com/python-caldav/caldav/pull/484
 * Refactoring of `multiget` in https://github.com/python-caldav/caldav/pull/492
 
 ### Documentation

caldav/config.py

@@ -0,0 +1,100 @@
+import json
+
+## This is being moved from my plann library.  The code itself will be introduced into caldav 2.0, but proper test code and documentation will come in a later release (2.1?)
+
+
+## TODO TODO TODO - write test code for all the corner cases
+## TODO TODO TODO - write documentation of config format
+def expand_config_section(config, section="default", blacklist=None):
+    """
+    In the "normal" case, will return [ section ]
+
+    We allow:
+
+    * * includes all sections in config file
+    * "Meta"-sections in the config file with the keyword "contains" followed by a list of section names
+    * Recursive "meta"-sections
+    * Glob patterns (work_* for all sections starting with work_)
+    * Glob patterns in "meta"-sections
+    """
+    ## Optimizating for a special case.  The results should be the same without this optimization.
+    if section == "*":
+        return [x for x in config if not config[x].get("disable", False)]
+
+    ## If it's not a glob-pattern ...
+    if set(section).isdisjoint(set("[*?")):
+        ## If it's referring to a "meta section" with the "contains" keyword
+        if "contains" in config[section]:
+            results = []
+            if not blacklist:
+                blacklist = set()
+            blacklist.add(section)
+            for subsection in config[section]["contains"]:
+                if not subsection in results and not subsection in blacklist:
+                    for recursivesubsection in expand_config_section(
+                        config, subsection, blacklist
+                    ):
+                        if not recursivesubsection in results:
+                            results.append(recursivesubsection)
+            return results
+        else:
+            ## Disabled sections should be ignored
+            if config.get("section", {}).get("disable", False):
+                return []
+
+            ## NORMAL CASE - return [ section ]
+            return [section]
+    ## section name is a glob pattern
+    matching_sections = [x for x in config if fnmatch(x, section)]
+    results = set()
+    for s in matching_sections:
+        if set(s).isdisjoint(set("[*?")):
+            results.update(expand_config_section(config, s))
+        else:
+            ## Section names shouldn't contain []?* ... but in case they do ... don't recurse
+            results.add(s)
+    return results
+
+
+def config_section(config, section="default"):
+    if section in config and "inherits" in config[section]:
+        ret = config_section(config, config[section]["inherits"])
+    else:
+        ret = {}
+    if section in config:
+        ret.update(config[section])
+    return ret
+
+
+def read_config(fn, interactive_error=False):
+    ## This can probably be refactored into fewer lines ...
+    try:
+        try:
+            with open(fn, "rb") as config_file:
+                return json.load(config_file)
+        except json.decoder.JSONDecodeError:
+            ## Late import.  yaml is external module,
+            ## and not included in the requirements as for now.
+            ## TODO: should wrap it in try: ... except: log readable error
+            import yaml
+
+            try:
+                with open(fn, "rb") as config_file:
+                    return yaml.load(config_file, yaml.Loader)
+            except yaml.scanner.ScannerError:
+                logging.error(
+                    "config file exists but is neither valid json nor yaml.  Check the syntax."
+                )
+
+    except FileNotFoundError:
+        ## File not found
+        logging.info("no config file found")
+    except ValueError:
+        if interactive_error:
+            logging.error(
+                "error in config file.  Be aware that the interactive configuration will ignore and overwrite the current broken config file",
+                exc_info=True,
+            )
+        else:
+            logging.error("error in config file.  It will be ignored", exc_info=True)
+    return {}

caldav/davclient.py

@@ -2,6 +2,7 @@
 import logging
 import os
 import sys
+import warnings
 from types import TracebackType
 from typing import Any
 from typing import cast
@@ -443,6 +444,8 @@ def __init__(
         """
         headers = headers or {}
 
+        ## Deprecation TODO: give a warning, user should use get_davclient or auto_calendar instead
+
         self.session = niquests.Session(multiplexed=True)
 
         log.debug("url: " + str(url))
@@ -495,6 +498,12 @@ def __init__(
         self._principal = None
 
     def __enter__(self) -> Self:
+        ## Used for tests, to set up a temporarily test server
+        if hasattr(self, "setup"):
+            try:
+                self.setup()
+            except:
+                self.setup(self)
         return self
 
     def __exit__(
@@ -504,6 +513,12 @@ def __exit__(
         traceback: Optional[TracebackType],
     ) -> None:
         self.close()
+        ## Used for tests, to tear down a temporarily test server
+        if hasattr(self, "teardown"):
+            try:
+                self.teardown()
+            except:
+                self.teardown(self)
 
     def close(self) -> None:
         """
@@ -845,7 +860,8 @@ def request(
 
 
 def auto_calendars(
-    configfile: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
+    config_file: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
+    config_section="default",
     testconfig=False,
     environment: bool = True,
     config_data: dict = None,
@@ -864,8 +880,26 @@ def auto_calendar(*largs, **kwargs) -> Iterable["Calendar"]:
     return next(auto_calendars(*largs, **kwargs), None)
 
 
-def auto_conn(
-    configfile: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
+def auto_conn(*largs, **kwargs):
+    """A quite stubbed verison of get_davclient was included in the
+    v1.5-release as auto_conn, but renamed a few days later.  Probably
+    nobody except my caldav tester project uses auto_conn, but as a
+    thumb of rule anything released should stay "deprecated" for at
+    least one major release before being removed.
+
+    TODO: remove in version 3.0
+    """
+    warnings.warn(
+        "auto_conn was renamed get_davclient",
+        DeprecationWarning,
+        stacklevel=2,
+    )
+    return get_davclient(*largs, **kwargs)
+
+
+def get_davclient(
+    config_file: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
+    config_section="default",
     testconfig=False,
     environment: bool = True,
     config_data: dict = None,
@@ -883,27 +917,29 @@ def auto_conn(
 
     * Data from the given dict
     * Environment variables prepended with "CALDAV_"
+    * Environment variables PYTHON_CALDAV_USE_TEST_SERVER and CALDAV_CONFIG_FILE will be honored if environment is set
     * Data from `./tests/conf.py` or `./conf.py` (this includes the possibility to spin up a test server)
     * Configuration file.  Documented in the plann project as for now.  (TODO - move it)
-
     """
     if config_data:
         return DAVClient(**config_data)
 
-    if testconfig:
+    if testconfig or (environment and os.environ.get("PYTHON_CALDAV_USE_TEST_SERVER")):
         sys.path.insert(0, "tests")
         sys.path.insert(1, ".")
         ## TODO: move the code from client into here
         try:
             from conf import client
 
+            idx = None
+            if name:
+                try:
+                    idx = int(name)
+                    name = None
+                except ValueError:
+                    pass
             try:
-                idx = int(name)
-                name = None
-            except ValueError:
-                idx = None
-            try:
-                conn = client(idx, name, **config_data)
+                conn = client(idx, name, **(config_data or {}))
                 if conn:
                     return conn
             except:
@@ -914,11 +950,28 @@ def auto_conn(
             sys.path = sys.path[2:]
 
     if environment:
-        raise NotImplementedError(
-            "Not possible to configure the caldav server through environmental variables yet"
-        )
-
-    if configfile:
-        raise NotImplementedError(
-            "Support for configuration file not made yet (TODO: copy the code from the plann tool)"
-        )
+        conf = {}
+        for conf_key in (x for x in os.environ if x.startswith("CALDAV_")):
+            conf[conf_key[7:]] = os.environ[conf_key].lower()
+        if conf:
+            return DAVClient(**conf)
+        config_file = os.environ.get("CALDAV_CONFIG_FILE")
+
+    if config_file:
+        ## late import in 2.0, as the config stuff isn't properly tested
+        from . import config
+
+        cfg = config.read_config(config_file)
+        if cfg:
+            section = config.config_section(cfg, config_section)
+            conn_params = {}
+            for k in section:
+                if k.startswith("caldav_") and section[k]:
+                    key = k[7:]
+                    if key == "pass":
+                        key = "password"
+                    if key == "user":
+                        key = "username"
+                    conn_params[key] = section[k]
+                if conn_params:
+                    return DAVClient(**conn_params)

tests/conf.py

@@ -275,7 +275,7 @@ def client(
             )
             kwargs_.pop(kw)
     conn = DAVClient(**kwargs_)
-    setup(conn)
+    conn.setup = setup
     conn.teardown = teardown
     conn.incompatibilities = kwargs.get("incompatibilities")
     conn.server_name = name

tests/test_caldav.py

@@ -8,9 +8,12 @@
 belong in test_caldav_unit.py
 """
 import codecs
+import json
 import logging
+import os
 import random
 import sys
+import tempfile
 import threading
 import time
 import uuid
@@ -41,6 +44,7 @@
 from caldav import compatibility_hints
 from caldav.davclient import DAVClient
 from caldav.davclient import DAVResponse
+from caldav.davclient import get_davclient
 from caldav.elements import cdav
 from caldav.elements import dav
 from caldav.elements import ical
@@ -432,6 +436,56 @@
 )
 
 
+@pytest.mark.skipif(
+    not caldav_servers,
+    reason="Requirement: at least one working server in conf.py. The tail object of the server list will be chosen, that is typically the LocalRadicale or LocalXandikos server.",
+)
+class TestGetDAVClient:
+    """
+    Tests for get_davclient and auto_calendars.
+
+    """
+
+    def testTestConfig(self):
+        with get_davclient(
+            testconfig=True, environment=False, name=-1, config_file=False
+        ) as conn:
+            assert conn.principal()
+
+    def testEnvironment(self):
+        os.environ["PYTHON_CALDAV_USE_TEST_SERVER"] = "1"
+        with get_davclient(environment=True, config_file=False, name="-1") as conn:
+            assert conn.principal()
+            for key in ("url", "username", "password", "proxy"):
+                if key in caldav_servers[-1]:
+                    os.environ[f"CALDAV_{key.upper()}"] = caldav_servers[-1][key]
+            with get_davclient(
+                testconfig=False, environment=True, config_file=False
+            ) as conn2:
+                assert conn2.principal()
+
+    def testConfigfile(self):
+        ## start up a server
+        with get_davclient(
+            testconfig=True, environment=False, name=-1, config_file=False
+        ) as conn:
+            config = {}
+            for key in ("url", "username", "password", "proxy"):
+                if key in caldav_servers[-1]:
+                    config[f"caldav_{key}"] = caldav_servers[-1][key]
+
+            with tempfile.NamedTemporaryFile(
+                delete=True, encoding="utf-8", mode="w"
+            ) as tmp:
+                json.dump({"default": config}, tmp)
+                tmp.flush()
+                os.fsync(tmp.fileno())
+                with get_davclient(
+                    config_file=tmp.name, testconfig=False, environment=False
+                ) as conn2:
+                    assert conn2.principal()
+
+
 @pytest.mark.skipif(
     not rfc6638_users, reason="need rfc6638_users to be set in order to run this test"
 )
@@ -490,7 +544,7 @@ def teardown_method(self):
             except error.NotFoundError:
                 pass
         for c in self.clients:
-            c.teardown(c)
+            c.__exit__()
 
     ## TODO
     # def testFreeBusy(self):
@@ -609,6 +663,7 @@ def setup_method(self):
             self.testcal_id2 = "pythoncaldav-test2"
 
         self.caldav = client(**self.server_params)
+        self.caldav.__enter__()
 
         if self.check_compatibility_flag("rate_limited"):
             self.caldav.request = _delay_decorator(self.caldav.request)