Config file, examples and misc

Various unrelated work got a bit mixed up here:

* Work on finding and parsing a config file, including some test code and documentation.  Fixes #485
* Brushing up the examples.  Fixing test code for some examples. Adding more information on the examples in the documentation.  Ref Road Map at #474
* Old test code utilizing vobjects has been rewritten from `obj.instance` to `obj.vobject_instance` to reduce the number of deprecation warnings when running tests


#507

Author: tobixen

CHANGELOG.md

@@ -10,14 +10,25 @@ This project should adhere to [Semantic Versioning](https://semver.org/spec/v2.0
 
 ### Deprecated
 
-* The `event.instance` property currently yields a vobject.  For quite many years people have asked for the python vobject library to be replaced with the python icalendar objects, but I haven't been able to do that due to backward compatibility.  In version 2.0 deprecation warnings will be given whenever someone uses the `event.instance` property.  In 3.0, perhaps `event.instance` will yield a `icalendar` instance.
+* The `event.instance` property currently yields a vobject.  For quite many years people have asked for the python vobject library to be replaced with the python icalendar objects, but I haven't been able to do that due to backward compatibility.  In version 2.0 deprecation warnings will be given whenever someone uses the `event.instance` property.  In 3.0, perhaps `event.instance` will yield a `icalendar` instance.  Old test code has been updated to use `.vobject_instance` instead of `.instance`.
 * `calendar.date_search` - use `calendar.search` instead.  (this one has been deprecated for a while, but only with info-logging)
 * `davclient.auto_conn` that was introduced just some days ago has already been renamed to `davclient.get_davclient`.
 
 ### Added
 
 * `event.component` is now an alias for `event.icalendar_component`.
-* `get_davclient` (earlier called `auto_conn`) is more complete now - it could already read from test config, now it can read from environment (including environment variable for reading from test config and for locating the config file).  While the `auto_conn` itself is tested in the functional tests, the code for reading the config file (and all the corner cases) is not tested.  It's allowable with a yaml config file, but the yaml module is not included in the dependencies yet ... so late imports as for now.  - https://github.com/python-caldav/caldav/pull/502 - https://github.com/python-caldav/caldav/issues/485
+* `get_davclient` (earlier called `auto_conn`) is more complete now - https://github.com/python-caldav/caldav/pull/502 - https://github.com/python-caldav/caldav/issues/485 - https://github.com/python-caldav/caldav/pull/507
+  * It can read from environment (including environment variable for reading from test config and for locating the config file).
+  * It can read from a config file.  New parameter `check_config_file`, defaults to true
+  * It will probe default locations for the config file (`~/.config/caldav/calendar.conf`, `~/.config/caldav/calendar.yaml`, `~/.config/caldav/calendar.json`, `~/.config/calendar.conf`, `/etc/calendar.conf`, `/etc/caldav/calendar.conf` as for now)
+  * Improved tests (but no test for inheritance yet).
+  * Documentation, linked up from the reference section of the doc.
+  * It's allowable with a yaml config file, but the yaml module is not included in the dependencies yet ... so late imports as for now, and the import is wrapped in a try/except-block
+* Looked through and brushed up the examples, two of them are now executed by the unit tests.  Added a doc section on the examples.
+
+### Fixes
+
+* Support for Lark/Feishu got broken in the 1.6-release.  Issue found and fixed by Hongbin Yang (github user @zealseeker) in https://github.com/python-caldav/caldav/issues/505 and https://github.com/python-caldav/caldav/pull/506
 
 ### Changed
 

caldav/config.py

@@ -1,4 +1,6 @@
 import json
+import logging
+import os
 
 """
 This configuration parsing code was just copied from my plann library (and will be removed from there at some point in the future).  It's lacking tests, documentation and ... generally just lacking.
@@ -71,23 +73,42 @@ def config_section(config, section="default"):
 
 
 def read_config(fn, interactive_error=False):
+    if not fn:
+        cfgdir = f"{os.environ.get('HOME', '/')}/.config/"
+        for config_file in (
+            f"{cfgdir}/caldav/calendar.conf",
+            f"{cfgdir}/caldav/calendar.yaml"
+            f"{cfgdir}/caldav/calendar.json"
+            f"{cfgdir}/calendar.conf",
+            "/etc/calendar.conf",
+            "/etc/caldav/calendar.conf",
+        ):
+            cfg = read_config(config_file)
+            if cfg:
+                return cfg
+        return None
+
     ## This can probably be refactored into fewer lines ...
     try:
         try:
             with open(fn, "rb") as config_file:
                 return json.load(config_file)
         except json.decoder.JSONDecodeError:
-            ## Late import.  yaml is external module,
+            ## Late import, wrapped in try/except.  yaml is external module,
             ## and not included in the requirements as for now.
-            ## TODO: should wrap it in try: ... except: log readable error
-            import yaml
-
             try:
-                with open(fn, "rb") as config_file:
-                    return yaml.load(config_file, yaml.Loader)
-            except yaml.scanner.ScannerError:
+                import yaml
+
+                try:
+                    with open(fn, "rb") as config_file:
+                        return yaml.load(config_file, yaml.Loader)
+                except yaml.scanner.ScannerError:
+                    logging.error(
+                        f"config file {fn} exists but is neither valid json nor yaml.  Check the syntax."
+                    )
+            except ImportError:
                 logging.error(
-                    "config file exists but is neither valid json nor yaml.  Check the syntax."
+                    f"config file {fn} exists but is not valid json, and pyyaml is not installed."
                 )
 
     except FileNotFoundError:

caldav/davclient.py

@@ -888,7 +888,7 @@ def request(
 
 
 def auto_calendars(
-    config_file: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
+    config_file: str = None,
     config_section="default",
     testconfig=False,
     environment: bool = True,
@@ -928,9 +928,10 @@ def auto_conn(*largs, config_data: dict = None, **kwargs):
 
 
 def get_davclient(
-    config_file: str = f"{os.environ.get('HOME')}/.config/calendar.conf",
-    config_section="default",
-    testconfig=False,
+    check_config_file: bool = True,
+    config_file: str = None,
+    config_section: str = None,
+    testconfig: bool = False,
     environment: bool = True,
     name: str = None,
     **config_data,
@@ -977,16 +978,26 @@ def get_davclient(
 
     if environment:
         conf = {}
-        for conf_key in (x for x in os.environ if x.startswith("CALDAV_")):
+        for conf_key in (
+            x
+            for x in os.environ
+            if x.startswith("CALDAV_") and not x.startswith("CALDAV_CONFIG")
+        ):
             conf[conf_key[7:].lower()] = os.environ[conf_key]
         if conf:
             return DAVClient(**conf)
-        config_file = os.environ.get("CALDAV_CONFIG_FILE")
+        if not config_file:
+            config_file = os.environ.get("CALDAV_CONFIG_FILE")
+        if not config_section:
+            config_section = os.enviorn.get("CALDAV_CONFIG_SECTION")
 
-    if config_file:
+    if check_config_file:
         ## late import in 2.0, as the config stuff isn't properly tested
         from . import config
 
+        if not config_section:
+            config_section = "default"
+
         cfg = config.read_config(config_file)
         if cfg:
             section = config.config_section(cfg, config_section)

docs/source/configfile.rst

@@ -0,0 +1,48 @@
+==================
+Config file format
+==================
+
+
+The :class:`davclient.get_davclient` method (and perhaps in 2.1, also ``davclient.get_calendar``) can read from a config file.  It will look for it in the following locations:
+
+* ``$HOME/.config/caldav/calendar.conf``
+* ``$HOME/.config/caldav/calendar.yaml``
+* ``$HOME/.config/caldav/calendar.json``
+* ``$HOME/.config/calendar.conf``
+* ``/etc/calendar.conf``
+
+The config file has to be valid json or yaml (support for toml and Apple pkl may be considered).
+
+The config file is expected to be divided in sections, where each section can describe locations and credentials to a CalDAV server, a CalDAV calendar or a collection of calendars/servers.  As of version 2.0, only the first is supported.
+
+A config section can be given either through parameters to :class:`davclient.get_davclient` or by enviornment variable ``CALDAV_CONFIG_SECTION``.  If no section is given, the ``default`` section is used.
+
+Connection parameters
+=====================
+
+The section should contain configuration keys and values.  All configuration keys starting with ``caldav_`` is considered to be connection parameters and is passed to the DAVClient object.  Typically,  ``caldav_url``, ``caldav_username`` and ``caldav_password`` should be passed.
+
+Calendar parameters
+===================
+
+Not implemented yet.
+
+Probably in version 2.1 or version 2.2, ``calendar_name``, ``calendar_id`` and ``calendar_url`` can be used to specify a calendar.
+
+Inheritance and collections
+===========================
+
+A section may ``inherit`` another section.  This may typically be used if having several sections in the config file corresponding to the same server/user but different calendars, or several sections corresponding to the same calendar server, but different users.
+
+If a section ``contains`` different other sections, it's efficiently a collection of calendars.  This is not relevant for 2.0 though.
+
+Simple example
+==============
+
+.. code-block: yaml
+
+   ---
+   default:
+       caldav_url: http://caldav.example.com/dav/
+       caldav_user: tor
+       caldav_pass: hunter2

docs/source/index.rst

@@ -15,6 +15,7 @@ Contents
    about
    tutorial
    reference
+   examples
 
 ====================
  Indices and tables

docs/source/reference.rst

@@ -10,6 +10,7 @@ Contents
 .. toctree::
    :maxdepth: 1
 
+   configfile.md
    caldav/davclient
    caldav/davobject
    caldav/collection

docs/source/tutorial.rst

@@ -5,7 +5,9 @@ Tutorial
 In this tutorial you should learn basic usage of the python CalDAV
 client library.  You are encouraged to copy the code examples into a
 file and add a ``breakpoint()`` inside the with-block so you can
-inspect the return objects you get from the library calls.
+inspect the return objects you get from the library calls.  Do not
+name your file `caldav.py` or `calendar.py`, this may break some
+imports.
 
 To follow this tutorial as intended, each code block should be run
 towards a clean-slate Radicale server.  To do this, you need:

examples/basic_usage_examples.py

@@ -8,6 +8,7 @@
 sys.path.insert(0, ".")
 
 import caldav
+from caldav.davclient import get_davclient
 
 ## DO NOT name your file calendar.py or caldav.py!  We've had several
 ## issues filed, things break because the wrong files are imported.
@@ -29,13 +30,10 @@ def run_examples():
     ## The client object stores http session information, username, password, etc.
     ## As of 1.0, Initiating the client object will not cause any server communication,
     ## so the credentials aren't validated.
+    ## get_davclient will try to read credentials and url from environment variables
+    ## and config file.
     ## The client object can be used as a context manager, like this:
-    with caldav.DAVClient(
-        url=caldav_url,
-        username=username,
-        password=password,
-        headers=headers,  # Optional parameter to set HTTP headers on each request if needed
-    ) as client:
+    with get_davclient() as client:
         ## Typically the next step is to fetch a principal object.
         ## This will cause communication with the server.
         my_principal = client.principal()
@@ -132,20 +130,20 @@ def read_modify_event_demo(event):
     ## event.icalendar_instance gives an icalendar instance - which
     ## normally would be one icalendar calendar object containing one
     ## subcomponent.  Quite often the fourth property,
-    ## icalendar_component is preferable - it gives us the component -
-    ## but be aware that if the server returns a recurring events with
-    ## exceptions, event.icalendar_component will ignore all the
-    ## exceptions.
-    uid = event.icalendar_component["uid"]
+    ## icalendar_component (now available just as .component) is
+    ## preferable - it gives us the component - but be aware that if
+    ## the server returns a recurring events with exceptions,
+    ## event.icalendar_component will ignore all the exceptions.
+    uid = event.component["uid"]
 
     ## Let's correct that typo using the icalendar library.
-    event.icalendar_component["summary"] = event.icalendar_component["summary"].replace(
+    event.component["summary"] = event.component["summary"].replace(
         "celebratiuns", "celebrations"
     )
 
     ## timestamps (DTSTAMP, DTSTART, DTEND for events, DUE for tasks,
     ## etc) can be fetched using the icalendar library like this:
-    dtstart = event.icalendar_component.get("dtstart")
+    dtstart = event.component.get("dtstart")
 
     ## but, dtstart is not a python datetime - it's a vDatetime from
     ## the icalendar package.  If you want it as a python datetime,
@@ -156,13 +154,13 @@ def read_modify_event_demo(event):
 
     ## We can modify it:
     if dtstart:
-        event.icalendar_component["dtstart"].dt = dtstart.dt + timedelta(seconds=3600)
+        event.component["dtstart"].dt = dtstart.dt + timedelta(seconds=3600)
 
     ## And finally, get the casing correct
     event.data = event.data.replace("norwegian", "Norwegian")
 
     ## Note that this is not quite thread-safe:
-    icalendar_component = event.icalendar_component
+    icalendar_component = event.component
     ## accessing the data (and setting it) will "disconnect" the
     ## icalendar_component from the event
     event.data = event.data
@@ -183,10 +181,7 @@ def read_modify_event_demo(event):
     ## Finally, let's verify that the correct data was saved
     calendar = event.parent
     same_event = calendar.event_by_uid(uid)
-    assert (
-        same_event.icalendar_component["summary"]
-        == "Norwegian national day celebrations"
-    )
+    assert same_event.component["summary"] == "Norwegian national day celebrations"
 
 
 def search_calendar_demo(calendar):

examples/get_events_example.py

@@ -3,8 +3,9 @@
 
 from caldav.davclient import get_davclient
 
-## Code contributed by Крылов Александр.  Minor changes and quite some
-## comments by Tobias Brox.
+## Code contributed by Крылов Александр.
+## Minor changes by Tobias Brox.
+## All comments by Tobias Brox.
 ## Set CALDAV_USERNAME, CALDAV_URL and CALDAV_PASSWORD through
 ## environment variables before running this example
 
@@ -37,6 +38,9 @@ def fill_event(component, calendar) -> dict[str, str]:
     cur["calendar"] = f"{calendar}"
     cur["summary"] = component.get("summary")
     cur["description"] = component.get("description")
+    ## month/day/year time? Never ever do that!
+    ## It's one of the most confusing date formats ever!
+    ## Use year-month-day time instead ... https://xkcd.com/1179/
     cur["start"] = component.start.strftime("%m/%d/%Y %H:%M")
     endDate = component.end
     if endDate:

examples/scheduling_examples.py

@@ -1,3 +1,7 @@
+## NOTE!  This is currently NOT tested.  It may and may not work.
+## Please reach out if you need help with scheduling ...
+## use https://github.com/python-caldav/caldav/issues
+## or scheduling-help@plann.no
 import sys
 import uuid
 from datetime import datetime