Merge branch 'dev'

Author: thiswillbeyourgithub

README.md

@@ -1,5 +1,8 @@
 <p align="center"><img src="https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/icon.png?raw=true" width="256"></p>
 
+> *I'm WDoc. I solve RAG problems.*
+> - WDoc, imitating Winston "The Wolf" Wolf
+
 # WDoc
 
 * **Goal and project specifications** use [LangChain](https://python.langchain.com/) to summarize, search or query documents. I'm a medical student so I need to be able to query from **tens of thousands** of documents, of different types ([Supported filetypes](#Supported-filetypes)). I also have little free time so I needed a tailor made summary feature to keep up with the news.
@@ -114,9 +117,56 @@
 6. To know more about each argument supported by each filetype, `wdoc --help`
 7. There is a specific recursive filetype I should mention: `--filetype="link_file"`. Basically the file designated by `--path` should contain in each line (`#comments` and empty lines are ignored) one url, that will be parsed by WDoc. I made this so that I can quickly use the "share" button on android from my browser to a text file (so it just appends the url to the file), this file is synced via [syncthing](https://github.com/syncthing/syncthing) to my browser and WDoc automatically summarize them and add them to my [Logseq](https://github.com/logseq/logseq/). Note that the url is parsed in each line, so formatting is ignored, for example it works even in markdown bullet point list.
 8. If you want to make sure your data remains private here's an example with ollama: `wdoc --private --llms_api_bases='{"model": "http://localhost:11434", "query_eval_model": "http://localhost:11434"}' --modelname="ollama_chat/gemma:2b" --query_eval_modelname="ollama_chat/gemma:2b" --embed_model="BAAI/bge-m3" my_task`
-9. Now say you just want to summarize a webpage: `wdoc summary --path="https://arstechnica.com/science/2024/06/to-pee-or-not-to-pee-that-is-a-question-for-the-bladder-and-the-brain/"`.
+9. Now say you just want to summarize [Tim Urban's TED talk on procrastination](https://www.youtube.com/watch?v=arj7oStGLkU): `wdoc summary --path 'https://www.youtube.com/watch?v=arj7oStGLkU' --youtube_language="english" --disable_md_printing`:
+> # Summary
+> ## https://www.youtube.com/watch?v=arj7oStGLkU
+> - The speaker, Tim Urban, was a government major in college who had to write many papers
+> - *He claims* his typical work pattern for papers was:
+>     - Planning to spread work evenly
+>     - Actually procrastinating until the last minute
+> - For his 90-page senior thesis:
+>     - Planned to work steadily over a year
+>     - *Actually* ended up writing 90 pages in 72 hours before the deadline
+>     - Pulled two all-nighters
+>     - Resulted in a 'very, very bad thesis'
+> - Urban is now a writer-blogger for 'Wait But Why'
+> - He wrote about procrastination to explain it to non-procrastinators
+> - *Humorously claims* to have done brain scans comparing procrastinator and non-procrastinator brains
+> - Introduces concept of 'Instant Gratification Monkey' in procrastinator's brain
+>     - Monkey takes control from the Rational Decision-Maker
+>     - Leads to unproductive activities like reading Wikipedia, checking fridge, YouTube spirals
+> - Monkey characteristics:
+>     - Lives in the present moment
+>     - No memory of past or knowledge of future
+>     - Only cares about 'easy and fun'
+> - Rational Decision-Maker:
+>     - Allows long-term planning and big picture thinking
+>     - Wants to do what makes sense in the moment
+> - 'Dark Playground': where procrastinators spend time on leisure activities when they shouldn't
+>     - Filled with guilt, dread, anxiety, self-hatred
+> - 'Panic Monster': procrastinator's guardian angel
+>     - Wakes up when deadlines are close or there's danger of embarrassment
+>     - Only thing the Monkey fears
+> - Urban relates his own experience procrastinating on preparing this TED talk
+> - *Claims* thousands of people emailed him about having the same procrastination problem
+> - Two types of procrastination:
+>     - 1. Short-term with deadlines (contained by Panic Monster)
+>     - 2. Long-term without deadlines (more damaging)
+>         - Affects self-starter careers, personal life, health, relationships
+>         - Can lead to long-term unhappiness and regrets
+> - *Urban believes* all people are procrastinators to some degree
+> - Presents 'Life Calendar': visual representation of weeks in a 90-year life
+> - Encourages audience to:
+>     - Think about what they're procrastinating on
+>     - Stay aware of the Instant Gratification Monkey
+>     - Start addressing procrastination soon
+> - *Humorously* suggests not starting today, but 'sometime soon'
+> Tokens used for https://www.youtube.com/watch?v=arj7oStGLkU: '4365' ($0.00060)
+> Total cost of those summaries: '4365' ($0.00060, estimate was $0.00028)
+> Total time saved by those summaries: 8.4 minutes
+> Done summarizing.
+
 
-<p align="center"><img src="https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/summary.png?raw=true" width="256"></p>
 
 ## Getting started
 *Tested on python 3.10 and 3.11.7*

WDoc/WDoc.py

@@ -79,7 +79,7 @@
 class WDoc:
     "This docstring is dynamically replaced by the content of WDoc/docs/USAGE.md"
 
-    VERSION: str = "1.1.9"
+    VERSION: str = "1.1.10"
     allowed_extra_args = extra_args_keys
     md_printer = md_printer
 
@@ -116,7 +116,7 @@ def __init__(
         query_eval_check_number: int = 4,
         query_relevancy: float = 0.1,
 
-        summary_n_recursion: int = 1,
+        summary_n_recursion: int = 0,
         summary_language: str = "the same language as the document",
 
         llm_verbosity: Union[bool, int] = False,
@@ -131,18 +131,36 @@ def __init__(
         DIY_rolling_window_embedding: Union[bool, int] = False,
         import_mode: Union[bool, int] = False,
         disable_md_printing: bool = False,
+        silent: bool = False,
+        version: bool = False,
 
         **cli_kwargs,
     ) -> None:
         "This docstring is dynamically replaced by the content of WDoc/docs/USAGE.md"
+        if version:
+            print(self.VERSION)
+            return
+        if notification_callback is not None:
+            @optional_typecheck
+            def ntfy(text: str) -> str:
+                out = notification_callback(text)
+                assert out == text, "The notification callback must return the same string"
+                return out
+            ntfy("Starting WDoc")
+        else:
+            @optional_typecheck
+            def ntfy(text: str) -> str:
+                return text
+        self.ntfy = ntfy
+
         if debug:
             def handle_exception(exc_type, exc_value, exc_traceback):
                 if not issubclass(exc_type, KeyboardInterrupt):
                     @optional_typecheck
                     def p(message: str) -> None:
                         "print error, in red if possible"
                         try:
-                            red(message)
+                            red(self.ntfy(message))
                         except Exception as err:
                             print(message)
                     p("\n--verbose was used so opening debug console at the "
@@ -158,13 +176,26 @@ def p(message: str) -> None:
             sys.excepthook = handle_exception
             faulthandler.enable()
 
+        elif notification_callback:
+            def print_exception(exc_type, exc_value, exc_traceback):
+                if not issubclass(exc_type, KeyboardInterrupt):
+                    message = "An error has occured:\n"
+                    message += "\n".join([line for line in traceback.format_tb(exc_traceback)])
+                    message += "\n" + str(exc_type) + " : " + str(exc_value)
+                    self.ntfy(message)
+                    sys.exit(1)
+
+            sys.excepthook = print_exception
+            faulthandler.enable()
+
         red(pyfiglet.figlet_format("wdoc"))
 
         # make sure the extra args are valid
         for k in cli_kwargs:
             if k not in self.allowed_extra_args:
                 raise Exception(
-                    red(f"Found unexpected keyword argument: '{k}'"))
+                    red(f"Found unexpected keyword argument: '{k}'\nThe allowed arguments are {','.join(self.allowed_extra_args)}")
+                        )
 
             # type checking of extra args
             if os.environ["WDOC_TYPECHECKING"] in ["crash", "warn"]:
@@ -376,19 +407,6 @@ def p(message: str) -> None:
                 raise Exception(
                     red(f"Can't find the price of {query_eval_modelname}"))
 
-        if notification_callback is not None:
-            @optional_typecheck
-            def ntfy(text: str) -> str:
-                out = notification_callback(text)
-                assert out == text, "The notification callback must return the same string"
-                return out
-            ntfy("Starting WDoc")
-        else:
-            @optional_typecheck
-            def ntfy(text: str) -> str:
-                return text
-            self.ntfy = ntfy
-
         if is_verbose:
             # os.environ["LANGCHAIN_TRACING_V2"] = "true"
             set_verbose(True)
@@ -786,6 +804,7 @@ def summarize_documents(
         if llmcallback.total_tokens != results['doc_total_tokens']:
             red(
                 f"Cost discrepancy? Tokens used according to the callback: '{llmcallback.total_tokens}' (${total_cost:.5f})")
+        self.summary_results = results
         return results
 
     @optional_typecheck
@@ -1196,6 +1215,7 @@ def query_task(self, query: Optional[str]) -> dict:
                 eval_args["n"] = self.query_eval_check_number
             else:
                 red(f"Model {self.query_eval_modelname} does not support parameter 'n' so will be called multiple times instead. This might cost more.")
+                assert self.query_eval_modelbackend != "openai"
             if "max_tokens" in self.eval_llm_params:
                 eval_args["max_tokens"] = 2
             else:
@@ -1607,6 +1627,7 @@ def retrieve_documents(inputs):
                 f"Number of documents after query eval filter: {len(output['filtered_docs'])}")
             red(
                 f"Number of documents found relevant by eval llm: {len(output['relevant_filtered_docs'])}")
+            red(f"Number of steps to combine intermediate answers: {len(all_intermediate_answers) - 1}")
             red(f"Time took by the chain: {chain_time:.2f}s")
 
             assert len(

WDoc/docs/USAGE.md

@@ -139,7 +139,7 @@
 
 ---
 
-* `--summary_n_recursion`: int, default `1`
+* `--summary_n_recursion`: int, default `0`
     * after summarizing, will go over the summary that many times to fix
     indentation, repetitions etc.
         * 0 means disabled.
@@ -222,11 +222,19 @@
     the default langchain SentenceTransformerEmbedding implementation
 
 * `--import_mode`: bool, default `False`
-    * if True, will return the answer from query instead of printing it
+    * if True, will return the answer from query instead of printing it.
+    The idea is to use if when you import WDoc instead of running
+    it from the cli. See `--silent`
 
 * `--disable_md_printing`: bool, default `True`
     * if True, instead of using rich to display some information, default to simpler colored prints.
 
+* `--silent`: bool, default False
+    * disable almost all prints. Can be handy if `--import_mode` is used.
+
+* `--version`: bool, default False
+    * display the version and exit
+
 * `--cli_kwargs`: dict, optional
     * Any remaining keyword argument will be parsed as a loader
     specific argument ((see below)[#loader-specific-arguments]).

WDoc/utils/flags.py

@@ -9,18 +9,14 @@
 kwargs = fire.Fire(lambda *args, **kwargs: kwargs)
 is_linux = platform.system() == "Linux"
 
-if "verbose" in kwargs and kwargs["verbose"]:
-    is_verbose = True
-else:
-    is_verbose = False
+def check_kwargs(arg):
+    if arg in kwargs and kwargs[arg]:
+        return True
+    return False
 
-if "debug" in kwargs and kwargs["debug"]:
-    is_debug = True
-    is_verbose = True
-else:
-    is_debug = False
+is_debug = check_kwargs("debug")
+is_verbose = is_debug or check_kwargs("verbose")
 
-if "disable_md_printing" in kwargs and kwargs["disable_md_printing"]:
-    disable_md_printing = True
-else:
-    disable_md_printing = False
+is_silent = check_kwargs("silent")
+
+md_printing_disabled = check_kwargs("disable_md_printing")

WDoc/utils/logger.py

@@ -14,7 +14,7 @@
 import warnings
 
 from .typechecker import optional_typecheck
-from .flags import disable_md_printing
+from .flags import md_printing_disabled, is_silent
 
 # ignore warnings from beautiful soup
 warnings.filterwarnings("ignore", category=UserWarning, module='bs4')
@@ -85,7 +85,8 @@ def printer(string: Union[str, Dict, List], **args) -> str:
         for k, v in colors.items():
             string = string.replace(v, "")
         logger.info(string)
-        tqdm.write(col + string + colors["reset"], **args)
+        if not is_silent:
+            tqdm.write(col + string + colors["reset"], **args)
         return inp
     return printer
 
@@ -100,7 +101,7 @@ def printer(string: Union[str, Dict, List], **args) -> str:
 @optional_typecheck
 def md_printer(message: str, color: Optional[str] = None) -> str:
     "markdown printing"
-    if not disable_md_printing:
+    if not md_printing_disabled:
         logger.info(message)
         md = Markdown(message)
         console.print(md, style=color)

WDoc/utils/misc.py

@@ -149,26 +149,29 @@ class DocDict(dict):
     allowed_keys = doc_kwargs_keys
     allowed_types = loader_specific_keys
 
+    def __check_values__(self, key, value):
+        if key not in self.allowed_keys:
+            raise Exception(
+                f"Cannot set key '{key}' in a DocDict. Allowed keys are "
+                f"'{','.join(self.allowed_keys)}'"
+            )
+        if key in self.allowed_types and value is not None:
+            assert isinstance(value, self.allowed_types[key]), (
+                f"Type of key {key} should be {self.allowed_types[key]},"
+                f"not {type(value)}"
+            )
+
     def __init__(self, *args, **kwargs):
         for arg in args:
             assert isinstance(arg, dict)
             for k, v in arg.items():
-                if k not in self.allowed_keys:
-                    raise Exception(f"Cannot set key '{k}' in a DocDict")
-                if k in self.allowed_types and v is not None:
-                    assert isinstance(v, self.allowed_types[k])
+                self.__check_values__(k, v)
         for k, v in kwargs.items():
-            if k not in self.allowed_keys:
-                raise Exception(f"Cannot set key '{k}' in a DocDict")
-            if k in self.allowed_types and v is not None:
-                assert isinstance(v, self.allowed_types[k])
+            self.__check_values__(k, v)
         super().__init__(*args, **kwargs)
 
     def __setitem__(self, key, value):
-        if key not in self.allowed_keys:
-            raise Exception(f"Cannot set key '{key}' in a DocDict")
-        if key in self.allowed_types and value is not None:
-            assert isinstance(value, self.allowed_types[key])
+        self.__check_values__(key, value)
         super().__setitem__(key, value)
 
 

bumpver.toml

@@ -1,5 +1,5 @@
 [bumpver]
-current_version = "1.1.9"
+current_version = "1.1.10"
 version_pattern = "MAJOR.MINOR.PATCH"
 commit_message = "bump version {old_version} -> {new_version}"
 tag_message = "{new_version}"

setup.py

@@ -31,15 +31,11 @@ def run(self):
         '<p align="center"><img src="https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/icon.png?raw=true" width="256"></p>',
         '![icon](https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/icon.png?raw=true)',
     )
-    long_description = long_description.replace(
-        '<p align="center"><img src="https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/summary.png?raw=true" width="256"></p>',
-        '![example](https://github.com/thiswillbeyourgithub/WDoc/blob/main/images/summary.png?raw=true)',
-    )
     assert 'align="center"' not in long_description
 
 setup(
     name="wdoc",
-    version="1.1.9",
+    version="1.1.10",
     description="A perfect AI powered RAG for document query and summary. Supports ~all LLM and ~all filetypes (url, pdf, epub, youtube (incl playlist), audio, anki, md, docx, pptx, oe any combination!)",
     long_description=long_description,
     long_description_content_type="text/markdown",