Release 2.3.0

Commits since last release

• [d99bd0a] by @thiswillbeyourgithub, 4 minutes ago:
  bump version 2.2.1 -> 2.3.0

bumpver.toml
setup.py
wdoc/wdoc.py

• [4a8f0a9] by @thiswillbeyourgithub, 44 minutes ago:
  enh: updated shell completions

shell_completions/wdoc_completion.bash
shell_completions/wdoc_completion.fish
shell_completions/wdoc_completion.zsh

• [f445717] by @thiswillbeyourgithub, 44 minutes ago:
  fix: onlinepdfloader is slow slow its better to just try downloading the file first

wdoc/docs/USAGE.md
wdoc/utils/loaders.py

• [09202fd] by @thiswillbeyourgithub, 48 minutes ago:
  fix: typo

wdoc/utils/embeddings.py

• [8fd0682] by @thiswillbeyourgithub, 3 hours ago:
  feat: add argument --oneoff to avoid prompting continuously

wdoc/docs/USAGE.md
wdoc/wdoc.py

• [4137319] by @thiswillbeyourgithub, 3 hours ago:
  prompt: tell each llm the name of the others

wdoc/utils/prompts.py

• [e15c736] by @thiswillbeyourgithub, 4 hours ago:
  move my sql_dict code to its own library
  https://github.com/thiswillbeyourgithub/PersistDict

setup.py
wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/customs/sql_dict.py

• [06f9420] by @thiswillbeyourgithub, 4 hours ago:
  gitignore

.gitignore

• [8a234ac] by @thiswillbeyourgithub, 5 hours ago:
  remove docstring

wdoc/utils/customs/sql_dict.py

• [75d43e1] by @thiswillbeyourgithub, 5 hours ago:
  enh: pass verbose argument to the custom caching

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py
wdoc/wdoc.py

• [425e699] by @thiswillbeyourgithub, 6 hours ago:
  update gitignore

.gitignore

• [12241c4] by @thiswillbeyourgithub, 6 hours ago:
  fix: forbid to use call if setitem was used

wdoc/utils/customs/sql_dict.py

• [7db4387] by @thiswillbeyourgithub, 6 hours ago:
  better documentation

wdoc/utils/customs/sql_dict.py

• [9e74454] by @thiswillbeyourgithub, 6 hours ago:
  rename example folder to scripts

README.md
scripts/AnkiFiltered/AnkiFilteredDeckCreator.py
scripts/NtfySummarizer/NtfySummarizer.py
scripts/NtfySummarizer/README.md
scripts/NtfySummarizer/TOPICS.EXAMPLE
scripts/NtfySummarizer/script.sh
scripts/TheFiche/README.md
scripts/TheFiche/TheFiche.py

• [633a222] by @thiswillbeyourgithub, 6 hours ago:
  typo

README.md

• [debbf21] by @thiswillbeyourgithub, 6 hours ago:
  added to the readme

README.md

• [ba68b44] by @thiswillbeyourgithub, 6 hours ago:
  better code to reset the cache if another python process modified it

wdoc/utils/customs/sql_dict.py

• [aa083b4] by @thiswillbeyourgithub, 7 hours ago:
  minor: password management

wdoc/utils/customs/sql_dict.py

• [782b200] by @thiswillbeyourgithub, 8 hours ago:
  upadte code to use my sql_dict

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py

• [5e03fb5] by @thiswillbeyourgithub, 8 hours ago:
  new: might as well create my own sql_dict for fun today

wdoc/utils/customs/sql_dict.py

• [9f8b4fd] by @thiswillbeyourgithub, 8 hours ago:
  reduce concurrency to 15

wdoc/docs/USAGE.md
wdoc/utils/env.py

• [07bf694] by @thiswillbeyourgithub (aider), 23 hours ago:
  feat: Add timeout argument to SQLite connections

wdoc/utils/customs/fix_llm_caching.py

• [e42f9c4] by @thiswillbeyourgithub (aider), 23 hours ago:
  feat: add SQLITE3_CONNECTION_TIMEOUT variable with default of 30

wdoc/utils/customs/fix_llm_caching.py

• [c2ccc6d] by @thiswillbeyourgithub, 26 hours ago:
  set default max concurrency for llms to 20

wdoc/docs/USAGE.md
wdoc/utils/env.py

• [dc6b162] by @thiswillbeyourgithub, 29 hours ago:
  fix: forgot to check if max_top_k was set, crashing in some cases

wdoc/utils/tasks/query.py
wdoc/wdoc.py

• [670bb8b] by @thiswillbeyourgithub, 34 hours ago:
  minor: bump version

wdoc/utils/customs/fix_llm_caching.py

• [4ab313f] by @thiswillbeyourgithub, 35 hours ago:
  fix: add a maybe needed commits

wdoc/utils/customs/fix_llm_caching.py

• [a40a100] by @thiswillbeyourgithub, 35 hours ago:
  enh: add vacuum and compression

wdoc/utils/customs/fix_llm_caching.py

• [349ba90] by @thiswillbeyourgithub, 35 hours ago:
  minor: add a method to get the columns in the db

wdoc/utils/customs/fix_llm_caching.py

• [52cc97a] by @thiswillbeyourgithub, 35 hours ago:
  minor: rename cache to storage

wdoc/utils/customs/fix_llm_caching.py

• [8cca75d] by @thiswillbeyourgithub, 2 days ago:
  untested: add expiration date to the sql cache

wdoc/utils/customs/fix_llm_caching.py
wdoc/utils/embeddings.py

• [b409642] by @thiswillbeyourgithub, 2 days ago:
  enh: add timestamp to the sql cache

wdoc/utils/customs/fix_llm_caching.py

• [615c967] by @thiswillbeyourgithub, 2 days ago:
  minor

examples/TheFiche/TheFiche.py

• [b26694a] by @thiswillbeyourgithub, 2 days ago:
  fix: modify md to always add a leading dash before ints in enumeration
  this avoids issues with markdown

examples/TheFiche/TheFiche.py

• [ef30912] by @thiswillbeyourgithub, 2 days ago:
  untested: rewrite how sqlite caching is done to reduce memory footprint by using it like a dict basically

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py

• [6876ef2] by @thiswillbeyourgithub, 2 days ago:
  enh: make all instance of the same db use the same cache

wdoc/utils/customs/fix_llm_caching.py

• [1e0cb6c] by @thiswillbeyourgithub, 2 days ago:
  new: print costs when using import_mode

wdoc/wdoc.py

• [f6c6b6d] by @thiswillbeyourgithub, 2 days ago:
  enh: display the number of cid and nid found per document, to help troubleshoot issues

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [16da77f] by @thiswillbeyourgithub, 2 days ago:
  fix: in some situation not all nids were returned

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [9aac581] by @thiswillbeyourgithub, 2 days ago:
  fix: must return int not str

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [2e89612] by @thiswillbeyourgithub, 2 days ago:
  tell the rephraser to take into account instructions to the other llm

wdoc/utils/prompts.py

• [cd665a9] by @thiswillbeyourgithub, 2 days ago:
  typo

wdoc/utils/prompts.py
wdoc/utils/tasks/query.py

• [dcd2654] by @thiswillbeyourgithub, 2 days ago:
  untested: make the compressed embeddings cache actually use the sqlite3 backend

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/embeddings.py

• [094d762] by @thiswillbeyourgithub, 2 days ago:
  enh: when using joblib only send the relevant batch

wdoc/utils/embeddings.py

• [4fdaec0] by @thiswillbeyourgithub, 2 days ago:
  enh: use pickle instead of dill because its much slower

wdoc/utils/customs/fix_llm_caching.py

• [b54cbe6] by @thiswillbeyourgithub, 2 days ago:
  enh: add BEGIN execute

wdoc/utils/customs/fix_llm_caching.py

• [501901a] by @thiswillbeyourgithub, 2 days ago:
  enh: always run .clear()

wdoc/utils/customs/fix_llm_caching.py

• [5a63f3e] by @thiswillbeyourgithub, 2 days ago:
  enh: use try finally block to close the connection

wdoc/utils/customs/fix_llm_caching.py

• [52edeb3] by @thiswillbeyourgithub, 2 days ago:
  enh: set sqlite3's check_same_thred to False now

wdoc/utils/customs/fix_llm_caching.py

• [2d8aa9c] by @thiswillbeyourgithub, 2 days ago:
  enh: keep locks opened less time

wdoc/utils/customs/fix_llm_caching.py

• [9c85df8] by @thiswillbeyourgithub, 3 days ago:
  make it easy to turn on or off check_same_thread

wdoc/utils/customs/fix_llm_caching.py

• [cd20832] by @thiswillbeyourgithub, 3 days ago:
  fix: chmod dir was not applied if dir already existed

wdoc/utils/customs/compressed_embeddings_cache.py

• [c801061] by @thiswillbeyourgithub, 3 days ago:
  fix: regex check arg has to be checked outside the memoize

wdoc/utils/customs/compressed_embeddings_cache.py

• [2c5a3a3] by @thiswillbeyourgithub, 3 days ago:
  enh: removed function _get_full_path that is not needed anymore

wdoc/utils/customs/compressed_embeddings_cache.py

• [b180eef] by @thiswillbeyourgithub, 3 days ago:
  enh: no need to create the directory every time we set a key

wdoc/utils/customs/compressed_embeddings_cache.py

• [e6c9186] by @thiswillbeyourgithub, 3 days ago:
  enh: add an arg to disable regex checking for keys

wdoc/utils/customs/compressed_embeddings_cache.py

• [aa2cc87] by @thiswillbeyourgithub, 3 days ago:
  enh: memoized hash checker for keys

wdoc/utils/customs/compressed_embeddings_cache.py

• [2360cac] by @thiswillbeyourgithub, 3 days ago:
  enh: no more try except as they are slow in python and not needed now that we check the db _z

wdoc/utils/customs/compressed_embeddings_cache.py

• [9cde74c] by @thiswillbeyourgithub, 3 days ago:
  enh: precompile regex check for hash keys

wdoc/utils/customs/compressed_embeddings_cache.py

• [f527cdc] by @thiswillbeyourgithub, 3 days ago:
  add versionning for db to anticipate breaking changes

wdoc/utils/customs/compressed_embeddings_cache.py
wdoc/utils/customs/fix_llm_caching.py

• [44e7c4c] by @thiswillbeyourgithub, 3 days ago:
  minor

wdoc/utils/customs/fix_llm_caching.py

• [8b4e9fe] by @thiswillbeyourgithub, 3 days ago:
  untested: remove unused kwargs in self.clear() that were not used as kwargs are slow

wdoc/utils/customs/fix_llm_caching.py

• [636e62c] by @thiswillbeyourgithub, 3 days ago:
  fix: use a threading lock for caching databases

wdoc/utils/customs/fix_llm_caching.py

• [f9b47d0] by @thiswillbeyourgithub, 3 days ago:
  rename a chain for clarificaiton

wdoc/wdoc.py

• [224eaba] by @thiswillbeyourgithub, 3 days ago:
  mention each new name in the readme

README.md

• [84f1ba0] by @thiswillbeyourgithub, 3 days ago:
  prompts: give name to each model

wdoc/utils/prompts.py

• [9822d95] by @thiswillbeyourgithub, 3 days ago:
  prompts: tell they are working at WDOC-CORP©

wdoc/utils/prompts.py

• [e5ac221] by @thiswillbeyourgithub, 3 days ago:
  prompts: tell to add [doubtful] source if really unsure

wdoc/utils/prompts.py

• [fdfd966] by @thiswillbeyourgithub, 3 days ago:
  fix: wdoc is now dealing with anki nids not cids

wdoc/wdoc.py

• [62766b4] by @thiswillbeyourgithub, 3 days ago:
  fix: anki filtered deck actually should use nids not cids

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [b3063d3] by @thiswillbeyourgithub, 3 days ago:
  add wdoc_filtered_deck:: before deckname

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [27c2ee4] by @thiswillbeyourgithub, 3 days ago:
  better tagging of anki filtered deck

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [d42f0b5] by @thiswillbeyourgithub, 4 days ago:
  bump version 2.2.0 -> 2.2.1

bumpver.toml
setup.py
wdoc/wdoc.py

• [5e66cd3] by @thiswillbeyourgithub, 4 days ago:
  enh: eval llm now applies a threshold

wdoc/docs/USAGE.md
wdoc/utils/tasks/query.py

• [201c55e] by @thiswillbeyourgithub, 4 days ago:
  fix: forgot to add the progress bars for task=search

wdoc/wdoc.py

• [c961386] by @thiswillbeyourgithub, 4 days ago:
  example: ankifilteredeck: new: bump version

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [e5f2062] by @thiswillbeyourgithub, 4 days ago:
  example: ankifilteredeck: new: by default set the LLM to the strong llm

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [32bb450] by @thiswillbeyourgithub, 4 days ago:
  new: retry with cache disabled if fails

wdoc/wdoc.py

• [05c705d] by @thiswillbeyourgithub, 4 days ago:
  fix: actually use caching inside the evaluate function just in case for partial results

wdoc/wdoc.py

• [930b18d] by @thiswillbeyourgithub, 4 days ago:
  example: ankifilteredeck: new: allow setting a task

examples/AnkiFiltered/AnkiFilteredDeckCreator.py

• [e150aeb] by @thiswillbeyourgithub, 4 days ago:
  fix: auto increasing top_k could fail in some cases

wdoc/wdoc.py