perf: Migrate logic to async

For backward compatibility, converters are added new "_async" methods, in complement of the original (sync). Sync method is marked deprecated. Youtube converted is executed in a thread context (the lib isn't natively async) and OpenAI integration use the AsyncOpenAI class, also marking sync class as deprecated.

Author: clemlesne

README.md

@@ -9,37 +9,39 @@
 
 > [!IMPORTANT]
 > Breaking changes between 0.0.1 to 0.1.0:
-> * Dependencies are now organized into optional feature-groups (further details below). Use `pip install 'markitdown[all]'` to have backward-compatible behavior. 
+>
+> * Dependencies are now organized into optional feature-groups (further details below). Use `pip install 'markitdown[all]'` to have backward-compatible behavior.
 > * convert\_stream() now requires a binary file-like object (e.g., a file opened in binary mode, or an io.BytesIO object). This is a breaking change from the previous version, where it previously also accepted text file-like objects, like io.StringIO.
 > * The DocumentConverter class interface has changed to read from file-like streams rather than file paths. *No temporary files are created anymore*. If you are the maintainer of a plugin, or custom DocumentConverter, you likely need to update your code. Otherwise, if only using the MarkItDown class or CLI (as in these examples), you should not need to change anything.
 
 MarkItDown is a lightweight Python utility for converting various files to Markdown for use with LLMs and related text analysis pipelines. To this end, it is most comparable to [textract](https://github.com/deanmalmgren/textract), but with a focus on preserving important document structure and content as Markdown (including: headings, lists, tables, links, etc.) While the output is often reasonably presentable and human-friendly, it is meant to be consumed by text analysis tools -- and may not be the best option for high-fidelity document conversions for human consumption.
 
 MarkItDown currently supports the conversion from:
 
-- PDF
-- PowerPoint
-- Word
-- Excel
-- Images (EXIF metadata and OCR)
-- Audio (EXIF metadata and speech transcription)
-- HTML
-- Text-based formats (CSV, JSON, XML)
-- ZIP files (iterates over contents)
-- Youtube URLs
-- EPubs
-- ... and more!
+* PDF
+* PowerPoint
+* Word
+* Excel
+* Images (EXIF metadata and OCR)
+* Audio (EXIF metadata and speech transcription)
+* HTML
+* Text-based formats (CSV, JSON, XML)
+* ZIP files (iterates over contents)
+* Youtube URLs
+* EPubs
+* ... and more!
 
 ## Why Markdown?
 
 Markdown is extremely close to plain text, with minimal markup or formatting, but still
 provides a way to represent important document structure. Mainstream LLMs, such as
-OpenAI's GPT-4o, natively "_speak_" Markdown, and often incorporate Markdown into their
+OpenAI's GPT-4o, natively "*speak*" Markdown, and often incorporate Markdown into their
 responses unprompted. This suggests that they have been trained on vast amounts of
 Markdown-formatted text, and understand it well. As a side benefit, Markdown conventions
 are also highly token-efficient.
 
 ## Prerequisites
+
 MarkItDown requires Python 3.10 or higher. It is recommended to use a virtual environment to avoid dependency conflicts.
 
 With the standard Python installation, you can create and activate a virtual environment using the following commands:
@@ -95,6 +97,7 @@ cat path-to-file.pdf | markitdown
 ```
 
 ### Optional Dependencies
+
 MarkItDown has optional dependencies for activating various file formats. Earlier in this document, we installed all optional dependencies with the `[all]` option. However, you can also install them individually for more control. For example:
 
 ```bash
@@ -144,7 +147,7 @@ More information about how to set up an Azure Document Intelligence Resource can
 
 ### Python API
 
-Basic usage in Python:
+Basic usage in Python. All functions are available in async and sync versions:
 
 ```python
 from markitdown import MarkItDown
@@ -164,13 +167,13 @@ result = md.convert("test.pdf")
 print(result.text_content)
 ```
 
-To use Large Language Models for image descriptions, provide `llm_client` and `llm_model`:
+To use Large Language Models for image descriptions, provide `llm_client` (AsyncOpenAI) and `llm_model` (str), and enable the `[openai]` optional dependency group:
 
 ```python
 from markitdown import MarkItDown
-from openai import OpenAI
+from openai import AsyncOpenAI
 
-client = OpenAI()
+client = AsyncOpenAI()
 md = MarkItDown(llm_client=client, llm_model="gpt-4o")
 result = md.convert("example.jpg")
 print(result.text_content)
@@ -187,7 +190,7 @@ docker run --rm -i markitdown:latest < ~/your-file.pdf > output.md
 
 This project welcomes contributions and suggestions. Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
-the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+the rights to use your contribution. For details, visit <https://cla.opensource.microsoft.com>.
 
 When you submit a pull request, a CLA bot will automatically determine whether you need to provide
 a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
@@ -212,13 +215,13 @@ You can help by looking at issues or helping review PRs. Any issue or PR is welc
 
 ### Running Tests and Checks
 
-- Navigate to the MarkItDown package:
+* Navigate to the MarkItDown package:
 
   ```sh
   cd packages/markitdown
   ```
 
-- Install `hatch` in your environment and run tests:
+* Install `hatch` in your environment and run tests:
 
   ```sh
   pip install hatch  # Other ways of installing hatch: https://hatch.pypa.io/dev/install/
@@ -233,7 +236,7 @@ You can help by looking at issues or helping review PRs. Any issue or PR is welc
   hatch test
   ```
 
-- Run pre-commit checks before submitting a PR: `pre-commit run --all-files`
+* Run pre-commit checks before submitting a PR: `pre-commit run --all-files`
 
 ### Contributing 3rd-party Plugins
 

packages/markitdown-mcp/src/markitdown_mcp/__main__.py

@@ -20,7 +20,9 @@
 @mcp.tool()
 async def convert_to_markdown(uri: str) -> str:
     """Convert a resource described by an http:, https:, file: or data: URI to markdown"""
-    return MarkItDown(enable_plugins=check_plugins_enabled()).convert_uri(uri).markdown
+    return (
+        await MarkItDown(enable_plugins=check_plugins_enabled()).convert_uri_async(uri)
+    ).markdown
 
 
 def check_plugins_enabled() -> bool:

packages/markitdown-sample-plugin/README.md

@@ -20,28 +20,25 @@ class RtfConverter(DocumentConverter):
     ):
         super().__init__(priority=priority)
 
-    def accepts(
+    async def accepts_async(
         self,
         file_stream: BinaryIO,
         stream_info: StreamInfo,
         **kwargs: Any,
     ) -> bool:
-	
-	# Implement logic to check if the file stream is an RTF file
-	# ...
-	raise NotImplementedError()
+        # Implement logic to check if the file stream is an RTF file
+        # ...
+        raise NotImplementedError()
 
-
-    def convert(
+    async def convert_async(
         self,
         file_stream: BinaryIO,
         stream_info: StreamInfo,
         **kwargs: Any,
     ) -> DocumentConverterResult:
-
-	# Implement logic to convert the file stream to Markdown
-	# ...
-	raise NotImplementedError()
+        # Implement logic to convert the file stream to Markdown
+        # ...
+        raise NotImplementedError()
 ```
 
 Next, make sure your package implements and exports the following:
@@ -98,7 +95,7 @@ In Python, plugins can be enabled as follows:
 from markitdown import MarkItDown
 
 md = MarkItDown(enable_plugins=True) 
-result = md.convert("path-to-file.rtf")
+result = await md.convert_async("path-to-file.rtf")
 print(result.text_content)
 ```
 

packages/markitdown-sample-plugin/src/markitdown_sample_plugin/_plugin.py

@@ -36,7 +36,7 @@ class RtfConverter(DocumentConverter):
     Converts an RTF file to in the simplest possible way.
     """
 
-    def accepts(
+    async def accepts_async(
         self,
         file_stream: BinaryIO,
         stream_info: StreamInfo,
@@ -54,7 +54,7 @@ def accepts(
 
         return False
 
-    def convert(
+    async def convert_async(
         self,
         file_stream: BinaryIO,
         stream_info: StreamInfo,

packages/markitdown-sample-plugin/tests/test_sample_plugin.py

@@ -1,4 +1,5 @@
 #!/usr/bin/env python3 -m pytest
+import asyncio
 import os
 
 from markitdown import MarkItDown, StreamInfo
@@ -12,11 +13,11 @@
 }
 
 
-def test_converter() -> None:
+async def test_converter() -> None:
     """Tests the RTF converter dirctly."""
     with open(os.path.join(TEST_FILES_DIR, "test.rtf"), "rb") as file_stream:
         converter = RtfConverter()
-        result = converter.convert(
+        result = await converter.convert_async(
             file_stream=file_stream,
             stream_info=StreamInfo(
                 mimetype="text/rtf", extension=".rtf", filename="test.rtf"
@@ -38,6 +39,6 @@ def test_markitdown() -> None:
 
 if __name__ == "__main__":
     """Runs this file's tests from the command line."""
-    test_converter()
+    asyncio.run(test_converter())
     test_markitdown()
     print("All tests passed.")

packages/markitdown/README.md

@@ -1,7 +1,7 @@
 # MarkItDown
 
 > [!IMPORTANT]
-> MarkItDown is a Python package and command-line utility for converting various files to Markdown (e.g., for indexing, text analysis, etc). 
+> MarkItDown is a Python package and command-line utility for converting various files to Markdown (e.g., for indexing, text analysis, etc).
 >
 > For more information, and full documentation, see the project [README.md](https://github.com/microsoft/markitdown) on GitHub.
 

packages/markitdown/pyproject.toml

@@ -10,9 +10,7 @@ readme = "README.md"
 requires-python = ">=3.10"
 license = "MIT"
 keywords = []
-authors = [
-  { name = "Adam Fourney", email = "adamfo@microsoft.com" },
-]
+authors = [{ name = "Adam Fourney", email = "adamfo@microsoft.com" }]
 classifiers = [
   "Development Status :: 4 - Beta",
   "Programming Language :: Python",
@@ -25,12 +23,13 @@ classifiers = [
 ]
 dependencies = [
   "beautifulsoup4",
-  "requests",
   "markdownify",
   "magika~=0.6.1",
   "charset-normalizer",
   "defusedxml",
   "onnxruntime<=1.20.1; sys_platform == 'win32'",
+  "aiohttp",
+  "openai",
 ]
 
 [project.optional-dependencies]
@@ -47,7 +46,7 @@ all = [
   "SpeechRecognition",
   "youtube-transcript-api~=1.0.0",
   "azure-ai-documentintelligence",
-  "azure-identity"
+  "azure-identity",
 ]
 pptx = ["python-pptx"]
 docx = ["mammoth", "lxml"]
@@ -75,16 +74,11 @@ features = ["all"]
 
 [tool.hatch.envs.hatch-test]
 features = ["all"]
-extra-dependencies = [
-  "openai",
-]
+parallel = true
 
 [tool.hatch.envs.types]
 features = ["all"]
-extra-dependencies = [
-  "openai",
-  "mypy>=1.0.0",
-]
+extra-dependencies = ["mypy>=1.0.0"]
 
 [tool.hatch.envs.types.scripts]
 check = "mypy --install-types --non-interactive --ignore-missing-imports {args:src/markitdown tests}"
@@ -93,20 +87,14 @@ check = "mypy --install-types --non-interactive --ignore-missing-imports {args:s
 source_pkgs = ["markitdown", "tests"]
 branch = true
 parallel = true
-omit = [
-  "src/markitdown/__about__.py",
-]
+omit = ["src/markitdown/__about__.py"]
 
 [tool.coverage.paths]
 markitdown = ["src/markitdown", "*/markitdown/src/markitdown"]
 tests = ["tests", "*/markitdown/tests"]
 
 [tool.coverage.report]
-exclude_lines = [
-  "no cov",
-  "if __name__ == .__main__.:",
-  "if TYPE_CHECKING:",
-]
+exclude_lines = ["no cov", "if __name__ == .__main__.:", "if TYPE_CHECKING:"]
 
 [tool.hatch.build.targets.sdist]
 only-include = ["src/markitdown"]