Skip to content

Commit baa3129

Browse files
dworthendworthen
committed
Add load_config to graphrag-common package.
1 parent 6b03af6 commit baa3129

36 files changed

Lines changed: 615 additions & 239 deletions

‎docs/examples_notebooks/api_overview.ipynb‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
"source": [
1717
"## API Overview\n",
1818
"\n",
19-
"This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations. "
19+
"This notebook provides a demonstration of how to interact with graphrag as a library using the API as opposed to the CLI. Note that graphrag's CLI actually connects to the library through this API for all operations.\n"
2020
]
2121
},
2222
{
@@ -28,11 +28,10 @@
2828
"from pathlib import Path\n",
2929
"from pprint import pprint\n",
3030
"\n",
31+
"import graphrag.api as api\n",
3132
"import pandas as pd\n",
32-
"from graphrag.config.load_config import load_config\n",
3333
"from graphrag.index.typing.pipeline_run_result import PipelineRunResult\n",
34-
"\n",
35-
"import graphrag.api as api"
34+
"from graphrag_common.config import load_config"
3635
]
3736
},
3837
{
@@ -49,16 +48,17 @@
4948
"metadata": {},
5049
"source": [
5150
"## Prerequisite\n",
51+
"\n",
5252
"As a prerequisite to all API operations, a `GraphRagConfig` object is required. It is the primary means to control the behavior of graphrag and can be instantiated from a `settings.yaml` configuration file.\n",
5353
"\n",
54-
"Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file."
54+
"Please refer to the [CLI docs](https://microsoft.github.io/graphrag/cli/#init) for more detailed information on how to generate the `settings.yaml` file.\n"
5555
]
5656
},
5757
{
5858
"cell_type": "markdown",
5959
"metadata": {},
6060
"source": [
61-
"### Generate a `GraphRagConfig` object"
61+
"### Generate a `GraphRagConfig` object\n"
6262
]
6363
},
6464
{
@@ -78,14 +78,14 @@
7878
"source": [
7979
"## Indexing API\n",
8080
"\n",
81-
"*Indexing* is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats."
81+
"_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
8282
]
8383
},
8484
{
8585
"cell_type": "markdown",
8686
"metadata": {},
8787
"source": [
88-
"## Build an index"
88+
"## Build an index\n"
8989
]
9090
},
9191
{
@@ -108,7 +108,7 @@
108108
"source": [
109109
"## Query an index\n",
110110
"\n",
111-
"To query an index, several index files must first be read into memory and passed to the query API. "
111+
"To query an index, several index files must first be read into memory and passed to the query API.\n"
112112
]
113113
},
114114
{
@@ -139,7 +139,7 @@
139139
"cell_type": "markdown",
140140
"metadata": {},
141141
"source": [
142-
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response."
142+
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
143143
]
144144
},
145145
{
@@ -155,7 +155,7 @@
155155
"cell_type": "markdown",
156156
"metadata": {},
157157
"source": [
158-
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model)."
158+
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
159159
]
160160
},
161161
{

‎docs/examples_notebooks/input_documents.ipynb‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
"\n",
1919
"Newer versions of GraphRAG let you submit a dataframe directly instead of running through the input processing step. This notebook demonstrates with regular or update runs.\n",
2020
"\n",
21-
"If performing an update, the assumption is that your dataframe contains only the new documents to add to the index."
21+
"If performing an update, the assumption is that your dataframe contains only the new documents to add to the index.\n"
2222
]
2323
},
2424
{
@@ -30,11 +30,10 @@
3030
"from pathlib import Path\n",
3131
"from pprint import pprint\n",
3232
"\n",
33+
"import graphrag.api as api\n",
3334
"import pandas as pd\n",
34-
"from graphrag.config.load_config import load_config\n",
3535
"from graphrag.index.typing.pipeline_run_result import PipelineRunResult\n",
36-
"\n",
37-
"import graphrag.api as api"
36+
"from graphrag_common.config import load_config"
3837
]
3938
},
4039
{
@@ -55,7 +54,7 @@
5554
"cell_type": "markdown",
5655
"metadata": {},
5756
"source": [
58-
"### Generate a `GraphRagConfig` object"
57+
"### Generate a `GraphRagConfig` object\n"
5958
]
6059
},
6160
{
@@ -73,14 +72,14 @@
7372
"source": [
7473
"## Indexing API\n",
7574
"\n",
76-
"*Indexing* is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats."
75+
"_Indexing_ is the process of ingesting raw text data and constructing a knowledge graph. GraphRAG currently supports plaintext (`.txt`) and `.csv` file formats.\n"
7776
]
7877
},
7978
{
8079
"cell_type": "markdown",
8180
"metadata": {},
8281
"source": [
83-
"## Build an index"
82+
"## Build an index\n"
8483
]
8584
},
8685
{
@@ -110,7 +109,7 @@
110109
"source": [
111110
"## Query an index\n",
112111
"\n",
113-
"To query an index, several index files must first be read into memory and passed to the query API. "
112+
"To query an index, several index files must first be read into memory and passed to the query API.\n"
114113
]
115114
},
116115
{
@@ -141,7 +140,7 @@
141140
"cell_type": "markdown",
142141
"metadata": {},
143142
"source": [
144-
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response."
143+
"The response object is the official reponse from graphrag while the context object holds various metadata regarding the querying process used to obtain the final response.\n"
145144
]
146145
},
147146
{
@@ -157,7 +156,7 @@
157156
"cell_type": "markdown",
158157
"metadata": {},
159158
"source": [
160-
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model)."
159+
"Digging into the context a bit more provides users with extremely granular information such as what sources of data (down to the level of text chunks) were ultimately retrieved and used as part of the context sent to the LLM model).\n"
161160
]
162161
},
163162
{

‎packages/graphrag-common/README.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,4 +48,65 @@ single2 = factory.create("some_other_strategy", {"value": "ignored"})
4848
assert single1 is single2
4949
assert single1.get_value() == "singleton"
5050
assert single2.get_value() == "singleton"
51+
```
52+
53+
## Config module
54+
55+
```python
56+
from pydantic import BaseModel, Field
57+
from graphrag_common.config import load_config
58+
59+
from pathlib import Path
60+
61+
class Logging(BaseModel):
62+
"""Test nested model."""
63+
64+
directory: str = Field(default="output/logs")
65+
filename: str = Field(default="logs.txt")
66+
67+
class Config(BaseModel):
68+
"""Test configuration model."""
69+
70+
name: str = Field(description="Name field.")
71+
logging: Logging = Field(description="Nested model field.")
72+
73+
# Basic
74+
# By default, ${} env variables in config file will be parsed and replaced
75+
# can disable by setting parse_env_vars=False
76+
config = load_config(Config, "path/to/config.[yaml|yml|json]")
77+
78+
# with .env file
79+
config = load_config(
80+
config_initializer=Config,
81+
config_path="config.yaml",
82+
dot_env_path=".env"
83+
)
84+
85+
# With overrides - provided values override whats in the config file
86+
# Only overrides what is specified - recursively merges settings.
87+
config = load_config(
88+
config_initializer=Config,
89+
config_path="config.yaml",
90+
overrides={
91+
"name": "some name"
92+
"logging": {
93+
"filename": "my_logs.txt"
94+
}
95+
},
96+
)
97+
98+
# Set the working directory to the directory of the config file
99+
config = load_config(
100+
config_initializer=Config,
101+
config_path="some/path/to/config.yaml",
102+
set_cwd=True
103+
)
104+
105+
# now cwd == some/path/to
106+
assert Path.cwd() == "some/path/to"
107+
108+
# And now throughout the codebase resolving relative paths in config
109+
# will resolve relative to the config directory
110+
Path(config.logging.directory) == "some/path/to/output/logging"
111+
51112
```

‎packages/graphrag-common/graphrag_common/config/__init__.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Copyright (c) 2024 Microsoft Corporation.
2+
# Licensed under the MIT License
3+
4+
"""The GraphRAG config module."""
5+
6+
from graphrag_common.config.load_config import ConfigParsingError, load_config
7+
8+
__all__ = ["ConfigParsingError", "load_config"]

0 commit comments

Comments
 (0)