| Field | Type | Required | Description | Example |
|---|---|---|---|---|
chunk_overlap |
Optional[int] | ➖ | Size of overlap between chunks in tokens to store in vector store to better capture relevant context | |
chunk_size |
int | ✔️ | Size of chunks in tokens to store in vector store (make sure it is not too big for the context if your LLM) | |
field_name_mappings |
List[models.DestinationPineconeFieldNameMappingConfigModel] | ➖ | List of fields to rename. Not applicable for nested fields, but can be used to rename fields already flattened via dot notation. | |
metadata_fields |
List[str] | ➖ | List of fields in the record that should be stored as metadata. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered metadata fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. user.name will access the name field in the user object. It's also possible to use wildcards to access all fields in an object, e.g. users.*.name will access all names fields in all entries of the users array. When specifying nested paths, all matching values are flattened into an array set to a field named by the path. |
Example 1: age Example 2: user Example 3: user.name |
text_fields |
List[str] | ➖ | List of fields in the record that should be used to calculate the embedding. The field list is applied to all streams in the same way and non-existing fields are ignored. If none are defined, all fields are considered text fields. When specifying text fields, you can access nested fields in the record by using dot notation, e.g. user.name will access the name field in the user object. It's also possible to use wildcards to access all fields in an object, e.g. users.*.name will access all names fields in all entries of the users array. |
Example 1: text Example 2: user.name Example 3: users.*.name |
text_splitter |
Optional[models.DestinationPineconeTextSplitter] | ➖ | Split text fields into chunks based on the specified method. |