Skip to content

Commit c6def0c

Browse files
committed
docs: make server-side batching the default import method
Restructure the batch-import guide so server-side (automatic) batching is the first and recommended way to import data, instead of a separate feature section: - Lead with server-side batching; present client-side batching as "Manual batching", the alternative for manual control or for the Go client. - The Go tab points to manual batching (the Go client has no server-side batching API) instead of "coming soon". - Note that server-side batching uses the gRPC API (enabled by default). - concepts/data-import.mdx: correct the now-stale "only the Python client supports server-side batch imports" (now Python, TypeScript, Java, and C#; Go does not yet), and fix a broken sentence.
1 parent d139873 commit c6def0c

2 files changed

Lines changed: 57 additions & 57 deletions

File tree

docs/weaviate/concepts/data-import.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Weaviate offers two flexible methods for importing data in bulk: **client-side b
1919

2020
:::tip
2121

22-
For **code examples**, check out the [How-to: Batch import](../manage-objects/import.mdx) guide. Currently, only the Python client supports server-side batch imports.
22+
For **code examples**, check out the [How-to: Batch import](../manage-objects/import.mdx) guide. Server-side batch imports are supported by the Python, TypeScript, Java, and C# clients. The Go client does not yet support them; use client-side batching instead.
2323

2424
:::
2525

@@ -40,7 +40,7 @@ Weaviate's server-side batching, also known as **automatic batching**, aims to p
4040
When an automatic batch import is initiated, the client opens a persistent connection to the server for the duration of the batch job.
4141

4242
- **Client sends data**: Your client sends objects to the server in chunks, at a rate that is based on server-provided feedback.
43-
- **Server manages queues**: The server places incoming objects into an internal. The queue is decoupled the network communication from the actual database ingestion (like vectorization and storage).
43+
- **Server manages queues**: The server places incoming objects into an internal queue. This queue decouples the network communication from the actual database ingestion (like vectorization and storage).
4444
- **Dynamic backpressure**: The server continuously monitors its internal queue size. It calculates an exponential moving average (EMA) of its workload and tells the client the ideal number of objects to send in the next chunk. This feedback loop allows the system to self-regulate, maximizing throughput without overwhelming the server.
4545
- **Asynchronous errors**: If an error occurs while processing an object (e.g., validation fails), the server sends the error message back to the client over a separate, dedicated stream without interrupting the flow of objects.
4646

docs/weaviate/manage-objects/import.mdx

Lines changed: 55 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -17,122 +17,122 @@ import CSharpCode from '!!raw-loader!/_includes/code/csharp/ManageObjectsImportT
1717
import GoCode from '!!raw-loader!/_includes/code/howto/go/docs/manage-data.import_test.go';
1818
import SkipLink from '/src/components/SkipValidationLink'
1919

20-
[Batch imports](../tutorials/import.mdx) are an efficient way to add multiple data objects and cross-references.
20+
[Batch imports](../tutorials/import.mdx) are an efficient way to add multiple data objects and cross-references. For most use cases, we recommend **server-side batching** as the starting point: the server tells the client how much data to send next, so you don't have to tune batch parameters yourself. When you need manual control over the batch size and concurrency — or you are using a client that does not yet support server-side batching — use [manual batching](#manual-batching) instead.
2121

22-
<details>
23-
<summary>Additional information</summary>
24-
25-
To create a bulk import job, follow these steps:
22+
## Server-side batching
2623

27-
1. Initialize a batch object.
28-
1. Add items to the batch object.
29-
1. Ensure that the last batch is sent (flushed).
24+
import SsbStatus from '/_includes/feature-notes/ssb-status.mdx';
3025

31-
</details>
26+
<SsbStatus/>
3227

33-
## Basic import
28+
With [server-side batch imports](../concepts/data-import.mdx#server-side-batching) (also called "automatic" batching), the client sends data in batch sizes determined by feedback from the server. This simplifies your code and helps the server manage its own load. The following example adds objects to a collection named `MyCollection`.
3429

35-
The following example adds objects to the `MyCollection` collection.
30+
Server-side batching uses the [gRPC API](#use-the-grpc-api), which current client versions enable by default.
3631

3732
<Tabs className="code" groupId="languages">
3833
<TabItem value="py" label="Python">
3934
<FilteredTextBlock
4035
text={PyCode}
41-
startMarker="# START BasicBatchImportExample"
42-
endMarker="# END BasicBatchImportExample"
36+
startMarker="# START ServerSideBatchImportExample"
37+
endMarker="# END ServerSideBatchImportExample"
4338
language="py"
4439
/>
45-
46-
### Error handling
47-
48-
<!-- TODO[g-despot]: Add link to external Python references once created for "reference page" -->
49-
50-
During a batch import, any failed objects or references will be stored and can be obtained through `batch.failed_objects` and `batch.failed_references`.
51-
Additionally, a running count of failed objects and references is maintained and can be accessed through `batch.number_errors` within the context manager.
52-
This counter can be used to stop the import process in order to investigate the failed objects or references.
53-
54-
Find out more about error handling on the Python client [reference page](/weaviate/client-libraries/python).
55-
5640
</TabItem>
5741
<TabItem value="ts" label="JavaScript/TypeScript">
5842
<FilteredTextBlock
5943
text={TSCode}
60-
startMarker="// START BasicBatchImportExample"
61-
endMarker="// END BasicBatchImportExample"
44+
startMarker="// START ServerSideBatchImportExample"
45+
endMarker="// END ServerSideBatchImportExample"
6246
language="ts"
6347
/>
6448
</TabItem>
6549
<TabItem value="go" label="Go">
66-
<FilteredTextBlock
67-
text={GoCode}
68-
startMarker="// START BasicBatchImportExample"
69-
endMarker="// END BasicBatchImportExample"
70-
language="go"
71-
/>
50+
51+
The Go client does not support server-side batching; use [manual batching](#manual-batching) instead.
52+
7253
</TabItem>
7354
<TabItem value="java" label="Java">
7455
<FilteredTextBlock
7556
text={JavaV6Code}
76-
startMarker="// START BasicBatchImportExample"
77-
endMarker="// END BasicBatchImportExample"
57+
startMarker="// START ServerSideBatchImportExample"
58+
endMarker="// END ServerSideBatchImportExample"
7859
language="java"
7960
/>
8061
</TabItem>
8162
<TabItem value="csharp" label="C#">
8263
<FilteredTextBlock
8364
text={CSharpCode}
84-
startMarker="// START BasicBatchImportExample"
85-
endMarker="// END BasicBatchImportExample"
65+
startMarker="// START ServerSideBatchImportExample"
66+
endMarker="// END ServerSideBatchImportExample"
8667
language="csharp"
8768
/>
8869
</TabItem>
8970
</Tabs>
9071

91-
## Server-side batching
72+
## Manual batching
9273

93-
import SsbStatus from '/_includes/feature-notes/ssb-status.mdx';
74+
Use manual (client-side) batching when you want to control the batch size and concurrency yourself, or when using a client that does not yet support server-side batching (such as the Go client). The following example adds objects to the `MyCollection` collection.
9475

95-
<SsbStatus/>
76+
<details>
77+
<summary>Additional information</summary>
78+
79+
To create a bulk import job manually, follow these steps:
80+
81+
1. Initialize a batch object.
82+
1. Add items to the batch object.
83+
1. Ensure that the last batch is sent (flushed).
9684

97-
Here's how to import objects into a collection named `MyCollection` using [server-side batch imports](../concepts/data-import.mdx#server-side-batching). The client will send data in batch sizes using feedback from the server.
85+
</details>
9886

9987
<Tabs className="code" groupId="languages">
10088
<TabItem value="py" label="Python">
10189
<FilteredTextBlock
10290
text={PyCode}
103-
startMarker="# START ServerSideBatchImportExample"
104-
endMarker="# END ServerSideBatchImportExample"
91+
startMarker="# START BasicBatchImportExample"
92+
endMarker="# END BasicBatchImportExample"
10593
language="py"
10694
/>
95+
96+
### Error handling
97+
98+
<!-- TODO[g-despot]: Add link to external Python references once created for "reference page" -->
99+
100+
During a batch import, any failed objects or references will be stored and can be obtained through `batch.failed_objects` and `batch.failed_references`.
101+
Additionally, a running count of failed objects and references is maintained and can be accessed through `batch.number_errors` within the context manager.
102+
This counter can be used to stop the import process in order to investigate the failed objects or references.
103+
104+
Find out more about error handling on the Python client [reference page](/weaviate/client-libraries/python).
105+
107106
</TabItem>
108107
<TabItem value="ts" label="JavaScript/TypeScript">
109108
<FilteredTextBlock
110109
text={TSCode}
111-
startMarker="// START ServerSideBatchImportExample"
112-
endMarker="// END ServerSideBatchImportExample"
110+
startMarker="// START BasicBatchImportExample"
111+
endMarker="// END BasicBatchImportExample"
113112
language="ts"
114113
/>
115114
</TabItem>
116-
<TabItem value="go" label="Go">
117-
118-
```go
119-
// Go support coming soon
120-
```
121-
115+
<TabItem value="go" label="Go">
116+
<FilteredTextBlock
117+
text={GoCode}
118+
startMarker="// START BasicBatchImportExample"
119+
endMarker="// END BasicBatchImportExample"
120+
language="go"
121+
/>
122122
</TabItem>
123123
<TabItem value="java" label="Java">
124124
<FilteredTextBlock
125125
text={JavaV6Code}
126-
startMarker="// START ServerSideBatchImportExample"
127-
endMarker="// END ServerSideBatchImportExample"
126+
startMarker="// START BasicBatchImportExample"
127+
endMarker="// END BasicBatchImportExample"
128128
language="java"
129129
/>
130130
</TabItem>
131131
<TabItem value="csharp" label="C#">
132132
<FilteredTextBlock
133133
text={CSharpCode}
134-
startMarker="// START ServerSideBatchImportExample"
135-
endMarker="// END ServerSideBatchImportExample"
134+
startMarker="// START BasicBatchImportExample"
135+
endMarker="// END BasicBatchImportExample"
136136
language="csharp"
137137
/>
138138
</TabItem>
@@ -171,7 +171,7 @@ The Java client v6 uses gRPC by default.
171171

172172
To use the gRPC API with the Go client, add the `GrpcConfig` field to your client connection code. Update `Secured` if you use an encrypted connection.<br/><br/>
173173

174-
```java
174+
```go
175175
cfg := weaviate.Config{
176176
Host: fmt.Sprintf("localhost:%v", "8080"),
177177
Scheme: "http",

0 commit comments

Comments
 (0)