Skip to content

Commit 4c0448f

Browse files
committed
feat: streaming upload, populated task.type, new postStream API (2.1.0)
- FilesAPI.upload streams chunked multipart instead of buffering the entire file in memory (eliminates the ~200 MB OOM cliff) - HttpClient.postStream: new public method for streaming POSTs (no retry because consumed ReadableStream bodies cannot be replayed) - client.getTask now uses response.type from /v1/tasks/{id} (api 1.38.5+); falls back to "" on older deployments - TaskStatusResponse.type? added to the public types - New named exports: buildMultipartStream, sanitizeFilename - Upload retries no longer happen on transient network errors (stream bodies can't be replayed); non-upload operations retain retry behaviour - 12 new streaming tests (90 total, all passing) - Minor bump: additive public API + behaviour change in upload retry semantics
1 parent 420c36f commit 4c0448f

18 files changed

Lines changed: 989 additions & 208 deletions

CHANGELOG.md

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Changelog
2+
3+
All notable changes to the `conversiontools` package are documented here.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
Release announcements with extended notes also live at
9+
<https://github.com/conversiontools/conversiontools-node/releases>.
10+
11+
## [Unreleased]
12+
13+
## [2.1.0] - 2026-05-20
14+
15+
### Added
16+
- **Streaming upload.** `FilesAPI.upload()` now sends the file as a chunked
17+
`multipart/form-data` body and no longer buffers the entire payload in memory.
18+
Safe for arbitrarily large files; eliminates the OOM cliff previous versions
19+
hit at ~200 MB.
20+
- `HttpClient.postStream(path, body, contentType, extraHeaders?)` — new public
21+
method for streaming POSTs. Skips the retry wrapper because a consumed
22+
`ReadableStream` body cannot be replayed.
23+
- New named exports: `buildMultipartStream`, `sanitizeFilename`.
24+
- Optional `type?: string` field on `TaskStatusResponse` to match the new API
25+
response shape (api 1.38.5+).
26+
27+
### Fixed
28+
- `client.getTask(id)` no longer hardcodes `task.type = ""`. When the API
29+
returns `type` (api 1.38.5+), it is populated on the returned `Task`
30+
instance. Older API deployments still fall back to `""`.
31+
32+
### Changed
33+
- Upload retries: prior versions could retry an upload up to 3 times after
34+
transient network failures because the body was a fully-buffered `FormData`.
35+
Streamed uploads cannot be replayed, so a failed upload now surfaces the
36+
error immediately. Non-upload operations retain the existing retry behaviour
37+
(3 retries, exponential backoff on 408/500/502/503/504).
38+
39+
### Notes
40+
- 12 new tests cover streaming behaviour: binary byte preservation, chunk
41+
boundaries, large-input streaming (no buffering), source-stream error
42+
propagation, sanitized filename, multipart envelope structure.
43+
44+
## [2.0.4] - 2026-03-31
45+
46+
### Changed
47+
- Update dependencies (minimatch, rollup, flatted, picomatch).
48+
49+
## [2.0.3] - 2026-02-21
50+
51+
### Added
52+
- `onProgress` callback in `downloadTo()` — track download progress with
53+
`loaded`, `total`, `percent` events.
54+
- `onDownloadProgress` config option now wired through `convert()`
55+
`task.downloadTo()`.
56+
57+
### Fixed
58+
- SDK version in the `User-Agent` header is now injected at build time from
59+
`package.json` (previously hardcoded as `2.0.0`).
60+
61+
## [2.0.2] - 2026-02-21
62+
63+
### Added
64+
- Config option to override the User-Agent.
65+
66+
## [2.0.1] - 2025-12-02
67+
68+
### Changed
69+
- Dependency bumps (glob and others).
70+
71+
## [2.0.0] - 2025-11-12
72+
73+
### Added
74+
- Complete rewrite with full TypeScript support (typed methods + options).
75+
- Native ESM + CommonJS dual module support.
76+
- Streaming download via `task.downloadStream()` / `task.downloadTo()`.
77+
- Real-time progress callbacks (upload, conversion, download).
78+
- Smart retry with exponential backoff for transient failures.
79+
- Modern stack: native `fetch`, Node 18+, zero heavy dependencies.
80+
- Webhook support for async task completion notifications.
81+
- Sandbox mode (`sandbox: true`) for testing without consuming quota.
82+
- Typed error classes for specific failure modes.
83+
84+
[Unreleased]: https://github.com/conversiontools/conversiontools-node/compare/v2.1.0...HEAD
85+
[2.1.0]: https://github.com/conversiontools/conversiontools-node/compare/v2.0.4...v2.1.0
86+
[2.0.4]: https://github.com/conversiontools/conversiontools-node/compare/v2.0.3...v2.0.4
87+
[2.0.3]: https://github.com/conversiontools/conversiontools-node/compare/v2.0.2...v2.0.3
88+
[2.0.2]: https://github.com/conversiontools/conversiontools-node/compare/v2.0.1...v2.0.2
89+
[2.0.1]: https://github.com/conversiontools/conversiontools-node/compare/v2.0.0...v2.0.1
90+
[2.0.0]: https://github.com/conversiontools/conversiontools-node/releases/tag/v2.0.0

dist/index.cjs

Lines changed: 112 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
'use strict';
22

3+
var crypto = require('crypto');
34
var fs = require('fs');
45
var path = require('path');
56
var stream = require('stream');
@@ -339,6 +340,69 @@ var HttpClient = class {
339340
...options
340341
});
341342
}
343+
/**
344+
* POST a streaming body (e.g. multipart upload).
345+
*
346+
* Bypasses the retry wrapper because a consumed ReadableStream cannot be
347+
* replayed. Use this for chunked uploads where buffering the whole body
348+
* in memory is unacceptable.
349+
*/
350+
async postStream(path2, body, contentType, extraHeaders) {
351+
const url = `${this.config.baseURL}${path2}`;
352+
const headers = {
353+
Authorization: `Bearer ${this.config.apiToken}`,
354+
"Content-Type": contentType,
355+
...extraHeaders
356+
};
357+
if (this.config.userAgent) {
358+
headers["User-Agent"] = this.config.userAgent;
359+
}
360+
const controller = new AbortController();
361+
const timeoutId = setTimeout(
362+
() => controller.abort(),
363+
this.config.timeout
364+
);
365+
try {
366+
let response;
367+
try {
368+
response = await fetch(url, {
369+
method: "POST",
370+
headers,
371+
body,
372+
signal: controller.signal,
373+
// `duplex: "half"` is required by fetch for streaming request bodies
374+
duplex: "half"
375+
});
376+
} catch (error) {
377+
if (error.name === "AbortError") {
378+
throw new TimeoutError(
379+
`Request timed out after ${this.config.timeout}ms`,
380+
this.config.timeout
381+
);
382+
}
383+
throw new NetworkError(
384+
`Network request failed: ${error.message}`,
385+
error
386+
);
387+
}
388+
this.extractRateLimits(response.headers);
389+
if (!response.ok) {
390+
await this.handleErrorResponse(response);
391+
}
392+
const data = await response.json();
393+
if (data.error) {
394+
throw new ConversionToolsError(
395+
data.error,
396+
"API_ERROR",
397+
response.status,
398+
data
399+
);
400+
}
401+
return data;
402+
} finally {
403+
clearTimeout(timeoutId);
404+
}
405+
}
342406
};
343407

344408
// src/utils/validation.ts
@@ -449,7 +513,10 @@ var FilesAPI = class {
449513
this.http = http;
450514
}
451515
/**
452-
* Upload a file from various sources
516+
* Upload a file from various sources.
517+
*
518+
* Streams the upload chunked to the API — never buffers the entire file
519+
* in memory. Safe for arbitrarily large inputs.
453520
*/
454521
async upload(input, options) {
455522
let stream$1;
@@ -475,20 +542,24 @@ var FilesAPI = class {
475542
if (options?.onProgress) {
476543
stream$1 = trackStreamProgress(stream$1, options.onProgress, fileSize);
477544
}
478-
const formData = new FormData();
479-
const chunks = [];
480-
for await (const chunk of stream$1) {
481-
if (typeof chunk === "string") {
482-
chunks.push(new TextEncoder().encode(chunk));
483-
} else if (chunk instanceof Buffer) {
484-
chunks.push(new Uint8Array(chunk));
485-
} else {
486-
chunks.push(chunk);
487-
}
488-
}
489-
const blob = new Blob(chunks);
490-
formData.append("file", blob, filename || "file");
491-
const response = await this.http.post("/files", formData);
545+
const boundary = `----conversiontools-${crypto.randomUUID()}`;
546+
const head = Buffer.from(
547+
`--${boundary}\r
548+
Content-Disposition: form-data; name="file"; filename="${sanitizeFilename(filename || "file")}"\r
549+
Content-Type: application/octet-stream\r
550+
\r
551+
`,
552+
"utf8"
553+
);
554+
const tail = Buffer.from(`\r
555+
--${boundary}--\r
556+
`, "utf8");
557+
const body = buildMultipartStream(head, stream$1, tail);
558+
const response = await this.http.postStream(
559+
"/files",
560+
body,
561+
`multipart/form-data; boundary=${boundary}`
562+
);
492563
if (response.error) {
493564
throw new ValidationError(response.error);
494565
}
@@ -569,6 +640,30 @@ var FilesAPI = class {
569640
return filename;
570641
}
571642
};
643+
function buildMultipartStream(head, source, tail) {
644+
return new ReadableStream({
645+
start(controller) {
646+
controller.enqueue(new Uint8Array(head));
647+
source.on("data", (chunk) => {
648+
const buf = typeof chunk === "string" ? Buffer.from(chunk) : chunk;
649+
controller.enqueue(new Uint8Array(buf));
650+
});
651+
source.on("end", () => {
652+
controller.enqueue(new Uint8Array(tail));
653+
controller.close();
654+
});
655+
source.on("error", (err) => {
656+
controller.error(err);
657+
});
658+
},
659+
cancel() {
660+
source.destroy?.();
661+
}
662+
});
663+
}
664+
function sanitizeFilename(name) {
665+
return name.replace(/[\r\n"\\]/g, "_");
666+
}
572667

573668
// src/api/tasks.ts
574669
var TasksAPI = class {
@@ -849,7 +944,7 @@ var Task = class {
849944
};
850945

851946
// src/client.ts
852-
var VERSION = "2.0.4";
947+
var VERSION = "2.1.0";
853948
var ConversionToolsClient = class {
854949
constructor(config) {
855950
validateApiToken(config.apiToken);
@@ -977,8 +1072,7 @@ var ConversionToolsClient = class {
9771072
return new Task(
9781073
{
9791074
id: taskId,
980-
type: "",
981-
// Type not available from status response
1075+
type: response.type ?? "",
9821076
status: response.status,
9831077
fileId: response.file_id,
9841078
error: response.error,

dist/index.cjs.map

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

dist/index.d.cts

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,8 @@ interface WaitOptions {
538538
*/
539539
interface TaskStatusResponse {
540540
error: string | null;
541+
/** Conversion type (added in API 1.38.5; absent on older deployments) */
542+
type?: string;
541543
status: TaskStatus;
542544
file_id: string | null;
543545
conversionProgress: number;
@@ -658,6 +660,14 @@ declare class HttpClient {
658660
* Make a POST request
659661
*/
660662
post<T = any>(path: string, body?: any, options?: Partial<RequestOptions>): Promise<T>;
663+
/**
664+
* POST a streaming body (e.g. multipart upload).
665+
*
666+
* Bypasses the retry wrapper because a consumed ReadableStream cannot be
667+
* replayed. Use this for chunked uploads where buffering the whole body
668+
* in memory is unacceptable.
669+
*/
670+
postStream<T = any>(path: string, body: ReadableStream<Uint8Array>, contentType: string, extraHeaders?: Record<string, string>): Promise<T>;
661671
}
662672

663673
/**
@@ -682,7 +692,10 @@ declare class FilesAPI {
682692
private readonly http;
683693
constructor(http: HttpClient);
684694
/**
685-
* Upload a file from various sources
695+
* Upload a file from various sources.
696+
*
697+
* Streams the upload chunked to the API — never buffers the entire file
698+
* in memory. Safe for arbitrarily large inputs.
686699
*/
687700
upload(input: string | NodeJS.ReadableStream | Buffer, options?: FileUploadOptions): Promise<string>;
688701
/**

dist/index.d.ts

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -538,6 +538,8 @@ interface WaitOptions {
538538
*/
539539
interface TaskStatusResponse {
540540
error: string | null;
541+
/** Conversion type (added in API 1.38.5; absent on older deployments) */
542+
type?: string;
541543
status: TaskStatus;
542544
file_id: string | null;
543545
conversionProgress: number;
@@ -658,6 +660,14 @@ declare class HttpClient {
658660
* Make a POST request
659661
*/
660662
post<T = any>(path: string, body?: any, options?: Partial<RequestOptions>): Promise<T>;
663+
/**
664+
* POST a streaming body (e.g. multipart upload).
665+
*
666+
* Bypasses the retry wrapper because a consumed ReadableStream cannot be
667+
* replayed. Use this for chunked uploads where buffering the whole body
668+
* in memory is unacceptable.
669+
*/
670+
postStream<T = any>(path: string, body: ReadableStream<Uint8Array>, contentType: string, extraHeaders?: Record<string, string>): Promise<T>;
661671
}
662672

663673
/**
@@ -682,7 +692,10 @@ declare class FilesAPI {
682692
private readonly http;
683693
constructor(http: HttpClient);
684694
/**
685-
* Upload a file from various sources
695+
* Upload a file from various sources.
696+
*
697+
* Streams the upload chunked to the API — never buffers the entire file
698+
* in memory. Safe for arbitrarily large inputs.
686699
*/
687700
upload(input: string | NodeJS.ReadableStream | Buffer, options?: FileUploadOptions): Promise<string>;
688701
/**

0 commit comments

Comments
 (0)