Skip to content

SPECIFICATION.md

Latest commit

 

History

History
447 lines (310 loc) · 28.8 KB

SPECIFICATION.md

File metadata and controls

447 lines (310 loc) · 28.8 KB
Raw

VOF Specification

Data Types

These are standard (to be preferred to alternatives) but optional (implemented as needed). They are not distinguished explicitly on the wire: applications agree out-of-band about when to use what, much like Protobuf, Thrift, etc. As such, types such as decimal and coordinates do not use explicit tags in the CBOR encoding, favoring compactness. See BINARY.md for details of VOF's own binary encoding proposed as slightly more compact than CBOR.

Type Names JSON CBOR
null Null Null
bool Boolean True, False
list Array Array
ndarray Array (nested) Array[[sizes…], values…]
uintmap Object Map
strmap Object Map
variant/enum Array[String,values…] / String list[Int,values…] / Int
record Object (keys: names) Map with uint keys
series 2D Array (row 0: names) / Empty Array list[[IDs…], values…] / list[]
string String (necessarily UTF-8) Text
bytes/data String base-64 URL encoded Bytes
uint Number / String if outside MIN/MAX for float64 Int
int/sint Number / String if outside MIN/MAX for float64 Int
decimal/dec String: optional - + 1+ digits + possibly . and 1+ digits Int (see below)
ratio String: optional - + digits + / + digits list[int,uint]
percent/pct String: decimal hundredths + % (i.e. "50%") dec ratio (i.e. 0.5)
float Number Float
date/_on uint as YYYYMMDD list[uint,uint,uint]
datetime/time uint as YYYYMMDDHHMM list[uint,uint,uint,uint,uint]
timestamp int Epoch int UNIX Epoch
timespan/span list[int,int,int] same
locale $locale enum IETF BCP-47 same
country/cntry $country enum ISO 3166-1 alpha-2 same
subdivision $subdivision enum ISO 3166-2 alpha-1/3
(no country prefix)		 same
currency/curr $currency enum ISO 4217 alpha-3 same
tax_code $tax enum "CC[_RRR]_X"
ISO 3166-1, ISO 3166-2, acronym		 same
unit $unit enum UN/CEFACT Rec. 20 same
text $locale record of strings
string for just one		 same
amount/price String: dec
+ optional space and curr		 list[dec,curr] / dec
tax/tax_amt String: dec
+ optional space and curr
+ mandatory space + tax_code		 list[dec,tax_code,curr]
list[dec,tax_code]
quantity String: dec
+ optional space and unit		 list[dec,unit] / dec
ip String: IPv4 or IPv6 notation bytes with 4 or 16 bytes
subnet/net String: CIDR notation list[ip,uint] CIDR notation
coords list[float,float] WGS84 same

Units of measure

We use codes from UN/CEFACT Recommendation 20. See full list at: unece.org. Quantities should default to EA (each) when they do not carry an explicit unit code. Here are some of the most common codes:

Category Units
Grouping EA Each • PR Pair • P3 Three-pack • P4 Four-pack • P5 Five-pack • P6 Six-pack • P8 Eight-pack • DZN Dozen • CEN Hundred
Form AY Assembly • CG Card • DC Disk • NF Message • NV Vehicle • RU Run • SET Set • SX Shipment • ZP Page
Time SEC Second • MIN Minute • HUR Hour • LH Labor hour • DAY Day • MON Month • ANN Year
Weight GRM Gram • KGM Kilogram • LBR Pound
Length CMT Centimetre • MTR Metre • INH Inch • FOT Foot • YRD Yard
Area CMK Square centimetre • MTK Square meter • INK Square inch • FTK Square foot • YDK Square yard
Volume MLT Millilitre • LTR Litre • INQ Cubic inch • ONZ Ounce • OZA Fluid ounce US • OZI Fluid ounce UK • QT Quart US • QTI Quart UK • GLL Gallon US • GLI Gallon UK
Energy KWH Kilowatt hour
Data 2P Kilobyte • 4L Megabyte

Namespaces: Variant, Enum, Record

A project or API has a root namespace, dot-delimited, i.e. com.example

Variant, Enum and Record types need unique namespaces in singular form, for example: com.example.order.line. Namespaces only allow characters: [$0-9A-Za-z._-] and are upper/lower case and hyphen/underscore insensitive.

Namespaces suggested by VOF are at root level and begin with a dollar sign, like com.example.$locale. Applications should not use dollar signs in their own namespaces. The namespaces are: $msg, $locale, $country, $subdivision, $currency, $tax, $unit.

Identifiers within a namespace are strings (JSON) and unsigned integers starting from 0 (CBOR, similar to Protobuf). To avoid version conflicts, identifiers must remain reserved forever when they are deprecated. Symbols are restricted to characters [0-9A-Za-z_-] and are upper/lower case and hyphen/underscore insensitive.

In CBOR, variants, enums and record fields are identified by a unique unsigned integer (similar to Protobuf), which should also be reserved forever. For a given context (for example, a company's HTTP API), encoders should maintain a global, namespaced symbol table. Applications calling encoder functions for these three types must provide a namespace (i.e. com.example.order.line) in which field names will be assigned integers starting from zero as they are first encountered. This table must be managed centrally and shared with other endpoints (much like Protobuf IDL .proto files must be shared among endpoints).

Decoders must use the canonical string (as specified in the symbol table) when decoding integers to strings. Thus, even if symbols are case insensitive, the string returned must match the initial declaration, allowing applications to use symbols like "USD", "en_US", etc. Encoders must accept aliases on input but only emit canonical symbols on output. (Thus even in JSON, a declared "en_US" specified as "en" in a Text must still be encoded as "en_US".)

Symbol tables are simple 7-bit ASCII files listing symbols in field order and are thus strictly append-only for each namespace, to preserve field IDs forever.

  • Lines are terminated by LF or CR-LF, stripped when reading;
  • Empty lines are ignored;
  • Lines starting with '#' are ignored;
  • Lines starting with a TAB are symbols in the current namespace;
  • Symbol lines may be appended whitespace delimited qualifiers;
  • Other lines are namespace declarations.
# VOF Symbol Table

com.example.$msg
	orders list:com.example.orders
	; ... custom fields

com.example.$currency
	USD
	CAD
	EUR

com.example.$locale
	en_US aka:en,en_CA
	fr_CA aka:fr

com.example.order
	id key
	modified_at req
	customer
	currency
	lines
	total

com.example.order.line
	i key
	product
	qty
	unit_price

In the above example, symbol 'customer' in namespace 'com.example.order' is ID 1.

Symbol Qualifiers

Qualifiers may be simple tags or "key:value" pairs or "key:v1,v2,..." multi-value pairs.

  • aka — Aliases for the current symbol, reserved forever. Implementations must fail when an application or symbol table file sets the same alias on multiple symbols.
  • key — In records, tags a field as being part of the primary key.
  • req — In record references, tags a field to always include along with keys.
  • list: — In the $msg namespace in your root path, tags fields which intended as lists of records with their full path. When reading, if multiple symbols are declared as lists of the same path, the last one prevails.

Readers should quietly ignore unknown qualifiers, however they should preserve them when updating the file.

NDArray

Fixed-size multi-dimensional lists. This is a list where the first item is a list of dimension sizes, followed by each value from zero-index onward, exactly as many values as the product of all dimension sizes.

For example, a 3D array of size 2x2x2 could be: [[2,2,2],1,2,3,4,5,6,7,8]

Record

In CBOR, records are normal maps with uint keys.

In VOF Binary, records are a list in which fields are positional (field ID 0 in first place, field ID 20 in 21st place, etc.) Missing fields are replaced by gap codes. Omit any trailing spacers when encoding, but tolerate them when decoding. For example, a 16-field record with just { 0:1, 3:2, 9:3 } becomes [1,gap(1),2,gap(4),3]

Series

Compact representation of a list of record where all the same fields are defined (typical of time series data, product price lists, etc.) In JSON and CBOR, this is a 2-D Array where the first row selects fields by name (by ID in CBOR) and each subsequent row is an Array with just those values. If numeric field IDs are available, that order should be used instead of alphanumeric, to make field order vary the least over time.

Any field missing from a record is encoded as Null. An empty series must be encoded with a singular empty Array (i.e. [] not [[]]).

In VOF Binary, this is a flat list where the first item is a list of numeric field IDs and the other items are the values of each record for the selected fields only, one after the other (no wrapping).

Decimal

Necessary for financial data. In JSON, it must be a string to bypass possible float conversions done by some libraries.

In CBOR, negatives are handled implicitly. The value is the significant digits multiplied by 10, plus the number of decimal places. For example, 2.150 would be 2152.

In VOF Binary, the absolute value is encoded unsigned and an optional Alt qualifier is prefixed if the value is negative. The significant digits are shifted left 2 bits and added a tag representing 0, 2, 4 or 9 decimal places. For example, 2.123 would require 4 decimal places as (21230 << 2) + 2 = 84922.

Date

Calendar date, sortable. Time zone is outside the scope of this type, derived from context as necessary. In JSON, a human-friendly YYYYMMDD number is used to avoid using strings.

In CBOR, it is a 3-uint list containing the full year, month 1..12 and day 1..31.

In VOF Binary, it is limited to years 1900+ and structured in 17 bits as (year << 9) + (month << 5) + day where:

  • year — Number of years since 1900 (i.e. 2025 is 125)
  • month — 1..12
  • day — 1..31

Datetime

Extends date with wall clock time with minute precision, still sortable and with implicit time zone. In JSON, a human-friendly YYYYMMDDHHMM number is used to avoid using stings.

In CBOR, it is a 5-uint list which adds hour 0..23 and minute 0..59.

In VOF Binary, it is structured in 28 bits as (year << 20) + (month << 16) + (day << 11) + (hour << 6) + minute where:

  • year — Number of years since 1900 (i.e. 2025 is 125)
  • month — 1..12
  • day — 1..31
  • hour — 0..23
  • minute — 0..59

Timestamp

In VOF Binary, 1,750,750,750 is subtracted from timestamps on the wire, which makes zero around June 2025 (instead of January 1970) to make typical timestamps smaller.

Timespan

Calendar duration expressed as half-months, days and seconds, each signed and applied in three steps in that order when it is used. For example, "one year minus one day" would be [24,-1,0].

Text

  • Decoders encountering multiple strings with the same locale code must pick one, preferrably the last one read.
  • Applications' $locale namespace should include aliases ("aka" qualifiers) for bare languages exactly once per language group, for whichever qualified locale is the application's default for the language.
  • Encoding CBOR or Binary must fail if there is no $locale namespace defined. JSON should gracefully downgrade to a regular strmap. Decoding CBOR or Binary without a $locale should gracefully skip over the value.
  • A text with a single string for the first declared locale (ID 0) when there is $locale, or for code "" (empty string) when there isn't, is canonically encoded as a bare string.
  • In JSON, the strmap keys must be made canonical (resolving aliases) when $locale exists.

JSON Encoding

  • The regular MIME type (application/json) for JSON encoded transfers is recommended for compatibility.

  • When possible, records should be sorted by ascending key when buffering the whole list is possible, to facilitate compression.

  • Number, decimal and ratio must strip leading zeros and trailing decimal zeros.

  • Integers must only encode as decimal when they are outside of JavaScript MIN/MAX_SAFE_INTEGER range.

CBOR Encoding

  • The regular MIME type (application/cbor) for CBOR encoded transfers is recommended.
  • When possible, maps should be sorted by ascending key when buffering the whole list is possible, to facilitate compression. This is usually done by using a CBOR encoder in deterministic "canonical" mode, but some implementations may require applications to pass sorted lists to encoders to achieve this manually.
  • You may begin a stream with CBOR's magic, tag 55799, although it is not necessary for disambiguation when decoding.
  • Variants without arguments should be unwrapped to a bare integer.

VOF Binary Encoding

  • The suggested MIME type for binary encoded transfers is application/x-vanilla-object. The suggested file name extension is .vo.
  • When possible, maps should be sorted by ascending key when buffering the whole list is possible, to facilitate compression.

API Best Practices

This section suggests a standard for designing HTTP APIs with the VOF data types.

URLs described would be relative to the root of where the API is served (i.e. https://api.example.com/api/12.1/) so for example GET …/users would mean getting /api/12.1/users.

With methods carrying content (PATCH, POST) should use Content-Type: application/json and X-Content-Type-Options: nosniff to prevent second-guessing based on contents.

If you need to issue multiple API requests within a few seconds, HTTP/1.1 keepalive use is encouraged.

HTTP Response Codes

Code Methods Notes
200 OK all Full success with a response body
207 Multi-Status POST, PATCH Partial success with a response body
400 Bad Request GET, POST Full failure, don't retry this query as-is
401 Unauthorized all Authentication failure
403 Forbidden all Authentication valid but insufficient rights
404 Not Found GET, PATCH, DELETE Incorrect URL or non-existent ID
429 Too Many Requests all Wait and try again
500 Internal Server Error all Wait and try again
501 Not Implemented all Fatal error, don't retry

Conventions

  • Reference: a record with only a primary key and, if applicable, a last modification timestamp. Where there may be a foo_id field in storage, in VOF APIs a foo reference field is preferable, as it is interchangeable with an inline instance without type ambiguity.
  • Use decimal and its derivatives for financial data which requires exact precision (quantities, amounts) and float64 for ratios and other non-financial data better suited for floating point numbers.
  • Record field types may only ever be changed for wire-compatible ones. For example, decimal could become string, but not the other way around.
  • Generated views/reports should be declared as record types in their module, probably with a Series result.
  • Variant/Enum/Record use Capitalized names. Dependent records are namespaced in their parent, i.e. Order.Line used by Order.
  • Fields use snake_case. Pluralize lists (i.e. lines)
  • Field names with multiple words should go from most to least precise (i.e. prefer item_qty over qty_item)
  • Suffix non-self-describing field names to clarify their type when the value might not be obvious: _id, _amt or _price, _qty, _tax

GET Parameters

These query string parameters are available for all GET requests. In order to avoid conflicts with field names, all these parameters end with a tilde (~). This also has the benefit of being visually distinctive (i.e. max~=20 for a result limit).

select~

(Default: *) By default, for GET responses, records are sent with all fields present and with references to other records (i.e. order customers, order line products), and only private child records inlined (i.e. an order's lines). This parameter allows specifying which fields to include, which record fields to inline and which to attach separately in the $msg. It is a comma-delimited list of field names with some modifiers:

  • * — Include all fields at this level.
  • !foo — Exclude field foo, only valid after *.
  • foo — Regular field to be included (scalar, reference or inline record).
  • foo() or foo(…) — Expand foo referenced record inline.
  • $foo or $foo(…) — If foo is a reference, attach the full record in $msg (de-duplicated).
The syntax is recursive, so `*,foo(bar(*,!x),baz)` means all fields, expand `foo` into its record but with only `bar` and `baz`, and exclude `x` from `bar`.

Full example: GET …/orders?user=12345&select~=id,ordered_on,grand_total,lines(qty,unit_price,product(id,name))

prune~

(Default: none.) List of a record's fields (expected to be lists of records) to filter based on query filters. For example, it could be desirable to restrict order lines in each returned order for a query filtering on order line product types.

For example, …/orders?prune~=lines&is_draft=$false&date=between:20250101:20251231&lines.product=in:ABC:DEF would return final orders placed in 2025 which have lines about products ABC or DEF, but each order would only include lines about products ABC or DEF.

max~ and page~

(Defaults: max~=100 and page~=1) Restricts results returning multiple records to fewer per call.

Row Filters

  • Format: field[!]=[operator:]value[:value2…]
  • Appending '!' to a field name negates its operator, like name!=has:Smith selects all names which do not include "Smith".
  • Filters are additive (all must be true).
  • Fields may be used more than once.
  • Record field members use '.' separators, i.e. order.lines.qty
  • Bare record fields match on their primary key, i.e. order.contact implies order.contact.id

Available operators (some with synonyms):

Operator Meaning
none equals exactly (i.e. id=1234)
lt/under/before field is less than (i.e. price=lt:10)
lte/upto field is less than or equal
gt/over/after field is greater than
gte/atleast field is greater than or equal
between inclusive, i.e. created_on=between:20250101:20251231
has string contains keyword
in(…) exactly one of these values, i.e. categ=in:123:234:345

Special values are prefixed with '$' and could include:

Value Meaning
$false Null, false, "0", number 0, empty string, empty list
$true Any non-false value
$today current date in the field's timezone
$now current datetime in the field's timezone

Using filters on children implies that parents without any matching children will not be included. By default, all direct children of included parents (i.e. order lines) are included.

PATCH Updates

A PATCH record is a possibly incomplete copy of an existing record with the primary key specified in the URL and/or in the record itself. The patch version of a record has the exact same structure as the record itself (like a normal REST PUT), with the following additional operations available for convenience:

Field Type Change Encoding
Any Unchanged Omit entirely
Any Unset Set to Null explicitly
Record list Unchanged item Omit entirely
Record list New item Record without ID or with new ID
Record list Edited item Record with ID and changed fields
Record list Deleted item Reference (record with only ID)
Other lists Any change Full replacement 
// Example patch on a hypothetical order record
{
  // Simple field updates
  delivered_on: null,
  deliver_by: 20250131,

  // Array replacement (simple values)
  labels: [ "red", "blue" ],
  notify_user_ids: [ 836583, 647684 ],  // Remove previous CC list, add these two

  // Discrete operations on arrays of private child records
  lines: [
    // Line i=1 unchanged
    { i: 2 },  // Delete line i=2
    { i: 3, qty: "1.3", subtotal: "13" },  // New values in line i=3
    { qty: "5", retail_amt: "5.05", subtotal: "25" },  // New line, server-assigned i
    { qty: "4", retail_amt: "1", subtotal: "4" },  // Other new line
  ]
}

Standard Endpoints

Unless specified otherwise, record types offer an endpoint corresponding to its namespace without the global prefix, in plural form and with forward slash separators. (i.e. com.example.order.account would be endpoint …/orders/accounts) Below, {path} represents this path (i.e. orders/accounts) and {Record} the main record type (i.e. "Order"). Some child-only types (i.e. order lines) don't necessarily have endpoints.

Ping endpoint: GET …/ should return a $msg with text set to "Pong!".

Simple endpoints:

Endpoint Request body Response $msg fields
GET …/{path}/{id} - One record in the type's field
GET …/{path}[?…] - Many records in the type's field
POST …/{path} {Record} without ID (new) One ref in the type's field
PATCH …/{path} {Record} with ID (existing) One ref in the type's field
DELETE …/{path}/{id1}[,…] - If any failed: none deleted, text set

Multiplexed endpoints are on the root path, which is reserved for protocol-level use:

Endpoint Request body Response $msg fields
POST …/ $msg records without IDs (new) References in affected types' fields
PATCH …/ $msg patches with IDs (existing) References in affected types' fields

Clients should collect records and all their descendants recursively in a single request. Records with old or no modified time field should be added as references, dependent and recent records should be included in full. On HTTP 200, response references confirm the records which have been created/updated. On HTTP 207, no action has been taken and references are for records which need to be added in full to the client's request in order to succeed.

Message $msg

Every HTTP response is a $msg record, which consists of a few meta-data fields plus one list field per record type in the project. This is a root-level namespace, like com.example.$msg.

Field Type Notes
text string optional Status details, error explanations
remaining uint optional If the current result set is limited, how many items follow
orders Order list Records of an example Order type
orders_sales Order/Sales list Hypothetical sales report rows
... ...

Implementation Considerations

Maps

Decoders should use the last value when a key is present multiple times.

Series

Encoders can either pre-scan the full series to collect the possible record fields, which is convenient but breaks streaming ability, or rely on the first record's fields and fail if any subsequent record includes extra fields when streaming is needed. This is considered an implementation detail.

Decoding and Compression

Encoders are encouraged to use Gzip or Zstd compression for VOF messages exceeding 100-200 bytes. Decoders can always know the format of VOF data by inspecting the first byte:

First byte Unique meaning
0x1F Gzip
0x28 Zstd
0x5B,6E,7B JSON (array, null, object)
0x80-DB,F6 CBOR (array, map, tag, magic, null)
0xE8..F3,FA..FD Binary (list, alt, tag, null)

Design Compromises

  • The decimal, date and datetime types were designed for financial systems based on SQLite and kept here for their compact sizes.
  • Negative decimals incur a one-byte penalty to optimize for positive cases which dominate in business applications.
  • The last size of decimal is 9 and not 7 in order to match the maximum precision allowed in some other business contexts such as ANSI X12.