Feature Request: Expose dt_metadata_add_metadata to Lua for script-defined per-image metadata

[radialmonster]

Is your feature request related to a problem? Please describe.

Lua scripts have no good way to persist structured per-image state across darktable restarts. For example, a publish/export plugin (such as one targeting Flickr, SmugMug, or any remote service) needs to remember, per local image, the remote photo ID it was uploaded as and which remote album(s) it belongs to. Today the only Lua-reachable persistent per-image store is image tags — a flat string namespace that's awkward and collision-prone for structured data. darktable's C core already has custom metadata fields (dt_metadata_add_metadata()), which are exactly the right shape, but there is no Lua binding to register one.

Describe the solution you'd like

Add a Lua binding darktable.register_metadata{ ... } that lets a script declare a persistent, per-image, named metadata field — a thin Lua wrapper around the existing C dt_metadata_add_metadata().

darktable.register_metadata{
  tagname = "Xmp.dtpublish.photo_id",  -- full XMP tag; last segment becomes the property name
  name    = "Remote photo ID",         -- label in the metadata editor
  visible = false,                     -- optional: hide from the editor
  private = true,                      -- optional: keep out of exported XMP
}

-- thereafter, on any dt_lua_image_t:
image.photo_id = "53812345678"   -- write
local id = image.photo_id        -- read

Nearly all the machinery already exists: definitions persist in data.meta_data, and the generic metadata_member getter/setter resolves fields dynamically — so this is essentially a thin, idempotent binding plus exposing the per-field member registration. It gives scripts a structured, schema'd per-image store instead of encoding state into tag strings.

Useful well beyond any one plugin: publish/export targets (remote photo IDs, album membership), reverse-geocode caches, AI-tagging confidence/model, external-DAM sync UUIDs + timestamps, and rating bridges.

Note — this needs a new field of the script's own, which is why the metadata editor can't substitute. The field a script wants is its own (e.g. Xmp.dtpublish.photo_id), so it doesn't collide with the user's real metadata — not an existing tag. The metadata-editor "add tag" dialog only lists tags from exiv2's known-tag database (dt_exif_get_exiv2_taglist()), so a user can add existing tags but cannot create a new one of the script's own — and it pushes a manual setup step onto every user. dt_metadata_add_metadata accepts any tag name, so a small Lua binding to it is the only way for a script to create its own field.

Describe alternatives you've considered

• Image tags — the only Lua-reachable persistent per-image store today. Works, but a flat string namespace: encoding structured/multi-value state (numeric ID + album set + timestamp) is awkward, collision-prone, has no schema, and pollutes the user's tag cloud.
• The built-in notes field (dt_lua_image_t.notes) — persistent and writable from Lua, but it's a single user-facing field (clobbers the user's own notes) and single-valued, so it can't hold structured per-service state.
• darktable.preferences — global / per-script only, with no per-image variant; can't map a value to an individual image.
• Have the user add the field in the metadata editor — the editor's add-tag dialog only offers exiv2-known tags (dt_exif_get_exiv2_taglist()), so a script can't get a new field of its own (Xmp.<plugin>.*) this way; it only covers reusing existing tags, and forces a manual setup step on every user.
• A C companion / patched build — dt_metadata_add_metadata() is already callable from C, but requiring a custom-compiled or forked darktable defeats the point of a drop-in Lua script. This request is exactly to make that capability reachable from Lua so no fork is needed.
• External sidecar DB maintained by the script — possible, but reinvents per-image storage darktable already has, and doesn't travel with the image or integrate with XMP.

Additional context

Full design proposal below, with source citations. All file:line references are to the current master.

---

Existing machinery (what already works)

These are the load-bearing facts a reviewer can confirm cheaply.

1. Definitions persist in data.meta_data (shared, survives restart). dt_metadata_add_metadata() (src/common/metadata.c:60) INSERTs the field definition into data.meta_data:

INSERT INTO data.meta_data
  (key, tagname, name, internal, visible, private, display_order)
  VALUES(NULL, ?1, ?2, ?3, ?4, ?5, ?6)

key is passed NULL, so SQLite auto-assigns it; the function then reads it back (metadata.c:80-84) and prepends the entry to the in-memory _metadata_list. data.meta_data lives in data.db (shared), distinct from the per-image values table main.meta_data. On next launch, dt_metadata_init() (metadata.c:189) reloads every definition. No new storage code is needed for persistence.

2. The generic getter/setter resolves the field at call time. metadata_member() (src/lua/image.c:312) backs every Lua image.<field> metadata accessor and resolves the field dynamically — read via dt_metadata_get_lock (image.c:319), write via dt_metadata_set (image.c:331). Because the key is looked up at call time, a newly-added field needs no new read/write code.

3. The dt_metadata_t struct already carries every option to expose (src/common/metadata.h:25-34): tagname, name, internal, visible, priv, display_order (+ DB-assigned key).

Proposed Lua API

Modeled on darktable.preferences.register (a single function taking a named-style argument list).

Lua option	dt_metadata_t field	Default	Notes
tagname	tagname	—	Required. Full XMP tag, e.g. Xmp.dtpublish.photo_id; must contain a . (the Lua property name is the last segment — see G2).
name	name	—	Required. Human label in the metadata editor.
visible	visible	true	Editor visibility (also participates in the export predicate — see semantics).
private	priv	false	"Do not export" — suppresses the field from exported XMP.
display_order	display_order	0	Sort order in the editor list.
(forced)	internal	false	Hard-coded false; a field with internal = true would be invisible to Lua. Not exposed.
(not exposed)	key	—	Assigned by SQLite.

"Hide from the editor" is visible = false; "keep out of exports" is private = true.

Semantics: internal vs visible vs priv

Flag	What it gates	Evidence
internal	Master gate. When true, the field is excluded from the Lua image type (src/lua/image.c:627 registers metadata_member only if(!metadata->internal)), from the editor grid (src/libs/metadata.c:812-813), from the editor prefs list (metadata.c:868), and from XMP export (src/common/exif.cc:5835). A Lua-registered field must have internal = false.
visible	Editor visibility (src/libs/metadata.c:479, :527). Does not affect Lua access. Does affect export: a field is written to XMP iff visible && !priv (src/common/exif.cc:5051, :5835).
priv	"Private" / do-not-export (private column, commented // do not export at src/libs/metadata.c:858). Export requires !priv. Does not affect Lua access.

⚠️ Key subtlety: because export tests visible && !priv, setting visible = false also keeps the field out of exported XMP. So for an opaque plugin bookkeeping field (e.g. a remote service's photo ID), visible = false alone gives "invisible in the editor and never exported".

Design gotchas the implementation must handle

G1 — Idempotency. dt_metadata_add_metadata() INSERTs unconditionally with no existence check (metadata.c:60-94). Since Lua scripts run on every launch, the binding must guard with dt_metadata_get_metadata_by_tagname() (metadata.c:107) and no-op if the field already exists, or it will accumulate duplicate rows.

G2 — Subkey collision. The Lua property is only the substring after the last . (dt_metadata_get_tag_subkey, metadata.c:174), and lookups match only that segment, returning the first hit (dt_metadata_get_key_by_subkey, metadata.c:151-172). So Xmp.darktable.notes and Xmp.acdsee.notes would both map to image.notes and collide. The binding should reject a tagname with no ., and reject one whose final segment already resolves to a different existing tagname.

G3 — The internal/visible overload. internal is the only flag controlling Lua visibility. Don't let callers set internal = true. "Hide from editor" = visible; "do not export" = priv.

G4 — Member-registration timing & the static qualifier. metadata_member is static int metadata_member(lua_State *L) (image.c:312), so a new translation unit can't reference it. The image-type member loop runs once at Lua init (image.c:621-633); a field added at script-run time is only usable if a member for its subkey is registered on dt_lua_image_t via dt_lua_type_register. Preferred fix: un-static metadata_member (or add an exported dt_lua_image_register_metadata_member(L, subkey) helper) and call it after a successful add, so the field works in the same session.

Who else benefits

• Publish/export plugins (e.g. Flickr, SmugMug, Piwigo-via-Lua) — remote photo IDs + album membership.
• Reverse-geocode caches — resolved place name / OSM IDs per lat-long, looked up once.
• AI / auto-tagging — model name, version, per-image confidence.
• External-DAM sync — foreign UUID + last-synced timestamp.
• Rating / flag bridges — mirror an external app's rating without clobbering darktable's native one.

Minimal patch checklist

1. New binding darktable.register_metadata (new src/lua/metadata.c or extend src/lua/image.c): validate tagname/name; G1 idempotency guard; G2 collision guard; build a dt_metadata_t with internal = FALSE; call dt_metadata_add_metadata; register the image-type member (needs G4).
2. G4 plumbing — un-static metadata_member or add an exported registration helper in src/lua/image.c.
3. API-version bump so scripts can feature-detect.
4. luadocs page for darktable.register_metadata; cross-link from dt_lua_image_t.
5. Register the new module in the Lua bootstrap if a new file is added.

Backward compatibility

Scripts should feature-detect and fall back to tags, never hard-depend on this:

if type(darktable.register_metadata) == "function" then
  darktable.register_metadata{ tagname = "Xmp.dtpublish.photo_id", name = "Remote photo ID",
                               visible = false, private = true }
  -- use image.photo_id
else
  -- fallback: encode state in a namespaced tag, e.g. "darktable|publish|id|<photo_id>"
end

Validation log — every claim above checked against source

#	Claim	Status	Evidence
1	dt_metadata_add_metadata persists the definition by INSERT into data.meta_data (distinct from per-image main.meta_data); key auto-assigned.	VERIFIED	INSERT data.meta_data VALUES(NULL, ...) src/common/metadata.c:66-69; key read back metadata.c:80-87; reload dt_metadata_init metadata.c:189-225.
2	Generic metadata_member resolves the key dynamically; new field needs no new read/write code.	VERIFIED	dt_metadata_get_key_by_subkey src/lua/image.c:315; read image.c:319; write image.c:331.
3	Startup loop registers image.<subkey> only when !internal; hiding from UI uses visible=false.	VERIFIED (clarified)	!metadata->internal src/lua/image.c:627; visible editor src/libs/metadata.c:479,527; export visible && !priv src/common/exif.cc:5051,5835. Clarification: visible=false also suppresses export.
4	Lua property name is only the last dotted segment; shared final segments collide.	VERIFIED	dt_metadata_get_tag_subkey metadata.c:174-178; member named image.c:630; first-match lookup metadata.c:151-172.
5	dt_metadata_add_metadata has no existence check → duplicate rows unless guarded.	VERIFIED	No dedup metadata.c:60-94; guard dt_metadata_get_metadata_by_tagname metadata.c:107.
6	metadata_member is static; patch must expose it or relocate registration.	VERIFIED	static int metadata_member src/lua/image.c:312; registration image.c:629-630.
7	No existing Lua binding registers metadata.	VERIFIED	Zero dt_metadata_add_metadata matches under src/lua; only callers are src/libs/metadata.c:1068 and the decl/def.

Net result: all claims verified. The only refinement (Claim 3): internal is the sole gate on Lua visibility, but visible is overloaded — it gates editor display and participates in the export predicate (visible && !priv), so visible = false also keeps a field out of exported XMP.

[wpferguson]

There is already darktable.metadata. in the Lua API. Right now the only implemented function is darktable.metadata.exists(<tag name>). The intention is to extend darktable.metadata. with more functionality (add, delete, etc.) but I didn't get there yet.

---

IIUC you are wanting to use the metadata editor, but from Lua? Currently if you define a metadata field in the metadata editor module, it is accessible from Lua, specifically dt_lua_image_t.<fieldname> where fieldname is the last part of the tag (creator for Xmp.dc.creator). The only thing you need to do before using is check if the metadata field exists (example: darktable.metadata.exists("creator") before using it since trying to access a field that doesn't exist will cause a crash. You don't need to check for the default defined fields because they are already "built in". The data will be stored and accessible per image.

---

EDIT: The darktable Lua documentation is at https://darktable-org.github.io/luadocs/

[radialmonster]

Thanks for taking a look, and for pointing me to darktable.metadata.exists() — I hadn't spotted that.

Let me restate our use case in case I described it poorly the first time, since I think the key thing is check vs. create. exists() and defining a field in the metadata-editor module both assume the field is already there (created by the user). What we're after is a way for the script itself to create its field on load — which I think is exactly the add you mentioned wanting to add to darktable.metadata.. The motivation is install-and-go for end users: a distributed script can't really ask people to open preferences and hand-add a custom field before it works, so being able to declare the field programmatically (and then guard with exists()) would close the loop.

While researching I went through the C side, and most of the pieces look like they're already in place — so add may be a fairly small addition to src/lua/metadata.c (which today only has exists). The relevant spots:

• dt_metadata_add_metadata() in src/common/metadata.c already does the work: it INSERTs the field definition into data.meta_data, so definitions persist across restarts (reloaded by dt_metadata_init()). No new storage code needed.
• metadata_member() in src/lua/image.c is the generic getter/setter behind image.<field>; it resolves the key dynamically (dt_metadata_get_key_by_subkey → dt_metadata_get_lock / dt_metadata_set), so a newly added field is read/written with no extra code. It's currently static, so exposing it (or a small helper) is the one bit of glue an add binding would need to register the image member in the same session.
• Edge cases I noticed: dt_metadata_add_metadata() has no dedup, so an idempotency guard via dt_metadata_get_metadata_by_tagname() matters (scripts re-run every launch); the Lua member is only the last .-segment of the tag, so two tags sharing a final segment collide; and the internal/visible/priv flags interact (a field has to be internal=false to be Lua-visible, and export tests visible && !priv). Full write-up is in the issue above.

One more thing worth being explicit about: the field a script needs is one of its own (e.g. Xmp.dtpublish.photo_id), so it doesn't collide with the user's real metadata — not an existing tag. And I don't think the metadata-editor route can create that: its "add tag" dialog is populated only from exiv2's known-tag list (dt_exif_get_exiv2_taglist() in metadata_tags.c), so a user can add existing tags but can't make a new one. dt_metadata_add_metadata takes any tag name, which is why a small Lua binding to it seems like the only way for a script to create its own field without asking users to do manual setup.

[wpferguson]

What we're after is a way for the script itself to create its field on load

Which probably wont help you, because the next thing you are going to want to do is use it. If it's an image metadata field it wont be populated unless you re-import the images so that the metadata fields get filled, or you select the images and do a exif metadata refresh from the actions on selected images module. If you are just adding text fields, then it will.

I'm not even sure it's a big deal because the user would only have to do it 1 time. To me, it's a convenience.

matters (scripts re-run every launch); the Lua member is only the last .-segment of the tag, so two tags sharing a final segment collide

It's worse. If you create two tags with the same final segment it will crash darktable with a segfault (IIRC because I did it once with a Canon tag and a Nikon tag then ended with the same final segment) because you are trying to add the same field to dt_lua_image_t twice. If you could add it twice, then which one would you get when you accessed the field?

has no dedup

check if it exists and if not then create

which today only has exists

We were at feature freeze, didn't have time for the rest

and the internal/visible/priv flags interact (a field has to be internal=false to be Lua-visible

adding through metadata editor doesn't let you set internal, nor should the API for obvious reasons :-)

---

The most requested changes concerning metadata are for image metadata (exif data). Adding it via metadata editor is probably the safest since it provides the tags in a drop down and you can just pick and choose. The provided tags in the drop down are what exiv2 knows about, not all the possible exif tags. So if you add an exif tag the exiv2 doesn't know, then it wont get populated.

---

What script are you wanting to write that needs the ability to create metadata fields?

---

EDIT: I was wrong about exists, it takes the full tag name. I'll blame it on release brain :-)

[radialmonster]

Thanks, this is genuinely helpful — and no problem at all on the feature freeze. You happened to ship a new release today (thank you for that!), and we happened to start really digging into this today, so the timing is pure coincidence.

For context on what we're building: a publishing plugin so users can publish straight from darktable to photo-sharing sites. That capability was removed from darktable core, and the suggested path forward was to do it via Lua — so that's what we're attempting. A key part is remembering, per image, its published status and a link/ID to the photo on the remote service.

On "you'll want to use it" — I think we're actually the case you said works. We're not reading anything from the image's embedded EXIF; the script generates the value (the remote photo ID) and writes it into the field itself, then reads it back later. So it's the "just adding text fields" path — no re-import or exif-refresh needed. The populate-from-file caveat is real but doesn't touch what we're doing.

Fair to call the create-on-load part a convenience — with one wrinkle: since the editor's add-tag dropdown only offers tags exiv2 knows, a user can't hand-create a field in the script's own namespace (e.g. Xmp.someplugin.id), only reuse existing tags, which risks stepping on real metadata. So for a script that wants its own field, add ends up being the only route rather than a shortcut. We're not blocked though — image tags work fine today; a dedicated field would just be a tidier store.

Good to know the same-final-segment case is an actual segfault, not just shadowing — that makes "exists() first, then create, and reject a subkey whose final segment already exists" essential whenever add does happen. And agreed the API shouldn't expose internal.

No urgency given the freeze — mostly wanted the use case (a script-owned, per-image field holding script-written values) on the radar for whenever add/delete get picked back up. Appreciate the time.

[radialmonster]

Following up to keep this honest, since digging further changed our own thinking a bit.

We went down the rabbit hole on durability — what survives a move/rename/reimport — and concluded that for our case (per-image bookkeeping the script writes), plain tags are actually the better store, not a custom metadata field:

• Tags are always serialized to the .xmp sidecar (as Xmp.dc.subject), so they survive remove-and-reimport / a library rebuild whenever sidecars are on.
• A custom field only reaches the sidecar if visible && !priv (exif.cc), so a hidden bookkeeping field is library.db-only and is lost on reimport — i.e. less portable than tags. Making it survive means making it visible, which then clutters the metadata editor.
• Our data is also naturally multi-valued (a photo can be in several albums on several accounts), which tags handle directly and a single field doesn't.

So we're going to build on tags, and we're not blocked — wanted to say that plainly rather than leave an overstated "tags are awkward" case sitting in the issue.

That said, darktable.metadata.add still seems worthwhile to us for a narrower, real niche: letting a script create a visible, user-meaningful metadata field in a namespace exiv2 doesn't already know (e.g. a reverse-geocoded place name a user wants to see/edit), and to round out the darktable.metadata. namespace you've already started. Just no urgency from our end — appreciate the back-and-forth either way.