Skip to content

Commit da9158b

Browse files
authored
Add shim_et nlohmann-json shim for OSS buck2 build (#20505)
The tokenizers submodule bump moved the fbsource//third-party/nlohmann-json dependency onto the tokenizers :headers target, which is reachable from the OSS buck2 CI build (multimodal_runner_lib -> text_token_generator -> tokenizers:headers). Since fbsource aliases to shim_et, this requires a shim_et//third-party/nlohmann-json package, which did not exist (only re2 had one). Add the shim, mirroring shim_et//third-party/re2, aliasing nlohmann-json to the header-only nlohmann_json target added to the tokenizers submodule (meta-pytorch/tokenizers#201), and bump the submodule pin to include it.
1 parent e03f777 commit da9158b

4 files changed

Lines changed: 15 additions & 2 deletions

File tree

extension/llm/tokenizers

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
load(":targets.bzl", "define_common_targets")
2+
3+
define_common_targets()
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
load("@fbsource//xplat/executorch/build:runtime_wrapper.bzl", "runtime")
2+
3+
def define_common_targets():
4+
# Only for OSS
5+
if runtime.is_oss:
6+
native.cxx_library(
7+
name = "nlohmann-json",
8+
exported_deps = ["root//extension/llm/tokenizers/third-party:nlohmann_json"],
9+
visibility = ["PUBLIC"],
10+
)

shim_et/xplat/executorch/build/env_interface.bzl

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ _EXTERNAL_DEPS = {
4545
"libtorch_python": "//third-party:libtorch_python",
4646
"log": [], # Intentionally not supporting OSS buck build log
4747
# Huggingface Tokenizer
48-
"nlohmann_json": [], # Intentionally not supporting OSS buck build HF tokenizer.
48+
"nlohmann_json": "//extension/llm/tokenizers/third-party:nlohmann_json",
4949
"prettytable": "//third-party:prettytable",
5050
"pybind11": "//third-party:pybind11",
5151
"pcre2": "//extension/llm/tokenizers/third-party:pcre2",

0 commit comments

Comments
 (0)