Skip to content

Commit 527a106

Browse files
pstjohnpstjohn
authored
Add llama3 recipe readme (#1374)
Adds an initial README for llama3, with the initial lingua-1b benchmark results. --------- Signed-off-by: Peter St. John <pstjohn@nvidia.com>
1 parent c42e7d6 commit 527a106

5 files changed

Lines changed: 305 additions & 3 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,12 +60,14 @@ A core use-case of the BioNeMo Framework is to help digital biology scientists a
6060
| Directory | Description | Support Status | 5D Parallel | Megatron-FSDP | TE | Sequence Packing | FP8 | Context Parallelism |
6161
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------- | ----------- | ------------- | ------ | ---------------- | ------ | ------------------- |
6262
| `models/`<br>`amplify` | TE accelerated protein BERT, pushed to HuggingFace | ✅ Active | ❌ | ✅ | ✅ | 🚧 WIP | ✅ | 🚧 WIP |
63-
| `models/`<br>`esm2` | TE accelerated protein BERT, pushed to HuggingFace | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | 🚧 WIP |
63+
| `models/`<br>`esm2` | TE accelerated protein BERT, pushed to HuggingFace | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ |
64+
| `models/`<br>`llama3` | TE accelerated Llama 3 | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP |
6465
| `models/`<br>`geneformer` | TE accelerated single-cell BERT | 🚧 WIP | ❌ | ✅ | 🚧 WIP | 🚧 WIP | 🚧 WIP | 🚧 WIP |
6566
| `recipes/`<br>`codonfm_ptl_te` | Recipe for [CodonFM](https://research.nvidia.com/labs/dbr/assets/data/manuscripts/nv-codonfm-preprint.pdf)'s Encodon using TE | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP |
6667
| `recipes/`<br>`esm2_accelerate_te` | Recipe for ESM2 TE + HF Accelerate | ✅ Active | ❌ | 🚧 WIP | ✅ | ❌ | ✅ | 🚧 WIP |
6768
| `recipes/`<br>`esm2_native_te` | Recipe for ESM2 TE + native PyTorch | ✅ Active | ❌ | ✅ | ✅ | ✅ | ✅ | 🚧 WIP |
6869
| `recipes/`<br>`geneformer_native_te_mfsdp_fp8` | Recipe for Geneformer HF model | 🚧 WIP | ❌ | ✅ | ✅ | ❌ | ✅ | 🚧 WIP |
70+
| `recipes/`<br>`llama3_native_te` | Recipe for Llama 3 TE + native PyTorch | ✅ Active | ❌ | 🚧 WIP | ✅ | ✅ | 🚧 WIP | 🚧 WIP |
6971
| `recipes/`<br>`vit` | Recipe for Vision Transformer | 🚧 WIP | ❌ | ✅ | ✅ | ❌ | ✅ | 🚧 WIP |
7072

7173
</small>

‎bionemo-recipes/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,9 +30,11 @@ The biological AI community is actively prototyping model architectures and need
3030
| ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | ------------ | ---- | ----------------- | --- | --------- | ------------------- | ------------------- | --- |
3131
| `models/amplify`,<br> [available on Hugging Face](https://huggingface.co/nvidia/AMPLIFY_350M) | TE accelerated protein BERT, [Amgen](https://www.biorxiv.org/content/10.1101/2024.09.23.614603v1) | ✅ | ✅ | ❌ | ❌ | 🚧 | ❌ | ❌ | ❌ |
3232
| `models/esm2`,<br> [available on Hugging Face](https://huggingface.co/nvidia/esm2_t48_15B_UR50D) | TE accelerated protein BERT, [Meta](https://www.biorxiv.org/content/10.1101/2022.07.20.500902v1) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ |
33-
| `recipes/`<br>`codonfm_ptl_te` | Recipe for [CodonFM](https://research.nvidia.com/labs/dbr/assets/data/manuscripts/nv-codonfm-preprint.pdf)'s Encodon using TE | 🚧 | ✅ | 🚧 | ✅ | 🚧 | 🚧 | 🚧 | 🚧 |
34-
| `recipes/`<br>`esm2_accelerate_te` | Recipe for `esm2/amplify` TE + HF Accelerate | 🚧 | ✅ | ✅ | 🚧 | ❌ | 🚧 | 🚧 | 🚧 |
33+
| `models/llama3` | TE accelerated Llama 3, [Meta](https://www.llama.com/models/llama-3/) | ✅ | ✅ | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 |
3534
| `recipes/`<br>`esm2_native_te` | Recipe for `esm2/amplify` + native PyTorch | mFSDP, FSDP2 | ✅ | ✅ | ✅ | ✅ | ✅ | 🚧 | 🚧 |
35+
| `recipes/`<br>`llama3_native_te` | Recipe for `llama3` + native PyTorch | FSDP2 | ✅ | ✅ | ✅ | 🚧 | 🚧 | 🚧 | 🚧 |
36+
| `recipes/`<br>`esm2_accelerate_te` | Recipe for `esm2/amplify` TE + HF Accelerate | 🚧 | ✅ | ✅ | 🚧 | ❌ | 🚧 | 🚧 | 🚧 |
37+
| `recipes/`<br>`codonfm_ptl_te` | Recipe for [CodonFM](https://research.nvidia.com/labs/dbr/assets/data/manuscripts/nv-codonfm-preprint.pdf)'s Encodon using TE | 🚧 | ✅ | 🚧 | ✅ | 🚧 | 🚧 | 🚧 | 🚧 |
3638
| `recipes/`<br>`geneformer_native_te_mfsdp_fp8` | Recipe for geneformer HF model | mFSDP | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 |
3739
| `recipes/`<br>`vit` | Recipe for vision transformer | mFSDP | ✅ | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 | 🚧 |
3840

0 commit comments

Comments
 (0)