Skip to content

Commit 731871f

Browse files
jackye1995jackye1995
authored
docs: introduce lance as a lakehouse format (#5209)
Also refreshes all the contents regarding table format, separate it into multiple documents for clarity. Closes #4136 #5133
1 parent 551ce82 commit 731871f

20 files changed

Lines changed: 2194 additions & 275 deletions

‎.typos.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ afe = "afe"
1111
typ = "typ"
1212
rabit = "rabit"
1313
flate = "flate"
14+
Ines = "Ines"
1415

1516
[default.expect]
1617
nprobs = "nprobes"

‎docs/mkdocs.yml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
site_name: Lance
2-
site_description: Modern columnar data format for ML and LLMs
2+
site_description: Open Lakehouse Format for Multimodal AI
33
site_url: https://lancedb.github.io/lance/
44
docs_dir: src
55

@@ -8,6 +8,7 @@ repo_url: https://github.com/lancedb/lance
88

99
theme:
1010
name: material
11+
custom_dir: overrides
1112
logo: logo/white.png
1213
favicon: logo/logo.png
1314
palette:
@@ -40,7 +41,11 @@ theme:
4041
markdown_extensions:
4142
- admonition
4243
- pymdownx.details
43-
- pymdownx.superfences
44+
- pymdownx.superfences:
45+
custom_fences:
46+
- name: mermaid
47+
class: mermaid
48+
format: !!python/name:pymdownx.superfences.fence_code_format
4449
- pymdownx.highlight:
4550
anchor_linenums: true
4651
line_spans: __span
@@ -70,3 +75,12 @@ extra:
7075
- icon: fontawesome/brands/twitter
7176
link: https://twitter.com/lancedb
7277

78+
copyright: © 2025 Lance Format. All rights reserved.
79+
80+
extra_css:
81+
- assets/stylesheets/termynal.css
82+
- assets/stylesheets/home.css
83+
84+
extra_javascript:
85+
- assets/javascript/termynal.js
86+

‎docs/overrides/home.html‎

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
{% extends "main.html" %}
2+
3+
{% block tabs %}
4+
{{ super() }}
5+
6+
<style>
7+
/* Prevent horizontal overflow */
8+
body {
9+
overflow-x: hidden;
10+
}
11+
12+
/* Hide main content for home page */
13+
.md-content {
14+
display: none;
15+
}
16+
17+
/* Hide table of contents */
18+
@media screen and (min-width: 60em) {
19+
.md-sidebar--secondary {
20+
display: none;
21+
}
22+
}
23+
24+
/* Hide navigation */
25+
@media screen and (min-width: 76.25em) {
26+
.md-sidebar--primary {
27+
display: none;
28+
}
29+
}
30+
31+
/* Make header static */
32+
.md-header {
33+
position: initial;
34+
}
35+
36+
.md-main__inner {
37+
margin: 0;
38+
}
39+
</style>
40+
41+
<!-- Hero Section -->
42+
<section class="mdx-container">
43+
<div class="container">
44+
<div class="intro-message">
45+
<div class="hero-logo">
46+
<img src="logo/white.png" alt="Lance Logo">
47+
<h1>Lance<sup>™</sup></h1>
48+
</div>
49+
<h3>The Open Lakehouse Format for Multimodal AI</h3>
50+
<hr class="intro-divider" />
51+
<ul class="list-inline">
52+
<li>
53+
<a href="quickstart" class="md-button md-button--primary">Get Started</a>
54+
</li>
55+
<li>
56+
<a href="format" class="md-button">Read the Spec</a>
57+
</li>
58+
<li>
59+
<a href="examples/python/llm_training" class="md-button">Train an LLM</a>
60+
</li>
61+
<li>
62+
<a href="https://discord.gg/msY9kdwSYw" class="md-button" target="_blank" rel="noopener">Join Discord</a>
63+
</li>
64+
</ul>
65+
</div>
66+
</div>
67+
</section>
68+
69+
<!-- What is Lance Section -->
70+
<section class="lance-intro-section">
71+
<div class="container">
72+
<div class="lance-intro-content">
73+
<h2>What is Lance<sup>™</sup>?</h2>
74+
<p>
75+
Lance contains a file format, table format, and catalog spec for multimodal AI,
76+
allowing you to build a complete open lakehouse on top of object storage to power your AI workflows.
77+
Lance brings high-performance vector search, full-text search, random access, and feature
78+
engineering capabilities to the lakehouse, while you can still get all the existing lakehouse benefits
79+
like SQL analytics, ACID transactions, time travel, and integrations with open engines (Apache Spark, Ray, Trino, DuckDB, etc.)
80+
and open catalogs (Apache Polaris, Unity Catalog, Apache Gravitino, Hive Metastore, etc.)
81+
</p>
82+
<a href="quickstart" class="md-button md-button--primary">Learn More</a>
83+
</div>
84+
</div>
85+
</section>
86+
87+
<!-- Feature 1: Expressive Hybrid Search -->
88+
<section class="lance-feature-section">
89+
<div class="container">
90+
<div class="lance-feature-content">
91+
<div class="lance-feature-text">
92+
<h2>Expressive Hybrid Search</h2>
93+
<p>
94+
Lance enables powerful hybrid search combining vector similarity, full-text search,
95+
and SQL analytics on the same dataset. All query types are accelerated by corresponding
96+
secondary indices as part of the Lance specification.
97+
</p>
98+
<p>
99+
Run semantic search on embeddings, BM25 search on keywords, and apply complex SQL predicates -
100+
all using a single table with a unified interface.
101+
</p>
102+
<a href="quickstart/vector-search" class="md-button">Learn More</a>
103+
</div>
104+
<div class="lance-feature-demo">
105+
<div id="termynal-hybrid-search"
106+
data-termynal=""
107+
data-ty-startdelay="500"
108+
data-ty-typedelay="40"
109+
data-ty-linedelay="700"
110+
style="width: 500px;">
111+
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
112+
<span data-ty="input" data-ty-prompt=">>>">ds = lance.dataset("s3://my-bucket/docs")</span>
113+
<span data-ty="input" data-ty-prompt=">>>">ds.to_table(full_text_query="machine learning")</span>
114+
<span data-ty="input" data-ty-prompt=">>>">ds.to_table(</span>
115+
<span data-ty="input" data-ty-prompt="..."> nearest={"column": "embedding", "q": query_vec, "k": 10},</span>
116+
<span data-ty="input" data-ty-prompt="..."> filter="year > 2020")</span>
117+
</div>
118+
</div>
119+
</div>
120+
</div>
121+
</section>
122+
123+
<!-- Feature 2: Lightning-fast Random Access -->
124+
<section class="lance-feature-section reverse">
125+
<div class="container">
126+
<div class="lance-feature-content">
127+
<div class="lance-feature-text">
128+
<h2>Lightning-fast Random Access</h2>
129+
<p>
130+
Lance delivers 100x faster random access compared to Parquet or Iceberg. With efficient
131+
row-addressing, you can access individual records across multiple files instantly,
132+
making it perfect for real-time ML serving, random sampling, and interactive applications.
133+
</p>
134+
<p>
135+
Unlike traditional columnar formats, Lance maintains high performance even when
136+
randomly accessing scattered rows across your entire dataset.
137+
</p>
138+
<a href="guide/read_and_write#random-access" class="md-button">Learn More</a>
139+
</div>
140+
<div class="lance-feature-demo">
141+
<div id="termynal-random-access"
142+
data-termynal=""
143+
data-ty-startdelay="1500"
144+
data-ty-typedelay="40"
145+
data-ty-linedelay="700"
146+
style="width: 500px;">
147+
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
148+
<span data-ty="input" data-ty-prompt=">>>">dataset = lance.dataset("s3://my-bucket/embeddings")</span>
149+
<span data-ty="input" data-ty-prompt=">>>">table = dataset.take([100, 5000, 1000000])</span>
150+
<span data-ty="input" data-ty-prompt=">>>">dataset.take([0, 1], columns=["id", "vector"])</span>
151+
</div>
152+
</div>
153+
</div>
154+
</div>
155+
</section>
156+
157+
<!-- Feature 3: Native Multimodal Data Support -->
158+
<section class="lance-feature-section">
159+
<div class="container">
160+
<div class="lance-feature-content">
161+
<div class="lance-feature-text">
162+
<h2>Native Multimodal Data Support</h2>
163+
<p>
164+
Store images, videos, audio, text, and embeddings in a single unified format.
165+
Lance's blob encoding efficiently handles large binary objects with lazy loading,
166+
while optimized vector storage accelerates similarity search.
167+
</p>
168+
<p>
169+
Perfect for AI/ML workloads where you need to store raw data alongside embeddings
170+
for multimodal retrieval and generation workflows.
171+
</p>
172+
<a href="guide/blob" class="md-button">Learn More</a>
173+
</div>
174+
<div class="lance-feature-demo">
175+
<div id="termynal-multimodal"
176+
data-termynal=""
177+
data-ty-startdelay="2500"
178+
data-ty-typedelay="40"
179+
data-ty-linedelay="700"
180+
style="width: 500px;">
181+
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
182+
<span data-ty="input" data-ty-prompt=">>>">import pyarrow as pa</span>
183+
<span data-ty="input" data-ty-prompt=">>>">schema = pa.schema([</span>
184+
<span data-ty="input" data-ty-prompt="..."> pa.field("video", pa.large_binary(),</span>
185+
<span data-ty="input" data-ty-prompt="..."> metadata={"lance-encoding:blob": "true"}),</span>
186+
<span data-ty="input" data-ty-prompt="..."> pa.field("embedding", pa.list_(pa.float32(), 128))])</span>
187+
<span data-ty="input" data-ty-prompt=">>>">lance.write_dataset(table, "multimodal.lance", schema=schema)</span>
188+
</div>
189+
</div>
190+
</div>
191+
</div>
192+
</section>
193+
194+
<!-- Feature 4: Data Evolution -->
195+
<section class="lance-feature-section reverse">
196+
<div class="container">
197+
<div class="lance-feature-content">
198+
<div class="lance-feature-text">
199+
<h2>Data Evolution > Schema Evolution</h2>
200+
<p>
201+
Schema evolution in most open table formats are metadata only and fast.
202+
But when trying to backfill column values in existing rows, a full table rewrite is typically required.
203+
Lance supports efficient schema evolution with backfill, making it perfect for ML
204+
feature engineering, embedding and media content management.
205+
</p>
206+
<p>
207+
Adding a new column with data is as simple as writing new Lance files to the Lance table -
208+
no need to rewrite your entire dataset.
209+
</p>
210+
<a href="guide/data_evolution" class="md-button">Learn More</a>
211+
</div>
212+
<div class="lance-feature-demo">
213+
<div id="termynal-evolution"
214+
data-termynal=""
215+
data-ty-startdelay="3500"
216+
data-ty-typedelay="40"
217+
data-ty-linedelay="700"
218+
style="width: 500px;">
219+
<span data-ty="input" data-ty-prompt=">>>">import lance</span>
220+
<span data-ty="input" data-ty-prompt=">>>">dataset = lance.dataset("my_data.lance")</span>
221+
<span data-ty="input" data-ty-prompt=">>>">@lance.batch_udf()</span>
222+
<span data-ty="input" data-ty-prompt="...">def add_embeddings(batch):</span>
223+
<span data-ty="input" data-ty-prompt="..."> vectors = model.encode(batch["text"])</span>
224+
<span data-ty="input" data-ty-prompt="..."> return {"embedding": vectors}</span>
225+
<span data-ty="input" data-ty-prompt=">>>">dataset.add_columns(add_embeddings)</span>
226+
</div>
227+
</div>
228+
</div>
229+
</div>
230+
</section>
231+
232+
<!-- Feature 5: Rich Ecosystem Integration -->
233+
<section class="lance-feature-section">
234+
<div class="container">
235+
<div class="lance-feature-content">
236+
<div class="lance-feature-text">
237+
<h2>Rich Ecosystem Integrations</h2>
238+
<p>
239+
As an open format, Lance integrates seamlessly with the Python data ecosystem and modern data platforms.
240+
Work with your favorite tools including Pandas, Polars, and PyTorch for data processing and machine learning.
241+
Connect with leading query engines like Apache DataFusion, DuckDB, Apache Spark, Trino, and Apache Flink/Fluss
242+
to run SQL analytics and distributed processing on your Lance datasets.
243+
</p>
244+
<a href="integrations/duckdb" class="md-button">View Integrations</a>
245+
</div>
246+
<div class="lance-feature-demo">
247+
<img src="assets/images/ecosystem-integrations.png" alt="Lance Ecosystem Integrations" style="max-width: 500px; width: 100%; height: auto; border-radius: 8px;">
248+
</div>
249+
</div>
250+
</div>
251+
</section>
252+
253+
<script
254+
src="assets/javascript/termynal.js"
255+
data-termynal-container="#termynal-hybrid-search|#termynal-random-access|#termynal-multimodal|#termynal-evolution">
256+
</script>
257+
258+
{% endblock %}
259+
260+
{% block content %}{% endblock %}
261+
{% block footer %}
262+
{{ super() }}
263+
{% endblock %}

‎docs/src/assets/images/ecosystem-integrations.png‎

146 KB
Loading

‎docs/src/assets/images/lance-mj.png‎

1.06 MB
Loading

0 commit comments

Comments
 (0)