Skip to content

Commit 4ac7ed8

Browse files
author
MPCoreDeveloper
committed
docs: Add cross-reference guide for design vs implementation documentation
1 parent d15c409 commit 4ac7ed8

File tree

1 file changed

+314
-0
lines changed

1 file changed

+314
-0
lines changed

‎docs/DOCUMENTATION_CROSS_REFERENCE.md‎

Lines changed: 314 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
# Documentation Cross-Reference Guide
2+
3+
## 📚 SharpCoreDB Documentation Ecosystem
4+
5+
### Two Documentation Tracks
6+
7+
SharpCoreDB has **two complementary documentation systems**:
8+
9+
#### 🏗️ Design Track: `docs/scdb/FILE_FORMAT_DESIGN.md`
10+
**Purpose:** Architectural design & specifications
11+
**Audience:** Architects, designers, future developers
12+
**Content:**
13+
- Overall design principles
14+
- Format specifications (struct definitions)
15+
- Comparison with SQLite, LiteDB
16+
- Performance optimization strategies
17+
- Future extension points
18+
19+
**Key sections:**
20+
- Executive Summary
21+
- File Structure Overview
22+
- Detailed Format Specification (Header, Registry, FSM, WAL, Table Directory)
23+
- Performance Optimizations
24+
25+
#### 🔧 Implementation Track: `docs/serialization/`
26+
**Purpose:** Practical guides for implementing/using serialization
27+
**Audience:** Developers using SharpCoreDB, implementers
28+
**Content:**
29+
- Real examples with actual hex dumps
30+
- Step-by-step serialization walkthroughs
31+
- Variable-length string handling (with evidence)
32+
- Free space management (practical examples)
33+
- Block registry lookups (O(1) explanation)
34+
- FAQ with solutions to real problems
35+
36+
**Structure:**
37+
```
38+
docs/serialization/
39+
├── README.md (navigation hub)
40+
├── SERIALIZATION_AND_STORAGE_GUIDE.md (3,200 lines)
41+
├── SERIALIZATION_FAQ.md (800 lines)
42+
├── BINARY_FORMAT_VISUAL_REFERENCE.md (900 lines)
43+
└── scripts/visualize_serialization.py (interactive tool)
44+
```
45+
46+
---
47+
48+
## 🗺️ Content Mapping
49+
50+
### Topic: File Header Structure
51+
52+
**In FILE_FORMAT_DESIGN.md:**
53+
```markdown
54+
### 1. File Header (512 bytes, fixed)
55+
56+
[StructLayout(LayoutKind.Sequential, Pack = 1)]
57+
public readonly struct ScdbFileHeader
58+
{
59+
public readonly ulong Magic; // 0x0000: "SCDB" + version
60+
public readonly ushort FormatVersion; // 0x0008: Format version (1)
61+
public readonly ushort PageSize; // 0x000A: Page size in bytes
62+
// ... (C# struct definitions)
63+
}
64+
```
65+
66+
**Purpose:** Formal specification
67+
**Detail Level:** Struct definitions, sizes, purpose of each field
68+
69+
**In BINARY_FORMAT_VISUAL_REFERENCE.md:**
70+
```markdown
71+
## 2. File Header Structure (512 bytes)
72+
73+
SCDB File Header Layout:
74+
75+
Offset Size Field Name Value
76+
0x0000 8 Magic 0x4243445310000000
77+
0x0008 2 FormatVersion 1
78+
0x000A 2 PageSize 4096 (default)
79+
```
80+
81+
**Purpose:** Visual reference
82+
**Detail Level:** Hex offsets, byte sizes, visual tables
83+
84+
---
85+
86+
### Topic: Block Registry
87+
88+
**In FILE_FORMAT_DESIGN.md:**
89+
```markdown
90+
### 2. Block Registry (Variable, page-aligned)
91+
92+
[StructLayout(LayoutKind.Sequential, Pack = 1)]
93+
public readonly struct BlockEntry
94+
{
95+
public readonly fixed byte Name[32];
96+
public readonly uint BlockType;
97+
public readonly ulong Offset;
98+
public readonly ulong Length;
99+
public readonly fixed byte Checksum[32];
100+
}
101+
102+
// Block Naming Convention:
103+
// Format: "namespace:identifier[:subtype]"
104+
// Examples:
105+
// - table:app_users:data
106+
// - table:app_users:index:pk_users
107+
```
108+
109+
**Purpose:** Specification & architecture
110+
**Detail Level:** Struct layout, naming conventions, design decisions
111+
112+
**In SERIALIZATION_AND_STORAGE_GUIDE.md:**
113+
```markdown
114+
## Block Registry
115+
116+
### Purpose
117+
118+
The **Block Registry** maps logical block names to physical file locations:
119+
120+
[code example + lookup flow + performance analysis]
121+
122+
### O(1) Lookups
123+
124+
ConcurrentDictionary = O(1) average lookup
125+
Block names stay in hash table
126+
```
127+
128+
**Purpose:** Implementation guide & explanation
129+
**Detail Level:** How it works in practice, performance implications
130+
131+
---
132+
133+
### Topic: Free Space Management
134+
135+
**In FILE_FORMAT_DESIGN.md:**
136+
```markdown
137+
### 3. Free Space Map (FSM)
138+
139+
Design: Inspired by PostgreSQL's FSM, uses a **two-level bitmap**:
140+
1. **L1 Bitmap:** 1 bit per page (allocated/free)
141+
2. **L2 Extent Map:** Tracks contiguous free extents
142+
143+
[C# struct definitions]
144+
145+
Allocation Strategy:
146+
1. Small allocations (<64 pages): Scan L1 bitmap
147+
2. Large allocations (≥64 pages): Use L2 extent map
148+
3. Defragmentation: Background VACUUM
149+
```
150+
151+
**Purpose:** Architectural design
152+
**Detail Level:** Design rationale, algorithm overview
153+
154+
**In SERIALIZATION_AND_STORAGE_GUIDE.md:**
155+
```markdown
156+
## Free Space Management
157+
158+
### How FSM Works
159+
160+
The **Free Space Map (FSM)** behaves vrije pagina's. Dit is een 2-level bitmap:
161+
162+
[Detailed explanation with code examples]
163+
164+
### File Growth Strategy
165+
166+
Exponential growth (10MB → 20MB → 40MB...)
167+
Phase 3 optimized: MIN_EXTENSION_PAGES = 2560
168+
169+
[Real-world numbers and examples]
170+
```
171+
172+
**Purpose:** Practical guide
173+
**Detail Level:** How to use it, examples, optimization tips
174+
175+
---
176+
177+
## ✅ When To Use Which
178+
179+
### Use FILE_FORMAT_DESIGN.md When...
180+
181+
- ❓ *"What is the binary structure of the header?"*
182+
- ❓ *"What are the struct layouts?"*
183+
- ❓ *"How does FSM compare to PostgreSQL?"*
184+
- ❓ *"What are the design principles?"*
185+
- ❓ *"What's the future extension strategy?"*
186+
187+
**Answer:** Go to `docs/scdb/FILE_FORMAT_DESIGN.md`
188+
189+
---
190+
191+
### Use docs/serialization/ When...
192+
193+
- ❓ *"How do I serialize a record?"*
194+
- ❓ *"What are the actual hex bytes for a string?"*
195+
- ❓ *"Do variable-length strings cause fragmentation?"*
196+
- ❓ *"How do I find a record?"*
197+
- ❓ *"What's the performance impact of strings?"*
198+
- ❓ *"How do column boundaries work?"*
199+
200+
**Answer:** Go to `docs/serialization/README.md` → pick the right doc
201+
202+
---
203+
204+
## 📊 Complementary Coverage
205+
206+
### FILE_FORMAT_DESIGN.md Covers:
207+
✅ Overall architecture
208+
✅ Struct definitions & layouts
209+
✅ Design decisions & rationale
210+
✅ Comparison with competitors
211+
✅ Future extension points
212+
✅ Performance optimization strategies
213+
❌ Actual hex dump examples
214+
❌ Variable-length string handling
215+
❌ Real-world fragmentation examples
216+
❌ O(1) lookup explanations
217+
❌ FAQ with solutions
218+
219+
### docs/serialization/ Covers:
220+
✅ Real hex dump examples
221+
✅ Variable-length strings (with evidence!)
222+
✅ Actual fragmentation examples
223+
✅ O(1) lookup walkthroughs
224+
✅ FAQ with solutions
225+
✅ Visual diagrams
226+
✅ Interactive Python tool
227+
✅ Performance comparisons
228+
❌ Overall architecture (that's in FILE_FORMAT_DESIGN)
229+
❌ Struct definitions (that's in FILE_FORMAT_DESIGN)
230+
❌ Design rationale (that's in FILE_FORMAT_DESIGN)
231+
232+
---
233+
234+
## 🔗 Cross-References
235+
236+
### From FILE_FORMAT_DESIGN.md to serialization docs
237+
238+
**Location:** Add to relevant sections
239+
240+
```markdown
241+
> **For practical examples and real-world usage:**
242+
> See `docs/serialization/SERIALIZATION_AND_STORAGE_GUIDE.md`
243+
```
244+
245+
### From serialization docs to FILE_FORMAT_DESIGN.md
246+
247+
**Location:** Add to relevant sections
248+
249+
```markdown
250+
> **For architectural design and struct definitions:**
251+
> See `docs/scdb/FILE_FORMAT_DESIGN.md`
252+
```
253+
254+
---
255+
256+
## 📋 Example: The Question "Do I need lots of free space?"
257+
258+
### Path 1: Designer's perspective
259+
1. **Question:** "Is FSM efficient? Any design flaws?"
260+
2. **Go to:** `docs/scdb/FILE_FORMAT_DESIGN.md` § "Free Space Map (FSM)"
261+
3. **Learn:** Two-level bitmap design, allocation strategy
262+
4. **Compare:** SQLite, LiteDB approaches
263+
264+
### Path 2: Developer's perspective
265+
1. **Question:** "Do variable-length strings waste space?"
266+
2. **Go to:** `docs/serialization/README.md`
267+
3. **Click:** "Do I need free space?" link
268+
4. **Learn:** Real examples showing 96.9% space savings!
269+
5. **Verify:** With visualize_serialization.py tool
270+
271+
---
272+
273+
## 🎯 Summary
274+
275+
| Dimension | FILE_FORMAT_DESIGN | serialization/ |
276+
|-----------|-------------------|----------------|
277+
| **Audience** | Architects | Developers |
278+
| **Purpose** | Design spec | Implementation guide |
279+
| **Detail** | Struct layout | Real examples |
280+
| **Format** | Formal | Practical |
281+
| **Code** | Struct definitions | Serialization code |
282+
| **Examples** | Design comparisons | Hex dumps, real data |
283+
| **Use case** | Understanding design | Solving problems |
284+
285+
---
286+
287+
## 🚀 Recommendation: Cross-Link Both
288+
289+
Since both are now complete and complementary, consider:
290+
291+
1. ✅ **Keep both separate** - They serve different purposes
292+
2. ✅ **Add cross-references** between them:
293+
- FILE_FORMAT_DESIGN.md → "See serialization/ for practical examples"
294+
- serialization/ → "See FILE_FORMAT_DESIGN.md for architecture"
295+
3. ✅ **Update main README.md** to mention both:
296+
- "Design documentation: `docs/scdb/`"
297+
- "Implementation guides: `docs/serialization/`"
298+
4. ✅ **Add to root docs/README.md**:
299+
```markdown
300+
## Documentation Structure
301+
302+
- **Design & Specifications:** `docs/scdb/FILE_FORMAT_DESIGN.md`
303+
- Architectural overview, struct definitions, design principles
304+
305+
- **Implementation Guides:** `docs/serialization/`
306+
- Practical tutorials, real examples, FAQ
307+
```
308+
309+
---
310+
311+
**Status:** ✅ Both documentation tracks complete and complementary
312+
**Cross-referencing:** Ready to implement
313+
**Organization:** Professional & maintainable
314+

0 commit comments

Comments
 (0)