-
Notifications
You must be signed in to change notification settings - Fork 3
Add enterprise-scale metadata organization guide and reference implementation #8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
717f4d9
Initial plan
Copilot 39c295a
Add enterprise-scale metadata organization example and guide
Copilot df2c3b2
Update documentation navigation and examples README
Copilot ad7601f
Fix deprecated substr method in utils
Copilot 95f7807
Merge branch 'main' into copilot/organize-raw-data-structure
hotlong ad37654
Remove Chinese README and update English README links
hotlong File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,339 @@ | ||
| # Organizing Metadata for Large Projects | ||
|
|
||
| When building enterprise-scale applications with ObjectQL, proper metadata organization becomes critical. This guide demonstrates best practices for structuring your object definitions, actions, hooks, and translations in a maintainable way. | ||
|
|
||
| ## The Challenge | ||
|
|
||
| As your application grows beyond 30-50 objects, a flat file structure becomes problematic: | ||
|
|
||
| **Problems with Flat Structure:** | ||
| ``` | ||
| src/objects/ | ||
| ├── user.object.yml | ||
| ├── organization.object.yml | ||
| ├── account.object.yml | ||
| ├── contact.object.yml | ||
| ├── opportunity.object.yml | ||
| ├── employee.object.yml | ||
| ├── department.object.yml | ||
| ├── invoice.object.yml | ||
| ├── payment.object.yml | ||
| ├── project.object.yml | ||
| ├── task.object.yml | ||
| ... (100+ files) | ||
| ``` | ||
|
|
||
| - ❌ Hard to find related objects | ||
| - ❌ Merge conflicts when multiple teams work simultaneously | ||
| - ❌ Unclear ownership boundaries | ||
| - ❌ Can't deploy modules independently | ||
| - ❌ Difficult to understand relationships | ||
|
|
||
| ## Recommended Structure: Domain-Driven Modules | ||
|
|
||
| For applications with **30+ objects** and **multiple teams**, organize by business domain: | ||
|
|
||
| ``` | ||
| src/ | ||
| ├── core/ # Foundation objects (user, org, etc.) | ||
| │ ├── objects/ | ||
| │ ├── i18n/ | ||
| │ └── index.ts | ||
| │ | ||
| ├── modules/ # Business domain modules | ||
| │ ├── crm/ # Customer management | ||
| │ │ ├── objects/ | ||
| │ │ ├── actions/ | ||
| │ │ ├── hooks/ | ||
| │ │ ├── i18n/ | ||
| │ │ ├── README.md | ||
| │ │ └── index.ts | ||
| │ │ | ||
| │ ├── hr/ # Human resources | ||
| │ ├── finance/ # Finance & accounting | ||
| │ └── project/ # Project management | ||
| │ | ||
| ├── extensions/ # Custom overrides | ||
| ├── shared/ # Shared utilities | ||
| └── index.ts # Application entry | ||
| ``` | ||
|
|
||
| ## Real-World Example | ||
|
|
||
| See the complete working example at: | ||
| ``` | ||
| examples/scenarios/enterprise-structure/ | ||
| ``` | ||
|
|
||
| This demonstrates: | ||
| - ✅ 20+ objects organized across 5 modules | ||
| - ✅ Domain-driven structure (CRM, HR, Finance, Project) | ||
| - ✅ Cross-module relationships | ||
| - ✅ Extension pattern for customization | ||
| - ✅ Comprehensive indexing strategy | ||
| - ✅ Multi-language support (en, zh-CN) | ||
| - ✅ Module documentation | ||
|
|
||
| ## Module Anatomy | ||
|
|
||
| Each module follows a consistent pattern: | ||
|
|
||
| ``` | ||
| modules/[domain]/ | ||
| ├── objects/ # Object definitions | ||
| │ ├── [object1].object.yml | ||
| │ └── [object2].object.yml | ||
| ├── actions/ # Custom actions | ||
| │ └── [object1].action.ts | ||
| ├── hooks/ # Lifecycle hooks | ||
| │ └── [object1].hooks.ts | ||
| ├── i18n/ # Translations | ||
| │ ├── en/ | ||
| │ └── zh-CN/ | ||
| ├── README.md # Module documentation | ||
| └── index.ts # Module exports | ||
| ``` | ||
|
|
||
| ### Module Documentation Template | ||
|
|
||
| Each module should have a README explaining: | ||
|
|
||
| 1. **Overview** - What business domain it covers | ||
| 2. **Objects** - List of objects with descriptions | ||
| 3. **Relationships** - How objects relate to each other | ||
| 4. **Team Ownership** - Who maintains this module | ||
| 5. **Dependencies** - What other modules/objects it depends on | ||
| 6. **Usage Examples** - Common query patterns | ||
|
|
||
| ## Object Naming Conventions | ||
|
|
||
| ### Prefixing Strategy | ||
|
|
||
| For large projects with multiple modules, use prefixes to avoid name collisions: | ||
|
|
||
| ```yaml | ||
| # ✅ Good: Clear module ownership | ||
| name: crm_account | ||
| name: finance_invoice | ||
| name: project_task | ||
|
|
||
| # ❌ Avoid: Risk of collision | ||
| name: account # Which account? CRM or Finance? | ||
| name: task # Project task or general task? | ||
| ``` | ||
|
|
||
| **When to prefix:** | ||
| - ✅ Multi-module applications (30+ objects) | ||
| - ✅ Plugin architectures | ||
| - ✅ When similar concepts exist across domains | ||
|
|
||
| **When NOT to prefix:** | ||
| - ❌ Core shared objects (`user`, `organization`) | ||
| - ❌ Small applications (< 30 objects) | ||
| - ❌ When it reduces clarity | ||
|
|
||
| ## Dependency Management | ||
|
|
||
| ### Dependency Layers | ||
|
|
||
| ``` | ||
| ┌─────────────────────────────────┐ | ||
| │ Application Layer │ | ||
| │ (modules/*) │ | ||
| │ - Can depend on Core │ | ||
| │ - Can depend on other modules │ | ||
| └─────────────────────────────────┘ | ||
| ↓ | ||
| ┌─────────────────────────────────┐ | ||
| │ Foundation Layer │ | ||
| │ (core/*) │ | ||
| │ - No dependencies │ | ||
| │ - Used by everyone │ | ||
| └─────────────────────────────────┘ | ||
| ``` | ||
|
|
||
| ### Cross-Module References | ||
|
|
||
| When modules need to reference each other's objects: | ||
|
|
||
| ```yaml | ||
| # In modules/finance/objects/invoice.object.yml | ||
| fields: | ||
| account: | ||
| type: lookup | ||
| reference_to: crm_account # Reference to CRM module | ||
| ``` | ||
|
|
||
| **Best Practices:** | ||
| 1. Document cross-module dependencies in module README | ||
| 2. Avoid circular dependencies | ||
| 3. Use core objects to break dependency cycles | ||
| 4. Consider extracting shared concepts to core layer | ||
|
|
||
| ## Index Strategy by Module | ||
|
|
||
| Different modules have different performance requirements: | ||
|
|
||
| ### High-Traffic Modules (CRM, Sales) | ||
| ```yaml | ||
| # Aggressive indexing | ||
| fields: | ||
| status: { type: select, index: true } | ||
| owner: { type: lookup, index: true } | ||
| created_at: { type: datetime, index: true } | ||
|
|
||
| indexes: | ||
| owner_status_idx: { fields: [owner, status] } | ||
| status_created_idx: { fields: [status, created_at] } | ||
| ``` | ||
|
|
||
| ### Low-Traffic Modules (Admin, Config) | ||
| ```yaml | ||
| # Minimal indexing | ||
| fields: | ||
| name: { type: text, index: true } | ||
| status: { type: select, index: true } | ||
| # Add more indexes only when needed | ||
| ``` | ||
|
|
||
| ## Extension Pattern | ||
|
|
||
| Use extensions to customize objects without modifying source: | ||
|
|
||
| **Original** (`core/objects/user.object.yml`): | ||
| ```yaml | ||
| name: user | ||
| fields: | ||
| name: { type: text } | ||
| email: { type: text } | ||
| ``` | ||
|
|
||
| **Extension** (`extensions/user.extension.object.yml`): | ||
| ```yaml | ||
| name: user # Same name triggers merge | ||
| fields: | ||
| employee_id: { type: text } | ||
| email: { required: true, unique: true } | ||
| ``` | ||
|
|
||
| **Result:** ObjectQL merges both definitions, adding `employee_id` and making `email` required. | ||
|
|
||
| ## Internationalization at Scale | ||
|
|
||
| ### Three-Layer i18n Strategy | ||
|
|
||
| ``` | ||
| 1. Core Layer (core/i18n/) | ||
| → Shared objects (user, organization) | ||
|
|
||
| 2. Module Layer (modules/[domain]/i18n/) | ||
| → Domain-specific objects | ||
|
|
||
| 3. Extension Layer (extensions/i18n/) | ||
| → Customer/regional customizations | ||
| ``` | ||
|
|
||
| ### Directory Structure | ||
| ``` | ||
| core/i18n/ | ||
| en/core.json | ||
| zh-CN/core.json | ||
|
|
||
| modules/crm/i18n/ | ||
| en/crm.json | ||
| zh-CN/crm.json | ||
| ``` | ||
|
|
||
| ## Migration Path | ||
|
|
||
| ### From Flat to Modular | ||
|
|
||
| **Step 1: Analyze** | ||
| Group existing objects by business domain. | ||
|
|
||
| **Step 2: Plan** | ||
| Create module structure, decide on prefixes. | ||
|
|
||
| **Step 3: Migrate Gradually** | ||
| ``` | ||
| src/ | ||
| ├── objects/ # Legacy (keep temporarily) | ||
| ├── modules/ # New structure | ||
| │ └── crm/ # Start with one module | ||
| └── index.ts # Loads from both | ||
| ``` | ||
|
|
||
| **Step 4: Update References** | ||
| Update imports and references to use new module structure. | ||
|
|
||
| **Step 5: Clean Up** | ||
| Remove legacy flat structure once migration is complete. | ||
|
|
||
| ## Project Size Guidelines | ||
|
|
||
| | Size | Objects | Teams | Recommended Structure | | ||
| |------|---------|-------|----------------------| | ||
| | **Small** | 1-30 | 1 | Flat `objects/` directory | | ||
| | **Medium** | 30-100 | 2-3 | Domain modules | | ||
| | **Large** | 100-500 | 5-10 | Modules + plugins | | ||
| | **Enterprise** | 500+ | 10+ | Monorepo with packages | | ||
|
|
||
| ## Testing Strategy | ||
|
|
||
| ### Module Tests | ||
| ```typescript | ||
| // modules/crm/__tests__/integration.test.ts | ||
| describe('CRM Module', () => { | ||
| it('should handle lead conversion', async () => { | ||
| const lead = await createLead(); | ||
| await convertLead(lead.id); | ||
| // Verify account, contact, opportunity created | ||
| }); | ||
| }); | ||
| ``` | ||
|
|
||
| ### Object Schema Tests | ||
| ```typescript | ||
| // modules/crm/objects/__tests__/account.test.ts | ||
| describe('Account Object', () => { | ||
| it('should have required fields', () => { | ||
| const schema = loadObjectSchema('crm_account'); | ||
| expect(schema.fields.name.required).toBe(true); | ||
| }); | ||
| }); | ||
| ``` | ||
|
|
||
| ## Complete Working Example | ||
|
|
||
| Explore the full example with 20+ objects: | ||
| ```bash | ||
| cd examples/scenarios/enterprise-structure | ||
| pnpm install | ||
| pnpm build | ||
| ``` | ||
|
|
||
| The example includes: | ||
| - Core module (user, organization, attachment) | ||
| - CRM module (account, contact, opportunity, lead) | ||
| - HR module (employee, department, position, timesheet) | ||
| - Finance module (invoice, payment, expense, budget) | ||
| - Project module (project, task, milestone, timesheet entry) | ||
| - Extensions pattern | ||
| - Multi-language support | ||
| - Comprehensive documentation | ||
|
|
||
| ## Key Takeaways | ||
|
|
||
| 1. **Start simple** - Don't over-engineer for small projects | ||
| 2. **Think in domains** - Organize by business capability, not technical layers | ||
| 3. **Document boundaries** - Make module ownership and dependencies explicit | ||
| 4. **Plan for scale** - Use prefixes and modules when you hit 30-50 objects | ||
| 5. **Test modules** - Each module should be testable independently | ||
| 6. **Version control** - Use module-level versioning for independent deployments | ||
|
|
||
| ## See Also | ||
|
|
||
| - [Data Modeling Guide](./data-modeling.md) | ||
| - [Plugin Development](./plugins.md) | ||
| - [Logic Hooks](./logic-hooks.md) | ||
| - [Complete Example](../../examples/scenarios/enterprise-structure/) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| # @example/enterprise-structure | ||
|
|
||
| ## 1.0.0 | ||
|
|
||
| ### Added | ||
|
|
||
| - Initial release with enterprise-scale metadata organization example | ||
| - Core module with user, organization, and attachment objects | ||
| - CRM module with account, contact, opportunity, and lead objects | ||
| - HR module with employee, department, position, and timesheet objects | ||
| - Finance module with invoice, payment, expense, and budget objects | ||
| - Project module with project, task, milestone, and timesheet entry objects | ||
| - Extension pattern demonstration with user extensions | ||
| - Comprehensive documentation for each module | ||
| - Shared utilities, constants, and validators | ||
| - Module index files for clean exports | ||
| - README with best practices and architecture guidance |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The guidance on when to use modular structure is inconsistent. The table shows "Small" projects handle 10-30 objects with flat structure and "Medium" starts at 30-100 with modules. However, the text says "Use modules when you hit 30-50 objects". This creates ambiguity about whether 30 objects should use flat or modular structure. Consider clarifying that 30 objects is the transition point.