Skip to content

Commit 0b53db6

Browse files
authored
🔧 Support multiple --mappings-file flags with deep merge capability (#67)
## 🧩 Feature Implementation This PR implements support for multiple `--mappings-file` flags as requested in issue #66. ## ✅ What Changed ### Core Functionality - **Modified argument parser** to accept multiple `--mappings-file` flags using `action='append'` - **Added deep merge capability** with `_deep_merge_dicts()` method for intelligent merging - **Implemented order-preserving merge** where later files override earlier ones for conflicting keys - **Deep merging for nested dictionaries** to preserve hierarchical structure ### Usage Examples **Basic multiple mappings:** ```bash struct generate \ --mappings-file ./common-mappings.yaml \ --mappings-file ./env-specific-mappings.yaml \ file://my-struct.yaml . ``` **Environment-based automation:** ```bash struct generate \ --mappings-file ./mappings/common.yaml \ --mappings-file ./mappings/${ENVIRONMENT}.yaml \ file://infrastructure.yaml \ ./output ``` ### Merging Behavior **Input files:** `common.yaml`: ```yaml mappings: teams: devops: devops-team environments: dev: database_url: postgres://dev-db:5432 ``` `production.yaml`: ```yaml mappings: teams: frontend: frontend-team environments: dev: debug: true prod: database_url: postgres://prod-db:5432 ``` **Merged result:** ```yaml mappings: teams: devops: devops-team # from common.yaml frontend: frontend-team # from production.yaml environments: dev: database_url: postgres://dev-db:5432 # from common.yaml debug: true # from production.yaml prod: database_url: postgres://prod-db:5432 # from production.yaml ``` ## 🛠️ Technical Implementation ### Deep Merge Algorithm - Recursively merges nested dictionaries - Preserves all keys from both dictionaries - Later values override earlier ones for conflicts - Maintains data type integrity ### Error Handling - Validates each mappings file exists before processing - Stops execution if any mappings file is missing or invalid - Provides clear error messages for debugging ### Backward Compatibility - Single `--mappings-file` usage remains unchanged - No breaking changes to existing functionality - Graceful handling when no mappings files are provided ## 📚 Documentation Updates - **Enhanced mappings.md** with detailed multiple files documentation - **Updated usage.md** to reflect new capability - **Added practical examples** showing environment-specific patterns - **Explained merging behavior** with clear examples ## 🧪 Testing Added comprehensive tests covering: - Deep merge functionality with nested dictionaries - Multiple mappings file loading and merging - Order preservation and override behavior - Integration with existing generate command flow ## 🚀 Benefits ### For CI/CD Automation ```bash # Clean separation of common vs environment-specific config struct generate \ --mappings-file ./mappings/common.yaml \ --mappings-file ./mappings/${ENVIRONMENT}.yaml \ file://infrastructure.yaml \ ./output ``` ### For DRY Principles - **Reusable common configurations** across environments - **Environment-specific overrides** without duplication - **Modular mapping organization** for better maintainability ### For Complex Projects - **Team-specific configurations** with shared defaults - **Multi-environment deployments** with inheritance - **Progressive configuration builds** from base to specific ## 🔗 Resolves Closes #66 --- This implementation provides the exact functionality requested in the issue while maintaining backward compatibility and adding comprehensive documentation and testing.
1 parent 5b7cfc8 commit 0b53db6

4 files changed

Lines changed: 166 additions & 13 deletions

File tree

docs/mappings.md

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ struct generate --mappings-file ./mymap.yaml file://my-struct.yaml .
151151

152152
### Multiple Mappings Files
153153

154-
You can specify multiple mappings files that will be merged:
154+
You can specify multiple mappings files that will be merged in order:
155155

156156
```sh
157157
struct generate \
@@ -160,7 +160,22 @@ struct generate \
160160
file://my-struct.yaml .
161161
```
162162

163-
Later files override earlier ones for conflicting keys.
163+
**Merging behavior:**
164+
165+
- Files are processed in the order specified
166+
- Later files override earlier ones for conflicting keys
167+
- Deep merging is performed for nested dictionaries
168+
- This enables clean separation of common vs environment-specific configuration
169+
170+
**Example with environment variable:**
171+
172+
```sh
173+
struct generate \
174+
--mappings-file ./mappings/common.yaml \
175+
--mappings-file ./mappings/${ENVIRONMENT}.yaml \
176+
file://infrastructure.yaml \
177+
./output
178+
```
164179

165180
## Practical Examples
166181

docs/usage.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ struct generate \
4444
- `--backup`: Specify backup directory for existing files
4545
- `--file-strategy`: Choose how to handle existing files (overwrite, skip, append, rename, backup)
4646
- `--log-file`: Write logs to specified file
47-
- `--mappings-file`: Provide external mappings file
47+
- `--mappings-file`: Provide external mappings file (can be used multiple times)
4848

4949
## Generate Schema Command
5050

struct_module/commands/generate.py

Lines changed: 28 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,24 @@ def __init__(self, parser):
2222
parser.add_argument('-f', '--file-strategy', type=str, choices=['overwrite', 'skip', 'append', 'rename', 'backup'], default='overwrite', help='Strategy for handling existing files').completer = file_strategy_completer
2323
parser.add_argument('-p', '--global-system-prompt', type=str, help='Global system prompt for OpenAI')
2424
parser.add_argument('--non-interactive', action='store_true', help='Run the command in non-interactive mode')
25-
parser.add_argument('--mappings-file', type=str,
26-
help='Path to a YAML file containing mappings to be used in templates')
25+
parser.add_argument('--mappings-file', type=str, action='append',
26+
help='Path to a YAML file containing mappings to be used in templates (can be specified multiple times)')
2727
parser.add_argument('-o', '--output', type=str,
2828
choices=['console', 'file'], default='file', help='Output mode')
2929
parser.set_defaults(func=self.execute)
3030

31+
def _deep_merge_dicts(self, dict1, dict2):
32+
"""
33+
Deep merge two dictionaries, with dict2 values overriding dict1 values.
34+
"""
35+
result = dict1.copy()
36+
for key, value in dict2.items():
37+
if key in result and isinstance(result[key], dict) and isinstance(value, dict):
38+
result[key] = self._deep_merge_dicts(result[key], value)
39+
else:
40+
result[key] = value
41+
return result
42+
3143
def _run_hooks(self, hooks, hook_type="pre"): # helper for running hooks
3244
if not hooks:
3345
return True
@@ -75,14 +87,20 @@ def execute(self, args):
7587
# Load mappings if provided
7688
mappings = {}
7789
if getattr(args, 'mappings_file', None):
78-
if os.path.exists(args.mappings_file):
79-
with open(args.mappings_file, 'r') as mf:
80-
try:
81-
mappings = yaml.safe_load(mf) or {}
82-
except Exception as e:
83-
self.logger.error(f"Failed to load mappings file: {e}")
84-
else:
85-
self.logger.error(f"Mappings file not found: {args.mappings_file}")
90+
for mappings_file_path in args.mappings_file:
91+
if os.path.exists(mappings_file_path):
92+
self.logger.info(f"Loading mappings from: {mappings_file_path}")
93+
with open(mappings_file_path, 'r') as mf:
94+
try:
95+
file_mappings = yaml.safe_load(mf) or {}
96+
# Deep merge the mappings, with later files overriding earlier ones
97+
mappings = self._deep_merge_dicts(mappings, file_mappings)
98+
except Exception as e:
99+
self.logger.error(f"Failed to load mappings file {mappings_file_path}: {e}")
100+
return
101+
else:
102+
self.logger.error(f"Mappings file not found: {mappings_file_path}")
103+
return
86104

87105
if args.backup and not os.path.exists(args.backup):
88106
os.makedirs(args.backup)

tests/test_commands.py

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -400,3 +400,123 @@ def mock_relpath_side_effect(file_path, base_path):
400400
structures = schema['definitions']['PluginList']['enum']
401401
assert 'builtin' in structures
402402
assert len(structures) == 1
403+
404+
def test_deep_merge_dicts():
405+
"""Test the _deep_merge_dicts functionality"""
406+
parser = argparse.ArgumentParser()
407+
command = GenerateCommand(parser)
408+
409+
dict1 = {
410+
'a': 1,
411+
'b': {
412+
'c': 2,
413+
'd': 3
414+
},
415+
'e': 4
416+
}
417+
418+
dict2 = {
419+
'b': {
420+
'd': 30,
421+
'f': 40
422+
},
423+
'g': 50
424+
}
425+
426+
result = command._deep_merge_dicts(dict1, dict2)
427+
428+
expected = {
429+
'a': 1,
430+
'b': {
431+
'c': 2,
432+
'd': 30, # overridden by dict2
433+
'f': 40 # added from dict2
434+
},
435+
'e': 4,
436+
'g': 50 # added from dict2
437+
}
438+
439+
assert result == expected
440+
441+
def test_multiple_mappings_files():
442+
"""Test loading and merging multiple mappings files"""
443+
parser = argparse.ArgumentParser()
444+
command = GenerateCommand(parser)
445+
446+
# Mock two mappings files
447+
mappings1_content = {
448+
'mappings': {
449+
'teams': {
450+
'devops': 'devops-team'
451+
},
452+
'environments': {
453+
'dev': {
454+
'database_url': 'postgres://dev-db:5432'
455+
}
456+
}
457+
}
458+
}
459+
460+
mappings2_content = {
461+
'mappings': {
462+
'teams': {
463+
'frontend': 'frontend-team'
464+
},
465+
'environments': {
466+
'dev': {
467+
'debug': True
468+
},
469+
'prod': {
470+
'database_url': 'postgres://prod-db:5432'
471+
}
472+
}
473+
}
474+
}
475+
476+
with patch('os.path.exists', return_value=True), \
477+
patch('builtins.open', new_callable=MagicMock) as mock_open, \
478+
patch('yaml.safe_load', side_effect=[mappings1_content, mappings2_content]), \
479+
patch.object(command, '_create_structure') as mock_create_structure:
480+
481+
# Create args with multiple mappings files
482+
args = argparse.Namespace(
483+
structure_definition='test.yaml',
484+
base_path='/tmp/test',
485+
mappings_file=['mappings1.yaml', 'mappings2.yaml'],
486+
backup=None,
487+
output='file',
488+
structures_path=None
489+
)
490+
491+
# Mock config loading
492+
with patch.object(command, '_load_yaml_config', return_value={'files': []}):
493+
command.execute(args)
494+
495+
# Verify both files were attempted to be opened
496+
assert mock_open.call_count == 2
497+
498+
# Verify _create_structure was called with merged mappings
499+
mock_create_structure.assert_called_once()
500+
call_args = mock_create_structure.call_args[0]
501+
merged_mappings = call_args[1] # Second argument is mappings
502+
503+
# Verify the mappings were merged correctly
504+
expected_mappings = {
505+
'mappings': {
506+
'teams': {
507+
'devops': 'devops-team',
508+
'frontend': 'frontend-team'
509+
},
510+
'environments': {
511+
'dev': {
512+
'database_url': 'postgres://dev-db:5432',
513+
'debug': True
514+
},
515+
'prod': {
516+
'database_url': 'postgres://prod-db:5432'
517+
}
518+
}
519+
}
520+
}
521+
522+
assert merged_mappings == expected_mappings

0 commit comments

Comments
 (0)