Skip to content

Add comprehensive MAP type support for SQLAlchemy ORM#589

Merged
laughingman7743 merged 7 commits into
masterfrom
feature/map-type-support
Aug 3, 2025
Merged

Add comprehensive MAP type support for SQLAlchemy ORM#589
laughingman7743 merged 7 commits into
masterfrom
feature/map-type-support

Conversation

@laughingman7743
Copy link
Copy Markdown
Member

Summary

This PR implements complete MAP type support for SQLAlchemy ORM, addressing GitHub issue #553. Users can now work with MAP types seamlessly in their PyAthena applications with full type safety and automatic conversion.

Implementation Details

Core Features

  • AthenaMap Type Class: Full SQLAlchemy type integration with configurable key/value types
  • Smart Converter: _to_map() function supporting both JSON and Athena native formats
  • SQL Compiler: Generates proper MAP<key_type, value_type> SQL syntax
  • Performance Optimized: Smart format detection avoids unnecessary parsing overhead

Usage Examples

Table Definition:

from pyathena.sqlalchemy.types import AthenaMap

products = Table('products', metadata,
    Column('id', Integer),
    Column('attributes', AthenaMap(String, String)),
    Column('metrics', AthenaMap(String, Integer))
)

Query Results:

# Automatic conversion to Python dict
result = connection.execute("SELECT MAP(ARRAY['name', 'category'], ARRAY['Laptop', 'Electronics'])")
data = result.fetchone()[0]  # {"name": "Laptop", "category": "Electronics"}

Testing

  • ✅ Unit tests for type creation and SQL compilation
  • ✅ Integration tests with actual Athena MAP queries
  • ✅ Converter tests for both JSON and native formats
  • ✅ All quality checks (lint, format, type checking) passing

Documentation

  • Complete usage guide with examples
  • Performance considerations and best practices
  • Migration guide from raw string handling
  • Updated feature overview in introduction

Test Plan

  • Create MAP types with different key/value combinations
  • Generate correct SQL syntax for table creation
  • Convert MAP strings to Python dictionaries
  • Handle both JSON and Athena native formats
  • Integration with actual Athena queries
  • All existing tests continue to pass

This follows the same architectural patterns established in the STRUCT type implementation (#587), ensuring consistency and maintainability.

Closes #553

🤖 Generated with Claude Code

This implements complete MAP type support addressing GitHub issue #553:

## Core Implementation
- Add _to_map converter function supporting JSON and Athena native formats
- Create AthenaMap and MAP type classes for SQLAlchemy integration
- Add visit_map methods to type compiler for SQL generation
- Smart format detection for optimal performance

## Features
- Full SQLAlchemy ORM support with MAP<key_type, value_type> syntax
- Automatic conversion between string and dictionary representations
- Support for both JSON and Athena native {key=value} formats
- Type-safe key and value type specifications
- Performance optimized with format detection

## Testing & Documentation
- Comprehensive unit tests for types, compiler, and converter
- Integration tests with actual Athena MAP queries
- Complete documentation with usage examples and best practices
- Migration guide for existing raw string handling

Closes #553

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@laughingman7743 laughingman7743 marked this pull request as draft August 2, 2025 14:05
laughingman7743 and others added 5 commits August 2, 2025 23:08
- Handle complex structures (arrays/structs) by returning None for string fallback
- Convert numeric keys to strings for test compatibility
- Skip parsing MAP values containing complex nested structures
- Maintain backward compatibility with existing behavior

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Move numeric key test from STRUCT to MAP (correct type)
- Update SQLAlchemy reflection test expectations for MAP conversion
- Add proper STRUCT test with named fields
- Ensure tests match the actual data type semantics

{1=2, 3=4} is MAP format, not STRUCT format.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Convert STRUCT, ARRAY, MAP, and complex combination tests to use @pytest.mark.parametrize
- Fix MAP type test issues by using JSON casting for complex structures
- Improve test readability and maintainability
- Reduce code duplication in test methods
- Fix line length issues for better code formatting
- All tests now run independently with clear failure identification

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Convert all STRUCT converter tests to use @pytest.mark.parametrize
- Consolidate 25 test cases into organized parametrized groups:
  - JSON format tests (5 cases)
  - Athena native format tests (8 cases)
  - Complex cases with special characters (3 cases)
  - Non-dict JSON format tests (3 cases)
  - DefaultTypeConverter tests (5 cases)
- Remove code duplication and improve test maintainability
- All 25 tests pass with clear individual case identification

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Update test_complex to expect MAP values as converted dictionaries instead of strings
- Update pandas test_as_pandas with same MAP expectations
- Fix DBAPI type assertions for MAP columns
- All CI failing tests now pass locally

MAP type converter now properly converts {1=2, 3=4} → {'1': 2, '3': 4}

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@laughingman7743 laughingman7743 marked this pull request as ready for review August 2, 2025 15:19
- Replace hasattr() with isinstance() for AthenaStruct and AthenaMap
- Add explicit type imports for better type safety
- Simplify struct field checking logic with guaranteed type safety
- More robust and performant type checking following SQLAlchemy patterns

This eliminates potential AttributeError risks and provides clearer code semantics.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@laughingman7743 laughingman7743 merged commit 2d2a66f into master Aug 3, 2025
5 checks passed
@laughingman7743 laughingman7743 deleted the feature/map-type-support branch August 3, 2025 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLAlchemy + Map Types

1 participant