Merge pull request #37 from VirtualFlyBrain/dev

Cleaned up caching

Author: Robbie1977

.github/workflows/examples.yml

@@ -22,9 +22,25 @@ jobs:
             pip install -r requirements.txt
             pip install deepdiff colorama
             pip install .
+      - name: Check SOLR availability
+        run: |
+          python -c "
+          import os
+          try:
+              import vfbquery as vfb
+              result = vfb.get_term_info('FBbt_00003748')
+              with open(os.environ['GITHUB_ENV'], 'a') as f:
+                  f.write('SOLR_AVAILABLE=true\n')
+              print('SOLR is available')
+          except Exception as e:
+              print('SOLR not available:', e)
+              with open(os.environ['GITHUB_ENV'], 'a') as f:
+                  f.write('SOLR_AVAILABLE=false\n')
+              exit(1)
+          "
       - name: Run examples from README.md
         run: |
-          cat README.md | grep -e '```python' -e '```' -e '^[^`]*$' | sed -e '/^```python/,/^```/!d' -e '/^```/d' -e 's/\(vfb.[^)]*)\)/print(\1)/g' > test_examples.py
+          cat README.md | grep -e '```python' -e '```' -e '^[^`]*$' | sed -e '/^```python/,/^```/!d' -e '/^```/d' -e 's/\(vfb\.[^(]*([^)]*)\)/print(\1)/g' > test_examples.py
           cat test_examples.py
           export VFBQUERY_CACHE_ENABLED=false
           python test_examples.py
@@ -33,8 +49,10 @@ jobs:
           python -m src.test.readme_parser
         env:
           PYTHONPATH: ${{ github.workspace }}
-      - name: Run examples from README.md and compare JSON outputs
+        if: env.SOLR_AVAILABLE == 'true'
+      - name: Run examples from README.md and validate structure
         run: |
           python -m src.test.test_examples_diff
         env:
           PYTHONPATH: ${{ github.workspace }}
+        if: env.SOLR_AVAILABLE == 'true'

.gitignore

@@ -12,3 +12,5 @@ test_results.py
 .pytest_cache
 .venv
 .vscode/settings.json
+temp_examples_output.txt
+json_block_*.json

CACHING.md

@@ -1,41 +1,43 @@
 # VFBquery Caching Guide
 
-VFBquery includes intelligent caching for optimal performance. Caching is **enabled by default** with production-ready settings.
+VFBquery includes intelligent SOLR-based caching for optimal performance. Caching is **enabled by default** with production-ready settings.
 
 ## Default Behavior
 
-VFBquery automatically enables caching when imported:
+VFBquery automatically enables SOLR caching when imported:
 
 ```python
 import vfbquery as vfb
 
-# Caching is already active with optimal settings:
+# SOLR caching is already active with optimal settings:
 # - 3-month cache duration
-# - 2GB memory cache with LRU eviction  
-# - Persistent disk storage
+# - Persistent across sessions
 # - Zero configuration required
 
 result = vfb.get_term_info('FBbt_00003748')  # Cached automatically
 ```
 
+## How It Works
+
+VFBquery uses a single-layer caching approach with SOLR:
+
+1. **First query**: Fetches data from Neo4j/Owlery and caches in SOLR
+2. **Subsequent queries**: Served directly from SOLR cache
+3. **Cache persistence**: Survives Python restarts and server reboots
+4. **Automatic expiration**: 3-month TTL matches VFB_connect behavior
+
 ## Runtime Configuration
 
-Adjust cache settings while your application is running:
+Control caching behavior:
 
 ```python
 import vfbquery as vfb
 
-# Modify cache duration
-vfb.set_cache_ttl(720)                    # 1 month  
-vfb.set_cache_ttl(24)                     # 1 day
-
-# Adjust memory limits
-vfb.set_cache_memory_limit(512)           # 512MB
-vfb.set_cache_max_items(5000)             # 5K items
+# Clear specific cache entries
+vfb.clear_solr_cache('term_info', 'FBbt_00003748')
 
-# Toggle disk persistence  
-vfb.disable_disk_cache()                  # Memory-only
-vfb.enable_disk_cache()                   # Restore persistence
+# Get SOLR cache statistics
+stats = vfb.get_solr_cache().get_cache_stats()
 ```
 
 ### Environment Control
@@ -48,39 +50,38 @@ export VFBQUERY_CACHE_ENABLED=false
 
 ## Performance Benefits
 
-VFBquery caching provides significant performance improvements:
+VFBquery SOLR caching provides significant performance improvements:
 
 ```python
 import vfbquery as vfb
 
-# First query: builds cache (~1-2 seconds)  
+# First query: builds SOLR cache (~1-2 seconds)  
 result1 = vfb.get_term_info('FBbt_00003748')
 
-# Subsequent queries: served from cache (<0.1 seconds)
+# Subsequent queries: served from SOLR cache (<0.1 seconds)
 result2 = vfb.get_term_info('FBbt_00003748')  # 54,000x faster!
+
+# Similarity queries are also cached
+similar = vfb.get_similar_neurons('VFB_jrchk00s')  # Cached after first run
 ```
 
 **Typical Performance:**
 
 - First query: 1-2 seconds  
 - Cached queries: <0.1 seconds
 - Speedup: Up to 54,000x for complex queries
+- **NBLAST similarity queries**: 10+ seconds → <0.1 seconds (cached)
 
 ## Monitoring Cache Performance
 
 ```python
 import vfbquery as vfb
 
-# Get cache statistics
-stats = vfb.get_vfbquery_cache_stats()
-print(f"Hit rate: {stats['hit_rate_percent']}%")
-print(f"Memory used: {stats['memory_cache_size_mb']}MB")
-print(f"Cache items: {stats['memory_cache_items']}")
-
-# Get current configuration
-config = vfb.get_cache_config()
-print(f"TTL: {config['cache_ttl_hours']} hours")
-print(f"Memory limit: {config['memory_cache_size_mb']}MB")
+# Get SOLR cache statistics
+cache = vfb.get_solr_cache()
+stats = cache.get_cache_stats()
+print(f"Total cached items: {stats['total_documents']}")
+print(f"Cache size: {stats['total_size_mb']:.1f}MB")
 ```
 
 ## Usage Examples
@@ -90,21 +91,21 @@ print(f"Memory limit: {config['memory_cache_size_mb']}MB")
 ```python
 import vfbquery as vfb
 
-# Caching is enabled automatically with optimal defaults
-# Adjust only if your application has specific needs
+# SOLR caching is enabled automatically with optimal defaults
+# Cache persists across application restarts
 
-# Example: Long-running server with limited memory
-vfb.set_cache_memory_limit(512)    # 512MB limit
-vfb.set_cache_ttl(168)             # 1 week TTL
+# Example: Long-running server
+result = vfb.get_term_info('FBbt_00003748')     # Fast on repeated runs
+instances = vfb.get_instances('FBbt_00003748')  # Cached automatically
 ```
 
 ### Jupyter Notebooks
 
 ```python
 import vfbquery as vfb
 
-# Caching works automatically in notebooks
-# Data persists between kernel restarts
+# SOLR caching works automatically in notebooks
+# Data persists between kernel restarts and notebook sessions
 
 result = vfb.get_term_info('FBbt_00003748')     # Fast on repeated runs
 instances = vfb.get_instances('FBbt_00003748')  # Cached automatically
@@ -114,14 +115,14 @@ instances = vfb.get_instances('FBbt_00003748')  # Cached automatically
 
 - **Dramatic Performance**: 54,000x speedup for repeated queries
 - **Zero Configuration**: Works out of the box with optimal settings
-- **Persistent Storage**: Cache survives Python restarts  
-- **Memory Efficient**: LRU eviction prevents memory bloat
-- **Multi-layer Caching**: Optimizes SOLR queries, parsing, and results
+- **Persistent Storage**: SOLR cache survives Python restarts and server reboots
+- **Server-side Caching**: Shared across multiple processes/instances
+- **Similarity Queries**: NBLAST and morphological similarity searches are cached
 - **Production Ready**: 3-month TTL matches VFB_connect behavior
 
 ## Best Practices
 
-- **Monitor performance**: Use `get_vfbquery_cache_stats()` regularly
-- **Adjust for your use case**: Tune memory limits for long-running applications  
-- **Consider data freshness**: Shorter TTL for frequently changing data
+- **Monitor performance**: Use SOLR cache statistics regularly
+- **Clear when needed**: Use `clear_solr_cache()` to force fresh data
+- **Consider data freshness**: SOLR cache TTL ensures data doesn't become stale
 - **Disable when needed**: Use environment variable if caching isn't desired