Merge pull request #115 from jugaad-py/fix/nse-bhavcopy-format-migration

[Fix] Resolve NSE bhavcopy format migration issues #81 & #83

Author: sevakram

CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.32.0] - 2026-03-16
+
+### Added
+- New `NSEDailyReports` class to handle NSE daily-reports API (39+ report types)
+- `NSEArchives.download_report()` method to download any NSE report by file key
+- `NSEArchives.list_available_reports()` method to discover available report types
+- Hybrid bhavcopy format support: UDiff (recent) + BHAVDATA-FULL (historical)
+- Tests for new report discovery and download functionality
+
+### Fixed
+- Issue #81: NSE bhavcopy downloads failing for dates >= July 8, 2024 (BadZipFile error)
+- Issue #83: Bhavcopy format change - implemented automatic fallback to BHAVDATA-FULL
+- Bhavcopy now works for all dates using hybrid approach:
+  - Recent dates (Jul 8, 2024+): Uses new UDiff format from daily-reports API
+  - Historical dates: Uses BHAVDATA-FULL format (available for all dates)
+
+### Changed
+- `bhavcopy_raw()` now automatically handles both UDiff and BHAVDATA-FULL formats
+- Data returned as-is per NSE updates (no backward compatibility guarantee)
+
 ## [0.31.2] - 2026-03-16
 
 ### Fixed

docs/API_REFERENCE.md

@@ -356,6 +356,51 @@ Download derivatives data and save to CSV.
 - Same as `derivatives_df()`
 - `output` (str): Path for output CSV file
 
+### NSE Daily Reports
+
+#### `list_available_reports()`
+Discover all available daily reports from NSE.
+
+**Parameters:** None
+
+**Returns:**
+```python
+list  # List of dictionaries with 'key' and 'name' for each report
+```
+
+**Example:**
+```python
+from jugaad_data.nse import list_available_reports
+
+reports = list_available_reports()
+# Returns: [{'key': 'CM-VOLATILITY', 'name': 'CM Volatility'}, ...]
+```
+
+#### `download_report(report_key, to_date=None, output=None)`
+Download any NSE daily report (39+ report types available).
+
+**Parameters:**
+- `report_key` (str): Report key (e.g., "CM-VOLATILITY", "NIFTY-50-ADVANCE-DECLINE")
+- `to_date` (datetime.date, optional): Date for report (defaults to NSE current date)
+- `output` (str, optional): Path for output file
+
+**Returns:** File saved at `output` path or temp directory
+
+**Example:**
+```python
+from jugaad_data.nse import download_report
+
+# List available reports first
+download_report("CM-VOLATILITY", output="/path/to/file.csv")
+```
+
+#### `stock_df(symbol, from_date, to_date)` - Format Note
+As of July 8, 2024, NSE transitioned from direct CSV format to compressed UDiff format. The library handles this automatically:
+- Dates >= July 8, 2024: Returns UDiff format (newer column structure)
+- Historical dates: Falls back to BHAVDATA-FULL format
+
+Both formats are returned as raw CSV data with no column mapping. See [HISTORICAL_DATA_GUIDE](HISTORICAL_DATA_GUIDE.md) for detailed column specifications.
+
 ---
 
 ## BSE Module

docs/HISTORICAL_DATA_GUIDE.md

@@ -18,6 +18,21 @@ A comprehensive guide to downloading historical market data using `jugaad-data`.
 
 Bhavcopies are complete daily market snapshots published by stock exchanges after market close. They contain all traded contracts with their OHLC prices.
 
+### Format Changes (July 8, 2024)
+
+On July 8, 2024, NSE transitioned from the old ZIP-based bhavcopy format to the **Unified Distilled File Format (UDiff)**. The library now automatically handles both formats:
+
+- **Recent Dates (≥ July 8, 2024):** Uses UDiff format from NSE's daily-reports API
+  - Format: More comprehensive data structure
+  - Availability: Current day + previous trading day (via API)
+  
+- **Historical Dates (< July 8, 2024):** Uses BHAVDATA-FULL format
+  - Format: CSV with delivery information
+  - Availability: All historical dates
+  - No ZIP decompression required
+
+**No action needed on your part** - the library automatically chooses the best available format!
+
 ### Types of Bhavcopies
 
 1. **Equity Bhavcopy** - All equity securities traded on NSE
@@ -31,23 +46,38 @@ Bhavcopies are complete daily market snapshots published by stock exchanges afte
 from datetime import date
 from jugaad_data.nse import bhavcopy_save
 
-# Download for specific date
+# Download for specific date (works for all dates, automatic format selection)
 bhavcopy_save(date(2020, 1, 1), "/path/to/directory")
+bhavcopy_save(date(2024, 9, 15), "/path/to/directory")  # Uses UDiff if API available, else BHAVDATA-FULL
 
 # Download for date range
-for d in range(1, 31):
+from datetime import timedelta
+start_date = date(2024, 1, 1)
+end_date = date(2024, 12, 31)
+current = start_date
+while current <= end_date:
     try:
-        bhavcopy_save(date(2020, 1, d), "/path/to/directory")
+        bhavcopy_save(current, "/path/to/directory")
+        print(f"Downloaded bhavcopy for {current}")
     except:
         pass  # Holiday or weekend
+    current += timedelta(days=1)
 ```
 
-**Output file:** `cm01Jan2020bhav.csv`
+**Output file:** `cm01Jan2020bhav.csv` or `cm15Sep2024bhav.csv`
 
 **CSV Columns:**
+For historical dates (BHAVDATA-FULL format):
+```
+SYMBOL, SERIES, DATE, PREV_CLOSE, OPEN_PRICE, HIGH_PRICE, LOW_PRICE, 
+LAST_PRICE, CLOSE_PRICE, AVG_PRICE, TTL_TRD_QNTY, TURNOVER_LACS, 
+NO_OF_TRADES, DELIV_QTY, DELIV_PER
+```
+
+For recent dates (UDiff format):
 ```
-SYMBOL, SERIES, OPEN, HIGH, LOW, CLOSE, LAST, PREVCLOSE, TOTTRDQTY, 
-TOTTRDVAL, DATE, TOTTRNUM, ISIN
+TradDt, BizDt, Sgmt, Src, FinInstrmTp, FinInstrmId, ISIN, TckrSymb, 
+SctySrs, XpryDt, ... (enhanced data structure)
 ```
 
 ### Download Full Bhavcopy (with Delivery Data)
@@ -64,6 +94,58 @@ full_bhavcopy_save(date(2020, 1, 1), "/path/to/directory")
 Delivery Quantity %
 ```
 
+### Download Other NSE Reports (39+ types available)
+
+The library now provides access to 39+ NSE reports through the daily-reports API:
+
+#### List Available Reports
+
+```python
+from jugaad_data.nse import NSEArchives
+
+nse = NSEArchives()
+
+# See all available reports
+reports = nse.list_available_reports()
+
+for file_key, info in reports.items():
+    print(f"{file_key}: {info['displayName']}")
+    print(f"  Available dates: {[d['date'] for d in info['dates']]}")
+```
+
+#### Download Specific Reports
+
+```python
+from jugaad_data.nse import NSEArchives
+
+nse = NSEArchives()
+
+# Download volatility data
+info = nse.download_report('CM-VOLATILITY', "/path/to/save")
+print(f"Downloaded: {info['file_name']}")
+print(f"Trading date: {info['trading_date']}")
+
+# Other available reports:
+# 'CM-UDIFF-BHAVCOPY-CSV'   - UDiff format bhavcopy (zip)
+# 'CM-BULK-DEAL'            - Bulk deals data
+# 'CM-BLOCK-DEAL'           - Block deals data
+# 'CM-SHORT-SELLING'        - Short selling data
+# 'CM-VOLATILITY'           - Daily volatility
+# 'CM-CIRCUIT'              - Circuit breaker updates
+# ... and 33+ more report types
+```
+
+**Return Value:**
+```python
+{
+    'file_path': '/path/to/save/filename.csv',  # Local file path
+    'file_name': 'CMVOLT_13032026.CSV',         # Actual file name from NSE
+    'trading_date': '13-Mar-2026',              # Trading date of report
+    'size': '292.78 KB',                        # File size
+    'cached': False                             # Whether file was cached
+}
+```
+
 ### Download F&O Bhavcopy
 
 ```python

docs/QUICKSTART.md

@@ -178,6 +178,31 @@ Output includes:
 - Government Securities rates
 - Index values (Sensex, Nifty 50)
 
+### Download NSE Daily Reports (39+ Report Types)
+
+NSE publishes 39+ different reports daily including volatility data, advance/decline data, and more.
+
+```python
+from jugaad_data.nse import list_available_reports, download_report
+
+# Discover all available reports
+reports = list_available_reports()
+for report in reports[:5]:  # Show first 5
+    print(f"{report['key']}: {report['name']}")
+
+# Download a specific report (e.g., CM Volatility)
+download_report("CM-VOLATILITY", output="cm_volatility.csv")
+
+# Download another report type
+download_report("NIFTY-50-ADVANCE-DECLINE", output="nifty_advance_decline.csv")
+```
+
+Available reports include:
+- CM-VOLATILITY: Equity segment volatility measures
+- NIFTY-50-ADVANCE-DECLINE: NIFTY 50 advance/decline statistics
+- NIFTY-NEXT-50-ADVANCE-DECLINE: NIFTY Next 50 statistics
+- And 36+ more report types
+
 ### Command Line Interface
 
 ```bash
@@ -226,6 +251,7 @@ $ jdata derivatives -s NIFTY -f 2020-01-01 -t 2020-01-23 -e 2020-01-23 -i OPTIDX
 ✅ Fetch live index and turnover data
 ✅ Fetch option chains
 ✅ Fetch current rates from RBI website
+✅ Download 39+ NSE daily reports (volatility, advance/decline, etc.)
 ✅ Pandas dataframe support
 ✅ Command-line interface