-
Notifications
You must be signed in to change notification settings - Fork 1
Historical Data Retrieval
- Introduction
- Core Components
- Parameter Analysis
- TVBar Structure
- Callback Implementation
- Error Handling
- Pagination and Data Management
- Best Practices
The historical data retrieval system in PyTradingView is designed to provide market data through the getBars method, which serves as the primary interface for obtaining historical market data. This document details the implementation requirements, parameter specifications, data structures, and best practices for implementing the getBars method to retrieve historical market data effectively. The system follows the TradingView datafeed API specification and provides a comprehensive framework for integrating with various data sources.
The historical data retrieval functionality is built around several core components that work together to provide market data to the TradingView charting library. The main components include the TVDatafeed class, which serves as the base implementation, and the BADatafeed class, which provides an example implementation. The system uses a modular architecture with separate data structures for bars, metadata, and parameters.
The getBars method is defined in the TVIDatafeedChartApi interface and must be implemented by any datafeed provider. The method is responsible for retrieving historical bar data based on the provided parameters and returning it through the appropriate callback mechanism. The implementation must handle various edge cases, including missing data, pagination, and error conditions.
classDiagram
class TVDatafeed {
+_configuration : TVDatafeedConfiguration
+_subscribers : Dict[str, Any]
+_quote_subscribers : Dict[str, Any]
+getBars(symbolInfo, resolution, periodParams, onResult, onError) void
+subscribeBars(symbolInfo, resolution, onTick, listenerGuid, onResetCacheNeededCallback) void
+unsubscribeBars(listenerGuid) void
}
class TVDatafeedConfiguration {
+exchanges : List[TVExchange]
+supported_resolutions : List[ResolutionString]
+units : Dict[str, List[TVUnit]]
+currency_codes : List
+supports_marks : bool
+supports_time : bool
+supports_timescale_marks : bool
}
TVDatafeed --> TVDatafeedConfiguration : "has"
TVDatafeed --> TVBar : "returns"
TVDatafeed --> TVHistoryMetadata : "returns"
TVDatafeed --> TVPeriodParams : "receives"
TVDatafeed --> TVLibrarySymbolInfo : "receives"
**Diagram sources **
The getBars method accepts several parameters that define the data retrieval request. These parameters include symbolInfo, resolution, periodParams, onResult, and onError. Each parameter plays a specific role in the data retrieval process and must be handled appropriately by the implementation.
The symbolInfo parameter contains comprehensive information about the financial instrument for which data is being requested. This includes the symbol name, description, type, exchange, timezone, and various configuration options. The TVLibrarySymbolInfo class provides a complete representation of the symbol's metadata, including trading sessions, price formatting, and supported resolutions.
The resolution parameter specifies the time interval for the requested bars. This is represented as a ResolutionString which can include values like "1", "5", "15", "30", "60", "240", "1D", "1W", and "1M" for different timeframes. The available resolutions are defined in the datafeed configuration and must be supported by the underlying data source.
The periodParams parameter contains the temporal boundaries for the data request. This includes the from_ timestamp (leftmost requested bar), to timestamp (rightmost requested bar, not inclusive), countBack (exact amount of bars to load), and firstDataRequest (flag indicating if this is the first call). The countBack parameter takes precedence over the from_ parameter when both are specified, enabling efficient pagination.
classDiagram
class TVPeriodParams {
+from_ : int
+to : int
+countBack : int
+firstDataRequest : bool
}
class TVLibrarySymbolInfo {
+name : str
+description : str
+type : str
+session : str
+exchange : str
+listed_exchange : str
+timezone : Timezone
+format : SeriesFormat
+pricescale : int
+minmov : int
+has_intraday : bool
+supported_resolutions : List[ResolutionString]
+volume_precision : int
}
class ResolutionString {
<<type alias>>
}
TVPeriodParams --> ResolutionString : "uses"
TVDatafeed --> TVPeriodParams : "receives"
TVDatafeed --> TVLibrarySymbolInfo : "receives"
**Diagram sources **
The TVBar class represents a single bar of market data and contains the core OHLCV (Open, High, Low, Close, Volume) information. Each bar includes a timestamp in milliseconds since Unix epoch in UTC timezone, which is critical for proper time series alignment across different timezones.
The time field has specific requirements based on the resolution:
- For intraday bars (1m, 5m, etc.), the time represents the start of the bar period
- For daily, weekly, and monthly bars, the time is expected to be at 00:00 UTC of the trading day
- The time must be in UTC timezone to ensure consistency across different client timezones
Volume precision is controlled by the volume_precision field in the TVLibrarySymbolInfo class, which specifies the number of decimal places to display for volume values. This allows for proper formatting of volume data for different asset classes, such as cryptocurrencies with high volume precision or stocks with integer volume values.
classDiagram
class TVBar {
+time : int
+open : float
+high : float
+low : float
+close : float
+volume : Optional[float]
+to_dict() dict
}
class TVHistoryMetadata {
+noData : Optional[bool]
+nextTime : Optional[int]
}
TVDatafeed --> TVBar : "returns"
TVDatafeed --> TVHistoryMetadata : "returns"
TVBar --> TVHistoryMetadata : "associated with"
**Diagram sources **
The getBars method uses a callback-based approach for returning data, with two primary callbacks: onResult and onError. This asynchronous pattern allows for non-blocking data retrieval and proper error handling.
The onResult callback accepts two parameters: an array of TVBar objects and an optional TVHistoryMetadata object. The metadata provides additional information about the data set, including whether there is no more data available (noData flag) and the time of the next available bar (nextTime). This enables the charting library to determine when to stop requesting historical data and whether there are gaps in the data series.
The onError callback is invoked when an error occurs during data retrieval. It accepts a string parameter containing the error message, which should be descriptive enough to diagnose the issue. Common error scenarios include network connectivity problems, invalid symbol information, or issues with the underlying data source.
sequenceDiagram
participant Chart as "TradingView Chart"
participant Datafeed as "TVDatafeed Implementation"
participant DataSource as "External Data Source"
Chart->>Datafeed : getBars(symbolInfo, resolution, periodParams, onResult, onError)
Datafeed->>DataSource : Request historical data
alt Data Available
DataSource-->>Datafeed : Return bar data
Datafeed->>Datafeed : Create TVBar objects
Datafeed->>Datafeed : Create TVHistoryMetadata
Datafeed->>Chart : onResult(bars, metadata)
else Error Occurred
DataSource-->>Datafeed : Error response
Datafeed->>Chart : onError(errorMessage)
end
**Diagram sources **
Proper error handling is critical for a robust historical data retrieval implementation. The system must handle various error conditions gracefully and provide meaningful feedback to the client application.
The TVHistoryMetadata class includes a noData flag that indicates when no more data is available from the server. This flag should be set to True when the requested time range extends beyond the available historical data, allowing the charting library to stop requesting additional data. Incorrect usage of this flag can lead to infinite loading states or missing data gaps in the chart.
Common issues to address include:
- Timestamp format errors: Ensure all timestamps are in milliseconds since Unix epoch in UTC timezone
- Missing data gaps: Handle cases where data is not available for certain time periods
- Invalid resolution requests: Verify that the requested resolution is supported by the data source
- Rate limiting: Implement appropriate delays between requests to avoid overwhelming the data source
flowchart TD
Start([getBars Request]) --> ValidateInput["Validate Input Parameters"]
ValidateInput --> InputValid{"Input Valid?"}
InputValid --> |No| HandleError["Invoke onError callback"]
InputValid --> |Yes| CheckData["Check Data Availability"]
CheckData --> DataAvailable{"Data Available?"}
DataAvailable --> |No| SetNoData["Set noData=True in metadata"]
DataAvailable --> |Yes| ProcessData["Process Raw Data"]
ProcessData --> CreateBars["Create TVBar objects"]
CreateBars --> ReturnResult["Invoke onResult callback"]
SetNoData --> ReturnResult
HandleError --> End([Method Complete])
ReturnResult --> End
**Diagram sources **
Effective pagination and data management are essential for handling large data requests efficiently. The countBack parameter plays a crucial role in pagination, allowing clients to request a specific number of bars rather than specifying a time range.
When implementing pagination:
- Prioritize the
countBackparameter over thefrom_parameter when both are specified - Use the
nextTimefield inTVHistoryMetadatato indicate when more data is available - Implement efficient data caching strategies to reduce redundant requests
- Consider the
firstDataRequestflag to optimize initial data loading
For large data requests, implement chunking strategies to avoid overwhelming the client or network. This can be achieved by:
- Limiting the maximum number of bars returned in a single request
- Implementing server-side pagination
- Using incremental loading based on viewport requirements
- Caching frequently requested data ranges
flowchart LR
A[Client Request] --> B{countBack > 0?}
B --> |Yes| C[Return countBack bars from end of range]
B --> |No| D{from_ and to specified?}
D --> |Yes| E[Return bars in time range]
D --> |No| F[Return default bar count]
C --> G[Set nextTime if more data available]
E --> G
F --> G
G --> H[Return data via onResult callback]
**Diagram sources **
Implementing the getBars method effectively requires adherence to several best practices that ensure reliability, performance, and compatibility with the TradingView platform.
Implement efficient data caching strategies to:
- Reduce redundant requests to external data sources
- Improve response times for frequently accessed data
- Handle rate limiting from data providers
- Support offline functionality when possible
Cache strategies should consider:
- Time-based expiration of cached data
- Memory usage limits
- Cache invalidation on data updates
- Persistent storage for critical data
The relationship between requested resolution and available data granularity is critical:
- Ensure the data source can provide data at the requested resolution
- Implement data aggregation for lower resolutions when necessary
- Handle cases where the requested resolution is not supported
- Provide appropriate error messages for unsupported resolutions
Optimize performance by:
- Minimizing data transfer size through efficient serialization
- Implementing connection pooling for data sources
- Using asynchronous operations for I/O-bound tasks
- Optimizing database queries for historical data retrieval
The BADatafeed class provides a concrete example of implementing the getBars method. It demonstrates proper error handling, mock data generation, and correct callback invocation. When implementing a production datafeed, replace the mock data logic with actual data retrieval from your data source while maintaining the same interface and error handling patterns.