Updating docs with logging and secrets.env to .env

Author: patrickfleith

README.md

@@ -46,7 +46,7 @@ pip install datafast
 
 ### 1. Environment Setup
 
-Make sure you have created a `secrets.env` file with your API keys.
+Make sure you have created a `.env` file with your API keys.
 HF token is needed if you want to push the dataset to your HF hub.
 Other keys depends on which LLM providers you use.
 ```
@@ -64,7 +64,7 @@ from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider
 from dotenv import load_dotenv
 
 # Load environment variables
-load_dotenv("secrets.env") # <--- your API keys
+load_dotenv() # <--- your API keys
 ```
 
 ### 3. Configure Dataset

docs/concepts.md

@@ -69,13 +69,41 @@ The prompt expansion system is key and enables:
 The datafast workflow follows a consistent pattern across all dataset types:
 
 1. **Configuration**: Define the dataset parameters, classes/topics, and generation settings
-2. **Prompt Design**: Create base prompts with mandatory and optional placeholders
-3. **Provider Setup**: Initialize one or more LLM providers
-4. **Generation**: Execute the generation process, which:
+2. **Logging Setup**: Configure logging to monitor the generation process (recommended)
+3. **Prompt Design**: Create base prompts with mandatory and optional placeholders
+4. **Provider Setup**: Initialize one or more LLM providers
+5. **Generation**: Execute the generation process, which:
     - Expands prompts based on configuration
     - Distributes generation across providers
     - Collects and processes responses
-5. **Output**: Save the resulting dataset to a file and optionally push to Hugging Face Hub
+6. **Output**: Save the resulting dataset to a file and optionally push to Hugging Face Hub
+
+## Logging and Monitoring
+
+Datafast includes comprehensive logging to provide visibility into the generation process:
+
+### Why Configure Logging?
+
+Without `configure_logger()`, your datafast scripts will run silently without:
+- Progress indicators during generation
+- Rate limiting warnings
+- Success completion messages
+- Detailed error information
+
+### Basic Usage
+
+```python
+from datafast.logger_config import configure_logger
+
+# Default: INFO level, console output with colors
+configure_logger()
+
+# With file logging for long-running jobs
+configure_logger(level="INFO", log_file="generation.log")
+
+# Debug mode for troubleshooting
+configure_logger(level="DEBUG", log_file="debug.log")
+```
 
 ## Dataset Diversity Mechanisms
 

docs/guides/generating_generic_pipeline_datasets.md

@@ -34,16 +34,20 @@ from datafast.schema.config import GenericPipelineDatasetConfig
 from datafast.llms import OpenRouterProvider
 ```
 
-In addition, we'll use `dotenv` to load environment variables containing API keys:
+In addition, we'll use `dotenv` to load environment variables containing API keys and configure logging to monitor the generation process:
 
 ```python
 from dotenv import load_dotenv
+from datafast.logger_config import configure_logger
 
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a `secrets.env` file with your API keys:
+Make sure you have created a `.env` file with your API keys:
 
 ```
 OPENROUTER_API_KEY=XXXX
@@ -214,10 +218,14 @@ Here's a complete working example:
 from datafast.datasets import GenericPipelineDataset
 from datafast.schema.config import GenericPipelineDatasetConfig
 from datafast.llms import OpenRouterProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 # Define prompt
 PROMPT = """I will give you a persona description.

docs/guides/generating_mcq_datasets.md

@@ -31,15 +31,19 @@ from datafast.schema.config import MCQDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider
 ```
 
-In addition, we'll use `dotenv` to load environment variables containing API keys.
+In addition, we'll use `dotenv` to load environment variables containing API keys and configure logging to monitor the generation process.
 ```python
 from dotenv import load_dotenv
+from datafast.logger_config import configure_logger
 
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a `secrets.env` file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use.
+Make sure you have created a `.env` file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use.
 
 ```
 GEMINI_API_KEY=XXXX
@@ -253,10 +257,14 @@ Here's a complete example for creating an MCQ dataset from a local JSONL file:
 from datafast.datasets import MCQDataset
 from datafast.schema.config import MCQDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load environment variables
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 def main():
     # 1. Define the configuration

docs/guides/generating_preference_datasets.md

@@ -21,19 +21,23 @@ Generating a preference dataset with `datafast` requires these imports:
 from datafast.datasets import PreferenceDataset
 from datafast.schema.config import PreferenceDatasetConfig
 from datafast.llms import OpenAIProvider, GeminiProvider, AnthropicProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 import json
 from pathlib import Path
 ```
 
-You'll need to load environment variables containing API keys:
+You'll need to load environment variables containing API keys and configure logging:
 
 ```python
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a `secrets.env` file with your API keys for the LLM providers you plan to use:
+Make sure you have created a `.env` file with your API keys for the LLM providers you plan to use:
 
 ```
 OPENAI_API_KEY=sk-XXXX
@@ -235,10 +239,14 @@ from pathlib import Path
 from datafast.schema.config import PreferenceDatasetConfig
 from datafast.datasets import PreferenceDataset 
 from datafast.llms import OpenAIProvider, GeminiProvider, AnthropicProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load environment variables with API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 # Load NASA lessons learned documents from JSONL file
 def load_documents_from_jsonl(jsonl_path: str | Path) -> list[str]:

docs/guides/generating_text_classification_datasets.md

@@ -26,15 +26,19 @@ from datafast.schema.config import ClassificationDatasetConfig, PromptExpansionC
 from datafast.llms import OpenAIProvider, AnthropicProvider
 ```
 
-In addition, we'll use `dotenv` to load environment variables containing API keys.
+In addition, we'll use `dotenv` to load environment variables containing API keys and configure logging to monitor the generation process.
 ```python
 from dotenv import load_dotenv
+from datafast.logger_config import configure_logger
 
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a `secrets.env` file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depends on which LLM providers you use. In our example, we use OpenAI and Anthropic.
+Make sure you have created a `.env` file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depends on which LLM providers you use. In our example, we use OpenAI and Anthropic.
 
 ```
 GEMINI_API_KEY=XXXX
@@ -236,10 +240,14 @@ Here's a complete example for creating a trail conditions classification dataset
 from datafast.datasets import ClassificationDataset
 from datafast.schema.config import ClassificationDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 # Configure dataset
 config = ClassificationDatasetConfig(

docs/guides/generating_text_datasets.md

@@ -31,15 +31,19 @@ from datafast.schema.config import RawDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider
 ```
 
-In addition, we'll use `dotenv` to load environment variables containing API keys.
+In addition, we'll use `dotenv` to load environment variables containing API keys and configure logging to monitor the generation process.
 ```python
 from dotenv import load_dotenv
+from datafast.logger_config import configure_logger
 
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a secrets.env file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use. In our example, we use OpenAI and Anthropic.
+Make sure you have created a .env file with your API keys. HF token is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use. In our example, we use OpenAI and Anthropic.
 
 ```
 GEMINI_API_KEY=XXXX
@@ -239,6 +243,14 @@ Here's a complete example script that generates a text dataset across multiple d
 from datafast.datasets import RawDataset
 from datafast.schema.config import RawDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider
+from datafast.logger_config import configure_logger
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 
 def main():
@@ -303,9 +315,6 @@ def main():
 
 
 if __name__ == "__main__":
-    from dotenv import load_dotenv
-
-    load_dotenv("secrets.env")
     main()
 ```
 

docs/guides/generating_ultrachat_datasets.md

@@ -31,15 +31,19 @@ from datafast.schema.config import UltrachatDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider
 ```
 
-In addition, use `dotenv` to load environment variables containing API keys:
+In addition, use `dotenv` to load environment variables containing API keys and configure logging to monitor the generation process:
 ```python
 from dotenv import load_dotenv
+from datafast.logger_config import configure_logger
 
 # Load environment variables containing API keys
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger to see progress, warnings, and success messages
+configure_logger()
 ```
 
-Make sure you have created a `secrets.env` file with your API keys. A Hugging Face token (HF_TOKEN) is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use.
+Make sure you have created a `.env` file with your API keys. A Hugging Face token (HF_TOKEN) is needed if you want to push the dataset to your HF hub. Other keys depend on which LLM providers you use.
 
 ```
 GEMINI_API_KEY=XXXX
@@ -231,10 +235,14 @@ Here's a complete example for creating an Ultrachat dataset:
 from datafast.datasets import UltrachatDataset
 from datafast.schema.config import UltrachatDatasetConfig
 from datafast.llms import AnthropicProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load environment variables
-load_dotenv("secrets.env")
+load_dotenv()
+
+# Configure logger
+configure_logger()
 
 def main():
     # 1. Define the configuration

docs/index.md

@@ -35,7 +35,7 @@ Currently we support the following LLM providers:
 
 ### 1. Environment Setup
 
-Make sure you have created a `secrets.env` file with your API keys.
+Make sure you have created a `.env` file with your API keys.
 HF token is needed if you want to push the dataset to your HF hub.
 Other keys depends on which LLM providers you use.
 ```
@@ -51,10 +51,14 @@ HF_TOKEN=hf_XXXXX
 from datafast.datasets import ClassificationDataset
 from datafast.schema.config import ClassificationDatasetConfig, PromptExpansionConfig
 from datafast.llms import OpenAIProvider, AnthropicProvider, GeminiProvider, OpenRouterProvider
+from datafast.logger_config import configure_logger
 from dotenv import load_dotenv
 
 # Load environment variables
-load_dotenv("secrets.env") # <--- your API keys
+load_dotenv() # <--- your API keys
+
+# Configure logger for visibility into generation process
+configure_logger() # <--- see progress, warnings, and success messages
 ```
 
 ### 3. Configure Dataset
@@ -135,6 +139,7 @@ Star this package to send positive vibes and support 🌟
 * **Multiple LLMs** used to boost dataset diversity 🤖
 * **Flexible prompt**: use our default prompts or provide your own custom prompts 📝
 * **Prompt expansion**: Combinatorial variation of prompts to maximize diversity 🔄
+* **Built-in logging**: Comprehensive logging with progress tracking, rate limiting warnings, and success messages 📊
 * **Hugging Face Integration**: Push generated datasets to the Hub 🤗
 
 !!! warning