Merge pull request #25 from mkaramuk/main

chore: peerBenchJS release 0.0.15

Author: Leibniz-01

peerBenchJS/packages/sdk/README.md

@@ -1,3 +1,3 @@
-# peerBench
+# peerBench SDK
 
-TypeScript SDK for peerBench framework
+A TypeScript SDK for building AI evaluation benchmarks and data processing pipelines.

peerBenchJS/packages/sdk/docs/implementing-a-collector.md

@@ -0,0 +1,204 @@
+# Implementing a Collector
+
+This guide demonstrates how to implement a new Collector for the peerBench SDK.
+
+## Overview
+
+Collectors are responsible for taking a source input and creating a structured data that later can be used by the Generators to generate new Prompts.
+
+## Basic Example
+
+Here's a simple collector implementation that demonstrates the basic structure:
+
+```typescript
+import { AbstractCollector } from "@/collectors/abstract/abstract-collector";
+
+interface MyCollectedData {
+  id: string;
+  title: string;
+  content: string;
+  metadata: Record<string, any>;
+}
+
+export class SimpleAPICollector extends AbstractCollector<MyCollectedData[]> {
+  readonly identifier = "simple-api-collector";
+
+  async collect(
+    source: unknown,
+    options?: Record<string, any>
+  ): Promise<MyCollectedData[] | undefined> {
+    // Type guard for input validation
+    if (typeof source !== "string") {
+      throw new Error("Source must be a string URL");
+    }
+
+    // Fetch data from external source
+    const response = await fetch(source);
+    if (!response.ok) {
+      throw new Error(`HTTP ${response.status}: ${response.statusText}`);
+    }
+
+    // Parse and transform the response
+    const rawData = await response.json();
+    return this.transformData(rawData);
+  }
+
+  private transformData(rawData: any): MyCollectedData[] {
+    return rawData.map((item: any) => ({
+      id: item.id,
+      title: item.title,
+      content: item.content,
+      metadata: item.metadata || {},
+    }));
+  }
+}
+```
+
+### Required Properties
+
+**`readonly identifier: string`**
+
+- A unique string that identifies your Collector
+- Should be descriptive and unique across all Collectors
+- Useful when you try to find this Collector among the others
+
+### Abstract Methods
+
+**`async collect(source: unknown, options?: Record<string, any>): Promise<T | undefined>`**
+
+- This is the main collection method you must implement
+- Takes a `source` as the input
+- Interpretation of what `source` parameter is depends on the implementation
+- For example if it is a URL, you should validate that it is a real URL string and fetch data from it
+- Or if it is a file path, you should be checking that file is exist
+- Optional `options` parameter for configurable collection behavior
+- Must return data matching your Collector's output type `T`, or `undefined` if the process fails
+- The generic type `T` represents what your collector **outputs**
+
+## Advanced Patterns
+
+### RSS Feed Collector
+
+For RSS feeds, extend `AbstractRSSCollector` instead:
+
+```typescript
+import { AbstractRSSCollector } from "@/collectors/abstract/abstract-rss-collector";
+import { z } from "zod";
+
+export class MyRSSCollector extends AbstractRSSCollector<MyRSSData[]> {
+  readonly identifier = "my-rss-collector";
+
+  // Define the expected RSS structure using Zod
+  feedSchema = z.object({
+    rss: z.object({
+      channel: z.object({
+        title: z.string(),
+        item: z.array(
+          z.object({
+            title: z.string(),
+            description: z.string(),
+            link: z.string(),
+            pubDate: z.string(),
+          })
+        ),
+      }),
+    }),
+  });
+
+  async collect(url: string): Promise<MyRSSData[] | undefined> {
+    const feed = await this.parseFeedXML(await this.fetchFeed(url));
+
+    // Process the validated RSS data
+    return feed.rss.channel.item.map((item) => ({
+      title: item.title,
+      description: item.description,
+      link: item.link,
+      publishedAt: new Date(item.pubDate),
+    }));
+  }
+}
+```
+
+### Collector with Authentication
+
+For APIs requiring authentication:
+
+```typescript
+export class AuthenticatedAPICollector extends AbstractCollector<MyData[]> {
+  readonly identifier = "authenticated-api";
+
+  async collect(
+    source: unknown,
+    options: {
+      includeSensitiveContent?: boolean;
+      limit?: number;
+      sortBy?: "relevance" | "date" | "popularity";
+      language?: string;
+    } = {}
+  ): Promise<MyData[] | undefined> {
+    if (typeof source !== "string") {
+      throw new Error("Source must be a search query string");
+    }
+
+    // Build the API request parameters using options
+    const params = new URLSearchParams();
+    params.set("q", source);
+
+    // Apply options to customize the API request
+    if (options.includeSensitiveContent) {
+      params.set("sensitive", "true");
+    }
+    if (options.limit) {
+      params.set("limit", options.limit.toString());
+    }
+    if (options.sortBy) {
+      params.set("sortBy", options.sortBy);
+    }
+    if (options.language) {
+      params.set("language", options.language);
+    }
+
+    const headers = {
+      Authorization: `Bearer ${this.apiKey}`,
+      "Content-Type": "application/json",
+    };
+
+    const response = await fetch(
+      `${this.baseUrl}/search?${params.toString()}`,
+      { headers }
+    );
+
+    if (!response.ok) {
+      throw new Error(`API request failed: ${response.status}`);
+    }
+
+    const data = await response.json();
+    return this.transformData(data, options.limit);
+  }
+
+  private transformData(rawData: any, limit?: number): MyData[] {
+    let results = rawData.results?.map(this.mapToMyData) || [];
+
+    if (limit && results.length > limit) {
+      results = results.slice(0, limit);
+    }
+
+    return results;
+  }
+}
+```
+
+## Examples
+
+See the `examples/collectors/` directory for complete working examples:
+
+- `file-system-collector.ts` - Local file collection
+- `news-api-collector.ts` - API-based collection
+
+These examples demonstrate real-world implementations and can serve as templates for your own collectors.
+
+## What's Next?
+
+Now that you understand how to implement a Collector, you're ready to learn about the next component in the peerBench SDK: **Generators**.
+
+**Next Documentation**: [Implementing a Generator](./implementing-a-generator.md)

peerBenchJS/packages/sdk/docs/implementing-a-generator.md

@@ -0,0 +1,117 @@
+# Implementing a Generator
+
+This guide demonstrates how to implement a new Generator for the peerBench SDK.
+
+## Overview
+
+Generators are responsible for taking the collected data from a Collector and generate one or more Prompts based on that data.
+
+## Basic Example
+
+Here's a simple generator implementation that demonstrates the basic structure:
+
+```typescript
+import { AbstractGenerator } from "@/generators/abstract/abstract-generator";
+import { Prompt, PromptTypes } from "@/types";
+import { z } from "zod";
+
+export class SimpleQuestionGenerator extends AbstractGenerator {
+  readonly identifier = "simple-question-generator";
+
+  inputSchema = z.object({
+    id: z.string(),
+    content: z.string(),
+    metadata: z.record(z.any()),
+  });
+
+  protected async generatePrompts(
+    input: z.infer<(typeof this)["inputSchema"]>,
+    options?: Record<string, any>
+  ): Promise<Prompt[]> {
+    // Transform validated input into Prompts
+    const question = `Answer the following question: ${input.content}`;
+    const fullPrompt = question;
+
+    // Use the buildPrompt helper method for Prompt creation
+    const prompt = await this.buildPrompt({
+      question,
+      correctAnswer: "Your expected answer here",
+      fullPrompt,
+      type: PromptTypes.Text,
+      metadata: input.metadata,
+    });
+
+    return [prompt];
+  }
+}
+```
+
+### Required Properties
+
+**`readonly identifier: string`**
+
+- A unique string that identifies your Generator
+- Should be descriptive and unique across all Generators
+- Useful when you try to find this Generator among the others
+
+**`inputSchema: z.ZodSchema`**
+
+- A Zod schema that validates the input data of the `generatePrompts` method
+- If you want to process data from different Collectors, simply you need to make this schema compatible with the output types of those Collectors
+- Base class handles the validation of the input so you just need to define the schema
+
+### Abstract Methods
+
+**`protected async generatePrompts(input: z.infer<(typeof this)["inputSchema"]>, options?: Record<string, any>): Promise<Prompt[]>`**
+
+- This is the main generation method you must implement
+- Takes validated input data that matches your `inputSchema`
+- Optional `options` parameter for configurable generation behavior
+- Must return an array of `Prompt` objects
+- The input is already validated by the base class
+
+## Type Compatibility Between Collectors and Generators
+
+The key to making Collectors and Generators work together is ensuring schema compatibility. When you define a Generator, the `inputSchema` specifies what input data structure the Generator expects.
+
+**Example**: If you have a Collector that outputs data matching your Generator's `inputSchema`, they are compatible:
+
+```typescript
+// Collector outputs data matching the schema
+export class FileCollector extends AbstractCollector<FileData[]> {
+  // ... implementation
+}
+
+// Generator expects data matching inputSchema
+export class QuestionGenerator extends AbstractGenerator {
+  inputSchema = z.object({
+    id: z.string(),
+    content: z.string(),
+    metadata: z.record(z.any()),
+  });
+
+  // ... implementation
+}
+
+// These can be used together if the Collector output matches the Generator's inputSchema
+const collector = new FileCollector();
+const generator = new QuestionGenerator();
+
+const collectedData = await collector.collect("path/to/file");
+const prompts = await generator.generate(collectedData); // Schema validation ensures compatibility
+```
+
+## Examples
+
+See the `examples/generators/` directory for complete working examples:
+
+- `multiple-choice-generator.ts` - Multiple choice question generation
+- `sentence-ordering-generator.ts` - Sentence ordering task generation
+
+These examples demonstrate real-world implementations and can serve as templates for your own Generators.
+
+## What's Next?
+
+Now that you understand how to implement a Generator, you're ready to learn about the next component in the peerBench SDK: **Scorers**.
+
+**Next Documentation**: [Implementing a Scorer](./implementing-a-scorer.md)