crypto-data-collector

This project fetches real-time and historical cryptocurrency prices using the CoinGecko API, stores the data in a Hyperbee database, and exposes the data through Hyperswarm RPC.

---

📦 Setup Instructions

1. Clone the repository

git clone git@github.com:PrashantHalaki/pears_assignment.git

2. Change into the project directory

cd pears_assignment

3. Install dependencies

npm install

---

⚙️ Prerequisites

4. Install Pear (if not installed)

Follow the official Pear installation guide:

👉 https://docs.pears.com/guides/getting-started

5. Install HyperDHT globally (if not already installed)

npm install -g hyperdht

---

🚀 Running the Project

6. Start the HyperDHT Bootstrap Node

This command sets up the bootstrap node. Run it in a separate terminal and keep it active:

hyperdht --bootstrap --host 127.0.0.1 --port 30001

7. Start the RPC server (data loader + server)

This starts the data collection pipeline, loads prices into Hyperbee, and starts the RPC server. The public key will be displayed in the console.

npm run dev

Example output:

⏰ Scheduled run: fetching crypto prices...
Starting RPC server...
RPC Server started on public key: <RPC Server Key>

8. Start the RPC client

In a new terminal, use the public key generated from the previous step:

npm run client <RPC_PUBLIC_KEY>

---

💻 RPC Client Commands

Once inside the RPC client terminal:

1. Fetch latest prices

latest btc,eth

Fetches the latest average prices for the listed coins.

---

2. Fetch historical prices

historical btc,eth 2023-01-01 2023-01-31

Returns historical prices between the provided date range.

---

3. Close the client session

close

Terminates the client gracefully.

---

📌 Notes

• Data is fetched from CoinGecko's public API every 30 seconds.
• Top 5 cryptocurrencies (by market cap) are monitored.
• For each currency, prices from the top 3 exchanges are averaged.
• Data is stored in Hypercore using Hyperbee.
• Communication is enabled through Hyperswarm RPC and HyperDHT.

---

🛠️ Project Structure

pears_assignment/
│
├── db/                         # DB related reusable files
│   ├── db.js                   # Hypercore and Hyperbee instance
│   ├── dht.js                  # Hyperdht instance
│   ├── getPrices.js            # getLatestPrices and getHistoricalPrices functions to read data from db
│   ├── index.js                # file to export reusable functions from db
│   ├── storePrices.js          # storePrices function to write data from db
│
├── config/                     # Configurations
│   ├── index.js                # All the envs and constants are available here
│
├── data/                       # Persistent local storage for Hypercore/Hyperbee
│
├── test/                       # Test cases folder
│   ├── index.test.js           # Empty test file TODO: Write unit test cases
│
│
├── src/
│   ├── server/                 # RPC server logic
│   │   ├── index.js            # Entry point for RPC server
│   │
│   ├── client/                 # RPC client to query server
│   │   └── index.js            # Sample client requesting prices
│   │
│   ├── collector/              # Data pipeline logic
│   │   ├── priceFetcher.js     # Fetches top 3 exchanges from CoinGecko API for each token
│   │   ├── topCoins.js         # Fetches top 5 usdt coins
│   │   └── index.js            # Fetches all prices
│   │
│   ├── job/                    # Data pipeline calling jobs
│   │   ├── index.js            # exports jobs and runs pipeline on demand when --once argument is passed
│   │   ├── pipeline.js         # Logic to call the store data
│   │   └── schedule.js         # Logic to schedule the data loader for defined time frame
│   │
│   └── utils/                  # Helper functions
│       ├── axiosInterface.js   # Axios interface to call coingeko APIs
│       └── sleep.js            # Sleep function to make the functions sleep for defined time frame
│
├── .env.example                # Example of all environmental variables in the app
├── index.js                    # Start of the app - Initializing scheduler and RPC server
├── .gitignore
├── package.json
├── package.lock.json
└── README.md

---

✅ Example Workflow

# Terminal 1
hyperdht --bootstrap --host 127.0.0.1 --port 30001

# Terminal 2
npm run dev

# Terminal 3
npm run client <RPC_PUBLIC_KEY>

---

Uncompleted Tasks – Pears Assignment

This document outlines the features and improvements that could not be completed within the given timeline. The reasons for each pending task are described below.

---

1. Axios Instance Issue

• Problem: The Axios instance used to fetch data from the CoinGecko public API throws a MODULE_NOT_FOUND: Cannot find module 'crypto' error when executed using pear run.
• Reason: Pear runtime has limited documentation and compatibility for third-party Node.js modules like Axios that rely on Node.js core modules (crypto in this case).
• Status: Raised the concern with provided contacts but haven’t received a response yet.
• Resolution: If any guidelines or examples are shared on making external HTTP requests via Pear, this can be fixed in a few minutes.

---

2. Unit Test Cases

• Plan: The intent was to first complete the entire pipeline (data collection, storage, scheduling, RPC setup) before adding unit tests.
• Status: Couldn’t add test cases due to tight time constraints.
• Note: The code is modular and can easily be covered with tests using frameworks like Jest or Mocha.

---

3. Scheduling Logic

• Current Approach: The data loading job is scheduled using a setTimeout-based recursive approach to simulate a 30-second interval.
• Reason: node-cron did not work in the Pear runtime environment.
• Recommendation: If Pear supports any built-in scheduling utilities or has a compatible cron-like tool, this logic can be replaced easily.

---

4. Code Explanation

• Missing Element: Due to time constraints, detailed inline comments and a structured code walkthrough are not included.
• Compensation: The variable names and function definitions follow clear and intuitive naming conventions, making the logic easy to follow for developers.

---

Summary

These pending tasks do not affect the core functionality. With slight documentation and runtime support, the issues mentioned above can be addressed quickly.

📝 License

© Prashant Halaki