feat: Remove market monitoring API and update documentation for identity analytics [DEV-4788] (#271)

* chore: Update docs & remove MM API && fix ledgerDenom querying

* fix linter error

* Fix querying by ledgerOperationType

* Update text

---------

Co-authored-by: Ankur Banerjee <ankurdotb@users.noreply.github.com>

Author: filipdjokic

README.md

@@ -120,28 +120,86 @@ Total account balance for specified account, in CHEQ.
 
 The standard Cosmos SDK REST API for account balances returns JSON with the account balances along with its denomination, usually the lowest denomination. This is hard to parse in applications such as Google Sheets (e.g., to monitor the account balance by fetching a response from a REST API directly in Google Sheets). This API returns a plain number that can be directly plugged into such applications, without having to parse JSON.
 
-### 🚨 Arbitrage
+### 📊 Identity Analytics
 
-#### Endpoint
+#### Endpoints
+
+##### Mainnet
+
+* **DID Documents**: [`data-api.cheqd.io/analytics/mainnet/did`](https://data-api.cheqd.io/analytics/mainnet/did)
+* **DID-Linked Resources (DLRs)**: [`data-api.cheqd.io/analytics/mainnet/resource`](https://data-api.cheqd.io/analytics/mainnet/resource)
+* **DIDs and DLRs combined** (harder to paginate): [`data-api.cheqd.io/analytics/mainnet`](https://data-api.cheqd.io/analytics/mainnet)
+
+##### Testnet
 
-- Results filtered by threshold value: [`data-api.cheqd.io/arbitrage`](https://data-api.cheqd.io/arbitrage)
-- Unfiltered results: [`data-api.cheqd.io/arbitrage/all`](https://data-api.cheqd.io/arbitrage/all)
+* **DID Documents**: [`data-api.cheqd.io/analytics/testnet/did`](https://data-api.cheqd.io/analytics/testnet/did)
+* **DID-Linked Resources (DLRs)**: [`data-api.cheqd.io/analytics/testnet/resource`](https://data-api.cheqd.io/analytics/mainnet/resource)
+* **DIDs and DLRs combined** (harder to paginate): [`data-api.cheqd.io/analytics/testnet`](https://data-api.cheqd.io/analytics/mainnet)
 
 #### Response
 
-Returns current price of CHEQ token among different markets along with an evaluation of whether they are at risk of arbitrage opportunities.
+Returns the identity transactions data for the specified network in JSON format.
+By default, the endpoint returns the data for the **last 30 days**.
+
+If `did` or `resource` is not specified, it will both DID and DLR analytics data, sorted by date in descending order.
+By default, the response is paginated with a limit of **100 results per page**. It's harder to paginate through responses on the combined endpoints, so where possible it is recommended to query DIDs and DLRs separately.
+
+#### Optional Query Parameters
+
+These optional query paramaters can be used to filter the results that are returned from the API.
+
+* `startDate`: Start date for the analytics data. If you provide *just* this parameter, the *current* date will be considered the end date.
+* `endDate`: End date for the analytics data. If you provide *just* this parameter, the start date will be considered as 30 days prior to this date.
+* `operationType`: Operation type for the analytics data. It accepts the following values:
+  * `createDid`
+  * `updateDid`
+  * `deactivateDid`
+  * `createResource`
+* `ledgerOperationType`: Ledger operation type for the analytics data. (Only relevant if the underlying operation types on ledger go through a breaking change in version number and you want to filter by specific operation type. Otherwise, use above.)
+  * `cheqd.did.v2.MsgCreateDid`
+  * `cheqd.did.v2.MsgUpdateDid`
+  * `cheqd.did.v2.MsgDeactivateDid`
+  * `cheqd.resource.v2.MsgCreateResource`
+* `denom`: Friendly denomination used for paying fees, e.g. `CHEQ` or `USDC`. Note that by default the results are returned in **main denomination**, i.e., the on-ledger value in `ncheq` is convert to CHEQ.
+* `ledgerDenom`: Ledger denomination used for paying fees, e.g. `ncheq` or `ibc/498A0751C798A0D9A389AA3691123DADA57DAA4FE165D5C75894505B876BA6E4`.
+* `feePayer`: Fee payer address, e.g., `cheqd1...xx`.
+* `success` (boolean; default: `true`): Whether transaction was successful or not. By default, only successful transactions are returned. Allows for specifically fetching failed transaction details.
+* `page` (default: `1`): Page number for pagination.
+* `limit` (default: `100`): Number of results per page.
 
 #### Rationale
 
-The CHEQ token trades on multiple markets/exchanges (e.g., [Osmosis](https://app.osmosis.zone), [Gate.io](https://www.gate.io/trade/CHEQ_USDT), [BitMart](https://www.bitmart.com/trade/en?layout=basic&symbol=CHEQ_USDT), [LBank](https://www.lbank.info/exchange/cheq/usdt), [Uniswap](https://app.uniswap.org/#/swap?inputCurrency=0x70edf1c215d0ce69e7f16fd4e6276ba0d99d4de7&outputCurrency=0xdac17f958d2ee523a2206206994597c13d831ec7&chain=mainnet)). This is typically established as CHEQ along with another token pair or currency.
+The purpose of this API is to provide more granular data and help us to understand the usage of identity modules and how they are used by the community.
 
-Fluctuations in the exchange rate between CHEQ and other tokens pairs can give rise to opportunities for arbitrage. Having a significant market arbitrage among different exchanges creates a [market inefficiencies](https://www.investopedia.com/terms/i/inefficientmarket.asp). Extreme market inefficiencies result [market failure](https://www.investopedia.com/terms/m/marketfailure.asp) and [deadweight loss](https://www.investopedia.com/terms/d/deadweightloss.asp).
+#### Example
 
-Having monitoring capabilities for arbitrage gives opportunities for the cheqd community to rectify potential liquidity issues and aware of exchange rate movements.
-
-#### Alerting via Zapier
+```bash
+curl -X GET "https://data-api.cheqd.io/analytics/mainnet?startDate=2024-01-01&endDate=2024-01-31&operationType=createDid&denom=CHEQ&&success=true&limit=1"
+```
 
-To alert a significant market arbitrages for CHEQ listings on different exchanges, we pull latest markets data from the [CoinGecko API for cheqd's ticker page](https://www.coingecko.com/en/coins/cheqd-network) via our Market Monitoring API [Monitor Markets API](https://github.com/cheqd/market-monitoring). If an arbitrage threshold is exceeded, a webhook trigger is sent to [Zapier](https://zapier.com/) for alerting via different channels (such as Slack).
+Response:
+
+```json
+{
+  "items": [
+    {
+      "didId": "did:cheqd:mainnet:f1a32c8b-bd76-450d-b5c2-46b1b92db74d",
+      "operationType": "createDid",
+      "feePayer": "cheqd15yqrljq6h7u8et30hqrwwaz8f9m62jvztlpc9t",
+      "amount": 50,
+      "denom": "CHEQ",
+      "blockHeight": "11709907",
+      "transactionHash": "36EEBB3E4551CC9BB1A70FACB9775AB8DD12F5ABF931A60322558C03F91822B9",
+      "createdAt": "2024-01-25T19:49:35.093Z",
+      "success": true
+    }
+  ],
+  "totalCount": 7,
+  "page": 1,
+  "limit": 1,
+  "totalPages": 7
+}
+```
 
 ## 🧑‍💻🛠 Developer Guide
 

src/handlers/allArbitrageOpportunities.ts

@@ -1,11 +1,9 @@
 import { updateCirculatingSupply } from '../helpers/circulating';
-import { filterArbitrageOpportunities } from './arbitrageOpportunities';
 import { Network } from '../types/network';
 import { syncNetworkData } from '../helpers/identity';
 
 export async function webhookTriggers(env: Env) {
 	console.log('Triggering webhook...');
-	await sendPriceDiscrepancies(env);
 
 	await updateCirculatingSupply(getHour(), env);
 	await syncIdentityData(env);
@@ -22,35 +20,6 @@ export async function syncIdentityData(env: Env) {
 	}
 }
 
-export async function sendPriceDiscrepancies(env: Env) {
-	try {
-		console.log('Sending price discrepancies...');
-
-		const arbitrageOpportunities = await filterArbitrageOpportunities(env);
-		const hasArbitrageOpportunities = arbitrageOpportunities.length > 0;
-		if (hasArbitrageOpportunities) {
-			console.log('Arbitrage opportunities...');
-			try {
-				const init = {
-					body: JSON.stringify({
-						arbitrage_opportunities: arbitrageOpportunities,
-					}),
-					method: 'POST',
-					headers: {
-						'content-type': 'application/json;charset=UTF-8',
-					},
-				};
-
-				await fetch(env.WEBHOOK_URL, init);
-			} catch (err: any) {
-				console.log(err);
-			}
-		}
-	} catch (e) {
-		console.log('Error at: ', 'sendPriceDiscrepancies');
-	}
-}
-
 function getHour(): number {
 	// This function only works when CIRCULATING_SUPPLY_GROUPS is set to 24
 	let hour = Number(new Date().getHours() + 1); // getHours() returns 0-23

src/handlers/arbitrageOpportunities.ts

@@ -14,7 +14,7 @@ import { DrizzleClient } from '../database/client';
 import { serializeBigInt } from './csv';
 import { Network } from '../types/network';
 import { validateDateRange } from './validate';
-
+import { DenomType, OperationType } from '../types/bigDipper';
 export const DEFAULT_LIMIT = 100;
 
 // Network-based table mapping
@@ -59,11 +59,11 @@ export function buildQueryConditions(
 	}
 
 	if (params.ledgerOperationType !== null) {
-		conditions.push(ilike(operationTypesTable.ledgerOperationType, params.ledgerOperationType));
+		conditions.push(eq(operationTypesTable.ledgerOperationType, params.ledgerOperationType as OperationType));
 	}
 
 	if (params.ledgerDenom !== null) {
-		conditions.push(ilike(denomTable.ledgerDenom, params.ledgerDenom));
+		conditions.push(eq(denomTable.ledgerDenom, params.ledgerDenom as DenomType));
 	}
 
 	if (params.denom !== null) {

src/handlers/webhookTriggers.ts

@@ -6,15 +6,11 @@ import { handler as liquidBalanceHandler } from './handlers/liquidBalance';
 import { handler as vestingBalanceHandler } from './handlers/vestingBalance';
 import { handler as vestedBalanceHandler } from './handlers/vestedBalance';
 import { handler as totalStakedCoinsHandler } from './handlers/totalStakedCoins';
-import { handler as allArbitrageOpportunitiesHandler } from './handlers/allArbitrageOpportunities';
-import { handler as arbitrageOpportunitiesHandler } from './handlers/arbitrageOpportunities';
 import { webhookTriggers } from './handlers/webhookTriggers';
 import { registerAnalyticsRoutes } from './routes/analytics';
 
 function registerRoutes(router: ReturnType<typeof Router>, env: Env, ctx: ExecutionContext) {
 	router.get('/', (request) => totalSupplyHandler(request, env));
-	router.get('/arbitrage', (request) => arbitrageOpportunitiesHandler(request, env));
-	router.get('/arbitrage/all', (request) => allArbitrageOpportunitiesHandler(request, env));
 	router.get('/balances/liquid/:address', (request) => liquidBalanceHandler(request, env));
 	router.get('/balances/total/:address', (request) => totalBalanceHandler(request, env));
 	router.get('/balances/vested/:address', (request) => vestedBalanceHandler(request, env));

src/helpers/analytics.ts

@@ -7,7 +7,6 @@ declare global {
 		GRAPHQL_API: string;
 		TESTNET_GRAPHQL_API: string;
 		CIRCULATING_SUPPLY_GROUPS: string;
-		MARKET_MONITORING_API: string;
 		ENVIRONMENT: string;
 		BIGDIPPER_URL: string;
 		WEBHOOK_URL: string;

src/index.ts

@@ -53,7 +53,6 @@ TESTNET_GRAPHQL_API = "https://testnet-gql.cheqd.io/v1/graphql"
 # Number of groups circulating supply watchlist is split into
 CIRCULATING_SUPPLY_GROUPS = "24"
 # Market monitoring API endpoint
-MARKET_MONITORING_API = "https://market-monitoring.cheqd.net"
 ENVIRONMENT = "production"
 
 # The necessary secrets are:
@@ -103,7 +102,7 @@ route = { pattern = "data-api-staging.cheqd.io/*", zone_id = "afe3b66243382f2714
 
 # Map of environment variables to set when deploying the Worker
 # Not inherited. @default `{}`
-vars = { ENVIRONMENT = "staging", TOKEN_EXPONENT = 9, REST_API = "https://api.cheqd.net", REST_API_PAGINATION_LIMIT = "50", GRAPHQL_API = "https://explorer-gql.cheqd.io/v1/graphql", CIRCULATING_SUPPLY_GROUPS = "24", MARKET_MONITORING_API = "https://market-monitoring-staging.cheqd.net", TESTNET_GRAPHQL_API = "https://testnet-gql.cheqd.io/v1/graphql" }
+vars = { ENVIRONMENT = "staging", TOKEN_EXPONENT = 9, REST_API = "https://api.cheqd.net", REST_API_PAGINATION_LIMIT = "50", GRAPHQL_API = "https://explorer-gql.cheqd.io/v1/graphql", CIRCULATING_SUPPLY_GROUPS = "24", TESTNET_GRAPHQL_API = "https://testnet-gql.cheqd.io/v1/graphql" }
 
 # The necessary secrets are:
 # - WEBHOOK_URL