AIP #61: AdEx v5: significantly cheaper channels

[elpiel]

Tracking issue

For the changes regarding AIP#61: AdEx v5: significantly cheaper channels (full details here)

Change log

• Channel & Campaign AIP #61 v5: Channel and Campaign #382 (PRs: AdEx v5: Issue #382 Channel and Campaign #383 )
• NewState & Accounting AIP #61 v5: NewState and Accounting #398
• Sentry

  • API documentation Sentry REST API documentation #475

    • AdEx V5: Init Sentry REST API documentation (using rustdoc) #443
    • Sentry REST API & functions documentation #473
    • Api routes documentation #524

  • Accounting & spenders AIP #61 v5: Spenders & Address accounting (postgres) #381

  • GET /v5/channel/:id/accounting AIP #61 v5: GET /v5/channel/:id/accounting #431 @simzzz

  • GET /v5/channel/:id/spender/:addr AIP #61 v5: GET /channel/:id/spender/:addr #391

  • GET /v5/channel/:id/spender/all AIP #61 v5: GET /channel/:id/spender/all #392

  • POST /v5/channel/:id/spender/:addr POST /v5/channel/:id/spender/:addr + tests #465 @simzzz

    • Change output to use SpenderResponse instead of just success: true Changed POST /v5/channel/:id/spender/:addr output to SpenderResponse #484 @simzzz

  • POST /v5/channel/:id/pay POST /v5/channel/:id/pay #477 @simzzz
    Channel Payout with authentication of the spender.
    Withdrawals of advertiser funds - re-introduces the PAY event with a separate route

  • GET /v5/channel/:id/get-leaf @simzzz (PR GET /v5/channel/:id/get-leaf route + tests #544)
    from AIP#61:

    add another helper route /get-leaf that is called like this: /get-leaf?spender=... or /get-leaf?earner=...; this will get the latest approved state (NewState/ApproveState pair), will find the given spender/earner in the balances tree, and produce a merkle proof for it; this is useful for the Platform to verify if a spender leaf really exists

    I've discussed this with Ivo, we will make spender/earner routes instead of using query parameters, e.g.:

    • /get-leaf/spender/0x....
    • /get-left/earner/0x...

  • POST /v5/campaign/:id/close AIP #61 v5 Sentry POST /v5/campaign/:id/close #417 (PR Issue 417 campaign close route #457)
    (newBudget = totalSpent, i.e. newBudget = oldBudget - remaining)

  • POST / PUT /v5/campaign (create / modify) AIP #61 v5: Channel and Campaign #382

    • update spender available funds when they try to mutate campaigns

  • GET /v5/units-for-slot Supermarket's units-for-slot route integration inside Sentry #471

  • Enforce limits ( PR Issue 382 campaign insert events #413 )

  • Events AIP #61 v5: Channel and Campaign #382 ( PR Issue 382 campaign insert events #413 )

    • enforce per-campaign spending limits atomically - we already do this in access

  • Distribute fees AIP #61 v5: Spenders & Address accounting (postgres) #381 ( PR Issue 382 campaign insert events #413 )

  • Analytics breakdowns by 1hr AIP #61 v5: Channel and Campaign #382

  • Campaign(s) remaining Campaign(s) remaining (redis) #415

• Validator
  • Worker flow Preparations for V5 validator workflow #426 V5 validator Leader/Follower ticks with timeouts & logging #433
    For details see JS issue AIP #61 v5: Validator Worker workflow adex-validator#360
    • worker will group the ticks (i.e. Accounting, NewState/ApproveState) per channel
    • leader/follower ticks V5 validator workflow - Follower & Leader ticks #428
    • Channel tick V5 validator Leader/Follower ticks with timeouts & logging #433
    • Tests @simzzz Leader/Follower tick tests + Heartbeat tests #490
  • Unified precision AIP #61 v5: Unified precision (8) #384 ( PR AdEx v5: Issue #384 Unified precision - primitives::UnifiedNum #386 Issue 384 unified precision #393 )
    • Targeting rules DSL - use UnifiedNum instead of BigNum (part of AIP #61 v5: Unified precision (8) #384)
  • Remove Fees (and move to Sentry) AIP #61 v5: Spenders & Address accounting (postgres) #381 ( PR Issue 382 campaign insert events #413 )
  • Outpace - spender & earner limits
    • sum(spenders) = sum(earners) - use the Balances<CheckedState>. The struct can be directly deserialized and will check the state of the Balances.
    • For Spender - Deposit > spent (use the necessary sentry routes to perform this validation)
• Adapter
  • Calculate total spendable amount AIP #61 v5: Adapter - total spendable amount #380
• Generic Deposit - instead of having adapter Deposit & spender Deposit, make the Deposit struct over a generic type ( impled in Channel v5 table #432 )
• AdView manager:
  • A way to test adview-manager generated code manually Initial AdView manager for serve-ing and testing generated code for ads #476 Adview manager serve ads preview #564 Infinity Behaviour + AdView Manager Tests #566
  • Make V5 changes
  • Finalize AIP#61: AdEx V5: adview-manager: AIP#31 Targeting changes #322

pub struct Deposit<T> {
    pub total: T,
    pub still_on_create2: T,
}

impl Deposit<UnifiedNum> {
   pub fn from_precision(precision: u8, deposit: Deposit<BigNum>) -> Option<Deposit<UnifiedNum>> {... }
}

• Remove SpenderLeaf Remove SpenderLeaf #442 @simzzz
• Integration tests - new mini test kit (see https://github.com/AdExNetwork/adex-validator/blob/v5/integrationTests.md)
  • PR Validator test harness - running sentry servers #437
    • A test_harness crate for running the tests
    • A way to run 2 sentry servers (Leader / Follower)
  • PR Validator stack - Running Workers tick & inserting events #454
    • A way to run both validator workers in single tick
    • Writing the test cases test_harness - additional multichain tests #491
  • Analytics AIP #61 v5: Analytics multi-chain support #494

deposit into a channel and create a campaign make sure it doesn't work before that submit events verify that payments appear in NewState/ApproveState, incl. fees verify that /analytics returns stuff

Remove old V4 routes that are no longer needed:

• Create Channel POST /channel Issue 382 Remove V4 Channel and fns cleanup #435
• Validator Channel POST /channel/validate Issue 382 Remove V4 Channel and fns cleanup #435
• Status of Channel GET /channel/:id/status Issue 382 Remove V4 Channel and fns cleanup #435
• Event aggregates GET channel/:id/events-aggregates (removed in Sentry REST API & functions documentation #473)

Sentry Authentication required (AuthRequired middleware) routes

• POST /v5/campaign (Auth is used to validate the campaign.creator)
• POST /v5/campaign/:id/close (Auth is used to validate the campaign.creator)
• POST /v5/channel/:id/spender/:addr
• GET /v5/channel/:id/spender/:addr
• POST /v5/channel/:id/spender/all
• GET channel/:id/events-aggregates (v4)
• POST channel/:id/validator-messages (v4)
• POST /v5/channel/:id/validator-messages
• /v5/channel/:id/pay
• /v5/analytics/for-publisher
• /v5/analytics/for-advertiser
• /v5/analytics/for-admin

[elpiel]

Here are a few questions and remarks on the current state of V5.
@Ivshti should take a look at the pending questions and we will discuss them together.

• PAY channel event (POST /v5/channel/:id/pay)
  • how will this event be handled in analytics and accounting?
  • How will this affect campaigns' remaining budget?
• A Channel is created implicitly when creating a Campaign and we don't have a route for creating channels. Validate if the following statement is the desirable outcome:
  • GET/POST /v5/channel/:id/spender/* will all return 404 if there is no Campaign with the given Channel.id. Updating the Spendable amount (POST /v5/channel/:id/spender/:addr) will not be executable in this case until a Campaign is created with the given Channel.
• On Campaign creation (POST /v5/campaign) in this case we perform:
  • Channel insertion in DB (if it does not exist)
  • Updating Spendable amount in the DB using the Adapter.
• On Campaign modification (POST /v5/campaign/:id) only when the budget is modified we also update the Spendable amount in the DB using the Adapter.
• How should we perform the GET /v5/channel/:id/get-leaf? Are these the correct steps to produce the given leaf:
  1. Generating a MerkleTree by getting the balance leaf of the spender/earner (see https://github.com/AmbireTech/adex-validator-stack-rust/blob/1bbb4e86a6797e80333c987abd784e623ebf8d06/adapter/src/lib.rs#L45-L70)
  2. making a singable state root for the Channel ( see https://github.com/AmbireTech/adex-validator-stack-rust/blob/1bbb4e86a6797e80333c987abd784e623ebf8d06/adapter/src/lib.rs#L28-L43)
  3. And finally hex encoding the result like we do with the Channel events
  • What should happen if a request does not include a spender or earner query parameter?
  • 400
• How should we be generating the SpenderLeaf merkle proof for GET /v5/channel/:id/spender/:addr & GET /v5/channel/:id/spender/all? We will drop this value from the response. API users will be forced to use get-leaf
• Should we keep the current behavior on GET /v5/campaign/list query parameter - isLeader?
  • query.validator is passed - filters campaign.channel.leader = query.validator
  • no query.validator but is an authenticated call (we have Bearer auth token) - filter by Campaign.channel.leader = Auth.uid
  • We will not keep this behavior. Instead use validator=0x... (for returning campaigns that have the validator as either leader or follower) & leader=0x... (returns only campaigns that the Channel.leader = 0x...)

[Ivshti]

• PAY event; the PAY event is outside the campaign, it is not associated with any campaign, and it affects Accounting directly without doing anything about the analytics; see "please note that we can't assume totalSpent equals the sum of all campaign spendings, because a spending can occur outside of campaign (cases like v4 CLOSE or PAY)"
• re non-existing channels, this behavior makes sense
• on campaign creation: makes sense
• when to update Spendable: also makes sense
• get-leaf: that's correct; if it doesn't conatin spender/earner, it's a 400 bad request
• sounds like the same procedure like get-leaf would be sufficient; actually, let's not return a leaf in this route for now, and force users to always call get-leaf
• this behavior is used by the validator when running through the campaigns to update them, so you decide whether to keep it depending on whether it's needed in the validator - I am pretty sure it is, so we'll need to keep it