-
Notifications
You must be signed in to change notification settings - Fork 67
feat: Experiment Center #1332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
feat: Experiment Center #1332
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
009ecd4
feat: Experiment Center
BcnCarlos 6587f3f
update
BcnCarlos eec90f3
Update
BcnCarlos bc45c67
update
BcnCarlos 2b0be56
Update
BcnCarlos e193c4a
Update
BcnCarlos d03911d
Update
BcnCarlos f625e3c
Update
BcnCarlos cdc57c5
update
BcnCarlos f83c0d3
update
BcnCarlos 61b99db
docs(experiment-center): apply review fixes; add Bucket C to-do
mswanson 5aab243
Update
BcnCarlos 103c5c7
Update
BcnCarlos 01e4942
Update
BcnCarlos 29d7699
update
BcnCarlos b4721d0
update
BcnCarlos 7d42e7e
Update
BcnCarlos c7e4ada
Update
BcnCarlos b06b385
Update
BcnCarlos ae8fb7d
Update
BcnCarlos 94857bf
Update
BcnCarlos 60f54b8
Update
BcnCarlos b263af5
update
BcnCarlos 6eb73f3
Update
BcnCarlos f267618
Update main/docs/customize/experiment-center/overview.mdx
BcnCarlos 05c4890
Update main/docs/customize/experiment-center/overview.mdx
BcnCarlos 09c3940
Update main/docs/customize/experiment-center/overview.mdx
BcnCarlos 3b045b9
Update main/docs/customize/experiment-center/overview.mdx
BcnCarlos 76ae842
Update main/docs/customize/experiment-center/overview.mdx
BcnCarlos e676d9c
Update main/docs/customize/experiment-center/entities.mdx
BcnCarlos 805a92b
Update main/docs/customize/experiment-center/entities.mdx
BcnCarlos ee38b75
Update main/docs/customize/experiment-center/entities.mdx
BcnCarlos 9927a85
Update main/docs/customize/experiment-center/integrations/page-templa…
BcnCarlos 6c23d13
Apply suggestion from @lrzhou25
BcnCarlos ecea94c
Apply suggestion from @lrzhou25
BcnCarlos ed3850a
Apply suggestion from @lrzhou25
BcnCarlos 0bc482f
Apply suggestion from @lrzhou25
BcnCarlos 159e77f
Apply suggestion from @lrzhou25
BcnCarlos 07ad4fd
Apply suggestion from @lrzhou25
BcnCarlos f5908dd
Apply suggestion from @lrzhou25
BcnCarlos 71aba59
Apply suggestion from @lrzhou25
BcnCarlos 6735e7c
Apply suggestion from @lrzhou25
BcnCarlos 2ab5a01
Apply suggestion from @lrzhou25
BcnCarlos 8233695
Apply suggestion from @lrzhou25
BcnCarlos 50e7d86
Apply suggestion from @lrzhou25
BcnCarlos 05a97ca
Apply suggestion from @lrzhou25
BcnCarlos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,152 @@ | ||
| --- | ||
| title: "Entities Details" | ||
| description: "Learn about Experiment Center entities: feature flags, variations, segments, experiments, and assignments." | ||
| sidebarTitle: Entities Details | ||
| --- | ||
|
|
||
| import { ReleaseStageNotice } from "/snippets/ReleaseStageNotice.jsx" | ||
|
|
||
| <ReleaseStageNotice | ||
| feature="Auth0 Experiment Center" | ||
| stage="beta" | ||
| terms="true" | ||
| contact="Auth0 Support" | ||
| /> | ||
|
|
||
| With the Auth0 [Management API](/docs/api/management/v2), you can define Experiment Center entity details, including properties, lifecycle states, and validation rules. | ||
|
|
||
| <Warning> | ||
| During Beta, Experiment Center runs on development tenants only. Production tenants are not supported. | ||
| </Warning> | ||
|
|
||
| ## Experiment | ||
|
|
||
| ### Experiment lifecycle | ||
|
|
||
| Experiments have five states: | ||
|
|
||
| | **Status** | **Meaning** | | ||
| |---|---| | ||
| | `draft` | Created but not running. Safe to test using query parameter overrides. | | ||
| | `active` | Running. Variant assignment and context injection are live for all auth transactions. | | ||
| | `paused` | Temporarily suspended. No new assignments. In-flight sessions keep their assigned variation. | | ||
| | `completed` | Concluded. No new assignments. Configuration retained for reference during manual promotion. | | ||
| | `archived` | Soft-deleted. Hidden from default list views. Analytics data retained. | | ||
|
|
||
| To transition lifecycles, use the Management API [`/api/v2/experimentation/experiments/{id}/status`](/docs/api/management/v2/experimentation/update-experiment-status) endpoint. | ||
|
|
||
| <Callout icon="file-lines" color="#0EA5E9" iconType="regular"> | ||
| Transitioning from `completed → active` is not allowed. If you need to run the same test again, create a new experiment. | ||
| </Callout> | ||
|
|
||
| ### The is_valid gate | ||
|
|
||
| Before an experiment can activate, it must pass a readiness check. | ||
|
|
||
| The Experiment Center stores an `is_valid` boolean on every experiment and recalculates it on every write. This gives you immediate feedback during setup. | ||
|
|
||
| You can also determine readiness explicitly using the Management API [`/api/v2/experimentation/experiments/{id}/validate`](/docs/api/management/v2/experimentation/validate-experiment) endpoint. | ||
|
|
||
| This returns `is_valid: true/false` and an `errors` array listing every blocker. The same check runs automatically when you attempt to activate via the status endpoint. | ||
|
|
||
| **Validation rules checked on activation:** | ||
|
|
||
| - Referenced feature flag is in `active` status | ||
| - At least one allocation exists | ||
| - All allocation `variation_id` values belong to the experiment's feature flag | ||
| - Exactly one allocation has `is_control: true` | ||
| - Allocation weights sum to 100 (percentage strategy) | ||
| - Exactly one allocation has `is_fallback: true` (segment strategy) | ||
| - No other experiment is currently active for the tenant (Beta only) | ||
|
|
||
| ### Assignment | ||
|
|
||
| **Query parameter overrides** | ||
|
|
||
| You can force a specific assignment on any `/authorize` request by passing query parameters: | ||
|
|
||
| | Parameter | Description | | ||
| |---|---| | ||
| | `experiment_id` | Enroll in a specific experiment | | ||
| | `variation_id` | Force a specific variation (requires `experiment_id`) | | ||
| | `segment_id` | Force a specific segment (requires `experiment_id`) | | ||
|
|
||
| This works for experiments in any status, including `draft`. Use this during development to verify that both variations render correctly before activating. | ||
|
|
||
| ## Feature flag | ||
|
|
||
| ### Feature flag lifecycle | ||
|
|
||
| Feature flags have a stored lifecycle with three states: | ||
|
|
||
| | **Status** | **Meaning** | | ||
| |---|---| | ||
| | `draft` | Created but not yet active. Cannot be referenced by an active experiment. | | ||
| | `active` | Ready for use. Required before any experiment that references this flag can activate. | | ||
| | `archived` | Terminal. No new experiments can reference this flag. Create a new flag if needed. | | ||
|
|
||
| To transition between statuses, use the Auth0 Management API [`/api/v2/experimentation/feature-flags/{id}/status`](/docs/api/management/v2/experimentation/update-feature-flag-status) endpoint. | ||
|
|
||
| **Activation gate:** A feature flag cannot transition to `active` until it has at least two variations. This ensures every active flag has at minimum a control and one treatment before it goes live. | ||
|
|
||
| ```text wrap lines | ||
| draft → active (requires ≥2 variations) | ||
| draft → archived | ||
| active → archived (terminal; no way back) | ||
| ``` | ||
|
|
||
| <Callout icon="file-lines" color="#0EA5E9" iconType="regular"> | ||
| Transitioning from `archived → active` is not allowed. If you archive a flag and need to run another test, create a new feature flag. | ||
| </Callout> | ||
|
|
||
| ### Variation | ||
|
|
||
| **Parameters and overrides** | ||
|
|
||
| Feature flag parameters define the configuration surface for your experiment. Each parameter has: | ||
|
|
||
| - A name (for example, `show_passkey_prompt`) | ||
| - A type: `string`, `boolean`, `number`, `array`, or `object` | ||
| - A default value (the baseline) | ||
|
|
||
| A variation's `overrides` specifies only the parameters that differ. At runtime, the Experiment Center merges the flag's baseline with the variation's overrides and delivers the full merged `config` object to ACUL, Actions, and page templates. Every parameter always has a value in `config`; you never need to write fallback logic. | ||
|
|
||
| <Callout icon="file-lines" color="#0EA5E9" iconType="regular"> | ||
| Parameters use **structured mode only**. Each parameter has a named key and a typed value. | ||
| </Callout> | ||
|
|
||
| ## Segment | ||
|
|
||
| ### Segment rules | ||
|
|
||
| A segment's `rules` is an array of rule objects. A request matches the segment if **any** rule in the array matches. A rule matches if its conditions satisfy the `match_type`: | ||
|
|
||
| - `match_type: "all"`: all conditions must be true (AND logic) | ||
| - `match_type: "any"`: at least one condition must be true (OR logic) | ||
|
|
||
| Each condition compares an attribute against a value using an operator: `equals`, `not_equals`, `in`, `not_in`, `exists`, `not_exists`. | ||
|
|
||
| ### Available condition attributes | ||
|
|
||
| Segments can only use attributes available at `/authorize` transaction start: | ||
|
|
||
| | **Category** | **Attributes** | | ||
| |---|---| | ||
| | Client | `client_id` | | ||
| | Connection | `connection`, `connection_type` | | ||
| | Organization | `organization_id` | | ||
| | Domain | `domain` | | ||
| | Device and browser | `device_type`, `browser`, `platform`, `user_agent` | | ||
| | Geographic (IP-derived) | `country`, `region` | | ||
|
|
||
| ## Learn more | ||
|
|
||
| <CardGroup cols={2}> | ||
| <Card title="Experiment Center Quickstart" icon="gear" href="/docs/customize/experiment-center/quickstart"> | ||
| Walk through creating a feature flag, variations, and an experiment end to end. | ||
| </Card> | ||
|
|
||
| <Card title="ACUL Integration" icon="puzzle-piece" href="/docs/customize/experiment-center/integrations/acul-integration-guide"> | ||
| Read experiment context inside Auth0 Custom Universal Login (ACUL) screens. | ||
| </Card> | ||
| </CardGroup> |
182 changes: 182 additions & 0 deletions
182
main/docs/customize/experiment-center/integrations/actions-integration-guide.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,182 @@ | ||
| --- | ||
| title: "Experiment Center and Auth0 Actions Integration" | ||
|
BcnCarlos marked this conversation as resolved.
|
||
| description: "How to read experiment context in Auth0 Actions and branch logic based on the assigned variation." | ||
| sidebarTitle: Actions Integration | ||
| --- | ||
|
|
||
| import { ReleaseStageNotice } from "/snippets/ReleaseStageNotice.jsx" | ||
|
|
||
| <ReleaseStageNotice | ||
| feature="Auth0 Experiment Center" | ||
| stage="beta" | ||
| terms="true" | ||
| contact="Auth0 Support" | ||
| /> | ||
|
|
||
| When an experiment is active, the Experiment Center injects the `ExperimentContext` object into supported Action triggers. | ||
|
|
||
| <Warning> | ||
| During Beta, Experiment Center runs on development tenants only. Production tenants are not supported. | ||
| </Warning> | ||
|
|
||
| ## Supported triggers | ||
|
|
||
| Experiment context is available for the following Auth0 [Actions](/docs/customize/actions/actions-overview): | ||
|
|
||
| * [`post_login`](/docs/customize/actions/explore-triggers/signup-and-login-triggers/login-trigger) trigger. | ||
| * [`pre_user_registration`](/docs/customize/actions/explore-triggers/signup-and-login-triggers/pre-user-registration-trigger) trigger. | ||
| * [`post_user_registration`](/docs/customize/actions/explore-triggers/signup-and-login-triggers/post-user-registration-trigger) trigger. | ||
|
|
||
|
|
||
| ## The event.experiment object | ||
|
|
||
| In supported triggers, the Experiment Center adds an `experiment` field to the event object: | ||
|
|
||
| ```typescript | ||
| interface ExperimentContext { | ||
| experiment_id: string; // The active experiment ID | ||
| variation_id: string; // The assigned variation ID | ||
| config: { // Merged configuration: baseline + overrides | ||
| [paramName: string]: { value: unknown }; | ||
| }; | ||
| is_control: boolean; // True when this is the control variation | ||
| } | ||
| ``` | ||
|
|
||
| When no experiment is active (or when the feature is not enabled for the tenant), `event.experiment` is `null` (not `undefined`). | ||
|
|
||
| ### config object contains the full merged configuration | ||
|
|
||
| The `config` object contains every parameter defined on the feature flag, merged with the assigned variation's overrides. You never need to look up baseline values or write fallback logic. If the parameter exists on the feature flag, it exists in `config`. | ||
|
|
||
|
|
||
| ## Null-safety pattern | ||
|
|
||
| Check for an active experiment before reading any properties: | ||
|
|
||
| ```javascript | ||
| exports.onExecutePostLogin = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; // No active experiment; nothing to do | ||
|
|
||
| // Safe to access ec.config, ec.variation_id, ec.is_control here | ||
| }; | ||
| ``` | ||
|
|
||
| Use this pattern in all three supported triggers. The early return keeps your Action clean and ensures it behaves correctly during periods when no experiment is running. | ||
|
|
||
|
|
||
| ## Example: post_login — conditional MFA policy | ||
|
|
||
| This example reads a boolean parameter and applies a different MFA policy depending on the variation. | ||
|
|
||
| ```javascript | ||
| // post_login Action | ||
| exports.onExecutePostLogin = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; | ||
|
|
||
| const enforceMfa = ec.config?.require_mfa?.value; | ||
|
|
||
| if (enforceMfa === true) { | ||
| // Treatment variation: enforce MFA for this user | ||
| api.multifactor.enable("any", { allowRememberBrowser: false }); | ||
| } | ||
| // Control variation: no MFA enforcement change (baseline behavior) | ||
| }; | ||
| ``` | ||
|
|
||
| `ec.config.require_mfa.value` is `true` for users in the treatment variation and `false` (the baseline) for users in the control variation. No fallback logic needed. | ||
|
|
||
|
|
||
|
|
||
| ## Example: post_login — set a custom claim based on variation | ||
|
|
||
| This example stamps the experiment assignment into the user's ID token as a custom claim. Some analytics pipelines read token claims instead of tenant logs. | ||
|
|
||
| ```javascript | ||
| // post_login Action | ||
| exports.onExecutePostLogin = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; | ||
|
|
||
| // Add experiment assignment to the ID token | ||
| api.idToken.setCustomClaim("https://example.com/experiment", { | ||
| experiment_id: ec.experiment_id, | ||
| variation_id: ec.variation_id, | ||
| }); | ||
| }; | ||
| ``` | ||
|
|
||
| Use a namespaced claim URL per the OIDC custom claim convention. The recipient (your app) can then read the claim from the token without querying tenant logs. | ||
|
|
||
|
|
||
|
|
||
| ## Example pre_user_registration — variation-based metadata | ||
|
|
||
| This example uses a registration-flow experiment to set `user_metadata` based on which variant the registering user lands in. | ||
|
|
||
| ```javascript | ||
| // pre_user_registration Action | ||
| exports.onExecutePreUserRegistration = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; | ||
|
|
||
| const onboardingVariant = ec.config?.onboarding_variant?.value; | ||
|
|
||
| if (onboardingVariant) { | ||
| api.user.setUserMetadata("onboarding_variant", onboardingVariant); | ||
| api.user.setUserMetadata("onboarding_experiment_id", ec.experiment_id); | ||
| } | ||
| }; | ||
| ``` | ||
|
|
||
| The metadata is written during registration, so it follows the user across sessions. You can query it later to understand which signup cohort a user belongs to. | ||
|
|
||
| ## Example: post_user_registration — trigger downstream enrollment | ||
|
|
||
| This example fires a webhook after registration based on the variation assigned to the new user. | ||
|
|
||
| ```javascript | ||
| // post_user_registration Action | ||
| exports.onExecutePostUserRegistration = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; | ||
|
|
||
| const enrollInNewProgram = ec.config?.enroll_welcome_program?.value; | ||
|
|
||
| if (enrollInNewProgram === true) { | ||
| await fetch("https://your-backend.example.com/api/enrollment", { | ||
| method: "POST", | ||
| headers: { "Content-Type": "application/json" }, | ||
| body: JSON.stringify({ | ||
| user_id: event.user.user_id, | ||
| email: event.user.email, | ||
| experiment_id: ec.experiment_id, | ||
| variation_id: ec.variation_id, | ||
| }), | ||
| }); | ||
| } | ||
| }; | ||
| ``` | ||
|
|
||
| Only users in the treatment variation trigger the enrollment webhook. Control users follow the standard post-registration path. | ||
|
|
||
| ## Use is_control | ||
|
|
||
| The parameter `is_control` is `true` when the user is in the control group (they received the baseline, no overrides applied). Use it when you need to track which users saw the unmodified experience, or when you want to skip optional processing for control users. | ||
|
|
||
| ```javascript | ||
| exports.onExecutePostLogin = async (event, api) => { | ||
| const ec = event.experiment; | ||
| if (!ec) return; | ||
|
|
||
| if (!ec.is_control) { | ||
| // Only track treatment users in your analytics system | ||
| // (control users are tracked separately, outside the Action) | ||
| await logTreatmentEvent(ec.experiment_id, ec.variation_id, event.user.user_id); | ||
| } | ||
| }; | ||
| ``` | ||
|
|
||
| For branching on behavior (not analytics), check `config` parameter values directly. They are more explicit and readable. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.