docs: add Experimentation guides (#7993)

Author: Zaimwa9

docs/docs/experimentation-ab-testing.md

@@ -4,76 +4,111 @@ sidebar_label: Experimentation (A/B Testing)
 sidebar_position: 4
 ---
 
-A/B testing enables you to experiment with design and functionality variants of your application. The data generated will allow you to make modifications to your app, safe in the knowledge that it will have a net positive effect.
+:::tip
 
-You can use Flagsmith to perform A/B tests. Using a combination of [multivariate flags](/managing-flags/core-management) and a 3rd party analytics tool like [Amplitude](https://amplitude.com/) or [Mixpanel](https://mixpanel.com/), you can easily perform complex A/B tests that will help improve your product.
+Flagsmith now runs experiments natively, with a managed warehouse and built-in statistics: see the
+[Experimentation overview](/experimentation/). This guide covers the do-it-yourself approach with a third-party
+analytics platform.
 
-Running A/B tests require two main components: a bucketing engine and an analytics platform. The bucketing engine is used to put users into a particular A/B testing bucket. These buckets will control the specific user experience that is being tested. The analytics platform will receive a stream of event data derived from the behaviour of the user. Combining these two concepts allows you to deliver seamless A/B test.
+:::
 
-We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't integrate with the platform you are using, you can still manually send the test data to the downstream platform manually.
+A/B testing enables you to experiment with design and functionality variants of your application. The data generated
+will allow you to make modifications to your app, safe in the knowledge that it will have a net positive effect.
+
+You can use Flagsmith to perform A/B tests. Using a combination of [multivariate flags](/managing-flags/core-management)
+and a 3rd party analytics tool like [Amplitude](https://amplitude.com/) or [Mixpanel](https://mixpanel.com/), you can
+easily perform complex A/B tests that will help improve your product.
+
+Running A/B tests require two main components: a bucketing engine and an analytics platform. The bucketing engine is
+used to put users into a particular A/B testing bucket. These buckets will control the specific user experience that is
+being tested. The analytics platform will receive a stream of event data derived from the behaviour of the user.
+Combining these two concepts allows you to deliver seamless A/B test.
+
+We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't
+integrate with the platform you are using, you can still manually send the test data to the downstream platform
+manually.
 
 By the end of this tutorial, you will be able to:
 
--   Set up a multivariate flag in Flagsmith for A/B testing.
--   Implement logic in your application to bucket users and display variants.
--   Send A/B test data to an analytics platform.
--   Understand how to use anonymous identities for A/B testing on unknown users.
+- Set up a multivariate flag in Flagsmith for A/B testing.
+- Implement logic in your application to bucket users and display variants.
+- Send A/B test data to an analytics platform.
+- Understand how to use anonymous identities for A/B testing on unknown users.
 
 ## Before you begin
 
 To follow this tutorial, you will need:
 
-- A basic understanding of [multivariate flags](/managing-flags/core-management) in Flagsmith. Remember: multivariate bucketing only works when users are identified (use real identities or persistent anonymous GUIDs).
-- Access to a third-party analytics platform (e.g., Amplitude, Mixpanel) where you can send custom events. You can explore Flagsmith [integrations](/third-party-integrations/analytics/segment) for this purpose.
+- A basic understanding of [multivariate flags](/managing-flags/core-management) in Flagsmith. Remember: multivariate
+  bucketing only works when users are identified (use real identities or persistent anonymous GUIDs).
+- Access to a third-party analytics platform (e.g., Amplitude, Mixpanel) where you can send custom events. You can
+  explore Flagsmith [integrations](/third-party-integrations/analytics/segment) for this purpose.
 - A development environment for your application where you can implement changes and integrate the Flagsmith SDK.
 
 ## Scenario - Testing a new Paypal button
 
-For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we are losing out on potential customers that would like to pay with PayPal. We're going to test whether adding PayPal to the payment options increases our checkout rate.
+For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we
+are losing out on potential customers that would like to pay with PayPal. We're going to test whether adding PayPal to
+the payment options increases our checkout rate.
 
-We have a lot of users on our platform, so we don't want to run this test against our entire user-base. We want 90% of our users to be excluded from the test. Then for our test, 5% of our users will see the new Paypal button, and the remaining 5% will not see it. So we will have 3 buckets:
+We have a lot of users on our platform, so we don't want to run this test against our entire user-base. We want 90% of
+our users to be excluded from the test. Then for our test, 5% of our users will see the new Paypal button, and the
+remaining 5% will not see it. So we will have 3 buckets:
 
 1. Excluded (Control) Users
 2. Paypal test button users
 3. Test users that don't see the Paypal button
 
-Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the multivariate value. In this example, we will only show the PayPal button if the value is set to `show`.
+Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We
+will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the
+multivariate value. In this example, we will only show the PayPal button if the value is set to `show`.
 
 ## Steps
 
-1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options:
+1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user
+   is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options:
 
    1. Control - 90% of users
    2. Paypal button - 5% of users
    3. Test users that don't see the Paypal button - 5% of users
 
-2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process. All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner.
-3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates to `show`, show the PayPal payment button. Otherwise, don't show the button.
-4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of the flag; in this case it would be one of either `control`, `show` or `hide`.
+2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process.
+   All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner.
+3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates
+   to `show`, show the PayPal payment button. Otherwise, don't show the button.
+4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of
+   the flag; in this case it would be one of either `control`, `show` or `hide`.
 5. Deploy our app, enable the flag and watch the data come in to your analytics platform.
 
 Here is what creating the flag would look like.
 
 ![Image](/img/ab-test-paypal-example.png)
 
-Once the test is set up, and the flag has been enabled, data will start streaming into the analytics platform. We can now evaluate the results of the tests based on the behavioral changes that the new button has created.
+Once the test is set up, and the flag has been enabled, data will start streaming into the analytics platform. We can
+now evaluate the results of the tests based on the behavioral changes that the new button has created.
 
 ## Handling Anonymous/Unknown Identities
 
-To do A/B testing you need to use identities. Without an identity to key from, it's impossible for the platform to serve a consistent experience to your users.
+To do A/B testing you need to use identities. Without an identity to key from, it's impossible for the platform to serve
+a consistent experience to your users.
 
-What if you want to run an A/B test in an area of your application where you don't know who your users are? For example on the homepage of your website? In this instance, you need to generate _anonymous identities_ values for your users. In this case we will generate a GUID for each user.
+What if you want to run an A/B test in an area of your application where you don't know who your users are? For example
+on the homepage of your website? In this instance, you need to generate _anonymous identities_ values for your users. In
+this case we will generate a GUID for each user.
 
-A GUID value is just a random string that has an extremely high likelihood of being unique. There's more info about generating GUID values [on Stack Overflow](https://stackoverflow.com/a/2117523).
+A GUID value is just a random string that has an extremely high likelihood of being unique. There's more info about
+generating GUID values [on Stack Overflow](https://stackoverflow.com/a/2117523).
 
 The general flow would be:
 
 1. A new browser visits your website homepage for the first time.
 2. You see that this is an anonymous user, so you generate a random GUID for that user and assign it to them.
 3. You send that GUID along with an identify call to Flagsmith. This will then segment that visitor.
-4. You add a cookie to the browser and store the GUID. That way, if the user returns to your page, they will still be in the same segment.
+4. You add a cookie to the browser and store the GUID. That way, if the user returns to your page, they will still be in
+   the same segment.
 
-These techniques will be slightly different depending on what platform you are developing for, but the general concept will remain the same.
+These techniques will be slightly different depending on what platform you are developing for, but the general concept
+will remain the same.
 
 ## Next steps
 

docs/docs/experimentation/_category_.json

@@ -0,0 +1,5 @@
+{
+    "label": "Experimentation (Beta)",
+    "position": 5,
+    "collapsed": true
+}

docs/docs/experimentation/analyse-an-experiment.md

@@ -0,0 +1,57 @@
+---
+title: Analyse an Experiment
+sidebar_label: Analyse an Experiment
+sidebar_position: 6
+description: Read the results page (lift, credible intervals, win probability) and decide when to roll out a winner.
+---
+
+:::info Enterprise beta
+
+Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join.
+
+:::
+
+The experiment's detail page shows who entered the experiment, how each variation performed, and whether you can trust
+the numbers.
+
+## Results
+
+Results are computed on demand: click **Refresh results** (at most once every 5 minutes). The **As of** timestamp shows
+how fresh the figures are.
+
+The summary scorecards show **Users enrolled**, the current **Winning variation**, its **Chance to be best** and its
+**Lift vs control**. See the [Statistics Glossary](/experimentation/statistics) for what each term means.
+
+When a variation reaches a 95% chance of beating control, a recommendation banner suggests rolling it out.
+
+![An experiment page with a winner recommendation banner](/img/experimentation/experiment-results-overview.png)
+
+The analysis table breaks each variation down against control:
+
+- **Exposures**: identities that entered on this variation.
+- The metric value (rate, count, sum or mean, depending on the metric).
+- **Delta**: how much better or worse the variation did than control, as a percentage.
+- **Credible Interval (95%)**: the range the true delta sits in with 95% certainty. If it crosses zero, the result is
+  inconclusive.
+- **Win Probability**: the chance the variation beats control.
+
+![Results scorecards and the analysis table with credible intervals](/img/experimentation/experiment-results-analysis.png)
+
+While a variation has fewer than 50 identities (or fewer than 5 conversions, for occurrence metrics), it shows
+**Collecting data**. Keep the experiment running.
+
+If a **sample ratio mismatch** warning appears, traffic didn't split the way you configured. Don't act on the results.
+
+## Exposures
+
+The **Exposures** panel charts enrolment over time, per variation, with each variation's share of the total.
+
+Identities that were served more than one variation are quarantined and shown as excluded. A small number is normal; a
+growing one means users are slipping between variations.
+
+## Ending the experiment
+
+When you have a conclusive result, click **End Experiment**. Results are frozen as final and the flag is unlocked.
+
+To roll out the winner, update the flag so every user gets the winning variation. Once it's permanent, clean up the flag
+and remove the losing variations.

docs/docs/experimentation/connect-a-warehouse.md

@@ -0,0 +1,49 @@
+---
+title: Connect a Warehouse
+sidebar_label: Connect a Warehouse
+sidebar_position: 2
+description: Enable the managed Flagsmith warehouse to store and query experiment events.
+---
+
+:::info Enterprise beta
+
+Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join.
+
+:::
+
+Experiment data (exposures and conversion events) is stored and queried in a data warehouse. Each environment has one
+warehouse connection, and it must be in place before you can create experiments.
+
+## Connect the Flagsmith warehouse
+
+The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure.
+
+1. Go to **Environment Settings > Warehouse**.
+2. Select **Flagsmith** and click **Enable**, then confirm.
+
+![The Warehouse tab with the Flagsmith managed warehouse selected](/img/experimentation/warehouse-tab.png)
+
+The connection is created immediately. To verify that events can flow, click **Send your first event** and Flagsmith
+sends a test event on your behalf. The connection shows **Pending Connection** while it waits for the first event, and
+switches to **Connected** as soon as the environment's first event arrives; an event sent from your own application
+counts too, not just the test event. Processing the first event can take up to a few hours, and the status refreshes
+about once a minute while you keep the Warehouse tab open.
+
+Once connected, the warehouse card shows the total number of events received and the number of unique event names, which
+is useful for checking that your instrumentation is arriving.
+
+<!-- Screenshot: connected warehouse card with event stats -->
+
+### Connection statuses
+
+The current status is shown on the warehouse card in **Environment Settings > Warehouse**.
+
+- **Created**: the connection exists but no events have been received yet.
+- **Pending Connection**: waiting for the first event to arrive.
+- **Connected**: events have been received; you are ready to run experiments.
+- **Errored**: something went wrong; contact support.
+
+## Coming soon
+
+Bring your own warehouse: connections for **Snowflake**, **BigQuery** and **Databricks**, so experiment results are
+computed directly in your own data platform.

docs/docs/experimentation/create-an-experiment.md

@@ -0,0 +1,66 @@
+---
+title: Create an Experiment
+sidebar_label: Create an Experiment
+sidebar_position: 4
+description: Set up an experiment on a multivariate flag. Name it, configure the rollout, and pick a primary metric.
+---
+
+:::info Enterprise beta
+
+Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.flagsmith.com/contact-us) to join.
+
+:::
+
+An experiment serves the variations of a [multivariate flag](/managing-flags/core-management) to a share of your users
+and measures the impact on a metric.
+
+## 0. Pre-requisites
+
+- A [connected warehouse](/experimentation/connect-a-warehouse).
+- A [**multivariate flag**](/managing-flags/core-management) with a variation for each treatment you want to test. The
+  flag's current value acts as the **control**. A flag can only be in one active experiment at a time.
+- A [metric](/experimentation/create-metrics). You can also create one from inside the wizard.
+
+On the **Experiments** page, click **Create Experiment**. The wizard has four steps.
+
+## 1. Setup
+
+Give the experiment a name and a hypothesis, and select the flag to experiment on. A good hypothesis names the change,
+the metric, the expected magnitude and the timeframe. For example: _"Redesigning the checkout button with a clearer CTA
+will increase conversion rates by at least 15% within 30 days."_
+
+![The Setup step: experiment name, hypothesis and feature flag](/img/experimentation/create-experiment-setup.png)
+
+## 2. Rollout configuration
+
+Two settings control who sees what:
+
+- **Rollout**: the percentage of identities that enter the experiment at all. Identities outside the rollout keep the
+  flag's normal behaviour and are not counted in the results.
+- **Variation split**: how identities _inside_ the rollout are distributed across control and the variations. Weights
+  must sum to 100, and control takes whatever the variations don't. Use **Split evenly** for an equal distribution.
+
+For example, with a rollout of 20% and a 50/50 split, 10% of your identities see the control, 10% see the variation, and
+the remaining 80% are untouched.
+
+Bucketing is deterministic on the identity key, so the same identity always lands in the same variation.
+
+![The Rollout configuration step: rollout percentage, variation split and distribution bar](/img/experimentation/create-experiment-rollout.png)
+
+## 3. Measurement
+
+Select the **primary metric** (the metric the experiment is judged on) and set the **expected direction**: increase,
+decrease, should not increase, or should not decrease. If the metric doesn't exist yet, click **Create Metric**.
+
+## 4. Review & Launch
+
+Review the configuration and click **Create Experiment**. The experiment starts immediately.
+
+:::caution
+
+While an experiment is running, its flag configuration and variations are locked. The rollout percentage can still be
+edited from the experiment page.
+
+:::
+
+A running experiment can be **ended** at any time. Its results are frozen as final, and this cannot be undone.