docs: add Experimentation guides#7993
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
2 Skipped Deployments
|
📝 WalkthroughWalkthroughAdds an Experimentation documentation section covering warehouse connections, metrics, experiment creation, event instrumentation, PayPal testing, result analysis, and anonymous identity handling. It also creates category navigation, updates the statistics glossary, reformats the existing A/B testing guide, and adds redirects for renamed documentation pages. Estimated code review effort: 3 (Moderate) | ~20 minutes Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 52af566f-521f-4125-bce3-1858e21df39d
⛔ Files ignored due to path filters (5)
docs/static/img/experimentation/create-experiment-rollout.pngis excluded by!**/*.pngdocs/static/img/experimentation/create-experiment-setup.pngis excluded by!**/*.pngdocs/static/img/experimentation/experiment-results-analysis.pngis excluded by!**/*.pngdocs/static/img/experimentation/experiment-results-overview.pngis excluded by!**/*.pngdocs/static/img/experimentation/warehouse-tab.pngis excluded by!**/*.png
📒 Files selected for processing (11)
docs/docs/experimentation-ab-testing.mddocs/docs/experimentation/_category_.jsondocs/docs/experimentation/analyse-an-experiment.mddocs/docs/experimentation/connect-a-warehouse.mddocs/docs/experimentation/create-an-experiment.mddocs/docs/experimentation/create-metrics.mddocs/docs/experimentation/example-paypal-button.mddocs/docs/experimentation/index.mddocs/docs/experimentation/run-an-experiment.mdxdocs/docs/experimentation/statistics.mddocs/vercel.json
| ## Scenario - Testing a new Paypal button | ||
|
|
||
| For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we are losing out on potential customers that would like to pay with PayPal. We're going to test whether adding PayPal to the payment options increases our checkout rate. | ||
| For this example, lets assume we have an app that currently accepts credit card payments only. We have a hunch that we |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the official PayPal spelling and correct the apostrophe.
Replace “Paypal” with “PayPal” throughout this section and change “lets assume” to “let’s assume”.
Also applies to: 55-60, 72-73
🧰 Tools
🪛 LanguageTool
[uncategorized] ~48-~48: The official name of this payment provider is spelled with two capital “P”.
Context: ...smith SDK. ## Scenario - Testing a new Paypal button For this example, lets assume w...
(PAYPAL)
[uncategorized] ~50-~50: It seems likely that a singular genitive (’s) apostrophe is missing.
Context: ... a new Paypal button For this example, lets assume we have an app that currently ac...
(AI_HYDRA_LEO_APOSTROPHE_S_XS)
Source: Linters/SAST tools
| We have a lot of users on our platform, so we don't want to run this test against our entire user-base. We want 90% of | ||
| our users to be excluded from the test. Then for our test, 5% of our users will see the new Paypal button, and the | ||
| remaining 5% will not see it. So we will have 3 buckets: | ||
|
|
||
| 1. Excluded (Control) Users | ||
| 2. Paypal test button users | ||
| 3. Test users that don't see the Paypal button | ||
|
|
||
| Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the multivariate value. In this example, we will only show the PayPal button if the value is set to `show`. | ||
| Because Flagsmith flags can contain both boolean states as well as multivariate flag values, we can make use of both. We | ||
| will use the boolean flag state to control whether to run the test. Then, if the flag is `enabled`, check the | ||
| multivariate value. In this example, we will only show the PayPal button if the value is set to `show`. | ||
|
|
||
| ## Steps | ||
|
|
||
| 1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options: | ||
| 1. Create a new [multivariate flag](/managing-flags/core-management) that will control which of the 3 buckets the user | ||
| is put into. We'll call this flag `paypal_button_test`. We will provide 3 variate options: | ||
|
|
||
| 1. Control - 90% of users | ||
| 2. Paypal button - 5% of users | ||
| 3. Test users that don't see the Paypal button - 5% of users | ||
|
|
||
| 2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process. All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner. | ||
| 3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates to `show`, show the PayPal payment button. Otherwise, don't show the button. | ||
| 4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of the flag; in this case it would be one of either `control`, `show` or `hide`. | ||
| 2. In our app, we want to [identify](/flagsmith-concepts/identities) each user before they start the checkout process. | ||
| All Flagsmith multivariate flags need us to identify the user, so we can bucket them in a reproducible manner. | ||
| 3. When we get to the checkout page, check the `value` of the `paypal_button_test` flag for that user. If it evaluates | ||
| to `show`, show the PayPal payment button. Otherwise, don't show the button. | ||
| 4. Send an event message to the analytics platform, adding the name/value pair of `paypal_button_test` and the value of | ||
| the flag; in this case it would be one of either `control`, `show` or `hide`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Make the rollout model consistent with the documented buckets.
The guide says 90% of users are excluded and that the boolean state gates the test, but the next steps define one multivariate flag with control/show/hide weights of 90/5/5 and then assume every user produces one of those values. The required targeting or rollout configuration for the 90% cohort is never specified.
Either document control as a 90% control variant and remove the “excluded”/separate-gate wording, or define a 10% rollout and explain the disabled/default path separately.
🧰 Tools
🪛 LanguageTool
[style] ~54-~54: Consider using a synonym to be more concise.
Context: ...s increases our checkout rate. We have a lot of users on our platform, so we don't want...
(A_LOT_OF)
[style] ~54-~54: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...this test against our entire user-base. We want 90% of our users to be excluded fr...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[uncategorized] ~55-~55: The official name of this payment provider is spelled with two capital “P”.
Context: ... test, 5% of our users will see the new Paypal button, and the remaining 5% will not s...
(PAYPAL)
[uncategorized] ~59-~59: The official name of this payment provider is spelled with two capital “P”.
Context: ...uckets: 1. Excluded (Control) Users 2. Paypal test button users 3. Test users that do...
(PAYPAL)
[uncategorized] ~60-~60: The official name of this payment provider is spelled with two capital “P”.
Context: ... users 3. Test users that don't see the Paypal button Because Flagsmith flags can con...
(PAYPAL)
[grammar] ~62-~62: Probable usage error. Use “and” after ‘both’.
Context: ...h flags can contain both boolean states as well as multivariate flag values, we can make u...
(BOTH_AS_WELL_AS)
[uncategorized] ~69-~69: The official name of this payment provider is spelled with two capital “P”.
Context: ...er is put into. We'll call this flag paypal_button_test. We will provide 3 variate...
(PAYPAL)
[uncategorized] ~72-~72: The official name of this payment provider is spelled with two capital “P”.
Context: ...ns: 1. Control - 90% of users 2. Paypal button - 5% of users 3. Test users t...
(PAYPAL)
[uncategorized] ~73-~73: The official name of this payment provider is spelled with two capital “P”.
Context: ...ers 3. Test users that don't see the Paypal button - 5% of users 2. In our app, we...
(PAYPAL)
[uncategorized] ~77-~77: The official name of this payment provider is spelled with two capital “P”.
Context: ...checkout page, check the value of the paypal_button_test flag for that user. If it ...
(PAYPAL)
[grammar] ~79-~79: An article may be missing.
Context: ...alytics platform, adding the name/value pair of paypal_button_test and the value of ...
(BUNCH_OF)
[uncategorized] ~79-~79: The official name of this payment provider is spelled with two capital “P”.
Context: ...platform, adding the name/value pair of paypal_button_test and the value of the fl...
(PAYPAL)
| Identities that were served more than one variation are quarantined and shown as excluded. A small number is normal; a | ||
| growing one means users are slipping between variations. | ||
|
|
||
| <!-- Screenshot: exposures panel with enrolment chart --> |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Replace the screenshot placeholder with the actual Exposures image.
This HTML comment renders nothing, leaving the Exposures section without the promised visual. Add the image with meaningful alt text, or remove the placeholder if no screenshot is intended.
| - **Variation split**: how identities _inside_ the rollout are distributed across control and the variations. Weights | ||
| must sum to 100, and control takes whatever the variations don't. Use **Split evenly** for an equal distribution. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Relevant file excerpt ==\n'
sed -n '1,120p' docs/docs/experimentation/create-an-experiment.md | cat -n
printf '\n== Search for allocation / split wording across experimentation docs ==\n'
rg -n --hidden -S "Variation split|Split evenly|control takes whatever|weights must sum to 100|rollout can still be edited|locked during execution|variation.*control|control.*variation" docs/docs
printf '\n== Open the referenced related doc excerpt if present ==\n'
if [ -f docs/docs/experimentation/run-an-experiment.mdx ]; then
sed -n '80,130p' docs/docs/experimentation/run-an-experiment.mdx | cat -n
fiRepository: Flagsmith/flagsmith
Length of output: 6399
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Search for experiment rollout / split implementation references ==\n'
rg -n --hidden -S "split evenly|variation split|rollout.*edit|edit.*rollout|locked while the experiment is running|cannot be edited|control takes whatever|weights must sum to 100" .
printf '\n== Search for API/serializer fields that suggest exact semantics ==\n'
rg -n --hidden -S "rollout|control|variation.*weight|weights.*100|experiment.*weight|split" backend web docs -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**'Repository: Flagsmith/flagsmith
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== Rollout step wording ==\n'
sed -n '1,120p' frontend/web/components/experiments/steps/RolloutStep.tsx | cat -n
printf '\n== Split editor shape ==\n'
sed -n '1,220p' frontend/web/components/experiments/RolloutSplitEditor/RolloutSplitEditor.tsx | cat -n
printf '\n== Example experiment wording ==\n'
sed -n '30,50p' docs/docs/experimentation/example-paypal-button.md | cat -nRepository: Flagsmith/flagsmith
Length of output: 8566
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,220p' frontend/web/components/experiments/rollout.ts | cat -nRepository: Flagsmith/flagsmith
Length of output: 4446
Make the split wording explicit. The entered weights are for the variations only; control is the derived remainder. Rephrase this so it’s clear that the variation weights do not need to total 100, and that control fills whatever is left.
|
|
||
| :::caution | ||
|
|
||
| While an experiment is running, its flag is locked and cannot be edited. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Clarify which parts of the flag are locked.
This says the flag cannot be edited while running, while docs/docs/experimentation/run-an-experiment.mdx line 100 says the rollout can be edited from the experiment page. State explicitly that flag configuration/variations are locked but the experiment rollout remains editable, if that is the intended behaviour.
| On the experiment page, check the **Exposures** panel: enrolment should track about 10% of checkout traffic, split | ||
| 50/50. Then watch the results as conversions arrive; see |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Measure rollout against identities, not raw traffic.
The setup page defines rollout as a percentage of identities, and exposures are deduplicated per identity. Therefore, 10% of identities will not necessarily equal 10% of checkout requests. Reword this as approximately 10% of eligible identities/exposures.
1 participant
Thanks for submitting a PR! Please check the boxes below:
docs/if required so people know about the feature.Changes
Documents the Experimentation feature end to end, as a new top-level sidebar category on docs.flagsmith.com:
Create an Experiment, Run an Experiment (JavaScript and Python instrumentation), and Analyse an Experiment.
A/B testing tutorial (branches on variation names via
useExperiment).How did you test this code?
npm run buildindocs/passes with no broken links in the new pages.make lint).