You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|`--region`| PostHog cloud region (when not specified, prompts for selection) | string || "us", "eu" |`POSTHOG_WIZARD_REGION`|
47
+
|`--default`| Use default options for all prompts | boolean |`true`||`POSTHOG_WIZARD_DEFAULT`|
48
+
|`--signup`| Create a new PostHog account during setup | boolean |`false`||`POSTHOG_WIZARD_SIGNUP`|
49
+
|`--integration`| Integration to set up | string || "nextjs", "astro", "react", "svelte", "react-native" ||
50
+
|`--force-install`| Force install packages even if peer dependency checks fail | boolean |`false`||`POSTHOG_WIZARD_FORCE_INSTALL`|
51
+
|`--install-dir`| Directory to install PostHog in | string |||`POSTHOG_WIZARD_INSTALL_DIR`|
51
52
52
53
> Note: A large amount of the scaffolding for this came from the amazing Sentry
53
54
> wizard, which you can find [here](https://github.com/getsentry/sentry-wizard)
54
55
> 💖
55
56
56
57
# Steal this code
57
58
58
-
While the wizard works great on its own, we also find the approach used by this project is [a powerful way to improve AI agent coding sessions](https://posthog.com/blog/envoy-wizard-llm-agent). Agents can run CLI tools, which means that conventional code like this can participate in the AI revolution as well – with all the benefits and control that conventional code implies.
59
+
While the wizard works great on its own, we also find the approach used by this
60
+
project is
61
+
[a powerful way to improve AI agent coding sessions](https://posthog.com/blog/envoy-wizard-llm-agent).
62
+
Agents can run CLI tools, which means that conventional code like this can
63
+
participate in the AI revolution as well – with all the benefits and control
64
+
that conventional code implies.
59
65
60
-
If you want to use this code as a starting place for your own project, here's a quick explainer on its structure.
66
+
If you want to use this code as a starting place for your own project, here's a
67
+
quick explainer on its structure.
61
68
62
69
## Entrypoint: `run.ts`
63
70
64
-
The entrypoint for this tool is `run.ts`. Use this file to interpret arguments and set up the general flow of the application.
71
+
The entrypoint for this tool is `run.ts`. Use this file to interpret arguments
72
+
and set up the general flow of the application.
65
73
66
74
## Analytics
67
75
68
-
Did you know you can capture PostHog events even for smaller, supporting products like a command line tool? `src/utils/analytics.ts` is a great example of how to do it.
76
+
Did you know you can capture PostHog events even for smaller, supporting
77
+
products like a command line tool? `src/utils/analytics.ts` is a great example
78
+
of how to do it.
69
79
70
-
This file wraps `posthog-node` with some convenience functions to set up an analytics session and log events. We can see the usage and outcomes of this wizard alongside all of our other PostHog product data, and this is very powerful. For example: we could show in-product surveys to people who have used the wizard to improve the experience.
80
+
This file wraps `posthog-node` with some convenience functions to set up an
81
+
analytics session and log events. We can see the usage and outcomes of this
82
+
wizard alongside all of our other PostHog product data, and this is very
83
+
powerful. For example: we could show in-product surveys to people who have used
84
+
the wizard to improve the experience.
71
85
72
86
## Leave rules behind
73
87
74
-
Supporting agent sessions after we leave is important. There are plenty of ways to break or misconfigure PostHog, so guarding against this is key.
88
+
Supporting agent sessions after we leave is important. There are plenty of ways
89
+
to break or misconfigure PostHog, so guarding against this is key.
75
90
76
-
`src/utils/rules/add-editor-rules.ts` demonstrates how to dynamically construct rules files and store them in the project's `.cursor/rules` directory.
91
+
`src/utils/rules/add-editor-rules.ts` demonstrates how to dynamically construct
92
+
rules files and store them in the project's `.cursor/rules` directory.
77
93
78
94
## Prompts and LLM interactions
79
95
80
-
LLM agent sessions are *anti-deterministic*: really, anything can happen.
96
+
LLM agent sessions are _anti-deterministic_: really, anything can happen.
81
97
82
-
But using LLMs for code generation is really advantageous: they can interpret existing code at scale and then modify it reliably.
98
+
But using LLMs for code generation is really advantageous: they can interpret
99
+
existing code at scale and then modify it reliably.
83
100
84
-
*If* they are well prompted.
101
+
_If_ they are well prompted.
85
102
86
-
`src/lib/prompts.ts` demonstrates how to wrap a deterministic fence around a chaotic process. Every wizard session gets the same prompt, tailored to the specific files in the project.
103
+
`src/lib/prompts.ts` demonstrates how to wrap a deterministic fence around a
104
+
chaotic process. Every wizard session gets the same prompt, tailored to the
105
+
specific files in the project.
87
106
88
-
These prompts are channeled using `src/utils/query.ts` to an LLM interface we host. This gives us more control: we can be certain of the model version and provider which interpret the prompts and modify the files. This way, we can find the right tools for the job and again, apply them consistently.
107
+
These prompts are channeled using `src/utils/query.ts` to an LLM interface we
108
+
host. This gives us more control: we can be certain of the model version and
109
+
provider which interpret the prompts and modify the files. This way, we can find
110
+
the right tools for the job and again, apply them consistently.
89
111
90
112
This also allows us to pick up the bill on behalf of our customers.
91
113
92
-
When we make improvements to this process, these are available instantly to all users of the wizard, no training delays or other ambiguity.
114
+
When we make improvements to this process, these are available instantly to all
115
+
users of the wizard, no training delays or other ambiguity.
93
116
94
-
## Testing locally
117
+
## Running locally
95
118
96
119
Run:
97
120
98
121
```bash
99
122
pnpm try --install-dir=[a path]
100
123
```
101
124
102
-
103
125
To build and use the tool locally:
104
126
127
+
```bash
128
+
bin/build
105
129
```
106
-
pnpm build
107
-
```
108
-
This compiles the TypeScript code and prepares the `dist` directory. Run this command any time you make changes to the wizard's source code.
130
+
131
+
This compiles the TypeScript code and prepares the `dist` directory. Run this
132
+
command any time you make changes to the wizard's source code.
109
133
110
134
```bash
111
135
pnpm link --global
112
136
```
113
-
This command makes your local version of the wizard available system-wide. You generally only need to do this once.
114
137
115
-
Then:
138
+
This command makes your local version of the wizard available system-wide. You
139
+
generally only need to do this once.
140
+
141
+
Then:
116
142
117
143
```bash
118
144
wizard [options]
119
145
```
146
+
120
147
The wizard will execute your last build.
121
148
122
-
## Publishing your tool
149
+
## Testing
123
150
124
-
To make your version of a tool usable with a one-line `npx` command:
151
+
To run unit tests, run:
125
152
126
-
1. Edit `package.json`, especially details like `name`, `version`
127
-
2. Run [`npm publish`](https://docs.npmjs.com/cli/v7/commands/npm-publish) from your project directory
128
-
3. Now you can run it with `npx yourpackagename`
153
+
```bash
154
+
bin/test
155
+
```
129
156
157
+
To run E2E tests run:
130
158
159
+
```bash
160
+
bin/test-e2e
161
+
```
131
162
163
+
E2E tests are a bit more complicated to create and adjust due to to their mocked
164
+
LLM calls. See the `e2e-tests/README.md` for more information.
132
165
166
+
## Publishing your tool
133
167
168
+
To make your version of a tool usable with a one-line `npx` command:
169
+
170
+
1. Edit `package.json`, especially details like `name`, `version`
171
+
2. Run [`npm publish`](https://docs.npmjs.com/cli/v7/commands/npm-publish) from
E2E Tests can be run locally from the root of the project with:
6
+
7
+
`pnpm test:e2e`
8
+
9
+
To run a specific test application
10
+
11
+
`pnpm test:e2e NextJS`
12
+
13
+
To record new fixtures:
14
+
15
+
`pnpm test:e2e-record`
16
+
17
+
## Writing Framework tests
18
+
19
+
Most of the E2E framework tests share a lot of common functionality such as
20
+
terminal inputs and test setup/teardown. To accomodate for the large amount of
21
+
frameworks we test, we expose the `createFrameworkTest` function.
22
+
23
+
For example usage, see one of the tests in `e2e-tests/tests` such as
24
+
`nextjs-app-router.test.ts`. Each framework test also requires a test
25
+
application to be defined in `test-applications`.
26
+
27
+
To adjust the default behaviour of the framework, also take a look at
28
+
`DEFAULT_WIZARD_STEPS` in `e2e-tests/utils/framework-test-utils.ts`
29
+
30
+
## Fixture Generation
31
+
32
+
To be able to mock our LLM calls in the E2E tests, we need to have a realistic
33
+
fixture. To generate them, we a call to the `/query` endpoint in PostHog. We
34
+
save this response as a fixture in `e2e-tests/fixtures`. The filename represents
35
+
the hashed request body to the endpoint. When we run the tests again, we reuse
36
+
those fixtures.
37
+
38
+
Whenever the request body to the LLM change we also regenerate the fixture. The
39
+
request body can change because of a few things:
4
40
5
-
```
6
-
test-applications/
7
-
|---- nextjs-app-router-test-app/
8
-
tests/
9
-
|---- nextjs-app-router.test.ts
10
-
```
41
+
- Prompt:
42
+
- The system propmt
43
+
- The provided framework files
44
+
- etc.
45
+
- LLM Model
46
+
- Response Schema
47
+
48
+
Because we use a set seed for the LLM, this means our tests are deterministic
49
+
and actually reflective of how they would work in production as well.
50
+
51
+
Two environment variables control our fixture management:
52
+
53
+
`RECORD_FIXTURES` performs a `/query` request to create a fixture if no matching
54
+
fixture is found the for the request body. Can be `true or false`. If `false`
55
+
and no matching fixture is found, the test will fail.
56
+
57
+
`CLEANUP_UNUSED_FIXTURES` deletes fixtures that were not used during an E2E jest
58
+
run. Should only be set to true when running all E2E tests. Can be `true` or
59
+
`false`
11
60
12
61
### Utilities
13
62
14
-
`utils/` contains helpers such as the wizard runner, assertion tools and file modifiers that can be used in (`*.test.ts`).
63
+
`utils/` contains helpers such as the wizard runner, assertion tools and file
64
+
modifiers that can be used in (`*.test.ts`).
15
65
16
66
#### Helpers
17
67
18
68
-`startWizardInstance` - Starts a new instance of `WizardTestEnv`.
19
69
20
70
-`initGit` - Initializes a temporary git repository in the test project.
21
71
-`cleanupGit` - Cleans up the temporary git repository in the test project.
22
-
-`revertLocalChanges` - Reverts local changes (git tracked or untracked) in the test project.
72
+
-`revertLocalChanges` - Reverts local changes (git tracked or untracked) in the
73
+
test project.
23
74
24
75
-`createFile` - Creates a file (optionally with content) in the test project.
25
76
-`modifyFile` - Modifies a file in the test project.
26
77
27
78
-`checkFileExists` - Checks if a file exists in the test project.
28
-
-`checkPackageJson` - Checks if the package package exists in the dependencies of the test project's `package.json`.
79
+
-`checkPackageJson` - Checks if the package package exists in the dependencies
80
+
of the test project's `package.json`.
29
81
30
82
-`checkIfBuilds` - Checks if the test project builds successfully.
31
-
-`checkIfRunsOnDevMode` - Checks if the test project runs on dev mode successfully.
32
-
-`checkIfRunsOnProdMode` - Checks if the test project runs on prod mode successfully.
83
+
-`checkIfRunsOnDevMode` - Checks if the test project runs on dev mode
84
+
successfully.
85
+
-`checkIfRunsOnProdMode` - Checks if the test project runs on prod mode
86
+
successfully.
33
87
34
88
#### `WizardTestEnv`
35
89
36
-
`WizardTestEnv` is a class that can be used to run the PostHog wizard in a test environment. It provides methods to run the wizard with specific arguments and stdio.
37
-
38
-
## Running Tests Locally
39
-
40
-
First, you need to create a `.env` file set the environment variables from the `.env.example` file in the root of the project.
41
-
42
-
Tests can be run locally from the root of the project with:
43
-
44
-
`pnpm test:e2e`
45
-
46
-
To run a specific test application
47
-
48
-
`pnpm test:e2e NextJS`
49
-
50
-
## Writing Tests
51
-
52
-
Each test file should contain a single test suite that tests the PostHog wizard for a specific framework. The test suite should contain a `beforeAll` and `afterAll` function that starts and stops the test application respectively.
90
+
`WizardTestEnv` is a class that can be used to run the PostHog wizard in a test
91
+
environment. It provides methods to run the wizard with specific arguments and
0 commit comments