Skip to content

Commit bd84cde

Browse files
jjoonleojjoonleo
authored
Merge pull request #478 from DevKor-github/codexd/463-rollout-monitoring-checklist
[codex] Add rollout monitoring checklist (#463)
2 parents a3aea0a + 2f217b9 commit bd84cde

3 files changed

Lines changed: 199 additions & 0 deletions

File tree

‎docs/Home.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ Welcome to the OnTime-front project documentation! This wiki contains everything
1010
- [Git Workflow](./Git.md) - Git strategy and commit message formats
1111
- [Android Release Signing](./Android-Release-Signing.md) - Required keystore inputs and release signing validation
1212
- [iOS Release Configuration](./iOS-Release-Configuration.md) - Required Dart defines and archive validation
13+
- [Release Rollout Monitoring](./Release-Rollout-Monitoring.md) - Release ownership, staged rollout gates, and post-launch monitoring
1314
- [Play Review Rejection Playbook](./Play-Review-Rejection-Playbook.md) - How to triage, fix, resubmit, or appeal Google Play review rejections
1415

1516
### Technical Deep Dives

‎docs/Release-Rollout-Monitoring.md‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
# Release Rollout Monitoring
2+
3+
Use this checklist after a Play Console submission is ready for review or a
4+
release has been approved for tester or production rollout. It assigns release
5+
ownership, defines rollout gates, and lists the signals the team must watch
6+
before widening availability.
7+
8+
## Ownership
9+
10+
- Release owner: assign one engineer before submission. This person owns Play
11+
Console status checks, rollout decisions, incident coordination, and final
12+
go/no-go notes.
13+
- Backup owner: assign one alternate before submission. This person must have
14+
access to the same dashboards and can pause rollout or coordinate rollback if
15+
the release owner is unavailable.
16+
- Handoff channel: record the owner, backup, release version, build number,
17+
release track, and expected monitoring windows in the team release channel or
18+
shared release note.
19+
- Required access: both owners need Google Play Console access, Firebase console
20+
access for the `ontime-c63f1` project, GitHub Actions access, and access to
21+
production backend logs or dashboards.
22+
23+
## Rollout Stages
24+
25+
1. Internal testing
26+
- Upload the signed AAB to Play Internal Testing as a draft release.
27+
- Confirm install, login, notification permission flow, Firebase
28+
initialization, FCM token registration, and the highest-risk user flows on
29+
at least one real Android device.
30+
- Do not proceed if install fails, Firebase initialization fails, login is
31+
broken, or the backend receives no token registration for a fresh install.
32+
2. Closed testing or limited tester rollout
33+
- Release to the agreed tester group after internal testing is clear.
34+
- Hold for at least one active test pass or one business day, whichever is
35+
more useful for the release risk.
36+
- Do not proceed if testers report launch blockers, sign-in blockers, alarm
37+
delivery regressions, or data-loss behavior.
38+
3. Production staged rollout
39+
- Start with the smallest practical Play production percentage.
40+
- Increase only after the first 24-hour checks are clean and no policy,
41+
crash, ANR, backend, or review signal suggests a release-caused regression.
42+
- Record every percentage increase with timestamp, owner, build number, and
43+
the dashboards checked.
44+
4. Full rollout
45+
- Move to full rollout only after the first 7-day checks are clean or the
46+
release owner explicitly accepts the remaining known risk.
47+
- Keep monitoring for one additional business day after reaching 100%.
48+
49+
## Pause And Rollback Criteria
50+
51+
Pause the rollout immediately when any of these signals appear release-caused:
52+
53+
- Play Console review rejection, policy warning, app content warning, or account
54+
email requiring action.
55+
- Crash rate or ANR rate materially above the previous production baseline.
56+
- Repeated startup, login, schedule creation, notification, alarm, or token
57+
registration failures from testers or monitoring.
58+
- Backend error spikes for authentication, schedule, preparation, alarm, or FCM
59+
token endpoints after the new build reaches users.
60+
- Ratings or reviews identify a reproducible blocker in the new version.
61+
- Support or team reports indicate data loss, missed alarms, broken sign-in, or
62+
privacy-sensitive behavior.
63+
64+
Rollback or supersede the release when pausing is not enough:
65+
66+
- If the issue is server-side and a backend fix can safely restore behavior,
67+
deploy the backend fix and keep the mobile rollout paused until metrics
68+
recover.
69+
- If the issue is client-side and affects already-updated users, prepare a new
70+
fixed build and keep rollout paused until it is approved.
71+
- If Play Console allows halting the staged rollout before wide exposure, halt
72+
it and document the affected percentage and build number.
73+
- If a policy issue caused rejection or removal risk, follow the rejection
74+
response playbook from #461 once available and do not resubmit until the facts,
75+
fix, and declarations are aligned.
76+
77+
## Monitoring Locations
78+
79+
- Play Console review status: check the release status, publishing overview, app
80+
content warnings, policy status, release notes, and tester or production track
81+
rollout percentage.
82+
- Play Console Android Vitals: monitor crash rate, user-perceived crash rate,
83+
ANR rate, excessive wakeups, battery, and device-specific clusters.
84+
- Play Store ratings and reviews: watch new public reviews, tester feedback, and
85+
low-rating comments that mention the current version or rollout date.
86+
- Policy email: monitor the Google Play developer account inbox and any team
87+
forwarding address for rejection, declaration, data safety, or policy notices.
88+
- Firebase console: check Cloud Messaging delivery health and any crash or event
89+
dashboard the project has enabled for `ontime-c63f1`.
90+
- GitHub Actions: confirm the release workflow artifact, generated build number,
91+
and deploy job match the build that reached Play Console.
92+
- Backend monitoring: check production API logs and dashboards for elevated
93+
status codes, latency, authentication failures, schedule/preparation failures,
94+
alarm status report failures, and FCM token registration failures.
95+
- Team channels and support inbox: watch for tester reports, screenshots, device
96+
details, and reproduction steps.
97+
98+
## First 24 Hours
99+
100+
- At submission: record version name, generated build number, track, rollout
101+
percentage, release owner, backup owner, and expected next check time.
102+
- Every 2 to 4 hours during local business hours: check Play review status,
103+
policy email, Android Vitals, Firebase health, backend errors, and team
104+
reports.
105+
- After approval or rollout start: verify installs from the active track and run
106+
a smoke test on a real Android device.
107+
- Before any percentage increase: compare crashes, ANRs, backend errors, and
108+
tester or review feedback against the pre-rollout baseline.
109+
- End of day: post a short status note with current rollout percentage, checked
110+
dashboards, issues found, and the next planned decision.
111+
112+
## First 7 Days
113+
114+
- Check Play Console, policy email, Android Vitals, reviews, Firebase, and
115+
backend dashboards at least once per business day.
116+
- Track whether crash and ANR clusters are new, increasing, or tied to a
117+
specific device, OS version, or app version.
118+
- Review low-rating feedback and tester reports for repeated symptoms before
119+
widening rollout.
120+
- Confirm backend errors and latency remain stable for schedule, preparation,
121+
alarm, auth, and FCM token endpoints.
122+
- Record each rollout increase with the reason it is acceptable and the signals
123+
checked.
124+
- After reaching full rollout, keep one final next-business-day monitoring pass
125+
before closing the release monitoring note.
126+
127+
## Release Monitoring Note Template
128+
129+
```md
130+
Release:
131+
Build:
132+
Track:
133+
Current rollout percentage:
134+
Release owner:
135+
Backup owner:
136+
Last checked:
137+
Next check:
138+
139+
Signals checked:
140+
- Play review/policy:
141+
- Android Vitals crashes/ANRs:
142+
- Ratings/reviews/tester feedback:
143+
- Firebase/FCM:
144+
- Backend errors/latency:
145+
- Team/support reports:
146+
147+
Decision:
148+
- Continue / pause / rollback / supersede
149+
150+
Notes:
151+
```

‎plans/issue_463_rollout_monitoring_checklist_plan.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Issue 463 Rollout Monitoring Checklist Plan
2+
3+
Parent track: #468
4+
Sub-issue: #463
5+
6+
## Scope
7+
8+
Create documentation for release ownership and rollout monitoring after Google
9+
Play submission. The document must define the primary owner, backup owner,
10+
rollout stages, pause and rollback criteria, monitoring locations, and the
11+
first 24-hour and first 7-day monitoring tasks.
12+
13+
## Files Likely Touched
14+
15+
- `docs/Release-Rollout-Monitoring.md`
16+
- `docs/Home.md`
17+
- `plans/issue_463_rollout_monitoring_checklist_plan.md`
18+
19+
## Implementation Approach
20+
21+
1. Keep the monitoring checklist separate from the reusable app release
22+
checklist because #462 is blocked on broader release-flow prerequisites.
23+
2. Use role placeholders instead of naming a specific person, because this
24+
thread does not have team assignment authority.
25+
3. Cover Play Console, Firebase/Play diagnostics, store reviews, policy email,
26+
and backend monitoring without requiring console access.
27+
4. Include concrete stage gates and criteria that a human release owner can
28+
apply during internal testing, closed testing, production staged rollout, and
29+
full rollout.
30+
31+
## Verification
32+
33+
- Review the Markdown for complete #463 acceptance criteria coverage.
34+
- Run a targeted text check for the new document and docs index entry.
35+
- Confirm `git diff` contains only #463-related documentation changes.
36+
37+
## Blockers
38+
39+
No implementation blockers. Human assignment is still required before using the
40+
checklist in a live release.
41+
42+
## Explicitly Left Out
43+
44+
- Reusable full release checklist completion for #462.
45+
- Play rejection response playbook content for #461.
46+
- Console actions, rollout changes, release uploads, or production monitoring
47+
execution.

0 commit comments

Comments
 (0)